Friday, 2016-11-18

*** thorst_ has quit IRC00:23
*** k0da has quit IRC00:27
*** thorst has joined #openstack-powervm00:30
*** thorst has quit IRC00:34
*** thorst has joined #openstack-powervm00:35
*** thorst has quit IRC00:43
*** seroyer has joined #openstack-powervm00:59
*** seroyer has quit IRC01:41
*** thorst has joined #openstack-powervm01:41
*** seroyer has joined #openstack-powervm01:43
*** thorst has quit IRC01:45
*** esberglu has joined #openstack-powervm02:19
*** esberglu has quit IRC02:23
*** thorst has joined #openstack-powervm02:33
*** thorst has quit IRC02:35
*** thorst has joined #openstack-powervm02:36
*** thorst has quit IRC02:44
*** esberglu has joined #openstack-powervm02:52
*** apearson has joined #openstack-powervm02:53
*** esberglu has quit IRC02:56
*** thorst has joined #openstack-powervm02:56
*** esberglu has joined #openstack-powervm02:56
*** thorst has quit IRC02:59
*** esberglu has quit IRC03:01
*** seroyer has quit IRC03:08
*** kotra03 has joined #openstack-powervm03:35
*** thorst has joined #openstack-powervm04:00
*** thorst has quit IRC04:08
openstackgerritCao Xuan Hoang proposed openstack/ceilometer-powervm: Changed author and author-email  https://review.openstack.org/39937204:18
*** arunman has joined #openstack-powervm04:33
*** thorst has joined #openstack-powervm05:06
*** tlian has quit IRC05:12
*** thorst has quit IRC05:14
*** apearson has quit IRC05:15
*** thorst has joined #openstack-powervm06:13
*** thorst has quit IRC06:19
*** esberglu has joined #openstack-powervm06:31
*** esberglu has quit IRC06:36
*** thorst has joined #openstack-powervm07:16
*** thorst has quit IRC07:23
*** esberglu has joined #openstack-powervm07:26
*** esberglu has quit IRC07:31
*** esberglu has joined #openstack-powervm07:31
*** esberglu has quit IRC07:35
*** openstackgerrit has quit IRC08:03
*** openstackgerrit has joined #openstack-powervm08:03
*** thorst has joined #openstack-powervm08:21
*** esberglu has joined #openstack-powervm08:21
*** esberglu has quit IRC08:26
*** thorst has quit IRC08:28
*** thorst has joined #openstack-powervm09:26
*** thorst has quit IRC09:34
*** AlexeyAbashkin has joined #openstack-powervm09:53
*** AlexeyAbashkin has quit IRC10:16
*** kotra03_ has joined #openstack-powervm10:22
*** kotra03 has quit IRC10:24
*** kotra03_ is now known as kotra0310:24
*** k0da has joined #openstack-powervm10:27
*** thorst has joined #openstack-powervm10:33
*** tblakes has joined #openstack-powervm10:39
*** thorst has quit IRC10:39
*** tblakes has quit IRC11:34
*** thorst has joined #openstack-powervm11:36
*** tblakes has joined #openstack-powervm11:36
*** thorst has quit IRC11:44
*** smatzek has joined #openstack-powervm11:46
*** smatzek has quit IRC11:47
*** smatzek has joined #openstack-powervm11:47
*** tblakes has quit IRC11:56
*** esberglu has joined #openstack-powervm12:01
*** esberglu has quit IRC12:05
*** esberglu has joined #openstack-powervm12:06
*** esberglu has quit IRC12:10
*** seroyer has joined #openstack-powervm12:31
openstackgerritManas Mandlekar proposed openstack/nova-powervm: Allow dynamic enable/disable of SRR capability  https://review.openstack.org/39957912:36
*** esberglu has joined #openstack-powervm12:56
*** tblakes_ has joined #openstack-powervm12:59
*** thorst has joined #openstack-powervm12:59
*** thorst_ has joined #openstack-powervm13:00
*** esberglu has quit IRC13:00
*** svenkat has joined #openstack-powervm13:00
*** tblakes_ is now known as tblakes13:03
*** thorst has quit IRC13:04
openstackgerritSridhar Venkat proposed openstack/networking-powervm: ProvisionRequest does not distinguish event source  https://review.openstack.org/39646713:12
openstackgerritSridhar Venkat proposed openstack/networking-powervm: ProvisionRequest does not distinguish event source  https://review.openstack.org/39646713:12
*** edmondsw has joined #openstack-powervm13:16
*** seroyer has quit IRC13:17
openstackgerritSridhar Venkat proposed openstack/networking-powervm: ProvisionRequest does not distinguish event source  https://review.openstack.org/39646713:31
*** tblakes has quit IRC13:32
*** esberglu has joined #openstack-powervm13:34
*** esberglu has quit IRC13:34
*** esberglu has joined #openstack-powervm13:34
*** tblakes has joined #openstack-powervm13:35
*** mdrabe has joined #openstack-powervm13:40
*** tlian has joined #openstack-powervm13:48
*** seroyer has joined #openstack-powervm13:48
*** apearson has joined #openstack-powervm13:49
*** seroyer has quit IRC14:01
*** seroyer has joined #openstack-powervm14:07
*** tblakes has quit IRC14:08
*** seroyer has quit IRC14:18
*** tjakobs_ has joined #openstack-powervm14:26
*** esberglu has quit IRC14:31
*** kriskend has joined #openstack-powervm14:56
*** tjakobs_ has quit IRC15:28
*** dwayne has quit IRC15:38
*** apearson has quit IRC15:48
*** k0da has quit IRC15:52
*** tblakes has joined #openstack-powervm15:59
*** apearson has joined #openstack-powervm16:01
*** seroyer has joined #openstack-powervm16:03
*** tjakobs_ has joined #openstack-powervm16:17
*** dwayne has joined #openstack-powervm16:55
*** svenkat has quit IRC16:59
*** dwayne has quit IRC17:06
thorst_efried: got a min to explain something to me/17:10
efriedHit me.17:11
thorst_in talking with svenkat, he pointed me to how the new updates you put in work for heal_and_optimize17:11
thorst_its no longer on an interval17:11
efrieduh17:11
thorst_but rather works off of new events, like a CACHE_CLEAR17:11
thorst_(and this isn't all that new)17:11
efriedIt's also on an interval.17:11
thorst_ahh, I see now.17:13
efriedDoes that fully address your concern?17:13
thorst_kinda....17:13
efriedWas there something else you wanted to discuss related to MGMTSWITCH and how it's going to be treated as a CNA?17:13
thorst_I'm wondering if the heal_and_optimize dumps things out about all the CNAs on the MGMTSWITCH that don't have a network bridge17:14
thorst_sea_agent.py line 26917:14
efriedI didn't change that code path, so it oughtta be doing  whatever it was doing before.17:15
thorst_yeah17:15
thorst_agree.17:15
thorst_I'm wondering if it was just bad before17:15
thorst_I'll ask svenkat.17:15
efried...except as we now produce the list of prov_reqs differently, so maybe.17:16
efried...except again that before, we were running on every dev that came to us.17:16
efriedNow we're at least not running on the VNICs.17:16
thorst_right17:17
efriedthorst_, so your concern here is whether we're constantly emitting a warning message about management CNAs?17:18
efriedNo actual harm, just an extraneous message?17:19
*** kotra03 has quit IRC17:20
openstackgerritTaylor Jakobson proposed openstack/nova-powervm: Fix iscsi attach issues  https://review.openstack.org/39971117:37
efriedthorst_, any idea where horizon settings live?  I can't connect to esberglu's CI nodes he set up yesterday.  Will use CLI if necessary, but...17:38
efried(adreznec ^^)17:38
*** dwayne has joined #openstack-powervm17:46
efriedthorst_ adreznec: reproduced that hang, first time around, on .3217:51
efriedIdea about adding a yield isn't going to fly, cause we don't have control over the thread that's doing the blocking (it's the external glance FUNC).17:52
efriedIma try throwing that func in a subthread and see if that does the trick.17:53
efried'cept I'm not totally sure how to restart the services here, without screen.17:53
*** apearson has quit IRC18:15
*** apearson has joined #openstack-powervm18:17
efriedthorst_ adreznec: screen is there; just gotta be jenkins to use it, which means logging in as jenkins directly so it can get to your tty.  I think our CI must set up keys, cause when I tried to ssh as jenkins, I popped right in.18:25
thorst_neat18:25
efriedShoved the _copy_retry in a subthread, and still getting the hang.  I'm kinda stymied here - what else to try?18:25
thorst_so its definitely hanging there?18:28
efriedOh, fo sho.18:29
efriedHere's a thing.18:29
efriedI checked the REST server, and it doesn't look like the upload request is in the Audit.log.18:29
efriedBut... actually that makes sense, cause the Audit.log entry doesn't get created until the request completes.18:29
efriedSo that tells us nothing.18:30
efriedthorst_, by the way, in the previous run, I tried deleting the pipe to see if it unwedged the compute process.  It didn't.18:32
efriedThe good news is, this is 100% reproducible.18:32
efriedDid we ever have a version of this that worked?18:33
thorst_not deleting it18:33
efriedno, I mean a version of the rest_api_pipe18:33
efriedOh, you're saying deleting it wouldn't unwedge it?  I should do a write to it, maybe?18:33
thorst_just echo something to it18:34
thorst_the echo will issue an EOF18:34
thorst_and then it'll go through18:34
thorst_because its a pipe, so it needs an EOF18:34
*** arunman has quit IRC18:35
efriedOkay, so this is interesting too.18:36
efriedWhere the pipe I deleted used to live (I hadn't removed the directory) there's now a regular file named REST_API_PIPE, filled with nulls (presumably contents of the image).18:37
efriedIt's stable at 917504 bytes.18:37
efried896KB18:37
thorst_yeah, because the fifo pipe died18:37
thorst_so the thread freed up18:37
thorst_weird...something was writing to the pipe?18:37
efriedWell, it tells us something interesting about the writer func18:37
efriedYeah, clearly the writer func stayed alive long enough to Back in a tick.18:38
efriedsudo shutdown -r now18:38
thorst_I don't wanna shut down now18:39
*** efried1 has joined #openstack-powervm18:40
efried1Okay, last thing I saw was: "(12:37:51 PM) thorst_: weird...something was writing to the pipe?"18:41
openstackgerritArun Mani proposed openstack/nova-powervm: Object upload using swift occasionally fails with 'TypeError' exception.  https://review.openstack.org/39886818:41
efried1So yeah; clearly the writer func reacts to the destination disappearing by recreating it somehow.18:41
*** svenkat has joined #openstack-powervm18:42
thorst_yeah...18:42
*** efried has quit IRC18:42
thorst_I say we should add some logs that tell us when we're writing18:43
efried1Problem is, we're not writing.18:43
efried1It's this external glance func, no?18:43
thorst_true.18:43
efried1So18:44
thorst_if its a held up system18:44
thorst_we could go find the code that does it18:44
efried1Couple ways that guy could be writing that would yield this behavior.18:44
efried1First is something akin to18:44
efried1while stuff_left; do18:44
efried1    echo chunk > file18:44
efried1done18:44
efried1correction: >>18:44
efried1Which would send an open/close with every chunk.18:45
efried1Second is, something went wrong, and the retry reissues the open18:45
thorst_well, we don't want to open/close each chunk18:45
efried1WE don't.  We don't have control.18:45
efried1Although it really shouldn't matter.18:45
efried1That >> loop should still work on a FIFO.18:46
thorst_gotcha18:46
thorst_is the thing writing with shutil?18:46
efried1No idea.  Time to go find out?18:46
thorst_yeah.  shutil can't copy to a pipe, but it'd raise an error18:47
efried1We could also involve REST and see if they can tell whether anything's available on the other end of the pipe.18:47
thorst_I seem to remember though I already did this18:47
thorst_and they don't do that18:47
efried1thorst_, anything you want to try while that process is still hung?18:48
efried1Echo something to the pipe?18:48
thorst_yeah18:48
efried1stand by.18:48
efried1Yup, that unblocked it.18:49
efried1and freaked out the compute process.18:49
efried1HttpError: HTTP error 400 for method PUT on path /rest/api/web/File/contents/39fb6a89-c9ef-4122-9974-7d01c9c8e7b5: Bad Request -- REST0036 Content of payload exceeded that specified in the Content-Length. Only 0 bytes were written.18:49
efried1And cleaned everything up properly ;-)18:50
efried1Okay, so it's not actually the writer that's wedging the compute process.18:51
efried1And here's my reasoning:18:51
efried1In the first run, when I deleted the pipe, nothing happened.  Notably, the pipe did not reappear yet.18:51
efried1It was when I killed the compute process that the pipe reappeared (as a regular file) and got stuff written to it.18:52
efried1So the writer was blocked behind the compute process.18:52
efried1Hm, I kinda wonder how much time the writer had to do stuff while the interrupt was being handled.  I used ^C.18:52
efried1But anyway, I'm not completely convinced the writer thread is happening within the compute process.18:53
thorst_I think its that.18:53
efried1It could very well be spawning another process.18:53
efried1which?18:53
thorst_I doubt its another process18:53
thorst_I think when you ^C'd, it got about 860 KB through it18:53
efried1Well, I can repro and kill -9 the process.18:54
efried1Pretty sure that breaks the pipe without a flush.18:54
efried1Easy to try...18:54
thorst_yep18:55
efried1oo, watching the logs, noticed an opportunity.18:59
efried1We could watch events for creation/deletion of LUs.18:59
efried1right now we're polling.18:59
efried1Oh, guess that only helps us for the local REST server.  Never mind.19:00
thorst_events come in to all subscribers19:01
thorst_not just local19:01
efried1But only from that REST server.19:02
efried1Remember, this SSP is shared among multiple REST servers.19:02
efried1So we would still have to poll.19:02
efried1Confirmed: when I used kill -9, the pipe file doesn't get created.  So likely the writer thread is indeed running under the compute process.19:02
efried1That's assuming the behavior is consistently reproducible.19:03
*** apearson has quit IRC19:06
*** apearson has joined #openstack-powervm19:10
thorst_efried1: I think one of the differencs between mine and yours is that you have the writer in the thread and then the reader in the main line code path19:16
thorst_and the reader is the thing getting hung.19:16
efried1thorst_, if you look on .32, I put both in subthreads.19:16
thorst_let me meander over there19:17
thorst_still not working though?19:17
*** k0da has joined #openstack-powervm19:18
thorst_are you sure that's running in a separate thread?19:18
thorst_doesn't the _init_ get called before it passes it into the thread19:19
efried1Are you looking on .32?19:21
thorst_yeah19:21
thorst_I'm assuming what you meant by 'I put it in a thread' is on line 39519:21
efried1yup19:21
efried1It's the REST API upload request that's going in a new thread now19:22
thorst_but you'r not in the thread until line 39619:22
thorst_and your init gets called at line 39519:22
efried1The only thing happening there is the open()19:22
efried1well, creation of the fifo, open, spawning of the writer thread...19:23
thorst_the open is where it hangs.19:23
efried1That's... weird.19:24
thorst_at least in my testing19:24
thorst_that was where it hung19:24
efried1But okay, let me work on it some more.19:24
thorst_but it had been OK generally because something was writing to it already19:24
efried1Right19:24
efried1That should be the case here19:24
efried1since the writer thread is already running when we do the open.19:24
thorst_agree...but pipo's are weirdo's19:24
thorst_I'd say put a pdb in there19:24
thorst_verify my assertion19:25
efried1roger that.19:25
efried1thorst_, I put a breakpoint right before the open - and never hit it.19:35
efried1i.e. hang happens before that.19:37
efried1which is super bizarre.19:37
efried1Because the FIFO is getting created.19:37
thorst_lol19:37
thorst_the plot thickens19:37
efried1And the only other thing in there is the spawning of the writer thread.  Which should be nonblocking.19:38
efried1Unless the thread pool is full.19:38
efried1One has to assume we're allowed more than one ThreadPoolExecutor per process.19:38
efried1Kinda silly otherwise, no?19:38
efried1Or maybe mkfifo isn't returning.19:39
efried1I'll put the pdb earlier and see which bit is actually hanging.19:40
efried1Here's another interesting point.19:42
efried1I can't ^C the compute process until I've deleted the REST_API_PIPE19:42
thorst_heh19:42
efried1"blocking on I/O" is one of the classic things that prevents SIGINT from going through.19:42
efried1This time the writer got off 4980736 bytes.19:44
efried14.75MB19:45
*** tblakes has quit IRC19:45
efried1If we're writing in 64K blocks, that's 76, compared to 14 previously.19:45
thorst_I think async was a no no for pipes19:46
efried1huh?19:46
efried1What else are pipes for?19:46
thorst_sorry - non-blocking19:46
efried1non-blocking which?19:47
thorst_write19:47
thorst_let me try to find where I saw that19:47
efried1oooooooohkay, I'm not hitting THAT breakpoint either.19:48
efried1which kinda leads me to believe I've been working on the wrong pypowervm this whole time.19:48
thorst_lol19:48
thorst_perhaps!19:48
efried1It had all the right patches in it.19:48
thorst_and maybe that's why CI was borked19:48
efried1oo19:49
thorst_so how I check that...19:49
efried1The dist-packages version also has the local2remote patch.19:50
thorst_you have19:50
thorst_<module 'pypowervm.adapter' from '/usr/local/lib/python2.7/dist-packages/pypowervm/adapter.pyc'>19:50
thorst_lol19:50
thorst_alright, so I think we found out why we were dying.19:51
thorst_fantastic.19:51
efried1Well, hold da phone.19:51
adreznec*record scratch*19:51
efried1The dist-packages one has the rest_api_pipe and _wrap_user_stream patches too.19:51
thorst_so put the pdb's in there and see if it hangs on the open19:52
efried1Well, maybe I start by putting my new subthread in there.19:52
thorst_wouldn't we want to debug where the freeze happens first?19:54
thorst_my vote is still on the open19:54
thorst_and I bet the fix is that we open the read for non block.19:54
thorst_and in fact, we probably open the read for non block before submitting the file writer.19:55
*** apearson has quit IRC19:56
efried1The hang happens even with the new subthread.  So okay, I'll pdb and see if it's the open, then try the open nonblocking.19:58
efried1Does that mean attempts to read from that handle will block until the open actually goes through?19:59
efried1Not sure how that really helps us, but will try.20:00
*** apearson has joined #openstack-powervm20:01
thorst_actually, not sure20:02
thorst_reading the python docs20:02
efried1thorst_, yup, it's the open.20:07
efried1thorst_, how do you specify the open as nonblocking?20:11
adreznecefried1: FYI Drew may be slow to respond, he's presenting right now20:12
efried1thx20:12
*** svenkat has quit IRC20:29
*** apearson has quit IRC20:34
*** seroyer has quit IRC20:41
thorst_efried1: sorry...yeah20:44
thorst_I looked into it20:44
thorst_and I have no clue20:44
thorst_surprise!20:44
efried1thorst_, I tried adding 'b' and a bufsize to the open, but neither had an effect (which doesn't really surprise me).20:56
efried1I'm going to isolate this fifo business in ipython and see if I can nail down the behavior.20:56
thorst_efried1: oK20:58
efried1reckon the 'b' should be there regardless.20:58
thorst_efried1: based on what I read, the b only matters for windows20:59
thorst_and I'm not ready to jump to 'pypowervm works on windows!'21:00
efried1Yeah, but "good for documentation"21:00
*** apearson has joined #openstack-powervm21:00
efried1thorst_ I think I get it.21:08
efried1http://stackoverflow.com/questions/39233663/nonblocking-fifo21:08
efried1First of all, have to use os.open so we can use the O_NONBLOCK flag.21:08
efried1Second, the writer has to wait until the reader opens, apparently.  Hopefully our writer is doing that.21:08
efried1I'll play around with this.21:09
*** seroyer has joined #openstack-powervm21:11
*** kriskend has quit IRC21:16
*** edmondsw has quit IRC21:43
thorst_efried1: neat21:43
*** apearson has quit IRC21:45
*** dwayne has quit IRC21:45
*** thorst_ has quit IRC21:46
*** apearson has joined #openstack-powervm21:58
*** mdrabe has quit IRC22:00
*** k0da has quit IRC22:30
*** smatzek has quit IRC22:47
*** tjakobs_ has quit IRC23:18
*** tlian has quit IRC23:19
*** tlian has joined #openstack-powervm23:26
*** apearson has quit IRC23:28
*** seroyer has quit IRC23:48
*** tlian has quit IRC23:54
*** tlian has joined #openstack-powervm23:59

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!