*** thorst_ has quit IRC | 00:23 | |
*** k0da has quit IRC | 00:27 | |
*** thorst has joined #openstack-powervm | 00:30 | |
*** thorst has quit IRC | 00:34 | |
*** thorst has joined #openstack-powervm | 00:35 | |
*** thorst has quit IRC | 00:43 | |
*** seroyer has joined #openstack-powervm | 00:59 | |
*** seroyer has quit IRC | 01:41 | |
*** thorst has joined #openstack-powervm | 01:41 | |
*** seroyer has joined #openstack-powervm | 01:43 | |
*** thorst has quit IRC | 01:45 | |
*** esberglu has joined #openstack-powervm | 02:19 | |
*** esberglu has quit IRC | 02:23 | |
*** thorst has joined #openstack-powervm | 02:33 | |
*** thorst has quit IRC | 02:35 | |
*** thorst has joined #openstack-powervm | 02:36 | |
*** thorst has quit IRC | 02:44 | |
*** esberglu has joined #openstack-powervm | 02:52 | |
*** apearson has joined #openstack-powervm | 02:53 | |
*** esberglu has quit IRC | 02:56 | |
*** thorst has joined #openstack-powervm | 02:56 | |
*** esberglu has joined #openstack-powervm | 02:56 | |
*** thorst has quit IRC | 02:59 | |
*** esberglu has quit IRC | 03:01 | |
*** seroyer has quit IRC | 03:08 | |
*** kotra03 has joined #openstack-powervm | 03:35 | |
*** thorst has joined #openstack-powervm | 04:00 | |
*** thorst has quit IRC | 04:08 | |
openstackgerrit | Cao Xuan Hoang proposed openstack/ceilometer-powervm: Changed author and author-email https://review.openstack.org/399372 | 04:18 |
---|---|---|
*** arunman has joined #openstack-powervm | 04:33 | |
*** thorst has joined #openstack-powervm | 05:06 | |
*** tlian has quit IRC | 05:12 | |
*** thorst has quit IRC | 05:14 | |
*** apearson has quit IRC | 05:15 | |
*** thorst has joined #openstack-powervm | 06:13 | |
*** thorst has quit IRC | 06:19 | |
*** esberglu has joined #openstack-powervm | 06:31 | |
*** esberglu has quit IRC | 06:36 | |
*** thorst has joined #openstack-powervm | 07:16 | |
*** thorst has quit IRC | 07:23 | |
*** esberglu has joined #openstack-powervm | 07:26 | |
*** esberglu has quit IRC | 07:31 | |
*** esberglu has joined #openstack-powervm | 07:31 | |
*** esberglu has quit IRC | 07:35 | |
*** openstackgerrit has quit IRC | 08:03 | |
*** openstackgerrit has joined #openstack-powervm | 08:03 | |
*** thorst has joined #openstack-powervm | 08:21 | |
*** esberglu has joined #openstack-powervm | 08:21 | |
*** esberglu has quit IRC | 08:26 | |
*** thorst has quit IRC | 08:28 | |
*** thorst has joined #openstack-powervm | 09:26 | |
*** thorst has quit IRC | 09:34 | |
*** AlexeyAbashkin has joined #openstack-powervm | 09:53 | |
*** AlexeyAbashkin has quit IRC | 10:16 | |
*** kotra03_ has joined #openstack-powervm | 10:22 | |
*** kotra03 has quit IRC | 10:24 | |
*** kotra03_ is now known as kotra03 | 10:24 | |
*** k0da has joined #openstack-powervm | 10:27 | |
*** thorst has joined #openstack-powervm | 10:33 | |
*** tblakes has joined #openstack-powervm | 10:39 | |
*** thorst has quit IRC | 10:39 | |
*** tblakes has quit IRC | 11:34 | |
*** thorst has joined #openstack-powervm | 11:36 | |
*** tblakes has joined #openstack-powervm | 11:36 | |
*** thorst has quit IRC | 11:44 | |
*** smatzek has joined #openstack-powervm | 11:46 | |
*** smatzek has quit IRC | 11:47 | |
*** smatzek has joined #openstack-powervm | 11:47 | |
*** tblakes has quit IRC | 11:56 | |
*** esberglu has joined #openstack-powervm | 12:01 | |
*** esberglu has quit IRC | 12:05 | |
*** esberglu has joined #openstack-powervm | 12:06 | |
*** esberglu has quit IRC | 12:10 | |
*** seroyer has joined #openstack-powervm | 12:31 | |
openstackgerrit | Manas Mandlekar proposed openstack/nova-powervm: Allow dynamic enable/disable of SRR capability https://review.openstack.org/399579 | 12:36 |
*** esberglu has joined #openstack-powervm | 12:56 | |
*** tblakes_ has joined #openstack-powervm | 12:59 | |
*** thorst has joined #openstack-powervm | 12:59 | |
*** thorst_ has joined #openstack-powervm | 13:00 | |
*** esberglu has quit IRC | 13:00 | |
*** svenkat has joined #openstack-powervm | 13:00 | |
*** tblakes_ is now known as tblakes | 13:03 | |
*** thorst has quit IRC | 13:04 | |
openstackgerrit | Sridhar Venkat proposed openstack/networking-powervm: ProvisionRequest does not distinguish event source https://review.openstack.org/396467 | 13:12 |
openstackgerrit | Sridhar Venkat proposed openstack/networking-powervm: ProvisionRequest does not distinguish event source https://review.openstack.org/396467 | 13:12 |
*** edmondsw has joined #openstack-powervm | 13:16 | |
*** seroyer has quit IRC | 13:17 | |
openstackgerrit | Sridhar Venkat proposed openstack/networking-powervm: ProvisionRequest does not distinguish event source https://review.openstack.org/396467 | 13:31 |
*** tblakes has quit IRC | 13:32 | |
*** esberglu has joined #openstack-powervm | 13:34 | |
*** esberglu has quit IRC | 13:34 | |
*** esberglu has joined #openstack-powervm | 13:34 | |
*** tblakes has joined #openstack-powervm | 13:35 | |
*** mdrabe has joined #openstack-powervm | 13:40 | |
*** tlian has joined #openstack-powervm | 13:48 | |
*** seroyer has joined #openstack-powervm | 13:48 | |
*** apearson has joined #openstack-powervm | 13:49 | |
*** seroyer has quit IRC | 14:01 | |
*** seroyer has joined #openstack-powervm | 14:07 | |
*** tblakes has quit IRC | 14:08 | |
*** seroyer has quit IRC | 14:18 | |
*** tjakobs_ has joined #openstack-powervm | 14:26 | |
*** esberglu has quit IRC | 14:31 | |
*** kriskend has joined #openstack-powervm | 14:56 | |
*** tjakobs_ has quit IRC | 15:28 | |
*** dwayne has quit IRC | 15:38 | |
*** apearson has quit IRC | 15:48 | |
*** k0da has quit IRC | 15:52 | |
*** tblakes has joined #openstack-powervm | 15:59 | |
*** apearson has joined #openstack-powervm | 16:01 | |
*** seroyer has joined #openstack-powervm | 16:03 | |
*** tjakobs_ has joined #openstack-powervm | 16:17 | |
*** dwayne has joined #openstack-powervm | 16:55 | |
*** svenkat has quit IRC | 16:59 | |
*** dwayne has quit IRC | 17:06 | |
thorst_ | efried: got a min to explain something to me/ | 17:10 |
efried | Hit me. | 17:11 |
thorst_ | in talking with svenkat, he pointed me to how the new updates you put in work for heal_and_optimize | 17:11 |
thorst_ | its no longer on an interval | 17:11 |
efried | uh | 17:11 |
thorst_ | but rather works off of new events, like a CACHE_CLEAR | 17:11 |
thorst_ | (and this isn't all that new) | 17:11 |
efried | It's also on an interval. | 17:11 |
thorst_ | ahh, I see now. | 17:13 |
efried | Does that fully address your concern? | 17:13 |
thorst_ | kinda.... | 17:13 |
efried | Was there something else you wanted to discuss related to MGMTSWITCH and how it's going to be treated as a CNA? | 17:13 |
thorst_ | I'm wondering if the heal_and_optimize dumps things out about all the CNAs on the MGMTSWITCH that don't have a network bridge | 17:14 |
thorst_ | sea_agent.py line 269 | 17:14 |
efried | I didn't change that code path, so it oughtta be doing whatever it was doing before. | 17:15 |
thorst_ | yeah | 17:15 |
thorst_ | agree. | 17:15 |
thorst_ | I'm wondering if it was just bad before | 17:15 |
thorst_ | I'll ask svenkat. | 17:15 |
efried | ...except as we now produce the list of prov_reqs differently, so maybe. | 17:16 |
efried | ...except again that before, we were running on every dev that came to us. | 17:16 |
efried | Now we're at least not running on the VNICs. | 17:16 |
thorst_ | right | 17:17 |
efried | thorst_, so your concern here is whether we're constantly emitting a warning message about management CNAs? | 17:18 |
efried | No actual harm, just an extraneous message? | 17:19 |
*** kotra03 has quit IRC | 17:20 | |
openstackgerrit | Taylor Jakobson proposed openstack/nova-powervm: Fix iscsi attach issues https://review.openstack.org/399711 | 17:37 |
efried | thorst_, any idea where horizon settings live? I can't connect to esberglu's CI nodes he set up yesterday. Will use CLI if necessary, but... | 17:38 |
efried | (adreznec ^^) | 17:38 |
*** dwayne has joined #openstack-powervm | 17:46 | |
efried | thorst_ adreznec: reproduced that hang, first time around, on .32 | 17:51 |
efried | Idea about adding a yield isn't going to fly, cause we don't have control over the thread that's doing the blocking (it's the external glance FUNC). | 17:52 |
efried | Ima try throwing that func in a subthread and see if that does the trick. | 17:53 |
efried | 'cept I'm not totally sure how to restart the services here, without screen. | 17:53 |
*** apearson has quit IRC | 18:15 | |
*** apearson has joined #openstack-powervm | 18:17 | |
efried | thorst_ adreznec: screen is there; just gotta be jenkins to use it, which means logging in as jenkins directly so it can get to your tty. I think our CI must set up keys, cause when I tried to ssh as jenkins, I popped right in. | 18:25 |
thorst_ | neat | 18:25 |
efried | Shoved the _copy_retry in a subthread, and still getting the hang. I'm kinda stymied here - what else to try? | 18:25 |
thorst_ | so its definitely hanging there? | 18:28 |
efried | Oh, fo sho. | 18:29 |
efried | Here's a thing. | 18:29 |
efried | I checked the REST server, and it doesn't look like the upload request is in the Audit.log. | 18:29 |
efried | But... actually that makes sense, cause the Audit.log entry doesn't get created until the request completes. | 18:29 |
efried | So that tells us nothing. | 18:30 |
efried | thorst_, by the way, in the previous run, I tried deleting the pipe to see if it unwedged the compute process. It didn't. | 18:32 |
efried | The good news is, this is 100% reproducible. | 18:32 |
efried | Did we ever have a version of this that worked? | 18:33 |
thorst_ | not deleting it | 18:33 |
efried | no, I mean a version of the rest_api_pipe | 18:33 |
efried | Oh, you're saying deleting it wouldn't unwedge it? I should do a write to it, maybe? | 18:33 |
thorst_ | just echo something to it | 18:34 |
thorst_ | the echo will issue an EOF | 18:34 |
thorst_ | and then it'll go through | 18:34 |
thorst_ | because its a pipe, so it needs an EOF | 18:34 |
*** arunman has quit IRC | 18:35 | |
efried | Okay, so this is interesting too. | 18:36 |
efried | Where the pipe I deleted used to live (I hadn't removed the directory) there's now a regular file named REST_API_PIPE, filled with nulls (presumably contents of the image). | 18:37 |
efried | It's stable at 917504 bytes. | 18:37 |
efried | 896KB | 18:37 |
thorst_ | yeah, because the fifo pipe died | 18:37 |
thorst_ | so the thread freed up | 18:37 |
thorst_ | weird...something was writing to the pipe? | 18:37 |
efried | Well, it tells us something interesting about the writer func | 18:37 |
efried | Yeah, clearly the writer func stayed alive long enough to Back in a tick. | 18:38 |
efried | sudo shutdown -r now | 18:38 |
thorst_ | I don't wanna shut down now | 18:39 |
*** efried1 has joined #openstack-powervm | 18:40 | |
efried1 | Okay, last thing I saw was: "(12:37:51 PM) thorst_: weird...something was writing to the pipe?" | 18:41 |
openstackgerrit | Arun Mani proposed openstack/nova-powervm: Object upload using swift occasionally fails with 'TypeError' exception. https://review.openstack.org/398868 | 18:41 |
efried1 | So yeah; clearly the writer func reacts to the destination disappearing by recreating it somehow. | 18:41 |
*** svenkat has joined #openstack-powervm | 18:42 | |
thorst_ | yeah... | 18:42 |
*** efried has quit IRC | 18:42 | |
thorst_ | I say we should add some logs that tell us when we're writing | 18:43 |
efried1 | Problem is, we're not writing. | 18:43 |
efried1 | It's this external glance func, no? | 18:43 |
thorst_ | true. | 18:43 |
efried1 | So | 18:44 |
thorst_ | if its a held up system | 18:44 |
thorst_ | we could go find the code that does it | 18:44 |
efried1 | Couple ways that guy could be writing that would yield this behavior. | 18:44 |
efried1 | First is something akin to | 18:44 |
efried1 | while stuff_left; do | 18:44 |
efried1 | echo chunk > file | 18:44 |
efried1 | done | 18:44 |
efried1 | correction: >> | 18:44 |
efried1 | Which would send an open/close with every chunk. | 18:45 |
efried1 | Second is, something went wrong, and the retry reissues the open | 18:45 |
thorst_ | well, we don't want to open/close each chunk | 18:45 |
efried1 | WE don't. We don't have control. | 18:45 |
efried1 | Although it really shouldn't matter. | 18:45 |
efried1 | That >> loop should still work on a FIFO. | 18:46 |
thorst_ | gotcha | 18:46 |
thorst_ | is the thing writing with shutil? | 18:46 |
efried1 | No idea. Time to go find out? | 18:46 |
thorst_ | yeah. shutil can't copy to a pipe, but it'd raise an error | 18:47 |
efried1 | We could also involve REST and see if they can tell whether anything's available on the other end of the pipe. | 18:47 |
thorst_ | I seem to remember though I already did this | 18:47 |
thorst_ | and they don't do that | 18:47 |
efried1 | thorst_, anything you want to try while that process is still hung? | 18:48 |
efried1 | Echo something to the pipe? | 18:48 |
thorst_ | yeah | 18:48 |
efried1 | stand by. | 18:48 |
efried1 | Yup, that unblocked it. | 18:49 |
efried1 | and freaked out the compute process. | 18:49 |
efried1 | HttpError: HTTP error 400 for method PUT on path /rest/api/web/File/contents/39fb6a89-c9ef-4122-9974-7d01c9c8e7b5: Bad Request -- REST0036 Content of payload exceeded that specified in the Content-Length. Only 0 bytes were written. | 18:49 |
efried1 | And cleaned everything up properly ;-) | 18:50 |
efried1 | Okay, so it's not actually the writer that's wedging the compute process. | 18:51 |
efried1 | And here's my reasoning: | 18:51 |
efried1 | In the first run, when I deleted the pipe, nothing happened. Notably, the pipe did not reappear yet. | 18:51 |
efried1 | It was when I killed the compute process that the pipe reappeared (as a regular file) and got stuff written to it. | 18:52 |
efried1 | So the writer was blocked behind the compute process. | 18:52 |
efried1 | Hm, I kinda wonder how much time the writer had to do stuff while the interrupt was being handled. I used ^C. | 18:52 |
efried1 | But anyway, I'm not completely convinced the writer thread is happening within the compute process. | 18:53 |
thorst_ | I think its that. | 18:53 |
efried1 | It could very well be spawning another process. | 18:53 |
efried1 | which? | 18:53 |
thorst_ | I doubt its another process | 18:53 |
thorst_ | I think when you ^C'd, it got about 860 KB through it | 18:53 |
efried1 | Well, I can repro and kill -9 the process. | 18:54 |
efried1 | Pretty sure that breaks the pipe without a flush. | 18:54 |
efried1 | Easy to try... | 18:54 |
thorst_ | yep | 18:55 |
efried1 | oo, watching the logs, noticed an opportunity. | 18:59 |
efried1 | We could watch events for creation/deletion of LUs. | 18:59 |
efried1 | right now we're polling. | 18:59 |
efried1 | Oh, guess that only helps us for the local REST server. Never mind. | 19:00 |
thorst_ | events come in to all subscribers | 19:01 |
thorst_ | not just local | 19:01 |
efried1 | But only from that REST server. | 19:02 |
efried1 | Remember, this SSP is shared among multiple REST servers. | 19:02 |
efried1 | So we would still have to poll. | 19:02 |
efried1 | Confirmed: when I used kill -9, the pipe file doesn't get created. So likely the writer thread is indeed running under the compute process. | 19:02 |
efried1 | That's assuming the behavior is consistently reproducible. | 19:03 |
*** apearson has quit IRC | 19:06 | |
*** apearson has joined #openstack-powervm | 19:10 | |
thorst_ | efried1: I think one of the differencs between mine and yours is that you have the writer in the thread and then the reader in the main line code path | 19:16 |
thorst_ | and the reader is the thing getting hung. | 19:16 |
efried1 | thorst_, if you look on .32, I put both in subthreads. | 19:16 |
thorst_ | let me meander over there | 19:17 |
thorst_ | still not working though? | 19:17 |
*** k0da has joined #openstack-powervm | 19:18 | |
thorst_ | are you sure that's running in a separate thread? | 19:18 |
thorst_ | doesn't the _init_ get called before it passes it into the thread | 19:19 |
efried1 | Are you looking on .32? | 19:21 |
thorst_ | yeah | 19:21 |
thorst_ | I'm assuming what you meant by 'I put it in a thread' is on line 395 | 19:21 |
efried1 | yup | 19:21 |
efried1 | It's the REST API upload request that's going in a new thread now | 19:22 |
thorst_ | but you'r not in the thread until line 396 | 19:22 |
thorst_ | and your init gets called at line 395 | 19:22 |
efried1 | The only thing happening there is the open() | 19:22 |
efried1 | well, creation of the fifo, open, spawning of the writer thread... | 19:23 |
thorst_ | the open is where it hangs. | 19:23 |
efried1 | That's... weird. | 19:24 |
thorst_ | at least in my testing | 19:24 |
thorst_ | that was where it hung | 19:24 |
efried1 | But okay, let me work on it some more. | 19:24 |
thorst_ | but it had been OK generally because something was writing to it already | 19:24 |
efried1 | Right | 19:24 |
efried1 | That should be the case here | 19:24 |
efried1 | since the writer thread is already running when we do the open. | 19:24 |
thorst_ | agree...but pipo's are weirdo's | 19:24 |
thorst_ | I'd say put a pdb in there | 19:24 |
thorst_ | verify my assertion | 19:25 |
efried1 | roger that. | 19:25 |
efried1 | thorst_, I put a breakpoint right before the open - and never hit it. | 19:35 |
efried1 | i.e. hang happens before that. | 19:37 |
efried1 | which is super bizarre. | 19:37 |
efried1 | Because the FIFO is getting created. | 19:37 |
thorst_ | lol | 19:37 |
thorst_ | the plot thickens | 19:37 |
efried1 | And the only other thing in there is the spawning of the writer thread. Which should be nonblocking. | 19:38 |
efried1 | Unless the thread pool is full. | 19:38 |
efried1 | One has to assume we're allowed more than one ThreadPoolExecutor per process. | 19:38 |
efried1 | Kinda silly otherwise, no? | 19:38 |
efried1 | Or maybe mkfifo isn't returning. | 19:39 |
efried1 | I'll put the pdb earlier and see which bit is actually hanging. | 19:40 |
efried1 | Here's another interesting point. | 19:42 |
efried1 | I can't ^C the compute process until I've deleted the REST_API_PIPE | 19:42 |
thorst_ | heh | 19:42 |
efried1 | "blocking on I/O" is one of the classic things that prevents SIGINT from going through. | 19:42 |
efried1 | This time the writer got off 4980736 bytes. | 19:44 |
efried1 | 4.75MB | 19:45 |
*** tblakes has quit IRC | 19:45 | |
efried1 | If we're writing in 64K blocks, that's 76, compared to 14 previously. | 19:45 |
thorst_ | I think async was a no no for pipes | 19:46 |
efried1 | huh? | 19:46 |
efried1 | What else are pipes for? | 19:46 |
thorst_ | sorry - non-blocking | 19:46 |
efried1 | non-blocking which? | 19:47 |
thorst_ | write | 19:47 |
thorst_ | let me try to find where I saw that | 19:47 |
efried1 | oooooooohkay, I'm not hitting THAT breakpoint either. | 19:48 |
efried1 | which kinda leads me to believe I've been working on the wrong pypowervm this whole time. | 19:48 |
thorst_ | lol | 19:48 |
thorst_ | perhaps! | 19:48 |
efried1 | It had all the right patches in it. | 19:48 |
thorst_ | and maybe that's why CI was borked | 19:48 |
efried1 | oo | 19:49 |
thorst_ | so how I check that... | 19:49 |
efried1 | The dist-packages version also has the local2remote patch. | 19:50 |
thorst_ | you have | 19:50 |
thorst_ | <module 'pypowervm.adapter' from '/usr/local/lib/python2.7/dist-packages/pypowervm/adapter.pyc'> | 19:50 |
thorst_ | lol | 19:50 |
thorst_ | alright, so I think we found out why we were dying. | 19:51 |
thorst_ | fantastic. | 19:51 |
efried1 | Well, hold da phone. | 19:51 |
adreznec | *record scratch* | 19:51 |
efried1 | The dist-packages one has the rest_api_pipe and _wrap_user_stream patches too. | 19:51 |
thorst_ | so put the pdb's in there and see if it hangs on the open | 19:52 |
efried1 | Well, maybe I start by putting my new subthread in there. | 19:52 |
thorst_ | wouldn't we want to debug where the freeze happens first? | 19:54 |
thorst_ | my vote is still on the open | 19:54 |
thorst_ | and I bet the fix is that we open the read for non block. | 19:54 |
thorst_ | and in fact, we probably open the read for non block before submitting the file writer. | 19:55 |
*** apearson has quit IRC | 19:56 | |
efried1 | The hang happens even with the new subthread. So okay, I'll pdb and see if it's the open, then try the open nonblocking. | 19:58 |
efried1 | Does that mean attempts to read from that handle will block until the open actually goes through? | 19:59 |
efried1 | Not sure how that really helps us, but will try. | 20:00 |
*** apearson has joined #openstack-powervm | 20:01 | |
thorst_ | actually, not sure | 20:02 |
thorst_ | reading the python docs | 20:02 |
efried1 | thorst_, yup, it's the open. | 20:07 |
efried1 | thorst_, how do you specify the open as nonblocking? | 20:11 |
adreznec | efried1: FYI Drew may be slow to respond, he's presenting right now | 20:12 |
efried1 | thx | 20:12 |
*** svenkat has quit IRC | 20:29 | |
*** apearson has quit IRC | 20:34 | |
*** seroyer has quit IRC | 20:41 | |
thorst_ | efried1: sorry...yeah | 20:44 |
thorst_ | I looked into it | 20:44 |
thorst_ | and I have no clue | 20:44 |
thorst_ | surprise! | 20:44 |
efried1 | thorst_, I tried adding 'b' and a bufsize to the open, but neither had an effect (which doesn't really surprise me). | 20:56 |
efried1 | I'm going to isolate this fifo business in ipython and see if I can nail down the behavior. | 20:56 |
thorst_ | efried1: oK | 20:58 |
efried1 | reckon the 'b' should be there regardless. | 20:58 |
thorst_ | efried1: based on what I read, the b only matters for windows | 20:59 |
thorst_ | and I'm not ready to jump to 'pypowervm works on windows!' | 21:00 |
efried1 | Yeah, but "good for documentation" | 21:00 |
*** apearson has joined #openstack-powervm | 21:00 | |
efried1 | thorst_ I think I get it. | 21:08 |
efried1 | http://stackoverflow.com/questions/39233663/nonblocking-fifo | 21:08 |
efried1 | First of all, have to use os.open so we can use the O_NONBLOCK flag. | 21:08 |
efried1 | Second, the writer has to wait until the reader opens, apparently. Hopefully our writer is doing that. | 21:08 |
efried1 | I'll play around with this. | 21:09 |
*** seroyer has joined #openstack-powervm | 21:11 | |
*** kriskend has quit IRC | 21:16 | |
*** edmondsw has quit IRC | 21:43 | |
thorst_ | efried1: neat | 21:43 |
*** apearson has quit IRC | 21:45 | |
*** dwayne has quit IRC | 21:45 | |
*** thorst_ has quit IRC | 21:46 | |
*** apearson has joined #openstack-powervm | 21:58 | |
*** mdrabe has quit IRC | 22:00 | |
*** k0da has quit IRC | 22:30 | |
*** smatzek has quit IRC | 22:47 | |
*** tjakobs_ has quit IRC | 23:18 | |
*** tlian has quit IRC | 23:19 | |
*** tlian has joined #openstack-powervm | 23:26 | |
*** apearson has quit IRC | 23:28 | |
*** seroyer has quit IRC | 23:48 | |
*** tlian has quit IRC | 23:54 | |
*** tlian has joined #openstack-powervm | 23:59 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!