| *** READ10 has quit IRC | 00:00 | |
| *** LinPeiWen has joined #openstack-nova | 00:33 | |
| *** brinzhang has joined #openstack-nova | 00:33 | |
| *** hongbin has quit IRC | 00:35 | |
| *** spatel has joined #openstack-nova | 00:49 | |
| *** hongbin has joined #openstack-nova | 00:57 | |
| *** redrobot has quit IRC | 01:08 | |
| *** spatel has quit IRC | 01:10 | |
| openstackgerrit | Yao wang proposed openstack/nova master: Use VIR_DOMAIN_XML_INACTIVE for detach volume https://review.opendev.org/749489 | 01:10 |
|---|---|---|
| *** READ10 has joined #openstack-nova | 01:25 | |
| *** Liang__ has joined #openstack-nova | 01:37 | |
| openstackgerrit | Yingji Sun proposed openstack/nova stable/train: Set different VirtualDevice.key https://review.opendev.org/752843 | 01:40 |
| *** yingjisun has joined #openstack-nova | 01:52 | |
| *** songwenping__ has joined #openstack-nova | 02:11 | |
| *** songwenping_ has quit IRC | 02:14 | |
| *** dannins has joined #openstack-nova | 02:15 | |
| *** READ10 has quit IRC | 02:18 | |
| *** hongbin has quit IRC | 02:23 | |
| *** hongbin has joined #openstack-nova | 02:31 | |
| openstackgerrit | Yao wang proposed openstack/nova master: Use VIR_DOMAIN_XML_INACTIVE for detach volume https://review.opendev.org/749489 | 02:37 |
| *** LinPeiWen has quit IRC | 03:09 | |
| *** sapd1 has joined #openstack-nova | 03:20 | |
| *** dave-mccowan has quit IRC | 03:23 | |
| *** psachin has joined #openstack-nova | 03:41 | |
| *** yingjisun has quit IRC | 04:04 | |
| *** zzzeek has quit IRC | 04:15 | |
| *** zzzeek has joined #openstack-nova | 04:17 | |
| *** hongbin has quit IRC | 04:20 | |
| *** ratailor has joined #openstack-nova | 04:23 | |
| *** hongbin has joined #openstack-nova | 04:24 | |
| *** hongbin has quit IRC | 04:29 | |
| *** rcernin has quit IRC | 04:31 | |
| *** evrardjp has quit IRC | 04:33 | |
| *** evrardjp has joined #openstack-nova | 04:33 | |
| *** hongbin has joined #openstack-nova | 04:38 | |
| *** rcernin has joined #openstack-nova | 04:40 | |
| *** jawad_axd has joined #openstack-nova | 05:11 | |
| *** LinPeiWen has joined #openstack-nova | 05:14 | |
| *** vishalmanchanda has joined #openstack-nova | 05:15 | |
| *** jawad_axd has quit IRC | 05:15 | |
| *** hongbin has quit IRC | 05:29 | |
| *** hongbin has joined #openstack-nova | 05:31 | |
| *** hongbin has quit IRC | 05:32 | |
| *** ociuhandu has joined #openstack-nova | 05:35 | |
| *** hongbin has joined #openstack-nova | 05:36 | |
| *** rcernin has quit IRC | 05:39 | |
| *** ociuhandu has quit IRC | 05:40 | |
| *** hongbin has quit IRC | 05:40 | |
| *** yingjisun has joined #openstack-nova | 05:42 | |
| *** sapd1 has quit IRC | 05:46 | |
| *** abhishekk has joined #openstack-nova | 05:47 | |
| *** jsuchome has joined #openstack-nova | 05:49 | |
| *** rcernin has joined #openstack-nova | 06:01 | |
| *** rcernin has quit IRC | 06:20 | |
| *** ralonsoh has joined #openstack-nova | 06:24 | |
| *** ircuser-1 has joined #openstack-nova | 06:42 | |
| *** bhagyashris is now known as bhagyashri|rover | 06:55 | |
| *** slaweq has joined #openstack-nova | 07:03 | |
| *** tosky has joined #openstack-nova | 07:03 | |
| *** sapd1 has joined #openstack-nova | 07:22 | |
| *** xek has joined #openstack-nova | 07:37 | |
| *** rcernin has joined #openstack-nova | 07:39 | |
| *** rcernin has quit IRC | 07:52 | |
| *** jawad_axd has joined #openstack-nova | 08:00 | |
| bauzas | good morning Nova | 08:01 |
| gibi | good morning bauzas | 08:10 |
| openstackgerrit | Elod Illes proposed openstack/nova stable/pike: libvirt: Provide VIR_MIGRATE_PARAM_PERSIST_XML during live migration https://review.opendev.org/747978 | 08:23 |
| * bauzas becomes nervous about the deadline for providing the recording for his Summit session :( | 08:26 | |
| kashyap | bauzas: When is the deadline? 01-Oct? | 08:27 |
| bauzas | yes... | 08:27 |
| brinzhang | gibi: can you review this patch https://review.opendev.org/#/c/749052? | 08:35 |
| *** derekh has joined #openstack-nova | 08:42 | |
| *** xek has quit IRC | 08:46 | |
| *** ociuhandu has joined #openstack-nova | 08:53 | |
| *** k_mouza has joined #openstack-nova | 08:54 | |
| *** jangutter has joined #openstack-nova | 09:03 | |
| *** jangutter has quit IRC | 09:03 | |
| *** jangutter_ has quit IRC | 09:03 | |
| *** jangutter has joined #openstack-nova | 09:04 | |
| kashyap | lyarwood: Morning, I've tried a rapid succession of attaching 20 disks (via `attach-device`) while there's consistent 40% of load on the CPUs ... can't reproduce so far. | 09:09 |
| kashyap | lyarwood: If you don't mind holding your nose tightly ... here's the repro: https://kashyapc.fedorapeople.org/device-attach-and-detach.bash | 09:09 |
| kashyap | (Fun aside: changing the target dev from sdX to vdX increases the execution of the above script by at least 20 seconds. So attaching 'virtio' disks seems to "slow"? Can't be so) | 09:10 |
| *** psachin has quit IRC | 09:11 | |
| kashyap | That said, I should rewrite it in Python and make it as close to what Nova is doing as possible | 09:20 |
| kashyap | Because, I learnt that if we suspsect a timing issue, 'virsh' has a lot more overhead. A naive connection-based approach is better | 09:23 |
| *** aj_mailing has quit IRC | 09:25 | |
| *** swp20 has joined #openstack-nova | 09:32 | |
| *** lemko has quit IRC | 09:35 | |
| *** lemko has joined #openstack-nova | 09:35 | |
| hemanth_n | sean-k-mooney: Appreciate if you can have a final look on this patch https://review.opendev.org/#/c/749175/ .. thank you | 09:50 |
| *** Liang__ has quit IRC | 09:54 | |
| lyarwood | kashyap:k~. | 09:55 |
| lyarwood | ops weird connection issue | 09:55 |
| lyarwood | kashyap: ack yeah that sounds like a sane approach, I don't think it's the rapid attach/detach that's the issue btw, just the overall load on the underlying host and libvirtd | 09:56 |
| kashyap | lyarwood: Oh, right; perhaps I should increase the load to 80% and do a detach? | 09:58 |
| kashyap | I'll vary the load (by measuring it in parallel with `sar -u 5`) and see | 09:58 |
| lyarwood | yup sounds good | 10:00 |
| tosky | lyarwood: nice work on the live migration/evacuate stuff, it seems that part is fine | 10:01 |
| tosky | now only the grenade one is missing | 10:02 |
| lyarwood | tosky: np thanks for leading the effort :) | 10:10 |
| lyarwood | gibi / stephenfin ; https://review.opendev.org/#/q/owner:self+topic:native-zuulv3-migration+status:open+project:openstack/nova ^ re the above, the live-migration and new evacuate job should be ready for review now. | 10:10 |
| *** yingjisun has quit IRC | 10:13 | |
| lyarwood | actually let me update the TODO for the ceph jobs | 10:14 |
| openstackgerrit | Lee Yarwood proposed openstack/nova master: zuul: Introduce nova-evacuate https://review.opendev.org/744883 | 10:14 |
| openstackgerrit | Lee Yarwood proposed openstack/nova master: zuul: Replace nova-live-migration with zuulv3 jobs https://review.opendev.org/752557 | 10:14 |
| lyarwood | there we go, should be good now if the gate behaves | 10:15 |
| tosky | gates and Monday, uhm | 10:16 |
| tosky | there are only 90 (check) + 16 (gate) jobs, so we may be lucky | 10:16 |
| lyarwood | hmmm I wonder if it's actually worth merging the nova-live-migration and nova-evacuate job now that the playbook is written | 10:19 |
| *** nightmare_unreal has joined #openstack-nova | 10:22 | |
| tosky | up to you and the time needed to run the tests vs the time needed to deploy | 10:32 |
| *** yingjisun has joined #openstack-nova | 10:33 | |
| *** yingjisun has quit IRC | 10:34 | |
| *** raildo has joined #openstack-nova | 10:36 | |
| gibi | lyarwood: I have some nits in https://review.opendev.org/#/c/744883 | 10:41 |
| gibi | lyarwood: will you do a follow up or respin it quickly? | 10:41 |
| gibi | brinzhang: sorry I think that refactor needs to wait for W | 10:42 |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: libvirt: Enable 'vmcoreinfo' feature by default https://review.opendev.org/752912 | 10:44 |
| sean-k-mooney | hemanth_n: sure ill go over it later today | 11:01 |
| hemanth_n | sean-k-mooney: thanks | 11:01 |
| lyarwood | gibi: ack can do | 11:01 |
| openstackgerrit | Lee Yarwood proposed openstack/nova master: zuul: Introduce nova-evacuate https://review.opendev.org/744883 | 11:09 |
| openstackgerrit | Lee Yarwood proposed openstack/nova master: zuul: Replace nova-live-migration with zuulv3 jobs https://review.opendev.org/752557 | 11:09 |
| *** ratailor has quit IRC | 11:17 | |
| *** ratailor_ has joined #openstack-nova | 11:17 | |
| *** sapd1 has quit IRC | 11:17 | |
| gibi | lyarwood: thanks | 11:22 |
| *** jawad_axd has quit IRC | 11:25 | |
| *** jawad_axd has joined #openstack-nova | 11:26 | |
| *** bbowen has quit IRC | 11:33 | |
| *** bbowen has joined #openstack-nova | 11:33 | |
| openstackgerrit | Yao wang proposed openstack/nova master: Use VIR_DOMAIN_XML_INACTIVE for detach volume https://review.opendev.org/749489 | 11:38 |
| *** artom has joined #openstack-nova | 11:39 | |
| *** xek has joined #openstack-nova | 11:42 | |
| sean-k-mooney | gibi: stephenfin i have figured out why the func test are failing. one issue is i was not adding the 0x prefix if you use "%(func)o" or "%(func)x" it does not genreate the prefix, the main issue however is device detach was only implemented for disk in the fake libvirt driver | 11:43 |
| sean-k-mooney | so im really not sure how the sriov detach func test actully are passing on master but once i fixed the 0x prefix and added a valid impleamtion of detach the tests seam to be passing. now i just need to run all the other func and unit tests before pushing | 11:44 |
| gibi | I guess it only verifies the compute.manager and resource_tracker part of the functionality | 11:48 |
| sean-k-mooney | ya it must never check the doamin xml. anyway that should now be correct too. | 11:49 |
| sean-k-mooney | i dont really know why my minimal chages to the test code caused it to now fail but im going to fix this in a seperate patch below my actul change | 11:49 |
| sean-k-mooney | although maybe i cant actully | 11:50 |
| sean-k-mooney | since im changin some code im adding in my patch | 11:50 |
| sean-k-mooney | and some code that was already there. | 11:50 |
| gibi | I did see that nova and libvirt generates the func number differnetly I think libvirt adds the prefix while nova doesn't | 11:51 |
| gibi | and libvirt accepts both | 11:51 |
| sean-k-mooney | we strip the prfix when generating the adress doing [2:] | 11:53 |
| sean-k-mooney | i was basically passing an unprefixed string and that endup passing the empty stiring to oslo | 11:53 |
| sean-k-mooney | oslo pciadress field then rejected it since the func part of the pci adress was empty | 11:54 |
| *** xek has quit IRC | 11:55 | |
| openstackgerrit | Claudiu Belu proposed openstack/nova master: hyperv: Configures chassis asset tags for VMs. https://review.opendev.org/752723 | 11:57 |
| *** martinkennelly has joined #openstack-nova | 12:10 | |
| *** vishalmanchanda has quit IRC | 12:11 | |
| *** priteau has joined #openstack-nova | 12:22 | |
| *** jawad_ax_ has joined #openstack-nova | 12:32 | |
| *** dave-mccowan has joined #openstack-nova | 12:35 | |
| *** jawad_axd has quit IRC | 12:36 | |
| *** jawad_axd has joined #openstack-nova | 12:42 | |
| *** jawad_ax_ has quit IRC | 12:46 | |
| *** xek has joined #openstack-nova | 12:48 | |
| openstackgerrit | sean mooney proposed openstack/nova master: libvirt: delegate ovs plug to os-vif https://review.opendev.org/602432 | 12:53 |
| *** nweinber has joined #openstack-nova | 12:53 | |
| *** Luzi has joined #openstack-nova | 12:55 | |
| *** xek has quit IRC | 12:59 | |
| *** ratailor_ has quit IRC | 13:06 | |
| *** lbragstad has joined #openstack-nova | 13:16 | |
| *** jawad_axd has quit IRC | 13:21 | |
| *** jawad_axd has joined #openstack-nova | 13:22 | |
| *** eharney has joined #openstack-nova | 13:34 | |
| gibi | sean-k-mooney: please remind me which patch is the one where you faced this pci function issue? | 13:37 |
| *** yingjisun has joined #openstack-nova | 13:37 | |
| sean-k-mooney | https://review.opendev.org/#/c/602432/18 | 13:40 |
| sean-k-mooney | the followup to the one you jsut reviewd | 13:40 |
| sean-k-mooney | this is the detach impleamtion https://review.opendev.org/#/c/602432/18/nova/tests/unit/virt/libvirt/fakelibvirt.py@1084 | 13:40 |
| gibi | sean-k-mooney: thanks. then I stoppped to early in my review on that series | 13:40 |
| gibi | continuing now | 13:40 |
| sean-k-mooney | the second hunk in that file is where i had dropped the 0x prefix | 13:41 |
| sean-k-mooney | that was the second issue | 13:41 |
| sean-k-mooney | gibi: the first patch really is a fix for a different bug but i just happen to need that to maintain the same behavior for my own patch since libvirt would nolonger be unplugging the vifs in this case now that os-vif handels the pluging. | 13:44 |
| sean-k-mooney | gibi: by the way i have already accpeted that these wont qualify for rc1 and they will be backported after wards so while i am glad for the review if something else needs review for rc1 then that obviously is higher priority. thanks in anycase. | 13:48 |
| gibi | sean-k-mooney: ack | 13:50 |
| gibi | at the moment I'm not tracking anything for RC1 that needs review | 13:51 |
| *** xek has joined #openstack-nova | 13:51 | |
| sean-k-mooney | ya same. i think lees ubuntu cloud archive patches have also merged in devstack at least the ussuri one i think so the libvirt bumps are really the only think im mindful off that we should still try to land | 13:52 |
| sean-k-mooney | i think we are in a pretty good state overall | 13:52 |
| gibi | yepp | 13:53 |
| *** Luzi has quit IRC | 13:54 | |
| sean-k-mooney | gmann: care to send https://review.opendev.org/#/c/747123/ through the gate now that the ussuri version has merged | 13:55 |
| *** eharney has quit IRC | 13:55 | |
| *** eharney has joined #openstack-nova | 13:58 | |
| gmann | sean-k-mooney: done | 13:58 |
| sean-k-mooney | :) | 13:58 |
| *** jsuchome has quit IRC | 14:03 | |
| openstackgerrit | Merged openstack/nova stable/train: Correctly disable greendns https://review.opendev.org/751622 | 14:14 |
| *** hemna has quit IRC | 14:30 | |
| *** hemna has joined #openstack-nova | 14:30 | |
| dansmith | artom: I didn't see any discussion of a test for this: https://review.opendev.org/#/c/751302/ | 14:31 |
| artom | dansmith, there wasn't | 14:31 |
| dansmith | artom: since we've had a few ordering things changed in there in recent memory, can't we have a test that pokes at eventlet internals or something to check that things happen like we expect? | 14:31 |
| artom | How would we test that thing specifically? | 14:31 |
| artom | dansmith, ah I see what you mean | 14:31 |
| *** hongbin has joined #openstack-nova | 14:31 | |
| artom | dansmith, yeah, could be done | 14:31 |
| artom | I guess? | 14:32 |
| dansmith | well, it would make it easier to validate that you've really changed behavior.. clearly the original code didn't think the ordering mattered, so if it does, it'd be nice if we could, you know, validate that | 14:32 |
| artom | dansmith, I agree with your point. Originally I 1. couldn't think of a way to test specifically this and 2. figured the NOTE I left was the best we could do to prevent future reorderings | 14:33 |
| dansmith | but I'm guessing the reviewers just assumed that it does matter and that your change fixed it sufficiently, | 14:33 |
| dansmith | and it'd just be nice to have a test where you can revert the code change and make sure the test validates some wrong behavior, which is (or used to be) common double checking for obscure fixes like this | 14:34 |
| *** eharney has quit IRC | 14:35 | |
| artom | dansmith, I think it still is - we try to do the "regression test that assers the wrong thing -> fix + test fix" thing when we can | 14:36 |
| artom | I suppose in this case we all assumed it wasn't realistic | 14:36 |
| dansmith | assumed but didn't even discuss or question, it seems... just sayin', this is pretty fundamental code, merged pretty quick and backported quick as well.. would be nice to have something to validate it | 14:37 |
| dansmith | like, maybe even the version of eventlet on stable/train behaves a little different or something | 14:38 |
| artom | dansmith, yep, I'mma try to hax something | 14:39 |
| dansmith | I think maybe one tricky thing is going to be that a test will run after eventlet has already been imported and this monkey patching has been done, which means you just need to have the test check that greendns is disabled in the internals or whatever, | 14:41 |
| dansmith | and then testing that the patch fixed it will just be running the same test but without the code change | 14:41 |
| dansmith | meaning, the test won't actually call monkey_patch() to do the test, but just examine the result of it and make sure it's good after the test harness did | 14:41 |
| artom | dansmith, I dunno - does the test framework actually call monkey_patch()? | 14:42 |
| artom | Lol yes it does: `import nova.monkey_patch # noqa` | 14:43 |
| artom | First line in nova/test.py | 14:43 |
| dansmith | well, I guess I'm not sure.. we have to in functional to make some of it work I thought | 14:43 |
| dansmith | heh, yeah okay | 14:43 |
| dansmith | in some cases you could get away with it, but in others we'd never be able to do certain things | 14:44 |
| artom | So it could be as stupid as the demonstrator I had in my bug | 14:45 |
| artom | Try to resolve 'lulz.fake', examine the stack... | 14:45 |
| dansmith | yeah | 14:45 |
| artom | Pass if it does NOT contain greendns | 14:45 |
| dansmith | ....yeah | 14:45 |
| *** vinay_m has joined #openstack-nova | 14:46 | |
| *** eharney has joined #openstack-nova | 14:47 | |
| *** dklyle has joined #openstack-nova | 14:56 | |
| *** mriedem has joined #openstack-nova | 14:58 | |
| *** xek has quit IRC | 15:01 | |
| *** k_mouza has quit IRC | 15:05 | |
| *** priteau has quit IRC | 15:10 | |
| *** k_mouza has joined #openstack-nova | 15:11 | |
| *** k_mouza has quit IRC | 15:15 | |
| *** k_mouza has joined #openstack-nova | 15:15 | |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: libvirt: 'video.vram' property must be an integer https://review.opendev.org/753011 | 15:18 |
| *** jawad_axd has quit IRC | 15:20 | |
| stephenfin | gibi, lyarwood: Real simple bugfix there ^ | 15:23 |
| *** redrobot has joined #openstack-nova | 15:23 | |
| *** mlavalle has joined #openstack-nova | 15:33 | |
| kashyap | stephenfin: Nice sleuthing ... TIL: "This appears to be a Python 3 thing, introduced by division of ints now returning a float." | 15:33 |
| gibi | stephenfin: should https://opendev.org/openstack/nova/src/branch/master/nova/tests/unit/virt/libvirt/test_driver.py#L6738 | 15:35 |
| gibi | stephenfin: fail now? | 15:36 |
| stephenfin | gibi: um, yes, it will | 15:37 |
| * stephenfin respins | 15:37 | |
| *** hongbin has quit IRC | 15:38 | |
| bauzas | sorry folks, was borked by internal stuff and the fact i have a presentation to make for a deadline in 10 days | 15:40 |
| bauzas | any reviews you want me to do ? | 15:40 |
| bauzas | stephenfin: remember the presentation deadline, btw. /o\ | 15:40 |
| bauzas | why on Earth did I wanted to provide a proposal ? | 15:40 |
| bauzas | what a curse | 15:40 |
| *** jawad_axd has joined #openstack-nova | 15:40 | |
| gibi | bauzas: at the moment I'm not tracking anything for RC1 that needs review | 15:41 |
| bauzas | cool | 15:41 |
| *** k_mouza has quit IRC | 15:41 | |
| bauzas | gibi: sorry, I hate myself being dragged from RC1 as it looks to me more priority | 15:41 |
| bauzas | and I promised to see a vGPU usage for both V100 and GVT-g, oh man | 15:42 |
| bauzas | I'm doomed | 15:42 |
| *** k_mouza has joined #openstack-nova | 15:42 | |
| gibi | bauzas: no worries | 15:43 |
| gibi | I think we are pretty well set for RC1 at the moment | 15:43 |
| *** yingjisun has quit IRC | 15:44 | |
| *** hongbin has joined #openstack-nova | 15:44 | |
| *** jawad_axd has quit IRC | 15:45 | |
| *** JamesBenson has joined #openstack-nova | 15:45 | |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: libvirt: 'video.vram' property must be an integer https://review.opendev.org/753011 | 15:55 |
| *** ralonsoh has quit IRC | 15:57 | |
| *** LinPeiWen has quit IRC | 16:00 | |
| openstackgerrit | Balazs Gibizer proposed openstack/nova master: Use cell targeted context to query BDMs for metadata https://review.opendev.org/752459 | 16:00 |
| openstackgerrit | Balazs Gibizer proposed openstack/nova master: Clean up the DynamicVendorData constructor https://review.opendev.org/752718 | 16:00 |
| artom | dansmith, looks like you get to dance around me yelling "I told you so" | 16:09 |
| artom | My unit test appears to show we're *still* not disabling greendns correctly | 16:09 |
| *** vishalmanchanda has joined #openstack-nova | 16:10 | |
| openstackgerrit | Artom Lifshitz proposed openstack/nova master: Unit test for 7c1d964faa https://review.opendev.org/753072 | 16:10 |
| artom | dansmith ^^ | 16:10 |
| *** sapd1 has joined #openstack-nova | 16:10 | |
| artom | I suppose I should have WIP'ed it, but it's not going to pass Zuul regardless | 16:11 |
| *** vinay_m has quit IRC | 16:24 | |
| *** ociuhandu has quit IRC | 16:25 | |
| *** vinay_m has joined #openstack-nova | 16:25 | |
| artom | Huh, by the time nova/monkey_patch.py runs, we've *already* imported evenlet | 16:25 |
| *** k_mouza has quit IRC | 16:33 | |
| *** _erlon_ has joined #openstack-nova | 16:37 | |
| dansmith | artom: meaning the tests defeat the fix because they import eventlet before they've run the thing that sets-before-import like cmd/* should? | 16:38 |
| artom | dansmith, no, I think the test shows that the fix doesn't work, because something else imports eventlet before Nova does | 16:38 |
| dansmith | but is that an artifact of our unit tests vs. real production stuff, or is there another place? | 16:40 |
| *** gyee has joined #openstack-nova | 16:40 | |
| dansmith | I thought you validated the fix locally on a real deployment and it had the desired effect? | 16:40 |
| artom | dansmith, I did not. | 16:40 |
| dansmith | ah, okay, I guess I thought from the bug report you had | 16:41 |
| artom | You need to hit a specific situation, which is hard to reproduce on the kind of deployment I have access to | 16:41 |
| artom | So I went on the demonstrator code snippet | 16:41 |
| dansmith | okay | 16:41 |
| artom | ... which doesn't work if something else has imported evenlet before you even do anything | 16:42 |
| *** vinay_m has quit IRC | 16:42 | |
| sean-k-mooney | artom: nothing else should be importing evenetlet | 16:42 |
| artom | sean-k-mooney, dunno what to tell you, but if I `if 'eventlet' in sys.modules` *before* we import it, it's there | 16:43 |
| sean-k-mooney | at least nothing else should be monkeypatching | 16:43 |
| artom | sean-k-mooney, monkeypatching doesn't matter here, the env variable we care about is processed by eventlet at import-time | 16:44 |
| artom | https://github.com/eventlet/eventlet/blob/af407c77f208ceefe5a35e39aed0cf3fdfc07cb9/eventlet/green/socket.py | 16:44 |
| sean-k-mooney | well i guess teh issue you will have is that you cant contol the order in which the tests run | 16:45 |
| sean-k-mooney | we import eventlets in a number of different tests | 16:46 |
| artom | As in directly `import eventlet`? | 16:46 |
| sean-k-mooney | https://github.com/openstack/nova/search?q=%22import+eventlet%22&unscoped_q=%22import+eventlet%22 | 16:46 |
| sean-k-mooney | yep | 16:46 |
| artom | Ah, so maybe that's it | 16:47 |
| sean-k-mooney | actully better https://github.com/openstack/nova/search?q=%22import+eventlet%22+path%3A%2Fnova%2Ftests&unscoped_q=%22import+eventlet%22+path%3A%2Fnova%2Ftests | 16:47 |
| dansmith | artom: that's what I was saying.. an artifact of the test infrastructure that doesn't really happen in prod | 16:47 |
| artom | I suppose passing a test name regex to tox doesn't prevent it from reading the entire tests tree and thus importing eventlet | 16:47 |
| dansmith | in prod we control the entry ordering exactly, but not so much in the test stuff | 16:48 |
| sean-k-mooney | there are 7 direct imports and 5 from eventlet import... | 16:48 |
| dansmith | artom: it's not tox, it's stestr | 16:48 |
| artom | dansmith, right, sorry | 16:48 |
| dansmith | and it does it in parallel and often in different orders based on the bucketing | 16:48 |
| artom | In that case we can't really have a unit test for this | 16:48 |
| artom | We would need a way to guarantee "first run" | 16:49 |
| artom | In order to be the first to import eventlet | 16:49 |
| artom | `del` and importlib.reload() don't work, I tried | 16:49 |
| sean-k-mooney | you could remove it form the module path but honestly im not sure this is worth it | 16:50 |
| dansmith | artom: this is what I was saying earlier, which might require some poking into eventlet | 16:50 |
| artom | dansmith, ok, it only me like 2 hours to catch up with you | 16:50 |
| sean-k-mooney | artom: do you jsut want to call the un patched socket lib | 16:50 |
| sean-k-mooney | *module | 16:50 |
| dansmith | sean-k-mooney: I do think *some* sort of validation is worth it, because artom didn't confirm by hand, says it's hard to do so, and five minutes ago, wasn't confident that the fix was actually a fix :) | 16:50 |
| artom | And you've been breathing smoke for 2 weeks, so your brain is at like 20% capacity | 16:51 |
| sean-k-mooney | dansmith: im more wondering what is the intent of the test | 16:51 |
| artom | sean-k-mooney, to make sure we've properly disbled greendns | 16:51 |
| sean-k-mooney | when called via a monkeypatched module | 16:51 |
| dansmith | sean-k-mooney: well, for one thing, a unit test of that monkeypatch code to make sure it's doing the things we expect seems like a good plan to me | 16:51 |
| artom | Yeah, so something like socket.gethostaddr() | 16:52 |
| artom | Like I did in my demonstrator code | 16:52 |
| dansmith | I think we started that around uwsgi time and have been lumping stuff in there from the start without really much validation | 16:52 |
| sean-k-mooney | artom: do you have a link to the orginial patch | 16:52 |
| sean-k-mooney | i just see the unit test but its hard to follow without context | 16:52 |
| artom | sean-k-mooney, https://review.opendev.org/#/c/751302/ | 16:52 |
| dansmith | artom: maybe we go with the approach that the profiler thing uses, which is another unittest run after a successful regular one so you can constrain the ordering? | 16:53 |
| artom | dansmith, that's just an extra line in tox.ini... | 16:54 |
| dansmith | right | 16:54 |
| dansmith | we've had multiple bugs related to the ordering of this monkeypatch thing, so .. maybe it's time :) | 16:54 |
| artom | So, an entirely separate file, outside of the unit tests, that just import nova.monkey_patch and tests stuff? | 16:55 |
| dansmith | like every line in that function is "be suuper careful to do this before that | 16:55 |
| sean-k-mooney | are we sure that the coment is correct | 16:55 |
| artom | sean-k-mooney, which comment | 16:55 |
| sean-k-mooney | that eventlet processes theis at import time and not when we call eventlet.monkeypatch | 16:55 |
| dansmith | sean-k-mooney: I'm not sure any of this is correct because apparently nobody has tested it :P | 16:55 |
| artom | sean-k-mooney, fairly certain, by looking at the eventlet code: https://github.com/eventlet/eventlet/blob/v0.26.0/eventlet/green/socket.py#L20 | 16:56 |
| sean-k-mooney | this one https://review.opendev.org/#/c/751302/2/nova/monkey_patch.py@33 | 16:56 |
| sean-k-mooney | ok so that is module scope | 16:56 |
| sean-k-mooney | so it will only be processed once the first tim ethe socket module is loaded | 16:57 |
| artom | 'zactly | 16:57 |
| sean-k-mooney | which wont be when we do import eventlet | 16:57 |
| sean-k-mooney | its import here https://github.com/eventlet/eventlet/blob/fc8cc0f6a77896234284ef40d7226d62345922c0/eventlet/patcher.py#L424 | 17:00 |
| *** aj_mailing has joined #openstack-nova | 17:00 | |
| artom | Ah, true | 17:00 |
| *** derekh has quit IRC | 17:00 | |
| artom | But... when is that called? | 17:00 |
| artom | Because it definitely *behaves* like it's processing EVENTLET_NO_GREENDNS at import time | 17:00 |
| *** k_mouza has joined #openstack-nova | 17:00 | |
| artom | IOW, it has to be set before importing eventlet | 17:01 |
| sean-k-mooney | form eventlet.monkey_patch | 17:01 |
| artom | Otherwise it has no effect | 17:01 |
| artom | Look at my Demonstration in https://bugs.launchpad.net/nova/+bug/1895322 | 17:01 |
| openstack | Launchpad bug 1895322 in OpenStack Compute (nova) ussuri "Nova is not actually disabling greendns" [Undecided,Fix committed] | 17:01 |
| sean-k-mooney | this is where its being patched as far as i can tell https://github.com/eventlet/eventlet/blob/fc8cc0f6a77896234284ef40d7226d62345922c0/eventlet/patcher.py#L274-L314 | 17:04 |
| *** k_mouza has quit IRC | 17:05 | |
| openstackgerrit | Artom Lifshitz proposed openstack/nova master: [WIP] Test for 7c1d964faa and monkey patching in general https://review.opendev.org/753072 | 17:10 |
| artom | dansmith, sean-k-mooney ^^ so that actually works | 17:10 |
| dansmith | artom: sure, but I think you can maybe also get away with it being a proper unittest and run it with stestr in the normal way | 17:10 |
| dansmith | if you make your module do stuff before it imports nova.test or whatever causes the import chain that leads to eventlet, I think stestr won't tickle anything else until you do | 17:11 |
| artom | dansmith, how though? I thought we established we can't have other tests importing eventlet before us | 17:11 |
| dansmith | if you don't discovery and don't run anything else | 17:11 |
| sean-k-mooney | artom: i think this is where we currently monkeyp[atch for test by the way https://github.com/openstack/nova/blob/master/nova/test.py#L24 | 17:11 |
| dansmith | artom: I still mean a special invocation like you have, but a unittest instead of a bare python file | 17:12 |
| sean-k-mooney | artom: so when youd do "from nova import test" | 17:12 |
| sean-k-mooney | that shoudl monkey patch | 17:12 |
| artom | Right, but unittest means stestr will read the whole tree... and so will cause eventlet to be imported, even if we don't execute the tests themselves | 17:12 |
| dansmith | no, | 17:13 |
| dansmith | not without discover I think | 17:13 |
| dansmith | for other unittest runners, if you provide an actual class module without discovery, it will try to import that path direct and then execute | 17:13 |
| artom | Oh, you literally mean `--no-discover` | 17:13 |
| sean-k-mooney | dansmith: we would have to put it in a seperate folder | 17:13 |
| dansmith | artom: right, like the profile test does | 17:13 |
| artom | *facepalm* OK yeah | 17:14 |
| sean-k-mooney | we could skip using a regex if we did that and use the --isolate parmater i think | 17:14 |
| *** k_mouza has joined #openstack-nova | 17:14 | |
| dansmith | sean-k-mooney: no, you can just make it skip for the regular tests, if necessary, or let it run in the regular ones if it will falsely claim success anyway :) | 17:14 |
| artom | Worth a try, anyways | 17:14 |
| dansmith | sean-k-mooney: you could just make it skip if sys.modules already has eventlet | 17:14 |
| sean-k-mooney | dansmith: ah | 17:14 |
| artom | This bare Python file is definitely teh uglies | 17:14 |
| sean-k-mooney | i tought you were going to sugges adding it to the test blacklist | 17:15 |
| sean-k-mooney | but having the test auto skip i think is fine | 17:15 |
| sean-k-mooney | that way we dont need to list a load of special tests if we add more like htis | 17:15 |
| dansmith | right | 17:15 |
| artom | Auto-skip is dangerous, no? Means we'd be silently missing potential errors | 17:15 |
| sean-k-mooney | no in the ci | 17:15 |
| sean-k-mooney | locallly sure | 17:15 |
| dansmith | artom: well, you can also do what the profile test does and set an envar to enable :) | 17:16 |
| sean-k-mooney | but if its run by tox | 17:16 |
| sean-k-mooney | it should be fine | 17:16 |
| artom | dansmith, ah, so folks can disable it if needed | 17:16 |
| artom | Whodda thunk precedent was so useful ;) | 17:16 |
| sean-k-mooney | artom: proably enable if needed | 17:16 |
| dansmith | artom: no, | 17:16 |
| sean-k-mooney | given its racy by defualt | 17:16 |
| dansmith | artom: it's disabled by default, enabled explicitly in that second stestr run command in tox | 17:17 |
| sean-k-mooney | you would set the env var wehn you run it explcitly | 17:17 |
| sean-k-mooney | yep what dansmith said | 17:17 |
| artom | dansmith, sorry, right | 17:17 |
| artom | I meant in the sense of... tox will run with it enabled | 17:17 |
| artom | Everything else won't | 17:18 |
| sean-k-mooney | tox -e py<whatever> sure | 17:18 |
| dansmith | seriously, there are a bunch of other things in that file with comments like "this is fragile if order isn't respected" comments... it'd be nice to get tests for those things too since we're clearly just flying by the seat of our pants on these things | 17:18 |
| sean-k-mooney | but in two calls to stester | 17:18 |
| artom | Wait, no, that second tox command that I'll add will enable and run it | 17:18 |
| *** k_mouza has quit IRC | 17:18 | |
| artom | Nothing else will because it's racy | 17:18 |
| * artom haas just now caught up | 17:18 | |
| dansmith | urllib3, oslo_context, threading things, | 17:19 |
| dansmith | artom: right, exactly.. you know, like the profiling test :) | 17:19 |
| sean-k-mooney | dansmith: ya we have a cople of example of this | 17:19 |
| artom | dansmith, you keep saying that like I'm some sort of genius that understands it the first time around | 17:19 |
| artom | ;) | 17:19 |
| sean-k-mooney | not sure why oslo_context would be on that list but why not | 17:19 |
| dansmith | artom: it just makes me feel better to highlight the first-mention-to-grok time gap :) | 17:19 |
| dansmith | sean-k-mooney: maybe if we had a test with comments in it, we'd know :/ | 17:20 |
| artom | dansmith, I'm pretty sure it's exponential with age (and inverse with sleep ) | 17:20 |
| sean-k-mooney | :) well it was more oslo is an openstack thing so hopefully we could fix the oslo issue there instead of nova | 17:20 |
| dansmith | artom: we should plot it on a graph | 17:20 |
| dansmith | sean-k-mooney: not likely if it's something oslo can't do if imported post-monkey-patch | 17:21 |
| artom | ∞ there's your graph | 17:21 |
| dansmith | heh | 17:21 |
| sean-k-mooney | https://bugs.launchpad.net/nova/+bug/1773102 | 17:22 |
| openstack | Launchpad bug 1773102 in OpenStack Compute (nova) queens "Abnormal request id in logs" [Medium,Fix committed] - Assigned to Radoslav Gerganov (rgerganov) | 17:22 |
| sean-k-mooney | thats the oslo_context bug | 17:22 |
| dansmith | ah yeah, and surely that's because it needs to grab pointers to the threading primitives for TLS before they get munched right? | 17:23 |
| sean-k-mooney | maybe although there might be a way to resolve that in a differnt way. anyway not important right now | 17:24 |
| sean-k-mooney | artom: so you going to add a second run combining the results with teh first and explictly enable that test in the second run ya | 17:24 |
| dansmith | https://bugs.launchpad.net/nova/+bug/1773102/comments/18 | 17:24 |
| openstack | Launchpad bug 1773102 in OpenStack Compute (nova) queens "Abnormal request id in logs" [Medium,Fix committed] - Assigned to Radoslav Gerganov (rgerganov) | 17:24 |
| dansmith | it'd be hard to solve it any other way, but.. yah | 17:25 |
| artom | dansmith, for all this talk of "do it like the profiler", I've tried it, and looks like it's still importing eventlet elsewhere | 17:25 |
| sean-k-mooney | artom: can you push it so we can try it | 17:26 |
| openstackgerrit | Artom Lifshitz proposed openstack/nova master: Unit test for 7c1d964faa https://review.opendev.org/753072 | 17:29 |
| artom | sean-k-mooney, ^^ | 17:29 |
| *** dtantsur is now known as dtantsur|afk | 17:31 | |
| * artom -> lunch | 17:31 | |
| dansmith | artom: can you try it with python3 -munittest path.to.test ? | 17:31 |
| dansmith | artom: wait, you're still importing nova.test in your module | 17:32 |
| dansmith | that means you do it before setUp() runs :) | 17:32 |
| sean-k-mooney | yep that would do it | 17:33 |
| dansmith | so that "importing eventlet elsewhere" is ... in your test :) | 17:33 |
| artom | Right, but nova.test imports eventlet *after setting the greendns env var* | 17:33 |
| artom | So that should be fine | 17:34 |
| artom | The problem is if some other unit tests straight up `import eventlet` without setting EVENTLET_NO_GREENDNS | 17:34 |
| dansmith | I thought something else does that is downstream of that, and that was the whole point? | 17:34 |
| artom | Something else does what? | 17:34 |
| dansmith | import eventlet | 17:34 |
| sean-k-mooney | that what we get when we run it directly http://paste.openstack.org/show/798154/ | 17:35 |
| artom | Weird | 17:36 |
| sean-k-mooney | basically we need to run this test using unitt test and not the base testcase and only import monkey patch inside teh test funtion i think | 17:38 |
| sean-k-mooney | im going to try that locally | 17:38 |
| dansmith | right | 17:38 |
| dansmith | although I moved the import into the test function and I still fail the test | 17:38 |
| dansmith | so, here's the other thing | 17:39 |
| dansmith | stestr forks and runs the worker children outside the main process | 17:39 |
| dansmith | which may not share the environment where the flag is being set | 17:39 |
| *** k_mouza has joined #openstack-nova | 17:40 | |
| artom | Sounds "right" - in the sense that, if we're the *only* ones importing nova.test, and that imports eventlet "properly" (setting EVENTLET_NO_GREENDNS), shouldn't matter when we do the importing | 17:40 |
| dansmith | still happens with bare unittest run | 17:41 |
| sean-k-mooney | http://paste.openstack.org/show/798157/ | 17:41 |
| sean-k-mooney | so it match on the function call | 17:42 |
| artom | *facepalm* | 17:42 |
| sean-k-mooney | test_greendns_is_disabled | 17:42 |
| dansmith | ah hah | 17:42 |
| dansmith | although I'm seeing it for realz: /home/dan/nova/.tox/py37/lib/python3.7/site-packages/eventlet/support/greendns.py | 17:43 |
| artom | Maybe premature though, changing it go 'greendns.py' still fails | 17:43 |
| dansmith | any chance the flag name is wrong, or requires a value different than "yes" ? | 17:44 |
| *** k_mouza has quit IRC | 17:44 | |
| artom | dansmith, I don't think it's that - remember, the bare python file worked | 17:44 |
| artom | worked == verified the correct behaviour | 17:44 |
| sean-k-mooney | dansmith https://github.com/eventlet/eventlet/blob/af407c77f208ceefe5a35e39aed0cf3fdfc07cb9/eventlet/green/socket.py#L20 | 17:45 |
| sean-k-mooney | if os.environ.get("EVENTLET_NO_GREENDNS", '').lower() != 'yes': | 17:45 |
| sean-k-mooney | so i think its right | 17:45 |
| dansmith | ack | 17:45 |
| sean-k-mooney | so i have a working copy locally | 17:46 |
| sean-k-mooney | just changed it to greendns.py | 17:46 |
| sean-k-mooney | but is that goign to be in the traceback | 17:46 |
| sean-k-mooney | http://paste.openstack.org/show/798159/ | 17:47 |
| dansmith | https://pastebin.com/kY9iCQpu | 17:47 |
| dansmith | this is how I'm getting to eventlet ^ | 17:47 |
| dansmith | unit/__init__.py calls objects.register_all() | 17:47 |
| dansmith | and eventlet is in there | 17:47 |
| sean-k-mooney | ya i was looking at that | 17:48 |
| sean-k-mooney | i didnt see which object it was | 17:48 |
| dansmith | so, like I surmised, something in the test infra is hitting it before we even get to us, but unfortunately it's in our module so it gets imported before we get imported | 17:48 |
| artom | Oh, smart just raising in evenlet itself and examining the trace | 17:48 |
| dansmith | if it wasn't in an __init__ then we'd be safe | 17:48 |
| sean-k-mooney | nova.objects.agent | 17:48 |
| artom | dansmith, so then... if I move it out of tests/unit ? | 17:49 |
| dansmith | artom: yeah I guess.. maybe functional would work by chance? | 17:49 |
| dansmith | lol, | 17:49 |
| sean-k-mooney | no | 17:49 |
| dansmith | functional moneky patches in __init__ ;P | 17:49 |
| sean-k-mooney | fucntion defietly wont | 17:49 |
| sean-k-mooney | yep | 17:49 |
| artom | dansmith, no, that as import nova.monkey_patch # noqa in __init__.py | 17:49 |
| artom | *has | 17:50 |
| sean-k-mooney | also fucntion is still under nova.test | 17:50 |
| sean-k-mooney | which import the objects | 17:50 |
| sean-k-mooney | i would have to be nova.evetlet.tests or soemthign like that | 17:50 |
| dansmith | sean-k-mooney: it's tests/unit/__init not tests/__init | 17:50 |
| sean-k-mooney | oh your right | 17:51 |
| dansmith | but yeah, unless we move that patch out of functional/__init it doesn't matter | 17:51 |
| sean-k-mooney | then ya just tests/ then | 17:51 |
| dansmith | yeah, | 17:51 |
| dansmith | and then it will be excluded from the regular stest execution anyway, | 17:51 |
| artom | So it's fine if it gets put in nova/tests | 17:51 |
| dansmith | which is what we wanted right? | 17:51 |
| sean-k-mooney | drop it at the top level and run it explcitly with tox | 17:51 |
| dansmith | yah | 17:51 |
| sean-k-mooney | ya it is | 17:52 |
| artom | Don't need the env var skip mechanism then, either | 17:53 |
| dansmith | right | 17:53 |
| sean-k-mooney | so this http://paste.openstack.org/show/798162/ | 17:55 |
| sean-k-mooney | actully setp is not needed | 17:55 |
| dansmith | yup | 17:56 |
| dansmith | but it'd be nice to get some other validations in there for the other things if we can | 17:56 |
| sean-k-mooney | ya im not super happy with the self.assertNotIn('.greendns', tb) | 17:56 |
| dansmith | artom: also, my -1 is going to be "change the title of the patch to what it does and not an obscure 'tests for $hash' message" | 17:56 |
| artom | dansmith, fair | 17:57 |
| artom | sean-k-mooney, I'll add comments? | 17:57 |
| dansmith | lots of people do that, but I hate it | 17:57 |
| *** hamalq has joined #openstack-nova | 17:57 | |
| artom | Tests for hash is what I do on the weekends :D | 17:57 |
| sean-k-mooney | actully it should be greendns.py | 17:58 |
| sean-k-mooney | its not the module path in the trace its the file path | 17:58 |
| dansmith | artom: it's legal here now, so doesn't make you seem cool and edgy like it used to | 17:58 |
| artom | dansmith, but what will I do for personality now? | 17:59 |
| dansmith | dirty jokes? | 18:01 |
| sean-k-mooney | so this works for me now that i moved it http://paste.openstack.org/show/798163/ | 18:01 |
| sean-k-mooney | oh this is what i ran http://paste.openstack.org/show/798164/ | 18:03 |
| *** hongbin has quit IRC | 18:05 | |
| *** hongbin has joined #openstack-nova | 18:11 | |
| openstackgerrit | Merged openstack/nova stable/rocky: libvirt: Handle VIR_ERR_DEVICE_MISSING when detaching devices https://review.opendev.org/742417 | 18:21 |
| CeeMac | Evening all | 18:26 |
| CeeMac | Anyone ever come across issues with guest operating system corruption following a volume retype operation ? | 18:27 |
| *** artom has quit IRC | 18:32 | |
| *** artom has joined #openstack-nova | 18:32 | |
| *** k_mouza has joined #openstack-nova | 18:34 | |
| *** ociuhandu has joined #openstack-nova | 18:34 | |
| *** k_mouza has quit IRC | 18:38 | |
| *** ociuhandu has quit IRC | 18:39 | |
| *** jawad_axd has joined #openstack-nova | 18:41 | |
| *** ociuhandu has joined #openstack-nova | 18:42 | |
| *** jawad_axd has quit IRC | 18:46 | |
| artom | Hrmpf, so it actually would work in nova/tests/functional because that __init__.py imports nova.monkey_patch, which is fine because we set EVENTLET_NO_GREENDNS in there | 18:48 |
| artom | But then we need the env var skip mechanism | 18:48 |
| artom | So perhaps better in nova/tests after all? | 18:48 |
| dansmith | I don't really care either way | 18:48 |
| artom | I guess the latter then, because less code. | 18:49 |
| artom | And I'll explain it all with a proper commit message | 18:49 |
| artom | ... *after* I pick up my kinds | 18:49 |
| artom | *kids | 18:49 |
| *** nightmare_unreal has quit IRC | 18:51 | |
| *** xek has joined #openstack-nova | 18:56 | |
| *** mriedem has quit IRC | 19:00 | |
| *** jawad_axd has joined #openstack-nova | 19:02 | |
| *** hongbin has quit IRC | 19:05 | |
| *** jawad_axd has quit IRC | 19:06 | |
| *** vishalmanchanda has quit IRC | 19:11 | |
| *** jawad_axd has joined #openstack-nova | 19:23 | |
| *** jawad_axd has quit IRC | 19:27 | |
| *** ociuhandu has quit IRC | 19:32 | |
| *** jawad_axd has joined #openstack-nova | 19:44 | |
| *** jawad_axd has quit IRC | 19:48 | |
| *** jawad_axd has joined #openstack-nova | 20:04 | |
| *** slaweq has quit IRC | 20:07 | |
| *** jawad_axd has quit IRC | 20:09 | |
| *** slaweq has joined #openstack-nova | 20:12 | |
| *** hongbin has joined #openstack-nova | 20:16 | |
| *** k_mouza has joined #openstack-nova | 20:25 | |
| *** ociuhandu has joined #openstack-nova | 20:25 | |
| *** nweinber has quit IRC | 20:26 | |
| *** k_mouza has quit IRC | 20:30 | |
| *** jawad_axd has joined #openstack-nova | 20:46 | |
| *** jawad_axd has quit IRC | 20:51 | |
| *** xek has quit IRC | 20:54 | |
| *** ociuhandu has quit IRC | 20:59 | |
| *** _erlon_ has quit IRC | 21:04 | |
| *** sapd1 has quit IRC | 21:05 | |
| *** slaweq has quit IRC | 21:06 | |
| *** tbachman has quit IRC | 21:17 | |
| *** sapd1 has joined #openstack-nova | 21:18 | |
| *** dave-mccowan has quit IRC | 21:58 | |
| *** JustAFlerkin has joined #openstack-nova | 22:02 | |
| *** dave-mccowan has joined #openstack-nova | 22:03 | |
| *** tbachman has joined #openstack-nova | 22:08 | |
| *** tbachman has quit IRC | 22:13 | |
| *** tacco has quit IRC | 22:17 | |
| *** k_mouza has joined #openstack-nova | 22:17 | |
| *** k_mouza has quit IRC | 22:22 | |
| *** jawad_axd has joined #openstack-nova | 22:29 | |
| *** tbachman has joined #openstack-nova | 22:30 | |
| *** jawad_axd has quit IRC | 22:34 | |
| *** k_mouza has joined #openstack-nova | 22:47 | |
| *** tosky has quit IRC | 22:49 | |
| *** mlavalle has quit IRC | 22:50 | |
| *** jawad_axd has joined #openstack-nova | 22:50 | |
| *** k_mouza has quit IRC | 22:52 | |
| *** jawad_axd has quit IRC | 22:55 | |
| *** tbachman has quit IRC | 23:00 | |
| *** rcernin has joined #openstack-nova | 23:02 | |
| *** zzzeek has quit IRC | 23:10 | |
| *** rcernin has quit IRC | 23:11 | |
| *** jawad_axd has joined #openstack-nova | 23:11 | |
| *** rcernin has joined #openstack-nova | 23:11 | |
| *** zzzeek has joined #openstack-nova | 23:13 | |
| *** jawad_axd has quit IRC | 23:15 | |
| *** eharney has quit IRC | 23:15 | |
| *** zzzeek has quit IRC | 23:17 | |
| *** zzzeek has joined #openstack-nova | 23:20 | |
| *** jawad_axd has joined #openstack-nova | 23:32 | |
| *** jawad_axd has quit IRC | 23:37 | |
| *** eharney has joined #openstack-nova | 23:37 | |
| *** spatel has joined #openstack-nova | 23:49 | |
| *** eharney has quit IRC | 23:50 | |
| *** eharney has joined #openstack-nova | 23:51 | |
| *** jawad_axd has joined #openstack-nova | 23:52 | |
| *** jawad_axd has quit IRC | 23:57 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!