Friday, 2018-08-24

*** tbachman has quit IRC00:05
*** hamzy has quit IRC00:18
*** hamzy has joined #openstack-nova00:19
*** r-daneel has quit IRC00:28
*** tbachman has joined #openstack-nova00:28
*** r-daneel has joined #openstack-nova00:28
*** moshele has joined #openstack-nova00:30
*** tbachman has quit IRC00:33
*** tbachman has joined #openstack-nova00:34
*** claudiub has quit IRC00:39
*** gyee has quit IRC00:44
*** imacdonn has quit IRC00:50
*** imacdonn has joined #openstack-nova00:50
*** ChanServ has quit IRC01:00
*** ChanServ has joined #openstack-nova01:03
*** barjavel.freenode.net sets mode: +o ChanServ01:03
*** mriedem_afk has quit IRC01:05
*** r-daneel has quit IRC01:05
*** itlinux has quit IRC01:07
*** itlinux has joined #openstack-nova01:07
*** moshele has quit IRC01:08
*** mrsoul has quit IRC01:23
*** Nel1x has joined #openstack-nova01:28
*** Dinesh_Bhor has joined #openstack-nova01:41
*** rmk has quit IRC01:46
*** raginbajin has quit IRC01:46
*** rmk has joined #openstack-nova01:47
*** mmedvede has quit IRC01:48
*** ericyoung has quit IRC01:48
*** raginbajin has joined #openstack-nova01:49
*** ericyoung has joined #openstack-nova01:49
*** masayukig has quit IRC01:49
*** dansmith has quit IRC01:49
*** Dinesh_Bhor has quit IRC01:50
*** dansmith has joined #openstack-nova01:51
*** mmedvede has joined #openstack-nova01:52
*** swamireddy has joined #openstack-nova01:56
*** Dinesh_Bhor has joined #openstack-nova01:58
*** lei-zh has joined #openstack-nova02:01
*** brinzhang has joined #openstack-nova02:03
*** threestrands has quit IRC02:08
*** hongbin has joined #openstack-nova02:21
*** gcb_ has quit IRC02:24
*** Nel1x has quit IRC02:28
*** imacdonn has quit IRC02:28
*** hamzy has quit IRC02:28
*** panda has quit IRC02:28
*** penick has quit IRC02:28
*** jdillaman has quit IRC02:28
*** nicolasbock has quit IRC02:28
*** jlvillal has quit IRC02:28
*** dims has quit IRC02:28
*** mtreinish has quit IRC02:28
*** skatsaounis has quit IRC02:28
*** ircuser-1 has quit IRC02:28
*** s1061123 has quit IRC02:28
*** mnaser has quit IRC02:28
*** TheJulia has quit IRC02:28
*** Kevin_Zheng has quit IRC02:28
*** burt has quit IRC02:28
*** mgariepy has quit IRC02:28
*** mdbooth has quit IRC02:28
*** egonzalez has quit IRC02:28
*** slaweq has quit IRC02:28
*** Miouge has quit IRC02:28
*** mordred has quit IRC02:28
*** rtjure has quit IRC02:28
*** Nel1x has joined #openstack-nova02:30
*** imacdonn has joined #openstack-nova02:30
*** hamzy has joined #openstack-nova02:30
*** panda has joined #openstack-nova02:30
*** penick has joined #openstack-nova02:30
*** jdillaman has joined #openstack-nova02:30
*** nicolasbock has joined #openstack-nova02:30
*** egonzalez has joined #openstack-nova02:30
*** mtreinish has joined #openstack-nova02:30
*** dims has joined #openstack-nova02:30
*** jlvillal has joined #openstack-nova02:30
*** skatsaounis has joined #openstack-nova02:30
*** ircuser-1 has joined #openstack-nova02:30
*** s1061123 has joined #openstack-nova02:30
*** Kevin_Zheng has joined #openstack-nova02:30
*** mnaser has joined #openstack-nova02:30
*** TheJulia has joined #openstack-nova02:30
*** burt has joined #openstack-nova02:30
*** mgariepy has joined #openstack-nova02:30
*** mdbooth has joined #openstack-nova02:30
*** slaweq has joined #openstack-nova02:30
*** Miouge has joined #openstack-nova02:30
*** mordred has joined #openstack-nova02:30
*** rtjure has joined #openstack-nova02:30
*** openstack has joined #openstack-nova13:21
*** ChanServ sets mode: +o openstack13:21
mnaserwe have plenty of capacity of both, but i have a proposed solution where it just defaults to requiring hardware virt but allowing user to control that value if need be13:21
*** ttsiouts has joined #openstack-nova13:21
sean-k-mooneymnaser: right but that is not something you can do via an image today13:21
mnasersean-k-mooney: it is, using the `img_hv_type` property13:21
sean-k-mooneymnaser: that would be better handeled in a weigher13:21
mnaserif you set it to `kvm`, it'll filter only those that can do hardware acceleration13:22
sean-k-mooneymnaser: no that just allowas you to select the hypervior not the arch13:22
mnasersean-k-mooney: combined with hw_architecture :)13:22
mnaserand https://review.openstack.org/#/c/566425/13:22
*** openstackstatus has joined #openstack-nova13:22
*** ChanServ sets mode: +v openstackstatus13:22
*** openstackstatus has quit IRC13:22
sean-k-mooneyhw_arcitecture is menat to defien the arch of the vm.13:22
sean-k-mooneymnaser: your relying on the fact that today kvm acclerate does not work when you are emulating a different architecture13:23
openstackgerritChris Dent proposed openstack/nova master: DNM: Make nova tests pass with externalized placement  https://review.openstack.org/59629113:23
openstackgerritBalazs Gibizer proposed openstack/nova-specs master: Add subtree filter for GET /resource_providers  https://review.openstack.org/59523613:24
mnasersean-k-mooney: i don't think that's likely going to change anytime soon™13:24
*** openstackstatus has joined #openstack-nova13:24
*** ChanServ sets mode: +v openstackstatus13:24
sean-k-mooneymnaser: well my first respocne to that patch is we should revert it with out a spec13:24
mnaserand that would mean we can't deliver multiarchitecture in our cloud13:25
mnaserso i'm not sure how that benefits anyone13:25
*** gbarros has quit IRC13:25
mnaserthe patch doesn't hurt anyone except help users deliver a better openstack experience13:25
sean-k-mooneymnaser: you can you just need a weigher.13:25
mnaserdoesnt the scheduler filter out N number of items to weight13:26
sean-k-mooneymnaser: my issue is you did this in a filter so its a hard requirement instead of a perference13:26
mnaserit is not13:26
mnaserif the hw_architecture field is empty13:26
mnaseror there is no default configured value13:26
mnaserthere is no enforcement13:26
sean-k-mooneyyes but i cant set hw_architecture and hv_type anymore13:27
mnaseryes you can13:27
mnaserthis patch only uses the configured default IF none is specified in the image.13:27
mnaseryou are more than welcome to set hw_arch to aarc64 and hv_type to qemu13:27
mnaserand you'll get software virtualized arm64 vm13:27
mnaserthis is just the default values if *none* are specified, which the current openstack nova behavior = put it anywhere.13:27
sean-k-mooneyso i can set hw_architecture=x86 and hv_type=kvm and land on a powerkvm node?13:28
*** tbachman has quit IRC13:28
mnasersean-k-mooney: no, if no values are set, you can end up anywhere yes13:28
mnaserbut the thing is, 99% of our customers *dont* send hw_arch and hv_type13:28
mnaserbecause they assume that the cloud is x86_64/kvm which is a reasonable assumption.13:29
sean-k-mooneymnaser: its not how rackspace used to run there public cloud13:29
mnasermultiarchitecture clouds are rare, so asking every single customer/user to tag their images with an architecture is not really smooth user experience, especially in a transition13:29
mnasersure.  so you're saying that for someone to upload an image into openstack, we need to start asking for 3 more parameters13:30
sean-k-mooneythey had a large amount of powerpc nodes at one point with xen as the hypervior running with x86 guests13:30
*** eharney has joined #openstack-nova13:30
mnaserwell, i'd say that's more of an outlier. i don't see how this helps solve things and make them easier for users overall in a multiarch environment13:31
mnaserbecause the rax env you're describing sounds still like a single arch one for the user facing stuff13:31
sean-k-mooneymnaser: well my point is if you implemented this as a weigher you would land on the same acrh in 99% of cases and fall back to hardware acllerated emulaton if not native nodes were available.13:32
mnasersean-k-mooney: but don't weighers run on a subset of compute nodes?13:33
sean-k-mooneythe chage is merged in anycase but i really dont think this should have been a bug.13:33
sean-k-mooneymnaser: am the run after all the filters13:33
sean-k-mooneyso you go through placment then the filters then the weighers13:33
mnaseri was pretty sure that weighers ran on a subset of hosts13:33
sean-k-mooneythey just reorder what the filters pass13:33
mnaserhence the values that would let you pick subsets of hosts when scheduling13:34
sean-k-mooneysorry i missed that last point?13:35
mnaseri thought weighers only ran on a subset of filtered hosts13:36
sean-k-mooneyyes13:36
mdboothmriedem: Briefly back to the topic of evacuation tests, am I right in thinking we probably won't be able to write a full job which does failure injection? If not, I don't think I don't think I'm blocked on that. If we *can* I'm happy to write it if you point me at the right repo and an example to crib from.13:37
*** Luzi has quit IRC13:37
mnasersean-k-mooney: was rax doing x86 on ppc via software or hardware emulation?13:37
mdbooth'example' wouldn't have to be that similar, just enough to orient myself. I haven't touched the top-level CI jobs before.13:38
sean-k-mooneymnaser: i belive xen allows cross arch hardware emulation13:38
mdboothmriedem: The ability to assert success in the non-failure case might still be interesting, I guess. It would essentially be a regression test.13:40
mnasersean-k-mooney: i am not too familiar with xen but i would assume that wouldn't be hardware assisted virt (i.e. using VT-x in intel, etc)13:40
mnaserbut that's something i dunno much about so i could be entirely wrong13:40
sean-k-mooneymnaser: well xen does use vt-x normally if you are running x86 on x8613:41
sean-k-mooneyas does kvm13:41
mnaserright, so this would be the same as software virtualization which can be really slow, which i'm trying to avoid having anyone end up with13:42
mnaserand all of this is optional, if you set none of those settings, it won't affect you.  but if you run a cloud where you want to make sure all VMs are hardware virt'd, then you can do that13:42
*** antonym_ is now known as antonym13:43
sean-k-mooneymnaser: its not so much that this change was done that i object to its that the semantics of the existing api were changed without a spec.13:47
mnasersean-k-mooney: but if you don't configure this very optional value, there will zero behavior change13:47
sean-k-mooneymnaser: yes and if you had those configure before an you upgrade then your cloud may now be broken if they do not have the same semantics13:48
mnasersean-k-mooney: you couldn't possible have them configured because they don't even exist before?13:48
mnasers/possible/possibly13:49
sean-k-mooneythe image properties13:49
openstackgerritStephen Finucane proposed openstack/nova master: api: Remove unnecessary default parameter  https://review.openstack.org/56445113:50
*** mlavalle has joined #openstack-nova13:50
*** itlinux-away has quit IRC13:53
*** awaugama has joined #openstack-nova13:53
*** gbarros has joined #openstack-nova13:53
*** gbarros has quit IRC13:54
*** gbarros has joined #openstack-nova13:55
mriedemmdbooth: what type of failure needs to be injected? like delete the disk before starting the evacuatoin?13:58
mdboothmriedem: Any failure in spawn on the destination13:59
mriedemcan't really do that from tempest, but could if run from a script13:59
mdboothI think I chose plug_vifs in my functional test13:59
dansmithmriedem: so on your gate job,13:59
dansmithmriedem: just force-downing the service isn't going to be enough I think, since it will still be running, potentially mucking with allocations and state correction, etc right?14:00
mdboothJust need to assert that if we fail in spawn in the dest, we only cleanup what we're supposed to and don't delete shared data.14:00
mdboothWhether that's shared instance dir or shared block.14:00
dansmithyou have to actually kill the n-cpu process and force-down it to avoid the wait14:00
dansmith(well, not mucking with allocations until it's deleted maybe anymore, but screaming at least)14:00
* mdbooth -> meeting14:01
dansmithbut I think it's more legit if you stop it14:01
mriedemdansmith: force down is what we do in functional tests14:01
mriedemthat's all the API checks for14:01
dansmithmriedem: I know it will make evac work,14:01
dansmithbut the n-cpu for the "down" one will still be running and doing stuff, which isn't how we describe evac as working properly in real life14:02
dansmithwe say you have to fence it14:02
*** jaosorior has joined #openstack-nova14:02
mriedemin a multinode job the post-test script runs on the control node which has an n-cpu process on it which we could kill14:02
mriedemwe'd have to force the server to that host so it's the source14:03
stephenfinmriedem, melwitt, dansmith: btw, given that sahid immediately replied yesterday, I figured it wasn't necessary to investigate the queue size thing any further myself. Sorry if I picked that up wrong14:03
mriedemstephenfin: it's fine14:03
mriedemafter looking at the existing release note and config option help for the rx/tx queue size stuff i think they were good enough to say that it was very limited to certain types of vifs14:04
mriedemand i think what moshe is trying to do is make it work for something else14:04
*** r-daneel has joined #openstack-nova14:04
stephenfinmriedem: aight, good to hear14:05
melwitt.14:06
mriedemmdbooth: btw yes i'm also interested in simply have regression testing for evac in a real env14:06
mriedem*having14:06
mriedemmelwitt: that last rocky change is in the gate14:07
melwittmriedem: I saw, thank you for getting that. the master change didn't land until late last night after I went to sleep14:09
*** med_ has joined #openstack-nova14:13
mriedemmdbooth: btw do you still care about this? https://review.openstack.org/#/c/572790/14:13
*** efried_afk is now known as efried14:16
*** efried is now known as senhor_granhular14:24
*** med_ has quit IRC14:28
*** jaosorior has quit IRC14:33
*** itlinux has joined #openstack-nova14:35
*** senhor_granhular is now known as fried_rice14:37
mriedemdansmith: a couple of thoughts on batch size floor in https://review.openstack.org/#/c/593131/14:38
*** pooja_jadhav has joined #openstack-nova14:38
*** cfriesen has joined #openstack-nova14:38
*** dpawlik_ has quit IRC14:39
dansmithmriedem: thanks will look shortly14:41
*** moshele has joined #openstack-nova14:44
*** ociuhandu has joined #openstack-nova14:44
*** ttsiouts has quit IRC14:46
*** links has quit IRC14:52
*** jlvillal is now known as jlviva-viva14:52
*** ociuhandu has quit IRC14:53
*** itlinux has quit IRC14:55
*** ykarel is now known as ykarel|away14:55
mriedemand i'm done with that stack \o/14:57
*** pcaruana has quit IRC14:59
mdboothmriedem: I do. Came up yesterday, in fact.15:02
mdboothmriedem: I don't think I had any strong objections there?15:02
mriedemi'm pretty sure you wanted to +1 it15:03
mdboothJust re-reading my last review. Sounds like I was prevaricating rather than objecting.15:04
*** r-daneel has quit IRC15:04
mdboothAh... the attach_mode thing.15:06
*** r-daneel has joined #openstack-nova15:06
*** ttsiouts has joined #openstack-nova15:07
*** Bhujay has quit IRC15:08
*** itlinux has joined #openstack-nova15:09
mdboothmriedem: So there's attach_mode and access_mode15:13
* mdbooth goes looking at cinder docs15:13
mdboothIt seems like we only *use* access_mode, but that code is checking attach_mode15:14
*** r-daneel_ has joined #openstack-nova15:18
*** r-daneel has quit IRC15:19
*** r-daneel_ is now known as r-daneel15:19
*** nicolasbock has joined #openstack-nova15:21
*** imacdonn has joined #openstack-nova15:22
openstackgerritClaudiu Belu proposed openstack/nova master: hyper-v: autospec classes before they are instantiated  https://review.openstack.org/34221115:23
openstackgerritClaudiu Belu proposed openstack/nova master: WIP: replace spec with autospec  https://review.openstack.org/55729915:23
mdboothmriedem: Who's a friendly cinder person?15:29
*** macza has joined #openstack-nova15:29
*** ykarel|away has quit IRC15:30
mriedemfor attachment record stuff it would be jgriffith15:31
*** whoami-rajat has quit IRC15:32
*** gbarros has quit IRC15:33
mdboothI've hopefully added the right one.15:33
*** giblet is now known as giblet_off15:34
giblet_offI will be back on Tuesday15:34
*** gbarros has joined #openstack-nova15:34
mdboothdansmith: Is your no timeouts messaging thing in Queens, btw? Or backportable to Queens?15:38
*** ShilpaSD has quit IRC15:45
*** ttsiouts has quit IRC15:46
mriedemit is not in queens15:46
mriedemand not really backportable15:46
mriedemas that would require a min required version of oslo.messaging15:46
mriedemand we don't bump mins on libs in stable15:46
mdboothmriedem: ack, thanks.15:47
mdboothmriedem: Means I can't promise a fix to a customer in our next stable release :/15:47
sean-k-mooneymdbooth: without it could you just set the time out really high to get a similar effect?15:48
mdboothsean-k-mooney: Yeah.15:48
mdboothsean-k-mooney: I still need to look way harder at the impact, anyway.15:49
mriedemmdbooth: you can fork whatever you want15:50
*** pooja_jadhav has quit IRC15:50
mdboothmriedem: Well I would, but dansmith won't let me.15:50
mdboothI was just going to rewrite in Go.15:51
*** jpena is now known as jpena|off15:53
mdboothIn all seriousness, though: pretty sure you were joking, but in case not we don't do that. At least not in any incompatible way.15:53
mdboothIf we can't do it, that's the way it is.15:54
*** tssurya has quit IRC15:54
*** Sundar_ has joined #openstack-nova15:58
*** Bhujay has joined #openstack-nova16:01
*** dtantsur is now known as dtantsur|afk16:01
*** itlinux has quit IRC16:02
*** manjeets has joined #openstack-nova16:03
*** itlinux has joined #openstack-nova16:06
*** sahid has quit IRC16:08
*** munimeha1 has joined #openstack-nova16:10
*** gbarros has quit IRC16:13
openstackgerritStephen Finucane proposed openstack/nova master: doc: Note NUMA topology requirements for numa-aware-vswitches  https://review.openstack.org/59639316:14
*** itlinux is now known as itlinux-away16:16
mdboothmelwitt: Do you remember anything about either adding or removing refresh_conn_info from hard reboot?16:16
*** itlinux-away is now known as itlinux16:16
mdboothIt's vaguely ringing a bell, but I can't find any evidence of it.16:16
melwittmdbooth: no, I never understood much around that. lyarwood would be the one to ask16:17
melwittmight be a WIP patch from him or something? not sure16:17
openstackgerritMerged openstack/nova stable/rocky: Correct the release notes related to nova-consoleauth  https://review.openstack.org/59589016:18
openstackgerritMerged openstack/nova master: tests: Move mocking to setUp  https://review.openstack.org/59580216:18
*** r-daneel_ has joined #openstack-nova16:18
*** udesale has quit IRC16:19
*** gyee has joined #openstack-nova16:19
*** r-daneel has quit IRC16:20
*** r-daneel_ is now known as r-daneel16:20
mdboothmelwitt: I think the change I'm thinking of is when we added the call to destroy() which calls cleanup() and therefore does other stuff like detaching volumes. We don't seem to refresh_conn_info, though16:20
mdboothmriedem: Is refresh_conn_info still relevant in the new world order?16:21
* mdbooth suspects it is16:21
melwittmdbooth: ah yeah, the "redo all the things" change16:21
mdboothmelwitt: Yeah.16:21
melwittdidn't have anything to do with refresh_conn_info tho (as you saw)16:22
mdboothI'm trying to come up with a recovery recommendation for this pre_live_migration timeout.16:22
mdboothThe bug is: we called cleanup on the source when the dest timed out, even though it continued to run. The result is that the instance still has all volumes connected on the source where it continues to run, but some bdms have conn_info for source and others for dest.16:23
mdboothSo really we want some operation which would cause a refresh_conn_info.16:24
mdboothOptions seem to be cold and live migrations.16:24
mdboothWonder if reboot ought to be another.16:24
mriedemmdbooth: refresh_conn_info with the new attachment record flow isn't the same as how things worked with the old flow16:25
smcginnismelwitt: Looks like that last stable/rocky patch merged, and you were right that the hash needs to be updated.16:25
mdboothmriedem: I'll take a look.16:26
melwittsmcginnis: doing it right now, thanks16:26
mriedemmdbooth: https://review.openstack.org/#/c/584648/20/nova/scheduler/client/report.py@2055 the difference is in the old flow, we'd always create a new connection to the storage backend which gives us a fresh connection_info from cinder,16:26
mriedemwith the new flow, we are getting the attachment record which has a stashed connection_info, which might be potentially stale16:26
mriedemwhich is what i think you're worried about16:26
mriedembtw this is related https://review.openstack.org/#/c/579004/16:27
mdboothmriedem: Ah, good point. So in that case attachment_id also be out of date.16:27
mdboothFrom that commit message: Note: Only hard reboots will currently update connection_info.16:28
mdboothI wonder if I missed something16:28
mriedemthat's just a result of the patch16:28
mriedembut i think the comment is wrong16:29
mdboothmriedem: That's super-useful, thanks.16:30
mriedemi am both very useful and very handsome16:30
mriedemi think we can all agree16:30
mdboothmriedem: Your modesty is also unparalleled in all of history ;)16:31
mriedemthat's very kind of you16:32
mriedemso,16:32
mriedemwithout digging into the logs,16:32
mriedemi expect this is blowing up b/c attachment_update isn't meant to be used this way once it's already got a connector16:32
mriedemlikely should be creating a new attachment record with new connector (and id) and deleting the old one16:32
*** gbarros has joined #openstack-nova16:34
*** gbarros has quit IRC16:37
mriedemcommented and pinged jgriffith16:38
mdboothOh, the irony!16:38
mdboothSo, if connection_info is wrong that's obviously going to break any attempt to attach16:39
mdboothBut in this particular case because the volumes are already attached they're already working16:39
*** fried_rice is now known as fried_rolls16:39
mdboothBut since the robustification patch added detach/attach in hard reboot, we actually broke it16:40
mdboothIt needs to be combined with refresh_conn_info16:40
openstackgerritDan Smith proposed openstack/nova master: Batch results per cell when doing cross-cell listing  https://review.openstack.org/59269816:40
openstackgerritDan Smith proposed openstack/nova master: List instances from all cells explicitly  https://review.openstack.org/59371716:41
openstackgerritDan Smith proposed openstack/nova master: Make instance_list perform per-cell batching  https://review.openstack.org/59313116:41
openstackgerritDan Smith proposed openstack/nova master: Record cell success/failure/timeout in CrossCellLister  https://review.openstack.org/59426516:41
openstackgerritDan Smith proposed openstack/nova master: Optimize global marker re-lookup in multi_cell_list  https://review.openstack.org/59457716:41
mriedemumm,16:41
mriedemwell,16:41
mriedemwe've talked about this hard refresh thing for a long time b/c of the ceph ip or auth changing and not wanting to live migrate your entire cloud to update the attachments16:42
mriedemi think coreycb is trying to fix this one case for reboot, and then see if we can extend to other operations16:44
mriedem"This only includes a fix for the 'openstack server reboot --hard' path  as I didn't want to get too deep before knowing if this is a proper fix.  A complete fix would also provide soft reboot, stop/start support,  perhaps more?"16:44
*** gbarros has joined #openstack-nova16:45
coreycbmriedem: thanks for your comments. i dropped that patch for a bit but will get back to it once we get rocky done.16:46
mdboothmriedem: I'm coming from a different angle. A weird unfixable bug means I've now got inconsistent state, and I want a thing to make everything right again.16:47
mriedemmdbooth: is that related to your other live migration patch?16:48
mriedemhttps://review.openstack.org/#/c/551302/16:48
mriedembw, you probably need some more spinning plates16:49
mdboothmriedem: No.16:50
openstackgerritDan Smith proposed openstack/nova master: Optimize global marker re-lookup in multi_cell_list  https://review.openstack.org/59457716:51
mdboothmriedem: I thought about that one, but I don't think it would fix this.16:51
mdboothI should resurrect that patch as being just a generally good idea, even if the original motivation for it has gone. It would have fixed a couple of other things since iirc.16:51
stephenfinmriedem: I'm not sure what I can do about this https://review.openstack.org/#/c/588570/16:53
stephenfinmriedem: Have tried drafting a functional test but it looks like it's going to require a good deal of changes to the fake libvirt driver, assuming it's even possible to validate16:54
mriedemmdbooth: fwiw i think i was ok with that change but wanted to see the same rollback stuff done in a follow up first16:55
mriedemstephenfin: ok...i'm about out of here for lunch and errands, so probably need to hit me next week when i'm in more of a "f it" kind of mood16:56
stephenfinmriedem: Heh, fair16:57
mriedembecause this definitely seems like new feature capability16:57
mriedemif it takes this much to fix a thing, it probably never worked in the first place right?16:57
mriedemthe bug says a certain configuration is not "optimal"16:58
stephenfinI think it was more a case of encoding faulty assumptions16:58
mriedemthis is optimizing things yes?16:58
stephenfinnamely, that asymmetric host topologies weren't a thing16:58
*** holser_ has quit IRC16:59
stephenfinYeah, optimization is probably fair16:59
stephenfinwithout the patch, you see issues when your hosts get a bit congested16:59
stephenfin...and you're using emulator threads16:59
mriedemwell, given (1) how large this is and multiple changes and (2) it sounds low priority bug-wise if it's not outright failing, and (3) i'm in no way an expert on this code, i'm hesitant to even get close to a backport of this17:00
stephenfinAlso fair17:00
mriedemif dan puts on his nfv shoes maybe he'd care17:00
mriedemidk17:00
*** med_ has joined #openstack-nova17:00
stephenfinI'll see if I can rope leakypipes into it next week, given that he reviewed it first time round17:00
stephenfinIf not...I'll think of something17:01
stephenfinAight, home time o/17:01
melwitthow do nova.conf get used with services running under uwsgi? how do you specify which conf17:02
*** mriedem is now known as mriedem_afk17:02
dansmithmelwitt: depends on what wsgi container you're using I think17:05
melwittdansmith: what does that mean? which nova service? or something external to that?17:07
dansmithno, like apache, ngnix, and then which wsgi module on each you use (uwsgi, etc)17:08
melwittoh, ok17:08
*** mdbooth has quit IRC17:09
dansmithI don't know much about uwsgi, so not sure where to point you to for it17:09
melwittcontext is, I was thinking this proposed devstack change is wrong https://review.openstack.org/#/c/564978/2/lib/nova but then, I don't get how the standalone metadata API service is currently working with querying instance metadata, for example, in the neutron-tempest-multinode-full job. it must be using the indirection api because it's standalone, but then how is it accessing the cell database?17:10
dansmithwith mod_wsgi you have a little app definition python file which is where you'd do it (or, I'd do it)17:10
melwittand I can't see how to see what nova.conf the uwsgi metadata service is running with17:10
melwittI see17:10
dansmithdo we run the standalone metadata api under wsgi? I don't think we do17:11
*** macza has quit IRC17:11
dansmitheven still,17:11
dansmiththe indirection_api thing isn't part of config so I don't think there's anything inconfig that is relevant17:12
dansmithhttps://github.com/openstack/nova/blob/master/nova/cmd/api_metadata.py#L45-L4517:12
dansmiththat's where it gets forced on for the standalone one17:12
melwittokay, maybe I got confused by something. because there's a separate n-api-metadata file, but that doesn't mean it's running standalone necessarily right?17:12
dansmithfile meaning.. log file?17:12
melwittI was thinking it meant it's standalone17:12
melwittyeah the log file17:12
dansmithyeah, but that doesn't mean it's running under wsgi...17:13
melwittI can see it's running under wsgi from the ps.txt.gz here http://logs.openstack.org/90/595890/2/check/neutron-tempest-multinode-full/a4a0803/logs/ps.txt.gz17:13
melwittbut I'm not sure how to tell whether it's the standalone service or the all-in-one17:13
*** macza has joined #openstack-nova17:13
dansmithapparently we are17:14
dansmithhttp://logs.openstack.org/77/594577/6/check/neutron-grenade/f10f831/logs/etc/nova/nova-metadata-uwsgi.ini.txt.gz17:14
dansmiththat's the uwsgi definition for it17:14
melwittoh yeah, that too17:14
dansmithnot sure if we capture that /usr/local/bin/nova-api-metadata-wsgi file though17:15
dansmithmaybe that's auto-generated17:15
melwittah, so maybe it's set in devstack somewhere17:15
dansmithbut again, what in the config are you looking for related to the indirection api?17:16
dansmithI'm not sure what you meant above about "it's standalone so much be using the indirection api"17:17
*** gbarros has quit IRC17:17
openstackgerritDan Smith proposed openstack/nova master: Batch results per cell when doing cross-cell listing  https://review.openstack.org/59269817:18
melwittno, sorry, I'm trying to determine whether that proposed change to devstack is right or not. it looks like it's trying to make the standalone service default to the cell database so it can find instance metadata. in devstack, it looks like if services_enabled includes n-api-meta, it is doing the standalone service which we said is not supposed to be used with multiple cells17:18
openstackgerritDan Smith proposed openstack/nova master: List instances from all cells explicitly  https://review.openstack.org/59371717:18
openstackgerritDan Smith proposed openstack/nova master: Make instance_list perform per-cell batching  https://review.openstack.org/59313117:18
openstackgerritDan Smith proposed openstack/nova master: Record cell success/failure/timeout in CrossCellLister  https://review.openstack.org/59426517:18
openstackgerritDan Smith proposed openstack/nova master: Optimize global marker re-lookup in multi_cell_list  https://review.openstack.org/59457717:18
melwittthen, I looked at an existing multinode job to see, how is the metadata API service currently working then without that change?17:19
dansmithwell, I think that change isn't right,17:19
dansmithbecause it means we won't do multi-cell mode _unless_ the standalone metadata service is enabled right?17:20
dansmithwhich is clearly not what we want17:20
melwitthere, it looks like the neutron multinode job is running with the standalone meta service http://logs.openstack.org/90/595890/2/check/neutron-tempest-multinode-full/a4a0803/logs/devstacklog.txt.gz#_2018-08-23_21_19_27_916 so how is it finding metadata? just confused17:20
*** dpawlik has joined #openstack-nova17:20
melwittoh, I was thinking that was an OR. my mistake I think17:21
dansmithAug 23 21:29:00.426509 ubuntu-xenial-inap-mtl01-0001520794 devstack@n-api-meta.service[4018]: DEBUG nova.api.openstack.wsgi_app [None req-848ebec7-678d-4d46-923a-2b41bf306753 None None] config files: ['/etc/nova/api-paste.ini', '/etc/nova/nova.conf'] {{(pid=4020) log_opt_values /usr/local/lib/python2.7/dist-packages/oslo_config/cfg.py:3009}}17:21
melwittoh, thank you17:21
dansmiththat standalone service is running from nova.conf17:22
melwittok, and nova.conf is pointing at the cell0 database17:22
dansmithwell, but,17:22
dansmithit's using indirection, which means it goes to superconductor17:23
dansmithwhich should be pointing at cell0 too17:23
*** gbarros has joined #openstack-nova17:23
melwittyeah... so shouldn't getting metadata not work? that's what's confusing me. but I see requests going successfully in the meta API log17:23
dansmithsorry, I'm not being clear17:24
mnaserthrowing this out here: how does barbican stable/rocky look right now? is it pretty much likely going to be the released version? (i.e. will/how much i get in trouble for deploying a new region on stable/rocky)17:24
mnaser:p17:24
mnaserer17:24
mnasers/barbican/nova/17:24
dansmithmelwitt: the standalone runner would be using indirection_api, going to superconductor and hitting cell0 and failing17:24
*** dpawlik has quit IRC17:25
dansmithmelwitt: but we're not running _our_ standalone metadata service, we're running metadata api as non-standalone, but in its own wsgi container17:25
melwittdansmith: oh, geez, okay17:25
dansmithso it's behaving like the combined global api service would have previously when we smashed them all together,17:25
dansmithbut with wsgi you yank them into life a little differently, without the nova-api service bootstrapping code17:26
dansmithwhich means it should be looking up the mapping for instances and talking to the right cell, like in global mode17:26
*** NostawRm has joined #openstack-nova17:26
melwittcray cray17:26
dansmithcalling it "non-standalone" is a bit of a misnomer, but compared to our old combined service bootstrapper thing, it's in "non-standalone" mode of operation17:27
dansmithbut.. it's fairly standalone in that it's running all by itself and has its own log file17:27
melwittdansmith: okay, so what is this then here? this looks like it means if service enabled n-api-meta, then remove the metadata from nova_enabled_apis and run it is _our_ standalone service. right? https://review.openstack.org/#/c/564978/2/lib/nova@47017:27
*** itlinux is now known as itlinux-away17:27
melwitt*as17:27
dansmiththat's probably now confused,17:28
dansmithas that used to be how devstack referred to *our* standalone service, and may still in some cases, idk, but now post-systemd and post-wsgi probably refers to the, uh, other one :)17:28
melwittmnaser: we think so, but we also thought we'd only have one RC and now we have 317:29
*** gbarros has quit IRC17:29
mnasermaybe ican help you make it 4, hah :P17:29
dansmithmelwitt: why are you even reviewing this with no details in the bug or commit message?17:29
*** itlinux-away is now known as itlinux17:29
*** itlinux is now known as itlinux-away17:29
mnaserthats reasonable.  ill proceed being careful and give input :)17:29
*** itlinux-away is now known as itlinux17:30
*** itlinux is now known as itlinux-away17:30
*** gbarros has joined #openstack-nova17:30
dansmithmelwitt: mriedem_afk: is this known? http://logs.openstack.org/71/594571/2/gate/nova-tox-functional-py35/fd2d9ac/testr_results.html.gz17:30
melwittmnaser: haha :) always happy to get your testing of the code, and sorry we've been behind on the cells orphaned records patch reviews. things have been just really crazy17:31
dansmithfailed on both jobs of one of my patches17:31
mnasermelwitt: nah its cool, my workaround for that was make sure our underlying infra doesnt break and thats ok enough :p17:31
mnaseri dont think things that break after unrelated nova issues would be priority number 1, imho17:31
melwittthanks for the understanding17:32
*** _hemna has joined #openstack-nova17:32
melwittdansmith: that's the first time I've seen that17:32
melwitthm, weird FileNotFoundError: [Errno 2] No such file or directory: '/home/zuul/src/git.openstack.org/openstack/nova/instances/.directio.test'17:33
melwittit tries to write a file to test for directio support17:34
*** itlinux-away has quit IRC17:35
melwittand that failed. does that mean the nova/instances/ dir missing from the test tree probably? hm17:35
dansmithjust not sure what would have changed17:36
melwittI can't remember if that was one of those wonky directories that get created during test runs, to put temp files in for tests17:36
melwittlike, what is that instances/ directory, I can't remember17:37
melwittoh wait, it's that the actual code is trying to write that file in a real life scenario and in the test env there is no nova/instances dir. but I don't know how that used to be working if it was working before17:37
melwittin func test land17:38
*** med_ has quit IRC17:41
melwittCONF.instances_path17:46
*** fried_rolls is now known as fried_rice17:47
*** eharney has quit IRC17:53
*** gbarros has quit IRC17:56
jrock_cfdgcould someone please help out? Trying to add a serial port with source mode='connect' to do low-level kernel debugging in windows instance; everytime i add device to instance it shuts down - is this even possible with nova? i can do this in libvirt w/out issue18:05
Sundar_fried_rice, efried: Do you have a moment?18:08
fried_riceSundar_: sure.18:08
openstackgerritMerged openstack/nova master: Make CELL_TIMEOUT a constant  https://review.openstack.org/59457018:08
fried_riceSundar_: I *am* trying to get to reviewing your spec really soon18:08
fried_riceSundar_: I've just been inundated with other things.18:09
Sundar_In the scheduling spec, we defined a way to extend the granular resource request syntax to support Cyborg: https://git.openstack.org/cgit/openstack/cyborg/tree/doc/specs/rocky/cyborg-nova-sched.rst?h=refs/changes/17/554717/18#n31518:09
Sundar_Just want to make sure you have reviewed it.18:09
Sundar_fied_rice: NP :) I understand. We all have to roll on different tracks at the same time.18:10
Sundar_And I may have added yet another track for you to roll on.18:11
fried_riceSundar_: I remember seeing this and being okay with it, but I would have to do a major context switch to be able to discuss it intelligently at this point.18:12
fried_riceI noticed you said the extended keys aren't interpreted by nova. Presumably that means they're interpreted by cyborg. How does that information get to cyborg? And does it translate to some new/different placement syntax? I assume not, that it just tells cyborg how to do certain things.18:14
*** cdent has quit IRC18:14
Sundar_fried_rice: Good to know that it is accepted. Yes, they will be interpreted by Cyborg (not sure if os-acc will also interpret it). The extra specs in the request are passed from Nova virt to os-acc to Cyborg.18:17
Sundar_No new placement APIs needed.18:17
melwittI have to run to an appointment, will be back later18:19
openstackgerritGhanshyam Mann proposed openstack/nova master: Merge security groups extension response into server view builder  https://review.openstack.org/58547518:34
openstackgerritGhanshyam Mann proposed openstack/nova master: Merge extended_status extension response into server view builder  https://review.openstack.org/59209218:35
*** moshele has quit IRC18:39
openstackgerritMerged openstack/nova master: tests: Create functional libvirt test base class  https://review.openstack.org/40705518:41
*** mriedem_afk is now known as mriedem18:45
*** eharney has joined #openstack-nova18:49
mriedemdansmith: i haven't seen that functional test failure18:50
*** gbarros has joined #openstack-nova18:58
mriedemdansmith: umm, could https://review.openstack.org/#/c/407055/ that just merged be a regression?19:00
mriedemor the change below it19:02
mriedemhttp://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22FileNotFoundError%3A%20%5BErrno%202%5D%20No%20such%20file%20or%20directory%3A%20'%2Fhome%2Fzuul%2Fsrc%2Fgit.openstack.org%2Fopenstack%2Fnova%2Finstances%2F.directio.test'%5C%22%20AND%20tags%3A%5C%22console%5C%22&from=7d19:02
mriedemjust started19:03
mriedemso those are probably it19:03
*** Bhujay has quit IRC19:04
mriedemhttps://bugs.launchpad.net/nova/+bug/178892219:05
openstackLaunchpad bug 1788922 in OpenStack Compute (nova) "SRIOVServersTest.test_create_server_with_VF intermittently fails due to "FileNotFoundError: [Errno 2] No such file or directory: '/home/zuul/src/git.openstack.org/openstack/nova/instances/.directio.test'"" [Undecided,New]19:05
mriedem-1ed https://review.openstack.org/#/c/564978/ which it looks like you and melwitt were talking about - the change has no context19:07
mriedem"this fixes something"19:07
*** Sundar_ has quit IRC19:10
*** mrjk has quit IRC19:13
mriedemoh you know, this func test might only be failing on one node provider19:14
mriedembut that means we aren't mocking something out properly and accidentally passing on other hosts19:14
mriedemmaybe19:14
openstackgerritEric Fried proposed openstack/nova master: [placement] split gigantor SQL query, add logging  https://review.openstack.org/59004119:23
*** erlon has quit IRC19:25
*** dpawlik has joined #openstack-nova19:39
*** mriedem has quit IRC19:40
*** dpawlik has quit IRC19:44
*** mriedem has joined #openstack-nova19:48
mriedemmelwitt: question in https://review.openstack.org/#/c/540258/19:49
*** mrjk has joined #openstack-nova19:53
*** eandersson has joined #openstack-nova20:06
openstackgerritMerged openstack/nova master: Stash the cell uuid on the context when targeting  https://review.openstack.org/59457120:09
mriedemmelwitt: have you thought about an agenda/schedule for the ptg? just thought of it since gorka was asking in the thread on cross-cell migration20:11
*** fried_rice is now known as efried_afk20:13
*** itlinux has joined #openstack-nova20:14
*** efried_afk is now known as fried_rice20:14
mriedemdansmith: i must not be understanding this comment https://review.openstack.org/#/c/592698/12/nova/tests/unit/compute/test_multi_cell_list.py@11820:23
dansmithmriedem: copied from L89 above, and yes20:24
dansmithit means that since key0 is the same on both it should actually sort the two instances by key120:25
dansmithbut since I copied it for this test and only have one instance it obviously doesn't need to be htere20:25
mriedemyeah...20:26
mriedemi was like,20:26
mriedembut it's only 1 record...20:26
mriedemcleanup patch on top of the series?20:26
dansmithwhatever you want20:27
dansmithif we're blocked on that functional test, I might as well just fix it20:28
openstackgerritDan Smith proposed openstack/nova master: Batch results per cell when doing cross-cell listing  https://review.openstack.org/59269820:29
openstackgerritDan Smith proposed openstack/nova master: List instances from all cells explicitly  https://review.openstack.org/59371720:29
openstackgerritDan Smith proposed openstack/nova master: Make instance_list perform per-cell batching  https://review.openstack.org/59313120:29
openstackgerritDan Smith proposed openstack/nova master: Record cell success/failure/timeout in CrossCellLister  https://review.openstack.org/59426520:29
openstackgerritDan Smith proposed openstack/nova master: Optimize global marker re-lookup in multi_cell_list  https://review.openstack.org/59457720:30
*** tbachman has joined #openstack-nova20:35
mriedemshouldn't be blocked http://status.openstack.org/elastic-recheck/#178892220:36
mriedembut logstash might be behind20:36
dansmithI had both functional jobs fail in one thing so it seemed like it was high frequency but okay20:36
mriedemyeah it might be higher fail rate than logstash/e-r is seeing right now20:40
mriedemor my query is missing something20:40
mriedemhmm, looking at this i think my query is only going to hit py27 jobs20:41
mriedemno i guess its hitting the opposite, only py3520:42
mriedem+2s up the board20:57
*** itlinux has quit IRC20:58
*** marvin_mhg has quit IRC21:04
openstackgerritEric Fried proposed openstack/nova master: Report client: Real get_allocs_for_consumer  https://review.openstack.org/58459921:13
openstackgerritEric Fried proposed openstack/nova master: Report client: get_allocations_for_provider_tree  https://review.openstack.org/58464821:13
openstackgerritEric Fried proposed openstack/nova master: Report client: _reshape helper, placement min bump  https://review.openstack.org/58503421:13
openstackgerritEric Fried proposed openstack/nova master: Report client: update_from_provider_tree w/reshape  https://review.openstack.org/58504921:13
openstackgerritEric Fried proposed openstack/nova master: Compute: Handle reshaped provider trees  https://review.openstack.org/57623621:13
*** munimeha1 has quit IRC21:15
openstackgerritMatt Riedemann proposed openstack/nova master: api-ref: fix volume attachment update policy note  https://review.openstack.org/59648921:29
*** awaugama has quit IRC21:36
*** elod has quit IRC21:42
openstackgerritMatt Riedemann proposed openstack/nova master: api-ref: add a warning about calling swap volume directly  https://review.openstack.org/59649221:47
openstackgerritEric Fried proposed openstack/nova master: Document no content on POST /reshaper 204  https://review.openstack.org/59649421:49
openstackgerritEric Fried proposed openstack/nova master: Compute: Handle reshaped provider trees  https://review.openstack.org/57623621:53
openstackgerritMatt Riedemann proposed openstack/nova master: Add functional test for live migrate with anti-affinity group  https://review.openstack.org/58893521:53
*** eharney has quit IRC22:07
openstackgerritEric Fried proposed openstack/nova master: Fix race condition in reshaper handler  https://review.openstack.org/59649722:07
melwittmriedem: thanks. I was going to make a similar comment on the devstack change but now I don't have to22:15
mriedemtroll proxy22:16
melwitthaha22:16
melwittmriedem: yeah, thinking monday I'll try and schedule some times slots for nova/cinder, nova/ironic, nova/neutron (assuming we need that to)22:17
melwitt*too22:17
melwittoriginally had in mind to figure it out this week, but this week was too busy with RC3 and other things22:18
mriedemi didn't think we had any nova/ironic stuff except finalizing on resource classes and dropping the old flavor stuff, but that seems to have hit a fan so will probably be a bigger discussion i guess22:18
mriedemi thought we could just drop a lot of code22:18
*** itlinux has joined #openstack-nova22:19
mriedemjroll: btw do you have that tripleo bug handy? i want to throw it in the ptg etherpad22:19
melwittI was thinking we'd do thursday as cross project meet day again but I need to find out if mlavalle and jungleboyj are ok with thursday. TheJulia said thursday afternoon is ok for ironic. she told me they have a thing where they want to discuss the possibility of being able to configure subsets of ironic nodes per nova-compute22:19
melwittfailure domain type stuff22:20
mlavallemelwitt: yes, Thursday works for us, as usual22:20
melwittcool, thanks22:20
melwittwe'll tune what time of day. I haven't scheduled the nova team photo yet22:21
mriedemsubsets of ironic nodes per compute...22:21
mriedemso like a mini az?22:21
jrollmriedem: https://bugs.launchpad.net/tripleo/+bug/1787910/22:21
openstackLaunchpad bug 1787910 in tripleo "OVB overcloud deploy fails on nova placement errors" [Critical,Triaged] - Assigned to Marios Andreou (marios-b)22:21
jrollmriedem: I don't think it's a problem for master, as people should be dropping those filters anyway22:22
* jroll hasn't heard about ironic node to nova-compute affinity in a while22:23
openstackgerritMerged openstack/nova-specs master: Repropose reshaper spec for Stein  https://review.openstack.org/59265022:23
jungleboyjmelwitt: hey. Thanks for bringing this up. Then to follow up with you today in the day got away from me.22:23
mriedemjroll: ok, added the bug anyway - would be nice to know what the summary of that issue was22:23
jungleboyjmelwitt: Thursday is fine for us. In the past we've done it in the morning but can be flexible.22:23
jrollmriedem: agree, I think the override is fine but the filter is doing something funky22:23
mriedemjroll: was tripleo still using the Exact* filters?22:23
jrollmriedem: I think just regular RamFilter22:24
jrollhttps://review.openstack.org/#/c/596093/5/environments/undercloud.yaml22:24
mriedemok, did the flavor values have garbage in them?22:24
jrollthey had flavor.memory_mb set, with flavor.properties[whatever:memory_mb]=022:24
melwittmriedem: something like that, I think. just sounded like something good for a discussion to get on the same page and figure out what's the best way to approach it22:24
melwittI don't know that much about it yet22:24
mriedemjroll: ah ok, so the filter say 0 ram on the HostState,22:25
mriedemand the filter rejected the host22:25
mriedem*saw22:25
jrollright22:25
mriedemi bet our docs on those filters don't say to stop using them...22:25
mriedemhttps://docs.openstack.org/nova/latest/admin/configuration/schedulers.html#ramfilter22:26
melwittjungleboyj: cool, that's right. on monday I'll schedule our team photo and then guess a time slot for nova/cinder and adjust with you if needed22:26
mriedemnope22:26
melwittdon't you still need them if CachingScheduler? or no?22:27
mriedemyes22:28
mriedembut we should probably say they are deprecated22:28
melwittso CachingScheduler people can't stop won't stop22:28
mriedemi can hork something out quick to start a discussion22:28
mriedemcan't stop won't stop as in puff daddy?22:28
melwittyeah. (I think?)22:29
mriedemi can't say i have the same answer for the aggregate ram/core/disk filters22:29
mriedem...22:29
melwittdidn't leakypipes have a spec up about that? aggregate affinity22:30
mriedemyes22:31
mriedemhttps://review.openstack.org/#/c/544683/22:32
melwittahoy22:33
jungleboyjmelwitt: oh. I need to take a look at when we scheduled our picture. Think it's Wednesday.22:46
*** gyee has quit IRC22:52
*** itlinux has quit IRC22:57
*** tbachman has quit IRC23:00
melwittmriedem: replied to instance group patch questions23:11
melwittI'll be respinning to add the code comment but will hold off until I know whether to adjust _get_group_details. I thought I have what's there to avoid changing existing behavior as part of the bug fix but maybe I've misunderstood your question23:18
*** slagle has joined #openstack-nova23:19
*** r-daneel has quit IRC23:24
melwittjungleboyj: ah, yeah. looks like team photos are only tues and wed this time23:25
jungleboyjOkay. So anytime Thursday is good for us then.23:27
openstackgerritMatt Riedemann proposed openstack/nova master: Deprecate Core/Ram/DiskFilter  https://review.openstack.org/59650223:28
mriedemmelwitt: jroll: bauzas: leakypipes: dansmith: it's time ^23:28
melwittjungleboyj: k, let's plan for thursday morning per tradition23:29
openstackgerritMatt Riedemann proposed openstack/nova master: Deprecate Core/Ram/DiskFilter  https://review.openstack.org/59650223:32
melwittmriedem: you wrapped lines on the commit message weirdly short. -1!23:32
* mriedem puts it on the tc office hours agenda23:33
melwitthaha23:33
mriedemi'll a heads up to the ML as well23:34
jungleboyjmelwitt: sounds great. Thank you.23:38
mriedemmelwitt: so i'm good to +W your setup_instance_group thing now,23:39
mriedemif you want to respin with that missing comment23:40
mriedemthe _get_group_details stuff should be cleaned up in stein so we don't risk a regression in a backport23:40
melwittyeah, let me add the comment since it's going to be backported and a follow on wouldn't get included23:40
mriedemyup just what i was thinking23:40
openstackgerritmelanie witt proposed openstack/nova master: Make scheduler.utils.setup_instance_group query all cells  https://review.openstack.org/54025823:45
*** macza has quit IRC23:46
*** gbarros has quit IRC23:50
melwittmriedem ^23:56
*** tristanC has quit IRC23:56
mriedemdone23:57
melwittyess23:58

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!