Friday, 2019-06-07

mnaseri mean00:00
melwittsean-k-mooney: well, at least I didn't think it's related, but wasn't thinking in that direction at all tbh00:00
*** tbachman has quit IRC00:00
mnaseroh this is interesting00:00
mnaserthe compute service is getting a messagingtimeout00:00
mnasereven though conductor is up00:00
melwittyeah, it should be recovering on its own00:00
sean-k-mooneymelwitt: ok i just wondered since you were suggesting the vnc console call00:00
mnaserjust looping on "Timed out waiting for reply.."00:01
*** tbachman has joined #openstack-nova00:01
*** panda has quit IRC00:01
melwittsean-k-mooney: haha no, we were just trying to see if the cast vs call behavior was consistent00:01
sean-k-mooneyjust got back to the start of the converstation so now have a bit more context00:01
mnaseri'll try restarting it (i have bunch of other computes that are broken so at least it will say if a restart fixes it)00:01
mnaserill still have a one in broken state00:01
melwittgood thing you have a lot of broken computes huh!00:02
mnaserhah00:02
mnaserlooks like it's not going up, the capabilities xml string just got logged and its stuck00:02
sean-k-mooneywe had an upstream or posible downstream bug that was similar to this00:02
sean-k-mooneynot to long ago00:02
mnaseri assume it'll just blow up now complaining the message timed out00:03
melwitthm, weird00:03
melwittsean-k-mooney: lettuce know if you have any links00:03
* mnaser restart in debug00:03
*** panda has joined #openstack-nova00:04
mnasersigh.00:04
mnasertwo restarts and it works now.00:04
mnaseror 3 rather00:05
mnaser(that specific compute node works rather)00:05
mnaserthe rest still broken00:05
melwitt3 restarts made it work but not 1?00:05
melwitt...00:05
mnaseryep, first restart it was hung00:06
melwittO.o00:06
mnaserrestarted once with debug enabled and accidnetally log levels without an s, restarted again right away with an s and it went up fine.00:06
mnaserwait00:06
mnaserit might actually have been okay00:06
mnaserthe first time :\00:06
melwittok, that is the only thing that could make sense there otherwise....mind blown00:07
mnaserlet me see whta happened with th broken ones00:07
sean-k-mooneyhehe if its a case that debug mode fixes it calling dibs on not finding out why :P00:07
mnaserhmm, looks like its been doing the same thing00:08
mnaserlooping forever in messagingtimeout00:08
mnaserwithout recover00:08
mnaserit looks like its flapping00:08
mnaserhttp://paste.openstack.org/show/752617/00:09
sean-k-mooneycould it be a network/loadblancer issue00:10
*** dpawlik has joined #openstack-nova00:10
melwittlooks like that "Recovering" message is a lie. it always resets itself to that when catching Exception00:11
*** slaweq has joined #openstack-nova00:11
sean-k-mooneyall of the rabbit nodes are now back up and in a quoram right00:11
melwitthttps://github.com/openstack/nova/blob/master/nova/servicegroup/drivers/db.py#L9800:11
melwittso it doesn't mean anything like it's reconnected, it hasn't00:11
mnasersean-k-mooney: yes00:13
mnaseroslo.messaging._drivers.impl_rabbit [-] Failed to consume message from queue: Queue.declare: (404) NOT_FOUND - failed to perform operation on queue 'compute' in vhost '/nova' due to timeout: NotFound: Queue.declare: (404) NOT_FOUND - failed to perform operation on queue 'compute' in vhost '/nova' due to timeout00:14
mnaseri wonder if the queue.declare is not recoverable00:14
*** dpawlik has quit IRC00:14
sean-k-mooneythis is looking more an more familar with the other rabbitmq issues we were debuging00:15
mnaseri wonder if this fails, and then when i restart the control plane, they declare the queues00:16
mnaserand then you kinda have two refs of bindings00:16
sean-k-mooneythis might be related https://bugs.launchpad.net/nova/+bug/179470600:16
openstackLaunchpad bug 1794706 in OpenStack Compute (nova) "The instance left stuck when oslo.messaging raised MessageDeliveryFailure exception" [Undecided,In progress] - Assigned to Tao Li (eric-litao)00:16
*** mriedem has joined #openstack-nova00:17
melwittif that's happening, mnaser should see a trace for MessageDeliveryFailure in the logs00:17
sean-k-mooneymnaser: when we were debugging the wsgi issue i think there was a question over would the connection recovery in oslo.messaging probaly redeclare the queue00:17
mnaseri have a bunch of those melwitt00:19
melwittohhhh damn00:20
mnaserUnable to connect to AMQP server on ctl1-xxxxx.vexxhost.cn:5671 after inf tries: Server unexpectedly closed connection00:20
mnaserlooks like it gives up00:20
sean-k-mooneythat is from here https://github.com/openstack/oslo.messaging/blob/master/oslo_messaging/_drivers/impl_rabbit.py#L76700:21
sean-k-mooneyi think https://github.com/openstack/oslo.messaging/blob/master/oslo_messaging/_drivers/impl_rabbit.py#L793 should redeclare teh queue00:22
sean-k-mooneybut im not 100% sure about that00:23
*** gyee has quit IRC00:23
sean-k-mooneywe should call that on reconnection https://github.com/openstack/oslo.messaging/blob/master/oslo_messaging/_drivers/impl_rabbit.py#L73200:23
*** slaweq has quit IRC00:24
sean-k-mooneyand before we execute the publish https://github.com/openstack/oslo.messaging/blob/master/oslo_messaging/_drivers/impl_rabbit.py#L74100:24
mnaserindeed that adds up00:24
sean-k-mooneybut i think that is only creating the consumer queues00:24
sean-k-mooneyit may not be creating the targe topic queue00:24
melwittmnaser: in which service(s) do you see MessageDeliveryFailure? I want to add more info that the LP bug00:25
mnasernova-compute00:25
sean-k-mooneyi wonder if _set_current_channel shoudl be ensuring all the exchanges and topic queus are created?00:26
melwitthm.. not what I expected. I thought instance stuck in scheduling state does not reach compute00:26
melwittso we wouldn't be able to handle it and put instance into ERROR state00:26
mnaseri wonder if this is a multithread/proc problem00:27
mnaseri mean i see a few reconnects after the disconnect00:27
mnaserand the reconnect is to different rabbitmq servers00:28
*** markvoelker has joined #openstack-nova00:29
sean-k-mooneymnaser: what is the topic/queue name its trying to publish too?00:29
*** mriedem has quit IRC00:30
mnaserthats not logged (not running in debug unfortunately)00:30
sean-k-mooneyoh it is00:30
sean-k-mooneyit was the compute queue on the /nova vhost00:30
sean-k-mooneythe /nova vhost is the exchange name right?00:30
mnaserlet me check00:31
mnaserseeing a bunch of nova.*, amq.*, reply_* and compute_* one00:31
mnaseri wonder if the compute_fanout was the missing one00:32
sean-k-mooneyin theory if we we declared the fanout consuemr using https://github.com/openstack/oslo.messaging/blob/master/oslo_messaging/_drivers/impl_rabbit.py#L108600:33
sean-k-mooneywhen we created the connection it should be in our consumeer list and we shoudl hav erecreated it00:33
melwittI added a comment to the LP bug00:34
sean-k-mooneybut if we dont make our selves a consumer of a queue then we wont recarte it00:34
openstackgerritTakashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (16)  https://review.opendev.org/57629900:34
openstackgerritTakashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (17)  https://review.opendev.org/57634400:34
openstackgerritTakashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (18)  https://review.opendev.org/57667300:35
openstackgerritTakashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (19)  https://review.opendev.org/57667600:35
mnasersean-k-mooney: but then if restarting recreates things, why dont the existing computes start working00:36
mnaserooooooh hold on00:36
mnasercompute.<hostname> queue only exists for the one that i restarted.00:36
mnaserso when it recovers, its not declaring that again.  nb: i don't know much about rabbitmq00:37
sean-k-mooneymnaser: yes its not the fanout queue its the topic queue for that host00:37
*** igordc has joined #openstack-nova00:38
sean-k-mooneye.g. the one that is used for calls00:39
sean-k-mooneymnaser: you could enable Durable queues on the rabbit side00:39
mnaseryeah.. i'm trying to find where that all gets declared00:40
sean-k-mooneywe might have a config option for that00:40
sean-k-mooneyin oslo its one of these two fucntion https://github.com/openstack/oslo.messaging/blob/master/oslo_messaging/_drivers/impl_rabbit.py#L1051-L108400:41
sean-k-mooneysetting https://docs.openstack.org/nova/latest/configuration/config.html#oslo-messaging-rabbit will likely help00:42
mnaseryeah i guess, i just kinda wanna get to the bottom of it too00:42
mnaseropenstack/nova doesnt seem to define any declare_topic_consumer00:43
sean-k-mooneywell it will help by masking the issue so the queue persist across rabbit reboots but ya we should fix it00:43
*** hongbin has quit IRC00:44
mnaserimstruggling to find what creates those queues in nova00:46
sean-k-mooneyi might be this https://github.com/openstack/nova/blob/master/nova/rpc.py#L66-L9200:47
mnaserno hits on declare_topic_consumer00:47
sean-k-mooneyactully i think its via this https://github.com/openstack/nova/blob/013aa1915c79cfcb90c4333ce1e16b3c40f16be8/nova/service.py#L184-L19200:51
mnaserhttps://github.com/openstack/oslo.messaging/blob/40c25c2bde6d2f5a756e7169060b7ce389caf174/oslo_messaging/_drivers/amqpdriver.py#L650-L66500:51
sean-k-mooneyyes so https://github.com/openstack/nova/blob/013aa1915c79cfcb90c4333ce1e16b3c40f16be8/nova/service.py#L18600:52
sean-k-mooneyis where in the base nova serivce calse we declar the target for an rpc to the service based on the topic and host00:52
mnaseri think that is not being recreated..00:54
*** frankwang has joined #openstack-nova00:54
openstackgerritDaniel Speichert proposed openstack/nova-specs master: Directly download and upload images to RBD  https://review.opendev.org/65890300:54
sean-k-mooneyi think so too00:55
sean-k-mooneyit intally gets created here https://github.com/openstack/nova/blob/013aa1915c79cfcb90c4333ce1e16b3c40f16be8/nova/cmd/compute.py#L56-L5800:55
sean-k-mooneywhen we start the agent00:55
mnaserand i think that ends up all the way down here when it starts listneing https://github.com/openstack/oslo.messaging/blob/40c25c2bde6d2f5a756e7169060b7ce389caf174/oslo_messaging/_drivers/amqpdriver.py#L650-L66500:56
mnaserbut declare_topic_consumer doesnt store state or anything, it just does it one time00:56
mnaserso if the queue disappears, declare_topic_consumer technically never runs00:56
sean-k-mooneyright so we need to add it to our consumers00:57
sean-k-mooneythat said00:58
sean-k-mooneyhttps://github.com/openstack/oslo.messaging/blob/40c25c2bde6d2f5a756e7169060b7ce389caf174/oslo_messaging/_drivers/impl_rabbit.py#L946-L96900:58
sean-k-mooneyi think that shoudl be happening already00:58
sean-k-mooneyso set_current_channel shoudl redeclar it here https://github.com/openstack/oslo.messaging/blob/40c25c2bde6d2f5a756e7169060b7ce389caf174/oslo_messaging/_drivers/impl_rabbit.py#L792-L79301:00
sapd1Hi everyone, I would like to regenerate metadata/properties in libvirt xml (I mean virsh dumpxml output) with running instances. How can I do that without reboot instances :D01:01
sean-k-mooneywe do not call self._set_current_channel here however https://github.com/openstack/oslo.messaging/blob/40c25c2bde6d2f5a756e7169060b7ce389caf174/oslo_messaging/_drivers/impl_rabbit.py#L75601:01
sean-k-mooneysapd1: you cant really what are you actully trying to do01:02
mnaseri didn't hit that ecxeption i think01:02
*** markvoelker has quit IRC01:02
sean-k-mooneysapd1: i mean you can update the xml with virsh edit but im not sure why you want to modify the metadata in the xml01:03
*** _hemna has quit IRC01:03
sean-k-mooneymnaser: ya if you are hitting the generic exception below that you shoudl end up redeclarign the queue01:03
sean-k-mooneymnaser: but that willl only happen after the heartbeat detects the connection is broken01:04
sean-k-mooneyor you try to send an rpc form the compute node01:04
mnaserFailed to declare consumer for topic 'compute': Queue.declare: (404) NOT_FOUND - failed to perform operation on queue 'compute' in vhost '/nova' due to timeout: NotFound: Queue.declare: (404) NOT_FOUND - failed to perform operation on queue 'compute' in vhost '/nova' due to timeout01:05
sean-k-mooneymnaser: i think the issue is the api is not ensuring the queue exists before calling the compute node01:05
mnasershould the api ensure it exists though, or should the service recreate it when recovering?01:06
sean-k-mooneywhy not both01:06
sapd1sean-k-mooney, ah. I put some key/value in properties option of each instance (our monitoring service), I have tried write a function to generate these properties when instances are created. But I want to update these properties after that.01:06
sean-k-mooneythe api could ensure it exists so that the message is there for when the compute connects01:06
*** slaweq has joined #openstack-nova01:06
sean-k-mooneysapd1: right ...01:07
sean-k-mooneysapd1: virsh edit is your best bet but when you say you added a fucntion do you mean in the nova code01:07
sean-k-mooneybecause as im sure you know we delete and recreate teh domain every time we start/stop/hard-reboot the instance01:08
sean-k-mooneyso if you did not modify nova those metadata values would be lost on basically any lifecycle operation01:09
sean-k-mooneygenerally form monitoring you better just pulling the instance uuid form the metadata and doing the corralation based on that01:09
sapd1sean-k-mooney, I have handled these actions bro, I have modified this function to generate xml with properties (https://github.com/openstack/nova/blob/b0e71d80c3/nova/virt/libvirt/driver.py#L427101:10
sapd1then if the instance is rebooted or hard-rebooted, It's still ok.01:10
sapd1sean-k-mooney, in my case, this is a special requirement :D01:11
mnaseroperation queue.declare caused a channel exception not_found: failed to perform operation on queue 'scheduler.ctl2-xxxxxx.vexxhost.cn' in vhost '/nova' due to timeout01:11
sean-k-mooneyhehe ok ya that shoudl work. any reaon not to push the extra metadata upstream or is it specifc to your deployment01:11
mnaserthat's from the rabbitmq logs01:11
mnaserit means that for some reason, its actually rabbitmq thats not lteting it declare?01:12
sean-k-mooneymnaser: im not sure what it means by channel exception not_found but ill ask google01:12
mnaseryeah im trying to ask google too01:12
sapd1sean-k-mooney, it's a specific for my deployment :)01:12
sean-k-mooneymnaser: Binding a non-existing queue or a non-existing exchange will fail with a 404 NOT_FOUND error01:13
sean-k-mooneyhttps://www.rabbitmq.com/channels.html01:14
mnaseri wonder if it tries to declare, fails, then binds, and fails, and then bombs out01:14
sapd1sean-k-mooney, Does my case related this function (https://github.com/openstack/nova/blob/6657339a7393226c72b9c8a293a6b92695ecb4cb/nova/virt/driver.py#L1374)?01:18
mnaserbleh, i've been at this for 2.5h -- i dunno what to do at this point, its a bit too much01:18
mnaserhttps://github.com/rabbitmq/rabbitmq-server/issues/95901:19
sean-k-mooneymnaser: i would proably set the quese to durable and enabel auto delete if this is an issue for you on production01:19
mnaseri wonder if this has to do with this01:19
sean-k-mooneyat least untill we figure this out more direclty01:19
*** slaweq has quit IRC01:19
mnaseryeah ill just work around it for now.. my brain is toast01:21
mnaserand there you go, restarted them all, went into spawning state01:22
mnaseri bet neutron is suffering from teh same thing01:22
sean-k-mooneyhttps://bugs.launchpad.net/oslo.messaging/+bug/166151001:22
openstackLaunchpad bug 1661510 in oslo.messaging "topic_send may loss messages if the queue not exists" [Medium,In progress] - Assigned to Gabriele Santomaggio (gsantomaggio)01:22
openstackgerritTakashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (20)  https://review.opendev.org/57668901:23
openstackgerritTakashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (21)  https://review.opendev.org/57670901:23
sean-k-mooneymnaser: i think https://bugs.launchpad.net/oslo.messaging/+bug/1661510 is basically the issue right01:23
openstackLaunchpad bug 1661510 in oslo.messaging "topic_send may loss messages if the queue not exists" [Medium,In progress] - Assigned to Gabriele Santomaggio (gsantomaggio)01:23
openstackgerritTakashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (22)  https://review.opendev.org/57671201:23
sean-k-mooneyhttps://bugs.launchpad.net/oslo.messaging/+bug/1437955 seams to be also related01:27
openstackLaunchpad bug 1437955 in oslo.messaging "RPC calls and responses do not use the mandatory flag (AMQP)" [Wishlist,Confirmed] - Assigned to Gabriele Santomaggio (gsantomaggio)01:27
mnasersean-k-mooney: that seems exactly it01:29
sean-k-mooneymnaser: melwitt so i cuess we want to encourage https://review.opendev.org/#/c/660373/01:30
mnasersean-k-mooney: I guess so!01:45
sean-k-mooneythe current approch would require nova to be modified to handel the retry whic may or may not be what we want01:45
sean-k-mooneyi left some feed back in the review01:46
sean-k-mooneybut its shoudl help even if we need to do some extra work as a consumer of oslo.messaging01:46
sean-k-mooneymnaser: i was tired and when to sleep for 4 hours so while its 2 am from me here it must be getting pretty late for you01:47
sean-k-mooneywell almost 3 but as i said i alreay got some sleep01:48
mnasersean-k-mooney: where are you based out of?01:54
sean-k-mooneyireland01:54
sean-k-mooneybut i tend to shift my day to overlap with the us more01:55
* sean-k-mooney because im not a morning person and that a valid excusse :P01:55
sean-k-mooneyi would normally end my day and go to sleep at aound 1 am but i was tired and went to bed at 8pm today but when i do that i never get more then about 4 hours and then i get up and work for a bit and typically go back to sleep around 3-4 ish01:57
sean-k-mooneyor 8-9 ish if my sleeping pattern is totally messed up01:58
*** ccstone has joined #openstack-nova01:59
*** markvoelker has joined #openstack-nova01:59
*** frankwang has quit IRC02:01
*** tinwood has quit IRC02:10
*** dpawlik has joined #openstack-nova02:11
*** slaweq has joined #openstack-nova02:11
*** tinwood has joined #openstack-nova02:11
*** bbowen__ has quit IRC02:15
*** dpawlik has quit IRC02:15
*** slaweq has quit IRC02:24
*** dpawlik has joined #openstack-nova02:27
*** sean-k-mooney has quit IRC02:28
*** dpawlik has quit IRC02:32
*** markvoelker has quit IRC02:32
*** JamesBenson has joined #openstack-nova02:50
*** threestrands has joined #openstack-nova02:54
*** sean-k-mooney has joined #openstack-nova03:00
*** bbowen__ has joined #openstack-nova03:07
*** whoami-rajat has joined #openstack-nova03:10
*** slaweq has joined #openstack-nova03:11
*** sean-k-mooney has quit IRC03:14
*** sean-k-mooney has joined #openstack-nova03:16
*** JamesBenson has quit IRC03:23
*** JamesBenson has joined #openstack-nova03:24
*** slaweq has quit IRC03:25
*** igordc has quit IRC03:25
openstackgerritDaniel Speichert proposed openstack/nova-specs master: Directly download and upload images to RBD  https://review.opendev.org/65890303:28
*** JamesBenson has quit IRC03:28
*** BjoernT has joined #openstack-nova03:31
*** BjoernT_ has joined #openstack-nova03:35
*** BjoernT has quit IRC03:36
*** ratailor has joined #openstack-nova03:54
*** threestrands has quit IRC04:12
*** slaweq has joined #openstack-nova04:16
*** sean-k-mooney has quit IRC04:19
*** lbragstad has quit IRC04:20
*** sean-k-mooney has joined #openstack-nova04:21
*** slaweq has quit IRC04:24
*** markvoelker has joined #openstack-nova04:30
*** udesale has joined #openstack-nova04:31
openstackgerritEric Fried proposed openstack/nova master: Introduces the openstacksdk to nova  https://review.opendev.org/64366404:32
openstackgerritEric Fried proposed openstack/nova master: Use OpenStack SDK for placement  https://review.opendev.org/65602304:32
*** dpawlik has joined #openstack-nova04:43
*** dpawlik has quit IRC04:48
*** BjoernT_ has quit IRC04:53
*** frankwang has joined #openstack-nova04:58
*** frankwang has quit IRC05:02
*** markvoelker has quit IRC05:03
*** slaweq has joined #openstack-nova05:13
*** slaweq has quit IRC05:24
*** dpawlik has joined #openstack-nova05:30
*** slaweq has joined #openstack-nova05:32
*** kaisers has quit IRC05:37
*** kaisers has joined #openstack-nova05:40
*** ratailor_ has joined #openstack-nova05:43
*** ratailor has quit IRC05:43
*** luksky has joined #openstack-nova05:44
*** slaweq has quit IRC05:44
*** markvoelker has joined #openstack-nova06:00
*** boxiang has joined #openstack-nova06:05
*** boxiang has quit IRC06:07
*** bhagyashris has joined #openstack-nova06:18
*** lpetrut has joined #openstack-nova06:19
*** sapd1_x has joined #openstack-nova06:21
*** hoonetorg has quit IRC06:21
*** maciejjozefczyk has joined #openstack-nova06:32
*** markvoelker has quit IRC06:32
*** itlinux_ has joined #openstack-nova06:34
*** itlinux has quit IRC06:34
*** pcaruana has joined #openstack-nova06:35
*** hoonetorg has joined #openstack-nova06:38
*** itlinux_ has quit IRC06:54
*** damien_r has joined #openstack-nova06:56
*** slaweq has joined #openstack-nova06:59
*** itlinux has joined #openstack-nova06:59
*** itlinux has quit IRC07:05
*** tssurya has joined #openstack-nova07:07
*** ttsiouts has joined #openstack-nova07:08
*** mvkr has joined #openstack-nova07:09
*** tesseract has joined #openstack-nova07:09
*** itlinux has joined #openstack-nova07:12
*** imacdonn has quit IRC07:16
openstackgerritBrin Zhang proposed openstack/nova master: WIP:Specify availability_zone to unshelve  https://review.opendev.org/66385107:20
*** dtantsur|afk is now known as dtantsur07:20
*** ttsiouts has quit IRC07:25
*** brinzhang has quit IRC07:26
*** ttsiouts has joined #openstack-nova07:26
*** ttsiouts_ has joined #openstack-nova07:28
*** markvoelker has joined #openstack-nova07:30
*** ttsiouts has quit IRC07:30
*** ttsiouts_ has quit IRC07:38
*** mcgigglier has joined #openstack-nova07:38
*** ttsiouts has joined #openstack-nova07:39
*** ccamacho has joined #openstack-nova07:41
*** ttsiouts has quit IRC07:43
*** jaosorior has joined #openstack-nova07:51
*** xek has joined #openstack-nova07:51
kashyapsean-k-mooney: Yeah, I _did_ see the obscure "vm_mode" yesterday, and reading its documentation.  It is a Xen-related term, but used as a general abstraction in Nova to represent any hypervisor07:55
*** rcernin has quit IRC07:55
kashyaptonyb: I wanted to do some tests on AArch64 host; got it from another dev.07:56
kashyapefried: No, that "pick guest arch" Blueprint is not for Train.  (It's Tech Debt)07:56
*** ralonsoh has joined #openstack-nova07:59
*** ttsiouts has joined #openstack-nova08:00
*** takashin has left #openstack-nova08:03
*** markvoelker has quit IRC08:03
*** JamesBenson has joined #openstack-nova08:08
*** JamesBenson has quit IRC08:12
*** sapd1_x has quit IRC08:15
*** trident has quit IRC08:35
*** derekh has joined #openstack-nova08:37
*** trident has joined #openstack-nova08:37
*** imacdonn has joined #openstack-nova08:37
tonyb[m]kashyap: okay have fun08:39
*** markvoelker has joined #openstack-nova09:00
*** mdbooth_ is now known as mdbooth09:05
*** _hemna has joined #openstack-nova09:12
*** davidsha has joined #openstack-nova09:12
*** _hemna has quit IRC09:16
*** cdent has joined #openstack-nova09:20
*** sapd1_x has joined #openstack-nova09:23
*** priteau has joined #openstack-nova09:25
*** zbr has joined #openstack-nova09:29
*** jaosorior has quit IRC09:30
*** markvoelker has quit IRC09:32
*** tkajinam has quit IRC09:37
*** lpetrut has quit IRC09:38
*** jaosorior has joined #openstack-nova09:47
*** jbernard has quit IRC09:49
openstackgerritBalazs Gibizer proposed openstack/nova master: Fold ComputeManager._resize to its only caller  https://review.opendev.org/66387309:50
*** bhagyashris has quit IRC09:55
*** lpetrut has joined #openstack-nova10:04
*** lpetrut has quit IRC10:04
*** lpetrut has joined #openstack-nova10:05
*** luksky has quit IRC10:11
*** jaosorior has quit IRC10:13
*** sapd1_x has quit IRC10:18
stephenfinRuntimeError: On Python 3, zvm sdk supports to Python 3.610:23
stephenfinand all I want to do is build docs10:23
stephenfinWe need to find some way to move that out of requirements.txt. More hassle than it's worth.10:24
*** ttsiouts has quit IRC10:24
*** ttsiouts has joined #openstack-nova10:25
*** ociuhandu has joined #openstack-nova10:25
*** ttsiouts has quit IRC10:29
*** ociuhandu has quit IRC10:30
*** ociuhandu has joined #openstack-nova10:31
openstackgerritStephen Finucane proposed openstack/nova stable/rocky: docs: Don't version links to reno docs  https://review.opendev.org/66217810:37
openstackgerritStephen Finucane proposed openstack/nova stable/rocky: tox: Use basepython of python3.5  https://review.opendev.org/66388710:37
openstackgerritStephen Finucane proposed openstack/nova stable/rocky: tox: Stop build *all* docs in 'docs'  https://review.opendev.org/66388810:37
*** zbr is now known as zbr|rover10:44
*** luksky has joined #openstack-nova10:49
*** francoisp has quit IRC10:53
*** udesale has quit IRC10:53
*** udesale has joined #openstack-nova10:53
*** janki has joined #openstack-nova10:57
*** ttsiouts has joined #openstack-nova10:57
*** cdent has quit IRC11:00
*** jaypipes has joined #openstack-nova11:01
*** ttsiouts has quit IRC11:01
*** ttsiouts has joined #openstack-nova11:01
*** cdent has joined #openstack-nova11:01
*** jaosorior has joined #openstack-nova11:07
*** jbernard has joined #openstack-nova11:16
*** xek has quit IRC11:29
*** markvoelker has joined #openstack-nova11:30
sean-k-mooneyjust an fyi but i have marked https://blueprints.launchpad.net/nova/+spec/libvirt-neutron-sriov-livemigration as implemented/complete since the documentation update we were waiting for is now merged in neutron.11:40
openstackgerritTheodoros Tsioutsias proposed openstack/nova-specs master: Enable rebuild for instances in cell0  https://review.opendev.org/64868611:48
*** xek has joined #openstack-nova11:53
*** ratailor_ has quit IRC12:01
*** francoisp has joined #openstack-nova12:03
*** markvoelker has quit IRC12:03
*** udesale has quit IRC12:10
*** udesale has joined #openstack-nova12:11
*** slaweq_ has joined #openstack-nova12:23
*** slaweq has quit IRC12:27
*** udesale has quit IRC12:27
*** udesale has joined #openstack-nova12:27
*** aloga has quit IRC12:27
*** lpetrut has quit IRC12:27
*** aloga has joined #openstack-nova12:28
openstackgerritMerged openstack/nova master: Make nova-next archive using --before  https://review.opendev.org/66100212:28
*** dansmith has quit IRC12:29
*** dansmith has joined #openstack-nova12:32
*** artom has quit IRC12:35
openstackgerritMerged openstack/nova master: Update usage in RT.drop_move_claim during confirm resize  https://review.opendev.org/64180612:35
openstackgerritMerged openstack/nova master: Robustify attachment tracking in CinderFixtureNewAttachFlow  https://review.opendev.org/65890412:35
*** rtjure has joined #openstack-nova12:40
*** markvoelker has joined #openstack-nova12:49
openstackgerritDr. Clemens Hardewig proposed openstack/nova master: Fixes instance resizing/migration when swap volume is on lvm  https://review.opendev.org/61862112:50
openstackgerritDr. Clemens Hardewig proposed openstack/nova master: This patch is to fix bug #1831657  https://review.opendev.org/66392612:50
openstackbug 1831657 in OpenStack Compute (nova) "Instance migration/Instance resize with lvm volumes" [Undecided,New] https://launchpad.net/bugs/183165712:50
openstackgerritStephen Finucane proposed openstack/nova master: Remove 'ComputeManager._reschedule'  https://review.opendev.org/66077412:57
openstackgerritStephen Finucane proposed openstack/nova master: Remove cells v1 parameter from 'ComputeTaskAPI.resize_instance'  https://review.opendev.org/65131112:57
openstackgerritStephen Finucane proposed openstack/nova master: Stop passing 'kwargs' to 'rebuild_instance'  https://review.opendev.org/65131212:57
openstackgerritStephen Finucane proposed openstack/nova master: Stop passing 'delete_type' to 'terminate_instance'  https://review.opendev.org/65131312:57
openstackgerritStephen Finucane proposed openstack/nova master: filters: Stop handling cells v1  https://review.opendev.org/65131412:57
openstackgerritStephen Finucane proposed openstack/nova master: Remove nova.compute.*API() shims  https://review.opendev.org/66052712:57
openstackgerritStephen Finucane proposed openstack/nova master: Add reno for removed cells v1 policies  https://review.opendev.org/66203112:57
openstackgerritStephen Finucane proposed openstack/nova master: Remove 'InstanceUnknownCell' exception  https://review.opendev.org/66241112:57
openstackgerritStephen Finucane proposed openstack/nova master: Ensure controllers all call super  https://review.opendev.org/66095012:57
stephenfingibi++12:58
stephenfinThanks for taking a look at those tests12:58
*** ttsiouts has quit IRC13:01
*** lbragstad has joined #openstack-nova13:03
*** janki has quit IRC13:04
*** janki has joined #openstack-nova13:04
*** cdent has quit IRC13:06
*** ttsiouts has joined #openstack-nova13:08
*** mvkr has quit IRC13:09
*** mriedem has joined #openstack-nova13:10
*** BjoernT has joined #openstack-nova13:20
*** priteau has quit IRC13:23
*** lbragstad_ has joined #openstack-nova13:26
*** dpawlik has quit IRC13:27
*** lbragstad has quit IRC13:27
*** BjoernT_ has joined #openstack-nova13:28
mriedemstephenfin: gibi: comments inline on https://review.opendev.org/#/c/660774/13:28
mriedemfor the notification sample test weirdness13:29
stephenfinin a meeting but I'll look after13:29
stephenfinmriedem: Actually, I did try to that. It failed miserably13:30
stephenfinThe issue that I pointed out previously is that I want the call to succeed in the other places it's called13:31
stephenfinAnd just stub out that one caller13:31
*** BjoernT has quit IRC13:31
gibistephenfin, mriedem: I can try to iterate on those tests sometime next week13:31
*** tbachman has quit IRC13:31
*** rtjure has quit IRC13:32
*** tbachman has joined #openstack-nova13:33
*** priteau has joined #openstack-nova13:38
*** lbragstad_ is now known as lbragstad13:38
mriedemi can play with it, maybe 3 sets of eyes can sort it out13:40
*** rtjure has joined #openstack-nova13:42
*** artom has joined #openstack-nova13:42
gibimriedem: one hint, in those resize sample tests there is only one compute (as far as I see) but still there is a successfull re-schedule attempt13:42
mriedemwhich is totally unrealistic but ok13:43
mriedemmaybe it's not unrealistic i guess, hard to know how this flow really works13:44
mriedemit would be unrealistic to reschedule and land back on the same host13:44
sean-k-mooneygibi:  a rescuderl would only work in that case if the retry filter is disabled13:44
gibimriedem: that two test are totally unrealistic :/13:45
mriedemwe don't go back through the scheduler on a reschedule13:45
mriedemsince queens13:45
gibimriedem: fault injection is hard13:45
mriedemin fact we could probably deprecate the RetryFilter13:45
mriedemi don't think it's ever used anymore13:45
mriedemanyway, bbiab13:46
*** dpawlik has joined #openstack-nova13:46
sean-k-mooneythat worth looking into13:46
sean-k-mooneyill file a bug so we dont forget13:46
*** JamesBenson has joined #openstack-nova13:52
*** JamesBenson has quit IRC13:53
*** JamesBenson has joined #openstack-nova13:53
stephenfinmriedem: Yeah, if you've ideas I'd be happy to hear them. Those tests really aren't good but I'm out of ideas at this point13:53
*** mlavalle has joined #openstack-nova13:55
*** ricolin has joined #openstack-nova13:55
*** slaweq_ is now known as slaweq14:00
*** hongbin has joined #openstack-nova14:06
*** janki has quit IRC14:08
*** liuyulong has joined #openstack-nova14:11
mriedemi'll tinker14:15
*** cdent has joined #openstack-nova14:21
openstackgerritMatt Riedemann proposed openstack/nova master: WIP: Deprecate RetryFilter  https://review.opendev.org/66395314:21
mriedemsean-k-mooney: let's see what blows up ^14:21
mriedemedleafe: ^14:21
*** igordc has joined #openstack-nova14:23
*** tbachman has quit IRC14:23
stephenfinmriedem: Would you be okay with the named kwargs thing if it was a separate patch? I think it's a nice readability improvement but it probably shouldn't be munged with other stuff alright14:24
stephenfinReferring to https://review.opendev.org/#/c/651311/14:24
mriedemi disagree that it's a readability improvement...14:26
mriedemby that logic we'd use named kwargs on everything14:26
mriedemstephenfin: i'm going to pull the series down and tinker with those notification tests for reschedule and if i come up with someting cleaner i can push that up and remove the kwargs in the next patch in the series while i'm at it14:27
sean-k-mooneymriedem: i have disabled the retry filter in one of my nfv job iteration and it more orless seam to be fine so i suspect it will not blow up or not blow up too badly14:27
*** dpawlik has quit IRC14:27
stephenfinmriedem: Not everything. Just things that have a lot of arguments (it's subjective but I'd say > 5)14:27
sean-k-mooneyim on a triage call but ill file the bug and coment on your patch when im done14:28
mriedemsean-k-mooney: bug?14:28
mriedemit's not really a bug that we forgot to deprecate the RetryFilter in Rocky14:28
sean-k-mooneyok14:28
sean-k-mooneyin that case ill just keep an eye on the test results :)14:29
openstackgerritStephen Finucane proposed openstack/nova master: Remove cells v1 parameter from 'ComputeTaskAPI.resize_instance'  https://review.opendev.org/65131114:30
openstackgerritStephen Finucane proposed openstack/nova master: Stop passing 'kwargs' to 'rebuild_instance'  https://review.opendev.org/65131214:30
openstackgerritStephen Finucane proposed openstack/nova master: Stop passing 'delete_type' to 'terminate_instance'  https://review.opendev.org/65131314:30
openstackgerritStephen Finucane proposed openstack/nova master: filters: Stop handling cells v1  https://review.opendev.org/65131414:30
openstackgerritStephen Finucane proposed openstack/nova master: Remove nova.compute.*API() shims  https://review.opendev.org/66052714:30
openstackgerritStephen Finucane proposed openstack/nova master: Add reno for removed cells v1 policies  https://review.opendev.org/66203114:30
openstackgerritStephen Finucane proposed openstack/nova master: Remove 'InstanceUnknownCell' exception  https://review.opendev.org/66241114:30
openstackgerritStephen Finucane proposed openstack/nova master: Ensure controllers all call super  https://review.opendev.org/66095014:30
openstackgerritStephen Finucane proposed openstack/nova master: Use named kwargs for resize_api  https://review.opendev.org/66395914:30
artombauzas, thanks for the note on https://review.opendev.org/#/c/644881/21, will address after this call. Also need to address mriedem's commit message feedback, which I completely forgot about the first time around14:36
mriedemoh right you want to backport that....14:37
mriedemi was going to say we don't need a release note for a virt driver interface change,14:37
mriedembut if we're backporting...14:37
* artom shall buy a spot on CNN to make sure the message goes out ;)14:38
mriedemwe did this recently for an ironic driver related fix and handled a TypeError in the backportable version and removed it in a follow up on master14:38
mriedemhttps://github.com/openstack/nova/blob/stable/rocky/nova/compute/manager.py#L764814:39
mriedemgiven that, we may want to follow the same pattern here14:39
mriedemit is admittedly bending over backward for out of tree drivers which is being exceedingly nice14:39
*** eharney has joined #openstack-nova14:40
artomWait, won't that leave vm_instance unset and blow up later?14:40
artomOh, actually no.14:40
artomvm_power_state will be unset, but it's only used in an if14:41
artomOK, I'll do something similar when I backport14:41
mriedemfor your change it would mean adding a try/except TypeError around the call to finish_revert_migration and if we get a TypeError, call again without the vifs_already_plugged kwarg14:41
mriedemwhen you backport? if we do this, you'd do it on master,14:41
mriedemand then follow up with a patch to remove that handling14:41
mriedemotherwise your backport has new extra logic in it which sucks to review14:42
artomOh yeah *facepalm*14:42
artomWhy remove on master though?14:42
artomThat same method signature TypeError thing can also happen with oot drivers on master, no?14:43
artomIf they're really slow to update14:43
*** tbachman has joined #openstack-nova14:44
sean-k-mooneyartom: we have customers still running kilo14:44
sean-k-mooneyartom: i assume those customers are also slow to update there out of tree stuff14:45
artomI doubt any of our customer have oot drivers :)14:45
openstackgerritStephen Finucane proposed openstack/nova master: Stop passing 'kwargs' to 'rebuild_instance'  https://review.opendev.org/65131214:45
openstackgerritStephen Finucane proposed openstack/nova master: Stop passing 'delete_type' to 'terminate_instance'  https://review.opendev.org/65131314:45
openstackgerritStephen Finucane proposed openstack/nova master: filters: Stop handling cells v1  https://review.opendev.org/65131414:45
openstackgerritStephen Finucane proposed openstack/nova master: Remove nova.compute.*API() shims  https://review.opendev.org/66052714:45
openstackgerritStephen Finucane proposed openstack/nova master: Add reno for removed cells v1 policies  https://review.opendev.org/66203114:45
openstackgerritStephen Finucane proposed openstack/nova master: Remove 'InstanceUnknownCell' exception  https://review.opendev.org/66241114:45
openstackgerritStephen Finucane proposed openstack/nova master: Ensure controllers all call super  https://review.opendev.org/66095014:45
sean-k-mooneywell not any that we support in anycase14:46
*** itlinux has quit IRC14:46
*** itlinux has joined #openstack-nova14:47
efriedI know for sure PowerVM oot driver is in production for real customers14:47
bauzasartom: cool with me14:48
efriedif you change the virt driver interface on stable, they'll need to patch.14:48
artomefried, right, I'm sure they exist, just most likely not among our customers14:48
mriedemartom: with your upstream hat on, you shouldn't care about your downstream customers and OOT issues14:48
mriedemiow, saying "our customers won't care" isn't a valid reason to not do it14:49
efriedartom: "our" meaning RH?14:49
mriedemthe upstream argument would be, "oot? f'em"14:49
mriedemefried: yes14:49
efriedI don't know what qualifies as a RH "customer", but the products that are based on the oot PowerVM driver are packaged as some form of RH distribution.14:50
bauzasmriedem: okay, I was thinking we could be good citizens and notify OOT maintainers, like asked in https://review.opendev.org/#/c/589085/14:50
efriedso whoever's using them is a RH "customer" in some capacity.14:50
bauzasanyway, I need to leave14:50
stephenfinWho's a good person to review resize flow bugfixes? https://review.opendev.org/#/c/662522/14:51
mriedemstephenfin: i want to say alex_xu already had a bug / patch for that14:52
mriedembut you'd have to ask him14:52
mriedemstephenfin: gibi: i changed those notification sample resize reschedule tests to move the patching out of the side effect method, the diff is in here https://review.opendev.org/#/c/660774/4/nova/tests/functional/notification_sample_tests/test_instance.py14:52
artommriedem, efried, yeah, I meant our = RH, but... I'm saying I'll do the nice things for oot drivers in my patch, as just replying to sean-k-mooney that RH customers are most likely *not* among those affected14:52
mriedemit's not a ton better, but it's different so see what you think14:52
*** itlinux has quit IRC14:52
mriedemhttps://review.opendev.org/#/c/620874/14:54
mriedemstephenfin: -1 on your fix for the same reason i -1'ed alex_xu's change14:55
mriedemsean-k-mooney: heh you even said, "this is a provisional +1 as i think this is correct assumeing we confirm the request_spec is not saved in its modifed form."14:56
mriedemdangerous to +1 before you know that to be true14:56
sean-k-mooneymriedem: i suspected it was true because we had whitebox test that were passing that implied the request spec was updated correctly on compute node but yes you are right i should have checked it end ot end first before +114:57
stephenfinmriedem: Yeah, we were trying to figure out if it was saved anywhere but I haven't been able to trace a save on the request_spec anywhere14:57
stephenfinIs there any reason we need to persist the request_spec on the object, tbh?14:58
mriedemit's very clearly saved in the conductor14:58
stephenfin*numa_topology14:58
mriedemhttps://github.com/openstack/nova/blob/master/nova/conductor/manager.py#L36714:58
mriedem^ is where i spend a good deal of my time fixing regressions14:58
mriedemhttps://review.opendev.org/#/q/topic:bug/1669054+(status:open+OR+status:merged)14:59
*** itlinux has joined #openstack-nova14:59
mriedemhttps://review.opendev.org/#/q/topic:bug/1830747+(status:open+OR+status:merged)14:59
stephenfinI saw that, but that's only for cold migrations?14:59
mriedemresize and cold migrate are the same flow stephen15:00
stephenfinoh yeah15:00
sean-k-mooneymriedem: so my original asstetion that we should use the temporay mutation context manager or somthing similar is proably correct?15:00
mriedemsean-k-mooney: correct you're right on the money here https://review.opendev.org/#/c/662522/1/nova/compute/api.py@356215:01
mriedemright on the MOONEY15:01
mriedemget it?!15:02
sean-k-mooneyhehe i should frame that and put it on my wall15:02
mriedemget a bad ass trucker hate made out of it15:02
sean-k-mooney"mriedem:  said i was right about somthing" :)15:02
mriedemyou're right about lots of things15:02
sean-k-mooney:) ok so it should be a fairly simple fix so are you ok to proceed with stephens version or alex's15:04
*** jbernard has quit IRC15:04
mriedemthey are both broken15:04
mriedemequally15:04
mriedemi've duplicated stephen's bug against alex's since alex's came first,15:04
mriedembut alex seems to have abandoned his change effectively15:04
mriedemby "simple fix" you mean shove the new numa topology into the migration contexT?15:05
mriedemi'm not sure we'd need to do that if we can, on revert, build the old numa topology from the old flavor that is in the migration context before we drop the migration context15:05
*** sapd1_x has joined #openstack-nova15:07
*** cdent has quit IRC15:08
*** tbachman has quit IRC15:08
openstackgerritMatt Riedemann proposed openstack/nova master: Remove 'ComputeManager._reschedule'  https://review.opendev.org/66077415:08
mriedemgibi: stephenfin: ^ updated i'll +W15:08
sean-k-mooneyby simpel fix i ment we jsut need to reset the change state on the field so that it does not get saved when the condoctor calls save.15:09
gibimriedem: +2, thanks15:09
mriedemsean-k-mooney: but we actually do need to persist the new numa topology on the request spec for any subsequent move operations15:10
sean-k-mooneymriedem: yes but we update it on the compute host15:10
mriedemotherwise i can (1) create my server with flavor1 and numa1, (2) resize to flavor2/numa2, confirm, (3) cold migrate and now i'm scheduling with flavor1's numa115:10
mriedemsean-k-mooney: the request spec?15:11
*** tbachman has joined #openstack-nova15:11
mriedemthat would be an up-call...15:11
mriedemthat also doesn't solve your revert problem15:11
sean-k-mooneyya we i think recalualate the numa toplogy in the driver and store it in the ... instance not he request spec never mind15:11
sean-k-mooneyso ya we need to actuly save it and then revert it15:12
stephenfinsean-k-mooney: Definitely? I know we used to mess with it in the filters https://github.com/openstack/nova/blob/master/nova/scheduler/filters/numa_topology_filter.py#L61-L6815:12
sean-k-mooneywell see the fact the numa_fit_instance_to_host fucntion modifes the request sepc is what made me suspect this would get updated by the compute node if we booted sucessfuly15:15
sean-k-mooneybut honestly we need 1 trace teh code and two test it locally to confirm15:15
stephenfinWe probably don't want to persist that change though, in any case15:15
sean-k-mooneyproably. the compute node really should not need to modify the request_spec15:16
sean-k-mooneywell in this instance15:16
mriedemthe compute service should never be saving changes to the RequestSpec, ever, since it's an up-call to the API DB15:16
mriedemdansmith: agree with me here? ^15:16
mriedemstephenfin: as i explained above, the RequestSpec.numa_topology is used in the scheduler so yes it needs to be persisted for subsequent move operations,15:17
mriedemhowever, on revert we have to make sure we revert the numa_topology in the request spec to the old flavor's values15:17
dansmithmriedem: of course15:17
sean-k-mooneywe only added the request spec to the api db around rocky correct15:17
mriedemwhich is my -115:17
mriedemjesus gd15:17
stephenfinAny changes from the compute node, I mean15:17
sean-k-mooneybuy you are right15:17
mriedemi would like to phone a friend15:18
mriedemsee this code on revert https://github.com/openstack/nova/blob/master/nova/compute/api.py#L341015:19
mriedemthat's the thing you need to concern yourself with15:19
mriedemand why i -1'ed alex_xu's change15:19
mriedemand yours15:19
mriedemsean-k-mooney: request spec has been around since mitaka or newton at least, which is why when i have to fix a regression bug b/c of the request spec, i have to backport it through 20 releases15:21
dansmithsean-k-mooney: reqspec has  been around a lot longer than rocky15:21
mriedemwhich makes me a sad panda15:21
sean-k-mooney oh i know the reqest spec has been15:22
sean-k-mooneyi just tought we only strated storing it in the api db more recently15:23
openstackgerritBalazs Gibizer proposed openstack/nova master: allow getting resource request of every bound ports of an instance  https://review.opendev.org/65511015:23
sean-k-mooneyi tought melwitt added it to the api db like 2-3 release ago but i could be wrong about that15:23
dansmithwe've had it in the api db since we've had an api db AFAIK15:24
mriedem4th api db migration15:24
dansmithyeah15:24
sean-k-mooneyah ok15:24
mriedemsweet laski15:24
mriedemapparently we aren't keeping track of runways anymore https://etherpad.openstack.org/p/nova-runways-train15:26
sean-k-mooneymriedem: i mentioned on irc that my one can be dropped form the runway15:28
*** bnemec is now known as beekneemech15:29
pandauh ?15:29
pandaoh, the sad me.15:29
pandaok.15:29
stephenfinpanda: Damn it man, look sadder!15:29
*** mcgigglier has quit IRC15:29
sean-k-mooneyit still need a second +2 but its more or less done at this point15:29
mriedemi've got it in a tab15:32
*** davidsha has quit IRC15:32
sean-k-mooneymriedem: oh incase you missed it i set https://blueprints.launchpad.net/nova/+spec/libvirt-neutron-sriov-livemigration to implemented since the neutron docs change has merged too and updated the spec link to the renderd version instead of gerrit15:33
mriedemack thanks15:33
sean-k-mooneyim going to go grab lunch ill be around later for an hour or so o/15:34
stephenfinsean-k-mooney: s/lunch/dinner/15:36
*** jbernard has joined #openstack-nova15:36
*** BjoernT has joined #openstack-nova15:37
*** ttsiouts has quit IRC15:38
*** ttsiouts has joined #openstack-nova15:39
*** cdent has joined #openstack-nova15:39
*** BjoernT_ has quit IRC15:40
openstackgerritBalazs Gibizer proposed openstack/nova master: Pass network API to the conducor's MigrationTask  https://review.opendev.org/65511115:40
openstackgerritBalazs Gibizer proposed openstack/nova master: Add request_spec to server move RPC calls  https://review.opendev.org/65572115:40
openstackgerritBalazs Gibizer proposed openstack/nova master: re-calculate provider mapping during migration  https://review.opendev.org/65511215:40
openstackgerritBalazs Gibizer proposed openstack/nova master: update allocation in binding profile during migrate  https://review.opendev.org/65642215:40
openstackgerritBalazs Gibizer proposed openstack/nova master: Extend NeutronFixture to handle migrations  https://review.opendev.org/65511415:40
*** gibi is now known as gibi_ogg15:41
*** gibi_ogg is now known as gibi_off15:41
*** ttsiouts has quit IRC15:43
openstackgerritLee Yarwood proposed openstack/nova master: libvirt: Use SATA bus for cdrom devices when using Q35 machine type  https://review.opendev.org/66301115:44
openstackgerritLee Yarwood proposed openstack/nova master: DNM: Run tempest-full-py3 with q35 machine type  https://review.opendev.org/66288715:44
*** markvoelker has quit IRC15:49
*** gyee has joined #openstack-nova15:51
*** wwriverrat has quit IRC15:54
openstackgerritArtom Lifshitz proposed openstack/nova master: Revert resize: wait for events according to hybrid plug  https://review.opendev.org/64488115:56
artommriedem, bauzas ^^15:56
* artom -> ph00d15:56
*** dtantsur is now known as dtantsur|afk15:58
*** jbernard has quit IRC16:00
*** jbernard has joined #openstack-nova16:03
openstackgerritDr. Clemens Hardewig proposed openstack/nova master: Fixes instance resizing/migration when swap volume is on lvm  https://review.opendev.org/61862116:04
openstackgerritDr. Clemens Hardewig proposed openstack/nova master: Fixes instance resizing/migration when swap volume is on lvm  https://review.opendev.org/66398616:04
*** stephenfin is now known as finucannot16:05
*** lbragstad is now known as lbragstad_runnin16:06
*** luksky has quit IRC16:11
*** xek has quit IRC16:11
*** panda is now known as panda|off16:13
openstackgerritStephen Finucane proposed openstack/nova master: Recalculate 'RequestSpec.numa_topology' on resize  https://review.opendev.org/66252216:14
finucannotsean-k-mooney, mriedem: ^ that should do the trick...I think16:14
finucannotmriedem: I'm going to have to rebase https://review.opendev.org/#/c/651311/ otherwise Gerrit won't let it merge, so want me to add the TODO?16:15
mriedemsure16:18
openstackgerritStephen Finucane proposed openstack/nova master: Remove cells v1 parameter from 'ComputeTaskAPI.resize_instance'  https://review.opendev.org/65131116:23
openstackgerritStephen Finucane proposed openstack/nova master: Stop passing 'kwargs' to 'rebuild_instance'  https://review.opendev.org/65131216:23
openstackgerritStephen Finucane proposed openstack/nova master: Stop passing 'delete_type' to 'terminate_instance'  https://review.opendev.org/65131316:23
openstackgerritStephen Finucane proposed openstack/nova master: filters: Stop handling cells v1  https://review.opendev.org/65131416:23
openstackgerritStephen Finucane proposed openstack/nova master: Remove nova.compute.*API() shims  https://review.opendev.org/66052716:23
openstackgerritStephen Finucane proposed openstack/nova master: Add reno for removed cells v1 policies  https://review.opendev.org/66203116:23
openstackgerritStephen Finucane proposed openstack/nova master: Remove 'InstanceUnknownCell' exception  https://review.opendev.org/66241116:23
openstackgerritStephen Finucane proposed openstack/nova master: Ensure controllers all call super  https://review.opendev.org/66095016:23
finucannotdone16:23
*** tbachman has quit IRC16:24
*** damien_r has quit IRC16:24
*** priteau has quit IRC16:25
finucannotwhoops, missed the comments in the following patches. Fixed those too16:28
*** mriedem has quit IRC16:28
openstackgerritStephen Finucane proposed openstack/nova master: Stop passing 'kwargs' to 'rebuild_instance'  https://review.opendev.org/65131216:28
openstackgerritStephen Finucane proposed openstack/nova master: Stop passing 'delete_type' to 'terminate_instance'  https://review.opendev.org/65131316:28
openstackgerritStephen Finucane proposed openstack/nova master: filters: Stop handling cells v1  https://review.opendev.org/65131416:28
openstackgerritStephen Finucane proposed openstack/nova master: Remove nova.compute.*API() shims  https://review.opendev.org/66052716:28
openstackgerritStephen Finucane proposed openstack/nova master: Add reno for removed cells v1 policies  https://review.opendev.org/66203116:28
openstackgerritStephen Finucane proposed openstack/nova master: Remove 'InstanceUnknownCell' exception  https://review.opendev.org/66241116:28
openstackgerritStephen Finucane proposed openstack/nova master: Ensure controllers all call super  https://review.opendev.org/66095016:28
*** mriedem has joined #openstack-nova16:29
mriedemdone16:29
mriedema couple of easy -1s to address in the next 2 after that16:29
finucannotyeah, just tackled those16:29
*** Sundar has joined #openstack-nova16:30
openstackgerritMatt Riedemann proposed openstack/nova master: Refresh instance network info on deletion  https://review.opendev.org/66076116:33
mriedemeasy fix ^16:34
*** tbachman has joined #openstack-nova16:39
finucannotIndeed. +W16:40
*** tesseract has quit IRC16:40
*** owalsh has quit IRC16:40
*** jaosorior has quit IRC16:42
*** cdent has quit IRC16:43
lyarwoodmdbooth: replied btw, the caps stub is still required.16:44
Sundarjohnthetubguy: Hi, are you good with https://review.opendev.org/#/c/603955/ after the updates?16:45
finucannotlyarwood: If you add TODOs for the few things mdbooth called out (I left my own comments on same), I'm +216:48
finucannotand then I'm going home16:49
efriedlyarwood: I gotta run, but you're hitting this code path in your tests16:50
efriedhttps://opendev.org/openstack/nova/src/branch/master/nova/virt/libvirt/driver.py#L483516:50
efriedwhich is patently wrong - should either be ... in (fields.Architecture.AARCH64,)  <== tuple -- or just ==16:50
*** markvoelker has joined #openstack-nova16:50
Sundarjohnthetubaguy: ^16:51
efriedbbiab16:51
*** efried is now known as fried_rolls16:51
*** owalsh has joined #openstack-nova16:52
*** gibi_off has quit IRC16:52
*** bbowen__ has quit IRC16:53
*** lbragstad_runnin is now known as lbragstad16:53
lyarwoodfried_rolls / finucannot: looking, thanks16:54
artomNote to self: bdms != bdsm16:59
mriedemtssurya: self.cells is a dict by the time we get to patch https://review.opendev.org/#/c/663388/17:00
mriedemsee the change below it17:00
lyarwoodartom: always a risky search term at work17:00
artomOh I don't need to search for it.17:00
openstackgerritDr. Clemens Hardewig proposed openstack/nova master: This patch is to fix bug #1831657  https://review.opendev.org/66400017:01
openstackbug 1831657 in OpenStack Compute (nova) "Instance migration/Instance resize with lvm volumes" [Undecided,New] https://launchpad.net/bugs/1831657 - Assigned to Dr. Clemens Hardewig (bringha1)17:01
*** derekh has quit IRC17:04
*** liuyulong has quit IRC17:06
tssuryamriedem: sorry, I am super stupid, I didn't pull the other two changes below it17:08
mriedemdon't worry about it17:08
tssuryawas working witht his and Jay's patches17:08
* mriedem makes note in personal records17:08
tssuryahehe17:08
*** tbachman has quit IRC17:09
*** ralonsoh has quit IRC17:09
*** tbachman has joined #openstack-nova17:13
*** factor has joined #openstack-nova17:14
*** ociuhandu_ has joined #openstack-nova17:15
openstackgerritLee Yarwood proposed openstack/nova master: libvirt: Use SATA bus for cdrom devices when using Q35 machine type  https://review.opendev.org/66301117:15
openstackgerritLee Yarwood proposed openstack/nova master: DNM: Run tempest-full-py3 with q35 machine type  https://review.opendev.org/66288717:15
lyarwoodfried_rolls / finucannot / mdbooth ; ^ thanks again for the reviews, just need to drop for dinner, I'll be back in ~45mins or so if there's any more follow up.17:15
lyarwoodaaaaaactually I've done it again and butchered the tests, 2 mins17:18
*** ociuhandu has quit IRC17:18
*** sapd1_x has quit IRC17:19
lyarwoodhaha, now I see why, sshfs has made the directory I was working on read-only, bloody wonderful17:19
*** ociuhandu_ has quit IRC17:20
*** ricolin has quit IRC17:24
*** markvoelker has quit IRC17:25
kashyaplyarwood: Nice!  The 'diffstat' is better.  I only looked now at the newer iteration17:28
* kashyap will read the comments on Tuesday; now --> cooking17:29
lyarwoodkashyap: thanks :)17:30
openstackgerritLee Yarwood proposed openstack/nova master: libvirt: Use SATA bus for cdrom devices when using Q35 machine type  https://review.opendev.org/66301117:30
openstackgerritLee Yarwood proposed openstack/nova master: DNM: Run tempest-full-py3 with q35 machine type  https://review.opendev.org/66288717:30
lyarwood^ right I'm done, apologies again for all the noise with this17:31
kashyap[Should also start a broader discussion on 'openstack-discuss' to _actually_ test 'q35' machine type in a proper job; 'i440fx' is a "second rate citizen", but it's not going to go away anytime soon, though]17:33
*** tssurya has quit IRC17:36
*** Sundar has quit IRC17:44
*** maciejjozefczyk has quit IRC17:55
*** luksky has joined #openstack-nova17:56
openstackgerritArtom Lifshitz proposed openstack/nova master: Revert resize: wait for events according to hybrid plug  https://review.opendev.org/64488118:01
openstackgerritArtom Lifshitz proposed openstack/nova master: Remove finish_revert_migration TypeError try/except  https://review.opendev.org/66401318:01
openstackgerritMatt Riedemann proposed openstack/nova master: DNM: test legacy dsvm job base-test parent  https://review.opendev.org/66401418:02
openstackgerritStephen Finucane proposed openstack/nova master: Recalculate 'RequestSpec.numa_topology' on resize  https://review.opendev.org/66252218:02
openstackgerritStephen Finucane proposed openstack/nova master: tests: Add '_setup_compute_services' helper  https://review.opendev.org/66310218:02
finucannotmriedem: Didn't address your two todos but that adds the functional test requested ^18:02
finucannotnow I'm *really* going home18:02
*** finucannot is now known as stephenfin18:02
artommriedem, I only *slightly* hate you :)18:03
artom^^18:03
*** aram1s has joined #openstack-nova18:05
openstackgerritDustin Cowles proposed openstack/nova master: Introduces SDK to IronicDriver and uses for node.get  https://review.opendev.org/64289918:06
openstackgerritDustin Cowles proposed openstack/nova master: Use SDK instead of ironicclient for node.list  https://review.opendev.org/65602718:06
openstackgerritDustin Cowles proposed openstack/nova master: Use SDK instead of ironicclient for validating instance and node  https://review.opendev.org/65602818:06
openstackgerritDustin Cowles proposed openstack/nova master: Use SDK instead of ironicclient for setting instance id  https://review.opendev.org/65969018:06
openstackgerritDustin Cowles proposed openstack/nova master: WIP: Use SDK instead of ironicclient for add/remove instance info from node  https://review.opendev.org/65969118:06
*** ccamacho has quit IRC18:18
*** markvoelker has joined #openstack-nova18:22
openstackgerritArtom Lifshitz proposed openstack/python-novaclient stable/stein: Use SHA256 instead of MD5 in completion cache  https://review.opendev.org/66402318:32
openstackgerritArtom Lifshitz proposed openstack/python-novaclient stable/rocky: Use SHA256 instead of MD5 in completion cache  https://review.opendev.org/66402518:32
mriedemsean-k-mooney: comments in https://review.opendev.org/#/c/647733/18:42
sean-k-mooneythanks ill take a look at them shortly18:42
mriedemor monday...18:42
mriedemb/c you shouldn't be working18:42
sean-k-mooney:) ill finish up in the next 20 mins or so.18:43
sean-k-mooneyya they all make sense. ill try swaping out the mocking on monday as you suggested. regarding hw_architecture = aarch64 and hw_video_model = vga  i dont know what will happen18:47
sean-k-mooneyi suspect qemu will bitch if its not supported18:47
*** markvoelker has quit IRC18:55
*** JamesBenson has quit IRC19:02
*** Sundar has joined #openstack-nova19:03
*** Nick_A has left #openstack-nova19:03
*** fried_rolls is now known as efried19:15
*** _hemna has joined #openstack-nova19:16
openstackgerritLee Yarwood proposed openstack/nova master: fup: in-line machine_type_mappings into get_default_machine_type  https://review.opendev.org/66403619:17
openstackgerritLee Yarwood proposed openstack/nova master: WIP fup: Replace call to get_capabilities within _has_uefi_support  https://review.opendev.org/66403719:17
openstackgerritEric Fried proposed openstack/nova master: WIP: Fix some bogus `in (not_a_tuple)` uses  https://review.opendev.org/66404119:20
*** _hemna has quit IRC19:20
efriedlyarwood: I suspect putting your patch together with ^ will have interesting side effects.19:21
lyarwoodefried: ack thanks, I'm heading offline now but I'll rebase on that and see what shakes out on Monday19:25
efriedlyarwood: They'll affect the tests (because `$string in $autospec_mock` doesn't work), but I don't think they can actually cause bugs in the code.19:25
*** jdillaman has quit IRC19:30
*** BjoernT has quit IRC19:31
sean-k-mooneyefried:  'foo' in 'foo' check if the 'foo' charater sequence extis in the second string 'foo'19:38
efriedright19:38
sean-k-mooneyefried: so 'f' in 'foo' is a substring match where as 'foo' in ('foo') is a contains check19:39
efriedcontains in what sense?19:39
sean-k-mooneye.g. is 'foo' one of the values of the tuple19:39
efried'foo' in ('foo') is the same as 'foo' in 'foo'.19:39
efriedBecause ('foo') is not a tuple.19:39
efriedIt's a string with redundant parens.19:39
efried('foo',) is a tuple.19:39
sean-k-mooneyah well yes that too19:40
sean-k-mooneyalthough 'foo' in ('foo',) should also be true19:40
efried 'foo' in ('foobar')  # True19:40
efried 'foo' in ('foobar',) # False19:40
sean-k-mooneyyes but that is not what i wrote19:41
efriedwhat you wrote where?19:41
sean-k-mooney>>> 'foo' in ('foo',)19:41
*** udesale has quit IRC19:41
sean-k-mooneyTrue19:41
efriedyes19:41
sean-k-mooneyif the string matachs a value in the tuple exactly it will be true19:42
efriedright19:42
efriedbut only if it's a tuple19:42
sean-k-mooneybut it wont to a substring match on each of the values of the tuple19:42
efriedright19:42
efriedwe are in violent agreement19:42
sean-k-mooneyyep :)19:42
efriedAnd I think we're not broken here, because there's no arch const that's a substring of another arch const (is there?)19:43
efriedbut it's by luck. It's still wrong and should be fixed.19:43
sean-k-mooneyi dont know if we have x86 and x86_64 but i think if we did we would threat them as an alias19:43
openstackgerritEric Fried proposed openstack/nova master: Fix some bogus `in (not_a_tuple)` uses  https://review.opendev.org/66404119:44
efriedEnhanced commit message which (hopefully) makes it clearer ^19:44
*** Sundar has quit IRC19:44
efried(also fixed the other two spots in the nova codebase that do the same)19:44
sean-k-mooneythis is the set we support https://github.com/openstack/nova/blob/master/nova/objects/fields.py#L112-L15319:44
sean-k-mooneytechnically tere a few19:45
sean-k-mooneyppc64 and ppc64le19:45
efriedmm, s390/s390x19:45
efriedppc is a substring of a bunch19:45
sean-k-mooneyya look like only ibm was broken19:45
efriedsh419:45
efriedxtensa19:45
efriedmips19:45
efriedarm719:46
sean-k-mooneyok19:46
sean-k-mooneylook like only x86 is not broken :P19:46
efriedthe one in question was specifically saying in (AARCH64)19:46
efriedwhich looks like it's probably okay.19:46
sean-k-mooneyi dont think we use most of these constants anyway19:46
sean-k-mooneyif only we had static types...19:49
*** _hemna has joined #openstack-nova19:49
*** markvoelker has joined #openstack-nova19:52
*** maciejjozefczyk has joined #openstack-nova20:02
*** mriedem has quit IRC20:08
*** maciejjozefczyk has quit IRC20:10
*** mriedem has joined #openstack-nova20:11
openstackgerritEdward Hope-Morley proposed openstack/nova master: Fix python3 compatibility of rbd get_fsid  https://review.opendev.org/66360720:14
*** Nick_A has joined #openstack-nova20:19
*** markvoelker has quit IRC20:20
*** _hemna has quit IRC20:23
*** igordc has quit IRC20:28
*** Sundar has joined #openstack-nova20:31
*** artom has quit IRC20:39
openstackgerritMatt Riedemann proposed openstack/nova master: Add SchedulerReportClient.get_provider_by_name  https://review.opendev.org/66406220:48
*** cfriesen has quit IRC20:54
mriedemwell zuul just got restarted so i'm out of here for the weekend21:06
mriedemsniff you jerks later21:06
*** mriedem has quit IRC21:06
openstackgerritEric Fried proposed openstack/nova master: Hacking N363: `in (not_a_tuple)`  https://review.opendev.org/66404121:09
*** markvoelker has joined #openstack-nova21:16
*** markvoelker has quit IRC21:21
*** Nick_A has left #openstack-nova21:26
*** whoami-rajat has quit IRC21:34
*** pcaruana has quit IRC21:51
*** luksky has quit IRC21:55
*** slaweq has quit IRC22:01
*** slaweq has joined #openstack-nova22:08
*** tbachman has quit IRC22:11
*** slaweq has quit IRC22:15
*** Sundar has quit IRC22:16
*** hamzy_ has joined #openstack-nova22:30
*** hamzy has quit IRC22:32
*** artom has joined #openstack-nova23:05
*** slaweq has joined #openstack-nova23:11
*** gyee has quit IRC23:12
*** Nick_A has joined #openstack-nova23:13
*** redkrieg has joined #openstack-nova23:14
redkriegIs there any way to run a virt-sparsify on a snapshot during creation?23:15
*** slaweq has quit IRC23:15
*** markvoelker has joined #openstack-nova23:17
*** hongbin has quit IRC23:20
*** tjgresha has joined #openstack-nova23:29
*** hoonetorg has quit IRC23:35
*** markvoelker has quit IRC23:38
*** hoonetorg has joined #openstack-nova23:49
*** lbragstad has quit IRC23:53

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!