| *** xek has quit IRC | 00:02 | |
| *** mriedem has quit IRC | 00:02 | |
| *** markvoelker has joined #openstack-nova | 00:03 | |
| *** adriant has quit IRC | 00:04 | |
| *** macz has quit IRC | 00:05 | |
| *** TxGirlGeek has quit IRC | 00:05 | |
| *** adriant has joined #openstack-nova | 00:07 | |
| *** adriant has quit IRC | 00:09 | |
| *** david-lyle is now known as dklyle | 00:09 | |
| *** markvoelker has quit IRC | 00:13 | |
| *** adriant has joined #openstack-nova | 00:24 | |
| *** adriant has quit IRC | 00:25 | |
| *** adriant has joined #openstack-nova | 00:28 | |
| *** gyee has quit IRC | 00:28 | |
| *** ociuhandu has joined #openstack-nova | 00:31 | |
| *** adriant has quit IRC | 00:32 | |
| *** ociuhandu has quit IRC | 00:35 | |
| *** slaweq has joined #openstack-nova | 00:47 | |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: WIP: Check target consumer before reverting allocations https://review.opendev.org/689049 | 00:51 |
|---|---|---|
| *** slaweq has quit IRC | 00:51 | |
| *** spatel has joined #openstack-nova | 00:59 | |
| *** adriant has joined #openstack-nova | 01:08 | |
| *** brinzhang has joined #openstack-nova | 01:12 | |
| *** brinzhang_ has quit IRC | 01:15 | |
| *** brinzhang_ has joined #openstack-nova | 01:33 | |
| *** nanzha has joined #openstack-nova | 01:33 | |
| *** brinzhang has quit IRC | 01:36 | |
| *** Liang__ has joined #openstack-nova | 01:37 | |
| *** slaweq has joined #openstack-nova | 01:47 | |
| *** nanzha has quit IRC | 01:48 | |
| *** nanzha has joined #openstack-nova | 01:49 | |
| *** slaweq has quit IRC | 01:53 | |
| *** nanzha has quit IRC | 01:54 | |
| *** nanzha has joined #openstack-nova | 01:55 | |
| *** brinzhang has joined #openstack-nova | 02:03 | |
| *** brinzhang_ has quit IRC | 02:06 | |
| *** markvoelker has joined #openstack-nova | 02:14 | |
| *** nanzha has quit IRC | 02:18 | |
| *** markvoelker has quit IRC | 02:18 | |
| *** nanzha has joined #openstack-nova | 02:19 | |
| *** mkrai_ has joined #openstack-nova | 02:21 | |
| *** brinzhang_ has joined #openstack-nova | 02:23 | |
| *** brault has quit IRC | 02:24 | |
| *** brault has joined #openstack-nova | 02:25 | |
| *** brinzhang has quit IRC | 02:26 | |
| *** mkrai_ has quit IRC | 02:41 | |
| *** mkrai__ has joined #openstack-nova | 02:41 | |
| *** brinzhang has joined #openstack-nova | 02:42 | |
| *** brinzhang_ has quit IRC | 02:45 | |
| *** slaweq has joined #openstack-nova | 02:48 | |
| *** spsurya has joined #openstack-nova | 02:50 | |
| *** slaweq has quit IRC | 02:53 | |
| *** mkrai__ has quit IRC | 02:57 | |
| *** mkrai__ has joined #openstack-nova | 02:58 | |
| *** dave-mccowan has quit IRC | 03:05 | |
| *** mkrai__ has quit IRC | 03:32 | |
| *** mkrai_ has joined #openstack-nova | 03:32 | |
| *** brinzhang_ has joined #openstack-nova | 03:36 | |
| *** psachin has joined #openstack-nova | 03:36 | |
| *** brinzhang has quit IRC | 03:39 | |
| *** slaweq has joined #openstack-nova | 03:49 | |
| *** slaweq has quit IRC | 03:54 | |
| *** gbarros has quit IRC | 03:56 | |
| *** brault has quit IRC | 03:56 | |
| *** brault has joined #openstack-nova | 03:56 | |
| openstackgerrit | zhufl proposed openstack/nova master: [Trivial]Add missing ws between words https://review.opendev.org/689921 | 03:58 |
| *** spatel has quit IRC | 04:02 | |
| *** spatel has joined #openstack-nova | 04:02 | |
| *** spatel has quit IRC | 04:03 | |
| *** udesale has joined #openstack-nova | 04:04 | |
| *** gbarros has joined #openstack-nova | 04:32 | |
| *** gbarros has quit IRC | 04:36 | |
| *** nanzha has quit IRC | 04:43 | |
| *** nanzha has joined #openstack-nova | 04:43 | |
| *** slaweq has joined #openstack-nova | 04:50 | |
| *** slaweq has quit IRC | 04:54 | |
| *** ileixe has joined #openstack-nova | 04:57 | |
| *** gbarros has joined #openstack-nova | 05:07 | |
| *** ratailor has joined #openstack-nova | 05:09 | |
| *** mkrai_ has quit IRC | 05:13 | |
| *** mkrai__ has joined #openstack-nova | 05:13 | |
| *** ralonsoh has joined #openstack-nova | 05:18 | |
| *** gbarros has quit IRC | 05:19 | |
| *** avolkov has joined #openstack-nova | 05:24 | |
| *** dakshina-ilangov has joined #openstack-nova | 05:24 | |
| *** slaweq has joined #openstack-nova | 05:29 | |
| *** janki has joined #openstack-nova | 05:30 | |
| *** brinzhang has joined #openstack-nova | 05:39 | |
| *** brinzhang_ has quit IRC | 05:42 | |
| *** slaweq has quit IRC | 05:59 | |
| *** brinzhang_ has joined #openstack-nova | 06:00 | |
| *** brinzhang has quit IRC | 06:04 | |
| *** nanzha has quit IRC | 06:15 | |
| *** nanzha has joined #openstack-nova | 06:15 | |
| *** rcernin has quit IRC | 06:16 | |
| *** markvoelker has joined #openstack-nova | 06:17 | |
| *** sapd1 has joined #openstack-nova | 06:21 | |
| *** markvoelker has quit IRC | 06:21 | |
| *** ttsiouts has joined #openstack-nova | 06:22 | |
| *** dpawlik has joined #openstack-nova | 06:27 | |
| *** igordc has joined #openstack-nova | 06:34 | |
| *** brinzhang has joined #openstack-nova | 06:34 | |
| *** ircuser-1 has quit IRC | 06:35 | |
| *** brinzhang_ has quit IRC | 06:38 | |
| *** ccamacho has joined #openstack-nova | 06:41 | |
| *** pcaruana has joined #openstack-nova | 06:47 | |
| *** brinzhang_ has joined #openstack-nova | 06:48 | |
| *** igordc has quit IRC | 06:48 | |
| *** mkrai__ has quit IRC | 06:48 | |
| *** brinzhang has quit IRC | 06:52 | |
| *** brinzhang_ has quit IRC | 06:53 | |
| *** trident has quit IRC | 06:53 | |
| *** damien_r has joined #openstack-nova | 06:53 | |
| *** damien_r has quit IRC | 06:53 | |
| *** damien_r has joined #openstack-nova | 06:54 | |
| *** trident has joined #openstack-nova | 06:58 | |
| *** dpawlik has quit IRC | 07:01 | |
| *** dpawlik has joined #openstack-nova | 07:02 | |
| *** slaweq has joined #openstack-nova | 07:04 | |
| *** udesale has quit IRC | 07:06 | |
| *** udesale has joined #openstack-nova | 07:06 | |
| *** larainema has joined #openstack-nova | 07:10 | |
| *** sridharg has joined #openstack-nova | 07:12 | |
| *** mjozefcz|lunch has joined #openstack-nova | 07:12 | |
| *** ttsiouts has quit IRC | 07:26 | |
| *** jawad_axd has joined #openstack-nova | 07:26 | |
| *** ttsiouts has joined #openstack-nova | 07:27 | |
| *** ttsiouts has quit IRC | 07:28 | |
| *** ttsiouts has joined #openstack-nova | 07:28 | |
| *** mjozefcz|lunch is now known as mjozefcz | 07:34 | |
| *** mjozefcz is now known as maciejjozefczyk | 07:34 | |
| *** jawad_axd is now known as gchristian | 07:35 | |
| *** jawad_axd has joined #openstack-nova | 07:36 | |
| *** trident has quit IRC | 07:40 | |
| *** udesale has quit IRC | 07:41 | |
| *** udesale has joined #openstack-nova | 07:42 | |
| *** priteau has joined #openstack-nova | 07:43 | |
| *** ratailor_ has joined #openstack-nova | 07:43 | |
| *** trident has joined #openstack-nova | 07:43 | |
| *** ivve has joined #openstack-nova | 07:43 | |
| *** janki has quit IRC | 07:46 | |
| *** ratailor has quit IRC | 07:46 | |
| *** mkrai_ has joined #openstack-nova | 07:51 | |
| *** jangutter has joined #openstack-nova | 07:52 | |
| bauzas | good morning Nova | 07:52 |
| *** janki has joined #openstack-nova | 07:52 | |
| gibi | bauzas: good morning | 07:56 |
| * bauzas just received his SIM card for a 4GB LTE coverage there in China | 07:57 | |
| gibi | bauzas: which provider did you use? | 07:57 |
| bauzas | gibi: just paid some "MySIM" vendor with amazon marketplace | 07:58 |
| bauzas | I'll see how it goes | 07:58 |
| bauzas | 20EUR for 4GB and 10 days | 07:58 |
| bauzas | gibi: https://www.amazon.fr/MySim-pr%C3%A9pay%C3%A9e-donn%C3%A9es-Roaming-Mobile/dp/B07LC32XMB/ref=cm_cr_arp_d_product_top?ie=UTF8 (sorry, French) | 07:58 |
| gibi | bauzas: thanks | 07:59 |
| bauzas | my phone is dual-SIM so I don't bother with having text and phone calls | 08:01 |
| gibi | bauzas: sounds like a good plan, my worldsim offers 2 GB for $128 in china | 08:02 |
| bauzas | I could pay some plan to my usual provider | 08:03 |
| bauzas | but it's waaay more expensive | 08:03 |
| gibi | yeah | 08:03 |
| bauzas | that said, if I was subscribing to another French provider called 'Free', then I'd get 25GB of data roaming by month, including China | 08:04 |
| bauzas | but that's 3G (even not 3G+) | 08:04 |
| bauzas | I mean, HSDPA | 08:05 |
| *** ttsiouts has quit IRC | 08:07 | |
| gibi | I think my current local provider offest wordwide (inc China) data 10EUR for 2GB for 3 days which seems usable | 08:07 |
| *** ttsiouts has joined #openstack-nova | 08:08 | |
| *** osmanlicilegi has left #openstack-nova | 08:09 | |
| bauzas | that's indeed reasonable, tho a bit expensive | 08:09 |
| bauzas | gibi: the fact is, my hotel is far from the conference venue plus I'll have the week-end there | 08:10 |
| alex_xu | $128 is crazy | 08:10 |
| bauzas | gibi: I can't just rely on Wifi connections | 08:10 |
| gibi | alex_xu: indeed it is | 08:11 |
| gibi | bauzas: yeah, I also have the weekend before the summit in china | 08:11 |
| gibi | arriving saturday morning | 08:11 |
| *** ttsiouts has quit IRC | 08:12 | |
| bauzas | gibi: heh, same here :) | 08:12 |
| bauzas | gibi: https://www.kayak.fr/trips/!B8vXZqcWMAbgwkrd | 08:12 |
| gibi | that saturday will be crazy to survive after the long flight | 08:12 |
| *** ttsiouts has joined #openstack-nova | 08:13 | |
| bauzas | I plan to sleep on plane | 08:15 |
| bauzas | I did it for going to Sydney as I made one stop in HK for the day | 08:15 |
| bauzas | it worked quite good | 08:15 |
| gibi | bauzas: I arrive with OS75 to PVG, landing at 06:50 in China | 08:15 |
| openstackgerrit | Adrian Chiris proposed openstack/nova master: Revert "vif: Resolve a TODO and update another" https://review.opendev.org/675776 | 08:15 |
| bauzas | didn't know OpenStack was running as airline :) | 08:16 |
| gibi | Austrian Airlines | 08:16 |
| gibi | :) | 08:16 |
| gibi | but the ticker fits well :) | 08:16 |
| bauzas | :p | 08:16 |
| bauzas | I arrive a bit later, around 9:10am | 08:16 |
| bauzas | so I mostly expect some visits during the afternoon hopefully | 08:17 |
| *** tesseract has joined #openstack-nova | 08:21 | |
| *** tesseract has quit IRC | 08:21 | |
| openstackgerrit | do3meli proposed openstack/nova master: [Trivial] Add missing ws between words https://review.opendev.org/689921 | 08:22 |
| bauzas | alex_xu: could you please make me a favor and tell me whether https://www.sosh.fr works in China ? I just wanna make sure I can reach my phone provider just in case | 08:23 |
| alex_xu | bauzas: just thinking how to ensure that, asking the local telco provider? | 08:25 |
| *** mkrai_ has quit IRC | 08:26 | |
| bauzas | alex_xu: yeah, for example | 08:27 |
| bauzas | I'm pretty sure it works | 08:27 |
| bauzas | but I'm not sure | 08:27 |
| alex_xu | bauzas: hah, let me try to call china mobile client service | 08:28 |
| *** jawad_axd has quit IRC | 08:28 | |
| *** andyzon has joined #openstack-nova | 08:29 | |
| openstackgerrit | Merged openstack/nova master: Func: bug 1849165: mig race with _populate_assigned_resources https://review.opendev.org/689866 | 08:31 |
| openstack | bug 1849165 in OpenStack Compute (nova) "_populate_assigned_resources raises "TypeError: argument of type 'NoneType' is not iterable" during active migration" [High,In progress] https://launchpad.net/bugs/1849165 - Assigned to Eric Fried (efried) | 08:31 |
| *** dpawlik has quit IRC | 08:31 | |
| alex_xu | bauzas: china mobile tell me you should ask SOSH :) | 08:32 |
| *** sapd1 has quit IRC | 08:33 | |
| bauzas | alex_xu: heh, no worries | 08:34 |
| bauzas | thanks for the call | 08:34 |
| bauzas | I just meant accessing the website | 08:34 |
| alex_xu | hah | 08:34 |
| bauzas | sorry for the confusion | 08:34 |
| alex_xu | no worries | 08:34 |
| bauzas | but I definitely appreciate you called your service provider ! | 08:35 |
| bauzas | appreciated* | 08:35 |
| bauzas | looks we both misunderstood :) | 08:35 |
| alex_xu | no, I understand you :) | 08:35 |
| bauzas | alex_xu: FWIW, I can reach the customer service by some internal chat app, that's why I just wanted to make sure it could work | 08:36 |
| *** derekh has joined #openstack-nova | 08:37 | |
| alex_xu | oh, so you need network first to reach the service | 08:37 |
| alex_xu | bauzas: but remember don't depend on google maps for travel, that may not works in china ;) | 08:38 |
| bauzas | alex_xu: yup, I know, hence my wonders about whether Sosh.fr works there | 08:41 |
| bauzas | anyway | 08:41 |
| bauzas | alex_xu: which application do you use for maps and directions in Shanghai, including routes and public transportation ? | 08:41 |
| bauzas | do you s/use/recommend | 08:42 |
| bauzas | my bad | 08:42 |
| bauzas | apparently this time, the Foundation doesn't provide some tips&tricks for navigating around Shanghai | 08:42 |
| alex_xu | I use local internet commpany apps, like baidu, alibaba, I'm not sure they have multiple language, but the Apple maps and bing maps should works. | 08:43 |
| alex_xu | bauzas: but you can try Baidu Maps, see if they works for franch or english, if not, bing maps or apple maps(if you use iphone) is safe choice I think | 08:44 |
| *** ricolin_ is now known as ricolin | 08:49 | |
| bauzas | alex_xu: I have an Android-based phone (Huawei) | 08:50 |
| bauzas | but I'll figure that out then | 08:50 |
| bauzas | my point is, I won't be able to use the Play Store while in China for downloading the apps that I want :D | 08:51 |
| bauzas | hence some bits of preparation | 08:51 |
| alex_xu | yea, good to download the apps first | 08:52 |
| alex_xu | when I travel to US, the app store is connecting back to china server, that is crazy slow... | 08:52 |
| *** tssurya has joined #openstack-nova | 08:53 | |
| bauzas | anyway, a good opportunity for de-googling myself :) | 08:54 |
| *** dakshina-ilangov has quit IRC | 08:54 | |
| *** dtantsur|afk is now known as dtantsur | 08:56 | |
| *** xek has joined #openstack-nova | 09:00 | |
| *** dpawlik has joined #openstack-nova | 09:01 | |
| openstackgerrit | kangyufei proposed openstack/python-novaclient master: Switch to Ussuri jobs https://review.opendev.org/690013 | 09:05 |
| *** ricolin_ has joined #openstack-nova | 09:15 | |
| *** ricolin has quit IRC | 09:17 | |
| *** ricolin_ is now known as ricolin | 09:20 | |
| *** Liang__ has quit IRC | 09:38 | |
| *** dpawlik has quit IRC | 09:48 | |
| *** nanzha has quit IRC | 09:51 | |
| *** mkrai_ has joined #openstack-nova | 09:52 | |
| *** nanzha has joined #openstack-nova | 09:53 | |
| *** dpawlik has joined #openstack-nova | 10:02 | |
| *** janki has quit IRC | 10:07 | |
| *** ociuhandu has joined #openstack-nova | 10:10 | |
| *** ttsiouts has quit IRC | 10:11 | |
| *** derekh has quit IRC | 10:16 | |
| *** pcaruana has quit IRC | 10:16 | |
| *** markvoelker has joined #openstack-nova | 10:19 | |
| *** pcaruana has joined #openstack-nova | 10:21 | |
| *** mkrai_ has quit IRC | 10:24 | |
| *** markvoelker has quit IRC | 10:24 | |
| *** ttsiouts has joined #openstack-nova | 10:40 | |
| *** SonPham has joined #openstack-nova | 10:40 | |
| *** SonPham has quit IRC | 10:42 | |
| *** SonPham has joined #openstack-nova | 10:44 | |
| SonPham | Hi. I want response data from nova (server.py) to nova-pythonclient after nova-pythonclient request to nova | 10:44 |
| SonPham | How can i do it | 10:44 |
| *** tbachman has quit IRC | 10:44 | |
| SonPham | Hi. I want response data from nova (server.py) to novaclient after novaclient request to nova | 10:44 |
| *** SonPham has quit IRC | 10:45 | |
| *** ttsiouts has quit IRC | 10:45 | |
| sean-k-mooney | bauzas: hi can you review this quickly, i tried to get this in to the train RCs but i think if fell through the cracks so it would be nice to land and backport it quickly https://review.opendev.org/#/c/675776/7 | 10:47 |
| *** SonPham has joined #openstack-nova | 10:48 | |
| *** derekh has joined #openstack-nova | 10:53 | |
| *** ttsiouts has joined #openstack-nova | 11:05 | |
| *** damien_r has quit IRC | 11:09 | |
| *** udesale has quit IRC | 11:18 | |
| *** bbowen has quit IRC | 11:30 | |
| *** tonyb has joined #openstack-nova | 11:31 | |
| *** dpawlik has quit IRC | 11:31 | |
| *** xek has quit IRC | 11:37 | |
| *** xek has joined #openstack-nova | 11:37 | |
| *** dpawlik has joined #openstack-nova | 11:43 | |
| *** ratailor__ has joined #openstack-nova | 11:43 | |
| *** ttsiouts has quit IRC | 11:44 | |
| *** ttsiouts has joined #openstack-nova | 11:44 | |
| *** dviroel has joined #openstack-nova | 11:46 | |
| *** ratailor_ has quit IRC | 11:46 | |
| *** SonPham has quit IRC | 11:50 | |
| *** ratailor__ has quit IRC | 11:53 | |
| *** dpawlik has quit IRC | 11:55 | |
| *** markvoelker has joined #openstack-nova | 12:01 | |
| *** dpawlik has joined #openstack-nova | 12:01 | |
| *** belmoreira has joined #openstack-nova | 12:03 | |
| efried | gibi: can I get a quick +A on https://review.opendev.org/#/c/688979/ please? | 12:05 |
| gibi | efried: looking... | 12:05 |
| efried | you were +2 before the reno | 12:07 |
| efried | Oh, I guess the predecessor still needs reviews | 12:07 |
| *** maciejjozefczyk has quit IRC | 12:07 | |
| gibi | I'm +A on the trait applying patch | 12:08 |
| * gibi is on a meeting in parallel | 12:08 | |
| efried | gibi: +2 on the ItemsMatcher fup. If you want to +2 the bottom one, maybe we can convince bauzas to +A both and all three of these will merge. | 12:09 |
| *** spatel has joined #openstack-nova | 12:09 | |
| *** damien_r has joined #openstack-nova | 12:11 | |
| *** spatel has quit IRC | 12:15 | |
| gibi | efried: I can add some unit tests for that fup. But as you are happy with the fup I'm happy with your original patch. | 12:16 |
| *** tbachman has joined #openstack-nova | 12:17 | |
| efried | gibi: like I said, I think we can live without test-for-test-util; when we use it in tests we'll uncover any bugs and fix them then. And it won't impact prod. | 12:18 |
| efried | but if you're losing sleep over it, I'll happily review tests :) | 12:19 |
| *** dpawlik has quit IRC | 12:20 | |
| *** maciejjozefczyk has joined #openstack-nova | 12:22 | |
| gibi | efried: sure, I got your comment. I would like to add test cases | 12:22 |
| gibi | efried: if the patch merges before I can add test cases then I will add those in a separate patch | 12:23 |
| efried | now we need to get that spy stuff merged | 12:23 |
| gibi | efried: that will be harder I think | 12:23 |
| efried | been at least three or four cases that could have made use of it. | 12:23 |
| *** derekh has quit IRC | 12:23 | |
| efried | since it was proposed | 12:23 |
| *** eharney has quit IRC | 12:25 | |
| gibi | efried: I think I saw two of those cases. It think not everybody sees the same value we thing the spy adds | 12:26 |
| efried | gibi: I'm almost convinced it should be added to the mock library itself. If we did that, we could probably iron out those last pesky wrinkles that are annoying us... | 12:27 |
| *** ttsiouts has quit IRC | 12:27 | |
| *** ttsiouts has joined #openstack-nova | 12:28 | |
| *** ttsiouts has quit IRC | 12:29 | |
| *** ttsiouts has joined #openstack-nova | 12:29 | |
| *** nanzha has quit IRC | 12:30 | |
| *** larainema has quit IRC | 12:32 | |
| *** nanzha has joined #openstack-nova | 12:34 | |
| *** dpawlik has joined #openstack-nova | 12:36 | |
| *** mdbooth has quit IRC | 12:37 | |
| *** mdbooth has joined #openstack-nova | 12:39 | |
| *** bbowen has joined #openstack-nova | 12:40 | |
| openstackgerrit | Alex Xu proposed openstack/nova master: Add test for the race between post live migration and update available resource https://review.opendev.org/690066 | 12:40 |
| alex_xu | efried: ^ I feel the race is case due to the post live migration, not the pre live migration | 12:41 |
| efried | alex_xu: ack, I was about to get around to trying to grok your comment :) | 12:41 |
| efried | Did you see the repro though? That triggers the race by running the periodic before the dest claim. | 12:42 |
| *** bbowen_ has joined #openstack-nova | 12:42 | |
| alex_xu | efried: yes, but I think that is due to the no numa instance without migration context, I feel it isn't race problem | 12:42 |
| alex_xu | efried: we assign migration.dest and instance.migration_context under the same lock, so it shouldn't happen race | 12:43 |
| efried | Only on cold migration | 12:43 |
| efried | on live migration the former happens on the conductor, the latter on the compute. | 12:44 |
| alex_xu | oh, I only see LM, didn't look at cold one | 12:44 |
| *** bbowen has quit IRC | 12:44 | |
| alex_xu | but it should be same, since both cold migration and LM using same move claim | 12:44 |
| alex_xu | efried: here is https://github.com/openstack/nova/blob/master/nova/compute/resource_tracker.py#L306 and https://github.com/openstack/nova/blob/master/nova/compute/resource_tracker.py#L370 | 12:46 |
| *** dtantsur is now known as dtantsur|brb | 12:47 | |
| efried | alex_xu: cold migration uses _claim_existing_migration but live migration uses _create_migration -- or the other way around, now I can't remember :( | 12:49 |
| alex_xu | really...i will hate that | 12:50 |
| *** spatel has joined #openstack-nova | 12:50 | |
| alex_xu | efried: quick grep the code, both cold and live using _claim_existing_migration | 12:51 |
| alex_xu | they create the migration obj in conductor | 12:51 |
| efried | alex_xu: If you can, probably best to wait for mriedem -- he understands these paths way better than I do. | 12:52 |
| alex_xu | efried: yea, no hurry, whatever the fix is right | 12:52 |
| efried | I put together these patches to further my understanding, but he was leading the way | 12:52 |
| *** nanzha has quit IRC | 12:53 | |
| efried | Like, mission accomplished, I understand things a little bit better than I did before, but it's still like |...me..................mriedem..| | 12:53 |
| *** derekh has joined #openstack-nova | 12:54 | |
| *** spatel has quit IRC | 12:55 | |
| efried | btw, I ran into the same COMPUTE_RESOURCE_SEMAPHORE deadlock while I was experimenting yesterday, and considered working around it by manually calling _populate_assigned_resources -- but I didn't think that was a realistic test. | 12:55 |
| alex_xu | yea...that deadlock spend hours for me, totally not worth... | 12:56 |
| efried | originally I was using a cold migration and injecting update_available_resource before _prep_resize, but that didn't trigger the problem. So I tried to inject it lower down, after _claim_existing_migration, and ran into the deadlock. | 12:59 |
| efried | That's when mriedem suggested I instead use a live migration and inject before pre_live_migration instead. | 13:00 |
| efried | That reproduced the race | 13:00 |
| efried | or... *a* race, anyway :) | 13:00 |
| *** nanzha has joined #openstack-nova | 13:01 | |
| *** nweinber__ has joined #openstack-nova | 13:02 | |
| *** xek has quit IRC | 13:03 | |
| *** spatel has joined #openstack-nova | 13:03 | |
| *** xek has joined #openstack-nova | 13:05 | |
| *** mriedem has joined #openstack-nova | 13:12 | |
| bauzas | sean-k-mooney: ah sorry, was at the gym, will review it ASAP | 13:13 |
| *** eharney has joined #openstack-nova | 13:14 | |
| *** derekh has quit IRC | 13:14 | |
| openstackgerrit | Ghanshyam Mann proposed openstack/nova master: Fix policy doc for host_status and extended servers attribute https://review.opendev.org/689833 | 13:16 |
| openstackgerrit | Ghanshyam Mann proposed openstack/nova master: Fix policy doc for host_status and extended servers attribute https://review.opendev.org/689833 | 13:16 |
| mriedem | hear ye hear ye we need stable/stein reviews https://review.opendev.org/#/q/status:open+project:openstack/nova+branch:stable/stein | 13:16 |
| bauzas | mriedem: ack | 13:17 |
| bauzas | sean-k-mooney: urgent revert, I guess ? | 13:17 |
| bauzas | sean-k-mooney: I can fast-approve it as per our policy | 13:17 |
| sean-k-mooney | bauzas: its not a pure revert | 13:21 |
| sean-k-mooney | e.g. we updated the comments to reflect our understanding of why we cant make the chagne | 13:21 |
| sean-k-mooney | bauzas: i also want to see the meloxn ci run an pass on it | 13:22 |
| bauzas | sean-k-mooney: ergh ok | 13:22 |
| bauzas | I'd have preferred a classic revert so we could spin another change | 13:22 |
| bauzas | anyway, it's done | 13:22 |
| sean-k-mooney | well we could just do a strait revert | 13:23 |
| sean-k-mooney | and then apply this after on master to fix the comments | 13:23 |
| sean-k-mooney | without the patch we get a traceback in nova while a vm is running with a macvtap interface | 13:24 |
| sean-k-mooney | it does not break the vm | 13:24 |
| sean-k-mooney | one of the periodic task that track resouce usage is unhappy | 13:24 |
| bauzas | cool | 13:27 |
| *** sridharg has quit IRC | 13:29 | |
| *** belmoreira has quit IRC | 13:29 | |
| *** udesale has joined #openstack-nova | 13:31 | |
| *** bbowen_ has quit IRC | 13:32 | |
| *** belmoreira has joined #openstack-nova | 13:37 | |
| *** xek has quit IRC | 13:40 | |
| *** xek has joined #openstack-nova | 13:40 | |
| *** damien_r has left #openstack-nova | 13:42 | |
| mriedem | gmann: in a multinode job, was is the "peers" group for hosts? and why would that need to be configured over just the "subnode" group vars to get configuration into the subnode compute? | 13:44 |
| gmann | mriedem: i think that was old name i remember, later we did 'peers' -> 'subnode' | 13:45 |
| *** jmlowe has quit IRC | 13:46 | |
| *** ociuhandu has quit IRC | 13:48 | |
| mriedem | ok i was wondering why we have both subnode and peers configured in some multinode jobs and it sounds like we could just move all the 'peers' config to 'subnode' vars config | 13:48 |
| mriedem | e.g. https://github.com/openstack/nova/blob/master/.zuul.yaml#L340-L352 | 13:49 |
| *** trident has quit IRC | 13:50 | |
| *** ociuhandu has joined #openstack-nova | 13:50 | |
| gmann | let me check if any other reason to keep that. | 13:50 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Make nova-next multinode and drop tempest-slow-py3 https://review.opendev.org/683988 | 13:51 |
| *** mlavalle has joined #openstack-nova | 13:51 | |
| *** trident has joined #openstack-nova | 13:54 | |
| mriedem | efried: want to do the backports for https://review.opendev.org/#/c/689842/ and the func test patch below it? | 13:57 |
| *** ociuhandu has quit IRC | 13:58 | |
| mriedem | sean-k-mooney: i'm assuming you'll be backporting this revert to train yeah? https://review.opendev.org/#/c/675776/ | 13:59 |
| *** jmlowe has joined #openstack-nova | 13:59 | |
| mriedem | not sure why that wasn't flagged for train-rc-potential | 14:00 |
| *** belmoreira has quit IRC | 14:00 | |
| mriedem | so we have 2 train regressions identified and fixed already, we should probably do a patch release soon | 14:03 |
| *** nweinber_ has joined #openstack-nova | 14:04 | |
| mdbooth | This is weird. utils.synchronized(instance.uuid) seems to be ignored in my functional test | 14:04 |
| sean-k-mooney | mriedem: yes i can do that if adrianc does not | 14:04 |
| mdbooth | i.e. I'm hitting breakpoints in critical sections locked by the same key concurrently | 14:05 |
| sean-k-mooney | mdbooth: we might be mocking the lock | 14:05 |
| sean-k-mooney | or the file acess | 14:05 |
| mdbooth | sean-k-mooney: Yeah, I was hoping somebody would shout if that was the case :) However, that's definitely where I'm heading next. | 14:06 |
| mdbooth | But why would we do that? | 14:06 |
| mriedem | https://github.com/openstack/nova/blob/master/nova/test.py#L239 | 14:06 |
| sean-k-mooney | mdbooth: we make most things serial in the fucntional tests | 14:06 |
| mdbooth | mriedem: Thanks! Why would you ever turn that off, btw? | 14:06 |
| sean-k-mooney | so i guess locking is not always needed | 14:07 |
| *** belmoreira has joined #openstack-nova | 14:07 | |
| mdbooth | Except if you didn't want it to work | 14:07 |
| mriedem | see the note above the condition | 14:07 |
| *** nweinber__ has quit IRC | 14:07 | |
| mdbooth | mriedem: Guessing that was written in the days before functional/ ? | 14:08 |
| sean-k-mooney | oh i guess it would also cause issues because we use the same instance uuids | 14:08 |
| sean-k-mooney | e.g. for parallel test executions on the same host | 14:08 |
| mdbooth | Because non-test code uses locks, and we're not going to 'fix' that code to not use locks | 14:08 |
| mdbooth | Just so we can execute it from a test | 14:08 |
| mdbooth | mriedem: For reference, the locks which are being ignored are the locks in _do_build_and_run_instance, and terminate_instance in ComputeManager | 14:10 |
| mriedem | mdbooth: it's not before functional tests existed | 14:11 |
| mdbooth | My test assumes that the locks works as expected. It would be weird to expect otherwise. | 14:11 |
| sean-k-mooney | mdbooth: most fucntional tests assume that there are not cuncuretn action taking place | 14:12 |
| mdbooth | It's kinda hard to write a test for a delete race when locks have different semantics in tests. | 14:12 |
| sean-k-mooney | since all the service runn in the same interperter instnace | 14:12 |
| mdbooth | Right. You can still have locks, though. | 14:12 |
| mriedem | i would prefer that we don't have tests that have to rely on locks | 14:12 |
| sean-k-mooney | yes but they would not race with anything | 14:12 |
| mdbooth | Well the locks are critical to the semantics of the code | 14:13 |
| sean-k-mooney | right but in any code path we should not have a singel action reneter the same critical section multiple times right | 14:13 |
| mdbooth | Specifically, in this case they prevent the delete from completing while it is still building, which is the actual behaviour. | 14:13 |
| sean-k-mooney | so the lock would always be aqired and never block | 14:13 |
| mdbooth | If we disable locks, I have to write a race test which assumes that an instance can be completely deleted while it is still being built. | 14:14 |
| mdbooth | That's obviously possible, but it's wrong. | 14:14 |
| mdbooth | i.e. possible to write. | 14:14 |
| mdbooth | But I'd be testing something which can't happen on a real system. | 14:14 |
| sean-k-mooney | mdbooth: well the instnace can at least be partly deleted while its building | 14:14 |
| mdbooth | sean-k-mooney: Absolutely, yes. | 14:14 |
| mdbooth | sean-k-mooney: That's the race I'm writing a test for. | 14:14 |
| sean-k-mooney | yep this is the thing stephen was working on | 14:15 |
| sean-k-mooney | did you see the message i left | 14:15 |
| mdbooth | sean-k-mooney: Did you leave it since this morning? | 14:15 |
| sean-k-mooney | no | 14:15 |
| mdbooth | sean-k-mooney: Then yeah. I've already written the test, though ;) | 14:15 |
| mdbooth | I've been badgering you to review it for a couple of days :P | 14:16 |
| * mdbooth is updating it following a suggestion from mriedem | 14:16 | |
| sean-k-mooney | yes i know :) and i will look at it now | 14:17 |
| mdbooth | In general it's better, but not having functioning locks would be super weird. | 14:17 |
| mdbooth | It would basically invalidate the test. | 14:17 |
| mdbooth | sean-k-mooney: Not worth it right now. I'm basically rewriting it. | 14:17 |
| *** ociuhandu has joined #openstack-nova | 14:17 | |
| sean-k-mooney | ok so you tried mocking _update_instance_after_spawn to do the api delete and now your rewriting it based on mriedem suggestion | 14:18 |
| sean-k-mooney | havent read all the comments since then but is see there is discussion on the patch | 14:19 |
| *** artom has quit IRC | 14:19 | |
| mdbooth | sean-k-mooney: My previous test worked fine. mriedem wants to deprecate IntegratedTestBase, thouogh, so requested a rewrite which doesn't use it. | 14:19 |
| *** andyzon has quit IRC | 14:19 | |
| mriedem | i just left more comments | 14:19 |
| sean-k-mooney | ah ok | 14:19 |
| mriedem | you're making this unnecessarily libvirt-specific when the fix isn't | 14:20 |
| mriedem | and relying on a refactor which is going to muddy backports | 14:20 |
| mdbooth | mriedem: Yeah, my rewrite also ditches libvirt | 14:20 |
| mriedem | iow you're making this harder than it needs to be imo | 14:20 |
| mriedem | ok good | 14:20 |
| *** andyzon has joined #openstack-nova | 14:20 | |
| sean-k-mooney | mriedem: so on that should i consume mdbooth refactor or avoid it for https://review.opendev.org/#/c/687957/3 | 14:20 |
| sean-k-mooney | give i do want to backport it would you be ok with backporting https://review.opendev.org/#/c/689186/2 | 14:21 |
| mriedem | it's common sense to avoid depending on as many things as possible if you intend on backporting a patch | 14:21 |
| gmann | mriedem: you can merge the peers and subnode groups setting. it was merged in base job also. Initially it was separate job for single and multinode jobs but during rocky PTG we discussed to merge both and have a single base job doing same network setting via subnode var | 14:21 |
| mriedem | sean-k-mooney: your fix is also not libvirt specific | 14:22 |
| mriedem | so i'm not sure why you need to depend on the libvirt functional test base stuff | 14:22 |
| mriedem | you should be able to write isolated functional tests for the rebuild + numa stuff | 14:22 |
| mdbooth | sean-k-mooney: I've pulled the non-placement bits out of the test mriedem pointed to. So it's still a small refactor, but I wanted to ability to not pull in a ton of placement stuff I don't need. | 14:22 |
| sean-k-mooney | that is a good point. i only depended on it because that is the only numa fucntional test we have | 14:22 |
| sean-k-mooney | but you are right i could proably remove all the libvirt stuff | 14:22 |
| sean-k-mooney | i should think about that | 14:23 |
| *** igordc has joined #openstack-nova | 14:23 | |
| * mdbooth needs still needs the locks in computemanager to work, though :/ | 14:24 | |
| sean-k-mooney | mdbooth: well we cant ignore placement on rebuild in general | 14:24 |
| sean-k-mooney | but yes its not relevent to what you are fixing/testing | 14:24 |
| mriedem | if a test needs to create a server then you are going to require the placement fixture somewhere in your functional test | 14:24 |
| mdbooth | Yeah. The resulting placement stuff could be a mixin, tbh. | 14:24 |
| mriedem | otherwise you can't get through the scheduler | 14:24 |
| sean-k-mooney | its also not technicall directly relevent to what im testing but if the traits chagned on rebuilt that in principal would need to be validated with placement | 14:25 |
| mdbooth | mriedem: Right, it's all the placement helper stuff which isn't needed in general. | 14:25 |
| *** andyzon has quit IRC | 14:25 | |
| sean-k-mooney | mriedem: yes | 14:25 |
| mdbooth | I'm still starting placement. | 14:25 |
| mriedem | mdbooth: if you need locks in the test you just add REQUIRES_LOCKING = True right? | 14:25 |
| mdbooth | mriedem: Possibly, but I also didn't expect this lock to be filesystem based. I'm going to investigate thoroughly before assuming it's all ok. | 14:26 |
| sean-k-mooney | mdbooth: the syconise decorators are always filesystem based | 14:27 |
| mdbooth | sean-k-mooney: I thought that was only for external=True | 14:27 |
| * mdbooth is reading the code, anyway | 14:28 | |
| mdbooth | re-reading | 14:28 |
| sean-k-mooney | oh am maybe | 14:28 |
| gibi | fyi tomorrow is a national holiday here in Hungary so I will be off tomorrow | 14:29 |
| *** gibi is now known as gibi_off | 14:29 | |
| mriedem | dansmith: i've abandoned my attempt at fixing https://bugs.launchpad.net/nova/+bug/1815082 but you might be interested in that | 14:29 |
| openstack | Launchpad bug 1815082 in OpenStack Compute (nova) rocky ""DBNonExistentTable: (sqlite3.OperationalError) no such table: services" when starting nova-metadata under uwsgi" [Medium,Triaged] | 14:29 |
| sean-k-mooney | mdbooth: i guess it default to false https://github.com/openstack/oslo.concurrency/blob/master/oslo_concurrency/lockutils.py#L284 | 14:29 |
| mriedem | dansmith: tl;dr trying to get the indirection api working in n-api-meta in certain cases if we don't have direct access to the db | 14:30 |
| openstackgerrit | Ghanshyam Mann proposed openstack/nova master: Fix policy doc for host_status and extended servers attribute https://review.opendev.org/689833 | 14:30 |
| mriedem | er i guess we use the indirection api for n-api-meta when running with evetlet but not wsgi | 14:30 |
| *** dpawlik has quit IRC | 14:30 | |
| dansmith | mriedem: hmm, what you have seems like it should work, although your comment about it hitting superconductor isn't right, unless nova-cpu.conf has api db creds | 14:32 |
| dansmith | mriedem: I could pick up on that, but unless someone is beating down the door for it and/or going to test it prior to merge, I'd probably set it aside too | 14:33 |
| *** andyzon has joined #openstack-nova | 14:33 | |
| efried | mriedem: I can do those backports, yeah. | 14:33 |
| *** slaweq has quit IRC | 14:33 | |
| sean-k-mooney | mriedem: before i go refactor my own work you would prefer i use ProviderUsageBaseTestCase instaead of _IntegratedTestBase and make it non libvirt specific | 14:33 |
| mriedem | jroll: i have given up on https://review.opendev.org/#/c/637217/ - someone else that cares about completing the ironic flavor data migration added in pike is going to have to take over | 14:33 |
| *** maciejjozefczyk has quit IRC | 14:34 | |
| mriedem | dansmith: i had a tempest patch that depended on the nova fix to test it | 14:34 |
| mriedem | https://review.opendev.org/#/q/status:abandoned+topic:bug/1815082 | 14:34 |
| mriedem | dansmith: looking at latest comments the issue was n-api-meta was starting before conductor which blew up | 14:35 |
| *** jaosorior has joined #openstack-nova | 14:35 | |
| mriedem | frickler: ^ i've abandoned those changes for your bug fyi | 14:36 |
| mriedem | sean-k-mooney: yes don't use _IntegratedTestBase for new tests | 14:36 |
| sean-k-mooney | ok thanks | 14:36 |
| mriedem | and yes it'd be cool if things that aren't libvirt specific didn't build on libvirt specific functional test base classes | 14:37 |
| sean-k-mooney | ya i think i can create a small numa file that is not in the libvirt folder to test this | 14:37 |
| sean-k-mooney | although there are few non libvirt numa things that will likely be added to it | 14:38 |
| *** maciejjozefczyk has joined #openstack-nova | 14:38 | |
| jroll | mriedem: thanks for the heads up | 14:39 |
| mriedem | sean-k-mooney: just my opinion, don't bend over backward based on something i'm not actively reviewing | 14:41 |
| *** dtantsur|brb is now known as dtantsur | 14:42 | |
| *** belmoreira has quit IRC | 14:43 | |
| sean-k-mooney | mriedem: ok be even so i tend to give you opipion weight even if you are not activly reviewing it since others often agree | 14:46 |
| *** belmoreira has joined #openstack-nova | 14:47 | |
| *** belmoreira has quit IRC | 14:47 | |
| *** cdent has joined #openstack-nova | 14:48 | |
| mdbooth | Whoa. It appears from tracing in oslo.concurrency that something is allowing a threading.Semaphore to be taken twice | 14:49 |
| mdbooth | I wonder... is threading.Semaphore re-entrant by default? | 14:49 |
| cdent | those have to be some most mdboothy statements I've ever seen | 14:50 |
| cdent | "hey, everything might be wrong!" | 14:50 |
| cdent | which I, of course, love | 14:51 |
| mdbooth | cdent: I usually hope to follow them up by being wrong, but I like to think them first :) | 14:51 |
| mdbooth | And unfortunately I'm not always wrong. | 14:51 |
| *** xek_ has joined #openstack-nova | 14:52 | |
| bnemec | It seems highly unlikely that a concurrency class would be non-reentrant. | 14:53 |
| mdbooth | bnemec: s/non-// | 14:53 |
| mdbooth | bnemec: I would *not* expect Semaphore to be re-entrant, and certainly in a regular environment it isn't: | 14:54 |
| *** xek has quit IRC | 14:54 | |
| bnemec | Are you sure the tracing is in the right place? We spent a bunch of time tracking down a double-locking problem at one point that was caused by logging in the wrong place. | 14:54 |
| mdbooth | >>> import threading >>> s = threading.Semaphore() | 14:54 |
| mdbooth | >>> with s: | 14:54 |
| mdbooth | ... with s: ... print("Hello world") | 14:54 |
| mdbooth | ... | 14:54 |
| mdbooth | Eurgh | 14:54 |
| mdbooth | Sorry | 14:54 |
| mdbooth | Anyway, takes same semaphore twice, hangs | 14:55 |
| bnemec | Oh, well no that wouldn't work. But that's not what reentrancy is. | 14:55 |
| mdbooth | bnemec: Wouldn't rule it out, but I don't think so. It's in a functional test, though, so my prime suspect is environmental. | 14:55 |
| bnemec | Reentrancy has to do with concurrent execution, which you can't get in a single thread. | 14:55 |
| mdbooth | bnemec: You might want to look that up ;) | 14:56 |
| bnemec | I did. Did you? https://en.m.wikipedia.org/wiki/Reentrancy_%28computing%29 | 14:56 |
| bnemec | "In computing, a computer program or subroutine is called reentrant if multiple invocations can safely run concurrently." | 14:56 |
| mdbooth | https://docs.python.org/3/library/threading.html#semaphore-objects | 14:56 |
| mdbooth | https://en.wikipedia.org/wiki/Reentrant_mutex | 14:57 |
| bnemec | Ah, we're talking about different things. | 14:57 |
| *** pcaruana has quit IRC | 14:58 | |
| mdbooth | So, Semaphore doesn't appear to be mocked, and yet I see 2 enters followed by 2 exits | 14:59 |
| * mdbooth strokes chin | 15:00 | |
| openstackgerrit | Eric Fried proposed openstack/nova stable/train: Func: bug 1849165: mig race with _populate_assigned_resources https://review.opendev.org/690099 | 15:00 |
| openstack | bug 1849165 in OpenStack Compute (nova) "_populate_assigned_resources raises "TypeError: argument of type 'NoneType' is not iterable" during active migration" [High,In progress] https://launchpad.net/bugs/1849165 - Assigned to Eric Fried (efried) | 15:00 |
| mdbooth | bnemec: To be clear, I'm *not* expecting to find a bug in oslo.concurrency. I'm expecting to find a fixture somewhere doing something I haven't thought of yet. | 15:01 |
| *** ratailor has joined #openstack-nova | 15:03 | |
| mriedem | smells like a bro down in here https://www.youtube.com/watch?v=CHnbC1N_G6w | 15:03 |
| *** ratailor has quit IRC | 15:03 | |
| mdbooth | Oh, great, it's a heisenbug >:( | 15:05 |
| *** slaweq has joined #openstack-nova | 15:06 | |
| adrianc | sean-k-mooney, i can do the backport for https://review.opendev.org/#/c/675776/, however it will take me some time to get a setup up to test it | 15:07 |
| bnemec | mdbooth: By any chance are you running tests in parallel? The external lock fixture from oslo.concurrency would make the locks independent. | 15:08 |
| *** mmethot has quit IRC | 15:09 | |
| openstackgerrit | Dan Smith proposed openstack/nova master: Add image precaching docs for aggregates https://review.opendev.org/687348 | 15:09 |
| *** mmethot has joined #openstack-nova | 15:09 | |
| openstackgerrit | Eric Fried proposed openstack/nova stable/train: Don't populate resources for not-yet-migrated inst https://review.opendev.org/690100 | 15:11 |
| efried | there must be an easier way to cherry pick a series | 15:11 |
| *** ivve has quit IRC | 15:13 | |
| *** otsukahy has joined #openstack-nova | 15:13 | |
| *** otsukahy has quit IRC | 15:13 | |
| *** trident has quit IRC | 15:13 | |
| *** maciejjozefczyk has quit IRC | 15:13 | |
| *** otsukahy has joined #openstack-nova | 15:14 | |
| *** ttsiouts has quit IRC | 15:14 | |
| sean-k-mooney | efried: cerry picking a serise is baically a rebase | 15:15 |
| sean-k-mooney | do you mean for backporting? | 15:15 |
| efried | yes | 15:15 |
| sean-k-mooney | ah | 15:15 |
| sean-k-mooney | i was refering to a feature branch before | 15:16 |
| sean-k-mooney | there is proably a way to do it in like 1 comamnd in git but i dont know what it is | 15:16 |
| sean-k-mooney | oh git cherry-pick takes a list of commit | 15:17 |
| sean-k-mooney | so if you do a git log betwen the base of the serise and top and just get the commits you shuld be able to pipe the revers of that to reverse and xargs to invoke the cherry pick | 15:18 |
| efried | will that pile them up in a series? | 15:21 |
| efried | that's probably the ticket, then. I'll try that next time. | 15:21 |
| sean-k-mooney | i think so also looking at the exmaple in the git man pages | 15:21 |
| sean-k-mooney | you might be able to do somting like this | 15:21 |
| sean-k-mooney | git cherry-pick start..end | 15:21 |
| dansmith | can't you do a range? | 15:21 |
| sean-k-mooney | where start and end are two comites | 15:21 |
| dansmith | yeah that | 15:21 |
| dansmith | works for any other git command that takes multiple commits | 15:22 |
| dansmith | if you have them all in a branch, then something like "git cherry-pick -x branch~4.." probably works | 15:22 |
| sean-k-mooney | they have an example fo "git cherry-pick ..master" | 15:22 |
| sean-k-mooney | that cherrypick anything on master but not locally | 15:22 |
| sean-k-mooney | so i assuem you can close the range | 15:22 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Require Migration object arg to migrate_instance_finish method https://review.opendev.org/690106 | 15:23 |
| umbSublime | I have a specific workload where I need to enable invTSC, for this to work with migrations I also need to set the tsc frequency. How can I configure the tsc frequency in nova.conf on my compute hosts (is it possible with cpu_model_extra_flags)? in qemu it looks like this: "<clock><timer name='tsc' frequency='3000000000'/></clock>" | 15:24 |
| sean-k-mooney | umbSublime: im pretty sure you cant | 15:25 |
| *** nsherry4 has joined #openstack-nova | 15:25 | |
| sean-k-mooney | you could enabel a specific cpu flag but i dont think you can spcify the frequency | 15:26 |
| *** nsherry4 has left #openstack-nova | 15:26 | |
| *** trident has joined #openstack-nova | 15:26 | |
| *** slaweq has quit IRC | 15:28 | |
| umbSublime | Would the solution in this case be to create a custom CPU model and use that in nova.conf ? | 15:29 |
| umbSublime | or is there an alternative workaround I didn't think about | 15:29 |
| sean-k-mooney | well this is really a feature request so i would start with the uscase | 15:29 |
| sean-k-mooney | you are trying to fake a specifc cpu frequece and provide a "stable" tsc to the guest correct? | 15:30 |
| umbSublime | I'm only the operator for this cloud. But the use case is a software that requires that flag. Only issue is with only the flag enbaled and the frequency unset, migrations are not possible | 15:30 |
| umbSublime | due to the nature of the flag | 15:30 |
| sean-k-mooney | so that if you migrate betwwen hosts with differen cpu fequecne it does not chagne | 15:30 |
| *** TxGirlGeek has joined #openstack-nova | 15:31 | |
| umbSublime | exactly there are some extensive discussions about that and how it was implemented in qemu/libvirt | 15:31 |
| umbSublime | when you set a specific frequency, there is a calculation made so that migration is possible and the value of the register is mirrored over | 15:31 |
| sean-k-mooney | ya so nova dose not currently suport hat | 15:32 |
| sean-k-mooney | *that | 15:32 |
| umbSublime | All I know is the software they run on the guest requires that invTSC is presented to the guest. From my operation pov I also need the VM to be migratable | 15:32 |
| sean-k-mooney | we could ad support if we had a defied usecase | 15:33 |
| sean-k-mooney | do they need invTSC set or do the want the TSC to be marked as reliable in the vm | 15:33 |
| umbSublime | It's a game engine software so using other counters isn't really possible | 15:33 |
| umbSublime | they need invTSC to be set | 15:33 |
| *** gyee has joined #openstack-nova | 15:33 | |
| sean-k-mooney | ok so setting tsc=reliable on the kernel commandline wont be viable | 15:34 |
| *** udesale has quit IRC | 15:34 | |
| *** psachin has quit IRC | 15:34 | |
| sean-k-mooney | im basicaly wonder if https://github.com/torvalds/linux/blob/master/Documentation/admin-guide/kernel-parameters.txt#L4832-L4838 | 15:35 |
| umbSublime | Would the custom cpu_model be a working temporary solution ? | 15:35 |
| sean-k-mooney | was another alternitive | 15:35 |
| umbSublime | Ohh I didn't mention it's a windows guest :/ | 15:35 |
| sean-k-mooney | oh :) | 15:35 |
| sean-k-mooney | so that would be a know to linux kernel args then | 15:35 |
| sean-k-mooney | in which case we shoudl be settin ghte hyperv features | 15:35 |
| sean-k-mooney | if you modified the libvirt xml defition and created a new cpu model it might work | 15:37 |
| sean-k-mooney | but i doubt it | 15:37 |
| umbSublime | If I virsh edit a running vm and add the frequency, then reboot the VM it is migrated just fine by openstack | 15:37 |
| sean-k-mooney | umbSublime: if you virsh edit it then it will be lost if nova regenerate the xml | 15:38 |
| sean-k-mooney | e.g. via hard reboot or any move opertaiton | 15:38 |
| umbSublime | Yes of course ~! | 15:38 |
| sean-k-mooney | e.g. i would not expect it to survie a migraton | 15:38 |
| sean-k-mooney | oh | 15:38 |
| sean-k-mooney | you mean | 15:38 |
| umbSublime | It was just to confirm that migration would work if forcing the freq. The next step was to configure in nova :) | 15:38 |
| sean-k-mooney | before you live migrate you remove the invTSC flag | 15:39 |
| umbSublime | nope both the invTSC and freq migrated with the VM | 15:39 |
| umbSublime | I was actually surprised by that | 15:39 |
| sean-k-mooney | well we dont update all the xml on live migration | 15:40 |
| sean-k-mooney | we just do a diff | 15:40 |
| sean-k-mooney | and modify what needs to be chagned | 15:40 |
| umbSublime | Ok that explains it ! | 15:40 |
| sean-k-mooney | but a hard reboot of the vm would loose any manual change you make | 15:40 |
| umbSublime | yes absolutely | 15:40 |
| sean-k-mooney | dumb question but do you neeed live migration. or would cold migration be enough | 15:40 |
| umbSublime | this was just for testing purposes | 15:40 |
| umbSublime | live-migration would really be the ideal goal (we need sometimes to update compute hosts without impacting guests) | 15:41 |
| umbSublime | If this really isn't possible at the moment, I'm sure that aspect could be discussed internally | 15:42 |
| sean-k-mooney | i think unfotuetly the best path forwad would be to add this as a new feature | 15:42 |
| sean-k-mooney | what release of openstack are you running | 15:42 |
| *** otsukahy has quit IRC | 15:43 | |
| umbSublime | Currently that stack is on Queens | 15:43 |
| sean-k-mooney | this is the only reference to invtsc in the nova code base | 15:44 |
| sean-k-mooney | https://github.com/openstack/nova/blob/1ec87c227c4a506ddc3355108a95390cdaa57d62/nova/conf/libvirt.py#L608-L613 | 15:44 |
| *** otsukahy has joined #openstack-nova | 15:44 | |
| umbSublime | hehe yes, that's the only one I found also. Most of the work to make exposing that flag to the guest possible was made in qemu/libvirt projects. There many discussions in the MLs about this | 15:45 |
| sean-k-mooney | umbSublime: can i suggest you file a bluepirnt for this. or maybe a bug fix. | 15:46 |
| sean-k-mooney | i would like to call it a bug so it could be backproted but its really a feautre since invtsc frequesce configuration was never previously supproted so it shoudl be a specless blueprint not a bug | 15:47 |
| *** otsukahy has quit IRC | 15:48 | |
| umbSublime | sean-k-mooney: I'll add it to my todo list :) (I've never done any of those, but would gladly do it) | 15:49 |
| *** andyzon has quit IRC | 15:50 | |
| umbSublime | Thanks a lot helping me out :) really appreciated | 15:50 |
| sean-k-mooney | no worries. sorry i could not point you at an exisitng knob to tweak | 15:51 |
| *** davee__ has quit IRC | 15:51 | |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Stop converting Migration objects to dicts for migrate_instance_start https://review.opendev.org/690115 | 15:51 |
| *** davee__ has joined #openstack-nova | 15:51 | |
| *** ircuser-1 has joined #openstack-nova | 15:54 | |
| mriedem | dansmith: this is the requirements bump you'll need for your osc change https://review.opendev.org/#/c/690097/ | 15:59 |
| mriedem | but the latest novaclient is breaking horizon... | 15:59 |
| dansmith | mriedem: woot, I was just looking for it | 15:59 |
| *** artom has joined #openstack-nova | 16:00 | |
| dansmith | guh | 16:00 |
| mriedem | https://bugs.launchpad.net/horizon/+bug/1849351 | 16:02 |
| openstack | Launchpad bug 1849351 in OpenStack Dashboard (Horizon) "horizon doesn't work with python-novaclient 16.0.0" [Undecided,New] | 16:02 |
| *** otsukahy has joined #openstack-nova | 16:03 | |
| *** otsukahy has quit IRC | 16:03 | |
| *** ricolin_ has joined #openstack-nova | 16:04 | |
| cdent | I think I know that bug. That's the "horizon needs more maintainers" bug. | 16:04 |
| dansmith | mriedem: do you know what the deal is? brin's list of changes does't include anything that pops out at me | 16:04 |
| mriedem | https://docs.openstack.org/releasenotes/python-novaclient/unreleased.html#upgrade-notes | 16:04 |
| mriedem | we removed the list_extensions stuff | 16:04 |
| mriedem | which horizon uses here https://github.com/openstack/horizon/blob/master/openstack_dashboard/api/nova.py#L1018 | 16:05 |
| mriedem | i'm guessing to enable/disable features based on extensions but since i think liberty everything is enabled and non-configurable | 16:05 |
| mriedem | unless you have out of tree hacks | 16:05 |
| mriedem | anyway, i'm cloning horizon to hack on it | 16:05 |
| dansmith | oh this is more of a bump for them just the latest novaclient release? | 16:05 |
| *** ricolin__ has joined #openstack-nova | 16:05 | |
| mriedem | horizon just has to remove usage of deprecated novaclient code that is now gone | 16:06 |
| *** jmlowe has quit IRC | 16:06 | |
| *** ricolin has quit IRC | 16:07 | |
| dansmith | novaclient hash 364cad41912e2c0f99a30f78b2835f3480a18d6e is what I was looking for | 16:07 |
| dansmith | I guess that's why we went from 15 to 16 | 16:07 |
| mriedem | efried: ooo hot damn this is working now https://review.opendev.org/#/c/683988/ | 16:07 |
| mriedem | dansmith: that and the cells v1 things that were removed also yeah | 16:07 |
| dansmith | ack | 16:08 |
| efried | \o/ | 16:08 |
| *** otsukahy has joined #openstack-nova | 16:09 | |
| *** ricolin_ has quit IRC | 16:09 | |
| *** macz has joined #openstack-nova | 16:12 | |
| mriedem | since stephen is MIA until after the summit, i wonder if we should move py27 jobs to experimental so we can run them on-demand but not automatically on every change? | 16:13 |
| mriedem | that would at least be py27 and functional | 16:13 |
| mriedem | maybe that's not worth the trouble right now | 16:13 |
| dansmith | shouldn't we change functional-py3whatever to functional and functional to functional-py27 ? | 16:14 |
| cdent | why bother? | 16:14 |
| cdent | that was in response to mriedem not dansmith | 16:14 |
| dansmith | just to make it default for people running local? | 16:14 |
| dansmith | cdent: I know :) | 16:14 |
| mriedem | dansmith: stephen's patch might do that...i'd have to check. but at some point i'm guessing we'll have functional-py37 too | 16:14 |
| mriedem | i think that's already in experimental | 16:14 |
| mriedem | yup https://github.com/openstack/nova/blob/master/.zuul.yaml#L97 | 16:15 |
| dansmith | mriedem: I've been typing py27 for unit for ten years almost, it's annoying to have the target name change every time we bump py3 versions | 16:15 |
| *** dpawlik has joined #openstack-nova | 16:15 | |
| dansmith | so I'd love to have a -efunctional and -eunit so I don't have to track | 16:15 |
| sean-k-mooney | dansmith: we did that in os-vif | 16:15 |
| mriedem | i've been typing -e functional-py36 for years now b/c that's the one i always broke :) | 16:15 |
| sean-k-mooney | https://github.com/openstack/os-vif/blob/master/tox.ini#L32-L43 | 16:16 |
| mriedem | anyway, just a thought, ignore me | 16:16 |
| sean-k-mooney | ya i have that in muscel memory too | 16:16 |
| sean-k-mooney | dansmith: oh we did it that way in os-vif because we only had py27 fucntional tests at the time | 16:17 |
| *** rpittau is now known as rpittau|afk | 16:17 | |
| sean-k-mooney | so when we added python3 func tests i decided to make functional py3 by defualt | 16:18 |
| *** igordc has quit IRC | 16:20 | |
| *** cdent has left #openstack-nova | 16:20 | |
| sean-k-mooney | dansmith: i would like to see a "-e unit" too by the way as i woudl liek a way to have just "tox" work on any system regardless of what version of python you have installed | 16:22 |
| dansmith | yeah | 16:22 |
| mriedem | this horizon fix is pretty simple, should have it up soon | 16:23 |
| sean-k-mooney | that said 90% of the time i run it by typeing "CTRL-r tox" to grab the command form my bash history so it does not really bother me. | 16:24 |
| openstackgerrit | Adrian Chiris proposed openstack/nova stable/train: Revert "vif: Resolve a TODO and update another" https://review.opendev.org/690126 | 16:27 |
| openstackgerrit | Merged openstack/nova master: Don't populate resources for not-yet-migrated inst https://review.opendev.org/689842 | 16:27 |
| openstackgerrit | Merged openstack/nova stable/queens: Stop sending bad values from libosinfo to libvirt https://review.opendev.org/688069 | 16:27 |
| openstackgerrit | Merged openstack/nova stable/stein: Ignore sqla-migrate inspect.getargspec deprecation warnings on py36 https://review.opendev.org/685072 | 16:27 |
| *** ricolin__ is now known as ricolin | 16:30 | |
| *** nanzha has quit IRC | 16:38 | |
| *** eharney has quit IRC | 16:39 | |
| mriedem | gah, this horizon thing is actually kind of a mess, | 16:39 |
| mriedem | they have their own config option for blacklisting compute API extensions, | 16:39 |
| mriedem | so i essentially have to remove that without a deprecation period in horizon | 16:39 |
| mriedem | and the novaclient change https://review.opendev.org/#/c/686516/ definitely jumped the gun on removal of the deprecated list_extensions stuff | 16:40 |
| mriedem | didn't even wait a full server side release | 16:40 |
| dansmith | blargh | 16:40 |
| mriedem | so, not sure what to do. i could plow ahead with the horizon thing, release note it and hope the horizon people are lax about their compat rules (makes me feel dirty) | 16:41 |
| *** pcaruana has joined #openstack-nova | 16:41 | |
| mriedem | or we revert the novaclient thing, blacklist 16.0.0, release a patch and then deprecate the option in horizon | 16:41 |
| mriedem | maybe it's time for the ML | 16:41 |
| dansmith | or release a 15.2.0 and keep everything on 15.x for a cycle? | 16:42 |
| *** dpawlik has quit IRC | 16:42 | |
| mriedem | always fun cleaning up the fallout from cores with itchy trigger fingers to delete shit | 16:42 |
| mriedem | dansmith: that doesn't solve the upper-constraints issue | 16:42 |
| mriedem | you need something for osc, and horizon needs to not be broken | 16:42 |
| dansmith | mriedem: can't we set u-c to 15.9 or something to keep everything from getting 16? | 16:43 |
| mriedem | dansmith: the novaclient change you need is only in 16.0.0 though | 16:43 |
| mriedem | iow you're tied to the sinking ship unless we revert this other novaclient patch | 16:43 |
| dansmith | right, that's why I said release 15.2, essentially backporting what we need to 15.x for a cycle | 16:43 |
| mriedem | you mean backport your aggregate-image-cache feature to stable/train? | 16:44 |
| mriedem | in novaclient i mean | 16:44 |
| dansmith | well, | 16:44 |
| dansmith | don't we have branches in the clients for number versions | 16:44 |
| mriedem | that's what we'd have to do i think, and that's not really kosher | 16:44 |
| dansmith | ? | 16:44 |
| mriedem | we have branches and stable/train is the 15.x series yeah, | 16:45 |
| mriedem | but still it would mean backporting the feature | 16:45 |
| dansmith | yeah so your concern is train would pull in 15.2? | 16:45 |
| mriedem | right | 16:45 |
| dansmith | but yeah, it's a backport | 16:45 |
| mriedem | simpler just to revert this rushed removal patch to me | 16:45 |
| mriedem | but i'll send to the ML to see what the horizon people say | 16:45 |
| dansmith | okay so what does that mean for versions? 16.1 becomes the next version with the stuff back in it? | 16:46 |
| dansmith | so we effectively bumped to 16 for no reason? | 16:46 |
| mriedem | 16.0.0 would be blacklisted, we'd revert https://review.opendev.org/#/c/686516/ and release as 16.0.1 i thnk | 16:46 |
| mriedem | *think | 16:46 |
| mriedem | not sure it needs to be a minor | 16:46 |
| dansmith | that's what I mean yeah | 16:46 |
| mriedem | u-c would get 16.0.1 | 16:47 |
| mriedem | you'd get your thing for osc, | 16:47 |
| mriedem | and horizon would have a release to deprecate it's option | 16:47 |
| mriedem | and we revert the revert in X | 16:47 |
| mriedem | which is what should have happened anyway according to our novaclient deprecation removal guidelines https://docs.openstack.org/python-novaclient/latest/contributor/deprecation-policy.html | 16:47 |
| dansmith | idk, I agree that plan would be the by-the-book way, | 16:48 |
| dansmith | but being a client it seems like it shouldn't be that big of a deal, but I'm clearly not the expert on the politics here | 16:48 |
| mriedem | it's not really politics, it's just that we screwed horizon without time to react, | 16:49 |
| mriedem | so they can find an alternative solution without novaclient (i don't know how to write that patch), they can drop their thing without a deprecation period, or we can try to make it right | 16:49 |
| mriedem | and option 5 is i just stop caring and let someone else sort out this mess :) | 16:50 |
| mriedem | like stephenfin! | 16:50 |
| dansmith | I mean the "getting everyone to agree on the solution" where some want the by-the-book and some might go a more pragmatic route | 16:50 |
| dansmith | anyhoo | 16:50 |
| *** pcaruana has quit IRC | 16:52 | |
| *** markvoelker has quit IRC | 16:54 | |
| dansmith | if we leave it to stephenfin then we're not going to get anything resolved until well after ptg | 16:56 |
| mriedem | i know, i was joking | 16:56 |
| dansmith | so hopefully that isn't the strategy | 16:56 |
| mriedem | crap, forgot i was preheating an oven, oops | 16:57 |
| *** dtantsur is now known as dtantsur|afk | 17:02 | |
| *** xek__ has joined #openstack-nova | 17:07 | |
| *** xek_ has quit IRC | 17:10 | |
| *** markvoelker has joined #openstack-nova | 17:12 | |
| *** spatel has quit IRC | 17:15 | |
| *** mkrai has joined #openstack-nova | 17:17 | |
| mriedem | will see if this does the trick https://review.opendev.org/690134 | 17:18 |
| *** tbachman has quit IRC | 17:22 | |
| *** priteau has quit IRC | 17:22 | |
| *** otsukahy has quit IRC | 17:24 | |
| *** eharney has joined #openstack-nova | 17:25 | |
| *** tbachman has joined #openstack-nova | 17:28 | |
| *** ozzzo has quit IRC | 17:32 | |
| *** tssurya has quit IRC | 17:32 | |
| *** ociuhandu_ has joined #openstack-nova | 17:32 | |
| *** mmethot_ has joined #openstack-nova | 17:35 | |
| *** mmethot has quit IRC | 17:35 | |
| *** ociuhandu has quit IRC | 17:36 | |
| *** ociuhandu_ has quit IRC | 17:39 | |
| *** ociuhandu has joined #openstack-nova | 17:43 | |
| *** andyzon has joined #openstack-nova | 17:46 | |
| *** ociuhandu has quit IRC | 17:47 | |
| dansmith | mriedem: were you going to send an email about this or just hope horizon peeps are okay with that? | 18:00 |
| *** ricolin has quit IRC | 18:01 | |
| *** jaosorior has quit IRC | 18:01 | |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Adds view builders for keypairs controller https://review.opendev.org/347289 | 18:02 |
| mriedem | dansmith: i started working up the email but then if that change is OK i don't think it's worth the email | 18:03 |
| mriedem | or revert discussion | 18:03 |
| *** otsukahy has joined #openstack-nova | 18:05 | |
| *** tbachman has quit IRC | 18:07 | |
| *** ttsiouts has joined #openstack-nova | 18:07 | |
| *** otsukahy has quit IRC | 18:09 | |
| *** mkrai has quit IRC | 18:13 | |
| *** igordc has joined #openstack-nova | 18:14 | |
| *** ivve has joined #openstack-nova | 18:15 | |
| *** tbachman has joined #openstack-nova | 18:18 | |
| *** ralonsoh has quit IRC | 18:22 | |
| *** CeeMac has joined #openstack-nova | 18:26 | |
| *** spsurya has quit IRC | 18:30 | |
| *** gbarros has joined #openstack-nova | 18:37 | |
| *** gbarros has quit IRC | 18:41 | |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: FakeDriver: adding and removing instances on live migration. https://review.opendev.org/243613 | 18:42 |
| *** ttsiouts has quit IRC | 18:44 | |
| *** ttsiouts has joined #openstack-nova | 18:44 | |
| *** gbarros has joined #openstack-nova | 18:45 | |
| *** igordc has quit IRC | 18:47 | |
| *** ttsiouts has quit IRC | 18:49 | |
| *** igordc has joined #openstack-nova | 18:56 | |
| *** dpawlik has joined #openstack-nova | 18:57 | |
| *** andyzon has quit IRC | 19:00 | |
| *** gbarros has quit IRC | 19:01 | |
| *** tbachman has quit IRC | 19:03 | |
| *** pcaruana has joined #openstack-nova | 19:04 | |
| *** jmlowe has joined #openstack-nova | 19:09 | |
| *** dpawlik has quit IRC | 19:09 | |
| *** gbarros has joined #openstack-nova | 19:12 | |
| *** gbarros has quit IRC | 19:19 | |
| *** pcaruana has quit IRC | 19:19 | |
| *** gbarros has joined #openstack-nova | 19:20 | |
| *** slaweq has joined #openstack-nova | 19:20 | |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: api-ref: mark device response param as optional for list/show vol attachments https://review.opendev.org/690383 | 19:24 |
| *** henriqueof has joined #openstack-nova | 19:27 | |
| *** gbarros has quit IRC | 19:27 | |
| *** otsukahy has joined #openstack-nova | 19:32 | |
| *** gbarros has joined #openstack-nova | 19:35 | |
| mriedem | this is pretty funny https://review.opendev.org/#/c/323250/ | 19:36 |
| mriedem | "Happy 3rd birthday patch!! Soon you will go to school!" | 19:36 |
| dansmith | lol | 19:37 |
| *** gbarros has quit IRC | 19:37 | |
| *** maciejjozefczyk has joined #openstack-nova | 19:40 | |
| *** gbarros has joined #openstack-nova | 19:40 | |
| *** slaweq has quit IRC | 19:46 | |
| *** dpawlik has joined #openstack-nova | 19:48 | |
| mriedem | unfortunately i still have to -1 them | 19:50 |
| *** slaweq has joined #openstack-nova | 19:51 | |
| mriedem | is there some way we could have an instance that is both deleted (instance.deleted != 0) and SOFT_DELETED? (instance.vm_state == 'soft-delete')? | 19:54 |
| *** nweinber_ has quit IRC | 20:00 | |
| *** maciejjozefczyk has quit IRC | 20:04 | |
| *** otsukahy has quit IRC | 20:05 | |
| *** otsukahy has joined #openstack-nova | 20:05 | |
| openstackgerrit | Matthew Booth proposed openstack/nova master: Add new simple test base for functional tests https://review.opendev.org/689186 | 20:18 |
| openstackgerrit | Matthew Booth proposed openstack/nova master: Functional test for UnexpectedDeletingTaskStateError https://review.opendev.org/689278 | 20:18 |
| openstackgerrit | Matthew Booth proposed openstack/nova master: Unplug VIFs as part of cleanup of networks https://review.opendev.org/663382 | 20:18 |
| openstackgerrit | Matthew Booth proposed openstack/nova master: Fix incorrect vm and task state after build failure race https://review.opendev.org/689388 | 20:18 |
| openstackgerrit | Matthew Booth proposed openstack/nova master: Make image fixtures simpler to reference https://review.opendev.org/690403 | 20:18 |
| mdbooth | mriedem: https://review.opendev.org/#/c/689186/3/nova/tests/functional/integrated_helpers.py is what I did to ProviderUsageBaseTestCase | 20:19 |
| mdbooth | tl;dr renamed it to SimpleTestBase and put all the provider test specific support code in a ProviderUsageBaseTestCase subclass | 20:20 |
| mdbooth | Then I subclassed SimpleTestBase, which is clean without the provider stuff | 20:21 |
| mriedem | i don't want to backport any of those refactors | 20:22 |
| mriedem | i thought i was pretty clear about that | 20:23 |
| mdbooth | I can't explain the locking thing I hit, btw, but it made no sense, went away when I looked harder, and never came back. I assume it was PEBKAC and I probably had an unsaved editor window open or something. | 20:23 |
| mdbooth | mriedem: Sure, if you like. | 20:23 |
| mdbooth | mriedem: For the backport we could just use ProviderUsageBaseTestCase | 20:23 |
| mriedem | yes, please | 20:23 |
| mdbooth | And for the image fixture thing we could just use a hardcoded image | 20:23 |
| mriedem | right now you have 2 refactor changes before the new regression test | 20:24 |
| mdbooth | But lets do that in the backport | 20:24 |
| *** tbachman has joined #openstack-nova | 20:24 | |
| melwitt | mriedem: without looking at the code first, I'd have thought "yes" because the instance would begin as SOFT_DELETED and then when the reclaim interval comes around and it's time to permadelete it, it would then become deleted != 0 as well. it would surprise me if the vm_state is cleared when that happens, but who knows | 20:26 |
| mriedem | melwitt: the periodic queries the db for instances which are SOFT_DELETED and not undergoing some task_state transition on the given host, | 20:27 |
| mriedem | then deletes them which changes the vm_state to DELETED and destroys the instance | 20:28 |
| mriedem | so if we updated the vm_state to DELETED it shouldn't be in the db query results in the periodic | 20:28 |
| *** gbarros has quit IRC | 20:28 | |
| mriedem | and if we didn't update that, like the update failed, then the instance won't be deleted either (deleted == 0) | 20:28 |
| mriedem | the db api query will return deleted instances by default since the filter being used doesn't include deleted=False | 20:29 |
| mriedem | but still, not sure how to get into deleted=id and vm_state=soft-delete at the same time | 20:29 |
| mriedem | i was thinking maybe some local delete in the api wonkiness but didn't see anything obvoius | 20:29 |
| mriedem | melwitt: also while i've got you, it looks like NOVA_CONSOLE_PROXY_COMPUTE_TLS in our nova-next job isn't getting honored anymore, i.e. we don't have the vencrypt config in that job | 20:30 |
| melwitt | ok, it sounds like by "could have" you mean can the periodic return an instance like that. I answered it as though it were, "can we have any records in that state?" and I thought yes. so... nevermind me | 20:30 |
| mriedem | because NOVA_CONSOLE_PROXY_COMPUTE_TLS=true in localrc but https://github.com/openstack/devstack/blob/1a46c898db9c16173013d95e2bc954992121077c/lib/nova#L175 | 20:30 |
| melwitt | mriedem: argh again huh... ok, I'll look. I've had to fix that before in the past | 20:31 |
| mriedem | i'm trying to figure out why https://review.opendev.org/#/c/683988/ is intermittenly failing novnc tests, | 20:31 |
| melwitt | I'll look up what I did last time and compare to see what has changed | 20:31 |
| mriedem | which you recently fixed for multinode jobs here https://review.opendev.org/#/c/675721/ | 20:31 |
| mriedem | i think it's just the conversion of the nova-next job from d-g to zuulv3 | 20:32 |
| mriedem | https://review.opendev.org/#/c/670196/ | 20:32 |
| mriedem | d-g would pass the value through in the localrc but zuulv3 is yaml'ing it and converting True to true | 20:33 |
| mriedem | yaml-ama-ding-dong | 20:33 |
| melwitt | this is what I had to do last time to fix enablement of tls between nova-compute and the proxy https://review.opendev.org/643045 | 20:33 |
| mriedem | we aren't even getting that far because https://review.opendev.org/#/c/643045/9/lib/nova@659 | 20:34 |
| mriedem | [[ ${NOVA_CONSOLE_PROXY_COMPUTE_TLS} = "True" ]] | 20:34 |
| mriedem | NOVA_CONSOLE_PROXY_COMPUTE_TLS: true | 20:34 |
| melwitt | oh, you're saying that flag is not getting reflected as true? O.o | 20:34 |
| KeithMnemonic | is this a valid command option, using --deleted and --limit -1. it seems to cause an infinite hang on Rocky. i did not check newwer releases yet | 20:34 |
| mriedem | melwitt: hard to say because local.conf here looks OK https://zuul.opendev.org/t/openstack/build/9726aebf48804b83bbebe290bd8187ac/log/controller/logs/local_conf.txt.gz#54 | 20:35 |
| melwitt | hmm | 20:35 |
| *** dpawlik has quit IRC | 20:35 | |
| melwitt | KeithMnemonic: I didn't think that limit -1 was a thing. unlimited happens by omitting --limit, I thought. but regardless of that, there shouldn't be an indefinite hang, that I would call a bug | 20:36 |
| mriedem | limit=-1 is a novaclient thing to page until there are no more results | 20:36 |
| melwitt | oh :\ I thought it did that on its own in the absence of --limit | 20:37 |
| KeithMnemonic | " --limit <num-servers> | 20:37 |
| KeithMnemonic | Maximum number of servers to display. If limit equals | 20:37 |
| KeithMnemonic | -1, all servers will be displayed. If limit is greater | 20:37 |
| KeithMnemonic | than 'osapi_max_limit' option of Nova API, | 20:37 |
| KeithMnemonic | 'osapi_max_limit' will be used instead. | 20:37 |
| KeithMnemonic | " | 20:37 |
| melwitt | nevermind, sorry | 20:37 |
| KeithMnemonic | from the cli help | 20:37 |
| KeithMnemonic | ok i can log a bug. i will see if this happens on something newer | 20:37 |
| mriedem | melwitt: nope, by default limit is 1000 results in the api | 20:37 |
| KeithMnemonic | i did not see anything open around this currently | 20:37 |
| melwitt | mriedem: that I know, but I thought the client would auto-page by itself without need to tell it | 20:37 |
| melwitt | and I was wrong | 20:37 |
| KeithMnemonic | in the env i tested, i only have maybe 20 deleted at most | 20:38 |
| melwitt | yeah, something is whack there, I think | 20:38 |
| KeithMnemonic | let me see what debug i can get to add to the bug to help isolate it | 20:39 |
| mdbooth | mriedem: Responded. I still think that's good to go. I'd prefer to make the requested changes in the backports. They're trivial enough. | 20:39 |
| melwitt | KeithMnemonic: yeah run with --debug to see where it gets stuck | 20:40 |
| *** otsukahy has quit IRC | 20:46 | |
| *** cgoncalves has quit IRC | 20:49 | |
| mriedem | melwitt: oh here is the vencrypt stuff, i was looking in the wrong config file (there are only 20 nova configs in a devstack job) | 20:50 |
| mriedem | https://zuul.opendev.org/t/openstack/build/9726aebf48804b83bbebe290bd8187ac/log/controller/logs/etc/nova/nova_cell1_conf.txt.gz#32 | 20:50 |
| *** colby_ has quit IRC | 20:50 | |
| mriedem | so i wonder if NOVA_CONSOLE_PROXY_COMPUTE_TLS=True just doesn't work for the subnode since we've never had a multinode job use that yet | 20:51 |
| melwitt | mriedem: ah yeah. awhile back I split things at stephenfin request (and I agree was a good idea) to only config which settings were required on the proxy and compute host, respectively. so they don't just all appear in all configs like they used to back in the day | 20:52 |
| melwitt | mriedem: the subnode is nova-compute only, right? if so, then it would make sense to *not* run the proxy service there and it might just be that we have a wrong assumption in the conditional that the service will necessarily be running if we're doing tls between the computes and the proxy? | 20:54 |
| mriedem | yeah subnode is nova-compute only | 20:54 |
| mriedem | no n-novnc service running on the subnode | 20:54 |
| melwitt | and maybe the intermittent is because depending on whether the instance being tested for vnc lands on the subnode or not? | 20:55 |
| mriedem | yup i think so | 20:55 |
| *** cgoncalves has joined #openstack-nova | 20:55 | |
| melwitt | yeah, so it sounds like we need to use some other condition for whether to do the stuff, not the presence of the tls proxy service thing | 20:56 |
| *** gbarros has joined #openstack-nova | 20:57 | |
| mriedem | ok so maybe for now i'll rev this patch, comment out the usage of NOVA_CONSOLE_PROXY_COMPUTE_TLS since it doesn't work for multinode jobs, and see if that at least gets this patch moving forward | 20:57 |
| melwitt | ok. and I can work on a change in devstack to make it work with multinode, and then stack a change on top of yours that Depends-On it, to see if it clears up the problem | 21:00 |
| melwitt | or sorry, I'll just do the devstack change and then you can uncomment and Depends-On to test | 21:00 |
| mriedem | sure that works, thanks | 21:01 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Make nova-next multinode and drop tempest-slow-py3 https://review.opendev.org/683988 | 21:01 |
| *** slaweq has quit IRC | 21:08 | |
| KeithMnemonic | so what i can tell is first it calls /server/detail?deleted=True and this returns a good list, then the next call is that same but with &marker=<id of last server> and it loops here | 21:11 |
| KeithMnemonic | so something about the marker. if i try and call it using curl i get a 401 but my guess is you that is not a valid test | 21:12 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Revert "Log CellTimeout traceback in scatter_gather_cells" https://review.opendev.org/690417 | 21:12 |
| mriedem | KeithMnemonic: you're getting a 401 with curl b/c you're not providing a token | 21:13 |
| KeithMnemonic | no i am | 21:13 |
| KeithMnemonic | it only 401 with &marker | 21:13 |
| KeithMnemonic | with just ?delete=True it is fine | 21:13 |
| mriedem | 401 is an auth issue so that doesn't make much sense | 21:13 |
| KeithMnemonic | http://paste.openstack.org/show/785488/ | 21:15 |
| KeithMnemonic | you can see something with adding &marker, but it could be my test is wrong | 21:15 |
| KeithMnemonic | i was just trying to curl what is was looping on | 21:15 |
| KeithMnemonic | the --debug show it loops once it adds on &marker=<last deleted UUID> | 21:16 |
| KeithMnemonic | https://bugs.launchpad.net/nova/+bug/1773945 | 21:19 |
| openstack | Launchpad bug 1773945 in OpenStack Compute (nova) "nova client servers.list crashes with bad marker" [Medium,Incomplete] | 21:19 |
| *** gbarros has quit IRC | 21:19 | |
| mriedem | i think i see the problem | 21:20 |
| KeithMnemonic | not quite my issue , but something with markers | 21:20 |
| mriedem | https://review.opendev.org/#/c/527564/1/nova/compute/api.py | 21:20 |
| mriedem | objects.BuildRequestList.get_by_filters returns an empty list if filtering on deleted=True, | 21:21 |
| mriedem | the code assumes that if there was a marker and we got a build request list (not MarkerNotFound), we found our marker and can set marker=None since we won't find it in a cell, | 21:21 |
| mriedem | but that's not the case here | 21:21 |
| mriedem | that goes beyond rocky https://review.opendev.org/#/q/I1aa3ca6cc70cef65d24dec1e7db9491c9b73f7ab | 21:22 |
| mriedem | https://review.opendev.org/#/c/527564/1/nova/objects/build_request.py@376 | 21:22 |
| mriedem | KeithMnemonic: in your reproduce do you have any deleted servers in the db? | 21:24 |
| KeithMnemonic | especially since i only have 18 deleted, it seems it should not need the marker right? | 21:24 |
| mriedem | you must otherwise you wouldn't get a marker | 21:24 |
| *** gbarros has joined #openstack-nova | 21:24 | |
| melwitt | maybe the deleted instances are located across separate cells? | 21:26 |
| mriedem | i meant i see the reason for the infinite loop | 21:26 |
| mriedem | KeithMnemonic: did you open a bug? | 21:26 |
| KeithMnemonic | not yet, was trying to get more debug to make the bug report useful | 21:27 |
| KeithMnemonic | that is when i found it looped on marker | 21:27 |
| melwitt | I was talking about why a marker when only 18 deleted? | 21:27 |
| melwitt | I dunno | 21:27 |
| mriedem | because limit is in the request i think | 21:27 |
| mriedem | limit == paging | 21:28 |
| KeithMnemonic | correct --limit -1 | 21:28 |
| melwitt | oh ok | 21:28 |
| KeithMnemonic | mriedem, ok i can open the bug, do you need anything specific in it or you have a handle on the issue? | 21:29 |
| mriedem | the paste is useful and recreate steps | 21:29 |
| KeithMnemonic | ok let me add all of that | 21:29 |
| mriedem | nice to know that regressions i introduced in queens are coming back to find me now | 21:31 |
| KeithMnemonic | lol | 21:32 |
| efried | mriedem: serves you right for not giving your features enough soak time. | 21:34 |
| mriedem | heh | 21:40 |
| mriedem | doesn't really fit here, but i can see that's been smoldering in your pocket for awhile | 21:40 |
| mriedem | i can never remember if you specify marker if the marker instance comes back as the first item in the list | 21:42 |
| mriedem | or if it's everything after the marker | 21:42 |
| mriedem | aha, ":param marker: the last item of the previous page; we returns the next | 21:43 |
| mriedem | results after this value." | 21:43 |
| *** otsukahy has joined #openstack-nova | 21:44 | |
| mriedem | KeithMnemonic: ok i've got a recreate functional test ready to push up whenever you have this bug filed | 21:52 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Add functional regression test for bug xxxxxx https://review.opendev.org/690428 | 22:00 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Fix listing deleted servers with a marker https://review.opendev.org/690429 | 22:00 |
| *** otsukahy has quit IRC | 22:01 | |
| *** otsukahy has joined #openstack-nova | 22:04 | |
| mriedem | ok well there is the reproduce and fix ^, will update once i have a bug id | 22:06 |
| *** otsukahy has quit IRC | 22:07 | |
| *** otsukahy has joined #openstack-nova | 22:07 | |
| *** otsukahy has quit IRC | 22:14 | |
| *** otsukahy has joined #openstack-nova | 22:15 | |
| *** TxGirlGeek has quit IRC | 22:18 | |
| *** mriedem has quit IRC | 22:23 | |
| *** otsukahy has quit IRC | 22:29 | |
| *** xek__ has quit IRC | 22:37 | |
| *** otsukahy has joined #openstack-nova | 22:40 | |
| *** otsukahy has quit IRC | 22:44 | |
| *** ivve has quit IRC | 22:45 | |
| *** ociuhandu has joined #openstack-nova | 22:46 | |
| *** ociuhandu has quit IRC | 22:51 | |
| *** tkajinam has joined #openstack-nova | 22:53 | |
| *** avolkov has quit IRC | 23:13 | |
| *** macz has quit IRC | 23:43 | |
| *** igordc has quit IRC | 23:55 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!