| *** mlavalle has quit IRC | 00:06 | |
| openstackgerrit | Ghanshyam Mann proposed openstack/nova master: Add test coverage of existing flavor_access policies https://review.opendev.org/713556 | 00:20 |
|---|---|---|
| *** tetsuro has joined #openstack-nova | 00:28 | |
| openstackgerrit | Ghanshyam Mann proposed openstack/nova master: Add test coverage of existing flavor_access policies https://review.opendev.org/713556 | 00:35 |
| *** lbragstad has joined #openstack-nova | 00:37 | |
| *** gyee has quit IRC | 00:38 | |
| *** ociuhandu has joined #openstack-nova | 00:42 | |
| openstackgerrit | Ghanshyam Mann proposed openstack/nova master: Introduce scope_types in os-flavor-access https://review.opendev.org/713559 | 00:46 |
| *** mriedem has left #openstack-nova | 00:47 | |
| *** larainema has quit IRC | 00:48 | |
| openstackgerrit | Ghanshyam Mann proposed openstack/nova master: Add test coverage of existing flavor_access policies https://review.opendev.org/713556 | 00:48 |
| openstackgerrit | Ghanshyam Mann proposed openstack/nova master: Introduce scope_types in os-flavor-access https://review.opendev.org/713559 | 00:48 |
| *** ociuhandu has quit IRC | 00:49 | |
| *** dave-mccowan has quit IRC | 00:55 | |
| *** nicolasbock has quit IRC | 01:01 | |
| openstackgerrit | Ghanshyam Mann proposed openstack/nova master: Add test coverage of existing flavor_access policies https://review.opendev.org/713556 | 01:05 |
| openstackgerrit | Ghanshyam Mann proposed openstack/nova master: Introduce scope_types in os-flavor-access https://review.opendev.org/713559 | 01:05 |
| openstackgerrit | Ghanshyam Mann proposed openstack/nova master: Introduce scope_types in os-flavor-access https://review.opendev.org/713559 | 01:06 |
| *** boldo has quit IRC | 01:08 | |
| *** brinzhang__ has joined #openstack-nova | 01:20 | |
| *** brinzhang_ has quit IRC | 01:23 | |
| *** ivve has quit IRC | 01:27 | |
| *** tbachman has quit IRC | 01:27 | |
| openstackgerrit | Merged openstack/nova master: FUP: Remove noqa and tone down an exception https://review.opendev.org/698054 | 01:28 |
| *** tbachman has joined #openstack-nova | 01:42 | |
| openstackgerrit | Kevin Zhao proposed openstack/nova master: fix scsi disk unit number of the attaching volume when cdrom bus is scsi https://review.opendev.org/712607 | 01:45 |
| *** mmethot has quit IRC | 01:53 | |
| *** mmethot has joined #openstack-nova | 01:53 | |
| *** yaawang has quit IRC | 01:56 | |
| *** yaawang has joined #openstack-nova | 01:57 | |
| *** tbachman has quit IRC | 01:57 | |
| openstackgerrit | Brin Zhang proposed openstack/python-novaclient master: Microversion 2.83: Add volume-attachment-update CLI https://review.opendev.org/712651 | 02:43 |
| *** factor has quit IRC | 02:52 | |
| *** tetsuro_ has joined #openstack-nova | 02:53 | |
| *** mkrai has joined #openstack-nova | 02:55 | |
| *** tetsuro has quit IRC | 02:56 | |
| openstackgerrit | melanie witt proposed openstack/nova master: DNM: try to get some debug info for bug 1844929 https://review.opendev.org/701478 | 03:04 |
| openstack | bug 1844929 in OpenStack Compute (nova) "grenade jobs failing due to "Timed out waiting for response from cell" in scheduler" [High,Confirmed] https://launchpad.net/bugs/1844929 | 03:04 |
| *** mmethot_ has joined #openstack-nova | 03:05 | |
| *** mmethot has quit IRC | 03:07 | |
| *** yaawang has quit IRC | 03:08 | |
| *** yaawang has joined #openstack-nova | 03:09 | |
| openstackgerrit | Yingji Sun proposed openstack/nova master: set different VirtualDevice.key https://review.opendev.org/713565 | 03:18 |
| *** dave-mccowan has joined #openstack-nova | 03:18 | |
| *** alex_xu has quit IRC | 03:29 | |
| *** zhanglong has joined #openstack-nova | 03:38 | |
| openstackgerrit | Yingji Sun proposed openstack/nova master: Set different VirtualDevice.key https://review.opendev.org/713565 | 03:53 |
| *** yaawang has quit IRC | 04:24 | |
| *** yaawang has joined #openstack-nova | 04:25 | |
| *** tetsuro has joined #openstack-nova | 04:29 | |
| *** tetsuro_ has quit IRC | 04:32 | |
| *** tkajinam has quit IRC | 04:33 | |
| *** tkajinam has joined #openstack-nova | 04:34 | |
| *** chenhaw has quit IRC | 04:49 | |
| *** udesale has joined #openstack-nova | 04:54 | |
| *** dave-mccowan has quit IRC | 04:56 | |
| *** dklyle has quit IRC | 05:04 | |
| *** ratailor has joined #openstack-nova | 05:13 | |
| *** TxGirlGeek has quit IRC | 05:26 | |
| *** lbragstad has quit IRC | 05:29 | |
| *** tetsuro_ has joined #openstack-nova | 05:32 | |
| *** tetsuro has quit IRC | 05:34 | |
| *** evrardjp has quit IRC | 05:35 | |
| *** evrardjp has joined #openstack-nova | 05:36 | |
| *** links has joined #openstack-nova | 05:40 | |
| openstackgerrit | Luyao Zhong proposed openstack/nova master: support live migration with vpmems https://review.opendev.org/687856 | 05:47 |
| openstackgerrit | Luyao Zhong proposed openstack/nova master: Track orphan instances and error migrations in resource tracker https://review.opendev.org/678451 | 05:47 |
| *** jdillaman has quit IRC | 05:51 | |
| *** jdillaman has joined #openstack-nova | 05:54 | |
| openstackgerrit | mashuting proposed openstack/nova master: Revert "Deprecate the vmwareapi driver" https://review.opendev.org/713401 | 06:19 |
| *** tetsuro has joined #openstack-nova | 06:40 | |
| *** tetsuro_ has quit IRC | 06:44 | |
| *** ociuhandu has joined #openstack-nova | 06:48 | |
| openstackgerrit | Kevin Zhao proposed openstack/nova master: fix scsi disk unit number of the attaching volume when cdrom bus is scsi https://review.opendev.org/712607 | 06:50 |
| *** ociuhandu has quit IRC | 06:52 | |
| *** tetsuro_ has joined #openstack-nova | 07:04 | |
| *** tetsuro has quit IRC | 07:07 | |
| *** dpawlik has joined #openstack-nova | 07:10 | |
| *** dpawlik has quit IRC | 07:26 | |
| *** ralonsoh has joined #openstack-nova | 07:31 | |
| *** mkrai has quit IRC | 07:33 | |
| *** dtantsur|afk is now known as dtantsur | 07:34 | |
| gibi | artom: ack, I will do a sweep | 07:39 |
| *** damien_r has joined #openstack-nova | 07:46 | |
| *** tesseract has joined #openstack-nova | 07:50 | |
| gibi | bauzas: do you still working on https://review.opendev.org/#/c/472685 ? | 07:55 |
| *** tosky has joined #openstack-nova | 07:57 | |
| *** mkrai has joined #openstack-nova | 07:58 | |
| *** dpawlik has joined #openstack-nova | 07:58 | |
| *** maciejjozefczyk has joined #openstack-nova | 07:59 | |
| *** slaweq has joined #openstack-nova | 08:04 | |
| *** maciejjozefczyk_ has joined #openstack-nova | 08:04 | |
| *** maciejjozefczyk has quit IRC | 08:05 | |
| gibi | stephenfin: is this still relevant? https://review.opendev.org/#/c/498820 | 08:08 |
| gibi | stephenfin: and this https://review.opendev.org/#/c/498819 ? | 08:08 |
| *** yoctozepto9 has joined #openstack-nova | 08:15 | |
| *** yoctozepto has quit IRC | 08:15 | |
| *** yoctozepto9 is now known as yoctozepto | 08:15 | |
| *** tkajinam has quit IRC | 08:18 | |
| *** rcernin has quit IRC | 08:21 | |
| *** amoralej|off is now known as amoralej | 08:24 | |
| *** slaweq has quit IRC | 08:28 | |
| *** slaweq has joined #openstack-nova | 08:29 | |
| *** xek has joined #openstack-nova | 08:31 | |
| *** tetsuro has joined #openstack-nova | 08:35 | |
| *** tetsuro_ has quit IRC | 08:38 | |
| *** lee1 has joined #openstack-nova | 08:43 | |
| *** lee1 is now known as lyarwood | 08:43 | |
| *** ociuhandu has joined #openstack-nova | 08:45 | |
| *** rpittau|afk is now known as rpittau | 08:45 | |
| *** yoctozepto has quit IRC | 08:49 | |
| *** yoctozepto has joined #openstack-nova | 08:59 | |
| *** tetsuro has quit IRC | 09:00 | |
| *** maciejjozefczyk_ is now known as maciejjozefczyk | 09:01 | |
| *** dpawlik has quit IRC | 09:03 | |
| *** dpawlik has joined #openstack-nova | 09:03 | |
| bauzas | gibi: morning | 09:03 |
| gibi | bauzas: hi! | 09:03 |
| bauzas | gibi: not really, I can abandon it | 09:03 |
| bauzas | gibi: how are you ? are you locked down too ? | 09:03 |
| gibi | bauzas: please abandon it. thanks | 09:04 |
| gibi | bauzas: general lockdown yes. groceries are open between 6:00 - 15:00 everything else is closed | 09:04 |
| bauzas | gibi: and you can walk outside home ? | 09:05 |
| * bauzas is glad to have a dog | 09:05 | |
| gibi | bauzas: yeah walks and forests are open | 09:05 |
| gibi | only the gatherings are forbidden | 09:05 |
| gibi | bauzas: do you only allowed to walk outside if you have a dog? | 09:06 |
| bauzas | okay, b/c for us, we have to stay home unless 5 reasons | 09:06 |
| bauzas | food, work (but only if very needed), urgent needing, and dogwalks yes :p | 09:06 |
| bauzas | you can't for example run outside | 09:07 |
| bauzas | only around your house (like 100m) | 09:07 |
| gibi | interesting distinction | 09:07 |
| *** dpawlik has quit IRC | 09:08 | |
| gibi | anyhow I mostly agree with the closedown | 09:09 |
| *** zhanglong has quit IRC | 09:09 | |
| bauzas | me too | 09:11 |
| gibi | my wife will make the flat soo clean as she has nothing else to really do :) | 09:11 |
| bauzas | just the fact that it's a bit difficult to both work and teach my children | 09:11 |
| gibi | yeah home schooling is an extra job | 09:12 |
| *** iurygregory has quit IRC | 09:12 | |
| *** ociuhandu has quit IRC | 09:12 | |
| *** ociuhandu has joined #openstack-nova | 09:12 | |
| gibi | but we have no kids so I only know that hardness from my coworkers and my sister | 09:13 |
| *** iurygregory has joined #openstack-nova | 09:14 | |
| gibi | artom: I abandoned a bunch of patches thanks for pushing me looking at that list. | 09:19 |
| *** martinkennelly has joined #openstack-nova | 09:32 | |
| *** derekh has joined #openstack-nova | 09:33 | |
| kashyap | bauzas: Running is still allowed here. | 09:42 |
| kashyap | gibi: We will only be able to lockdown is the right decision retrospectively. | 09:43 |
| *** mmethot has joined #openstack-nova | 09:52 | |
| *** mmethot_ has quit IRC | 09:54 | |
| bauzas | https://www.thelocal.fr/20200318/frances-coronavirus-lockdown-form-your-questions-answered | 09:58 |
| bauzas | kashyap: ^ as you can read, running is possible but with a very short distance | 09:59 |
| *** yoctozepto has quit IRC | 09:59 | |
| bauzas | within* | 09:59 |
| * kashyap nods | 09:59 | |
| bauzas | (and alone) | 09:59 |
| kashyap | Yeah | 10:00 |
| bauzas | the main issue is that kids have zero possibilities to go outside, they're literally locked down for 3 or more weeks | 10:00 |
| bauzas | I'm glad I have a house with garden | 10:00 |
| *** corvus_ has joined #openstack-nova | 10:01 | |
| *** mkrai_ has joined #openstack-nova | 10:02 | |
| *** mkrai has quit IRC | 10:02 | |
| *** yoctozepto has joined #openstack-nova | 10:03 | |
| *** corvus has quit IRC | 10:04 | |
| openstackgerrit | Balazs Gibizer proposed openstack/nova master: Support unshelve with qos ports https://review.opendev.org/704759 | 10:05 |
| openstackgerrit | Balazs Gibizer proposed openstack/nova master: Enable unshelve with qos ports https://review.opendev.org/705475 | 10:06 |
| *** ociuhandu has quit IRC | 10:09 | |
| *** ociuhandu has joined #openstack-nova | 10:10 | |
| stephenfin | gibi: FYI https://github.com/testing-cabal/subunit/commit/26d31fa7c34019fad9038addf8114bbb4b656c92 | 10:13 |
| *** yoctozepto2 has joined #openstack-nova | 10:13 | |
| *** yoctozepto has quit IRC | 10:13 | |
| *** yoctozepto2 is now known as yoctozepto | 10:13 | |
| stephenfin | That should kill the "subunit parser error" errors we were getting any time we logged too many warning | 10:14 |
| stephenfin | *s | 10:14 |
| *** ociuhandu has quit IRC | 10:15 | |
| gibi | stephenfin: woot \o/ | 10:16 |
| gibi | we just need to bump the subunit dependency to 1.4.0 | 10:17 |
| stephenfin | It's already done in openstack/requirements. We just need to fix lower-constraints now, yeah | 10:17 |
| gibi | stephenfin: hit me with any review where I can help | 10:18 |
| gibi | stephenfin: I really appreciate your effort on fixing this long outstanding bug | 10:18 |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: Bump python-subunit minimum to 1.4.0 https://review.opendev.org/713611 | 10:21 |
| *** ociuhandu has joined #openstack-nova | 10:21 | |
| stephenfin | gibi: ^ | 10:21 |
| stephenfin | Also, if you're in the reviewing mood, these two patches are super trivial and good to go https://review.opendev.org/#/c/704533/1 https://review.opendev.org/#/c/710276/2 | 10:22 |
| lyarwood | stephenfin: morning morning, how was your lock-in yesterday? | 10:25 |
| gibi | stephenfin: regarding https://review.opendev.org/#/c/710276/2/tox.ini , those thing that you removed are the default now? | 10:26 |
| lyarwood | stephenfin: Could you hit https://review.opendev.org/#/q/status:open+topic:workaround_native_luksv1 & https://review.opendev.org/#/q/status:open+topic:rbd_block_device today if you have time? | 10:26 |
| bauzas | stephenfin: with thoughts ot Saint Patrick, I gladly drank a beer yesterday night to celebrate his memory | 10:26 |
| *** ociuhandu has quit IRC | 10:27 | |
| openstackgerrit | Merged openstack/os-resource-classes master: setup.cfg: Configure 'python-requires' https://review.opendev.org/704533 | 10:46 |
| *** jawad_axd has joined #openstack-nova | 10:50 | |
| *** tbachman has joined #openstack-nova | 10:51 | |
| lyarwood | Random question, trying to force an instance to launch on a given host in a fresh stable/queens devstack env using --availability-zone nova:unassigned-hostname but it's always failing in n-sch somehow (shouldn't it skip the filters?) | 11:03 |
| *** nicolasbock has joined #openstack-nova | 11:03 | |
| lyarwood | ^ is there any config updates required to make that work with a default devstack env on stable/queens? | 11:03 |
| * lyarwood is trying to reproduce some bdm reservation weirdness when launching an instance on a specific host fails | 11:04 | |
| *** jaosorior has quit IRC | 11:06 | |
| bauzas | lyarwood: which exact command do you issue ? | 11:08 |
| lyarwood | openstack server create --image cirros-0.3.5-x86_64-disk --flavor m1.xlarge --block-device-mapping vdb=4776528c-f18c-4750-844d-41cbfaf802ab --availability-zone nova:unassigned-hostname test | 11:09 |
| bauzas | lyarwood: and IIRC, --az <host>:<node>:az | 11:09 |
| bauzas | lemme double-check the API | 11:09 |
| lyarwood | https://docs.openstack.org/nova/queens/admin/availability-zones.html | 11:10 |
| lyarwood | ^ is what I was going off | 11:10 |
| lyarwood | and the customers reproducer downstream | 11:10 |
| lyarwood | that uses az:host | 11:10 |
| bauzas | sec, checking | 11:10 |
| lyarwood | compute_api.parse_availability_zone also uses that | 11:10 |
| bauzas | https://github.com/openstack/nova/blob/master/nova/compute/api.py#L475 | 11:11 |
| bauzas | lyarwood: but look | 11:11 |
| bauzas | if you don't provide a tuple of 3 items (az, host, node) but only 2 (az, host) then you need to double ":" | 11:12 |
| bauzas | lyarwood: ^ | 11:12 |
| bauzas | https://github.com/openstack/nova/blob/master/nova/compute/api.py#L488 | 11:12 |
| bauzas | holy snap | 11:12 |
| bauzas | I'm wrong | 11:12 |
| lyarwood | yeah you only need 1 | 11:13 |
| bauzas | "::" is for (az, host | 11:13 |
| bauzas | damn | 11:13 |
| bauzas | (az, node) | 11:13 |
| bauzas | lyarwood: I guess you have nova service-list telling you that you have nova-compute on some random host literrally named 'unassigned-hostname' ? | 11:14 |
| lyarwood | $ openstack compute service list --service nova-compute | 11:14 |
| lyarwood | +----+--------------+---------------------+------+---------+-------+----------------------------+ | 11:14 |
| lyarwood | | ID | Binary | Host | Zone | Status | State | Updated At | | 11:14 |
| lyarwood | +----+--------------+---------------------+------+---------+-------+----------------------------+ | 11:14 |
| lyarwood | | 2 | nova-compute | unassigned-hostname | nova | enabled | up | 2020-03-18T11:14:39.000000 | | 11:14 |
| lyarwood | +----+--------------+---------------------+------+---------+-------+----------------------------+ | 11:15 |
| lyarwood | yeah | 11:15 |
| *** tesseract-RH has joined #openstack-nova | 11:16 | |
| lyarwood | Mar 18 07:02:06 unassigned-hostname nova-scheduler[3674]: DEBUG nova.scheduler.manager [None req-27d6db0a-3f02-4350-8bab-7cc9f93639d0 admin admin] Got no allocation candidates from the Placement API. This may be a temporary occurrence as compute nodes start up and begin reporting inventory to the Placement service. - ah | 11:16 |
| lyarwood | maybe we are bailing on before we even get to filtering the hosts | 11:17 |
| *** tesseract has quit IRC | 11:18 | |
| *** ociuhandu has joined #openstack-nova | 11:19 | |
| *** jaosorior has joined #openstack-nova | 11:20 | |
| lyarwood | https://github.com/openstack/nova/blob/78e53bd388a5c2305da5249f1f217923d3fe2aac/nova/scheduler/manager.py#L134-L139 yup that's it | 11:20 |
| bauzas | lyarwood: we don't go with checking the filters if you specify a target like this | 11:20 |
| lyarwood | yeah it's failing before we get anywhere near there | 11:20 |
| lyarwood | I think it's the hostname causing issues | 11:20 |
| lyarwood | I'll unstack, set something valid and try again | 11:20 |
| bauzas | weirdo | 11:21 |
| lyarwood | yup indeed | 11:21 |
| *** ociuhandu has quit IRC | 11:35 | |
| *** martinkennelly has quit IRC | 11:43 | |
| stephenfin | lyarwood: Sorry, meetings. Yup, I'll take a look | 11:45 |
| stephenfin | and my lock was me cooped up at home, figuring out how traefik worked with a single beer | 11:45 |
| *** mkrai_ has quit IRC | 11:46 | |
| *** ivve has joined #openstack-nova | 11:47 | |
| stephenfin | gibi: replied on https://review.opendev.org/#/c/710276/ | 11:48 |
| *** martinkennelly has joined #openstack-nova | 11:48 | |
| lyarwood | stephenfin: my deepest condolences | 11:48 |
| stephenfin | I should have been clearer in the commit message - I'd forgotten half the reason I'd done some of that myself. Can edit it if you'd like | 11:48 |
| lyarwood | bauzas: so I'm still getting no allocation candidates from placement | 11:49 |
| stephenfin | yo, bauzas, can you hit https://review.opendev.org/#/c/710239 please? | 11:50 |
| * bauzas goes for lunch but will back in 1 hour | 11:51 | |
| lyarwood | ah wait I'm being stupid | 11:52 |
| *** sapd1 has quit IRC | 11:53 | |
| openstackgerrit | Merged openstack/nova master: bug-fix: Reject live migration with vpmem https://review.opendev.org/708110 | 11:55 |
| *** rpittau is now known as rpittau|bbl | 12:01 | |
| *** ralonsoh has quit IRC | 12:05 | |
| openstackgerrit | Merged openstack/nova master: VMware VMDK detach: get adapter type from instance VM https://review.opendev.org/653738 | 12:09 |
| *** ralonsoh has joined #openstack-nova | 12:09 | |
| *** zhanglong has joined #openstack-nova | 12:10 | |
| openstackgerrit | Luyao Zhong proposed openstack/nova master: support live migration with vpmems https://review.opendev.org/687856 | 12:14 |
| *** nweinber has joined #openstack-nova | 12:18 | |
| openstackgerrit | Luyao Zhong proposed openstack/nova master: Track orphan instances and error migrations in resource tracker https://review.opendev.org/678451 | 12:21 |
| *** ratailor has quit IRC | 12:32 | |
| lyarwood | eh when did ubuntu make upgrade and dist-upgrade different? | 12:42 |
| lyarwood | I hit this again https://bugs.launchpad.net/os-vif/+bug/1749972 because virt-builder is just running upgrade when you build with --update | 12:42 |
| openstack | Launchpad bug 1749972 in os-vif "`brctl setageing $bridge 0` fails on Ubuntu 16.04 4.4.0-21-generic" [Undecided,Invalid] | 12:42 |
| lyarwood | *sigh* | 12:43 |
| sean-k-mooney | lyarwood: its always been different | 12:43 |
| sean-k-mooney | lyarwood: dist-upgrade upgrades your kernel and all other packages | 12:43 |
| sean-k-mooney | upgrade will hold back some packages that would require a hots reboot to take effect | 12:44 |
| *** udesale_ has joined #openstack-nova | 12:44 | |
| lyarwood | sean-k-mooney: ack thanks, I haven't used Ubuntu as a daily OS in something like 12 years so I guess I had just forgotten this | 12:44 |
| *** jaosorior has quit IRC | 12:45 | |
| sean-k-mooney | ya i use it daily so i swap between the two depening on what behavior i want | 12:45 |
| jkulik | upgrade will also hold back packages needing new dependencies to be installed or other packages to be uninstalled. not only reboots. at least in debian. | 12:45 |
| sean-k-mooney | jkulik: correctg | 12:46 |
| sean-k-mooney | upgrade is more concervitive | 12:46 |
| *** udesale has quit IRC | 12:47 | |
| sean-k-mooney | lyarwood: how did that affect that bug | 12:47 |
| sean-k-mooney | was brctl not upgraded? | 12:47 |
| sean-k-mooney | lyarwood: also why are you using 16.04? aren't all stable brnaches that use that em at this point? | 12:49 |
| *** amoralej is now known as amoralej|lunch | 12:53 | |
| lyarwood | sean-k-mooney: deploying stable/queens | 12:54 |
| sean-k-mooney | i think queens runs fine on 18.04 as well | 12:54 |
| sean-k-mooney | but ok | 12:55 |
| lyarwood | sean-k-mooney: as you said in that bug it appears that something in the kernel changed during the lifecycle of that release | 12:55 |
| lyarwood | sean-k-mooney: yeah I was just copying CI | 12:55 |
| sean-k-mooney | i had completely forgoten about that bug report so thanks for marking it invalid | 12:55 |
| lyarwood | np | 12:56 |
| * lyarwood -> lunch brb | 12:56 | |
| *** rpittau|bbl is now known as rpittau | 13:01 | |
| artom | gibi, thanks for the cleanup :) | 13:01 |
| artom | I would have been even more agressive - like https://review.opendev.org/#/c/525335 is from 2 years ago with no response | 13:02 |
| artom | But I won't tell you how to do your job ;) | 13:02 |
| openstackgerrit | waleed mousa proposed openstack/os-vif master: [Follow Up] OVS DPDK port representors support https://review.opendev.org/705018 | 13:03 |
| *** ygk_12345 has joined #openstack-nova | 13:04 | |
| luyao | artom: Hi | 13:04 |
| ygk_12345 | hi all | 13:04 |
| ygk_12345 | i have rocky setup. | 13:04 |
| ygk_12345 | when I try to launch an instance on a specific compute node , it is throwing this error in the nova-compute logs | 13:05 |
| ygk_12345 | Failed to retrieve resource provider tree from placement API for UUID ea3af0ff-fb69-449a-b484-5ac722010a1f | 13:05 |
| *** tesseract has joined #openstack-nova | 13:05 | |
| gibi | artom: you recently asked questions on https://review.opendev.org/#/c/525335 so I allow the author to answer. :) | 13:07 |
| sean-k-mooney | ygk_12345: does the vm launch correctly | 13:07 |
| *** lbragstad has joined #openstack-nova | 13:07 | |
| ygk_12345 | sean-k-mooney no . it is going into error state | 13:08 |
| *** tesseract-RH has quit IRC | 13:08 | |
| *** obre has quit IRC | 13:08 | |
| sean-k-mooney | ygk_12345: i cant recall which release we made placmenert mandatory but have you deployed the placement service | 13:08 |
| ygk_12345 | sean-k-mooney its rocky 18.1.9 branch in openstack ansible | 13:08 |
| sean-k-mooney | did you delete the compute node and recreate it? | 13:09 |
| ygk_12345 | sean-k-mooney i tried upgrading the working rocky setup to stein . it failed and so rolled back to rocky now | 13:09 |
| ygk_12345 | yes | 13:09 |
| ygk_12345 | sean-k-mooney how to delete the compute node ? | 13:10 |
| ygk_12345 | sean-k-mooney i re ran the nova playbook | 13:10 |
| sean-k-mooney | so this is not a nova issue. its and issue with osa and the operation you performed. | 13:10 |
| ygk_12345 | sean-k-mooney so whats the workaround now ? | 13:11 |
| sean-k-mooney | i think we have disucssed this before but if you delete the compute service and you restart the agent without altering the host name of the compute host then it will create a new compute service record in the db with a differnet uuid | 13:11 |
| sean-k-mooney | but that will prevent the the agent form creating the resouce provider with the new uuid as the RP name will still be in use | 13:12 |
| sean-k-mooney | can you validate that by checking if there is a placmenet RP with the hostname? | 13:12 |
| ygk_12345 | sean-k-mooney i have altered the hostname and it created the new record in resource_providers db in nova_api table | 13:12 |
| luyao | artom: If you see my message, could you help look at https://review.opendev.org/#/c/687856/15/nova/compute/manager.py@8336, I change the order of cleanup during post live migration. We do post_live_migration_at_destination first and then cleanup source host in existing code logic, if there are some specific reason ? Or if the order doesn't matter? | 13:12 |
| ygk_12345 | sean-k-mooney how to do that ? | 13:12 |
| *** obre has joined #openstack-nova | 13:15 | |
| *** ccamacho has quit IRC | 13:16 | |
| artom | gibi, fair enough :) | 13:16 |
| sean-k-mooney | you can use the plamcent openstack client plugin to list the resouce providers and grep for the hostname | 13:17 |
| gibi | artom: ping me next week and I will close out the next patch of reviews including the one you commented :) | 13:17 |
| ygk_12345 | sean-k-mooney is placement included at all in rocky ? | 13:17 |
| artom | luyao, looking... | 13:18 |
| artom | gibi, hehe, ack | 13:18 |
| sean-k-mooney | ygk_12345: it was intoduced in pike maybe earlier | 13:18 |
| *** sapd1 has joined #openstack-nova | 13:18 | |
| sean-k-mooney | ygk_12345: so yes it should be included in rocky | 13:19 |
| luyao | artom: thanks | 13:19 |
| *** vishalmanchanda has joined #openstack-nova | 13:19 | |
| ygk_12345 | sean-k-mooney how to install the placement plugin for the openstack client | 13:20 |
| sean-k-mooney | i think its just "pip install osc-placement" | 13:21 |
| ygk_12345 | sean-k-mooney should I do it in the utility node in openstack ansible ? | 13:21 |
| sean-k-mooney | well "python3 -m pip install --user osc-placement" woudl be more modern way of doing it | 13:21 |
| lyarwood | bauzas: random scheduler question for you, if we fail to schedule an instance and it moves to ERROR can we ever retry launching that instance or can we only delete it? | 13:21 |
| lyarwood | bauzas: working through an issue where failure to schedule an instance with attached volumes leaves these volumes in a reserved state | 13:22 |
| sean-k-mooney | ygk_12345: this really isnt a supprot channel and definetly not for openstack ansible so i dont know. | 13:22 |
| lyarwood | bauzas: but if we are able to schedule and/or go straight to a specific host and then fail the volumes become available | 13:22 |
| lyarwood | I assume we want things to stay reserved *if* we can relaunch the instances somehow | 13:23 |
| ygk_12345 | sean-k-mooney oh ok | 13:23 |
| *** brinzhang__ has quit IRC | 13:31 | |
| *** brinzhang__ has joined #openstack-nova | 13:31 | |
| *** zhanglong has quit IRC | 13:31 | |
| *** zhanglong has joined #openstack-nova | 13:32 | |
| *** ociuhandu has joined #openstack-nova | 13:34 | |
| luyao | stephenfin: Hi, I saw you rechecked my vpmem-live-migration patch. FYI, I submitted a new version, there are a bug in the previous one causing the tempest test failed under mixed-version compute services, I think I addressed it for now, you can look at it if you have time. | 13:35 |
| bauzas | lyarwood: I'm back | 13:37 |
| kashyap | lyarwood: I can't see how your patch is the reason for the 'tempest-integrated-compute' failure | 13:37 |
| kashyap | (Not that you're claiming it :)) | 13:37 |
| *** ociuhandu has quit IRC | 13:38 | |
| lyarwood | kashyap: which one sorry? | 13:39 |
| luyao | artom: I'm afraid I can't response to you promptly in IRC , so I'll appreciate that if you can leave comments under that patch. | 13:39 |
| kashyap | lyarwood: The blockCopy one; sorry - https://zuul.opendev.org/t/openstack/build/1796646647854d128ba128b45d5c7a14 | 13:39 |
| artom | luyao, yeah, that's fine | 13:40 |
| * kashyap goes for a bike ride, to alleviate some cabin fever | 13:40 | |
| bauzas | lyarwood: to answer your question, you could rebuild I think but not just 'create' again | 13:42 |
| *** umbSublime_ has joined #openstack-nova | 13:42 | |
| *** lbragstad_ has joined #openstack-nova | 13:42 | |
| artom | dansmith, when you're about, you don't have to look at the entire patch, but just the https://review.opendev.org/#/c/687856/15/nova/conductor/tasks/live_migrate.py@57 bit, about using compute service versions to judge support for a thing | 13:42 |
| *** jraju__ has joined #openstack-nova | 13:43 | |
| lyarwood | bauzas: so we can't just `nova start $instance` | 13:43 |
| lyarwood | bauzas: we have to rebuild? | 13:43 |
| *** links has quit IRC | 13:44 | |
| lyarwood | bauzas: either way, I'd assume that given that's an option the volumes should stay reserved right? | 13:44 |
| *** umbSublime has quit IRC | 13:44 | |
| *** lbragstad has quit IRC | 13:44 | |
| *** amoralej|lunch is now known as amoralej | 13:46 | |
| *** liuyulong has joined #openstack-nova | 13:46 | |
| bauzas | lyarwood: 'start' recreates the guest XML but doesn't ask for cinder and neutron | 13:49 |
| bauzas | (AFAIR) | 13:49 |
| lyarwood | bauzas: it should update the attachment but it wouldn't recreate it | 13:50 |
| bauzas | yup, this | 13:51 |
| bauzas | either way ,if the instance was set to ERROR on scheduling stage, then the related cell is cell° | 13:51 |
| bauzas | cell0 sorry | 13:51 |
| bauzas | so you're messed up | 13:52 |
| bauzas | that's why a rebuild or resize is the only option | 13:52 |
| sean-k-mooney | the nova start api action calls power on the virt driver which in the libvirt driver calls hard reboot because reasons | 13:56 |
| *** Liang__ has joined #openstack-nova | 13:57 | |
| *** Liang__ is now known as LiangFang | 13:57 | |
| sean-k-mooney | so yes it recreates the xml. it might rettrive attachemnt info form cinder but it will not modify the sate of the volume in cinder | 13:57 |
| *** dave-mccowan has joined #openstack-nova | 13:57 | |
| *** zhanglong has quit IRC | 13:57 | |
| lyarwood | sean-k-mooney / bauzas ; ack thanks | 13:57 |
| lyarwood | so I'm thinking that either way the volume needs to remain reserved | 13:58 |
| lyarwood | if we ERROR out during scheduling (current behaviour) | 13:58 |
| *** zhanglong has joined #openstack-nova | 13:58 | |
| lyarwood | or if we ERROR out in the virt driver during spawn (currently moves back to available) | 13:58 |
| sean-k-mooney | well if you stop an instance then start it we should not hit the scuduler | 13:58 |
| sean-k-mooney | we dont for a hard reboot | 13:58 |
| lyarwood | right | 13:58 |
| sean-k-mooney | a rebuild need to hit the schduler to assert the new image does not contian metadata that invaldiate the current host | 13:59 |
| sean-k-mooney | but start and reboot dont cause nothing changes | 13:59 |
| lyarwood | right | 13:59 |
| sean-k-mooney | lyarwood: whats the issue you are looking at by the way | 13:59 |
| lyarwood | sean-k-mooney: as above, when we ERROR out during scheduling attached volumes remain in a reserved state (as we don't delete their attachments in cinder), if we fail during spawn the compute layer moves the volume back to available (by deleting the attachments in cinder). | 14:01 |
| nightmare_unreal | hello where can I see list of bugs for openstack CLI client or nova related bugs | 14:01 |
| sean-k-mooney | lyarwood: ah ok | 14:01 |
| lyarwood | sean-k-mooney: so I'm thinking that for the latter we should keep the volume reserved to avoid something or someone else coming along and using it | 14:01 |
| lyarwood | if it isn't multiattach etc etc | 14:01 |
| * lyarwood goes to write a functional test | 14:02 | |
| *** zhanglong has quit IRC | 14:02 | |
| sean-k-mooney | lyarwood: the later being during spawn when we delete the attachemtn we leave it in a reserved sate | 14:02 |
| lyarwood | sean-k-mooney: when we delete the attachments the volume moves back to an available state | 14:03 |
| lyarwood | sean-k-mooney: I'd like to change this to keep it reserved by not deleting the attachments | 14:03 |
| sean-k-mooney | in the case wehre start fails or boot fails or both | 14:04 |
| sean-k-mooney | for boot it will end up in cell 0 and have host so no attachemts | 14:04 |
| sean-k-mooney | *no host | 14:04 |
| lyarwood | for spawn, so we have a host and a cell already | 14:05 |
| lyarwood | brb | 14:06 |
| *** dklyle has joined #openstack-nova | 14:11 | |
| *** yoctozepto1 has joined #openstack-nova | 14:12 | |
| *** yoctozepto has quit IRC | 14:13 | |
| *** yoctozepto1 is now known as yoctozepto | 14:13 | |
| *** brinzhang__ has quit IRC | 14:14 | |
| *** ccamacho has joined #openstack-nova | 14:21 | |
| sean-k-mooney | nightmare_unreal: nova bugs are here https://bugs.launchpad.net/nova | 14:24 |
| sean-k-mooney | nightmare_unreal: the openstack client uses storyborad | 14:24 |
| sean-k-mooney | so here https://storyboard.openstack.org/#!/project/openstack/python-openstackclient but this is all the releated projects https://storyboard.openstack.org/#!/project_group/80 | 14:25 |
| nightmare_unreal | thanks :) | 14:26 |
| *** ygk_12345 has quit IRC | 14:30 | |
| dansmith | artom: your sigh about migration func testing is lack of review I assume? | 14:34 |
| dansmith | I'm sure I'm at least partially to blame there | 14:34 |
| artom | dansmith, no, mostly towards myself | 14:35 |
| dansmith | oh okay | 14:35 |
| artom | For not staying on top of it | 14:35 |
| artom | I wanted to do a push, well, starting now | 14:36 |
| artom | I still have stephenfin's feedback from the previous millenium to address (about the 2/3 NUMA nodes split) | 14:36 |
| artom | But if we manage to land it before the VPMEM live migration patch, it would make things easier | 14:37 |
| dansmith | ack | 14:37 |
| dansmith | I'm pretty un-jazzed about that patch | 14:37 |
| openstackgerrit | Balazs Gibizer proposed openstack/nova master: Support unshelve with qos ports https://review.opendev.org/704759 | 14:38 |
| openstackgerrit | Balazs Gibizer proposed openstack/nova master: Enable unshelve with qos ports https://review.opendev.org/705475 | 14:38 |
| artom | dansmith, live migration with VPMEM? | 14:38 |
| dansmith | yes | 14:38 |
| artom | In terms of current implementation or the concept in general? | 14:39 |
| artom | I'm just neutral-jazzed - I got involved because my pawprints are all over that code after NUMA live migration | 14:39 |
| dansmith | well, I'm rather negative on the vpmem stuff in general, but that ship has sailed..But yeah, the current patch more acutely | 14:39 |
| openstackgerrit | Balazs Gibizer proposed openstack/nova master: Merge qos related renos for Ussuri https://review.opendev.org/706766 | 14:40 |
| dansmith | it smacks of "let me just move around all these critical things so they work the way I need them" | 14:40 |
| artom | Yeah... | 14:40 |
| lyarwood | sean-k-mooney: https://bugs.launchpad.net/neutron/+bug/1867936 seen this before? | 14:40 |
| openstack | Launchpad bug 1867936 in neutron "test_update_delete_extra_route failing due to timeout when creating subnets" [Undecided,New] | 14:40 |
| artom | dansmith, you've essentially put words on a vague feeling I had | 14:41 |
| sean-k-mooney | lyarwood: only in the failure of one of your patches | 14:41 |
| artom | Which is why I would really be more comfortable if we at least had NUMA LM func tests in-tree before merging | 14:41 |
| gibi | dansmith: hi! added the global service version check to https://review.opendev.org/#/c/705475 | 14:42 |
| artom | Those at least exercise those code paths | 14:42 |
| dansmith | artom: well, I certainly don't think that's unreasonable, as long as we can get that moving | 14:42 |
| artom | dansmith, ack, lemme get on that | 14:42 |
| lyarwood | sean-k-mooney: yeah I've seen it a few times, was just wondering if someones already raised a more specific bug report for this | 14:42 |
| dansmith | artom: using your patch to validate what we have in tree so we can more directly assess the changes this one makes would be helpful | 14:43 |
| dansmith | gibi: ack | 14:43 |
| artom | dansmith, the bottom two of https://review.opendev.org/#/c/705545/ would be ready, I'd say | 14:43 |
| sean-k-mooney | lyarwood: there have been some issue with the dhcp agent by the way. e.g. it not seting up the dhcp config in a timely manner | 14:43 |
| dansmith | gibi: I feel like we might be at the point with the cyborg set where someone else needs to take over it.. I'm not fully caught up yet this morning but I think feedback has been pending for a while now | 14:43 |
| sean-k-mooney | this might just be a side effect of that althogh i though that neuton had fixed that | 14:43 |
| artom | dansmith, if you have any -1s to get in on those, do it now ;) | 14:44 |
| lyarwood | sean-k-mooney: kk | 14:44 |
| dansmith | artom: at least the bottom one is -W, should it not be? | 14:44 |
| gibi | dansmith: I agree. I can spend some review time on that series but I cannot promise to take over the impl | 14:44 |
| artom | dansmith, it's to indicate that the top 2 aren't ready | 14:44 |
| artom | It shouldn't stop you from -1'ing :) | 14:45 |
| dansmith | gibi: no, not asking the latter, but it would be good if we could have the next few above your +2s reviewed and then make a big effort there | 14:45 |
| gibi | dansmith: OK. I will look into those patches soon | 14:45 |
| dansmith | gibi: thanks, I'll start in on your unshelve patch | 14:46 |
| gibi | dansmith: thanks | 14:46 |
| gibi | dansmith: I'm working on retrofitting the service version check to the other qos move operations implemented in Ussuri | 14:47 |
| gibi | dansmith: but the pattern is there in the unshelve patch | 14:48 |
| *** dtantsur is now known as dtantsur|brb | 14:48 | |
| dansmith | gibi: okay, things that are already landed are kinda already out of the bag, unless you mean gating everything on this service version for people that haven't rolled yet | 14:48 |
| gibi | dansmith: I'm planning to gate it on the service version I added at the unshelve patch | 14:49 |
| gibi | dansmith: does it make sense to you? | 14:49 |
| gibi | dansmith: I don't think many people rolled Ussuri yet | 14:49 |
| dansmith | okay.. technically that is kinda making history go backwards, but for the practical case where people are just deploying releases, it makes sense | 14:49 |
| dansmith | I don't really want to get too far into that line of thinking, because it leads us towards "break things during the release as long as we fix before GA" but... this isn't quite that | 14:50 |
| gibi | dansmith: I consider it as a bug that was introduced in Ussuri and fixed in Ussuri | 14:50 |
| * dansmith nods | 14:51 | |
| * gibi going back to code | 14:51 | |
| nightmare_unreal | how can i report a bug , it's related to openstack client | 14:51 |
| nightmare_unreal | use storyboard ? | 14:51 |
| *** mkrai has joined #openstack-nova | 14:52 | |
| stephenfin | nightmare_unreal: yeah, I think so | 14:53 |
| *** macz_ has joined #openstack-nova | 14:55 | |
| *** corvus_ is now known as jeblair | 14:56 | |
| *** jeblair is now known as corvus | 14:56 | |
| *** lbragstad__ has joined #openstack-nova | 14:57 | |
| *** lbragstad_ has quit IRC | 14:59 | |
| *** priteau has joined #openstack-nova | 15:00 | |
| lyarwood | sean-k-mooney: https://zuul.opendev.org/t/openstack/build/b62f4ef603904e05bc34e2a32a4d2866/log/logs/screen-n-cpu.txt#6754 - so the nova-live-migration evacuation failure looks like another race between libvirtd and n-cpu btw | 15:03 |
| lyarwood | at least in this example | 15:04 |
| lyarwood | ah ignore me that's an earlier negative test | 15:08 |
| *** TxGirlGeek has joined #openstack-nova | 15:10 | |
| dansmith | gibi: maybe you can save me some time checking on this, but... it looks like we do set reqspec.requested_resources in various places, | 15:15 |
| dansmith | but this patch just blindly sets that on reqspec (as you do elsewhere) | 15:16 |
| dansmith | is the _only_ time any of requested_resources gets used is for port qos right now and you're just taking the shortcut of being aggressive there? | 15:16 |
| *** mlavalle has joined #openstack-nova | 15:16 | |
| dansmith | I note the comment to that effect, but I just want to make sure you're positive that it's only used for that, and that it's always safe to blow those away | 15:17 |
| gibi | dansmith: I'm positive it is use only for that (except the cyborg case where I'm not sure what is merged yet) | 15:17 |
| dansmith | nothing for it is merged, so that's safe | 15:17 |
| gibi | but I do remember that the cyborg change was proper and did not overwrite the requested_resources | 15:18 |
| gibi | dansmith: so I think we are safe there | 15:18 |
| dansmith | seems a little risky to just be blind about these early cases, and that it's going to be a source of bugs later where cyborg uses it for spawn, for example but unshelve overwrites those | 15:18 |
| openstackgerrit | Lee Yarwood proposed openstack/nova master: gate: Ensure n-cpu is stopped on the subnode during evacuation testing https://review.opendev.org/713674 | 15:19 |
| lyarwood | sean-k-mooney: ^ so I wonder if n-cpu running on the subnode is borking things here even if it is marked as down in the API? | 15:19 |
| lyarwood | I can see the subnode refreshing the instance network info cache for example | 15:20 |
| lyarwood | while we are trying to evacuate | 15:20 |
| gibi | dansmith: the field is not persisted so I don't think we will overwrite existing data from the past. every user of the requested_resources needs to re-fill it for each operation | 15:20 |
| dansmith | ahh, okay that makes sense | 15:20 |
| sean-k-mooney | lyarwood: well the mark host down api has a prequisite that the host is actully down and the agents are stoped | 15:25 |
| sean-k-mooney | lyarwood: if we voilate that expectation then perhaps it coudl cause issues | 15:26 |
| sean-k-mooney | lyarwood: in the evacuate case we proably should be storping n-cpu on the compute host as it is not properly fenced if it is running and able to connect to the message bus/db | 15:27 |
| lyarwood | sean-k-mooney: yup, if it's pinging neutron behind our backs I'm assuming that's going to cause issues when it comes to evacuting the instance from the node | 15:28 |
| sean-k-mooney | well i have not check how the info cache works but unless its per host then it could replace the binding info in the cache with the stale prot bindin info for the orginal host | 15:29 |
| sean-k-mooney | although im not sure if that would cause an issue | 15:29 |
| sean-k-mooney | its certenly safter to not have it messing with the info cache | 15:29 |
| sean-k-mooney | the warning is pretty clear in the api docs too on this point "tting a service forced down without completely fencing it will likely result in the corruption of VMs on that host." | 15:30 |
| *** vishalmanchanda has quit IRC | 15:31 | |
| *** udesale_ has quit IRC | 15:32 | |
| openstackgerrit | Balazs Gibizer proposed openstack/nova master: Add service version check for live migrate with qos https://review.opendev.org/713679 | 15:32 |
| openstackgerrit | Balazs Gibizer proposed openstack/nova master: Add service version check for evacuate with qos https://review.opendev.org/713680 | 15:32 |
| gibi | dansmith: fyi I retrofitted the service version check to the rest of the Ussuri qos move code ^^ | 15:33 |
| openstackgerrit | Lee Yarwood proposed openstack/nova master: WIP gate: Ensure n-cpu is stopped on the subnode during evacuation https://review.opendev.org/713674 | 15:36 |
| openstackgerrit | Balazs Gibizer proposed openstack/nova stable/stein: Fix intermittently failing regression case https://review.opendev.org/713682 | 15:37 |
| openstackgerrit | Lee Yarwood proposed openstack/nova master: WIP gate: Ensure n-cpu is stopped on the subnode during evacuation https://review.opendev.org/713674 | 15:37 |
| openstackgerrit | Balazs Gibizer proposed openstack/nova stable/stein: Fix intermittently failing regression case https://review.opendev.org/713682 | 15:37 |
| *** LiangFang has quit IRC | 15:41 | |
| *** jawad_axd has quit IRC | 15:45 | |
| openstackgerrit | Balazs Gibizer proposed openstack/nova master: [Community goal] Update contributor documentation https://review.opendev.org/712420 | 15:52 |
| dansmith | gibi: can you look at my concern here real quick? https://review.opendev.org/#/c/713679/1/nova/api/openstack/compute/migrate_server.py | 16:01 |
| *** mkrai has quit IRC | 16:02 | |
| gibi | dansmith: looking | 16:02 |
| openstackgerrit | Merged openstack/os-resource-classes master: Switch to hacking 2.x https://review.opendev.org/710276 | 16:03 |
| gibi | dansmith: make sense. I can reword the API error, and LOG the detailed info | 16:03 |
| dansmith | gibi: okay, thanks | 16:04 |
| dansmith | also, vmware ci did report on the un-deprecate patch... but it failed and only ran two tempest tests (both of which failed) | 16:04 |
| gibi | dansmith: I will respin th whole series and fix your nits on the bottom too. | 16:04 |
| dansmith | gibi: same api concern here: https://review.opendev.org/#/c/705475/6/nova/api/openstack/compute/shelve.py | 16:05 |
| dansmith | gibi: cool, I'll circle back quickly when you do | 16:05 |
| gibi | dansmith: yepp, and in the evac patch too | 16:05 |
| gibi | dansmith: regarding vmware. I think it is fair to say that we need those self selected tempest test to pass | 16:05 |
| dansmith | I think we need to see it working on a wide scale for a substantial period of time, | 16:06 |
| dansmith | because this pattern of making it work for two weeks until it breaks again doesn't really translate to quality | 16:06 |
| gibi | dansmith: yes, you have a point | 16:07 |
| gibi | dansmith: does it only triggers to vmware related patcheS? because then there will be not many changes that triggers them | 16:08 |
| dansmith | AFAICT, it's manually triggered it seems | 16:08 |
| gibi | that feels like cheating for a CI machinery | 16:09 |
| *** sapd1 has quit IRC | 16:09 | |
| *** lbragstad__ has quit IRC | 16:14 | |
| *** lbragstad has joined #openstack-nova | 16:23 | |
| *** slaweq has quit IRC | 16:28 | |
| *** mlavalle has quit IRC | 16:35 | |
| *** mlavalle has joined #openstack-nova | 16:35 | |
| openstackgerrit | Lee Yarwood proposed openstack/nova master: WIP gate: Ensure n-cpu is stopped on the subnode during evacuation https://review.opendev.org/713674 | 16:39 |
| *** mlavalle has quit IRC | 16:42 | |
| *** mlavalle has joined #openstack-nova | 16:42 | |
| *** mlavalle has quit IRC | 16:44 | |
| *** mlavalle has joined #openstack-nova | 16:45 | |
| *** johanssone has quit IRC | 16:51 | |
| *** mlavalle has quit IRC | 16:53 | |
| *** mlavalle has joined #openstack-nova | 16:53 | |
| *** jraju__ has quit IRC | 16:54 | |
| *** johanssone has joined #openstack-nova | 16:56 | |
| *** johanssone has quit IRC | 16:57 | |
| openstackgerrit | Ghanshyam Mann proposed openstack/nova master: Add new default roles in os-flavor-access policies https://review.opendev.org/713697 | 16:59 |
| *** tesseract has quit IRC | 16:59 | |
| *** johanssone has joined #openstack-nova | 17:03 | |
| *** johanssone has quit IRC | 17:03 | |
| openstackgerrit | Ghanshyam Mann proposed openstack/nova master: Add new default roles in os-flavor-access policies https://review.opendev.org/713697 | 17:05 |
| *** johanssone has joined #openstack-nova | 17:05 | |
| gmann | johnthetubaguy: list flavor-access policy is little wired. old defaults are admin-or-owner which i think should be admin as multiple projects can have access to single flavor so owner is not suitable things for flavor. ^^ updating it with more accurate new defaults. you can see details in bug https://bugs.launchpad.net/nova/+bug/1867840 | 17:05 |
| openstack | Launchpad bug 1867840 in OpenStack Compute (nova) "os-flavor-access API policy should be admin only" [Undecided,New] - Assigned to Ghanshyam Mann (ghanshyammann) | 17:05 |
| *** johanssone has quit IRC | 17:05 | |
| *** johanssone has joined #openstack-nova | 17:06 | |
| gmann | i did not fix the old default as bug because that can break existing deployment. so thinking to keep supporting them and cleanup during we move to system scope by default | 17:06 |
| *** johanssone has quit IRC | 17:08 | |
| *** dtantsur|brb is now known as dtantsur | 17:09 | |
| *** johanssone has joined #openstack-nova | 17:14 | |
| *** johanssone has quit IRC | 17:15 | |
| *** johanssone has joined #openstack-nova | 17:20 | |
| *** johanssone has quit IRC | 17:20 | |
| *** johanssone has joined #openstack-nova | 17:23 | |
| *** gyee has joined #openstack-nova | 17:27 | |
| *** johanssone has quit IRC | 17:27 | |
| *** johanssone has joined #openstack-nova | 17:30 | |
| *** johanssone has quit IRC | 17:32 | |
| *** evrardjp has quit IRC | 17:36 | |
| *** evrardjp has joined #openstack-nova | 17:36 | |
| *** priteau has quit IRC | 17:42 | |
| *** johanssone has joined #openstack-nova | 17:46 | |
| *** lbragstad has quit IRC | 17:47 | |
| *** lbragstad has joined #openstack-nova | 17:48 | |
| openstackgerrit | Merged openstack/nova master: db: Remove unused ec2 DB APIs https://review.opendev.org/698880 | 17:53 |
| *** johanssone has quit IRC | 17:54 | |
| *** derekh has quit IRC | 18:00 | |
| *** johanssone has joined #openstack-nova | 18:02 | |
| *** johanssone has quit IRC | 18:02 | |
| *** johanssone has joined #openstack-nova | 18:03 | |
| openstackgerrit | Merged openstack/nova master: libvirt: Check the guest support UEFI https://review.opendev.org/685678 | 18:05 |
| *** johanssone has quit IRC | 18:08 | |
| openstackgerrit | Balazs Gibizer proposed openstack/nova master: Support unshelve with qos ports https://review.opendev.org/704759 | 18:10 |
| openstackgerrit | Balazs Gibizer proposed openstack/nova master: Enable unshelve with qos ports https://review.opendev.org/705475 | 18:10 |
| *** dtantsur is now known as dtantsur|afk | 18:11 | |
| *** jawad_axd has joined #openstack-nova | 18:13 | |
| *** johanssone has joined #openstack-nova | 18:13 | |
| openstackgerrit | Balazs Gibizer proposed openstack/nova master: Add service version check for live migrate with qos https://review.opendev.org/713679 | 18:13 |
| openstackgerrit | Balazs Gibizer proposed openstack/nova master: Merge qos related renos for Ussuri https://review.opendev.org/706766 | 18:13 |
| openstackgerrit | Balazs Gibizer proposed openstack/nova master: Add service version check for evacuate with qos https://review.opendev.org/713680 | 18:15 |
| *** amoralej is now known as amoralej|off | 18:15 | |
| gibi | dansmith: updated the whole qos series ^^ | 18:15 |
| * gibi leaveas for today | 18:15 | |
| dansmith | gibi: sweet | 18:15 |
| gibi | dansmith: thanks for the review | 18:15 |
| dansmith | np, will hit those soon | 18:15 |
| *** johanssone has quit IRC | 18:19 | |
| sean-k-mooney | lyarwood: it looks like your evacuate change is hitting some where errors https://zuul.opendev.org/t/openstack/build/daabc75c8992406f8a089a808c2553b9/log/logs/screen-n-cpu.txt#7789 | 18:22 |
| *** johanssone has joined #openstack-nova | 18:22 | |
| sean-k-mooney | it disconnect an iscsi volume then reports rbd issues and finally fails with a virtual interface createion faild error | 18:23 |
| sean-k-mooney | although i dont know why that is being raided | 18:23 |
| *** johanssone has quit IRC | 18:25 | |
| *** johanssone has joined #openstack-nova | 18:34 | |
| *** johanssone has quit IRC | 18:37 | |
| *** johanssone has joined #openstack-nova | 18:43 | |
| *** nicolasbock has quit IRC | 18:47 | |
| *** johanssone has quit IRC | 18:47 | |
| *** johanssone has joined #openstack-nova | 18:50 | |
| *** nicolasbock has joined #openstack-nova | 18:54 | |
| *** jawad_axd has quit IRC | 18:54 | |
| *** maciejjozefczyk has quit IRC | 18:56 | |
| *** johanssone has quit IRC | 18:57 | |
| *** prometheanfire has joined #openstack-nova | 19:02 | |
| lyarwood | sean-k-mooney: which one was that sorry? | 19:03 |
| lyarwood | sean-k-mooney: oh the evacuation fix | 19:03 |
| lyarwood | sean-k-mooney: it looked like the same issue again in console.log | 19:03 |
| lyarwood | sean-k-mooney: I didn't even think to look at n-cpu sorry | 19:04 |
| * lyarwood checks | 19:04 | |
| *** johanssone has joined #openstack-nova | 19:04 | |
| *** johanssone has quit IRC | 19:04 | |
| sean-k-mooney | the n-cpu logs honestly confuse me | 19:05 |
| sean-k-mooney | i did not look at teh code but i think the virutal interface creation excpetion is a redheiring | 19:05 |
| *** johanssone has joined #openstack-nova | 19:06 | |
| *** johanssone has quit IRC | 19:06 | |
| lyarwood | sean-k-mooney: https://zuul.opendev.org/t/openstack/build/daabc75c8992406f8a089a808c2553b9/log/logs/screen-n-cpu.txt#7760 it's the same issue | 19:07 |
| lyarwood | sean-k-mooney: we hit the vif timeout first and call disconnect_volume | 19:08 |
| lyarwood | sean-k-mooney: I think the RBD cleanup failures are due to the domain still being active on the subnode | 19:08 |
| * lyarwood wonders why we are connecting iscsi volumes | 19:09 | |
| lyarwood | oh right because ceph.sh only sets up ephemeral storage on ceph | 19:09 |
| lyarwood | not c-vol | 19:09 |
| prometheanfire | looks like alembic is still failing for nova / neutron https://review.opendev.org/708706 | 19:10 |
| sean-k-mooney | ah ok | 19:10 |
| sean-k-mooney | the ceph thing was confusing me | 19:10 |
| sean-k-mooney | and ya that kind of makes sense | 19:10 |
| sean-k-mooney | so we not only need to stop the compute agent but the vms on the sub node too | 19:11 |
| sean-k-mooney | which i guess makes sense | 19:11 |
| *** johanssone has joined #openstack-nova | 19:11 | |
| sean-k-mooney | so we should stop the compute agent. then use libvirt to stop all the vms on the subnode | 19:12 |
| sean-k-mooney | then do an evacuate | 19:12 |
| *** johanssone has quit IRC | 19:14 | |
| lyarwood | sean-k-mooney: kk, I'll give that a go. | 19:16 |
| lyarwood | sean-k-mooney: here's a thought, shouldn't all devstack@* services be stopped on the subnode? | 19:16 |
| sean-k-mooney | i was consider doing a power off on the subnode | 19:17 |
| sean-k-mooney | but i think we need logs from it | 19:17 |
| sean-k-mooney | but yes you could do "sudo systemctl stop devstack@*" | 19:18 |
| sean-k-mooney | that would be totally valid to do provide we dont install cpeh there | 19:18 |
| sean-k-mooney | if we are installing ceph in the devstack slice then that woudl be an issue | 19:19 |
| *** noonedeadpunk has joined #openstack-nova | 19:21 | |
| *** johanssone has joined #openstack-nova | 19:21 | |
| noonedeadpunk | hi! is there any reason (like security or other) why flavor transition from public to private or vice versa is not supported? | 19:22 |
| sean-k-mooney | noonedeadpunk: private flavors was a half impleneted feature that is not commonly used | 19:22 |
| sean-k-mooney | noonedeadpunk: so support was not added. | 19:23 |
| noonedeadpunk | oh really? Like I was using it pretty much in several clouds | 19:23 |
| openstackgerrit | Lee Yarwood proposed openstack/nova master: WIP gate: Ensure n-cpu is stopped on the subnode during evacuation https://review.opendev.org/713674 | 19:23 |
| lyarwood | sean-k-mooney: ^ awful hacks, no idea if that would even work but there we go. This would actually be easier to do if this was a playbook... | 19:23 |
| noonedeadpunk | Ok, I see. Just wondering if it makes sense to look into it to propose a patch | 19:24 |
| noonedeadpunk | As there's no blockers which are aware for everyone, will try that out | 19:25 |
| sean-k-mooney | noonedeadpunk: its an api change which would require a microverion bump and a spec | 19:26 |
| sean-k-mooney | so it cant be done as a bug fix | 19:26 |
| sean-k-mooney | https://docs.openstack.org/api-ref/compute/?expanded=remove-flavor-access-from-tenant-removetenantaccess-action-detail#flavors-access-flavors-os-flavor-access | 19:26 |
| sean-k-mooney | the flavor acess api is what is normally used instead | 19:26 |
| noonedeadpunk | sean-k-mooney: the thing I'm missing is transition from is_public: true to is_public: false.... | 19:28 |
| sean-k-mooney | yep | 19:29 |
| noonedeadpunk | but yeah, it needs api version bump... | 19:29 |
| noonedeadpunk | anyway thanks for answering:) | 19:30 |
| sean-k-mooney | noonedeadpunk: most flavor fileds are imuntable after its created | 19:30 |
| sean-k-mooney | the update endpoint only supports updating the description | 19:31 |
| noonedeadpunk | I can understand about flavor properties, as it will mix up isntances | 19:31 |
| sean-k-mooney | https://docs.openstack.org/api-ref/compute/?expanded=update-flavor-description-detail#update-flavor-description | 19:31 |
| noonedeadpunk | but is_public is kinda visability thing, which should be safe to update in that terms | 19:31 |
| sean-k-mooney | in theory yes. | 19:32 |
| sean-k-mooney | although updating it wont change the status of the is_public flag in the embded flavor in an instnace | 19:32 |
| gmann | but it might cause restriction access to few tenants which are not in access list. | 19:32 |
| openstackgerrit | melanie witt proposed openstack/nova master: DNM: try to get some debug info for bug 1844929 https://review.opendev.org/701478 | 19:33 |
| openstack | bug 1844929 in OpenStack Compute (nova) "grenade jobs failing due to "Timed out waiting for response from cell" in scheduler" [High,Confirmed] https://launchpad.net/bugs/1844929 | 19:33 |
| sean-k-mooney | gmann: well this is not using the os-flavor-acces api | 19:34 |
| sean-k-mooney | this is setting the is_public flag in the flavor itself | 19:34 |
| sean-k-mooney | the os-flavor-acces api is normally what peopel shoudl use | 19:34 |
| sean-k-mooney | so to make a flavor private you just add acess to only the admin/service project | 19:34 |
| sean-k-mooney | then no other tenats can see it | 19:35 |
| gmann | yeah but we restrict the public flavor to process in flavor-access api | 19:35 |
| gmann | yeah, removing all other tenants make it private | 19:35 |
| sean-k-mooney | noonedeadpunk: does ^ makes sense | 19:36 |
| sean-k-mooney | noonedeadpunk: is_public on the flavor is a hold over from before we had per tenant contol over flaovr visablity | 19:37 |
| noonedeadpunk | hm, let me check that | 19:37 |
| noonedeadpunk | (actully deploy latest sandbox and test) | 19:38 |
| sean-k-mooney | lyarwood: looking at https://github.com/openstack/devstack-plugin-ceph/blob/master/devstack/lib/ceph#L1213-L1251 it looks like ceph is not in the devstack slice | 19:38 |
| sean-k-mooney | so stop devstack@* should be ok | 19:38 |
| *** ociuhandu has joined #openstack-nova | 19:39 | |
| sean-k-mooney | you would still have to do "killall qemu" or use virish stop on all active domains too to stop the rbd device in use error | 19:39 |
| lyarwood | sean-k-mooney: right and I don't think we deploy anything on the subnode in this case anyway | 19:39 |
| gmann | only thing is you would not be able to add projects or list access info on public flavor | 19:39 |
| sean-k-mooney | gmann: you can use the flavor acess api to do that | 19:40 |
| sean-k-mooney | horizon allows you to take a standard public flavor and add tenants too it | 19:40 |
| *** rpittau is now known as rpittau|afk | 19:40 | |
| * lyarwood calls it a night \o | 19:41 | |
| gmann | sean-k-mooney: no, after 2.7 add project is restricted on non-public and list access is for non-public only from stating | 19:41 |
| sean-k-mooney | gmann: i tought the os-flaovr-acess api basically ignored the is_public atribte | 19:41 |
| gmann | https://github.com/openstack/nova/blob/c9f5b583b6072f542d1757e35fd6305b9698496a/nova/api/openstack/compute/flavor_access.py#L50-L75 | 19:41 |
| sean-k-mooney | gmann: right so if you use an older microversion then what happens | 19:41 |
| gmann | with older than 2.7 yes you can add. | 19:42 |
| sean-k-mooney | i suspect horizon is using older then 2.7 | 19:42 |
| sean-k-mooney | why did we make that change? | 19:42 |
| gmann | well, public means for everyone so we do not really need to add access things right | 19:43 |
| sean-k-mooney | gmann: basicaly i tought we effectivly deprecated is_public and ignored it | 19:44 |
| sean-k-mooney | gmann: so if you use falvor acess it effectivly became private | 19:44 |
| sean-k-mooney | excpet to the tenants listed | 19:44 |
| sean-k-mooney | gmann: if that is not the case then i agree that this should be mutable in the flavor api | 19:46 |
| sean-k-mooney | you cant really gracefully retire a flavor othersize. | 19:46 |
| sean-k-mooney | it looks like it was change as a result of this bug https://bugs.launchpad.net/nova/+bug/1361476 | 19:47 |
| openstack | Launchpad bug 1361476 in OpenStack Compute (nova) "flavor access create should check public/private first" [Low,Fix released] - Assigned to Sergey Nikitin (snikitin) | 19:47 |
| gmann | humm, but will change in is_public on embedded instances effect like make that flavor stale. the reason we do not allow flavor modification on other parameters. | 19:47 |
| sean-k-mooney | my expectation is that if a flaovr is public and i add a tenatn via os-flavor-access then any tenant other then those listed via flavor access would not see it | 19:48 |
| sean-k-mooney | gmann: is public on the embeded instance has no meaning | 19:48 |
| gmann | and keep is_public same ? | 19:49 |
| sean-k-mooney | so i dont think its an issue | 19:49 |
| gmann | listing flavor? | 19:49 |
| sean-k-mooney | yes | 19:49 |
| sean-k-mooney | so my expectation is is_public would still be public/true | 19:49 |
| sean-k-mooney | and flavor list would not list | 19:49 |
| sean-k-mooney | it | 19:49 |
| sean-k-mooney | unless you were in the tenant on the access list or an admin | 19:49 |
| sean-k-mooney | so basicaly i think https://bugs.launchpad.net/nova/+bug/1361476 was invalid | 19:50 |
| openstack | Launchpad bug 1361476 in OpenStack Compute (nova) "flavor access create should check public/private first" [Low,Fix released] - Assigned to Sergey Nikitin (snikitin) | 19:50 |
| gmann | i mean we have is_public in list flavor filters if anyone listing by their instance's flavor is_public flag | 19:51 |
| sean-k-mooney | well actully maybe not | 19:51 |
| sean-k-mooney | gmann: can you say that again | 19:51 |
| sean-k-mooney | looking at the bug i would expect "nova flavor-access-list --flavor 1" to work but i would not expect the flavor to be in "openstack flavor list" unless you were in a tenant with acess or an admin | 19:52 |
| gmann | i mean will is_public change the flavor signature or not? even that does not make any change in real configueation but still a attribute in flavor user facing dict | 19:52 |
| sean-k-mooney | is_public only changes if you can see the flaovr or not | 19:53 |
| sean-k-mooney | so i think that is just metadata about the flaovr | 19:53 |
| sean-k-mooney | not a part of it | 19:53 |
| gmann | yeah. kind of. | 19:54 |
| sean-k-mooney | kind of like the description wich we allow to be udpated | 19:54 |
| gmann | or we just remove this flag. | 19:54 |
| sean-k-mooney | i think the only benift to it is the defualt polciy | 19:54 |
| sean-k-mooney | e.g. should it be visable by default or not | 19:55 |
| sean-k-mooney | but i certenly dont think we should be blocking the use of the flaovr acess api based on it | 19:55 |
| sean-k-mooney | i would be ok with removing it too but im not sure how others would feel | 19:56 |
| sean-k-mooney | e.g. do operators use it frequently. if soo i think it should be kept and mutable. if not remove and just use flavor acess api for this | 19:56 |
| gmann | ok, even list public flavor on list access is not wrong which is why that bug did the change in add access | 19:56 |
| *** ociuhandu has quit IRC | 19:57 | |
| gmann | sorry, need to go for lunch, ttyl | 19:57 |
| sean-k-mooney | no worries | 19:57 |
| *** gmann is now known as gmann_afk | 19:57 | |
| sean-k-mooney | im going for food too | 19:57 |
| *** ociuhandu has joined #openstack-nova | 19:58 | |
| noonedeadpunk | it was pretty interesting discussion. Like I thought it is a bit more simple than it is | 19:59 |
| sean-k-mooney | well the code is simple to change but it has several other implications obvirously | 19:59 |
| sean-k-mooney | noonedeadpunk: this is why we generally require a spec for api changes as this type of discussion normally happens when we dig into it | 20:00 |
| noonedeadpunk | yeah and it is fair | 20:00 |
| *** ralonsoh has quit IRC | 20:01 | |
| noonedeadpunk | will try to write down it as it seems that some patching is required anyway | 20:01 |
| *** ociuhandu has quit IRC | 20:02 | |
| *** ociuhandu has joined #openstack-nova | 20:02 | |
| *** nweinber has quit IRC | 20:20 | |
| *** gmann_afk is now known as gmann | 20:33 | |
| gmann | yeah, spec can be better idea to discuss if something we miss. sean-k-mooney proposal looks ok to me for now. | 20:36 |
| *** ociuhandu has quit IRC | 20:37 | |
| *** ociuhandu has joined #openstack-nova | 20:38 | |
| *** dklyle has quit IRC | 20:39 | |
| *** ociuhandu has quit IRC | 20:39 | |
| *** ociuhandu has joined #openstack-nova | 20:40 | |
| *** damien_r has quit IRC | 20:40 | |
| *** nicolasbock has quit IRC | 20:40 | |
| *** dklyle has joined #openstack-nova | 20:40 | |
| *** damien_r has joined #openstack-nova | 20:44 | |
| openstackgerrit | Merged openstack/nova master: Lowercase ironic driver hash ring and ignore case in cache https://review.opendev.org/711680 | 20:50 |
| *** ociuhandu has quit IRC | 20:56 | |
| *** ociuhandu has joined #openstack-nova | 20:57 | |
| openstackgerrit | melanie witt proposed openstack/nova stable/train: Lowercase ironic driver hash ring and ignore case in cache https://review.opendev.org/713739 | 21:00 |
| *** ociuhandu has quit IRC | 21:02 | |
| *** slaweq has joined #openstack-nova | 21:15 | |
| *** openstackstatus has joined #openstack-nova | 21:16 | |
| *** ChanServ sets mode: +v openstackstatus | 21:16 | |
| *** xek has quit IRC | 21:17 | |
| *** ociuhandu has joined #openstack-nova | 21:18 | |
| *** ociuhandu has quit IRC | 21:27 | |
| openstackgerrit | Lee Yarwood proposed openstack/nova master: WIP gate: Ensure n-cpu is stopped on the subnode during evacuation https://review.opendev.org/713674 | 21:36 |
| *** ociuhandu has joined #openstack-nova | 21:41 | |
| *** ociuhandu has quit IRC | 21:46 | |
| *** slaweq has quit IRC | 22:00 | |
| *** david-lyle has joined #openstack-nova | 22:01 | |
| *** ociuhandu has joined #openstack-nova | 22:01 | |
| *** dklyle has quit IRC | 22:03 | |
| *** ociuhandu has quit IRC | 22:06 | |
| *** david-lyle has quit IRC | 22:08 | |
| *** nicolasbock has joined #openstack-nova | 22:22 | |
| *** nicolasbock has quit IRC | 22:27 | |
| *** nicolasbock has joined #openstack-nova | 22:28 | |
| *** rcernin has joined #openstack-nova | 22:44 | |
| *** tkajinam has joined #openstack-nova | 22:50 | |
| *** ivve has quit IRC | 22:50 | |
| *** macz_ has quit IRC | 22:56 | |
| *** threestrands has joined #openstack-nova | 22:58 | |
| *** sean-k-mooney1 has joined #openstack-nova | 23:02 | |
| *** sean-k-mooney has quit IRC | 23:04 | |
| *** spatel has joined #openstack-nova | 23:27 | |
| *** alex_xu has joined #openstack-nova | 23:34 | |
| alex_xu | dansmith: artom, thanks for the review https://review.opendev.org/#/c/687856/15/nova/compute/manager.py@8346, I reply that, but yes, we tried differnt options, and back and forward many times ourselve, looking for suggestion :) | 23:36 |
| *** spatel has quit IRC | 23:51 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!