| *** tetsuro has joined #openstack-placement | 00:40 | |
| openstackgerrit | 98k proposed openstack/os-traits master: Add python 3.6 unit test job https://review.openstack.org/616749 | 03:18 |
|---|---|---|
| *** openstackstatus has quit IRC | 04:59 | |
| *** openstack has joined #openstack-placement | 07:11 | |
| *** ChanServ sets mode: +o openstack | 07:11 | |
| *** e0ne has quit IRC | 07:31 | |
| openstackgerrit | Brin Zhang proposed openstack/nova-specs master: Support admin to specify project to create snapshot https://review.openstack.org/616843 | 07:52 |
| *** takashin has left #openstack-placement | 08:03 | |
| *** tssurya has joined #openstack-placement | 08:07 | |
| *** helenaAM has joined #openstack-placement | 08:31 | |
| *** tetsuro has quit IRC | 09:20 | |
| *** e0ne has joined #openstack-placement | 09:39 | |
| *** ttsiouts has joined #openstack-placement | 10:05 | |
| *** ttsiouts has quit IRC | 10:10 | |
| *** ttsiouts has joined #openstack-placement | 10:11 | |
| *** ttsiouts has quit IRC | 10:15 | |
| *** ttsiouts has joined #openstack-placement | 10:20 | |
| *** ttsiouts has quit IRC | 10:21 | |
| *** ttsiouts has joined #openstack-placement | 10:22 | |
| *** ttsiouts has quit IRC | 10:26 | |
| *** ttsiouts has joined #openstack-placement | 10:30 | |
| *** e0ne has quit IRC | 10:56 | |
| *** ttsiouts has quit IRC | 10:59 | |
| *** ttsiouts has joined #openstack-placement | 11:00 | |
| *** tssurya has quit IRC | 11:04 | |
| *** e0ne has joined #openstack-placement | 11:44 | |
| *** rubasov has quit IRC | 13:07 | |
| *** mriedem has joined #openstack-placement | 14:08 | |
| *** tssurya has joined #openstack-placement | 14:57 | |
| openstackgerrit | Merged openstack/placement master: Harden placement init under wsgi https://review.openstack.org/612444 | 14:57 |
| *** tssurya has quit IRC | 16:22 | |
| *** e0ne has quit IRC | 16:35 | |
| *** helenaAM has quit IRC | 16:59 | |
| *** ttsiouts has quit IRC | 17:51 | |
| *** e0ne has joined #openstack-placement | 18:09 | |
| *** e0ne has quit IRC | 18:33 | |
| *** e0ne has joined #openstack-placement | 18:36 | |
| mriedem | so i'm wondering if delete_consumers_if_no_allocations is broken | 18:40 |
| mriedem | big ass triage in https://bugs.launchpad.net/nova/+bug/1798688 | 18:40 |
| openstack | Launchpad bug 1798688 in OpenStack Compute (nova) "AllocationUpdateFailed_Remote: Failed to update allocations for consumer. Error: another process changed the consumer after the report client read the consumer state during the claim" [High,Triaged] | 18:40 |
| mriedem | tl;dr tempest has a test which shelve offloads a server and then immediately unshelves it, | 18:40 |
| mriedem | and we're hitting consumer generation conflicts during the unshelve | 18:40 |
| mriedem | presumably because the consumer/allocations aren't gone by the time we unshelve | 18:40 |
| mriedem | even though shelve offload deletes the allocations for the server | 18:41 |
| mriedem | and by "delete" I mean PUT /allocations/consumer with an empty allocations dict | 18:41 |
| mriedem | https://review.openstack.org/#/c/591597/8/nova/scheduler/client/report.py | 18:41 |
| *** mriedem is now known as mriedem_hangry | 18:55 | |
| *** e0ne has quit IRC | 18:58 | |
| *** mriedem_hangry is now known as mriedem | 19:34 | |
| *** e0ne has joined #openstack-placement | 20:01 | |
| *** e0ne has quit IRC | 20:04 | |
| *** efried is now known as fried_rice | 20:09 | |
| fried_rice | mriedem: Shouldn't we have a retry covering that race? | 20:09 |
| mriedem | fried_rice: we used to, before the consumer generatoin stuff | 20:19 |
| fried_rice | consumer generation stuff should allow us to do a *better* retry :( | 20:20 |
| mriedem | https://review.openstack.org/#/c/583667/29/nova/scheduler/client/report.py | 20:20 |
| fried_rice | Bogus, who would approve THAT patch?? | 20:20 |
| mriedem | i have a debug patch up to try and figure out what the existing allocations GET response looks like | 20:20 |
| mriedem | hard to know whats going on without that | 20:20 |
| mriedem | https://review.openstack.org/617016 | 20:20 |
| mriedem | also, i think tihs is always true https://review.openstack.org/#/c/583667/29/nova/scheduler/client/report.py@1805 | 20:22 |
| mriedem | i don't see a consumer generation in the allocation candidates response | 20:22 |
| mriedem | probably b/c those are not suppose to be specific to a consumer | 20:22 |
| mriedem | how/where does the root_provider_uuid get set on a provider? | 20:30 |
| mriedem | i'm looking at https://review.openstack.org/#/c/613304/ | 20:30 |
| mriedem | and trying to figure out how that could happen - old computes? | 20:31 |
| mriedem | oh so if i create a resource provider but omit parent_provider_uuid, then it's automatically the root, right? | 20:32 |
| mriedem | b/c it doesn't have a parent | 20:32 |
| fried_rice | should be, yeah | 20:32 |
| fried_rice | I don't think you're allowed to set the root provider UUID on a provider. It happens automatically. | 20:33 |
| fried_rice | If you don't set the parent ID, the node is a root node, period. | 20:33 |
| mriedem | right so https://bugs.launchpad.net/nova/+bug/1799892 | 20:34 |
| openstack | Launchpad bug 1799892 in OpenStack Compute (nova) "Placement API crashes with 500s in Rocky upgrade with downed compute nodes" [Undecided,New] | 20:34 |
| fried_rice | and yes, the test case in that patch is theoretically old computes, or manual diddling. | 20:34 |
| mriedem | any providers created before queens https://review.openstack.org/#/c/377138/ wouldn't have parent_provider_uuid or root_provider_id set | 20:34 |
| mriedem | and we apparently didn't have an online data migration to set the root_provider_uuid for existing providers... | 20:35 |
| mriedem | and at some point along the way to supported nested resource providers in the scheduler, allocation candidates assumes all providers have a root_provider_uuid set | 20:35 |
| fried_rice | we *do* have a data migration | 20:36 |
| fried_rice | We even set the root provider ID on reads if we find it unset | 20:36 |
| fried_rice | But the hole here may have been that we're looking at this particular provider in a place that doesn't have that migration thing going. | 20:37 |
| mriedem | oh i see https://review.openstack.org/#/c/377138/62/nova/objects/resource_provider.py@917 | 20:39 |
| mriedem | yeah so when we get the provider, we migrate the data | 20:39 |
| mriedem | but maybe when getting allcoation candidates, we're going straight through the db rather than the objects | 20:39 |
| fried_rice | just so | 20:39 |
| mriedem | this is where placement-manager db online_data_migrations is going to be useful | 20:41 |
| mriedem | stuff like this i mean | 20:42 |
| mriedem | fried_rice: btw, are you berlin bound or didn't get approval? | 20:42 |
| fried_rice | I got approval a week ago. Also saw an ENT this morning who told me my eyeball won't explode out of my head on ascent. I leave Sunday, arrive Monday, leave Friday. | 20:43 |
| mriedem | having an eyeball will be good | 20:44 |
| mriedem | the bruise will also be great if you're giving a talk | 20:44 |
| fried_rice | I'm not giving a talk, but the bruise is still nice. | 20:44 |
| mriedem | did Kevin hook you up with that placement / public clouds one? | 20:44 |
| mriedem | oh | 20:44 |
| mriedem | maybe he was shy | 20:44 |
| fried_rice | It's actually disappointingly non-spectacular. I don't bruise very easily. | 20:44 |
| mriedem | so full of iron youare | 20:45 |
| mriedem | like a human bean | 20:45 |
| fried_rice | But between that and my hair being longer, I look even more like Ash Williams :P | 20:45 |
| mriedem | i'm also growing my hair out | 20:45 |
| mriedem | gets real big and wavy | 20:45 |
| fried_rice | Tired of people comparing you to Doogie Howser? | 20:45 |
| mriedem | like eric from that 70s show | 20:45 |
| mriedem | i don't look anything like doogie | 20:45 |
| mriedem | kevin bacon on the other hand.. | 20:45 |
| fried_rice | it's more of a general mien | 20:46 |
| openstackgerrit | Matt Riedemann proposed openstack/placement master: Add recreate test for bug 1799892 https://review.openstack.org/617031 | 20:52 |
| openstack | bug 1799892 in OpenStack Compute (nova) rocky "Placement API crashes with 500s in Rocky upgrade with downed compute nodes" [Medium,Triaged] https://launchpad.net/bugs/1799892 | 20:52 |
| mriedem | fried_rice: jaypipes: i've got a question in https://review.openstack.org/#/c/613305/ which maybe you can answer | 21:02 |
| * fried_rice reads | 21:02 | |
| fried_rice | mriedem: Which question specifically? | 21:03 |
| mriedem | https://review.openstack.org/#/c/613305/2/nova/api/openstack/placement/objects/resource_provider.py@3002 | 21:04 |
| fried_rice | oy, that's not going to be a quick answer, with my sandbox in its current state | 21:05 |
| fried_rice | something tickling my brain says I looked at that when I first reviewed this patch... | 21:06 |
| fried_rice | mebbe not | 21:07 |
| fried_rice | I was in a chatty mood at that time, I would have said something. | 21:07 |
| fried_rice | mriedem: Okay, I get it. Have you looked at the other usage of that method via get_trees_matching_all ? | 21:10 |
| mriedem | i didn't dig into all that uses it no | 21:13 |
| mriedem | most of this giant resource_provider.py module is greek to me by now | 21:13 |
| fried_rice | mriedem: Replied. | 21:13 |
| fried_rice | mriedem: I suspect we can't get there because we only hit that "trees" path if we find proper tree-ness in the initial stages of GET /a_c. But I suppose it may be possible. | 21:15 |
| mriedem | idk | 21:16 |
| mriedem | without a test for that scenario it's hard to know what could break | 21:16 |
| fried_rice | right. If anyone can swing that test, tetsuro can. | 21:16 |
| fried_rice | I think it was here, I found a hole in the test coverage that he closed up. | 21:17 |
| *** mriedem has quit IRC | 22:29 | |
| *** s10 has joined #openstack-placement | 23:28 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!