Wednesday, 2019-08-21

« Tuesday, 2019-08-20 Index Thursday, 2019-08-22 »
*** markvoelker has joined #openstack-nova00:11
*** markvoelker has quit IRC00:15
*** bnemec has joined #openstack-nova00:16
*** markvoelker has joined #openstack-nova00:26
*** bnemec has quit IRC00:35
*** gyee has quit IRC00:41
*** bbowen__ has quit IRC00:48
*** bbowen__ has joined #openstack-nova00:48
*** spsurya has joined #openstack-nova00:59
*** hongbin has joined #openstack-nova01:04
*** ricolin has joined #openstack-nova01:05
openstackgerritsean mooney proposed openstack/nova stable/pike: lxc: make use of filter python3 compatible  https://review.opendev.org/67650201:15
*** dtruong has quit IRC01:24
*** dtruong has joined #openstack-nova01:25
*** igordc has quit IRC01:35
*** boxiang has joined #openstack-nova01:41
openstackgerritTakashi NATSUME proposed openstack/nova master: Fix missing rule description in HACKING.rst  https://review.opendev.org/67758901:43
*** liuyulong has quit IRC01:51
*** igordc has joined #openstack-nova01:58
*** sapd1_x has joined #openstack-nova02:07
openstackgerritMerged openstack/nova master: [Trivial]Remove unused helper filter_and_format_resource_metadata  https://review.opendev.org/67603802:21
*** sapd1_x has quit IRC02:26
*** sapd1_x has joined #openstack-nova02:28
*** igordc has quit IRC02:49
*** mkrai_ has joined #openstack-nova02:52
openstackgerritTakashi NATSUME proposed openstack/nova master: Add a hacking rule for non-existent assertions  https://review.opendev.org/67759603:00
*** BjoernT has joined #openstack-nova03:12
*** sapd1_x has quit IRC03:15
*** BjoernT has quit IRC03:17
*** BjoernT has joined #openstack-nova03:17
openstackgerritBoxiang Zhu proposed openstack/nova master: Fix live migration break group policy simultaneously  https://review.opendev.org/65196903:26
*** igordc has joined #openstack-nova03:33
*** psachin has joined #openstack-nova03:33
*** janki has joined #openstack-nova03:41
*** mkrai_ has quit IRC04:00
*** hongbin has quit IRC04:06
*** mkrai has joined #openstack-nova04:30
*** mkrai has quit IRC04:30
*** jaosorior has quit IRC04:34
*** mkrai has joined #openstack-nova04:34
*** tetsuro has joined #openstack-nova04:42
openstackgerritmelanie witt proposed openstack/nova master: Verify archive_deleted_rows --all-cells in post test hook  https://review.opendev.org/67284004:44
openstackgerritmelanie witt proposed openstack/nova master: Verify archive_deleted_rows --all-cells in post test hook  https://review.opendev.org/67284004:44
*** ratailor has joined #openstack-nova05:02
*** BjoernT has quit IRC05:06
*** udesale has joined #openstack-nova05:10
*** tetsuro has quit IRC05:12
*** BjoernT has joined #openstack-nova05:12
*** udesale has quit IRC05:14
*** tetsuro has joined #openstack-nova05:18
openstackgerritTakashi NATSUME proposed openstack/nova master: Add a hacking rule for non-existent assertions  https://review.opendev.org/67759605:23
*** BjoernT has quit IRC05:29
*** sapd1_x has joined #openstack-nova05:33
*** tetsuro has quit IRC05:33
*** ash2307 has left #openstack-nova05:42
*** sridharg has joined #openstack-nova05:42
openstackgerritMerged openstack/nova master: doc: Trivial fixes to API version history  https://review.opendev.org/64956105:46
openstackgerritTakashi NATSUME proposed openstack/nova master: Add a hacking rule for useless assertions  https://review.opendev.org/67761005:46
*** jaosorior has joined #openstack-nova05:49
openstackgerritMerged openstack/nova master: Add blocker migration for completing services.uuid migration  https://review.opendev.org/67362005:50
*** sapd1_x has quit IRC06:04
*** dpawlik has joined #openstack-nova06:20
*** jaosorior has quit IRC06:21
*** igordc has quit IRC06:30
*** tetsuro has joined #openstack-nova06:47
*** tetsuro has quit IRC06:48
*** tetsuro has joined #openstack-nova06:48
*** tetsuro has quit IRC06:50
*** tetsuro has joined #openstack-nova06:50
*** sridharg has quit IRC06:52
*** aojea has joined #openstack-nova07:01
*** brinzhang_ has joined #openstack-nova07:01
alex_xuefried: sean-k-mooney dansmith, here is the refined version about using db persistent https://etherpad.openstack.org/p/vpmems-non-virt-driver-specific-new. I also sent email to summary the pros/cons of two ways http://lists.openstack.org/pipermail/openstack-discuss/2019-August/008625.html07:02
*** jaosorior has joined #openstack-nova07:04
*** tetsuro has quit IRC07:04
*** brinzhang has quit IRC07:05
*** sridharg has joined #openstack-nova07:05
*** brinzhang has joined #openstack-nova07:05
*** brinzhang_ has quit IRC07:06
*** trident has quit IRC07:10
*** maciejjozefczyk has joined #openstack-nova07:12
*** sridharg has quit IRC07:12
*** rcernin has quit IRC07:14
*** boxiang has quit IRC07:14
*** boxiang has joined #openstack-nova07:14
*** trident has joined #openstack-nova07:17
*** udesale has joined #openstack-nova07:24
*** sapd1_x has joined #openstack-nova07:25
*** xek has joined #openstack-nova07:28
*** ivve has joined #openstack-nova07:39
*** rpittau|afk is now known as rpittau07:40
*** bhagyashris has joined #openstack-nova07:50
*** ralonsoh has joined #openstack-nova07:52
*** zbr is now known as zbr|ooo07:56
*** takashin has left #openstack-nova08:00
*** jaosorior has quit IRC08:00
openstackgerritBrin Zhang proposed openstack/nova master: Specify availability_zone to unshelve  https://review.opendev.org/66385108:07
*** boxiang has quit IRC08:17
*** boxiang has joined #openstack-nova08:17
*** mkrai has quit IRC08:25
*** mkrai has joined #openstack-nova08:25
*** boxiang has quit IRC08:28
*** boxiang has joined #openstack-nova08:28
*** tkajinam has quit IRC08:29
*** shilpasd has joined #openstack-nova08:33
*** ociuhandu has joined #openstack-nova08:35
*** tetsuro has joined #openstack-nova08:46
*** elod_off is now known as elod08:47
*** tetsuro has quit IRC08:51
*** mkrai has quit IRC08:53
openstackgerritGhanshyam Mann proposed openstack/python-novaclient master: Microversion 2.77: Support Specifying AZ to unshelve  https://review.opendev.org/66513609:00
*** Alon_KS has joined #openstack-nova09:18
*** jaosorior has joined #openstack-nova09:19
*** jawad_axd has joined #openstack-nova09:20
*** Conqueror has joined #openstack-nova09:22
*** derekh has joined #openstack-nova09:26
*** mkrai has joined #openstack-nova09:28
*** shilpasd has quit IRC09:32
*** sapd1_x has quit IRC09:53
openstackgerritMerged openstack/nova master: [Trivial]Remove unused helper _get_instances_by_filters  https://review.opendev.org/67603609:56
*** klindgren_ has quit IRC09:59
*** klindgren has joined #openstack-nova09:59
*** rpittau is now known as rpittau|bbl10:14
*** shilpasd has joined #openstack-nova10:28
*** bhagyashris has quit IRC10:31
aspiersalex_xu: I will have another PS for you very soon10:34
aspiersalex_xu: I'm adding the missing tests10:34
*** jchhatbar has joined #openstack-nova10:47
*** janki has quit IRC10:49
*** shilpasd has quit IRC10:49
*** ociuhandu has quit IRC10:49
*** hemna has quit IRC10:54
*** owalsh is now known as owalsh|away11:08
openstackgerritMerged openstack/nova master: Libvirt: add support for vPMU configuration.  https://review.opendev.org/67133811:15
*** ociuhandu has joined #openstack-nova11:20
*** ociuhandu has quit IRC11:25
*** udesale has quit IRC11:26
*** nnsingh has joined #openstack-nova11:27
*** udesale has joined #openstack-nova11:27
*** nnsingh has left #openstack-nova11:27
*** yikun has joined #openstack-nova11:28
*** tbachman has quit IRC11:42
*** shilpasd has joined #openstack-nova11:42
alex_xuaspiers: thanks a lot11:51
*** mdbooth has joined #openstack-nova11:52
aspiersalex_xu: just finished now, submitting ...11:52
openstackgerritAdam Spiers proposed openstack/nova master: Split fake host capabilities into reusable variables  https://review.opendev.org/67771011:52
openstackgerritAdam Spiers proposed openstack/nova master: Track libvirt host/domain capabilities for multiple machine types  https://review.opendev.org/67315111:53
mdboothHeads up: the libvirt tpool proxy doesn't work reliably: https://bugs.launchpad.net/nova/+bug/1840912 Potentially the cause of a bunch of interesting behaviours11:53
openstackLaunchpad bug 1840912 in OpenStack Compute (nova) "libvirt calls aren't reliably using tpool.Proxy" [Undecided,New]11:53
openstackgerritBrin Zhang proposed openstack/nova master: Specify availability_zone to unshelve  https://review.opendev.org/66385111:53
alex_xuaspiers: I will try another around tomorrow11:55
aspiersalex_xu: thanks. I just updated the test comment and added some simple tests to test_host.py11:56
aspiersalex_xu: so it should be easy to review11:56
alex_xuaspiers: cool11:58
*** udesale has quit IRC12:02
*** rpittau|bbl is now known as rpittau12:10
openstackgerritStephen Finucane proposed openstack/nova master: docs: Rewrite host aggregate, availability zone docs  https://review.opendev.org/66713312:27
stephenfinalex_xu: Think you could send this on its way? https://review.opendev.org/#/c/675570/12:28
*** udesale has joined #openstack-nova12:32
*** tbachman has joined #openstack-nova12:33
openstackgerritAdam Spiers proposed openstack/nova master: Split fake host capabilities into reusable variables  https://review.opendev.org/67771012:36
openstackgerritAdam Spiers proposed openstack/nova master: Track libvirt host/domain capabilities for multiple machine types  https://review.opendev.org/67315112:36
openstackgerritAdam Spiers proposed openstack/nova master: Extract new base class for provider usage functional tests  https://review.opendev.org/67696412:37
openstackgerritAdam Spiers proposed openstack/nova master: Provide HW_CPU_X86_AMD_SEV trait when SEV is supported  https://review.opendev.org/63868012:37
openstackgerritAdam Spiers proposed openstack/nova master: Add extra spec parameter and image property for memory encryption  https://review.opendev.org/66442012:37
openstackgerritAdam Spiers proposed openstack/nova master: Extract SEV-specific bits on host detection  https://review.opendev.org/63633412:37
openstackgerritAdam Spiers proposed openstack/nova master: Add <launchSecurity> and <driver iommu='on' /> to config.py  https://review.opendev.org/63631812:37
openstackgerritAdam Spiers proposed openstack/nova master: Apply SEV-specific guest config when SEV is required  https://review.opendev.org/64456512:38
openstackgerritAdam Spiers proposed openstack/nova master: Enable booting of libvirt guests with AMD SEV memory encryption  https://review.opendev.org/66661612:38
*** jchhatba_ has joined #openstack-nova12:39
shilpasdstephenfin: Hi, small doubt, at https://github.com/openstack/nova/blob/master/nova/api/openstack/compute/views/versions.py#L4312:40
shilpasdfor multiple choices reply, links getting printed12:41
shilpasdand if we given wrong request path, that getting added to response12:41
stephenfinshilpasd: Could you give me an example?12:41
stephenfinExpected vs. actual12:41
shilpasdactual:  "links": [{"href": "http://10.232.48.206/compute/v2.1/compute/v4.1/servers/detail", "rel": "self"}]12:42
*** jchhatbar has quit IRC12:42
shilpasdexpected:  "links": [{"href": "http://10.232.48.206/compute/v2/servers/detail", "rel": "self"}]12:43
*** jchhatba_ has quit IRC12:43
sean-k-mooneyits reading v4.112:43
sean-k-mooneyas the server uuid12:43
sean-k-mooneyactully no its not12:43
shilpasdi have purposfully given v4.1 in req path12:44
sean-k-mooneyya i guessed that12:44
sean-k-mooneycompute/v4.1/servers/detail was what you used12:45
shilpasdcurl -g -i -X GET http://<host_ip>/compute/v4.1/servers/detail -H "Accept: application/json" -H "OpenStack-API-Version: compute 2.74 <AUTH_TOKEN>" -H "X-OpenStack-Nova-API-Version: 2.74"12:45
shilpasdyes12:45
shilpasdsean-k-mooney: yes12:45
aspiersCan anyone comment on whether six.iteritems() should be avoided in nova? Currently our hacking extensions allow it despite http://lists.openstack.org/pipermail/openstack-dev/2015-June/066391.html12:46
sean-k-mooneywell we proably should be validaing the links if nothing else12:46
shilpasdi referred https://docs.openstack.org/api-guide/compute/versions.html12:46
stephenfinI'm trying to figure out where we parse the URL, but it sounds like a legit bug12:46
stephenfinI assume you meant to request 'http://<host_ip>/compute/v2.1/servers/detail' ?12:46
*** ociuhandu has joined #openstack-nova12:46
*** ociuhandu has quit IRC12:47
stephenfinI'd ask cdent but he's not around12:47
shilpasdstephenfin: yes12:47
*** ociuhandu has joined #openstack-nova12:47
shilpasdbut purposefully requested 'http://<host_ip>/compute/v4.1/servers/detail'12:47
sean-k-mooneyaspiers: am i dont htink the time it examples in that post are good tests12:48
aspierssean-k-mooney: read the whole thread12:48
aspiersalso currently there are only two uses of six.iteritems in the whole of nova12:48
sean-k-mooneywe have removed som uses of it recently12:49
sean-k-mooneypersonally given we intend to kill python 3 support in 2 months i could live with items at this point12:49
stephenfinaspiers: Personally, I'd just avoid it unless you know you need it12:50
aspiersstephenfin: yes, I've just removed it from my patches12:50
sean-k-mooneybut as a general preference i dont think we should write slow code by default12:50
aspiersstephenfin: but I'm wondering whether we should add a hacking check12:50
sean-k-mooneyaspiers: i think that is overkill12:50
stephenfinWe can, but we have less than three months of needing to care about Python 2.7 left12:51
stephenfinso yeah, overkill12:51
sean-k-mooneymaybe in Ussuri12:51
aspiersok12:51
aspierssean-k-mooney: in other news, https://review.opendev.org/#/c/67771012:51
sean-k-mooneye.g. in ussuri we should not be writing new code that uses six at all12:51
stephenfinsean-k-mooney: Within reason. I'll take readable code over slightly faster code in 90% of cases12:51
stephenfinYeah, if anything we could start stripping out six12:51
stephenfinwhere it made sense, that is12:52
sean-k-mooneystephenfin: sure but it adds up. if the perfomace is close readablity win if it way more readabel but way slower we should not use it in performance critical parts of nova or any project12:52
aspierssean-k-mooney: need you to check https://review.opendev.org/#/c/673151/13..15/nova/tests/unit/virt/libvirt/test_host.py12:53
sean-k-mooneyaspiers: ah your adding the fake data for capablites too. i had that in my orginial version but then dropped it as it was not needed12:53
aspierssean-k-mooney: not adding, more like moving and updating12:54
sean-k-mooneythat looks like a potentally good approch to take12:54
sean-k-mooneyrefactoring :)12:54
*** eharney has joined #openstack-nova12:55
sean-k-mooneyim in two minds about your test__get_machine_types fucntion12:56
sean-k-mooneyi dont like use hardcoding data like that in tests that are that coupled but i guess its ok12:56
sean-k-mooneyi guess its fine since it matching the fixture but it would be nic if you could get teh expected arch/machine types form the fixture12:57
sean-k-mooneyill think about it over lunch but ill proably +1 goes to actully have lunch now12:58
aspierssean-k-mooney: thanks :)12:59
*** lbragstad has joined #openstack-nova13:00
*** ociuhandu has quit IRC13:00
*** davee_ has joined #openstack-nova13:05
*** nweinber__ has joined #openstack-nova13:14
mdboothdansmith: I've got a much better feeling about this one: https://bugs.launchpad.net/nova/+bug/184091213:17
openstackLaunchpad bug 1840912 in OpenStack Compute (nova) "libvirt calls aren't reliably using tpool.Proxy" [Undecided,New]13:17
*** davee_ has quit IRC13:18
*** davee_ has joined #openstack-nova13:19
*** dulek has joined #openstack-nova13:20
dulekHi! This is probably trivial, but I see this in n-cpu logs on a fresh DevStack instance: `libvirtError: Cannot access storage file '/opt/stack/data/nova/instances/403824ef-422f-4e23-9e3a-d897766aa70d/disk' (as uid:107, gid:107)`. Any ideas how to get rid of that?13:20
*** tesseract has joined #openstack-nova13:22
*** tesseract has quit IRC13:22
dansmithmdbooth: okay, have to do something first but then I'll look13:29
mdboothdansmith: That's just the bug description, btw. Patch incoming. Patch might touch a couple of testing weird spots, so I'm expecting discussion/rework.13:30
*** brinzhang_ has joined #openstack-nova13:31
dansmithmdbooth: I'll be looking for why this wasn't a thing in the first eight years of its life, of course :)13:33
mdboothdansmith: I think it was ;) I suspect there's a point 2 to this bug which is why libvirt is being slow enough for this customer to trigger it.13:33
mdboothHowever, we tested it and it's definitely not using the proxy13:33
*** brinzhang has quit IRC13:34
*** ratailor has quit IRC13:36
*** munimeha1 has joined #openstack-nova13:39
*** shilpasd has quit IRC13:39
*** mriedem has joined #openstack-nova13:39
*** eharney has quit IRC13:40
dansmithmdbooth: ah, I see, looks like a good lead13:42
openstackgerritAdam Spiers proposed openstack/nova master: Make memtune parameters consistent with libvirt docs and code  https://review.opendev.org/63630113:46
*** zbr|ooo is now known as zbr13:47
*** udesale has quit IRC13:47
*** boxiang has quit IRC13:48
*** jawad_axd has quit IRC13:48
*** boxiang has joined #openstack-nova13:49
*** jawad_axd has joined #openstack-nova13:49
*** jawad_axd has quit IRC13:49
*** BjoernT_ has joined #openstack-nova13:50
*** jawad_axd has joined #openstack-nova13:50
*** jawad_ax_ has joined #openstack-nova13:52
openstackgerritAdam Spiers proposed openstack/nova master: Track libvirt host/domain capabilities for multiple machine types  https://review.opendev.org/67315113:53
*** jawad_axd has quit IRC13:55
*** bnemec has joined #openstack-nova13:55
*** jawad_ax_ has quit IRC13:56
openstackgerritMatthew Booth proposed openstack/nova master: libvirt: Fix service-wide pauses caused by un-proxied libvirt calls  https://review.opendev.org/67773613:57
*** pvradu has joined #openstack-nova14:00
*** pvradu has quit IRC14:05
*** shilpasd has joined #openstack-nova14:10
openstackgerritAdam Spiers proposed openstack/nova master: Extract new base class for provider usage functional tests  https://review.opendev.org/67696414:10
openstackgerritAdam Spiers proposed openstack/nova master: Provide HW_CPU_X86_AMD_SEV trait when SEV is supported  https://review.opendev.org/63868014:10
openstackgerritAdam Spiers proposed openstack/nova master: Add extra spec parameter and image property for memory encryption  https://review.opendev.org/66442014:10
openstackgerritAdam Spiers proposed openstack/nova master: Extract SEV-specific bits on host detection  https://review.opendev.org/63633414:10
openstackgerritAdam Spiers proposed openstack/nova master: Add <launchSecurity> and <driver iommu='on' /> to config.py  https://review.opendev.org/63631814:11
openstackgerritAdam Spiers proposed openstack/nova master: Apply SEV-specific guest config when SEV is required  https://review.opendev.org/64456514:11
openstackgerritAdam Spiers proposed openstack/nova master: Enable booting of libvirt guests with AMD SEV memory encryption  https://review.opendev.org/66661614:11
*** gregwork has quit IRC14:14
*** mrhillsman has quit IRC14:14
*** portdirect has quit IRC14:14
*** coreycb has quit IRC14:14
*** eharney has joined #openstack-nova14:15
*** kmalloc has quit IRC14:15
*** mrhillsman has joined #openstack-nova14:15
*** jmlowe has quit IRC14:16
*** mordred has quit IRC14:16
*** jbryce has quit IRC14:16
*** mgagne has quit IRC14:16
*** csatari has quit IRC14:16
*** logan- has quit IRC14:16
*** arne_wiebalck has quit IRC14:16
*** irclogbot_3 has quit IRC14:17
*** logan_ has joined #openstack-nova14:17
*** mgagne has joined #openstack-nova14:17
*** ebbex has quit IRC14:18
*** irclogbot_2 has joined #openstack-nova14:18
*** ebbex has joined #openstack-nova14:18
*** portdirect has joined #openstack-nova14:18
*** logan_ is now known as logan-14:18
*** coreycb has joined #openstack-nova14:18
*** gregwork has joined #openstack-nova14:18
*** jmlowe has joined #openstack-nova14:19
*** csatari has joined #openstack-nova14:19
*** kmalloc has joined #openstack-nova14:19
*** jbryce has joined #openstack-nova14:19
*** arne_wiebalck has joined #openstack-nova14:19
*** Jeffrey4l_ has quit IRC14:19
openstackgerritMatthew Booth proposed openstack/nova master: libvirt: Fix service-wide pauses caused by un-proxied libvirt calls  https://review.opendev.org/67773614:20
*** Jeffrey4l has joined #openstack-nova14:21
*** mordred has joined #openstack-nova14:26
*** mdbooth has quit IRC14:31
*** mdbooth has joined #openstack-nova14:35
*** pvradu has joined #openstack-nova14:36
openstackgerritAdam Spiers proposed openstack/nova master: libvirt: Parse the 'os' element from domainCapabilities  https://review.opendev.org/67379014:36
*** ociuhandu has joined #openstack-nova14:39
mriedemmdbooth: can we avoid the libvirt-python test dependency? i think that will mean local dev needs to install a native package, from what i remember years ago anyway14:39
mdboothmriedem: Yeah, I threw that in there to have a starting point, tbh14:40
mriedemand it was a source of problems before too14:40
*** davee_ has quit IRC14:40
mriedemok14:40
mdboothmriedem: I mostly wanted to get a full tempest run against it.14:40
mriedemso WIP it?14:40
mdboothmriedem: Will do.14:40
mriedemthanks14:41
*** jaosorior has quit IRC14:41
mdboothmriedem: I really want eyes on it, though :) Also, ideas for useful testing without using libvirt.14:41
mdboothMight look at something using fakelibvirt14:42
mriedemi assumed fakelibvirt14:42
mriedemsince that's what we always use14:42
sean-k-mooneymdbooth: you porbaly could mock the call to inspect14:42
sean-k-mooneyand pass in a set of different moduels for it to use and assert they were wrapped14:42
mdboothsean-k-mooney: The test I really want to do is that Host.list_instance_domains() returns <list of tpool.Proxy(virDomain)>14:43
*** hemna has joined #openstack-nova14:43
mdboothIf I can get that, I'll be confident that a bunch of things are lined up14:43
sean-k-mooneyi think that would be doable with out the really libvirt module14:44
mdboothI think so14:44
*** tbachman has quit IRC14:44
mriedemwere you going to open an issue against eventlet for the not-handling-list thing?14:44
sean-k-mooneywe might have to asser it returns <list of tpool.Proxy(mock)>14:44
*** mlavalle has joined #openstack-nova14:45
sean-k-mooneyor fakelibvirt.virDomain but we should be able to cover the behaivor i think14:45
openstackgerritStephen Finucane proposed openstack/nova master: Remove 'hw:cpu_policy', 'hw:mem_page_size' extra specs from API samples  https://review.opendev.org/67533814:48
openstackgerritStephen Finucane proposed openstack/nova master: scheduler: Flatten 'ResourceRequest.from_extra_specs', 'from_image_props'  https://review.opendev.org/67489414:48
openstackgerritStephen Finucane proposed openstack/nova master: Add support for translating CPU policy extra specs, image meta  https://review.opendev.org/67180114:48
openstackgerritStephen Finucane proposed openstack/nova master: libvirt: Fold in argument to '_update_provider_tree_for_vgpu'  https://review.opendev.org/67672914:48
openstackgerritStephen Finucane proposed openstack/nova master: Add reshaper for PCPU  https://review.opendev.org/67489514:48
*** dtantsur has joined #openstack-nova14:48
*** ratailor has joined #openstack-nova14:49
dtantsurhey folks. TheJulia and I might be seeing a race condition during ironic node take over.14:49
dtantsurit seems that when the hash ring rebalances, the old n-cpu may clean up the node records, including the resource provide in placement14:50
sean-k-mooneynode takeover e.g. when a node is moved to a different ironic compute service14:50
dtantsuryep14:50
*** my_nd has joined #openstack-nova14:50
sean-k-mooneyis this for all node or just ones in mantaince mode14:50
dtantsurit was for a normal node14:50
dtantsurapparently under some condition the new (receiving) n-cpu may not re-create the resource provider for the node it took over14:51
sean-k-mooneydtantsur: ok mnaser noted there is an issue with nodes in mainance mode being deleted14:51
*** pvradu has quit IRC14:51
dtantsurI think it was fixed by mriedem recently14:51
sean-k-mooneyproably i just head about it a day or two ago14:52
*** lbragstad has quit IRC14:52
mnasersean-k-mooney, dtantsur, TheJulia: https://review.opendev.org/#/c/675704/ is the stack you might want to look at14:52
openstackgerritMatt Riedemann proposed openstack/nova master: Remove deprecated [neutron]/url option  https://review.opendev.org/67775514:52
sean-k-mooneydtantsur: its possible the reciving node check if the RP exists pefore teh souce node deletes it14:52
dtantsuryep, that's my guess14:53
dtantsurrelevant log lines: https://etherpad.openstack.org/p/zCQ8vPcGX614:53
* dtantsur pokes jroll14:53
sean-k-mooneythe first patch looks relevent https://review.opendev.org/#/c/675704/14:53
jroll?14:54
mriedemmordred: efried: i'm removing the old deprecated [neutron]/url option in https://review.opendev.org/677755 and i noticed that the neutron install guide still uses it: https://bugs.launchpad.net/neutron/+bug/1840930 - as i mentioned in that neutron bug, we could just replace with endpoint_override but i don't think that's what we want, right?14:54
openstackLaunchpad bug 1840930 in neutron "Networking service in neutron - install guide says to configure nova with [neutron]/url which is deprecated" [Undecided,New]14:54
mriedemmordred: efried: we want to avoid endpoint_override and just have config for letting KSA do it's thing with the service catalog14:54
dtantsurjroll: see scrollback, potential race condition on take over14:54
dtantsuryou may know this code better than me14:54
TheJuliaSo an important note from what dtantsur is mentioning is that we're seeing this in our grenade job, which means this is when stein is still running from what we see in the logs14:54
mriedemstein doesn't have the fix yet14:55
mriedemwaiting for stable reviews14:55
mriedemhttps://review.opendev.org/#/q/topic:bug/1839560+branch:stable/stein14:55
sean-k-mooneyTheJulia: is this only before we do the upgrade.14:55
jrollthere we go14:55
jrolldtantsur: side question: does the new compute service re-add it on the following RT run?14:56
dtantsurjroll: nope, never14:56
TheJuliasean-k-mooney: well, we never actually get to perform the upgrade14:56
jrolloh boy14:56
dtantsurit becomes orphaned essentially14:56
efriedmriedem: Right; for people who just gotta specify, they can s/url/endpoint_override/, but it is better to just let the service catalog do its thing.14:56
mriedemefried: yeah, i think the question is - for an install guide, we don't want to document the exceptional path14:56
sean-k-mooneyTheJulia: i was wondering if ye tried it manually but ya it would fail tempest and not upgrade so make sense14:56
mriedemso i was just going to remove the url mention from the install guide14:56
*** mdbooth has quit IRC14:57
mriedemdtantsur: TheJulia: i'm curious, this has been a thing since rocky, so why are you hitting it now?14:57
mriedemnew tests?14:57
TheJuliasean-k-mooney: we wouldn't even make it to tempest, there is a resource check where placement is consulted14:57
TheJuliaif we don't have the number of "baremetal" nodes, we eventually time out and fail14:57
*** davee_ has joined #openstack-nova14:57
dtantsurmriedem: I'm curious as well. The simplest explanation is that it's become more likely for some unrelated reason.14:57
mriedemhttps://review.opendev.org/#/c/571535/ is since rocky i mean14:57
efriedmriedem: We should just be able to document "do this the same way you would do any service backed by the catalog". mordred might have a central document somewhere that recommends best practices for that. If we don't have that doc, we should.14:57
* dtantsur remembers how switching to pypy on one of his projects uncovered a few completely unexpected races because of how much faster it was14:58
sean-k-mooneyTheJulia: well if we can recreate it reliably we proably should add a depens on against the stien backport and see if that resolves it14:58
TheJuliaWe've seen some fun variations in performance so I guess any changes are going to cause us to find races more often....14:58
mriedemjroll: before the fix, the old compute will delete the comptue node and resource provider, but the compute node record is soft deleted; the new compute will try to create the compute node record with the same uuid - which has a unique constraint on it - and fail, so it won't re-create the resource provider14:58
TheJuliasean-k-mooney: could work if it survives a couple rechecks... I think the overall job failure rate has been only 20-30%.14:59
*** mdbooth has joined #openstack-nova14:59
mriedemi don't know if this matters for y'all, but when the old compute deletes the compute node and resource provider, it will also delete any allocations for an instance on that node/provider, and the new compute won't re-create those allocations...15:00
sean-k-mooneythat is still high enough to be concerned15:00
mriedemback in ocata/pike the new could RT would re-create the allocations15:00
mriedembut that code was removed...15:00
dtantsurouch15:00
jrollmriedem: yep, that makes sense. we're seeing something similar in ocata recently, but I can't decide if this is related (as the compute_nodes table is also affected)15:00
jrollum, that allocations thing seems... really bad15:00
mriedemjroll: the "re-use the node uuid for the compute node uuid" thing was only rocky15:00
jrollmriedem: right, that's just why I was asking for more info15:01
sean-k-mooneyjroll: deleteing all the allocation is require to be able to delet eh placmeent RP15:01
mriedemjroll: https://review.opendev.org/#/c/602481/15:01
sean-k-mooneyif we dont do that then we cant delete the rp wehn we delete the compute service reliably15:02
jrollsean-k-mooney: sure, but in an ironic world where we churn these "compute nodes" all the time, that seems scary15:02
mriedem^ removed the thing so that the ironic driver would always tell the RT to report the allocations15:02
jrollmaybe I'm misunderstanding how this works15:02
jrolland I really don't have time to deep dive this part of nova today :(15:02
mriedemsince i never think if ironic node re-balancing stuff, and we don't test it (at least in nova) i guess i didn't think about this15:02
mriedemi would also think that johnthetubaguy (who added this workaround i think) and stackhpc would have noticed a problem by now15:03
mriedembut i don't know what version of openstack they are giving to customers15:03
mriedemhttps://review.opendev.org/#/c/602481/ was stein15:03
sean-k-mooneywhen we are rebalcne we are moving the compute nodes between compute service right. and in an ironic case we have mupliple compute nodes (1 per server) each with there one placment RP associates to a singel compute service correct15:04
sean-k-mooneyjust making sure i have that right in my head15:04
mriedemironic node : compute node : instance = 1:1:115:04
jrollcorrect, until the end - there is a placement RP for each ironic node / compute node15:04
mriedem+ : resource provider15:04
sean-k-mooneyjroll: yep that is want i ment15:05
sean-k-mooneyone placment RP per compute/ironic node15:05
jrollcool, you are correct then15:05
sean-k-mooneydurign a reblance we are not actully delet ing he nova compute service we are just moveing the comnpute nodes so we really should not need to update placmeent15:06
sean-k-mooneyor rather delete the compute node/placmente RP15:06
*** priteau has joined #openstack-nova15:07
mriedemthe compute manager doesn't know you're doing a re-balance15:07
sean-k-mooneyall we should need to do on the nova side is modify the service to compute node mapping but i dont know how we do that today15:07
jrollexcept that a rebalance happens when the number of compute services changes. so if you shut down a compute service (which deletes compute nodes and RPs), ironic nodes get moved to another, and those compute nodes and RPs are re-created15:07
mriedemit knows, "oh i was managing node x before, but now the driver says i'm not, so delete"15:07
mriedemjroll: yeah and before rocky, those new compute nodes and RPs would all have a unique uuid15:08
mriedemsince rocky, they re-use the node uuid15:08
jrollyeah15:08
mriedemso i think with my recent fix that's all fine,15:09
mriedemthe question is what happens with allocations against the re-balanced node resource provider15:09
mriedemb/c i'm guessing since stein those are gone15:09
sean-k-mooneymriedem: right but im wonder that im not "managing because the drive said so, therefor delete is required"15:09
jrollyeah, that's what worries me15:09
mriedemi guess someone needs to do some testing15:09
sean-k-mooneyin a libvirt case we would normaly delete the compute service and have it delete the compute nodes under neat it15:10
mriedemsean-k-mooney: the compute service doesn't delete itself,15:10
mriedemthat happens in the API15:10
mriedemDELETE /os-services/{service_id}15:10
sean-k-mooneywell yes15:10
mriedemwhich is related to a number of other fun issues15:10
* TheJulia reads back15:10
sean-k-mooneyi know that but that is what i ment15:10
mriedemwhich are in the ML15:10
sean-k-mooneywe externally trigger the deletion of the compute service when removing a libvir host15:11
sean-k-mooneydoes vmware do something similar to ironic15:11
sean-k-mooneythey have multiple compute nodes in the same service too right15:11
mriedemhttp://lists.openstack.org/pipermail/openstack-discuss/2019-June/007135.html15:11
mriedemsean-k-mooney: no, vmware driver does not do that since kilo or liberty15:12
mriedemTheJulia: tl;dr, i think you need https://review.opendev.org/#/q/topic:bug/1839560+branch:stable/stein for your grenade failures,15:13
mriedemand there might be a regression due to https://review.opendev.org/#/c/602481/ where allocations in placement are lost during a re-balance15:14
mriedembut someone needs to test that15:14
TheJuliamriedem: sean-k-mooney: I added a depends-on flag to one of my test patches for the patches above, if I see the issue re-appear on that job, I'll let you all know15:14
mriedemso if the lost allocations on re-balance is a real thing, i might have a solution, which i wrote down in a comment in https://review.opendev.org/#/c/602481/15:18
mriedemb/c the code that deletes the resource provider and it's allocations pre-dates the removal of the RT allocation healing code, and unsurprisingly we lost track of all those random one off moving parts15:19
mriedemespecially since we don't have functional testing of that stuff in tree15:19
*** ivve has quit IRC15:21
*** gyee has joined #openstack-nova15:22
*** pvradu has joined #openstack-nova15:25
* efried <== doc15:33
*** efried is now known as efried_afk15:33
openstackgerritAdam Spiers proposed openstack/nova master: libvirt: Parse the 'os' element from domainCapabilities  https://review.opendev.org/67379015:37
gibimriedem: hi! I have no problem moving the small parts of the cross-cell resize series to the front. I can re-apply my +2 if you rebase the series15:39
mriedemgibi: ok thanks15:39
openstackgerritMatt Riedemann proposed openstack/nova master: FUP for I5576fa2a67d2771614266022428b4a95487ab6d5  https://review.opendev.org/67776915:46
*** tbachman has joined #openstack-nova15:50
*** mkrai has quit IRC15:50
*** mkrai has joined #openstack-nova15:53
*** mgagne has quit IRC15:54
*** mgagne has joined #openstack-nova15:55
aspiersanyone object to me splitting libvirt/test_driver.py into smaller pieces? at 23k LoC it's almost *double* the size of the next largest file in nova (compute/test_compute.py in case you're curious)15:55
aspiersmy editor really hates such a huge file, and so do I15:55
aspiersefried_afk, sean-k-mooney: ^^^15:56
*** igordc has joined #openstack-nova15:56
aspiersof course this will probably cause a few merge conflicts so maybe we need to time it right15:56
sean-k-mooneyaspiers: you mean it does not like 10s of 1000s or lines15:56
aspierssean-k-mooney: yes, syntax highlighting something that big is really expensive15:57
aspiersunless your highlighting algorithm cuts corners15:57
*** pvradu has quit IRC15:58
sean-k-mooneypycharm does not mind too much15:58
sean-k-mooneyonce it has built the index15:58
aspiersYikes this is a huge list https://review.opendev.org/#/q/project:openstack/nova+path:nova/tests/unit/virt/libvirt/test_driver.py+status:open+branch:master15:59
aspiersStill, we could split stuff out in smaller steps to be less disruptive16:00
aspiersI was thinking to move all the capabilities / domcaps tests out for a start16:00
aspierslibvirt/driver.py is 10k LoC, that should also be split up16:01
*** pvradu has joined #openstack-nova16:02
aspiersanyway I'm done for today16:02
*** KeithMnemonic1 has joined #openstack-nova16:02
aspiersprobably won't do much the rest of this week16:02
aspierssince I've been doing way too much overtime recently16:02
*** KeithMnemonic has quit IRC16:03
*** rpittau is now known as rpittau|afk16:05
*** pvradu has quit IRC16:05
*** ricolin has quit IRC16:05
*** markvoelker has quit IRC16:08
sean-k-mooneyaspiers: yes it shoudl but with the rate of change that is hard to do without a dedicated effort16:12
*** ratailor has quit IRC16:12
sean-k-mooneyaspiers: and ya dont burn out.16:12
mriedemefried_afk: did cern reach out to you about the deepcopy of the ProviderTree? i think it came up last week, but was reminded after reading https://techblog.web.cern.ch/techblog/post/nova-ironic-at-scale/16:21
openstackgerritMatthew Booth proposed openstack/nova master: libvirt: Fix service-wide pauses caused by un-proxied libvirt calls  https://review.opendev.org/67773616:21
*** markvoelker has joined #openstack-nova16:22
mriedemaspiers: you mean moving driver.py tests out of test_driver.py? but leave the code in driver.py? that would be confusing to me, and becomes a mess for git history tracking16:22
mriedemthere is probably a ton of copy/paste patterns in libvirt driver tests that could be a simple cleanup to start with, rather than moving things *out* of the test_driver.py module16:23
*** damien_r has quit IRC16:35
*** dtantsur is now known as dtantsur|afk16:37
*** mdbooth has quit IRC16:37
*** igordc has quit IRC16:41
*** boxiang has quit IRC16:42
*** spsurya has quit IRC16:43
*** boxiang has joined #openstack-nova16:43
*** davee_ has quit IRC16:45
*** Garyx has quit IRC16:45
*** xek has quit IRC16:47
*** Garyx has joined #openstack-nova16:49
*** davee_ has joined #openstack-nova16:54
*** ivve has joined #openstack-nova16:56
*** bbobrov has quit IRC17:00
*** derekh has quit IRC17:00
openstackgerritMatt Riedemann proposed openstack/nova master: WIP: Get instances outside of update_available_resource lock  https://review.opendev.org/67779017:02
*** psachin has quit IRC17:04
*** mkrai has quit IRC17:19
*** damien_r has joined #openstack-nova17:19
*** priteau has quit IRC17:21
*** lbragstad has joined #openstack-nova17:21
openstackgerritMerged openstack/nova master: Move calls to ovs-vsctl to privsep.  https://review.opendev.org/63928217:29
openstackgerritMerged openstack/nova master: Move setting of device trust to privsep.  https://review.opendev.org/63928317:32
*** ociuhandu has quit IRC17:32
*** mdbooth has joined #openstack-nova17:33
*** nweinber_ has joined #openstack-nova17:35
*** macz has joined #openstack-nova17:36
*** nweinber__ has quit IRC17:37
*** mdbooth has quit IRC17:38
*** my_nd has quit IRC17:43
efried_afkaspiers: I love the idea of splitting huge files. sean-k-mooney, syntax highlighting isn't the problem so much as the inspection. Every change causes it to reinspect the file, which takes forever on monsters like test_libvirt. If I'm spending any amount of time there, I have to turn inspection off.17:44
*** damien_r has quit IRC17:45
dansmithefried_afk: without any other context or target articles,17:45
dansmiththat first sentence sounds like it belongs on your online dating profile17:46
dansmith"I like long walks on the beach and splitting huge files"17:46
*** efried_afk is now known as dansmith117:46
*** dansmith1 is now known as efried17:46
sean-k-mooneyefried_afk: i have not had that problem but it does annoy me that thost files are so large17:47
efrieddansmith: It was a straight copy/paste17:47
dansmithefried: I figured17:47
dansmithMale seeking cybernetic female analog capable of introspection of long python files and/or automated splitting thereof17:47
efriedmriedem: I don't recall anything specific about a ProviderTree deepcopy...17:47
*** shilpasd has quit IRC17:48
efriedoo, that's good dansmith. Clearly you have a lot of experience here.17:48
dansmithmy kink knows no bounds17:48
efrieddansmith: are you happy with https://review.opendev.org/#/c/671072/ series at this point?17:51
efriedgibi: also ^17:52
dansmithefried: I haven't circled back17:52
openstackgerritEric Fried proposed openstack/nova master: Remove deprecated [neutron]/url option  https://review.opendev.org/67775517:56
aspiersmriedem: I would happily move the corresponding code out from driver.py at the same time if that was feasible17:56
aspiersI'm not sure why it would become a mess for git history tracking? even git blame has for a long time been able to figure out when stuff gets moved between files17:57
aspiersefried: exactly17:57
mriedemi'm not a fan of the tests for something in one module living in another module17:57
aspiersmriedem: that said, pointers on stuff to deduplicate are welcome17:57
mriedemand like i said, i think you can just refactor the duplication in test_driver.py for lots of tests to probably make it smaller17:58
aspiersI think our messages crossed :) ^^^17:58
mriedemthis isn't an exercise i'm the one that's wanting to take on, so that's up to you,17:58
mriedembut i know there is a ton of just copy test foo and tweak 1-2 lines for test bar17:58
mriedemi also have no problem with the thing in pycharm17:59
efriedaspiers: git history is an issue, unfortunately - afaik blame will show the new commit on the new file. You would have to back up to see the previous iteration, which is awkward. (I've run into this a number of times already in placement, and it's pretty debilitating.)18:00
mriedemcorrect, it sucks18:00
mriedemthe mox removal series is a prime example18:00
mriedemit also causes issues with merge conflicts for backports18:00
sean-k-mooneywell perhaps this is jsut somethin we need to be aware of and think should we be moving things out of the driver into modules used by the driver in the future18:00
aspiersefried: no, blame really can show where it came from18:01
sean-k-mooneyaspiers: yes but git blame on github crashes on those files18:01
sean-k-mooneyso you have to do it locally18:01
aspiers?18:01
efriedwhat's the blame trick, without iterating backward across commits?18:01
aspiersmriedem's point about backports is a much more convincing argument against18:01
aspiershowever, if our policy is "you can't move stuff around, ever" then we're in for a miserable future18:02
mriedemi'm not against refactoring giant methods in the libvirt driver module, deduplicating test code in test_driver, etc. i just don't want to see random tests moving out of test_driver into some other module when the code being tested is still in driver.py18:02
aspiersmriedem: sure, that's why I said <aspiers> mriedem: I would happily move the corresponding code out from driver.py at the same time if that was feasible18:03
mriedemaspiers: no one said that was the policy18:03
sean-k-mooneyaspiers: try opening https://github.com/openstack/nova/blame/master/nova/virt/libvirt/driver.py18:03
*** igordc has joined #openstack-nova18:03
aspiersmriedem: ok good :)18:03
aspierswas beginning to sound that way18:03
mriedemas a nova troll i have a reputation to maintain18:04
aspiers:)18:04
mriedemas immovable object18:04
sean-k-mooneyi think we just dont want test to be move with out the code also moving and am not sure that patches that only move code are a good idea18:04
*** lbragstad_ has joined #openstack-nova18:04
aspiersefried: try git gui blame for instance18:04
*** lbragstad_ has quit IRC18:04
aspierssome tooltips include "Copied Or Moved Here By:"18:04
aspierscan't remember if github or gitea do the same18:05
sean-k-mooneyaspiers: i think i makes sense to consider moving things if you are refacoritn the code already.18:05
aspiersbut local is way faster anyway IME18:05
aspierssean-k-mooney: definitely agree with that18:05
aspierswell, it's an idea to bear in mind anyway18:06
aspiers23k LoC in one file does not seem sane to me ;-)18:06
sean-k-mooney23K of code to test 9.75K of code and we still miss things...18:07
sean-k-mooneythe driver is not that bad coverage wise but ther are other part that are less awsome...18:07
efriedaspiers: git gui appears to be... a GUI :( :( :(18:08
aspiersefried: shocker! who'd have thought?18:08
efriedguess it would take some getting used to.18:08
sean-k-mooneyaspiers: that is not a standard git command is it a python extentions18:09
efriedanyway, I agree enormous files suck, and being stuck with them because of sucky reasons also sucks.18:09
efriedsean-k-mooney: apt install git-gui18:09
efriedthat's if you're running ubuntu.18:09
efriedIf you're running another distro... just shoot yourself and be done.18:09
sean-k-mooneyPopOS which is based on ubuntu yes18:10
aspiersefried: watch your mouth!18:10
efried(sorry, I had a bad experience with rhel a decade ago and still haven't recovered)18:10
aspiersyou ran RHEL as a desktop? eww18:10
sean-k-mooneyalthough i have a rhel8 system for the last while... not sure i will keep it but it currently works18:10
aspiersFedora is OK18:10
efriedyeah, that was my problem18:10
aspierstry openSUSE :)18:10
efriedis that still a thing??18:10
efriedIs it written in FORTRAN?18:10
* aspiers just about manages not to swear18:11
sean-k-mooneyaspiers: i used tumbelweed for a while and mostly liked it18:11
aspiersok, good time to leave18:11
* aspiers goes for a run18:11
sean-k-mooneynot sure i liked yast but it was nice to have up to date packages18:11
*** markvoelker has quit IRC18:16
*** mchlumsky has joined #openstack-nova18:18
*** markvoelker has joined #openstack-nova18:22
*** markvoelker has quit IRC18:30
*** ralonsoh has quit IRC18:35
*** bbobrov has joined #openstack-nova18:35
*** ralonsoh has joined #openstack-nova18:35
mriedemmelwitt: a couple of things in the archive patch https://review.opendev.org/#/c/507486/18:49
mriedemand a couple of nits in the nova-next patch on top of that18:49
mriedemboth should be small18:50
*** nweinber_ has quit IRC18:50
mriedemmelwitt: efried: i think we're ok to merge https://review.opendev.org/#/c/675218/6 now yeah?18:50
melwittthanks18:51
* efried reinstates +218:51
melwittyeah I think all is well18:51
*** factor has quit IRC18:52
mriedemonto openstack resource provider inventory set --aggregate18:55
*** maciejjozefczyk has quit IRC18:56
melwittheh18:57
*** mriedem has quit IRC18:58
*** mriedem has joined #openstack-nova18:59
*** ralonsoh has quit IRC19:05
melwittI was panicking with the potential loss of my launchpad bug text after a 503 "TimeoutError" but found when I reloaded the page, it posted everything I had written19:10
melwittphhheeewwww19:10
mriedemhappens a lot, just have to keep refreshing19:13
openstackgerritAdam Spiers proposed openstack/nova master: Add extra spec parameter and image property for memory encryption  https://review.opendev.org/66442019:17
openstackgerritAdam Spiers proposed openstack/nova master: Extract SEV-specific bits on host detection  https://review.opendev.org/63633419:21
openstackgerritAdam Spiers proposed openstack/nova master: Add <launchSecurity> and <driver iommu='on' /> to config.py  https://review.opendev.org/63631819:21
mriedemmelwitt: i think i have an idea for how to make one of multiple rps fail to set inventory in aggregate in https://review.opendev.org/#/c/640898/1119:22
mriedemadd allocations to one of them19:22
openstackgerritmelanie witt proposed openstack/nova master: Make a failure to purge_db fail in post_test_hook.sh  https://review.opendev.org/67780619:24
melwittmriedem: nice! thanks, I was stumped about that19:25
openstackgerritAdam Spiers proposed openstack/nova master: Add extra spec parameter and image property for memory encryption  https://review.opendev.org/66442019:25
openstackgerritAdam Spiers proposed openstack/nova master: Extract SEV-specific bits on host detection  https://review.opendev.org/63633419:26
openstackgerritAdam Spiers proposed openstack/nova master: Add <launchSecurity> and <driver iommu='on' /> to config.py  https://review.opendev.org/63631819:26
*** nweinber_ has joined #openstack-nova19:27
mriedemmelwitt: same, until about 5 minutes ago :)19:28
melwitt\o/19:29
mriedemlooking at the placement code i'm not totally sure if it handles the "drop inventory below what's allocated but don't remove the resource class" or if it's just "can't remove a resource class inventory that has allocations" - i see the latter in the placement code but not sure about the former, but i'm pretty sure it enforces that19:29
mriedemefried: ^ right?!19:29
mriedemhmm19:31
mriedemhttps://review.opendev.org/#/c/640898/1119:31
efriedmriedem: You can reduce inventory below what's allocated. You can't set total to zero (I'm 95% sure of that, would have to check).19:31
mriedemops19:31
mriedemefried: not before 1.26 anyway..19:31
mriedemyeah so if your allocations exceed capacity it only logs a warning19:32
mriedemhttps://github.com/openstack/placement/blob/master/placement/objects/resource_provider.py#L61919:32
mriedemif you try to remove a resource class entirely that has allocations it'll fail https://github.com/openstack/placement/blob/master/placement/objects/resource_provider.py#L8419:32
mriedemso melwitt i guess one of the providers has to have some resource class inventory and a consumer is using that rc, and then in set inventories you drop/omit that rc,19:33
mriedemit could just be some CUSTOM_FOO in your test19:33
melwittah ok19:33
melwittsophisticated19:33
mriedemefried: can't set total to 0 but you can set reserved = total with 1.2619:35
efriedyes19:35
efriedwe did that specifically so you could take an ironic node out of the pool19:35
mriedemyeah i know19:35
mriedemmel is trying to trigger a failure on one provider's PUT inventories out of a set of providers in an aggregate19:36
mriedemanyway, think it's figured out, i just didn't realize placement would let you do the other thing here https://github.com/openstack/placement/blob/master/placement/objects/resource_provider.py#L61919:36
efrieddoes it matter what the failure is?19:36
efriedYou could PUT a resource class that doesn't exist yet19:36
melwittlike a custom class placement doesn't yet know about?19:37
efriedyeah19:37
melwittahhh that would be easier19:37
efried(careful if you're trying to drive this stuff through the report client - some paths will automatically ensure RCs for you)19:38
melwittI was thinking inside the box of the standard classes19:38
efriedwell, you could make those fail too by setting total=0 :)19:38
melwittthis is for osc-placement so should be safe there19:38
efriedor total=foo for that matter19:39
melwittok, well, thanks for making this sound really stupidly easy :P19:39
efriedhell, getting an inventory PUT to *fail* is easy!19:39
efriedgetting it to *succeed* is the hard part19:39
melwittwell hang on19:39
melwittneeds to fail for only one RP though19:40
efriedare you putting multiple inventories at once?19:40
openstackgerritAdam Spiers proposed openstack/nova master: Apply SEV-specific guest config when SEV is required  https://review.opendev.org/64456519:40
efriedlike via a single REST call?19:40
melwittthis is a new --aggregate command which will loop over RPs in an aggregate and do the same inventory set on all of them19:40
melwittand I need to fail one in the middle19:40
efriedright, so you need to fail one (entire) REST call in a loop of REST calls19:40
melwittfor test coverage for a "log error and continue" behavior19:40
melwittyeah19:41
efriedso yeah, you can just make a bogus inventory value in that one call19:41
efriedwhich is kinda what you're talking about doing I guess.19:41
openstackgerritAdam Spiers proposed openstack/nova master: Enable booting of libvirt guests with AMD SEV memory encryption  https://review.opendev.org/66661619:41
melwittyeah but that's not what the command wants to do, it does the same 'set' for each19:41
melwittideally I'd want to muck with one of the RPs ahead of time such that it would fail an identical request while the others succeed19:42
melwittso I think I will have to do the allocation thing19:43
*** markvoelker has joined #openstack-nova19:44
efriedmelwitt: Yeah, you can add a CUSTOM_FOO to (only) that RP and create an allocation against it. Then since you're PUTting the same inventory (which presumably lacks CUSTOM_FOO) against all the providers, it'll fail for that one since it would amount to removing the CUSTOM_FOO inventory.19:45
efriedIs that what mriedem suggested?19:45
melwittyes19:45
efriedsmart man19:45
melwittyup19:45
*** markvoelker has quit IRC19:49
*** BjoernT_ is now known as BjoernT19:50
* mriedem writes it down19:54
*** davee_ has quit IRC19:54
melwittmriedem: how are you finding the good link to the post_test_hook run under ara-report?19:55
*** factor has joined #openstack-nova19:55
mriedemyou know to get to the ara-report right? https://0bde141365cc25860c5b-0b668b63edb9a2dd0711fb78b2335400.ssl.cf1.rackcdn.com/672840/11/check/nova-next/daad30a/ara-report/19:56
melwittyeah19:56
mriedemi find the post.yaml with 1 task after the devstack-tempest run19:56
melwittoh, I was clicking the > arrow19:56
melwittand just saw a ton of hashes19:56
mriedemexpand that, then click on the 'changed' box19:56
melwittthanks19:57
mriedemthat gives you a popup,19:57
mriedemthen hit the chain link icon in the top right19:57
melwittgot it, thank you19:58
mriedemefried: reading your reply to alex's thread, why do you say that the numa and pci fields in the instance migratoin context are hypervisor specific? as far as i know they aren't, since they are used by libvirt and i think the hyperv driver as well during moves20:05
efriedmriedem: then they're specific to libvirt and hyperv.20:06
efriedthat's luck more than anything else20:06
mriedemyou mean because they wouldn't work for powervm or what?20:06
efriedyeah, that's the example that springs to my mind obviously.20:06
efriedalso pretty n/a for ironic if I'm not mistaken.20:07
mriedemironic doesn't migrate20:07
mriedembut yeah20:07
efriedso like, *really* n/a :P20:08
mriedemthose fields are only populated if the instance has numa/pci stuff20:08
mriedemwhich baremetal instances won't have20:08
efriedright. And in alex_xu's new world, `resources` would only get populated if the instance has local resources to track.20:08
efriedAny time we just need a count, we leave that bit empty and just use the placement inventory structure.20:09
mriedemyeah i haven't read the whole message yet20:15
mriedemnor the etherpad20:15
mriedemonly what i've gathered from irc conversations20:15
mriedemefried: mordred: also, this is the patch to remove [neutron]/url from the neutron nova/neutron compute install guide in favor of a KSA-tastic world https://review.opendev.org/67781020:16
mriedemhopefully i didn't misrepresent in the commit message20:16
openstackgerritAdam Spiers proposed openstack/nova master: Enable booting of libvirt guests with AMD SEV memory encryption  https://review.opendev.org/66661620:19
efriedmriedem: nicely done, +120:22
mriedem\o/20:27
mriedemi know you sdk/kds/sta people are particular20:27
mriedemheh, i just realized that archive_deleted_rows returns 1 meaning, we archived some stuff, keep going until you get rc=0,20:28
mriedembut if the command blows up in some unexpected way, we return 1 (rather than like 255)20:28
efriedthat's eek20:28
mriedemdansmith: that seems not ideal yeah? ^20:28
mriedemhttps://github.com/openstack/nova/blob/master/nova/cmd/manage.py#L262520:29
dansmithhmm,20:29
dansmithI thought we had an error case20:29
dansmithoh20:29
dansmithyou mean if it bubbles up to main20:30
mriedemright20:30
mriedemTypeError or something dumb20:30
dansmithyeah20:31
dansmithhonestly, that "return 1" there in main probably conflicts with other things too, no?20:32
dansmith255 is a common thing for "I dunno wtf happened"20:32
mriedemright, nova-status returns 255, that's what got me looking at this20:33
openstackgerritMatt Riedemann proposed openstack/nova master: Document archive_deleted_rows return codes  https://review.opendev.org/67781920:38
efrieddansmith: couple easy ones, if you have a sec: https://review.opendev.org/#/c/676707/ and successor?20:38
dansmithmriedem: yeah I dunno what danger might lurk by changing that now, but surely seems like the right thing20:40
mriedemhttps://bugs.launchpad.net/nova/+bug/184097820:40
openstackLaunchpad bug 1840978 in OpenStack Compute (nova) "nova-manage commands with unexpected errors returning 1 conflict with expected cases of 1 for flow control" [Undecided,New]20:40
mriedemyeah, just said that in the bug report,20:40
mriedemi think it's just a simple upgrade reno yeah?20:40
dansmithmriedem: well, more like the impact to tripleo, osa, etc20:41
dansmiththis potentially affects lots of manage commands20:41
dansmithalthough this is really SHTF level fail cases, so nobody should really be depending on that20:41
mriedemi counted simple_cell_setup, map_instances and heal_allocations as returning 1 in non-error cases20:43
efriednobody should be relying on 1 being a real failure, because if they are, they're f'ed anyway.20:44
dansmithmriedem: no, I mean cases in all the other commands that could raise20:44
efriedso I think this is a situation where it's okay to change it.20:44
dansmithefried: that's my point, the only case would be somewhere that someone tests "is something done yet" by checking if something fails, and uses ret=1 to check20:44
openstackgerritEric Fried proposed openstack/nova master: Revert "Filter UnsupportedServiceVersion warning"  https://review.opendev.org/67782520:44
efriedright, which is invalid.20:45
efriedI'm agreeing with you20:45
efriedmark your calendar20:45
dansmithI don't think it's invalid20:46
openstackgerritMatt Riedemann proposed openstack/nova master: Revert "Filter UnsupportedServiceVersion warning"  https://review.opendev.org/67782520:46
dansmiththere are probably cases where there's no other way to tell if a thing is done yet or not20:46
*** damien_r has joined #openstack-nova20:49
*** damien_r has quit IRC20:49
openstackgerritMatt Riedemann proposed openstack/nova master: Revert "Filter UnsupportedServiceVersion warning"  https://review.opendev.org/67782520:50
mriedemefried: just rebased on top of the change mentioned in the commit20:50
efriedyeah, good plan20:50
efriedthough the warning would be gone anyway since 0.34.0 is in u-c20:51
*** damien_r has joined #openstack-nova20:58
*** damien_r has quit IRC20:58
*** nweinber_ has quit IRC20:59
openstackgerritMatt Riedemann proposed openstack/nova master: Document archive_deleted_rows return codes  https://review.opendev.org/67781921:06
openstackgerritMatt Riedemann proposed openstack/nova master: Change nova-manage unexpected error return code to 255  https://review.opendev.org/67783221:06
openstackgerritJames E. Blair proposed openstack/nova master: Fix issues building PDF docs  https://review.opendev.org/67702821:14
*** altlogbot_3 has quit IRC21:16
openstackgerritMatt Riedemann proposed openstack/nova master: Document map_instances return codes in table format  https://review.opendev.org/67783521:16
* mriedem goes to mow21:18
*** mriedem has quit IRC21:18
*** BjoernT has quit IRC21:22
*** rcernin has joined #openstack-nova21:27
*** altlogbot_0 has joined #openstack-nova21:37
*** altlogbot_0 has quit IRC21:38
*** altlogbot_2 has joined #openstack-nova21:41
*** altlogbot_2 has quit IRC21:42
*** takashin has joined #openstack-nova21:50
*** ivve has quit IRC21:57
*** markvoelker has joined #openstack-nova21:59
*** bnemec has quit IRC21:59
*** munimeha1 has quit IRC22:06
*** mlavalle has quit IRC22:16
*** markvoelker has quit IRC22:24
openstackgerritMerged openstack/nova master: Deprecate Aggregate[Core|Ram|Disk]Filters  https://review.opendev.org/67349622:27
openstackgerritMerged openstack/nova master: Re-use DB MetaData during archive_deleted_rows  https://review.opendev.org/67521822:27
*** threestrands has joined #openstack-nova22:34
*** hemna has quit IRC22:35
*** markvoelker has joined #openstack-nova22:35
*** rcernin has quit IRC22:40
*** markvoelker has quit IRC22:40
*** eharney has quit IRC22:41
*** rcernin has joined #openstack-nova22:43
*** tkajinam has joined #openstack-nova22:56
*** macz has quit IRC23:00
« Tuesday, 2019-08-20 Index Thursday, 2019-08-22 »

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!