Tuesday, 2019-07-30

brinzhang_mriedem: I am sorry later see, yeah, I missed the log under the below section, got it, thanks.00:00
*** mdbooth has quit IRC00:01
*** mdbooth has joined #openstack-nova00:01
*** mdbooth_ has quit IRC00:02
brinzhang_efried: Hi, are you around?00:03
*** betherly has joined #openstack-nova00:04
*** betherly has quit IRC00:09
*** threestrands has joined #openstack-nova00:10
*** artom has joined #openstack-nova00:25
*** tetsuro has joined #openstack-nova00:25
*** betherly has joined #openstack-nova00:25
*** gyee has quit IRC00:28
*** betherly has quit IRC00:30
openstackgerritMerged openstack/nova master: Privsep the ebtables modification code.  https://review.opendev.org/63543500:38
*** betherly has joined #openstack-nova00:46
*** betherly has quit IRC00:51
*** zhouyao has joined #openstack-nova00:51
zhouyaohttps://review.opendev.org/#/c/621646/ Any one can review this?00:52
*** tetsuro has quit IRC00:56
*** ricolin_ has joined #openstack-nova00:58
*** bhagyashris has joined #openstack-nova01:00
*** brinzhang has joined #openstack-nova01:00
*** brinzhang_ has quit IRC01:02
*** panda has quit IRC01:06
*** panda has joined #openstack-nova01:08
*** igordc has quit IRC01:15
*** betherly has joined #openstack-nova01:18
*** bbowen has joined #openstack-nova01:18
*** betherly has quit IRC01:22
*** mdbooth_ has joined #openstack-nova01:25
*** mdbooth has quit IRC01:28
*** mdbooth_ has quit IRC01:30
*** tetsuro has joined #openstack-nova01:31
*** zhouyao has quit IRC01:36
*** zhouyao has joined #openstack-nova01:43
*** brinzhang_ has joined #openstack-nova01:50
*** brinzhang_ has quit IRC01:51
*** brinzhang_ has joined #openstack-nova01:52
*** brinzhang_ has quit IRC01:52
*** brinzhang_ has joined #openstack-nova01:53
*** brinzhang has quit IRC01:53
*** brinzhang_ has quit IRC01:54
*** brinzhang_ has joined #openstack-nova01:54
*** brinzhang_ has quit IRC01:56
*** brinzhang_ has joined #openstack-nova01:56
*** brinzhang has joined #openstack-nova01:57
*** brinzhang_ has quit IRC02:01
*** yikun has joined #openstack-nova02:04
*** yaawang has joined #openstack-nova02:05
gmannefried: i thought it was merged. This is followup nit comments fix of approved spec - https://review.opendev.org/#/c/669196/102:07
gmannefried: do i need to follow the exception process for this or i can just restore it ? and and ping melwitt johnthetubaguy to review.02:08
openstackgerritMerged openstack/nova master: Add functional regression test for bug 1837955  https://review.opendev.org/67335702:12
openstackbug 1837955 in OpenStack Compute (nova) "MaxRetriesExceeded sometime fails with messaging exception" [Medium,In progress] https://launchpad.net/bugs/1837955 - Assigned to Erik Olof Gunnar Andersson (eandersson)02:12
alex_xuefried: this is an update of existing standrad cpu resource tracking https://review.opendev.org/#/c/666032/02:18
openstackgerritMerged openstack/nova master: Cleanup when hitting MaxRetriesExceeded from no host_available  https://review.opendev.org/67285502:19
*** sridharg has joined #openstack-nova02:23
*** zhouyao has quit IRC02:28
*** sridharg has quit IRC02:50
*** hongbin has joined #openstack-nova02:53
*** epoojad1 has quit IRC02:54
*** tetsuro has quit IRC02:55
*** brinzhang_ has joined #openstack-nova03:01
*** brinzhang has quit IRC03:04
openstackgerritBrin Zhang proposed openstack/nova-specs master: Add flavor group  https://review.opendev.org/66356303:25
*** mdbooth has joined #openstack-nova03:27
*** threestrands has quit IRC03:30
*** threestrands has joined #openstack-nova03:30
*** threestrands has quit IRC03:31
*** hongbin has quit IRC03:34
*** psachin has joined #openstack-nova03:36
*** brinzhang_ has quit IRC03:43
*** jaosorior has quit IRC04:05
*** jmlowe has joined #openstack-nova04:14
*** jmlowe has quit IRC04:19
*** udesale has joined #openstack-nova04:23
*** jmlowe has joined #openstack-nova04:24
*** jmlowe has quit IRC04:30
*** gary_perkins has quit IRC04:32
*** gary_perkins has joined #openstack-nova04:33
*** whoami-rajat has joined #openstack-nova04:40
*** ratailor has joined #openstack-nova04:47
*** belmoreira has joined #openstack-nova04:54
*** jaosorior has joined #openstack-nova05:04
*** Luzi has joined #openstack-nova05:05
*** ricolin__ has joined #openstack-nova05:21
openstackgerritya.wang proposed openstack/os-traits master: Add new traits to flag live migration features  https://review.opendev.org/67340205:23
*** ricolin_ has quit IRC05:24
*** d34dh0r53 has quit IRC05:25
*** huaqiang has joined #openstack-nova05:25
openstackgerritGhanshyam Mann proposed openstack/nova master: Multiple API cleanup changes  https://review.opendev.org/66688905:38
openstackgerritya.wang proposed openstack/os-traits master: Add new traits to flag live migration features  https://review.opendev.org/67340205:48
*** brinzhang has joined #openstack-nova05:58
openstackgerritSundar Nadathur proposed openstack/nova master: ksa auth conf and client for cyborg access  https://review.opendev.org/63124205:59
openstackgerritSundar Nadathur proposed openstack/nova master: WIP: Add Cyborg device profile groups to request spec.  https://review.opendev.org/63124305:59
openstackgerritSundar Nadathur proposed openstack/nova master: WIP: Create and bind Cyborg ARQs.  https://review.opendev.org/63124405:59
openstackgerritSundar Nadathur proposed openstack/nova master: WIP: Get resolved Cyborg ARQs and add PCI BDFs to VM's domain XML.  https://review.opendev.org/63124505:59
*** epoojad2 has joined #openstack-nova06:06
*** belmoreira has quit IRC06:13
*** brinzhang_ has joined #openstack-nova06:16
*** d34dh0r53 has joined #openstack-nova06:17
*** brinzhang has quit IRC06:19
openstackgerritZhengMa proposed openstack/nova master: Check container_format in nova, simply abort instance creation directly from image, if container_format of the image is 'compressed'.  https://review.opendev.org/67340706:24
openstackgerritZhengMa proposed openstack/nova master: Check container_format in nova before instance creation simply abort instance creation directly from image, if container_format of the image is 'compressed'.  https://review.opendev.org/67340706:28
*** maciejjozefczyk has joined #openstack-nova06:30
*** dpawlik has joined #openstack-nova06:33
*** dpawlik has quit IRC06:38
*** dpawlik has joined #openstack-nova06:43
*** slaweq has joined #openstack-nova06:53
*** brinzhang has joined #openstack-nova06:54
*** rpittau|afk is now known as rpittau06:58
*** jaosorior has quit IRC07:01
*** rcernin has quit IRC07:02
*** belmoreira has joined #openstack-nova07:08
*** aojea has joined #openstack-nova07:10
*** tesseract has joined #openstack-nova07:11
*** tssurya has joined #openstack-nova07:15
*** xek has joined #openstack-nova07:20
openstackgerritBrin Zhang proposed openstack/nova master: Add delete_on_termination to volume-attach API  https://review.opendev.org/67313307:31
*** ricolin__ is now known as ricolin07:44
*** brault has joined #openstack-nova07:47
*** ivve has joined #openstack-nova07:49
openstackgerritYongli He proposed openstack/nova master: Add server sub-resource topology API  https://review.opendev.org/62147607:49
*** jangutter has joined #openstack-nova07:55
*** mkrai has joined #openstack-nova07:57
*** ralonsoh has joined #openstack-nova07:58
*** priteau has joined #openstack-nova08:08
*** cdent has joined #openstack-nova08:09
kashyapaspiers: Morning.  On your pastebin, it certainly looks better.08:15
kashyapaspiers: On the absence of 'shim' binaries, yeah, I'm a bit surprised08:15
kashyapaspiers: But you confirm that in `grub` (and `dmesg` - assuming you haven't ran over the ring buffer limit) shows Secure Boot is enabled, that's good enough :-)08:16
*** lpetrut has joined #openstack-nova08:18
bhagyashrisstephenfin: Hi, I just encountered in one issue and I have posted comment on your patch https://review.opendev.org/#/c/671793/5 .08:22
bhagyashrisstephenfin: Just to info you I am currently working on the upgrade patch  https://review.opendev.org/#/c/672224/1 (Fixing review comments given by you) and also testing those change on top of your changes. Once it’s fixed  I push the patche soon.08:22
*** purplerbot has quit IRC08:22
*** purplerbot has joined #openstack-nova08:22
*** derekh has joined #openstack-nova08:27
*** ttsiouts has joined #openstack-nova08:32
*** jaosorior has joined #openstack-nova08:45
*** ivve has quit IRC08:48
*** johnthetubaguy has quit IRC08:51
*** johnthetubaguy has joined #openstack-nova08:53
*** belmoreira has quit IRC08:54
*** belmoreira has joined #openstack-nova08:55
*** belmoreira has quit IRC08:56
*** tkajinam has quit IRC09:03
openstackgerritBrin Zhang proposed openstack/nova master: Add delete_on_termination to volume-attach API  https://review.opendev.org/67313309:05
mdboothsean-k-mooney: I haven't looked closely, but unlikely. The downstream bug requires a 24 hour delay, which you're unlikely to find in the gate.09:05
*** johnthetubaguy has quit IRC09:09
*** johnthetubaguy has joined #openstack-nova09:11
aspierskashyap: yeah, once I figured out that I needed a) a newer kernel to avoid some panics and b) qxl video since virtio doesn't work well with SEV yet, I got the guest fully booted and it works beautifully :)09:12
kashyapaspiers: Sweet.  You're probably the Second Man(tm) on this channel to have booted SB VM and came out unscathed :D09:13
aspierskashyap: I didn't even know it was SB until I saw that on the vnc :)09:13
kashyapHehe09:13
aspierswas only trying for SEV, but I guess q35 enables both09:14
aspiersah well, q35 only needed for *truly* secure boot IIRC from your spec09:14
aspierswhich with hindsight I realise https://review.opendev.org/#/c/673151/ probably accidentally helped quite a lot09:14
aspierseven though it's now abandoned X-D09:14
aspierskashyap: BTW the whole SEV series is blocked on review of that patch ... hint hint no pressure ;-)09:15
kashyapaspiers: It's on the top of my pile.  Just trying to shuffle things with highest prio before I disappear for PTO on 06-Aug (to 23-Aug)09:16
aspiersthanks :)09:16
kashyapaspiers: (And yes, 'q35' is mandatory for SB)09:17
kashyapHehe, and first time I came across the 'bamboo' canonical machine type for PPC!09:17
aspierspretty cool huh09:17
aspiers"g3beige"09:17
*** ivve has joined #openstack-nova09:17
kashyapaspiers: A quick question from line-709 here: https://review.opendev.org/#/c/673151/4/nova/virt/libvirt/host.py09:21
kashyapaspiers: SEV "typically requires" 'q35' or it _definitely_ requires 'q35'?09:22
aspiersthat's explained in the spec09:22
kashyapTypically implies, you can do away without 'q35' as well09:22
kashyapAh, I'll open that.09:22
aspiersq35 is required in order to bind all the virtio devices to the PCIe bridge so that they use virtio 1.0 and not virtio 0.9, since QEMU’s iommu_platform feature is added in virtio 1.0 only.09:22
aspiersThe gurus went back and forth on this point, and seemed confused themselves09:23
aspiersso I didn't want to write "definitely"09:23
aspiersadding a hint for other reviewers09:23
kashyap(No need to add 'definitely', but to my mind, removing 'typically' will clarify: "in all cases that matter you want 'q35' with SEV")09:24
kashyapaspiers: Yeah, I see the virtio 1.0 vs. 0.9 point.09:25
aspiershit reload09:25
kashyapAnd yes, for SEV you do want 1.0-only.09:25
kashyapAh, thanks.09:25
*** mkrai has quit IRC09:33
openstackgerritLee Yarwood proposed openstack/nova master: WIP compute: Lock while rebooting an instance  https://review.opendev.org/67346309:42
*** dpawlik has quit IRC09:45
*** dpawlik has joined #openstack-nova09:51
*** dansmith has quit IRC09:52
*** Nick_A has quit IRC09:54
*** ianw has quit IRC09:55
*** dpawlik has quit IRC09:57
*** dansmith has joined #openstack-nova09:58
*** ianw has joined #openstack-nova09:59
*** betherly has joined #openstack-nova10:01
*** sapd1_x has joined #openstack-nova10:02
*** bhagyashris has quit IRC10:06
*** jackyzhu has joined #openstack-nova10:07
*** jackyzhu has quit IRC10:07
*** boxiang_ has joined #openstack-nova10:08
sean-k-mooneymdbooth: ok we have a persitent proablem that happens ocationally in the gate where we get a disk not found error10:13
sean-k-mooneypersumably because the root disk is deleted for some reason10:13
*** ttsiouts has quit IRC10:14
*** ttsiouts has joined #openstack-nova10:15
*** ociuhandu has joined #openstack-nova10:17
*** ttsiouts has quit IRC10:19
*** tbachman has quit IRC10:21
*** sapd1_x has quit IRC10:22
*** shilpasd has joined #openstack-nova10:22
*** mkrai has joined #openstack-nova10:23
*** aojea has quit IRC10:24
*** ociuhandu_ has joined #openstack-nova10:25
*** ociuhandu has quit IRC10:25
*** epoojad2 has quit IRC10:26
*** avolkov has joined #openstack-nova10:26
kashyapaspiers: Also, for future reference you might want to keep your guest XML + the full QEMU command-line with SB somewhere.10:27
* kashyap bbiab10:27
*** dpawlik has joined #openstack-nova10:29
*** aojea has joined #openstack-nova10:29
*** belmoreira has joined #openstack-nova10:33
*** udesale has quit IRC10:38
aspierskashyap: responding10:42
aspierskashyap: shall I add the SB bp?10:42
aspiersto the commit message10:42
*** tbachman has joined #openstack-nova10:43
*** boxiang_ has quit IRC10:47
*** boxiang_ has joined #openstack-nova10:47
openstackgerritAdam Spiers proposed openstack/nova-specs master: Fix URL to blueprint for Secure Boot  https://review.opendev.org/67347710:48
*** belmoreira has quit IRC10:48
aspiersI'm gonna assume the answer to that is yes :)10:50
openstackgerritAdam Spiers proposed openstack/nova master: Track libvirt host/domain capabilities for multiple machine types  https://review.opendev.org/67315110:52
*** aojea has quit IRC10:52
openstackgerritHuachang Wang proposed openstack/nova master: Add the support of CPU feature 'AVX512-VNNI'  https://review.opendev.org/67348010:52
aspierskashyap: https://review.opendev.org/#/c/673151/4..5//COMMIT_MSG (view the other files too)10:53
openstackgerritLee Yarwood proposed openstack/nova master: WIP compute: Take an instance.uuid lock when rebooting  https://review.opendev.org/67346310:54
*** mkrai has quit IRC10:54
*** aojea has joined #openstack-nova10:55
kashyapaspiers: Hi, you could, as the SB series will most likely be using bits from there anyway.10:59
kashyap(Ah, you did; just saw)10:59
*** bhagyashris has joined #openstack-nova11:00
*** belmoreira has joined #openstack-nova11:01
lyarwoodmdbooth: https://review.opendev.org/#/c/673463/ - thoughts on testing this? I'm stuck between func, unit or something else tbh.11:01
* lyarwood -> lunch11:01
*** belmoreira has quit IRC11:02
*** ttsiouts has joined #openstack-nova11:04
openstackgerritMerged openstack/nova master: doc: fix links for server actions in api guide  https://review.opendev.org/67310111:08
openstackgerritMerged openstack/nova master: Fix wrong huge pages in doc  https://review.opendev.org/67325211:09
*** belmoreira has joined #openstack-nova11:09
sean-k-mooneyi think we made the sufixs case insensitive by the way but i do prefer uppercase so i guess im ok with ^11:13
*** brinzhang_ has quit IRC11:15
*** brinzhang_ has joined #openstack-nova11:16
*** brtknr has quit IRC11:19
*** brault has quit IRC11:20
huaqiangls11:24
sean-k-mooneyhuaqiang: password: :)11:26
*** ociuhandu_ has quit IRC11:34
*** belmoreira has quit IRC11:35
*** ociuhandu has joined #openstack-nova11:35
*** belmoreira has joined #openstack-nova11:36
*** dpawlik has quit IRC11:52
openstackgerritBrin Zhang proposed openstack/python-novaclient master: Microversion 2.76: Add delete_on_termination to volume-attach API  https://review.opendev.org/67348511:55
*** mkrai has joined #openstack-nova12:07
*** jaosorior has quit IRC12:07
*** ociuhandu has quit IRC12:14
*** dpawlik has joined #openstack-nova12:16
openstackgerritsean mooney proposed openstack/nova master: Deprecate Aggregate[Core|Ram|Disk]Filters  https://review.opendev.org/67349612:21
sean-k-mooneyi just posted to the mailing list about ^ http://lists.openstack.org/pipermail/openstack-discuss/2019-July/008118.html12:22
*** eharney has joined #openstack-nova12:23
*** psachin has quit IRC12:25
*** maciejjozefczyk has quit IRC12:25
*** maciejjozefczyk has joined #openstack-nova12:27
yaawangefried: Can you take a look at this patch(https://review.opendev.org/#/c/673402/) if you have any time?12:29
yaawangefried: This is a part of the implementation of spec expose-auto-converge-post-copy, which is in os-traits project, and the implementation/testing in nova are depend on this patch.12:29
openstackgerritMerged openstack/nova master: Stop sending bad values from libosinfo to libvirt  https://review.opendev.org/66919012:31
*** ricolin has quit IRC12:35
*** bhagyashris has quit IRC12:42
*** bbowen has quit IRC12:50
*** ociuhandu has joined #openstack-nova12:50
*** belmoreira has quit IRC12:51
*** jaosorior has joined #openstack-nova12:54
*** mkrai has quit IRC12:55
*** mkrai_ has joined #openstack-nova12:55
*** belmoreira has joined #openstack-nova12:55
mdboothlyarwood: Testing races is really hard. I've written comprehensive race tests before, but the problem is that they're generally hard to understand so I just get moaned at.12:59
mdboothlyarwood: Fudge it and don't test, or write a test that looks like it's useful, but is simple enough that it isn't.13:00
*** mkrai_ has quit IRC13:07
*** boxiang_ has quit IRC13:10
*** boxiang_ has joined #openstack-nova13:10
* cdent hugs mdbooth 13:11
*** mriedem has joined #openstack-nova13:13
mriedemeandersson: were you going to work on backports for these or should i? https://review.opendev.org/#/q/topic:bug/1837955+(status:open+OR+status:merged)13:14
*** ttsiouts has quit IRC13:14
*** ttsiouts has joined #openstack-nova13:15
openstackgerritHuachang Wang proposed openstack/nova master: doc: correct the information of 'cpu_map'  https://review.opendev.org/67327213:18
*** BjoernT has joined #openstack-nova13:19
*** ttsiouts has quit IRC13:19
efriedyaawang: +2, but we need a non-Intel to +A. cdent is looking.13:21
efriedbrinzhang: I'm here now13:22
efriedgmann: Sorry, my search picked up some things it shouldn't have accidentally. Restored.13:22
efriedalex_xu: Yup, sorry about that.13:24
*** mkrai_ has joined #openstack-nova13:24
artomefried, btw, is force-abandoning specs past freeze a new thing? IIRC it would be a procedural -2 in the past?13:28
efriedartom: There's not a formal/written/official process afaik.13:28
artomI don't really care one way or another, but was surprised by the emails I got last night13:28
efriedI decided to abandon because then the author can restore rather than me having to take action.13:28
artomefried, as, as you were ;)13:29
artom*ack13:29
efriedJust trying to keep the repo clean-ish.13:29
*** bbowen has joined #openstack-nova13:30
*** belmoreira has quit IRC13:31
*** belmoreira has joined #openstack-nova13:31
*** BjoernT_ has joined #openstack-nova13:34
*** BjoernT has quit IRC13:35
*** ratailor has quit IRC13:36
openstackgerritMerged openstack/os-traits master: Add new traits to flag live migration features  https://review.opendev.org/67340213:37
mriedemmelwitt: how is https://review.opendev.org/#/c/582407/ different from https://review.opendev.org/#/c/562041/ ?13:38
mdboothlyarwood: Commented on https://review.opendev.org/#/c/673463/213:40
openstackgerritEric Fried proposed openstack/os-traits master: Followup: spec ref for auto-converge/post-copy  https://review.opendev.org/67352313:41
lyarwoodmdbooth: thanks13:44
*** ttsiouts has joined #openstack-nova13:45
openstackgerritMatt Riedemann proposed openstack/nova master: Add functional recreate test for bug 1764556  https://review.opendev.org/56204113:45
openstackbug 1764556 in OpenStack Compute (nova) ""nova list" fails with exception.ServiceNotFound if service is deleted and has no UUID" [Medium,In progress] https://launchpad.net/bugs/1764556 - Assigned to melanie witt (melwitt)13:45
openstackgerritMatt Riedemann proposed openstack/nova master: Add functional regression test for bug 1778305  https://review.opendev.org/58240713:45
openstackgerritMatt Riedemann proposed openstack/nova master: Don't generate service UUID for deleted services  https://review.opendev.org/58240813:45
openstackbug 1778305 in OpenStack Compute (nova) "Nova may erronously look up service version of a deleted service, when hostname have been reused" [Medium,In progress] https://launchpad.net/bugs/1778305 - Assigned to melanie witt (melwitt)13:45
*** mkrai_ has quit IRC13:51
*** yikun has quit IRC13:52
*** mlavalle has joined #openstack-nova13:53
*** factor has joined #openstack-nova13:54
yaawangefried: cdent Thanks, I saw your patch :)13:54
*** Luzi has quit IRC13:55
*** mkrai_ has joined #openstack-nova13:56
yaawangefried: cdent Will you release a new versoin of os-traits?13:56
efriedyaawang: I already updated the hash for the proposed 0.16.0 https://review.opendev.org/67329413:57
efriedyaawang: feel free to +1 that patch. Once cdent re-acks it, the release team should push it by the end of the week.13:57
efried...which he has now done13:58
yaawangGreat, I'll push the nova code after os-traits released, so it'll easy to pass the test :)14:00
openstackgerritMerged openstack/os-traits master: Followup: spec ref for auto-converge/post-copy  https://review.opendev.org/67352314:01
*** rpittau is now known as rpittau|afk14:02
efriedyaawang: You'll need to wait until upper-constraints is bumped. A bot will propose a patch to the openstack/requirements project after the release patch merges.14:04
efriedyaawang: Then in your patch, you can update the requirements.txt and lower-constraints.txt to 0.16.014:05
*** tbachman has quit IRC14:07
*** tbachman has joined #openstack-nova14:09
yaawangGet you point :)14:09
*** brtknr has joined #openstack-nova14:10
*** BjoernT_ has quit IRC14:11
*** BjoernT has joined #openstack-nova14:11
*** BjoernT_ has joined #openstack-nova14:15
openstackgerritMatt Riedemann proposed openstack/nova stable/stein: Add functional regression test for bug 1837955  https://review.opendev.org/67353214:16
openstackbug 1837955 in OpenStack Compute (nova) stein "MaxRetriesExceeded sometime fails with messaging exception" [Medium,Confirmed] https://launchpad.net/bugs/183795514:16
openstackgerritMatt Riedemann proposed openstack/nova stable/stein: Cleanup when hitting MaxRetriesExceeded from no host_available  https://review.opendev.org/67353314:16
*** BjoernT has quit IRC14:16
*** mlavalle has quit IRC14:17
*** mlavalle has joined #openstack-nova14:19
openstackgerritMatt Riedemann proposed openstack/nova stable/rocky: Add functional regression test for bug 1837955  https://review.opendev.org/67353614:25
openstackbug 1837955 in OpenStack Compute (nova) stein "MaxRetriesExceeded sometime fails with messaging exception" [Medium,In progress] https://launchpad.net/bugs/1837955 - Assigned to Matt Riedemann (mriedem)14:25
sean-k-mooneymriedem: thanks for reviewing the deprecateion patch. i want to get stephenfin to take a look as i know there is a way i can pull out the repeated content and link it in once. ill look at the other comments you left and adress them after i grab lunch14:26
mriedemeandersson: i've started backports. just fyi the backport of your fix gets tricky in rocky b/c the _cleanup_when_reschedule_fails method doesn't exist in rocky14:26
mriedemsean-k-mooney: you inline it with a doc reference i think, not sure that's really worth it though14:27
*** BjoernT_ has quit IRC14:27
sean-k-mooneyok well i just wanted to get a basic version up to link in the mailing list post14:27
sean-k-mooneyim happy to simplfy14:27
sean-k-mooneyi more or less based it off your deprecation patch for the non aggreate versions but i grant you its proably too verbose as it is14:28
sean-k-mooneyanyway ill be back in 30 mins or so14:28
*** mkrai_ has quit IRC14:30
*** dpawlik has quit IRC14:34
*** BjoernT_ has joined #openstack-nova14:34
*** factor has quit IRC14:39
*** factor has joined #openstack-nova14:39
*** tbachman has quit IRC14:39
*** icarusfactor has joined #openstack-nova14:43
*** belmoreira has quit IRC14:45
openstackgerritMatt Riedemann proposed openstack/nova stable/rocky: Reproduce bug #1819460 in functional test  https://review.opendev.org/67354614:45
openstackbug 1819460 in OpenStack Compute (nova) rocky "instance stuck in BUILD state due to unhandled exceptions in conductor" [Undecided,New] https://launchpad.net/bugs/181946014:45
*** factor has quit IRC14:46
*** Nick_A has joined #openstack-nova14:48
*** mlavalle has quit IRC14:50
*** icarusfactor has quit IRC14:50
*** belmoreira has joined #openstack-nova14:50
*** icarusfactor has joined #openstack-nova14:50
*** belmoreira has quit IRC14:51
*** mlavalle has joined #openstack-nova14:51
*** dklyle has quit IRC14:52
*** dklyle has joined #openstack-nova14:53
*** mkrai_ has joined #openstack-nova14:53
*** takamatsu has joined #openstack-nova14:54
*** icarusfactor has quit IRC14:55
*** factor has joined #openstack-nova14:56
openstackgerritMatt Riedemann proposed openstack/nova stable/rocky: Handle placement error during re-schedule  https://review.opendev.org/67355014:57
*** belmoreira has joined #openstack-nova14:59
melwittmriedem: they are similar but one condition causes ServiceTooOld to be raised and the other causes ServiceNotFound to be raised. have to look some more to see exactly what condition results in the different behaviors15:03
openstackgerritMatt Riedemann proposed openstack/nova stable/rocky: Cleanup when hitting MaxRetriesExceeded from no host_available  https://review.opendev.org/67355315:05
mriedemmelwitt: was it the service version?15:05
mriedemthat was about the only thing that stood out15:05
mriedemservicetooold would be the version related thing15:05
mriedemeandersson: ^ there is the rocky backport15:06
melwittmriedem: yeah, when a failure occurs at this code https://github.com/openstack/nova/blob/16.1.4/nova/objects/service.py#L36215:08
melwittand it looks like the other bug, the ServiceNotFound is raised from the db.service_update call, after the service version check passed15:09
*** tbachman has joined #openstack-nova15:10
mriedemyeah ok15:10
mriedemi didn't make the too old <> not found connection15:11
*** belmoreira has quit IRC15:12
*** hamzy has quit IRC15:12
*** ricolin has joined #openstack-nova15:14
mriedembauzas: https://review.opendev.org/#/c/562041/ needed to be rebased and remove the set_nodes/restore_nodes stuff that's been removed on master, otherwise it's the same - can i get a fast re-approve there?15:16
mriedemthe rest of the series above it is approved15:16
*** belmoreira has joined #openstack-nova15:27
*** mriedem has quit IRC15:33
openstackgerritMatt Riedemann proposed openstack/nova stable/rocky: Cleanup when hitting MaxRetriesExceeded from no host_available  https://review.opendev.org/67355315:35
*** mriedem has joined #openstack-nova15:35
*** mkrai_ has quit IRC15:39
bauzasmriedem: sure15:39
*** whoami-rajat has quit IRC15:39
bauzasdone15:39
*** belmoreira has quit IRC15:42
*** ttsiouts has quit IRC15:42
*** ttsiouts has joined #openstack-nova15:42
*** tssurya has quit IRC15:46
*** ttsiouts has quit IRC15:47
openstackgerritMatt Riedemann proposed openstack/nova stable/queens: Add functional regression test for bug 1837955  https://review.opendev.org/67356715:48
openstackbug 1837955 in OpenStack Compute (nova) stein "MaxRetriesExceeded sometime fails with messaging exception" [Medium,In progress] https://launchpad.net/bugs/1837955 - Assigned to Matt Riedemann (mriedem)15:48
*** aojea has quit IRC15:50
*** gyee has joined #openstack-nova15:52
*** aojea has joined #openstack-nova15:53
*** shilpasd has quit IRC15:56
openstackgerritSylvain Bauza proposed openstack/nova master: WIP: Add a placement audit command  https://review.opendev.org/67011216:03
bauzasmriedem: quick shot on a total rework of the nova-manage command16:05
bauzasmriedem: I'm rushing to write the func tests before I enter the PTO season16:05
*** aojea has quit IRC16:07
*** whoami-rajat has joined #openstack-nova16:07
*** panda has quit IRC16:09
*** lpetrut has quit IRC16:11
*** panda has joined #openstack-nova16:13
*** Sundar has joined #openstack-nova16:17
openstackgerritMatt Riedemann proposed openstack/nova stable/rocky: Cleanup when hitting MaxRetriesExceeded from no host_available  https://review.opendev.org/67355316:17
*** yonglihe has quit IRC16:22
*** arne_wiebalck has quit IRC16:22
*** yonglihe has joined #openstack-nova16:26
*** arne_wiebalck has joined #openstack-nova16:28
*** rajinir has quit IRC16:28
openstackgerritMatt Riedemann proposed openstack/nova stable/queens: Cleanup when hitting MaxRetriesExceeded from no host_available  https://review.opendev.org/67357616:29
*** rajinir has joined #openstack-nova16:30
*** factor has quit IRC16:31
*** factor has joined #openstack-nova16:31
sean-k-mooneyefried: by the way i would be totally fine with making the "cpu priortisiation" use case a docs only feature. e.g. in the install guide or admin docs have a topic doc that explains how to use the provider.yaml to add the custom trati and host to configure the cpu_shared_set, cpu_dedicated_set to map to the high and low priority cores defined in your speed select policy16:31
sean-k-mooneyefried: rather then being totally downstream only16:32
*** trident has quit IRC16:32
*** igordc has joined #openstack-nova16:33
sean-k-mooneywith the cpu resocuse tracking in placmenet spec and provider.yaml everything  is there to allow you to staticly configure this without addtional code chagnes16:33
sean-k-mooneywe likely will end up looking to support speed select and cache via that mechcanium with train eventuraly for OSP so if we collaberated on upstream does it would both sever as a concreate example of how to use the provider.yaml and de duplicate the work16:35
*** rajinir has quit IRC16:36
*** trident has joined #openstack-nova16:37
*** rajinir has joined #openstack-nova16:38
stephenfindansmith: I need to determine if the NUMATopology o.vo I'm looking at was created on a pre-Train compute, because some fields I'm adding won't be present. Can I check the version of that object easily?16:39
artom    b'reference = NUMAMigrateData(cell_pins={2=set([6,7]),3=set([8,9])},cpu_pins={0=set([0,1]),1=set([2,3])},emulator_pins=set([4,5]),sched_priority=8,sched_vcpus=set([6,7]))'16:39
artom    b'actual    = NUMAMigrateData(cell_pins={2=set([6,7]),3=set([8,9])},cpu_pins={0=set([0,1]),1=set([2,3])},emulator_pins=set([4,5]),sched_priority=8,sched_vcpus=set([6,7]))'16:39
artomUmm16:39
sean-k-mooneyartom: you are forgeting about the reset changed thing16:40
sean-k-mooneywhen your are comparing two ovo you should convert them to primatives and compare those16:40
artomMaybe. But I still have the right to be damn angry about it.16:41
sean-k-mooneyothersize even if they have the same data they will be considered different of one has a files change state different then the other object16:41
sean-k-mooneyartom: you have hit this in the past no?16:41
artomBut... thanks sean-k-mooney :)16:41
sean-k-mooneywe migth have a unitlity funciton for this too16:41
artomsean-k-mooney, don't think so, the past was list ordering and stuch16:41
artom*such16:41
sean-k-mooneyah ok16:42
sean-k-mooneyi have hit that in the past so it burn into my memory16:42
artomMy memory is apparently resistant to burning :(16:42
stephenfindansmith: In case it matters, the object is saved in serialized form in the 'ComputeNode.numa_topology' (StringField) field and I'm having to deserialize it using this https://github.com/openstack/nova/blob/master/nova/objects/numa.py#L243-L25016:42
sean-k-mooneystephenfin: you can check it in two ways one the objec thousl have a version embeded or two you can just check if the field is present16:43
stephenfinsean-k-mooney: Is it embedded? I'm looking at the 'VersionedObject' base class and I can't see anything obvious16:45
stephenfinthere's VERSION but that's a class variable, not an instance variable16:45
sean-k-mooney its in the primitave dict16:45
efriedsean-k-mooney: what you said16:45
sean-k-mooneyhttps://github.com/openstack/oslo.versionedobjects/blob/master/oslo_versionedobjects/base.py#L40816:45
stephenfinThat's only present in the serialized form though, right?16:46
*** jmlowe has joined #openstack-nova16:46
sean-k-mooneystephenfin: that is what we retire form the db no?16:46
*** ociuhandu has quit IRC16:46
stephenfinWhich I guess I can use since I'm deserializing things as noted above16:47
*** ociuhandu has joined #openstack-nova16:47
*** yonglihe_ has joined #openstack-nova16:47
stephenfinIt is, but it feels a bit wrong to be modifying that 'obj_from_db_obj' method to do this stuff16:47
*** yonglihe has quit IRC16:47
*** yonglihe_ is now known as yonglihe16:47
stephenfinI'd much rather have done it where I'm actually using it, but I guess beggars can't be choosers, heh16:47
sean-k-mooneywhat are you trying to do16:48
stephenfinsec16:48
stephenfinsean-k-mooney: I need to work around this https://review.opendev.org/#/c/671800/8/nova/virt/hardware.py@191216:49
stephenfinback in 2 mins16:49
sean-k-mooneywe defien a cpu_usage property on teh ovo that know how to retrun teh correct value16:50
sean-k-mooneyfor the old object cpu_usage would be in the object field for the new value it would be the sum of shared_cpu_usage+dedicated_cpu_usage16:52
*** mkrai_ has joined #openstack-nova16:52
stephenfinYup, correct. I don't think we actually need to worry about supporting stuff reading old NUMATopology objects though16:52
sean-k-mooneya property can just use "in" to determin if the field is set16:52
stephenfinI was trying to figure out what actually sets NUMAToplogy.cpu_usage16:53
stephenfinI figured it would be driver, but it seems it's not16:53
sean-k-mooneyits set by the resouce tracker16:53
stephenfinInstead, the driver (libvirt) returns the object with those fields set to 016:53
stephenfinYeah16:53
stephenfinwhich seems a little dumb, tbh, when the compute node could be doing this for us16:53
sean-k-mooneyits update by a perodic task16:53
sean-k-mooneywell the resouce tracker runs on the compute node16:54
stephenfinSorry, the driver16:54
sean-k-mooneyand this code is ment to be virt driver independent16:54
sean-k-mooneyits update by https://github.com/openstack/nova/blob/master/nova/compute/resource_tracker.py#L67616:55
stephenfinI mean, why can't we the driver populate this field when it creates it https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L661416:55
stephenfinYeah, via a call to '_update_available_resource'16:55
stephenfinwhich calls '_update_usage_from_instances' (plural), which calls '_update_usage_from_instance' (singular) multiple times for each instance16:56
sean-k-mooneywell we start by calling https://github.com/openstack/nova/blob/master/nova/compute/resource_tracker.py#L69516:56
sean-k-mooney resources = self.driver.get_available_resource(nodename)16:56
sean-k-mooneywhich gets the resouce form the driver16:57
sean-k-mooney_update_usage_from_instances is to take care of cleaning up after/during migrations16:57
sean-k-mooneyat least i thinkt that is what that does16:58
stephenfinself.driver.get_available_resource returns the NUMATopology object with the usage fields set to 016:58
stephenfinand the '_update_usage_from_instance' is what actually sets those16:59
stephenfinwhich isn't intuitive at all16:59
stephenfinAt least, to me16:59
openstackgerritMatt Riedemann proposed openstack/nova master: Remove Request Spec Migration upgrade status check  https://review.opendev.org/67358017:01
*** derekh has quit IRC17:01
sean-k-mooneywe proably could optimise it so the first call calulated everythign but i think this is called in a perodic task17:02
sean-k-mooneyand im not sure if just the inner fucntion is called  _update_available_resource17:02
sean-k-mooneyor if update_available_resource is called17:02
*** BjoernT_ has quit IRC17:03
*** markvoelker has quit IRC17:03
openstackgerritMerged openstack/nova-specs master: Fix URL to blueprint for Secure Boot  https://review.opendev.org/67347717:04
*** mkrai_ has quit IRC17:05
*** cdent has quit IRC17:05
*** ociuhandu has quit IRC17:07
sean-k-mooneyefried: by the way "what you said" were you agreeing with adding a topic doc for speed select using provider.yaml ectra17:08
sean-k-mooneyor was that in relation to something else17:08
efriedsean-k-mooney: I was basically +1ing the concept that the way we're looking at mixed shared+dedicated and providers.yaml is, at least from Intel's perspective, all about enabling those use cases in a way that keeps nova from having to know too much.17:09
efriedsean-k-mooney: I would be fully supportive of the idea of having an in-tree doc that described the CPU priority setup as an *example* of how you could piece together providers.yaml etc.17:10
efriedBut I wouldn't think we should have it as a suggested, supported, etc. thing.17:10
sean-k-mooneyah right, well i could keep my pm happy by wrighign some doc to show how to support those usecase with trian17:10
*** ricolin has quit IRC17:11
sean-k-mooneydo you know if there has been any progress on making RMD listent to libvirt events and read the flaovr info17:12
sean-k-mooneywe could also enable cache and memory bandwith in a similar way if RMD has been updateded as we spoke about at the ptg17:13
efriedsean-k-mooney: They're working on it, but I don't think there's anything to show yet.17:14
efriedFYI I think they're planning to hack the domain xml file....17:14
sean-k-mooneyso it will never be supported got it17:15
sean-k-mooneywhy?17:15
sean-k-mooneythe instance and flavor uuid are in the xml and they can do the enforcement without needing to modify the xml17:16
efriedsean-k-mooney: I thought they needed to update the xml to assign cacheways to the VM?17:16
sean-k-mooneyno they can do it out of band via the resouce control file system17:17
efriedokay, neat.17:17
sean-k-mooneythey basically have two options they can tell the kernel to apploy the policy to a set of host cores. e.g. the ones the vm is pinned too or they can use the process id17:18
sean-k-mooneyat least for cache17:18
*** jmlowe has quit IRC17:18
*** ralonsoh has quit IRC17:23
stephenfinsean-k-mooney: Should we prevent one thread sibling being used for VCPUs and another for PCPUs?17:28
stephenfinI think yes, but I'd like a sanity check :)17:29
stephenfins/and another/and the other/17:29
stephenfinOr others, for something like POWER17:29
sean-k-mooneywe could17:29
sean-k-mooneywe dont have too17:29
sean-k-mooneydedicated cpus does not imply isolate17:30
stephenfinNope, but the isolate policy does17:30
sean-k-mooneywe could issue a warning17:30
sean-k-mooneynot any more17:30
sean-k-mooneyisolate means find a host with SMT disabled17:30
stephenfinTrue17:30
sean-k-mooneyso it wont be an issue17:30
sean-k-mooneywe shoudl also document how to add/remove the hyperthreind trait using provider.yaml17:31
stephenfinTrue. I guess I need to decide what I'm doing with '(un)pin_cpus_with_siblings' on that front17:31
stephenfinYeah, I haven't actually given that part of the spec much thought yet17:31
sean-k-mooneyactully we cant do that with provider.ymal17:32
*** brault has joined #openstack-nova17:32
*** brault has quit IRC17:32
sean-k-mooneybecause its a standard trait17:32
efriednot because it's a standard trait17:32
efriedbecause we're not supporting trait removal in this pass17:32
efriedremoving a standard trait is totally something you will eventually be able to do with providers.yaml.17:33
stephenfinI was going to automatically report it via the libvirt driver17:33
stephenfinCan we not do that?17:33
sean-k-mooneyefried: the virt diver does not own the hypertreading trait17:33
sean-k-mooneyso its not allowed to report it17:33
sean-k-mooneyper the cpu resouce trackign spec17:33
stephenfinsean-k-mooney: Can we make it own it?17:33
efriedwho owns it?17:33
sean-k-mooneythe operator17:33
stephenfinI forget the context on this, tbh17:33
efriedokay, if the operator owns it, they can use providers.yaml to have it added to the compute.17:33
efriedor they can just ... add it.17:33
efriedstandard/custom isn't the dividing line for what you can do out of band17:34
sean-k-mooneyit has its own section https://specs.openstack.org/openstack/nova-specs/specs/train/approved/cpu-resources.html#add-hw-cpu-hyperthreading-trait17:34
*** tesseract has quit IRC17:34
efriedIf we haven't declared this trait to be in the set of ones compute "owns" then the operator can do whatever they want with it.17:35
sean-k-mooneyyes17:35
stephenfinefried: We can still report it though, right?17:35
stephenfinWe just don't override it17:35
stephenfinNot sure that makes sense but I don't really know how virt-reported traits work yet17:35
sean-k-mooneystephenfin: since we dont have a way to remove direver reported tratis we should not unless we have a config option to disable it17:35
sean-k-mooneystephenfin: the reason that this exits is two fold. one peole wanted to be able to leave hypertreaind enabled on the host for the shared cpus and not list the hyper tread in the dedicated set17:37
sean-k-mooneythe other reason is with numa peopl wanted to have teh flexablity to decler the trait diffently per socket/numa node17:38
sean-k-mooneyit was motivated by small edge deployment of 1-5 nodes17:39
sean-k-mooneywhere its not resonable to dedicate entire host to one confiuration or the other17:39
sean-k-mooneysince we can nolnger do this per instance17:39
sean-k-mooneyvia cpu tread polices17:39
stephenfinHow about we report the trait but only if all hyperthreads are enabled/disabled17:39
stephenfini.e. there's either no cpu set configuration17:39
stephenfinor there is, and hyper threads aren't listed in either option17:40
stephenfinor the host doesn't have hyperthreads17:40
sean-k-mooneyall hypertreads? e.g. you whitelist both thread siblings17:40
sean-k-mooneyi think that overly complex to debug17:40
stephenfinHmm, that's true17:41
sean-k-mooneyi think a config to enable auto reporting or just making it work with provider.yaml is the simplest17:41
stephenfinYeah, config option wfm17:41
stephenfin*disable* auto-reporting though17:42
sean-k-mooneysure17:42
stephenfinI'd rather cater to the 90% than the 10%17:42
sean-k-mooneybut that should be in the driver section17:42
stephenfinYup17:42
sean-k-mooneyand ya im fine with disableing it if you know what your doing and want to specfy something custum17:43
sean-k-mooneystephenfin: you realise however that if its auto enabeld that isolated vms will not be moveable if you have hypertreading on everything17:43
sean-k-mooneyor bootable17:43
sean-k-mooneyi guess the same is true of require17:44
stephenfinYup17:44
stephenfinJust go toggle the config option and you're sorted17:44
efriedjust note that if virt reports it, you can't disable it.17:44
stephenfinefried: wdym?17:44
sean-k-mooneyefried: you can if there is a config option in the libvirt section17:44
efriedI mean, you could, like with a conf option or something, but that would be a new and different way of doing things.17:44
efried^17:44
sean-k-mooneyright but nothing reports this trait currently17:45
efriedSorry, let me backpedal a bit17:45
sean-k-mooneywe standardised it for this spec and have not implemented using it yet17:45
efriedI think it would be fine if, within the actual libvirt driver (in update_provider_tree), there was a CONF-based conditional which, if true (and other conditions are met), would report the hyperthreading trait on the compute node RP.17:46
efriedBut it should not be one of the automatic traits controlled by the RT outside of the virt driver.17:46
sean-k-mooneyright17:47
sean-k-mooneyi agree with that17:47
efriedclear as mud stephenfin?17:47
stephenfinWhat's the difference between virt-driver reported traits and resource tracker-provided traits?17:48
sean-k-mooneystephenfin: he is saying don do it here https://github.com/openstack/nova/blob/master/nova/compute/resource_tracker.py#L1001-L100317:48
sean-k-mooneywhich but do it in the driver update provider tree17:49
efriedexactly17:49
efriedit sucks how messy this has gotten17:50
stephenfinOh, so do it here instead? https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L6797-L680517:50
stephenfinGot it, if so17:50
efriedYes, more or less.17:50
sean-k-mooneyyes17:50
sean-k-mooneylike im doing here https://review.opendev.org/#/c/666914/5/nova/virt/libvirt/driver.py17:50
*** francoisp has quit IRC17:50
sean-k-mooneyfor my storage bus traits17:50
efriedNot sure I would do it from within _get_cpu_traits, but that's the right neighborhood.17:51
sean-k-mooneyi would not17:51
sean-k-mooneyadd a seperate function17:51
efriedI guess it's a CPU trait...17:51
efriedanyway, we're on the same page.17:51
sean-k-mooneyim adding 3 more traits functions to the libvirt driver17:52
stephenfinYeah, but that general area17:52
sean-k-mooneytraits.update(self._get_storage_bus_traits())17:52
stephenfinCool, I'll do just that17:52
sean-k-mooneyand just calling them in sequen to build up the dict of tratis17:52
sean-k-mooneyit makes the code simpler17:53
efriedDoesn't bother me to expand trait logic in a given upt. What's bugging me is the growing number of different sources-of-truth for traits, and trying to keep track of their precedence.17:53
efriedvirt driver, RT, direct placement, and soon providers.yaml (which will funnel through the RT)17:54
sean-k-mooneystephenfin: the follow up patch is a better example https://review.opendev.org/#/c/666915/6/nova/virt/libvirt/driver.py@682017:54
sean-k-mooneyefried: does direct placmeent actully work?17:54
sean-k-mooneyor do we overwrite them still17:54
efriedwe're not supposed to17:55
efriedunless you muck with ones the compute supposedly "owns".17:55
efriedso like, you can't switch a CPU capability trait on or off17:55
sean-k-mooneythat is why we pass the bool for if the trait should be present in the dict of tratis right17:55
efriedor like MULTI_ATTACH17:55
efriedpartly, yeah - the bool is so we can get both sides of the "I own it" equation. Otherwise we would only be able to switch on ones that were off.17:56
sean-k-mooneyefried: we could remove one of the soucrces of truth if we wanted too17:56
stephenfinsean-k-mooney++ efried++17:56
sean-k-mooneye.g. we could move the traits that are set in the RT into the driver17:56
stephenfinand with that, I'm off home o/17:56
efriedo/ stephenfin17:56
sean-k-mooneyby impleemnting the common code in a function that all driver call17:56
sean-k-mooneyon the other hand i dont realy want to change that right now17:57
*** Sundar has quit IRC17:59
sean-k-mooneyefried: by the way are you implementing provider.yaml or is that being done by someone else17:59
openstackgerritAndreas Jaeger proposed openstack/os-resource-classes master: Update api-ref location  https://review.opendev.org/67360417:59
sean-k-mooneyit looks like there are two implementations https://review.opendev.org/#/q/topic:bp/provider-config-file+(status:open+OR+status:merged)18:00
efriedsean-k-mooney: dustinc is doing it.18:00
efried...based on some stuff I did last year, yeah.18:00
sean-k-mooneyah ok18:01
dustincsean-k-mooney: yeah I am working on it as we chat18:01
dustincthe newer one18:01
sean-k-mooneyhttps://review.opendev.org/#/c/673341/118:01
sean-k-mooneyis the other one still relevent18:01
sean-k-mooneyor can it be abandoned18:01
sean-k-mooneyhttps://review.opendev.org/#/c/622622/118:01
dustincdefer to efried, but I imagine it can be abandoned18:02
dustincthere is more to that one than what I am doing, dunno if it will be relevant for something else18:02
sean-k-mooneydustinc: well it will live in gerrit for ever.18:02
*** priteau has quit IRC18:02
efriedI can abandon it. Can always restore later (or more likely, steal stuff from it)18:02
efriedyeah18:02
sean-k-mooneyill star https://review.opendev.org/#/c/673341/1 so i know to review that one instead18:03
dustincthanks!18:03
efriedabandoned the other.18:03
sean-k-mooneyon of the things  i do like about gerrit is 1 the history but more so if you use topic properly its easy to find stuff that is reslated18:04
sean-k-mooney*related18:04
*** hamzy has joined #openstack-nova18:11
aspierssean-k-mooney: agreed. I was thinking earlier that it should be possible to make a human-readable Change-Id18:22
aspiersthen more useful metadata could be written to express logical dependencies between commits18:23
*** BjoernT has joined #openstack-nova18:24
*** BjoernT has quit IRC18:28
*** BjoernT has joined #openstack-nova18:32
*** BjoernT has quit IRC18:35
*** BjoernT_ has joined #openstack-nova18:35
*** bbowen has quit IRC18:36
*** BjoernT_ has quit IRC18:40
openstackgerritMerged openstack/os-resource-classes master: Update api-ref location  https://review.opendev.org/67360418:46
openstackgerritmelanie witt proposed openstack/nova master: Add functional recreate test for bug 1764556  https://review.opendev.org/56204118:49
openstackbug 1764556 in OpenStack Compute (nova) ""nova list" fails with exception.ServiceNotFound if service is deleted and has no UUID" [Medium,In progress] https://launchpad.net/bugs/1764556 - Assigned to melanie witt (melwitt)18:49
openstackgerritmelanie witt proposed openstack/nova master: Add functional regression test for bug 1778305  https://review.opendev.org/58240718:49
openstackgerritmelanie witt proposed openstack/nova master: Don't generate service UUID for deleted services  https://review.opendev.org/58240818:49
openstackbug 1778305 in OpenStack Compute (nova) "Nova may erronously look up service version of a deleted service, when hostname have been reused" [Medium,In progress] https://launchpad.net/bugs/1778305 - Assigned to melanie witt (melwitt)18:49
openstackgerritMerged openstack/nova master: Run 'tempest-ipv6-only' job in gate  https://review.opendev.org/67190018:49
openstackgerritEric Fried proposed openstack/nova master: DNM: Where have all the global_request_id gone?  https://review.opendev.org/67298618:51
mriedemjust debugged a unit test for nearly an hour and it was so dumb, my brain must be mush19:11
*** eharney has quit IRC19:15
*** mriedem has quit IRC19:20
*** BjoernT has joined #openstack-nova19:21
*** tbachman has quit IRC19:22
*** amodi has quit IRC19:25
*** tbachman has joined #openstack-nova19:29
openstackgerritMatt Riedemann proposed openstack/nova master: Add blocker migration for completing services.uuid migration  https://review.opendev.org/67362019:32
*** mriedem has joined #openstack-nova19:32
aspierskashyap: I just tried the same SEV experiment on Leap 15.0 with the same guest image (SLES15) and it gets stuck at the UEFI boot prompt, any ideas?19:40
openstackgerritMerged openstack/nova master: Replace "integrated-gate-py3" template with new "integrated-gate-compute"  https://review.opendev.org/67155119:42
*** jmlowe has joined #openstack-nova19:57
*** whoami-rajat has quit IRC19:57
eanderssonIs it safe to change overcommit ratios from 16 to 1 on a compute (it was never intended to have 16).20:10
eanderssonWill it update the placement inventory db automatically when the service is restarted?20:10
eandersson*updaate20:11
*** hamzy has quit IRC20:14
sean-k-mooneyeandersson: if you have overcommited unintentially all that will happen is no vm willl be allowed on that host untill you drop below the new overcommit limit20:14
sean-k-mooneyso yes20:14
eanderssonPerfect - I assumed as much, but my lab env was acting weird (probably needs to be rebuilt).20:14
* eandersson needs a better lab.20:15
*** xek has quit IRC20:15
*** BjoernT has quit IRC20:15
*** BjoernT has joined #openstack-nova20:16
mriedemsean-k-mooney: remember last week when i was having problems with an ubuntu 18.04 devstack vm, weird libvirt hang issues and such?20:18
mriedemi've got another one today with lots of random uwsgi crashes,20:18
sean-k-mooneyyep20:18
mriedemi realized i'm running on py2720:18
mriedemwhich we probably don't run on the gate much anywhere anymore20:18
mriedemso that might have something to do with it20:18
sean-k-mooneyoh maybe20:18
mriedemuwsgi seg faults20:18
sean-k-mooneywe still run in the gate with py27 i think20:19
sean-k-mooneyat least i think we have at lest 1 tempest job20:19
*** BjoernT_ has joined #openstack-nova20:19
mriedemyeah we do, tempest-full is py2720:19
sean-k-mooneyi have been using python 3 however for ubuntu 18.0420:20
sean-k-mooneyi actuly had the opisite issue not that long ago trying to run some  py3 test on centos 720:20
*** BjoernT has quit IRC20:20
sean-k-mooneywe might need to cap something in upper constratits20:21
sean-k-mooneywe have started to see more and more upstream libs drop py2 support20:22
mriedemheh n-cpu just crashed too20:22
mriedemwtf20:22
*** jmlowe has quit IRC20:23
*** BjoernT has joined #openstack-nova20:23
sean-k-mooneyi can restack with py 2.7 and see if it breaks20:23
sean-k-mooneyi assume your running master20:24
*** BjoernT_ has quit IRC20:24
*** dpawlik has joined #openstack-nova20:25
*** jmlowe has joined #openstack-nova20:25
*** BjoernT_ has joined #openstack-nova20:27
*** BjoernT has quit IRC20:29
sean-k-mooneymy dev vm is restacking now with Python 2.7.15 on kernel 4.18.0-25-generic20:34
*** BjoernT_ is now known as BjoernT20:34
sean-k-mooneyill let you know if it breaks20:34
mriedemhallelujah i finally created a vm20:35
mriedemyes this is master, devstack created today20:35
*** dpawlik has quit IRC20:35
*** bbowen has joined #openstack-nova20:36
*** jmlowe has quit IRC20:36
*** kashyap has quit IRC20:37
*** jmlowe has joined #openstack-nova20:38
*** BjoernT_ has joined #openstack-nova20:39
*** BjoernT has quit IRC20:40
*** belmoreira has joined #openstack-nova20:44
aspierswe're going to need a new version of that T-shirt: "OMG it worked on devstack"20:46
*** eharney has joined #openstack-nova20:46
sean-k-mooneyhaha it always works with devstack. devstack is life :)20:47
sean-k-mooneyjust stop running it in production20:47
*** hamzy has joined #openstack-nova20:47
*** igordc has quit IRC20:50
sean-k-mooneywell it stacked and i seam to be able to boot vms ok. ill leave my setup on python2 for a week and see if i hit any issues20:54
*** jmlowe has quit IRC20:56
sean-k-mooneymriedem: by the way are you using nested virt. i am but maybe that is related21:02
sean-k-mooneybut the first level vm where devstack is running and the second level vm booted by nova are using kvm21:03
mriedemyes nested virt with qemu21:03
mriedemoh wait,21:04
mriedemi assumed qemu, but looks like nova-cpu.conf is using kvm21:04
mriedemthat's likely the reason for the weird failures and hangs with libvirtd21:04
mriedemwtf21:04
sean-k-mooneyright well if you have nested vert enable i would expect ti to use kvm21:04
mriedemdevstack should be detecting and using qemu i think in most cases - devstack-gate (and i think zuulv3 now) force to qemu21:04
sean-k-mooneythey hardcode that21:05
sean-k-mooneydevstack prefers kvm21:05
sean-k-mooneyi think21:05
mriedemhttp://logs.openstack.org/40/673340/1/check/tempest-full/f05f8ae/controller/logs/etc/nova/nova-cpu_conf.txt.gz21:05
mriedemvirt_type = qemu21:05
sean-k-mooneyi hardcode virt_type=kvm in my local.conf21:05
mriedemno i'm pretty sure we use qemu exclusively in the gate b/c of weird shit failing randomly with nested virt and kvm21:05
sean-k-mooneyyes21:06
sean-k-mooneynested vert is very kenel specfic21:06
*** bbowen has quit IRC21:06
sean-k-mooneyis this still on the vexhost cloud21:08
sean-k-mooneythey use centos7 host i think and they have been quite stable with nested virt from what i have heard but i know the default ubunut 18.04 kernel has some issues21:09
*** BjoernT has joined #openstack-nova21:10
sean-k-mooneykernel 4.19+ has the nested virt fixes21:10
mriedemyes this is on vexxhost21:10
mriedem18.04 image21:10
mriedem$ uname -r -v21:12
mriedem4.15.0-20-generic #21-Ubuntu SMP Tue Apr 24 06:16:15 UTC 201821:12
mriedemanyway, i haven't had issues before with these images, but i'm guessing they were properly detecting and configuring nova to use qemu rather than kvm21:12
sean-k-mooneyim using 4.18.0-25-generic21:12
mriedemso it might be something in the devstack logic with 18.0421:12
*** BjoernT_ has quit IRC21:12
mriedemi think the gate jobs are setup to just hard-code virt_type=qemu21:12
mriedemfungi: ^ do you know off hand?21:12
mriedemclarkb probably does21:12
sean-k-mooneymriedem: it does21:12
sean-k-mooneymriedem: https://github.com/openstack/devstack/blob/master/.zuul.yaml#L22321:13
mriedemah yup21:13
mriedemjust found that too21:14
sean-k-mooneyanyway i woudl advise doing an apt-get dist upgrade and grab a new kernel if you want to fix the nested virt issue21:14
sean-k-mooneyoh im running the ubuntu Hardware Enablement Stack (HWE) kernel which is why mine is newer21:17
*** gyee has quit IRC21:23
openstackgerritMatt Riedemann proposed openstack/nova master: neutron: log something more useful in _get_instance_nw_info  https://review.opendev.org/67369421:24
mriedemit's not a contest sean gawd21:24
mriedemhaving all sorts of fun,21:28
mriedemlike hanging to create a server,21:28
mriedemallocations are removed and task_state is reset to None on error, but there is no traceback21:28
mriedemJul 30 21:15:11 devstack nova-compute[29645]: DEBUG nova.compute.manager [None req-a4247c77-5e4c-4e48-ba09-00f5d2903f23 admin admin] [instance: 33f64d9c-0a1a-410c-b5ba-e32ab501098d] Starting instance... {{(pid=29645) _do_build_and_run_instance /opt/stack/nova/nova/compute/manager.py:1955}}21:28
mriedemJul 30 21:16:11 devstack nova-compute[29645]: INFO nova.compute.manager [None req-a4247c77-5e4c-4e48-ba09-00f5d2903f23 admin admin] [instance: 33f64d9c-0a1a-410c-b5ba-e32ab501098d] Successfully reverted task state from None on failure for instance.21:28
mriedemJul 30 21:16:12 devstack nova-compute[29645]: INFO nova.scheduler.client.report [None req-a4247c77-5e4c-4e48-ba09-00f5d2903f23 admin admin] Deleted allocation for instance 33f64d9c-0a1a-410c-b5ba-e32ab501098d21:28
mriedemso we must be leaking an error somewhere and not logging it21:28
mriedemdefinitely hitting something like https://review.opendev.org/#/c/666857/ as a result21:30
sean-k-mooneyya maybe21:31
*** belmoreira has quit IRC21:32
sean-k-mooneybut in all seriousnes i used ot have nested virt issue before i changed to the hwe kernel so if you keep having issue give it a try and see if it helps21:33
sean-k-mooneyi have been hitting some gate isseu where vms just stay in building too21:33
sean-k-mooneyso i suspec your are right about leaking another error like that one21:34
mriedemi really haven't had weird issues like this in forever, just started happening but i don't think the image i'm using has changed (my devstack startup script has the image id hard-coded)21:34
sean-k-mooneyya its strange21:36
*** BjoernT has quit IRC21:36
mriedemyeah,21:36
mriedem| d5474e0f-3a32-4036-a89f-8a57dfbd237e | Ubuntu 18.04 LTS (x86_64) [2018-05-09]    | active |21:36
mriedemso that tells me it's devstack/nova that changed21:36
mriedemor dependent packages21:36
*** belmoreira has joined #openstack-nova21:36
sean-k-mooneywell you said you are runnign python 2 and like sphinx other libs are starting to kill there python2 support21:37
sean-k-mooneyso we might have an uncapped dep that is not happy on python 221:37
mriedemheh of course when i restart with more debug logging it works21:38
mriedemyeah i updated my local.conf to make sure the next one i use is py3 and uses qemu21:38
sean-k-mooneyi need to actully check python3.7 agian21:39
sean-k-mooneyi got simmilar issue to use using 3.7 back in january or febuary21:39
sean-k-mooneythe nova compute agent would hang when spawning a vm21:40
sean-k-mooneyeven though the functional tests and unit tests ran fine21:40
*** maciejjozefczyk has quit IRC21:41
*** gyee has joined #openstack-nova21:42
sean-k-mooneyanyway sorry i couldnt help narrow it down the next vm i create ill stack before i update the kernel and ill let you know if i run into the same issues21:44
sean-k-mooneyi need to set a vm to fix a dpdk issue so ill proably do that later in the week21:45
sean-k-mooneynight o/21:45
*** belmoreira has quit IRC21:53
*** slaweq has quit IRC21:53
*** boxiang_ has quit IRC21:54
*** boxiang_ has joined #openstack-nova21:54
efriedWhy do we have user/flavors.rst and admin/flavors.rst? Does "user" encompass "admin" in this sense?22:01
eanderssonsean-k-mooney, testing the patch out in the lab now, looks good so far22:04
eandersson(for rocky)22:04
*** panda has quit IRC22:15
*** panda has joined #openstack-nova22:18
*** ivve has quit IRC22:23
*** eharney has quit IRC22:26
openstackgerritMatt Riedemann proposed openstack/nova master: rt: remove misleading log in _remove_deleted_instances_allocations  https://review.opendev.org/67370622:27
*** READ10 has joined #openstack-nova22:36
mriedemefried: it's from the great docs migration in pike22:39
mriedemthere are a few things like that, like with quotas22:39
mriedemi think one of the flavor docs say they should be combined - or maybe i'm thinking of quotas22:39
mriedemthey are different audiences, like by default only admins can create flavors - there are other admin-only things in /users but i think over time that probably moves to /admin, not sure22:40
mriedeme.g. https://docs.openstack.org/nova/latest/user/#maintenance22:40
efriedyeah, that's what I thought22:40
efriedadmins create flavors, so why would there be flavors docs in the user section?22:41
efriedmaybe donnyd has an opinion, and/or wants to be the one to make it so22:41
mriedemusers need to know things about flavors like for vcpu/ram/disk, how flavors define swap and ephemeral disk, stuff like that22:41
mriedemgetting into extra specs is a bit more of a stretch22:41
donnydYea makes no sense for users to learn about flavors other than they need to pick one22:42
donnydso i agree efried22:42
efriedI guess if they need to `show` a flavor and understand what they're looking at in order to pick one, maybe it makes sense for there to be a user doc at that level. Is that what you were talking about mriedem?22:42
mriedemsomething like that22:43
donnydThey should know what a flavor means in the user guide, like if you pick a flavor that translates to you get x / y / and z in your instance22:43
efriedOkay. This went from "cut/paste user/flavors.rst into admin/flavors.rst" to "do something harder" so I'll punt, certainly at 5:45pm22:44
mriedemi think the overview in the user/flavors docs are ok, and the extra specs could maybe move to admin/flavors22:44
mriedemyeah this isn't really something i'm going to put a lot of brain power in right now22:45
efriedI'm outta here. See y'all tomorrow. o/22:45
mriedemusers might care about extra specs though for advanced vms, sriov, stuff like that22:45
mriedemso idk22:45
mriedemo/22:45
mriedemsame here22:45
*** mriedem has quit IRC22:45
openstackgerritDustin Cowles proposed openstack/nova master: WIP: Provider config file  https://review.opendev.org/67334122:52
*** READ10 has quit IRC22:56
*** tkajinam has joined #openstack-nova22:57
*** rcernin has joined #openstack-nova23:01
*** brault has joined #openstack-nova23:06
*** brault has quit IRC23:10
*** slaweq has joined #openstack-nova23:11
*** slaweq has quit IRC23:15
*** avolkov has quit IRC23:35
*** mlavalle has quit IRC23:40
*** betherly has quit IRC23:41

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!