Tuesday, 2023-04-18

frickler5,15 kernel should be new cirros. that failure very much looks like memory corruption to me04:39
opendevreviewMerged openstack/placement master: Bugtracker link update  https://review.opendev.org/c/openstack/placement/+/87676806:38
gibidansmith: I agree that kernel crash points to cirros. Obviously if we see such crashes whathever happens after it (ie detach failure) is probably a consequence of the crash.07:50
gibidansmith: in the original detach failure case there was no kernel crash, so this is definitely different07:50
kashyapgibi: On that device-detach patch from the QEMU upstream, do we have a way to test it upstream? -- https://www.mail-archive.com/qemu-devel@nongnu.org/msg952944.html08:01
kashyapgibi: I can create an RPM build of QEMU with that patch for Fedora ... if we there's a Fedora job that can test it08:02
kashyapBut probably a Ubuntu package is preferred, I guess08:02
gibiI don't think we have a Fedora based job upstream :/08:11
gibikashyap: but if you can build the fix to Fedora then you can at least try my libvirt only reproduction to see if that is fixed08:12
kashyapgibi: Oh, right indeed.  I'll give it a go this week and update you on the bug08:13
gibicool,thanks08:13
sean-k-mooneygibi: bauzas  i now want to add a new hw_vif_model specifically igb10:40
sean-k-mooneyhttps://www.qemu.org/docs/master/system/devices/igb.html10:40
sean-k-mooneyqemu added sriov virtualsiation support to it last year10:40
gibiwut? that feels super useful for testing if it does what I think it does10:41
sean-k-mooneyunfortunetly it will take a while before our cloud providers have this10:41
sean-k-mooneygibi: yep thats why they added it10:41
sean-k-mooneyit would allow us to take a neutron port and then create VFs from it10:41
gibithen I'm +10010:41
sean-k-mooneysupport was added in libvirt 9.3.0 10:42
sean-k-mooneythat may be qemu 9.3.0 libvirts docs kind of suck10:42
sean-k-mooneyat makeing it cleare if they mean the qemu or libvirt version10:42
sean-k-mooneyill finish the filter stuff firts and you know actully try and do it locally before proposing anything but if it work ill bring this up in a team meeting10:44
sean-k-mooneyjust not today10:44
sean-k-mooneyon a side note this is the alpine image i created https://fileshare.seanmooney.info/10:45
sean-k-mooneyim going to update the nova job patch to skip trying to build it and just pull that and see if it works after i finish the docs chagnes on teh schduler patch10:46
zigoHow do I migrate away a VM with status SUSPENDED ?!?11:16
sean-k-mooneyzigo: i suspended means suspended to disk. i think a cold migration might work but i would have to check the api11:26
sean-k-mooneylive migration wont because its not actully running11:28
sean-k-mooneylooking at the micate api it does not say but resize says You can only resize a server when its status is ACTIVE or SHUTOFF.11:29
sean-k-mooneyzigo: so my guess is without resuming the instnace its not possible11:29
sean-k-mooneybauzas: you have reviewd this in the past can you take alook at this backprot again https://review.opendev.org/c/openstack/nova/+/79044711:45
zigosean-k-mooney: Thanks.11:59
bauzassean-k-mooney: me clicks12:37
dansmithgibi: sean-k-mooney: yep, it could be totally different problems. However, we're at the point of suspecting the guest is not playing ball, and sometimes we just see no detach and other times a kernel crash13:32
dansmithsometimes I see only one trace in the klog when that happens, other times several like this13:32
dansmithso I'm just saying that I think expecting cirros to behave like a real guest is already suspect (in my mind)13:33
gibiyeah, I agree13:44
sean-k-mooneyso i have the alpine image hosted at https://fileshare.seanmooney.info/alpine.qcow213:51
sean-k-mooneyill see if i can just sub that in later today into a job and see what breaks13:51
dansmithmost of the guest crashes were in kmalloc things,13:54
dansmithwhich makes me wonder if there's more interaction with the host there13:55
dansmithactually they all go through load_module() in the kernel and are failing there13:58
bauzasdansmith: sorry I didn't had time to look at how many failures were having this kind of crash14:13
bauzasreminder : nova meeting in 15 mins15:45
dansmithbauzas: reminder, nova meeting 24 seconds ago16:00
bauzas#startmeeting nova16:00
opendevmeetMeeting started Tue Apr 18 16:00:33 2023 UTC and is due to finish in 60 minutes.  The chair is bauzas. Information about MeetBot at http://wiki.debian.org/MeetBot.16:00
opendevmeetUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.16:00
opendevmeetThe meeting name has been set to 'nova'16:00
bauzasdansmith: you beated me on the clock16:00
bauzaswalcome falks16:00
elodilles:D16:01
elodilleso/16:01
bauzas#link https://wiki.openstack.org/wiki/Meetings/Nova#Agenda_for_next_meeting16:01
auniyalo/16:01
bauzaslet's try to do a quick one16:01
bauzasso let's start16:01
bauzas#topic Bugs (stuck/critical) 16:02
bauzas#info No Critical bug16:02
bauzas(welp, somehow)16:02
gibio/16:02
bauzas#link https://bugs.launchpad.net/nova/+bugs?search=Search&field.status=New 13 new untriaged bugs (-5 since the last meeting)16:02
bauzaskudos to gibi16:02
bauzasgibi: any bug you wanna raise today ?16:02
gibitriaged couple of one https://etherpad.opendev.org/p/nova-bug-triage-2023041116:02
bauzasthanks16:02
gibinothing to raise today16:03
bauzasvery quickly looked at them, thanks for fishing them16:04
bauzas#info Add yourself in the team bug roster if you want to help https://etherpad.opendev.org/p/nova-bug-triage-roster16:04
bauzasso, next in the roster is melwitt16:04
bauzasnot sure she's around16:04
bauzasI'll ask her later if she can help16:04
bauzasif not, I'll take the baton16:04
bauzas#info bug baton is being passed to melwitt/bauzas16:05
bauzas#topic Gate status 16:05
bauzas#link https://bugs.launchpad.net/nova/+bugs?field.tag=gate-failure Nova gate bugs 16:05
bauzas#link https://etherpad.opendev.org/p/nova-ci-failures16:05
bauzasdo people want to discuss about the recent failures we got ?16:05
bauzasI didn't had time to look at logsearch yet16:05
Ugglao/16:06
bauzaswell, fair enough16:06
bauzaslet's say I haven't seen yet a lot of failures in my internal gerrit email directory16:07
bauzasmoving on so16:07
bauzas#link https://zuul.openstack.org/builds?project=openstack%2Fnova&project=openstack%2Fplacement&pipeline=periodic-weekly Nova&Placement periodic jobs status16:08
bauzasas always, all greens16:09
bauzas#info Please look at the gate failures and file a bug report with the gate-failure tag.16:09
bauzas#info STOP DOING BLIND RECHECKS aka. 'recheck' https://docs.openstack.org/project-team-guide/testing.html#how-to-handle-test-failures16:09
bauzas#topic Release Planning 16:09
bauzasso16:09
bauzas#link https://releases.openstack.org/bobcat/schedule.html16:09
bauzas#info Nova deadlines are set in the above schedule16:09
bauzasyou'll see some nova details in the above schedule ^16:09
bauzasbasically,16:09
bauzas#info Bobcat-1 is in 3 weeks16:09
bauzasand ditto the stable branch review day 16:10
bauzasanything to discuss about the schedule ?16:11
bauzas#topic Review priorities 16:12
bauzas#link https://review.opendev.org/q/status:open+(project:openstack/nova+OR+project:openstack/placement+OR+project:openstack/os-traits+OR+project:openstack/os-resource-classes+OR+project:openstack/os-vif+OR+project:openstack/python-novaclient+OR+project:openstack/osc-placement)+(label:Review-Priority%252B1+OR+label:Review-Priority%252B2)16:12
bauzas#info As a reminder, cores eager to review changes can +1 to indicate their interest, +2 for committing to the review16:12
bauzas#topic Stable Branches 16:12
bauzaselodilles: your turn16:13
elodilles#info final Xena release (24.2.1) is out - ready to transition stable/xena to EM16:13
bauzashuzzah16:13
elodilleshopefully we haven't merged anything harmful :)16:13
bauzasand I +1d the EM patch16:13
bauzasso we're ready 16:13
elodillesyepp, I will +2+W tomorrow morning16:13
bauzasone week in advance nah ?16:13
bauzasif so, \o/16:14
elodillesalmost 1 week, yes :]16:14
elodillesso we are good i think :)16:14
bauzasxena was a nice warrior but we should do yoga now16:14
elodillesyepp16:14
elodillesnow the general things:16:14
elodilles#info stable gates are not blocked, but sometimes patches need many rechecks16:14
elodilles#info stable branch status / gate failures tracking etherpad: https://etherpad.opendev.org/p/nova-stable-branch-ci16:14
elodillesthat's all from me16:15
bauzascool and thanks16:17
elodillesnp16:17
bauzasauniyal: I guess you wanted to add the two other items ?16:17
auniyal#help Please review these backport patches for zed and yoga release16:17
bauzasas a weekly reminder16:17
auniyalwill be releasing this week 16:17
auniyal#link https://etherpad.opendev.org/p/release-liaison-PatchesToReview16:17
bauzasack gtk, so I'll try to do homework then16:17
auniyalthats all bauzas, thanks16:17
bauzasauniyal: thanks16:17
bauzas#topic Open discussion 16:18
elodillesthanks in advance auniyal :)16:18
bauzas(this time, I'm refreshing)16:18
bauzasaaaand nothing, even after Ctrl-R16:18
bauzasso, anything to say before we end this meeting ?16:18
bauzasI thought we wanted to discuss about some volume detach issues16:18
bauzasbut not sure if the reporter is around, can't see jsanemet here16:19
auniyalbauzas, I have a feature spec on dangling volume its on review, is it same topic ?16:19
auniyal#link https://review.opendev.org/c/openstack/nova-specs/+/87875716:20
bauzasauniyal: nope, jorge wanted to discuss about the volume detach issues we were having in CI16:20
auniyalack16:20
bauzasbut given he's not around, we'll defer it to next week (if so)16:20
bauzasauniyal: fwiw, your spec is part of my review backlog16:20
auniyalthanks bauzas 16:21
bauzasokay, then, let's close early16:21
bauzasfolks, have a nice day, be it pretty done or early for you :)16:22
bauzasthanks all16:22
bauzas#endmeeting16:22
opendevmeetMeeting ended Tue Apr 18 16:22:21 2023 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)16:22
opendevmeetMinutes:        https://meetings.opendev.org/meetings/nova/2023/nova.2023-04-18-16.00.html16:22
opendevmeetMinutes (text): https://meetings.opendev.org/meetings/nova/2023/nova.2023-04-18-16.00.txt16:22
opendevmeetLog:            https://meetings.opendev.org/meetings/nova/2023/nova.2023-04-18-16.00.log.html16:22
opendevreviewDan Smith proposed openstack/nova master: Remove silent failure to find a node on rebuild  https://review.opendev.org/c/openstack/nova/+/88063216:22
opendevreviewDan Smith proposed openstack/nova master: Stop ignoring missing compute nodes in claims  https://review.opendev.org/c/openstack/nova/+/88063316:22
bauzasdansmith: looking at it then16:23
bauzasheh, first time I'm seeing such gerrit comment : "Code-Review-1 (copy condition: "changekind:TRIVIAL_REBASE OR is:MIN")"16:24
dansmithbauzas: I'm really not trying to nitpick, I just want to make sure we're sending what we should be16:27
bauzasthat was me who nitpicked16:28
bauzasanyway; +2d16:28
dansmithbauzas: all the exit paths from that rebuild send a dedicated notification about the "phase" of the rebuild and I just didn't want to add a path that doesn't16:28
bauzasI see your point16:28
opendevreviewribaudr proposed openstack/nova master: Attach Manila shares via virtiofs (db)  https://review.opendev.org/c/openstack/nova/+/83119316:32
opendevreviewribaudr proposed openstack/nova master: Attach Manila shares via virtiofs (objects)  https://review.opendev.org/c/openstack/nova/+/83940116:32
opendevreviewribaudr proposed openstack/nova master: Attach Manila shares via virtiofs (manila abstraction)  https://review.opendev.org/c/openstack/nova/+/83119416:32
opendevreviewribaudr proposed openstack/nova master: Attach Manila shares via virtiofs (drivers and compute manager part)  https://review.opendev.org/c/openstack/nova/+/83309016:32
opendevreviewribaudr proposed openstack/nova master: Attach Manila shares via virtiofs (api)  https://review.opendev.org/c/openstack/nova/+/83683016:32
opendevreviewribaudr proposed openstack/nova master: Check shares support  https://review.opendev.org/c/openstack/nova/+/85049916:32
opendevreviewribaudr proposed openstack/nova master: Add metadata for shares  https://review.opendev.org/c/openstack/nova/+/85050016:32
opendevreviewribaudr proposed openstack/nova master: Add instance.share_attach notification  https://review.opendev.org/c/openstack/nova/+/85050116:32
opendevreviewribaudr proposed openstack/nova master: Add instance.share_detach notification  https://review.opendev.org/c/openstack/nova/+/85102816:32
opendevreviewribaudr proposed openstack/nova master: Add shares to InstancePayload  https://review.opendev.org/c/openstack/nova/+/85102916:32
opendevreviewribaudr proposed openstack/nova master: Add helper methods to attach/detach shares  https://review.opendev.org/c/openstack/nova/+/85208516:32
opendevreviewribaudr proposed openstack/nova master: Add libvirt test to ensure metadata are working.  https://review.opendev.org/c/openstack/nova/+/85208616:32
opendevreviewribaudr proposed openstack/nova master: Add virt/libvirt error test cases  https://review.opendev.org/c/openstack/nova/+/85208716:32
opendevreviewribaudr proposed openstack/nova master: Add share_info parameter to reboot method for each driver (driver part)  https://review.opendev.org/c/openstack/nova/+/85482316:32
opendevreviewribaudr proposed openstack/nova master: Support rebooting an instance with shares (compute and API part)  https://review.opendev.org/c/openstack/nova/+/85482416:32
opendevreviewribaudr proposed openstack/nova master: Add instance.share_attach_error notification  https://review.opendev.org/c/openstack/nova/+/86028216:33
opendevreviewribaudr proposed openstack/nova master: Add instance.share_detach_error notification  https://review.opendev.org/c/openstack/nova/+/86028316:33
opendevreviewribaudr proposed openstack/nova master: Add share_info parameter to resume method for each driver (driver part)  https://review.opendev.org/c/openstack/nova/+/86028416:33
opendevreviewribaudr proposed openstack/nova master: Support resuming an instance with shares (compute and API part)  https://review.opendev.org/c/openstack/nova/+/86028516:33
opendevreviewribaudr proposed openstack/nova master: Add helper methods to rescue/unrescue shares  https://review.opendev.org/c/openstack/nova/+/86028616:33
opendevreviewribaudr proposed openstack/nova master: Support rescuing an instance with shares (driver part)  https://review.opendev.org/c/openstack/nova/+/86028716:33
opendevreviewribaudr proposed openstack/nova master: Support rescuing an instance with shares (compute and API part)  https://review.opendev.org/c/openstack/nova/+/86028816:33
opendevreviewribaudr proposed openstack/nova master: Docs about Manila shares API usage  https://review.opendev.org/c/openstack/nova/+/87164216:33
opendevreviewribaudr proposed openstack/nova master: Mounting the shares as part of the initialization process  https://review.opendev.org/c/openstack/nova/+/88007516:33
sean-k-mooneybauzas: we missed talking about the release note for hyperv and vmawre experimental/deprecated status16:38
sean-k-mooneyor more speicifclly backporting that to antelope16:38
bauzasshit16:39
bauzasmy brain derailed16:39
sean-k-mooneyso i would obviosly like to do both (merge on master and backport)16:39
sean-k-mooneyso that the warning goes out early16:40
sean-k-mooneyand then wait and see what happens with os-win before doing anythin else16:40
sean-k-mooneyif the TC retrie os-win in bobcat then we can look at remvoing the supprot in 2024.116:41
bauzassean-k-mooney: my only concern is that the relnotes for 27.0.0 won't see it16:42
sean-k-mooneyas long as its not breaking any ci jobs and we have sent the warning i dont think thre is any pressure on us to do anything else16:42
bauzasshow*16:42
bauzasso operators may miss it16:42
sean-k-mooneywe can proably fix that16:42
bauzaswell, they would need to upgrade to latest 2023.1 release before they would see the log, hence my point16:43
sean-k-mooneybut they shoudl not just be looking at 27.0.0 before going to 29.0.016:43
sean-k-mooneyyep but i dont think we will ahve peopel upgradign for 27.0.0 directly to 29.0.0 without any updates to 27.x.y16:44
dansmithI think backporting a reno to antelope is a good idea to increase our warning interval, but I do not think that buys us more time in terms of how soon we can do something16:44
sean-k-mooneyits a concern but i think its a minor one16:44
dansmithbecause people may deploy a .0 and never update16:44
sean-k-mooneymaybe16:44
dansmithmeaning I don't think we can reasonably deprecate in 2023.2 and remove it in 2024.116:44
sean-k-mooneybut i think CVEs ectra would force an update16:44
dansmithnobody is forced to update16:44
sean-k-mooneyya what sucks is we agree do deprecate in antelope and we just forgot to merge the patch16:45
dansmithIMHO, this is the extra work we take on by committing to the longer support envelope16:45
sean-k-mooneybecause it was not assocated with any bug or blueprint16:45
bauzassean-k-mooney: we had quite the same situation in newton16:45
dansmithyeah, sucks, but.. we just can't rewrite history16:45
bauzaswithout that cadence16:45
bauzasbut we forgot something that merged post-rc116:46
bauzasand I still remember the shitstorm that happened next16:46
bauzasit's still engraved in me16:46
sean-k-mooneyso removal in 2024.2 it is i guess16:46
sean-k-mooneyassuming nothign happens btween now and then16:46
bauzasI'd rather prefer to ask rosmaita again and the tc about the reno tooling things16:47
bauzasto see whether we could forward port the deletion notes to B16:47
bauzasbrian's proposal was quite ok to me16:47
bauzasbut the tooling has to be correctly documented16:47
dansmiththat doesn't change anything about the decision, just the mechanics of the notice, to be clear16:48
bauzasyup, that's why I'm still afraid of removing anything in B16:48
bauzasthe plan isn't sold yet16:48
dansmithdeprecating?16:48
dansmithwe can't remove these in B because we didn't warn about it16:48
bauzasremoving anything, and by extension deprecating (in this case)16:48
sean-k-mooneyya like if we really wanted to we could just move the 2023.1 git tag so that 27.0.0 has the release note btu that has other problems so i dont really wnat to go there16:49
dansmithno.16:49
sean-k-mooneybauzas: ya not sure where removal is coming form in b16:49
bauzassean-k-mooney: well, we can't do this or then the distros would hate us16:49
sean-k-mooneythat is not what is beign proposed16:49
bauzassorry, I meant the general case16:49
sean-k-mooneybauzas: hehe i know16:49
sean-k-mooneybut if we really wanted to get it in the 27.0.0 release notes im sure that possibel16:50
bauzasI'm saying I'm still quite unhappy with any change proposing to deprecate or remove any thing in B, including in particular the hyperv driver that we speak16:50
sean-k-mooneyanyway i think we can likely move forward with the patch and waith for it to go out in 2024.116:50
sean-k-mooneythen see whtat the state is for 2024.2 and if its still functioanl at that point or not16:51
bauzasdansmith: I haven't read the tc agenda for today, but do you think they will discuss the slurp things we discussed before the PTG ?16:51
sean-k-mooneybauzas: i find it very unreasoonble to disallow deprecateion or removals in B16:51
bauzassean-k-mooney: my PTG concern is still legit, we haven't still a solid plan on forward-porting the notes to C16:52
sean-k-mooneybut i agree that we can only remove thing in B if they were deprecated in A or older16:52
dansmithbauzas: neither have I so I dunno16:52
sean-k-mooneysure but i dont think we should block the removal of deprecated fucntionalty on resolvign it16:53
dansmithtbh, I'm not getting the problem here16:54
dansmithwe should backport a reno to 2023.1 about the deprecation,16:54
dansmithwe should have another one in 2024.1 for such a large thing16:55
dansmithwe should have a deprecation log message in 2023.1, 2023.2, 2024.116:55
dansmithI think bauzas' concern is about remembering to do the extra reno in 2024.1, right?16:55
sean-k-mooneyyes bauzas and i swtiched topic slightly to talke about deprecation/removals in B in general16:55
bauzasdansmith: correct16:55
sean-k-mooneyyes16:55
bauzasdansmith: your plan looks good then16:55
bauzasprovided we don't forget to mention the deprecation in 2024.116:56
dansmithsean-k-mooney: well, you seemed to go back to the specific with the "block removing on resolving"16:56
sean-k-mooneywell in 2024 we would be annouching that we intend to remvoe it effectivly16:56
sean-k-mooneyif that what we decied at that point16:56
sean-k-mooneywere as right now we are just saying its untested16:57
bauzassean-k-mooney: dansmith: I apologize, I mixed two problems16:57
dansmithbauzas: to be honest, the removal of it is not that important, IMHO16:57
sean-k-mooneyand deprecated but no plan to remove it in any partical release16:57
bauzasdon't disagree16:57
bauzasdansmith: ^16:57
dansmithbauzas: saying it's deprecated is a lot more important.. we can remove it when it becomes an _actual_ problem16:57
bauzastrue16:57
bauzasok, looks like we're settled then16:57
dansmithwhich, if it's been long enough, doesn't matter when we do it16:57
bauzasI can vote16:57
sean-k-mooneyanyway i need to go to another meeting now16:59
sean-k-mooneybauzas: as an fyi i plan to submit the patch to remove the AZ filter next week when i have time to write it16:59
bauzasack16:59
sean-k-mooneyit was ment to be done like in xena or yoga. i have a version somewhere but im jsut going to do it from scratch17:00
bauzassean-k-mooney: you know that by merging this without having a tooling in place, we basically leave myself to be responsible of the release note forward-porting, right? :)17:01
bauzas(just saying this here as well, as a side discussion occurs somewhere else)17:02
bauzashttps://review.opendev.org/c/openstack/nova/+/863910 is sent to the gate17:09
dansmithso,17:12
dansmithmarking it as experimental back then made more sense than I think it does now17:12
dansmith(I realize I'm +2 on that)17:12
dansmithI think we do need to mark it as actually deprecated to be clean here, because we know it has no maintenance horizon, the dependent library is abandoned and in danger, etc17:13
sean-k-mooneydansmith: we are technially doing both17:18
sean-k-mooneyits deprecated in the relesae notes17:18
dansmithno it's not17:18
sean-k-mooneybut we use experimantal in the logs becasue we did not want to call it deprecated for reasons17:18
dansmithhttps://review.opendev.org/c/openstack/nova/+/863910/1/releasenotes/notes/hyperv-experimental-antelope-372e18a05cafc295.yaml17:18
dansmithit says "may be removed" but no 17:18
dansmithnot "deprecated"17:19
sean-k-mooneydeprecations:17:19
sean-k-mooneyits in the deprecation section17:19
dansmithmeh17:19
dansmithit's too vague at this point.. we should be saying it's deprecated and will be removed17:20
dansmiththe log is probably even more important than the reno17:20
dansmith2% of deployments are using hyperv...we really should not leave any room for misinterpreting what we know is going to happen17:21
sean-k-mooneyok im fine with refining it but i was ok with the fact it was in the deprecation section to track that it was deprecated.17:23
opendevreviewMerged openstack/nova stable/yoga: Unify placement client singleton implementations  https://review.opendev.org/c/openstack/nova/+/85899717:55
opendevreviewMerged openstack/nova stable/yoga: Avoid n-cond startup abort for keystone failures  https://review.opendev.org/c/openstack/nova/+/85899818:17
opendevreviewsean mooney proposed openstack/nova master: add hypervisor version weigher  https://review.opendev.org/c/openstack/nova/+/88023119:23
opendevreviewsean mooney proposed openstack/nova master: add hypervisor version weigher  https://review.opendev.org/c/openstack/nova/+/88023119:25

Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!