Tuesday, 2023-10-03

opendevreviewsean mooney proposed openstack/nova master: [codespell] fix typos in api-ref  https://review.opendev.org/c/openstack/nova/+/89709400:33
opendevreviewsean mooney proposed openstack/nova master: [codespell] apply codespell to the releasenotes  https://review.opendev.org/c/openstack/nova/+/89709500:33
opendevreviewsean mooney proposed openstack/nova master: [codespell] doc,devstack and gate typos  https://review.opendev.org/c/openstack/nova/+/89709600:33
*** kopecmartin|off is now known as kopecmartin07:04
bauzascores, could you please quickly merge https://review.opendev.org/c/openstack/nova-specs/+/895815 before our Bobcat GA ?07:36
gibibauzas: Is it better to mark the ironic shard implemented with a note that it is reverted? Do we expect that shard spec will be reproposed in C? 08:12
bauzasgibi: have you seen the next change in the series ?08:13
bauzashttps://review.opendev.org/c/openstack/nova-specs/+/896679/1/specs/2023.2/implemented/ironic-shards.rst08:13
gibibauzas: I saw the note yes08:14
gibimy question is will there be a new spec in C to merge the shard implementation again?08:14
bauzasgibi: I hope so08:14
gibithen we should not put the current shard spec to implemented08:14
bauzasjohnthetubaguy: do you know if you could be able to create a new spec in Caracal for ironic-shards ?08:15
gibiwe should only put the shard spec to implemented that was implemented and *released*08:15
bauzasokay, then let me change the Launchpad blueprint08:16
opendevreviewSylvain Bauza proposed openstack/nova-specs master: Move Bobcat implemented specs  https://review.opendev.org/c/openstack/nova-specs/+/89581508:23
opendevreviewSylvain Bauza proposed openstack/nova-specs master: explain why ironic-shards was reverted in the spec  https://review.opendev.org/c/openstack/nova-specs/+/89667908:23
gibithanks09:32
sean-k-mooneysure ill review it now09:37
sean-k-mooneydone09:38
sean-k-mooneyback to codespell...09:40
bauzasthanks09:50
opendevreviewMerged openstack/nova-specs master: Move Bobcat implemented specs  https://review.opendev.org/c/openstack/nova-specs/+/89581509:52
opendevreviewMerged openstack/nova-specs master: explain why ironic-shards was reverted in the spec  https://review.opendev.org/c/openstack/nova-specs/+/89667909:52
dvo-plvsean-k-mooney, Hello, I had resolved your comment, Could you please review it again in a free time ? https://review.opendev.org/c/openstack/nova-specs/+/89592410:18
sean-k-mooneyyep looks fine to me10:26
sean-k-mooneyeasy reporposal ^ if anyone else wants to review that10:26
dvo-plvThank you10:27
dvo-plvyes, bauzas told me yesterday, that its review week, so I would like to wait for it as far as it is in his agenda10:28
sean-k-mooneywell the bobcat release is mostly done now and we are getting ready for the ptg so its not really formal10:30
sean-k-mooneybut this week is when most of us are starting to focus more on the next cycle10:31
sean-k-mooneythats why im getting all the liniting/spellecheking/style enhancment stuff i have wanted to do for the last year or so ready this week10:32
sean-k-mooneyi.e. before next cycle dev really picks up get the tooling to help use review and land code faster in place10:32
opendevreviewsean mooney proposed openstack/nova master: [codespell] fix typos in tests  https://review.opendev.org/c/openstack/nova/+/89721311:02
opendevreviewsean mooney proposed openstack/nova master: [codespell] fix final typos and enable ci  https://review.opendev.org/c/openstack/nova/+/89721411:02
opendevreviewsean mooney proposed openstack/nova master: [codespell] ignore codespell in git blame  https://review.opendev.org/c/openstack/nova/+/89721511:02
opendevreviewsean mooney proposed openstack/nova master: fix sphinx-lint errors in docs and add ci  https://review.opendev.org/c/openstack/nova/+/89708911:08
opendevreviewsean mooney proposed openstack/nova master: ignore sphinx-lint series in git blame  https://review.opendev.org/c/openstack/nova/+/89721811:11
opendevreviewsean mooney proposed openstack/nova master: remove deprecated vmware virt driver  https://review.opendev.org/c/openstack/nova/+/89701711:15
opendevreviewsean mooney proposed openstack/nova master: remove deprecated vmware virt driver  https://review.opendev.org/c/openstack/nova/+/89701711:17
opendevreviewsean mooney proposed openstack/nova-specs master: resubmit per-process-healthchecks spec for 2024.1  https://review.opendev.org/c/openstack/nova-specs/+/89722512:24
opendevreviewsean mooney proposed openstack/nova master: [WIP] add initial healthcheck support  https://review.opendev.org/c/openstack/nova/+/82501512:26
opendevreviewsean mooney proposed openstack/nova master: [WIP] add healthcheck manager to manager base  https://review.opendev.org/c/openstack/nova/+/82784412:26
opendevreviewsean mooney proposed openstack/nova master: [WIP] add healthcheck tracker to nova context  https://review.opendev.org/c/openstack/nova/+/82946812:26
opendevreviewsean mooney proposed openstack/nova master: [WIP] add healthcheck utils and constants  https://review.opendev.org/c/openstack/nova/+/82946912:26
opendevreviewsean mooney proposed openstack/nova master: add healthcheck endpoint to proxy commands  https://review.opendev.org/c/openstack/nova/+/83070312:26
opendevreviewAmit Uniyal proposed openstack/nova master: Adds device tagging functional tests  https://review.opendev.org/c/openstack/nova/+/89516212:35
opendevreviewAmit Uniyal proposed openstack/nova master: Device tags: don't pass pf_interface=True to get_mac_by_pci_address  https://review.opendev.org/c/openstack/nova/+/67059312:35
opendevreviewAmit Uniyal proposed openstack/nova master: WIP: Device tagging: expose target PCI address, not source  https://review.opendev.org/c/openstack/nova/+/67212712:35
opendevreviewribaudr proposed openstack/nova master: Allow config to support virtiofs (driver)  https://review.opendev.org/c/openstack/nova/+/88652213:15
opendevreviewribaudr proposed openstack/nova master: Attach Manila shares via virtiofs (db)  https://review.opendev.org/c/openstack/nova/+/83119313:15
opendevreviewribaudr proposed openstack/nova master: Attach Manila shares via virtiofs (objects)  https://review.opendev.org/c/openstack/nova/+/83940113:15
opendevreviewribaudr proposed openstack/nova master: Attach Manila shares via virtiofs (manila abstraction)  https://review.opendev.org/c/openstack/nova/+/83119413:15
opendevreviewribaudr proposed openstack/nova master: Attach Manila shares via virtiofs (drivers and compute manager part)  https://review.opendev.org/c/openstack/nova/+/83309013:15
opendevreviewribaudr proposed openstack/nova master: Mounting the shares as part of the initialization process  https://review.opendev.org/c/openstack/nova/+/88007513:15
opendevreviewribaudr proposed openstack/nova master: Deletion of associated share mappings on instance deletion  https://review.opendev.org/c/openstack/nova/+/88147213:15
opendevreviewribaudr proposed openstack/nova master: Add metadata for shares  https://review.opendev.org/c/openstack/nova/+/85050013:15
opendevreviewribaudr proposed openstack/nova master: Add share_info parameter to reboot method for each driver (driver part)  https://review.opendev.org/c/openstack/nova/+/85482313:15
opendevreviewribaudr proposed openstack/nova master: Support rebooting an instance with shares (compute manager part)  https://review.opendev.org/c/openstack/nova/+/85482413:15
opendevreviewribaudr proposed openstack/nova master: Add share_info parameter to resume method for each driver (driver part)  https://review.opendev.org/c/openstack/nova/+/86028413:15
opendevreviewribaudr proposed openstack/nova master: Support resuming an instance with shares (compute manager part)  https://review.opendev.org/c/openstack/nova/+/86028513:15
opendevreviewribaudr proposed openstack/nova master: Add helper methods to rescue/unrescue shares  https://review.opendev.org/c/openstack/nova/+/86028613:15
opendevreviewribaudr proposed openstack/nova master: Support rescuing an instance with shares (driver part)  https://review.opendev.org/c/openstack/nova/+/86028713:15
opendevreviewribaudr proposed openstack/nova master: Support rescuing an instance with shares (compute manager part)  https://review.opendev.org/c/openstack/nova/+/86028813:15
opendevreviewribaudr proposed openstack/nova master: Allow to mount manila share using Cephfs protocol  https://review.opendev.org/c/openstack/nova/+/88386213:15
opendevreviewribaudr proposed openstack/nova master: Check shares support (compute manager)  https://review.opendev.org/c/openstack/nova/+/88575113:15
opendevreviewribaudr proposed openstack/nova master: Add share lock/unlock and restrict visibility  https://review.opendev.org/c/openstack/nova/+/89034013:15
opendevreviewribaudr proposed openstack/nova master: Check shares support (only API exception)  https://review.opendev.org/c/openstack/nova/+/88575213:15
opendevreviewribaudr proposed openstack/nova master: Attach Manila shares via virtiofs (API)  https://review.opendev.org/c/openstack/nova/+/83683013:15
opendevreviewribaudr proposed openstack/nova master: Check shares support (API)  https://review.opendev.org/c/openstack/nova/+/85049913:15
opendevreviewribaudr proposed openstack/nova master: Add helper methods to attach/detach shares  https://review.opendev.org/c/openstack/nova/+/88575313:15
opendevreviewribaudr proposed openstack/nova master: Add instance.share_attach notification  https://review.opendev.org/c/openstack/nova/+/85050113:15
opendevreviewribaudr proposed openstack/nova master: Add instance.share_detach notification  https://review.opendev.org/c/openstack/nova/+/85102813:15
opendevreviewribaudr proposed openstack/nova master: Add shares to InstancePayload  https://review.opendev.org/c/openstack/nova/+/85102913:15
opendevreviewribaudr proposed openstack/nova master: Add instance.share_attach_error notification  https://review.opendev.org/c/openstack/nova/+/86028213:15
opendevreviewribaudr proposed openstack/nova master: Add instance.share_detach_error notification  https://review.opendev.org/c/openstack/nova/+/86028313:15
opendevreviewribaudr proposed openstack/nova master: Add libvirt test to ensure metadata are working.  https://review.opendev.org/c/openstack/nova/+/85208613:15
opendevreviewribaudr proposed openstack/nova master: Add virt/libvirt error test cases  https://review.opendev.org/c/openstack/nova/+/85208713:15
opendevreviewribaudr proposed openstack/nova master: Docs about Manila shares API usage  https://review.opendev.org/c/openstack/nova/+/87164213:15
bauzassean-k-mooney: please try to avoid telling our contributors that we would have new linting enhancements until we have a consensus, please13:31
sean-k-mooneybauzas: i didnt say we will i said i wanted to get them ready before the ptg13:32
bauzasokay13:32
sean-k-mooneyi would very strongly like to do this however13:33
bauzastbc, I'm okay with a new project, my only concerns are around to Nova13:34
sean-k-mooneyim mainly concerned about nova and our deliverables13:35
atmarkHello! Yesterday, I upgraded Victoria to Wallaby using kolla-ansible. Out of 48 computes, I found 2 computes duplicated entries in  `openstack compute service list`. It won't let me delete rouge  entry 14:56
atmarkI tried deleting the resource allocations and tried to do heal allocations but it throws an error 14:57
dansmithatmark: won't let you delete service records because why? it thinks they have instances running on them?14:58
atmarkdansmith: yes14:59
atmarkthese are duplicate entries https://paste.openstack.org/show/bBM1U8uQkSgpsNh6gLFU/14:59
bauzasreminder : nova meeting in 1 hour by now here15:00
dansmithatmark: okay you'll want to find the instances that are assigned to the orphaned computes15:01
dansmithI'm surprised to see that they have identical names15:02
atmarkcompute01's nova-compute.log https://paste.openstack.org/show/bGYG9FcesMGyFRgYH08x/ 15:02
opendevreviewMerged openstack/nova-specs master: Amend spec to add more details around encryption secrets  https://review.opendev.org/c/openstack/nova-specs/+/88790515:04
dansmithhuh okay I guess we can have duplicate hostnames in service records, even for the same binary15:06
atmarkdansmith: so I found the and tried to heal allocations but throws not descriptive `Error:` https://paste.openstack.org/show/bFSAcoAlJvZdpCrxzRyh/ 15:06
atmarkfound the instances*15:06
dansmithatmark: yeah that's not going to help because they have different nodes in the DB as well15:06
dansmithatmark: so the best thing you can do, if able, is to migrate the affected instances away from the duplicate services and then delete the dupes after they're empty15:08
atmarkCold migrate? Getting `No server with a name or ID of '' exists. when doing live migrate 15:11
dansmithany migration15:12
dansmithis that the whole error message?15:12
atmarkyes, that's from the openstack client 15:13
opendevreviewMerged openstack/nova-specs master: Re-propose using extend volume completion action for 2024.1  https://review.opendev.org/c/openstack/nova-specs/+/89564815:15
dansmithlike it's complaining about the instance id being unknown?15:15
dansmithare you able to do a server show on that instance?15:16
atmarkdansmith: disregard the previous error, my for loop syntax to live migrate was wrong 15:24
atmarkgetting `Compute service of compute25.cnco1  is unavailable at this time.` 15:24
dansmithheh okay15:24
dansmithatmark: is that just live or did you try cold as well?15:24
atmarklive15:24
atmarklet me try cold15:24
dansmithso I think this might be because you've got these duplicated service records, so it's probably picking the wrong one.. usually what happens is people get duplicated *node* records, but the service record shows as up, so this works15:25
dansmithpicking the wrong one meaning it thinks the compute node is down even though it's not (obviously).. the migrating is what we need to correct the node mismatch, which you *also* have15:26
atmark`Service is unavailable at this time.` for cold migrate 15:26
dansmithcan you select out the host and node fields for one of the affected instances direct from the database?15:29
atmarkdansmith: https://paste.openstack.org/show/b0oiRoqWDHHgataIK7jO/15:35
dansmithatmark: and can you query out the relevant records from services and compute_nodes?15:37
dansmithit's weird that the compute is having trouble finding the right compute node record (placement errors in the compute logs) and that the instance is being associated with the downed duplicate service record15:38
atmarkthese are instances affected https://paste.openstack.org/show/bM6wA9HPHbuNzGmNEqPB/15:38
dansmithyeah so just a little background, we associate instances, nodes and services very loosely with these hostnames, which is terrible and fragile, as you're noticing.. work has been done recently to try to make this better (if that's any consolation)15:39
atmarkservices and compute_nodes table https://paste.openstack.org/show/bujkfzse6gko5nV197ue/15:43
dansmithhmm, okay, so there is only one non-deleted service of each name right?15:44
dansmitha record is deleted when there is a non-zero value in the deleted column15:44
dansmithI'm not sure why you're seeing them in the services list through the openstack client in that case15:47
atmarkfor compute01, I ran `openstack resource provider allocation delete $consumerid`, deleted the resource provider for the compute, restarted the nova compute to let it recreate resource provider entry a15:47
atmarkupon healing allocation, it wont just throws an error 15:47
atmarkit just throws an non descriptive error*15:48
dansmithyeah, well (a) you shouldn't have done that but (b) I think it's failing to create a new provider with the same name15:48
dansmithalso that has nothing to do with these records, FWIW15:49
dansmithatmark: your compute nodes list doesn't show the deleted field, can you update with that field?15:49
atmarkhttps://paste.openstack.org/show/b6Ng2S3XwYQoL8G0mJvr/ with deleted field 15:50
dansmithso the uuid in our compute01 logs shows a node uuid of bf7a7e8e-c486-42b2-8dc4-00b2d5f70f8615:52
dansmithbut I don't see that in your nodes list15:53
dansmithcan you see if any compute node has that uuid in the DB?15:53
dansmithatmark: I guess I should also ask - do you have multiple compute cells?15:53
atmark(a) - I followed what's in RH KB https://access.redhat.com/solutions/5308941. After the upgrade, nova-compute on those was throwing `Conflicting resource provider name` 15:53
dansmithare you running OSP?15:54
atmarkatmark: Yes, two cells: cnco1 and cnco2 but I only use cnco1 cell 15:55
atmarkdansmith: No. I have kolla-ansible deployment 15:55
dansmithmeaning there are no compute nodes in the other cell? but it's mapped?15:55
atmarkshow databases https://paste.openstack.org/show/b26YFJxzw33hNLa9pVLw/ 15:56
opendevreviewElod Illes proposed openstack/nova stable/victoria: libvirt: Abort live-migration job when monitoring fails  https://review.opendev.org/c/openstack/nova/+/83732115:56
dansmithre: cells I'm just trying to suss out if there are any duplicates happening because of that.. can you check the services table in the other cell to see if any of them have the same hostname of either of these affected ones just to be sure?15:56
atmarknova_cnco1 is where all computes are mapped 15:56
dansmithatmark: we're going to have to pause here for the compute meeting in a few minutes but if you're still around after we can resume15:57
atmarkgot it15:57
atmarkappreaciate the help 15:57
atmarkre:  can you check the services table in the other cell - nova_cell0?  15:58
dansmithno you said you have a cnco2 cell15:59
atmarknova_cell0 https://paste.openstack.org/show/by9WvQgKARdZRppniRoY/ 16:00
dansmithcell0 is a different thing16:00
bauzas#startmeeting nova16:00
opendevmeetMeeting started Tue Oct  3 16:00:09 2023 UTC and is due to finish in 60 minutes.  The chair is bauzas. Information about MeetBot at http://wiki.debian.org/MeetBot.16:00
opendevmeetUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.16:00
opendevmeetThe meeting name has been set to 'nova'16:00
bauzassorry guys, meeting by now16:00
elodilleso/16:00
bauzasatmark: I can also help you once we're done with the meeting, shall be quick16:00
dansmitho/16:00
bauzasheya folks16:00
opendevreviewMerged openstack/nova master: Add job to test with SQLAlchemy master (2.x)  https://review.opendev.org/c/openstack/nova/+/88623016:00
bauzas#link https://wiki.openstack.org/wiki/Meetings/Nova#Agenda_for_next_meeting16:01
gibio/16:01
bauzasok, I guess we can start16:01
bauzas#topic Bugs (stuck/critical) 16:01
Ugglao/16:01
bauzas#info No Critical bug16:02
bauzas#link https://bugs.launchpad.net/nova/+bugs?search=Search&field.status=New 44 new untriaged bugs (-1 since the last meeting)16:02
bauzas#info Add yourself in the team bug roster if you want to help https://etherpad.opendev.org/p/nova-bug-triage-roster16:02
bauzasgibi: happy with triaging some bugs ?16:02
bauzasor do you want to skip it given your other prios ?16:02
gibibauzas: I can try16:03
bauzasgibi: no worries, as I always say, it's a stretch goal, best-effort16:03
bauzasand fwiw, in general when it's me, the number of bugs is raising :blushes: :facepalm:16:04
bauzasbut thanks, appreciated given the known situation16:04
bauzas#info bug baton is gibi16:04
bauzasmoving on16:04
bauzas#topic Gate status 16:04
bauzas#link https://bugs.launchpad.net/nova/+bugs?field.tag=gate-failure Nova gate bugs 16:04
bauzas#link https://zuul.openstack.org/builds?project=openstack%2Fnova&project=openstack%2Fplacement&pipeline=periodic-weekly Nova&Placement periodic jobs status16:04
bauzas#info Please look at the gate failures and file a bug report with the gate-failure tag.16:05
bauzasall greens16:05
bauzasnothing I spotted 16:05
bauzasthat's a silent week and I appreciate it :)16:05
bauzasany CI failures you guys spotted N?16:05
bauzas(doh about the N?, thanks french keyboard layout)16:06
bauzaslooks nothing ditto16:06
bauzasthat's cool \o/16:06
bauzas#topic Release Planning 16:06
bauzas#link https://releases.openstack.org/caracal/schedule.html16:06
bauzas#info Nova deadlines aren't set yet until we agree on them at the PTG16:07
bauzas#info Bobcat GA planned tomorrow16:07
bauzasmostly a FYI16:07
bauzas#info Specs can be reproposed for 2024.1 Caracal timeframe16:07
bauzasI've been a round of reviews but I still need to do my homework on some 16:07
bauzasI've been doing*16:08
bauzas(with the verb, it's better)16:08
bauzas#info Caracal-1 milestone in 6 weeks16:08
bauzasI don't expect any deadline around caracal-1 so hold my beer16:08
sean-k-mooneywell16:09
sean-k-mooneythe only "deadlien" we really had around m1 ins the past was a gentil16:09
sean-k-mooneyencurrament to please have your specs ready for review16:09
bauzasyeah, for sure, good point16:10
sean-k-mooneybecause many are on PTO for a lot of the time between m2 and m116:10
sean-k-mooneyso prehaps not a deadline16:10
sean-k-mooneybut i do want to do a spec review day16:10
bauzasonce we agree on the deadlines, be sure I'll probably tell this16:10
bauzasyeah, sure too16:10
sean-k-mooneysomewhere between m1 and decemeber 13th ish16:10
bauzasfwiw, I started looking at some specs16:10
bauzasbut we can align ourselves on a review day, this is helpful16:11
bauzasand yeah about the calendar16:11
sean-k-mooneywhen is m1 this release16:11
bauzasfor some unexpected reason, people leave around Christmas period, doh.16:11
* sean-k-mooney has not looked at the schedule yet16:11
bauzasC-1 is mid-Nov16:11
sean-k-mooneyack16:11
bauzasC-2 is the second week of January16:12
sean-k-mooneyso like m-1 + 4 weeks is when we should try to get most specs reviewd and landed by16:12
bauzasso yeah, basically, it would be appreciated if specs could be written in advance of the XMas period16:12
sean-k-mooneym-2 is very early in jan ya16:12
bauzassure, and that's one of the topics we already have in the PTG etherpad :)16:13
sean-k-mooneycool16:13
bauzasanyway, yeah, people can propose anytime they want, the sooner the better16:13
bauzasbut we'll clarify that after PTG16:13
bauzasthis actually goes well as a good transition16:14
bauzas#topic Caracal vPTG planning 16:14
bauzas#info Sessions will be held virtually October 23-2716:14
bauzas#info Register yourselves on https://ptg2023.openinfra.dev/ even if the event is free16:14
bauzas(I stupidly did it twice, heh)16:14
bauzas#link https://etherpad.opendev.org/p/nova-caracal-ptg PTG etherpad16:14
bauzas#info add your own topics into the above etherpad if you want them to be discussed at the PTG16:15
bauzastime for collecting your inputs !16:15
bauzasagain, the sooner the better16:15
bauzasthe sooner we know the topics, the easiest it would be for us to organize our sessions and plan in advance the schedule16:15
bauzastl;dr: don't hold your thoughts, provided you care about something for Caracal16:16
bauzasmoving on16:16
bauzas#topic Review priorities 16:16
bauzas#link https://review.opendev.org/q/status:open+(project:openstack/nova+OR+project:openstack/placement+OR+project:openstack/os-traits+OR+project:openstack/os-resource-classes+OR+project:openstack/os-vif+OR+project:openstack/python-novaclient+OR+project:openstack/osc-placement)+(label:Review-Priority%252B1+OR+label:Review-Priority%252B2)16:16
bauzas#info As a reminder, people eager to review changes can +1 to indicate their interest, +2 for asking cores to also review16:17
bauzasI shall actually regenerate a new Gerrit dashboard with review prios16:17
bauzasthat's easy16:17
bauzas#action bauzas to propose a Gerrit dashboard for review priorities16:18
bauzas#topic Stable Branches 16:18
bauzaselodilles: WAAAZUUUP ?16:18
elodilles:)16:18
elodillesas you said, we are in a calm period, even for stable branches16:18
elodillesi'm not aware of any broken stable gates16:19
bauzas\o/16:19
elodilles#info stable branch status / gate failures tracking etherpad: https://etherpad.opendev.org/p/nova-stable-branch-ci16:19
elodillesplease add here issues if you find one ^^^16:19
elodillesthat's all from me16:19
auniyal#info request to review https://review.opendev.org/q/topic:bug%252F199673216:19
bauzaselodilles: bobcat will become our stable branch tomorrow16:19
bauzaselodilles: I assume with the new TC resolution that nothing happens with the other branches in terms of maintenance ?16:20
elodilleswell, 2023.2 is already open for all our repositories16:20
elodillesand yoga was next to transition to EM in early november,16:20
bauzasoh right, we did cut the last rc16:20
elodillesi don't know yet what will be the new process around yoga16:21
bauzasI need to read something at bedtime16:21
bauzaswhich is the new TC resolution about EM16:21
elodillesanyway, i suggest to release any needed yoga patches, if you have one :)16:21
bauzas(should hopefully find sleep after that :D)16:21
bauzas(naaah, kidding)16:21
elodillesbecause in a month yoga will transition to somehwere :)16:22
gibi:)16:22
bauzasfwiw, I have no personal interest into yoga, in whether form it can be :)16:22
bauzasincluding the real strecthing thing16:23
bauzasexcept it will be our first SLURP-ish release to disappear16:23
dansmithSLURPy?16:23
elodilles:)16:23
bauzashah16:24
bauzasanyway, that questions me about the future of non-SLURP releases if the precedent SLURP release goes EOL16:24
auniyalbauzas  few from my end ... \o 16:24
bauzassurely people can non-skip releases, so it can exist16:25
bauzasbut its interest reduces16:25
sean-k-mooneywell each reasle is supported for 18 months16:25
bauzasauniyal: you're next in the pipe16:25
sean-k-mooneyand non slurps are drop after that16:25
auniyalack thakns16:25
auniyalsean-k-mooney there is a backport patch for stable/zed https://review.opendev.org/c/openstack/nova/+/885344 created by gibi16:25
sean-k-mooneyonly slurps can move form stable/x to unmained/x16:25
auniyalits for anti-affinity check count,  can you please review this .16:25
bauzasanyway, not a question we shall discuss here, mostly a TC-wide question16:26
sean-k-mooneyso the current stables got granfatherd in16:26
auniyalI am asking you because you had reviewed it for master and its merged in master.16:26
sean-k-mooneythere is a section in the tc doc for that16:26
auniyalit's created till ussuri, if its alright can you please approve them as well. thanks  :) 16:26
sean-k-mooneyso zed is still ok to have patches merged16:26
bauzasok, anuyal, I guess it's your turn now16:26
JayFsean-k-mooney: ++ that matches my understanding16:26
auniyal:( 16:26
sean-k-mooneywe should however do a review of the old stable branches and determin if they are healty 16:26
sean-k-mooneyliek we are ment to do for unstable16:27
bauzassean-k-mooney: yeah, sounds an idea16:27
auniyalbauzas added already thats all from me16:27
sean-k-mooneythat specific patch seams to have passed ci so i hope that means zed is healty16:27
bauzasbecause we EOLd Train with pain16:27
bauzasand I would want us to be a little more aggresive in terms of cleanup :)16:28
sean-k-mooneyi would have to check the resolution but i think everyting pre Wallaby should be removed under the propsal16:28
sean-k-mooneybut i dont recall off the top of my head16:28
sean-k-mooneyhttps://github.com/openstack/governance/blob/master/resolutions/20230724-unmaintained-branches.rst#transition16:29
sean-k-mooneyhe last 3 active Extended Maintenance branches are automatically transitioned to Unmaintained branches.16:29
elodilles++16:29
sean-k-mooneyso thats W,X,Y16:29
sean-k-mooneyunless im off by one which i could be16:30
elodilles(this is the merged resolution: https://governance.openstack.org/tc/resolutions/20230724-unmaintained-branches.html )16:30
sean-k-mooney so we dont need to fully figure this out now16:30
bauzasas I said, I need to do bedtime readings16:31
bauzasnothing urges16:31
sean-k-mooneybut we can prepare to EOL Ussuri and victoria and asses Wallaby16:31
elodillessean-k-mooney: i remember, X,W,V, hence, i'm not sure about Y :)16:31
bauzasbut yeah, EOLing up to Victoria is honestly a goal for me :)16:31
sean-k-mooneyeoling Victoria allows us to drop 18.0416:32
sean-k-mooneyform all our ci16:32
elodillesyeah, ussuri is the last 18.04 based series16:32
sean-k-mooneyso that i think has a lot of value16:33
bauzasyeah16:33
dansmithwhy?16:33
bauzasin terms of CI16:33
dansmithI mean I'm fine with EOLing V but is 18.04 adding a lot of pain I'm unaware of?16:33
bauzasif nova stops supporting ussuri, that would pull the other projects to do the same16:33
sean-k-mooneydansmith: 18.04 is EOL form a canonical point of view16:34
sean-k-mooneyso its not entirly unsecurity patched16:34
sean-k-mooneyit went out of supprot in march this year16:34
bauzasworth adding it into a PTG topic then16:35
dansmithokay just for that reason, fine, I just am not aware of any actual problems16:35
bauzasI take the action16:35
sean-k-mooneywe dont get any similar benifts again until zed16:35
dansmithbauzas: you said this meeting would be quick16:36
bauzasok, moving on16:36
bauzasand yeah16:36
sean-k-mooneyzed droped centos 8 and python 3.616:36
bauzasmy bad, I opened a can16:36
elodilles:D16:36
bauzas#topic Open discussion 16:36
bauzas(nothing)16:36
bauzasso, anything else or can I return you 23 mins of your time ?16:37
sean-k-mooneynot form me16:37
bauzasthanks all16:37
bauzas#endmeeting16:37
opendevmeetMeeting ended Tue Oct  3 16:37:29 2023 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)16:37
opendevmeetMinutes:        https://meetings.opendev.org/meetings/nova/2023/nova.2023-10-03-16.00.html16:37
opendevmeetMinutes (text): https://meetings.opendev.org/meetings/nova/2023/nova.2023-10-03-16.00.txt16:37
opendevmeetLog:            https://meetings.opendev.org/meetings/nova/2023/nova.2023-10-03-16.00.log.html16:37
elodillesthanks o/16:37
* sean-k-mooney has one more meeting at the top of the hour and im done for today16:38
dansmithatmark: okay so, there should never be compute services in cell0.. that's why you're seeing the duplicates from the api I think, which we should resolve to start16:38
dansmithatmark: I think you probably also have that node uuid in the compute_nodes table there16:38
dansmithand that if you look, these two compute nodes are pointing at cell0 and not the proper cell, which explains your other issues16:39
sean-k-mooneyif it actully got that far it proably create teh placement RP which could cause name collisions if you repoint it ot cell116:39
dansmithso can you check compute_nodes in cell0, and also check to see if there are any instances in cell0 (other than ones in the build state)16:39
dansmithsean-k-mooney: yes and atmark has already deleted some of that, so we have more work to cleanup there16:40
sean-k-mooneyack16:40
atmarkdansmith: i do have those two computes in nova_cell0 16:43
dansmithatmark: okay, now things are starting to make sense :)16:43
auniyalsean-k-mooney hopefully you noticed my request in meeting16:45
atmarkcompute_tables in nova_cell0 https://paste.openstack.org/show/bYtEaxBqcGpSkDhThQY2/16:45
atmarkcompute_nodes*16:45
auniyalif not please have a look when you have time16:45
dansmithatmark: aha, there's the missing bf7* uuid16:45
sean-k-mooneyauniyal: you asked baout https://review.opendev.org/c/openstack/nova/+/885344? but it already has a +2w form dansmith 16:45
sean-k-mooneywas there something else16:46
auniyalyeah just noticed thanks dansmith16:46
auniyalcan you please look for others in topic16:46
sean-k-mooneythe others are in other branches16:46
sean-k-mooneyso after it has merged i can perhaps take another look16:47
atmarkdansmith: bf7* is new resource provider id for compute01. it used to be c017*  16:47
auniyalyes same patch backport for other stable branches16:47
sean-k-mooneythere is noting to do until the zed patch merges16:47
auniyaltill ussuri16:47
dansmithatmark: it's the one I was looking for from the compute logs.. but it needs to be deleted and rebuilt entirely at this point16:47
auniyalif you ad your +2 I'll look for seond later16:47
sean-k-mooneyya so we can look at https://review.opendev.org/c/openstack/nova/+/885345 next but it will fail in the gate until the zed one is merged16:48
dansmithatmark: so before we proceed, we need to check the instances table in there16:48
sean-k-mooneymaybe tomorrow can you ping me in the morning16:49
atmarkok, one sec16:49
auniyalack,16:49
auniyaltanks16:49
sean-k-mooneyi dont have time to load context and review before i finish for today but i can look at thing in the morning16:49
atmarkdansmith: cell0's instances table https://paste.openstack.org/show/bZuA2J0x09iTDI5JDmKH/ 16:50
dansmithatmark: okay excellent16:50
dansmiththat is both good and expected16:50
dansmithatmark: so first thing to do is figure out what's wrong with those two computes. my guess is that during the upgrade, they got their transport_url pointed at the cell0/superconductor instead of the cell conductor16:51
atmarkI see find that in nova.conf right ?16:53
dansmithyep16:53
atmarkthe transport_url  in nova.conf  points to rabbitmq 16:54
opendevreviewDan Smith proposed openstack/nova master: WIP: Warn if we find compute services in cell0  https://review.opendev.org/c/openstack/nova/+/89724616:55
dansmithatmark: right it always points at rabbit.. I'm not familiar with kolla but are you running two sets of conductors, one for the top-level and one for the cells?16:56
dansmitheasy way to tell is compare your transport_url to a compute that is not borked :)16:56
atmarkin the controller i can see transport_url points to nova_cnco1 vhost of rabbitmq16:56
dansmithstart with the transport_url on the computes, and compare one of the broken ones to a non-broken one16:56
atmarkdansmith: ok, the transport_url on broken computes doesn't have nova_cnco1 vhost 16:57
dansmithokay, that's the problem, so fix that and restart nova-compute there16:58
atmarkjust a sec17:01
atmarkdansmith: done17:14
atmarktook a while, had to the push  change  through kolla-ansible 17:15
dansmithatmark: okay is it happier on startup now? there might be more work to clean up, but it might also have just worked it out17:15
dansmithby it I mean the compute service in its log17:15
dansmithnow that they're not pointing at the wrong transport (you fixed both right?) you can clear the service and node records from cell0's database which should resolve the service list duplication17:16
dansmithif the computes are happy in the log, you probably need to do the placement heal part now17:16
atmarkyes, both are fixed. Still seeing  `Failed to retrieve allocations for resource provider c017*` on compute01's nova-compute.log https://paste.openstack.org/show/b8j8iC0eV6TOK1QLVBfN/17:18
atmarkis that expected?17:19
dansmithatmark: yeah, so what happened is you deleted some stuff, the confused computes re-created their providers with the wrong uuid (the new one in the cell0) and now that it's back, it wants to use/recreate the old one17:19
dansmithatmark: so probably best to nuke the allocations *and* the RP for each, then run the heal and then get them started back up17:20
dansmithit's why I said "you shouldn't have deleted the allocations" earlier, despite the doc saying it's appropriate in some circumstances17:20
dansmithbut it's cool, this is confusing and messed up obviously :)17:20
atmarkclear the service and node records from cell0's database - I'm safe to remove these records https://paste.openstack.org/show/bxfAMF8OBcUCz2GsHiin/ ? 17:24
atmarkshould I remove entry or just update the deleted field ? 17:26
dansmithatmark: you can just set the deleted field to id if you want (that's what service delete does) but really they should go away.. but fine to just mark as deleted to test if you want17:26
opendevreviewDan Smith proposed openstack/nova master: WIP: Warn if we find compute services in cell0  https://review.opendev.org/c/openstack/nova/+/89724617:40
atmarkdansmith: For compute25 - I didn't have to do anything. compute01 - I deleted the RP, restarted nova-compute and then ran heal so issue is now resolved. I learned something new.  Thanks a lot. 17:47
dansmithatmark: awesome, glad to hear it.. so you cleaned up the service *and* node records from cell0 right?17:47
atmarkIf I unset alloactions instead of alloactions, will I face the same issue? https://docs.openstack.org/nova/latest/admin/troubleshooting/orphaned-allocations.html#solution 17:47
atmarkdansmith: yes, I decided to remove the records instead of updating deleted field. 17:48
dansmithcool17:48
dansmithatmark: not sure what you're asking about. unset .. "allocations instead of allocations"17:48
atmark"allocation unset instead of allocation remove"17:49
dansmithatmark: no I think it would have ended up the same way17:50
atmarkis it safe to remove these records in instances table in nova_cell0? https://paste.openstack.org/show/bP2VO4c7Gh6Tb4SIm5Wi/17:56
dansmithyes on the compute nodes.. the instances were all deleted in the last dump, so they belong there and will be cleaned after an archive17:59
dansmithinstances *do* belong in cell0, if they failed the schedule17:59
atmarkgot it. thanks again 18:10
opendevreviewMerged openstack/nova stable/zed: Fix failed count for anti-affinity check  https://review.opendev.org/c/openstack/nova/+/88534418:19
opendevreviewDan Smith proposed openstack/nova master: WIP: Warn if we find compute services in cell0  https://review.opendev.org/c/openstack/nova/+/89724619:30

Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!