Tuesday, 2025-05-20

bauzassean-k-mooney: yeah, apparently the mirrors were fixed06:56
bauzasthat's why I uploaded again my series (because I don't have the same kernel version (I use a virtual host)06:56
bauzasand yeah, bindep is not installing automatically mdevctl06:57
opendevreviewSylvain Bauza proposed openstack/nova master: WIP : Add mtty support to nova-next  https://review.opendev.org/c/openstack/nova/+/92214007:50
bauzassean-k-mooney: if you're here, I'm curious why in nova-next, compute1 doesn't get my series https://9eed2832462b93d99d63-9a2c0e6b238d9eb37d7a57d0c5074dee.ssl.cf1.rackcdn.com/openstack/93034cf7903d4f8abe5fa23883c761b8/compute1/logs/screen-n-cpu.txt10:37
bauzasthe controller uses it correctly : https://9eed2832462b93d99d63-9a2c0e6b238d9eb37d7a57d0c5074dee.ssl.cf1.rackcdn.com/openstack/93034cf7903d4f8abe5fa23883c761b8/controller/logs/screen-n-cpu.txt10:38
sean-k-mooneythe same git repo will be copied to all node by the workspace prepare but my guess is you have not configured something correctly in the devstack config10:39
sean-k-mooney mel had to fix the job config to turn on the compilation fo the sampels in the comptue node for example https://review.opendev.org/c/openstack/nova/+/948581/7/.zuul.yaml10:40
sean-k-mooney bauzas  we can confirm if it clooned the corrct content in the logs lets dot that fist but i think that is highly unlikely to be the problem here10:43
bauzasdoh, I'll then set the compilation in the subnode section10:45
bauzassean-k-mooney: fwiw, I just updated my WIP to install mdevctl10:45
bauzasso melwitt's series shouldn't be needed10:45
bauzass/series/changre10:45
sean-k-mooneyack. she was working on that because you wre on pto to see if she could move it forward but she didnt intent to keep working on it. she just didnt want to directly update your patch10:46
sean-k-mooney2025-05-20 08:30:05.668555 | compute1 | + functions-common:git_clone:676           :   cd /opt/stack/nova10:50
sean-k-mooney2025-05-20 08:30:05.673150 | compute1 | + functions-common:git_clone:677           :   git show --oneline10:51
sean-k-mooney2025-05-20 08:30:05.673265 | compute1 | + functions-common:git_clone:677           :   head -110:51
sean-k-mooney2025-05-20 08:30:05.691351 | compute1 | 6e752ff1c3 WIP : Add mtty support to nova-next10:51
sean-k-mooney2025-05-20 08:30:05.695399 | compute1 | + functions-common:git_clone:678           :   cd /opt/stack/devstack10:51
sean-k-mooney2025-05-20 08:30:05.699127 | compute1 | + lib/nova:install_nova:984                :   setup_develop /opt/stack/nova10:51
sean-k-mooneyso on the compute its definetly usign your change10:51
sean-k-mooneydevstack git clone say the git repo already existed (because it was prepared by zuul) and skip the clone and just used it10:51
sean-k-mooneybauzas: i guess that may be an issue with the error message 10:53
sean-k-mooneyi.e. if it returns https://9eed2832462b93d99d63-9a2c0e6b238d9eb37d7a57d0c5074dee.ssl.cf1.rackcdn.com/openstack/93034cf7903d4f8abe5fa23883c761b8/compute1/logs/screen-n-cpu.txt10:53
sean-k-mooneywhen the mtty device is not found10:53
sean-k-mooneyits conflating incorrect or invalid with not found so we can proably make that more obvious10:53
sean-k-mooneybauzas: anyway let see if enaableign the plugin on the comptue helps and if that resolves teh error10:56
sean-k-mooneyi can take a look again when we have new results10:56
opendevreviewSylvain Bauza proposed openstack/nova master: WIP : Add mtty support to nova-next  https://review.opendev.org/c/openstack/nova/+/92214012:18
bauzasstupid me again, forgot to upload it12:18
opendevreviewLajos Katona proposed openstack/os-vif master: VS Trunk: Add bridge_name to external_ids  https://review.opendev.org/c/openstack/os-vif/+/94973613:16
dansmithbauzas: gibi: Can you guys fast approve this trivial revert of the glance location thing? https://review.opendev.org/c/openstack/nova/+/950336?usp=email13:45
dansmithas I suspected, not enough testing on the glance side was actually done and this is triggering a behavioral change and tempest fails13:45
sean-k-mooneyi can do it fi you want13:46
dansmithyep, thanks13:46
bauzas+Wd too13:56
opendevreviewTakashi Kajinami proposed openstack/nova-specs master: Re-propose "libvirt: AMD SEV-ES support" for 2025.2  https://review.opendev.org/c/openstack/nova-specs/+/95042914:25
opendevreviewTakashi Kajinami proposed openstack/nova master: libvirt: Allow using the system page (4KiB)  https://review.opendev.org/c/openstack/nova/+/95043014:37
UgglaNova meeting in ~55mn15:05
*** ralonsoh is now known as ralonsoh_out15:08
UgglaNova meeting in ~5mn15:55
Uggla#startmeeting nova16:01
opendevmeetMeeting started Tue May 20 16:01:41 2025 UTC and is due to finish in 60 minutes.  The chair is Uggla. Information about MeetBot at http://wiki.debian.org/MeetBot.16:01
opendevmeetUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.16:01
opendevmeetThe meeting name has been set to 'nova'16:01
UgglaHello everyone16:01
gibio/16:02
dansmitho/16:02
gmaano/16:02
masahitoo/16:02
bauzaso/16:02
fwieselo/16:02
Uggla#topic Bugs (stuck/critical) 16:03
elodilleso/16:04
Uggla#info No Critical bug 16:04
Uggla#info https://bugs.launchpad.net/nova/+bug/2110545 from last week.16:04
Uggla#info We merged https://review.opendev.org/c/openstack/nova/+/949622 proposal to skip the the problematic job.16:04
gibiwe probably need to decrease the severity of it now that it is not blocking the gate16:04
gibimoved it to High now16:04
Ugglaok I was about to ask about it16:05
Ugglahttps://review.opendev.org/c/openstack/nova/+/922140 bauzas working on a patch to enable nova-next.16:05
Ugglabauzas, am i correct with ^ ?16:05
bauzasI still need to understand why compute1 doesn't like me16:05
bauzasbut let's not discuss my problem now16:05
bauzasI would appreciate some devstack/zuul expert to help me finding the cause that's it16:06
Ugglacool, it is on a good path to be resolved.16:07
bauzastldr: multi-node's compute1 apparently doesn't get the right repo branch 16:07
Ugglabauzas, anything else you want to add ?16:07
Ugglaseems not so moving on16:08
Uggla#topic Gate status 16:08
Uggla#link https://bugs.launchpad.net/nova/+bugs?field.tag=gate-failure Nova gate bugs 16:08
Uggla#link https://etherpad.opendev.org/p/nova-ci-failures-minimal16:08
Uggla#link https://zuul.openstack.org/builds?project=openstack%2Fnova&project=openstack%2Fplacement&branch=stable%2F*&branch=master&pipeline=periodic-weekly&skip=0 Nova&Placement periodic jobs status16:08
Uggla#info Please look at the gate failures and file a bug report with the gate-failure tag.16:09
bauzasUggla: sorry, nope16:09
Uggla#info Please try to provide meaningful comment when you recheck16:09
Uggla#topic tempest-with-latest-microversion job status 16:09
Uggla#link https://zuul.opendev.org/t/openstack/builds?job_name=tempest-with-latest-microversion&skip=016:10
Ugglagmaan, do you want to say something about it ?16:10
gmaanone update, keypair test passing. I was working for server tags tests which need more update as its schema missing since starting16:11
gmaanit in-progress https://review.opendev.org/c/openstack/tempest/+/94968216:11
gmaansorry instance action 16:11
gmaanthat is all for this week16:11
Ugglagmaan, thx16:11
Uggla#topic Release Planning 16:11
Uggla#link https://releases.openstack.org/flamingo/schedule.html16:11
Uggla#info Nova deadlines are set in the above schedule 16:12
Uggla             #info Only 2 weeks before Nova Spec Soft Freeze, do not forget to submit your specs. 16:12
Uggla#topic Review priorities 16:12
Uggla#link https://etherpad.opendev.org/p/nova-2025.2-status16:12
UgglaFYI I have requested #openstack-nova channel ops for sean-k-mooney, bauzas, gibi, dansmith, melwitt, Uggla16:12
gibithanks116:13
sean-k-mooneyack. i suspect we will very rarely need to use that16:13
sean-k-mooneybut it would have been nice to have been able to fix the topic before16:13
UgglaI think we should start reviewing and approving specs.16:14
gibiyeah it is only for the topic16:14
bauzasnoted16:14
UgglaLet's start with : https://review.opendev.org/c/openstack/nova-specs/+/945549, https://review.opendev.org/c/openstack/nova-specs/+/947542, https://review.opendev.org/c/openstack/nova-specs/+/94950416:15
UgglaThe 2 last ones are reproposal, so it will be glad to approved them.16:15
UgglaThe first one is about cloud hypervisor. It was reviewed by Dan16:16
UgglaBut that would cool to have feedbacks from another core.16:16
dansmithfwiesel: I reviewed that spec, but I'm not going to approve it, FWIW16:16
sean-k-mooneyi pushed my pending comments...16:16
dansmithoops, sorry Uggla: ^16:16
sean-k-mooneyi really need to stop forgetign to do that16:17
dansmithI don't have time to really review the implementation in detail and so I don't want to be the +2 on the spec16:17
sean-k-mooneyif i recall correctly they have set up a ppa to rpovide the libvirt integration.16:17
Ugglacool, the author pinged me and was worried, so that's cool if he has new feedback.16:17
sean-k-mooneythat solves part of the how can we test this in ci aspect16:17
sean-k-mooneyi think were we are right now is16:19
sean-k-mooneywithout seeign the code and some work towards the integration16:19
UgglaAlso FYI I think the document is up2date regarding the current spec status, so feel free to pick specs for review.16:19
sean-k-mooneywe cant really make much more of a judvemnt call on the desgin16:19
masahitohi, we started to update the supporting trait tracking by the provider.yaml feature nova-spec with PoC. https://review.opendev.org/c/openstack/nova-specs/+/93758716:19
sean-k-mooneyso we coudl approve the spec btu we cant really commit to landign the feature until they start building it out and imporantly the devstack supprot and ci for the same16:20
Ugglasean-k-mooney, sure I think approving the spec is good. So author can "check" that box.16:21
sean-k-mooneyUggla: the vtpm spec is on my radar to review this week. if the cinder one actully makes progress i dont have issues with that either but i have not reviewd it since last cycle. im not sure if we really want to go into details on those in real time16:21
gibiis it a general rule that we should not +2 a spec if we are not commiting to review the impl?16:21
* gibi trying to figure out how to vote 16:22
sean-k-mooneyin the past no.16:22
dansmithsean-k-mooney: agreed, I think we need to see a little sniff of the implementation in CI16:22
dansmithgibi: not a hard rule, I just think for something like this I'm going to avoid putting my name on such an effort16:22
sean-k-mooneybut there is at least some expecation i guess that if you do +2 and the authour has questions you will at least try to guide them16:22
gibidansmith: ack16:22
sean-k-mooneydansmith: i thinik if you know you wont have time then signaling that is also valuable16:23
gibisean-k-mooney: yeah by voting on the spec I agree I will be around to answer questions about the spec16:23
UgglaSomething to add ?16:25
sean-k-mooneyi think its good to call out the spec that are ready for review but i think we need to actully do the review async16:25
sean-k-mooneyso unless you have specific question i would suggest we move on and if we want to have a spec review day16:25
gibi+116:26
sean-k-mooneywe can do that but do that at/around the soft spec freeeze in  june16:26
Ugglasure this topic is just to highlight that specs are waiting and that we should start to review them not to be overwhelmed.16:26
sean-k-mooneymasahito: your sepc also has a poc implemantion if i recall correct16:27
sean-k-mooneyi belive gibi has alrady started to look at it too16:27
masahitoyup.16:27
masahitois it okay to add it into the priority?16:27
sean-k-mooneymasahito: since you are here did you have any questions for us that you wanted input on 16:27
sean-k-mooneyyou can certenlly put it in the etherpad as ready for review if you think it is16:28
gibisean-k-mooney: yeah I looked at the PoC and I liked what I saw16:28
masahitonothing now. i think the spec and poc need first review round now.16:28
sean-k-mooneygibi: ack i opened it but didnt get to read it yet. masahito: ack16:28
Ugglaand so moving on.16:29
Uggla#topic OpenAPI 16:29
Uggla#link: https://review.opendev.org/q/topic:%22openapi%22+(project:openstack/nova+OR+project:openstack/placement)+-status:merged+-status:abandoned16:29
sean-k-mooneymasahito: please add it here https://etherpad.opendev.org/p/nova-2025.2-status#L3416:29
Uggla21 patches remaining.16:29
sean-k-mooneyactully slightly more16:30
sean-k-mooneystephen as a branch on his github fork with all of them16:30
sean-k-mooneybut is propsoing them in batches. i was intended to review them today but didnt get to it so i will be reviewign them tomrrow16:30
masahitosean-k-mooney: got it. thanks16:31
sean-k-mooneyim hoping gmaan  or other will also have time in the next week or two to reviwe some of them16:31
UgglaThis topic is to follow progress on openapi. Because we said in PTG we would try to close it.16:31
gmaanyeah, I will try if I can do during end of this week otherwise next week sometime16:31
UgglaI have reviewed 2 of them.16:32
gmaanI would not be able to review all of them together as it exhaust eyes checking all fields for many APIs :) but I will target some set 16:32
Ugglagmaan, I expect to progress of them week after week.16:33
sean-k-mooneyya i can ususlly do 5-10 over the course of an hour or two and then i need to clear my mind16:33
sean-k-mooneythey are not hard persay but its hard to maintain concentration on them for long periods of time16:33
gmaanyeah16:33
Ugglayep my idea is to do a first pass on them and then if gmaan and sean-k-mooney you can review and merge that would be great.16:34
Ugglaany as time is flying, I'd like to move on the next topic.16:35
Uggla#topic Stable Branches 16:35
Ugglaelodilles, the floor is yours.16:35
elodillesack, thanks, not so much things this time16:35
elodilles#info stable branches (stable/2025.1 and stable/2024.*) seem to be in OK state16:36
elodilles#info stable branch status / gate failures tracking etherpad: https://etherpad.opendev.org/p/nova-stable-branch-ci16:36
elodillesthat's all from me16:36
elodillesUggla: back to you16:36
Ugglathx elodilles 16:36
elodillesnp16:36
Uggla#topic vmwareapi 3rd-party CI efforts Highlights16:37
Ugglafwiesel, anything to say ?16:37
fwieselHi, no updates from my side.16:37
Ugglaok16:37
Uggla#topic Gibi's news about eventlet removal. 16:37
Uggla#link Series: https://gibizer.github.io/categories/eventlet/16:37
Ugglagibi, do you want to say something ? I think you are chasing for reviews.16:38
gibiyepp 16:38
gibithe nova-scheduler series is ready for core review16:38
gibi(I've added it to the status etherpad as well)16:39
gibithe series starts here https://review.opendev.org/c/openstack/nova/+/94796616:39
gibithe first 11 patches are ready to go, the last two depends on oslo.service to land and release the threading backend support16:39
gibithat is it16:40
Ugglacool thx gibi 16:40
Uggla#topic Open discussion16:41
UgglaI can see topic on the agenda.16:41
Ugglas/can/can't/16:41
Ugglaso unless if we have points, maybe we can try to triage bugs.16:42
Uggla#topic Bug scrubbing 16:43
Uggla#link: https://etherpad.opendev.org/p/nova-bug-selection-for-triaging#L416:43
UgglaFirst one : https://bugs.launchpad.net/nova/+bug/211022216:44
UgglaNova server image uploaded to S3 uses singlepart instead of multipart upload16:45
gibithat feels like a feature request16:45
Ugglayep I was about to say the same.16:45
Ugglait is more an optimization request.16:46
UgglaSo I could probably set the bug to invalid and ask for a blueprint.16:47
gibiit is a small feature even a specless one if we can assume that we can pass the size to the glance client at uploat16:47
Ugglaok I'll do that for this one.16:48
UgglaNext one:     https://bugs.launchpad.net/nova/+bug/2109727 - removing an host from an aggregate should be accepted even if the host no longer exists16:48
UgglaI think it is a valid one as opened by Sylvain16:48
Ugglabauzas, ^16:50
gibiI hope bauzas could push a functional reproducer for that16:51
bauzassorry was taking a break16:51
bauzasnot sure if I have time for the reproducer but I can try16:52
sean-k-mooneybauzas: i guess that only matteres if your going to work on the bug16:52
Ugglabauzas, at least this is something valid.16:52
sean-k-mooneyif not then the peroson who picks it up can16:52
UgglaSo I'll move the bug to valid.16:54
UgglaNext one :     https://bugs.launchpad.net/nova/+bug/2108974 - Keypairs lost during cross-cell resize in instance_extra16:54
opendevreviewMerged openstack/nova master: Revert "Support glance's new location API"  https://review.opendev.org/c/openstack/nova/+/95033616:54
gibiit has reproducer steps so somebody should run them and if reproducible mark it valid16:55
UgglaIs resizing cross-cell supported ?16:56
dansmithyeah16:56
dansmithit uses shelve on the backend16:57
dansmithand it has to copy between databases, so it might be missing something16:57
gibiThe config drive associated with the server, if there is one, will be re-generated on the destination host in the target cell. Therefore if the server was created with personality files they will be lost. However, this is no worse than evacuating a server that had a config drive when the source and destination compute host are not on shared storage or when shelve offloading and unshelving a server 16:58
gibiwith a config drive. If necessary, the resized server can be rebuilt to regain the personality files.16:58
gibiso maybe such config drive regen is the problem16:59
gibi(the above is from https://docs.openstack.org/nova/latest/admin/configuration/cross-cell-resize.html#limitations )16:59
dansmithah yep16:59
gibithe bug says it is lost form metadata response as well17:01
Ugglayep I understand it is not only config drive.17:01
gibiand it also points to `select keypairs from dst_cell.instance_extra ` I did not know we have a copy of a keypair there17:01
gibibut it says it is in the source cell17:02
gibiso that is probably not copied to the dest cell DB17:02
gibiI vote that this is valid17:02
UgglaI can try to reproduce, but I'm unsure how to easily deploy a multi cell env.17:02
gibiwe have multicell job we just need to see if we can enable some tempest there to catch this17:03
sean-k-mooneythats in the sepc for what its porth17:04
sean-k-mooneyat least im 99% sure we called out that any files injected in the config drive would be lost with a cross cell resize17:05
sean-k-mooneyhttps://specs.openstack.org/openstack/nova-specs/specs/ussuri/implemented/cross-cell-resize.html#known-issues17:06
sean-k-mooneylimiation 317:06
sean-k-mooneyServers created with personality files, commonly known as file injection, that are resized across cells will lose the personality files since they are not persisted in the database17:06
gibisean-k-mooney: keypairs are not personality files17:06
gibiafaik17:06
dansmithcorrect17:06
dansmithbut they mentioned instance_extra17:07
dansmithso if there's something we're not copying there and then regen'ing the configdrive...17:07
sean-k-mooneyoh it the keypari17:07
sean-k-mooneyok that shoudl be copied and regenerated17:07
sean-k-mooneyi actully otught keyparis were in teh api db17:08
sean-k-mooneynot the cell db but i guess we embded them when we first create teh instance in the cell db?17:08
dansmithkeypairs themselves are I think (maybe) but the instance's attachment to them is in the main one of course17:08
sean-k-mooneyack17:08
sean-k-mooneyim wondering if we could see this in ci. i think this is valid in anycase17:09
sean-k-mooneywe could create a repodcuer functional tests. in thoery tempest coudl detect this also17:10
sean-k-mooneywe just need to curl the metadata after the resize17:10
sean-k-mooneyor check the config drive i guess17:10
sean-k-mooneypresumable it workign now because the keypari is already in the authorized keys17:10
Ugglasean-k-mooney, I'd like to try to do a reproducer.17:11
gibiyepp we should be able to catch this in tempest17:11
sean-k-mooneythe workaround for now is just pass the keypari again on rebuild i guess17:12
UgglaSo if you all agree I will set this bug to valid and I'll try to do a reproducer. But I may need your help.17:12
sean-k-mooneyack, sure shout if you hit issues17:12
UgglaI will also update the bug with the workaround and remind limitation.17:13
UgglaWe are overtime, so I think we can wrap for today.17:14
sean-k-mooneyim not sure if we have multi cell functional tests today so that could be "fun" to make work17:14
sean-k-mooneyreproducing it in tempest will be simpler in this specific case.17:14
sean-k-mooneyoh we do https://github.com/openstack/nova/blob/master/nova/tests/functional/test_cross_cell_migrate.py17:15
dansmithmatt riedemann did that work, so I was *sure* there was functional testing for it :D17:15
sean-k-mooneyack. getting diffent config to work in the fucntionla tests is still a bit fo a pain so i was unsure if we did or not17:16
Ugglaanything else ?17:17
UgglaThanks all and thanks for the extended time for bug triage.17:17
Uggla #endmeeting17:17
Uggla#endmeeting17:17
opendevmeetMeeting ended Tue May 20 17:17:40 2025 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)17:17
opendevmeetMinutes:        https://meetings.opendev.org/meetings/nova/2025/nova.2025-05-20-16.01.html17:17
opendevmeetMinutes (text): https://meetings.opendev.org/meetings/nova/2025/nova.2025-05-20-16.01.txt17:17
opendevmeetLog:            https://meetings.opendev.org/meetings/nova/2025/nova.2025-05-20-16.01.log.html17:17
elodillesthanks o/17:18
masahitothank17:19
gibiUggla: thanks17:21
Ugglayou are welcome.17:21
sean-k-mooneyUggla: so https://github.com/openstack/nova/blob/master/nova/tests/functional/test_cross_cell_migrate.py#L816-L826 test_cold_migrate_target_host_in_other_cell17:24
sean-k-mooneyget you to the point where the vm has be moved to a differnt cell 17:24
sean-k-mooneyyou shoudl be able to use that as a baseline to create a repoducer17:24
Ugglaok but it is a fn test right ?17:25
UgglaYou have said that a tempest one should be easier.17:25
UgglaAm i wrong ?17:25
sean-k-mooneythat was assumign we did nto have functional test coverage for this which we do17:26
sean-k-mooneyso it should be pretty simple to extend since all the infra is in place17:26
Ugglaok I will try to reproduce with the fn ones though.17:27
sean-k-mooney_resize_and_validate accepts a server and returns server, source_rp_uuid, target_rp_uuid, old_flavor, new_flavor17:27
sean-k-mooneyso you can create a server with a key pair17:27
sean-k-mooneyassert that its in the first cell db17:28
sean-k-mooneythen call that to move it17:28
sean-k-mooneyand then check the second db17:28
sean-k-mooneyi think that woudl work17:28
Ugglasean-k-mooney, OK I'll try.17:29
*** jcosmao is now known as Guest1642319:26

Generated by irclog2html.py 4.0.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!