Wednesday, 2026-04-22

opendevreviewmelanie witt proposed openstack/nova master: Increase wait retries for resize reschedule functional test  https://review.opendev.org/c/openstack/nova/+/98574100:08
melwittgmaan: just an update, the graceful shutdown was unrelated and was just the test cleanup happening after the wait for server state failed ^00:09
gmaanmelwitt: ohk, thanks for fix, +201:00
melwittthanks gmaan 01:01
*** bauzas4 is now known as bauzas02:07
opendevreviewRico Lin proposed openstack/nova master: compute: add [compute] skip_inventory_in_use_errors  https://review.opendev.org/c/openstack/nova/+/98575006:25
opendevreviewRico Lin proposed openstack/nova master: compute: add skip_inventory_in_use_errors opt  https://review.opendev.org/c/openstack/nova/+/98575006:26
opendevreviewMerged openstack/nova master: Increase wait retries for resize reschedule functional test  https://review.opendev.org/c/openstack/nova/+/98574108:01
elodillesbauzas Uggla : i know this is PTG week, but i've prepared nova release patches, mainly because of we could do a final 2024.2 Dalmatian release before it moves to EOL. could you please review them? https://review.opendev.org/q/project:openstack/releases+is:open+intopic:nova09:02
UgglaHi elodilles, I'm gonna have a look09:03
elodillesthx in advance o/09:06
opendevreviewribaudr proposed openstack/nova master: ironic: Optimize get_num_instances to avoid unnecessary API calls  https://review.opendev.org/c/openstack/nova/+/98578509:26
bauzaselodilles: ack, will look09:32
elodillesthanks, too o/09:48
Ugglabauzas, elodilles, if you can have a look at 985791: Add Nova deadlines to the 2026.2 Hibiscus schedule | https://review.opendev.org/c/openstack/releases/+/985791  (no hurries)10:15
elodillesUggla: ack, will do10:29
opendevreviewsean mooney proposed openstack/nova master: [dnm] testing with python 3.14  https://review.opendev.org/c/openstack/nova/+/98580211:22
sean-k-mooneyUggla: gmaan sambork  so i hacked up some py 3.14 testing yesteday and got some interesting result on https://review.opendev.org/c/openstack/devstack/+/98573511:23
sean-k-mooneywe can boot vms but some things fail11:24
sean-k-mooneyfor example some of the novnc tests11:24
sean-k-mooneythe dnm to nova above shoudl give us better results fro our edgecases11:24
sean-k-mooneybasicly i have a WIP serise for devtack to use uv to provide python and replace pip with uv pip11:25
sean-k-mooneywhat that allows use to do is use basicly any python verion on any distro within reason11:25
sean-k-mooneyso with that dnm all our jobs regrdless of the disto will run on python 3.14 and give ues a good overviewr of compatiablity 11:26
sean-k-mooneygibi: tkajinam sambork  so the novnc issue iw the pikel issue we were discussing in the eventlet removeal room https://zuul.opendev.org/t/openstack/build/6f2857af77aa41eabb9f28a077081d89/log/controller/logs/screen-n-novnc-cell1.txt#1611:27
sean-k-mooneywe can see that tempest.api.compute.volumes.test_attach_volume.AttachVolumeMultiAttachTest.test_snapshot_volume_backed_multiattach and other complex test passed so for the most part we seam to be in a relitively good place11:32
samborknice findings, I'm curious if this will pass with threading mode (because patch for proxies runing with threading are still under review https://review.opendev.org/c/openstack/nova/+/976089)11:55
sean-k-mooneyyou could add a second dnm on top of your patch11:55
sean-k-mooneyyou just need to depend on the devstack dnm11:55
sean-k-mooneybut ya im not sure if the pikel issue is eventlet related or not11:56
sean-k-mooneysambork: or actully once my currnt dnm reports back11:56
sean-k-mooneyi can rebase it on yoru review11:56
sean-k-mooneyi just eamiled the list so other project can tyr this themselve if they want11:57
elodillesUggla: Nova deadlines patch LGTM, i've +2'd it >>> https://review.opendev.org/c/openstack/releases/+/98579112:02
elodillesbauzas Uggla : i've corrected the version bump on the epoxy release patch, can you please revisit that? https://review.opendev.org/c/openstack/releases/+/98565112:03
sean-k-mooneyelodilles: im not sure the ptg dicussion for that has happend yet https://etherpad.opendev.org/p/nova-2026.2-ptg#L17912:03
sean-k-mooneyelodilles: if it did there were not any notes taken12:03
sean-k-mooneyi left some comment on it yesterday for whenever that happens12:04
bauzaselodilles: ack12:04
elodillessean-k-mooney: it was discussed on Monday12:04
elodillessean-k-mooney: and the notes are at the end of the lines o:)12:04
sean-k-mooneyoh well no agreement was recrored12:04
sean-k-mooneythe light green?12:05
elodillesyes12:05
sean-k-mooneybecause that just loosk liek comemnt folks had added12:05
sean-k-mooneywiht no agreement recorreded12:05
elodillesindeed, it is not commented at 'AGREED' section, but otherwise based on the discussion the commented dates were agreed, but correct me Uggla if i'm wrong 12:07
sean-k-mooneyack i was questioning the spec deadline12:07
sean-k-mooneybut im not agents the dates over all12:07
sean-k-mooneywe had less time for it spec last cycle (i..e it was before m2) so this proposal is just repeating that that patten again this cycle12:08
sean-k-mooneybut it effectily put the spec deadlien at the half way point of the cyle instead of 2/3rd 12:09
sean-k-mooneywhich is fine as long as we also apply that to blueprint and we adverise ti to folks early12:09
elodillessean-k-mooney: there were no hard opinions afair, so maybe this can be negotiated further12:10
sean-k-mooneythe only real point i wanted to contibut is spec and blueprint approve deadline shoudl be the same, and tell peopel early because several contibutors in other poejct that wanted to do cross project things with nova didnt knwo we had the deadline in decemeber for it last cycle12:16
sean-k-mooneyi had to tell multiple peopel that they missed the spec deadlien in late decemebr early and early january because they were exectign m212:16
sean-k-mooneyso as long as we tell peopel in advance then thats fine. just uodatign the secheudler isnt enouch to "tell peopel" becausemost project dont recrod the deadliens in the schdule and dont expect that to be where its comunicated12:17
bauzasUggla: gentle reminder that I won't be able to join the nova sessions until 3pm UTC (hopefully), maybe a bit later12:18
bauzassean-k-mooney: feel free to chime in https://review.opendev.org/c/openstack/releases/+/98579112:18
Ugglabauzas 👍12:18
*** ykarel_ is now known as ykarel13:29
opendevreviewTakashi Kajinami proposed openstack/nova-specs master: libvirt: AMD SEV-SNP support  https://review.opendev.org/c/openstack/nova-specs/+/98337614:44
opendevreviewTakashi Kajinami proposed openstack/nova-specs master: libvirt: AMD SEV-SNP support  https://review.opendev.org/c/openstack/nova-specs/+/98337614:47
tkajinammhen, ^^^ I've updated the spec to capture the firmware descriptor hack15:16
tkajinamit seems that the required content is available in c10s and even c9s (!!!) while it's not in ubuntu 24.04 or even 26.04 :-(15:17
opendevreviewsean mooney proposed openstack/nova master: [dnm] testing with python 3.14  https://review.opendev.org/c/openstack/nova/+/98580215:22
tkajinamoh I didn't notice that I've been holding the moderator role16:07
tkajinam(probably I got it because I joined first today...16:07
clarkbyes the first person to join gets it. You can give it to someone else16:07
tkajinamI hope I transferred it to Uggla correctly ...16:08
Ugglatkajinam seems ok thanks16:08
tkajinamUggla, is it still continued, right ?16:09
Ugglatkajinam yep no pb. 16:09
tkajinamI left the meeting and saw message saying "the others are all leaving"16:09
tkajinamwhich scared me really ... :-(16:09
Ugglatkajinam no worries we are all in the room, it's working fine.16:09
tkajinamthat's good16:10
tkajinamleaving now. will try to attend the attestation session tomorrow (but no guarantee)16:10
Ugglatkajinam, thanks Takashi, yes please rest.16:10
opendevreviewMerged openstack/nova master: [py313-threading]Reenable last scatter-gather unit test  https://review.opendev.org/c/openstack/nova/+/98434916:31
opendevreviewMerged openstack/nova master: [py313-threading]Reenable test_show_simple_tenant_usage_policy  https://review.opendev.org/c/openstack/nova/+/98460616:32
melwittgibi: thank you for the review on https://review.opendev.org/c/openstack/nova/+/985741, I am very motivated to try and stabilize the CI 😆 17:28
gibimelwitt: I'm always happy to review stabiliziation patches17:29
stephenfingibi: I checked and yeah, I have a WIP commit from 22-Jan in my local oslo.service repo where I started adding a LauncherBase command (and some others) 😅17:30
gibi:)17:48
stephenfinAnd this is the first thing that comes to mind wrt the requested oslo.service issue https://review.opendev.org/c/openstack/oslo.service/+/985859 I'd prefer to rework this to make that more natural but it's emmintly doable17:51
* stephenfin calls it17:51
sean-k-mooneygibi: sambork so i said as much in the patch but the good news is the novnc proxy fails on 3.14 with eventlet but works in threaded mode18:22
melwittsean-k-mooney: this is random but I spent some time trying to get multinode ipv6 to work to look at ceph live migration, I made my own page bc I didn't want to mess yours up too much https://review.opendev.org/c/openstack/nova/+/984623 I come to the conclusion that this is known to not be possible (https://opendev.org/openstack/neutron/src/commit/2bc1bcc675ded515c8dd8da5e513010f6cb59794/zuul.d/tempest-singlenode.yaml#L744) but I don't18:31
melwitt understand why18:31
melwitton the subnode, the interface that's found to support ipv6 just will not route to the controller and I don't get why18:32
sean-k-mooneydont worrry about messing up ming beu totally fine to do it in your onw patch too18:32
sean-k-mooney ok so is the limitaiotn with ovn18:33
melwittthis isn't super important obviously but just wanted to let you know what I did haha18:33
sean-k-mooneywe could jsut swap to ml2 ovs18:34
sean-k-mooneylike the alt config job18:34
melwittsometimes ip route list finds host_ip_iface=enX0 sometimes it finds host_ip_iface=ens318:34
sean-k-mooneythat depens on the disto/kernle18:34
sean-k-mooneythe systemd/bios dev name stuff18:35
melwittah ok. yeah I ended up having to make DNM https://review.opendev.org/c/openstack/devstack/+/984605 to handle was it debian? I can't remember18:35
sean-k-mooneyyes we moved the ceph jobs ot debian 13 becuase of some qemu/ceph isues18:36
sean-k-mooneybasiclly to get newer versions with less bugs18:36
sean-k-mooneyso this is debian 1318:36
melwittok, so my next experiment can be configuring ml2 ovs in my dnm patch18:36
melwittthat's helpful to know, thanks18:36
sean-k-mooneyyou coudl do that or you could enable ceph in the nova alt config job18:36
sean-k-mooneybut eihter would worth a try18:37
melwittah ok18:37
sean-k-mooneythe alt config job is already ml2 ovs i bleive18:37
sean-k-mooneyas is nova next18:37
sean-k-mooneybut one has legacy ip tables and hte other the ovs firewall driver18:38
melwittgotcha18:38
sean-k-mooneymelwitt: the other thin we may or many not need too do is modify the multi node bridge18:41
melwittthat is uncharted territory for me but I will keep in mind18:42
sean-k-mooneymelwitt: if you have not looked at the multi node job in detail, it create linuxor ovs? briges on each of the hosts and and interconenct them with vxlan tunnels18:42
sean-k-mooneyso that we get a flat l2 network that we can shove vlans through18:42
sean-k-mooneywe might need to do somethign like assign ipv6 ips to that bridge and use those instead18:43
melwittis that something that's done in devstack?18:43
sean-k-mooneyyes18:43
sean-k-mooneywell no18:43
sean-k-mooneyin the devstack job18:43
melwittoh interesting18:43
sean-k-mooneyits done by a role in zuul-jobs18:43
melwittI'll go look at it18:43
melwittahhh roles, that explains why I never saw anything18:44
sean-k-mooneyhttps://opendev.org/zuul/zuul-jobs/src/branch/master/roles/multi-node-bridge18:44
sean-k-mooneyah its using ovs in the upstream verison i have a version of this that uses linux bridge 18:44
melwittneat18:44
sean-k-mooneyif you have ever looked at the multi node nodeset you will see https://github.com/openstack/devstack/blob/master/.zuul.yaml#L133-L14018:45
sean-k-mooneythe swtich is the contoler and the peers are the subnode18:45
sean-k-mooneythat use for cretating the tunnels and bridges18:45
sean-k-mooneyby https://opendev.org/zuul/zuul-jobs/src/branch/master/roles/multi-node-bridge18:46
sean-k-mooneyso its not a full mesh but a set of point to point lings between the conotler node and the rest18:46
melwittI never knew what that did. that's cool18:46
sean-k-mooneyso i dont recall all the history but i knwo we use that to test vlan network sbut i belive its orgial use was to make sure we just had l2 connectivity betweeen each fo the subnodes as we coudl not alwasy depend on that18:48
sean-k-mooneybut like this role in some from has existied since we use jenkins18:49
sean-k-mooneymelwitt: so just looking at the job run https://zuul.opendev.org/t/openstack/build/dcc395a124c04f4fbc3368ddf97400d3/log/job-output.txt18:49
sean-k-mooneywhat exactly was the issues the ovn setup ?18:50
sean-k-mooneybecasue the interfac echeted or soemthin else18:50
melwitttbh I don't really know other than the compute1 could not get to the /identity url on the controller. and I assumed that meant some aspect of networking was not connected somehow18:51
sean-k-mooney:) as a compute person blamin gnetworkign and storage first alwasy means less work18:51
sean-k-mooneybut ya ill take a look quickly and ssee if i can see teh actual failure18:52
melwittthe first problem was on compute1 it would never find a ipv6 interface because the awk command or whatever was not able to work with debian. so I made a devstack dnm patch for that. then it got farther to the point where compute1 is trying to do a very basic GET to /identity and it just hangs18:52
melwittthe second problem was compute1 tls cert was for the controller commonName so it did not work (the code simply copied controller's certs to subnode) so I added to my dnm patch to make the subnode actually make a cert for itself, for its commonName. then it got farther and then crapped out trying to GET /identity18:54
sean-k-mooneyso we have a role that does that copy18:54
melwittat that point I didn't know what to do18:54
sean-k-mooneybut we dont normally need a cert for the compute18:54
sean-k-mooneywe just need the ca18:55
sean-k-mooneyso it trust the contoler18:55
melwittI got a cert validation error when subnode tried to contact controller so that's why I did it. oh, ok18:55
melwittnot sure why what was there originally didn't work. maybe it was a one-off18:55
sean-k-mooneymaybe or maybe its because of how the name/ip is included in the cert18:56
sean-k-mooneylike if we are connetiv via the ipv6 adress rather then the name we might need to do soemthign differnt18:57
melwittyeah the complaint mentioned the commonName not matching or something like that18:57
sean-k-mooneyoh https://opendev.org/zuul/zuul-jobs/src/branch/master/roles/multi-node-hosts-file18:57
sean-k-mooneyso that only adds the ipv4 adresses to /etc/hosts18:57
sean-k-mooneyhttps://opendev.org/zuul/zuul-jobs/src/branch/master/roles/multi-node-hosts-file/tasks/main.yaml so maybe that a factor18:58
melwitthm ok18:58
sean-k-mooneywhat that does is loop over all the hsot in the inventory adn update /etc/host so that contoller maps to the correct ip18:59
sean-k-mooneythat may or may not be related jsut noticed the doc says its only for ipv4 which the code confirms18:59
melwittI see19:01
sean-k-mooneythisd does nto prove anyting but if you look at the zuul-info.* files https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_dcc/openstack/dcc395a124c04f4fbc3368ddf97400d3/zuul-info/index.html19:05
sean-k-mooneywhat i do notice is the compute seams to have much hihger latency then the contoler19:05
sean-k-mooneythey both have the saem default route19:07
sean-k-mooneybtu they took difent path back to open dev (ill ignore the fact the ipv4 trace rotes is actully v6 in both)19:07
sean-k-mooneywe also see fe80::ac5a:77ff:fe03:adb2 dev ens3 lladdr ae:5a:77:03:ad:b2 router DELAY  vs fe80::3457:e8ff:fe88:9852 dev ens3 lladdr 36:57:e8:88:98:52 router REACHABLE 19:08
sean-k-mooneyboth those got a routabel ip in teh same ipv6 subnet 2001:41d0:302:1000::cbc/56 s19:09
sean-k-mooneyso they shoudl be able to comunicate fine19:09
melwitt:/19:10
sean-k-mooneyhttps://[2001:41d0:302:1000::1ab]/identity19:10
sean-k-mooneyso that what its trying to conenct too19:10
sean-k-mooneybut that is not the adress of the contoler19:11
sean-k-mooneythat is the adress of the compute19:11
sean-k-mooneyso the compute is trying to conenct to its own ipv6 adress to talk to keystone19:11
sean-k-mooneyand that is because SERVICE_HOST19:11
sean-k-mooneyis not set properly19:12
sean-k-mooneyyour setting   SERVICE_HOST: ""19:13
sean-k-mooneythat will work on singel node19:13
sean-k-mooneybut we actully need to set that to the ipv6 adress of the contoler19:13
melwittoh, bleh ok. 19:13
sean-k-mooneyor the host naem in /etc/hosts need to resove to the ipv6 adres19:13
melwittI just copied it from the base ipv6 job out of desperation19:14
melwittok. sorry about that19:14
sean-k-mooneydont be sorry. but that why i didnt set service host in my orgianl patch19:16
sean-k-mooneythe orcstarte devstack role shoudl populate that for us19:17
melwittI wonder why does the base job do that then19:17
sean-k-mooneybut it may or may not work out of the box for ipv619:17
sean-k-mooneymelwitt: my gess it its a hack because it use the ipv4 adress or similar19:17
sean-k-mooneybut lets find out19:17
opendevreviewmelanie witt proposed openstack/nova master: DNM test ceph ipv6  https://review.opendev.org/c/openstack/nova/+/98462319:19
sean-k-mooneyso i tought it was configred via https://github.com/openstack/devstack/blob/03ece8f88040be9b0b14dd1cfe93076ad2419a80/playbooks/pre.yaml#L3719:20
sean-k-mooneybut no...19:20
sean-k-mooneyah https://github.com/openstack/devstack/blob/03ece8f88040be9b0b14dd1cfe93076ad2419a80/.zuul.yaml#L51419:22
sean-k-mooneyso devstack-minimal is doing  SERVICE_HOST: "{{ hostvars['controller']['nodepool']['private_ipv4'] }}"19:22
sean-k-mooneymelwitt: can you try changing that to  SERVICE_HOST: "{{ hostvars['controller']['nodepool']['private_ipv6'] }}" im goign to doubel check that thats a thing19:22
melwittsure19:23
sean-k-mooneyok that will not work but we can get the ip adress in a difennt place 19:24
sean-k-mooneyhttps://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_dcc/openstack/dcc395a124c04f4fbc3368ddf97400d3/zuul-info/inventory.yaml19:24
sean-k-mooneyis the inventory file and that has private_ipv6: null19:24
melwittah19:25
sean-k-mooneybut if we look in teh ansibel fact https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_dcc/openstack/dcc395a124c04f4fbc3368ddf97400d3/zuul-info/host-info.controller.yaml19:26
sean-k-mooneywe defnilly have access to the ip19:26
sean-k-mooneyi think this will work. one sec whil i type it out19:26
sean-k-mooneySERVICE_HOST: ""{{hostvars['controller']['ansible_defult_ipv6']['address']}}"19:28
sean-k-mooneywell - the extra "19:28
sean-k-mooneySERVICE_HOST: "{{hostvars['controller']['ansible_defult_ipv6']['address']}}"19:28
melwittyes I will edit the typos :)19:28
sean-k-mooneyyou will also need to set it for the subvar19:29
melwittack19:29
sean-k-mooneywe can proably fix this in the devstack defintion to be based on if you have selecte ipv4 or ipv6 for the serivce ip later19:30
sean-k-mooneyin the job defintion we are seting  SERVICE_IP_VERSION: 6 so we have something to switch off19:31
sean-k-mooneywiththat said if i look at the job vriable for the compute 19:31
sean-k-mooneyhttps://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_dcc/openstack/dcc395a124c04f4fbc3368ddf97400d3/zuul-info/inventory.yaml19:31
sean-k-mooneyi see a lot of other ip v4 stuff there but we can revisti that if needed19:32
opendevreviewmelanie witt proposed openstack/nova master: DNM test ceph ipv6  https://review.opendev.org/c/openstack/nova/+/98462319:32
sean-k-mooneymelwitt: as an asside we started looking at this because we wanted to test a patch that was related to it19:33
sean-k-mooneymelwitt: do you recall if we you knwo merged that bug fix?19:33
sean-k-mooneythe libvirt migration one19:33
melwittright. no we did not merge it yet and I have been trying to make this work in order to verify it19:34
sean-k-mooneyack19:34
melwittI'm not opposed to merging it without verification but it would be really great to be able to 19:34
sean-k-mooneyoh ya i agree it would be good to have it19:34
sean-k-mooneyi just lost context on where we were at with the patch19:34
melwittafter seeing that XML ipv6 brackets thing, I am feeling like I really want to see this work haha19:35
sean-k-mooneyyep19:35
sean-k-mooneyok im going to wrap for today19:35
melwittit looks fine to me. just wanted to see if I could do this. if not, I'll just review it without19:36
sean-k-mooneyill try and remember to check the resutls of tha trun but feel free to ping me to take another look 19:36
melwittcool thanks for the help19:36
sean-k-mooneymelwitt: by the way i said this before but there is currently no ipv6 + ceph covage anywhere. i dont even think there is singel node19:37
sean-k-mooneyso defintly nice ot fix that at some point19:37
melwitt++19:37
isaacvicente[m]hi guys o/ i would like to start solving this bug: https://bugs.launchpad.net/nova/+bug/203591119:39
isaacvicente[m]given the last time it was assigned to someone, its ok to assign to myself? i replied on the bug but got no reponse19:39
melwittisaacvicente[m]: yeah I would go ahead and assign it to yourself19:40
isaacvicente[m]nice, i started looking into it, and i wonder where I would do the changes. Trying to understand better the bug and the code base itself I asked chatgpt-codex-5.3 where the changes would be. It said that drivers under nova/virt/libvirt/volumes has two functions: connect_volume and disconnect_volume. I would then add the os-brick context manager guard_connections on those functions. I'm not sure if this is the right19:48
isaacvicente[m]direction. Does it make sense?19:48
isaacvicente[m]* nice, i started looking into it, and i wonder where I would do the changes. Trying to understand better the bug and the code base itself I asked chatgpt-codex-5.3 where the changes would be. It said that drivers under nova/virt/libvirt/volumes has two functions: connect_volume and disconnect_volume. I would then add the os-brick context manager guard_connection on those functions. I'm not sure if this is the right19:49
isaacvicente[m]direction. Does it make sense?19:49

Generated by irclog2html.py 4.1.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!