Tuesday, 2025-07-08

mhenHi! #openstack-placement seems to be empty. Is this the right channel for Placement?08:37
gibimhen: yes it is the right one09:19
mhenI've got a question regarding Placement:09:20
mhenis my interpretation correct that Placement offers no means of adjusting its WSGI pipeline? It seems hardcoded here: https://github.com/openstack/placement/blob/ce677d96f2a5e18dcc5e10a2046cac9780a67cde/placement/deploy.py#L108-L11709:20
opendevreviewKamil Sambor proposed openstack/nova master: Switch from Eventlet timeout  https://review.opendev.org/c/openstack/nova/+/95432409:20
mhenFor other services, there is usually an api-paste.ini or something similar.09:21
mhenSo the audit middleware (https://docs.openstack.org/keystonemiddleware/latest/audit.html) cannot be used with Placement?09:21
gibimhen: good question let me double check09:23
gibimhen: it seems you are right we have a hard coded pipeline with no api-paste configuratbility09:24
mhenalright, thanks for confirming09:27
opendevreviewBalazs Gibizer proposed openstack/nova master: [pci]Keep used dev in Placement regardless of dev_spec  https://review.opendev.org/c/openstack/nova/+/95414912:51
opendevreviewBalazs Gibizer proposed openstack/nova master: Reproduce bug/2115905  https://review.opendev.org/c/openstack/nova/+/95433612:51
sean-k-mooneygibi: i reviewed the first couple of patches in your eventlet work and im back up to where i left off before (the stats priting one) for the most part they look fine to me13:25
sean-k-mooneyi also looked at kamils patch, https://review.opendev.org/c/openstack/nova/+/949754/17 i think ThreadingEventWithResult should just be a future not a event with a result, i woudl be interested to know if you agree but i left that as a comment in gerrit so no rush13:27
zigoHi there! I filled-up a new bug: https://bugs.launchpad.net/nova/+bug/211618613:56
zigoI tried using an older version of pymysql (the one in Bookworm), it didn't help. I tried doing a gc.disable() to check if that was another instance of the GC issue in Python 3.13, and it didn't help either. Anyone has an idea of what's going on?13:56
gibisean-k-mooney: on the ThreadingEventWithResult: I wanted a solution that changes as minimal in the current flow of the code as possible. Hence the direct replication of the Eventlet.Event behavior with threading.Event + a variable approach. The Future based approach has the drawback in my mind that the concurrent Future interface explicitly says that setting the result of the Future should happen by 13:59
gibithe implementation of an Executor. So it seems using a Future in that nova code would create a lot more code churn if it is even possible.13:59
sean-k-mooneyam so future are usabel with raw thread13:59
sean-k-mooneyso an executor is not stritly required13:59
gibihttps://docs.python.org/3/library/concurrent.futures.html#concurrent.futures.Future.set_result 14:00
gibiThis method should only be used by Executor implementations and unit tests.14:00
sean-k-mooneyhum14:01
sean-k-mooneythat does not conform to what i would normally consider a future then14:01
sean-k-mooneywe would have to use a queue isnterad which is less obvious14:02
gibiwe started with a queue actually :) If you look back of previous PSs it is there14:03
sean-k-mooneyto me a future is concpetiulaly the same as a a queue of lenth 1 with blocking or non blocking gets dependign on if you pass a timeout to result14:04
gibithen we learned that eventlet.Event has the behavior that it sends the same piece of result to all waiters so a queue with a lenght of 1 is not optimal14:04
sean-k-mooneyah i see14:04
gibias it requires requeueing the same result :/14:04
sean-k-mooneywell that fan out or multi subsciber behaivor seams non standard too14:05
gibiyep it is not14:05
sean-k-mooneyare we actuly reallying on that14:05
sean-k-mooneyi think for now im ok with the current approch then14:05
gibiI'm not sure but I'm not willing to risk14:05
sean-k-mooneybut we may want to see if we can simplicy in the future14:05
gibiyepp simplifying in the future is the way14:05
gibifirst have something that works and then optimize14:06
sean-k-mooneyso the asyncio future does nto have that limiation for what its worth14:10
sean-k-mooneyhttps://docs.python.org/3/library/asyncio-future.html#asyncio.Future.set_result14:10
sean-k-mooneyi ws not expectign thos to diverge in that regard14:12
gibiyeah it is strange14:16
opendevreviewBalazs Gibizer proposed openstack/nova master: Reproduce bug/2115905  https://review.opendev.org/c/openstack/nova/+/95433614:22
opendevreviewBalazs Gibizer proposed openstack/nova master: [pci]Keep used dev in Placement regardless of dev_spec  https://review.opendev.org/c/openstack/nova/+/95414914:22
sean-k-mooneylooking at https://peps.python.org/pep-3148/ i think the landed on the current interface by emulating the java and in https://peps.python.org/pep-3156/#futures the intentioally diverged 14:23
sean-k-mooneyanyway that enough of a distaction14:25
UgglaNova meeting in  ~1h15:02
gmaanUggla: Due to Board meeting,  I might not be able to join today meeting.15:31
Ugglagmaan, ok15:31
Ugglagmaan, thanks for the notification15:31
gibiI know Dan is out this week, so we might have a quick meeting if no quorum is present15:34
Uggla#startmeeting nova16:00
opendevmeetMeeting started Tue Jul  8 16:00:14 2025 UTC and is due to finish in 60 minutes.  The chair is Uggla. Information about MeetBot at http://wiki.debian.org/MeetBot.16:00
opendevmeetUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.16:00
opendevmeetThe meeting name has been set to 'nova'16:00
UgglaHello everyone16:00
fwieselo/16:01
elodilleso/16:01
bauzaso/16:03
UgglaLet's start slowly so people could join16:03
Uggla#info No Critical bug 16:03
Ugglaoops16:03
Uggla#topic Bugs (stuck/critical)16:03
Uggla#info No Critical bug 16:03
Uggla#topic Gate status16:03
Uggla#link https://bugs.launchpad.net/nova/+bugs?field.tag=gate-failure Nova gate bugs 16:04
Uggla#link https://etherpad.opendev.org/p/nova-ci-failures-minimal16:04
Uggla#link https://zuul.openstack.org/builds?project=openstack%2Fnova&project=openstack%2Fplacement&branch=stable%2F*&branch=master&pipeline=periodic-weekly&skip=0 Nova&Placement periodic jobs status16:04
Uggla#info Please look at the gate failures and file a bug report with the gate-failure tag.16:04
Uggla#info Please try to provide a meaningful comment when you recheck16:04
Uggla#info https://bugs.launchpad.net/nova/+bug/2115980 opened by gibi. It seems nova next has regular failure due to this.16:04
Ugglagibi, do you have more info about ^16:05
clarkbthe underlying issue is that zuul launcher can in some exceptional circumstances assign nodes to multiple providers within the same nodeset. This is due to zuul launcher wanting to be able to support k8s pods and openstack vms in the same nodeset16:05
clarkbunfortauntely there were a few bugs (I think 2 or 3 at this point) that allowed it to mix providers more often than expected. The last of these known bugs was fixed around 2200UTC yseterday16:06
gibiyepp the latest from me is that I'm not seeing the failure today any more16:06
clarkbwith each of the bugfixes the occurence of this issue should be lessened. And hopefully if we've fixed the bugs then the issue goes away16:06
gibiI will keep the bug around for couple of days then I will close it if the problem does not re-occure16:07
Ugglathanks clarkb, I appreciate your inputs.16:08
Ugglathanks gibi16:08
Ugglaskipping next topic because gmann is not available.16:08
Uggla#topic Release Planning 16:09
Uggla#link https://releases.openstack.org/flamingo/schedule.html16:09
Uggla#info Nova deadlines are set in the above schedule16:09
UgglaSpec freeze was last week. So no new specs.16:09
Uggla#topic Review priorities 16:10
Uggla#link https://etherpad.opendev.org/p/nova-2025.2-status16:10
UgglaWe had a lot of interest just before the spec freeze around io threads.16:12
UgglaAlthough, the specs were not approved this cycle. We may open a 2026.1 "directory" to keep the momentum around this activity.16:12
Uggla#topic OpenAPI 16:13
Uggla#link: https://review.opendev.org/q/topic:%22openapi%22+(project:openstack/nova+OR+project:openstack/placement)+-status:merged+-status:abandoned16:14
Uggla#info 19 remaining. Same as last week.16:14
Uggla#topic Stable Branches16:14
Ugglaelodilles, please go ahead.16:14
elodillesi was on vacation last week, so i might missed some stuff16:14
elodillesotherwise, i'm not aware of any issue with stable gates16:15
elodillesso that's all, to be quick o:)16:15
elodillesUggla: back to you16:15
Ugglathx elodilles16:15
Uggla#topic vmwareapi 3rd-party CI efforts Highlights16:16
fwieselHi, no updates from my side.16:16
Ugglafwiesel, thanks.16:16
Uggla#topic Gibi's news about eventlet removal. 16:16
Uggla#link Blog: https://gibizer.github.io/categories/eventlet/16:16
Uggla#link nova-scheduler series is ready for core review, starting at https://review.opendev.org/c/openstack/nova/+/94796616:16
Ugglagibi, do you want to say something about eventlet removal ?16:16
gibionly that we are slowly landing the approved patches as the gate allows16:17
gibiwe have one patch left that is approved but not landed yet16:17
gibithe series is in reviewable state16:17
gibiI'm only tracking minor comment at the moment16:18
gibithat is all16:18
Ugglathx gibi16:18
Uggla#topic Open discussion 16:19
Uggla#info (Uggla) I will be on PTO the 15th and 22nd July. bauzas will run the meeting on my behalf.16:19
UgglaThis is the only thing I wanted to share, thanks bauzas ^16:20
UgglaSomeone wants to discuss about something else ?16:20
gibiI have one topic, mostly a heads up. The grenade job semi frequently times out while dumping DBs. Recent occurence https://zuul.opendev.org/t/openstack/build/1328f6c40c5845fa800753d338551e0616:20
gibiI did not see any logs telling me why dumping the DB takes so much time. But maybe we just need to disable DB dumping on the gate by default16:21
UgglaWhat is the purpose of the dump ?16:22
gibiI guess to troubleshoot DB schema upgrade 16:22
gibibut I did not tracked down where we enable the dumpgin16:22
gibiI will bring this up next week when we have more cores around16:23
gibiand If my time allows then I will propose a patch to disable it16:23
Ugglayep no worries, tbh the question was more to satisfy my curiosity.16:23
UgglaDo you have some energy for bug triage ?16:25
gibiIf it just you and me are here for triage then maybe one or two bugs16:26
Ugglagibi, let's do it.16:27
Uggla#topic Bug scrubbing 16:27
Uggla#info up to 167 :( Increased a lot on the 16th of June.16:27
Uggla#link: https://etherpad.opendev.org/p/nova-bug-selection-for-triaging#L416:27
Uggla#link: https://truc.uggla.fr/ I kicked off this draft page to follow open bugs.16:27
UgglaFisrt bug: https://bugs.launchpad.net/nova/+bug/2116186 - Py3.13: nova-conductor crash during report status16:29
UgglaHere I'm more looking at what we should do with py3.13 bugs? I set a "python313" tag on the bug at least.16:29
gibiI know zigo wants help fixing them. I'm not sure how much capacity we have to help there.16:30
gibiWe will have more priority fixing them once we want to make py313 suppoted officially16:31
UgglaI'm a bit torn about looking at 3.13 bug. Should we wait eventlet removal first ?16:32
gibito look at the bugs we need motivation, when the decision happen that openstack officially supports 3.13 we will have such motivation.16:33
UgglaAnd should we keep this kind of but to new or set them to deferred ? What's your opinion ?16:33
gibievetlet might become supportable on top of 3.13 so it is unrelated16:34
Ugglagibi, you mean some people working in that direction ? ^ 16:35
gibiI have no hard opinion. I would keep it as is for now as I don't see the value of the deferred paperwork here16:35
gibiUggla: yes, herve and the folks are working on that16:35
Ugglaoh so fixing eventlet (OO)16:36
gibieventlet is more supportable in 313 than in 31416:37
Ugglaok good to know.16:38
UgglaRegarding the bug, I keep it in the current state.16:38
UgglaNext one: https://bugs.launchpad.net/nova/+bug/2115726, I think it is just a matter to triage it as it looks you worked on it?16:39
gibiit needs dicussion with the core team I have at least followup question to Sean's statements in the bug. 16:40
gibieither we doc the behavior or we rip it off, but if we rip it off we need to consider other adjacent features giving the same result as what Sean does not like16:41
sean-k-mooneyya i dont think 3.14 is even remotely possible16:41
UgglaHi sean-k-mooney16:41
sean-k-mooneyo/16:42
sean-k-mooneymy issue with this si form my perspecitive this behvior was never docudment or approved in any of the specs that altered the fucntionaltiy16:43
sean-k-mooneyi dont belive we shoudl document casese where we didnt to proper input validation and supprot them just becasue we didnt catch it in code review16:43
sean-k-mooneynot when it impact both the number of nested RPs in palcmenet and the dbrow count in the pci devices table16:43
sean-k-mooneyif this was a deliberate chocie by using address="*" that different16:44
gibithe dev_spec has the notion of partial match this is an edge case of such partial match, so if we remove this then we should remove other partial matches that potentially create a lot of rows16:44
gibiwhat about address=""?16:44
sean-k-mooneywell that shoudl also be an error16:45
gibiwhat about vendor_if=8086 without product id?16:45
sean-k-mooneyits not a viald adress or glob16:45
gibisean-k-mooney: we do support partial address match toady16:45
gibiyou dont need the *16:45
sean-k-mooneyvendor_id is ment ot be combdiend with product id or an address16:45
sean-k-mooneygibi: that is again not one of the docuemnted cases that shoudl work16:46
sean-k-mooneyit can work if you have one of the ohter matchers16:46
gibiI do belive that both globbing by address pieces and both partial address string match is explicitly implemented16:46
sean-k-mooneyan empty string is not a partial match16:46
sean-k-mooneyor a blog16:46
sean-k-mooney*glob16:47
gibiit is by regex16:47
gibiBoth traditional glob style and regular expression syntax is supported.16:47
gibihttps://docs.openstack.org/nova/latest/configuration/config.html#pci.device_spec16:48
sean-k-mooneyregex supprot requries the dict syntaxz16:48
sean-k-mooneyit not supproted in the string syntax16:48
gibiI'm note sure16:48
sean-k-mooneyi am pretty sure16:48
gibianyhow there is a lot of things that works today and we need discussion with quorum what to remove and what to document16:48
sean-k-mooneyi agree16:49
* bauzas apologies, was in a long convoy on the phone16:49
gibidevice_spec = { "vendor_id":"8086", "product_id":"10c9", "address": "0000:0c:"}16:49
sean-k-mooneyi would consieder supprot {} as a wild card to be a feature nto a bug16:49
gibithis works today16:49
Ugglaok so, can we consider it triaged or valid ?16:49
UgglaAs I understand something is required here.16:49
sean-k-mooneyUggla: we both agree its valid but not what the actull issue is and the actual fix16:49
bauzaslet's just document this16:49
sean-k-mooneyi consider it a input validation bug16:50
bauzasat least for now16:50
sean-k-mooneyim not okk wiht documenting it as supproted16:50
sean-k-mooneyif we want to say that it work but is not supproted andy you cant rely on it then sure16:50
sean-k-mooneybut this was never inteded to work so im not oke with commiting to supprot it going forward16:50
bauzassean-k-mooney: I'm not saying we should document it as "it's supported"16:50
bauzasI just want to document something like "note: if you use {}, then blah"16:51
bauzasor "bug note" if you prefer16:51
bauzasor known issue16:51
sean-k-mooneyi can by into this as a known issue16:51
gibiMy original plan was to describe the two types of dev_spec fields. The one that we use to filter the devices and the one that we use to enhance the devices. Then in the filtering fields I would go an detail how partial match works. But based on Sean's view I would go -1 or -2 so I'm not going to do it16:52
sean-k-mooneygibi: those are not seperate things16:52
sean-k-mooneyyou are to enrich devices sepereatly form filtering16:52
sean-k-mooneythere are ways to do that today beucase of bug16:53
sean-k-mooneybut that was not how this was intened to work form my perspective16:53
gibitraits, resource_class, otu are clearly not filtering anything16:53
gibithey are just enhancing the already filtered data16:53
sean-k-mooneycorrect but they shoudl only be applied to the same devspec that they are defiend on16:53
gibiand they do 16:54
sean-k-mooneyok so your not advocating for somethign like {managed=no}16:54
sean-k-mooneyto set that on all devices16:54
sean-k-mooneylike that is an invalid devspec form my perspetive16:54
gibiif you say {managed=no} then your dev_spec is matching to all devices today and enhancing them with the managed flag16:55
sean-k-mooneyya so that is not valid in my opion16:55
sean-k-mooneythat is a very severy bug that should prevent startup16:55
gibisame as {address="", managed=no}16:55
sean-k-mooneywhich is also not valid16:55
gibidoes  {address="0000", managed=no} valid?16:56
gibiit is accepted today16:56
gibiand matching most of the devices :)16:56
sean-k-mooneyif you read our docs then no16:56
sean-k-mooneyand i don tcondier what works to defien the contact16:56
gibiso the docs is incomplete16:56
sean-k-mooneywha twe wrote in the specs does16:56
gibithen we would never fix a doc bug :)16:57
bauzasI agree with gibi here16:57
bauzaswe need a doc, please16:57
sean-k-mooneythe docs are correct16:57
sean-k-mooneythe code is wrong16:57
gibianyhow. We need a list of what is supported and what is not supported and we need to agree on it16:57
gibithen we can fix whatever need fixing16:57
gibiwe currently cannot agree on what is supported and what is not16:57
gibiso I suggest tabling this16:57
sean-k-mooneyby the way {address="0000", managed=no} is technially valid today16:58
gibimaybe a PTG discussion is in order with preparation listing all the cases that accepted today16:58
sean-k-mooneybut that only because we supprot partlial adresses16:58
UgglaCan we move to the next one. As I'm unsure we manage to get out with something here.16:58
gibiso address="0" is also a partial address as well as ""16:58
gibiwhere is the line?16:58
sean-k-mooneywe draw it based on what we agreed in the specs orginally16:59
sean-k-mooneyand if thre was a bug in the spec we fix it with a new one or a bug that we dicusss in detail16:59
gibiI agree that we need to discuss in detail17:00
Ugglayep please can we stop on that one.17:00
gibiwe need a detailed plan what to rip off we cannot rip off address="" support if we keep address="0" I tink17:00
bauzasI need to leave now17:00
bauzaso/17:00
UgglaCan we have a look at the latest one: https://bugs.launchpad.net/nova/+bug/2115870  I guess it is valid looking aat the patch attached.17:01
gibiI have no context on volume retype17:02
sean-k-mooneylets leave that for now and come back next week17:02
Ugglaok end for today.17:03
UgglaThanks all17:03
Uggla#endmeeting17:03
opendevmeetMeeting ended Tue Jul  8 17:03:35 2025 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)17:03
opendevmeetMinutes:        https://meetings.opendev.org/meetings/nova/2025/nova.2025-07-08-16.00.html17:03
opendevmeetMinutes (text): https://meetings.opendev.org/meetings/nova/2025/nova.2025-07-08-16.00.txt17:03
opendevmeetLog:            https://meetings.opendev.org/meetings/nova/2025/nova.2025-07-08-16.00.log.html17:03
sean-k-mooneyUggla: for what its worth im not sure we have enough info in the bug to triage that17:04
sean-k-mooneywe dont know the souce/dest volume types, we dont knwo if it just change the qos ro actull moved data17:04
elodillesthanks o/17:05
Ugglasean-k-mooney, I guess the intend is clearer in the patch.17:05
sean-k-mooneyi.e. if this si between netapp san and ceph ectra17:05
sean-k-mooneythis is realted to how we store the nvram17:05
sean-k-mooneybut lets loop back to this when we have had tim to look ath this properly17:06
sean-k-mooneythere is potically some race condtions happenign here too so i think we need to look at this in detail17:06
Ugglasean-k-mooney, no problem. I agree the bug is not accurate enough. But the description of the patch looks clearer.17:07
UgglaI think I'll ask for a reproducer. Because that will clarify the need.17:08
sean-k-mooneyi think i know why this si happeing and i think its a side aeffect to a previous incorrect bugfix17:10
sean-k-mooneyhttps://review.opendev.org/c/openstack/nova/+/92235417:11
sean-k-mooneythat reverts https://review.opendev.org/c/openstack/nova/+/906380 which was written for https://launchpad.net/bugs/1997352 but i dont think that patchs was correct17:12
sean-k-mooneythere fix is properly makign the nvram deletion contionall https://review.opendev.org/c/openstack/nova/+/954210/2/nova/virt/libvirt/guest.py17:13
sean-k-mooneyand making sure we dont delete it as part of swap volume17:13
sean-k-mooneybut we also shoudl not be undefining it in most cases17:14
sean-k-mooneythe only time its valid to delete it is when deleting the vm or when rebuilding form uefi ot non uefi17:14
gmaansean-k-mooney: stephenfin: manager role code is ready, all tempest test are also green https://review.opendev.org/q/topic:%22bp/policy-manager-role-default%22+status:open 23:25
sean-k-mooney awsome ill try and do a pass tomorow23:26

Generated by irclog2html.py 4.0.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!