mhen | Hi! #openstack-placement seems to be empty. Is this the right channel for Placement? | 08:37 |
---|---|---|
gibi | mhen: yes it is the right one | 09:19 |
mhen | I've got a question regarding Placement: | 09:20 |
mhen | is my interpretation correct that Placement offers no means of adjusting its WSGI pipeline? It seems hardcoded here: https://github.com/openstack/placement/blob/ce677d96f2a5e18dcc5e10a2046cac9780a67cde/placement/deploy.py#L108-L117 | 09:20 |
opendevreview | Kamil Sambor proposed openstack/nova master: Switch from Eventlet timeout https://review.opendev.org/c/openstack/nova/+/954324 | 09:20 |
mhen | For other services, there is usually an api-paste.ini or something similar. | 09:21 |
mhen | So the audit middleware (https://docs.openstack.org/keystonemiddleware/latest/audit.html) cannot be used with Placement? | 09:21 |
gibi | mhen: good question let me double check | 09:23 |
gibi | mhen: it seems you are right we have a hard coded pipeline with no api-paste configuratbility | 09:24 |
mhen | alright, thanks for confirming | 09:27 |
opendevreview | Balazs Gibizer proposed openstack/nova master: [pci]Keep used dev in Placement regardless of dev_spec https://review.opendev.org/c/openstack/nova/+/954149 | 12:51 |
opendevreview | Balazs Gibizer proposed openstack/nova master: Reproduce bug/2115905 https://review.opendev.org/c/openstack/nova/+/954336 | 12:51 |
sean-k-mooney | gibi: i reviewed the first couple of patches in your eventlet work and im back up to where i left off before (the stats priting one) for the most part they look fine to me | 13:25 |
sean-k-mooney | i also looked at kamils patch, https://review.opendev.org/c/openstack/nova/+/949754/17 i think ThreadingEventWithResult should just be a future not a event with a result, i woudl be interested to know if you agree but i left that as a comment in gerrit so no rush | 13:27 |
zigo | Hi there! I filled-up a new bug: https://bugs.launchpad.net/nova/+bug/2116186 | 13:56 |
zigo | I tried using an older version of pymysql (the one in Bookworm), it didn't help. I tried doing a gc.disable() to check if that was another instance of the GC issue in Python 3.13, and it didn't help either. Anyone has an idea of what's going on? | 13:56 |
gibi | sean-k-mooney: on the ThreadingEventWithResult: I wanted a solution that changes as minimal in the current flow of the code as possible. Hence the direct replication of the Eventlet.Event behavior with threading.Event + a variable approach. The Future based approach has the drawback in my mind that the concurrent Future interface explicitly says that setting the result of the Future should happen by | 13:59 |
gibi | the implementation of an Executor. So it seems using a Future in that nova code would create a lot more code churn if it is even possible. | 13:59 |
sean-k-mooney | am so future are usabel with raw thread | 13:59 |
sean-k-mooney | so an executor is not stritly required | 13:59 |
gibi | https://docs.python.org/3/library/concurrent.futures.html#concurrent.futures.Future.set_result | 14:00 |
gibi | This method should only be used by Executor implementations and unit tests. | 14:00 |
sean-k-mooney | hum | 14:01 |
sean-k-mooney | that does not conform to what i would normally consider a future then | 14:01 |
sean-k-mooney | we would have to use a queue isnterad which is less obvious | 14:02 |
gibi | we started with a queue actually :) If you look back of previous PSs it is there | 14:03 |
sean-k-mooney | to me a future is concpetiulaly the same as a a queue of lenth 1 with blocking or non blocking gets dependign on if you pass a timeout to result | 14:04 |
gibi | then we learned that eventlet.Event has the behavior that it sends the same piece of result to all waiters so a queue with a lenght of 1 is not optimal | 14:04 |
sean-k-mooney | ah i see | 14:04 |
gibi | as it requires requeueing the same result :/ | 14:04 |
sean-k-mooney | well that fan out or multi subsciber behaivor seams non standard too | 14:05 |
gibi | yep it is not | 14:05 |
sean-k-mooney | are we actuly reallying on that | 14:05 |
sean-k-mooney | i think for now im ok with the current approch then | 14:05 |
gibi | I'm not sure but I'm not willing to risk | 14:05 |
sean-k-mooney | but we may want to see if we can simplicy in the future | 14:05 |
gibi | yepp simplifying in the future is the way | 14:05 |
gibi | first have something that works and then optimize | 14:06 |
sean-k-mooney | so the asyncio future does nto have that limiation for what its worth | 14:10 |
sean-k-mooney | https://docs.python.org/3/library/asyncio-future.html#asyncio.Future.set_result | 14:10 |
sean-k-mooney | i ws not expectign thos to diverge in that regard | 14:12 |
gibi | yeah it is strange | 14:16 |
opendevreview | Balazs Gibizer proposed openstack/nova master: Reproduce bug/2115905 https://review.opendev.org/c/openstack/nova/+/954336 | 14:22 |
opendevreview | Balazs Gibizer proposed openstack/nova master: [pci]Keep used dev in Placement regardless of dev_spec https://review.opendev.org/c/openstack/nova/+/954149 | 14:22 |
sean-k-mooney | looking at https://peps.python.org/pep-3148/ i think the landed on the current interface by emulating the java and in https://peps.python.org/pep-3156/#futures the intentioally diverged | 14:23 |
sean-k-mooney | anyway that enough of a distaction | 14:25 |
Uggla | Nova meeting in ~1h | 15:02 |
gmaan | Uggla: Due to Board meeting, I might not be able to join today meeting. | 15:31 |
Uggla | gmaan, ok | 15:31 |
Uggla | gmaan, thanks for the notification | 15:31 |
gibi | I know Dan is out this week, so we might have a quick meeting if no quorum is present | 15:34 |
Uggla | #startmeeting nova | 16:00 |
opendevmeet | Meeting started Tue Jul 8 16:00:14 2025 UTC and is due to finish in 60 minutes. The chair is Uggla. Information about MeetBot at http://wiki.debian.org/MeetBot. | 16:00 |
opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 16:00 |
opendevmeet | The meeting name has been set to 'nova' | 16:00 |
Uggla | Hello everyone | 16:00 |
fwiesel | o/ | 16:01 |
elodilles | o/ | 16:01 |
bauzas | o/ | 16:03 |
Uggla | Let's start slowly so people could join | 16:03 |
Uggla | #info No Critical bug | 16:03 |
Uggla | oops | 16:03 |
Uggla | #topic Bugs (stuck/critical) | 16:03 |
Uggla | #info No Critical bug | 16:03 |
Uggla | #topic Gate status | 16:03 |
Uggla | #link https://bugs.launchpad.net/nova/+bugs?field.tag=gate-failure Nova gate bugs | 16:04 |
Uggla | #link https://etherpad.opendev.org/p/nova-ci-failures-minimal | 16:04 |
Uggla | #link https://zuul.openstack.org/builds?project=openstack%2Fnova&project=openstack%2Fplacement&branch=stable%2F*&branch=master&pipeline=periodic-weekly&skip=0 Nova&Placement periodic jobs status | 16:04 |
Uggla | #info Please look at the gate failures and file a bug report with the gate-failure tag. | 16:04 |
Uggla | #info Please try to provide a meaningful comment when you recheck | 16:04 |
Uggla | #info https://bugs.launchpad.net/nova/+bug/2115980 opened by gibi. It seems nova next has regular failure due to this. | 16:04 |
Uggla | gibi, do you have more info about ^ | 16:05 |
clarkb | the underlying issue is that zuul launcher can in some exceptional circumstances assign nodes to multiple providers within the same nodeset. This is due to zuul launcher wanting to be able to support k8s pods and openstack vms in the same nodeset | 16:05 |
clarkb | unfortauntely there were a few bugs (I think 2 or 3 at this point) that allowed it to mix providers more often than expected. The last of these known bugs was fixed around 2200UTC yseterday | 16:06 |
gibi | yepp the latest from me is that I'm not seeing the failure today any more | 16:06 |
clarkb | with each of the bugfixes the occurence of this issue should be lessened. And hopefully if we've fixed the bugs then the issue goes away | 16:06 |
gibi | I will keep the bug around for couple of days then I will close it if the problem does not re-occure | 16:07 |
Uggla | thanks clarkb, I appreciate your inputs. | 16:08 |
Uggla | thanks gibi | 16:08 |
Uggla | skipping next topic because gmann is not available. | 16:08 |
Uggla | #topic Release Planning | 16:09 |
Uggla | #link https://releases.openstack.org/flamingo/schedule.html | 16:09 |
Uggla | #info Nova deadlines are set in the above schedule | 16:09 |
Uggla | Spec freeze was last week. So no new specs. | 16:09 |
Uggla | #topic Review priorities | 16:10 |
Uggla | #link https://etherpad.opendev.org/p/nova-2025.2-status | 16:10 |
Uggla | We had a lot of interest just before the spec freeze around io threads. | 16:12 |
Uggla | Although, the specs were not approved this cycle. We may open a 2026.1 "directory" to keep the momentum around this activity. | 16:12 |
Uggla | #topic OpenAPI | 16:13 |
Uggla | #link: https://review.opendev.org/q/topic:%22openapi%22+(project:openstack/nova+OR+project:openstack/placement)+-status:merged+-status:abandoned | 16:14 |
Uggla | #info 19 remaining. Same as last week. | 16:14 |
Uggla | #topic Stable Branches | 16:14 |
Uggla | elodilles, please go ahead. | 16:14 |
elodilles | i was on vacation last week, so i might missed some stuff | 16:14 |
elodilles | otherwise, i'm not aware of any issue with stable gates | 16:15 |
elodilles | so that's all, to be quick o:) | 16:15 |
elodilles | Uggla: back to you | 16:15 |
Uggla | thx elodilles | 16:15 |
Uggla | #topic vmwareapi 3rd-party CI efforts Highlights | 16:16 |
fwiesel | Hi, no updates from my side. | 16:16 |
Uggla | fwiesel, thanks. | 16:16 |
Uggla | #topic Gibi's news about eventlet removal. | 16:16 |
Uggla | #link Blog: https://gibizer.github.io/categories/eventlet/ | 16:16 |
Uggla | #link nova-scheduler series is ready for core review, starting at https://review.opendev.org/c/openstack/nova/+/947966 | 16:16 |
Uggla | gibi, do you want to say something about eventlet removal ? | 16:16 |
gibi | only that we are slowly landing the approved patches as the gate allows | 16:17 |
gibi | we have one patch left that is approved but not landed yet | 16:17 |
gibi | the series is in reviewable state | 16:17 |
gibi | I'm only tracking minor comment at the moment | 16:18 |
gibi | that is all | 16:18 |
Uggla | thx gibi | 16:18 |
Uggla | #topic Open discussion | 16:19 |
Uggla | #info (Uggla) I will be on PTO the 15th and 22nd July. bauzas will run the meeting on my behalf. | 16:19 |
Uggla | This is the only thing I wanted to share, thanks bauzas ^ | 16:20 |
Uggla | Someone wants to discuss about something else ? | 16:20 |
gibi | I have one topic, mostly a heads up. The grenade job semi frequently times out while dumping DBs. Recent occurence https://zuul.opendev.org/t/openstack/build/1328f6c40c5845fa800753d338551e06 | 16:20 |
gibi | I did not see any logs telling me why dumping the DB takes so much time. But maybe we just need to disable DB dumping on the gate by default | 16:21 |
Uggla | What is the purpose of the dump ? | 16:22 |
gibi | I guess to troubleshoot DB schema upgrade | 16:22 |
gibi | but I did not tracked down where we enable the dumpgin | 16:22 |
gibi | I will bring this up next week when we have more cores around | 16:23 |
gibi | and If my time allows then I will propose a patch to disable it | 16:23 |
Uggla | yep no worries, tbh the question was more to satisfy my curiosity. | 16:23 |
Uggla | Do you have some energy for bug triage ? | 16:25 |
gibi | If it just you and me are here for triage then maybe one or two bugs | 16:26 |
Uggla | gibi, let's do it. | 16:27 |
Uggla | #topic Bug scrubbing | 16:27 |
Uggla | #info up to 167 :( Increased a lot on the 16th of June. | 16:27 |
Uggla | #link: https://etherpad.opendev.org/p/nova-bug-selection-for-triaging#L4 | 16:27 |
Uggla | #link: https://truc.uggla.fr/ I kicked off this draft page to follow open bugs. | 16:27 |
Uggla | Fisrt bug: https://bugs.launchpad.net/nova/+bug/2116186 - Py3.13: nova-conductor crash during report status | 16:29 |
Uggla | Here I'm more looking at what we should do with py3.13 bugs? I set a "python313" tag on the bug at least. | 16:29 |
gibi | I know zigo wants help fixing them. I'm not sure how much capacity we have to help there. | 16:30 |
gibi | We will have more priority fixing them once we want to make py313 suppoted officially | 16:31 |
Uggla | I'm a bit torn about looking at 3.13 bug. Should we wait eventlet removal first ? | 16:32 |
gibi | to look at the bugs we need motivation, when the decision happen that openstack officially supports 3.13 we will have such motivation. | 16:33 |
Uggla | And should we keep this kind of but to new or set them to deferred ? What's your opinion ? | 16:33 |
gibi | evetlet might become supportable on top of 3.13 so it is unrelated | 16:34 |
Uggla | gibi, you mean some people working in that direction ? ^ | 16:35 |
gibi | I have no hard opinion. I would keep it as is for now as I don't see the value of the deferred paperwork here | 16:35 |
gibi | Uggla: yes, herve and the folks are working on that | 16:35 |
Uggla | oh so fixing eventlet (OO) | 16:36 |
gibi | eventlet is more supportable in 313 than in 314 | 16:37 |
Uggla | ok good to know. | 16:38 |
Uggla | Regarding the bug, I keep it in the current state. | 16:38 |
Uggla | Next one: https://bugs.launchpad.net/nova/+bug/2115726, I think it is just a matter to triage it as it looks you worked on it? | 16:39 |
gibi | it needs dicussion with the core team I have at least followup question to Sean's statements in the bug. | 16:40 |
gibi | either we doc the behavior or we rip it off, but if we rip it off we need to consider other adjacent features giving the same result as what Sean does not like | 16:41 |
sean-k-mooney | ya i dont think 3.14 is even remotely possible | 16:41 |
Uggla | Hi sean-k-mooney | 16:41 |
sean-k-mooney | o/ | 16:42 |
sean-k-mooney | my issue with this si form my perspecitive this behvior was never docudment or approved in any of the specs that altered the fucntionaltiy | 16:43 |
sean-k-mooney | i dont belive we shoudl document casese where we didnt to proper input validation and supprot them just becasue we didnt catch it in code review | 16:43 |
sean-k-mooney | not when it impact both the number of nested RPs in palcmenet and the dbrow count in the pci devices table | 16:43 |
sean-k-mooney | if this was a deliberate chocie by using address="*" that different | 16:44 |
gibi | the dev_spec has the notion of partial match this is an edge case of such partial match, so if we remove this then we should remove other partial matches that potentially create a lot of rows | 16:44 |
gibi | what about address=""? | 16:44 |
sean-k-mooney | well that shoudl also be an error | 16:45 |
gibi | what about vendor_if=8086 without product id? | 16:45 |
sean-k-mooney | its not a viald adress or glob | 16:45 |
gibi | sean-k-mooney: we do support partial address match toady | 16:45 |
gibi | you dont need the * | 16:45 |
sean-k-mooney | vendor_id is ment ot be combdiend with product id or an address | 16:45 |
sean-k-mooney | gibi: that is again not one of the docuemnted cases that shoudl work | 16:46 |
sean-k-mooney | it can work if you have one of the ohter matchers | 16:46 |
gibi | I do belive that both globbing by address pieces and both partial address string match is explicitly implemented | 16:46 |
sean-k-mooney | an empty string is not a partial match | 16:46 |
sean-k-mooney | or a blog | 16:46 |
sean-k-mooney | *glob | 16:47 |
gibi | it is by regex | 16:47 |
gibi | Both traditional glob style and regular expression syntax is supported. | 16:47 |
gibi | https://docs.openstack.org/nova/latest/configuration/config.html#pci.device_spec | 16:48 |
sean-k-mooney | regex supprot requries the dict syntaxz | 16:48 |
sean-k-mooney | it not supproted in the string syntax | 16:48 |
gibi | I'm note sure | 16:48 |
sean-k-mooney | i am pretty sure | 16:48 |
gibi | anyhow there is a lot of things that works today and we need discussion with quorum what to remove and what to document | 16:48 |
sean-k-mooney | i agree | 16:49 |
* bauzas apologies, was in a long convoy on the phone | 16:49 | |
gibi | device_spec = { "vendor_id":"8086", "product_id":"10c9", "address": "0000:0c:"} | 16:49 |
sean-k-mooney | i would consieder supprot {} as a wild card to be a feature nto a bug | 16:49 |
gibi | this works today | 16:49 |
Uggla | ok so, can we consider it triaged or valid ? | 16:49 |
Uggla | As I understand something is required here. | 16:49 |
sean-k-mooney | Uggla: we both agree its valid but not what the actull issue is and the actual fix | 16:49 |
bauzas | let's just document this | 16:49 |
sean-k-mooney | i consider it a input validation bug | 16:50 |
bauzas | at least for now | 16:50 |
sean-k-mooney | im not okk wiht documenting it as supproted | 16:50 |
sean-k-mooney | if we want to say that it work but is not supproted andy you cant rely on it then sure | 16:50 |
sean-k-mooney | but this was never inteded to work so im not oke with commiting to supprot it going forward | 16:50 |
bauzas | sean-k-mooney: I'm not saying we should document it as "it's supported" | 16:50 |
bauzas | I just want to document something like "note: if you use {}, then blah" | 16:51 |
bauzas | or "bug note" if you prefer | 16:51 |
bauzas | or known issue | 16:51 |
sean-k-mooney | i can by into this as a known issue | 16:51 |
gibi | My original plan was to describe the two types of dev_spec fields. The one that we use to filter the devices and the one that we use to enhance the devices. Then in the filtering fields I would go an detail how partial match works. But based on Sean's view I would go -1 or -2 so I'm not going to do it | 16:52 |
sean-k-mooney | gibi: those are not seperate things | 16:52 |
sean-k-mooney | you are to enrich devices sepereatly form filtering | 16:52 |
sean-k-mooney | there are ways to do that today beucase of bug | 16:53 |
sean-k-mooney | but that was not how this was intened to work form my perspective | 16:53 |
gibi | traits, resource_class, otu are clearly not filtering anything | 16:53 |
gibi | they are just enhancing the already filtered data | 16:53 |
sean-k-mooney | correct but they shoudl only be applied to the same devspec that they are defiend on | 16:53 |
gibi | and they do | 16:54 |
sean-k-mooney | ok so your not advocating for somethign like {managed=no} | 16:54 |
sean-k-mooney | to set that on all devices | 16:54 |
sean-k-mooney | like that is an invalid devspec form my perspetive | 16:54 |
gibi | if you say {managed=no} then your dev_spec is matching to all devices today and enhancing them with the managed flag | 16:55 |
sean-k-mooney | ya so that is not valid in my opion | 16:55 |
sean-k-mooney | that is a very severy bug that should prevent startup | 16:55 |
gibi | same as {address="", managed=no} | 16:55 |
sean-k-mooney | which is also not valid | 16:55 |
gibi | does {address="0000", managed=no} valid? | 16:56 |
gibi | it is accepted today | 16:56 |
gibi | and matching most of the devices :) | 16:56 |
sean-k-mooney | if you read our docs then no | 16:56 |
sean-k-mooney | and i don tcondier what works to defien the contact | 16:56 |
gibi | so the docs is incomplete | 16:56 |
sean-k-mooney | wha twe wrote in the specs does | 16:56 |
gibi | then we would never fix a doc bug :) | 16:57 |
bauzas | I agree with gibi here | 16:57 |
bauzas | we need a doc, please | 16:57 |
sean-k-mooney | the docs are correct | 16:57 |
sean-k-mooney | the code is wrong | 16:57 |
gibi | anyhow. We need a list of what is supported and what is not supported and we need to agree on it | 16:57 |
gibi | then we can fix whatever need fixing | 16:57 |
gibi | we currently cannot agree on what is supported and what is not | 16:57 |
gibi | so I suggest tabling this | 16:57 |
sean-k-mooney | by the way {address="0000", managed=no} is technially valid today | 16:58 |
gibi | maybe a PTG discussion is in order with preparation listing all the cases that accepted today | 16:58 |
sean-k-mooney | but that only because we supprot partlial adresses | 16:58 |
Uggla | Can we move to the next one. As I'm unsure we manage to get out with something here. | 16:58 |
gibi | so address="0" is also a partial address as well as "" | 16:58 |
gibi | where is the line? | 16:58 |
sean-k-mooney | we draw it based on what we agreed in the specs orginally | 16:59 |
sean-k-mooney | and if thre was a bug in the spec we fix it with a new one or a bug that we dicusss in detail | 16:59 |
gibi | I agree that we need to discuss in detail | 17:00 |
Uggla | yep please can we stop on that one. | 17:00 |
gibi | we need a detailed plan what to rip off we cannot rip off address="" support if we keep address="0" I tink | 17:00 |
bauzas | I need to leave now | 17:00 |
bauzas | o/ | 17:00 |
Uggla | Can we have a look at the latest one: https://bugs.launchpad.net/nova/+bug/2115870 I guess it is valid looking aat the patch attached. | 17:01 |
gibi | I have no context on volume retype | 17:02 |
sean-k-mooney | lets leave that for now and come back next week | 17:02 |
Uggla | ok end for today. | 17:03 |
Uggla | Thanks all | 17:03 |
Uggla | #endmeeting | 17:03 |
opendevmeet | Meeting ended Tue Jul 8 17:03:35 2025 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 17:03 |
opendevmeet | Minutes: https://meetings.opendev.org/meetings/nova/2025/nova.2025-07-08-16.00.html | 17:03 |
opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/nova/2025/nova.2025-07-08-16.00.txt | 17:03 |
opendevmeet | Log: https://meetings.opendev.org/meetings/nova/2025/nova.2025-07-08-16.00.log.html | 17:03 |
sean-k-mooney | Uggla: for what its worth im not sure we have enough info in the bug to triage that | 17:04 |
sean-k-mooney | we dont know the souce/dest volume types, we dont knwo if it just change the qos ro actull moved data | 17:04 |
elodilles | thanks o/ | 17:05 |
Uggla | sean-k-mooney, I guess the intend is clearer in the patch. | 17:05 |
sean-k-mooney | i.e. if this si between netapp san and ceph ectra | 17:05 |
sean-k-mooney | this is realted to how we store the nvram | 17:05 |
sean-k-mooney | but lets loop back to this when we have had tim to look ath this properly | 17:06 |
sean-k-mooney | there is potically some race condtions happenign here too so i think we need to look at this in detail | 17:06 |
Uggla | sean-k-mooney, no problem. I agree the bug is not accurate enough. But the description of the patch looks clearer. | 17:07 |
Uggla | I think I'll ask for a reproducer. Because that will clarify the need. | 17:08 |
sean-k-mooney | i think i know why this si happeing and i think its a side aeffect to a previous incorrect bugfix | 17:10 |
sean-k-mooney | https://review.opendev.org/c/openstack/nova/+/922354 | 17:11 |
sean-k-mooney | that reverts https://review.opendev.org/c/openstack/nova/+/906380 which was written for https://launchpad.net/bugs/1997352 but i dont think that patchs was correct | 17:12 |
sean-k-mooney | there fix is properly makign the nvram deletion contionall https://review.opendev.org/c/openstack/nova/+/954210/2/nova/virt/libvirt/guest.py | 17:13 |
sean-k-mooney | and making sure we dont delete it as part of swap volume | 17:13 |
sean-k-mooney | but we also shoudl not be undefining it in most cases | 17:14 |
sean-k-mooney | the only time its valid to delete it is when deleting the vm or when rebuilding form uefi ot non uefi | 17:14 |
gmaan | sean-k-mooney: stephenfin: manager role code is ready, all tempest test are also green https://review.opendev.org/q/topic:%22bp/policy-manager-role-default%22+status:open | 23:25 |
sean-k-mooney | awsome ill try and do a pass tomorow | 23:26 |
Generated by irclog2html.py 4.0.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!