16:00:04 <gibi> #startmeeting nova
16:00:04 <opendevmeet> Meeting started Tue Jun  6 16:00:04 2023 UTC and is due to finish in 60 minutes.  The chair is gibi. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:00:04 <opendevmeet> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
16:00:04 <opendevmeet> The meeting name has been set to 'nova'
16:00:11 <gibi> #chair bauzas
16:00:11 <opendevmeet> Current chairs: bauzas gibi
16:00:15 <gibi> #chair dansmith
16:00:15 <opendevmeet> Current chairs: bauzas dansmith gibi
16:00:16 <dansmith> o/
16:00:22 <auniyal> o/
16:00:28 <elodilles> o/
16:01:02 <gibi> bauzas has an appointment so I try to chair this
16:01:14 <gibi> #link https://wiki.openstack.org/wiki/Meetings/Nova#Agenda_for_next_meeting
16:01:26 <gibi> #topic Bugs (stuck/critical)
16:01:30 <gibi> lets see
16:01:40 <gibi> #info No Critical bug
16:02:25 <gibi> #link https://bugs.launchpad.net/nova/+bugs?search=Search&field.status=New 18 new untriaged bugs (+3 since the last meeting)
16:02:31 <gibi> #info Add yourself in the team bug roster if you want to help https://etherpad.opendev.org/p/nova-bug-triage-roster
16:02:52 <gibi> last week the baton was at bauzas
16:03:14 <gibi> so I'm not sure if we have any news from him now
16:03:50 <gibi> the next on the roster is me
16:04:02 <bauzas> I'll take it next week
16:04:08 <gibi> but I will be mostly away next week
16:04:15 <Uggla_> o/
16:04:15 <bauzas> I didn't had time to look at them this week
16:04:38 <bauzas> Ditto due to the summit
16:04:48 <gibi> so moving down the list the next on it is melwitt
16:04:48 <bauzas> But I can try to look at them
16:04:56 <bauzas> (sorry on my phone)
16:05:18 <gibi> melwitt: could you take the baton?
16:05:24 <auniyal> gibi, last to last week I looked into this bug: https://bugs.launchpad.net/nova/+bug/2018719, I could not reproduce  so added comment to ask for more info
16:06:09 <gibi> auniyal: ack
16:06:42 <gibi> I guess logging in to the rescue image depends on the actual image so you are right
16:07:15 <gibi> I will ping melwitt later about the bug baton
16:07:27 <gibi> any other bugs we need to discuss?
16:08:45 <auniyal> nothing from my side, thanks
16:09:35 <gibi> #topic Gate status
16:09:56 <gibi> #link https://bugs.launchpad.net/nova/+bugs?field.tag=gate-failure Nova gate bugs
16:10:01 <gibi> #link https://etherpad.opendev.org/p/nova-ci-failures
16:10:16 <dansmith> lots of little test failures lately which is making it challenging to get a clean result
16:10:27 <dansmith> but nothing outstanding as a super common thing to go tackle that I've seen
16:10:50 <gibi> I saw two different guest failures one case an disk io error
16:10:59 <gibi> the other was probably some metadata error
16:11:08 <gibi> but I agree I did not see a pattern yet
16:11:22 <dansmith> I have seen some IO errors related to volumes yeah, but I don't know what that's coming from
16:12:43 <gibi> I don't see any new bug reported tagged with gate-failure. If I see a pattern in tomorrows reject then I will file some
16:13:04 <gibi> s/reject/recheck/
16:13:44 <bauzas> haven't seen any gate failure
16:13:52 <bauzas> (still otp)
16:14:14 <gibi> #link https://zuul.openstack.org/builds?project=openstack%2Fnova&project=openstack%2Fplacement&pipeline=periodic-weekly Nova&Placement
16:14:34 <gibi> periodics look good
16:15:11 <gibi> any other gate issues to raise?
16:15:21 <gibi> #info Please look at the gate failures and file a bug report with the gate-failure tag.
16:15:59 <dansmith> nothing from me
16:16:14 <bauzas> I'm back
16:16:21 <gibi> then the usual announcement
16:16:22 <gibi> #info STOP DOING BLIND RECHECKS aka. 'recheck' https://docs.openstack.org/project-team-guide/testing.html#how-to-handle-test-failures
16:16:47 <dansmith> fwiw,
16:17:11 <dansmith> I think we're doing quite well on the blind recheck thing.. not that we shouldn't remind people, but we could probably un-ALL-CAPS-ify that now :D
16:17:13 <bauzas> gibi: wants me to take again the chair seat ?
16:17:29 <gibi> dansmith: cool. I'm OK to uncap it :)
16:17:41 <dansmith> it's been tracked in the TC meeting and we'
16:17:45 <gibi> bauzas: the char is yours :)
16:17:49 <dansmith> we seem to be settling around pretty good behavior
16:17:49 <bauzas> dansmith: lol, I'll change it :)
16:19:01 <bauzas> #topic Release Planning
16:19:07 <bauzas> #link https://releases.openstack.org/bobcat/schedule.html
16:19:15 <bauzas> #info Nova deadlines are set in the above schedule
16:19:20 <bauzas> #info Nova spec review day today
16:19:25 <bauzas> as a reminder ^
16:19:39 <gibi> I deeply missed that :/
16:19:42 <bauzas> tbh, I wasn't able to do my duty but I'll do this later tonight
16:20:00 <bauzas> (some internal discussion ate my whole afternoon)
16:20:12 <bauzas> so, yeah, would be nice
16:20:24 <gibi> I saw that there is a spec proposal for continuing the PCI in placement work
16:20:24 <bauzas> nothing to tell apart this
16:20:33 <bauzas> gibi: indeed, someone proposed
16:20:40 <gibi> I need to review that
16:20:43 <bauzas> cool
16:20:57 <gibi> but other can chime in there too :)
16:20:58 <bauzas> as a reminder, if folks don't have time to review specs today, that's fine (c)
16:21:09 <bauzas> but please try to look at them this week
16:21:10 <gibi> there is alway tomorrow :)
16:21:22 <bauzas> at least before the Summit in case people discuss there
16:21:31 <bauzas> anyway, good related point,
16:21:40 <bauzas> #topic pPTG Planning
16:21:45 <bauzas> #info please add your topics and names to the etherpad https://etherpad.opendev.org/p/vancouver-june2023-nova
16:21:57 <bauzas> crickets in there ^
16:22:12 <bauzas> so I'll write an -discuss ML thread for this
16:22:20 <gibi> nah, I added one thing now :D
16:22:23 <bauzas> in case ops or devs want to discuss with us
16:22:30 <bauzas> hehe
16:23:23 <bauzas> + I'll tell ops during our forum meet&greet about our PTG
16:23:36 <bauzas> #info The table #24 is booked for the whole two days. See the Nova community there !
16:23:40 <bauzas> that's it
16:23:42 <bauzas> moving on
16:23:48 <bauzas> #topic Review priorities
16:23:56 <bauzas> #link https://review.opendev.org/q/status:open+(project:openstack/nova+OR+project:openstack/placement+OR+project:openstack/os-traits+OR+project:openstack/os-resource-classes+OR+project:openstack/os-vif+OR+project:openstack/python-novaclient+OR+project:openstack/osc-placement)+(label:Review-Priority%252B1+OR+label:Review-Priority%252B2)
16:24:01 <bauzas> #info As a reminder, cores eager to review changes can +1 to indicate their interest, +2 for committing to the review
16:24:06 <bauzas> #topic Stable Branches
16:24:12 <bauzas> elodilles is maybe afk
16:24:18 <bauzas> so lemme add his points
16:24:31 <bauzas> #info stable gates should be OK (from stable/2023.1 to stable/train)
16:24:37 <bauzas> #info stable branch status / gate failures tracking etherpad: https://etherpad.opendev.org/p/nova-stable-branch-ci
16:24:42 <bauzas> huzzah for this
16:24:51 <bauzas> and my point
16:24:57 <bauzas> #info train-eol patch proposed https://review.opendev.org/c/openstack/releases/+/885365
16:25:10 <bauzas> I'd appreciate if nova-cores could comment it ^
16:25:19 <dansmith> I will
16:25:20 * gibi just proposed a train backport today
16:25:31 <dansmith> I noticed the cinder people are taking a more aggressive approach
16:25:40 <bauzas> gibi: okay, then -1 my patch and I'll modify it to await for your backport merge
16:25:47 <bauzas> dansmith: yup, saw it too
16:25:53 <gibi> bauzas: I think I'm OK to drop that backport
16:26:05 <gibi> I did that to see if the patch works
16:26:12 <bauzas> gibi: as you want, just tell me your insights in the train-eol patch
16:26:18 <gibi> ack
16:26:44 <bauzas> dansmith: fwiw I'm afraid of EOLing the whole EM branches
16:27:09 <dansmith> shrug.. my argument for train applies to all the EM ones too
16:27:16 <dansmith> maybe we wait and see how it goes for cinder ;)
16:27:38 <bauzas> but since we haven't backported the os-brick CVE fix in Ussuri and Victoria, I could understand
16:28:01 <bauzas> but yeah, let's see what it's happening for cinder :D
16:28:09 * bauzas takes his popcorn :)
16:28:26 <bauzas> anyway, moving on
16:28:35 <bauzas> #topic Open discussion
16:28:45 <bauzas> none in the agenda
16:28:54 <bauzas> anything that someone wants to tell ?
16:30:01 <gibi> one thing
16:30:05 <bauzas> shoto
16:30:11 <bauzas> shoot even
16:30:34 <gibi> there was a request for opinion about openstack on k8s
16:30:40 <gibi> let me find the link
16:30:55 <sean-k-mooney> its a thing that people do
16:31:36 <bauzas> okidoki, let's wait
16:32:15 <gibi> 13:01 < mdbooth> I'll be running a forum session on Kubernetes on OpenStack in Vancouver next week. It's for users and developers of all related projects to talk to each other. Etherpad is here if there's anything you'd like to discuss: https://etherpad.opendev.org/p/openinfra-2023-kubernetes-on-openstack
16:32:20 <gibi> 13:01 < mdbooth> I'll be running a forum session on Kubernetes on OpenStack in Vancouver next week. It's for users and developers of all related projects to talk to each other. Etherpad is here if there's anything you'd like to discuss: https://etherpad.opendev.org/p/openinfra-2023-kubernetes-on-openstack
16:32:30 <gibi> sorry for the duplicate
16:32:47 <sean-k-mooney> im wondering why mdboot is runnign that session but ok
16:32:51 <dansmith> that's k8s on openstack not what you said right?
16:32:57 <gibi> sorry I mixed up
16:32:59 <sean-k-mooney> that k8s on openstack
16:33:01 <sean-k-mooney> ya
16:33:05 <bauzas> ok, so lemme add the link
16:33:17 <sean-k-mooney> that makes more sense why mdbooth is involved
16:33:26 <bauzas> #link https://etherpad.opendev.org/p/openinfra-2023-kubernetes-on-openstack OpenInfra Forum session for discussing about k8s on openstack
16:33:37 <bauzas> gibi: the other way btw. :)
16:33:41 <gibi> there is some nova related question in the etherpad
16:34:24 <gibi> about getting notified if anything changed with servers
16:34:30 <gibi> I offered the nova notification inteface
16:34:35 <bauzas> yeah and it's a public API :)
16:34:38 <gibi> but apperantly they want something public
16:34:58 <sean-k-mooney> the notifcation are the only interface we have currently
16:34:58 <gibi> the API is documented but the message bus access then to be non public
16:35:03 <bauzas> since someone worked on notifications objects like 6 years ago (guess who :p )
16:35:14 <gibi> bauzas: hah :D
16:35:24 <bauzas> gibi: are you sure that the message bus can't be public ?
16:35:33 <dansmith> it shouldn't be
16:35:39 <gibi> yeah
16:35:43 <dansmith> we have instance events.. that's what they want I think
16:35:48 <sean-k-mooney> the notification bus is semi privladged
16:35:50 <gibi> and our notifications tend to contain infra information
16:35:52 <dansmith> I think they just need an async way to get those
16:35:58 <sean-k-mooney> the notifications can have private infor in them
16:36:02 <sean-k-mooney> depening on what you configure
16:36:04 <sean-k-mooney> like the bdms
16:36:30 <gibi> yeah, bottom line the notification API is designed for consumed by admins or other openstack service not endusers
16:36:43 <bauzas> ah they want it to be consumable by endusers ?
16:36:43 <dansmith> right, it would leak bad things between tenants for sure
16:36:47 <dansmith> not just infra things
16:36:52 <gibi> yeah
16:36:58 <bauzas> urth
16:37:01 <bauzas> urgh
16:37:01 <sean-k-mooney> you could have a multi tentant service that converts the notificaton in to a webhook callback or similar
16:37:09 <dansmith> instance events/actions is the right thing I think, it's just only polling currently
16:37:13 <sean-k-mooney> but that still not greate
16:37:28 <sean-k-mooney> dansmith: ya the event stream would work but im not sure that
16:37:37 <gibi> yeah, a websocket around instance actions wouldbe nice
16:37:41 <sean-k-mooney> even if it was event based they woud lwant ot liste per instance
16:37:53 <sean-k-mooney> more liek open a websocket and get all instance events for a project?
16:38:09 <dansmith> they probably want to be able to register a handler with a scope (one instance, all my instances) that lives for a period of time that we call when there's a new event
16:38:14 <sean-k-mooney> or that you are allowed to see based on teh scope of the keystone token
16:38:17 <bauzas> I think everytime someone asks us to monitor some instance action, we tell them 'lookup the notifications'
16:38:25 <bauzas> but this is for admin usage
16:38:29 <dansmith> websocket will require a lot of standby resources that I think would be hard for us to manage
16:38:45 <gibi> true
16:38:45 <dansmith> anyway,
16:38:58 <bauzas> so, they want some enduser public subscription mechanism for asynchronously being notified on my instance state changes ?
16:39:00 <dansmith> not sure how many people will be there to make any sort of headway on that topic, since I think those people are likely here :)
16:39:11 <dansmith> bauzas: yeah
16:39:24 <bauzas> sounds a client thing to mre
16:39:26 <bauzas> me
16:39:31 <gibi> dansmith: I will be there hence collecting ideas here now :)
16:39:32 <sean-k-mooney> lets see if they acan at least expand on the usecases
16:39:58 <dansmith> bauzas: a client can poll (or long poll) but that's much less efficient
16:39:59 <gibi> yeah I will pull out some specific use case and try to limit the scope to something very simple on our side
16:40:04 <dansmith> especially when instance actions could be days apart
16:40:21 <bauzas> but yeah, someone could provide some tool that would listen to the notification bus and scramples all the admin-only data
16:40:29 <gibi> bauzas: exactly
16:40:39 <bauzas> sorry, by client I meant something unrelated to nova
16:40:45 <sean-k-mooney> so the way to do this in the past was ceilometer put the relevetn events in AODH
16:40:51 <dansmith> they could, but that's basically re-constructing the tenant isolation that nova already has, so it's a big new surface to secure and new services to run
16:40:54 <sean-k-mooney> and then you woudl set up alarms on the events you cared about
16:41:15 <dansmith> sean-k-mooney: that's all intended to be operator-focused, not for users to get status/events on their instances right?
16:41:27 <sean-k-mooney> no
16:41:34 <gibi> if they only need a trigger to re-read the instance action API then most of of the data can be hidden from our notifications
16:41:39 <sean-k-mooney> aodh and celoimeter provided user facing events/metrics
16:41:51 <dansmith> okay I didn't realize
16:41:52 <bauzas> yeah
16:41:53 <sean-k-mooney> they didnt actully expose the full notificaiton
16:42:01 <bauzas> ceilometer was the fit
16:42:04 <sean-k-mooney> jsut instance boot started and instance boot finsined events
16:42:19 <bauzas> and that's why we never had this in nova
16:42:27 <dansmith> honestly, I feel like this is probably something nova can/should be doing
16:42:40 <dansmith> nowadays this is how stuff plugs together
16:42:51 <sean-k-mooney> ya i think its something we coudl do
16:42:56 <sean-k-mooney> but we need to think about how
16:42:58 <dansmith> making an external tool reconstruct what we already know is kinda :/
16:43:18 <gibi> yeah
16:43:24 <dansmith> it could be a service like console that you run if you want, and scale separately to handle the amoun tof load you want to tolerate
16:43:31 <bauzas> dansmith: I'm still struggling to find how we would ensure the tenancy isolation by the message bus, but I'm open to ideas
16:43:45 <bauzas> unless we create a bus per tenant
16:43:46 <sean-k-mooney> if its in nova
16:43:47 <dansmith> bauzas: we wouldn't?
16:43:48 <sean-k-mooney> we can just filter
16:44:36 <bauzas> I'm maybe misunderstanding the proposal, but I thought we were about saying that we may emit project-related notifications
16:44:46 <dansmith> not at the rabbit level
16:44:51 <dansmith> let's let gibi collect some data,
16:44:59 <dansmith> and then we probably need a high-bandwidth conversation about options
16:45:01 <gibi> bauzas: we def need to understand their use case better
16:45:02 <bauzas> dansmith: ah ok
16:45:18 <bauzas> dansmith: then we need to construct some HTTP/2 layer with keystone auth
16:45:33 <bauzas> or something like that
16:45:38 <dansmith> bauzas: not necessarily
16:45:56 <dansmith> it just depends.. but it should be HTTP-something, either an event stream or callbacks
16:46:07 <sean-k-mooney> you would do somethign like "openstack project event subsribe (instnace.action.)*" which woudl return a websock url that would only stream the relevent events for the current project based on the keystone token
16:46:27 <dansmith> yep, could be something like that
16:46:38 <bauzas> sounds very console-ish
16:46:40 <sean-k-mooney> so like the console you woudl first create it and then get a handel for where to collect the data
16:46:42 <bauzas> but okay
16:46:44 <gibi> I can simplify it down to give me a server uuid and I stream you data about notifications affecting the server, but only with very limited data provided
16:46:58 <dansmith> bauzas: exactly.. it's the same sort of arrangement, and the same target audience
16:47:38 <bauzas> ok, then it sounds we have an agreement on the direction, let's not overpaper the technical details
16:47:42 <sean-k-mooney> and if its a seperate binary nova-event-proxy
16:47:53 <dansmith> gibi: yeah I just don't think you should need 100 websockets you have to read from if you have 100 instances in your wordpress deployment
16:47:58 <dansmith> sean-k-mooney: right
16:48:02 <sean-k-mooney> then its scalablity and wether its deployed is up ot the operator
16:48:07 <dansmith> yup
16:48:10 <gibi> dansmith: ahh true, we can do it pre project then
16:48:13 <gibi> per
16:48:18 <bauzas> yeah, devil is in the details of the productization
16:48:21 <dansmith> gibi: or even server group
16:48:42 <sean-k-mooney> anyway lets see what they actully bring up
16:48:42 <bauzas> gibi: honestly, the granularity sounds per project to me
16:48:53 <sean-k-mooney> and see if this type of solution would work for them or not
16:48:57 <dansmith> well, nfv people are all one project in some cases, so that probably won't work for them
16:49:01 <bauzas> gibi: are you done with this topic now that we drafted a solution for you ? :D
16:49:11 <gibi> I'm done
16:49:15 <bauzas> cool
16:49:16 <gibi> thanks for the discussion\
16:49:22 <gibi> I will link this to the etherpad
16:49:27 <gibi> and I will report back from the summit
16:49:27 <bauzas> cool
16:49:37 <bauzas> I'll be back watching you at the Summit anyway
16:49:51 <bauzas> so if you promise too many things, I could yell :p
16:49:58 <gibi> bauzas: please do so
16:49:59 <gibi> :D
16:50:12 <gibi> I don't need another 3 years of "notification" work
16:50:22 <dansmith> heh
16:50:28 <dansmith> still got scars eh/
16:50:30 <bauzas> ok, I was balancing the idea to paperwork the scaphandre and manila series but I'm exhausted of today
16:50:54 <bauzas> so, let's skip it and pretend it will be discussed in two weeks from now
16:51:01 <gibi> dansmith: time makes all these memory nicer and nicer actaully
16:51:06 <dansmith> heh
16:51:18 <gibi> so bauzas' has a good point watching me :)
16:51:32 <dansmith> gibi: https://www.youtube.com/watch?v=dLjNzwEULG8
16:51:48 <bauzas> gibi: you're fortunate, canadians don't open carry
16:52:05 <bauzas> sorry, was a terrible joke :)
16:52:16 <gibi> dansmith: I need to check this out after the meeting :)
16:52:20 <bauzas> anyway, I think we're done for today
16:52:29 <gibi> indeed
16:52:40 <bauzas> thanks all
16:52:45 <bauzas> and thanks gibi for the chair
16:52:50 <bauzas> #endmeeting