16:00:06 <bauzas> #startmeeting nova
16:00:06 <opendevmeet> Meeting started Tue Mar 14 16:00:06 2023 UTC and is due to finish in 60 minutes.  The chair is bauzas. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:00:06 <opendevmeet> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
16:00:06 <opendevmeet> The meeting name has been set to 'nova'
16:00:12 <bauzas> howdy folks
16:00:18 <dansmith> o/
16:00:21 <bauzas> #link https://wiki.openstack.org/wiki/Meetings/Nova#Agenda_for_next_meeting
16:01:04 <bauzas> ok, let's start, people can join meanwhile
16:01:10 <gibi> o/
16:01:10 <bauzas> #topic Bugs (stuck/critical)
16:01:15 <bauzas> #info No Critical bug
16:01:19 <bauzas> #link https://bugs.launchpad.net/nova/+bugs?search=Search&field.status=New 16 new untriaged bugs (+2 since the last meeting)
16:01:26 <elodilles> o/
16:01:30 <auniyal> o/
16:02:02 <bauzas> elodilles: I saw you did a bit of triage
16:02:07 <elodilles> yepp
16:02:08 <bauzas> I also looked at the new bugs todayt
16:02:16 <elodilles> thanks!
16:02:21 <bauzas> (to see whether we were regressing)
16:02:31 <bauzas> elodilles: nothing you want to discuss ?
16:02:43 <auniyal> bauzas, regarding this - https://bugs.launchpad.net/nova/+bug/2011567
16:02:52 <auniyal> The message right now is, "fill this in after the PTG", shall we update it to something else.
16:02:57 <gmann> o/
16:03:21 <bauzas> auniyal: as I said as a comment, we don't have cycle priorities
16:03:40 <bauzas> so, either we delete the Bobcat file, or meh
16:04:07 <sean-k-mooney> we have kept it for other release
16:04:17 <sean-k-mooney> its been blank for years at this point
16:04:18 <bauzas> yup, that's what I wrote
16:04:33 <bauzas> we can provide some upstream priorities if we want
16:04:40 <auniyal> okay, the msg seems like we missed to update after PTG
16:04:47 <bauzas> but, if we don't have anyone, should we delete the file ?
16:05:03 <sean-k-mooney> im ok either way
16:05:10 <bauzas> we could discuss this in the PTG if we want
16:05:14 <sean-k-mooney> sure
16:05:18 <bauzas> kk
16:05:18 <auniyal> ack
16:05:21 <bauzas> let's do this then
16:05:42 <bauzas> auniyal: I also closed your other bug report https://bugs.launchpad.net/nova/+bug/2011564
16:05:56 <bauzas> auniyal: as I said, those links are just examples
16:05:57 <auniyal> yes, that I missed the example
16:06:03 <auniyal> my bad
16:06:09 <bauzas> ok
16:06:15 <bauzas> moving on then
16:06:20 <elodilles> about your question bauzas : there was a serialproxy TR. serialproxy example client seems very old and not really functional (bug:  https://bugs.launchpad.net/nova/+bug/2009956 )
16:07:07 <elodilles> i guess low prio is OK for that
16:07:16 <elodilles> but correct me if i was wrong
16:07:32 <bauzas> elodilles: mmm, good question
16:07:46 <bauzas> 'low' is good for me
16:07:57 <sean-k-mooney> we expect the serial proxy to be used with netcat in generalright
16:08:48 <sean-k-mooney> oh actully we expeort it as a websocket
16:08:59 <sean-k-mooney> so you need a websocket client
16:09:04 <bauzas> sean-k-mooney: netcat or any script like https://docs.openstack.org/nova/latest/contributor/testing/serial-console.html
16:09:07 <elodilles> yes, rather a websocket client
16:09:29 <sean-k-mooney> ya so i dont really consider the example in the docs to be maintained
16:09:38 <sean-k-mooney> if its not tested its broken by default
16:09:41 <sean-k-mooney> and we dont test that
16:09:56 <elodilles> ack
16:10:24 <bauzas> well, then should we deprecate this API ?
16:10:27 <sean-k-mooney> so we proably shoudl fix this and  by either providing a new nova console script that we test or delete the current example and document using a diffent client
16:10:29 <bauzas> as a signal
16:10:32 <sean-k-mooney> no
16:10:36 <sean-k-mooney> it works fine in horizon
16:10:40 <bauzas> saying "sorry, we don't really test it"
16:10:42 <sean-k-mooney> or it did the last time i used it
16:10:56 <sean-k-mooney> we shoudl remove the example script
16:11:05 <bauzas> sean-k-mooney: looks to me elodilles tested this with master and it wasn't longer working
16:11:18 <sean-k-mooney> elodilles: did you test it with horizon
16:11:24 <sean-k-mooney> or that test script
16:11:55 <bauzas> I can try to test it with https://github.com/vi/websocat
16:11:56 <elodilles> sean-k-mooney: well, I don't know whether my devstack was properly configured, but it didn't work
16:12:13 <elodilles> neither with any old clients
16:12:46 <elodilles> so i don't know whether this is a problem in serial proxy or the clients
16:12:51 <sean-k-mooney> ack we shoudl look into this more i dont currently have a devstack but i can try and find time to deploy one and see
16:13:11 <sean-k-mooney> elodilles: if you can try configuring horizon to use it that would be the main way its used
16:13:16 <bauzas> me too
16:13:22 <elodilles> sean-k-mooney: ack
16:13:42 <bauzas> maybe let's put the importance to Medium, as this is an API
16:13:53 <sean-k-mooney> https://docs.openstack.org/nova/xena/admin/remote-console-access.html#serial
16:14:02 <sean-k-mooney> its not an api
16:14:18 <sean-k-mooney> but it is a fully supprot compenet
16:14:43 <bauzas> ah yes
16:14:45 <bauzas> you're right
16:14:56 <bauzas> a specific service
16:15:00 <sean-k-mooney> yep
16:15:08 <bauzas> with another port
16:15:11 <sean-k-mooney> its just an alternivte to the novnc proxy
16:15:15 <elodilles> anyway, i've changed its prio to Medium
16:15:24 <bauzas> anyway, let's not try to find the solution here
16:15:30 <elodilles> ++
16:15:45 <bauzas> elodilles: if you want, I can help you, ping me tomorrow
16:16:05 <bauzas> I don't think this is an Antelope regression btw.
16:16:36 <elodilles> bauzas: ack, thanks
16:16:44 <bauzas> moving on
16:16:54 <bauzas> #info Add yourself in the team bug roster if you want to help https://etherpad.opendev.org/p/nova-bug-triage-roster
16:17:00 <bauzas> #info bug baton is being passed to auniyal
16:17:05 <bauzas> auniyal: you're ok with that ?
16:17:11 <auniyal> yes,
16:18:16 <bauzas> cool
16:18:25 <bauzas> #topic Gate status
16:18:29 <bauzas> #link https://bugs.launchpad.net/nova/+bugs?field.tag=gate-failure Nova gate bugs
16:18:35 <bauzas> #link https://etherpad.opendev.org/p/nova-ci-failures
16:18:55 <bauzas> I'll be honest, I haven't uploaded a lot of implementation changes this week
16:19:08 <bauzas> so I had not found any problem
16:19:22 <bauzas> so looks to me the gate is now better :)
16:19:33 <elodilles> :]
16:19:34 <dansmith> yeah, lots of small changes have improved things a lot
16:19:37 <bauzas> but we'll see after a few time, when we're back
16:19:56 <bauzas> for the moment, the gate doesn't run a lot of jobs
16:20:02 <bauzas> job runs* I mean
16:20:29 <bauzas> and in general, we know the gate becomes bad around the milestones
16:20:39 <bauzas> so we'll see how it goes later
16:20:46 <bauzas> #link https://zuul.openstack.org/builds?project=openstack%2Fnova&project=openstack%2Fplacement&pipeline=periodic-weekly Nova&Placement periodic jobs status
16:20:50 <bauzas> all greens here
16:20:56 <bauzas> #info Please look at the gate failures and file a bug report with the gate-failure tag.
16:21:00 <bauzas> #info STOP DOING BLIND RECHECKS aka. 'recheck' https://docs.openstack.org/project-team-guide/testing.html#how-to-handle-test-failures
16:21:05 <bauzas> #topic Release Planning
16:21:10 <bauzas> #link https://releases.openstack.org/antelope/schedule.html
16:21:15 <bauzas> #info This week is the last time for creating a new RC if we need it.
16:21:23 <bauzas> #link https://etherpad.opendev.org/p/nova-antelope-rc-potential
16:21:33 <bauzas> #link https://bugs.launchpad.net/nova/+bugs?field.tag=antelope-rc-potential no open regression bugs were found.
16:21:46 <bauzas> so, yeah I looked at the open bugs and I haven't found any regression
16:22:30 <bauzas> I also tried to look at the 'In Progress' bugs
16:22:32 <bauzas> https://bugs.launchpad.net/nova/+bugs?search=Search&field.status=In%20Progress&orderby=-datecreated&start=0
16:22:34 <bauzas> by Age
16:24:17 <bauzas> amirite by saying none of the 8 in-progress bug reports we have since 2 months are not regressions ? I hope so
16:25:30 <bauzas> anyway, will continue to look at those bugs
16:25:47 <bauzas> because as I said, this is the last week for a new release candidate
16:26:12 <bauzas> #link https://releases.openstack.org/bobcat/schedule.html
16:26:26 <gibi> would be nice to have a release where we dont need RC2
16:26:41 <bauzas> you'll be interested in the below :
16:26:46 <bauzas> #link https://review.opendev.org/c/openstack/releases/+/877094 Proposed deadlines for Bobcat
16:27:15 <bauzas> we'll discuss those proposal dates at the PTG
16:27:21 <bauzas> vPTG I mean
16:27:49 <bauzas> but you can start to write comments if you wish
16:28:41 <bauzas> moving on I guess
16:28:43 <bauzas> oh
16:28:51 <bauzas> I was about to forge
16:28:53 <bauzas> forget
16:29:16 <bauzas> I created all bobcat launchpad series for placement, novaclient and nova
16:29:51 <bauzas> now, we said to try to remove placement from storyboard
16:30:01 <bauzas> so I started to look and it looks difficult
16:30:26 <bauzas> #info as a reminder, please no longer use Storyboard for creating feature requests or bug reports https://storyboard.openstack.org/#!/project/list?q=placement
16:30:45 <bauzas> but I'll try to see if I can do anything with https://opendev.org/ttygroup/boartty/
16:31:03 <bauzas> moving on
16:31:55 <bauzas> #topic vPTG Planning
16:32:06 <bauzas> usual reminder
16:32:08 <bauzas> #link https://www.eventbrite.com/e/project-teams-gathering-march-2023-tickets-483971570997 Register your free ticket
16:32:15 <bauzas> #link https://etherpad.opendev.org/p/nova-bobcat-ptg PTG etherpad
16:32:20 <bauzas> now this is time
16:32:24 <bauzas> #info please add your topics before end of this week, so I could provide an agenda on Monday.
16:32:33 <bauzas> this is important
16:32:49 <bauzas> as I don't see a lot of topics yet in the ptg etherpad
16:33:00 <bauzas> (well, I actually wrote the majority of them)
16:33:26 <bauzas> also, if you want to discuss with other teams, please add your topics in the cross-project agenda
16:33:43 <bauzas> and then I'll ping the other PTLs in order to find some time
16:34:09 <bauzas> as a reminder, I asked for 4 days of 4 hours
16:34:47 <bauzas> but if the agenda remains quite empty as it is, I'll probably unbook the Friday rooms
16:35:26 <sean-k-mooney> if it was in person i would suggest using the firday slot for an unconfernce/hackaton
16:35:38 <sean-k-mooney> but i kind of hate the idea of doing that virutally
16:35:44 <bauzas> don't get me on that direction :)
16:35:52 <sean-k-mooney> so sure
16:36:04 <sean-k-mooney> we could keep the slot and decied durign the ptg
16:36:09 <bauzas> I mean, after 3 vPTGs, I'm quite done with them
16:36:12 <sean-k-mooney> i.e. if there is a topic we ant to come back too
16:36:23 <bauzas> surely
16:36:44 <bauzas> and I'm pretty sure people will add topics on the last time like every cycle..
16:36:56 <bauzas> (tbc, I'm also doing this game)
16:37:19 <bauzas> but even with that, I just feel Friday will be either off or hackathon
16:37:45 <bauzas> fwiw, my main priorities for the beginning of Bobcat is to review some series we accepted
16:37:56 <bauzas> like the manila one, which I'll probably help too
16:38:00 <sean-k-mooney> if we have noting to do on firday im  fine with just say "thanks for coming folks" and doing a spec review day or something
16:38:13 <bauzas> sure
16:38:20 <bauzas> that's an idea
16:38:23 <bauzas> anyway
16:39:18 <bauzas> fwiw, the ptg main agenda is at the moment not large https://ptg.opendev.org/ptg.html
16:39:37 <bauzas> hopefully projects will book their rooms next week
16:40:25 <bauzas> (one thing I also think we miss with *virtual* PTGs is those kind of cross-project large discussions we had before)
16:40:48 <bauzas> but meh, moving on
16:41:09 <bauzas> #topic Review priorities
16:41:14 <bauzas> #link https://review.opendev.org/q/status:open+(project:openstack/nova+OR+project:openstack/placement+OR+project:openstack/os-traits+OR+project:openstack/os-resource-classes+OR+project:openstack/os-vif+OR+project:openstack/python-novaclient+OR+project:openstack/osc-placement)+(label:Review-Priority%252B1+OR+label:Review-Priority%252B2)
16:41:18 <bauzas> #info As a reminder, cores eager to review changes can +1 to indicate their interest, +2 for committing to the review
16:41:25 <bauzas> #topic Stable Branches
16:41:27 <bauzas> elodilles: your time
16:41:40 <elodilles> well, nothing special
16:41:49 <elodilles> i mean, not so many patches merge
16:41:59 <elodilles> but as far as i see:
16:42:09 <elodilles> #info stable gates seem to be OK - though it's hard to merge patches due to intermittent failures
16:42:16 <elodilles> #info stable branch status / gate failures tracking etherpad: https://etherpad.opendev.org/p/nova-stable-branch-ci
16:42:31 <elodilles> that's all :X
16:43:15 <bauzas> elodilles: dansmith: afaik, we haven't backported the mysqld memory reduction changes into stable branches ?
16:43:15 <dansmith> a bunch of the things we've fixed haven't been backported to stable,
16:43:21 <dansmith> so I expect that will remain challenging
16:43:22 <bauzas> hah, jinx
16:43:40 <bauzas> we could work on that
16:43:48 <dansmith> I haven't been monitoring the mysql memory thing, but it seems like that _has_ helped yeah/
16:44:27 <opendevreview> Alexey Stupnikov proposed openstack/nova master: Don't remove cached base images for failed resize ops  https://review.opendev.org/c/openstack/nova/+/877410
16:44:31 <dansmith> there was a potential for negative impacts, but I've heard no complaints
16:45:19 <elodilles> dansmith: do you have a topic set for those patches? so that I could see whether some could be backported?
16:45:21 <bauzas> mmmh
16:45:56 <dansmith> elodilles: this in devstack plus the flag enabled: https://review.opendev.org/c/openstack/devstack/+/873646
16:46:26 <elodilles> dansmith: thanks, i'll have a look
16:46:29 <bauzas> example here https://review.opendev.org/c/openstack/nova/+/874664
16:47:02 <bauzas> dansmith: but that means we would need to backport the devstack change too right?
16:47:12 <dansmith> that's why I said "this in devstack"
16:47:21 <dansmith> meaning you need it in the devstack branc you're running on
16:47:25 <elodilles> the nova patch is at least part of stable/2023.1 :)
16:47:27 <dansmith> which is kinda :/
16:48:13 <sean-k-mooney> so the memory patch proably shoudl be backported in devstack
16:48:14 <bauzas> yeah
16:48:23 <sean-k-mooney> becuase i think that might be useful on other stable branches
16:48:28 <bauzas> https://review.opendev.org/c/openstack/devstack/+/873646/3/lib/databases/mysql I understand dansmith's concerns
16:48:52 <bauzas> but it looks to me mysql doesn't bubble up the memory
16:48:54 <dansmith> I don't have specific concerns
16:49:08 <dansmith> if it's really that impactful, it's probably worth it,
16:49:18 <bauzas> do we have some mysql monitoring in devstack ?
16:49:19 <dansmith> it's just that it takes the devstack patch on each branch, plus job changes to enable
16:49:31 <dansmith> bauzas: we have my performance.json which has the info in it
16:49:56 <dansmith> gmann specifically didn't want to enable by default until bobcat (understandable)
16:50:02 <dansmith> so backporting to stable is kinda the opposite of that :)
16:50:14 <bauzas> yeah, I can understand
16:50:24 <bauzas> we shouldn't default this to all the jovs
16:50:26 <bauzas> obs
16:50:31 <bauzas> damn, jobs even
16:50:32 <sean-k-mooney> why not
16:50:38 <dansmith> the concerns are that it could slow down mysql and introduce other performance regressions that manifest as failures
16:50:44 <dansmith> it hasn't seemed to have done that in practice,
16:50:46 <sean-k-mooney> did we see a change in the job execution time
16:50:57 <sean-k-mooney> right
16:51:00 <dansmith> but the thought was to minimize the risk, in an already risky environment
16:51:05 <bauzas> sean-k-mooney: because of the fact we don't really monitor mysqld runs
16:51:12 <bauzas> in the logs
16:51:15 <dansmith> bauzas: what does that mean?
16:51:15 <sean-k-mooney> and reducing memory pressure might actully reduce swapping and speed up the job
16:51:51 <dansmith> bauzas: I think we do monitor it plenty, it's just that it's a pretty core function and breaking it could have lots and lots of wide impacts, both obvious and non-obvious
16:51:52 <bauzas> dansmith: correct me if I'm wrong, but do we trace the mysql performance in the logs, you said we have that in performance.yaml
16:52:06 <bauzas> I should take a look at this file
16:52:10 <bauzas> (tbh)
16:52:17 <dansmith> performance.json
16:52:30 <dansmith> there's a lot of data in there, but memory is probably the only relevant bit
16:52:39 <sean-k-mooney> we dont messure query reponce time as far as i know but we moditor memory usage
16:52:45 <dansmith> I'm just saying I don't know what "monitor mysqld runs" means in this context
16:53:16 <dansmith> yeah, no query time logging, but that would need to be done in aggregate to have any sort of meaningful result I think
16:53:29 <sean-k-mooney> as long as we are not seeing errors form teh services (timeouts) or longer overall job runs that all we really need to know
16:53:30 <dansmith> *and* it's time-based which is nearly impossible to compare across runs in the gate
16:54:01 <dansmith> sean-k-mooney: right, well, not having it on by default in master yet, we don't really have that large of a sample
16:54:13 <dansmith> we have a few jobs that are already atypical opt-ed into it
16:54:15 <bauzas> dansmith: I'll look at what we get from performance.yaml
16:54:31 <dansmith> anyway, I was good to go default on, but there's definitely risk so we just have to keep that in mind
16:54:48 <dansmith> bauzas: please stop saying performance.yaml :)
16:54:56 <dansmith> it's performance.json dammit :D
16:55:00 <bauzas> my question was more about the fact that given we have less large temporary tables and innodb pool sizes, I'd love to see some mysql insights about botyh
16:55:14 <bauzas> oh f, you're right
16:55:17 <bauzas> pardon my YAML
16:55:28 <sean-k-mooney> ...
16:55:38 <bauzas> anyway, we're quite at the end of the meeting
16:55:58 <bauzas> #topic Open discussion
16:56:10 <bauzas> the agenda is free from any item
16:56:16 <bauzas> so, anything to say ?
16:57:22 <bauzas> looks not
16:57:29 <bauzas> thanks all
16:57:34 <bauzas> #endmeeting