Tuesday, 2023-03-14

opendevreviewAmit Uniyal proposed openstack/nova stable/wallaby: fup: Print message logging uncaught nova-manage exceptions  https://review.opendev.org/c/openstack/nova/+/87733405:18
Ugglagibi, Hi I agree that currently nothing prevent leaking the share if you delete the VM.08:45
bauzasUggla: you tested it ?08:47
bauzasthat was my question 08:48
Ugglayes08:48
bauzasok, so 08:49
bauzashttps://review.opendev.org/c/openstack/nova/+/831193/24/nova/db/main/models.py#67708:49
bauzasthat means the FK would be not relationed08:50
Ugglagibi, I'm surprised regarding the mount error not tracked, any idea about what is causing the issue. Firewalling maybe ?08:51
gibiUggla: what manila config you used for testing? I tried the default that is DHSS=True + GenericDriver08:53
Ugglagibi, DHSS=false LVM driver08:55
gibiUggla: is there a reason why GenericDriver + DHSS=true would not work?08:55
gibi(I'm restacking with LVM DSSF=false now)08:55
gibi*DHSS08:56
Ugglagibi, to be honest I have not tested with DHSS=false thinking that in our context that was not necessary. So I don't know. :(08:57
bauzasso, IMHO, we should delete the share mapping in the instance delete call08:58
bauzasas we can't use the delete cascade SQL support08:58
Ugglabauzas, yep and add the semaphore to avoid any race.08:58
bauzas+ in init_instance(), recreating the share like I said08:59
bauzasand maybe a periodic for making sure we don't leak any shares08:59
bauzasUggla: can you look at what happens to the foreign key ?09:00
Ugglabauzas, yes I'll try to simulate it again.09:01
bauzascool09:03
opendevreviewHiroki Narukawa proposed openstack/nova master: libvirt: retry libvirt connection on live_migration_monitor  https://review.opendev.org/c/openstack/nova/+/86707710:57
opendevreviewRajesh Tailor proposed openstack/nova master: Fix duplicate cell creation with same name  https://review.opendev.org/c/openstack/nova/+/87694013:35
bauzasgibi: sean-k-mooney: other cores, we need this https://review.opendev.org/c/openstack/nova-specs/+/876887 to be merged quick soon please14:17
bauzasquite* soon14:17
sean-k-mooneyah the redirects sure ill look now14:21
gibigood for me14:21
sean-k-mooneysame looks fine14:22
sean-k-mooneyim happy the script worked with the new name format14:22
opendevreviewMerged openstack/nova-specs master: Move Antelope implemented specs  https://review.opendev.org/c/openstack/nova-specs/+/87688714:30
opendevreviewMaxim Monin proposed openstack/nova master: Server Rescue leads to Server ERROR state if base image is deleted  https://review.opendev.org/c/openstack/nova/+/87238514:36
bauzasthanks folks14:38
gibibauzas: I'm happy that you are only asking me to review these realtively easy patches as otherwise I would have to say no :)15:17
bauzas#startmeeting nova16:00
opendevmeetMeeting started Tue Mar 14 16:00:06 2023 UTC and is due to finish in 60 minutes.  The chair is bauzas. Information about MeetBot at http://wiki.debian.org/MeetBot.16:00
opendevmeetUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.16:00
opendevmeetThe meeting name has been set to 'nova'16:00
bauzashowdy folks16:00
dansmitho/16:00
bauzas#link https://wiki.openstack.org/wiki/Meetings/Nova#Agenda_for_next_meeting16:00
bauzasok, let's start, people can join meanwhile16:01
gibio/16:01
bauzas#topic Bugs (stuck/critical) 16:01
bauzas#info No Critical bug16:01
bauzas#link https://bugs.launchpad.net/nova/+bugs?search=Search&field.status=New 16 new untriaged bugs (+2 since the last meeting)16:01
elodilleso/16:01
auniyalo/16:01
bauzaselodilles: I saw you did a bit of triage16:02
elodillesyepp16:02
bauzasI also looked at the new bugs todayt16:02
elodillesthanks!16:02
bauzas(to see whether we were regressing)16:02
bauzaselodilles: nothing you want to discuss ?16:02
auniyalbauzas, regarding this - https://bugs.launchpad.net/nova/+bug/201156716:02
auniyalThe message right now is, "fill this in after the PTG", shall we update it to something else.16:02
gmanno/16:02
bauzasauniyal: as I said as a comment, we don't have cycle priorities 16:03
bauzasso, either we delete the Bobcat file, or meh16:03
sean-k-mooneywe have kept it for other release16:04
sean-k-mooneyits been blank for years at this point16:04
bauzasyup, that's what I wrote16:04
bauzaswe can provide some upstream priorities if we want16:04
auniyalokay, the msg seems like we missed to update after PTG16:04
bauzasbut, if we don't have anyone, should we delete the file ?16:04
sean-k-mooneyim ok either way16:05
bauzaswe could discuss this in the PTG if we want16:05
sean-k-mooneysure16:05
bauzaskk16:05
auniyalack16:05
bauzaslet's do this then16:05
bauzasauniyal: I also closed your other bug report https://bugs.launchpad.net/nova/+bug/201156416:05
bauzasauniyal: as I said, those links are just examples16:05
auniyalyes, that I missed the example16:05
auniyalmy bad 16:06
bauzasok16:06
bauzasmoving on then16:06
elodillesabout your question bauzas : there was a serialproxy TR. serialproxy example client seems very old and not really functional (bug:  https://bugs.launchpad.net/nova/+bug/2009956 )16:06
elodillesi guess low prio is OK for that16:07
elodillesbut correct me if i was wrong16:07
bauzaselodilles: mmm, good question16:07
bauzas'low' is good for me16:07
sean-k-mooneywe expect the serial proxy to be used with netcat in generalright16:07
sean-k-mooneyoh actully we expeort it as a websocket16:08
sean-k-mooneyso you need a websocket client16:08
bauzassean-k-mooney: netcat or any script like https://docs.openstack.org/nova/latest/contributor/testing/serial-console.html16:09
elodillesyes, rather a websocket client16:09
sean-k-mooneyya so i dont really consider the example in the docs to be maintained16:09
sean-k-mooneyif its not tested its broken by default16:09
sean-k-mooneyand we dont test that16:09
elodillesack16:09
bauzaswell, then should we deprecate this API ?16:10
sean-k-mooneyso we proably shoudl fix this and  by either providing a new nova console script that we test or delete the current example and document using a diffent client16:10
bauzasas a signal16:10
sean-k-mooneyno16:10
sean-k-mooneyit works fine in horizon16:10
bauzassaying "sorry, we don't really test it"16:10
sean-k-mooneyor it did the last time i used it16:10
sean-k-mooneywe shoudl remove the example script16:10
bauzassean-k-mooney: looks to me elodilles tested this with master and it wasn't longer working16:11
sean-k-mooneyelodilles: did you test it with horizon16:11
sean-k-mooneyor that test script16:11
bauzasI can try to test it with https://github.com/vi/websocat16:11
elodillessean-k-mooney: well, I don't know whether my devstack was properly configured, but it didn't work16:11
elodillesneither with any old clients16:12
elodillesso i don't know whether this is a problem in serial proxy or the clients16:12
sean-k-mooneyack we shoudl look into this more i dont currently have a devstack but i can try and find time to deploy one and see16:12
sean-k-mooneyelodilles: if you can try configuring horizon to use it that would be the main way its used16:13
bauzasme too16:13
elodillessean-k-mooney: ack16:13
bauzasmaybe let's put the importance to Medium, as this is an API 16:13
sean-k-mooneyhttps://docs.openstack.org/nova/xena/admin/remote-console-access.html#serial16:13
sean-k-mooneyits not an api16:14
sean-k-mooneybut it is a fully supprot compenet16:14
bauzasah yes16:14
bauzasyou're right16:14
bauzasa specific service 16:14
sean-k-mooneyyep16:15
bauzaswith another port16:15
sean-k-mooneyits just an alternivte to the novnc proxy16:15
elodillesanyway, i've changed its prio to Medium16:15
bauzasanyway, let's not try to find the solution here16:15
elodilles++16:15
bauzaselodilles: if you want, I can help you, ping me tomorrow16:15
bauzasI don't think this is an Antelope regression btw.16:16
elodillesbauzas: ack, thanks16:16
bauzasmoving on16:16
bauzas#info Add yourself in the team bug roster if you want to help https://etherpad.opendev.org/p/nova-bug-triage-roster16:16
bauzas#info bug baton is being passed to auniyal16:17
bauzasauniyal: you're ok with that ?16:17
auniyalyes,16:17
bauzascool16:18
bauzas#topic Gate status 16:18
bauzas#link https://bugs.launchpad.net/nova/+bugs?field.tag=gate-failure Nova gate bugs 16:18
bauzas#link https://etherpad.opendev.org/p/nova-ci-failures16:18
bauzasI'll be honest, I haven't uploaded a lot of implementation changes this week16:18
bauzasso I had not found any problem 16:19
bauzasso looks to me the gate is now better :)16:19
elodilles:]16:19
dansmithyeah, lots of small changes have improved things a lot16:19
bauzasbut we'll see after a few time, when we're back16:19
bauzasfor the moment, the gate doesn't run a lot of jobs16:19
bauzasjob runs* I mean16:20
bauzasand in general, we know the gate becomes bad around the milestones16:20
bauzasso we'll see how it goes later16:20
bauzas#link https://zuul.openstack.org/builds?project=openstack%2Fnova&project=openstack%2Fplacement&pipeline=periodic-weekly Nova&Placement periodic jobs status16:20
bauzasall greens here16:20
bauzas#info Please look at the gate failures and file a bug report with the gate-failure tag.16:20
bauzas#info STOP DOING BLIND RECHECKS aka. 'recheck' https://docs.openstack.org/project-team-guide/testing.html#how-to-handle-test-failures16:21
bauzas#topic Release Planning 16:21
bauzas#link https://releases.openstack.org/antelope/schedule.html16:21
bauzas#info This week is the last time for creating a new RC if we need it.16:21
bauzas#link https://etherpad.opendev.org/p/nova-antelope-rc-potential16:21
bauzas#link https://bugs.launchpad.net/nova/+bugs?field.tag=antelope-rc-potential no open regression bugs were found.16:21
bauzasso, yeah I looked at the open bugs and I haven't found any regression16:21
bauzasI also tried to look at the 'In Progress' bugs 16:22
bauzashttps://bugs.launchpad.net/nova/+bugs?search=Search&field.status=In%20Progress&orderby=-datecreated&start=016:22
bauzasby Age16:22
bauzasamirite by saying none of the 8 in-progress bug reports we have since 2 months are not regressions ? I hope so16:24
bauzasanyway, will continue to look at those bugs16:25
bauzasbecause as I said, this is the last week for a new release candidate16:25
bauzas#link https://releases.openstack.org/bobcat/schedule.html16:26
gibiwould be nice to have a release where we dont need RC216:26
bauzasyou'll be interested in the below :16:26
bauzas#link https://review.opendev.org/c/openstack/releases/+/877094 Proposed deadlines for Bobcat16:26
bauzaswe'll discuss those proposal dates at the PTG16:27
bauzasvPTG I mean16:27
bauzasbut you can start to write comments if you wish16:27
bauzasmoving on I guess16:28
bauzasoh 16:28
bauzasI was about to forge16:28
bauzasforget16:28
bauzasI created all bobcat launchpad series for placement, novaclient and nova16:29
bauzasnow, we said to try to remove placement from storyboard16:29
bauzasso I started to look and it looks difficult16:30
bauzas#info as a reminder, please no longer use Storyboard for creating feature requests or bug reports https://storyboard.openstack.org/#!/project/list?q=placement16:30
bauzasbut I'll try to see if I can do anything with https://opendev.org/ttygroup/boartty/16:30
bauzasmoving on16:31
bauzas#topic vPTG Planning 16:31
bauzasusual reminder 16:32
bauzas#link https://www.eventbrite.com/e/project-teams-gathering-march-2023-tickets-483971570997 Register your free ticket16:32
bauzas#link https://etherpad.opendev.org/p/nova-bobcat-ptg PTG etherpad16:32
bauzasnow this is time16:32
bauzas#info please add your topics before end of this week, so I could provide an agenda on Monday.16:32
bauzasthis is important16:32
bauzasas I don't see a lot of topics yet in the ptg etherpad16:32
bauzas(well, I actually wrote the majority of them)16:33
bauzasalso, if you want to discuss with other teams, please add your topics in the cross-project agenda16:33
bauzasand then I'll ping the other PTLs in order to find some time16:33
bauzasas a reminder, I asked for 4 days of 4 hours16:34
bauzasbut if the agenda remains quite empty as it is, I'll probably unbook the Friday rooms16:34
sean-k-mooneyif it was in person i would suggest using the firday slot for an unconfernce/hackaton16:35
sean-k-mooneybut i kind of hate the idea of doing that virutally16:35
bauzasdon't get me on that direction :)16:35
sean-k-mooneyso sure 16:35
sean-k-mooneywe could keep the slot and decied durign the ptg16:36
bauzasI mean, after 3 vPTGs, I'm quite done with them16:36
sean-k-mooneyi.e. if there is a topic we ant to come back too16:36
bauzassurely16:36
bauzasand I'm pretty sure people will add topics on the last time like every cycle..16:36
bauzas(tbc, I'm also doing this game)16:36
bauzasbut even with that, I just feel Friday will be either off or hackathon16:37
bauzasfwiw, my main priorities for the beginning of Bobcat is to review some series we accepted16:37
bauzaslike the manila one, which I'll probably help too16:37
sean-k-mooneyif we have noting to do on firday im  fine with just say "thanks for coming folks" and doing a spec review day or something16:38
bauzassure16:38
bauzasthat's an idea16:38
bauzasanyway16:38
bauzasfwiw, the ptg main agenda is at the moment not large https://ptg.opendev.org/ptg.html16:39
bauzashopefully projects will book their rooms next week16:39
bauzas(one thing I also think we miss with *virtual* PTGs is those kind of cross-project large discussions we had before)16:40
bauzasbut meh, moving on16:40
bauzas#topic Review priorities 16:41
bauzas#link https://review.opendev.org/q/status:open+(project:openstack/nova+OR+project:openstack/placement+OR+project:openstack/os-traits+OR+project:openstack/os-resource-classes+OR+project:openstack/os-vif+OR+project:openstack/python-novaclient+OR+project:openstack/osc-placement)+(label:Review-Priority%252B1+OR+label:Review-Priority%252B2)16:41
bauzas#info As a reminder, cores eager to review changes can +1 to indicate their interest, +2 for committing to the review16:41
bauzas#topic Stable Branches 16:41
bauzaselodilles: your time16:41
elodilleswell, nothing special16:41
elodillesi mean, not so many patches merge16:41
elodillesbut as far as i see:16:41
elodilles#info stable gates seem to be OK - though it's hard to merge patches due to intermittent failures16:42
elodilles#info stable branch status / gate failures tracking etherpad: https://etherpad.opendev.org/p/nova-stable-branch-ci16:42
elodillesthat's all :X16:42
bauzaselodilles: dansmith: afaik, we haven't backported the mysqld memory reduction changes into stable branches ?16:43
dansmitha bunch of the things we've fixed haven't been backported to stable,16:43
dansmithso I expect that will remain challenging16:43
bauzashah, jinx16:43
bauzaswe could work on that 16:43
dansmithI haven't been monitoring the mysql memory thing, but it seems like that _has_ helped yeah/16:43
opendevreviewAlexey Stupnikov proposed openstack/nova master: Don't remove cached base images for failed resize ops  https://review.opendev.org/c/openstack/nova/+/87741016:44
dansmiththere was a potential for negative impacts, but I've heard no complaints16:44
elodillesdansmith: do you have a topic set for those patches? so that I could see whether some could be backported?16:45
bauzasmmmh16:45
dansmithelodilles: this in devstack plus the flag enabled: https://review.opendev.org/c/openstack/devstack/+/87364616:45
elodillesdansmith: thanks, i'll have a look16:46
bauzasexample here https://review.opendev.org/c/openstack/nova/+/87466416:46
bauzasdansmith: but that means we would need to backport the devstack change too right?16:47
dansmiththat's why I said "this in devstack"16:47
dansmithmeaning you need it in the devstack branc you're running on16:47
elodillesthe nova patch is at least part of stable/2023.1 :)16:47
dansmithwhich is kinda :/16:47
sean-k-mooneyso the memory patch proably shoudl be backported in devstack16:48
bauzasyeah16:48
sean-k-mooneybecuase i think that might be useful on other stable branches 16:48
bauzashttps://review.opendev.org/c/openstack/devstack/+/873646/3/lib/databases/mysql I understand dansmith's concerns 16:48
bauzasbut it looks to me mysql doesn't bubble up the memory16:48
dansmithI don't have specific concerns16:48
dansmithif it's really that impactful, it's probably worth it,16:49
bauzasdo we have some mysql monitoring in devstack ?16:49
dansmithit's just that it takes the devstack patch on each branch, plus job changes to enable16:49
dansmithbauzas: we have my performance.json which has the info in it16:49
dansmithgmann specifically didn't want to enable by default until bobcat (understandable)16:49
dansmithso backporting to stable is kinda the opposite of that :)16:50
bauzasyeah, I can understand16:50
bauzaswe shouldn't default this to all the jovs16:50
bauzasobs16:50
bauzasdamn, jobs even16:50
sean-k-mooneywhy not16:50
dansmiththe concerns are that it could slow down mysql and introduce other performance regressions that manifest as failures16:50
dansmithit hasn't seemed to have done that in practice,16:50
sean-k-mooneydid we see a change in the job execution time16:50
sean-k-mooneyright16:50
dansmithbut the thought was to minimize the risk, in an already risky environment16:51
bauzassean-k-mooney: because of the fact we don't really monitor mysqld runs16:51
bauzasin the logs16:51
dansmithbauzas: what does that mean?16:51
sean-k-mooneyand reducing memory pressure might actully reduce swapping and speed up the job16:51
dansmithbauzas: I think we do monitor it plenty, it's just that it's a pretty core function and breaking it could have lots and lots of wide impacts, both obvious and non-obvious16:51
bauzasdansmith: correct me if I'm wrong, but do we trace the mysql performance in the logs, you said we have that in performance.yaml16:51
bauzasI should take a look at this file16:52
bauzas(tbh)16:52
dansmithperformance.json16:52
dansmiththere's a lot of data in there, but memory is probably the only relevant bit16:52
sean-k-mooneywe dont messure query reponce time as far as i know but we moditor memory usage16:52
dansmithI'm just saying I don't know what "monitor mysqld runs" means in this context16:52
dansmithyeah, no query time logging, but that would need to be done in aggregate to have any sort of meaningful result I think16:53
sean-k-mooneyas long as we are not seeing errors form teh services (timeouts) or longer overall job runs that all we really need to know16:53
dansmith*and* it's time-based which is nearly impossible to compare across runs in the gate16:53
dansmithsean-k-mooney: right, well, not having it on by default in master yet, we don't really have that large of a sample16:54
dansmithwe have a few jobs that are already atypical opt-ed into it16:54
bauzasdansmith: I'll look at what we get from performance.yaml16:54
dansmithanyway, I was good to go default on, but there's definitely risk so we just have to keep that in mind16:54
dansmithbauzas: please stop saying performance.yaml :)16:54
dansmithit's performance.json dammit :D16:54
bauzasmy question was more about the fact that given we have less large temporary tables and innodb pool sizes, I'd love to see some mysql insights about botyh16:55
bauzasoh f, you're right16:55
bauzaspardon my YAML16:55
sean-k-mooney...16:55
bauzasanyway, we're quite at the end of the meeting16:55
bauzas#topic Open discussion 16:55
bauzasthe agenda is free from any item 16:56
bauzasso, anything to say ?16:56
bauzaslooks not16:57
bauzasthanks all16:57
bauzas#endmeeting16:57
opendevmeetMeeting ended Tue Mar 14 16:57:34 2023 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)16:57
opendevmeetMinutes:        https://meetings.opendev.org/meetings/nova/2023/nova.2023-03-14-16.00.html16:57
opendevmeetMinutes (text): https://meetings.opendev.org/meetings/nova/2023/nova.2023-03-14-16.00.txt16:57
opendevmeetLog:            https://meetings.opendev.org/meetings/nova/2023/nova.2023-03-14-16.00.log.html16:57
sean-k-mooneybauzas: for what its worth i remember turnign mysql for low memory foot print many years ago when i first started workign on openstack16:59
sean-k-mooneythere used to be a blog or doc that told you how to do it16:59
sean-k-mooneyi dont know where that was cause its been years but this type of change is not new to me16:59
bauzassean-k-mooney: AFAIR, you can use mysql's information schema tables to get the recommended pool sizes17:00
bauzasbut I was a mysql operator a decade ago and my brain is fried now17:00
sean-k-mooneyprobably but i dont really have concern with the value that are bing used17:00
sean-k-mooneyi think we used to do this in the intel nfv ci17:00
sean-k-mooneyto reduce memoryusage because we were deploying with ovs-dpdk17:01
bauzasfor the temp tables, well that's less of a concern17:01
bauzasiirc, if the tmp tables aren't large enough, this just goes on disk17:01
sean-k-mooneyno it was for the pools as well17:01
bauzasso this becomes an I/O performance question17:02
sean-k-mooneyright i dont think it will be in our usage17:02
sean-k-mooneywe will see but i would prefer to see by turning this on by default in all the jobs17:02
sean-k-mooneyand seeign if we see any regressions17:02
bauzasso, I double-checked and yeah the pools are for caching17:03
sean-k-mooneyyep17:03
bauzasso in case the pools aren't large enough, this is just a disk io performance concern as well17:03
sean-k-mooneyboth of result sets and quey execution plans i belive17:03
sean-k-mooneybauzas: yep but we are already swappign to disk in several jobs17:04
sean-k-mooneyif we can reduce that it likely will improve perfromce over all17:04
bauzasyeah, now I refresh my mind17:04
sean-k-mooneybasically what im saying is any perfromance hit form the mysql turning will likely be offset by reduced memory pressure over all17:05
bauzasI remember we can call the innodb engine to get the performance metrics17:05
auniyalthis :(  https://zuul.openstack.org/status#864055, its always fail on gate18:11
* gibi gibi_pto18:37
*** gibi is now known as gibi_pto18:38

Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!