16:00:44 <gibi> #startmeeting nova
16:00:44 <opendevmeet> Meeting started Tue May  7 16:00:44 2024 UTC and is due to finish in 60 minutes.  The chair is gibi. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:00:44 <opendevmeet> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
16:00:44 <opendevmeet> The meeting name has been set to 'nova'
16:01:01 <erlon> o/
16:01:12 <tkajinam> o/
16:01:16 <gibi> bauzas is off today, so I will be your chair
16:01:16 <auniyal> o/
16:01:25 <elodilles> o/
16:01:45 <sean-k-mooney> dansmith: using local.sh works too
16:01:51 <sean-k-mooney> o/
16:01:54 <fwiesel> o/
16:01:54 <gibi> #link https://wiki.openstack.org/wiki/Meetings/Nova#Agenda_for_next_meeting
16:01:55 <dansmith> o/
16:02:35 <gibi> #topic Bugs (stuck/critical)
16:02:51 <gibi> #info No Critical bug
16:03:12 <gibi> is there any bug that needs discussion today?
16:03:26 <erlon> Must be critical I suppose?
16:04:33 <auniyal> gibi, not critical but seems imp to me https://bugs.launchpad.net/nova/+bug/2063342
16:04:38 <tkajinam> this is not very critical now but there is one remaining follow-up about excludes cleanup. would be nice if it can get 2nd +2 https://review.opendev.org/c/openstack/os-traits/+/917713
16:04:40 <gibi> erlon: if you feel it worth discussing with the foks then sure
16:05:50 <erlon> So, I need some reviews on the RBAC support patch. Its the last one fron the series: https://review.opendev.org/c/openstack/nova/+/811521?usp=search
16:06:40 <gibi> erlon: OK, noted.
16:06:50 <erlon> thanks
16:07:16 <gibi> auniyal: my immediate reaction that bug is to ask if they tried to archive deleted rows
16:07:20 <erlon> Should I add it to the etherepad or you do?
16:07:58 <gibi> erlon: I will
16:08:11 <erlon> Thanks
16:08:22 <gibi> auniyal: but reading it more it is about instance action
16:08:35 <gibi> growing without bound for non deleted instances
16:09:26 <sean-k-mooney> erlon's patch is for a blueprint that is not approved
16:09:27 <auniyal> yes, can we add  some preodic updated nova can do to DB
16:09:28 <gibi> and I see they proposed a nova-manage command to trim the action list. We at least need a blueprint for it
16:09:49 <sean-k-mooney> so it need to be appoved here first before we can proceed with it
16:10:02 <sean-k-mooney> https://blueprints.launchpad.net/nova/+spec/shared-security-groups
16:10:14 <erlon> what does that require? to aprove?
16:11:06 <sean-k-mooney> agreement that this does not requrie a spec and that we are ok to proceed with adding supprot for neutron SRBC for security groups to nova
16:11:52 <sean-k-mooney> basically this is a case of the new feature was added to neutron a few release ago and the nova side was not done
16:11:56 <erlon> ok, so I suppose that the blueprint was mostly a formality
16:12:26 <sean-k-mooney> well no the blueprint is alwasy requried for features like this but the quetion is does it also need a spec
16:13:14 <erlon> a spec seems a overkill
16:13:15 <gibi> erlon: does it need any nova API, db, RPC, or object model changes? Does it have upgrade impct?
16:13:33 <sean-k-mooney> in this particalr case there is not direct api impact. do we want to move this to the opendicussion stection by the way
16:13:36 <sean-k-mooney> or resolve this now
16:14:16 <gibi> lets resolve it
16:14:17 <sean-k-mooney> the reason i say there is no direct api impact is the api bechiaov changes but the requesst does not and i dont think we need a microversion
16:14:28 <gibi> OK.
16:14:42 <sean-k-mooney> so without this feature you can only reference secuirty groups that are created in your project
16:14:42 <gibi> I see it depends on a new neutron extension
16:14:55 <erlon> @gibi, no, just adds some calls to neutron client
16:15:04 <sean-k-mooney> with this feature you can use a secrutiy group defiend in another project but shread with your project
16:15:27 <sean-k-mooney> so before the change you would have got a error and after the vm would boot
16:15:46 <gibi> sean-k-mooney: that is a reasonable behavior change
16:16:02 <sean-k-mooney> yep its tradign a vm in error due to SecurityGroupNotFound: Security group 0c649378-1cf8-48e0-9eb4-b72772c35a62 not found. in swpan
16:16:05 <sean-k-mooney> for a booting vm
16:16:20 <sean-k-mooney> so i dont think that a breaking change and therfor no api impact
16:17:24 <sean-k-mooney> gibi: erlon  has a patch for review https://review.opendev.org/c/openstack/nova/+/811521 and the code changes are confied to the neutorn module in nova
16:17:25 <gibi> There is a preformance impact discussed in the impl, does that sever enough to pull it out to a spec?
16:17:54 <sean-k-mooney> the performace impact is 1 addtion call to list security group that are shared with you
16:18:26 <gibi> as far as I see if you don't use shared SG then you don't have the extra overhead
16:18:36 <sean-k-mooney> based on the trace back in the bug it got to the comptue manger beforfe that happend
16:18:39 <sean-k-mooney> so i think that ok
16:18:54 <gibi> OK. I'm not against approving this without a spec
16:18:54 <sean-k-mooney> am well no we dont know fi its shared or not
16:19:14 <sean-k-mooney> but we only need to make the call if neutron has the extention
16:19:34 <gibi> sean-k-mooney: I mean if the deployment doesn't use share SGs the the first query will find the grouo
16:19:53 <gibi> scratch it, I'm wrong
16:20:20 <sean-k-mooney> well if we reorded https://review.opendev.org/c/openstack/nova/+/811521/8/nova/network/neutron.py
16:20:25 <sean-k-mooney> we proably could avoid that ya
16:20:30 <gibi> anyhow I'm OK to approve this and continue discussing the impl
16:20:43 <gibi> do we feel we have chorum without bauzas ?
16:20:45 <sean-k-mooney> so im not against doing this as specless and ya defering to the gerrit review
16:20:56 <sean-k-mooney> dansmith: what do you think?
16:21:01 <gibi> quorum even
16:21:25 <sean-k-mooney> i dont know if gmann is around either
16:21:42 <sean-k-mooney> im tentivly ok with saying yes but mabey we can double check next week
16:22:08 <sean-k-mooney> "yes" -> approve this as specless and move discuuion to gerrit review
16:22:10 <gibi> OK. lets discuss it next week with the others around
16:22:20 <dansmith> yeah, sorry, multitasking
16:23:22 <gibi> ack, lets get back to this next week then
16:23:23 <dansmith> looks like basic enablement of a neutron feature though,
16:23:41 <dansmith> so unless it breaks some nova-side model it seems like specless would be fine
16:23:46 <gibi> OK
16:23:57 <dansmith> but also think it's probably fine/good to wait for bauzas anyway
16:24:13 <gibi> then I will let bauzas know that is it OK to approve from our side
16:24:17 <dansmith> ++
16:24:20 <gibi> thanks
16:24:58 <gibi> #action bauzas to approve //blueprints.launchpad.net/nova/+spec/shared-security-groups if he don't disagree
16:25:01 <opendevreview> ribaudr proposed openstack/nova-specs master: Update and Re-propose "Allow Manila shares to be directly attached to an instance when using libvirt" for Dalmatian  https://review.opendev.org/c/openstack/nova-specs/+/913997
16:25:52 <gibi> tkajinam: I will check https://review.opendev.org/c/openstack/os-traits/+/917713 after the meeting
16:26:07 <gibi> any other bug to discuss?
16:27:12 <tkajinam> gibi, thx
16:27:26 <gibi> moving on
16:27:39 <gibi> #topic Gate status
16:27:49 <gibi> #link https://bugs.launchpad.net/nova/+bugs?field.tag=gate-failure Nova gate bugs
16:28:03 <gibi> #link https://etherpad.opendev.org/p/nova-ci-failures-minimal
16:28:11 <gibi> #link https://zuul.openstack.org/builds?project=openstack%2Fnova&project=openstack%2Fplacement&pipeline=periodic-weekly Nova&Placement periodic jobs status
16:28:32 <gibi> emulation job failed on zed and antelope
16:29:02 <gibi> ohh actually failed on every branch
16:29:27 <sean-k-mooney> fun anything obvious
16:29:54 <gibi> this is the master failure https://f267574198a517c003d1-a40b9478ae1bd073dc32f331038fe6d7.ssl.cf5.rackcdn.com/periodic-weekly/opendev.org/openstack/nova/master/nova-emulation/825cc9a/testr_results.html
16:30:30 <gibi> I did not dig deeper
16:30:45 <auniyal> I added this https://review.opendev.org/c/openstack/tempest/+/918439 for same - is there a way we can run priodic depends on this ?
16:30:45 <auniyal> it only adds req-id
16:30:45 <gibi> would be nice to see if the failures on each branch are the same or not
16:30:49 <sean-k-mooney> Instance failed to spawn: oslo_messaging.rpc.client.RemoteError: Remote error: DBConnectionError (pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query')
16:30:59 <sean-k-mooney> so maybe an oom kill
16:31:05 <sean-k-mooney> of the db
16:31:24 <auniyal> I saw this for antelope https://zuul.openstack.org/build/183dfe690ad44431b023ef95ec347bd9
16:31:28 <gibi> OK, I can check the logs tomorrow for possible oom
16:31:57 <dansmith> emulation being slower, oom kill seems likely I  bet
16:32:02 <sean-k-mooney> we can move on lill quickly tak a look
16:32:18 <gibi> #action gibi to check the nova-emulation periodic failures for oom kill events
16:32:31 <gibi> any other gate issue we need to discuss today?
16:33:24 <sean-k-mooney> ay 04 08:51:02 np0037439600 kernel: nova-compute invoked oom-killer: gfp_mask=0x100cca(GFP_HIGHUSER_MOVABLE), order=0, oom_score_adj=0
16:33:32 <gibi> nice catch
16:33:46 <sean-k-mooney> oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/system.slice/mysql.service,task=mysqld,pid=63525,uid=116
16:33:56 <sean-k-mooney> so nova trigered it and it kills mysql
16:34:05 <sean-k-mooney> which is why the job was unhappy
16:34:23 <gibi> could you please file a bug for it? if you cannot the I will file it tomorrow
16:34:41 <sean-k-mooney> sure this might jut need swap or some tweak to the concurancy
16:34:46 <gibi> ack, thanks
16:34:50 <gibi> moving on
16:34:52 <gibi> #topic Release Planning
16:35:01 <gibi> #link https://releases.openstack.org/dalmatian/schedule.html
16:35:08 <gibi> #link https://review.opendev.org/c/openstack/releases/+/918422 nova specific schedule is proposed
16:35:17 <gibi> #info Dalmatian-1 in 1 week
16:35:25 <gibi> #info Spec review day is next Tuesday (14th of May)
16:36:13 <gibi> anything else about the release?
16:37:10 <gibi> #topic Review priorities
16:37:16 <gibi> #link https://etherpad.opendev.org/p/nova-dalmatian-status
16:37:30 <gibi> bauzas update the status etherpad
16:37:40 <gibi> *updated
16:37:53 <gibi> any questions about the priorities?
16:38:41 <gibi> #topic Stable Branches
16:38:45 <gibi> elodilles: your turn
16:38:52 <elodilles> yepp, so
16:38:59 <elodilles> #info no recent merges, but stable/2024.1 gate should be OK
16:39:09 <elodilles> #info stable/2023.2 gate is broken due to grenade-skip-level job (fix: https://review.opendev.org/c/openstack/grenade/+/918452 )
16:39:42 <elodilles> and it seems meanwhile, that the fix is not enough, as e.g. keystone haven't transitioned to unmaintained yet :S
16:40:16 <elodilles> #info stable/2023.1 gate is probably also broken due to grenade job, probably same issue
16:40:27 <elodilles> i'll try to look further tomorrow
16:40:56 <gibi> thanks
16:40:59 <elodilles> i've updated the stable gate pad:
16:41:02 <elodilles> #info stable branch status / gate failures tracking etherpad: https://etherpad.opendev.org/p/nova-stable-branch-ci
16:41:23 <elodilles> will continue to update that, too
16:41:31 <elodilles> that's all from me
16:41:38 <gibi> thanks elodilles
16:41:46 <elodilles> np
16:41:56 <gibi> any other stable topic from others?
16:42:43 <gibi> #topic vmwareapi 3rd-party CI efforts Highlights
16:42:55 <gibi> from the agenda
16:42:56 <gibi> #info Fixing some "lost" changes
16:43:16 <fwiesel> Hi, that's me...
16:43:21 <gibi> sure go ahead
16:43:45 <fwiesel> So, the way I get the changes are via ssh stream, if that gets restarted and there is a new change, that one would get lost.
16:43:56 <opendevreview> Merged openstack/os-traits master: Bump min versions to exclude known bad versions  https://review.opendev.org/c/openstack/os-traits/+/917713
16:43:59 <fwiesel> I work on polling all past changes to fix that.
16:44:13 <fwiesel> That's from my side, unless there are any questions?
16:44:42 <gibi> fwiesel: thanks for the update
16:44:48 <gibi> and the continued effort on the CI
16:45:52 <gibi> if no question from others then we move to our last topic today
16:46:17 <gibi> #topic Open discussion
16:46:22 <gibi> nothing on the agenda
16:46:34 <gibi> does anybody has anything else for today?
16:47:19 <tkajinam> nothing from me.
16:47:35 <gibi> It seem we are done for today.
16:47:38 <gibi> thanks for joining
16:47:46 <gibi> have a nice rest of the day
16:47:50 <gibi> #endmeeting