16:01:26 <bauzas> #startmeeting nova
16:01:26 <opendevmeet> Meeting started Tue Jun 27 16:01:26 2023 UTC and is due to finish in 60 minutes.  The chair is bauzas. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:01:26 <opendevmeet> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
16:01:26 <opendevmeet> The meeting name has been set to 'nova'
16:01:31 <bauzas> hey folks \o
16:01:33 <elodilles> o/
16:01:34 <auniyal> o/
16:01:39 <gibi> o\
16:01:40 <bauzas> #link https://wiki.openstack.org/wiki/Meetings/Nova#Agenda_for_next_meeting
16:02:54 <bauzas> ok let's start while the folks are arriving :)
16:03:02 <bauzas> #topic Bugs (stuck/critical)
16:03:10 <bauzas> #info No Critical bug
16:03:14 <bauzas> #link https://bugs.launchpad.net/nova/+bugs?search=Search&field.status=New 31 new untriaged bugs (+4 since the last meeting)
16:03:19 <Uggla_> o/
16:03:22 <bauzas> I only had time to look at two bugs this week
16:03:24 <bauzas> sorriy
16:03:37 <bauzas> I can try to look at the bugs this new week
16:05:24 <bauzas> okay, nobody is against
16:05:32 <bauzas> #info Add yourself in the team bug roster if you want to help https://etherpad.opendev.org/p/nova-bug-triage-roster
16:05:39 <bauzas> #info bug baton is being passed to bauzas
16:05:51 <gibi> I would be super suprised if somebody want to stole the baton :D
16:05:52 <bauzas> do people want to discuss on bugs ?
16:05:59 <bauzas> gibi: :)
16:06:29 <bauzas> looks not
16:06:34 <bauzas> #topic Gate status
16:06:40 <bauzas> #link https://bugs.launchpad.net/nova/+bugs?field.tag=gate-failure Nova gate bugs
16:07:12 <bauzas> it looks to me we have a new bug :(
16:07:14 <bauzas> #link https://bugs.launchpad.net/nova/+bug/2025096
16:07:29 <dansmith> it's been skipped for now,
16:07:43 <dansmith> but that appears to just have not been tested on ceph and something is wrong with it in that environment
16:08:03 <dansmith> I'm not sure about the non-ceph assertion in the comment, I'll have to look
16:08:12 <bauzas> dansmith: ack
16:08:13 <dansmith> but that's a test we should have landed a year ago
16:08:16 <dansmith> and it just got merged
16:08:19 <bauzas> I was looking at Zuul
16:09:19 <bauzas> anyway, like you say, the job is not failed https://zuul.openstack.org/builds?job_name=nova-ceph-multistore&skip=0
16:10:34 <bauzas> dansmith: https://b932a1446345e101b3ef-4740624f0848c8c3257f704064a4516f.ssl.cf5.rackcdn.com/883557/2/check/nova-ceph-multistore/d7db64f/testr_results.html
16:10:48 <bauzas> the test is not skipped
16:11:23 <dansmith> https://review.opendev.org/c/openstack/devstack-plugin-ceph/+/887003
16:11:42 <dansmith> we should be inheriting from that afaik
16:12:04 <bauzas> ahah
16:12:06 <bauzas> ok
16:12:30 <dansmith> why do you say it's not skipped?
16:12:45 <bauzas> gmann: could you add a new change for skipping test_rebuild_volume_backed_server on nova-ceph-multistore
16:12:47 <bauzas> ?
16:13:14 <bauzas> dansmith: oh wait, verifying
16:13:42 <dansmith> the patch specifically calls out the nova job
16:14:02 <bauzas> yeah, you're right, the test is now skipped
16:14:07 <bauzas> gmann: nevermind what I said
16:14:17 <bauzas> https://6354f24cae23db064c15-540aa9666e0f6140cd4132fd113878d1.ssl.cf5.rackcdn.com/876075/14/check/nova-ceph-multistore/617bfe5/testr_results.html I don't see the test
16:14:46 <bauzas> dansmith: sorry, I misunderstood what you said
16:15:28 <dansmith> ..okay
16:15:55 <bauzas> dansmith: sometimes my brain doesn't run correctly
16:16:04 <bauzas> :(
16:16:25 <bauzas> anyway, moving on
16:16:32 <bauzas> #link https://zuul.openstack.org/builds?project=openstack%2Fnova&project=openstack%2Fplacement&pipeline=periodic-weekly Nova&Placement periodic jobs status
16:16:43 <bauzas> nothing to relate
16:16:50 <bauzas> #info Please look at the gate failures and file a bug report with the gate-failure tag.
16:16:56 <bauzas> #info STOP DOING BLIND RECHECKS aka. 'recheck' https://docs.openstack.org/project-team-guide/testing.html#how-to-handle-test-failures
16:17:06 <bauzas> (I could remove this note next week)
16:17:32 <bauzas> moving on then, unless folks want to discuss on some other CI failures
16:17:59 <bauzas> #topic Release Planning
16:18:04 <bauzas> #link https://releases.openstack.org/bobcat/schedule.html
16:18:10 <bauzas> #info Nova deadlines are set in the above schedule
16:18:15 <bauzas> #info Nova spec review day today (June 27th)
16:18:24 <bauzas> I started to review a few specs
16:18:27 <bauzas> I approved one
16:18:43 <bauzas> I still have two other specs to look at
16:19:09 <bauzas> if I don't have time to look at them today, I'll continue to review the specs tomorrow
16:19:31 <bauzas> and tomorrow, I'll add an etherpad about the already merged specs and their code
16:19:58 <bauzas> as an important reminder,
16:20:00 <bauzas> #link https://releases.openstack.org/bobcat/schedule.html#b-nova-spec-freeze Nova Spec Freeze on July 6th !
16:20:08 <bauzas> which is next week
16:20:26 <bauzas> most of the specs I've seen already have implementation changes
16:20:41 <bauzas> so thanks folks for that
16:21:11 <bauzas> anything anyone that wants to discuss on either specs or our release planning ?
16:21:52 <bauzas> -
16:22:01 <bauzas> moving on then
16:22:09 <bauzas> #topic Stable Branches
16:22:16 <bauzas> elodilles: wants to discuss anything ?
16:22:26 <elodilles> bauzas: nope, actually
16:22:31 <bauzas> cool cool
16:22:38 <elodilles> not much happened recently
16:22:40 <elodilles> around stable
16:22:46 <bauzas> #info stable gates should be OK (from stable/2023.1 to stable/train)
16:22:51 <bauzas> #info stable branch status / gate failures tracking etherpad: https://etherpad.opendev.org/p/nova-stable-branch-ci
16:22:57 <bauzas> elodilles: no news is good :)
16:23:08 <elodilles> :)
16:23:08 <bauzas> #info train-eol patch proposed https://review.opendev.org/c/openstack/releases/+/885365
16:23:14 <bauzas> elodilles: I've seen your comment
16:23:23 <elodilles> yepp, it needs an update
16:23:51 <bauzas> so you're also asking to EOL placement, os-vif, os-resource-classes, and the clients ?
16:24:04 <bauzas> I can work on it
16:24:22 <elodilles> yes, i guess those would be good to EOL together
16:24:24 <elodilles> with nova
16:24:36 <elodilles> as no reason to keep them in EM, right?
16:24:40 <elodilles> without nova...
16:24:45 <bauzas> elodilles: also, my question is, shall we abandon the changes in the stable branches, or would it be done automatically ?
16:24:58 <gibi> os-traits and os-resource classes should be tight to the placement eol
16:25:12 <elodilles> they need to be abandoned, there's no automation
16:25:14 <bauzas> elodilles: for os-vif and novaclient, sounds ok
16:25:23 <bauzas> for placement dependencies, I'm quite torn
16:25:29 <gibi> but os-traits might also be used be neutron
16:25:51 <bauzas> gibi: good call, then let's stick to os-vif and novaclient
16:26:01 <elodilles> i guess if nova eol's it's train branch then maybe other projects will do the same
16:26:06 <bauzas> we could discuss about EOLing placement and its deps on another change
16:26:20 <gibi> bauzas: os-vif is also used by neutron for sure
16:26:21 <elodilles> (and i won't insist to keep open neutron either in that case o:)
16:26:38 <bauzas> elodilles: I've heard other projects that would EOL *all* their EM branches but I haven't seen yet any change :)
16:26:49 <gibi> python-novaclient is used by cinder and neutron for the external events no?
16:26:54 <elodilles> bauzas: yes, i only seen the cinder patch yet
16:27:02 <bauzas> gibi: good point, let's then be conservative and only EOL novaclient
16:27:02 <gibi> also heat probably uses nova-pythonclient
16:27:28 <bauzas> elodilles: what have we done when we EOL'd Stein ?
16:27:30 <gibi> can we delete all the train branches at once? :D
16:27:36 <bauzas> I thought nova did it in advance
16:27:47 <elodilles> heat is already EOL'd up to ussuri
16:28:27 <elodilles> bauzas: if you ask about the open patches: i ussually abandon the patches to be able to delete the branch :)
16:28:57 <elodilles> so i can abandon them whenever there is a decision
16:28:58 <bauzas> elodilles: so I need to click on *every* open patch to abandon it ?
16:29:27 <bauzas> disclaimer, my office room is 28°C now, my brain is absolutely fried.
16:29:31 <elodilles> bauzas: i can do that (there aren't many open patches)
16:30:09 <bauzas> so, what's the outcome of this long conversation ,
16:30:11 <bauzas> ?
16:30:19 <elodilles> here the temperature is quite moderate now 21°C :)
16:30:19 <bauzas> I guess only nova and novaclient can be EOL now
16:30:43 <bauzas> elodilles: it's 28°C *inside* my room (I don't have any A/C)
16:30:47 <gibi> bauzas: I'm not sure about python-novaclient
16:31:01 <bauzas> okay, so let's be strict
16:31:05 <bauzas> and only EOL nova
16:31:11 <sean-k-mooney> EOLign should not break others as we are just closing the branch and tagging it
16:31:13 <bauzas> elodilles: fancy revisiting your comment ?
16:31:22 <sean-k-mooney> and libs like os-vif are installed form pypi by default
16:32:17 <sean-k-mooney> at most we would need a overdied-checkout for os-vif to poitn to the eol tag unless i missed something
16:32:19 <elodilles> bauzas: i still think it's better to EOL all the other nova projects in the same go
16:32:32 <sean-k-mooney> ya same ^
16:32:59 <bauzas> gibi: agree then ?
16:33:02 <gibi> elodilles: sure, but then somebody need to handle the fallout
16:33:24 <elodilles> gibi: by fallout you mean?
16:33:46 <gibi> other project's job is failing to check out the deleted branch
16:34:05 <gibi> as an exmple
16:34:06 <gibi> example
16:34:34 <elodilles> yeah, that probably needs to be fixed (and honestly, more like EOL'ing everything else
16:34:43 <bauzas> I guess we need to work on this closely
16:35:15 <bauzas> AFAIK I tried to look at the jobs to find the ones that use stable/train branch of nova
16:35:29 <elodilles> btw, was there any result of the PTG forum session about the EM story?
16:35:33 <bauzas> and unfortunately, I have no crazy idea on how to get a list of such jobs
16:35:50 <elodilles> i haven't seen news on ML about that
16:36:12 <bauzas> since the job definitions are in the project repos, I would have to discover every single git repo in order to ensure this isn't the case :(
16:36:29 <gibi> elodilles: the result was to follon on the ML
16:36:31 <bauzas> elodilles: about that, the conclusion is mostly that the TC will discuss this
16:36:42 <bauzas> and yeah, follow on the ML
16:37:07 <elodilles> bauzas: yes, looking for those jobs is a bit tricky, the "easiest" maybe to compare with ussuri's .zuul.yaml and what jobs were removed there
16:37:21 <bauzas> tl;dr: no agreement was there to stop the EM experience, but a clarification was needed.
16:37:46 <bauzas> elodilles: you mean stein ?
16:38:07 <elodilles> bauzas: OK, so we either rush forward with the train EOL patch or wait then until tc discussion happens?
16:38:10 * bauzas is lost in translation
16:38:26 <bauzas> (and again, the room temperature doesn't help)
16:38:36 <sean-k-mooney> the TC discussion is about EM as a whole and would affect everythin before ZED
16:38:57 <bauzas> elodilles: not sure any conclusion would hit our EOL of train
16:38:59 <sean-k-mooney> i think we can EOL train regardless of what happens for the newer EM branchs
16:39:04 <bauzas> righjt
16:39:07 <dansmith> for sure
16:39:09 <elodilles> bauzas: not stein, ussuri. the job, that is removed in ussuri probably was last used in train, thus its definition can be deleted (if it is outside of nova repo)
16:39:19 <bauzas> and IIRC, EM will continue to exist in some form
16:40:09 <bauzas> maybe the M of EM carries too much context, but the idea is that we'll continue to have other branches
16:40:14 <gibi> I agree to drop train regardless of the TC discussion
16:40:23 <bauzas> those branches could be somewhere else, but that's an alternative to discuss yet
16:40:44 <bauzas> elodilles: again, lost in translation but I'll ping you tomorrow about it
16:40:58 <elodilles> OK, anyway, i think the 'fallout' will happen regardless we EOL only nova. EM / EOL'ing would be confusing if even within a project would be mixed
16:41:23 <elodilles> bauzas: sure, we can check it outside of the meeting :)
16:41:47 <bauzas> on a morning, with a fresh room temperature, a rested brain and a coffee, please.
16:42:15 <elodilles> bauzas: ++ :)
16:42:17 <bauzas> anyway, doesn't sound there is a change in direction
16:42:34 <bauzas> I guess we're done on that topic anyway
16:42:41 <elodilles> ++
16:42:46 <bauzas> #topic Open discussion
16:42:49 <bauzas> I have one thing
16:42:55 <bauzas> (bauzas) Specless blueprint approval request for https://blueprints.launchpad.net/nova/+spec/num-instances-weigher
16:43:33 <bauzas> tl;dr: some operators said during the PTG that they were having some problems for spreading between heteregenous hardware
16:43:48 <bauzas> so we found we missed a weigher
16:44:23 <bauzas> we already have a filter for making sure a host can't have more than a max number of instances
16:44:32 <gibi> +1 to have this
16:44:38 <gibi> it is a quick and easy win
16:44:38 <bauzas> but it doesn't spread the hosts
16:45:11 <dansmith> yup
16:45:20 <bauzas> there is a current discussion on the already-provided code change https://review.opendev.org/c/openstack/nova/+/886232
16:46:11 <bauzas> the concern is that we have a negative default value for spreading
16:46:15 <sean-k-mooney> whihc is not realted to if we shoudl have the weigher. im ok wigher jsut dislike that we have to default to -1 to spread when positive multipes are normally spread since we spread by default
16:46:48 <sean-k-mooney> yep so after some other discussion i woudl just prefer you retrined the number of instance * -1.0
16:46:54 <sean-k-mooney> and default to 1.0
16:47:11 <bauzas> sean-k-mooney: yeah and I understand your concern but my point is that I don't want to have a max-instances value that would be the same for all hosts
16:47:14 <sean-k-mooney> that is a prefernce not a blocker
16:47:34 <sean-k-mooney> yep which is why i provided two suggestion orginally
16:47:52 <gibi> hm, how the -1 relates to the max-instance?
16:48:05 <sean-k-mooney> it does not
16:48:11 <sean-k-mooney> i provided 2 suggestsions
16:48:22 <sean-k-mooney> return max_instace- current
16:48:32 <bauzas> sean-k-mooney: okay, so you would be okay if _weigh_object() would just do -1 * host_state.num_instances ?
16:48:32 <sean-k-mooney> or return current * -1.0
16:48:42 <sean-k-mooney> yep
16:48:50 <sean-k-mooney> tottaly fine with that
16:48:52 <gibi> The -1.0 one works for me. I would not complicate it with the max-instance
16:49:23 <bauzas> sean-k-mooney: as I said in https://review.opendev.org/c/openstack/nova/+/886232/comment/565abda0_70ab7fde/ we already have negative values for spreading
16:49:29 <bauzas> https://docs.openstack.org/nova/latest/configuration/config.html#filter_scheduler.io_ops_weight_multiplier
16:49:43 <sean-k-mooney> yep i pointed that out to you in the review
16:50:02 <sean-k-mooney> and noted that its unforturneate because its teh opisctie behavior of the cpu ram and disk weighers
16:50:17 <sean-k-mooney> and i didnt want to proliferate that by adding more useage
16:50:33 <sean-k-mooney> but as i also said its just a prefrence if other dont mind then fine
16:50:37 <bauzas> okay, if really you want it to be positive, then I'll just multiply by -1.0
16:51:01 <bauzas> but I'm not sure it's really a problem to have a negative value for spreadinh
16:51:23 <bauzas> anyway, looks to me an implementation nit
16:51:24 <sean-k-mooney> it just makes it harder to reason about
16:51:35 <sean-k-mooney> if it would not break peopel i would change the io_ops_weigher
16:51:39 <bauzas> do people agree on having https://blueprints.launchpad.net/nova/+spec/num-instances-weigher to be specless ?
16:51:40 <sean-k-mooney> but its really not worth it
16:51:41 <dansmith> I wasn't aware we had any sort of weigher convention where one value means pack or spread
16:51:47 <dansmith> bauzas: yes, specless
16:51:55 <sean-k-mooney> ya specless is good with me
16:52:00 <bauzas> cool, then we'll revisit this conversation in gerrit
16:52:09 <sean-k-mooney> ack
16:52:09 <dansmith> I've heard you say that, but I've never heard that before
16:52:10 <bauzas> anyone fancy to review the code itself
16:52:18 <dansmith> I guess I'll have to go look at the existing weighers and see
16:52:23 <bauzas> dansmith: I can't remember it too
16:52:42 <bauzas> in general our weighers have different default values
16:52:45 <gibi> +1 on specless
16:52:59 <bauzas> like https://docs.openstack.org/nova/latest/configuration/config.html#filter_scheduler.build_failure_weight_multiplier
16:53:03 <dansmith> bauzas: yeah I thought they were all pretty much individually contextual
16:53:09 <bauzas> which defaults to 1000000.0
16:53:10 <sean-k-mooney> in generall they all spread by default and positive measn spread. those are the two ruels of tumb i learend early on
16:53:21 <bauzas> anyway
16:53:28 <sean-k-mooney> if we broke that at some poin thats unfortunet btu its what we are stuck with
16:53:39 <bauzas> #agreed https://blueprints.launchpad.net/nova/+spec/num-instances-weigher is approved as a specless bp
16:53:47 <gibi> \o/
16:53:55 <bauzas> I guess we can continue this in gerrit :!)
16:53:57 <bauzas> :)
16:54:34 <bauzas> okay, so I don't see any other item
16:54:42 <bauzas> anyone wanting to discuss on something ?
16:55:09 * gibi sends the nice cool weather towards France
16:55:41 <bauzas> thanks :)
16:55:50 <Uggla_> :)
16:56:01 <bauzas> I guess I'll need to stop sooner than later
16:56:06 <bauzas> anyway
16:56:09 <bauzas> thanks all
16:56:11 <bauzas> #endmeeting