16:01:09 <gibi> #startmeeting nova
16:01:10 <openstack> Meeting started Thu May  6 16:01:09 2021 UTC and is due to finish in 60 minutes.  The chair is gibi. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:01:11 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
16:01:14 <openstack> The meeting name has been set to 'nova'
16:01:18 <bauzas> \o
16:01:21 <elod> o/
16:01:32 <gibi> o/
16:02:35 <artom> ~o~
16:02:39 <gibi> #topic Bugs (stuck/critical)
16:02:55 <gibi> No critical bugs
16:03:04 <gibi> #link 12 new untriaged bugs (-15 since the last meeting): #link https://bugs.launchpad.net/nova/+bugs?search=Search&field.status=New
16:03:37 <gibi> is there any bug we need to discuss?
16:04:38 <gibi> #topic Gate status
16:04:44 <gibi> Placement periodic job status #link https://zuul.opendev.org/t/openstack/builds?project=placement&pipeline=periodic
16:04:52 <gibi> there should be jobs there but I don't see them
16:05:19 <gibi> sean-k-mooney: did you have an idea what is wrong?
16:06:32 <sean-k-mooney> no but i can look into it
16:06:34 * artom doesn't see a periodic pipeline...
16:06:44 <gibi> thanks
16:07:05 <bauzas> I can help here if you want sean-k-mooney
16:07:19 <elod> yes, only jobs in check pipeline are visible ( https://zuul.opendev.org/t/openstack/builds?project=openstack%2Fplacement )
16:07:41 <bauzas> out of curiousity, where are we going to pushing the results ?
16:07:49 <bauzas> publish*
16:07:53 <sean-k-mooney> https://zuul.openstack.org/builds?project=openstack%2Fplacement&pipeline=periodic-weekly
16:07:56 <sean-k-mooney> they are tere
16:08:07 <sean-k-mooney> and they are green
16:08:07 <gibi> sean-k-mooney: cool thanks
16:08:12 <gibi> and they are green \o/
16:08:23 <artom> Oh, the search isn't substring, it's exact match
16:08:29 <sean-k-mooney> the periodic pipline is nightly
16:08:33 <sean-k-mooney> i used the weekly one
16:08:43 <gibi> sean-k-mooney: thanks for the jobs I will update the agenda with the proper link
16:09:01 <bauzas> so, we're going to check them by looking at zuul directly ? ok
16:09:03 <gibi> bauzas: what do you mean by publishing the results?
16:09:27 <bauzas> gibi: I mean, periodic jobs are there for publishing outputs
16:09:28 <sean-k-mooney> bauzas: ya we might be able to add a grapha dashbord or somehting
16:09:49 <bauzas> since they're not change-related, a webpage or something else
16:09:59 <sean-k-mooney> bauzas: no in this case they are to ensure that updates to upper constratis or new lib release dont break placement
16:10:00 <bauzas> but looking at zuul sounds ok
16:10:07 <bauzas> sean-k-mooney: I know the job
16:10:18 <bauzas> sean-k-mooney: I was wondering what to do with the result :)
16:10:27 <sean-k-mooney> ah
16:10:49 <bauzas> sean-k-mooney: we could trigger something, ideally a bug report
16:10:50 <sean-k-mooney> well ya we are just making sure they pass and if they dont then we should check it after the meeting
16:11:19 <artom> Make it a recurring item on the meeting?
16:11:21 <bauzas> but yeah, scrubbing zuul seems good at first step
16:11:28 <bauzas> provided we keep the meetings :)
16:11:43 <sean-k-mooney> anyway i think we can move on
16:11:46 <artom> If there are no more meetings we're either all dead or working at Facebook
16:12:03 <gibi> yepp this will be a recurring item on the meeting to look at it
16:12:11 <bauzas> artom: I don't know what I'd prefer the least
16:12:14 <gibi> and if it is not green the it triggers me to file a bug
16:12:21 <artom> bauzas, they're basically the same thing
16:12:49 <bauzas> moving on ?
16:12:50 <gibi> any other gate issue we need to discuss?
16:13:27 <gibi> #topic Release Planning
16:13:31 <gibi> Milestone 1 is 27th of May. Nothing special is expected at that time.
16:13:37 <gibi> As in previous cycles we plan to have spec freeze at M2 only.
16:13:41 <gibi> See the nova specific schedule on #link https://wiki.openstack.org/wiki/Nova/Xena_Release_Schedule
16:13:59 <gibi> any schedule questions? or release related topic?
16:14:31 <bauzas> eek, time flies
16:14:37 <sean-k-mooney> does the release team still want use to do lib release per milestone
16:14:50 <gibi> I think so
16:14:53 <sean-k-mooney> technially we dont have too but that was there perference
16:14:58 <bauzas> lemme look at the docs
16:15:04 <sean-k-mooney> ok so we should get os-vif ectra read for m1
16:15:38 <sean-k-mooney> that fine we can do it next week or the week after
16:15:41 <gibi> a lib release is basically a free thing, I guess some script will propose a patch we need to approve
16:15:58 <gibi> sean-k-mooney: so you don't have to manually propose a release
16:16:17 <sean-k-mooney> right i just need to see if there is anything that should merge in it
16:16:23 <gibi> yepp
16:16:46 <gibi> anything else about the release?
16:17:20 <gibi> #topic Stable Branches
16:17:28 <gibi> stable gate should be OK
16:17:33 <gibi> train-em tagging will happen next week, so train final release (+ for the rest of the maintained stable branches) prepared:
16:17:38 <gibi> train 20.6.1, ussuri 21.2.1, victoria 22.2.1, wallaby 23.0.1 -- https://review.opendev.org/q/project:openstack/releases+intopic:nova+status:open
16:17:42 <gibi> EOM(from elod)
16:17:58 <bauzas> I need to do some wallaby scrubbing
16:18:16 <gibi> I guess if somebody needs a train fix then this is the ultimate last time to release it
16:18:22 <bauzas> I guess we're planning a stable/wallaby release soon ?
16:18:34 <gibi> bauzas: it is proposed
16:18:42 <elod> bauzas: I've proposed the patch,
16:18:44 <bauzas> haven't seen it
16:18:52 <gibi> https://review.opendev.org/q/project:openstack/releases+intopic:nova+status:open
16:18:57 <elod> but please -1 it if you want something there
16:19:02 <elod> to wait
16:19:17 <artom> "if somebody needs a train fix" *cries in OSP 16*
16:19:49 <bauzas> we have a couple of gibi's backports https://review.opendev.org/q/project:openstack/nova+branch:stable/wallaby+is:open
16:20:08 <bauzas> artom: don't sell your drug here, there are kids
16:20:14 <gibi> that is a big, but clean, backport
16:20:25 <gibi> I'm OK if it needs more time to settle
16:20:31 <bauzas> gibi: your choice
16:20:51 <gibi> bauzas: it is more lyarwood's and elod's choice ;)
16:21:00 <bauzas> I'm subtle
16:21:00 <elod> i guess we can release train even if we keep wallaby release open
16:21:09 <bauzas> ya
16:21:28 <elod> ok
16:21:57 <gibi> if nothing else then moving on
16:22:16 <gibi> #topic Sub/related team Highlights
16:22:20 <gibi> Libvirt (bauzas)
16:22:48 <bauzas> nothing important to mention, highly focusing on closing some nasty bug on vGPUs that could help mnaser
16:23:02 <bauzas> (mdevs are stupidely not persistent)
16:23:20 <bauzas> apart from it, I haven't seen other libvirt related asks
16:23:44 <bauzas> and we're early in the cycle so we have time for thinking about the versions we'd like to support
16:24:00 <bauzas> that's it.
16:24:17 <sean-k-mooney> have we started to do the verion bump yet
16:24:29 <sean-k-mooney> it might be nice to do that before m1
16:25:02 <sean-k-mooney> ah soory so no we have not
16:25:47 <bauzas> nope, haven't seen it
16:25:56 <gibi> good point. We need a volunteer to look at the possible versions we can bump to
16:26:06 <bauzas> I can take it
16:26:10 <gibi> thanks
16:26:17 <sean-k-mooney> well that is in the code already as next_*
16:26:27 <sean-k-mooney> but we need to know what that should be updted too
16:26:28 <gibi> true, then finding a new next
16:26:33 <sean-k-mooney> yep
16:26:45 <gibi> anything else?
16:26:51 <sean-k-mooney> we also have the option of not bumping it if there is not meaningful cleanup to be had
16:27:05 <bauzas> well
16:27:15 <bauzas> I guess it's a valuable ask
16:28:10 <gibi> yepp
16:28:11 <gibi> moving on
16:28:12 <gibi> ezeket gondolom nem
16:28:15 <gibi> sorry
16:28:20 <gibi> #topic Open discussion
16:28:26 <artom> I mean, you *have* to translate that now
16:28:30 <gibi> (artom) Normalize (via workaround config option?) the Rabbit queue name to lowercase to avoid things like https://bugzilla.redhat.com/1942079
16:28:31 <openstack> bugzilla.redhat.com bug 1942079 in openstack-nova "Instances are stuck in scheduling/building state when scheduled on specific compute host" [High,New] - Assigned to nova-maint
16:28:48 <artom> So the bug itself is mostly useless at this point, filled with noise
16:28:52 <gibi> artom: these I don't think ... (it was really a fragment only)
16:29:52 <artom> But the problem is that Rabbit exchange names can be either lower case or upper case
16:30:08 <artom> And we're currently not forcing them one way or another, and relying on things like hostname
16:30:18 <sean-k-mooney> well rather they are case sensivite
16:30:28 <artom> So the proposal is to normalize them to lower case.
16:31:00 <artom> Gate that behind a workaround config option, because of the upgrade impact
16:31:11 <artom> And the question is: full spec or specless blueprint?
16:31:43 <bauzas> there is an upgrade impact, right?
16:31:52 <sean-k-mooney> yes
16:31:59 <artom> Yeah, it would have to be opt-in
16:32:03 <bauzas> then you have your answer...
16:32:23 <gibi> artom: yepp a small spec about the upgrade impact would be nice
16:32:24 <sean-k-mooney> tl;dr if you service is UPPERCASE or mixed today so will the queue name be
16:32:57 <bauzas> that's my point
16:33:10 <bauzas> and controllers and computes should expect the same queue name, right?
16:33:12 <sean-k-mooney> so there is a rolling upgrade impact as all nodes have to agree
16:33:18 <gibi> I guess we need coordination between rpc client and server side to change the name
16:33:18 <sean-k-mooney> bauzas: yep
16:33:35 <artom> gibi, ack, sounds like bauzas's on your side, spec it is then
16:33:37 <bauzas> exactly, we need to paper it
16:33:52 <gibi> artom: dont need to be big one, and you can ping me to read it
16:33:58 <bauzas> artom: sorry, I did shoot you
16:34:05 <artom> Maybe in it we can figure out a way to migrate deployments to the new thing
16:34:17 <bauzas> but I think this isn't trivial if we do care of our customers upgrading :D
16:34:27 <artom> ... if they currently have uppercase exchange names. But that seems hard/impossible.
16:34:44 <artom> gibi, ack, appreciated :)
16:34:49 <artom> bauzas, you... shoot me?
16:34:55 <bauzas> one way to consider it would be "sorry, dude, you made a big mistake, hence UNSUPPORTED. Bye."
16:35:16 <bauzas> but nah, we're gentle
16:35:24 <bauzas> artom: because you were full of hope
16:35:56 <artom> There are easier ways to drain me of all hope
16:36:04 <artom> Much, much easier :(
16:36:07 <gibi> I thin kwe can move on :)
16:36:29 <sean-k-mooney> artom: well the upgrade plan would be  add config option, add nova status check, then flip default and then always do it right
16:36:45 <sean-k-mooney> we just need to do it over a few releaes which we can detail in the spec
16:37:13 <artom> sean-k-mooney, spec review time :)
16:37:13 <gibi> something like that yes ^^
16:37:17 <artom> (Well, once I post it)
16:37:17 <gibi> :)
16:37:17 <sean-k-mooney> althoug i was hoping we could treat this as a bug fix
16:37:25 <bauzas> sean-k-mooney: some programmibility maybe with service checks too
16:37:28 <sean-k-mooney> with it off by defualt
16:38:05 <sean-k-mooney> the other side of this coin is our db schem
16:38:13 <sean-k-mooney> which is currently case insenitve by defualt
16:38:29 <sean-k-mooney> even though in python we are case sensitive
16:38:30 <artom> Doesn't that depend on the DBMS in use?
16:38:30 <bauzas> right, and we store the message queues
16:38:37 <bauzas> in the  API DB
16:38:45 <sean-k-mooney> artom: it depens on the coalation type
16:38:46 <artom> Though I guess... we only support MariaDB and SQLite for CI at this point?
16:38:53 <sean-k-mooney> which we do not specify
16:39:09 <bauzas> sounds a problem to me
16:39:09 <artom> Right, so whatever's the default
16:39:22 <artom> Yeah, OTOH have we had bugs around that?
16:39:24 <bauzas> because we would need to update the API DB
16:39:24 <sean-k-mooney> whcih is utf8_general_ci
16:39:34 <sean-k-mooney> ci being case insensitive
16:39:55 <bauzas> we can't rely on the collation the operator decidd
16:39:58 <bauzas> decided*
16:40:08 <sean-k-mooney> well we do today
16:40:16 <bauzas> hmmm
16:40:19 <sean-k-mooney> that is how this issue came about
16:40:26 <dansmith> what does the schema sensitivity have to do with it really?
16:40:32 <gibi> but collation only about ordering, we still store the data without normalizing it
16:40:41 <dansmith> that's just for matching, right? it still stores the thing you gave it, I think
16:40:50 <melwitt> yes
16:41:06 <sean-k-mooney> if the name is HOST and we query by host it will match the same reccord
16:41:07 <artom> Yeah, it affects queries like where foo like bar, or where foo = bar
16:41:09 <dansmith> so why can't we just start warning people who have non-lowercase exchanges, and then in a future release start normalizing them on input?
16:41:10 <artom> So FOO would match foo
16:41:24 <bauzas> dansmith: that's one option
16:41:34 <bauzas> with a nova status check
16:41:38 <bauzas> like sean-k-mooney mentioned
16:41:47 <dansmith> yeah, that and maybe also at runtime
16:41:48 <artom> But then how can they realistically move?
16:42:02 <melwitt> that's what I was wondering
16:42:03 <artom> Drain the compute node, remove it, re-add with the lowercase exchange?
16:42:13 <dansmith> artom: the concern is people with mixed case records in their cell mappings right?
16:42:15 <artom> What if the UPPERCASE exchange isn't node-specific
16:42:20 <sean-k-mooney> yep
16:42:33 <sean-k-mooney> dansmith: they had some instace with uppercase name and some with lowercase
16:42:36 <bauzas> artom: you hit the issue because we can't find the transport url, right?
16:42:40 <sean-k-mooney> depening on when the vm was booted on the host
16:42:47 <dansmith> yeah, so we can easily make nova-manage update them with the lowercase variant without deleting the record right?
16:42:55 <sean-k-mooney> yep
16:42:59 <bauzas> dansmith: I was considering it
16:43:11 <dansmith> it's just the cell mapping that is the issue, not something that scales with instances
16:43:25 <sean-k-mooney> so maybe we need to detail this in the spec and esiced if its a feature or bug later
16:43:31 <bauzas> we need to lowercase for new computes but we somehow need a way to correct the transport url
16:43:36 <melwitt> oh, I was thinking they would have to change the exchange names in rabbit if they had used mixed case
16:43:46 <dansmith> melwitt: they would
16:43:53 <bauzas> is rabbit case-sensitive ?
16:43:58 <sean-k-mooney> bauzas: yes
16:44:00 <dansmith> sean-k-mooney said it was, and I imagine it is
16:44:16 <artom> dansmith said that sean-k-mooney said it was, and I imagine it is
16:44:20 <bauzas> okok, so yeah we need both
16:44:38 <sean-k-mooney> bauzas: our custoemr had 2 queues the one the conduictor set the message too and the one the comptue agent was litenting too
16:44:40 <artom> OK, sounds like there's meat for a spec here
16:44:47 <dansmith> presumably we could also just make that one table be case-sensitive right?
16:44:48 <bauzas> but I guess that existing computes will have to register to another queue
16:44:58 <dansmith> cell_mappings is tiny, even on giant clouds
16:45:06 <sean-k-mooney> dansmith: maybe
16:45:15 <bauzas> hmmm
16:45:24 <sean-k-mooney> in this case i think its the service table we care about
16:45:26 <dansmith> we could just make it case sensitive too, if we want to avoid people who make poor naming decisions not have to retool their exchanges
16:45:33 <dansmith> sean-k-mooney: eh?
16:45:52 <sean-k-mooney> we might be able to do it on the column
16:45:54 <dansmith> sean-k-mooney: there's no transport url in service right?
16:45:55 <bauzas> dansmith: that's a reasonable thing, because lowercasing would mean changing their queues they monitor
16:46:01 <bauzas> that's an operating breaking change
16:46:09 <sean-k-mooney> dansmith: i think its the queue not the exchange
16:46:14 <dansmith> bauzas: yeah people with mixed case now would have to quiesce their whole cloud (or a whole cell) to make the change
16:46:15 <sean-k-mooney> that was the issue
16:46:30 <dansmith> not ideal, but I dunno.. not that big of a deal I'd think
16:46:53 <dansmith> still, I don't see how this impacts anything other than cell_mappings
16:47:05 <bauzas> correct me wrong, but the queue names are written by us, not opt-driven
16:47:16 <bauzas> if I*
16:47:21 <sean-k-mooney> bauzas: they are created using the valud of conf.host
16:47:30 <bauzas> ah
16:47:32 <sean-k-mooney> as part of the topic/name
16:47:37 <bauzas> gotcha
16:47:38 <dansmith> oh, I see
16:47:44 <dansmith> that's where service comes in
16:47:47 <sean-k-mooney> the condocutor gets that form the servcice record host value
16:47:48 <bauzas> so people writing hostnames with uppercase are stepping into this
16:47:53 <sean-k-mooney> the comptue agent gets it form the conf
16:48:10 <bauzas> ok, I see now the bug
16:48:18 <dansmith> sean-k-mooney: and why does the case insensitivity of service matter?
16:48:22 <dansmith> just if they change their conf right?
16:48:29 <sean-k-mooney> yep
16:48:40 <dansmith> this is very meh, to me
16:48:47 <sean-k-mooney> they did as part of an FFU... to make vlaidation work...
16:49:14 <dansmith> so we provide them a way to fix it if they want, but we don't need to make the runtime system overly concerned with this kind of thing right?
16:49:35 <sean-k-mooney> yes we are doing that
16:49:46 <dansmith> we should optimize for systems deployed with config management, and if they are and need to change, config management can automate running a tool for them too
16:49:47 <sean-k-mooney> we were just wondering if we could harden this so it cant happen
16:49:53 <sean-k-mooney> by makeing it always lowercase
16:49:54 <artom> I think the catch is that said system cannot change the conf.host setting
16:50:04 <artom> Because that's written in a bunch of places, and is really hard to update
16:50:19 <dansmith> artom: I don't think I understand that
16:50:25 <sean-k-mooney> lets talk about this on the nova channel after the meeting
16:50:36 <sean-k-mooney> unless this is the last item
16:50:49 <gibi> I have one small thing still
16:51:13 <gibi> (gibi) Follow up on the review priority patch https://review.opendev.org/c/openstack/project-config/+/787523
16:51:34 <gibi> so last week sean-k-mooney brought it up, but we still missing feedback on the review prioprity proposal
16:51:52 <gibi> so it is a reminder to look at it and leave some feedback
16:51:55 <gibi> EOM
16:52:03 <sean-k-mooney> i rebased it today with the 0..+1 range
16:52:49 <artom> So is the question just whether it's going to be 0..+1 or 0..+2?
16:53:01 <artom> IOW, do we need the +1/+2 granularity?
16:53:04 <gibi> nope, it is question about the process I drafted in the comment
16:53:20 <gibi> https://review.opendev.org/c/openstack/project-config/+/787523/1#message-a8433d63f865d2fe4f20284e92a2e91772aa4ae6
16:53:43 <sean-k-mooney> im generally in favor of what you proposed
16:53:49 <bauzas> +0/+1 sounds good to me
16:53:55 <sean-k-mooney> you were goign to codify that in the contibutor gide yes
16:54:02 <gibi> sean-k-mooney: yes
16:54:31 <bauzas> just one thing we haven't discussed AFAICR
16:54:31 <sean-k-mooney> i guess here https://github.com/openstack/nova/blob/master/doc/source/contributor/code-review.rst
16:54:48 <bauzas> when and how the cores are going to decide which changes to flag ?
16:55:21 <sean-k-mooney> that part of the process gibi was suggesting
16:55:24 <bauzas> can people propose a patch for a priority ?
16:55:30 <bauzas> like, flagging with +1 ?
16:55:38 <gibi> bauzas: good point. I think it should be similar to how we decided if something goes into a runway slot
16:55:39 <bauzas> and then, cores accepting it by +2 it ?
16:55:47 <artom> bauzas, I like that
16:55:52 <bauzas> so, it would be a +0/+1/+2
16:55:57 <sean-k-mooney> yes
16:55:58 <gibi> bauzas: right now the proposal is to only have 0/+1 and onyl core can set it to +1
16:56:02 <sean-k-mooney> i can split that by group
16:56:08 <bauzas> I know
16:56:19 <bauzas> but I was considering to open the proposal
16:56:23 <bauzas> hence the +1
16:56:30 <bauzas> like any other change
16:56:43 <bauzas> considering we don't -1
16:56:47 <sean-k-mooney> i guess the workflow as written woudl be bring it up in the meeting or irc
16:56:55 <bauzas> if we don't accept a proposal, we just reset it to +0
16:57:11 <gibi> do we make that a place to ping-poing?
16:57:22 <bauzas> sean-k-mooney: I don't like meetings
16:57:28 <sean-k-mooney> hehe
16:57:37 <bauzas> sean-k-mooney: because a certain percentage of our contribs couldn't attend it
16:57:40 <sean-k-mooney> can i suggest moving it to the review
16:57:50 <bauzas> yeah, we're over time soon
16:57:52 <sean-k-mooney> ill update it to whatever the concensou is
16:57:54 <gibi> sean-k-mooney: ++
16:58:04 <artom> Yeah... one potential problem is people just spamming +1
16:58:06 <gibi> lets continute discussing the proposal in the review
16:58:12 <artom> At which point +1 is the new 0, and becomes meaningless
16:58:12 <bauzas> I opened the review tab, I'll decently throw some notes
16:58:40 <gibi> artom: yepp that was my thinking too
16:58:56 <gibi> anyhow. Is there any other topic for today/
16:58:57 <gibi> ?
16:59:47 <gibi> if not then thank you for joining
16:59:58 <gibi> o/
17:00:10 <gibi> #endmeeting