16:01:09 #startmeeting nova 16:01:10 Meeting started Thu May 6 16:01:09 2021 UTC and is due to finish in 60 minutes. The chair is gibi. Information about MeetBot at http://wiki.debian.org/MeetBot. 16:01:11 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 16:01:14 The meeting name has been set to 'nova' 16:01:18 \o 16:01:21 o/ 16:01:32 o/ 16:02:35 ~o~ 16:02:39 #topic Bugs (stuck/critical) 16:02:55 No critical bugs 16:03:04 #link 12 new untriaged bugs (-15 since the last meeting): #link https://bugs.launchpad.net/nova/+bugs?search=Search&field.status=New 16:03:37 is there any bug we need to discuss? 16:04:38 #topic Gate status 16:04:44 Placement periodic job status #link https://zuul.opendev.org/t/openstack/builds?project=placement&pipeline=periodic 16:04:52 there should be jobs there but I don't see them 16:05:19 sean-k-mooney: did you have an idea what is wrong? 16:06:32 no but i can look into it 16:06:34 * artom doesn't see a periodic pipeline... 16:06:44 thanks 16:07:05 I can help here if you want sean-k-mooney 16:07:19 yes, only jobs in check pipeline are visible ( https://zuul.opendev.org/t/openstack/builds?project=openstack%2Fplacement ) 16:07:41 out of curiousity, where are we going to pushing the results ? 16:07:49 publish* 16:07:53 https://zuul.openstack.org/builds?project=openstack%2Fplacement&pipeline=periodic-weekly 16:07:56 they are tere 16:08:07 and they are green 16:08:07 sean-k-mooney: cool thanks 16:08:12 and they are green \o/ 16:08:23 Oh, the search isn't substring, it's exact match 16:08:29 the periodic pipline is nightly 16:08:33 i used the weekly one 16:08:43 sean-k-mooney: thanks for the jobs I will update the agenda with the proper link 16:09:01 so, we're going to check them by looking at zuul directly ? ok 16:09:03 bauzas: what do you mean by publishing the results? 16:09:27 gibi: I mean, periodic jobs are there for publishing outputs 16:09:28 bauzas: ya we might be able to add a grapha dashbord or somehting 16:09:49 since they're not change-related, a webpage or something else 16:09:59 bauzas: no in this case they are to ensure that updates to upper constratis or new lib release dont break placement 16:10:00 but looking at zuul sounds ok 16:10:07 sean-k-mooney: I know the job 16:10:18 sean-k-mooney: I was wondering what to do with the result :) 16:10:27 ah 16:10:49 sean-k-mooney: we could trigger something, ideally a bug report 16:10:50 well ya we are just making sure they pass and if they dont then we should check it after the meeting 16:11:19 Make it a recurring item on the meeting? 16:11:21 but yeah, scrubbing zuul seems good at first step 16:11:28 provided we keep the meetings :) 16:11:43 anyway i think we can move on 16:11:46 If there are no more meetings we're either all dead or working at Facebook 16:12:03 yepp this will be a recurring item on the meeting to look at it 16:12:11 artom: I don't know what I'd prefer the least 16:12:14 and if it is not green the it triggers me to file a bug 16:12:21 bauzas, they're basically the same thing 16:12:49 moving on ? 16:12:50 any other gate issue we need to discuss? 16:13:27 #topic Release Planning 16:13:31 Milestone 1 is 27th of May. Nothing special is expected at that time. 16:13:37 As in previous cycles we plan to have spec freeze at M2 only. 16:13:41 See the nova specific schedule on #link https://wiki.openstack.org/wiki/Nova/Xena_Release_Schedule 16:13:59 any schedule questions? or release related topic? 16:14:31 eek, time flies 16:14:37 does the release team still want use to do lib release per milestone 16:14:50 I think so 16:14:53 technially we dont have too but that was there perference 16:14:58 lemme look at the docs 16:15:04 ok so we should get os-vif ectra read for m1 16:15:38 that fine we can do it next week or the week after 16:15:41 a lib release is basically a free thing, I guess some script will propose a patch we need to approve 16:15:58 sean-k-mooney: so you don't have to manually propose a release 16:16:17 right i just need to see if there is anything that should merge in it 16:16:23 yepp 16:16:46 anything else about the release? 16:17:20 #topic Stable Branches 16:17:28 stable gate should be OK 16:17:33 train-em tagging will happen next week, so train final release (+ for the rest of the maintained stable branches) prepared: 16:17:38 train 20.6.1, ussuri 21.2.1, victoria 22.2.1, wallaby 23.0.1 -- https://review.opendev.org/q/project:openstack/releases+intopic:nova+status:open 16:17:42 EOM(from elod) 16:17:58 I need to do some wallaby scrubbing 16:18:16 I guess if somebody needs a train fix then this is the ultimate last time to release it 16:18:22 I guess we're planning a stable/wallaby release soon ? 16:18:34 bauzas: it is proposed 16:18:42 bauzas: I've proposed the patch, 16:18:44 haven't seen it 16:18:52 https://review.opendev.org/q/project:openstack/releases+intopic:nova+status:open 16:18:57 but please -1 it if you want something there 16:19:02 to wait 16:19:17 "if somebody needs a train fix" *cries in OSP 16* 16:19:49 we have a couple of gibi's backports https://review.opendev.org/q/project:openstack/nova+branch:stable/wallaby+is:open 16:20:08 artom: don't sell your drug here, there are kids 16:20:14 that is a big, but clean, backport 16:20:25 I'm OK if it needs more time to settle 16:20:31 gibi: your choice 16:20:51 bauzas: it is more lyarwood's and elod's choice ;) 16:21:00 I'm subtle 16:21:00 i guess we can release train even if we keep wallaby release open 16:21:09 ya 16:21:28 ok 16:21:57 if nothing else then moving on 16:22:16 #topic Sub/related team Highlights 16:22:20 Libvirt (bauzas) 16:22:48 nothing important to mention, highly focusing on closing some nasty bug on vGPUs that could help mnaser 16:23:02 (mdevs are stupidely not persistent) 16:23:20 apart from it, I haven't seen other libvirt related asks 16:23:44 and we're early in the cycle so we have time for thinking about the versions we'd like to support 16:24:00 that's it. 16:24:17 have we started to do the verion bump yet 16:24:29 it might be nice to do that before m1 16:25:02 ah soory so no we have not 16:25:47 nope, haven't seen it 16:25:56 good point. We need a volunteer to look at the possible versions we can bump to 16:26:06 I can take it 16:26:10 thanks 16:26:17 well that is in the code already as next_* 16:26:27 but we need to know what that should be updted too 16:26:28 true, then finding a new next 16:26:33 yep 16:26:45 anything else? 16:26:51 we also have the option of not bumping it if there is not meaningful cleanup to be had 16:27:05 well 16:27:15 I guess it's a valuable ask 16:28:10 yepp 16:28:11 moving on 16:28:12 ezeket gondolom nem 16:28:15 sorry 16:28:20 #topic Open discussion 16:28:26 I mean, you *have* to translate that now 16:28:30 (artom) Normalize (via workaround config option?) the Rabbit queue name to lowercase to avoid things like https://bugzilla.redhat.com/1942079 16:28:31 bugzilla.redhat.com bug 1942079 in openstack-nova "Instances are stuck in scheduling/building state when scheduled on specific compute host" [High,New] - Assigned to nova-maint 16:28:48 So the bug itself is mostly useless at this point, filled with noise 16:28:52 artom: these I don't think ... (it was really a fragment only) 16:29:52 But the problem is that Rabbit exchange names can be either lower case or upper case 16:30:08 And we're currently not forcing them one way or another, and relying on things like hostname 16:30:18 well rather they are case sensivite 16:30:28 So the proposal is to normalize them to lower case. 16:31:00 Gate that behind a workaround config option, because of the upgrade impact 16:31:11 And the question is: full spec or specless blueprint? 16:31:43 there is an upgrade impact, right? 16:31:52 yes 16:31:59 Yeah, it would have to be opt-in 16:32:03 then you have your answer... 16:32:23 artom: yepp a small spec about the upgrade impact would be nice 16:32:24 tl;dr if you service is UPPERCASE or mixed today so will the queue name be 16:32:57 that's my point 16:33:10 and controllers and computes should expect the same queue name, right? 16:33:12 so there is a rolling upgrade impact as all nodes have to agree 16:33:18 I guess we need coordination between rpc client and server side to change the name 16:33:18 bauzas: yep 16:33:35 gibi, ack, sounds like bauzas's on your side, spec it is then 16:33:37 exactly, we need to paper it 16:33:52 artom: dont need to be big one, and you can ping me to read it 16:33:58 artom: sorry, I did shoot you 16:34:05 Maybe in it we can figure out a way to migrate deployments to the new thing 16:34:17 but I think this isn't trivial if we do care of our customers upgrading :D 16:34:27 ... if they currently have uppercase exchange names. But that seems hard/impossible. 16:34:44 gibi, ack, appreciated :) 16:34:49 bauzas, you... shoot me? 16:34:55 one way to consider it would be "sorry, dude, you made a big mistake, hence UNSUPPORTED. Bye." 16:35:16 but nah, we're gentle 16:35:24 artom: because you were full of hope 16:35:56 There are easier ways to drain me of all hope 16:36:04 Much, much easier :( 16:36:07 I thin kwe can move on :) 16:36:29 artom: well the upgrade plan would be add config option, add nova status check, then flip default and then always do it right 16:36:45 we just need to do it over a few releaes which we can detail in the spec 16:37:13 sean-k-mooney, spec review time :) 16:37:13 something like that yes ^^ 16:37:17 (Well, once I post it) 16:37:17 :) 16:37:17 althoug i was hoping we could treat this as a bug fix 16:37:25 sean-k-mooney: some programmibility maybe with service checks too 16:37:28 with it off by defualt 16:38:05 the other side of this coin is our db schem 16:38:13 which is currently case insenitve by defualt 16:38:29 even though in python we are case sensitive 16:38:30 Doesn't that depend on the DBMS in use? 16:38:30 right, and we store the message queues 16:38:37 in the API DB 16:38:45 artom: it depens on the coalation type 16:38:46 Though I guess... we only support MariaDB and SQLite for CI at this point? 16:38:53 which we do not specify 16:39:09 sounds a problem to me 16:39:09 Right, so whatever's the default 16:39:22 Yeah, OTOH have we had bugs around that? 16:39:24 because we would need to update the API DB 16:39:24 whcih is utf8_general_ci 16:39:34 ci being case insensitive 16:39:55 we can't rely on the collation the operator decidd 16:39:58 decided* 16:40:08 well we do today 16:40:16 hmmm 16:40:19 that is how this issue came about 16:40:26 what does the schema sensitivity have to do with it really? 16:40:32 but collation only about ordering, we still store the data without normalizing it 16:40:41 that's just for matching, right? it still stores the thing you gave it, I think 16:40:50 yes 16:41:06 if the name is HOST and we query by host it will match the same reccord 16:41:07 Yeah, it affects queries like where foo like bar, or where foo = bar 16:41:09 so why can't we just start warning people who have non-lowercase exchanges, and then in a future release start normalizing them on input? 16:41:10 So FOO would match foo 16:41:24 dansmith: that's one option 16:41:34 with a nova status check 16:41:38 like sean-k-mooney mentioned 16:41:47 yeah, that and maybe also at runtime 16:41:48 But then how can they realistically move? 16:42:02 that's what I was wondering 16:42:03 Drain the compute node, remove it, re-add with the lowercase exchange? 16:42:13 artom: the concern is people with mixed case records in their cell mappings right? 16:42:15 What if the UPPERCASE exchange isn't node-specific 16:42:20 yep 16:42:33 dansmith: they had some instace with uppercase name and some with lowercase 16:42:36 artom: you hit the issue because we can't find the transport url, right? 16:42:40 depening on when the vm was booted on the host 16:42:47 yeah, so we can easily make nova-manage update them with the lowercase variant without deleting the record right? 16:42:55 yep 16:42:59 dansmith: I was considering it 16:43:11 it's just the cell mapping that is the issue, not something that scales with instances 16:43:25 so maybe we need to detail this in the spec and esiced if its a feature or bug later 16:43:31 we need to lowercase for new computes but we somehow need a way to correct the transport url 16:43:36 oh, I was thinking they would have to change the exchange names in rabbit if they had used mixed case 16:43:46 melwitt: they would 16:43:53 is rabbit case-sensitive ? 16:43:58 bauzas: yes 16:44:00 sean-k-mooney said it was, and I imagine it is 16:44:16 dansmith said that sean-k-mooney said it was, and I imagine it is 16:44:20 okok, so yeah we need both 16:44:38 bauzas: our custoemr had 2 queues the one the conduictor set the message too and the one the comptue agent was litenting too 16:44:40 OK, sounds like there's meat for a spec here 16:44:47 presumably we could also just make that one table be case-sensitive right? 16:44:48 but I guess that existing computes will have to register to another queue 16:44:58 cell_mappings is tiny, even on giant clouds 16:45:06 dansmith: maybe 16:45:15 hmmm 16:45:24 in this case i think its the service table we care about 16:45:26 we could just make it case sensitive too, if we want to avoid people who make poor naming decisions not have to retool their exchanges 16:45:33 sean-k-mooney: eh? 16:45:52 we might be able to do it on the column 16:45:54 sean-k-mooney: there's no transport url in service right? 16:45:55 dansmith: that's a reasonable thing, because lowercasing would mean changing their queues they monitor 16:46:01 that's an operating breaking change 16:46:09 dansmith: i think its the queue not the exchange 16:46:14 bauzas: yeah people with mixed case now would have to quiesce their whole cloud (or a whole cell) to make the change 16:46:15 that was the issue 16:46:30 not ideal, but I dunno.. not that big of a deal I'd think 16:46:53 still, I don't see how this impacts anything other than cell_mappings 16:47:05 correct me wrong, but the queue names are written by us, not opt-driven 16:47:16 if I* 16:47:21 bauzas: they are created using the valud of conf.host 16:47:30 ah 16:47:32 as part of the topic/name 16:47:37 gotcha 16:47:38 oh, I see 16:47:44 that's where service comes in 16:47:47 the condocutor gets that form the servcice record host value 16:47:48 so people writing hostnames with uppercase are stepping into this 16:47:53 the comptue agent gets it form the conf 16:48:10 ok, I see now the bug 16:48:18 sean-k-mooney: and why does the case insensitivity of service matter? 16:48:22 just if they change their conf right? 16:48:29 yep 16:48:40 this is very meh, to me 16:48:47 they did as part of an FFU... to make vlaidation work... 16:49:14 so we provide them a way to fix it if they want, but we don't need to make the runtime system overly concerned with this kind of thing right? 16:49:35 yes we are doing that 16:49:46 we should optimize for systems deployed with config management, and if they are and need to change, config management can automate running a tool for them too 16:49:47 we were just wondering if we could harden this so it cant happen 16:49:53 by makeing it always lowercase 16:49:54 I think the catch is that said system cannot change the conf.host setting 16:50:04 Because that's written in a bunch of places, and is really hard to update 16:50:19 artom: I don't think I understand that 16:50:25 lets talk about this on the nova channel after the meeting 16:50:36 unless this is the last item 16:50:49 I have one small thing still 16:51:13 (gibi) Follow up on the review priority patch https://review.opendev.org/c/openstack/project-config/+/787523 16:51:34 so last week sean-k-mooney brought it up, but we still missing feedback on the review prioprity proposal 16:51:52 so it is a reminder to look at it and leave some feedback 16:51:55 EOM 16:52:03 i rebased it today with the 0..+1 range 16:52:49 So is the question just whether it's going to be 0..+1 or 0..+2? 16:53:01 IOW, do we need the +1/+2 granularity? 16:53:04 nope, it is question about the process I drafted in the comment 16:53:20 https://review.opendev.org/c/openstack/project-config/+/787523/1#message-a8433d63f865d2fe4f20284e92a2e91772aa4ae6 16:53:43 im generally in favor of what you proposed 16:53:49 +0/+1 sounds good to me 16:53:55 you were goign to codify that in the contibutor gide yes 16:54:02 sean-k-mooney: yes 16:54:31 just one thing we haven't discussed AFAICR 16:54:31 i guess here https://github.com/openstack/nova/blob/master/doc/source/contributor/code-review.rst 16:54:48 when and how the cores are going to decide which changes to flag ? 16:55:21 that part of the process gibi was suggesting 16:55:24 can people propose a patch for a priority ? 16:55:30 like, flagging with +1 ? 16:55:38 bauzas: good point. I think it should be similar to how we decided if something goes into a runway slot 16:55:39 and then, cores accepting it by +2 it ? 16:55:47 bauzas, I like that 16:55:52 so, it would be a +0/+1/+2 16:55:57 yes 16:55:58 bauzas: right now the proposal is to only have 0/+1 and onyl core can set it to +1 16:56:02 i can split that by group 16:56:08 I know 16:56:19 but I was considering to open the proposal 16:56:23 hence the +1 16:56:30 like any other change 16:56:43 considering we don't -1 16:56:47 i guess the workflow as written woudl be bring it up in the meeting or irc 16:56:55 if we don't accept a proposal, we just reset it to +0 16:57:11 do we make that a place to ping-poing? 16:57:22 sean-k-mooney: I don't like meetings 16:57:28 hehe 16:57:37 sean-k-mooney: because a certain percentage of our contribs couldn't attend it 16:57:40 can i suggest moving it to the review 16:57:50 yeah, we're over time soon 16:57:52 ill update it to whatever the concensou is 16:57:54 sean-k-mooney: ++ 16:58:04 Yeah... one potential problem is people just spamming +1 16:58:06 lets continute discussing the proposal in the review 16:58:12 At which point +1 is the new 0, and becomes meaningless 16:58:12 I opened the review tab, I'll decently throw some notes 16:58:40 artom: yepp that was my thinking too 16:58:56 anyhow. Is there any other topic for today/ 16:58:57 ? 16:59:47 if not then thank you for joining 16:59:58 o/ 17:00:10 #endmeeting