Thursday, 2021-05-06

*** jamesmcarthur has quit IRC00:20
*** jamesmcarthur has joined #openstack-meeting-300:33
*** hemanth_n has joined #openstack-meeting-301:34
*** imtiazc has quit IRC01:52
*** purplerbot has quit IRC01:52
*** otherwiseguy has quit IRC01:52
*** andreaf has quit IRC01:52
*** purplerbot has joined #openstack-meeting-301:53
*** otherwiseguy has joined #openstack-meeting-301:53
*** andreaf has joined #openstack-meeting-301:53
*** markmcclain has quit IRC01:55
*** markmcclain has joined #openstack-meeting-301:57
*** macz_ has joined #openstack-meeting-302:43
*** macz_ has quit IRC02:47
*** psachin has joined #openstack-meeting-303:37
*** jamesmcarthur has quit IRC03:45
*** jamesmcarthur has joined #openstack-meeting-303:46
*** jamesmcarthur has quit IRC03:51
*** jamesmcarthur has joined #openstack-meeting-304:16
*** jamesmcarthur has quit IRC04:49
*** jamesmcarthur has joined #openstack-meeting-305:03
*** jamesmcarthur has quit IRC05:09
*** jamesmcarthur has joined #openstack-meeting-305:21
*** jamesmcarthur has quit IRC05:28
*** jamesmcarthur has joined #openstack-meeting-305:43
*** slaweq has joined #openstack-meeting-306:15
*** ralonsoh has joined #openstack-meeting-306:31
*** _mlavalle_1 has joined #openstack-meeting-307:12
*** belmoreira has joined #openstack-meeting-307:15
*** mlavalle has quit IRC07:15
*** jamesmcarthur has quit IRC07:43
*** tosky has joined #openstack-meeting-307:47
*** jamesmcarthur has joined #openstack-meeting-307:59
*** hemanth_n has quit IRC08:10
*** e0ne has joined #openstack-meeting-308:14
*** e0ne has quit IRC08:14
*** e0ne has joined #openstack-meeting-308:16
*** e0ne has quit IRC08:47
*** rubasov has quit IRC09:01
*** jamesmcarthur has quit IRC09:02
*** psahoo has joined #openstack-meeting-309:51
*** rubasov has joined #openstack-meeting-310:00
*** jamesmcarthur has joined #openstack-meeting-310:04
*** rubasov has quit IRC10:30
*** rubasov has joined #openstack-meeting-311:28
*** artom has quit IRC11:48
*** jamesmcarthur has quit IRC12:06
*** jamesmcarthur has joined #openstack-meeting-312:06
*** dustinc has joined #openstack-meeting-313:07
*** njohnston has joined #openstack-meeting-313:16
*** artom has joined #openstack-meeting-314:25
*** psachin has quit IRC15:27
*** macz_ has joined #openstack-meeting-315:35
*** macz_ has quit IRC15:35
*** macz_ has joined #openstack-meeting-315:35
bauzasding dong ?16:00
gibi#startmeeting nova16:01
openstackMeeting started Thu May  6 16:01:09 2021 UTC and is due to finish in 60 minutes.  The chair is gibi. Information about MeetBot at http://wiki.debian.org/MeetBot.16:01
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.16:01
*** openstack changes topic to " (Meeting topic: nova)"16:01
openstackThe meeting name has been set to 'nova'16:01
bauzas\o16:01
elodo/16:01
gibio/16:01
artom~o~16:02
gibi#topic Bugs (stuck/critical)16:02
*** openstack changes topic to "Bugs (stuck/critical) (Meeting topic: nova)"16:02
gibiNo critical bugs16:02
gibi#link 12 new untriaged bugs (-15 since the last meeting): #link https://bugs.launchpad.net/nova/+bugs?search=Search&field.status=New16:03
gibiis there any bug we need to discuss?16:03
gibi#topic Gate status16:04
*** openstack changes topic to "Gate status (Meeting topic: nova)"16:04
gibiPlacement periodic job status #link https://zuul.opendev.org/t/openstack/builds?project=placement&pipeline=periodic16:04
*** sean-k-mooney has joined #openstack-meeting-316:04
gibithere should be jobs there but I don't see them16:04
*** claw426069 has joined #openstack-meeting-316:05
gibisean-k-mooney: did you have an idea what is wrong?16:05
*** claw426069 is now known as claw426016:05
sean-k-mooneyno but i can look into it16:06
* artom doesn't see a periodic pipeline...16:06
gibithanks16:06
bauzasI can help here if you want sean-k-mooney16:07
elodyes, only jobs in check pipeline are visible ( https://zuul.opendev.org/t/openstack/builds?project=openstack%2Fplacement )16:07
bauzasout of curiousity, where are we going to pushing the results ?16:07
bauzaspublish*16:07
sean-k-mooneyhttps://zuul.openstack.org/builds?project=openstack%2Fplacement&pipeline=periodic-weekly16:07
sean-k-mooneythey are tere16:07
sean-k-mooneyand they are green16:08
gibisean-k-mooney: cool thanks16:08
gibiand they are green \o/16:08
artomOh, the search isn't substring, it's exact match16:08
sean-k-mooneythe periodic pipline is nightly16:08
sean-k-mooneyi used the weekly one16:08
gibisean-k-mooney: thanks for the jobs I will update the agenda with the proper link16:08
bauzasso, we're going to check them by looking at zuul directly ? ok16:09
gibibauzas: what do you mean by publishing the results?16:09
bauzasgibi: I mean, periodic jobs are there for publishing outputs16:09
sean-k-mooneybauzas: ya we might be able to add a grapha dashbord or somehting16:09
bauzassince they're not change-related, a webpage or something else16:09
sean-k-mooneybauzas: no in this case they are to ensure that updates to upper constratis or new lib release dont break placement16:09
bauzasbut looking at zuul sounds ok16:10
bauzassean-k-mooney: I know the job16:10
bauzassean-k-mooney: I was wondering what to do with the result :)16:10
sean-k-mooneyah16:10
bauzassean-k-mooney: we could trigger something, ideally a bug report16:10
sean-k-mooneywell ya we are just making sure they pass and if they dont then we should check it after the meeting16:10
artomMake it a recurring item on the meeting?16:11
bauzasbut yeah, scrubbing zuul seems good at first step16:11
bauzasprovided we keep the meetings :)16:11
sean-k-mooneyanyway i think we can move on16:11
artomIf there are no more meetings we're either all dead or working at Facebook16:11
gibiyepp this will be a recurring item on the meeting to look at it16:12
bauzasartom: I don't know what I'd prefer the least16:12
gibiand if it is not green the it triggers me to file a bug16:12
artombauzas, they're basically the same thing16:12
bauzasmoving on ?16:12
gibiany other gate issue we need to discuss?16:12
gibi#topic Release Planning16:13
*** openstack changes topic to "Release Planning (Meeting topic: nova)"16:13
gibiMilestone 1 is 27th of May. Nothing special is expected at that time.16:13
gibiAs in previous cycles we plan to have spec freeze at M2 only.16:13
gibiSee the nova specific schedule on #link https://wiki.openstack.org/wiki/Nova/Xena_Release_Schedule16:13
gibiany schedule questions? or release related topic?16:13
bauzaseek, time flies16:14
sean-k-mooneydoes the release team still want use to do lib release per milestone16:14
gibiI think so16:14
sean-k-mooneytechnially we dont have too but that was there perference16:14
bauzaslemme look at the docs16:14
sean-k-mooneyok so we should get os-vif ectra read for m116:15
sean-k-mooneythat fine we can do it next week or the week after16:15
gibia lib release is basically a free thing, I guess some script will propose a patch we need to approve16:15
gibisean-k-mooney: so you don't have to manually propose a release16:15
sean-k-mooneyright i just need to see if there is anything that should merge in it16:16
gibiyepp16:16
gibianything else about the release?16:16
gibi#topic Stable Branches16:17
*** openstack changes topic to "Stable Branches (Meeting topic: nova)"16:17
gibistable gate should be OK16:17
gibitrain-em tagging will happen next week, so train final release (+ for the rest of the maintained stable branches) prepared:16:17
gibitrain 20.6.1, ussuri 21.2.1, victoria 22.2.1, wallaby 23.0.1 -- https://review.opendev.org/q/project:openstack/releases+intopic:nova+status:open16:17
gibiEOM(from elod)16:17
bauzasI need to do some wallaby scrubbing16:17
gibiI guess if somebody needs a train fix then this is the ultimate last time to release it16:18
bauzasI guess we're planning a stable/wallaby release soon ?16:18
gibibauzas: it is proposed16:18
elodbauzas: I've proposed the patch,16:18
bauzashaven't seen it16:18
gibihttps://review.opendev.org/q/project:openstack/releases+intopic:nova+status:open16:18
elodbut please -1 it if you want something there16:18
elodto wait16:19
artom"if somebody needs a train fix" *cries in OSP 16*16:19
bauzaswe have a couple of gibi's backports https://review.opendev.org/q/project:openstack/nova+branch:stable/wallaby+is:open16:19
bauzasartom: don't sell your drug here, there are kids16:20
gibithat is a big, but clean, backport16:20
gibiI'm OK if it needs more time to settle16:20
bauzasgibi: your choice16:20
gibibauzas: it is more lyarwood's and elod's choice ;)16:20
bauzasI'm subtle16:21
elodi guess we can release train even if we keep wallaby release open16:21
bauzasya16:21
elodok16:21
gibiif nothing else then moving on16:21
gibi#topic Sub/related team Highlights16:22
*** openstack changes topic to "Sub/related team Highlights (Meeting topic: nova)"16:22
gibiLibvirt (bauzas)16:22
bauzasnothing important to mention, highly focusing on closing some nasty bug on vGPUs that could help mnaser16:22
bauzas(mdevs are stupidely not persistent)16:23
bauzasapart from it, I haven't seen other libvirt related asks16:23
bauzasand we're early in the cycle so we have time for thinking about the versions we'd like to support16:23
bauzasthat's it.16:24
sean-k-mooneyhave we started to do the verion bump yet16:24
sean-k-mooneyit might be nice to do that before m116:24
sean-k-mooneyah soory so no we have not16:25
bauzasnope, haven't seen it16:25
gibigood point. We need a volunteer to look at the possible versions we can bump to16:25
bauzasI can take it16:26
gibithanks16:26
sean-k-mooneywell that is in the code already as next_*16:26
sean-k-mooneybut we need to know what that should be updted too16:26
gibitrue, then finding a new next16:26
sean-k-mooneyyep16:26
gibianything else?16:26
sean-k-mooneywe also have the option of not bumping it if there is not meaningful cleanup to be had16:26
bauzaswell16:27
bauzasI guess it's a valuable ask16:27
gibiyepp16:28
gibimoving on16:28
gibiezeket gondolom nem16:28
gibisorry16:28
gibi#topic Open discussion16:28
*** openstack changes topic to "Open discussion (Meeting topic: nova)"16:28
artomI mean, you *have* to translate that now16:28
gibi(artom) Normalize (via workaround config option?) the Rabbit queue name to lowercase to avoid things like https://bugzilla.redhat.com/194207916:28
openstackbugzilla.redhat.com bug 1942079 in openstack-nova "Instances are stuck in scheduling/building state when scheduled on specific compute host" [High,New] - Assigned to nova-maint16:28
artomSo the bug itself is mostly useless at this point, filled with noise16:28
gibiartom: these I don't think ... (it was really a fragment only)16:28
artomBut the problem is that Rabbit exchange names can be either lower case or upper case16:29
artomAnd we're currently not forcing them one way or another, and relying on things like hostname16:30
sean-k-mooneywell rather they are case sensivite16:30
artomSo the proposal is to normalize them to lower case.16:30
artomGate that behind a workaround config option, because of the upgrade impact16:31
artomAnd the question is: full spec or specless blueprint?16:31
bauzasthere is an upgrade impact, right?16:31
sean-k-mooneyyes16:31
artomYeah, it would have to be opt-in16:31
bauzasthen you have your answer...16:32
gibiartom: yepp a small spec about the upgrade impact would be nice16:32
sean-k-mooneytl;dr if you service is UPPERCASE or mixed today so will the queue name be16:32
bauzasthat's my point16:32
bauzasand controllers and computes should expect the same queue name, right?16:33
sean-k-mooneyso there is a rolling upgrade impact as all nodes have to agree16:33
gibiI guess we need coordination between rpc client and server side to change the name16:33
sean-k-mooneybauzas: yep16:33
artomgibi, ack, sounds like bauzas's on your side, spec it is then16:33
bauzasexactly, we need to paper it16:33
gibiartom: dont need to be big one, and you can ping me to read it16:33
bauzasartom: sorry, I did shoot you16:33
artomMaybe in it we can figure out a way to migrate deployments to the new thing16:34
bauzasbut I think this isn't trivial if we do care of our customers upgrading :D16:34
artom... if they currently have uppercase exchange names. But that seems hard/impossible.16:34
artomgibi, ack, appreciated :)16:34
artombauzas, you... shoot me?16:34
bauzasone way to consider it would be "sorry, dude, you made a big mistake, hence UNSUPPORTED. Bye."16:34
bauzasbut nah, we're gentle16:35
bauzasartom: because you were full of hope16:35
artomThere are easier ways to drain me of all hope16:35
artomMuch, much easier :(16:36
gibiI thin kwe can move on :)16:36
sean-k-mooneyartom: well the upgrade plan would be  add config option, add nova status check, then flip default and then always do it right16:36
sean-k-mooneywe just need to do it over a few releaes which we can detail in the spec16:36
artomsean-k-mooney, spec review time :)16:37
gibisomething like that yes ^^16:37
artom(Well, once I post it)16:37
gibi:)16:37
sean-k-mooneyalthoug i was hoping we could treat this as a bug fix16:37
bauzassean-k-mooney: some programmibility maybe with service checks too16:37
sean-k-mooneywith it off by defualt16:37
sean-k-mooneythe other side of this coin is our db schem16:38
sean-k-mooneywhich is currently case insenitve by defualt16:38
sean-k-mooneyeven though in python we are case sensitive16:38
artomDoesn't that depend on the DBMS in use?16:38
bauzasright, and we store the message queues16:38
bauzasin the  API DB16:38
sean-k-mooneyartom: it depens on the coalation type16:38
artomThough I guess... we only support MariaDB and SQLite for CI at this point?16:38
sean-k-mooneywhich we do not specify16:38
bauzassounds a problem to me16:39
artomRight, so whatever's the default16:39
artomYeah, OTOH have we had bugs around that?16:39
bauzasbecause we would need to update the API DB16:39
sean-k-mooneywhcih is utf8_general_ci16:39
sean-k-mooneyci being case insensitive16:39
bauzaswe can't rely on the collation the operator decidd16:39
bauzasdecided*16:39
sean-k-mooneywell we do today16:40
bauzashmmm16:40
sean-k-mooneythat is how this issue came about16:40
dansmithwhat does the schema sensitivity have to do with it really?16:40
gibibut collation only about ordering, we still store the data without normalizing it16:40
dansmiththat's just for matching, right? it still stores the thing you gave it, I think16:40
melwittyes16:40
sean-k-mooneyif the name is HOST and we query by host it will match the same reccord16:41
artomYeah, it affects queries like where foo like bar, or where foo = bar16:41
dansmithso why can't we just start warning people who have non-lowercase exchanges, and then in a future release start normalizing them on input?16:41
artomSo FOO would match foo16:41
bauzasdansmith: that's one option16:41
bauzaswith a nova status check16:41
bauzaslike sean-k-mooney mentioned16:41
dansmithyeah, that and maybe also at runtime16:41
artomBut then how can they realistically move?16:41
melwittthat's what I was wondering16:42
artomDrain the compute node, remove it, re-add with the lowercase exchange?16:42
dansmithartom: the concern is people with mixed case records in their cell mappings right?16:42
artomWhat if the UPPERCASE exchange isn't node-specific16:42
sean-k-mooneyyep16:42
sean-k-mooneydansmith: they had some instace with uppercase name and some with lowercase16:42
bauzasartom: you hit the issue because we can't find the transport url, right?16:42
sean-k-mooneydepening on when the vm was booted on the host16:42
dansmithyeah, so we can easily make nova-manage update them with the lowercase variant without deleting the record right?16:42
sean-k-mooneyyep16:42
bauzasdansmith: I was considering it16:42
dansmithit's just the cell mapping that is the issue, not something that scales with instances16:43
sean-k-mooneyso maybe we need to detail this in the spec and esiced if its a feature or bug later16:43
bauzaswe need to lowercase for new computes but we somehow need a way to correct the transport url16:43
melwittoh, I was thinking they would have to change the exchange names in rabbit if they had used mixed case16:43
dansmithmelwitt: they would16:43
bauzasis rabbit case-sensitive ?16:43
sean-k-mooneybauzas: yes16:43
dansmithsean-k-mooney said it was, and I imagine it is16:44
artomdansmith said that sean-k-mooney said it was, and I imagine it is16:44
bauzasokok, so yeah we need both16:44
sean-k-mooneybauzas: our custoemr had 2 queues the one the conduictor set the message too and the one the comptue agent was litenting too16:44
artomOK, sounds like there's meat for a spec here16:44
dansmithpresumably we could also just make that one table be case-sensitive right?16:44
bauzasbut I guess that existing computes will have to register to another queue16:44
dansmithcell_mappings is tiny, even on giant clouds16:44
sean-k-mooneydansmith: maybe16:45
bauzashmmm16:45
sean-k-mooneyin this case i think its the service table we care about16:45
dansmithwe could just make it case sensitive too, if we want to avoid people who make poor naming decisions not have to retool their exchanges16:45
dansmithsean-k-mooney: eh?16:45
sean-k-mooneywe might be able to do it on the column16:45
dansmithsean-k-mooney: there's no transport url in service right?16:45
bauzasdansmith: that's a reasonable thing, because lowercasing would mean changing their queues they monitor16:45
bauzasthat's an operating breaking change16:46
sean-k-mooneydansmith: i think its the queue not the exchange16:46
dansmithbauzas: yeah people with mixed case now would have to quiesce their whole cloud (or a whole cell) to make the change16:46
sean-k-mooneythat was the issue16:46
dansmithnot ideal, but I dunno.. not that big of a deal I'd think16:46
dansmithstill, I don't see how this impacts anything other than cell_mappings16:46
bauzascorrect me wrong, but the queue names are written by us, not opt-driven16:47
bauzasif I*16:47
sean-k-mooneybauzas: they are created using the valud of conf.host16:47
bauzasah16:47
sean-k-mooneyas part of the topic/name16:47
bauzasgotcha16:47
dansmithoh, I see16:47
dansmiththat's where service comes in16:47
sean-k-mooneythe condocutor gets that form the servcice record host value16:47
bauzasso people writing hostnames with uppercase are stepping into this16:47
sean-k-mooneythe comptue agent gets it form the conf16:47
bauzasok, I see now the bug16:48
dansmithsean-k-mooney: and why does the case insensitivity of service matter?16:48
dansmithjust if they change their conf right?16:48
sean-k-mooneyyep16:48
dansmiththis is very meh, to me16:48
sean-k-mooneythey did as part of an FFU... to make vlaidation work...16:48
dansmithso we provide them a way to fix it if they want, but we don't need to make the runtime system overly concerned with this kind of thing right?16:49
sean-k-mooneyyes we are doing that16:49
dansmithwe should optimize for systems deployed with config management, and if they are and need to change, config management can automate running a tool for them too16:49
sean-k-mooneywe were just wondering if we could harden this so it cant happen16:49
sean-k-mooneyby makeing it always lowercase16:49
artomI think the catch is that said system cannot change the conf.host setting16:49
artomBecause that's written in a bunch of places, and is really hard to update16:50
dansmithartom: I don't think I understand that16:50
sean-k-mooneylets talk about this on the nova channel after the meeting16:50
sean-k-mooneyunless this is the last item16:50
gibiI have one small thing still16:50
gibi(gibi) Follow up on the review priority patch https://review.opendev.org/c/openstack/project-config/+/78752316:51
gibiso last week sean-k-mooney brought it up, but we still missing feedback on the review prioprity proposal16:51
gibiso it is a reminder to look at it and leave some feedback16:51
gibiEOM16:51
sean-k-mooneyi rebased it today with the 0..+1 range16:52
artomSo is the question just whether it's going to be 0..+1 or 0..+2?16:52
artomIOW, do we need the +1/+2 granularity?16:53
gibinope, it is question about the process I drafted in the comment16:53
gibihttps://review.opendev.org/c/openstack/project-config/+/787523/1#message-a8433d63f865d2fe4f20284e92a2e91772aa4ae616:53
sean-k-mooneyim generally in favor of what you proposed16:53
bauzas+0/+1 sounds good to me16:53
sean-k-mooneyyou were goign to codify that in the contibutor gide yes16:53
gibisean-k-mooney: yes16:54
bauzasjust one thing we haven't discussed AFAICR16:54
sean-k-mooneyi guess here https://github.com/openstack/nova/blob/master/doc/source/contributor/code-review.rst16:54
bauzaswhen and how the cores are going to decide which changes to flag ?16:54
sean-k-mooneythat part of the process gibi was suggesting16:55
bauzascan people propose a patch for a priority ?16:55
bauzaslike, flagging with +1 ?16:55
gibibauzas: good point. I think it should be similar to how we decided if something goes into a runway slot16:55
bauzasand then, cores accepting it by +2 it ?16:55
artombauzas, I like that16:55
bauzasso, it would be a +0/+1/+216:55
sean-k-mooneyyes16:55
gibibauzas: right now the proposal is to only have 0/+1 and onyl core can set it to +116:55
sean-k-mooneyi can split that by group16:56
bauzasI know16:56
bauzasbut I was considering to open the proposal16:56
bauzashence the +116:56
bauzaslike any other change16:56
bauzasconsidering we don't -116:56
sean-k-mooneyi guess the workflow as written woudl be bring it up in the meeting or irc16:56
bauzasif we don't accept a proposal, we just reset it to +016:56
gibido we make that a place to ping-poing?16:57
bauzassean-k-mooney: I don't like meetings16:57
sean-k-mooneyhehe16:57
bauzassean-k-mooney: because a certain percentage of our contribs couldn't attend it16:57
sean-k-mooneycan i suggest moving it to the review16:57
bauzasyeah, we're over time soon16:57
sean-k-mooneyill update it to whatever the concensou is16:57
gibisean-k-mooney: ++16:57
artomYeah... one potential problem is people just spamming +116:58
gibilets continute discussing the proposal in the review16:58
artomAt which point +1 is the new 0, and becomes meaningless16:58
bauzasI opened the review tab, I'll decently throw some notes16:58
*** ralonsoh has quit IRC16:58
gibiartom: yepp that was my thinking too16:58
gibianyhow. Is there any other topic for today/16:58
gibi?16:58
gibiif not then thank you for joining16:59
gibio/16:59
gibi#endmeeting17:00
*** openstack changes topic to "OpenStack Meetings || https://wiki.openstack.org/wiki/Meetings/"17:00
openstackMeeting ended Thu May  6 17:00:10 2021 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)17:00
openstackMinutes:        http://eavesdrop.openstack.org/meetings/nova/2021/nova.2021-05-06-16.01.html17:00
openstackMinutes (text): http://eavesdrop.openstack.org/meetings/nova/2021/nova.2021-05-06-16.01.txt17:00
openstackLog:            http://eavesdrop.openstack.org/meetings/nova/2021/nova.2021-05-06-16.01.log.html17:00
*** sean-k-mooney has left #openstack-meeting-317:02
*** macz_ has quit IRC17:15
*** jamesmcarthur has quit IRC17:22
*** jamesmcarthur has joined #openstack-meeting-317:23
*** jamesmcarthur has quit IRC17:27
*** macz_ has joined #openstack-meeting-317:34
*** psahoo has quit IRC17:51
*** belmoreira has quit IRC18:01
*** jamesmcarthur has joined #openstack-meeting-318:17
*** jamesmcarthur has quit IRC18:37
*** jamesmcarthur has joined #openstack-meeting-318:37
*** jamesmcarthur has quit IRC18:42
*** claw4260 has left #openstack-meeting-318:56
*** e0ne has joined #openstack-meeting-319:29
*** e0ne has quit IRC20:01
*** e0ne has joined #openstack-meeting-320:03
*** slaweq has quit IRC20:38
*** e0ne has quit IRC20:42
*** e0ne has joined #openstack-meeting-321:00
*** artom has quit IRC21:07
*** artom has joined #openstack-meeting-321:16
*** e0ne has quit IRC23:04
*** macz_ has quit IRC23:14
*** tosky has quit IRC23:17
*** dustinc has quit IRC23:34

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!