16:00:01 <gibi> #startmeeting nova
16:00:02 <openstack> Meeting started Thu Apr  9 16:00:01 2020 UTC and is due to finish in 60 minutes.  The chair is gibi. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:00:03 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
16:00:05 <openstack> The meeting name has been set to 'nova'
16:00:10 <gibi> o/
16:00:17 <dansmith> o/
16:00:45 <elod> o/
16:00:53 <stephenfin> o/
16:01:04 <gmann> o/
16:01:31 <melwitt> o/
16:01:52 <gibi> #topic Last meeting
16:01:57 <gibi> #link Minutes from last meeting: http://eavesdrop.openstack.org/meetings/nova/2020/nova.2020-04-02-16.00.log.html
16:02:07 <lyarwood> o/
16:02:15 <gibi> anything from the last meeting where follow up is needed?
16:03:10 <gibi> #topic Bugs (stuck/critical)
16:03:17 <gibi> no critical bug
16:03:22 <gibi> #link 111 new untriaged bugs (+3 since the last meeting): https://bugs.launchpad.net/nova/+bugs?search=Search&field.status=New
16:03:36 <gibi> I guess from next week everybody will focus on bugs ;)
16:03:53 <gibi> is there any bug that needs attention?
16:04:34 <lyarwood> triaged here means importance is set and status != new right?
16:04:44 <lyarwood> there's a few at the top of the list we can likely move
16:04:56 <gibi> lyarwood: yeas, basically you verified that this is really a bug
16:05:01 <lyarwood> kk
16:05:09 <gibi> https://wiki.openstack.org/wiki/Nova/BugTriage#Tags
16:05:12 <lyarwood> thanks
16:05:13 <bauzas> \o
16:05:36 <gibi> #topic Gate status
16:05:41 <gibi> The Gate is in FF mode (i.e. clogged)
16:05:50 * bauzas can't make promises on bug triage but has reasonable expectations for next week
16:05:50 <gibi> but it feels less clogged than last time
16:05:56 <gibi> Recently the hit rate of https://bugs.launchpad.net/nova/+bug/1823251 is increased so I re-heated the patch that temporarily disables the unstable tests https://review.opendev.org/#/c/718629/
16:05:57 <openstack> Launchpad bug 1823251 in OpenStack Compute (nova) "Spike in TestNovaMigrationsMySQL.test_walk_versions/test_innodb_tables failures since April 1 2019 on limestone-regionone" [High,Confirmed]
16:06:02 <gmann> yeah, takes time to get the node for testing
16:06:17 <gibi> any other gate issue that needs attention?
16:07:14 <melwitt> I've seen several hits around "multiple networks found" and proposed this https://review.opendev.org/716809 but it's the wrong approach
16:07:39 <melwitt> that is, this tempest test involves passing a network during server create, and it's supposed to get that network from creds
16:07:49 <melwitt> but the network is not present in the creds for some reason
16:08:05 <melwitt> could use some help understanding that one
16:08:26 <melwitt> other than that, I can mention that the fix for https://bugs.launchpad.net/nova/+bug/1844929 is approved and I'll be rechecking it through the gate
16:08:28 <openstack> Launchpad bug 1844929 in OpenStack Compute (nova) "grenade jobs failing due to "Timed out waiting for response from cell" in scheduler" [High,In progress] - Assigned to melanie witt (melwitt)
16:08:40 <gmann> melwitt: i did not get chance to look into logs, I will do after this FF thngs
16:08:49 <melwitt> gmann: cool thanks
16:09:07 <gibi> melwitt, gmann: thanks
16:09:14 <dansmith> melwitt: does your cells one have a e-r query to watch for falling hits after mege?
16:09:16 <dansmith> *merge
16:09:21 <melwitt> dansmith: yes
16:09:24 <dansmith> sweet
16:09:37 <dansmith> if I could order popcorn from the empty shelves of the grocery stores, I would.
16:09:51 <melwitt> :P
16:09:57 <gibi> :)
16:09:58 <bauzas> melwitt: will follow-up on this change tomorrow if needed
16:10:23 <gibi> #topic Release Planning
16:10:32 <gibi> happy Feature Freeze day everyone!
16:10:55 <gibi> we also have the final novaclient release today for Ussuri
16:11:02 <gibi> I posted an info mail to the ML #link http://lists.openstack.org/pipermail/openstack-discuss/2020-April/014018.html
16:11:12 <gibi> and started the release TODO etherpad #link https://etherpad.openstack.org/p/nova-ussuri-rc-potential
16:11:35 <gibi> please review the cycle highlight patch #link https://review.opendev.org/#/c/712498/ as it is also due today
16:11:52 <gibi> (we can update it later, but the today's version will be used for marketing)
16:12:19 <gibi> who wants to volunteer for writing a release notes prelude?
16:12:25 <bauzas> I can
16:12:30 <gibi> bauzas: thanks!
16:12:31 <bauzas> (I'm used to)
16:12:57 <gibi> I noted your nick on the etherpad
16:13:04 <bauzas> tbc, we need to merge the prelude change before RC1
16:13:13 <bauzas> so we have two weeks for filling it
16:13:24 <bauzas> but I'll start with a proposal and people will be able to comment
16:13:25 <gibi> bauzas: good, that is plenty of time
16:13:39 <bauzas> (or even co-author if they wish)
16:14:34 <bauzas> gibi: and sure for naming me in the etherpad
16:14:42 <gibi> gmann pinged me about an FFE for the policy work, lets discuss that at the end
16:14:49 <gmann> ok
16:15:05 <gibi> any other thing we need to discuss for FF and the novaclient release?
16:15:45 <bauzas> like I proposed, I'll be around for rechecking tomorrow
16:16:09 <gibi> I will try to look at the gate time to time during the extended weekend
16:16:10 <bauzas> if people have +Wd changes before tonight, they can ping me for asking a necessary +W if they got merge conflicts in the gate
16:16:22 <gibi> ^^ +1
16:17:06 <bauzas> if so, please raise your request for fast-approval on your change in https://etherpad.openstack.org/p/nova-ussuri-rc-potential
16:17:51 <gibi> yeah, one more thing. FFE requests are expected by ML post. We have 2 weeks for RC1 so I feel we need to close the FFE items end of Wednesday next week
16:18:14 <bauzas> and we're on a short week
16:18:18 <gibi> which means there wont be too much time to argue about FFE requests
16:18:51 * gibi try to set expectations
16:19:07 <gibi> anything else before moving on to stable status?
16:19:31 <sean-k-mooney> do we know if the olso chagnes will land for the policy thing
16:19:38 <sean-k-mooney> we can cover that later too
16:19:50 <gibi> I have not context on that
16:20:07 <gibi> s/not/no/
16:20:17 <sean-k-mooney> the waring on deprecated default policies
16:20:17 <gibi> others?
16:20:48 <gmann> sean-k-mooney: wanring one is done - https://review.opendev.org/#/c/717879/
16:20:49 <sean-k-mooney> gmann i think has patches submitted and requested an oslo FFE for them
16:21:06 <gmann> sean-k-mooney: new flag for migration is pending - https://review.opendev.org/#/c/717943/
16:21:12 <sean-k-mooney> cool
16:21:54 <gibi> gmann: do we need follow ups on the nova side for that ^^ ?
16:22:16 <gibi> if yes, please make a note on the etherpad #link https://etherpad.openstack.org/p/nova-ussuri-rc-potential
16:22:24 <gmann> I am testing that on gate but due to new config options, flag is set true properly. I tested locally and ot worked
16:22:33 <gmann> gibi: +1, i will add
16:22:55 <gibi> gmann: thanks
16:23:52 <gibi> #topic Stable Branches
16:24:03 <gibi> lyarwood do we need to release stable/train and stable/stein as we are close to ussuri?
16:24:22 <gibi> I mean close to have stable/ussuri
16:24:41 <lyarwood> gibi: we can but I'd like to flush the queue post M3
16:24:55 <bauzas> good call on flushing the queues
16:24:58 <lyarwood> gibi: I'll look into that next week once the gate settles down
16:24:59 <gibi> lyarwood: agree.
16:25:07 <melwitt> usually good to release stable branches each milestone (if there's enough changes)
16:25:10 <bauzas> we could find bugs on ussuri that 'd drop down the releases
16:25:19 <gibi> lets flush the queue and release before RC!
16:25:21 <gibi> RC1
16:25:24 <lyarwood> yup ack
16:25:27 <gibi> thanks
16:25:36 <gibi> anything else on the stable land?
16:25:58 <bauzas> also, worth saying that we will refuse bugs on ussuri between RC1 and release date
16:26:07 <bauzas> for ussuri I mean
16:26:16 <lyarwood> bugs are fine until RC however right?
16:26:23 <bauzas> so, next two weeks are pretty straightforward
16:26:25 <melwitt> unless they're regressions
16:26:30 <bauzas> that ^
16:26:38 <bauzas> lyarwood: before RC1, yeah any bug is fine
16:26:42 <gibi> bugs I fine until RC1 if they are regressions or low risk
16:26:47 <bauzas> after RC1, then we branch
16:26:51 <lyarwood> ack cool just confirming
16:27:04 <gibi> after RC1 we only merge bugfixes that warrant a new RC
16:27:07 <melwitt> I mean after RC1 if there's a regression that's grounds for another RC
16:27:17 <gibi> ^^ +1
16:27:18 <lyarwood> yup understood
16:27:53 <gibi> #topic Sub/related team Highlights
16:28:03 <gibi> Placement (tetsuro)
16:29:01 <gibi> API (gmann)
16:29:09 <gmann> no other things to mention than FFE for policy work which we will discuss at the end
16:29:21 <gibi> gmann: thanks
16:29:26 <gibi> #topic Stuck Reviews
16:29:31 <gibi> nothing on the agenda
16:29:38 <gibi> do we have stuck review to discuss?
16:30:46 <gibi> #topic Open discussion
16:30:53 <gibi> (gmann): FFE for the remaining policy work.
16:30:57 <gibi> gmann: your turn ^^
16:32:18 <gmann> for policy work, we left with few policy. as rough count it is- ~7 policy patches up but not merged + 3 API policies to push patches
16:32:57 <gmann> and i am thinking to update the deprecated API policy also to be consistent because they are still valid for older version
16:33:23 <gibi> I can spend time on reviewing those on Tuesday, Wednesday next week
16:33:25 <gmann> i need more time to finish those. tried my best to push hard since last couple of week but could not finish
16:33:49 <gmann> gibi: thanks.
16:33:59 <gibi> gmann: what if we don't do the deprecated part of the work for U? Is the rest still usable?
16:35:03 <gmann> gibi: yes, that is usable and by default all the new changes (scope and defaults) are not enabled so existing deployment would break
16:35:14 <gmann> that is what we tested with test cases
16:35:43 <gibi> you mean "would not break"
16:35:51 <bauzas> alas. hopefully :)
16:36:17 <gmann> no, it will keep working until they enable scope flag which is disabled by default
16:36:34 <gibi> OK, so the feauter is disable by default. cool
16:36:57 <gibi> other, what do you think about granting FFE for the policy work?
16:37:04 <gibi> others
16:37:09 <gmann> and if they enable the scope flag enforce_scope and new flag enforce_new_defaults then everything move to new system
16:37:32 <bauzas> I have pros and cons
16:38:11 <bauzas> I feel it's reasonable to assume that the changes are self-contained and the risk is low
16:38:21 <gmann> my hope is in ussuri we rlease so that operator when they switch the flag to adopt new things all policy get the changes
16:38:53 <bauzas> on the other hand, just by looking at https://review.opendev.org/#/q/topic:bp/policy-defaults-refresh+%28status:open+OR+status:merged%29 I can see there is still a massive effort to make for getting all the patches merged
16:38:55 <gmann> instead of few policy left to have new things
16:39:05 <melwitt> I'm thinking it would be good to have the complete set of new policy defaults together in one release
16:39:15 <gmann> yeah, +1
16:39:17 <melwitt> and scope types support
16:39:18 <bauzas> and looking at the timings, we are short
16:39:29 <bauzas> the RC1 period is two weeks
16:39:43 <bauzas> and we have this Friday and Monday where people on or off
16:39:44 <gmann> bauzas: other than first patch which is adding tests cases, changes are very straight forward
16:40:25 <gmann> straight forward means changing the policy on same line with decision of correct scope and dfaults
16:40:26 <bauzas> so, yeah, I'm intended to say yes, but I have concerns about whether we can merge the whole series by Wednesday
16:40:53 <bauzas> gmann: sure, but like I said to you, there is https://review.opendev.org/718348 that's not merged yet
16:41:12 <bauzas> and if we say we want the whole series by Ussuri, this one necessarly has to work
16:41:47 <gmann> Wednesday ?
16:42:20 <bauzas> I don't know, I don't want to give a specific deadline
16:42:24 <gibi> as I meantioned above we have two weeks til RC1 so it would be good to done with FFEs at next Wednesday
16:42:29 <gmann> I am pretty confident to make all work (other than deprecated APIs) up before Tuesday
16:42:45 <bauzas> but honestly, given 2 weeks of RC1, giving more than one extra week for FFEs seems unreasonable
16:43:19 <gibi> if we have things up by Tuesday then it might need still couple days to review
16:43:27 <bauzas> honestly, I don't know what to say, stephenfin did most of the reviews so he should speak IMHO
16:43:45 <gibi> still I tend to agree on granting a FFE due to the impact we can make if we finish
16:43:49 <bauzas> me too
16:43:50 <gmann> gibi: and considering the gate and rebase things if happens
16:44:07 <bauzas> I'm .9 on this one but i have concerns
16:44:11 <stephenfin> No arguments from me, fwiw. They're pretty easy to review
16:44:33 <gibi> stephenfin: will you have bandwidth for review next week for the policy patches?
16:44:39 <bauzas> who would review besides stephenfin ? gibi, me ?
16:44:47 <bauzas> hah, jinxed
16:44:48 <stephenfin> Sure. Just not Monday (I'm off)
16:44:58 <gibi> yeah I'm also back on Tuesday
16:44:59 <stephenfin> johnthetubaguy is the other person that's been doing most of the reviews
16:45:02 <gmann> johnthetubaguy: review also but he is not online now so cannot say
16:45:06 <gmann> yeah
16:45:06 <melwitt> I can help review
16:45:11 <gibi> cool
16:45:15 <gmann> thanks melwitt
16:45:20 <melwitt> I will be here on monday
16:45:34 <bauzas> then...
16:45:36 <gibi> anybody else has concerns?
16:46:31 <gibi> #info FFE is granted for the policy work. We are aiming for finishing it next week.
16:46:43 <gmann> thanks everyone
16:46:55 <gibi> gmann: please drop a mail to the ML about it, just for formality.
16:47:11 <gmann> on including this on cycle highlights page. I am working on adding doc and reno. can we get those today and link to page which is deadline today ?
16:47:19 <gmann> gibi: yeah. noted
16:47:25 <bauzas> gmann: that looks difficult
16:47:39 <bauzas> we can't tell about things that haven't merged yet, right?
16:47:49 <gibi> gmann: could you write like 2-3 senteces to me after the meeting that I can put it into the highlight?
16:47:58 <gmann> ok
16:48:00 <bauzas> and i don't know whether we can patch the highlights later on
16:48:04 <gibi> bauzas: I'm not sure
16:48:09 <gibi> bauzas: we can patch
16:48:13 <gibi> Im sure about that
16:48:14 <bauzas> then
16:48:18 <melwitt> we can patch highlights later but they won't go into marketing materials
16:48:24 <bauzas> let's patch once it's merged, nope ?
16:48:26 <gibi> the only limit is that the marketing folks will start using the today's version
16:48:27 <gmann> we can patch later but only things is it can miss if market team pick before modification
16:48:31 <gmann> yeah
16:48:40 <gmann> if they pick monday or later then even better
16:49:58 <gmann> so i can write the 2-3 lines and in parallel work on doc/reno which can be linked later ?
16:49:58 <gibi> we already merged parts of the policy support
16:50:13 <gibi> so we can write a generic highlight that is true already
16:50:24 <stephenfin> we already have improved policy support, so we wouldn't be lying
16:50:31 <gibi> yes ^^
16:50:36 <gibi> that is what I'm thinking
16:50:41 <bauzas> lol
16:50:42 <gibi> and the details will be in the reno
16:50:47 <stephenfin> the lie would be to say it's complete :)
16:50:56 <gibi> exaclty
16:50:57 <stephenfin> yup, reno ftw
16:51:02 <bauzas> that's what happens when you ask nerds to provide you marketing materials
16:51:12 <gibi> I don't feel bad about it
16:52:12 <gibi> OK. I will update the cycle highlight patch today
16:52:26 <gibi> anything else to discuss?
16:52:39 <gmann> thanks. I will write the summary and provide to you
16:52:47 <gibi> gmann: awesome
16:53:53 <gmann> thanks everyone for supporting it /o\.
16:54:27 <gibi> if nothing else then I will close the meeting and go to pray for God Zuul a bit
16:54:44 <artom> Quick questions - RC1 is in 2 weeks, so that's the time window we have to merge bug fixes?
16:54:54 <sean-k-mooney> yes
16:54:57 * artom didn't realize it was this short
16:55:13 <artom> And after that it's only regressions, or really bad stuff?
16:55:14 <sean-k-mooney> after rc1 the master branch switich to Victoria
16:55:17 <gibi> artom: between now and RC1 we fix bugs, between RC1 and GA we only fix release breaking bugs
16:55:18 <bauzas> artom: we can merge bugs *after* RC1 but those will be officilally Victoria
16:55:52 <bauzas> if you want them to be Ussuri, hold your breath for a couple of weeks or shout it's a regression fix
16:56:03 <sean-k-mooney> between rc1 and relase teh stable branch is till manage by the core team
16:56:22 <bauzas> again, we can merge bugs
16:56:23 <sean-k-mooney> once the releae is done normal backports can be don to stable/usuri
16:56:37 <bauzas> they are just on the master branch, which won't be Ussuri
16:56:38 <artom> Understood, thanks all
16:56:46 <melwitt> yeah for regressions after RC1 that causes another RC and then the fix backported to stable/ussuri
16:57:03 <artom> Aha, so RC1 = master becomes Victoria
16:57:08 <sean-k-mooney> yep
16:57:13 <artom> (And presumably stable/ussuri is cut)
16:57:21 <sean-k-mooney> yes
16:57:39 <artom> Understood
16:57:43 <sean-k-mooney> i think we are done o/
16:57:48 <gibi> OK
16:57:52 <gibi> thanks you folks
16:57:57 <gibi> and happy Easter!
16:58:04 <bauzas> ++
16:58:07 <gibi> #endmeeting