Tuesday, 2023-02-28

*** thuvh1 is now known as thuvh07:15
*** thuvh1 is now known as thuvh07:28
opendevreviewJorge San Emeterio proposed openstack/nova master: WIP: Creating an example of the refactor privileged functions will go through.  https://review.opendev.org/c/openstack/nova/+/87549708:12
gibio/08:12
bauzashola08:13
gibibauzas: you mentioned things to clean up before RC1, do you have a list?08:13
bauzasgibi: indeed,08:13
bauzashttps://etherpad.opendev.org/p/nova-antelope-rc-potential08:13
gibibauzas: ack, on it08:14
* bauzas is working on your comments for https://review.opendev.org/c/openstack/nova/+/87493208:14
bauzasgibi: actually, we only have the prelude and the logging revert patch to review08:15
bauzas(for the moment)08:15
bauzasalso, I forgot to tell about RBAC defaults, shit.08:16
bauzas(in the cycle highlights)08:16
bauzashttps://etherpad.opendev.org/p/nova-antelope-blueprint-status it wasn't in the etherpad, neither in the Antelope blueprints08:16
bauzasgibi: about your question about SLURP, well, maybe we should wait for dansmith to be here08:18
bauzasgibi: but AFAIK, we try in Nova to support a N-2 compute 08:19
bauzaswhen N is a SLURP release08:19
bauzasgibi: dansmith tried at least to have a job for testing it08:19
gibibauzas: I'm fine to make such a statement but then lets make it a clear statement somewhere that we support N-2 computes not just easing on the constraint in the code. 08:25
gibii.e. our change in what is supported is hard to discover as it mentioned nowhere deployer facing08:26
bauzasgibi: yep, I see your point08:32
bauzasfwiw, we haven't yet agreed on this, which should be done at the vPTG08:32
bauzasgibi: got a second for thinking about 2023.1 SLURP implications ?08:39
* bauzas is reading carefully https://governance.openstack.org/tc/resolutions/20220210-release-cadence-adjustment.html08:39
gibibe back in 5min to think08:40
opendevreviewSylvain Bauza proposed openstack/nova master: Update min service version for Antelope  https://review.opendev.org/c/openstack/nova/+/87493208:47
* gibi is back08:47
bauzasright in time08:47
bauzasI totally flipped my mind08:48
gibiso what I quoted from that dock is #608:48
gibis/docks/doc/08:48
bauzasyup08:48
bauzassee my latest revision08:48
gibilooking...08:48
bauzasmy wonder is, do we have a grenade-skip-level job in place that validates our N-2 computes ?08:48
opendevreviewJorge San Emeterio proposed openstack/nova master: WIP: Creating an example of the refactor privileged functions will go through.  https://review.opendev.org/c/openstack/nova/+/87549708:49
bauzasyeah, we do08:49
bauzashttps://github.com/openstack/nova/blob/master/.zuul.yaml#L75908:49
bauzashttps://zuul.openstack.org/builds?job_name=grenade-skip-level&project=openstack%2Fnova&skip=008:50
opendevreviewSylvain Bauza proposed openstack/nova master: Update min service version for Antelope  https://review.opendev.org/c/openstack/nova/+/87493208:58
opendevreviewSylvain Bauza proposed openstack/nova master: DNM (yet) Update min support for Bobcat  https://review.opendev.org/c/openstack/nova/+/87562108:58
gibibauzas: replied inline09:00
gibithe current patch still proposes allowing Antelope to support Yoga computes, but it does not document it09:00
gibiahh you pushed PS309:01
gibiI think my comments still valid for PS3 too09:02
bauzasgibi: tl,dr : I don't disagree with your very valid points, we haven't properly documented the level of support operators shall expect09:03
bauzasbut I think I understood the pattern09:03
bauzaswe'll remove the grenade-skip-level patch in Bobcat09:04
bauzasgibi: ideally, I'd clarify this in our meeting09:06
bauzaswith dansmith and others around09:06
gibiohh on that note, I might not be able to join today 09:07
bauzasI also wonder whether we should remove the grenade-skip-level job run on check pipeline at the same time of https://review.opendev.org/c/openstack/nova/+/875621/ in https://github.com/openstack/nova/blob/master/.zuul.yaml#L759-L76009:07
bauzasgibi: ok, then we'll try to discuss this first with dan before you run off09:08
gibiI think having a non voting skip level during Bobcat does not hurt if we clearly communicate what support (none) is expected in Bobcat for Zed computes09:09
bauzasgibi: I don't disagree but maybe it shouldn't be on the check pipeline09:10
bauzasgibi: maybe periodic + experimental09:10
gibiI'm fine either way. For me having a job we ignore is a non issue. I more affaraid of miscommunication of support09:11
bauzasand right after Bobcat RC1, we should set the job to *voting*09:11
opendevreviewJorge San Emeterio proposed openstack/nova master: WIP: Creating an example of the refactor privileged functions will go through.  https://review.opendev.org/c/openstack/nova/+/87549709:11
bauzasthe point here is that I'm always context loading at every end of a cycle, this is not great09:11
bauzasI really want we agree on a pattern to follow09:12
gibisure. have a pattern, document it, and then it is easier to follow09:12
bauzasso new PTLs should just copy/paste09:12
bauzasI'm glad we had the grenade-skip-level job running despite I completely forgot about it09:13
bauzasthis was added way earlier 09:13
bauzashttps://github.com/openstack/nova/commit/219c360dc19f79ce7df8a92f6487aee97a64ee4c#diff-108978819c05ae183d88ec87959c2341a94cfc3f9465e3aeee82d554217b4f5809:13
gibithe ptl guide has the timeline so I guess that is a good place to document the pattern to follow09:14
bauzasoh yeah09:14
bauzasI used it a lot09:14
bauzas+ the rc1 etherpads 09:14
bauzasI'll do a bit of scrubbing09:15
opendevreviewSylvain Bauza proposed openstack/nova master: DNM (yet) Update min support for Bobcat  https://review.opendev.org/c/openstack/nova/+/87562109:26
sean-k-mooneybauzas: we do not need a skip level upgrade job for 2023.209:28
bauzasyet again a violent agreement ? :)09:29
bauzassean-k-mooney: see my latest rev of https://review.opendev.org/c/openstack/nova/+/87562109:29
sean-k-mooneyi was just reading that but i dont think we need it in the periodic09:29
bauzassean-k-mooney: I'll wait for dansmith's insights on it09:30
bauzasif he's OK with only experimental, I'm fine.09:30
sean-k-mooneyit technically does not need to be there either but sure09:30
bauzasbut I'd somehow like the idea of having a periodic for ensuring we don't regress09:30
sean-k-mooneythere are no skip level upgrades to bobcat09:30
bauzasI know09:31
bauzas:)09:31
sean-k-mooneycool09:31
bauzas(well, I loaded again the context in my mind, I dare say)09:31
sean-k-mooneyso that job bascially is only needed half the year09:31
bauzaswell, we had it during the whole zed cycle too :)09:31
bauzasmy biggest concern that I raised to gibi is that it takes me always a lot of time to load again the context 09:32
bauzasso I want to eventually write docs about it09:32
sean-k-mooneyi think that was because that is when it was written09:32
sean-k-mooneybut sure it should proably get listed in the ptl guide or something like that09:32
bauzasyup, scroll it up09:32
bauzasI'm on it09:33
gibi:)09:33
bauzashttps://review.opendev.org/c/openstack/nova/+/831229 was when we added the job09:33
bauzasit was on the yoga timeframe09:33
bauzasand then we left it 09:33
sean-k-mooneyi see what release was it upgrading form during zed09:41
bauzasgibi: sean-k-mooney: do you remember how we tracked the default policy changes from gmann ? It isn't a blueprint AFAICS09:41
sean-k-mooneybauzas: its a blueprint/spec09:41
bauzasI'm maybe blind09:41
sean-k-mooneywell09:42
bauzassean-k-mooney: I'm not talking of https://blueprints.launchpad.net/nova/+spec/policy-service-role-default09:42
sean-k-mooneyok so the service role was 09:42
sean-k-mooneyya i asked for both to be blueprint/specs09:42
sean-k-mooneybecause i was unhappy with having no tracker at all09:42
sean-k-mooneyi think its in the etherpad09:42
bauzashttps://review.opendev.org/c/openstack/nova/+/86621809:43
bauzaswasn't tracked properly09:43
bauzashence the miss for the cycle highlights09:43
bauzasdoh09:43
sean-k-mooneywell i responded to gmann 09:43
sean-k-mooneyi dont think it need to be in nova sycle highlights09:44
sean-k-mooneyi think it shoudl go out in the overall release prelude09:44
bauzasthis is anyway too late for the cycle highlights09:44
bauzasI'll just mention it in the prelude09:44
sean-k-mooneywe should ahve one section that covers all the secure rbac work for all projects09:44
bauzasand I'll actually look at the generated relnotes to see if I missed anything else09:44
bauzassean-k-mooney: don't disagreed on your point for the cycle highlights09:45
bauzasthis is more a cross-project effort09:45
bauzasbut let's mention the nova related implications in the prelude09:45
sean-k-mooneysure09:45
sean-k-mooneyit should also be in the general release notes in the upgrades section09:46
gibi+109:46
sean-k-mooneyhttps://review.opendev.org/c/openstack/nova/+/866218/13/releasenotes/notes/enable-enforce-scope-and-new-defaults-14db8c75b263b599.yaml09:46
bauzasI'm pretty sure the marketing folks will handle that info correctly despite not being said in our cycle highlights :)09:46
sean-k-mooneyi would have preferd not to have it in the cycle highlights anyway09:47
sean-k-mooneyas i said i think its better to comunitcate this in a cross project way since it really needs to be enabled in a corrdenated way09:47
sean-k-mooneywith that said we really do need to complete the service role this cycle09:48
sean-k-mooneyto keep on track with the comunity goal schduel09:48
opendevreviewMaksim Malchuk proposed openstack/nova stable/xena: Fix to implement 'pack' or 'spread' VM's NUMA cells  https://review.opendev.org/c/openstack/nova/+/82980409:49
sean-k-mooneyim going to revert the logging patch now https://review.opendev.org/c/openstack/nova/+/87358409:49
bauzassean-k-mooney: yes about the service role, but gmann still has a -W on it09:49
sean-k-mooneybauzas: unless you object?09:49
bauzassean-k-mooney: nope, go for it, it's in the list-of-patches-to-merge-before-next-day09:50
sean-k-mooneycool its in the gate09:50
* bauzas amends the prelude with gmann's untracked feature and SLURP mentions09:50
sean-k-mooneythis is the base of the first offical SLU release09:51
sean-k-mooneydo we need to mentiaon anything beyond that09:52
sean-k-mooneywe dont offically support it form yoga09:52
sean-k-mooneythat was a test of our tooling/process09:52
sean-k-mooneyso while it should work i would not adivse people to use it09:52
sean-k-mooneythere wasnt an expecation taht all project support Yoga to Antelope09:53
opendevreviewSylvain Bauza proposed openstack/nova master: Add the 2023.1 Antelope prelude section  https://review.opendev.org/c/openstack/nova/+/87538010:02
stephenfinsean-k-mooney: https://review.opendev.org/c/openstack/oslo.db/+/875627 that will fix the annoying warnings you've been seeing in the functional tests10:21
*** elodilles is now known as elodilles_afk10:28
sean-k-mooneynice unfortuhnetly that wont be in 2023.1 at this point10:37
stephenfinyeah, we'll have to backport it10:37
sean-k-mooneyi assume once we cut rc1 tomrrow you would like your SQLA2.0 seriese to land sooner rather then later10:40
sean-k-mooneyi.e. droping the legacy migration and the other patches you have up10:40
stephenfinyes, please. get it in early10:40
sean-k-mooneyya that is what i was thinking10:41
sean-k-mooneyill try and do a pass of it next week10:41
sean-k-mooney                conf.get_location(key, group=group.name).location ==10:43
sean-k-mooney                cfg.Locations.opt_default10:43
sean-k-mooneyso i tought here was a cleaner way to do that10:43
sean-k-mooneyeither usign in or a method to check it the option was set from the default 10:44
stephenfinThere could be. I'm simply not aware of it if so10:44
sean-k-mooneyi have not used it personally but i vaguely recall it form the docs10:45
sean-k-mooneythe patch looks fine to me but i just want to check if that is a thing quickly10:45
sean-k-mooneystephenfin: ok no you are doing it the way you are ment too https://docs.openstack.org/oslo.config/latest/reference/locations.html10:46
sean-k-mooneythat is what i was recalling10:46
sean-k-mooneyi tought there was an is_default() or similar10:47
stephenfinyeah, I explicitly wanted to raise a warning if the application (i.e. nova) was setting it too10:47
stephenfinsince that's still wrong10:47
sean-k-mooneyactully you could use is_user_controlled10:47
sean-k-mooneyah10:48
sean-k-mooneyok so you want to also support set_default and set_override10:48
sean-k-mooneyso is_user_controlled is not correct10:48
stephenfinI want to also raise for the two of those, yes10:48
stephenfinexactly10:48
sean-k-mooneycool captured that in the review an dset Backport-Candidate on it10:50
sean-k-mooneyim debating if having Backport-Candidate would be useful for nova et al 10:52
sean-k-mooneyis it useful for oslo?10:53
stephenfinnot really, no10:54
sean-k-mooneyingeneral if the patch is not intentioanlly written to be backportable it wont be10:54
stephenfinThe biggest problem is lack of reviewers, not lack of things to review. Also, we don't tend to make many changes nowadays so there aren't many backports10:54
sean-k-mooneyso if its just set after the fact its probaly less useful10:55
sean-k-mooneyya reviews are a problem everywhere10:55
sean-k-mooneyalthough oslo with all its repos proably have it wrose then most10:55
opendevreviewJorge San Emeterio proposed openstack/nova master: WIP: Have schema for 'lock' action be applied to all microversions.  https://review.opendev.org/c/openstack/nova/+/87565311:46
*** elodilles_afk is now known as elodilles12:44
opendevreviewSamuel Kunkel proposed openstack/nova master: fix: amd-sev handle missing img properties  https://review.opendev.org/c/openstack/nova/+/87426413:00
opendevreviewAmit Uniyal proposed openstack/nova stable/wallaby: libvirt: Delegate OVS plug to os-vif  https://review.opendev.org/c/openstack/nova/+/79044713:03
opendevreviewAmit Uniyal proposed openstack/nova stable/wallaby: libvirt: Delegate OVS plug to os-vif  https://review.opendev.org/c/openstack/nova/+/79044713:07
ratailor_#openstack-nova can I get reviews for these patches https://review.opendev.org/q/Ia738a0972b050f549f446c85171d3f33e60ada4f and https://review.opendev.org/q/Id4c8c5f3b32985ac7d3d7c833b82e0876f7367c1 ?13:09
opendevreviewJorge San Emeterio proposed openstack/nova master: WIP: Have schema for 'lock' action be applied to all microversions.  https://review.opendev.org/c/openstack/nova/+/87565313:20
opendevreviewJorge San Emeterio proposed openstack/nova master: WIP: Have schema for 'lock' action be applied to all microversions.  https://review.opendev.org/c/openstack/nova/+/87565313:21
*** lowercase is now known as Guest617313:42
*** lowercase_ is now known as lowercase13:42
lowercasestephenfin: I looked into our next enviornment, which also contains a build_requests table with no columns.13:43
lowercasein an identical state to the dev13:43
opendevreviewAmit Uniyal proposed openstack/nova stable/wallaby: libvirt: Delegate OVS plug to os-vif  https://review.opendev.org/c/openstack/nova/+/79044713:45
lowercaseand i just checked one of my production databases, also include a build_requests in the same state. select * from build_requests; Empty set (0.000 sec)13:46
opendevreviewAlexey Stupnikov proposed openstack/nova stable/victoria: Test aborting queued live migration  https://review.opendev.org/c/openstack/nova/+/84574813:52
opendevreviewAlexey Stupnikov proposed openstack/nova stable/victoria: Add functional tests to reproduce bug #1960412  https://review.opendev.org/c/openstack/nova/+/84575313:52
opendevreviewAlexey Stupnikov proposed openstack/nova stable/victoria: Clean up when queued live migration aborted  https://review.opendev.org/c/openstack/nova/+/84575413:52
opendevreviewSylvain Bauza proposed openstack/nova master: Update to the PTL guide  https://review.opendev.org/c/openstack/nova/+/87573013:59
opendevreviewSylvain Bauza proposed openstack/nova master: Update to the PTL guide  https://review.opendev.org/c/openstack/nova/+/87573014:05
bauzaselodilles: I'm done with the meeting wikipage, feel free to amend it anytime you want14:46
elodillesbauzas: ack, thanks14:47
elodillesbauzas: well, actually, it is done o:)14:48
elodillesno need to update :X14:48
bauzaselodilles: I'll name you Barry Allen14:48
elodillesjust a sec, have to google this. :S14:50
dansmithbauzas: why remove the grenade-slip-level job for bobcat? it's not required, but we might as well keep it.. is it because it hasn't yet been updated for zed->bobcat?14:51
bauzasdansmith: good question I was about getting a coffee before pinging you to ask you what you would prefer14:54
dansmithpersonally I'd rather leave it in place as much as we can, as I think it just adds extra coverage14:54
dansmithhowever, I need to look at it again,14:54
bauzasdansmith: okay, so we would leave non-voting in a non-SLURP branch and just make it voting after Bobcat RC1 ?14:54
dansmithas it might be that we need to change it so that it tests the right releases, which we can't do until after rc of course14:55
dansmithI would leave it voting personally14:55
bauzaseven for like  B>D ?14:55
dansmithyeah, I mean,14:56
dansmithare you thinking we'll run into issues with deprecations?14:56
dansmithunless it's something specifically in that test I think we'll be okay.. we've had it enabled for a while and never found any problems in nova (IIRC)14:56
bauzaswell,14:56
bauzasI'm rather thinking about for example when we remove a RPC compat code 14:57
dansmiththis isn't a live test, so I don't think that will matter14:58
bauzaslike for when removing https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L1130614:58
*** kopecmartin_ is now known as kopecmartin15:00
bauzasanyway, if you prefer to continue the job, we could be just making it to voting, and in case after C** RC1, if we see it has some problem, then we could make it non-voting15:00
bauzasI mean, make it voting after Bobcat RC1, and leave it voting even after C* RC1 unless we see a problem15:01
dansmithsure, I mean, during bobcat it doesn't *have* to be voting, but I think it would be a good idea15:02
dansmithat least we'll notice what we're breaking and we can make it n-v if that's what we need to proceed15:02
sean-k-mooneyas long as it moves to 22.04 form 20.04 so we can drop focal testing im ok with that15:04
sean-k-mooneyi think we need that change to bump our min libvirt version which we skipped doing now for a few releases15:05
sean-k-mooneyso we should do it in bobcat and do it early15:05
stephenfinlowercase: I assume you mean no rows? There should be columns. The question is if they're the columns we're expecting, i.e. `vm_state`15:06
dansmithack, I think we should be able to switch it to zed->bobcat and 22.04 as soon as bobcat opens.. gmann you okay with that?15:06
bauzasbobcat will open in 2 days :)15:08
lowercaseshow tables; -> build_requests is present. select * from build_requests ; -> Empty set (0.001 sec) show full columns from build_requests ; -> there are about 23 columns there from a quick count.15:09
dansmithbauzas: I was going to say it actually matters when the relevant projects have branches, but that's the grenade holdup.. since this is two back, we should be able to do it immediately, yeah15:09
dansmithbauzas: the only concern would be people using the job to merge things on their rc branches that need coverage.. but I think we're going to need a grenade-skip-level-previous (or something) job anyway to keep stable happy, so maybe this is the time to practice that15:10
bauzashmmm15:10
lowercasestephenfin: see previous response15:11
stephenfinlowercase: can you dump the output of 'describe build_requests;' to a pastebin?15:12
bauzasdansmith: have you seen my service version change ? We shouldn't longer need to use this https://github.com/openstack/grenade/blob/master/.zuul.yaml#L40115:12
stephenfinlowercase: and 'select * from alembic_version;'15:13
dansmithbauzas: something other than 875621?15:13
stephenfinlowercase: This is for the next environment, of course15:13
bauzasdansmith: https://review.opendev.org/c/openstack/nova/+/87493215:13
lowercasestephenfin: this is my qa cluster that is on xena, and is not being upgraded to yoga yet. d67eeaabee3615:13
dansmithbauzas: ah, yeah I had but I didn't connect the dots15:15
bauzasdansmith: so are you agreeing on the change ? (like, just continuing to support Yoga)15:15
stephenfinlowercase: Cool, so alembic_version looks correct. Now to ensure that the columns that migration b30f573d3377 removes are present there, as expected (the 'describe' command)15:15
bauzasdansmith: and just after Antelope RC1, bumping the service version to Antelope15:16
dansmithbauzas: yeah I mean, I think I'm okay with that.. the change doesn't really change anything (other than get ready for the future with the missing service versions) right?15:16
lowercasestephenfin: https://paste.centos.org/view/529a12ed15:17
dansmithbauzas: meaning I'm happy to loosen our requirements a cycle before we officially do15:17
bauzasdansmith: yeah the problem that I have is that I need to understand again every cycle how we can support SLURP 15:17
bauzasso I want to make sure we do this correctlyu15:17
stephenfinlowercase: Okay, they're all there as expected. I would expect the migrations to apply cleanly in that environment. So the question is how is that test environment created?15:19
lowercasestephenfin: the test and qa environments are created using openstack-ansible and I do a really good job ensuring it stays very close to our remaining environments 15:20
stephenfinlowercase: In that case, I suspect either (a) someone has changed something behind alembic's back or (b) alembic ran but failed to record the updated version in the alembic_version table. I think you already ruled out (a)?15:20
lowercaseuhh.. so i just dropped that table in test and it now errors with 1146, \"Table 'nova_api.build_requests' doesn't exist\")", "[SQL: ALTER TABLE build_requests DROP COLUMN vm_state]", "(Background on this error at: https://sqlalche.me/e/14/f405)"]} bahaha15:21
lowercasein test, this is the one that is going to yoga15:21
stephenfinlowercase: that's expected. You've modified the schema behind alembic's back. It (intentionally) doesn't do introspection. It expects the schema to be in a given state and makes changes based on that15:22
* bauzas runs errand for a sec15:22
stephenfinThe only thing that should be issuing CREATE, DROP or ALTER operations is alembic. Not the operator.15:23
lowercasestephenfin: that's a fair statement. its just funny that im going to need to reverse engineer the creation of a table just so the script can remove it15:23
stephenfinlowercase: It's not removing the table - it's modifying it15:23
lowercaseoh yep. okay i see that now. just dropping everything inside of it and emptying the schema15:24
stephenfinlowercase: Nah, it doesn't drop everything inside of it either. As the name would suggest, the build requests table contains build requests. A record is created in there when a build is requested. It's deleted once that build is completed (either success or failure)15:26
stephenfinThat's why it looks empty. If you watched the table while creating instances though, you'd see stuff appearing and disappearing.15:27
stephenfinhttps://github.com/openstack/nova/blob/master/nova/conductor/manager.py#L554-L56315:27
stephenfinlowercase: Also, this is the schema from a Zed-based DevStack deployment https://paste.opendev.org/show/bm3psmcu51ltEvP1sT9D/15:29
lowercasei just dumped the table out of qa cluster, and im using that to build the dev cluster to get it to pass15:32
lowercaseokay, that looks pretty good.. rerunning the code!15:33
lowercasestephenfin: "Error: (pymysql.err.OperationalError) (1091, \"Can't DROP INDEX `shadow_instance_extra_idx`; check that it exists\")", "[SQL: ", "DROP INDEX shadow_instance_extra_idx ON shadow_instance_extra]",15:40
lowercaseregredably, im getting swamped over here. I don't have time to look into this further. I'll take some time to look at this and get back to you.15:41
bauzasreminder : nova meeting in 9 mins here15:51
bauzasI'll also need to bail out quickly after the meeting15:51
bauzas#startmeeting nova16:02
opendevmeetMeeting started Tue Feb 28 16:02:00 2023 UTC and is due to finish in 60 minutes.  The chair is bauzas. Information about MeetBot at http://wiki.debian.org/MeetBot.16:02
opendevmeetUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.16:02
opendevmeetThe meeting name has been set to 'nova'16:02
bauzassorry, bit late here16:02
Ugglao/16:02
elodilleso/16:02
bauzas#link https://wiki.openstack.org/wiki/Meetings/Nova#Agenda_for_next_meeting16:02
bauzaslet's start16:02
bauzas#topic Bugs (stuck/critical) 16:02
dansmitho/16:02
bauzas#info No Critical bug16:02
bauzas#link https://bugs.launchpad.net/nova/+bugs?search=Search&field.status=New 17 new untriaged bugs (+1 since the last meeting)16:02
bauzasUggla: any bug you wanna raise ?16:03
gibio/16:03
* gibi might need to disappeare suddenly16:03
bauzasif not, let's skip16:03
bauzas#info Add yourself in the team bug roster if you want to help https://etherpad.opendev.org/p/nova-bug-triage-roster16:03
bauzassean-k-mooney: can you be the next bug baton owner ?16:03
Ugglabauzas, no nothing special16:03
bauzasack16:03
sean-k-mooneyi guess so16:04
bauzascool ta16:04
bauzas#info bug baton is being passed to sean-k-mooney16:04
bauzasmoving on so16:04
bauzas#topic Gate status 16:04
bauzas#link https://bugs.launchpad.net/nova/+bugs?field.tag=gate-failure Nova gate bugs 16:04
bauzas#link https://etherpad.opendev.org/p/nova-ci-failures16:04
bauzasI'll be honest, I didn't had a lot of time to look at the failures16:05
bauzashave you seen some CI failure ? 16:05
dansmithdefinitely improving16:05
bauzasindeed16:05
dansmiththe volume detach on cleanup failures have all but gone away except for the live mgiration tests,16:05
dansmithand I have a patch up for that right now16:05
dansmithboth of those are failures in cinder tests that have gone on for years without getting proper attention16:06
dansmithbut they've been spiking a lot lately, so hopefully this will really improve things16:06
bauzasok16:06
gibiglad to hear that16:06
* bauzas looks at https://zuul.openstack.org/builds?project=openstack%2Fnova&branch=master&result=FAILURE&skip=016:06
elodillesbtw, can we expect the same for stable branches with this fix as well? (less volume detach and cleanup failures)16:07
dansmithelodilles: the fixes were in tempest16:07
dansmithelodilles: so that's branchless and should apply to stable right?16:07
bauzasI don't see a lot of failures for the same job16:07
elodillesso it means that where the tempest is not pinned16:07
dansmithah, right16:07
elodilles(non-EM branches)16:07
elodillescool, thanks!16:07
bauzasok, let's then move on16:08
bauzas#link https://zuul.openstack.org/builds?project=openstack%2Fnova&project=openstack%2Fplacement&pipeline=periodic-weekly Nova&Placement periodic jobs status16:08
bauzasall greens16:08
bauzas#info Please look at the gate failures and file a bug report with the gate-failure tag.16:09
bauzas#info STOP DOING BLIND RECHECKS aka. 'recheck' https://docs.openstack.org/project-team-guide/testing.html#how-to-handle-test-failures16:09
bauzas#topic Release Planning 16:09
bauzas#link https://releases.openstack.org/antelope/schedule.html16:09
bauzas#info Antelope-rc1 is in 0.2 weeks16:09
bauzas(which is on Thursday)16:09
bauzas#link https://etherpad.opendev.org/p/nova-antelope-rc-potential16:09
elodilleseven the rc1 patches are generated! ;)16:09
elodilles* release patches16:10
bauzaselodilles cool but we need to update at least the nova one16:10
elodillesack, that is expected16:10
elodillesdon't forget to signal this (if not yet done)16:11
elodilleswith a -1 on it16:11
bauzasat least we need to merge first https://review.opendev.org/c/openstack/nova/+/874932 (min servion version), https://review.opendev.org/c/openstack/nova/+/873584 (logging revert) and https://review.opendev.org/c/openstack/nova/+/875380 (reno prelude)16:11
elodilles(and thanks in advance o:))16:11
bauzaselodilles: indeed, doing it now16:11
bauzasso16:12
bauzasabout https://review.opendev.org/c/openstack/nova/+/874932 I think we are OK with how to use rolling upgrades with SLURP and non-SLURP releases16:13
bauzassean-k-mooney had a concern with https://review.opendev.org/c/openstack/nova/+/875380 but we'll discuss it tomorrow16:14
bauzasanything people want to discuss with RC1 ?16:14
gibibauzas: will you add a reno for https://review.opendev.org/c/openstack/nova/+/87493216:15
gibibauzas: it now allows Antelope to support Yoga computes16:15
bauzasgibi: I provided a new phrase in the prelude16:15
gibiwhich is N-216:15
bauzasgibi: but like you said, we could also modify the rolling-upgrade doc 16:16
gibiyeah, my point is we either do a proper documented N-2 support, or we keep the N-1 check for now16:16
bauzasprelude patch is https://review.opendev.org/c/openstack/nova/+/87538016:16
gibibut right now the code patch allows N-2 while our doc says otherwises16:17
bauzasgibi: okay, then I'll add a new change before we create the RC116:17
gibiin the current form I cannot +2 https://review.opendev.org/c/openstack/nova/+/87493216:17
gibias it is inconsistent with our doc16:17
gibibut as I said I'm OK to move the doc to match with the code or move the code to match with the doc16:18
bauzasgibi: so you want me to modify https://docs.openstack.org/nova/latest/admin/upgrades.html#rolling-upgrade-process ?16:18
gibibauzas: I would like to see a consistent documentation about Antelope supports regarding old compute service versions16:18
gibiif we go with Antelop supporting Yoga computes then yes https://docs.openstack.org/nova/latest/admin/upgrades.html#rolling-upgrade-process needs a change16:19
gibi(or we just fall back to only support N-1 and modify https://review.opendev.org/c/openstack/nova/+/874932 to only support Zed computes in Antelope)16:19
bauzascool, then I'll just add a doc modification for it in https://docs.openstack.org/nova/latest/admin/upgrades.html#rolling-upgrade-process16:19
bauzasshit16:20
bauzasadd a doc modif in https://review.opendev.org/c/openstack/nova/+/87493216:20
gibiOK16:20
bauzasdo people prefer here to only support Zed instead of Yoga ? https://review.opendev.org/c/openstack/nova/+/87493216:20
bauzasdansmith: sean-k-mooney: ^16:21
dansmithfor bobcat or antelope?16:21
bauzasMHO is that I'd prefer to modify the rolling upgrade document and say yes, we can support Yoga computes for Antelope services16:21
dansmithas I think I've mentioned, for antelope I think it would be nice to have a best-effort trial run of N-2 support, so antelope would support yoga16:21
dansmithyeah that ^16:22
sean-k-mooneyya i think yoga makes sense16:22
bauzasdansmith: yeah, gibi asked either to modify the doc or not modify it but only support Zed computes with Antelope conductors16:22
bauzasI'd prefer the former16:22
dansmithyeah16:22
sean-k-mooneywith that said for bobcat technially it should be antelope but i assume dansmith would prefer zed16:22
dansmithsean-k-mooney: I'd prefer we just try to do N-2 always unless we come up with a blocking reason we can't yeah16:23
bauzasthat's another change https://review.opendev.org/c/openstack/nova/+/875621/ that needs to be merged *after* RC116:23
dansmithI think we'll be able to do it16:23
sean-k-mooneyright whihc is not strictly requried by the governance resolution but im ok to try that until it breaks16:23
bauzasbut looks like dansmith prefers to support Zed computes with Bobcat so meh16:23
gibithen modify the upgrade doc to state that in Antelope we support N-2 (Zed) computes as best effort16:23
dansmithsean-k-mooney: if nothing else, it'll highlight the thing(s) that prevent us from doing it, which are good for planning16:23
dansmithgibi: best effort is the right terminology yeah16:24
bauzasok, so let's try to make the grenade-skip-level job voting 16:24
dansmiths/make/keep/ right?16:24
bauzasit's n-v as I speak16:24
sean-k-mooneyvoting based on 22.04 wiht zed ->bobcat16:25
bauzasand only in check16:25
dansmithoh, I thought I looked16:25
bauzasonly in check pipeline16:25
dansmithokay, I see.. yeah "make" then16:25
bauzassean-k-mooney: cool then I can modify https://review.opendev.org/c/openstack/nova/+/875621/ to make grenade-skip-level *voting* 16:26
bauzas(and adding it to the gate pipeline)16:26
bauzasthat one would be merged *AFTER* 2023.1 RC1 to be clear, so it would be a 2023.2 change16:27
bauzasagreed ?16:27
sean-k-mooneyyes16:27
bauzasok, and about your point with focal, sure16:27
bauzaswe need to test the N-2 rolling upgrade with ubuntu 22.04 only, right?16:28
sean-k-mooneyyes16:28
bauzascool16:28
sean-k-mooneybecause i want us to bump our min libvirt/qemu16:28
bauzaslooks like we have consensus16:28
dansmithI think we can only switch that once we convert it to start from zed, which it doesn't currently do16:28
* bauzas can't remember when we supported Jellyfish 16:28
sean-k-mooneyand i htink the  ones we chose last were released in 20.1016:29
sean-k-mooneyyes it will need to wait for changing to zed16:29
bauzaswhat's the TC supported release for Yoga ?16:29
sean-k-mooneywhich might need a grenade change16:29
dansmithsean-k-mooney: it will yeah16:29
sean-k-mooneyhttps://github.com/openstack/governance/blob/master/reference/runtimes/yoga.rst16:29
sean-k-mooney20.0416:29
sean-k-mooneyits technically the same for zed 20.0416:30
sean-k-mooneybut canoncial released zed on 22.0416:30
bauzasthat's what I was looking for16:30
dansmithI need to talk to gmann first but I will propose the skip-level job change16:30
bauzasso16:30
sean-k-mooneydevstack works fin with zed on 22.04 the last time i tried it so i expect it to work16:31
bauzasare we saying that we would use focal or jammy for the N-2>N skip-level job ?16:31
sean-k-mooneyfocal for now jammy when we change to zed16:31
dansmithjammy once we upgrade to zed->bobcat16:31
bauzascool16:31
bauzasI agree then16:31
bauzas#action bauzas to make our grenade-skip-level job voting after antelope rc1 on focal16:32
bauzasI guess we're done with that topic16:32
bauzaslast point tho16:32
bauzas#info who's interested in running for being a release liaisonĀ ?16:32
bauzaswe discussed it last week16:32
bauzasas a reminder, https://docs.openstack.org/project-team-guide/release-management.html#release-liaisons16:33
bauzastl;dr: look at Gerrit to see whether the releases team created a patch for reviews16:33
bauzasand say yay or nay on that patch16:33
bauzasthat job is basically backed by a French person that knows it for a while16:34
bauzasbut I'd prefer not to be alone in order to avoid elodilles to hassle me with pings16:34
bauzas:p16:34
elodillesnote: release liaisons are added as reviewers automatically (usually) to related release patches16:35
bauzason a side note, I have a non-urgent patch for cleaning up the PTL guide https://review.opendev.org/c/openstack/nova/+/87573016:35
sean-k-mooneyi have done it now for 2-3 releases  i can try and find time when pinged to look but if other are intereested let us know16:35
bauzasthat may help this person to know what to look and when16:35
bauzassean-k-mooney: we can have multiple liaisons16:36
sean-k-mooneyyep16:36
bauzasI very briefly discuss this with Uggla and he was showing a very slight interest in that16:36
bauzasdiscussed*16:36
bauzasbut maybe he left the meeting now, he had another appointment16:37
sean-k-mooneywhich is a good point it does not have to be a person form the core team nessisarly16:37
auniyalo/ I can !16:37
sean-k-mooneywe are not in rush16:37
bauzasauniyal: ack, noted.16:37
bauzasI'll discuss this with you later16:37
auniyalack16:38
bauzasthanks for the offer16:38
bauzasmoving on16:40
bauzas#topic vPTG Planning 16:40
bauzasthe usual now reminder16:40
bauzas#link https://www.eventbrite.com/e/project-teams-gathering-march-2023-tickets-483971570997 Register your free ticket16:40
bauzasand16:40
bauzas#link https://etherpad.opendev.org/p/nova-bobcat-ptg Draft PTG etherpad16:40
bauzasthe etherpad starts to grow16:40
bauzasI haven't yet been asked how many slots we gonna want for the vPTG16:41
bauzasbe sure I'll tell you once I'm reachend16:41
bauzasreached*16:41
bauzas#topic Review priorities 16:41
bauzas#link https://review.opendev.org/q/status:open+(project:openstack/nova+OR+project:openstack/placement+OR+project:openstack/os-traits+OR+project:openstack/os-resource-classes+OR+project:openstack/os-vif+OR+project:openstack/python-novaclient+OR+project:openstack/osc-placement)+(label:Review-Priority%252B1+OR+label:Review-Priority%252B2)16:41
bauzas#info As a reminder, cores eager to review changes can +1 to indicate their interest, +2 for committing to the review16:41
bauzas#topic Stable Branches 16:42
bauzaselodilles: your time16:42
elodillesthanks16:42
elodillesactually, nothing special to report,16:43
elodilles#info stable gates seem to be OK16:43
elodilles#info stable branch status / gate failures tracking etherpad: https://etherpad.opendev.org/p/nova-stable-branch-ci16:43
elodillesuse this if you see any new errors ^^^16:43
elodillesthat's all from me16:43
bauzasall good16:44
bauzasno news is good news16:44
bauzas#topic Open discussion 16:44
bauzasnothing on the agenda16:44
bauzasI guess we can call it a wrap ?16:45
auniyalo/16:45
auniyalThere are some bugs that are either fixed but not closed for some reason or do not have much information so people can work on them (IMO), which makes our backlog big.16:45
auniyal    Os-vif:16:45
auniyalhttps://bugs.launchpad.net/os-vif/+bug/165411716:45
auniyalhttps://bugs.launchpad.net/os-vif/+bug/167062816:45
auniyalhttps://bugs.launchpad.net/os-vif/+bug/170226216:45
auniyalfixed here: https://review.opendev.org/c/openstack/os-vif/+/479861/16:45
bauzasthat's a topic I added for the vPTG16:45
auniyalokay16:46
bauzasI'd like us to scrub our bug number by abandoning very old reports16:46
bauzasbut I'd prefer to discuss it at the PTG16:46
auniyalack bauzas 16:46
bauzasand yeah, some bug reports may not be automatically closed if the gerrit patch forgot to correctly write the commit msg tag16:46
bauzasso the launchpad sync is not automatically done16:47
bauzasauniyal: feel free to close the bug report with a comment16:48
bauzasthe status is then Fixed Released in LP16:48
bauzasany other questions ?16:48
bauzasI have to go errand in a sec16:48
bauzaslooks not16:49
bauzasif so16:49
bauzasthanks all16:49
bauzas#endmeeting16:50
opendevmeetMeeting ended Tue Feb 28 16:50:02 2023 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)16:50
opendevmeetMinutes:        https://meetings.opendev.org/meetings/nova/2023/nova.2023-02-28-16.02.html16:50
opendevmeetMinutes (text): https://meetings.opendev.org/meetings/nova/2023/nova.2023-02-28-16.02.txt16:50
opendevmeetLog:            https://meetings.opendev.org/meetings/nova/2023/nova.2023-02-28-16.02.log.html16:50
elodillesthanks o/16:50
* bauzas disappears16:50
stephenfinWeird one. Anyone have any idea why this doesn't work as expected? https://paste.opendev.org/show/bpzDcEI75QIIb9QuGFb8/17:21
stephenfinIf I run that, it prints out "Exception: Method 'baz' is not allowed" instead of "Exception: Method 'foo' is not allowed" as expected (baz instead of foo)17:21
stephenfinIt's a reproducer for bug in Keystone tests. I've fixed the tests but I can't yet grok why it was happening.17:22
gmanndansmith: sean-k-mooney bauzas : on skip-level job. as 2023.2 is NON_SLURP so we will not run skip-level job right17:31
gmannin 2023.3 cycle we will run it from 2023.1 to 2023.3 upgrade on jammy17:31
dansmithgmann: I want nova to stick to testing N-2->N even in non-slurp releases17:31
dansmithgmann: perhaps I should just clone grenade-skip-level to grenade-skip-level-continuous and update that to be z->b ?17:32
dansmithor z->master17:32
gmanndansmith: ohk17:32
gmannso that will be on focal right as zed is tested on focal as testing runtime17:32
gmannzed (focal) -> 2023.2 (focal ?)17:33
gmannbut 2023.2 on focal is not tested 17:33
dansmithI thought zed was both?17:33
dansmithah, I see, it wasn't.. dang17:33
gmann2023.1 was both, let me check17:33
dansmithyeah, zed was just 20.0417:34
dansmithwell, I can try it and if it doesn't work I guess we can pung17:34
gmannhttps://governance.openstack.org/tc/reference/runtimes/zed.html17:34
dansmith*punt17:34
gmannyeah17:34
dansmithbecause sean-k-mooney wants to bump the minimum libvirt version17:34
gmannas 2023.1 starting was tested ion jammy so zed might work17:34
gmannohk17:34
gmannin 2023.2 right? or bump in 2023.1 ?17:35
dansmithsean-k-mooney: wants to bump in 2023.2, but presumably to still include what would be needed  for 2023.117:35
dansmithi.e. jammy17:36
gmannbut is not that will be covered in normal grenade job from 2023.1 to 2023.2 ? or we want N-2 thing also due to libvirt bump ? and so does zed on jammy just because of grenade testing but main purpose is to check zed->2023.2 libvirt bump work fine or not?17:39
dansmithnot to test the libvirt bump specifically17:40
dansmithbut rather to keep us with a working N-2->N even though we don't *have* to support it17:40
dansmithif we end up breaking something and have to drop the job, then fine, but I would rather highlight what breaks N-2 upgrades if we can17:41
gmannok17:41
gmanni think it should work as zed was the edge when we move to jammy 17:42
dansmithyeah, will see17:42
opendevreviewDan Smith proposed openstack/nova master: Add continuous skip-level job for nova  https://review.opendev.org/c/openstack/nova/+/87577317:43
gmannso for that I will not filter out the 2023.2 from here (we filter out NON-SLURP  here) which I do while release time https://github.com/openstack/grenade/blob/master/.zuul.yaml#L39217:43
opendevreviewDan Smith proposed openstack/nova master: Add continuous skip-level job for nova  https://review.opendev.org/c/openstack/nova/+/87577317:44
stephenfinGot it. It's late binding https://stackoverflow.com/q/3431676/ Sigh, TIL17:44
dansmithgmann: oh hmm, that's just the thing that prevents it from running on stable right?17:45
gmannyeah17:45
gmannbecause it is in integrated gate template also17:45
dansmithyeah, so that's okay for now right? we only care about master17:45
gmannyes, we can adjust the integrated template not to run it for other projets does not want to run and keep generic job able-to-run-everywhere17:47
gmannand that is good so that project can control if they want to run or skip instead of we stop it for everyone17:48
dansmithgmann: well, I just inherited it for nova ^17:48
dansmithif other projects want to do the same, we can add it to grenade and not the integrated jobs, but if not, we might as well keep it in nova for now right?17:48
gmanndansmith: ohk it need to run on jammy otherwise default one will run on focal17:49
dansmithright17:49
gmanndansmith: in that case let me filter out the base job for 2023.2 not  to run so that it does not run by default as part of integrated gate and you can override 'branches' variant in nova inherited job to run on 2023.2 (master)17:51
dansmithgmann: yeah I assumed that was what you would do once 2023.2 opens right?17:53
gmanndansmith: yeah17:53
dansmithgmann: ubuntu-jammy is not the right nodeset?17:55
opendevreviewDan Smith proposed openstack/nova master: Add continuous skip-level job for nova  https://review.opendev.org/c/openstack/nova/+/87577317:56
gmanndansmith: openstack-single-node-jammy in devstck set the controller things more than just ubuntu-jammy so for devstck based job openstack-single-node-jammy will work17:56
dansmithokay17:57
gmannhttps://github.com/openstack/devstack/blob/master/.zuul.yaml#L1117:57
dansmithI copied from tempest-tox-plugin-sanity-check which I guess is not a devstack job17:57
* dansmith just grep'd for jammy17:58
opendevreviewSylvain Bauza proposed openstack/nova master: DNM (yet) Update min support for Bobcat  https://review.opendev.org/c/openstack/nova/+/87562117:58
bauzasdansmith: sean-k-mooney: just updated the next-after-rc1 patch for make grenade-skip-level voting on both check and gate pipes17:58
opendevreviewSylvain Bauza proposed openstack/nova master: Add service version for Antelope  https://review.opendev.org/c/openstack/nova/+/87493218:04
opendevreviewSylvain Bauza proposed openstack/nova master: DNM (yet) Update min support for Bobcat  https://review.opendev.org/c/openstack/nova/+/87562118:04
bauzassean-k-mooney: dansmith: and did the cleanups for the antelope patch18:04
sean-k-mooney"Nova does now currently support the18:49
sean-k-mooney    coexistence of N and N+2 or greater18:49
sean-k-mooney" 18:50
sean-k-mooneythat a little clunky18:50
sean-k-mooney"""     As of OpenStack 2023.1 (Antelope), Nova enables the18:52
sean-k-mooney    coexistence of N and N-2 (zed)18:52
sean-k-mooney"""18:52
sean-k-mooneyill comment that inline but this is how i woudl write that.18:52
sean-k-mooneyhowever while we are treating this as a dress rehersal im not sure i agree with avocating this.18:53
sean-k-mooneywe expect it to work18:53
sean-k-mooneybut i dont know if we want to comit to fixing issues with yoga to antelope18:53
sean-k-mooneyi think we should consider it experimental rather then something you should use in production in general18:54
dansmithI haven't reviewed yet, but agree, we should make it clear that it's a best-effort thing, but I'd also be in favor of just explaining that we don't *fail* on N-2, but not go so far as saying we support it or that it's a good idea18:54
sean-k-mooneybauzas: comments inline https://review.opendev.org/c/openstack/nova/+/874932/4/doc/source/admin/upgrades.rst18:57
opendevreviewMerged openstack/nova master: Fix logging in MemEncryption-related checks  https://review.opendev.org/c/openstack/nova/+/87338820:28

Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!