15:00:17 <TheJulia> #startmeeting ironic
15:00:18 <openstack> Meeting started Mon Dec  7 15:00:17 2020 UTC and is due to finish in 60 minutes.  The chair is TheJulia. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:00:19 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
15:00:19 <TheJulia> o/
15:00:21 <openstack> The meeting name has been set to 'ironic'
15:00:21 <iurygregory> o/
15:00:24 <dtantsur> o/
15:00:25 <TheJulia> Good morning everyone!
15:00:26 <rpioso> \o
15:00:27 <stendulker> o/
15:00:33 <bdodd> o/
15:00:38 <kaifeng> o/
15:00:46 <TheJulia> It is time for our weekly meeting of baremetal cloud irony!
15:00:53 <rloo> o/
15:01:00 <TheJulia> Our agenda can be found on the wiki today.
15:01:03 <TheJulia> #link https://wiki.openstack.org/wiki/Meetings/Ironic#Agenda_for_next_meeting
15:01:11 <TheJulia> Looksl ike only one discussion item, so maybe today will go quick?!?
15:01:23 <TheJulia> #topic Announcements / Reminders
15:02:13 <rpittau> o/
15:02:33 <TheJulia> #info We are approaching end of year. Numerous contributors are expected to be out the last half of December. Please remember to provide review feedback on sprint-1 and early sprint 2 items if applicable and present. If an existing item won't make it, now would be a good time to signal so.
15:03:02 <dtantsur> we have releases due this week, right?
15:03:09 <TheJulia> umm... good question!
15:03:11 <TheJulia> checking
15:03:16 <bfournie> o/
15:03:22 <JayF> o/
15:03:34 <TheJulia> dtantsur: next week
15:03:42 <ajya> o/
15:03:56 <dtantsur> okay, cool. let's try to do it rather earlier than later
15:04:18 <TheJulia> ++
15:04:48 <rpittau> I can help with the releases if needed
15:04:54 <TheJulia> I can always put in the actual change for the releases team later in the week, but items need reviews.
15:05:15 <dtantsur> I'll be here next week, but only Mon-Wed (same the week after that)
15:06:18 <TheJulia> k
15:06:30 <TheJulia> Anyway, shall we proceed?
15:06:46 <rpittau> Let's
15:06:58 <dtantsur> yep
15:07:22 <TheJulia> #topic Review action items from the previous meeting
15:09:24 <TheJulia> We had one action item which was me to sync with the stable team regarding skipping backports on branches in the "extended maintenance phase". Basically they feel it is a stable policy violation, and then discussion kind of devolved because their view switched to the community as a whole and not as a project or as a contributor. The resulting consensus seemed to be we need to do what is reasonable for the
15:09:24 <TheJulia> situation that exists at that time.
15:09:51 <TheJulia> rloo: ^^
15:09:58 <TheJulia> Moving on!
15:10:07 <TheJulia> #topic Review subteam status reports
15:10:19 <TheJulia> #link https://etherpad.opendev.org/p/IronicWhiteBoard
15:10:25 <dtantsur> This makes EM quite costy..
15:10:35 <rloo> thx TheJulia!
15:10:47 <rloo> i'll update the etherpad...
15:10:48 <TheJulia> dtantsur: They don't seem to care and we're not really interested in discussing further, I pointed that out and they just let it drop
15:11:02 <TheJulia> So I think our EM path is to kind of just do whatever is reasonable to ourselves.
15:11:17 <TheJulia> Which likely need to include starting to drop extensive testing on older branches
15:11:26 <kaifeng> i wondner if other projects backporting patches to the EM releases
15:11:54 <rloo> (I think there ought to be some community ... understanding... about this but ...)
15:12:00 <TheJulia> kaifeng: nova is doing skip patching, so master -> appropriate current stables and then jumping to the queens em that they care about
15:12:21 <TheJulia> rloo: the Problem is the community, they fallback on the community in the past said they don't want to do this
15:12:46 <TheJulia> so their stance was frozen in time and not considering the realities of those that actually have to get stuff in to EM.
15:12:54 <TheJulia> Anyway!
15:13:01 <TheJulia> #link https://etherpad.opendev.org/p/IronicWhiteBoard
15:13:06 <TheJulia> Line 267
15:13:22 <rloo> but if some projects are doing it, i think that ought to be communicated to the community, vs separate projects doing things their own way. having said that, i don't really want to get involved and don't want this to block/delay whatever.
15:13:47 <kaifeng> skip patching doesn't seem upgrade compatible th
15:13:54 <kaifeng> s/th/tbh/
15:14:27 <TheJulia> kaifeng: depends on the patch I guess, if it is not data model, then there is low risk as long as someone is going to a newer release version and not just going from say Queens -> Rocky
15:14:30 <rloo> kaifeng: good point. i think though, that the stuff that is backported, is to fix bugs. not add features. so it won't prevent upgrades, and if one upgrades enough, they'll get the bug fix.
15:15:14 <TheJulia> rloo: I think the same is true, they did not want to be in the middle of such a discussion either.
15:15:33 <TheJulia> dtantsur: rpittau: No luck on nvme secure erase?
15:15:36 <rloo> TheJulia: to confirm -- we're only talking about backporting patches as per the stable team's rules wrt backporting. we're just not going to backport to 'all' the interim releases.
15:16:05 <rloo> wrt rules, i mean wrt what qualifies to be backported.
15:16:06 <rpittau> TheJulia: nothing for now
15:16:12 <TheJulia> rloo: lets continue this in open discussion
15:16:17 <rloo> ok
15:16:41 <TheJulia> rpittau: Could you put a note saying that it is going to not make sprint1 and will likely make sprint 2?
15:16:42 <dtantsur> TheJulia: nothing from me, and I think janders will only get to it after the holidays
15:16:48 <TheJulia> k
15:17:00 <TheJulia> kaifeng: I see you have an approved spec \o/
15:17:03 <rpittau> TheJulia: will do
15:17:09 <TheJulia> w/r/t node history
15:17:30 <kaifeng> TheJulia: yeah, thanks everyone for the review!
15:18:04 <TheJulia> ajya, bdodd: reminder: redfish raid is presently listed for sprint 2. I see there is some discussion on the change. I'm curious if it can be updated at this point?
15:18:29 <TheJulia> iurygregory: same status on oslo.privsep?
15:18:47 <iurygregory> TheJulia, going to update things this week
15:18:49 <bdodd> I'm working on updates to the RAID patches
15:18:50 <iurygregory> I was on PTO
15:18:54 <TheJulia> \o/
15:18:58 <TheJulia> Okay, thanks
15:19:54 <TheJulia> zer0c00l: Looks like you have a little discussion on the anaconda patch, I can drop my +2 if you want for now. You'll likely want to consider revising maybe.
15:20:34 <TheJulia> Everything looks good to me, are we good to proceed to priorities for the week?
15:20:35 <rloo> ++ for revising
15:21:40 <TheJulia> vote changed
15:21:42 <TheJulia> or... changing
15:21:46 <JayF> I certainly want my reviews taken into account before it merges :D
15:21:51 * TheJulia needs a gerrit stop-watch
15:22:33 <TheJulia> #topic Deciding on priorities for the coming week
15:22:46 <TheJulia> #link https://etherpad.opendev.org/p/IronicWhiteBoard
15:22:46 <rloo> (I added a note wrt anaconda revision desired)
15:23:19 <TheJulia> Starting at line 120
15:23:32 <TheJulia> Looks like we got some stuff merged last week, so I'll clean that up. I added items aaround 198
15:25:22 * arne_wiebalck cannot join the meeting today, just briefly: SIG meeting tmrw with rpioso on Redfish interop profiles, NTR for the SIG otherwise, bye o/
15:26:26 <TheJulia> Thanks arne_wiebalck
15:26:32 <TheJulia> Any objections to the items to add?
15:27:34 <JayF> lgtm
15:27:59 <TheJulia> Okay, I;kk do the shuffling after the meeting
15:28:02 <TheJulia> err I'll
15:28:06 <TheJulia> Onward!
15:28:22 * TheJulia hears crickets
15:28:27 <TheJulia> #topic Discussion
15:28:54 <TheJulia> One quick discussion item, should we have a final meeting for the month next week?
15:29:13 <dtantsur> I'll be likely here for the next 2 Mondays
15:29:24 <dtantsur> won't be around on the 4th of January though
15:29:45 <TheJulia> I was about to suggest we skip the 21st, 28th, and Janurary 4th
15:29:54 <dtantsur> sounds reasonable to me
15:30:13 <TheJulia> Which is a long gap, but I think a number of us need the heads down time be it code or in our pillows
15:30:46 <rloo> ++ so dec 14 is last meeting of the year!
15:31:04 <TheJulia> Seems that way if there are no objections :)
15:31:07 <dtantsur> prepare your santa hats \o/
15:31:14 <TheJulia> Oh wait, are we doing that?!?
15:31:19 <TheJulia> eek!
15:31:21 <TheJulia> :)
15:31:29 <dtantsur> we also have two more SPUCs planned
15:31:32 <TheJulia> ++
15:31:35 <TheJulia> Spucs are good
15:31:36 * dtantsur had to miss last Friday, sorry for that
15:31:36 <iurygregory> I don't have santa hats =(
15:31:50 <TheJulia> I assume santa hats should now be on SPUCs
15:31:50 <rpittau> I have a wizard goofy hat, is that ok?
15:31:55 <TheJulia> maybe ;)
15:32:01 <iurygregory> rpittau, yes!
15:32:21 <TheJulia> Anyway, I'll send out the "end of year email" and we'll switch to etherpad updates as we have the last couple of end of year holiday seasons
15:32:33 <TheJulia> If there are any questions or concerns, please feel free to reach out to me.
15:32:47 <TheJulia> Anyway! Onward to the Baremetal SIG
15:32:50 <TheJulia> #topic Baremetal SIG
15:33:44 <TheJulia> #Info Baremetal SIG session covering Redfish Interop profiles - Tomorrow, December 8th, at 2PM UTC
15:33:55 <TheJulia> #link https://etherpad.opendev.org/p/bare-metal-sig
15:34:19 <TheJulia> arne_wiebalck: one reminder, record the talk portion so we can build some content for youtube :)
15:34:32 <rpioso> TheJulia: Ack
15:34:36 <TheJulia> Since we have no RFE's on the agenda, we'll go directly to Open Discussion
15:34:45 <TheJulia> #topic Open Discussion
15:35:31 <TheJulia> rloo: where were we?
15:35:55 <rloo> TheJulia: for myself, I think what I'd like is a clarification of this process, so everyone (in ironicland) knows.
15:36:00 <TheJulia> I think we just try and do the best we can, applicable to stable policy, and just start killing EM branch testing that doesn't make sense
15:36:08 <rloo> if i recall, i was fine with what we discussed at ptg
15:36:25 <rloo> what EM branch testing are we doing now?
15:36:26 <TheJulia> I can write a doc update then
15:36:33 <TheJulia> rloo: basically trying to keep everything alive
15:36:35 <TheJulia> which is insane
15:36:46 <rloo> right. i agree. i thought we had already killed some branch testing.
15:36:56 <TheJulia> Some, but not tons
15:37:25 <TheJulia> We can likely dial it back by a 50%
15:37:31 <JayF> Does OpenStack's branch model fit this concept? e.g. can we put "r" in unmaintained while "q" is in EM?
15:37:33 <TheJulia> s/a//
15:37:39 <rloo> there were 2 things? 1 kill some branch testing. 2. allow backports to skip some branches. eg if someone wants to backport to rocky but not stein. (we're talking only branches in EM, can't skip branches in M)
15:38:24 <TheJulia> JayF: EM is up to anyone wantint to put patches forth to burn the time to get the patch in, but forcing someone to go through a ton of EM branches and keeping them all perfectly alive with full testing, is a huge time sink
15:38:26 <rloo> and then maybe 3. is it ok to backport to an EM branch were there is no upstream testing.
15:38:57 <TheJulia> or limited testing, like we know some scenarios are more likely to fail than others
15:39:03 <JayF> TheJulia: I don't disagree in concept, I'm just wondering if there's a clear way to communicate that to users. OpenStack not having an "EM ... but more maintained than the other EM" branch type is unfortunate
15:39:05 <TheJulia> At least inCI
15:39:27 <dtantsur> it won't help to kill "some" branch testing, our jobs tend to break in large groups :)
15:39:37 <TheJulia> JayF: unfortunately it is up to those that care about an EM branch to carry that load
15:40:21 <rloo> dtantsur: what do you mean by that? keep all branch testing then? or kill all of them? :)
15:40:28 <TheJulia> dtantsur: well, huge breakages are generally easy to resolve once we identify the issue. The problem is noise and overall load crushing the ability for the job to actually pass
15:40:28 <JayF> I'm just thinking, we're going through a process right now to determine which release to upgrade to. As a contributor, I know to look for either Queens or a "M" release or newer. I don't think an external user would easily be able to determine that Queens is a better choice than Rocky, for instance
15:40:35 <dtantsur> I mean, it's not about removing one job, it's about removing most of them
15:40:48 <rloo> fwiw, stein is EM now.
15:40:54 <dtantsur> and I'll personally have reservations about +2'ing something that only passed unit tests..
15:41:16 <TheJulia> Yeah, I'd prefer to some tests remain functional
15:41:48 <dtantsur> I still want to see a model similar to Linux kernel and some other projects:
15:41:50 <TheJulia> just, not 10+ integration jobs
15:42:04 <dtantsur> we maintain any branch for a while. then whoever cares can provide support, everything else is killed off.
15:42:19 <TheJulia> dtantsur: that is kind of where we're headed I think
15:42:23 <rloo> dtantsur: are you including branches in Maint, or only EM?
15:42:25 <TheJulia> to be totally honest
15:42:37 <dtantsur> rloo: Maint is the "we maintain any branch" part. EM is "whoever cares" part.
15:42:39 <openstackgerrit> Verification of a change to openstack/bifrost failed: Support testing secure boot  https://review.opendev.org/c/openstack/bifrost/+/760791
15:42:52 <rloo> (M=maintenance, EM=extended maintenance)
15:43:16 <TheJulia> EM's naming is not the best, but I remember when that was a huge fight
15:43:20 <rloo> dtantsur: does upstream community/stable say anything wrt CI for M branches?
15:43:23 <JayF> dtantsur: TheJulia: ++ the linux kernel did this a couple of years back to solve this exact problem
15:43:39 <dtantsur> rloo: I don't recall any firm position. I think it's up for us as a project.
15:43:52 <rloo> shall we take EM first, that seems easiest?
15:43:53 <TheJulia> rloo: aiui, it is up to the projects.
15:44:05 <dtantsur> rloo: I think we're only talking about EM now
15:44:27 <dtantsur> the main maintenance is not long, and the stable policy requires us to provide a high level of maintenance for them
15:44:38 <TheJulia> I'd like to point out some of the major projects also only have 1-3 integrated scenario jobs on older branches. We've got many more
15:44:55 <dtantsur> one of the reasons I started removing the iscsi deploy :)
15:45:00 <TheJulia> ++
15:45:11 <dtantsur> but yeah. our problem is low concurrency of our tests
15:45:18 <dtantsur> and a relatively large test time
15:45:20 <TheJulia> And limited CI resources
15:45:28 <TheJulia> 8 GB of ram, no ability to touch swap
15:45:45 <dtantsur> if Nova can boot a VM with cirros in 30 seconds and 256M of RAM, they're in a much better position
15:45:47 <rloo> i'm totally good with 0 CI for EM (except unit tests)
15:45:57 <TheJulia> to quote an infra person, "Swap is the mind or CI job killer"
15:46:04 <dtantsur> rloo: would you +2 any patch under these conditions?
15:46:16 <rloo> i would if we agree to that 'policy'. it is a backport.
15:46:44 <JayF> I think patches exist where I'd be hesitant to +2 without any integration tests
15:46:55 <JayF> clean backsports of minor fixes? sure, lets forgo the integration tests
15:46:55 <dtantsur> same for me
15:46:58 <iurygregory> same
15:47:07 <rloo> hmm. what if said person tested downstream?
15:47:08 <JayF> but I've seen some backports that I'd want to see a deploy work for
15:47:25 <dtantsur> rloo: there are people who I'd trust in this case, there are people (most of them) who I'd not
15:47:28 <JayF> Then we go down a path of why isn't "trust me, I tested it" good enough everywhere :|
15:47:45 <rloo> i just don't know how reasonable it is, to expect upstream to provide CI, given the state of things upstream.
15:47:49 <dtantsur> right
15:48:20 <dtantsur> bifrost jobs can potentially have a longer lifetime now that we default to using virtual environments
15:48:24 <rloo> so if we don't +2, then an alternative is for the person/company to patch downstream. which is also fine with me (that's what we do at verizon media)
15:48:28 <dtantsur> because of their simplicity
15:49:24 <TheJulia> Pure downstream patching adds a lot of cost as well, but it all becomes what makes the most sense in the situation
15:49:27 <rloo> honestly, i haven't been contributing upstream. for the folks that have -- what is your preference?
15:49:56 <rloo> we have a limited number of people working upstream. i think they ought to have a bigger vote wrt where they want to put their time/energy.
15:49:56 <JayF> rloo: TBH I think our input, contributor or not, is interesting here because we're soon-to-be consumers, likely, of one of these EM branches
15:50:17 <TheJulia> It seems like the action is to kind of write down the overall feeling/perception, and be able to point to that whilst also possibly beginning to nuke some but not all integration jobs *where it makes sense*
15:50:29 <rloo> right, we aren't the only ones (I hope) that will be consumers of those EM. having said that though, things aren't 'free'.
15:50:30 <dtantsur> JayF: would you become a "patron" of an EM branch?
15:50:47 <JayF> I don't care so much what we do w/r/t deciding support. Putting resources where they matter (e.g. targetting queens and not other EMs) makes sense; I'm just concerned about upstream users (vs rh users) realizing that Queens is more maintained than other EMs
15:51:10 <dtantsur> How does kernel solve this messaging problem?
15:51:12 * TheJulia likes this idea dtantsur just raised
15:51:16 <JayF> dtantsur: ^ is my concern, that we communicate it well. I think OpenStack's branch system is almost designed to make this sorta "branch skipping" awkward and hard to communicate
15:51:39 <JayF> dtantsur: kernel specifically calls out longterm releases on kernel.org
15:51:45 <JayF> dtantsur: so you get a menu of maintained things to pick from
15:51:47 <dtantsur> Honestly, the lack of mutual understanding around EM branches is already problematic
15:51:53 <rloo> so... we introduce a new 'class' of branches. M -> EM, and EM can include .. sponsored M?
15:52:14 <dtantsur> Can we maybe pull our EM branches from releases.openstack.org (if they're anyhow mentioned there) and only provide this information in our docs?
15:52:36 <dtantsur> I'm not convinced this issue is solved by adding more concepts
15:52:47 <JayF> can we pull EM branches *except queens* from releases?
15:52:57 <TheJulia> The whole idea around EM is that it can "sponsored" or have a "patron" if the resources are supplied by the interested party/org. The thing is they will never ever be released again so patches are merged at the will of the project or the EM maintainers who care
15:52:59 <dtantsur> I think openstack as a whole should do a better jobs communicating that the exactly quality of EM branches is up to a project
15:53:10 <JayF> dtantsur: ++
15:53:55 <TheJulia> This may be worth of taking to the TC, in order for them to drive that visibility
15:54:14 <dtantsur> are you still on TC? ;)
15:54:16 <TheJulia> since, in reality, what is needed for that to be documented OR the EM branch model to be re-visited
15:54:33 <JayF> I think a fair summary is: 1) Nobody has objections to the idea of targetting your resources on a single EM branch and not spreading the love. 2) We need a good way for operators to know that Queens is more maintained than Rocky (and soon Stein), but OpenStack's release system doesn't make that easy
15:54:37 <TheJulia> No, Board until January when the new board is seated, that is if I'm not re-elected.
15:55:03 <TheJulia> JayF: that sounds right
15:55:27 <JayF> I'd suggest taking a summary of what you wanna do, call out that sticking point, and hit the ML
15:56:04 <JayF> This is the sorta thing we should have a longer form discussion about  ... but we can have that discussion while effectively implementing the new plan
15:56:59 <TheJulia> Agreed
15:57:34 <rloo> ++
15:58:35 <TheJulia> Okay, well, we're almost at time. Thanks everyone
15:58:45 <dtantsur> a heads-up
15:58:59 <dtantsur> CentOS 8.3 is out, watch out for related failures
15:59:05 <dtantsur> </heads-up>
15:59:06 <TheJulia> joy
15:59:18 <dtantsur> we have a minor breakage in openshift land
15:59:30 <TheJulia> #info Heads up - Centos 8.3 has been released, keep an eye out for failures.
15:59:39 <TheJulia> If anyplace explodes it will be metalsmith
15:59:45 <dtantsur> or bifrost. or IPA-builder.
15:59:47 <dtantsur> :)
15:59:51 <TheJulia> Yup
15:59:59 <TheJulia> Anywya, thanks everyone! Have a wonderful week!
16:00:16 <dtantsur> thank you!
16:00:31 <JayF> \o
16:01:26 <TheJulia> #endmeeting