15:00:39 <TheJulia> #startmeeting ironic
15:00:40 <TheJulia> o/
15:00:40 <openstack> Meeting started Mon Apr 27 15:00:39 2020 UTC and is due to finish in 60 minutes.  The chair is TheJulia. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:00:42 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
15:00:43 <dtantsur> o/
15:00:44 <openstack> The meeting name has been set to 'ironic'
15:00:47 <cdearborn> o/
15:00:48 <erbarr> o/
15:00:49 <arne_wiebalck> dtantsur: yes, we should merge this one and fix the other issue
15:00:50 <rpittau> o/
15:00:52 <kaifeng_> o/
15:00:52 * iurygregory was too fast
15:00:52 <ajya> \o
15:00:54 <iurygregory> o/
15:00:59 <arne_wiebalck> o/
15:01:03 <rpioso> o/
15:01:11 <arne_wiebalck> dtantsur: I think TheJulia was on it as well
15:01:13 <rloo> o/
15:01:29 <stendulker> o/
15:01:36 <TheJulia> dtantsur: I can always revise arne_wiebalck's patch after the meeting
15:01:41 <TheJulia> Good morning everyone!
15:02:07 <TheJulia> Our agenda can be found on the wiki, as always.
15:02:09 <TheJulia> #link https://wiki.openstack.org/wiki/Meetings/Ironic#Agenda_for_next_meeting
15:02:17 <TheJulia> Hopefully this will go quickly :)
15:02:27 <mgoddard> \o
15:02:40 <TheJulia> Since I suspect we should skip status reports this week given it is basically release week
15:02:48 <TheJulia> well, last week was, but yeah. :)
15:03:09 <TheJulia> #topic Announcements / Reminders
15:03:58 <TheJulia> #info Final Ussuri release is due May 13th, we should expect devstack to possibly break us at any moment.
15:04:18 <TheJulia> It seems like we're on track to be done maybe today or tomorrow?
15:04:42 <TheJulia> Seems we also need to register our PTG attendance response, which I believe I already took care of.
15:05:00 <TheJulia> Does anyone else have anything to announce or remind us of?
15:05:26 <iurygregory> TheJulia, do we have the slots defined?
15:05:53 <iurygregory> or this can be done after 28?
15:05:59 <TheJulia> iurygregory: we do not. I'm trying to remember if we got an etherpad going
15:06:09 <TheJulia> That can be done after the 28th as I understand it
15:06:14 <dtantsur> #link https://etherpad.opendev.org/p/Ironic-VictoriaPTG-Planning
15:06:18 <dtantsur> this ^^?
15:06:30 <iurygregory> http://lists.openstack.org/pipermail/openstack-discuss/2020-April/014423.html
15:06:38 <iurygregory> dtantsur, yup
15:06:58 <TheJulia> That works!
15:07:43 <dtantsur> oh, one more thing
15:08:14 <dtantsur> If you remember my flame war email about making ironic more independent, one of the follow-ups was to understand what exactly we need in terms of release cadence, etc
15:08:35 <dtantsur> I'd like to set up a call, ideally this week, to gather ideas and try to come up with a proposal
15:08:46 <iurygregory> ++ sounds good
15:08:50 <TheJulia> ++
15:09:05 <dtantsur> #link https://doodle.com/poll/gtxzdhatabx8agy6 The doodle for release cadence discussion
15:09:18 <TheJulia> dtantsur: thanks!
15:09:27 <dtantsur> A small word of caution: this is not mandatory to attend. If you don't care much, you're welcome to skip it
15:09:43 <TheJulia> ++
15:10:01 <TheJulia> Looks like we didn't have any action items from our last meeting, except maybe SPUC II :)
15:10:19 <iurygregory> it's done ^
15:10:33 <iurygregory> http://lists.openstack.org/pipermail/openstack-discuss/2020-April/014410.html
15:10:43 <TheJulia> iurygregory: Is there an agreed upon date/time??
15:10:50 <TheJulia> #link http://lists.openstack.org/pipermail/openstack-discuss/2020-April/014410.html
15:11:21 <iurygregory> TheJulia, so far only 5 people voted
15:11:30 <TheJulia> Indeed :(
15:11:40 <TheJulia> #link https://doodle.com/poll/2q5zmv3g6uy2475e
15:11:57 <iurygregory> would be May 8 so far =) since we have holiday this week...
15:12:13 <dtantsur> yeah, May 1st is a holiday for a lot of people
15:12:16 <TheJulia> Anyway, since we are basically at the release window, I guess we should skip ahead to priorities for the week?
15:12:29 <dtantsur> ++
15:12:53 <TheJulia> #topic Deciding on priorities for the coming week
15:13:02 <TheJulia> #link https://etherpad.openstack.org/p/IronicWhiteBoard
15:13:20 <TheJulia> Starting at line 131
15:14:29 <iurygregory> can we add  https://review.opendev.org/#/c/703098/ ? =)
15:14:29 <patchbot> patch 703098 - ironic - Native zuulv3 grenade job for ironic - 15 patch sets
15:14:40 <dtantsur> CI changes are backportable
15:14:42 <dtantsur> JFYI
15:15:01 <iurygregory> gotcha
15:15:09 <TheJulia> I'd prefer to hold off until after we cut the branch and then just backport them
15:15:34 <iurygregory> makes sense to me
15:16:02 <TheJulia> anyone have the links handy for the stuff we're waiting on to merge?
15:17:59 <iurygregory> nope =(
15:18:24 <dtantsur> I'm only aware of the patches that has already been added
15:18:42 <TheJulia> Okay, then we're in really good shape, ci permitting
15:18:53 <iurygregory> yup
15:19:03 * TheJulia considers making an awful joke, but decides that it is a bad idea
15:19:16 * dtantsur considers which sacrifice to make to the CI gods
15:19:23 <rpittau> all the ci fixes are in the gate, I *think*
15:19:26 <TheJulia> dtantsur: rubber chickens I believe
15:19:29 <dtantsur> ++
15:19:33 <TheJulia> Does anyone have things for after the stable branch is cut?
15:19:39 <rpittau> I'll recheck a bunch of stuff as soon as those are merged
15:19:51 <dtantsur> as to awful jokes, that's what we have twitter for, no?
15:20:02 <TheJulia> dtantsur: true
15:20:21 <TheJulia> rpioso: Excellent! Thanks!
15:20:27 <dtantsur> https://review.opendev.org/703778 is for post-branch mering
15:20:27 <patchbot> patch 703778 - ironic - Add RPC objects for deployment API - 10 patch sets
15:20:49 <iurygregory> if we have patches that have changes in l-c they are ok to backport ?
15:21:03 <TheJulia> l-c?
15:21:09 <iurygregory> lower-constraints
15:21:12 <iurygregory> =)
15:21:15 <dtantsur> iurygregory: l-c alone or also requirements?
15:21:35 <dtantsur> lower-constraints.txt is not, to my best knowledge, a production artefact of ours
15:21:47 <dtantsur> changing requirements.txt is a whole different story
15:21:52 <iurygregory> well some required changes in requirements
15:22:06 <iurygregory> e.g https://review.opendev.org/#/c/720790/4
15:22:06 <patchbot> patch 720790 - ironic - Stop configuring install_command in tox. - 4 patch sets
15:22:18 <dtantsur> changing in requirements are only acceptable when we know for sure that the current lower bounds don't work at all
15:22:21 <dtantsur> * changes
15:22:24 <dtantsur> ** on stable branches
15:22:48 <iurygregory> I only changed because the job said it was wrong XD
15:23:32 <TheJulia> So, I suspect we should merge that prior to releasing ironic. We've already released basically everything else except sushy-tools/virtualbmc afaik
15:23:35 <dtantsur> I've approved teh change
15:23:57 <TheJulia> so if the gods of CI are friendly, ~4 hours
15:24:02 * TheJulia can live with that
15:24:03 <dtantsur> It would be a contentious change to backport
15:24:09 <TheJulia> Yeah
15:24:19 <dtantsur> TheJulia: with release mapping already in place, we can probably survive devstack and grenade branching
15:24:37 <TheJulia> dtantsur: if our branch is not there, we won't
15:24:52 <TheJulia> grenade will break once the mappings are changed
15:24:55 <dtantsur> I *think* it works nowadays, although in a weird fashion
15:25:01 <dtantsur> grenade falls back to master on unknown branches
15:25:08 <TheJulia> oh! then that is good!
15:25:09 <dtantsur> so it will be master->master testing
15:25:15 <TheJulia> okay
15:25:20 <iurygregory> yeah
15:25:20 <dtantsur> which is bad, but fine if only for a brief momemnt
15:25:21 <TheJulia> I'm glad that has been changed
15:25:29 <dtantsur> TheJulia: I *think* it has been the case for a while
15:25:30 <TheJulia> Is there anything else for the release?
15:25:37 <dtantsur> what used to break us is missing release mapping on our side
15:25:54 <dtantsur> I may be terribly wrong, of course, consult your lawyer before trusting me
15:26:01 <TheJulia> dtantsur: eh, I guess it was ~3 cycles ago that we had such issues last
15:26:13 <TheJulia> I remember two cycles ago, we were early and were like "why has this not been changed yet?"
15:26:16 <TheJulia> Anyway!
15:26:32 * dtantsur also suspects the release team will wait a reasonable amount of time
15:26:33 <TheJulia> I think we're good to move on
15:26:36 <dtantsur> ++
15:26:56 <iurygregory> ++
15:27:00 <TheJulia> We seem to have no discussion topics, so we could stop by Baremetal SIG
15:27:03 <TheJulia> arne_wiebalck: ^^^
15:27:12 <TheJulia> or we could just go directly to Open Discussion
15:27:16 <rloo> i believe dtantsur is correct wrt grenade
15:27:17 <Qianbiao> Hello, i got a question. If i submit "raid configuration support" patch after unsuri released, may it be cherry-picked to stable/unsuri ?
15:27:18 <arne_wiebalck> nothing to see here either
15:27:29 <dtantsur> Qianbiao: features are generally not backported, sorry
15:27:50 <TheJulia> Pass go, collect two code reviews and a get out of discussion free card! (Making a monopoly game reference)
15:28:08 <iurygregory> nice reference
15:28:21 <TheJulia> Qianbiao: We can make an early Victoria cycle release once it merges
15:28:34 <Qianbiao> dtantsur any chance that i commit it now?
15:28:35 <TheJulia> It is not unheard of us to do such. :)
15:28:39 <Qianbiao> TheJulia ok.
15:29:12 <openstackgerrit> Merged openstack/python-ironic-inspector-client stable/ussuri: Document the new standalone CLI  https://review.opendev.org/723575
15:29:35 <TheJulia> Well, with arne_wiebalck having chimed in, lets go to Open Discussion
15:29:38 <TheJulia> #topic Open Discussion
15:29:52 <TheJulia> Hopefully everyone had a wonderful weekend!
15:30:05 <dtantsur> They become shorter and shorter, these weekends
15:30:08 <dtantsur> has anyone else noticed?
15:30:37 <iurygregory> nope .-.
15:30:45 <rpittau> dtantsur: kind of, yeah :/
15:31:10 <TheJulia> They have :(
15:32:30 <kaifeng_> anyone noticed the weird issue in the ML wrt the NCs and ICs?
15:32:58 <TheJulia> kaifeng_: NCs? ICs? I'm guessing ML is mailing list?
15:32:59 <dtantsur> I'm not even sure what NC and IC are..
15:33:34 <kaifeng_> heh, actually i am not the inventor, NC = nova compute, IC = ironic conductor
15:34:19 <TheJulia> I've not seen anything, but pulling up the ml now
15:34:29 * dtantsur used to n-cpu and ir-cond
15:34:50 <kaifeng_> http://lists.openstack.org/pipermail/openstack-discuss/2020-April/014337.html
15:35:49 <dtantsur> I wonder why it hasn't landed in my inbox..
15:35:50 <kaifeng_> what makes me confused in the problem is an image is downloaded in one conductor host while pxe-ed from another conductor host
15:36:33 <dtantsur> take over takes some time
15:36:47 <kaifeng_> i am not sure how that can happen, except there is a take over, but this is not mentioned in the text
15:36:48 <arne_wiebalck> or does not work at all
15:36:56 <dtantsur> lol
15:37:01 <dtantsur> this is.. concerning
15:37:03 <iurygregory> lol
15:37:08 <rpittau> oO
15:37:10 <iurygregory> would be a problem
15:37:20 <dtantsur> can we put CI testing of take over on our backlog please?
15:37:30 <dtantsur> I recall fixing a lot of bugs with it at some point
15:37:30 <iurygregory> ++
15:37:59 <arne_wiebalck> I meant take over in Ironic and Nova, since he was changing both.
15:38:04 <TheJulia> yeah, we've had to fix a number of things. I suspect we're missing some context.
15:38:08 <iurygregory> ouch
15:38:11 <iurygregory> >.< nova
15:38:12 <arne_wiebalck> We stopped doing this.
15:38:33 <dtantsur> I don't think the nova's take over can affect the deploy process
15:38:49 <dtantsur> it's only about resource tracker updates, everything else is re-balanced when going through ir-api
15:39:54 <kaifeng_> because we will utilize mutiple n-cpu and ir-cond in production, this issue seems suspicious
15:40:13 <arne_wiebalck> if multiple compute nodes claim an overlapping set of nodes, there will be a fine mess
15:40:19 <TheJulia> I think they may be expecting the hash rings in both ironic and nova to be calculated exactly the same way
15:40:40 <dtantsur> it's a fair (and wrong) expectation
15:40:57 <TheJulia> indeed
15:40:58 <dtantsur> but I don't see how a discrepancy between the two hash rings could cause a node to be handled by two conductors
15:41:18 <dtantsur> but if they try to deploy while take over is in progress..
15:41:27 <dtantsur> I don't think anybody has tried that. All bets are off.
15:41:39 <TheJulia> this is true. The two minutes in between the actions could be a lot of activity
15:42:41 <arne_wiebalck> I don't think it is the interaction between the two hash rings, it is (or was) more a bug in the implementation in one of them.
15:43:13 <arne_wiebalck> FWIU, the given nova-compute did not let go :)
15:43:18 <arne_wiebalck> giving
15:43:45 <TheJulia> I think we're going to need more in the way of logs to really understand what is occuring
15:44:16 <TheJulia> it feels like there is more than one issue :\
15:44:58 <kaifeng_> indeed, just the "unstable" scares
15:45:00 <arne_wiebalck> "simple" take over works fine, we have added and removed Ironic controllers several times
15:45:32 <openstackgerrit> Christopher Dearborn proposed openstack/ironic master: Fix supported sushy-oem-idrac version  https://review.opendev.org/723588
15:45:34 <arne_wiebalck> "simple" == I haven't tried to do that while mass-instantiating nodes
15:45:40 <TheJulia> I guess n-cpu and ir-cond is running on the same machine too
15:46:17 <TheJulia> I guess the question becomes can ir-cond be scaled first, and then api/n-cpu
15:46:43 <arne_wiebalck> we have these on different nodes
15:46:47 * TheJulia wonders if this is also the latest stable queens or an older stable/queens release
15:46:59 <TheJulia> I seem to remember there were some queens timeframe fixes
15:47:07 <dtantsur> not unlikely
15:48:00 <dtantsur> oh, https://review.opendev.org/#/c/723588/ should ideally merge before branching
15:48:01 <patchbot> patch 723588 - ironic - Fix supported sushy-oem-idrac version - 1 patch set
15:48:07 * dtantsur has just noticed
15:48:22 <rpittau> ^ I was going to say :)
15:49:04 <dtantsur> 3 minutes to approval, I bet the fasted cdearborn has observed :D
15:49:07 <rpittau> this has been rebased also https://review.opendev.org/717689
15:49:07 <patchbot> patch 717689 - ironic-prometheus-exporter - Use hacking for pep8 tests - 7 patch sets
15:49:35 * cdearborn almost fell out of my chair!
15:49:54 * rpioso did :-)
15:50:27 <dtantsur> :D
15:50:37 <dtantsur> iurygregory: https://review.opendev.org/#/c/720793/ seemingly needs a rebase?
15:50:37 <patchbot> patch 720793 - ironic-prometheus-exporter - Stop configuring install_command in tox. - 4 patch sets
15:50:45 <iurygregory> dtantsur, yeah just noticed
15:50:53 <TheJulia> ugh. \o/
15:51:09 <iurygregory> I thought rpittau had everything done so I used his patch as base XD
15:51:23 <TheJulia> Does anyone want to respond to the hash ring thread on the mailing list?
15:51:48 <dtantsur> I responded with a couple of comments just to revive it
15:51:56 <TheJulia> okay
15:51:57 <dtantsur> we may need to ask for logs, not sure how doable it is
15:52:11 <TheJulia> Yeah :\
15:52:40 <TheJulia> is there anything else to discuss today?
15:53:31 <rpittau> not strictly related to release, but TheJulia when you have a minute I think this can be merged now https://review.opendev.org/708827
15:53:32 <patchbot> patch 708827 - ironic-python-agent (stable/train) - Rescan after restarting the md device - 2 patch sets
15:54:02 <dtantsur> re that release cadence doodle: please vote by ~ noon UTC tomorrow, as the most likely candidate now is on Wednesday
15:54:42 <TheJulia> rpittau: thanks
15:55:45 <TheJulia> dtantsur: Thanks!
15:55:48 <TheJulia> Thanks everyone!
15:56:00 <dtantsur> thank you TheJulia
15:56:07 <rpittau> thank you TheJulia :)
15:56:55 <TheJulia> Have a wonderful week everyone!
15:57:01 <TheJulia> #endmeeting