Thursday, 2020-04-30

*** diablo_rojo has quit IRC00:08
*** tinwood has quit IRC02:10
*** tinwood has joined #openstack-release02:12
openstackgerritJulia Kreger proposed openstack/releases master: Create stable/ussuri for ironic
*** ykarel|away is now known as ykarel04:15
*** evrardjp has quit IRC04:35
*** evrardjp has joined #openstack-release04:36
*** vishalmanchanda has joined #openstack-release05:07
*** tetsuro has joined #openstack-release05:29
*** udesale has joined #openstack-release05:41
*** ysandeep|away is now known as ysandeep05:42
openstackgerritMerged openstack/releases master: Create stable/ussuri for adjutant
openstackgerritMerged openstack/releases master: Create stable/ussuri for adjutant-ui
openstackgerritMerged openstack/releases master: Release manila 8.1.2 off of stable/stein
openstackgerritMerged openstack/releases master: Release Horizon 15.3.0
openstackgerritMerged openstack/releases master: add some doc about process on R-2
*** e0ne has joined #openstack-release06:51
openstackgerritMerged openstack/reno master: Handle empty config files
openstackgerritMerged openstack/releases master: add doc related to R-2 and requirements update
openstackgerritMerged openstack/releases master: [doc][process R-2] clarifying our role
openstackgerritMerged openstack/releases master: add useful links for R-2 3rd step is doc
*** e0ne has quit IRC06:57
*** slaweq has joined #openstack-release06:58
*** e0ne has joined #openstack-release06:58
*** e0ne has quit IRC07:05
*** amoralej|off is now known as amoralej07:20
*** tosky has joined #openstack-release07:28
*** rpittau|afk is now known as rpittau07:29
*** e0ne has joined #openstack-release07:34
*** e0ne has quit IRC07:36
openstackgerritTobias Urdin proposed openstack/releases master: Release puppet-ceph 3.1.0
*** ysandeep is now known as ysandeep|lunch08:08
*** witek has joined #openstack-release08:10
*** jbadiapa has joined #openstack-release08:14
*** ykarel is now known as ykarel|lunch08:39
*** ysandeep|lunch is now known as ysandeep09:04
openstackgerritMerged openstack/releases master: Create stable/ussuri for ironic
*** dtantsur|afk is now known as dtantsur09:23
dtantsurfolks, I see the automatic post-release patches changing UPPER_CONSTRAINTS_FILE to TOX_CONSTRAINTS_FILE, is it intended?09:26
dtantsurit will be inconsistent depending on not only the branch, but also when a project got branched09:26
dtantsurttx: I suspect you may be online ^^09:26
*** e0ne has joined #openstack-release09:31
*** brtknr has quit IRC09:37
*** brtknr has joined #openstack-release09:42
dtantsuralso, folks, I don't see ironic-prometheus-exporter new release on
dtantsureven though it did get published to pypi10:06
*** rpittau is now known as rpittau|bbl10:21
*** ykarel|lunch is now known as ykarel10:25
*** zxiiro has quit IRC10:42
*** udesale_ has joined #openstack-release11:12
*** tetsuro has quit IRC11:12
*** udesale has quit IRC11:15
*** ysandeep is now known as ysandeep|brb11:18
ttxUPPER_CONSTRAINTS_FILE is the deprecated name for TOX_CONSTRAINTS_FILE11:23
dtantsurttx: ugh, good to know. we're using UPPER_CONSTRAINTS_FILE in non-tox context as well, I think11:24
ttxlooking up the tarball thing now11:25
ttxdtantsur: it's not just ironic-prometheus-exporter... ironic 15.0.0 also got published to pypi but not on tarballs.o.o11:28
ttxchecking if this affects all others11:28
ttxsame for manila 8.1.2, seems to be everyone11:30
ttxmanila 9.1.2 was published ok, 2 days ago11:31
ttxvitrage 7.1.0 ok on 2020-04-27 22:5911:32
ttxmanila 9.1.2 ok on 2020-04-28 14:5011:36
ttxironic-prometheus-exporter 1.0.0 FAIL on 2020-04-29 12:1911:37
ttxso this affects everything after manila 9.1.2, which means:11:38
ttxironic-prometheus-exporter 2.0.0, sushy-tools 0.9.0, virtualbmc 2.1.0, manila 8.1.2, horizon 15.3.0, ironic 15.0.0, ironic-prometheus-exporter 2.0.011:40
ttxsomething that happened between 2020-04-28 14:50 and 2020-04-29 12:1911:40
ttxrelease-managers: please stop approving bew releases until we solve this issue11:42
ttxjobs report success though:
ttxso it looks like an AFS failure to me11:46
ttxI'll continue the discussion on #opendev11:49
ttxdtantsur: yeah, not cool!11:51
ttxdtantsur: thanks for catching the issue early, before we have thousands of releases to fix11:52
dtantsurnp, it was actually RDO CI that caught it11:52
*** ysandeep|brb is now known as ysandeep11:54
*** rpittau|bbl is now known as rpittau12:22
*** amoralej is now known as amoralej|lunch12:33
*** slaweq has quit IRC12:35
*** slaweq has joined #openstack-release12:36
*** slaweq_ has joined #openstack-release12:38
*** slaweq has quit IRC12:41
*** ykarel is now known as ykarel|afk13:14
*** amoralej|lunch is now known as amoralej13:19
*** ysandeep is now known as ysandeep|afk13:56
openstackgerritMatthew Treinish proposed openstack/reno master: Set parallel_read_safe to True
*** ykarel|afk is now known as ykarel14:09
*** dtantsur has quit IRC14:24
*** dtantsur has joined #openstack-release14:29
*** ricolin has quit IRC14:37
ttxdtantsur: the website should have caught up with the uploads now14:39
ttxI see ironic 15.0.0 and prometheus-exporter 2.0.014:40
ttxrelease-managers: we are back in business14:40
*** armax has joined #openstack-release14:44
*** armstrong has joined #openstack-release14:47
smcginnisGlad it just needed to sync and we didn't need to rebuild all of those.14:51
openstackgerritMerged openstack/releases master: Release Glance RC2 for Ussuri
*** ricolin has joined #openstack-release15:01
*** slaweq_ is now known as slaweq15:03
openstackgerritMerged openstack/reno master: Set parallel_read_safe to True
*** AJaeger has joined #openstack-release15:16
AJaegersmcginnis, ttx, dhellmann, should we do a new reno release?15:17
*** ricolin has quit IRC15:17
ttxAJaeger: we usually try to not disturb the waters so close to release15:30
AJaegerOk, then let's wait until afterwards...15:37
smcginnisYeah, some good things in there to get out, but it will be safer if we wait a couple weeks.15:42
AJaegeris it on somebody todo list so that we don't forget?15:48
smcginnisI usually don't like WIP release requests since they kind of waste gate resources, but I'll put one up and -W it so we don't forget about it.15:54
ttxyes, submit it with W-115:55
smcginnisBTW, with that reno parallel extension change, and an in-flight openstackdocstheme patch to do the same, the cinder repo's release notes build goes from 5 minutes and 44 seconds on my machine to 50 seconds.15:56
AJaegersmcginnis: great!15:56
openstackgerritSean McGinnis proposed openstack/releases master: Release reno 3.1.0
AJaegeryes, we need to get openstackdocstheme released soonish - but first a few changes need to merge.15:57
AJaegerthx, smcginnis15:57
smcginnisNo problem.15:57
smcginnis#startmeeting releaseteam16:00
openstackMeeting started Thu Apr 30 16:00:01 2020 UTC and is due to finish in 60 minutes.  The chair is smcginnis. Information about MeetBot at
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.16:00
*** openstack changes topic to " (Meeting topic: releaseteam)"16:00
openstackThe meeting name has been set to 'releaseteam'16:00
smcginnis#link Agenda16:00
smcginnisPing list: ttx armstrong diablo_rojo, diablo_rojo_phon16:00
smcginnis#topic Review week tasks completion16:01
*** openstack changes topic to "Review week tasks completion (Meeting topic: releaseteam)"16:01
smcginnisWe got all of the RC1 deadline exceptions through.16:01
smcginnisgenerate stable branches for all cycle deliverables that are still missing one16:02
smcginnisWe have a few pending patches for this, but most have been done now.16:02
ttxyes os-vif pycadf, karbor16:02
smcginnisI did note in the commit messages that we would merge by tomorrow if no response.16:03
smcginnisI can probably take a little time later today to see if anything else has merged or is in flight, but I would be surprised if that was the case.16:03
ttxlet me see if we can ping PTLs in this channel16:03
ttxgibi, bauzas: see
ttxknikolla: see
smcginnisI don't see jiaopengju around for karbor.16:05
ttxyes, but he was not CCed on the patch16:05
ttxadding now16:05
smcginnisWas he assigned by the TC? I wonder why he didn't get added.16:06
ttxno, he applied normally16:06
ttxI think16:06
diablo_rojo_phonHe did.16:06
smcginnisHmm, my script log shows he was added by email address.16:06
ttxor not16:07
ttxanyway, would be good to give him until Monday16:07
smcginnisSure, we can wait.16:07
*** rpittau is now known as rpittau|afk16:07
smcginnisI can check if there is any reason to wait though.16:07
ttxgood idea16:08
smcginnisIf there are no patches out there and nothing merged, then we might as well get it done.16:08
smcginnis"After all the projects enabled in devstack by default have been branched, we can engage with the QA, I18n and Requirements PTLs to finalize the stable branch setup"16:08
smcginnisThanks hberaud for getting those steps done.16:08
hberaudsmcginnis: you're welcome16:08
smcginnisAnd "Ensure that all projects that are publishing release notes have the notes link included in their deliverable file"16:08
*** witek has quit IRC16:09
gibittx: I'm on the nova meeting in parallel, I will chack back later16:09
smcginnisI usually run that script again a couple times since some teams take awhile to merge the bot proposed patches to add the release note landing pages.16:09
ttxgibi: thanks! It's the stable branch for os-vif.16:09
gibittx: ack16:09
*** zxiiro has joined #openstack-release16:09
smcginnisThanks gibi16:09
smcginnisLast thing is just about letting teams iterate on RCs.16:10
gmanntwo meeting in parallel. QA stuff for ussuri devstack, grenade, d-g is all merged now.16:10
smcginnisThanks gmann16:10
smcginnisI've seen at least one RC2, so that is happening. Countdown email will have a reminder about final RC deadline.16:10
smcginnis#topic Assign next week tasks16:10
*** openstack changes topic to "Assign next week tasks (Meeting topic: releaseteam)"16:10
smcginnis"Process any remaining stable branching exception" - one for all of us.16:10
smcginnisPlease review if and when you can.16:11
smcginnis"Notify the documentation team that it should be safe to apply their process to create the new release series landing pages"16:11
smcginnisThis had always already been done by the team when I had checked in with them.16:11
smcginnisBut now that there isn't a docs team, I guess we notify sig chairs?16:11
evrardjplet's see how that goes16:12
evrardjpit's the trial by fire16:12
ttxyes, can be an email to openstack-discuss16:12
ttxso that someone else can pick it up16:12
smcginnisAnyone want to sign up for that email?16:12
evrardjpit probably should be sending to stephenfin and asettle16:13
smcginnisOr we can just make sure stephenfin has a bunch of IRC pings in here. :)16:13
evrardjpthat's what I secretly thought doing16:14
smcginnisOK, thanks hberaud for already signing up for the release-test.16:14
smcginnisevrardjp: ;)16:14
evrardjpbut maybe stephenfin is not having pings in here16:14
smcginnisAnd the check for last minute RCs.16:14
evrardjptbh I am not really sure what this entails, else I would take the action item16:14
* stephenfin shakes fist at evrardjp16:14
evrardjpyou're welcome.16:14
smcginnisHehe, three times and he appears.16:14
* evrardjp hands stephenfin a beer16:14
smcginnisLast task is the normal countdown email, which I will do.16:15
evrardjpsmcginnis: yeah, I wonder why. Probably out of a Tim Burton movie?16:15
stephenfinI _think_ AJaeger already handled the creation of the series landing pages16:15
evrardjpok I am taking the action item to follow up on that then16:15
smcginnisSee, always so proactive that I don't think we've ever had to initiate the process.16:15
evrardjpI will sync with stephenfin and AJaeger16:16
smcginnisOops :)16:16
smcginnis#topic Review countdown email16:16
*** openstack changes topic to "Review countdown email (Meeting topic: releaseteam)"16:16
smcginnis#link draft16:17
smcginnisLooks fairly straight forward. Not as much to say this close to the end other than be ready.16:18
smcginnisDates check out I think.16:18
smcginnis#topic Python runtime validation and automation16:18
*** openstack changes topic to "Python runtime validation and automation (Meeting topic: releaseteam)"16:18
smcginnisThis idea came up last week, so I made a note here to make sure the idea was brought up.16:18
smcginnisWe have automation now to switch over job templates like ussuri=>victoria.16:18
smcginnisThanks evrardjp for getting most of that started.16:19
evrardjppinging the wrong person there, I didn't do nothing16:19
smcginnisI then manually (well, semi-manually) proposed patches to repos that had merged those job changes to add the py38 metadata to their package info.16:19
smcginnisevrardjp: You started it at least.16:19
evrardjpok if you say so ! :)16:20
smcginnisSo the question came up as to whether that can be automated.16:20
smcginnisThe change itself isn't too hard.16:20
smcginnisThe tricky part would be to have the awareness programatically to know what python version to add.16:20
smcginnisPython runtimes are decided by the TC.16:20
smcginnisIt's all in rst right now, but was thinking we could get that into data/runtimes.yaml or something.16:21
smcginnisThen we would be able to script figuring out what runtimes were needed for a cycle and be able to propose patches to do the updates.16:21
hberaudif it's in a script then TC could store this info in this script and update it if needed for current cycle16:21
smcginnisThere could be a sphinx extension that pulls the info into the governance docs even.16:22
evrardjpwhile the idea is very simple and noble, the text on each page of the PTI changes16:22
hberaudI mean I suppose we will have a script to run...16:22
smcginnisBut the other tricky part would be that right now our branching automation only knows and cares about series names.16:22
smcginnisSo really just a couple string passed along.16:22
evrardjpthough I suppose we could have a support matrix, and restructure our docs.16:22
smcginnisWe would need to include more data to be able to do this.16:22
smcginnisevrardjp: I was thinking it would just be an rst directive that would generate a bullet list of the runtimes or something.16:23
evrardjpyeah I understood that part16:23
evrardjpI think it's fine16:24
smcginnisAnyway, not looking to solve this here. I just wanted to raise the idea up to the group to get people thinking about it in case there were any better ideas. Or strong motivation to try something out.16:24
evrardjpI don't foresee anyone complaining about this kind of structured data16:24
smcginnisOtherwise any time we add a new python runtime, we will just have to know to add that to the package metadata.16:24
smcginnisWhich, given how we've been changing runtimes often, I do wonder if this will not really be needed much for a bit.16:24
smcginnisDepends on how soon distros decide to pick up py39 once that is out.16:25
smcginnisI would guess we might have a few cycles before that happens.16:25
smcginnisSo trade off if it's worth putting a bunch of automation in place at this point or not.16:25
smcginnisMight be easier to just stick a note somewhere saying it needs to be done.16:25
hberaudI agree16:26
smcginnisNot even really a release team issue to deal with, just that we have the automation for the bot proposed patches that do other cycle-based updates.16:26
evrardjpbecause it's a multiple teams effort, I think it's better coordinated by human16:26
smcginnisAnyway, I can check it off the list as having raised the idea and putting it out there. ;)16:26
smcginnis#topic Open discussion16:26
*** openstack changes topic to "Open discussion (Meeting topic: releaseteam)"16:27
evrardjpor allow me to rephrase: It's good that we have someone kicking the TC to have the runtimes done as soon as possible, so that the rest of the chain can consume this16:27
ttxsorry catching up16:27
evrardjpand if that's done manually it's probably not much effort to just edit things16:27
ttxyes Python runtimes could be defined in a YAML that would end up generating a governance webpage16:27
ttxand then we could programmatically read that16:27
evrardjpttx: the problem is that we recently had conversations about justifications of versions in PTI16:28
ttxah ok16:28
evrardjpbut I guess we could have a "description"/explanation field next to the "version" :)16:28
evrardjp compared to for example16:28
evrardjpbut I don't think the data there will be much of a big deal, it's maybe a lot of automation for a two liner.16:29
smcginnisevrardjp: What was this discussion about? Not defining versions?16:29
evrardjpsmcginnis: IIRC, it was not so easy to have consensus on what needed to be there. But the problem was that this _changed_ over time16:30
evrardjpso when should the automation pickup this?16:31
*** haleyb has quit IRC16:31
*** ykarel is now known as ykarel|afk16:31
smcginnisI thought it wasn't really a consensus thing. What distros are shipping at the start of the cycle, what runtimes do they ship by default = what runtimes we need to support.16:31
evrardjpwhen do we assume "final"16:31
evrardjpsmcginnis: that's the idea16:31
evrardjpbut wording wasn't very simple IIRC16:31
evrardjpit's all fuzzy in my head16:31
smcginnisOh well.16:31
evrardjp is the policy, it's more than a few lines :)16:32
fungias to when 3.9 is likely to be added, ubuntu has been tending to add packages for each new minor python version to their most recent lts pretty much as soon as the python community releases them16:32
smcginnisI did add notes to our process doc to ping the TC on these runtimes at least, so we can make sure a job template is defined in time for the automation to work of switching from one cycle template to the next.16:32
smcginnisfungi: OK, so maybe we will see that sooner than I thought.16:32
fungipython 3.9 final is scheduled for october16:33
fungiso likely our "w" cycle16:33
evrardjpoh I think I remember now. I think it's fine if we just use a data file.16:33
evrardjpI don't think it has changed on merging, my bad.16:33
evrardjpafter first merge*16:33
smcginnisWell, we can think about it for a bit. Not a big rush right now16:34
smcginnisNothing else from me. Wrap up early?16:34
fungiyeah, python 3.9 is scheduled to release the week before openstack victoria, so very likely in focal (20.04 lts) by the start of w16:34
evrardjp(my part, not what fungi said)16:34
evrardjpwe can wrap up, sorry for the long rant in my head16:35
evrardjptl;dr: we can automate just fine!16:35
smcginnisOK, thanks all. I appreciate the time spent on release management.16:35
*** evrardjp has quit IRC16:35
ttxneed to run...16:35
*** openstack changes topic to "OpenStack Release Managers office - Come here to discuss how to release OpenStack components - Logged at"16:35
openstackMeeting ended Thu Apr 30 16:35:48 2020 UTC.  Information about MeetBot at . (v 0.1.4)16:35
ttxsmcginnis: thanks!16:35
openstackMinutes (text):
hberaudsmcginnis: thanks16:35
*** evrardjp has joined #openstack-release16:36
*** ykarel|afk is now known as ykarel|away16:36
evrardjpthanks for being our fearless leader!16:37
evrardjpsmcginnis: I have double checked only the nodejs runtime was changed after definition in the PTI. Same applies for distro. Python didn't change recently.16:37
evrardjpI mean the versions present in the PTI haven't changed after their initial merge.16:37
evrardjppython versions*16:37
evrardjpso I think it would not be too hard16:37
smcginnisI suppose if there did ever end up being a reason for changing them after the cycle had started, we could still use the data/*yaml file to manually run a script to propose whatever patches were needed to handle that change.16:38
*** udesale_ has quit IRC16:39
evrardjpyeah, I think if that happens, we can just get pinged by the TC to say "hey, could you please update the openstack job template for this release" ?16:40
evrardjpor rephrased it differently: The TC should probably update the bloody jobs if they are already defined :p16:40
smcginnisHaha, yep.16:40
smcginnisAt least that part is done as a template update.16:40
smcginnisSo it would just be a matter of updating the template, not going to every repo and making a change.16:40
smcginnisDropping a version would probably be fine doing that.16:41
smcginnisAdding a new version that could potentially break someone would be a little more controversial.16:41
evrardjptechnically that what could happen for 3.7 iirc16:41
smcginnisAt least we still have jobs running 3.7, so if we decided to add that back into the victoria template it shouldn't be an issue.16:41
evrardjpI don't see the TC taking controversial decisions anytime soon :p16:41
smcginnisPersonally, I'd prefer to stick with 36 and 38 and making the assumption that 37 is fine based on that coverage.16:42
evrardjpyeah it makes sense to me16:42
evrardjpbut hey16:42
smcginnisfungi: Oh, I had meant to bring up the AFS sync issues.16:42
evrardjpI think we had that conversation in another channel another time :p16:43
fungismcginnis: is it still a problem?16:43
evrardjpoh yeah, are those more frequent?16:43
smcginnisfungi: Not the tarball thing from this moring (though that's good to note too) but the job failures we get on simultaneous jobs running their docs publishing at the same time and failing.16:43
smcginnisfungi: Wondering if there is anything at all we can do to lock on that or make it less likely to fail.16:43
fungioh, the race condition where multiple releases are trying to publish into the same file tree16:44
smcginnisOf recognize if it is failing because of that condition and not reporting failure.16:44
fungithat job could use a mutex16:44
fungithen only one copy of it can run at a time16:44
smcginnisIs there a filesystem or something that it could write out a lock to serialize that step?16:44
fungithat's basically what a mutex is, though it'll be for the full build duration not just the synchronize task16:45
*** haleyb has joined #openstack-release16:46
smcginnisHah, yeah, I know what a mutex is, but didn't know if we had a practical way to implement that here.16:47
smcginnisThe other option I suppose would be to try to catch that error and just ignore it and assume that another job took care of the update. Since that's basically what we do right now. I'd like to avoid all the failure announcements that get sent out if they are for things we are just going to ignore.16:48
smcginnisTo make it less likely that we ignore a legitimate failure.16:48
fungiyep, so the mutex i mentioned would simply rely on zuul's sempahore construct:
fungiif you don't specify a max value, it defaults to 1, effectively operating like a mutex for any job referencing it16:56
openstackgerritMerged openstack/releases master: Create stable/ussuri for pycadf
openstackgerritMerged openstack/releases master: Create stable/ussuri for os-vif
smcginnisfungi: OK, so we could add a semaphore stanza to the publish-openstack-docs definition. Then all of those would be serialized, but we would avoid this failure from happening.16:57
fungiif those jobs complete relatively quickly, then serializing them is probably not a big deal (other jobs in the same buildset may just complete sooner)16:57
smcginnisAnd theoretically, those jobs are pretty quick anyway.16:57
fungiyeah, exactly16:57
smcginnisThe tagging jobs would definitely be the long pole.16:57
fungiyou'd define the semaphore like with the example in the docs there, and then reference it from the job via
fungithe job.semaphore.resources-first option may also be pertinent to this case17:00
smcginnisfungi: Happen to have any examples of other jobs using this?17:02
AJaegersmcginnis: translation jobs, let me find a link17:02
AJaegercheck the usage further down17:03
fungiAJaeger beat me to it17:03
smcginnisPerfect, thanks!17:03
smcginnisI will look at adding this to the publish job now.17:03
*** dtantsur is now known as dtantsur|afk17:06
*** vishalmanchanda has quit IRC17:29
*** amoralej is now known as amoralej|off17:36
*** diablo_rojo has joined #openstack-release18:00
*** jbadiapa has quit IRC18:03
*** AJaeger has left #openstack-release18:04
*** armstrong has quit IRC19:17
*** witek has joined #openstack-release19:39
*** slaweq has quit IRC19:57
*** slaweq has joined #openstack-release19:58
*** witek has quit IRC20:03
*** slaweq has quit IRC20:29
*** e0ne has quit IRC20:44
*** e0ne has joined #openstack-release20:45
*** cgoncalves has quit IRC20:53
*** Jeffrey4l has quit IRC20:53
*** huats has quit IRC20:53
*** openstackgerrit has quit IRC20:53
*** huats_ has joined #openstack-release20:53
*** Jeffrey4l has joined #openstack-release20:53
*** cgoncalves has joined #openstack-release20:55
*** e0ne has quit IRC21:00
*** smcginnis has quit IRC21:06
*** smcginnis has joined #openstack-release21:07
*** tosky has quit IRC23:11

Generated by 2.17.2 by Marius Gedminas - find it at!