Thursday, 2016-04-07

*** cgalan has joined #openstack-release00:06
*** mriedem has left #openstack-release00:12
*** mriedem has joined #openstack-release00:19
*** amrith is now known as _amrith_00:21
*** Qiming has quit IRC00:28
*** _amrith_ is now known as amrith00:40
lifelessdims: when is M fully out - so we can release pbr?01:09
dimsby this time tomorrow lifeless01:09
dimswe can break stuff friday :)01:11
lifelessI'll release fixtures today :)01:13
*** thinrichs has quit IRC01:15
dimsdear god :)01:19
lifelessalaski and nove have been waiting patiently for a bugfix01:20
lifelessleast I can do01:20
*** Qiming has joined #openstack-release01:26
*** sdake has quit IRC01:28
*** sdake has joined #openstack-release01:32
*** hongbin has joined #openstack-release01:47
*** cody-somerville has joined #openstack-release01:53
*** amrith is now known as _amrith_01:59
*** mriedem has quit IRC02:16
*** sdake_ has joined #openstack-release02:20
*** sdake has quit IRC02:22
openstackgerritQiming Teng proposed openstack/releases: Release senlin-dashboard mitaka 0.2.1
*** hongbin has quit IRC02:54
*** openstackstatus has quit IRC03:01
*** sdake_ has quit IRC03:06
*** sdake has joined #openstack-release03:07
*** sdake has quit IRC03:17
*** sdake has joined #openstack-release03:18
*** prometheanfire has quit IRC03:22
*** prometheanfire has joined #openstack-release03:23
*** sdake has quit IRC03:28
*** Daviey has quit IRC03:50
*** Daviey has joined #openstack-release03:50
*** sekrit has quit IRC03:51
*** sekrit has joined #openstack-release04:05
*** amotoki has joined #openstack-release04:20
*** TravT_ has joined #openstack-release04:41
*** TravT has quit IRC04:43
dhellmannttx: bon matin04:51
*** armax has quit IRC04:59
openstackgerritDoug Hellmann proposed openstack/releases: Revert "treat individual release jobs as warnings"
*** cody-somerville has quit IRC06:02
dhellmannttx: ping06:15
*** cody-somerville has joined #openstack-release06:17
*** pcaruana has joined #openstack-release06:26
openstackgerritMerged openstack/releases: support warnings from project_config validation
openstackgerritMerged openstack/releases: use a global for the zuul layout path file name
ttxdhellmann: o/06:46
*** vgridnev has joined #openstack-release06:46
dhellmannttx: greetings! I've started merging a few of the little validation changes we need to make the release patch landable06:47
dhellmannnotes are in
ttxhm senlin-dashboard mitaka 0.2.106:49
dhellmannyeah, I was going to let that wait06:49
dhellmannit looks like a legit bug, but not critical06:49
ttxyes, we granted the RC because the bug was annoying and senlin is not that critical, but today is really pushing it06:49
ttxwe should probably restore rules once we process the yaml rather than before06:51
openstackgerritMerged openstack/releases: treat individual release jobs as warnings
dhellmanndo I have that out of order?06:52
AJaegerdhellmann, ttx, dims: is ready for MItaka release - we even include now a link to
ttxdhellmann: you did, I fixed it06:53
dhellmannAJaeger: excellent, we're working on that right now06:53
dhellmannttx: ah, thanks06:53
ttxSummary of the week so far...06:53
ttxWe fixed most of the tarballs situation, except monasca-ceilometer which is pretty far from having what was necessary to generate tarballs06:54
ttx(and monasca-thresh which is java)06:54
ttxsahara-extra appeared to be OK, not sure why we ended up flagging it06:55
dhellmannso those are still not being listed06:55
ttxthose are no-tarballed06:55
ttxi.e. the tag is there and people can look up the git repo but that's it06:55
dhellmannsahara-extra didn't use the job template for release jobs, which is one of the rules we had to relax06:55
ttxah, ok.06:55
dhellmannI'm working on the project-config fix06:55
ttxGranted 3 RCs, all pointing to wide holes in testing: keystone federation, Mistral functional testing and Senlin tests06:56
AJaegerdhellmann: is there anyting in project-config that you need taken care of urgently?06:56
ttxAJaeger: nothing urgent imho06:56
AJaegergreat ;)06:56
dhellmannAJaeger: nothing urgent, thanks!06:56
ttxWe have teh PR machine lined up at 9am CT (1400 utc), so we should officially announce on our side shortly before that06:57
dhellmannI make it 8:57 now, is my clock off?06:57
* dhellmann should have checked what tz he's in06:57
ttxclock is correct. Announce must go shortly before 4pm our time06:58
AJaeger8:57 is 6:57 UTC06:58
dhellmannoh, wait, CT central US time not central EUROPE time :-)06:58
AJaegerdhellmann: where are you? 8:57 is Europe...06:58
ttxyes. CET != CVT. I suspected you got that one wrong06:58
dhellmannAJaeger: : lyon06:58
AJaegerdhellmann: Ah! In "my" timezone, just around the corner (ok, I'm in Germany) ;)06:59
ttxdhellmann: we can wrap it up quickly this morning and you can pop in shortly before 4pm maybe ? Otherwise I guess I can send tit06:59
dhellmannttx: gah, yes, I thought we were talking about doing this this morning -- I'm not going to be near a computer at 14:0006:59
ttxor maybe you can cron it; though that sounds risky07:00
dhellmannyeah, I wouldn't want to do cron07:00
dhellmannI can stage it and send the announcement from my phone, I think07:00
ttxLet's try that, worst case scenario I'll send it07:01
dhellmannI need to do the build now, though07:01
ttxSorry I should have been clearer. Release processing used to takje so much time we never announced in the EU morning, but it's true it's a possibility07:01
ttxnow that we took LP out of the equation07:01
dhellmannyeah, this is going to take a while, but I hope not as long as before07:01
openstackgerritTony Breeds proposed openstack/releases: Add yaml2ical release for 0.6.0
dhellmannnp, I should have asked for a UTC time :-)07:02
ttxPR is always in CT due to Austin-centricity in marketing07:02
dhellmannttx: do you want to look over that set of versions and hashes in the proposal?07:02
ttxI'll quickly check the change yes07:02
dhellmannttx: it includes unmanaged projects because the script doesn't check. I don't know if we should remove those.07:03
ttxno no it's fine07:04
ttxthe only thing I did before was to process managed first07:04
ttxsince I wasn't sure LP timeouts would not delay me at some point07:04
ttxweird link for ironic-inspector at
ttx(the build from the change)07:05
ttxmight be a test artifact though07:06
ttxOtherwise all RCs check out, and the releases.o.o page shows proper collapse of versions07:06
* dhellmann looks at the link07:07
ttxthat is in the table07:07
ttxAt "other projects"07:07
ttxshows `release notes < >`__ instead of release notes for some reason07:08
dhellmannttx: oh, their release notes link is wrong in the deliverable file, we can fix that07:08
dhellmannstand by07:08
ttxheh, first time I notice07:08
ttxI usually pick up those things07:09
dhellmannbadly formatted yaml07:09
openstackgerritDoug Hellmann proposed openstack/releases: fix formatting of ironic-inspector release note link
ttxlet's merge that and then go. How much more time do you have this morning ?07:11
ttxI'd like to allow some time to fix release notes if they end up doing the wrong thing07:11
dhellmannlet's go ahead then and see how things go07:11
*** ihrachys has joined #openstack-release07:11
dhellmannI can spend a little time on it this morning07:12
ttxyes we can fix the link after07:12
dhellmannok, I'm going to merge the change with the shas now and start processing it07:12
ttxok go for it07:12
dhellmannI've staged the announcement email as a draft that I can reach from my phone, and scheduled an alarm for 8:50 AM Austin time to send it07:13
ttxAnything between 15:30 and 15:55 our time will work07:13
dhellmannok, I moved it up a bit earlier then07:14
ttxIf not sent at 15:55 our time I'll step in07:14
dhellmannsounds good07:14
ttxand send the draft as etherpadded07:14
dhellmanncool, that's the text I copied07:15
ttxAlso I'll handle shortly before that07:15
dhellmannok, I've removed my WIP flag there07:16
ttxYou can remove your W-1 at too07:17
patchbotttx: patch 300667 - openstack-infra/project-config - Give project teams access to approve stable/mitaka...07:17
ttxI'll rebase it just in case07:17
ttxI'll place a W-1 on it that I can remove this afternoon07:18
dhellmannremoved my wip07:19
dhellmannthe validation on this patch is going to take a while since it touches so many repos -- lots of cloning07:19
ttxHmm, can't W-1, I'll comment to wait until my signal07:20
ttxnot a big deal anyway at this stage07:20
openstackgerritMerged openstack/releases: tags for mitaka final releases
dhellmannthere we go, starting tagging locally07:22
ttxalright, let's the games begin07:22
ttxlet the games*07:22
dhellmannaodh merge release tags job failed :-(07:24
ttxdid it ever succeed ?07:24
* ttx checks07:24
dhellmannastara-horizon, too07:24
dhellmannno idea07:24
ttxaodh 1.1.2 worked alright07:25
dhellmannwaiting for zuul to catch up and start the barbican job07:25
dhellmannI don't know how critical that job is07:26
dhellmannit worked for barbican and some of the other repos07:26
ttxI think it can be fixed afterwards, unless it breaks reno07:26
dhellmannyeah, can you start a list?07:27
dhellmannpausing again for zuul to start ceilometer07:28
ttxbarbican is fine07:28
dhellmanncongress worked, too. maybe we got some bad nodes on the first few jobs07:30
ttxok most are working, looking into the specific failures07:30
ttx! [remote rejected] HEAD -> refs/publish/master/merge/release-tag (change closed)07:31
openstackttx: Error: "remote" is not a valid command.07:31
*** daemontool has joined #openstack-release07:32
ttxIt's like it thinks we are trying to push it on an existing change07:32
dhellmanndo you think it's safe to queue all of these up or should I keep pausing?07:33
ttxI think it's safe. All it does is propose a change07:33
ttxsome projects may need us to manually propose those07:33
dhellmannok, pushing ahead then07:33
ttxOh. Oh.07:34
ttxI think failure is the way this job has to submit no change07:34
dhellmannthat's confusing :-)07:35
ttxBasically it pushes nothing and then ends up amending a commit that is already merged07:35
ttxI think07:35
ttxgit merge returns "Already up-to-date"07:35
ttxthen git commit --amend touches a commit that is already merged, hence Gerrit rejecting the change07:36
ttxDigging deeper now but i think it's safe07:36
*** dtantsur|afk is now known as dtantsur07:36
ttxYes, when it works it returns "Merge made by the 'ours' strategy" and the the commit --amend amends the merge commit07:37
ttxI'll take notes, we can fix that post-release07:37
dhellmannok, cool07:37
ttxso it affects things that did not create a stable branch until the very last release07:38
ttx(so they have all the RC tags in master)07:39
dhellmannmakes sense07:39
* dhellmann wishes he had a shorter passphrase07:39
ttxI'll admit I changed mine to a shorter one after handling releases07:42
dhellmannI'm not seeing the release notes builds updated, but I don't know if we have a cache in front of that web server07:43
dhellmannI checked keystone and glance07:43
AJaegerdhellmann: have the post jobs run already?07:44
dhellmannAJaeger: release notes are rebuilt on tags07:44
*** cody-somerville has quit IRC07:45
AJaeger was just updated a few minutes ago07:45
AJaegerBut the version looks broken ;(07:45
dhellmannok, all the tags are pushed07:46
dhellmannAJaeger: should be updated, but I still see RC1 there07:47
dhellmannAJaeger: the version in the page footer is going to reflect the master branch version07:47
dhellmannI wonder if it's a git cache issue07:47
AJaegerlast modified has "2016-04-07T09:39:11 CEST" - so 8 minutes ago07:48
AJaegerso, the job run and published - but can't help right now with why it doesn't do the right thing ;(07:48
ttxLooks like release notes collapsing is not working... I thought we tested that successfully with release-test though07:48
dhellmannyeah, so did I07:49
dhellmannI'm trying it locally07:50
ttxrunning the other checks on the page while you investigate07:58
dhellmannI think I found it. If I'm right we'll need a reno release and then to retrigger the release notes jobs somehow08:01
*** vgridnev has quit IRC08:02
dhellmannalternately we can add a flag to the releasenotes file in each repo08:02
dhellmannlanding that would trigger the build we want08:02
openstackgerritDoug Hellmann proposed openstack/reno: default to collapsing pre-releases in sphinxext
dhellmannttx: ^^08:03
dimsttx : dhellmann : good morning08:03
AJaegerdhellmann: release notes jobs will be retriggered with any merged, won't it?08:03
dhellmannAJaeger: yes, that's right, but we don't know that all projects will land something today08:03
ttxMaybe we can live with that08:03
AJaegerdhellmann: I know - but we could ask PTLs to do so08:03
ttxdhellmann: we could cover managed projects08:04
ttxThe page with Rcs and all is not incorrect or ugly08:04
ttxit's just promoting something we'd rather hide at this point08:04
ttxso I'd fix it best-effort only08:04
ttxrather than "must have be08:04
ttxfore announce"08:05
dims+2A on
ttxAJaeger: if we land stable/mitaka recent translations for example, would that retrigger the job alright ? We could cover a few with that08:06
ttx(once that reno change is shipped)08:06
ttxcovers most of the base08:07
ttxthen we can go hunting if we still miss a refresh in a few hours08:07
dimssounds good to me08:08
dimsor when one of the infra folks arrive, they usually have a way to trigger jobs by hand too right?08:08
openstackgerritDoug Hellmann proposed openstack/reno: default to collapsing pre-releases in sphinxext
dhellmanndims: that needed an edit, give me a minute to test locally before re-approving08:09
dimsdhellmann : ack08:09
ttxI'll add a column in the release tracker for checking release notes refresh08:10
AJaegerttx, yes, that would retrigger it already.08:10
AJaeger,n,z is full list08:10
ttxOK, so let's keep that card up our sleeves08:10
ttxsince we still have approval rights on stable/mitaka :)08:10
AJaegerttx, oh, you pasted that one already - the list is short, so will only cover a few projects08:11
ttxack, but better than nothing08:11
ttxcovers most of the managed stuff08:11
AJaegerdims: yes, infra admins can enqueue jobs manually in the queue... Not sure how involved that is08:11
dimsAJaeger : y, worst case08:11
ttxworst case we have release notes with RC and beta mentions. Not a big deal08:12
openstackgerritDoug Hellmann proposed openstack/reno: default to collapsing pre-releases in sphinxext
dimsttx : AJaeger : that zanata url is a mix of master and stable/08:12
dhellmanndims: ok, I'm just going to force it for now08:12
dimsdhellmann : sure thing08:13
* dims puts on a pot of coffee08:13
AJaegerdims: I should have read ttx's comment first ;(08:13
AJaegerdims: ttx has stable/mikata in it08:13
dhellmannok, the change I just made to reno should fix the issue so as soon as it clears the check queue and can merge we can release that08:15
dhellmannas AJaeger pointed out, work in any of these projects today should update their notes pages08:15
dimsdhellmann : i was eyeballing zaqar, i was not expecting any changes and i see some -
dhellmannI'll make a note to add better test coverage for the sphinx extension08:16
dimsdhellmann : nova too -
dimsdhellmann : weren't we supposed to use the same SHA as the last it would not pick up later changes from branch?08:17
dhellmanndims: zaqar had an rc208:17
AJaegerdims, what about - they had an rc208:17
dhellmannnova had rc308:17
dimsah gotcha08:18
dimsi was going by the email "Changes in nova" AJaeger  and dhellmann08:18
ttxOK added a Relnotes column with the things that need a refresh today08:18
dimsguess the code picks up the first SHA and not the last SHA08:19
AJaegerdims: Ah, the announcement email - yes, that's confusing.08:19
dimsit even shows diffs08:19
dhellmannoh, bug in the announce script08:20
dhellmanndims: is ready for +2a08:21
patchbotdhellmann: patch 302586 - reno - default to collapsing pre-releases in sphinxext08:21
ttxI thought we would not issue anything before the announce :/08:21
dhellmannttx: these are the automated emails08:23
dimsy we don't have a toggle to mute the announce temporarily08:23
dhellmannI've added that as a thing to do for newton08:24
dhellmannttx: did all of the tarballs build ok?08:24
openstackgerritMerged openstack/reno: default to collapsing pre-releases in sphinxext
ttxdhellmann: are the emails listing all the release notes for mitaka or only post-RC1 ?08:24
dhellmannttx: it looks like they are incorrectly detecting the prior release and then showing the notes since then08:25
dimsdhellmann : right08:25
dimsfirst SHA and not the last SHA08:25
dhellmannthat script needs to handle the rc->final transition case differently08:25
dhellmannthere are a bunch of improvements to make to that08:26
ttxThat makes up for a pretty confusing announcement08:26
dhellmannyeah, I didn't anticipate that08:26
ttxwe have way too many things we only exercise on release day.08:27
openstackgerritDoug Hellmann proposed openstack/releases: reno 1.6.2
ttxI used to try to keep those to a minimum, using the same process for milestones and final... but we added a lot of new things08:28
ttxFeels like we need to send something out to explain the confusion now08:29
dhellmannthat announce script was written with non-services in mind, too, so there are a bunch of cases that just haven't come up before08:29
dhellmannttx: good idea. do you want to draft something while I finish this reno release?08:29
*** openstackstatus has joined #openstack-release08:29
*** ChanServ sets mode: +v openstackstatus08:29
ttxJust wondering how we can send something without saying "it's released" at this stage08:30
ttxI'll try08:30
dhellmannttx: just call it an error in the build script, that's pretty close to accurate08:30
dhellmann"an error in the build script led to sending erroneous release announcements"08:31
dhellmannthe check queue is starting to get long again08:31
AJaegerhorizon is running mass rechecks since npm works again ;)08:32
dhellmannI'm going to go ahead and release reno and then merge that new tag patch. I'll use the local copy of the file to ensure the results are the same.08:33
dimsdhellmann : +208:33
-openstackstatus- NOTICE: jobs depending on npm are now working again08:34
ttxdhellmann: prepared draft at the bottom of
dhellmannttx: looks good, I'll send that to -announce and -dev now08:38
*** dtantsur is now known as dtantsur|brb08:38
ttxmaybe just announce since -dev was not hit ?08:38
ttxbah, can't hurt to hit -dev I guess08:38
ttxyour call08:38
dhellmannyeah, both seems safe so that's what I did08:40
ttxdhellmann: looks like your morning is properly ruined now. But that was a long shot :)08:40
dhellmannI hope the weather holds out for an afternoon in vienne08:40
openstackgerritMerged openstack/releases: reno 1.6.2
*** daemontool has quit IRC08:41
ttxToo much new process with tooling depending on you that we can spare you on release day08:41
dhellmannthe reno fix is released08:41
dhellmannyeah, the builds went fine but the release notes stuff and announcements are all new and I should have tested them more08:41
ttxAJaeger: dhellmann: how long shall we wait before pushing random changes to regenerate ?08:41
dhellmannwe want the pip cache updated, I think, but I don't know how long that takes08:42
dimsdhellmann : gets updated when it's used i think, so we should be good dhellmann08:43
* ttx watches queue08:43
dhellmanndims: oh? in that case, ttx, maybe we should test one08:43
AJaegerdhellmann: if it'S on pypi, should be safe to approve.08:43
ttxlet's se there is a ceilo change in the post queue08:44
AJaegerIt will take some time to push a change through gate and then it sits in post queue...08:44
ttxChange 6a0f84d in the post queue. Should trigger a release notes refresh for ceilo08:44
AJaegeryep, let's check that08:44
ttxIf that one works I'll push the translations08:44
AJaeger uses reno 1.6.1 ;(08:50
ttxdhellmann, dims; let's merge now08:50
patchbotttx: patch 302546 - releases - fix formatting of ironic-inspector release note link08:50
ttxdo we need the upper-constraints bump ?08:50
AJaegerttx, not needed for *post* jobs08:51
AJaegerI mean: u-c does not work in post jobs, thus is not used there - so, no need to update08:51
ttxso why did it fail to pick it up ? too early ?08:52
ttxwe have an horizon change up08:52
AJaegerI guess too early ;(08:52
AJaegerLet's wait for next job ;)08:52
ttxlet's see if that one picks it up08:52
AJaegerswift has no release-notes? ;(08:52
ttxAJaeger: it doesn't need collapsing08:53
dimslooking at infra08:53
AJaegerttx, swift has no releasenotes job08:53
ttxah, that too08:53
* ttx grabs coffee08:54
dimshmm, i may have had the wrong info
dimsthough */5 means every 5 mins08:56
AJaegerdesignate still hsa 1.6.1 ;(08:58
* AJaeger is confused08:58
* AJaeger might be impatient as well08:59
* AJaeger remembers that normally new releases show up "quickly"08:59
ttxdhellmann: maybe you can go back to your touristy morning. Lovely weather to walk in vieux-Lyon08:59
ttxIf the new reno doesn't cut it and we can't fix it ourselves, I'll text you09:00
dhellmannttx: ok, that would be good09:00
* ttx considers jumping in his car and joining for lunch09:00
dimslol :) leave me and Andreas here by ourselves :)09:00
dimshave fun dhellmann !09:00
dimsit's pouring rain here..will get 2 inches today. we had 6-8 inches snow the last few days09:01
dhellmannthis horizon reno build is installing tons of packages09:01
dhellmanndims: wow09:01
dhellmannah, there it hit the tox step09:02
dimsso....send pics dhellmann :)09:02
ttxdhellmann: enjoy!09:02
AJaegerdhellmann: have fun!09:02
AJaegerstill 1.6.1 ;(09:02
dhellmannyeah, I think we need to wait for a cache to be updated somewhere09:03
dimsy, i am going to look at the logs to see if u-c gets hit09:03
dhellmanndims: I think reno is in the blacklist for u-c so it shouldn't matter09:03
dimsah ok09:03
dhellmannok, I'm going to drop off. I'll check in later, but email or text if you need something from me before that announcement deadline09:04
AJaegeryes, it's in blacklist09:04
AJaegerdims: u-c is not used for post jobs!09:05
dimsAJaeger : ack i remember you mentioning it09:05
AJaegerdims, should we update global-requirements for reno? Not urgent - but developers should get what we require09:10
dimsAJaeger : sure, go for it. i'll approve09:10
dims"Changes to update the minimum version of a library developed by the OpenStack community can be approved by one reviewer, as long as the constraints are correct and the tests pass."09:11
dims^^ FTW :)09:11
AJaegerDidn't know that...09:12
ttxwe have a barbican build lined up09:15
ttxBarbie can do it09:16
*** vgridnev has joined #openstack-release09:17
dimsgo barbican!09:18
AJaegerFirst it needs a node to run on - but the longer that takes, the higher the chances ;)09:22
*** kzaitsev_mb has joined #openstack-release09:23
ttxbut the longer we wait the less likely we'll have them all rebuilt in time :)09:24
AJaegeryeah ;(09:24
AJaegerttx, from entering to leaving of gate is 2hs - so I suggest you go ahead and just approve...09:25
AJaegerin 2+ hours we should have reno09:26
ttxhmmm, ok09:26
AJaegerwe can also wait for barbican...09:26
AJaegerI'm just seeing how long it takes09:26
ttxwe have nova and horizon queued09:27
ttxand heat09:27
ttxI'll queue things that are not there09:27
*** breton has joined #openstack-release09:28
ttxAlright, added neutron, ceilometer and keystone09:29
ttxAJaeger: we might want to prepare for plan B, which would be to see if we can trigger the jobs separately09:30
dimsttx : i know fungi or clarkb have tickled a job for me in the past09:31
AJaegerttx, that needs an infra-root to do it - so wait for fungi to wake up. Unless jhesketh is still awake09:31
ttxLet's wait until fungi is up09:31
AJaegerwait, fungi is travelling today, so might not be available09:31
* jhesketh is around09:31
dimshey jhesketh09:31
AJaegerjhesketh: for how much longer?09:31
jheskethas long as needed09:31
ttxjhesketh: we may need to trigger release notes generation jobs for a number of projects, once a new version of reno kicks in09:32
AJaegerWe wait for reno 1.6.2 to be available for building (currently 1.6.1 is still used) and then want to retrigger these09:32
dimsjhesketh : any way to trigger bandersnatch to pull in 1.6.2?09:33
ttxThe post queue is unfortunately too starved for us to regenerate those before we need to announce release09:33
ttxdims: it should have triggered already -- so maybe we need to investigate and see if it did09:33
jheskethI haven't done a manual bandersnatch before, but I can take a look09:34
ttxi.e. look at the bandersnatch pool and see if reno is there already09:34
ttxmaybe it's there already09:34
jhesketh1.6.2 doesn't appear to be in the mirror-update yet09:37
AJaegerIt's here
AJaegerjhesketh: thanks for checking!09:38
ttxWould be great to somehow pull it in asap09:38
* jhesketh is looking09:40
jheskethI can't see a way to ask bandersnatch to download a particular package09:43
jheskethdoes anybody know if that's a feature09:43
jheskethotherwise it should run every 15min09:43
ttxjhesketh: package has been here for an hour now09:44
ttxso it's weird that it wouldn't get pulled in09:44
AJaegerttx, indeed09:45
jheskethhmm, bandersnatch is currently running09:45
ttxjhesketh: been running for how long ?09:46
jheskethwell it just finished, but I can't see the package so I'm trying to figure out what is going on09:47
dimsjhesketh : found some notes here - (curl with PURGE)09:48
* AJaeger just build something locally and it downloaded reno 1.6.209:48
dimsAJaeger : found the urls to the mirrors in in that bandersnatch.html - definitely not there (example
* dims going to be off and on. need to wake up kids and get them ready for school09:52
*** kzaitsev_mb has quit IRC09:52
* AJaeger has lunch now - back later...09:53
*** dtantsur|brb is now known as dtantsur09:55
ttxjhesketh: do you see "stale serial" errors in /var/log/bandersnatch for reno ?09:55
jheskethttx: no, there's nothing in the logs for reno09:55
*** kzaitsev_mb has joined #openstack-release09:59
ttx has 1.6.2 for sure, and none of our mirrors have10:01
jheskethso this is the only clue I have thus far:
jheskethbandersnatch thinks there is only one package that needs updating and it keeps failing to update it (AnyBlok)10:08
jheskethpurging also doesn't appear to help10:08
jheskethnot sure if that's blocking on other packages, or if bandersnatch isn't finding other packages (such as reno) to update10:09
ttx"1 packages to sync."10:10
dimsjhesketh : i just tried this -
*** kzaitsev_mb has quit IRC10:12
dimslet's see if that makes any difference10:12
jheskethdims: yep, I tried that already10:13
jheskethdidn't pick up on any new reno10:13
*** Qiming has quit IRC10:13
dimsouch ok10:13
*** Kiall has quit IRC10:14
*** Kiall has joined #openstack-release10:14
*** kzaitsev_mb has joined #openstack-release10:15
ttxmirrors last modified yesterday 14:1010:16
ttxwhile there were a lot of updates since then10:16
ttxthat would be when AnyBlok 0.8.2 was released10:19
dimsttx : i still see some jobs for reno -
dimsttx : especially the wheel jobs10:20
dimsnot sure how they factor into this situation10:20
*** sdague has joined #openstack-release10:20
ttxthey should not, we did reno-pypi-both-upload and seems correct10:21
jheskethso reading bandersnatch source it looks like anyblok may be stopping the rest from working10:21
ttxjhesketh: I just issues a PURGE on AnyBlok in case you didn't10:22
*** daemontool has joined #openstack-release10:22
jheskethttx: yeah I tried that and we have a wrapper script that does that automatically anyway10:22
dimsy i did that too :)10:22
ttxjhesketh: it is stopping the rest from working. Since yesterday 14:1010:23
jheskethso, if I look at it looks like bandersnatch will not do anything more until it completes its previous todo list10:24
jheskethwhich it writes out to a file10:24
jheskethcat /afs/
jheskethAnyBlok 204920610:24
dimsttx : i see other problems with pypi mirroring ( fixtures===2.0.0 is not available as well10:24
dimsttx : jhesketh :
dimsttx : jhesketh : so definitely busted10:24
jheskethso if I'm understanding it properly it should be safe to remove the todo10:24
jheskethand it'll unblock it10:25
jheskethbut if somebody else wants to double check the bandersnatch code for me that'd be handy10:25
* jhesketh doesn't want to bork the mirror10:25
dimsit's already borked :)10:25
jheskethheh, I'm sure I can make it worse ;-)10:25
dimsjhesketh : want to delete that file by hand? instead of fixing the code?10:25
ttxjhesketh: I think removing the todo won't break it further10:26
jheskeththe more I read the code though the safer it seems to remove the file10:26
ttxworst case scenario it triggers a larger sync10:26
jheskethdims: that's what I'm suggesting, but I want to make sure it's safe, but as I say I think it is10:26
dimsy, looks safe to me :)10:26
ttxjhesketh: try to remove it before the next run in 3 min10:26
jheskethI have a feeling it'll be a large sync once removed, yes10:26
ttxAnother worst case scenario is that it will stop updating. But that is what it does already10:27
jheskethsigh, I have to do something odd to mv it since it's afs...10:28
ttxdims: barbican ran and picked up reno==1.6.1, sigh10:29
jheskethokay, moved10:29
ttxjhesketh: just in time10:29
jheskethheh, worse case I'd just trigger it again10:29
jheskethbut in this case I'm going to wait 30sec10:29
ttxhmm, dropping out for lunch10:31
jheskethlooks like it's writing contents to the todo10:31
jheskethand reno is in there10:31
jheskethhopefully anyblok doesn't stuck it again, but it seems to be working through the list10:32
dimsjhesketh : w00t10:32
ttxOK our next candidate in the queue is horizon 92f758710:32
ttxjhesketh: thx!10:33
* ttx lunches10:33
jheskethdon't thank me yet10:34
* jhesketh watches it closely10:34
dimsjhesketh : it's progress :)10:34
jheskeththere it is
dimsi see it!10:38
dimsofficial thanks jhesketh ! :)10:40
jheskethany time10:40
jheskethdoes that unblcok ya'll?10:40
* jhesketh didn't read the scrollbac10:40
jheskethor will you need some things enqueued10:40
dimsjhesketh : it means we can queue the zanata updates. not sure if ttx wanted to ask for help requeing other jobs that already ran10:41
jheskethokay, I'll be around for a bit (hopefully til more roots are online) so feel free to ping me10:42
dimsthanks a ton Joshua!10:42
dimsttx : i see 3 reviews here, approving*zanata.*10:55
*** cgalan has quit IRC10:59
ttxdims: that was probably unnecessary since those all had changes in the queue anyway. But doesn't hurt11:04
ttxreno 1.6.2 picked up11:04
ttxnova at least got synced11:05
ttxand collapsed ok11:05
ttxsame for keystone and neutron11:08
openstackgerritMerged openstack/releases: fix formatting of ironic-inspector release note link
ttxjd__: if you could merge some change in aodh to retrigger release notes generation, that would be awesome.11:14
jd__ttx anything in master? I can probably do that11:15
ttxjd__: or in stable11:15
ttxceilometer got something in recently so it's covered11:15
ttxredrobot: if you could merge some change in barbican to retrigger release notes generation11:15
jd__damn there are fewer patches waiting in queue that I would expect for Aodh… we're doing a too good job :(11:16
ttxthat would be great11:16
jd__I'll make a false patch :-)11:16
ttxjd__: dammit11:16
ttxjd__: don't sweat it, we'll likely have to ask infra to do some manual triggering11:16
ttxit's just that if you had something almoist there now was the moment to push it11:16
jd__ttx: ok, let me know otherwise, I can pimp some doc :)11:16
ttxsmcginnis: if you could get some change merged in cinder (master or stable) now that would be great (we need to retrigger release notes generation)11:17
ttxnikhil, flaper87: if you could get some change merged in glance (master or stable) now that would be great (we need to retrigger release notes generation)11:18
ttxSergeyLukjanov: if you could get some change merged in sahara (master or stable) now that would be great (we need to retrigger release notes generation)11:19
ttx_amrith_: if you could get some change merged in trove (master or stable) now that would be great (we need to retrigger release notes generation)11:19
ttxflwang, flaper87: if you could get some change merged in zaqar (master or stable) now that would be great (we need to retrigger release notes generation)11:19
SergeyLukjanovttx, approved11:21
patchbotSergeyLukjanov: patch 302217 - sahara - Update the Administrator Guide link11:21
ttxSergeyLukjanov: awesome, thx!11:22
* ttx looks up stable/mitaka for candidate changes to push11:22
AJaegerjhesketh, ttx, dims: Glad to see this got resolved! Thanks, jhesketh !11:22
dimsttx : there's a bunch here -
ttxyeah, we still need them to push it11:26
ttxhmm, there are a few requirements updates for stable that we could push11:27
*** Qiming has joined #openstack-release11:28
patchbotttx: patch 237220 - barbican (stable/liberty) - Updated from global requirements11:28
ttxnot so many, since we jhave most the bases covered already11:28
ttxdims: if you +2 this one I'll approve it ^11:29
dims+2 ttx11:30
ttxdims: please review/approve
patchbotttx: patch 302518 - releases - Revert "treat individual release jobs as warnings"11:33
openstackgerritThierry Carrez proposed openstack/releases: update status of mitaka and newton
openstackgerritMerged openstack/releases: Revert "treat individual release jobs as warnings"
*** vgridnev has quit IRC11:39
*** _amrith_ is now known as amrith11:44
amrithttx, what's that you say?11:46
ttxamrith: we need some change to land in trove so that release notes end up being regenerated. If there is anything that you can approve there, now is the time to do it11:48
dimsamrith : if you have some review close to merging either in master or stable, please +2A it, there was a hiccup in the reno release notes job earlier and we need 1.6.2 of reno to run11:49
ttx might do it11:49
patchbotttx: patch 302581 - trove - Merge tag '5.0.0'11:49
amrithttx, dims, ok11:49
ttxthose need to be approved anyway11:49
amrithI'm looking at that one ttx11:49
amrithseems benign enough except the left file seems to include merge conflicts so I'm wondering what's going on.11:49
ttxamrith: yeah, that's a Gerrit bug, see reference above11:50
amrithsure, that's what monty calls 'disgusting' :)11:50
ttxit is disgusting tbh11:50
ttxand every 6 months we get to explain why11:50
amrithdone, I +2/+111:51
amrithin 1h 13m, if the gods smile on us, it'll merge.11:51
ttxjust in, time for the announcement hopefully11:52
ttxdims: probably not worth asking people to merge stuff now, it won't make it anyway11:52
*** ChanServ changes topic to "OpenStack Release Managers office - Where weekly 1:1 sync points between release manager and PTLs happen - Logged at"11:53
dimsttx : ack. no harm if it happens over the day, not everyone may go looking at the web site release notes that quickly11:53
dimsttx : so only thing left is to send the release email?11:53
ttxsure, but I think we'll ask infra to retrigger jobs for the things we still miss11:53
amrithso, do you need one in trove-client as well?11:53
ttxamrith: no11:54
amrithI have a good candidate there :)11:54
ttxthose are fine as is, since they had no RCs11:54
amrithit is a disgusting change and I'd like an excuse to merge it :)11:54
*** gordc has joined #openstack-release11:55
bauzasttx: do we need something to provide for the mitaka GA ?11:56
ttxbauzas: no, Nova is all set11:56
bauzasttx: okay, no need to update the releeses file ?11:57
bauzasit's manualN11:57
ttxbauzas: no that was done for you11:57
bauzasI possibly missed then11:57
ttxwe have a script that proposes promoting last RCs to final automatically11:58
patchbotbauzas: patch 302452 - releases - tags for mitaka final releases (MERGED)11:59
bauzasokay, fine by me then :)11:59
dimsttx : +1 to (ask infra to retrigger jobs for the things we still miss)11:59
*** vgridnev has joined #openstack-release12:03
*** jhesketh has quit IRC12:07
*** jhesketh has joined #openstack-release12:07
dimsttx : stepping out for about 30 mins12:12
ttxwe'll start chasing down infra folks in ~1 hour12:13
vgridnevhey release team, I see that sahara was tagged with 4.0.0 tag, is it right that stable/mitaka for us is opened for post-release fixes?12:19
ttxvgridnev: it is but we haven't restored the ACL just yet12:20
ttxso the branch is still udner our control for a few more hours12:21
ttxuntil we announce (2 hours)12:21
vgridnevttx, thanks!12:21
*** doug-fish has joined #openstack-release12:26
*** kzaitsev_mb has quit IRC12:38
*** doug-fish has quit IRC12:42
smcginnisttx: We've had some recent merges. Still need something sent through?12:42
*** doug-fish has joined #openstack-release12:42
AJaegersmcginnis: has still RC1 - so, not recent enough ;)12:43
ttxsmcginnis: yep, what AJaeger said12:44
smcginnisAJaeger, ttx: I see a few that look like they're ready. I'll send something through.12:45
AJaegersmcginnis: one is enough ;)12:45
smcginnisWhat happens to the Mitaka release notes in master now? Do they need to be moved?12:46
ttxsmcginnis: no, nothing needs to meve12:46
smcginnisttx: How does that work to track where the change belongs? Git history?12:47
ttxsmcginnis: yes12:47
*** doug-fish has quit IRC12:47
*** sigmavirus24_awa is now known as sigmavirus2412:47
ttxbetter to wait for dhellmann for him to explain how the magic works12:47
smcginnisttx: Good enough for me. Just want to make sure I understand the process.12:47
dimsttx : am back12:50
dimsttx : TODO shows 6 left. right?12:51
ttxyes, let me check12:52
amrithttx, change merged in trove12:52
ttxamrith: ack thx12:53
amrithyou must have some double-secret super powres, or dims must've done some gitfu because it was done in 56m which is a new speed record. I just got a call from the Guiness Book of Records ...12:53
*** doug-fish has joined #openstack-release12:54
ttxjhesketh, fungi: is there a possibility to artificially trigger -releasenotes jobs for a number of projects ? Looks like there won't be changes in time for release announcement12:54
ttxamrith: we burn a lot of candles around here12:55
amrithI tried that but the smoke alarms went off :)12:55
amrithi also have no eyebrows now12:55
dimslol amrith12:55
*** kzaitsev_mb has joined #openstack-release12:55
*** doug-fis_ has joined #openstack-release12:56
amrithy'all have serious stuff to do, I'll bugger off. holler if I can be of assistance.12:56
dimsthanks amrith12:57
*** doug-fish has quit IRC12:59
jheskethttx: it's possible to enqueue changes into a pipeline, but it'll run all of the jobs (for that pipeline, such as post) not just an individual one13:02
jheskethttx: which jobs do you need?13:02
ttxhmm, probably a bad idea there13:02
ttxfor example, we need aodh-releasenotes to run (to generate and upload release notes using reno 1.6.2)13:03
ttxI was thinking jenkins-level manual triggering.13:04
ttxAlternatively we could overwrite content on docs.o.o/releasenotes13:04
AJaegerttx, I have access to docs.o.o.13:05
AJaegerso, if you want me to upload a page...13:05
ttxAJaeger: we'd have to run the job locally and push content13:05
AJaegerttx, which repos?13:05
AJaegerlet's see whether that can be done manually...13:05
ttxlet me see what the job does13:05
AJaegerI know what the job does ;)13:06
ttxah, good then :)à13:06
AJaegertox -e releasenotes; then upload releasenotes/something to the right place13:06
ttxWe need at least aodh, barbican, cinder, glance, manila13:06
ttxI think zaqar has no release notes anyway, checking13:06
AJaegerAnd just the mitaka pages, correct?13:06
ttxyeah, then upload at least mitaka.html13:07
jheskethI'm just looking to see which jobs are in the post queue.. it could be the others are harmless13:08
ttxwe also have mistral, murano, murano-agent, muranbo-dashboard, searchlight and senlin but maybe can wait a bit more for them. Depending on how painful it is to run one13:08
ttxyou can skip murano-agent it doesn't seem to have notes13:09
* AJaeger tries aodh to see what needs to be done...13:09
ttxhmm, we have manila in queue now so no need to do it13:10
*** TravT_ has quit IRC13:10
kzaitsev_mbweird, I was pretty sure we fixed a couple of bugs in murano-agent, maybe we've missed adding notes there, though =/13:11
ttxSo... aodh, barbican, cinder glance, mistral, murano, murano-dashboard, searchlight and senlin13:11
jheskethso I just checked aodh and the jobs in the post pipeline are :13:11
jhesketh- '{name}-branch-tarball'13:11
jhesketh- '{name}-docs'13:11
jhesketh- '{name}-upstream-translation-update'13:11
jhesketh- '{name}-releasenotes'13:11
AJaegerttx, done13:11
jhesketh(clearly the others might be different)13:11
jheskethbut they don't seem too bad13:11
*** TravT has joined #openstack-release13:11
AJaegerjhesketh: I'm on it - easy enough to do ;)13:11
ttxyeah, simpler to patch docs13:11
jheskethAJaeger: awesome, thanks :-)13:12
dimsjhesketh : AJaeger : want me to get the SHA's for the tags we cut?13:12
ttxAJaeger: looks good13:12
ttxdims: I don't think that will be necessary13:13
dimsok. easy enough to find if needed (
AJaegerdims: I'm just building from head of the files13:14
dimsAJaeger : ok13:14
AJaegerThanks to gvfs, I have the ftp tree mounted locally and can script and copy easily ;)13:14
dimsAJaeger : ah tricks of the trade :)13:15
ttxAJaeger: that leaves sahara, trove and manila but they are in the POST queue so hopefully will make it soon13:16
AJaegerdims, I didn't know that gvfs worked so nicely ;)13:16
amrithdims, ttx, when will mitaka revert to being just a normal stable branch? i.e. ACL change to be stable branch rules like liberty?13:16
ttxamrith: we'll ask infra to merge the ACL revert just after release is completed13:16
amrithttx, thx13:17
ttxshould be ok in ~1 h13:17
dimsamrith : watch this
patchbotdims: patch 300667 - openstack-infra/project-config - Give project teams access to approve stable/mitaka...13:17
*** daemontool has quit IRC13:17
* dims waiting to RT ttx's tweet :)13:20
openstackgerritMerged openstack/releases: update status of mitaka and newton
*** doug-fis_ has quit IRC13:23
ttxoh hmm we missed something13:23
* ttx fixes13:23
AJaegerttx, dims: aodh, barbican, cinder glance, mistral, murano, murano-dashboard, searchlight and senlin releasenotes done13:24
AJaegerttx, let me do sahara, trove and manila quickly as well...13:24
dimsAJaeger : checking...thanks!13:24
openstackgerritThierry Carrez proposed openstack/releases: Mark mitaka as being released, rather than planned
dimsAJaeger : cinder shows rc1 (
ttxdims: ^13:25
AJaegerdims, reload, please13:25
amrithAJaeger, won't the trove jobs for translation have that zanata failure I emailed you about yesterday?13:25
AJaegeramrith: the zanata failure is fixed...13:25
dimsAh. cache. thanks13:25
ttxAJaeger: let me know when all done and I'll check them all to make sure we didn't miss any13:26
amrithah yes, I see it now,
*** mriedem has joined #openstack-release13:26
ttxdims: review please13:26
dimsy on it13:26
dimsttx : +2, please +A when ok13:27
ttxa bit at the bottom of the check queue unfortunately13:27
AJaegerttx, dims: Done.13:27
dimsAJaeger : all the pages i saw were good :)13:27
ttxchecking them all now13:27
*** daemontool has joined #openstack-release13:28
dimsAJaeger : trove is rc1 ( guessing its in queue?13:28
ttxAJaeger: missing trove13:28
ttxit's in queue13:28
AJaegerttx, you're right - still building ;(13:29
* amrith looks for trove13:29
ttxbut not anywhere near the top13:29
AJaegerthat tox trove releasenotes takes ages - compared to other projects ;(13:30
amrithyes, trove is RC2 and still in queue . a long way to go.13:30
dimsttx : 30 mins to go13:30
amrithAJaeger, anything I could do to help with that? I can speak sternly to it :)13:30
dimsi mean for the deadline13:30
amrithseriously, for next time around ...13:30
AJaegeramrith: cut down the list of requirements ;)13:30
AJaegeramrith: no worries - if you need them all, you need them...13:31
amrithfor the releasnotes job, I could give you another install-command with few requirements13:31
amriththat would be quick13:31
AJaegeramrith: what we do here today should be the exception13:31
amriththe releasenotes job shouldn't need all of the projects requirements13:31
amrithoh, I was thinking for newton, I'd just strip out requirements.txt and test-requirements.txt from the install command for the releasenotes job13:32
dimsamrith : also, this one was a snafu with a reno sphinx doc thing. causing us to request regenerating them13:32
amrithif we don't already do that13:32
ttxamrith: yeah, that was a one-time fail13:33
AJaegeramrith: I'm not sure it's worth maintaining separate requirements list for these cases.13:33
AJaeger done13:34
amrithhmm, ok. I'll request a post-mortem specific to trove and see how we can improve. for now, I think it is steady-on-the-wheel, it'll be done soonish13:34
AJaegerttx, dhellmann, dims: One nit for those perfectionists that some of us are: Every project orders the list of releases differently at their index page - compare and or
* AJaeger won't send patches for these13:36
dimsAJaeger : ah, good eye :)13:37
*** bdemers has quit IRC13:40
dhellmannAjaeger: we can straighten that out for Newton, thanks13:41
dhellmannttx: I'm ready to send the announcement13:41
ttxdhellmann: hi!13:41
AJaegerttx, shall I apply the change for by hand on the server?13:41
dimsAJaeger : lots of variations -
ttxSo we are missing one thing13:41
ttxAJaeger: no13:41
dhellmannBrought to you from the ancient roman ruins in vienne13:42
ttxAJaeger: oh wait13:42
ttxAJaeger: yes, that would be nice13:42
dimsAJaeger ++13:42
ttxdhellmann: we still had "Projected reelase date" on the mitaka page13:42
AJaegerwait - that'S on, isn't it? I cannot change that one ;(13:42
ttxdhellmann: I have a change in to fix that but it's lost in check queue13:42
patchbotttx: patch 302753 - releases - Mark mitaka as being released, rather than planned13:42
dhellmannIs the date wrong? :)13:43
dimswe could ask infra to promote13:43
ttxThe date is right. It's just that it should say "originally released"13:43
ttxdims: bah, it will get in eventually13:43
dhellmannYeah I see the patch now13:43
ttxand we need trove relnotes fixed too13:43
ttxAJaeger: how is that going ?13:44
ttxoh, done now13:44
dhellmannThanks for catching that. We should put that into a data file to about repetition13:44
dhellmannHow did the relno rebuilds go?13:44
ttxdhellmann: so we could technically hold until the last minute to reduce the window where that line is slightly incorrect, but we'll still have a window13:45
*** bdemers has joined #openstack-release13:45
AJaegerttx, see backscroll - published13:45
ttxso fire at will13:45
ttxdhellmann: they went well, just didn't cover everything. We had a pip mirror refresh issue13:45
dimsgo for it13:45
ttxso reno 1.6.2 only appeared around 1pm13:45
dimsAJaeger : fixed a bunch by hand13:45
dhellmannOh wow13:45
dimsbandersnatch was stuck13:46
dhellmannThanks ajaeger13:46
ttxyeah AJaeger ended up running the job and scping to docs13:46
dimsjhesketh helped unclog that13:46
* dhellmann owes the team a round of drinks for today13:46
dhellmannSo, we're ok to announce?13:46
ttxyeah, if you want to mention thanks on the announce, don't forget AJAeger and jhesketh13:46
ttxdhellmann: +2A13:47
nikhilttx: this one should be in the merge queue
patchbotnikhil: patch 302618 - glance - Removed one extra enter key13:47
ttxnikhil: we workedaround the issue, thanks anyway13:47
nikhilah cool13:47
*** bdemers_ has joined #openstack-release13:48
*** bdemers has quit IRC13:49
fungittx: jhesketh: we can retrigger individual jobs too. i've rerun just the releasenotes jobs for dhellmann before. it's a different mechanism than enqueuing through zuul (we have to put work requests directly into gearman instead) but it's not too hard13:51
dhellmannEmail sent and blog post published13:52
jheskethfungi: right, I had also thought you could probably invoke them from jenkisn too.. however I didn't bother to look further as I imagined setting all the zuul variables would be fiddly13:52
dimsdhellmann : ttx : AJaeger : everyone : congrats!13:53
dhellmann#success we have declared Mitaka released, congratulations everyone!13:53
openstackstatusdhellmann: Added success to Success page13:53
AJaegerdhellmann: congrats!13:53
ttxdhellmann: yay13:54
dhellmannThanks to everyone who chipped in today!13:54
fungijhesketh: i have prior runs of for releasenotes jobs in my command history on zuul, so pretty quick for me to repeat13:54
fungiif it becomes necessary13:54
jheskethfungi: ah neato13:54
fungidid it a bunch when we were first testing out that job13:54
fungithere were more than a few false starts with it ;)13:54
AJaegerfungi, thanks - not needed anymore. I uploaded myself for the few projects we still needed it...13:54
dhellmannOk, I'm going to find a glass of wine to celebrate.13:55
ttxdhellmann: go back to drinking Coteaux du Lyonnais13:55
* dhellmann departs13:55
*** kzaitsev_mb has quit IRC13:56
ttxlet the PR fest start13:56
ttxfungi: you can get merged now13:57
patchbotttx: patch 300667 - openstack-infra/project-config - Give project teams access to approve stable/mitaka...13:57
dimsttx : dhellmann : that was fun! thanks13:57
fungittx: can do13:57
fungisorry i wasn't around earlier. mother in law's birthday yesterday so we went on a brief road trip to visit overnight so am in the car on my way home right now13:58
jheskethdhellmann: thank you for the kind mention in the announce :-)13:58
jheskethit's really my pleasure to help13:58
openstackgerritSean McGinnis proposed openstack/releases: Release os-brick 1.3.0
smcginnisAnd let the fun begin... :]14:00
ttxfungi: between AJaeger and jhesketh we managed to pass all the hurdles14:01
fungithat's because you and they are awesome14:02
fungigreat work everyone!14:02
smcginnisYes, thanks everyone. All your work to make things happen is very much appreciated.14:03
openstackgerritMerged openstack/releases: Mark mitaka as being released, rather than planned
AJaegerdhellmann: do you want to un WIP now?14:06
patchbotAJaeger: patch 298866 - openstack-infra/project-config - update tag ACLs for all official projects14:06
ttxfungi: we did just merge but the change didn't kick in on releases.o.o14:06
*** sdake has joined #openstack-release14:07
ttxjob looks alright14:07
ttx should show "Originally Released" now14:07
AJaegerttx, the job is probably in the post queue14:08
*** hongbin has joined #openstack-release14:08
fungiyeah, that should have high priority14:09
ttxignore me14:09
AJaegerrelease-post it is - yeah, wait a minute ;)14:09
ttxfungi: the bandersnatch situation this morning was interesting14:09
ttxit was stuck since yesterday14:09
fungioh, i missed that in scrollback14:10
ttxtalk about Murphy's law14:10
AJaegerttx, reload, plesae ;)14:10
ttxwe had a reno change urgently needed14:10
ttxfungi: he had to clear the todo file that bandersnatch leaves out in case of interrupted work14:10
AJaegerotherwise we would have had much less fun here ;)14:10
fungithere used to be a bug in bandersnatch where it would hang indefinitely, but i thought that was fixed sometime last year14:10
* AJaeger will be back later...14:11
fungiremoving the todo files firces a complete bandersnatch resync, takes many hours to complete normally14:11
ttxit wasn't hanging. It was repeatedly failing to process changes14:11
*** sdake has quit IRC14:12
fungigot it, that's often due to having gotten a very weird response from (or more likely their fastly cdn)14:12
ttx9:33+ in this channel if you want to follow what we did14:13
fungithe sun is right in my lap. making it virtually impossible to see the screen14:13
fungioh, and my signal keeps dropping completely. that doesn't help14:13
ttxnothing urgent at all14:13
fungiyeah, i'll take a closer lok when i get home, if it's all working as intended again14:14
fungialso planning to help clarkb replace the nodepool server at that point14:14
amrithguys, I see that merged. does something else have to happen before project groups get to +2 on stable-mitaka? I'm still getting only +1 and,access still has the earlier settings.14:15
patchbotamrith: patch 300667 - openstack-infra/project-config - Give project teams access to approve stable/mitaka... (MERGED)14:15
ttxamrith: it may take a few to kick in, ask in #infra maybe14:16
fungiheh, my interface says it's on 2g with one bar signal strength14:16
amrithttx, thx14:16
fungimosh is awesome14:16
amrithwill wait a bit and then ask on infra if it doesn't clear. enjoy the bubbly14:16
ttxmosh is awesome indeed.14:17
fungiamrith: gerrit aqcl updates are not instantaneous. takes upwards of 30-45 minutes from patch merge until application14:17
amrithfungi, thx14:18
ttxAlright, all planned release day tasks completed14:18
fungiamrith: and even then, your browser may cache the gerrit vote dialogues, needing force-refreshes on the vote screen14:21
* fungi typed that several minutes ago, just took that long for the bits to get through14:21
fungithe north carolina coastal plain is a wasteland of cellular dead zones14:21
*** sdake has joined #openstack-release14:21
amrithfungi, I'm a windows user. I even rebooted my laptop :)14:21
fungiamrith: my condolences14:22
amrithhey, it's at least better than the rooten fruit ...14:22
amrithI get to do custom builds when I want to14:22
amrithso I'm happy14:22
*** Qiming_ has joined #openstack-release14:32
*** Qiming has quit IRC14:32
TravTttx dhellmann we have a critical bug we've fixed on searchlight master and are going to backport to stable. is it too late to do a rc3 for mitaka?14:36
ttxTravT: mitaka is out already14:37
ttxso yes, it's too late. You can do a point release though14:37
TravTcan point releases happen at any time? or is there a regular timing for that?14:38
ttxat any time.14:38
ttxJust request them using the releases repo14:38
TravTok, that's good to know.14:38
ttxHm, looks like that could be made a lot clearer in
ttxTravT: thx!14:39
*** thinrichs has joined #openstack-release14:40
dimsprometheanfire : wow! nice :)14:48
prometheanfiredims: :P14:50
prometheanfiredims: consisted of copying the ebuild and editing three lines14:50
dimsprometheanfire : way more simpler than what we had to do :)14:51
prometheanfirewe can build debs/rpms too iirc14:51
prometheanfireI considered packaging it all in deb and giving people that as a joke14:51
*** kzaitsev_mb has joined #openstack-release14:52
*** sigmavirus24 is now known as sigmavirus24_awa14:55
*** sigmavirus24_awa is now known as sigmavirus2414:55
*** Qiming_ has quit IRC14:57
*** kzaitsev_mb has quit IRC14:58
*** Qiming has joined #openstack-release14:59
*** thinrichs has quit IRC15:01
*** thinrichs has joined #openstack-release15:02
openstackgerritMerged openstack/releases: newton: release python-novaclient 3.4.0
*** armax has joined #openstack-release15:30
*** Qiming has quit IRC15:41
*** kzaitsev_mb has joined #openstack-release15:46
*** pcaruana has quit IRC15:53
clarkbttx: dims: to confirm mitaka is completely done and infra can replace the nodepool servers at any time without interrupting release tasks right?15:55
*** pabelanger has joined #openstack-release15:56
*** thinrichs has quit IRC15:57
*** thinrichs has joined #openstack-release15:58
pabelangerGreetings, over in -infra were looking to update the script to add support for puppet modules: Would just like to confirm this is okay with the release team as not wanting to affect mitaka release process15:58
clarkbpabelanger: lol everyone queuing up to ind out if release team is finished so we can break things/party15:59
pabelangerclarkb: break all the things!15:59
clarkbin my case its already broken, so mostly doing damage control :)15:59
*** thinrichs has left #openstack-release16:00
*** thinrichs has joined #openstack-release16:00
clarkblooks like there are a bunch of requirements updates16:00
pabelangerclarkb: I new it was release day, when I seen 10 patches is the gate queue for openstack/requirements :)16:00
clarkblikely related to the release that should go through soon16:00
AJaegerclarkb, pabelanger : dhellmann is trinking some wine to celebrate - the release should be done based on the discussions from 2 hours earlier16:00
clarkbAJaeger: ya I think we let these requiremnts changes merge first then we can proceed to fix/break things16:01
pabelangerYa, I am in no rush to land the patch, we can ask EmilienM to want another day or so16:01
EmilienMno rush at all16:01
clarkboh it should be done in the next half hour16:02
*** armax has quit IRC16:02
*** armax has joined #openstack-release16:02
ttxclarkb: yes16:03
ttxnobody screamed yet16:03
ttxok, I'll be back in a few hours to check things out16:03
EmilienMdhellmann: hi Doug, we might want to review again since Mitaka is out. I'm also pushing efforts to have tarball like other projects, in - any feedback is welcome, thanks16:07
patchbotEmilienM: patch 297382 - governance - Update release tags for some Puppet OpenStack modules16:07
*** dtantsur is now known as dtantsur|afk16:14
*** vgridnev has quit IRC16:21
*** vgridnev has joined #openstack-release16:21
*** cgalan has joined #openstack-release16:29
*** david-lyle has quit IRC16:36
*** daemontool has quit IRC16:44
*** costingalan has joined #openstack-release16:52
thinrichsI just saw a patch tagging 3.0.0 on Congress.  I don't understand why that necessary since there's already a 3.0.0 tag that Doug added.16:55
patchbotthinrichs: patch 302558 - congress - Merge tag '3.0.0'16:55
*** stevebaker has quit IRC17:00
*** vgridnev has quit IRC17:04
*** david-lyle has joined #openstack-release17:06
*** pcaruana has joined #openstack-release17:08
*** david-lyle has quit IRC17:11
AJaegerthinrichs: see
dimsclarkb : +1 go for it17:16
thinrichsAJaeger: thanks—missed that I'm assuming b/c not marked with [all].17:17
*** david-lyle has joined #openstack-release17:17
thinrichsAJaeger: follow up.  I'm getting an error for that patch b/c pbr wants setup.cfg to say version is 3.0.117:18
AJaegersetup.cfg should not have a version ;(17:18
AJaegerremove it.17:18
AJaegerwe use pbr and that gets the version from git.17:19
AJaegerthinrichs: there was some discussion about this half a year ago, I don't have references.17:19
thinrichsAJaeger.  Will do.  Will I still be able to merge the 3.0.0 tag after the setup.cfg is fixed?17:19
AJaegerthinrichs: don't know - best discuss with release team17:20
thinrichsAJaeger: ok17:20
*** david-lyle has quit IRC17:23
openstackgerritamrith proposed openstack/releases: Please tag python-troveclient v2.2.1
openstackgerritamrith proposed openstack/releases: Please tag python-troveclient v2.2.0
AJaegeramrith: don't you want to have merged?17:35
amrithAJaeger, it looks open17:36
amrithor at least it should be merging17:36
amrithwhy do you ask?17:36
amrithit is for master17:36
AJaegeramrith: waiting for an approval in master of python-troveclient...17:36
AJaegeramrith: I noticed your change aboe and wondered whether you want releasenotes publishing done first - or afterwards17:37
amrithoh, longish explanation follows17:37
amrithshort answer, tagging the release can go ahead17:37
amrithI'm tagging at a hash that is some weeks old17:37
amrithto pick up (literally) one fix17:37
amrithI am (NOT) tagging the current HEAD in the branch17:38
AJaegeramrith: as long as you are aware of it, no problem from my side ;)17:38
amrithwell, don't count on it. I think I'm doing the right thing but history has shown that I'm often mistaken.17:38
amrithso, ...17:38
amrithif I shouldn't ask to tag something other than the head, please let me know.17:38
amrithwhich begs the question, why hasn't 301899  merged?17:39
amrithit isn't in zuul'land17:40
AJaeger310899 has a depends-on a change that is not merged17:40
AJaegeramrith: Asking to tag an older version is fine IMHO17:40
amrithAJaeger, yes, foudn that change17:41
AJaegerand I'm not part of release team, so can only give my private comments17:41
amrithI'll get it reviewed as well17:41
amrithI'll check with the release team on the tag request17:41
*** ihrachys has quit IRC17:41
amrithfolks, would some of you please look at it is holding up a change in infra right now17:49
patchbotamrith: patch 301894 - python-troveclient - Add reno for release notes management17:49
AJaegeramrith: wrong channel?17:50
amrithyes, wrong channel17:50
amriththanks AJaeger17:50
amrithmeant to put that in #openstack-trove (which I just did)17:51
*** kzaitsev_mb has quit IRC17:55
*** david-lyle has joined #openstack-release17:56
*** david-lyle has quit IRC17:56
*** david-lyle has joined #openstack-release17:58
*** vgridnev has joined #openstack-release17:58
*** TravT has quit IRC18:06
*** sdake_ has joined #openstack-release18:26
*** doug-fish has joined #openstack-release18:26
*** sdake has quit IRC18:26
*** cgalan has quit IRC18:31
*** costingalan has quit IRC18:31
*** cody-somerville has joined #openstack-release18:33
*** AJaeger has left #openstack-release18:38
*** TravT has joined #openstack-release18:42
*** njohnston has quit IRC18:49
openstackgerritSean McGinnis proposed openstack/releases: Add Cinder deadlines for Newton
*** thinrichs has quit IRC19:06
*** nikhil has quit IRC19:07
openstackgerritMatthew Treinish proposed openstack/releases: Add tempest 11.0.0 release
*** nikhil has joined #openstack-release19:11
mtreinishdims, dhellmann: ^^^ I assume you wanted me to do that again19:12
dimsmriedem : tempest release? i'll look in a bit19:12
*** david-lyle_ has joined #openstack-release19:13
*** vgridnev has quit IRC19:14
mtreinishdims: wrong Matt :)19:14
mtreinishdims: I already pushed the tag and stuff, but dhellmann asked me to update the yaml last time19:14
dimslol. sorry mtreinish :) been a long day19:14
mtreinishheh, no worries19:14
dimshad to work EU time today :)19:15
*** david-lyle has quit IRC19:15
*** sigmavirus24 is now known as sigmavirus24_awa19:17
*** david-lyle_ has quit IRC19:19
*** ihrachys has joined #openstack-release19:24
*** david-lyle has joined #openstack-release19:29
*** david-lyle has quit IRC19:31
*** david-lyle has joined #openstack-release19:31
*** thinrichs has joined #openstack-release19:34
* ttx pops in19:36
openstackgerritSteve Martinelli proposed openstack/releases: release keystoneauth 2.5.0
*** david-lyle has quit IRC19:37
dimshey ttx : no fires on stable/mitaka :)19:38
*** thinrichs has quit IRC19:38
dimsttx : a few of us are looking at some stuff from master (fixtures, novaclient)19:38
*** thinrichs has joined #openstack-release19:39
*** TravT has quit IRC19:39
stevemardims: *cough* keystoneauth19:41
dims:) give me a few mins. dust settling down on a novaclient/horizon and another one about fixtures...19:42
ttxI'll wave my hands and look the other way19:43
dimsstevemar : gonna wait for the CI jobs
patchbotdims: patch 300965 - releases - release keystoneauth 2.5.019:43
dimsstevemar : same SHA, so we should be good19:43
stevemardims: i bumped the SHA to the latest in the most recent patch set19:44
dimsstevemar : ok so i'll wait to eyeball list-changes19:45
*** sdake_ is now known as sdake20:23
*** thinrichs has quit IRC20:39
*** thinrichs has joined #openstack-release20:40
*** mriedem has quit IRC20:46
dimsstevemar : looks good to me
dimsstevemar : i don't see anything that could be backward incompat so version number looks good20:50
stevemardims: totally forgot about it, yeah, looks fine, should be a small change set20:50
dimsshipping it stevemar20:52
stevemardims: can you poke the related requirements change too?20:52
stevemardims: ++20:52
dimsstevemar : once it ships20:52
openstackgerritMerged openstack/releases: release keystoneauth 2.5.0
*** david-lyle has joined #openstack-release20:56
*** pcaruana has quit IRC21:01
stevemardims: thank you sir21:12
dimsmy pleasure stevemar21:16
dimsstevemar : rebase please?
patchbotdims: patch 300967 - requirements - bump keystoneauth to 2.5.021:18
stevemardims: i guess we can wait til jenkins votes :)21:22
*** ihrachys has quit IRC22:05
*** doug-fish has quit IRC22:23
*** thinrichs has quit IRC22:40
*** gordc has quit IRC22:46
*** daemontool has joined #openstack-release22:48
*** thinrichs has joined #openstack-release22:50
*** sdake has quit IRC22:54
*** sdague has quit IRC22:57
*** sdake has joined #openstack-release22:58
*** hongbin has quit IRC23:05
*** Qiming has joined #openstack-release23:09
*** sdake has quit IRC23:12
*** mriedem has joined #openstack-release23:14
armaxdhellmann: ping23:37
dimsarmax : dhellmann is in EU TZ this week. anything i can help with?23:59
armaxah thanks23:59

Generated by 2.14.0 by Marius Gedminas - find it at!