15:00:03 <smcginnis> #startmeeting releaseteam
15:00:04 <openstack> Meeting started Fri Nov 17 15:00:03 2017 UTC and is due to finish in 60 minutes.  The chair is smcginnis. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:00:05 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
15:00:07 <openstack> The meeting name has been set to 'releaseteam'
15:00:11 <smcginnis> Ping list: dhellmann, dims, fungi, tonyb, lbragstad, ttx
15:00:29 <smcginnis> An hour earlier thanks to DST change.
15:00:44 <dhellmann> o/
15:00:58 <smcginnis> #link https://etherpad.openstack.org/p/queens-relmgt-tracking Tracking etherpad
15:01:01 <fungi> and this is why i do everything in utc ;)
15:01:07 <smcginnis> Morning dhellmann. How's the jetlag coming along?
15:01:18 <smcginnis> fungi: Smart. ;)
15:01:44 <dhellmann> my wife is a bit of a night owl, so she likes it when I travel west and come home able to stay up later than usual. I'm a bit less thrilled, since I still need to wake up early...
15:01:47 <smcginnis> fungi: I would do metric too, but the impedence mismatch with everyone around me would be too much. :D
15:01:59 <fungi> also there's no good metric time standard
15:02:03 <smcginnis> dhellmann: Similar situation here.
15:02:36 <ttx> o/
15:02:39 <smcginnis> #topic Cycle progress
15:03:02 <smcginnis> We are currently R-15 in the cycle.
15:03:16 <smcginnis> Just a few short weeks away from q-2.
15:03:50 <ttx> Is there any standing issue on the release infra side?
15:03:58 <ttx> I've seen a few threads open
15:04:05 <smcginnis> We've had a few release notes failures lately, but I think those may be resolved now.
15:04:09 <smcginnis> Or just about resolved.
15:04:17 <smcginnis> fungi: Are you aware of anything else?
15:04:20 <smcginnis> dhellmann: ^^
15:04:22 <fungi> release notes job discussion underway in #-infra right now
15:04:40 <dhellmann> EmilienM just posted about a failure related to oslosphinx in #openstack-release
15:04:51 <dhellmann> I think the fix there is to not use the deprecated lib and update to the new theme
15:04:53 <fungi> mostly around dropping tox from the release notes build, in preparation for doing the same for docs builds
15:05:13 <dhellmann> did we announce that the release notes job was being changed?
15:05:18 <fungi> (per mordred's updates to the pti)
15:05:33 <smcginnis> dhellmann: I think cinder-specs still has a patch out for that. Probably others yet.
15:05:49 <fungi> dhellmann: not sure, it came up in the discussions about trying to fix failing release notes builds for horizon plugins?
15:05:57 <dhellmann> smcginnis: I think that's a different thing?
15:06:11 <dhellmann> yeah, we talked in channel but I wasn't sure if there was an email about it
15:06:11 <smcginnis> dhellmann: Sorry, I was referring to the discussion in -releases
15:06:28 <smcginnis> dhellmann: I don't believe there was an email about the release notes change.
15:06:39 <smcginnis> I was actually surprised we merged that already.
15:06:42 <dhellmann> ok, we should do that
15:06:56 <dhellmann> yeah, that made it through faster than I expected
15:07:06 <smcginnis> There are at least a series of patches making changes so services don't need to be installed to run reno.
15:08:30 <fungi> that much got discussed on the ml, i thought. just not formally announced
15:08:48 <dhellmann> ok, I'll look for that thread
15:09:07 <smcginnis> Who would be the best to follow up on that thread and make it official? Infra or release team?
15:09:15 <dhellmann> we probably should have written a new job, tested it with some projects, then moved everyone to it
15:09:16 <fungi> around determining that installing projects to identify their version string wasn't necessary for release notes builds
15:09:40 <smcginnis> dhellmann: That would have been a smoother transition.
15:09:59 <smcginnis> As we've seen, some folks are getting annoyed that things that were working before are suddenly broken.
15:10:05 <fungi> i wasn't following the transition--been away from the computer a lot this week. i guess that's not how it was done?
15:10:09 <smcginnis> I can't imagine why.
15:10:53 <dhellmann> fungi : it doesn't seem so. I still don't fully understand the v3 stuff so it wasn't obvious at the time, but if we're having projects failing for new reasons I assume the existing job was changed
15:11:03 <fungi> i agree a trial implementation would have been nice
15:11:09 <AJaeger> fungi mentioned release-notes discussions? Anything for me to chime in?
15:11:31 <dhellmann> AJaeger : we were wondering if the ramifications of the job rewrite were announced before that patch landed
15:11:38 <dhellmann> we're seeing people with new questions about new failures in the job
15:12:05 <fungi> AJaeger: mostly just some projects were surprised by the release notes jobs changing when they had been previously working for them. i guess we could have tried a new job just on the projects which were already broken to make things smoother. worth considering for next time
15:12:07 <AJaeger> dhellmann: We had temporary failures but fixed them directly.
15:12:27 <dhellmann> AJaeger : https://bugs.launchpad.net/tripleo/+bug/1732815
15:12:27 <openstack> Launchpad bug 1732815 in tripleo "CI: releasenotes job broken with Could not import extension oslosphinx (exception: No module named oslosphinx) " [Critical,Triaged]
15:12:34 <AJaeger> The check/gate worked for the repos - but as dhellmann's emails showed, the post/release publishing failed ;(
15:12:47 <AJaeger> dhellmann: I fixed that ;)
15:13:17 <dhellmann> AJaeger : ok. EmilienM came to #openstack-release at the top of the hour to ask about it, so maybe he hadn't seen the fix yet
15:13:37 <EmilienM> where is the fix?
15:13:49 <fungi> hindsight and all, but we probably would have found that bug even if we were just working through a replacement job for the horizon plugins, without also inflicting the bug on projects which already had working builds
15:13:53 <dhellmann> we need someone to write an email to the list describing the job change
15:13:53 <AJaeger> dhellmann: but that failure in 1732815 was not known before to me
15:13:59 <dhellmann> fungi : yeah
15:14:19 <dhellmann> ah, so the issue with tripleo's job isn't fixed
15:14:35 <AJaeger> dhellmann: I'm wrong - I didn't fix python-tripleoclient yet.
15:14:46 <AJaeger> We could install oslosphinx as well in the releasenotes job to fix it
15:14:50 <dhellmann> ok. switching the releasenotes build to use openstackdocstheme instead of oslosphinx should fix the problem.
15:14:54 <fungi> does python-tripleoclient do something unusual?
15:15:03 <dhellmann> well, oslosphinx is deprecated and no one is supposed to be using it any more
15:15:07 <AJaeger> dhellmann: exactly, that should fix it
15:15:14 <smcginnis> AJaeger: We may want to do that until everyone has switched over. I know not all had yet.
15:15:16 <fungi> aha, so most projects were already on openstackdocstheme
15:15:24 <dhellmann> updating that was part of the migration effort last cycle
15:15:36 <dhellmann> but yeah, as smcginnis points out, maybe that's a good temporary measure
15:15:49 <fungi> it's entirely possible that if all the horizon plugins we were working on fixing builds for were already using openstackdocstheme we wouldn't have caught 1732815
15:16:01 <smcginnis> I'm aware of at least two now that have not completed the switch. Likely more.
15:16:02 <dhellmann> I see 320 repos using oslosphinx, but it's not clear how many have releasenotes
15:16:46 <dhellmann> AJaeger : maybe the thing to do is to try installing the test-requirements.txt for a project, but ignore the failure? like we do with the project's requirements
15:17:36 <fungi> that seems like a reasonable stepping stone to the docs-requirements.txt mandated by the pti
15:18:26 <dhellmann> I can't find the patch that redefined the job, does someone have that link handy?
15:19:09 * AJaeger just pushed " https://review.openstack.org/521104 ensure-reno: Install oslosphinx"
15:19:28 <AJaeger> dhellmann: I56909152975f731a9d2c21b2825b972195e48ee8, let me get a link...
15:19:43 <dhellmann> AJaeger : I just commented on 512104
15:20:00 <AJaeger> dhellmann: https://review.openstack.org/#/c/520362/
15:20:07 * mordred waves
15:20:48 <dhellmann> do we agree on the approach now?
15:21:03 <smcginnis> dhellmann: Can you restate it to make sure we all agree on the same thing?
15:21:12 <AJaeger> dhellmann: I think it works fine with just oslosphinx, nothing else should be needed to get installed
15:21:22 <dhellmann> update the job to optionally install test-requirements.txt, like we do with requirements.txt
15:21:37 <dhellmann> AJaeger : we need to stop making assumptions about these jobs. we keep breaking things.
15:22:04 <AJaeger> dhellmann: so, you would install *only* test-requirements.txt?
15:22:36 <dhellmann> we should make the new job try to reproduce closely what the old job was doing, but be resilient if installing something fails
15:22:50 * mordred likes that - or also maybe optionally install docs/requirements.txt if it's there, if not install test-requirements.txt if it's there?
15:22:51 <smcginnis> ++
15:22:52 <dhellmann> the issue with the horizon/neutron jobs was they couldn't install a dependency
15:23:13 <dhellmann> sure, just loop over a list of all of these files and try to install based on whatever ones are found
15:23:21 <mordred> since docs/requirements.txt files, once they exist, should contain sphinx/openstackdocstheme requirements for the projects where appropriate
15:23:29 <mordred> cool. I can get that updated in the stack
15:23:50 <dhellmann> then we need to start a mailing list thread explaining to people what is going on and what changes they can make to anticipate the end state for the job
15:24:08 <AJaeger> note: 10 repos have doc/requirements.txt, 5 fuel and 5 other
15:24:09 <dhellmann> and we probably need to leave cleaning that job up so it doesn't install extra stuff until after the first of the year, if not the end of the cycle
15:24:14 <fungi> down-side to making things install everything whether or not they need it is that it becomes increasingly hard to clean that up later
15:24:35 <dhellmann> we have the patches AJaeger has proposed to stop setting the version
15:24:42 <dhellmann> that was the main reason projects needed themselves installed
15:24:45 <AJaeger> mordred: I can abandon 521104  and let you do it cleanly...
15:24:47 <fungi> like, we're getting started removing the deprecated bindep fallback list finally, and we really have no way to know what that's going to break
15:24:57 <dhellmann> we still have some using oslosphinx and we can address those
15:25:02 <ttx> it's still ok to approve jobs now (and let random releasenotes job fail), or should we wait to approve in some cases?
15:25:13 <dhellmann> it should be ok to release things
15:25:18 <dhellmann> the release notes build every time any patch merges
15:25:24 <ttx> right
15:25:33 <dhellmann> it's just that we only see the failure notices for jobs in the release queues
15:25:44 <smcginnis> Good point.
15:25:45 <mordred> related to this, but also a different topic to tee up potentially - is publishing server projects to PyPI - which would also remove some of the weirdness with horizon and neutron plugin repos
15:25:59 <dhellmann> do you mean just publish them all?
15:26:14 <smcginnis> There is definitely a group of users who do want them published.
15:26:26 <mordred> well, yes - that's the overall thing I'd like to advocate - but we have a potential weirdness issue with keystone
15:26:31 <dhellmann> I would be fine with that. I think ttx has given reasons not to in the past?
15:26:41 <AJaeger> we can now merge https://review.openstack.org/520763 - we don't need the horizon/neutron variants of releasenotes anymore and thus have the same setup for post and check - the extra requirements in post are the problem
15:26:49 <mordred> in that there is a keystone on pypi already - even though it seems abandoned - we need to contact its maintainer and see if we can take it over
15:26:55 <dhellmann> before we get too far into that, who is going to write the email about what's going on with the release notes job?
15:26:56 <fungi> i'm unsure whether publishing them to pypi solves that, because the plugins/drivers want to test and build against development branches of those services and not lag a release cycle behind them
15:27:02 <ttx> there was a discussion in a far past that it was encouraging installing from pip
15:27:14 <mordred> fungi: that's all handled by tox-siblings just fine though
15:27:22 <fungi> ahh, good point
15:27:23 <ttx> but I'd say with "some" being present that ship has sailed
15:27:24 <dhellmann> fungi : good point
15:27:32 <ttx> or that bridge was burnt
15:27:32 <mordred> ttx: ++
15:27:39 <ttx> or whatever analogy works best
15:28:00 <mordred> ttx: also there was the 'pip install nova does not provide a working nova' argument, but again, I think ship-sailed/bridge-burnt
15:28:03 <ttx> basically we have 3 things
15:28:18 <ttx> things that were at some point published and look weird on Pypi
15:28:25 <ttx> things that were always published
15:28:26 <dhellmann> if we keep the "do not publish to pypi" job we can move projects as needed
15:28:31 <fungi> `pip install openstackclient` doesn't provide a working openstackclient either, because you still need to configure it
15:28:33 <ttx> things that were never published
15:28:40 * AJaeger needs to really step out now...
15:28:47 <fungi> thanks AJaeger!
15:28:50 <ttx> We tried to clean up 1s and remove 3s
15:28:50 <dhellmann> thanks, AJaeger
15:29:00 <mordred> dhellmann: ++ - I think that's the way forward regardless - if we're ok with it as an overall approach, we could start publshing horizon and neutron and at least unstick that stuff
15:29:04 <ttx> but there was always a good reason to keep some 3s
15:29:24 <dhellmann> fwiw, at Dreamhost we installed all of the services with pip then built our own debs of the results
15:29:25 <smcginnis> Never published?
15:29:45 <smcginnis> ttx: Do you mean 2's?
15:29:56 <fungi> mordred: how does tox-siblings handle projects with different names on pypi than their repo names? (or does it?)
15:29:57 <ttx> would there be a way to explain that that it might not result in a functional install and you should rtfm?
15:30:09 <fungi> e.g., keystone
15:30:19 <ttx> see "things" mentioned above
15:30:25 <dhellmann> using tox-siblings requires a job variant, doesn't it?
15:30:26 <mordred> fungi: it reads the setup.cfg files of the repos
15:30:31 <ttx> 2= things that were always published
15:30:31 <fungi> ahh, okay
15:30:55 <smcginnis> ttx: Right, you were saying there are good reasons to keep some things never published?
15:30:58 <mordred> dhellmann: oh - yes indeed - so it actually won't solve the releasenotes job issue that AJaeger's version patches are solving
15:31:01 <dhellmann> we were trying to avoid having job variants for the release notes jobs
15:31:03 <ttx> smcginnis: oh, yes. sorry
15:31:19 <ttx> smcginnis: always good reason to keep things published
15:31:20 <dhellmann> mordred : right
15:31:41 <dhellmann> so we've drifted into 2 discussions without finishing the first
15:31:44 <ttx> so rather than flight it, maybe we should just publish them all for consistency
15:31:51 <dhellmann> who is going to write the email explaining the changes to the release notes job?
15:31:54 <ttx> fight*
15:32:01 * ttx shuts up
15:32:03 <mordred> dhellmann: yah - I think the current stack will fix the variants issue - I apologize for context-shift - the variants need in releasenotes just reminded me of the horizon/neutron pip weirdness issue
15:32:08 <smcginnis> ttx: That would simplify a lot of things, even if the results aren't always "useful".
15:32:33 <mordred> dhellmann: I can write a draft if you like when I'm done with this patch stack
15:32:36 <fungi> i would volunteer to write that announcement, but i wasn't around for the changes and didn't review them so expect someone who was involved may do a better job of explaining
15:32:39 <dhellmann> mordred : thank you
15:32:42 <smcginnis> dhellmann: I was thinking AJaeger would be a good one for that, but maybe someone else from infra would be willing?
15:32:47 <smcginnis> mordred: Thanks
15:33:14 <smcginnis> #action mordred to write draft email explaining release notes job changes
15:33:37 <dhellmann> do we need to discuss the change to publish more projects to pypi?
15:33:48 <smcginnis> Second item - publishing all to pypi - what do we need to move forward there?
15:33:50 <dhellmann> if we start changing jobs, we will break some validation in the releases repo
15:34:03 <dhellmann> so we need to announce the change in advance and probably change that validation logic to be more flexible again
15:34:13 <mordred> ++
15:34:13 <dhellmann> we should also make it opt-in, I guess?
15:34:19 <dhellmann> let projects choose to publish themselves?
15:34:25 <mordred> that's probably a great start, yeah
15:34:28 <dhellmann> maybe go ahead and propose horizon and neutron
15:34:34 <mordred> and gives us time/flexibility to deal with keystone
15:34:39 <dhellmann> and then just allow whatever other projects want to do
15:34:48 <dhellmann> and the keystone team can sort their thing out, right
15:35:05 <fungi> opt-in initially, but we could say that in the future we expect to make it mandatory for projects used as dependencies by other projects at least
15:35:33 <dhellmann> sure, that would let us get rid of the constraints installer script, I think
15:35:38 <mordred> +100 to making it mandatory for projects needed as dependencies and banning the current neutron/horizon approach
15:35:57 <fungi> i.e., that we plan to stop allownig official deliverables to declare dependencies on things not published to pypi
15:36:05 <dhellmann> prometheanfire and dirk may be happy about that
15:36:50 <dhellmann> cool, so someone needs to write all of that up
15:36:53 * mordred would LOVE to get rid of tox_install.sh ... once we do, I also think writing a $something - maybe in the PTI, maybe elsewhere, which explains that it is our intent to not need scripts like that ... but we can deal with that later
15:37:11 <mordred> dhellmann: I can do that after the other email - it seems like this is my brainspace today :)
15:37:16 <ttx> wondering if we should not just do it
15:37:23 <ttx> as part of the release process
15:37:34 <fungi> i think dropping tox_install.sh will be predicated on us getting the local ansible for developers to run jobs situation well-trodden
15:37:36 <dhellmann> ttx: I'm not sure I see what you mean?
15:37:41 <smcginnis> So we would need to 1) update validation logic to be more flexible, 2) start ML thread on the plan to change things to publish, and 3) work with projects to not have implicit dependencies on non-published services?
15:37:55 <dhellmann> I can handle the validation logic change
15:38:00 <ttx> I find it confusing that only some of our services are there
15:38:14 <ttx> and that requires two code paths
15:38:30 <mordred> ttx: yah - I think we eventually want all of them to be there - it just may take a minute or two to get them there
15:38:37 <ttx> oh sure
15:38:41 <smcginnis> Yeah, for consistency sake, I think I would rather have all or nothing. Simplifies our tooling too.
15:38:42 <dhellmann> yeah
15:38:59 <fungi> and as discussed previously, we'd like to not break ci for those projects in the process
15:39:02 <dhellmann> but we know already that we have 1 project that we can't publish
15:39:08 <dhellmann> do we know that is the only one?
15:39:13 <smcginnis> #action dhellmann to update validation logic be more flexible
15:39:24 <fungi> there are at least two i know of which have name collisions on pypi
15:39:30 <ttx> arh
15:39:33 <smcginnis> #action mordred to draft ML post about pypi publishing plans
15:40:16 <ttx> oh well, maybe ad-hoc publishing is the solution then
15:40:24 <smcginnis> #info Need to look for and work through naming conflicts for other projects on pypi
15:40:27 <ttx> i.e. continue having two options
15:40:57 <smcginnis> Maybe all packages should get prepended "openstack-".
15:40:59 <dhellmann> smcginnis : https://review.openstack.org/521115 should cover my piece
15:41:02 <smcginnis> openstack-keystone
15:41:09 <fungi> keeping the repo name and module name unchanged but using a different package name on pypi to avoid collisions should be reasonably tractable
15:41:16 <smcginnis> dhellmann: That was quick!
15:41:30 <ttx> 2-minute rule
15:42:00 <dhellmann> fungi : good point, we could just change the dist name for keystone to openstack-keystone
15:42:11 <ttx> fungi: ok
15:42:14 <fungi> right, exactly that
15:42:19 <dhellmann> although the python package name would still collide
15:42:32 <fungi> mordred says tox-siblings already handles that case by parsing setup
15:42:37 <ttx> so maybe the idea is to have them all there, and if name collision on first upload use an alias
15:42:42 <smcginnis> Less likely someone would want to install both openstack-keystone and keystone, right?
15:42:56 <dhellmann> I mean if someone did pip install keystone and pip install openstack-keystone they would be trying to write to the same directory
15:43:16 <dhellmann> I would hope, if "not-openstack-keystone" is deprecated that no one is using it
15:43:22 <dhellmann> but who knows
15:43:38 <dhellmann> I guess if they are using it, they already have the path collision problem
15:43:58 <smcginnis> venv or containers FTW.
15:44:06 <fungi> we might be able to do some hacky install_requires tricks with a greedy version exclusion to make it not install them together
15:44:41 <smcginnis> Should we stew on this part for a bit. Not sure we are going to solve it in the next 15 minutes.
15:44:56 <fungi> yeah, no need to design in-meeting
15:45:27 <dhellmann> yeah, we can also proceed under the assumption that we'll solve that somehow
15:45:56 <smcginnis> Back to having a plan come together. ;)
15:46:06 <fungi> love it
15:46:12 <smcginnis> #topic Cycle highlights
15:46:27 <smcginnis> I held off on anything more on this with the summit.
15:46:35 <smcginnis> But I think we have everything in place that we wanted to.
15:46:46 <smcginnis> ttx: We did not want those linked somewhere for now, correct?
15:47:14 <smcginnis> We would just provide the link to journalists asking for updates, IIRC.
15:47:41 <ttx> right
15:47:51 <fungi> eventually the idea was to have a per-release page aggregating the highlights entries across deliverables, right?
15:48:13 <ttx> no need to link to it from releases.o.o/
15:48:21 <fungi> is that being published, and we're just not going to link to it prominently?
15:48:22 <smcginnis> #action smcginnis to draft ML post announcing the plan for PTLs to add cycle highlights.
15:48:48 <smcginnis> fungi: Yep: https://releases.openstack.org/queens/highlights.html
15:48:58 <fungi> got it
15:49:04 <fungi> seems reasonable
15:49:12 <smcginnis> So as teams add highlights to their deliverables, those will be compiled into the output there.
15:49:18 <smcginnis> Grouped by team.
15:49:34 <fungi> a cool design, for sure
15:49:52 <smcginnis> I think we want something like the "top 3 highlights", but we do not have anything restricting that content at the moment.
15:49:54 <ttx> yes, should make everyone's lives much easier
15:50:21 <ttx> it's fine if there are 2 or 4, it's more what to shoot for
15:50:43 <smcginnis> Depending how this goes, I think we may want to just link it off of https://releases.openstack.org/queens/ like we do for the schedule.
15:50:52 <smcginnis> But that can be something to discuss in the future.
15:51:13 <smcginnis> First we need to get people to start using it.
15:51:15 <ttx> yeah, let's see how it goes
15:51:45 <smcginnis> #topic Open Discussion
15:51:52 <ttx> I think in our discussion we came up with  good reasons not to link it
15:52:06 <ttx> like you want marketers to extract themes from it
15:52:10 <smcginnis> ttx: I couldn't remember the reason for that, but I'm sure there is a good one.
15:52:22 <smcginnis> ttx: Oh, marketing themes sounds logical.
15:52:23 <ttx> it's basically an input doc, not an ouput
15:52:30 <smcginnis> Good point.
15:52:46 <ttx> raw data rather than something you want regular humans to consume
15:53:16 <smcginnis> Just wanted to bring up, next Thursday is a holiday in the US. Do we still want the meeting next Friday, or will folks be travelling or be in a food coma?
15:53:17 <ttx> took me a minute to remember it
15:53:33 <ttx> I'll be around, but ok to skip
15:53:43 <smcginnis> I think I will be travelling, but shouldn't be a problem to still hold the meeting.
15:53:55 <smcginnis> Might actually be a nice excuse to hide for a bit. :)
15:54:00 <fungi> i'll be in a car at meeting time, but don't skip on my account
15:54:05 <dhellmann> I'll be off friday
15:54:14 <ttx> yay skip
15:54:23 <smcginnis> #info No meeting next week
15:54:27 <smcginnis> That was easy.
15:54:56 <smcginnis> Anything else in the next 5 minutes to discuss?
15:55:36 <dhellmann> https://review.openstack.org/521119 and https://review.openstack.org/521120 should help address the release notes job issue
15:55:37 <smcginnis> OK, let's wrap then. Thanks everyone.
15:55:49 <smcginnis> dhellmann: Oh, great!
15:56:35 <smcginnis> Alright, see you all in channel.
15:56:43 <smcginnis> #endmeeting