15:00:03 #startmeeting releaseteam 15:00:04 Meeting started Fri Nov 17 15:00:03 2017 UTC and is due to finish in 60 minutes. The chair is smcginnis. Information about MeetBot at http://wiki.debian.org/MeetBot. 15:00:05 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 15:00:07 The meeting name has been set to 'releaseteam' 15:00:11 Ping list: dhellmann, dims, fungi, tonyb, lbragstad, ttx 15:00:29 An hour earlier thanks to DST change. 15:00:44 o/ 15:00:58 #link https://etherpad.openstack.org/p/queens-relmgt-tracking Tracking etherpad 15:01:01 and this is why i do everything in utc ;) 15:01:07 Morning dhellmann. How's the jetlag coming along? 15:01:18 fungi: Smart. ;) 15:01:44 my wife is a bit of a night owl, so she likes it when I travel west and come home able to stay up later than usual. I'm a bit less thrilled, since I still need to wake up early... 15:01:47 fungi: I would do metric too, but the impedence mismatch with everyone around me would be too much. :D 15:01:59 also there's no good metric time standard 15:02:03 dhellmann: Similar situation here. 15:02:36 o/ 15:02:39 #topic Cycle progress 15:03:02 We are currently R-15 in the cycle. 15:03:16 Just a few short weeks away from q-2. 15:03:50 Is there any standing issue on the release infra side? 15:03:58 I've seen a few threads open 15:04:05 We've had a few release notes failures lately, but I think those may be resolved now. 15:04:09 Or just about resolved. 15:04:17 fungi: Are you aware of anything else? 15:04:20 dhellmann: ^^ 15:04:22 release notes job discussion underway in #-infra right now 15:04:40 EmilienM just posted about a failure related to oslosphinx in #openstack-release 15:04:51 I think the fix there is to not use the deprecated lib and update to the new theme 15:04:53 mostly around dropping tox from the release notes build, in preparation for doing the same for docs builds 15:05:13 did we announce that the release notes job was being changed? 15:05:18 (per mordred's updates to the pti) 15:05:33 dhellmann: I think cinder-specs still has a patch out for that. Probably others yet. 15:05:49 dhellmann: not sure, it came up in the discussions about trying to fix failing release notes builds for horizon plugins? 15:05:57 smcginnis: I think that's a different thing? 15:06:11 yeah, we talked in channel but I wasn't sure if there was an email about it 15:06:11 dhellmann: Sorry, I was referring to the discussion in -releases 15:06:28 dhellmann: I don't believe there was an email about the release notes change. 15:06:39 I was actually surprised we merged that already. 15:06:42 ok, we should do that 15:06:56 yeah, that made it through faster than I expected 15:07:06 There are at least a series of patches making changes so services don't need to be installed to run reno. 15:08:30 that much got discussed on the ml, i thought. just not formally announced 15:08:48 ok, I'll look for that thread 15:09:07 Who would be the best to follow up on that thread and make it official? Infra or release team? 15:09:15 we probably should have written a new job, tested it with some projects, then moved everyone to it 15:09:16 around determining that installing projects to identify their version string wasn't necessary for release notes builds 15:09:40 dhellmann: That would have been a smoother transition. 15:09:59 As we've seen, some folks are getting annoyed that things that were working before are suddenly broken. 15:10:05 i wasn't following the transition--been away from the computer a lot this week. i guess that's not how it was done? 15:10:09 I can't imagine why. 15:10:53 fungi : it doesn't seem so. I still don't fully understand the v3 stuff so it wasn't obvious at the time, but if we're having projects failing for new reasons I assume the existing job was changed 15:11:03 i agree a trial implementation would have been nice 15:11:09 fungi mentioned release-notes discussions? Anything for me to chime in? 15:11:31 AJaeger : we were wondering if the ramifications of the job rewrite were announced before that patch landed 15:11:38 we're seeing people with new questions about new failures in the job 15:12:05 AJaeger: mostly just some projects were surprised by the release notes jobs changing when they had been previously working for them. i guess we could have tried a new job just on the projects which were already broken to make things smoother. worth considering for next time 15:12:07 dhellmann: We had temporary failures but fixed them directly. 15:12:27 AJaeger : https://bugs.launchpad.net/tripleo/+bug/1732815 15:12:27 Launchpad bug 1732815 in tripleo "CI: releasenotes job broken with Could not import extension oslosphinx (exception: No module named oslosphinx) " [Critical,Triaged] 15:12:34 The check/gate worked for the repos - but as dhellmann's emails showed, the post/release publishing failed ;( 15:12:47 dhellmann: I fixed that ;) 15:13:17 AJaeger : ok. EmilienM came to #openstack-release at the top of the hour to ask about it, so maybe he hadn't seen the fix yet 15:13:37 where is the fix? 15:13:49 hindsight and all, but we probably would have found that bug even if we were just working through a replacement job for the horizon plugins, without also inflicting the bug on projects which already had working builds 15:13:53 we need someone to write an email to the list describing the job change 15:13:53 dhellmann: but that failure in 1732815 was not known before to me 15:13:59 fungi : yeah 15:14:19 ah, so the issue with tripleo's job isn't fixed 15:14:35 dhellmann: I'm wrong - I didn't fix python-tripleoclient yet. 15:14:46 We could install oslosphinx as well in the releasenotes job to fix it 15:14:50 ok. switching the releasenotes build to use openstackdocstheme instead of oslosphinx should fix the problem. 15:14:54 does python-tripleoclient do something unusual? 15:15:03 well, oslosphinx is deprecated and no one is supposed to be using it any more 15:15:07 dhellmann: exactly, that should fix it 15:15:14 AJaeger: We may want to do that until everyone has switched over. I know not all had yet. 15:15:16 aha, so most projects were already on openstackdocstheme 15:15:24 updating that was part of the migration effort last cycle 15:15:36 but yeah, as smcginnis points out, maybe that's a good temporary measure 15:15:49 it's entirely possible that if all the horizon plugins we were working on fixing builds for were already using openstackdocstheme we wouldn't have caught 1732815 15:16:01 I'm aware of at least two now that have not completed the switch. Likely more. 15:16:02 I see 320 repos using oslosphinx, but it's not clear how many have releasenotes 15:16:46 AJaeger : maybe the thing to do is to try installing the test-requirements.txt for a project, but ignore the failure? like we do with the project's requirements 15:17:36 that seems like a reasonable stepping stone to the docs-requirements.txt mandated by the pti 15:18:26 I can't find the patch that redefined the job, does someone have that link handy? 15:19:09 * AJaeger just pushed " https://review.openstack.org/521104 ensure-reno: Install oslosphinx" 15:19:28 dhellmann: I56909152975f731a9d2c21b2825b972195e48ee8, let me get a link... 15:19:43 AJaeger : I just commented on 512104 15:20:00 dhellmann: https://review.openstack.org/#/c/520362/ 15:20:07 * mordred waves 15:20:48 do we agree on the approach now? 15:21:03 dhellmann: Can you restate it to make sure we all agree on the same thing? 15:21:12 dhellmann: I think it works fine with just oslosphinx, nothing else should be needed to get installed 15:21:22 update the job to optionally install test-requirements.txt, like we do with requirements.txt 15:21:37 AJaeger : we need to stop making assumptions about these jobs. we keep breaking things. 15:22:04 dhellmann: so, you would install *only* test-requirements.txt? 15:22:36 we should make the new job try to reproduce closely what the old job was doing, but be resilient if installing something fails 15:22:50 * mordred likes that - or also maybe optionally install docs/requirements.txt if it's there, if not install test-requirements.txt if it's there? 15:22:51 ++ 15:22:52 the issue with the horizon/neutron jobs was they couldn't install a dependency 15:23:13 sure, just loop over a list of all of these files and try to install based on whatever ones are found 15:23:21 since docs/requirements.txt files, once they exist, should contain sphinx/openstackdocstheme requirements for the projects where appropriate 15:23:29 cool. I can get that updated in the stack 15:23:50 then we need to start a mailing list thread explaining to people what is going on and what changes they can make to anticipate the end state for the job 15:24:08 note: 10 repos have doc/requirements.txt, 5 fuel and 5 other 15:24:09 and we probably need to leave cleaning that job up so it doesn't install extra stuff until after the first of the year, if not the end of the cycle 15:24:14 down-side to making things install everything whether or not they need it is that it becomes increasingly hard to clean that up later 15:24:35 we have the patches AJaeger has proposed to stop setting the version 15:24:42 that was the main reason projects needed themselves installed 15:24:45 mordred: I can abandon 521104 and let you do it cleanly... 15:24:47 like, we're getting started removing the deprecated bindep fallback list finally, and we really have no way to know what that's going to break 15:24:57 we still have some using oslosphinx and we can address those 15:25:02 it's still ok to approve jobs now (and let random releasenotes job fail), or should we wait to approve in some cases? 15:25:13 it should be ok to release things 15:25:18 the release notes build every time any patch merges 15:25:24 right 15:25:33 it's just that we only see the failure notices for jobs in the release queues 15:25:44 Good point. 15:25:45 related to this, but also a different topic to tee up potentially - is publishing server projects to PyPI - which would also remove some of the weirdness with horizon and neutron plugin repos 15:25:59 do you mean just publish them all? 15:26:14 There is definitely a group of users who do want them published. 15:26:26 well, yes - that's the overall thing I'd like to advocate - but we have a potential weirdness issue with keystone 15:26:31 I would be fine with that. I think ttx has given reasons not to in the past? 15:26:41 we can now merge https://review.openstack.org/520763 - we don't need the horizon/neutron variants of releasenotes anymore and thus have the same setup for post and check - the extra requirements in post are the problem 15:26:49 in that there is a keystone on pypi already - even though it seems abandoned - we need to contact its maintainer and see if we can take it over 15:26:55 before we get too far into that, who is going to write the email about what's going on with the release notes job? 15:26:56 i'm unsure whether publishing them to pypi solves that, because the plugins/drivers want to test and build against development branches of those services and not lag a release cycle behind them 15:27:02 there was a discussion in a far past that it was encouraging installing from pip 15:27:14 fungi: that's all handled by tox-siblings just fine though 15:27:22 ahh, good point 15:27:23 but I'd say with "some" being present that ship has sailed 15:27:24 fungi : good point 15:27:32 or that bridge was burnt 15:27:32 ttx: ++ 15:27:39 or whatever analogy works best 15:28:00 ttx: also there was the 'pip install nova does not provide a working nova' argument, but again, I think ship-sailed/bridge-burnt 15:28:03 basically we have 3 things 15:28:18 things that were at some point published and look weird on Pypi 15:28:25 things that were always published 15:28:26 if we keep the "do not publish to pypi" job we can move projects as needed 15:28:31 `pip install openstackclient` doesn't provide a working openstackclient either, because you still need to configure it 15:28:33 things that were never published 15:28:40 * AJaeger needs to really step out now... 15:28:47 thanks AJaeger! 15:28:50 We tried to clean up 1s and remove 3s 15:28:50 thanks, AJaeger 15:29:00 dhellmann: ++ - I think that's the way forward regardless - if we're ok with it as an overall approach, we could start publshing horizon and neutron and at least unstick that stuff 15:29:04 but there was always a good reason to keep some 3s 15:29:24 fwiw, at Dreamhost we installed all of the services with pip then built our own debs of the results 15:29:25 Never published? 15:29:45 ttx: Do you mean 2's? 15:29:56 mordred: how does tox-siblings handle projects with different names on pypi than their repo names? (or does it?) 15:29:57 would there be a way to explain that that it might not result in a functional install and you should rtfm? 15:30:09 e.g., keystone 15:30:19 see "things" mentioned above 15:30:25 using tox-siblings requires a job variant, doesn't it? 15:30:26 fungi: it reads the setup.cfg files of the repos 15:30:31 2= things that were always published 15:30:31 ahh, okay 15:30:55 ttx: Right, you were saying there are good reasons to keep some things never published? 15:30:58 dhellmann: oh - yes indeed - so it actually won't solve the releasenotes job issue that AJaeger's version patches are solving 15:31:01 we were trying to avoid having job variants for the release notes jobs 15:31:03 smcginnis: oh, yes. sorry 15:31:19 smcginnis: always good reason to keep things published 15:31:20 mordred : right 15:31:41 so we've drifted into 2 discussions without finishing the first 15:31:44 so rather than flight it, maybe we should just publish them all for consistency 15:31:51 who is going to write the email explaining the changes to the release notes job? 15:31:54 fight* 15:32:01 * ttx shuts up 15:32:03 dhellmann: yah - I think the current stack will fix the variants issue - I apologize for context-shift - the variants need in releasenotes just reminded me of the horizon/neutron pip weirdness issue 15:32:08 ttx: That would simplify a lot of things, even if the results aren't always "useful". 15:32:33 dhellmann: I can write a draft if you like when I'm done with this patch stack 15:32:36 i would volunteer to write that announcement, but i wasn't around for the changes and didn't review them so expect someone who was involved may do a better job of explaining 15:32:39 mordred : thank you 15:32:42 dhellmann: I was thinking AJaeger would be a good one for that, but maybe someone else from infra would be willing? 15:32:47 mordred: Thanks 15:33:14 #action mordred to write draft email explaining release notes job changes 15:33:37 do we need to discuss the change to publish more projects to pypi? 15:33:48 Second item - publishing all to pypi - what do we need to move forward there? 15:33:50 if we start changing jobs, we will break some validation in the releases repo 15:34:03 so we need to announce the change in advance and probably change that validation logic to be more flexible again 15:34:13 ++ 15:34:13 we should also make it opt-in, I guess? 15:34:19 let projects choose to publish themselves? 15:34:25 that's probably a great start, yeah 15:34:28 maybe go ahead and propose horizon and neutron 15:34:34 and gives us time/flexibility to deal with keystone 15:34:39 and then just allow whatever other projects want to do 15:34:48 and the keystone team can sort their thing out, right 15:35:05 opt-in initially, but we could say that in the future we expect to make it mandatory for projects used as dependencies by other projects at least 15:35:33 sure, that would let us get rid of the constraints installer script, I think 15:35:38 +100 to making it mandatory for projects needed as dependencies and banning the current neutron/horizon approach 15:35:57 i.e., that we plan to stop allownig official deliverables to declare dependencies on things not published to pypi 15:36:05 prometheanfire and dirk may be happy about that 15:36:50 cool, so someone needs to write all of that up 15:36:53 * mordred would LOVE to get rid of tox_install.sh ... once we do, I also think writing a $something - maybe in the PTI, maybe elsewhere, which explains that it is our intent to not need scripts like that ... but we can deal with that later 15:37:11 dhellmann: I can do that after the other email - it seems like this is my brainspace today :) 15:37:16 wondering if we should not just do it 15:37:23 as part of the release process 15:37:34 i think dropping tox_install.sh will be predicated on us getting the local ansible for developers to run jobs situation well-trodden 15:37:36 ttx: I'm not sure I see what you mean? 15:37:41 So we would need to 1) update validation logic to be more flexible, 2) start ML thread on the plan to change things to publish, and 3) work with projects to not have implicit dependencies on non-published services? 15:37:55 I can handle the validation logic change 15:38:00 I find it confusing that only some of our services are there 15:38:14 and that requires two code paths 15:38:30 ttx: yah - I think we eventually want all of them to be there - it just may take a minute or two to get them there 15:38:37 oh sure 15:38:41 Yeah, for consistency sake, I think I would rather have all or nothing. Simplifies our tooling too. 15:38:42 yeah 15:38:59 and as discussed previously, we'd like to not break ci for those projects in the process 15:39:02 but we know already that we have 1 project that we can't publish 15:39:08 do we know that is the only one? 15:39:13 #action dhellmann to update validation logic be more flexible 15:39:24 there are at least two i know of which have name collisions on pypi 15:39:30 arh 15:39:33 #action mordred to draft ML post about pypi publishing plans 15:40:16 oh well, maybe ad-hoc publishing is the solution then 15:40:24 #info Need to look for and work through naming conflicts for other projects on pypi 15:40:27 i.e. continue having two options 15:40:57 Maybe all packages should get prepended "openstack-". 15:40:59 smcginnis : https://review.openstack.org/521115 should cover my piece 15:41:02 openstack-keystone 15:41:09 keeping the repo name and module name unchanged but using a different package name on pypi to avoid collisions should be reasonably tractable 15:41:16 dhellmann: That was quick! 15:41:30 2-minute rule 15:42:00 fungi : good point, we could just change the dist name for keystone to openstack-keystone 15:42:11 fungi: ok 15:42:14 right, exactly that 15:42:19 although the python package name would still collide 15:42:32 mordred says tox-siblings already handles that case by parsing setup 15:42:37 so maybe the idea is to have them all there, and if name collision on first upload use an alias 15:42:42 Less likely someone would want to install both openstack-keystone and keystone, right? 15:42:56 I mean if someone did pip install keystone and pip install openstack-keystone they would be trying to write to the same directory 15:43:16 I would hope, if "not-openstack-keystone" is deprecated that no one is using it 15:43:22 but who knows 15:43:38 I guess if they are using it, they already have the path collision problem 15:43:58 venv or containers FTW. 15:44:06 we might be able to do some hacky install_requires tricks with a greedy version exclusion to make it not install them together 15:44:41 Should we stew on this part for a bit. Not sure we are going to solve it in the next 15 minutes. 15:44:56 yeah, no need to design in-meeting 15:45:27 yeah, we can also proceed under the assumption that we'll solve that somehow 15:45:56 Back to having a plan come together. ;) 15:46:06 love it 15:46:12 #topic Cycle highlights 15:46:27 I held off on anything more on this with the summit. 15:46:35 But I think we have everything in place that we wanted to. 15:46:46 ttx: We did not want those linked somewhere for now, correct? 15:47:14 We would just provide the link to journalists asking for updates, IIRC. 15:47:41 right 15:47:51 eventually the idea was to have a per-release page aggregating the highlights entries across deliverables, right? 15:48:13 no need to link to it from releases.o.o/ 15:48:21 is that being published, and we're just not going to link to it prominently? 15:48:22 #action smcginnis to draft ML post announcing the plan for PTLs to add cycle highlights. 15:48:48 fungi: Yep: https://releases.openstack.org/queens/highlights.html 15:48:58 got it 15:49:04 seems reasonable 15:49:12 So as teams add highlights to their deliverables, those will be compiled into the output there. 15:49:18 Grouped by team. 15:49:34 a cool design, for sure 15:49:52 I think we want something like the "top 3 highlights", but we do not have anything restricting that content at the moment. 15:49:54 yes, should make everyone's lives much easier 15:50:21 it's fine if there are 2 or 4, it's more what to shoot for 15:50:43 Depending how this goes, I think we may want to just link it off of https://releases.openstack.org/queens/ like we do for the schedule. 15:50:52 But that can be something to discuss in the future. 15:51:13 First we need to get people to start using it. 15:51:15 yeah, let's see how it goes 15:51:45 #topic Open Discussion 15:51:52 I think in our discussion we came up with good reasons not to link it 15:52:06 like you want marketers to extract themes from it 15:52:10 ttx: I couldn't remember the reason for that, but I'm sure there is a good one. 15:52:22 ttx: Oh, marketing themes sounds logical. 15:52:23 it's basically an input doc, not an ouput 15:52:30 Good point. 15:52:46 raw data rather than something you want regular humans to consume 15:53:16 Just wanted to bring up, next Thursday is a holiday in the US. Do we still want the meeting next Friday, or will folks be travelling or be in a food coma? 15:53:17 took me a minute to remember it 15:53:33 I'll be around, but ok to skip 15:53:43 I think I will be travelling, but shouldn't be a problem to still hold the meeting. 15:53:55 Might actually be a nice excuse to hide for a bit. :) 15:54:00 i'll be in a car at meeting time, but don't skip on my account 15:54:05 I'll be off friday 15:54:14 yay skip 15:54:23 #info No meeting next week 15:54:27 That was easy. 15:54:56 Anything else in the next 5 minutes to discuss? 15:55:36 https://review.openstack.org/521119 and https://review.openstack.org/521120 should help address the release notes job issue 15:55:37 OK, let's wrap then. Thanks everyone. 15:55:49 dhellmann: Oh, great! 15:56:35 Alright, see you all in channel. 15:56:43 #endmeeting