21:02:47 <ttx> #startmeeting crossproject
21:02:47 <devananda> ttx: i continue to wonder why i'm not on the courtesy pings for this meeting :p
21:02:48 <openstack> Meeting started Tue Mar 17 21:02:47 2015 UTC and is due to finish in 60 minutes.  The chair is ttx. Information about MeetBot at http://wiki.debian.org/MeetBot.
21:02:49 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
21:02:53 <openstack> The meeting name has been set to 'crossproject'
21:02:55 <morganfainberg> ttx, why must you jinx it like that :P
21:02:59 <ttx> devananda: you're now added
21:03:31 <ttx> Our agenda for today:
21:03:34 <ttx> #link http://wiki.openstack.org/Meetings/CrossProjectMeeting
21:03:41 <ttx> #topic Progress on Swift and Keystone developing next (incompatible) version of client libs in openstack-sdk
21:03:52 <ttx> Back in December we discussed how Swift and Keystone were developing the next-generation client libs via openstack-sdk
21:03:59 <ttx> #link http://eavesdrop.openstack.org/meetings/crossproject/2014/crossproject.2014-12-02-21.01.html
21:04:04 <ttx> Back then we said we'd talk again about how that went in February/March, so here we are
21:04:16 <ttx> notmyname is not around, but reported that nothing has happened on Swift's side
21:04:21 <ttx> mostly due to other priorities
21:04:26 <morganfainberg> i can report much the same on Keystone side
21:04:35 <ttx> morganfainberg: hi!
21:04:49 <ttx> morganfainberg: is it still in the cards for keystone ?
21:04:58 <morganfainberg> not this cycle.
21:05:03 <ttx> mordred: heh
21:05:08 <ttx> arh tab fail
21:05:13 <ttx> morganfainberg: heh
21:05:20 <morganfainberg> mordred, aha! for once it's the inverse tab-fail
21:05:31 <briancurtin> fwiw, SDK has made a ton of progress and will soon be at a better point that other projects can leverage it like those were planned (pending their availability/priority, of course)
21:05:50 <jeblair> careful; you have summoned mordred to a meeting with the words "client" and "incompatible" in the topic.
21:05:54 <morganfainberg> ttx, we did discuss we are going to split all the auth logic out of keystoneclient, session, plugins, etc.
21:05:56 <mordred> aroo?
21:06:03 <morganfainberg> ttx aiming for liberty early
21:06:06 <ttx> briancurtin: would you say it's a sane approach for projects that want a next-gen CLI to piggyback on openstack-sdk ?
21:06:10 * bknudson was planning to change auth-token to use keystoneclient, maybe should use sdk instead.
21:06:20 <morganfainberg> this will be consumable by SDK and anything else.
21:06:39 <dhellmann> ttx: now that openstackclient is official, perhaps we should be encouraging projects to add their next-gen CLIs there?
21:06:41 <mordred> ttx: CLI should use python-openstackclient
21:06:42 <devananda> naive question on openstack-sdk -- is there a goal to have this used for intra-serice communication?
21:06:49 <mordred> which is what projects are already doing
21:06:49 <briancurtin> ttx: yes. SDK will soon be able to provide a consistent REST client *library* from which CLIs could be build
21:06:55 <morganfainberg> dhellmann, ++ more people should have CLIs in OSC :)
21:07:08 <ttx> jeblair: I did it on puprose, I was surprised that idea flew so easily in december
21:07:09 <briancurtin> devananda: i dont work on any services myself, but the goal is to make libraries taht can be used intra-service - yes
21:07:13 <mordred> briancurtin, ttx: I think that that would be more OSC moving to OpenStack SDK
21:07:27 <dhellmann> mordred: ++
21:07:31 <stevemar> mordred, which will happen, eventually
21:07:46 <briancurtin> ^probably, i dont know 100% what next-gen CLI means so i spoke a bit early
21:08:04 <ttx> morganfainberg: does mordred's plan fit in your long-term view for the next-gen CLI ?
21:08:11 <morganfainberg> ttx, i'
21:08:17 <jeblair> devananda: people tell me that the python-*client stuff is really focusing on inter-service communication, and that's why it's not a good place to develop user-side client interfaces
21:08:18 <mordred> aroo?
21:08:19 <morganfainberg> ve already pushed the CLI stuff over to OSC
21:08:20 <devananda> interesting. so do you see eg. Nova using the common client instead of python-glanceclient at some point?
21:08:30 <devananda> jeblair: that is what I had heard as well
21:08:39 <morganfainberg> i let them drive the best approach, it doesn't let me drop bitrotting things from ksc though
21:08:40 <jeblair> devananda: if the services move to the common client and take that same attitude with them, i'm going to be upset
21:08:42 <fungi> devananda: briancurtin: by intra-service do you mean when nova needs to talk to neutron, using openstack-sdk instead of python-neutronclient to accomplish that?
21:08:47 <mordred> I would counsel stongly not putting the cart before the horse
21:08:50 <briancurtin> fungi: yes
21:08:53 <fungi> er, what devananda typed faster than i did
21:08:54 <mordred> let's get the thign we don't have right now
21:08:54 <devananda> fungi: that is my question, yes
21:08:58 <mordred> which is a usable end-user library
21:08:59 <ttx> mordred: topic is, back in december while you were asleep both keystojen and swift declared wanting to write incompatible next-gen CLI
21:09:02 <jeblair> mordred: ++
21:09:03 <mordred> we don't have that at all
21:09:05 <devananda> I do *not* think that is the right goal, fwiw
21:09:09 <mordred> because python-*client is CRAP
21:09:23 <morganfainberg> so the net approach is split out the stuff i don't want to get clobbered by KSC wonky-compat stuff into it'w own thing
21:09:37 <devananda> I would much prefer the python*clients to stay focused on intra-service interactions, and the common / unified client to really be focused on client-side usability
21:09:41 <morganfainberg> so SDK and shade and anything else can do things with it w/o getting compat things.
21:09:42 <mordred> devananda: ++
21:09:45 <morganfainberg> anyway.
21:09:53 <sdague> honestly, ++ on using the sdk for cross service interactions
21:09:54 <mordred> shade also wants to port to openstacksdk when it's ready
21:09:54 <dhellmann> devananda: yes, I understood that to be the plan, at least for now.
21:10:02 <fungi> so we'll be doing stable branches of the sdk presumably to accomplish inter-service communication hopefully
21:10:03 <briancurtin> devananda: SDK isn't *focused* on intra-service, but by just providing a consistent and usable interface, you get it for free
21:10:17 <devananda> thanks all. didn't mean to derail - -just wanted some clarification.
21:10:38 <dhellmann> sdague: now that we're using virtualenvs and have stable branches of libs in the gate, that would work (eventually)
21:10:40 <fungi> because the "backward compatible one branch" model has pretty much been identified as not working for stable intra-service purposes
21:11:04 <morganfainberg> briancurtin, i'll ping you about auth stuff net week.
21:11:08 <mordred> yah - it's a different use case
21:11:08 <morganfainberg> next*
21:11:15 <briancurtin> sounds good
21:11:25 <mordred> fungi: ++
21:11:26 <ttx> OK, i'm a bit confused
21:11:33 <mordred> ttx: I can't imagine why
21:11:43 <ttx> probably because this day is too long
21:11:48 <mordred> yah
21:11:54 <devananda> ttx: much ...
21:11:56 <morganfainberg> ttx, i haven't even had coffee yet :(
21:12:07 <mordred> just please please please don't screw the SDK up with intra-service stuff yet
21:12:07 <morganfainberg> the day is turning into a nightmare.
21:12:09 * fungi hasn't even had beer yet
21:12:12 <devananda> morganfainberg: oh the horror!
21:12:15 <ttx> morganfainberg: could you summarize what you plan to do for keystone, between openstack-sdk and openstackclient and all ?
21:12:19 <mordred> give it some time to be useful for the use case which is currently a nightmare
21:12:23 <jeblair> mordred: ++
21:12:26 <devananda> mordred: ++
21:12:26 <ttx> fungi: I didn't have my guinness yet
21:12:27 <sdague> mordred: in what way would it be screwed by that?
21:12:36 <mordred> sdague: release policy being completely different
21:12:40 <morganfainberg> ttx, so in short, we're going to split out, targeted for liberty some bits from keystoneclient that are common [auth]
21:12:44 <mordred> for intra-service - it is important to know version
21:12:49 <mordred> for end user, it is explicitly important to not
21:12:53 <devananda> sdague: different audience => subtly different requirements
21:12:57 <morganfainberg> ttx, the parts that should never have compat carried with it.
21:13:04 <devananda> (or maybe not so subtly different)
21:13:07 <morganfainberg> this is session, auth, plugins, adapters
21:13:13 <mordred> because an end user does not know if you installed essex or havana and should not
21:13:22 <mordred> and some end users may want to connect to both
21:13:35 <ttx> morganfainberg: and that split-stuff would end up... in openstackclient ?
21:13:37 <sdague> mordred: so, I'm not convinced it's actually important in the intra service case either, but that's fine
21:13:48 <morganfainberg> and then we will focus on using SDK for "next gen" client-y things that interact with the API in non-auth ways
21:13:54 <mordred> sdague: awesome- except we just added stable branches for client libs
21:14:09 <morganfainberg> ttx, no, that will be it's own lib, like middleware that just handles auth processes. I expect SDK to consume it
21:14:16 <sdague> mordred: we added that because pip is drunk
21:14:19 <mordred> sdague: in any case- I'm just asking that we get end-user solid first
21:14:21 <ttx> morganfainberg: ah, starts making more sense
21:14:24 <mordred> before we add another use case
21:14:34 <fungi> yeah, and this paradox has led to all sorts of testing and packaging pain too. does a distro package the version of glanceclient which their packaged version od nova needs to talk to their packaged version of glance, or do they package a version of glanceclient that they want their cloud users talking to random glance service providers with?
21:14:36 <sdague> if pip wasn't drunk, we could largely have avoided that
21:14:48 <morganfainberg> ttx, it lets us drop the compat worries for the shared stuff. it also means i can drop some bitrotting stuff in keystoneclient because we will be changing dependency trees
21:14:52 <mordred> I'm just asking that we get end-user solid first before we add another use case
21:15:14 <mordred> since we have largely screwed our end users constantly since the inception
21:15:25 <morganfainberg> ttx, it's an effort to not carry keystone-compatibility and keystone-specific API code for commonly shared / used code.
21:15:30 <sdague> mordred: sure, that's fine
21:15:53 <mordred> \o/
21:16:15 <lifeless> sdague: is there a spec on the pip drunkeness somewhere? I'd like edumacation
21:16:21 <ttx> ok, I think I got it . Wondering how much of that plan has potential to be replicated elsewhere
21:16:22 <sdague> lifeless: it has no resolver
21:16:45 <fungi> #link https://github.com/pypa/pip/issues/988
21:16:51 <lifeless> sdague: lets not derail here; I know that much - after the meeting perhaps?\
21:17:12 <dhellmann> morganfainberg: is all of that written down somewhere else?
21:17:34 <ttx> morganfainberg: basically I'd like your specific plan to be part of a long-term vision on lcient libs and CLI
21:17:46 <morganfainberg> ttx, i'll be proposing the infra/goverance changes next week with a code split. but the target for release will be liberty. that will document the whole plan between gov. repo and infra bits
21:17:59 <ttx> (that was the idea for this status update, let projects experiment and see what can be adopted elsewhere)
21:18:12 <ttx> morganfainberg: cool
21:18:17 <morganfainberg> dhellmann, ^
21:18:29 <dhellmann> morganfainberg: ++
21:18:43 <ttx> Alright, last thoughts / questions on that topic ?
21:19:01 <morganfainberg> the tl;dr is "help make it so SDK can be successful so we can move stuff to it sanely"
21:19:27 <bknudson> do we have any idea how close it is to parity?
21:19:27 <ttx> sounds like a plan. Next topic...
21:19:30 <bknudson> with keystoneclient?
21:19:39 <ttx> arh, bknudson's last second question.
21:19:44 <morganfainberg> bknudson, bigger discussion
21:19:53 <bknudson> ok, don't worry about it.
21:20:01 <morganfainberg> bknudson, i don't think we're there yet.
21:20:08 <ttx> #topic Add library stable release procedures/policy
21:20:14 <ttx> #link https://review.openstack.org/#/c/155072/
21:20:23 <briancurtin> bknudson: on parity, we haven't made that a priority since we've only been using the auth bits. once we stabilize API, getting to parity with any other clients isn't too hard
21:20:28 <ttx> This one is really in the final stage before final approval by the TC, so if you have a strong opinion on it, please voice it now.
21:20:39 <ttx> If it is still consensual by the end of the week I'll put it on next TC meeting agenda for final rubberstamping.
21:20:57 <morganfainberg> in short, stable releases of clients?
21:20:58 <ttx> Because we want to apply it for kilo
21:21:17 <dhellmann> morganfainberg: yes, in addition to oslo and other libs
21:21:22 <bknudson> backporting security fixes for libraries.
21:21:24 <morganfainberg> hmm.
21:21:35 <morganfainberg> i think i voiced support for this a while back. cool
21:21:51 <bknudson> and keystonemiddleware?
21:22:00 <morganfainberg> yes middleware would adhere to this
21:22:03 <dhellmann> bknudson: if it is a library, it is covered by this policy
21:22:10 <fungi> driven in part by attempting to maintain requirements stability and the fact that our servers require the clients to do intra-service communication (per earlier topic)
21:22:19 <dhellmann> fungi: right
21:22:28 <morganfainberg> dhellmann, middleware is a little different, but it falls into the same policy easily (and should)
21:22:47 <dhellmann> this is mostly driven by the fact that we are capping requirements in stable branches for applications, so I'm trying to describe a process for allowing patch releases into the stable branches for fixes
21:22:55 <devananda> dhellmann: does this apply to python-*clients as well?
21:22:56 <ttx> So please, please, report on that spec if you agree or disagree. Not having an opinion is fine too.
21:23:00 <dhellmann> morganfainberg: middleware isn't useful on its own, right? so it's a library?
21:23:03 <dhellmann> devananda: yes
21:23:08 <devananda> huh. ok.
21:23:21 <adam_g> one question: is maintenance and release mgmt of stable client libraries the responsibility of the individual client teams or stable-maint?
21:23:22 <dhellmann> devananda: because the alternative is they can never use oslo libraries
21:23:30 <morganfainberg> dhellmann, it's ... well it could be useful on it's own w/ keystone. but uhm. lets just say yes it'sa libary.
21:23:37 <dhellmann> adam_g: how is the work split up for the apps now?
21:23:40 <devananda> dhellmann: yah. it makes sense. i'm just thinking of the situation between ironic and nova
21:23:46 <dhellmann> morganfainberg: :-)
21:23:47 <morganfainberg> dhellmann, for simplicity
21:23:51 <devananda> dhellmann: that is, nova refuses to accept python-ironicclient in their requirements.txt file
21:24:00 <ttx> adam_g: currently, individual client teams, usually driven by VMT (since only security updates will likely result in point releases)
21:24:11 <devananda> dhellmann: but you can't use nova + ironic without installing it. and thus it should be capped on stable branches
21:24:11 <jogo> devananda: there is a case of this right now for glanceclient
21:24:14 <jogo> needing a stable branch
21:24:17 <fungi> devananda: the python-.*client libraries are used by services to talk to one another, so can't introduce dependency changes in new client releases without capping them in stable branches, and can't get security fixes to the stable branches if we cap them without branching them
21:24:23 <dhellmann> devananda: there's a separate spec somewhere that may deal with optional requirements like that
21:24:42 <ttx> fungi: we could however only create the branch if there is a backport
21:24:43 <dhellmann> right, what fungi said
21:24:47 <adam_g> dhellmann, as it is today, release management is generally handled by stable-maint-core, backports and general maintanence by individual project teams + stable-maint-core
21:24:51 <devananda> yah. the reasoning to have stable branches of clients -- and thereby cap their requirements -- makes sense
21:24:57 <adam_g> ttx, thanks
21:25:03 <devananda> it's one reason I join the pip-hate bandwagon
21:25:16 <dhellmann> ttx: technically we could, but if we do that we're going to screw up eventually, so let's choose a process that doesn't require anyone to make a decision on that
21:25:18 <dhellmann> branches are cheap
21:25:25 <devananda> because in ironic right now, we're trying to make sure tip of client continues to work with stable server and current server release
21:25:43 <jogo> devananda: https://bugs.launchpad.net/python-glanceclient/+bug/1423165
21:25:45 <openstack> Launchpad bug 1423165 in Cinder "https: client can cause nova/cinder to leak sockets for 'get' 'show' 'delete' 'update'" [Undecided,New]
21:25:49 <dhellmann> devananda: right, and that could still be a goal under this system if you wanted it to be
21:25:50 <sdague> devananda: you can't do that if you have releases
21:25:53 <ttx> dhellmann: except we don't want to encourage backports in python-*client, so we might want to ahve an additional policy there
21:25:55 <adam_g> devananda, i think we'd want to strive for that even with stable branches
21:26:04 <sdague> the requirements get into conflict pretty fast
21:26:12 <ttx> (i.e. "they start in Phase III stable support")
21:26:21 <devananda> ttx: right. backports on clients doesnt seem to be a thing we are doing today, but if they have stable branches, we may need to
21:26:25 <jeblair> dhellmann: what is "$release" and "$SERIES" in that spec?
21:26:32 <dhellmann> sdague: oh, I thought he meant "can talk to" not "can run in the same site-packages"
21:26:34 <devananda> ttx: eg, for security fixes
21:26:42 <dhellmann> jeblair: sorry, I wasn't consistent. Both are things like kilo or liberty
21:26:54 <jeblair> oooh, okay.
21:26:55 <sdague> dhellmann: but, the point is, it needs to live in the nova address space
21:26:59 <dhellmann> I think ttx suggested $SERIES and I didn't find all of the $release instances and replace them
21:26:59 <fungi> yeah, i think the clients need functional tests against devstack or something like that, but cease to need to be tested as intra-service communication libraries from a backward-compat perspective
21:27:08 <devananda> sdague: OH. yes, I wasn't clear.
21:27:08 <dhellmann> sdague: sure, in one use case
21:27:16 <sdague> otherwise, kind of pointless, as that's a primary consumer
21:27:20 <jeblair> dhellmann: yeah, that's probably worth fixing as i was really having trouble understanding it.  :)
21:27:25 <dhellmann> jeblair: noted
21:27:52 <ttx> ok, other questions on that one ?
21:28:10 <fungi> i'd imagine for example novaclient's functional tests would start to get run against supported stable release environments to ensure backward-compat
21:28:13 <jeblair> dhellmann: i'm glad you put in the thing that said you didn't want to use stable/1.2.3, because i thought that's what you meant by $release and why you were distinguishing it from $SERIOUS
21:28:16 <devananda> ttx: i revert my position in our earlier conversation
21:28:18 <jeblair> $SERIES even
21:28:25 <devananda> ttx: we will need stable branches of ironicclient
21:28:32 <dhellmann> jeblair: I was $SERIOUS about it ;-)
21:28:40 <ttx> devananda: ok
21:28:41 <morganfainberg> jeblair, oh god stable/X.X.X would be awful.
21:28:47 <morganfainberg> jeblair, stable/series ++
21:29:02 <dhellmann> I'll clarify that in a new draft in the morning
21:29:04 <fungi> we tried something akin to stable/x.y.z and there are some issues there
21:29:09 <ttx> ok, let's move on to next topic then
21:29:10 <fungi> but better left for beertalk
21:29:19 <ttx> #topic Managing stable branch requirements
21:29:23 <ttx> #link https://review.openstack.org/#/c/161047/
21:29:26 <jeblair> fungi: i believe the only issue was that the process wasn't followed
21:29:39 <ttx> This one is about providing a more stable test environment for stable branches
21:29:44 <ttx> by freezing the transitive dependency set and using that in tests
21:29:59 <ttx> adam_g: I'll let you present it deeper
21:30:02 <adam_g> ive pushed up a first pass at the required devstack changes for this at https://review.openstack.org/#/c/165195/
21:30:04 <adam_g> #link https://review.openstack.org/#/c/165195/
21:30:10 <fungi> jeblair: well, lack of existing tooling to match them to equivalent stable series branches for integration testing
21:30:12 <ttx> This one is also unlikely to affect that many people, but those with an opinion usually have a strong one
21:30:17 <ttx> So it is time to review it and provide feedback
21:30:34 <adam_g> the idea is to maintain a stable list of pinned transitive deps in requirements (in addition to global-requirements.txt), and use those when testing in the gate. this list is updated anytime global-requirements.txt changes
21:30:44 <bknudson> fewer gate breakages will be a positive affect.
21:30:50 <morganfainberg> bknudson, ++
21:31:28 <bknudson> updated automatically?
21:31:29 <sdague> adam_g: I feel like this tooling should all be in requirements project, not in devstack
21:31:35 <morganfainberg> sdague, ++
21:31:39 <adam_g> i think the spec is ready for review, the only concerns i have at this point is 1) requiring devs updating global-requirements.txt to also compile and propose the static list 2) relying on pip-compile, which lives outside of openstack/stackforge
21:31:45 <morganfainberg> i would like to see it clearly published centrally
21:31:46 <dhellmann> yeah, if we have this in multiple repos we'll get wedged
21:32:01 <morganfainberg> i think it would be a net-win for our distro packagers as well
21:32:22 <adam_g> sdague, we need to change how we install things a bit, and that'll require something in devstack
21:32:26 <bknudson> how are distros going to use this?
21:32:30 <fungi> i don't think it will likely be much help to distro packagers
21:32:33 <morganfainberg> bknudson, same way they use g-r
21:32:48 <adam_g> sdague, but yeah, any functionality can be pushed into reqs and devstack can just call that instead
21:32:50 <bknudson> not having breaks will help, of course.
21:32:51 <fungi> morganfainberg: by the time this exists, they've already settled on what versions to package
21:32:52 <dhellmann> yeah, I expect distros will continue to use the more lenient file
21:32:53 <sdague> morganfainberg: honestly, I don't think it will be. Because we are picking a very narrow requirements set they'll basically have to reinvent this all if they want to support anything other than big bang upgrades
21:32:54 <morganfainberg> bknudson, they can confirm what we're testing against and what the transitive deps are. i know packagers would like that
21:33:03 <morganfainberg> fungi, future looking even?
21:33:05 <morganfainberg> not just kilo
21:33:10 <fungi> morganfainberg: for any branch
21:33:15 <morganfainberg> eh ok
21:33:18 <bknudson> we create packages at a certain level and ship with and then don't change.
21:33:23 <morganfainberg> oh it's a one-off when branching
21:33:25 <morganfainberg> ooooh
21:33:32 <fungi> morganfainberg: master is a moving target. by the time we branch, they've already decided on versions of packages
21:33:36 <morganfainberg> not auto-update on every g-r- patch?
21:33:40 <adam_g> sdague, we're not picking a narrow requirement set, we're still using the ranges we provide in requirements.txt--we're just being narrow about what we use at a given point in time in our gate runs
21:33:55 * morganfainberg shrugs.
21:33:59 <adam_g> all the while satisyfing what we're publishing in requiremnts.txt for downstreams
21:34:17 <jeblair> i have read the spec, but i am not certain why we want to solve this problem
21:34:19 <dhellmann> frankly, I'd like us to run for a while with the caps to see how big of a deal breaking changes are before we start being more strict
21:34:21 <morganfainberg> i guess it would be good for CD but thats a different story.
21:34:40 <jeblair> i am generally opposed to making it so that our software _only_ works in the gate
21:34:47 <jeblair> and does not work for a user who just tries to run it
21:34:55 <ttx> This is all about preserving our ability to produce stable branches at all. If we don't simplify testing, we can't maintain them. So it should be seen as a net win
21:35:21 <ttx> because the alternative is not pretty
21:35:28 <sdague> dhellmann: yeh, I'd agree, have we seen wedges with the current capping?
21:35:30 <adam_g> jeblair, currently, a user who tries to "run it" may get something totally different than what was run in the gate 10 minutes ago, wrt to dependencies
21:35:36 <dhellmann> ttx: we're already expressing upper bounds on *our* requirements. How often are the stable tests still breaking because of transitive dependencies?
21:35:41 <bknudson> we also have the issue now where we don't test with the min version req'd, so we don't know if it actually works ... e.g. cryptography 0.4
21:35:49 <jeblair> adam_g: yes, but the next time the gate runs, it will get what they got (or something later), and it will break, and we will know
21:36:00 <dhellmann> jeblair: ++
21:36:04 <ttx> dhellmann: I think there was a case last week, mentioned by jogo on the review
21:36:14 <jogo> ttx: correct https://bugs.launchpad.net/devstack/+bug/1430592
21:36:16 <openstack> Launchpad bug 1430592 in devstack "testtools-1.7.0 triggering pkg_resources.VersionConflict: (unittest2 0.5.1 (/usr/lib/python2.7/dist-packages), Requirement.parse('unittest2>=1.0.0'))" [Undecided,In progress] - Assigned to Ian Wienand (iwienand)
21:36:16 <dhellmann> ttx: so 1. And how long did it take to fix?
21:36:33 <jogo> #link https://bugs.launchpad.net/devstack/+bug/1430592 is the exact case this spec should prevent
21:36:36 <adam_g> jeblair, the "it will break" part is what the spec is trying to address
21:36:38 <dhellmann> ok, that's not even a runtime dependency
21:36:52 <jeblair> adam_g: but this proposal will virtually guarantee that users will be unable to run the software without the extra setup that this does -- which is work that is not handled by pip
21:36:59 <dhellmann> adam_g: but having it break is a useful piece of information
21:37:00 <sdague> also, that was a pip is drunk moment right, it should have upgraded unittest2 but it did not
21:37:05 <jeblair> adam_g: basically, this works around pip _so much_ that we would be better off simply not using it
21:37:20 <jogo> jeblair: using breaking stable breaches as a forcing mechanism to make people pay attention to stable branches means we block development for large chunks of time
21:37:22 <jeblair> adam_g: because right now we say you can install with pip, but this means you can not.
21:37:28 <sdague> jeblair: yeh, this feels like at this point it's basically just ghetto pip2
21:37:35 <jeblair> jogo: it's not a forcing system, it's reality
21:37:42 <dhellmann> jogo: capping a new requirement doesn't take a lot of time, though
21:37:43 <jeblair> jogo: the software _really is broken_ and needs to be fixed
21:37:48 <sdague> done in shell scripts, which is weird
21:38:09 <jogo> dhellmann: so that isn't always true, figuring out subtle breakage can be really hard
21:38:09 <bknudson> capping requirement does take time especially when you don't know exactly when it broke
21:38:17 <jeblair> jogo: this merely masks the problem and means that the only way to use the software is via the exact mechanism in the gate.
21:38:37 <dhellmann> jogo, bknudson : so let's track when we start seeing new versions of transitive dependencies in the gate to make that easier?
21:39:05 <fungi> i too agree that trying to work around "our software doesn't work out in the world" is akin to sticking our developer heads in teh sand so we can continue to make progress on other things
21:39:17 <ttx> The problem is, (1) the expertise to solve thsoe is pretty limited, and those people ususally work on master; (2) the tooling to follow such issues is pretty inexistent
21:39:37 <jogo> honestly using pip with free floating deps to install this in the real world doesn't sound like a great idea to me
21:39:43 <bknudson> the stable branches do work out in the world, because packagers pin versions.
21:40:08 <dhellmann> jeblair, fungi : how hard would it be to track the pip freeze output for each job so when a job starts failing we could point out that it has different versions than the last time it passed?
21:40:09 <fungi> bknudson: that is basically an argument in favor of having the distros do our stable branch testing for us
21:40:15 <jeblair> jogo: that's entirely valid.  i think some projects do hard-specify all transitive deps and versions.
21:40:37 <jeblair> jogo: i think if we want to go that way, we should make that our only supported installation mechanism....
21:40:43 <ttx> jeblair: so you suggest we fix the stable-gate-fixing staffing problem ?
21:40:49 <fungi> dhellmann: we basically already do?
21:41:13 <fungi> dhellmann: unless you're asking for more automation and reporting around it, that's already usually one of the first things i check
21:41:14 <jogo> also it wouldn't be hard to make a bot to automatically propose changes to the pinned stable branch deps
21:41:15 <dhellmann> fungi: the current freeze output is in each log, but is there a diff since the last known good job run?
21:41:42 <jogo> so we can test new versions of libs and only approve using them if they don't break things
21:41:50 <fungi> dhellmann: i just pull up a recent passing job, an early failing job (identified via logstash) and then diff the pip freeze from them
21:42:00 <dhellmann> fungi: ah, ok, sure
21:42:14 <dhellmann> fungi: better tooling for that would be useful
21:42:27 <fungi> an argument can certainly be made for adding automation and reporting around that, but we'd need to define what it would look like
21:42:27 <jeblair> dhellmann, fungi: yeah, i think there's room for that
21:42:51 <fungi> guessing something akin to subunit2sql
21:42:54 <jogo> this also comes down to a human problem, I do not want to keep putting out these fires
21:43:06 <ttx> nobody wants
21:43:07 <dhellmann> fungi: "your job ran with blah, blah, blah installed and that list is different from the job at time X that passed in this way..."
21:43:32 <dhellmann> jogo: just because we don't want to do this, doesn't mean we want you to put out all the fires
21:44:04 <jogo> dhellmann: so who wants to do it instead?
21:44:07 <dhellmann> frankly, I think if we leave things broken for a little bit we'll find some people willing to spend time debugging these things. So far we're doing it all for them.
21:44:11 <ttx> shooting down this solution without proposing an alternative to fixing our stable maint gate issues is not really acceptable
21:44:13 <fungi> often times the only way to get new people to help fight those fires (or even recognize that they are) is to step back and let them burn
21:44:21 <dhellmann> and I use "we're" very loosely there, since I don't do it much :-)
21:44:32 <jogo> dhellmann: so part of the idea here is to decouple stable branch stuff from impacting master branch
21:44:43 <jeblair> ttx, jogo, adam_g: i think we need to pick an installation method and stick with it.  either use pip the way we are now, or use a different installation method where we use fixed versions everywhere.
21:44:49 <dhellmann> ttx: I am not yet convinced that this problem happens so often any more than we still have that problem, though. Are you?
21:44:56 <jogo> jeblair: what about master branch?
21:45:18 <ttx> dhellmann: I trust the people who fight those every day, like jogo & adam_g -- if they push a solution, I want to support them
21:45:32 <jogo> dhellmann: I wasn't sure it would happen much but it did a week ago
21:45:42 <ttx> rather than push them to burnout by rejecting their attempts
21:45:48 <dhellmann> ttx: ok. I do, too, but I think this adds more issues downstream, as jeblair pointed out.
21:46:02 <jogo> dhellmann: this puts more burdon on stable teams yes
21:46:07 <jeblair> ttx: this is not a "stable maint gate" issue.  this is a "stable branch" issue.  the solution _must not be_ to make the gate different than reality.  the solution must fix reality.  :)
21:46:19 <ttx> dhellmann: if the alternative is to have dead stable branches, not that much
21:46:19 * adam_g would love to know who downstream actually uses stable branches via pip
21:46:28 <morganfainberg> jeblair, there is no spoon.
21:46:40 <dhellmann> adam_g: I know of some deployments that work that way
21:46:58 <ttx> jeblair: so far the only alternative I heard is to drop stabel branches completely. I would rather not have it come down to that
21:46:59 <dhellmann> adam_g: some folks follow the stable branches instead of master because they're stable :-)
21:47:35 <jeblair> ttx: i think lots of alternatives have been suggested.  let me try to recap for you.
21:47:37 <adam_g> dhellmann, weiiird
21:47:50 <ttx> jeblair: key word is "suggested". We need people to work on those
21:47:51 <dhellmann> adam_g: not at all, these folks do their own packaging
21:47:54 <jogo> jeblair: I agree, we can make how devstack installs stable align with how downstream folks who don't use packages install stable branches
21:48:15 <jeblair> 0) status quo -- because capping was a big part of the problem, let's try that and see how big the transitive problem really is first.
21:48:15 <adam_g> dhellmann, exactly, they do their own packaging and likely run with some static set of dependencies (either satisfied by a distro or their own controlled pip mirrors)
21:48:32 <dhellmann> adam_g: no, they do not use a distro. They make their own virtualenv-based packages with pip.
21:48:51 <ttx> jeblair: We've been looking for someone to drive this for months
21:48:51 <ttx> adam_g kindly volunteered, but if his solution won't fly, we need a backup plan
21:48:51 <ttx> with another volunteer to push it
21:49:02 <bknudson> dhellmann: what do they do when they have a problem due to the package not working with the old code?
21:49:17 <ttx> (network issues, I'm lagging now)
21:49:23 <jeblair> 1) the minimal change to adam_g's proposal that i would be okay with is to make the installation mechanism for stable branches use fully pinned deps always -- gate or end user.
21:50:05 <jeblair> 2) that could extend to changing how we install the master branch too
21:50:15 <jeblair> (i don't necessarily think that's a good idea)
21:50:26 <fungi> the problem he's trying to work around i think is that #2 "fully-pinned" is sort of a fairy tale
21:50:31 <dhellmann> an issue with (1) is that we can't sync those pins into the projects because then we'd never be able to change the pins because the tests couldn't install the incompatible projects together, so we need to just manage the pins in g-r
21:50:32 <fungi> er, #1
21:50:37 <ttx> adam_g: what do you think of jeblair's suggestions ?
21:50:45 <adam_g> #1 makes distro packagers lives a mess
21:51:05 <jeblair> adam_g: how so?
21:51:21 <fungi> adam_g: why are distro packagers caring what we have pinned in stable branches?
21:51:25 <dhellmann> and ftr, we haven't done (2) in the past because we want to catch incompatibilities in new releases of things we use early
21:51:30 <dhellmann> #writeitdown
21:51:37 <jeblair> dhellmann: indeed
21:51:42 <fungi> i thought they'd already settled on packaged versions of things when we branched stable from master?
21:51:43 <adam_g> jeblair, fungi because our stable branche requirements.txt ends up dictating what they need to ship in addition to our projects
21:52:03 <ttx> jeblair: I agree that we could do (0) -- it's just that those transitive deps issues seem to be common enough
21:52:17 <fungi> adam_g: don't they do that when we release? stable branch pinning is a post-major-release activity
21:52:55 <jeblair> adam_g: but if your proposal doesn't include changing them, then we are asking them to use known-broken requirements.
21:53:48 <adam_g> fungi, pinning them in stable means there will be potential problems when re-packaging our stable point releases, ie if distro has released some security fixes to a pinned dep
21:53:53 <jeblair> ttx: i feel like there is not universal agreement on that.  i don't want to argue against jogo who says he's tired of it, but i also do think we need some real time with caps to see if they have an effect.  :(
21:54:03 <ttx> OK, so it looks like the discussion can continue on the review --which was all +1 before this discussion
21:54:18 <jeblair> ttx: i will write a response there.
21:54:25 <jogo> jeblair: so since adding caps, we have already seen one failure
21:54:27 <jeblair> i think this was useful
21:54:31 <ttx> so I guess there is still value in raising specs at this meeting
21:54:35 * morganfainberg has some interesting thoughts suddenly.
21:54:48 <morganfainberg> let me stew on them and i'll write up my response to that review.
21:54:52 <jogo> https://bugs.launchpad.net/devstack/+bug/1430592
21:54:54 <openstack> Launchpad bug 1430592 in devstack "testtools-1.7.0 triggering pkg_resources.VersionConflict: (unittest2 0.5.1 (/usr/lib/python2.7/dist-packages), Requirement.parse('unittest2>=1.0.0'))" [Undecided,In progress] - Assigned to Ian Wienand (iwienand)
21:55:12 <jogo> honestly I didn't think this would happen much, but it did
21:55:26 <jeblair> that almost seems self-inflicted
21:55:33 <dhellmann> jogo: so another alternative is to cap "bad actors" as they come up
21:55:40 <dhellmann> or pin, rather
21:55:54 <jogo> dhellmann: no fires please, not less
21:56:01 <adam_g> s/as they come up/when we are wedged/
21:56:02 <bknudson> you pin it but then another packages unpins it.
21:56:03 <jogo> unless you are volunteering to put them all out
21:56:18 <dhellmann> adam_g: "as we learn that we have incompatibilities in the world" :-)
21:56:41 <morganfainberg> i mean. really silly question... is it possible to pull an entry point and programatically populate to setuptools the requirements?
21:57:00 <jeblair> jogo: wasn't that case related to installing from debs, not pip?
21:57:02 <dhellmann> morganfainberg: ?
21:57:14 <fungi> jogo: so can you describe how the proposed solution would have protected us from that bug while still allowing us to find out it's a problem?
21:57:20 <morganfainberg> e.g. g-r publishes as a package, and pbr or similar tool loads it, pushes the known versions of whatever pinned/capped to setuptools for install
21:57:55 <adam_g> we can have a perioidic job that runs against global-requirements.txt and finds conflicts
21:57:57 <morganfainberg> dhellmann, that way at least the fix is "push an updated g-r package" not chase down differnt dependency wonkyness in many openstack projects
21:57:59 <jogo> fungi: yes, it would have prevented it, because we would have had a pinned list of *all* dependencies, transient and all
21:58:24 <ttx> ok, please continue discussion on the review, and moving on to final topic
21:58:31 <ttx> #topic Open discussion & announcements
21:58:34 <jogo> and as adam_g just said, we have a periodic, say nightly, job that tries proposing new caps and if it passes the gate its good
21:58:35 <dhellmann> morganfainberg: I think I proposed that way back when pbr was just a twinkle in mordred's eye. It feels a bit weird to manage dependencies with an extra download, though
21:58:38 <morganfainberg> dhellmann, and then projects just say "i use 'testtools'" but the version is loaded from f-r.
21:58:47 <ttx> We had 1:1s syncs today in #openstack-relmgr-office, logs at:
21:58:49 <ttx> #link http://eavesdrop.openstack.org/meetings/ptl_sync/2015/ptl_sync.2015-03-17-09.00.html
21:58:52 <ttx> We discussed the last features and bugfixes before the kilo-3 tag
21:58:53 <mordred> aroo?
21:58:54 <morganfainberg> dhellmann, let me re-open that w/ mordred and some folks later today.
21:59:00 <ttx> (which coincides with Feature Freeze, so expect your favorite ML to be filled with FFE requests next week)
21:59:15 <dhellmann> morganfainberg: you want to use the package data api to load a file, though, not setuptools
21:59:33 <morganfainberg> dhellmann, will work out details on right things post meeting
21:59:36 * morganfainberg lets ttx carry on
21:59:47 <ttx> Well, we are mostly done
21:59:51 <ttx> Anything else, anyone ?
21:59:57 <mordred> ttx: I'm hungry?
21:59:57 <ttx> Announcement ?
22:00:01 <ttx> jeblair: outage ?
22:00:06 <ttx> mordred: Noma reopened
22:00:11 <mordred> ttx: I KNOW!
22:00:14 <mordred> ttx: TC meeting there?
22:00:20 <jeblair> ttx: gerrit outage saturday, email will be sent shortly.
22:00:44 <ttx> mordred: also "La Maison des bois" burnt down
22:00:52 <ttx> some equilibrium in the force
22:00:53 <fungi> also gerrit ip address change going along with the outage
22:00:54 <jeblair> (this was originally announced in februrary -- this is our server + OS upgrade/move)
22:01:01 <jeblair> and what fungi said ;)
22:01:04 <morganfainberg> fungi, fun times ahead.
22:01:17 <ttx> ok, closing down
22:01:21 <ttx> #endmeeting