21:03:09 <ttx> #startmeeting crossproject
21:03:11 <openstack> Meeting started Tue Feb 24 21:03:09 2015 UTC and is due to finish in 60 minutes.  The chair is ttx. Information about MeetBot at http://wiki.debian.org/MeetBot.
21:03:12 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
21:03:13 <sdague> sure
21:03:14 <openstack> The meeting name has been set to 'crossproject'
21:03:16 <ttx> Our agenda for today:
21:03:17 <jogo> sdague: ++
21:03:20 <ttx> #link http://wiki.openstack.org/Meetings/CrossProjectMeeting
21:03:33 <ttx> #topic PSA from the Gatekeepers
21:03:40 <ttx> sdague: open fire
21:03:42 <notmyname> here
21:03:42 <sdague> so semver: X.Y.Z
21:04:00 <sdague> the Z is meant to be extremely minor, guarunteed compatible changes
21:04:08 <sdague> not just the next release
21:04:17 <SergeyLukjanov> o/
21:04:23 <mtreinish> sdague:
21:04:27 <sdague> definitely not a new release that has non Z bumps of dependencies  :)
21:04:33 <mtreinish> well that's a paste buffer fail
21:04:38 <ttx> yes, adding a dep is NOT minor
21:04:43 * devananda steps afk for a few minutes
21:04:53 <mtreinish> there it is: http://semver.org/
21:04:54 <sdague> or bumping keystoneclient a Y
21:05:02 <lifeless> its also in our pbr docs :)
21:05:05 <bknudson> we're going to have a lot of Y.0s.
21:05:09 <sdague> mtreinish: yep, good reference
21:05:15 <ttx> lifeless: damn people don't read
21:05:16 <lifeless> thats fine
21:05:23 <morganfainberg> bknudson, keystonelcient and middleware does have a lot of Y.0's
21:05:28 <sdague> bknudson: fortunately, there is an infininite number of numbers
21:05:29 <morganfainberg> bknudson, for that reason.
21:05:36 <sdague> so we have some room
21:05:41 <mtreinish> bknudson: and that's fine
21:06:05 <mikal> .
21:06:06 <clarkb> sdague: thankfully they are countably infinite
21:06:14 <ttx> sdague: more on that topic ?
21:06:38 <sdague> ttx: just that a lot of library releases recently have been defaulting to Z bumps
21:06:45 <sdague> and broken things pretty badly in the process
21:07:01 <ttx> question is... is that just a mistake, or were they thinking it really was only worth a Z ?
21:07:04 <sdague> so please really think about that, python-ceilometerclient was the latest one
21:07:08 <mtreinish> ttx: probably worth point out: http://lists.openstack.org/pipermail/openstack-dev/2015-February/057699.html
21:07:11 <sdague> right, I don't know
21:07:11 <lifeless> I might up-prioritise teh pbr semver finalisation
21:07:18 <lifeless> which will go a long way to fixing this
21:07:20 <ttx> because it feels like 75% of them are wrong
21:07:21 <bknudson> there's no review of release #s... just update the tag... seems easy to make a mistake.
21:07:37 <ttx> if only we could review the proposed tag
21:07:38 * eglynn puts hand up ... the python-ceiloclient was a thought-fail on my part
21:07:59 <sdague> anyway, mostly a PSA, so we try to do less of it in the future
21:08:04 <sdague> and I'm done
21:08:11 <lifeless> I think its worth a -dev mail
21:08:17 <lifeless> would you like me to send one?
21:08:25 <sdague> lifeless: sure
21:08:28 <ttx> lifeless: can't hurt
21:08:30 <ttx> alright, back to our regular schedule
21:08:37 <ttx> #topic Proposed evolution in release management tracking for Liberty
21:08:46 <ttx> #link https://wiki.openstack.org/wiki/Release_Cycle_Management/Liberty_Tracking
21:08:51 * devananda returns from afk
21:08:54 <ttx> So this is the result of a long discussion I had with various PTLs and release liaisons over the Kilo cycle
21:09:04 <ttx> Basically, we spend a lot of time and 1:1 syncs trying to craft a prediction of what will land in the next milestone
21:09:15 <ttx> And most of the time that prediction is blatantly false (we postpone a lot of stuff) and useless (nobody really consumes it)
21:09:25 <ttx> In particular, product managers appear to be more interested in a general idea of what's being worked on and its completion rate
21:09:33 <ttx> ...rather than a failed attempt at listing milestone we hope that will land
21:09:47 <ttx> So as far as release tracking is concerned, the idea would be to focus on communicating what landed at each milestone / release
21:09:59 <ttx> And then, encourage assignees to directly update the completion rate of their features on the blueprint page
21:10:08 <ttx> That doesn't prevent teams from having cycle goals and specifically track those
21:10:18 <ttx> But as far as release management is concerned I would only focus on what actually landed
21:10:28 <ttx> ...since my influence on what gets done everywhere appears to have... diluted
21:10:41 <ttx> Also note that this proposed change is orthogonal to the idea of reorganizing development cycles which is the new ML hotness
21:10:54 <ttx> Even if we switched to releasing more often, release management would still focus on what was done and what people are working on
21:11:05 <ttx> ...leaving goal setting and prioritization to each project team
21:11:17 <ttx> That should all free up more time to provide project teams with useful tools to facilitate their own tracking
21:11:21 <asalkeld> ttx: sounds like a great direction IMO
21:11:23 <ttx> How does that evolution sound ?
21:11:30 <devananda> +100
21:11:32 <zaneb> +1 I never understood how being able to predict accurately what would be in the release the day before the release was helpful to anyone
21:11:42 <morganfainberg> wfm
21:11:42 <ttx> zaneb: we used to be good at prediction
21:11:51 <rockyg> ttx: ++
21:12:00 <ttx> when I was more deeply involved in each project
21:12:03 <nikhil_k> +1
21:12:09 <ttx> but that doesn't scale
21:12:16 <ttx> Also I originally thought downstream users were consuming our per-milestone predictions, but apparently they are not
21:12:20 <morganfainberg> we need ttx clones for that to scale :P
21:12:22 <ttx> Maybe it's because those are unreliable, maybe that was never an interesting piece of info
21:12:32 <ttx> Anyway, focusing on setting goals/priorities, listing what's being worked on and tracking what actually landed sounds like a better use of our collective time
21:12:32 <jokke_> +1
21:12:37 <jungleboyj> ttx +1
21:12:42 <zaneb> ttx ++
21:12:45 <jroll> +1
21:12:48 <david-lyle> +1
21:12:51 <devananda> fwiw, I've started dragging BadCub into doing PM work for us (Ironic)
21:12:51 <eglynn> +1
21:13:14 <devananda> morganfainberg: ++ for ttx clones
21:13:21 <ttx> Also we'd still track RC bugs the same way we always did. This is just about meilstone targeting of stuff and keeping that accurate all the time
21:13:39 <ttx> anyway, we'd start that for Liberty
21:13:40 <notmyname> ttx: on that linked wiki page, you propose dropping 1:1s. but you didn't mentione that just now
21:14:01 <jokke_> ttx: getting too big to be fully up to date with everything? :P
21:14:07 <ttx> right... 1:1s were mostly used to make sure the milestoen page is up to date
21:14:23 <ttx> notmyname: I think two of them differed
21:14:34 <nikhil_k> I see a lot of value in 1:1 or some place to be able to do so
21:14:37 <ttx> The oslo one where I would actually coordinate release processes with Doug
21:14:49 <ttx> The swift one where we would exchange on what's going on in swift
21:15:04 <notmyname> I have found the weekly 1:1s to be useful and helpful
21:15:27 <devananda> ttx: I get value out of our 1:1's - but it's not in keeping the milestone page up to date ahead of time.
21:15:27 <ttx> Not exactly sure yet but I'll probably keep syncing with oslo and other delegated release teams
21:15:31 <rockyg> Having the equivalent of 1:1s with the PMs of any project that has them would go a long ways
21:15:36 <thingee> ttx: +1
21:15:51 <ttx> notmyname: I feel like it's the wrong forum to communicate though
21:16:14 <ttx> I see value in regular touch points on project status
21:16:39 <ttx> as a lightweight way to give a general idea of where you stand, without formal status reports
21:16:44 <rockyg> Release/PM liasons
21:16:48 <notmyname> ttx: the regularly scheduled time was the most helpful part. if we're going to regularly communicate, that's hard without a formal time. in part because we're in such different time zones
21:16:55 <ttx> but i'm not sure that has much to do with release management vs. general comm
21:17:16 <ttx> Most of the 1:1s end up being completely useless
21:17:16 <notmyname> I'd agree it's general communication more than release managerment (for me/swift)
21:17:35 <ttx> like .. let's see your milestona page, hmm.. keep up the good work... next
21:17:59 <notmyname> well, I do like the rest of the proposal to track reality instead of predictions :-)
21:18:06 <ttx> swift version: "any idea when your next release will be ? no... ok next
21:18:19 <david-lyle> information that used to be covered in the project meeting was relegated to the 1:1, where now?
21:18:25 <ttx> anyway, not totally sure how i'll evolve the format of 1:1s
21:18:27 <notmyname> to be fair, very few swift 1:1s have been like that :-)
21:18:53 <ttx> but I note that some of you like them enough to ask that they stay
21:19:04 <thingee> Generally I've communicated in these meetings deadline dates to ttx. Advice on actions or how courageous I want to be in a milestone have came from chats with previous PTL's or TC
21:19:06 <ttx> they just are vbery costly in time for me
21:19:24 <notmyname> ttx: realize that you've got a bunch of tech people actually asking for regular meetings. this is crazy!
21:19:27 <ttx> maybe we could mix several projects and do them at the same time
21:19:39 <ttx> like a morning slot and an end of day slots
21:19:48 <jokke_> ttx is using some kind of black magic
21:20:16 <ttx> (would avoid me repeating the same "warning, kilo-3 coming up" to 12 different projects)
21:20:27 <mestery> ttx: ++ to that idea!
21:20:39 <jokke_> ttx: you should be able to write really simple script to do that :D
21:20:47 <ttx> #action ttx to think about evolving 1:1 sync rather than removing them
21:21:02 <rockyg> Instead of project meeting, have a release meeting slot and anyone who wants to discuss the release schedule/status shows up
21:21:21 <notmyname> I'd worry about bystander effect where something isn't said because of other people talking or waiting for someone else to say something (if they became 1:*s instead of 1:1s)
21:21:26 <rockyg> But leave project meeting.
21:21:31 <notmyname> but that can probably be dealt with
21:21:40 <ttx> OK, let's move on, i'll think about it and propose on the ML
21:21:41 <david-lyle> I'm ok with their removal, I would like to have perhaps some common reminders at the beginning of this meeting
21:21:44 <SergeyLukjanov> probably all ptls could prepare a weekly report weekly to share status
21:21:48 <ttx> just wanted to see if the general direction was good or not
21:21:54 <SergeyLukjanov> ttx, like moving part of the responsibility to ptls
21:22:03 <notmyname> 1:1s not as a release thing but as a "communicate with the chair of the TC" thing
21:22:12 <devananda> notmyname: ++
21:22:29 <ttx> could do formlal offie hours
21:22:34 <devananda> also as a collective body of notes afterwards of what each PTL thought worth sharing
21:22:36 <ttx> formal* office*
21:22:38 <notmyname> PTLs responsible (more) for releases and release tracking. 1:1s for communicating that to the rest of the openstack org
21:22:38 <morganfainberg> ttx, that probably is a good compromise
21:23:03 <morganfainberg> it lets us quickly sync up as needed but not mandate it if nothing needs ot be said
21:23:05 <notmyname> devananda: +1
21:23:16 <ttx> ack
21:23:28 <rockyg> And if there are no updates for the week, PTL responsible for cancelling 1:1
21:24:02 <ttx> ok, moving on
21:24:05 <ttx> #topic openstack-specs discussion
21:24:12 <ttx> * Add TRACE definition to log guidelines (https://review.openstack.org/#/c/145245)
21:24:22 <ttx> This one should be ready to move to final approval at next week TC meeting
21:24:28 <ttx> Minor nits can be fixed on subsequent patches
21:24:38 <ttx> so if you disagree, file your -1 asap
21:24:50 <ttx> otherwise I'll have it rubberstamped by the tc next sweek
21:24:53 <ttx> or week
21:25:03 <ttx> * Cross-Project spec to eliminate SQL Schema Downgrades (https://review.openstack.org/#/c/152337/)
21:25:22 <ttx> This one may be ready too. Just needs to pile up approvals on the latest patchset. If it gets enough I'll put it on next week TC agenda as well
21:25:48 <ttx> mikal: if you still ahve an objection you might want to repost it
21:26:01 <ttx> * Eventlet Best Practices (https://review.openstack.org/#/c/154642/)
21:26:09 <ttx> So for this one, some people are wondering if openstack-specs is the right place
21:26:19 <ttx> Sounds like development tips and tricks rather than an actionable change or transition
21:26:28 <ttx> what do you all think ?
21:26:29 <morganfainberg> this feels like a devleoper doc? not a "things for people to do"
21:26:47 <morganfainberg> so. -specs is a little odd imo
21:26:52 <ttx> do we have a place for general development tips ?
21:26:58 <bknudson> wiki
21:27:02 <ttx> apart from undiscoverable wiki pages ?
21:27:09 <mtreinish> ttx: blog posts?
21:27:25 <ttx> mtreinish: maybe
21:27:45 <dhellmann> I think the source of this is in Oslo where we've started moving some of our "policies" that don't correspond to direct actions into the specs repo where we can vote on them, rather than just having them in the wiki.
21:27:48 <jokke_> ttx: I think that should be in wiki not in the OS-specs
21:28:06 <dhellmann> bnemec may just move this to the oslo.concurrency docs, but wanted a little more visibility to start
21:28:28 <sdague> nova has a devref section, it might be a good thing to propose a wider version of that
21:28:49 <dhellmann> but in general Oslo is not adding a lot of content to the wiki these days, because we're finding lib and spec repos better at hosting them
21:29:07 <dhellmann> sdague: good idea, maybe you can leave that comment on the spec if you haven't already?
21:29:13 <devananda> ttx: on the sql downgrades spec, do we actually have sufficient feedback from operators to know that they aren't going to rage when we do that?
21:29:15 <rockyg> The results belong in developer docs, including reviewer checklists, but the discussion and final resolution that leads to the docs needs a home
21:29:36 <bknudson> so openstack-specs becomes a cross-project developer docs?
21:29:37 <sdague> dhellmann: yeh, will do
21:29:40 <dhellmann> sdague: thanks
21:29:41 <ttx> hrm, looks like I have network issues
21:29:58 <jokke_> rockyg & all: do we need devdocs repo?
21:30:08 <ttx> If I disappear, someone can take over the meeting agenda and finish it
21:30:16 <sdague> bknudson: no, I'm suggesting there is something else that we need. devref at a global level
21:30:16 <morganfainberg> devananda, i would like to hold off until post operator summit in philly (cc ttx) for the downgrade thing
21:30:22 <morganfainberg> see if we can get some more feedback directly
21:30:27 <sdague> morganfainberg: good idea
21:30:33 <morganfainberg> if someone who is going to be there can poke at people
21:30:33 <dhellmann> bknudson: maybe? since they were sort of "guidelines" we thought this would be a good place to get reviews
21:30:38 <morganfainberg> since i will not be at the operator summit
21:30:42 <rockyg> like sdague said, devref for all O.O projects
21:30:47 <sdague> morganfainberg: I can bring it up
21:30:47 <morganfainberg> s/summit/midcycle
21:30:50 <morganfainberg> sdague, thanks
21:31:04 <sdague> morganfainberg: can you poke Tom about it, so he can figure out where on the agenda it should go
21:31:10 <morganfainberg> yes will do
21:31:13 <ttx> morganfainberg: ok, could you annotate the review as such ? like -1ing it so that I don't miss it ?
21:31:16 <bknudson> I feel like a cross-project devref has been discussed before...
21:31:22 <morganfainberg> he's usually awake late pacific time, right?
21:31:32 <morganfainberg> ttx, will do.
21:31:58 <devananda> morganfainberg: ++. or even until post-summit
21:32:24 <ttx> So back to central devref... I'll kick off a thread on that
21:32:46 <morganfainberg> devananda, i think this is a case where the operator focused thing at philly will be easier to get clear feedback that an the summit. but in either case.
21:32:51 <ttx> #action ttx to start thread on central devref for "Eventlet Best Practices" like things
21:32:54 <dhellmann> ttx: if we think that's a good idea, I can propose creating a repository
21:33:00 <morganfainberg> devananda, a minor delay would be good *kicks that enter key*
21:33:07 <ttx> dhellmann: you take the action ?
21:33:11 <dhellmann> ttx: sure
21:33:20 <ttx> I'd rather have athread first
21:33:35 <dhellmann> #action dhellmann start thread on publishing a central developer reference
21:33:36 <ttx> to check if the idea is good before moving to implementation
21:33:46 <ttx> * Add library stable release procedures/policy (https://review.openstack.org/#/c/155072/)
21:34:10 <ttx> dhellmann: up to introducing that one ?
21:34:10 <bknudson> I like the idea of a cross-project devref... would make it easier to see where projects have diverged and could consolidate.
21:34:22 <dhellmann> ttx: sure
21:34:46 <dhellmann> this spec is related to the requirements management changes we've made, where we are now capping libraries in stable branches
21:35:02 <dhellmann> in the past, we've not had stable branches for the clients and we've only had them for some oslo libraries
21:35:21 <dhellmann> my proposal  is that since we are now capping all libraries, we should have stable branches for all of them and do backports
21:35:36 <ttx> rare backports
21:35:44 <dhellmann> sdague brought up semver earlier in the meeting, and this assumes we're all following the semver rules
21:35:47 <ttx> as in security-fix-backports
21:35:57 <dhellmann> right, or serious crashes or something
21:35:58 <dhellmann> not features
21:36:13 <ttx> or OMG-that-bug-eats-my-data-backport
21:36:18 <jogo> dhellmann: would these stable branches be lazy created, as on only made when needed?
21:36:32 <dhellmann> jogo: in order for some of the CI jobs to work correctly, we need to create them all proactively
21:36:33 <bknudson> we haven't done security fix backports for clients... also I'm not sure that packagers would pick the same version # for their packages...
21:36:49 <dhellmann> jogo: some of the jobs do things like check out stable/foo and fall back to master
21:36:55 <morganfainberg> dhellmann, i've been thinking that stable client branches might need to occur in general.
21:37:04 <dhellmann> morganfainberg: right, that would be part of this, too
21:37:06 <bknudson> if we did happen to pick the same version to ship with the release then having security backports would be nice.
21:37:33 <morganfainberg> it would actually solve an issue i have with keystoneclient
21:37:38 <dhellmann> bknudson: well, the idea is we would start saying "inside your cloud, use these versions of the client; outside of your cloud use whatever you want"
21:37:40 <ttx> bknudson: distros increment their side of the version number, not ours
21:37:48 <morganfainberg> because then we could easily remove things that are moribund / bit rotting after a reasonable time
21:37:58 <jogo> dhellmann: hmm, it may be easier to change the tests versus creating all those branches when most wont be used
21:37:59 <morganfainberg> rather than carrying it indefinitely
21:38:25 <morganfainberg> it changes how clients can be viewed overall
21:38:31 <dhellmann> jogo: most of this can be automated, and the work is distributed among all of the teams, so I don't think it's going to be that much work
21:38:38 <rockyg> morganfainberg: +1
21:38:48 <dhellmann> if we delay creating the branches, then fixing a critical bug becomes a big infra push instead of just a patch in the right place
21:38:58 <jogo> dhellmann: true
21:39:07 <dhellmann> fewer people have the ACLs to create branches than approve backports
21:39:31 <jokke_> how about <next minor and the next lib/client release after RC1 needs to bump minor?
21:39:35 <dhellmann> so it just becomes part of our end-of-cycle release process, and most teams only have one or two anyway -- oslo is going to be impacted the most
21:40:05 <dhellmann> jokke_: I have some timing details in the proposal
21:40:16 <lifeless> mail sent on semver
21:40:28 <jokke_> that would leave the X.Y.<what-ever-needed> as "stable" branch of lib
21:40:30 <dhellmann> we'll synchronize using the global requirements caps
21:40:42 <dhellmann> jokke_: yes, that's the idea
21:41:03 <dhellmann> oslo has been doing this, so the proposal is just to expand it to everyone else now, too
21:41:07 <sdague> so, I actually think the issue is g-r sync with libraries
21:41:13 <jokke_> dhellmann: ++
21:41:27 <sdague> because g-r works really well when everything releases all at once
21:41:40 <sdague> but with adhoc releasing, we can see how it breaks down
21:42:03 <sdague> and the adhoc releasing has been really good for other reasons, but put new strains on the old system that didn't anticipate it
21:42:05 <bknudson> dhellmann: have you had a security backport? I don't think I've seen one.
21:42:12 <jokke_> dhellmann: although that saves us only on OS ... wish we could expand that to 3rd parties as well :P
21:42:24 <dhellmann> bknudson: we haven't had a security issue, but we did have some backports of bug fixes into juno
21:42:26 <jogo> sdague: and g-r artificially bumps minimums for a lot of things
21:42:54 <ttx> so.. please provide comments on that review if you care
21:43:02 <dhellmann> right
21:43:32 <ttx> unless there are more pressing questions/comments on this one, I propose we move on ?
21:44:03 <ttx> taking silence as yes
21:44:05 <ttx> * Replace eventlet with asyncio (https://review.openstack.org/#/c/153298/) vs. Replace eventlet + monkey-patching with threads (https://review.openstack.org/#/c/156711)
21:44:20 <ttx> So those are two competing way forward on how to move from eventlet
21:44:32 <ttx> even if we did not formally decide to move on
21:44:41 <ttx> I think it's not bad to lay the options out there
21:44:59 <asalkeld> option 3 == stay with eventlet
21:45:13 <devananda> ttx: on this, I'm not yet convinced that there's significant enough benefit for us to mandate a whole bunch of work when, yea, we haven't even agreed that we need to move on right now
21:45:21 <ttx> I think both are missing an analysis of the switching cost and a perf analysis
21:45:23 <bknudson> keystone went to running server in apache httpd... instead.
21:45:25 <morganfainberg> asalkeld, option 4 == mod_wsgi/uwsgi/unicorns
21:45:38 <dhellmann> the issues with eventlet that caused us to start this discussion (the python 3 support) are getting better, but are still not 100% resolved afaik
21:45:59 <morganfainberg> bknudson, ++ and we're slating a deprecation of eventlet this cycle and complete removal in M
21:46:04 <sdague> morganfainberg: mod_wsgi works less well for non API services
21:46:08 <ttx> devananda: yes, I agree with you
21:46:12 <morganfainberg> sdague, fair point
21:46:15 <ttx> just wondering what to do with those
21:46:16 <devananda> sdague: ++
21:46:17 <dhellmann> ttx: ++ on the performance analysis. I think haypo is working on a version of ceilometer that doesn't need eventlet as a demo of the scale of changes that need to be made.
21:46:21 <sdague> the fact that keystone has no workers makes it special in that way
21:46:28 <SergeyLukjanov> ++ for perf analysis
21:46:33 <ttx> I liked zzzek(sp?) analysis
21:46:39 <devananda> ironic-api works fine in apache mod_wsgi, but the bulk of our work (like nova) is not in the API service
21:46:44 <morganfainberg> ttx, a few more eees in there but close enough.
21:46:53 <eglynn> one clear difference between the options IIUC is that the asyncio plan involves limiting the initial change to oslo-messaging and ceilometer
21:46:57 <eglynn> (or some subset thereof)
21:47:01 <dhellmann> yeah, we recognize that any solution has to support WSGI, SQLAlchemy, and the message bus
21:47:06 <eglynn> whereas the threads apprach would require an audit across all projects
21:47:07 <devananda> is this actually about performance at all?
21:47:14 * asalkeld not crazy about asyncio
21:47:15 <devananda> or just about py3 support (or lack thereof)
21:47:20 <eglynn> more correctness than perf
21:47:24 <ttx> The only benefit I see to syncio is that it's the "future in Python"
21:47:28 <ttx> asyncio*
21:47:32 <dhellmann> eglynn: well, that initial proposal is just to give people an idea of the changes, but would eventually apply to all projects
21:48:02 <dhellmann> devananda: mostly py3 and developer issues, but we don't want performance to degrade heavily
21:48:03 <eglynn> dhellmann: true, once proven out to some extent presumably
21:48:07 <morganfainberg> if eventlet grows py3 support - there is no reason we can't stay with it. It does obscure things in wierd ways because of the monkey patching, where asyncio is explicit development [afaict]
21:48:09 <dhellmann> eglynn: right
21:48:24 <morganfainberg> so, it becomes the question of "which way do we want to go"
21:48:27 <eglynn> unfortunately haypo doesn't seem to be around to represent the asyncio case
21:48:37 <dhellmann> morganfainberg: the monkey-patching is starting to cause conflicts with import order in some of the applications
21:48:42 <morganfainberg> dhellmann, aye.
21:48:48 <sdague> eventlet 0.17.0 (released yesterday) is passing all it's tests on py3
21:48:50 <sdague> fwiw
21:48:50 <morganfainberg> dhellmann, as said because monkeypatching is wierd.
21:48:58 <dhellmann> we had a bug filed against oslo.concurrency, let me see if I can find that
21:49:07 <morganfainberg> dhellmann, actually we hit something in keystone regarding log stuff and threading.lock issues
21:49:10 <devananda> moving away from monkey patching would be great. but is it worth a massive amount of work? I dunno -- seems like a per-project decision
21:49:12 <ttx> I don't expect us to settle it in the next 10 minutes... I just wonder what to exactly do with those
21:49:15 <devananda> not something we should be mandating
21:49:19 <jogo> so this sounds like something that we may not need to have a openstack wide decision on
21:49:19 <dhellmann> sdague: yeah, I haven't looked yet to see if their py3 support now monkeypatches everything. At one point it didn't.
21:49:25 <morganfainberg> dhellmann, it has a lot of subtle pitfalls
21:49:33 <jokke_> devananda: ++
21:49:34 <jogo> not sure what the risk is for projects each doing there own thing.
21:50:02 <dhellmann> #link https://bugs.launchpad.net/oslo.concurrency/+bug/1418541
21:50:03 <openstack> Launchpad bug 1418541 in neutron "processutils checks whether stdlib is monkey patched during import" [Undecided,Fix committed] - Assigned to Ihar Hrachyshka (ihar-hrachyshka)
21:50:09 <bknudson> if it works really great for ceilometer or whoever picks it then that can convince others.
21:50:12 <sdague> jogo: that was one of the reasons I think there should be a higher level statement of some sort
21:50:17 <dhellmann> jogo: well, for one thing we don't want to have to support all of the various options in oslo
21:50:20 <eglynn> jogo: difficulty in on-boarding devs moving between projects for one thing
21:50:27 <jokke_> jogo: I guess we will see in a while after we get the feedback from keystone apache trial ;)
21:50:29 <sdague> eglynn: yes, exactly
21:50:39 <devananda> dhellmann: if oslo switches, what is the immediate impact to projects?
21:50:40 <dhellmann> because we don't want to have to have forks of libs for eventlet, asyncio, and threading support
21:50:41 <ttx> yeah, I'm not very excited at the prospect to learn 5 different async models
21:50:51 <SergeyLukjanov> ttx, ++
21:50:59 <sdague> eglynn: also, the difficulty for ops to track down issues
21:51:01 <morganfainberg> jokke_, we have recommended apache be used for 4 cycles now, and most scaled up deployments use it. so i think the feedback has been "yep this is working" ;)
21:51:02 <jogo> eglynn: I would be really surprised if having multiple options would make moving between projects harder
21:51:03 <notmyname> is anyone else concerned that the 2 projects that are proposing to be test cases for dropping eventlet have plugins (middleware) for every other project?
21:51:06 <jogo> dhellmann: yeah that is fair
21:51:08 <ttx> think about those of us that are cross-project, and how to facilitate their work rather than complicate it
21:51:09 <dhellmann> devananda: I want Oslo to push for a decision, but follow the overall project on this.
21:51:15 <devananda> dhellmann: I'm guessing it would break them. which means coordinating the roll-out of that change in oslo becomes a support headache
21:51:19 <morganfainberg> jokke_, and devstack defaults to apache for keystone too
21:51:21 <sdague> if every project has different concurrency models
21:51:31 <devananda> dhellmann: which means oslo can't make the change unless every project agrees?
21:51:36 <devananda> I hope I'm wrong on that
21:51:43 <ttx> i mean, I'm fine with migrating from one to another, but not very thrilled at the idea of encouraging everyone to pick
21:51:44 <eglynn> jogo: a lot of the perceived problem is the subtle bugs that arise, and each approach has a different class of subtle pitfalls IIUC
21:51:50 <dhellmann> devananda: oslo.messaging already supports both threads and eventlet, but the application has to invoke it differently
21:51:58 <bknudson> does having keystone running in httpd cause cross-project problems?
21:52:03 <jogo> well we are really taking about 3 models? thread based (putting mod_wsgi in here), eventlet, asynicio
21:52:14 <bknudson> we're kind of stuck since we need apache for federation.
21:52:14 <dhellmann> so we're not going to just say "oslo is doing this" and start breaking things -- that's why we're having this as a cross-project discussion instead of an oslo spec
21:52:30 <morganfainberg> bknudson, as far as i know there are minor issues with docs and questions about restarting keystone. but largely no.
21:52:32 <bknudson> I know it makes the test logs kind of iffy.
21:52:33 <sdague> jogo: mod_wsgi really is a 4th thing, as it's process forks
21:52:37 <devananda> dhellmann: gotcha
21:52:42 <morganfainberg> bknudson, and logging is ... well logging.
21:52:42 <ttx> jogo: so we'd say "one of those" rather than "whatever you want" ?
21:52:50 <jogo> ttx: sure
21:52:54 <devananda> sdague: mod_wsgi is orthogonal since it doesn't address non-API services
21:52:57 <ttx> jogo: sounds slightly better
21:53:01 <sdague> devananda: sure
21:53:13 <dhellmann> sdague, jogo : I'm lumping "wsgi" into one thing, since it really shouldn't matter to the code which container it's running in
21:53:15 <sdague> bknudson: wsgi should be fine for API surfaces
21:53:44 <morganfainberg> sdague, i would say wsgi would be a good choice for api surfaces. typically would be my choice if I could encourage all the projects to move that way.
21:53:51 <morganfainberg> sdague, but that is a tall order as well
21:54:01 <devananda> morganfainberg: ++ for api surfaces
21:54:28 <dhellmann> morganfainberg: all of our rest APIs are WSGI now, served with the eventlet WSGI container. We should be able to plug them into something else if we stop using eventlet.
21:54:46 <dhellmann> unless you found out otherwise in keystone?
21:54:51 <ttx> hmm, so rather than dive more into details... what is the way forward for these specs ? Iterate until they ahve a clearer cost/benefit analysis ?
21:54:58 <devananda> dhellmann: you probably already said it, but what's the impact on oslo to (continue) support(ing) both eventlet and asyncio?
21:54:59 <morganfainberg> dhellmann, i meand mod_wsgi sorry missed a 'mod_' in there. mod_wsgi/uwsgi/unicorns
21:55:03 <bknudson> we've got separate launchers, one for eventlet-server and one for apache.
21:55:16 <jogo> ttx: I like sdague's idea of a higher level spec of some sort
21:55:20 <morganfainberg> dhellmann, but yeah there is some magic needing in some cases because things are... well.. odd with eventlet
21:55:27 <dhellmann> ttx: yeah, it would be good if folks would post some of these questions on the specs so the authors know what sort of additional research is needed
21:55:31 <ttx> jogo: agree -- the need to change and the options
21:55:38 <morganfainberg> dhellmann, so there is a bit of extra wrapping needed in most cases.
21:55:50 <dhellmann> morganfainberg: right, that's why we want to stop using eventlet :-)
21:55:53 <jogo> having a single spec with all the recommended options, along with tradeoffs, would be a nice end result
21:55:59 <morganfainberg> :-)
21:56:01 <dhellmann> sdague: what higher level spec would you want to see?
21:56:13 <bknudson> here's the keystone launchers: http://git.openstack.org/cgit/openstack/keystone/tree/keystone/server
21:56:13 <ttx> sdague: would you take the action of suggesting a meta spec and/or merging the options into a metaspec ?
21:56:25 <sdague> dhellmann: well, a statement of direction of where we are headed
21:56:37 <sdague> because "in all the directions" seems ... unproductive
21:56:39 <ttx> dhellmann: I think both options need to tackle the cost/benefit analysis for change
21:57:01 <dhellmann> sdague: we have 2 competing proposals and we want folks to express opinions between them. Does a meta-spec make that easier?
21:57:02 <sdague> ttx: honestly, I'm not sure I'm a good person to do that, I'm mostly just asking questions.
21:57:08 <ttx> ok, I'll take it
21:57:21 <sdague> dhellmann: oh, I'm fine with the existing proposals to flesh this out
21:57:28 <ttx> #action ttx to suggest some metaspec on the various async potential plans
21:57:32 <sdague> sorry, I didn't mean we needed a 3rd thing to explore that
21:57:47 <lifeless> dhellmann: so I'd like to get rid of the message bus as it stands today; its a source of bugs (they way we use it, not it per se)
21:57:48 <sdague> I think we need to be driving to a set of project wide guidelines at the end of the day
21:57:49 <dhellmann> yeah, ok, I think ttx misunderstood? or maybe he wants it?
21:57:50 <ttx> I think they can all work on the same spec with options
21:58:00 <dhellmann> lifeless: that is a whole different spec
21:58:15 <sdague> and a whole different set of bugs :)
21:58:25 <lifeless> jogo: the risk is having different constraints placed on oslo
21:58:34 <dhellmann> ttx: ok, the template has a single proposal section, so I suggested the two camps write separate specs
21:58:55 <ttx> dhellmann: I'm fine with bending the template on that specific case
21:58:57 <lifeless> dhellmann: yes, I know it is. Just catching up as quickly as I can w/backlog
21:59:07 <ttx> still a long way to go anyway
21:59:13 <dhellmann> I thought we'd roll the rejected one into the approved one as a proposed alternative, at the end, but whatever you like is fine
21:59:16 <dhellmann> lifeless: ok
21:59:18 <ttx> There was a last spec on the docker:
21:59:21 <ttx> docket*
21:59:27 <ttx> freudian slip
21:59:29 <dhellmann> heh
21:59:32 <ttx> * Return request ID to caller (https://review.openstack.org/156508) -- proposer not able to attend, please provide feedback on spec
21:59:40 <ttx> #topic Open discussion & announcements
21:59:45 <ttx> We had 1:1s syncs today in #openstack-relmgr-office, logs at:
21:59:48 <ttx> #link http://eavesdrop.openstack.org/meetings/ptl_sync/2015/ptl_sync.2015-02-24-09.04.html
22:00:29 <ttx> (that last spec above is about cross-project request ID tracking)
22:00:45 <ttx> (which is definitely a good thing to have)
22:00:47 <notmyname> ttx: seems to be cinder-specific
22:00:49 <rockyg> Operators really need something in that area
22:00:58 <ttx> notmyname: yeah, I was a bit confused at first read
22:01:10 <ttx> seems to start general and then be only cinder
22:01:11 <mtreinish> ttx: there was previous work on that at some point
22:01:14 <jokke_> who ever is attending to that ops meetup ... get them to comment in as well
22:01:35 <sdague> do we have some history for why that was landed here?
22:01:40 <rockyg> jokke_: good point
22:01:51 <sdague> thingee: ?
22:01:53 <ttx> sdague: nope
22:01:59 <jungleboyj> notmyname: It started in Cinder, I believe, but as it has been further investigated it is broader.
22:02:10 <ttx> probably needs to be less cindery
22:02:11 <rockyg> I think it was because a change in one project would cascade to others
22:02:17 <ttx> anyway, time is up
22:02:21 <jungleboyj> Requires changes in Nova and anyone else using cinder
22:02:27 <ttx> thx everyone!
22:02:28 <ttx> #endmeeting