15:59:59 <dims_> #startmeeting oslo
16:00:00 <openstack> Meeting started Mon Jun  8 15:59:59 2015 UTC and is due to finish in 60 minutes.  The chair is dims_. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:00:01 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
16:00:04 <openstack> The meeting name has been set to 'oslo'
16:00:12 <sileht> o/
16:00:14 <dims_> our agenda:
16:00:14 <dims_> #link https://wiki.openstack.org/wiki/Meetings/Oslo
16:00:18 <ihrachyshka> o/
16:00:25 <kgiusti> o/
16:00:27 <dims_> courtesy ping for jd__, dhellmann, bnemec, flaper87, harlowja, viktors, rpodolyaka, zzzeek, sileht, kgiusti, dansmith, kragniz, amotoki, lintan, toabctl
16:00:27 <dims_> courtesy ping for redrobot, jungleboyj, zhiyan, therve, amotoki, GheRivero, bknudson, ihrachyshka, jogo, dougwig, sreshetnyak, amrith,haypo, johnsom
16:00:33 <harlowja_at_home> yo yo
16:00:34 <redrobot> o/
16:00:37 <bknudson> hi
16:00:38 <ozamiatin> o/
16:00:41 <jungleboyj> o/
16:00:44 <kragniz> o/
16:00:45 <jecarey> o/
16:00:50 <bnemec> o/
16:00:50 <Alec> o/
16:01:03 <jd__> o/
16:01:43 <johnsom> o/
16:01:51 <dims_> giving another min for those who are still enjoying their weekend :)
16:02:03 <jungleboyj> :-)
16:02:15 <zzzeek> \o
16:02:31 <dims_> #topic Red flags for/from liaisons
16:03:08 <bknudson> for keystone, I don't think we'll be able to switch from timeutils.isotime
16:03:11 <dims_> so we had a couple of hiccups with isotime (one in oslo.serialization with nova and another in keystone with oslo.utils). are there other issues?
16:03:13 <bknudson> https://review.openstack.org/#/c/189147/
16:04:19 <dims_> jd__: ^^ i am leaning towards bknudson's and ayoung's pov to undeprecate for those uses where they are sticking the timestamp into a token and that can't change
16:04:33 <harlowja_at_home> :(
16:04:44 <harlowja_at_home> should we deprecate the deprecation ?
16:04:48 <flaper87> o/
16:04:56 <flaper87> (as in, I'm here)
16:05:20 <harlowja_at_home> ^ was a joke (mostly)
16:05:26 <harlowja_at_home> not flaper87 stuff (my own), ha
16:05:36 <dims_> harlowja_at_home: 2 messages instead of one? :)
16:05:39 <jungleboyj> Do two deprecations make a new feature?
16:05:42 <harlowja_at_home> :)
16:05:45 <jd__> dims_: I don't think it's a good idea to maintain code because that encourage spreading malpractice
16:05:51 <harlowja_at_home> why double, when we could triple it!
16:06:20 <dims_> jd__: right, but then we don't want folks to copy it as they will
16:06:53 <dims_> bknudson: the main issue with keystone is that the unit tests check for deprecation and fail fast right?
16:07:01 <jd__> I prefer that they copy it and have that in their corner rather than continuing spreading the evil
16:07:02 <bknudson> we need a standard format for timestamps, but isoformat has issues.
16:07:15 <jd__> we should not maintain that code
16:07:36 <dims_> bknudson: what are the issues? (instead of going back to the old impl)
16:08:07 <bknudson> one issue is that if the microseconds happens to be 0 it will not render the .microseconds
16:08:27 <bknudson> so you'll usually get H:M:S.micro but sometimes only get H:M:S
16:08:37 <dims_> gotcha.
16:08:57 <bknudson> there may be other differences in the output, too. I didn't look closely.
16:09:04 <dims_> jd__: can we add a toggle for that?
16:09:52 <bknudson> I don't think we should be using deprecated code and having warnings in the logs.
16:10:12 <bknudson> If it's going to be removed, we need to figure out eventually how to deal with it.
16:10:13 <jd__> dims_: that sounds "useless" but acceptable
16:10:22 <dims_> bknudson: how about we work offline on adding the 0's and bring it back to everyone if keystone folks still have an issue with the suggested replacement
16:10:46 <dims_> jd__: customer is always right :)
16:10:54 <bknudson> I'm not sure what's being proposed -- un-deprecate?
16:11:20 <dims_> bknudson: add support for 0's in isoformat
16:11:29 <bknudson> isoformat is python stdlib
16:11:51 <bknudson> monkeypatching?
16:12:18 <bknudson> https://docs.python.org/2/library/datetime.html#datetime.datetime.isoformat
16:12:40 <jd__> yeah that means maintaining yet again a function for no gain
16:12:42 <harlowja_at_home> un-un-un-deprecate it :-P
16:12:55 <bknudson> Is the format of isotime documented?
16:12:58 <jd__> I guess keystone can do that with their own function if they think it's useful?
16:13:06 <jd__> bknudson: you mean ISO8601?
16:13:42 <bknudson> the way I was proposing to do that is to copy isotime... we can clean it up since we've got a copy.
16:14:02 <bknudson> but as you can see the other cores are none too happy
16:14:22 <bknudson> jd__: iso8601 allows all sorts of formats.
16:14:36 <bknudson> you can have  T or use a space...
16:14:46 <jd__> bknudson: that's a format with choices :)
16:14:55 <jd__> and you can have zeros or not too
16:15:13 <bknudson> looks like you can leave off the :
16:15:18 <jd__> do we want people to add .0 everywhere? I don't think so
16:15:32 <bknudson> you can leave off the seconds and minutes, too
16:15:56 <dims_> k so let's take this offline please
16:16:01 <harlowja_at_home> hmmm, anyone able to get the iso8601 spec from the iso folks :)
16:16:09 <dims_> (after this meeting)
16:16:15 <dims_> any other issues from other liaisons?
16:16:40 <dims_> #topic Releases for this week
16:16:40 <dims_> Do we neeed any releases? Here are the unreleased changes
16:16:40 <dims_> #link http://paste.openstack.org/show/274337/
16:16:57 <jungleboyj> No issues from Cinder.
16:17:05 <dims_> please look at the paste and let me and doug know. he can cut releases tomorrow
16:17:05 * harlowja_at_home should get a taskflow release out this week
16:17:10 <dims_> harlowja_at_home: +1
16:17:19 <harlowja_at_home> tooz maybe to jd__ ?
16:17:36 <harlowja_at_home> automaton  and futurist maybe (when they are ready)
16:17:40 <harlowja_at_home> that'd be sweet
16:17:42 <jd__> next week I guess
16:17:49 <harlowja_at_home> k
16:18:14 <toabctl> pbr would be good to unblock sphinx >= 1.3 (but I don't know if there are other blockers)
16:18:23 <dims_> lifeless: ^^
16:18:54 <dims_> not sure if lifeless is around toabctl, will ping him when i see him
16:19:08 <dims_> switching topics
16:19:08 <toabctl> dims_: ok
16:19:12 <dims_> #topic New libraries and drivers - how is it going?
16:19:12 <dims_> #link specs - http://specs.openstack.org/openstack/oslo-specs/
16:19:37 <dims_> anyone doing oslo.service, oslo.cache, oslo.reports here today?
16:19:44 <ozamiatin> I am updating the https://review.openstack.org/#/c/187338
16:19:52 <harlowja_at_home> its going good i think (for my stuff)
16:20:13 <harlowja_at_home> https://review.openstack.org/#/c/189041/ i think ironic will use, although that does start to bring up more questions around periodics in other projects...
16:20:33 <dims_> ozamiatin: cool
16:20:53 <harlowja_at_home> oslo.service will have the periodic code?
16:20:54 <dims_> harlowja_at_home: Need cores there or just reviews? for futurist/automaton
16:21:35 <dims_> y oslo.service will have the periodic code from what i remember
16:21:54 <harlowja_at_home> either or, that brings up another question, i have volunteers that wouldn't mind being core reviewers there (at least a guy from taskflow said he'd be ok with being a automaton core), whats the policy there
16:22:20 <harlowja_at_home> dims_, hmmm, k, maybe should talk with oslo.service folks, perhaps they can use https://review.openstack.org/#/c/189041/
16:22:30 <dims_> harlowja_at_home: ping me please and we can send an email to the -dev list
16:22:33 <harlowja_at_home> k
16:23:12 <harlowja_at_home> from my memory the oslo.service periodic is tied to eventlet, this should decouple that
16:23:12 <dims_> harlowja_at_home: +1 to ping oslo.service folks (eezhova is not here i think)
16:23:30 <harlowja_at_home> cools
16:23:36 <dims_> k next up
16:23:39 <dims_> #topic Functional testing
16:23:39 <dims_> Any drivers need additional testing CI jobs? or help setting one up?
16:23:55 <sileht> harlowja_at_home, +1 for decoupling
16:24:01 <dims_> zzzeek: i saw something from fungi about alternate mysql driver
16:24:05 <dims_> on friday
16:24:21 <zzzeek> dims_: yes … i remember speaking to someone named fungi....
16:24:28 <dims_> https://review.openstack.org/#/c/184392/
16:24:29 <zzzeek> dims_: we’re just doing pymysql
16:24:44 <zzzeek> dims_: oh right
16:24:51 <harlowja_at_home> sileht, cool (https://review.openstack.org/#/c/189041/ has it running in threads/processes/ and greenthreads to!, freakin amazin!, ha)
16:24:52 <zzzeek> dims_: oslo.db test suite was fialing in some places
16:24:58 <zzzeek> dims_: and i said, “id look at it” :)
16:25:12 <dims_> ok i told fungi that i'll ping you :)
16:25:29 <zzzeek> dims_: OK so is this patch ready for me to go in and make it work ?
16:26:00 <dims_> zzzeek: i believe so and he was willing to help as well
16:26:23 <zzzeek> dims_: OK.   I just need to get the tests to pass shouldnt be too big a deal
16:26:37 <zzzeek> dims_: different error messages, names, etc
16:26:45 <dims_> ack zzzeek
16:26:50 <dims_> kgiusti: sileht: ozamiatin: any additional CI for oslo.messaging?
16:26:54 <kgiusti> dims_: flaper87 and I are still working towards amqp1 integration testing
16:27:06 <zzzeek> dims_: I dont understnad his last comment “I know for a fact this is going to require some heavy changes to a lot of the opportunistic tests, but not sure what the plan is for the existing mysql-connector and mysql-python tests (keep them? remove them?).”
16:27:29 <kgiusti> dims_: we've got a new release of pyngus staged that will address some of the required dependency problems on proton
16:27:35 <fungi> zzzeek: dims_: yep, i mostly didn't know how many alternate drivers we wanted to leave tests in for
16:27:50 <fungi> so was looking for suggestions (or takeover if you prefer)
16:27:55 <zzzeek> fungi: OK if those tests just run using mocked messages, no reason to just leave them in
16:28:02 <zzzeek> fungi: er, there’s no reason to remove them
16:28:07 <dims_> kgiusti: nice
16:28:37 <zzzeek> fungi: ideally oslo.db shoudl run the same with any DBAPI
16:29:56 <dims_> zzzeek, i had one more for you, were you able to run enginefacade with nova?
16:30:25 <zzzeek> dims_: well that’s a port of the code.  originally mbooth was working on it and rpodolyaka i thought indicated he’d give it some attention
16:30:30 <kgiusti> dims_: fyi: https://review.openstack.org/#/c/186671
16:30:55 <zzzeek> dims_: unless oyu mean, does nova as is work with the new oslo.db changes.
16:31:16 <kgiusti> dims_: also needs https://review.openstack.org/#/c/189287/ and https://review.openstack.org/#/c/186671/
16:31:35 <rpodolyaka1> dims_: zzzeek: haven't tried it with nova yet. reviewed patch to oslo.db today. I hope we'll merge it this week
16:31:36 <dims_> zzzeek: i had pointed out how to run nova with oslo.db master + pending oslo.db review.
16:31:50 <dims_> rpodolyaka1: ack
16:32:10 <zzzeek> dims_: yes, i was curious about that for when we start actualyl triyng to port nova
16:32:10 <dims_> kgiusti: ack will look at them later
16:32:23 <dims_> zzzeek: gotcha
16:32:31 <dims_> #topic Ongoing work & Review priorities
16:32:36 <zzzeek> dims_: mdbooth did a bunch of work on that and he might want to still be doing it, not sure
16:32:46 <dims_> zzzeek: ack
16:33:11 <dims_> Starting an etherpad for tracking priorities and focusing effort, i need help organizing it…
16:33:11 <dims_> #link https://etherpad.openstack.org/p/liberty-oslo-priorities-tracking
16:33:18 <harlowja_at_home> ah, new etherpad, sweet
16:33:40 <dims_> Example from Nova - https://etherpad.openstack.org/p/liberty-nova-priorities-tracking
16:34:00 <harlowja_at_home> man nova has a lot of work to do
16:34:02 <dims_> harlowja_at_home: as an experiment, let's see if it is useful or not
16:34:07 <dims_> haha
16:34:37 <dims_> any suggestions? or help welcome. just jump in and make changes works too
16:34:39 <harlowja_at_home> dims_, soooo should we have more categories
16:34:57 <dims_> ++
16:35:02 <harlowja_at_home> guess let's see
16:35:18 <dims_> k switching topics
16:35:20 <dims_> #topic Open discussion
16:35:24 <dims_> floor is open
16:36:26 <dims_> i filed several reviews to sync projects to latest oslo-incubator and fix up openstack-common.conf - if you see one please take a look or feel free to take over
16:36:28 * harlowja_at_home can rant on about the new libraries, if people wonder wtf they are
16:36:32 <harlowja_at_home> *rant == explain, ha
16:36:43 <dims_> harlowja_at_home: go for it
16:37:12 <Alec> ozamiatin: what is the exact status of the coding for the new zmq driver? did you start coding pieces or it is just the spec in gerrit right now?
16:37:13 <harlowja_at_home> so people probably wonder if they haven't seen wtf are automaton and/or futurist
16:37:33 <harlowja_at_home> sooo automaton is a state-machine library (contains the structure to make a state-machine and run it)
16:37:45 <harlowja_at_home> ironic is using a copy of it (to be removed someday) and taskflow also
16:37:48 <jungleboyj> dims_: I am meaning to check into the syncs for Cinder.  Hope to have some time this week.
16:37:54 <ozamiatin> Alec: I have started to implement the first spec with REQ/REP which is already approved
16:38:01 <harlowja_at_home> jungleboyj, cinder i think might have a use for automaton to (but not sure)
16:38:05 <dims_> jungleboyj: thanks
16:38:19 <harlowja_at_home> i'll put up some examples of using it, but basically it allows u to define transitions, states, and movement between them
16:38:25 <dims_> ozamiatin: cool, if u need a feature branch we can ping dhellmann
16:38:39 <harlowja_at_home> in ironic it is the thing powering http://docs.openstack.org/developer/ironic/dev/states.html (the diagram is created from the machine itself)
16:38:48 <Alec> ozamiatin: quite a lot to do to finish your spec, I see someone +2 it already
16:39:05 <Alec> and I barely started putting comments in there ;(
16:39:05 <harlowja_at_home> in taskflow it is the thing powering http://docs.openstack.org/developer/taskflow/states.html#engine
16:39:23 <ozamiatin> Alec, it's WIP status so it wouldn't be accidentally merged
16:39:26 <harlowja_at_home> so thats what automaton is :-P
16:39:41 <ozamiatin> Alec, feel free to put -1 if you disagree
16:39:46 <harlowja_at_home> people may have various usages of it, or some usages that i can't even think about!
16:40:04 <toabctl> harlowja_at_home: is there already a pypi release for automation?or at least a git repo?
16:40:07 <Alec> ozamiatin :I don't want to slow you down, if the plan is to make small steps and adjust I'm fine
16:40:18 <rpodolyaka1> harlowja_at_home: is current state persisted somehow? or it's in-memory only?
16:40:20 <harlowja_at_home> toabctl, why of course @ https://github.com/openstack/automaton
16:40:27 <harlowja_at_home> rpodolyaka, only in memory
16:40:35 <rpodolyaka1> harlowja_at_home: ack, thanks
16:40:45 <ozamiatin> dims_, feature branch sounds good
16:40:57 <harlowja_at_home> rpodolyaka, could be persisted somewhere, but thats a bigger scope, not impossible to do
16:41:08 <dims_> ozamiatin: let's ping dhellmann tomorrow
16:41:16 <ozamiatin> dims_, ok
16:41:21 <Alec> ozamiatin: I think adding more known ports could be a better approach if you do not want to see a name server, the fanout is a cause of concern for scale for me
16:41:45 <Alec> ozamiatin: also the management of new nodes
16:41:58 <rpodolyaka1> harlowja_at_home: yeah, I was just thinking this could help us to make code more declarative in other places as well (like state of floating IPs, etc)
16:42:08 <harlowja_at_home> rpodolyaka, def
16:42:38 <Alec> ozamiatin: what are the real forces pushing for zmq now that rabbitmq is solidly entrenched? This is goimg to be a long road
16:42:54 <ozamiatin> Alec, fanout is a big topic actually. I think we need to build some small prototypes and test them to make a final decision
16:42:54 <jungleboyj> harlowja_at_home: I will take a look at your link and get back to you from a Cinder perspective.
16:43:14 <Alec> ozamiatin: have you considered true multicast at all?
16:43:19 <harlowja_at_home> futurist @ https://github.com/openstack/futurist is right now a layer ontop of https://docs.python.org/3.4/library/concurrent.futures.html that adds things like a synchrnous executor (and a eventlet powered one to); with https://review.openstack.org/#/c/189041/ it can also gain periodic running functionality
16:43:41 <ozamiatin> Alec, do you mean PGM?
16:43:42 <harlowja_at_home> jungleboyj, cool, i believe hemana was thinking about it somehow for all the state validation stuffs there
16:43:56 <jungleboyj> harlowja_at_home: Ah, ok.  Good to know.
16:44:09 <Alec> ozamiatin: yes - unicasting for fanout could work at small scale but when you get to thousands of nodes that can be a rpoblem
16:44:49 <Alec> ozamiatin: I've had some experience with pgm which I can share- it can be used but only if we're ok with the limitations
16:45:10 <harlowja_at_home> futurist is used (or may be used) by oslo.messaging (unifying the executors there), taskflow (already used for running things), and ironic (i think wants to use the peridoci parallel stuff)
16:45:22 <harlowja_at_home> ^ u may have other ideas of using it also!
16:45:26 <ozamiatin> Alec, as I know PGM needs some hardware support to use?
16:45:45 <Alec> ozamiatin: you need multicast on your  control plane
16:46:05 <Alec> ozamiatin: but I'd say if you really want to scale out, that could be an OK requirement
16:46:07 <harlowja_at_home> sileht, correct me if i am wrong about the oslo.messaging usage :)
16:46:38 <Alec> ozamiatin: secondly you need to be realistic wrt to multicast throughput
16:46:51 <dims_> almost forgot to ask, does anyone have any stuck reviews? or specs?
16:46:57 <sileht> harlowja_at_home, you right the code is here: https://review.openstack.org/#/c/140985/
16:46:58 <ozamiatin> Alec, can you please start some e-mail thread and describe your vision about multicast?
16:47:09 <harlowja_at_home> dims_, oh right, i was also gonna ask about 2.6 support dropping
16:47:12 <harlowja_at_home> nearly forgot
16:47:13 <Alec> ozamiatin: ok will do
16:47:20 <ozamiatin> Alec, thanks!
16:47:32 <harlowja_at_home> for projects that have it, is there a timeline anywhere, (probably somewhere listed)
16:47:43 <dims_> +1 to start thread ozamiatin Alec
16:48:06 <dims_> harlowja_at_home: will need to talk to dhellmann on that one about latest ideas
16:48:10 <harlowja_at_home> k
16:48:17 <Alec> dims: coudl you anwer the first more general question wrt zmq support
16:48:35 <dims_> Alec: sorry, repeat it please?
16:48:49 <harlowja_at_home> dims_,  thx, don't want to be spreading around the 2.6 disease to new libraries if i can help it :-P
16:49:06 <dims_> new libraries, avoid it i'd say
16:49:12 <Alec> dims: looks like there is some interest from certain groups to push for zmq, do we know exactly who and how badly, given rabbitMQ is solidly entrenched now
16:49:33 <harlowja_at_home> ya, except those libraries get pulled into things that use 2.6 still (taskflow, tooz, oslo.messaging?) :-/
16:49:40 <harlowja_at_home> so its like a little infectious disease, lol
16:49:48 <Alec> dimsL reason I ask is we need something really solid and better than rabbitMQ to make it worth
16:49:53 <harlowja_at_home> aka the 2.6plague
16:49:54 <harlowja_at_home> lol
16:49:55 <dims_> Alec: so the main reason it's coming up is rabbitmq has issues when it comes to HA
16:50:26 <dims_> Alec: we are shackled to rabbitmq at the moment
16:50:34 <Alec> dims: yes that is known but I want to make sure we have it analyzed correctly before pushing out a solution
16:50:54 <dims_> Alec: sure, that's the reason for the spec process
16:51:15 <dims_> Alec: definitely it's a long road
16:51:22 <sileht> Alec, also zmq will not become the default, just an alternative to experiment and see if we can do better with zmq than rabbit
16:51:31 <dims_> sileht: right
16:51:31 <Alec> dims: from my experience with openstack scale, I can see that infra may not be the only issue
16:51:41 <ozamiatin> sileht: ++
16:52:02 <sileht> Alec, it's same kind of experiment we do we amqp1
16:52:28 <dims_> Alec: what other issues do you see? which may be missing in discussions or spec so far
16:52:33 <Alec> meaning suppose we come up with a "perfect" zmq driver, we may not be out of the woods yet because the other issue I see is how apps are using oslo messaging
16:52:51 <Alec> give you one example on a real deployment at scale
16:52:56 <dims_> Alec: absolutely
16:52:58 <Alec> rabbit MQ HA
16:53:10 <Alec> neutron server gets flooded by all the agents
16:53:36 * harlowja_at_home do i need to get some y! operators in here to talk about scale + rabbitmq, let me know if i need to
16:53:38 <Alec> and neutron server too slow to absorb the work load, and agents time out and resend more of the same requetss
16:53:43 <sileht> Alec, neutron have a really bad design (/me is thinking of this very big all in one daemon)
16:54:04 <dims_> harlowja_at_home: we'll set one up may be a hangout
16:54:14 <Alec> 1 problem is rabbitMQ keeps everything and too long, which causes lots of stale requests
16:54:15 <harlowja_at_home> dims_, cool beans, let me know, and i can drag in some folks, ha
16:54:16 <sileht> Alec, neutron guys should really split the code into many daemon like all other does
16:54:37 <sileht> Alec, I don't think, this is not a oslo.messaging issue
16:54:50 <dims_> sileht: so we have to tell them there's a problem and follow up i think
16:54:57 <dims_> sc68cal around?
16:55:02 <Alec> that's my point, I think we need to also fix how apps are using oslo messaging
16:55:19 <dims_> Alec: ack, that will help all drivers
16:55:27 <ozamiatin> Alec, definitely yes
16:55:51 <harlowja_at_home> ^ sounds like a best practices guide/docs, fixing all the projects 'ourselves' is something i've learned isn't sustainable
16:55:59 <dims_> Alec: please bring this up on the email you promised as well, i'll ask folks to respond
16:56:21 <Alec> ok I'll start an email thread on that because too much to say in IRC
16:56:22 <harlowja_at_home> *imho projects need to fix themselves if they can
16:56:23 <dims_> harlowja_at_home: y, it takes a while for what we propose to percolate to all the projects
16:56:34 <harlowja_at_home> ya
16:56:55 <sileht> for neutron this is not only 'how apps are using oslo messaging', but how ANY rpc application should work
16:57:42 <sileht> for example nova prepare a big payload to send to nova-compute to create a vm with all needed information
16:58:04 <sileht> neutron does multiple rpc round-trip to get all the neutr-agent need to create the port
16:58:24 <dims_> k 2 mins left :)
16:58:41 <dims_> any other thoughts, please head to oslo channel
16:58:47 <Alec> as a general comment the main problem I see is lack of visibility in oslo messaging (at the traffic level)
16:59:24 <dims_> thanks Alec
16:59:33 <dims_> thanks everyone, see you all next week
16:59:36 <dims_> #endmeeting