19:00:45 <flaper87> #startmeeting marconi
19:00:46 <openstack> Meeting started Thu May 16 19:00:45 2013 UTC.  The chair is flaper87. Information about MeetBot at http://wiki.debian.org/MeetBot.
19:00:47 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
19:00:49 <openstack> The meeting name has been set to 'marconi'
19:00:54 <flaper87> w0000000t
19:01:01 <flaper87> marconi folks around?
19:01:08 <malini> yes Sir
19:01:22 * flaper87 wonders what's alessio's nick
19:01:35 <aababilov> Hi! It's Alessio!
19:01:41 <flaper87> aababilov: there you are
19:01:49 <aababilov> si! :)
19:02:04 <flaper87> soooo, it looks like it'll be the three of us today
19:02:06 <flaper87> https://wiki.openstack.org/wiki/Meetings/Marconi
19:02:14 <flaper87> that's the schedule for today
19:02:29 <flaper87> before we get there, is there anything we should know share or something?
19:03:00 <flaper87> kk
19:03:10 <flaper87> #topic blueprints
19:03:15 <flaper87> #link https://blueprints.launchpad.net/marconi
19:03:42 <flaper87> I know last week you guys went through the blueprints, I think we should do that again today and see which of those are really esential
19:03:54 <kgriffs> kk
19:04:09 <flaper87> so, we've got the base blueprints for transports and sotrages which are obviously essential
19:04:15 <kgriffs> I guess we can move grizzly-debt down a notch
19:04:25 <kgriffs> (or two)
19:04:27 <flaper87> and we've also got the transport wsgi and mongodb that are essential as well
19:04:29 <flaper87> kgriffs: +1
19:04:47 <flaper87> #link https://blueprints.launchpad.net/marconi/+spec/grizzly-debt
19:04:50 <flaper87> kgriffs: done
19:05:15 <flaper87> #agreed setting grizzly-dept to medium
19:05:34 <flaper87> #link https://blueprints.launchpad.net/marconi/+spec/config-module
19:05:47 <flaper87> I think this one is ready
19:06:02 <flaper87> what do you guys think?
19:06:23 <kgriffs> yeah, it's been stable for a while now
19:06:41 <flaper87> cool, so, I guess we should set it as implemented
19:06:42 <kgriffs> BTW, I'm thinking to move grizzly-debt to H2
19:06:55 <flaper87> kgriffs: makes sense
19:07:02 * kgriffs does that
19:07:10 <flaper87> kgriffs: what about adding work items to it?
19:07:20 <flaper87> like Pay dept on transport
19:07:24 <flaper87> Pay debt on storage
19:07:26 <flaper87> and so on
19:07:39 <flaper87> so we can have a single blueprint and a more organized tasks for that BP
19:07:58 <kgriffs> I like that idea. We haven't been using that feature yet.
19:08:00 <kgriffs> +1
19:09:01 <aababilov> +1
19:09:26 <flaper87> cool
19:09:28 <aababilov> I really wondered what exactly means " fix all the broken windows"
19:09:44 <flaper87> #agreed add work items to grizzly-dept
19:09:46 <aababilov> do we have appropriate FIXMEs?
19:10:01 * flaper87 set grizzly-debt as implemented by mistake.
19:10:05 <flaper87> I put it back as started
19:10:32 * kgriffs seeded with a few work items
19:10:44 <flaper87> awesome
19:10:53 <aababilov> ok, 'cause now I see just one FIXME and 7 TODOs
19:11:05 <flaper87> so, I think essential blueprints are set
19:11:20 <flaper87> aababilov: mmh, perhaps your seeing bugs instead of blueprints ?
19:11:27 <flaper87> aababilov: https://blueprints.launchpad.net/marconi
19:12:00 <aababilov> no, I definitely look at https://blueprints.launchpad.net/marconi/+spec/grizzly-debt
19:12:03 <flaper87> we have qa-clusters set for havana-1 and marked as High
19:12:27 <flaper87> aababilov: ah sorry, I misunderstood
19:12:30 <malini> we need those set up for the performance testing
19:12:45 <flaper87> malini: does that have to happen for H-1 ?
19:13:04 <flaper87> H-1 ends by the end of this month, AFAIK
19:13:16 <flaper87> we've to push that back
19:13:31 <kgriffs> right, H1 ends on 30th
19:13:53 <malini> ok..we might take longer to get the salt scripts etc. done for tht
19:14:10 <flaper87> ok, pushed it back to H-2
19:14:15 <flaper87> we'll revisit it later
19:14:17 <aababilov> there is no ref to Gerrit on https://blueprints.launchpad.net/marconi/+spec/qa-cluster but progress is good
19:14:53 <flaper87> FYI, this is the list of H-1 bps https://launchpad.net/marconi/+milestone/havana-1
19:14:55 <flaper87> aababilov: gooood point
19:14:55 <aababilov> https://review.openstack.org/: Service Temporarily Unavailable
19:15:08 <kgriffs> flaper87: oz_akan is working on that, may have it done in a week or two
19:15:19 <malini> tht one is just setting up the servers etc. So do you expect anything in gerrit for tht ?
19:15:21 <flaper87> kgriffs: qa-clusters you mean?
19:15:46 <kgriffs> right
19:15:58 <flaper87> right, I see him there, so, I guess we better wait for his update
19:16:21 <kgriffs> oz_akan said he couldn't make it to the meeting, will email an update
19:16:41 <flaper87> sounds good
19:16:48 <flaper87> next: https://blueprints.launchpad.net/marconi/+spec/service-hooks
19:17:00 <kgriffs> I think we can keep it in h1 for now - malini do you know if he still plans on finishing this in the next week or two?
19:17:26 <malini> he has plans to finish it soon
19:17:34 <kgriffs> excellent
19:17:47 <flaper87> #action oz_akan to send update about the progress on qa-cluster
19:17:50 <flaper87> :D
19:17:54 <kgriffs> flaper87: re service-hooks I bumped up that priority because it's required to implement input validation
19:18:24 <kgriffs> but we can postpone to h2. Methinks fixing bugs and closing functionality gaps is more important
19:18:41 <flaper87> kgriffs: +1
19:18:42 <kgriffs> (i.e., postpone both service-hooks and input-validation)
19:19:05 <flaper87> ok, doing it now if there are no objections
19:19:59 <flaper87> #info launchpad is REALLY STUPID pushes back a parent and not its dependency (not even a small note)
19:20:32 <flaper87> next
19:20:37 <flaper87> I guess the same applies for this one https://blueprints.launchpad.net/marconi/+spec/message-pagination
19:21:24 <aababilov> what is the pagination model?
19:21:40 <aababilov> what has client to specify: the market or the offset?
19:22:02 <aababilov> markeR
19:22:08 <flaper87> aababilov: the idea is to have a way to paginate messages and queues through the API but it has to be fast and simple for other transports as well
19:22:23 <flaper87> so far we don't have pagination so we decided to put it in a separated bp
19:23:06 <flaper87> kgriffs: news about this one? https://blueprints.launchpad.net/marconi/+spec/v1-obvious-optimizations
19:23:24 <flaper87> and the pagination one?
19:23:48 <flaper87> (a couple of more minutes for blueprints and then we'll move on to other topics)
19:24:38 <flaper87> I think the later can stay for H-1
19:25:32 <flaper87> I think we're in good shape for H-1 https://launchpad.net/marconi/+milestone/havana-1
19:25:56 * flaper87 is basically talking to himself :P
19:26:41 <flaper87> ooooooooooooooooooke-doke
19:26:44 <flaper87> moving on
19:27:12 <flaper87> #topic marconiclient adopting oslo's apiclient
19:27:37 <flaper87> so, I like the idea in general
19:27:52 <flaper87> and having a common client lib throughout openstack makes sense to me
19:28:07 <flaper87> (fucking gerrit is down)
19:28:18 <flaper87> aababilov: so, a couple of thoughts about your patch
19:28:19 <aababilov> but I have comments in my email
19:28:33 <kgriffs> sorry, got distracted.
19:28:37 * kgriffs is reading log
19:28:48 <flaper87> 1) I'd like to move that outside openstack/common and put it in common/
19:28:56 <aababilov> should I say a couple of words about the library's purpose?
19:29:11 <flaper87> the reason is that openstack/common belongs  to oslo
19:29:22 <flaper87> and as for now, we're implementing it as part of marconiclient
19:29:28 <flaper87> aababilov: go ahead
19:29:53 <flaper87> kgriffs: aababilov is the guy who has been working on the common client library for openstack
19:30:04 <aababilov> apiclient contains code that's common for python-*client
19:30:14 <kgriffs> flaper87: re pagination, we have the idea of marker, limit defined. we just have to solve the fifo+guaranteed delivery
19:30:36 <kgriffs> aababilov: excellent. nice progress on that
19:30:50 <aababilov> 1) HttpClient (that reissues authentication request for expired tokens)
19:31:04 <aababilov> 2) pluggable authentication:
19:31:04 <aababilov> a) keystone;
19:31:09 <flaper87> kgriffs: right, I saw it is marked as Good Progress, should it stay in H-1?
19:31:10 <aababilov> b) endpoint + token
19:31:13 <aababilov> c) nova legacy
19:31:18 <aababilov> d) whatever
19:31:26 <kgriffs> flaper87: yeah, I plan on tackling that in the very near future
19:31:33 <flaper87> kgriffs: ++
19:31:34 <aababilov> 3) rich exception hierarchy
19:31:47 <aababilov> 4) Manager and Resource base classes
19:32:00 <flaper87> aababilov: all that sounds good to me
19:32:01 <aababilov> 5) utils for building CLI tools (temporarily)
19:32:45 <aababilov> what do we have now?
19:32:49 <kgriffs> does it do async requests?
19:33:08 <flaper87> kgriffs: so, the idea would be to use marconiclient as a "test" environment for the api. I think both the api and marconiclient can benefit from this interaction
19:33:13 <flaper87> and early adopting stage
19:33:18 <flaper87> kgriffs: good question
19:33:19 <aababilov> no. Could you show me a sample implementation?
19:33:39 <aababilov> (implementation of async req)
19:33:48 <aababilov> ok, I'll proceed
19:33:53 <kgriffs> we were playing around with that in the python-marconiclient prototype
19:34:01 <aababilov> gerrit is down, but believe me
19:34:18 <kgriffs> https://github.com/painterjd/python-marconiclient
19:34:39 <aababilov> I have written updates for almost all python*clients, except of quantum and swift
19:34:46 <aababilov> all unit tests pass
19:34:50 <flaper87> #link https://github.com/kennethreitz/grequests
19:34:58 <aababilov> CLI utilities work and are backwards-compatible
19:34:59 <flaper87> aababilov: kgriffs perhaps based on that ^
19:35:07 <kgriffs> the reason I ask, is that 1) a event queue client naturally lends itself well to an event-driven model
19:35:09 <aababilov> thanks!
19:35:34 <kgriffs> and 2 ) to get your thoughts on eventlet vs greenlet vs that-py3-thing
19:35:49 <kgriffs> (AKA tulip)
19:35:58 * flaper87 thinks tulip is cool
19:36:39 <aababilov> grequests look nice
19:36:40 <kgriffs> re grequests, that would be cool
19:36:46 <flaper87> :)
19:36:49 * flaper87 wants pop-tarts
19:36:54 <wirehead_> My vague sense at the moment is that everybody seems to be a bit gung ho to play with Tulip
19:36:55 <kgriffs> maybe we contribute and make it magically work with tulip and py3
19:37:27 * kgriffs gives flaper87 a chocolate pop-tart
19:37:34 <flaper87> w00000t
19:37:42 <wirehead_> And that's across different OpenStack teams.
19:37:54 <kgriffs> there was a discussion at the summit about migrating away from eventlet
19:37:58 <aababilov> but my point is to accept apiclient in some form and to begin improving it
19:38:05 <kgriffs> (the other reason I bring this up)
19:38:09 <kgriffs> yes
19:38:10 <flaper87> aababilov: that's the plan
19:38:17 <kgriffs> async can come later
19:38:38 <kgriffs> feel free to use marconi as a guinea pig
19:38:41 <flaper87> kgriffs: there was a question in the review about whether to have the client under openstack/common/apiclient or common/apiclient
19:38:42 <aababilov> could we accept apiclient to oslo? it's an incubator, isn't it?
19:38:47 <kgriffs> (for apiclient in general)
19:38:52 <aababilov> marconiclient will be the first user
19:38:57 <flaper87> my suggestion is to put it in common
19:39:05 <kgriffs> flaper87: kk
19:39:09 <flaper87> because openstack/common/ is oslo's holy ground
19:39:22 <flaper87> and dude, you don't want to mess with it
19:39:51 <flaper87> aababilov: yep, it's an incubator lib BUT, it is always good to have the base API sorted out before letting anything land there
19:39:58 <flaper87> so, first give it a try somewhere
19:40:02 <flaper87> and then move it there
19:40:09 <kgriffs> oic
19:40:13 <flaper87> so that people *know* how that thing should work
19:40:20 <kgriffs> heh
19:40:21 <aababilov> sure, but what will be the criterions to put it to oslo?
19:40:34 <aababilov> marconiclient lacks unit tests
19:40:43 * kgriffs is finally caught up with the conversation
19:40:45 <flaper87> aababilov: yep, it lacks of everything
19:40:53 <aababilov> novaclient and keystoneclient have them
19:41:20 <aababilov> we have integration tests in tempest
19:41:26 <aababilov> I could update horizon
19:41:27 <flaper87> aababilov: yep but we need many unittests 1) for apiclient (which are the ones that will be ported to oslo) and 2) for marconiclient
19:42:10 <kgriffs> how quickly could you get apiclient accepted for one of the mature projects?
19:42:30 <aababilov> it depends on maintainers
19:42:33 <flaper87> TBH, I wouldn't do that
19:42:48 <aababilov> it took 3-4 days to update all clients
19:43:15 <flaper87> I'd rather test it and improve it on marconi than going out there and implement it in mature clients
19:43:28 <kgriffs> +1
19:43:46 <kgriffs> i was just thinking it may be a real chore to get it accepted into those other projects
19:43:55 <kgriffs> (without any track record)
19:44:04 <flaper87> exactly
19:44:27 <flaper87> so, we've got 15 mins left and I'd love to share some thoughts about zmq transport
19:44:27 <aababilov> well, apiclient is not written from scratch
19:44:46 <aababilov> it's mostly an aggregation of code that already resides in *clients
19:44:49 <flaper87> aababilov: yeah, that's cool, I mean, it's based on other clients sweat
19:45:00 <flaper87> aababilov: TBH, that last sentence scares a bit
19:45:21 <flaper87> code from different places put in the same package...
19:45:33 <flaper87> it's not bad but it definitely neeeds to be tested a LOT
19:45:44 <flaper87> that's my thinking
19:46:04 <flaper87> so, again, I'd suggest you, if you agree, to use marconiclient as test environment
19:46:15 <flaper87> and both projects will ebnefit from each other
19:46:26 <flaper87> you can always go ahead and propose it to other clients as well
19:46:40 <flaper87> but if something changes in 1 client, you'll have to update all of them
19:46:54 <aababilov> sure, I agree to use marconiclient!
19:46:59 <flaper87> cooooooooool
19:47:12 <flaper87> aababilov: feel free to join #openstack-marconi :)
19:47:31 <aababilov> done :)
19:47:34 <flaper87> any other thoughts here ?
19:47:57 <flaper87> cool
19:47:59 <flaper87> moving on
19:48:07 <flaper87> #topic zmq transport and flaper87 crazy ideas
19:48:12 <flaper87> #link https://etherpad.openstack.org/marconi-zmq
19:48:17 <flaper87> so, I was working on that ^
19:48:32 <flaper87> which seems incomplete
19:48:43 <flaper87> and that's what it actually is
19:49:12 <flaper87> and then thought that, if we've to implement an RPC like API for the zmq stuff, why don't we use the same rpc like protocol for all the transports?
19:49:15 * flaper87 runs away
19:49:22 * flaper87 runs fast, fast faaaaaaaaaaaaaaast
19:49:43 <kgriffs> heh
19:49:47 <flaper87> the idea would be to use the same "RPC" like for HTTP communications as well
19:49:51 <flaper87> benefits:
19:49:55 <flaper87> 1) same code
19:50:01 <flaper87> 2) same verifications
19:50:08 <flaper87> 3) less code to maintain
19:50:15 <flaper87> drawbacks:
19:50:30 <flaper87> 1) RPC over HTTP, erm, that sounds like a bunch of POST requests
19:51:00 <flaper87> 2) it's all json based and more data will flight to / from the server
19:51:01 <kgriffs> RPC over HTTP is fuuuuuugly
19:51:18 <flaper87> other benefit I see is that the same protocol can be used for websocket
19:51:28 <flaper87> so, that's the crazy idea
19:51:38 <kgriffs> websocket is ok
19:51:50 <flaper87> if it really sucks I'll just STFU
19:52:12 <kgriffs> well, we have a couple paradigms
19:52:16 <aababilov> gerrit is alive!
19:52:28 <kgriffs> one is RPC, one is REST
19:52:38 <flaper87> yeah
19:52:49 <kgriffs> just thinking out loud. so...
19:52:54 <flaper87> go ahead
19:53:09 <flaper87> you've 8 mins before you'll have to shut your brain down
19:53:21 <kgriffs> it would be interesting to layer HTTP and RPC over the controllers
19:53:37 <kgriffs> just means the controllers have to support the combined set of everything REST and RPC need
19:53:41 <kgriffs> :p
19:53:51 <flaper87> #link https://github.com/openstack/glance/blob/master/glance/common/rpc.py
19:54:03 <flaper87> kgriffs: there you can have an idea of how that could work ^
19:54:31 <kgriffs> kk
19:54:36 <kgriffs> I'll have to noodle on this
19:54:45 <flaper87> so, I'd say, lets think this a bit more and elaborate the idea a bit better
19:55:08 <flaper87> #action flaper87 elaborate RPC idea better and get more pros / cons
19:55:26 <flaper87> I'll work on the RPC spec anyway because we'll need that for zmq
19:55:27 <kgriffs> for now, I vote keeping the HTTP transport RESTful, but I'm open to doing something oslo-ish with the other transports
19:55:38 <flaper87> kgriffs: cool
19:55:44 <flaper87> so, speaking of oslo
19:56:00 <flaper87> kgriffs: did you read the pad?
19:56:08 <flaper87> there are some benefits from using it
19:56:42 * flaper87 just realized he jumped the tests status update
19:56:48 <flaper87> malini: SO SO Sorry
19:57:02 <flaper87> how much time you need for that?
19:57:11 <malini> just a couple of min
19:57:15 <flaper87> ETOOMANY TOPICS
19:57:20 <malini> but finish up the zmq stuff first
19:57:50 <flaper87> malini: not much to say for now, just wanted to share those 2 was for doing it so we can start thinking about that
19:58:07 <flaper87> plus, I wanted to tell the crazy idea
19:58:09 <flaper87> :P
19:58:11 <flaper87> moving on
19:58:13 <kgriffs> flaper87: I saw the pad, will do some ponderin'
19:58:21 <flaper87> kgriffs: cool, thanks a lot
19:58:28 <flaper87> I'll share with cppcabrera as well
19:58:33 <flaper87> #topic system tests status update
19:58:38 <flaper87> malini: go ahead
19:58:44 <malini> I have fixed the large bulk of the flake8 errors in the system tests. The remaining few are module import errors.   That one looks a little tricky & we are still figuring out how to fix tht.
19:58:44 <flaper87> 2 mins
19:59:09 <kgriffs> does it look like a flake8 bug?
19:59:16 * kgriffs famous last words
19:59:18 <flaper87> malini: cool, did you try asking mordred as well? (Monty)
19:59:25 <mordred> aroo?
19:59:26 <flaper87> he implemented that for most projects
19:59:27 <malini> sure..
19:59:32 <flaper87> mordred: there you are :D
19:59:37 <mordred> sup
19:59:41 <flaper87> malini: ^
19:59:54 <malini> I am having trouble getting some of my modules mergeed
20:00:01 <malini> flake8 claims its not a module
20:00:07 <flaper87> we've just 1 min left
20:00:17 <mordred> malini: can you point me towards a failing review?
20:00:57 <malini> sure.. will ping u offline..we are almost out of time here
20:00:57 <flaper87> guys, I've got to end the meeting now! malini thanks for your hard work on tests. ++ for you
20:01:10 <malini> thanks :)
20:01:18 <flaper87> Thanks guys, really great meeting full of interesting topics
20:01:23 <flaper87> #endmeeting