16:03:26 <kgriffs> #startmeeting marconi
16:03:27 <openstack> Meeting started Mon Oct 21 16:03:26 2013 UTC and is due to finish in 60 minutes.  The chair is kgriffs. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:03:28 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
16:03:30 <openstack> The meeting name has been set to 'marconi'
16:03:35 <flaper87> <o/ \o>
16:03:43 <kgriffs> #link https://wiki.openstack.org/wiki/Meetings/Marconi#Agenda
16:03:59 <kgriffs> #topic Review teh actionz from teh last timez
16:04:12 <flaper87> :D
16:04:21 <kgriffs> #link http://eavesdrop.openstack.org/meetings/marconi/2013/marconi.2013-10-07-16.03.html
16:04:38 <kgriffs> so, looks like flaper87 was the only one with anything to do this past week.
16:04:40 <kgriffs> ;)
16:04:42 <alcabrera> lol
16:05:20 <flaper87> erm, mmh, erm erm mmhh
16:05:36 <flaper87> I totally failed on that but I got feedback on the devstack patch
16:05:38 <flaper87> does that count?
16:05:40 <flaper87> :D
16:05:46 <kgriffs> heh
16:05:55 <flaper87> I'll get to it this week
16:05:58 <flaper87> promissed
16:06:01 <kgriffs> #action flaper87 to research heat
16:06:11 <kgriffs> ok
16:06:13 <kgriffs> otherwise
16:06:18 <kgriffs> quick not regarding API freeze
16:06:43 <kgriffs> as Murphy's Law dictates, immediately after freezing the API we started getting people complaining about certain aspects of it
16:07:11 <kgriffs> So, i started a page to track feedback. We will need to review it at some point.
16:07:12 <kgriffs> https://wiki.openstack.org/wiki/Marconi/specs/api/next
16:07:13 <zyuan> :)
16:07:22 <alcabrera> #link https://wiki.openstack.org/wiki/Marconi/specs/api/next
16:07:39 <alcabrera> #info Now tracking API feedback for future versions.
16:08:25 <flaper87> interesting notes
16:08:47 <flaper87> I would like us to break them down on an etherpad / bugs and comment there
16:08:57 <flaper87> I can help with bugs creation
16:09:12 <kgriffs> yeah, let's hold off for a bit and see what else comes in
16:09:33 <flaper87> sure, I'd say we should start putting them in an etherpad then
16:09:35 <zyuan> i think we need to discuss them first
16:09:37 <kgriffs> fwiw, I thought this post was pretty relevant. ;)
16:09:37 <flaper87> and add some comments
16:09:37 <kgriffs> http://sethgodin.typepad.com/seths_blog/2013/09/when-to-speak-up.html
16:10:12 <flaper87> heheh
16:10:21 <kgriffs> flaper87: let me set that up for next week. We've got sharding to focus on for now.
16:10:21 <alcabrera> yup - speak up before it is done, though getting people to break the inertia before use is tough. :P
16:10:37 <flaper87> yeah, I meant to do it off-line
16:10:39 <flaper87> not right now
16:10:50 <flaper87> sorry for not being explicit enough :D
16:10:56 <kgriffs> ok, feel free to set up etherpadz or whateverz
16:11:02 <flaper87> okeyz
16:11:11 <flaper87> neeeext topicz
16:11:13 <flaper87> :D
16:11:52 <kgriffs> #topic teh updatez on teh shardzes
16:11:59 <alcabrera> lol
16:12:11 <alcabrera> yeah, there's a lot going on there~
16:12:29 * flaper87 didn't know kgriffs knows french
16:12:38 <alcabrera> So, with a link to summarize the state of things
16:12:41 <alcabrera> #link https://etherpad.openstack.org/p/sharding-merge-strategy
16:12:49 <alcabrera> Wr've gotten a lot of the foundational stuff down.
16:12:53 <alcabrera> *We've
16:13:05 <alcabrera> Pipeline enhancements, a sharding driver stubbed out
16:13:31 <alcabrera> It's been a load of development towards making sharding a real thing.
16:13:37 <alcabrera> (for marconi)
16:13:58 <alcabrera> the proposed admin API is currently under review
16:14:26 <alcabrera> once that's merged, we'll have a sharding and a catalogue queues storage driver for controlling shard registration
16:15:09 <kgriffs> ok, any big questions or concerns or blockers?
16:15:24 <zyuan> requires_mongodb
16:15:24 <flaper87> not from me!
16:15:37 <zyuan> does not work T_T
16:15:37 <alcabrera> the only blocker is already being addressed - let's keep reviewing the admin API branch and get those storage drivers in. :)
16:15:40 <flaper87> The whole idea and work looks good
16:15:55 <flaper87> zyuan: mmh, it does work for here :(
16:16:01 <kgriffs> ok, rock on
16:16:14 <zyuan> just not on jenkins with py26. discuss latter.
16:16:20 <kgriffs> we'll continue working through those patches in #openstack-marconi
16:16:43 <alcabrera> awesome
16:16:46 <kgriffs> #topic The future of the proxy
16:17:16 * alcabrera notes how serious the tone suddenly became - 90% fewer z's
16:17:29 <alcabrera> *100%
16:18:00 * kgriffs has to budget his z's - doesn't want to run out early
16:18:03 <alcabrera> kgriffs: want to lead the discussion on this one?
16:18:09 <kgriffs> sure
16:18:24 <kgriffs> so, given that we will have storage-layer sharding
16:18:35 <kgriffs> how far will that allow Marconi to scale?
16:19:00 <kgriffs> (without any proxy thingy)
16:19:09 <flaper87> Far enough, imho!
16:19:51 <alcabrera> it seems like it would go for some time. Some potential bottlenecks are CPU/connections|node, but those seem far off atm.
16:20:09 <flaper87> I was also thinking that Marconi it'sef could be use as a shard in case of more partitions are needed
16:20:32 <flaper87> meaning, we coul have a Marconi storage backend that talks to a remote Marconi server
16:20:45 <flaper87> in case that's ever needed
16:20:54 <kgriffs> my mind just exploded
16:21:01 <flaper87> LOL
16:21:06 <kgriffs> cool idea
16:21:09 <alcabrera> lol
16:21:24 <alcabrera> That's like nova in docker in nova in...
16:21:34 <flaper87> :D
16:21:42 <flaper87> I'll create a bp for that then :D
16:22:09 <amitgandhi> so a marconi storage backend would talk to another marconi transport?
16:22:12 <kgriffs> ok, so the proposal is to stamp a big-ole YAGNI on the proxy?
16:22:26 <alcabrera> That's my favored outcome for the proxy.
16:22:32 <flaper87> I'd say so
16:22:50 <flaper87> amitgandhi: yeah, it'd talk to another Marconi
16:23:01 <flaper87> that bp will obviously depend on the client being ready
16:23:02 <alcabrera> That'd be -2372 LOC for removing the proxy - easier to manage.
16:23:10 <flaper87> shall we vote?
16:23:14 <amitgandhi> biggest fear there for me is the latency of it
16:23:22 <alcabrera> flaper87: good idea
16:23:35 <kgriffs> #startvote Remove the proxy, salvaging what we can for the sharding feature? Yes, No, Abstain
16:23:35 <openstack> Begin voting on: Remove the proxy, salvaging what we can for the sharding feature? Valid vote options are Yes, No, Abstain.
16:23:36 <openstack> Vote using '#vote OPTION'. Only your last vote counts.
16:23:40 <flaper87> amitgandhi: yeah, that needs to be examined a lot!
16:23:50 <alcabrera> #vote yes
16:23:55 <flaper87> #vote yes
16:23:56 <amitgandhi> #vote yes
16:24:06 <zyuan> #vote Abstain
16:24:22 <kgriffs> #vote yez
16:24:23 <openstack> kgriffs: yez is not a valid option. Valid options are Yes, No, Abstain.
16:24:32 <alcabrera> lol
16:24:32 <zyuan> ...
16:24:35 * kgriffs bots have no humor
16:24:43 <kgriffs> #vote yes
16:25:00 <kgriffs> going once...
16:25:09 <kgriffs> twice...
16:25:15 <zyuan> when?
16:25:19 <kgriffs> #endvote
16:25:19 <openstack> Voted on "Remove the proxy, salvaging what we can for the sharding feature?" Results are
16:25:20 <openstack> Yes (4): amitgandhi, alcabrera, kgriffs, flaper87
16:25:21 <openstack> Abstain (1): zyuan
16:25:40 <kgriffs> ok, once sharding is packaged up with a pretty bow, we can nuke the proxy code
16:25:46 <zyuan> ok
16:25:55 <kgriffs> let's leave it around until then for reference
16:26:00 <zyuan> +1
16:26:18 <alcabrera> I've already started the salvaging process.
16:26:32 <kgriffs> anything else on that topic?
16:26:37 <alcabrera> the admin API feature branch used many of the idioms and storage patterns I put together while working on the proxy.
16:26:42 <alcabrera> *uses
16:26:53 <alcabrera> Plus, it has pagination. :P
16:27:02 <alcabrera> Anyway, that's it from me on the proxy.
16:27:15 <alcabrera> (nuke it from orbit)
16:27:15 <kgriffs> #topic API Spec server sidez
16:27:24 <alcabrera> more zzzzs
16:27:30 <flaper87> o/
16:27:32 <flaper87> o/
16:27:34 <kgriffs> flaper87: go for it
16:27:35 <flaper87> o/
16:27:44 * flaper87 runs towards it
16:27:51 * flaper87 is still running
16:27:54 * flaper87 is getting tired
16:27:57 * flaper87 fainted
16:27:59 <flaper87> ok
16:28:01 <flaper87> seriously
16:28:35 <zyuan> that's the first issue?
16:28:48 <flaper87> The idea there is to have something similar to what we have in the client but adapted a bit for the server
16:28:49 <flaper87> it all starts from the thought that we don't have a good way to version our server-side API
16:28:49 <flaper87> ... yet.
16:28:52 <flaper87> so, this is what we've in the client
16:29:11 <flaper87> #link https://review.openstack.org/#/c/50638/7/marconiclient/queues/v1/api.py
16:29:47 <flaper87> In that JSON, I'm specifying the API that can be validated with json schema.
16:29:52 <flaper87> (if needed)
16:30:09 <flaper87> although it seems a bit HTTP tight, it actually isn't.
16:30:31 <flaper87> The whole point is that HTTP is the most partitioned protocol we have right now
16:30:51 <flaper87> other protocols can simple translate that to something they need
16:30:55 <flaper87> and can use
16:31:14 <flaper87> so, hope that explains a bit the idea
16:31:19 <flaper87> thoughts so far?
16:31:21 <alcabrera> it's true - zmq is just data + Maybe framing
16:31:28 <alcabrera> HTTP is a bit more enveloped
16:31:35 <alcabrera> I don;t know about websockets
16:31:47 <alcabrera> And I'm pretty sure nanomsg is similar to zmq (data + Maybe frames)
16:31:49 <flaper87> For example, current wsgi controllers would register routes based on that dict
16:31:50 <zyuan> zmq's frame is invisible
16:32:16 <alcabrera> zyuan: it's invisible as long as you don't use the XSUB/XPUB patterns
16:32:40 <zyuan> :(
16:32:46 <alcabrera> flaper87: as for the idea in general, I'm pretty positive towards it.
16:32:58 <alcabrera> It seems like it would greatly reduce the effort required to add a new transport.
16:33:10 * flaper87 is pretty sure it'll need tweaked a bit
16:33:17 <flaper87> need to be*
16:33:29 <alcabrera> Instead of implementing each of the resources for each protocol, it seems like only the low-level transport.api.py.thing would have to be implemented
16:33:37 <flaper87> but, yeah, the idea is to have a well defined API schema
16:33:45 <alcabrera> and the resources just use the appropriate lower-level to forward/serialize data.
16:34:04 <alcabrera> I'm +2 on the API schema
16:34:08 <kgriffs> hmm
16:35:32 <flaper87> that would also give us a home endpoint for free and eventually help us a lot with API discovery
16:35:50 <flaper87> also, it will unified the API definition throughout transports
16:36:03 <flaper87> as for now, we basically need to 'define' it for each transport
16:36:05 <kgriffs> so, the proposal is to define the API using json-schema
16:36:43 <flaper87> kgriffs: ish, the idea is to have an API schema. Jsonschema has proven to be good for the job
16:37:01 <flaper87> it can be validated but it doesn't "have to" plus, it is flexible enough
16:37:40 <kgriffs> so, one schema that all the transports are based on?
16:37:52 <flaper87> kgriffs: yup
16:38:14 <flaper87> Transport would get an 'Api' instance that they'll use to 'register' endpoints
16:38:34 <flaper87> that Api instance contains the API version, schema and other things related to the API
16:39:11 <flaper87> this is what the base API looks like in the client https://github.com/openstack/python-marconiclient/blob/master/marconiclient/transport/api.py
16:39:16 <flaper87> #link https://github.com/openstack/python-marconiclient/blob/master/marconiclient/transport/api.py
16:39:50 <flaper87> I can work on a POC for it
16:40:01 <flaper87> and we can discuss it a bit further based on that
16:40:14 <alcabrera> flaper87: To some extent, the client impl. is a pre-POC. :D
16:40:22 <flaper87> alcabrera: good point :D
16:40:28 <kgriffs> yeah, let's do that
16:40:29 * flaper87 takes that back
16:40:33 <flaper87> damn
16:40:34 <alcabrera> lol
16:40:40 * flaper87 facepalm
16:40:43 <kgriffs> ?
16:40:43 <flaper87> :P
16:40:51 <flaper87> kgriffs: just kidding!
16:41:03 <flaper87> I'll do some experiments on marconi-queues
16:41:09 <flaper87> and show the results
16:41:31 <flaper87> but I'd like to get a general thought now about the whole idea
16:41:43 <flaper87> if you guys see some limitations
16:41:45 <flaper87> etc
16:42:21 <kgriffs> do you see this complicating, simplifying, or keeping the WSGI driver code the same?
16:42:38 <kgriffs> (keeping roughly same in complexity, I mean)
16:42:41 <flaper87> I see this simplifying the WSGI code
16:42:50 <kgriffs> in what ways?
16:44:00 <flaper87> 1) The way routes are being defined, 2) We could reuse Controller classes in different versions
16:44:43 <flaper87> Number 2 obviously depends on the changes between versions
16:45:13 <flaper87> but what it definitely improves is API's consistency between transports
16:45:52 <flaper87> let me work on a POC
16:45:58 <flaper87> I think that will make this clearer
16:46:19 <kgriffs> #action flaper87 to work on a POC for server API schema thingy
16:46:41 <alcabrera> awesome. That'll be good to see growing. :)
16:46:44 <flaper87> cool
16:46:49 <kgriffs> flaper87: i think I see where you are headed
16:47:01 <kgriffs> btw, could you also consider the question of api extensions?
16:47:22 <flaper87> kgriffs: yes, will do!
16:47:26 <kgriffs> just want to make sure whatever we implement takes that into consideration
16:47:32 <kgriffs> ok
16:47:34 <flaper87> any thoughts so far?
16:47:49 <flaper87> like "WTF are you planning?" or "Yup, that could work"
16:47:57 <kgriffs> so, I like the idea of defining the API in code, rather than on a wiki page
16:48:09 <alcabrera> kgriffs: +1
16:48:14 <flaper87> +1
16:48:21 <flaper87> we can generate Wiki pages out of it
16:48:23 <flaper87> :D
16:48:26 <flaper87> ok, cool!
16:48:26 <alcabrera> It gives us the opportunity to - yes, that
16:48:28 <alcabrera> ^
16:48:30 <flaper87> lets move on
16:48:42 <flaper87> there are other topics to talk about
16:48:44 <alcabrera> next topic?
16:48:53 <kgriffs> yeah, the basic idea is good, I just want to make sure it doesn't unnecessarily complicate the transport drivers
16:49:14 <kgriffs> #topic API versioning strategy?
16:49:33 <kgriffs> so, I put out some feelers re api versioning a few weeks back
16:49:52 <kgriffs> it sounds like the community prefers extensions and major api revs over media type versioning
16:50:03 <kgriffs> does that sound about right?
16:50:07 <zyuan> only media type?
16:50:16 <zyuan> without url?
16:50:38 <kgriffs> like, if a client requests queues/my-queue/messages with
16:50:56 <kgriffs> accept: application/vnd.openstack.marconi.messages.v2
16:51:08 <kgriffs> then we return a different representation of the messages resource
16:51:28 <zyuan> :( that will be curl un-friendly
16:51:50 <alcabrera> hmmm...
16:52:02 <kgriffs> so, this is the recommended approach instead - http://www.infoq.com/presentations/OpenStack-Extensions
16:52:07 <kgriffs> (from what I could gather)
16:52:25 <kgriffs> that's my first thought
16:52:25 <alcabrera> Cool, I'll check that out.
16:52:44 <kgriffs> second is whether to target v2 API for Havana, or stick with v1 plus maybe some extensions
16:52:55 <kgriffs> #link http://www.infoq.com/presentations/OpenStack-Extensions
16:53:04 <flaper87> kgriffs: Icehouse, you mena
16:53:05 <flaper87> mena
16:53:08 <flaper87> mean
16:53:08 <kgriffs> oops, yeah
16:53:12 <kgriffs> Icehouse
16:53:17 * alcabrera kgriffs is a time traveler
16:53:22 <alcabrera> **notices
16:53:24 <zyuan> kgriffs: can briefly introduce the link above?
16:53:25 <kgriffs> the idea being, we start out with a polished API for our first integrated release
16:53:51 <kgriffs> zyuan: basically, you allow api extensions by introducing new resources (with new URLs), new headers, etc.
16:53:58 <alcabrera> Icehouse is expected for 2014-04-04
16:54:01 <flaper87> kgriffs: we could also target some minor releases for v1 depending on what needs to be changed
16:54:12 <alcabrera> #info Icehouse is expected for 2014-04-04
16:54:13 <flaper87> lets note that our client doens't fully support v1 yet
16:54:21 <kgriffs> you also provide a way for clients to discover said extensions
16:54:32 <flaper87> it'd be a bit ackward to release API v2 and leave the client behind
16:54:42 <flaper87> although that clients have a different release cycle
16:54:43 <kgriffs> yes, i agree
16:54:48 <zyuan> what an api extension looks like?
16:55:10 <kgriffs> well, v2 would be a minor bump i suppose
16:55:17 <kgriffs> i guess we could just do v1.1 instead
16:55:37 * flaper87 votes for 1.1
16:55:52 <flaper87> 1 -> 2 will create a LOT of confussion
16:55:58 <alcabrera> v1.1 works for me, given the changes requested are rather small atm
16:56:10 <alcabrera> v2.0 makes me think of routing/exchanges and tags/topics
16:56:11 <kgriffs> that would be more open-stacky, although the RESTafarian in me screams our in agony
16:56:25 <kgriffs> flaper87: noted
16:56:39 <kgriffs> ok, so let's think about doing a v1.1 for Icehouse
16:56:55 <zyuan> /v1.1/ ?
16:57:18 <zyuan> if so, LGTM
16:57:37 <kgriffs> then 2.0 stuff we want to start working on we can have first as extensions - that will force us to start making extensions work for 3rd-parties
16:57:42 <kgriffs> zyuan: right
16:58:02 <zyuan> actually, i think we can provide extension between releases
16:58:07 <flaper87> plus, the summit is 2 weeks from tomorrow, we'll definitely get a lot of feedback tehre
16:58:09 <flaper87> there
16:58:14 <zyuan> and merge them to new releases
16:58:20 <flaper87> we should also wait on that before taking any final decission
16:58:23 <kgriffs> yeah, that is one of the purposes of extensions in general
16:58:33 <zyuan> great
16:58:33 <kgriffs> POCs
16:59:29 <zyuan> we don't have much time left; can we discuss the object size / json doc length stuff?
16:59:37 <kgriffs> yeah, just one moment
16:59:47 <kgriffs> #info Target v1.1 for Icehouse
17:00:00 <kgriffs> #topic Updates on bugs
17:00:05 <kgriffs> zyuan: go for it
17:00:28 <zyuan> currently, we define document limit in terms of JSON char count
17:00:51 <zyuan> but obviously that does not work for other content like msgpack
17:00:57 <flaper87> zyuan: just a comment about that review. I'm ok with marking WIP in the summary but lets also use the WIP button in gerrit, pls! I didn't notice it was a WIP and reviewed it
17:01:10 <zyuan> LOL, ok...
17:01:25 <flaper87> :P
17:01:46 <zyuan> in addition, currently the code run in production does not perform real JSON char count checking
17:02:01 <zyuan> I short cut it with content-length checking
17:02:25 <zyuan> but if we enable the JSON char count checking.... that would be not just a little slow
17:02:39 <zyuan> so my proposal is to redefine "document size"
17:02:51 <zyuan> -- in terms of the total size of sub obejcts
17:03:07 <zyuan> for example, int and double has size 8
17:03:18 <zyuan> empty array, {} has size 8
17:03:19 <zyuan> etc.
17:03:31 <zyuan> so that any type of media type can have same size
17:03:56 <zyuan> and the checking is cheaper than JSON char count (unless we have a sizing parser in C)
17:04:17 <zyuan> prototype: https://review.openstack.org/#/c/51921/
17:04:34 <zyuan> comment?
17:05:22 <flaper87> I like the idea overall, I'm a bit worried about the iteration cost (I mean, going deeper on nested objects) but I think that's still cheaper than counting chars
17:05:30 <kgriffs> I think it would be helpful to have a python sizing parser as a baseline
17:05:38 <alcabrera> The recursion is elegant, and I don't expect it to be too expensive CPU-wise.
17:05:45 <kgriffs> given a dict, it walks it and counts up JSON chars
17:06:09 <alcabrera> If this becomes a CPU-bound task, then pypy would likely shine here.
17:06:15 <alcabrera> Given it's pure Python.
17:06:23 <zyuan> then you pretty much need N iteration, where N is JSON char counts...
17:06:26 <kgriffs> I am still concerned about confusion on the part of the users/library developers when they get back an error response and they have to figure out why their message was too large.
17:06:26 <alcabrera> Err, becomes a CPU bottleneck
17:06:30 <zyuan> it has to be done in C
17:06:30 <flaper87> alcabrera: yeah, but we can't expect all deployments to use pypy
17:06:39 <alcabrera> flaper87: true. :)
17:07:03 <zyuan> i implemented it with recusion just because i don't how to do it without
17:07:05 <flaper87> kgriffs: that also had me thinking
17:07:06 <kgriffs> also, we don't have much data on average message complexity
17:07:24 <kgriffs> only what we can extrapolate from Rackspace's use of the internal precursor to Marconi
17:07:26 <zyuan> ... and i suspect a manual recursion to be even slower...
17:08:15 <zyuan> kgriffs: i hope so?
17:08:37 <kgriffs> i mean, if messages are usually not very complex, then we may be doing premature optimization here
17:08:46 <kgriffs> but we are still speculating
17:08:58 <ekarlso> is RAX using marconi or ?
17:09:20 <flaper87> I guess we could examine this a bit further and re-schedule the discussion
17:09:24 <kgriffs> we use a primitive sort of Marconi called RSE (really simple events)
17:09:33 <flaper87> like I said, I like the idea overall
17:09:53 <kgriffs> so, requirements
17:10:09 <kgriffs> 1. Users must not be confused
17:10:22 <kgriffs> 2. It needs to be fast for the common case
17:10:52 <zyuan> (as a numerical anaysis user, i feel *really* confused with JSON char counts)
17:11:18 <zyuan> (i agree, performance improvement, hopefully)
17:11:30 <kgriffs> 3. Don't optimize prematurely; a naive python character counter may be good enough, for example
17:11:34 <kgriffs> anyway, we are out of time
17:11:46 <kgriffs> let's keep discussing
17:11:50 <flaper87> lets re-schedule this for next week
17:11:54 * flaper87 gtg now
17:12:03 * kgriffs has to go too
17:12:06 <kgriffs> #endmeeting