19:05:16 <kgriffs> #startmeeting marconi
19:05:16 <openstack> Meeting started Thu Jul 18 19:05:16 2013 UTC.  The chair is kgriffs. Information about MeetBot at http://wiki.debian.org/MeetBot.
19:05:17 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
19:05:19 <openstack> The meeting name has been set to 'marconi'
19:05:20 <cppcabrera> Woot
19:05:23 <flaper87> \o/
19:05:24 <kgriffs> #topic review actions from last time
19:05:36 <kgriffs> #link http://eavesdrop.openstack.org/meetings/marconi/2013/marconi.2013-07-11-19.03.html
19:05:52 <kgriffs> 1.	oz_akan_ and flaper87 to investigate mongo insert performance
19:06:35 <flaper87> kgriffs: It wasn't possible to do that, we'll have to schedule it for next week
19:06:41 <kgriffs> ok
19:06:48 <kgriffs> #action oz_akan_ and flaper87 to investigate mongo insert performance
19:07:20 <kgriffs> 2. flaper87 to document client cleanup
19:07:36 <flaper87> kgriffs: I started it here: https://etherpad.openstack.org/apiclient-marconi and folks helped me out
19:07:46 <flaper87> Alessio added a bunch of stuff there
19:07:52 <kgriffs> sweet. can we get these turned into bugs/bp's?
19:08:13 <cppcabrera> Yup. I can do that today.
19:08:21 * cppcabrera volunteers for action
19:08:23 <kgriffs> imaginary internet bonus points (IIBP's) for anyone who helps!
19:08:36 <flaper87> cppcabrera: I'll help you with that
19:08:50 <cppcabrera> Awesome, flaper87!
19:08:59 <kgriffs> #action cppcabrera, flaper87 to turn apiclient-marconi etherpad into bps/bugs
19:09:21 <kgriffs> 3. flaper87 to merge patches
19:09:28 <kgriffs> well, that's rather specific, isn't it?
19:09:32 <kgriffs> :p
19:09:33 <cppcabrera> Hahaha
19:09:39 <flaper87> wheheheheheh
19:09:41 <cppcabrera> Common lib patches were merged.
19:09:56 <flaper87> yeah, and we pointed out what needs to be done next
19:10:03 <kgriffs> sweet. nice work
19:10:05 <cppcabrera> I think the input validation framework is still pending? Is that correct?
19:10:14 <zyuan> yes.
19:10:15 <flaper87> yup
19:10:21 <kgriffs> 4. flaper87, cppcabrera to finalize client design
19:10:43 <cppcabrera> We certainly rocked out on that item last week. All kinds of brainstorming happened.
19:10:51 <Sriram> +1
19:10:57 <flaper87> indeed +1
19:10:57 <kgriffs> still some loose ends there?
19:10:59 <cppcabrera> #link https://wiki.openstack.org/wiki/Python_Marconi_Client
19:11:03 <flaper87> cppcabrera: good work on that!
19:11:09 <cppcabrera> #link https://wiki.openstack.org/wiki/Marconi/specs/zmq/api/v1
19:11:21 * flaper87 didn't know that page exists.
19:11:26 <flaper87> glad it does
19:11:28 <flaper87> cppcabrera: +1
19:11:34 <zyuan> there are many thoughts on the input validation, while the auther is not in the office.
19:11:49 <cppcabrera> A few loose ends remain, mostly in regards to adding polling support (drafting that now), and... some I can't think of yet. :P
19:12:28 <kgriffs> #action cppcabrera to tie up loose ends re python-marconicient v1 design
19:12:48 <kgriffs> 5. zyuan to write up metadata handling proposal and review next time
19:12:52 <cppcabrera> +1. I'll pick all your brains on that. :D
19:13:04 <zyuan> wrote, as in the comment
19:13:13 * kgriffs wonders if cppcabrera is secretly a zombie
19:13:17 <zyuan> see card #287
19:13:34 <zyuan> QAQ
19:13:36 <kgriffs> ok, let's take that topic now
19:13:45 <kgriffs> #topic metadata handling proposal
19:14:03 <kgriffs> zyuan: you have the floor. :D
19:14:34 <kgriffs> (15 minutes)
19:14:51 <kgriffs> (for this topic)
19:15:17 <zyuan> first i want to clearify
19:15:21 <kgriffs> I've discussed this at length already with zyuan, so I'll sit on the sidelines for a bit
19:15:41 <zyuan> the checking is costly.
19:16:22 <zyuan> because the checking is to check have a user used a _name we already defined"
19:16:56 <zyuan> not just to check whether the user have _ prefix in their metadata
19:17:29 <zyuan> to distinguish "their" metadata from ours, we have to check all the _ prefixed metadata one by one
19:17:52 <zyuan> 1st it decreases runtime performance
19:18:05 <zyuan> 2nd it makes it hard for us to define new _names
19:18:18 <zyuan> (the list will become longer and longer)
19:18:31 <zyuan> (and can hardly be driver-specific)
19:18:57 <cppcabrera> I agree on the performance portion. It's O(n), given n is the number of metadata fields defined. The actual check can be against a dictionary of reserved names O(1). So O(n) * O(1).
19:19:21 <flaper87> TBH, I was thinking about that as forbidding anything starting with _
19:19:30 <zyuan> cppcabrera: trust me, never think O(1) is cheap
19:19:33 <flaper87> that will avoid having to have a secondary list with reserved words
19:19:49 <zyuan> O(1) is cost -- when compared with 1
19:20:04 <flaper87> if we don't want to use _ for reserved metadata, we can pick another prefix
19:20:09 <zyuan> flaper87: i considered about that.
19:20:15 <zyuan> like C++'s std namespace
19:20:21 <zyuan> we can have our own
19:20:47 <cppcabrera> Hmm... namespaced metadata... that sounds pretty awesome.
19:20:51 <zyuan> but the core problem is not changed: what if a user expend the std namespace and add their own stuff?
19:21:12 <zyuan> (some programmers really open the std namespace....)
19:21:42 <cppcabrera> No need to check user meta - that can be bundled as "metadata" : { "user" : { "mine" : ...} } and ours can be in "metadata" : { "private": { ... } }
19:21:54 <cppcabrera> What if used that kind of scheme?
19:21:57 <flaper87> well, the limit is up to us. What about just keeping everything in a separate metadata field that we *don't* return?
19:22:13 <cppcabrera> flaper87: Roughly what I was thinking.
19:22:23 <flaper87> cppcabrera: :P
19:22:27 <zyuan> then user can't see their configurations
19:22:30 <flaper87> my turn to squize your brain
19:22:30 <kgriffs> flaper87: we would need a different resource type to support setting it
19:22:32 <flaper87> :D
19:22:36 <cppcabrera> :)
19:22:47 <zyuan> when i was implementing per-queue grace, i found that we have to return it
19:22:50 <flaper87> kgriffs: that would make acl easier, wouldn't it ?
19:23:01 <kgriffs> so, either same resource type with reserved sub-document name (namespace) or two different resources
19:23:07 <flaper87> btw, just thinking outloud
19:23:41 <amitgandhi> reserved namespacing +1
19:24:14 <zyuan> as i said, a reserved namespace does not solve the problem intrinsincally.
19:24:21 <kgriffs> zyuan: pls elaborate
19:24:31 <zyuan> check or not check is the 1st problem
19:25:03 <zyuan> how to reserve -- 2nd problem
19:25:30 <flaper87> zyuan: if we don;t answer the 2nd we won't be able to answer the 1st, IMHO
19:25:51 <kgriffs> let's assume we namespace reserved variables
19:25:57 <kgriffs> (for the sake of argument)
19:26:06 <kgriffs> how does that effect 1st problem?
19:26:10 <kgriffs> affect
19:26:27 <zyuan> my opinion is, it does not.
19:27:29 <zyuan> if you want to enforce the checking, you need to perform the checking wherever you have "reserved" things.
19:28:24 <cppcabrera> I does change the check a lot, in my eyes. It changes the check from verifying whether *any* field in metadata is a reserved one to checking whether the PATCH operation contains a document that identifies itself with the key "reserved".
19:28:50 <cppcabrera> Hmm...
19:29:05 <kgriffs> but we want to allow them to PATCH that portion, no?
19:29:13 <cppcabrera> Right. I got things mixed. :P
19:29:14 <zyuan> oh, then you need to verify *any* fields in your reserved interface
19:29:25 <zyuan> s/interface/namespace/
19:29:36 <cppcabrera> I was thinking of the "create metadata" and thought I meant patch, heh.
19:29:53 <cppcabrera> But yeah, we disallow users from creating new metadata fields in the reserved space.
19:30:01 <cppcabrera> s/space/namespace
19:30:02 <zyuan> there is no PATCH opt for queue; only PUT
19:30:23 <cppcabrera> zyuan: Yup, I remembered after I posted my sentence. :)
19:30:35 <Sriram> yes. i think there is a patch for only for claims.
19:30:36 <zyuan> then you have to check the names under the reserved namespace
19:30:51 <flaper87> but, the reserved metadata should be used internally, right?
19:30:55 <Sriram> *ignore first for
19:31:06 <kgriffs> so, sounds like namespacing maybe helps psychologically, but not necessarily technically?
19:31:36 <zyuan> kgriffs: yes. it may looks clearer, but the problem is still there.
19:31:50 <zyuan> (but to me, i think _ is enough...)
19:32:04 <zyuan> (if your argument is like "who will touch 'metadata'")
19:32:08 <flaper87> I think we should think about a real use-case
19:32:14 <flaper87> (For the sake of the argument)
19:32:18 <kgriffs> go for it
19:32:20 <zyuan> (i can answer "who will touch _names?")
19:32:22 * cppcabrera wishes we had a whiteboard to think this one out
19:32:32 * kgriffs budgets a few more minutes for this topic
19:32:42 <Sriram> +1, a real use-case will help.
19:32:48 <cppcabrera> +1 flaper87
19:33:07 <kgriffs> stand by
19:33:12 <zyuan> example, if we have a field (w/o namespace/prefix)
19:33:17 <zyuan> called "grace"
19:33:23 <zyuan> with prefix
19:33:26 <flaper87> so, when we first talk about reserved emtadata, IIRC, we were thinking about having different custom TTL per queue
19:33:37 <flaper87> I mean, queue level TTL, IIRC
19:33:42 <zyuan> it's "_grace"
19:33:47 <flaper87> or grace
19:34:03 <zyuan> if you want to do the checking, you need to keep a list of those names
19:34:03 <kgriffs> yes, that's a good example - queue-level defaults if not specified per message/claim
19:34:16 <zyuan> and theck the global namespace
19:34:24 <cppcabrera> +1 for context
19:34:29 <zyuan> if we use a reserved namespace instead
19:34:32 <flaper87> so, having a queue level default means we have to read it everytime a message will be posted
19:34:35 <zyuan> say, "metadata"
19:34:44 <zyuan> then the global namespace become
19:34:48 <kgriffs> flaper87: yes, but we could cache I suppose
19:35:00 <zyuan> {"metadata" : {"grace": 60}}
19:35:01 <flaper87> which means that it won't be enough to have *just* the queue id when doing that
19:35:02 <kgriffs> (memcached/redis colocated on app server)
19:35:13 <flaper87> kgriffs: right right, just bringing the worst case scenario
19:35:24 <flaper87> well, actually, that helps with the context
19:35:32 <zyuan> then you need to open the "metadata" field and keep checking the reserved names....
19:35:33 <kgriffs> anyway, let's hold off on that tangent until another time
19:35:33 <flaper87> so, lets think we have this cache system
19:35:54 <zyuan> problems become more: "metadata" may not be a dict....
19:36:02 <flaper87> what would we put into that cache if we have everything mixed?
19:36:17 <flaper87> we would have to separate *our* metadata from user's and store it in the cache
19:36:31 <flaper87> so, that makes me think that a separate field for that would make sense
19:37:06 <zyuan> i think it's ok to store anything metadata in the cache
19:37:34 <zyuan> it's actually more simple, and cheap if we have a configurable json parser
19:37:56 <zyuan> i mean, a proposed parser which can stop parsing at some level of the elements
19:38:18 <zyuan> and don't forget, we have size limit on queue metadata -- as a whole
19:38:28 <flaper87> zyuan: but that requiers more memory, right? And we'll have to update it everytime the metadata is updated. (re putting everything on cache)
19:39:09 <zyuan> we have size limit, the space should not matter that much
19:39:28 <kgriffs> we could also compress with snappy
19:39:45 <zyuan> the thing matter may be the performance -- with our own namespace, the processing logic becomes more complex
19:39:51 <flaper87> but you'd be counting the reserved metadata size as part of the queue metadata size
19:39:54 <kgriffs> but idk if cache is faster than mongo if you have it centralized
19:39:55 <flaper87> unless we split them
19:40:17 <kgriffs> here is how splitting into two resource types may look:
19:40:17 <kgriffs> https://etherpad.openstack.org/queuing-scratch
19:40:24 <zyuan> which is not nessasory, from my point of view
19:40:29 <cppcabrera> +1 kgriffs (a whiteboard!)
19:41:20 <cppcabrera> i like this separation of config from metadata...
19:41:25 <kgriffs> meta would have no size checks except valid JSON and size
19:41:25 <cppcabrera> *I
19:41:50 <cppcabrera> config would have type checks, since we know what they are ahead of time.
19:42:05 <flaper87> I prefer the second one
19:42:22 <zyuan> then what's the queue metadata for? what if user PUT garbage into the /meta
19:42:25 <kgriffs> for our first release, we just don't reference the config/settings/whatever resource in the home document
19:42:46 <kgriffs> so, we kick the can down the road and have more time to come up with a good solution
19:42:58 <kgriffs> I think we just ignore garbage fields
19:43:15 <kgriffs> but still check overal doc size for sanity
19:44:02 <cppcabrera> Sounds good to me.
19:44:07 <flaper87> +1
19:44:16 <kgriffs> zyuan: I'm not saying this solves the fundamental problem of schema checking vs. backwards compat with clients, but it lets us divide and conquer
19:44:51 <Sriram> yes, this looks good.
19:45:01 <malini> this is one of the longest 15 minutes :)
19:45:02 <zyuan> soo... what's the plan for the 1st release?
19:45:18 <kgriffs> if we go with this, then what happens if you GET /v1/queues/foo
19:45:34 <kgriffs> 405 ?
19:46:11 <zyuan> PUT has no content at all?
19:46:19 <kgriffs> or do we leave as-is, and later add /queues/foo/defaults
19:46:54 <kgriffs> the original design was based on swift semantics, where they overload metadata headers to do fancy things
19:47:11 <kgriffs> I wonder if that is the best approach after all?
19:47:13 <cppcabrera> x-delete-at, x-delete-after...
19:47:28 <zyuan> /queues/foo/defaults looks too fancy to me....
19:47:51 <flaper87> can we have /queues/foo/meta/foo ?
19:47:56 <flaper87> in the first release?
19:48:08 <flaper87> then we'll add more things when needed
19:48:12 <flaper87> s/meta/metadata/
19:48:14 <kgriffs> what's the lat "foo"
19:48:21 <kgriffs> s/lat/last
19:48:24 <kgriffs> ?
19:48:30 <kgriffs> (almost time for a vote)
19:48:45 <flaper87> ah remove that foo
19:48:54 <flaper87> /queue/foo/metadata/
19:48:54 <zyuan> my option: leave things as-is
19:49:24 <flaper87> if we leave things as they are, we, most likely, will have to break backward compatibility in our next release
19:49:35 <kgriffs> zyuan: as-is, including reserved prefix?
19:49:35 <zyuan> i don't like to invent needs
19:50:13 <zyuan> kgriffs: we have that in doc, and we don't care the guys want to challge us
19:51:05 <zyuan> does the current design close the door of a more fancy metadata?
19:51:35 <flaper87> no if we decide to have reserved words (and checks) within the existing metadata
19:51:49 <flaper87> but if we want to split them, then yes
19:52:31 <zyuan> i'm OK with two metadata plan; i'm c++ guy :)
19:53:04 <flaper87> so, vote? as-is or add new metadata resource to the queue ?
19:53:07 <kgriffs> #startvote Should we keep queue metadata API as is (a), create a metadata type (b), or keep as-is, but not reserve a prefix, adding one or more new settings types later? a, b, c
19:53:08 <openstack> Begin voting on: Should we keep queue metadata API as is (a), create a metadata type (b), or keep as-is, but not reserve a prefix, adding one or more new settings types later? Valid vote options are a, b, c.
19:53:09 <openstack> Vote using '#vote OPTION'. Only your last vote counts.
19:53:39 <flaper87> #vote a
19:53:47 <flaper87> #vote b
19:53:49 <flaper87> sorry
19:53:51 <flaper87> arrg
19:53:53 <flaper87> :)
19:54:06 <kgriffs> fortunately, only last vote counts. :D
19:54:13 <flaper87> indeed :P
19:54:26 <cppcabrera> #vote b
19:54:28 <Sriram> #vote b
19:54:32 <zyuan> #vote a
19:54:55 <zyuan> ...
19:54:57 <kgriffs> anyone else in the room?
19:55:18 <zyuan> the explaination looks... confusing....
19:55:41 <zyuan> what exactly (b) is?
19:55:54 <kgriffs> that is what flaper87 proposed
19:55:57 <megan_w> kgriffs: in the room, but trust you guys to decide
19:56:04 <kgriffs> :D
19:56:29 <kgriffs> "/queue/foo/metadata"
19:56:54 <zyuan> then (c) looks not so good.
19:57:14 <zyuan> why adding new routes in the future conflict with reserving _names now?
19:58:42 <kgriffs> zyuan: it is cleaner, if we add _foo, then harder to migrate that to a different resource later
19:58:46 <kgriffs> #showvote
19:58:47 <openstack> a (1): zyuan
19:58:48 <openstack> b (3): cppcabrera, Sriram, flaper87
19:58:58 <kgriffs> anyone want to change their vote?
19:59:04 <zyuan> clean... he he
19:59:06 <flaper87> kgriffs: u gotta vote
19:59:09 <kgriffs> you have 10 seconds
19:59:15 <bryansd> #vote b
19:59:31 <kgriffs> #vote b
19:59:33 * bryansd votes for pop-tarts, too
19:59:37 <kgriffs> heh
19:59:42 <Sriram> +1 for pop tarts :D
19:59:48 <kgriffs> #endvote
19:59:49 <openstack> Voted on "Should we keep queue metadata API as is (a), create a metadata type (b), or keep as-is, but not reserve a prefix, adding one or more new settings types later?" Results are
19:59:50 <openstack> a (1): zyuan
19:59:50 * cppcabrera votes for donuts
19:59:51 <openstack> b (5): bryansd, kgriffs, cppcabrera, Sriram, flaper87
19:59:53 * flaper87 gives bryansd pop-tarts
20:00:13 <kgriffs> ok, so this is obviously a breaking change. Let's get it done ASAP
20:00:14 <cppcabrera> That's all the time we have, I believe. Any closing comments?
20:00:21 * bryansd is bringing doughnuts tomorrow
20:00:31 <kgriffs> so, quick one about priorities
20:00:49 <zyuan> i'm looking at the board and in deep thinking...
20:00:55 <kgriffs> let's finalize API, continue focus on bugs and robustness and performance
20:01:12 <flaper87> kgriffs: +1 (and client)
20:01:16 <kgriffs> right
20:01:33 <kgriffs> focus on HTTP transport for now, ZMQ is lower priority
20:01:43 <flaper87> +1
20:01:51 <oz_akan_> we need to finalize placement service design
20:02:06 <cppcabrera> +1
20:02:11 <flaper87> oz_akan_: we're planning to discuss it on Monday
20:02:40 <flaper87> since Amit will be out tomorrow
20:02:44 <kgriffs> ok, let's talk metadata implementation in #openstack-marconi
20:02:51 <flaper87> rock on
20:02:55 <flaper87> great work everyone
20:02:58 <oz_akan_> ok
20:03:17 <flaper87> Those bugs are afraid of us
20:03:31 <kgriffs> #action oz_akan, amit, flaper87 to finalize placement service strategy
20:03:47 <kgriffs> #endmeeting