16:03:43 <kgriffs> #startmeeting marconi-api
16:03:44 <openstack> Meeting started Tue Oct 29 16:03:43 2013 UTC and is due to finish in 60 minutes.  The chair is kgriffs. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:03:45 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
16:03:48 <openstack> The meeting name has been set to 'marconi_api'
16:04:03 <kgriffs> #link https://wiki.openstack.org/wiki/Marconi/specs/api/next
16:04:09 <mpanetta> kgriffs: Much better
16:04:28 <kgriffs> #topic Auto-generate client UUID if not given
16:05:04 <kgriffs> we discussed this briefly yesterday
16:05:22 <kgriffs> I propose giving this a thumbs-down
16:05:50 <kgriffs> BUT
16:06:12 <kgriffs> I could be convinced otherwise, if only because other projects don't require it.
16:06:26 <kgriffs> on the other hand, other projects don't have the echo cancellation thing
16:06:50 <kgriffs> which breaks if you auto-generate, so that may cause confusion from users anyway
16:06:56 <kgriffs> flaper87: thoughts?
16:07:08 <flaper87> sorry, I was writing an email :D
16:07:09 <flaper87> back
16:07:18 * flaper87 is a slow email writer
16:07:32 <flaper87> -1 for auto-generated UUIDs
16:07:50 <flaper87> let me give my reasoning
16:07:57 * alcabrera listens
16:08:03 <flaper87> 1) Just 'some' endpoints will use that
16:08:15 <flaper87> 2) I expect users to use marconi with the client library
16:08:38 <flaper87> 3) If users are using marconi with CURL then I don't think it'll be hard for them to add an uuid to the call
16:08:53 <flaper87> 4) I'm +1 for consistency in the API, as much as possible!
16:08:56 <flaper87> that's it
16:09:38 <kgriffs> makes sense
16:09:47 <alcabrera> works for me.
16:10:13 <kgriffs> I would just add to #1 that we want people to be able to mix and match messaging patterns without having to worry about when UUID matters
16:11:10 <flaper87> kgriffs: +1
16:11:23 <kgriffs> #info strike out auto-generate client UUID
16:11:27 <kgriffs> #topic  Clearly define whether client ID is required for every request, and enforce it in the implementation
16:11:49 <kgriffs> currently we do not clearly define it in the spec, and it is only enforced in a few places
16:12:01 <alcabrera> Ah, I thought we were enforcing it every where... hmm...
16:12:07 <kgriffs> I would like to propose always requiring it so that it can be logged and used for analytics
16:12:13 <flaper87> kgriffs: +1
16:12:20 <alcabrera> Yeah, I'm +1 here.
16:12:42 <flaper87> btw, besides openstackgerrit, is anyone writing this down?
16:12:46 <kgriffs> re logging, we will need a way to "bind" params to the logger, so it gets included in every subsequent log line. Can oslo.logging do that?
16:12:55 <flaper87> I can do it if you guys want
16:13:00 <kgriffs> flaper87: I am making notes on the wiki page
16:13:15 <flaper87> kgriffs: you rock, I haven't told you that enough!
16:13:39 <alcabrera> :)
16:14:08 <kgriffs> flaper87: we all rock in our own ways. :D
16:14:20 <kgriffs> for example, you are great at community building
16:14:35 <kgriffs> aaaanyway
16:14:38 * flaper87 is great at eating m&ms
16:14:45 <alcabrera> cpallares: thanks for clearing out the duplicated patches from the review queue! Greatly appreciated. :)
16:14:55 * kgriffs suddenly has a craving for chocolate
16:15:24 <mpanetta> kgriffs: The stuff you got me is almost gone, soo good, thank you again!
16:15:28 <cpallares> alcabrera: Just glad to help out :D
16:16:22 <kgriffs> #info Always require client id, and log the id in every log line for tracing, analytics, support, etc.
16:16:34 <kgriffs> #topic Consider allowing opaque string for client ID rather than UUID (will need to understand what else people want to use?)
16:16:51 <kgriffs> so... I'm not sure on this one
16:17:01 <zyuan> kgriffs: the door is closed
16:17:10 <kgriffs> like flaper87 said, it isn't *that* hard to generate one
16:17:30 <kgriffs> and saying it has to be uuid makes validation easier, and reduces confusion IMO
16:17:36 <kgriffs> any objections to striking this?
16:18:11 * flaper87 listens to the silence
16:18:31 <kgriffs> #info keep UUID requirement for Client-ID
16:18:43 <kgriffs> #topic Remove deprecated "partial results" semantics from message posting
16:18:52 <zyuan> +0.99
16:19:01 <kgriffs> I added this one, since it is not longer possible to get partial results from the mongo driver
16:19:15 <flaper87> you can get partial claims
16:19:17 <kgriffs> the only reason to keep it is if we expect other backends to need it
16:19:37 <kgriffs> flaper87: true, this proposal is just when posting messages
16:19:45 <flaper87> as for now, I'm not sure about other backends needing it!
16:19:50 <flaper87> kgriffs: ah ok, sorry! I missed that
16:19:53 <zyuan> i would suggest try optional fields for "other backends"
16:20:08 <kgriffs> mmm. makes me think of extensions
16:20:36 <kgriffs> ok, so get rid of the notion of a "partial" message post? Any objections?
16:20:59 <flaper87> kgriffs: would it make sense to investigate other storage technologies?
16:21:05 <flaper87> before taking a final decision
16:21:41 <flaper87> mmh, actually
16:21:53 <flaper87> I think the right quesiton here is: Do we want to allow this?
16:21:59 <kgriffs> yeah
16:22:07 <kgriffs> I guess that is what I was getting at as well
16:22:18 <kgriffs> it does complicate clients
16:22:39 <flaper87> I'm leaning towards removing partial message posting completely
16:22:44 <alcabrera> hmm...
16:23:05 <alcabrera> I like the concept of all-or-nothing here.
16:23:20 <kgriffs> alcabrera: do you think you will have any trouble implementing that with Redis, for example?
16:23:33 <flaper87> it's different for claims, because one can re-claim stuff
16:23:36 <kgriffs> SQL db's will have no problem, obviously
16:23:41 <flaper87> but for messages there's too much to be aware off
16:24:09 <alcabrera> it shouldn't be any problem with Redis. IIRC, posting all the messages is a single communication with Redis.
16:24:22 <flaper87> btw, what are we returning if the max_retries is reached in mongodb's backend?
16:24:28 <kgriffs> and redis will not ack on partial success?
16:24:59 <kgriffs> flaper87: probably 503
16:25:27 <flaper87> ok!
16:25:50 <alcabrera> kgriffs: turns out I'm doing one communication per message, so... it's not atomic with Redis, either. :P
16:25:51 <kgriffs> #topic Include Client ID in claim data
16:25:51 <flaper87> cool, so, voting
16:26:03 <flaper87> erm, I mean, we all agreed
16:26:04 <alcabrera> kgriffs: no partial ACK
16:26:05 <flaper87> :D
16:26:05 <kgriffs> alcabrera: oic. You would need to batch them, then
16:26:10 <alcabrera> yup
16:26:29 <kgriffs> so, client ID in claim data
16:26:48 <kgriffs> if we do that, then we can include claim info when listing messages
16:27:09 <kgriffs> someone mentioned that would be nice for auditing and stuff
16:27:27 <kgriffs> cons are extra storage space
16:27:28 <flaper87> mmh, agreed, however, the claim does not belong to that specific client
16:27:34 <flaper87> it was created by that client, though
16:27:40 <kgriffs> yes, true
16:27:48 <flaper87> any client holding that same claim id can still consume the claim
16:27:48 <kgriffs> "created_by"
16:27:52 <flaper87> kk
16:28:04 <zyuan> listing?
16:28:09 <kgriffs> uuid is how many bytes, zyuan?
16:28:10 <zyuan> is that helpful?
16:28:15 <flaper87> 34 ?
16:28:30 <alcabrera> 36
16:28:33 <flaper87> damnit!
16:28:36 <kgriffs> zyuan: someone thought it would be
16:28:37 <flaper87> :D
16:28:38 <alcabrera> len(str(uuid.uuid4())) => 36
16:28:43 <zyuan> 16
16:29:03 <zyuan> an 128 bit integer
16:29:03 <kgriffs> in bson it is 16, right?
16:29:08 <zyuan> anywhere
16:29:31 <alcabrera> ahh, uuid as int
16:29:35 <kgriffs> so, 16 bytes extra per claimed message. seems reasonable.
16:29:52 <zyuan> waitwaitwait
16:29:59 <alcabrera> uuid.uuid1().int - gtk
16:30:02 <zyuan> in message, it's hex form
16:30:07 <zyuan> ~36
16:30:08 <kgriffs> (plus a little overhead for the field name)
16:30:23 <kgriffs> zyuan: only when serialized, right?
16:30:35 <zyuan> 16 in storage
16:30:35 <kgriffs> i mean, in mongo we just stick it in the claim doc
16:30:40 <flaper87> right
16:30:41 <zyuan> ~36 in JSON
16:30:46 <kgriffs> ok
16:30:53 <kgriffs> seems like space wouldn't be a big deal
16:31:02 <alcabrera> also gtk - uuid.UUID(uuid_as_str).int works
16:31:16 <flaper87> and it is 32
16:31:24 <flaper87> (whithout - )
16:31:28 <flaper87> without
16:31:39 <flaper87> which is what we're using, IIRC
16:32:03 <kgriffs> ok, let's tentatively plan this for v1.1
16:32:21 <flaper87> +1
16:32:25 <alcabrera> +1
16:32:44 <flaper87> alcabrera: str(uuid) adds -, uuid.hex gives you the real hex repr
16:32:58 <zyuan> we use str with -
16:33:06 <zyuan> we accept hex only as well
16:33:21 <flaper87> zyuan: kk, thanks for the hint!
16:33:54 <kgriffs> two more things i'd like to try and triage today
16:34:03 <kgriffs> #topic List claims for a given queue
16:34:08 <kgriffs> user request
16:34:14 <kgriffs> not sure how useful it is
16:34:18 <zyuan> 0 interests
16:34:22 <flaper87> 0 interest
16:34:31 <flaper87> -1 from me!
16:34:37 <zyuan> this is how the conversation wents
16:34:48 <flaper87> that may work as an admin endpoint or something
16:35:09 <flaper87> but I don't think it'll be of any help for real messaging systems
16:35:24 <alcabrera> sounds useful in the context of /stats - # active claims on a given queue.
16:35:26 <flaper87> also, that will make it difficult to support all kind of storage
16:35:34 <kgriffs> alcabrera: +1
16:35:37 <alcabrera> so... +1 as an admin endpoint
16:35:45 <alcabrera> -1 as a user endpoint
16:35:55 <kgriffs> i think a user could find it useful
16:36:02 <kgriffs> maybe use it for autoscaling or something
16:36:05 <kgriffs> (the count)
16:36:08 <kgriffs> (not the list)
16:36:13 <zyuan> too much, too trikey
16:36:44 <flaper87> I agree with zyuan here!
16:36:55 <mpanetta> Sounds like a way to introduce a race...
16:36:56 <kgriffs> yes, the stat would be difficult to do and make fast
16:37:17 <kgriffs> knowing claimed vs free should be sufficient
16:37:22 <flaper87> I'd rather use N queues or N messages for autoscalling - or a ratio of those 2.
16:37:55 <kgriffs> TBH, I think this request came from someone who just wanted to make sure their claim succeeded
16:37:59 <alcabrera> +1 - message # sounds like it'd the core load metric
16:38:27 <zyuan> if you get it from claim, then it *is* successed...
16:38:32 <kgriffs> ok, so strike this one?
16:38:50 <zyuan> .rej
16:38:58 <flaper87> reject!
16:39:25 <alcabrera> reject~
16:41:14 <kgriffs> #info do not implement listing claims per queue
16:41:26 <kgriffs> #info existing stats should suffice for scaling metrics
16:41:31 <kgriffs> #topic Automatically create queues the first time a message is posted to it
16:41:32 <kgriffs> last one
16:41:36 <kgriffs> (last topic)
16:41:52 <kgriffs> flaper87: this one was your suggestion, right?
16:42:09 <flaper87> kgriffs: correct!
16:42:26 <mpanetta> Ooo
16:42:34 <zyuan> hmm
16:42:38 <flaper87> I think this one could be expanded a bit, TBH!
16:43:14 <zyuan> i think we are going to have control panel to create queue
16:43:23 <alcabrera> hmmm
16:43:29 <zyuan> now we are going to see queues created by typos in the panel
16:43:40 <flaper87> I mean, depending on the storage backend, we may want to stop treating queues as resources
16:43:56 <flaper87> but that's up to the storage
16:43:59 <zyuan> i know
16:44:08 <alcabrera> so pros - one less call to post a message (-PUT queue), and cons: a typo creates a queue.
16:44:09 <flaper87> I just wanted to mention some of the things that are possible
16:44:26 <flaper87> alcabrera: not sure about the type creates a queue
16:44:29 <zyuan> and... it can actually decrease 204 vs 404 complains
16:44:33 <zyuan> hehe
16:44:44 <flaper87> I mean, not sure what you mean
16:44:46 <flaper87> :D
16:45:01 <zyuan> so, if queue is nolonger regarded as a resource
16:45:11 <zyuan> it's just fine if we don;t return 404 if it's not there
16:45:21 <zyuan> it's just an attribute of message
16:45:23 <alcabrera> To clarify on that con -
16:46:06 <flaper87> but, mmh, we won't be able to have metadata for that queue
16:46:08 <flaper87> mmh
16:46:12 <alcabrera> If someone develops an application where they have workers reading from queues/feed, and they have producers that POST to queues/feeed/messages... :x
16:46:15 <flaper87> if we don't treat it as resource, I mean
16:46:27 <flaper87> btw, this is v2 material!
16:46:36 <flaper87> just want to make that point clear!
16:47:31 <zyuan> yea. i can't quickly think of how this is being implemented, but i suspect that it has some value
16:47:49 <kgriffs> #info lazy queues would be for v2 if we do it, not v1.1
16:48:05 <alcabrera> +1 for v2 feature
16:48:07 <zyuan> lgtm
16:48:07 <flaper87> the value I see in this is the lazyness of the API, it would work mostly as mongodb collections do!
16:48:15 <alcabrera> I'm rather in favor of it
16:48:27 <flaper87> plus other things, obviously
16:48:36 <flaper87> but that's the one in terms of API semantic
16:48:56 <alcabrera> ooohh, idea for dealing with that whole typo issue... :D
16:49:06 <flaper87> haha
16:49:13 <alcabrera> So, what if...
16:49:16 <alcabrera> We had a query flag
16:49:24 <alcabrera> That by derfault allowed lazy creation
16:49:38 <alcabrera> But if users preferred the checking behavior (maybe for debug dev?)
16:49:45 <alcabrera> That they could set to True
16:49:47 <flaper87> alcabrera: that could work!
16:49:49 <alcabrera> E.g.
16:49:52 <alcabrera> strict=False
16:49:55 <alcabrera> something like that
16:49:55 <flaper87> however, I'd let the client and API discovery take care of that
16:50:01 <flaper87> :D
16:50:26 <flaper87> the API discovery should be ready before this work start
16:50:31 <alcabrera> +1
16:50:32 <kgriffs> #info mitigate typos with a sticy/lazy flag on the operation
16:50:32 <flaper87> and the client should fully support v1
16:50:48 <kgriffs> s/sticy/strict
16:50:55 <flaper87> which would make it easier to migrate it to v2
16:50:57 <flaper87> I hope
16:51:00 * flaper87 hopes
16:51:05 <flaper87> everybody hopes
16:51:13 <alcabrera> agreed
16:52:06 <kgriffs> ok
16:52:09 <kgriffs> that's all folks!
16:52:12 <kgriffs> #endmeeting