16:03:43 #startmeeting marconi-api 16:03:44 Meeting started Tue Oct 29 16:03:43 2013 UTC and is due to finish in 60 minutes. The chair is kgriffs. Information about MeetBot at http://wiki.debian.org/MeetBot. 16:03:45 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 16:03:48 The meeting name has been set to 'marconi_api' 16:04:03 #link https://wiki.openstack.org/wiki/Marconi/specs/api/next 16:04:09 kgriffs: Much better 16:04:28 #topic Auto-generate client UUID if not given 16:05:04 we discussed this briefly yesterday 16:05:22 I propose giving this a thumbs-down 16:05:50 BUT 16:06:12 I could be convinced otherwise, if only because other projects don't require it. 16:06:26 on the other hand, other projects don't have the echo cancellation thing 16:06:50 which breaks if you auto-generate, so that may cause confusion from users anyway 16:06:56 flaper87: thoughts? 16:07:08 sorry, I was writing an email :D 16:07:09 back 16:07:18 * flaper87 is a slow email writer 16:07:32 -1 for auto-generated UUIDs 16:07:50 let me give my reasoning 16:07:57 * alcabrera listens 16:08:03 1) Just 'some' endpoints will use that 16:08:15 2) I expect users to use marconi with the client library 16:08:38 3) If users are using marconi with CURL then I don't think it'll be hard for them to add an uuid to the call 16:08:53 4) I'm +1 for consistency in the API, as much as possible! 16:08:56 that's it 16:09:38 makes sense 16:09:47 works for me. 16:10:13 I would just add to #1 that we want people to be able to mix and match messaging patterns without having to worry about when UUID matters 16:11:10 kgriffs: +1 16:11:23 #info strike out auto-generate client UUID 16:11:27 #topic Clearly define whether client ID is required for every request, and enforce it in the implementation 16:11:49 currently we do not clearly define it in the spec, and it is only enforced in a few places 16:12:01 Ah, I thought we were enforcing it every where... hmm... 16:12:07 I would like to propose always requiring it so that it can be logged and used for analytics 16:12:13 kgriffs: +1 16:12:20 Yeah, I'm +1 here. 16:12:42 btw, besides openstackgerrit, is anyone writing this down? 16:12:46 re logging, we will need a way to "bind" params to the logger, so it gets included in every subsequent log line. Can oslo.logging do that? 16:12:55 I can do it if you guys want 16:13:00 flaper87: I am making notes on the wiki page 16:13:15 kgriffs: you rock, I haven't told you that enough! 16:13:39 :) 16:14:08 flaper87: we all rock in our own ways. :D 16:14:20 for example, you are great at community building 16:14:35 aaaanyway 16:14:38 * flaper87 is great at eating m&ms 16:14:45 cpallares: thanks for clearing out the duplicated patches from the review queue! Greatly appreciated. :) 16:14:55 * kgriffs suddenly has a craving for chocolate 16:15:24 kgriffs: The stuff you got me is almost gone, soo good, thank you again! 16:15:28 alcabrera: Just glad to help out :D 16:16:22 #info Always require client id, and log the id in every log line for tracing, analytics, support, etc. 16:16:34 #topic Consider allowing opaque string for client ID rather than UUID (will need to understand what else people want to use?) 16:16:51 so... I'm not sure on this one 16:17:01 kgriffs: the door is closed 16:17:10 like flaper87 said, it isn't *that* hard to generate one 16:17:30 and saying it has to be uuid makes validation easier, and reduces confusion IMO 16:17:36 any objections to striking this? 16:18:11 * flaper87 listens to the silence 16:18:31 #info keep UUID requirement for Client-ID 16:18:43 #topic Remove deprecated "partial results" semantics from message posting 16:18:52 +0.99 16:19:01 I added this one, since it is not longer possible to get partial results from the mongo driver 16:19:15 you can get partial claims 16:19:17 the only reason to keep it is if we expect other backends to need it 16:19:37 flaper87: true, this proposal is just when posting messages 16:19:45 as for now, I'm not sure about other backends needing it! 16:19:50 kgriffs: ah ok, sorry! I missed that 16:19:53 i would suggest try optional fields for "other backends" 16:20:08 mmm. makes me think of extensions 16:20:36 ok, so get rid of the notion of a "partial" message post? Any objections? 16:20:59 kgriffs: would it make sense to investigate other storage technologies? 16:21:05 before taking a final decision 16:21:41 mmh, actually 16:21:53 I think the right quesiton here is: Do we want to allow this? 16:21:59 yeah 16:22:07 I guess that is what I was getting at as well 16:22:18 it does complicate clients 16:22:39 I'm leaning towards removing partial message posting completely 16:22:44 hmm... 16:23:05 I like the concept of all-or-nothing here. 16:23:20 alcabrera: do you think you will have any trouble implementing that with Redis, for example? 16:23:33 it's different for claims, because one can re-claim stuff 16:23:36 SQL db's will have no problem, obviously 16:23:41 but for messages there's too much to be aware off 16:24:09 it shouldn't be any problem with Redis. IIRC, posting all the messages is a single communication with Redis. 16:24:22 btw, what are we returning if the max_retries is reached in mongodb's backend? 16:24:28 and redis will not ack on partial success? 16:24:59 flaper87: probably 503 16:25:27 ok! 16:25:50 kgriffs: turns out I'm doing one communication per message, so... it's not atomic with Redis, either. :P 16:25:51 #topic Include Client ID in claim data 16:25:51 cool, so, voting 16:26:03 erm, I mean, we all agreed 16:26:04 kgriffs: no partial ACK 16:26:05 :D 16:26:05 alcabrera: oic. You would need to batch them, then 16:26:10 yup 16:26:29 so, client ID in claim data 16:26:48 if we do that, then we can include claim info when listing messages 16:27:09 someone mentioned that would be nice for auditing and stuff 16:27:27 cons are extra storage space 16:27:28 mmh, agreed, however, the claim does not belong to that specific client 16:27:34 it was created by that client, though 16:27:40 yes, true 16:27:48 any client holding that same claim id can still consume the claim 16:27:48 "created_by" 16:27:52 kk 16:28:04 listing? 16:28:09 uuid is how many bytes, zyuan? 16:28:10 is that helpful? 16:28:15 34 ? 16:28:30 36 16:28:33 damnit! 16:28:36 zyuan: someone thought it would be 16:28:37 :D 16:28:38 len(str(uuid.uuid4())) => 36 16:28:43 16 16:29:03 an 128 bit integer 16:29:03 in bson it is 16, right? 16:29:08 anywhere 16:29:31 ahh, uuid as int 16:29:35 so, 16 bytes extra per claimed message. seems reasonable. 16:29:52 waitwaitwait 16:29:59 uuid.uuid1().int - gtk 16:30:02 in message, it's hex form 16:30:07 ~36 16:30:08 (plus a little overhead for the field name) 16:30:23 zyuan: only when serialized, right? 16:30:35 16 in storage 16:30:35 i mean, in mongo we just stick it in the claim doc 16:30:40 right 16:30:41 ~36 in JSON 16:30:46 ok 16:30:53 seems like space wouldn't be a big deal 16:31:02 also gtk - uuid.UUID(uuid_as_str).int works 16:31:16 and it is 32 16:31:24 (whithout - ) 16:31:28 without 16:31:39 which is what we're using, IIRC 16:32:03 ok, let's tentatively plan this for v1.1 16:32:21 +1 16:32:25 +1 16:32:44 alcabrera: str(uuid) adds -, uuid.hex gives you the real hex repr 16:32:58 we use str with - 16:33:06 we accept hex only as well 16:33:21 zyuan: kk, thanks for the hint! 16:33:54 two more things i'd like to try and triage today 16:34:03 #topic List claims for a given queue 16:34:08 user request 16:34:14 not sure how useful it is 16:34:18 0 interests 16:34:22 0 interest 16:34:31 -1 from me! 16:34:37 this is how the conversation wents 16:34:48 that may work as an admin endpoint or something 16:35:09 but I don't think it'll be of any help for real messaging systems 16:35:24 sounds useful in the context of /stats - # active claims on a given queue. 16:35:26 also, that will make it difficult to support all kind of storage 16:35:34 alcabrera: +1 16:35:37 so... +1 as an admin endpoint 16:35:45 -1 as a user endpoint 16:35:55 i think a user could find it useful 16:36:02 maybe use it for autoscaling or something 16:36:05 (the count) 16:36:08 (not the list) 16:36:13 too much, too trikey 16:36:44 I agree with zyuan here! 16:36:55 Sounds like a way to introduce a race... 16:36:56 yes, the stat would be difficult to do and make fast 16:37:17 knowing claimed vs free should be sufficient 16:37:22 I'd rather use N queues or N messages for autoscalling - or a ratio of those 2. 16:37:55 TBH, I think this request came from someone who just wanted to make sure their claim succeeded 16:37:59 +1 - message # sounds like it'd the core load metric 16:38:27 if you get it from claim, then it *is* successed... 16:38:32 ok, so strike this one? 16:38:50 .rej 16:38:58 reject! 16:39:25 reject~ 16:41:14 #info do not implement listing claims per queue 16:41:26 #info existing stats should suffice for scaling metrics 16:41:31 #topic Automatically create queues the first time a message is posted to it 16:41:32 last one 16:41:36 (last topic) 16:41:52 flaper87: this one was your suggestion, right? 16:42:09 kgriffs: correct! 16:42:26 Ooo 16:42:34 hmm 16:42:38 I think this one could be expanded a bit, TBH! 16:43:14 i think we are going to have control panel to create queue 16:43:23 hmmm 16:43:29 now we are going to see queues created by typos in the panel 16:43:40 I mean, depending on the storage backend, we may want to stop treating queues as resources 16:43:56 but that's up to the storage 16:43:59 i know 16:44:08 so pros - one less call to post a message (-PUT queue), and cons: a typo creates a queue. 16:44:09 I just wanted to mention some of the things that are possible 16:44:26 alcabrera: not sure about the type creates a queue 16:44:29 and... it can actually decrease 204 vs 404 complains 16:44:33 hehe 16:44:44 I mean, not sure what you mean 16:44:46 :D 16:45:01 so, if queue is nolonger regarded as a resource 16:45:11 it's just fine if we don;t return 404 if it's not there 16:45:21 it's just an attribute of message 16:45:23 To clarify on that con - 16:46:06 but, mmh, we won't be able to have metadata for that queue 16:46:08 mmh 16:46:12 If someone develops an application where they have workers reading from queues/feed, and they have producers that POST to queues/feeed/messages... :x 16:46:15 if we don't treat it as resource, I mean 16:46:27 btw, this is v2 material! 16:46:36 just want to make that point clear! 16:47:31 yea. i can't quickly think of how this is being implemented, but i suspect that it has some value 16:47:49 #info lazy queues would be for v2 if we do it, not v1.1 16:48:05 +1 for v2 feature 16:48:07 lgtm 16:48:07 the value I see in this is the lazyness of the API, it would work mostly as mongodb collections do! 16:48:15 I'm rather in favor of it 16:48:27 plus other things, obviously 16:48:36 but that's the one in terms of API semantic 16:48:56 ooohh, idea for dealing with that whole typo issue... :D 16:49:06 haha 16:49:13 So, what if... 16:49:16 We had a query flag 16:49:24 That by derfault allowed lazy creation 16:49:38 But if users preferred the checking behavior (maybe for debug dev?) 16:49:45 That they could set to True 16:49:47 alcabrera: that could work! 16:49:49 E.g. 16:49:52 strict=False 16:49:55 something like that 16:49:55 however, I'd let the client and API discovery take care of that 16:50:01 :D 16:50:26 the API discovery should be ready before this work start 16:50:31 +1 16:50:32 #info mitigate typos with a sticy/lazy flag on the operation 16:50:32 and the client should fully support v1 16:50:48 s/sticy/strict 16:50:55 which would make it easier to migrate it to v2 16:50:57 I hope 16:51:00 * flaper87 hopes 16:51:05 everybody hopes 16:51:13 agreed 16:52:06 ok 16:52:09 that's all folks! 16:52:12 #endmeeting