19:05:40 <kgriffs> #startmeeting marconi
19:05:41 <openstack> Meeting started Thu Mar  7 19:05:40 2013 UTC.  The chair is kgriffs. Information about MeetBot at http://wiki.debian.org/MeetBot.
19:05:42 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
19:05:44 <openstack> The meeting name has been set to 'marconi'
19:06:11 <kgriffs> #topic Process update
19:07:41 <kgriffs> So, we've been hard at work setting up tasks and adding blueprints to organize everything that needs to be done before the Portland summit
19:08:15 <kgriffs> #info Smaller work items are tracked in Trello: https://trello.com/b/7NLODgbr
19:08:38 <flaper87> o/
19:09:05 <kgriffs> for larger work items/complicated stuff we will continue adding blueprints in the next couple of weeks.
19:09:14 <kgriffs> flaper87: welcome!
19:09:40 * flaper87 thought the meeting was at 21:00 his time.....
19:09:57 <flaper87> so, what's the current topic?
19:10:04 <kgriffs> process update
19:11:27 <kgriffs> also, I just want to let everyone know we now have a HACKING file and there is enough code in the repo for folks to start getting the lay of the land.
19:11:49 * flaper87 dances
19:11:50 <kgriffs> anyway, does anyone have any questions about the current state of Marconi?
19:12:21 <flaper87> o/
19:12:33 <flaper87> about the message post api
19:12:50 * flaper87 wonders if that's off-topic
19:13:09 <kgriffs> no worries, let's move on
19:13:40 <kgriffs> #topic API - Post Message
19:13:57 <flaper87> cool
19:14:00 <flaper87> so
19:14:03 <kgriffs> #info API working draft - https://etherpad.openstack.org/queuing-api
19:14:57 <flaper87> the current spec says that when a message is posted, marconi returns an empty response with the Location header! But, when multiple messages are posted (bulk) it just doesn't return anything
19:15:08 <flaper87> hopefully I didn't miss-read any of that
19:16:05 <flaper87> so, That doesn't feel right and I think we should give a consistent response either case
19:16:06 <kgriffs> well, as written, it will return just the location of the messages resource
19:16:15 <kgriffs> agreed
19:16:21 <flaper87> here's my thought
19:17:08 <flaper87> What about *always* returning a list of dicts containing the location of the messages and other information (some metadata related to the message)? OR
19:17:15 <flaper87> just returning a list of ids / locations
19:19:06 <flaper87> thoughts?
19:19:15 <kgriffs> so, the list of id's would be returned in the same order as messages posted?
19:19:16 <edsrzf> I think it would be good to get the IDs of all the messages posted, one way or another
19:19:32 <flaper87> edsrzf: +1
19:19:36 <kgriffs> or does that matter?
19:19:51 <flaper87> kgriffs: mmh, good point, I think that matters
19:19:52 <flaper87> I mean
19:20:06 <flaper87> it doesn't but having that small "coolness" is certainly good
19:20:34 <flaper87> but, clients should already know in what order messages where inserted
19:20:39 <flaper87> since they did the request
19:21:06 <flaper87> I think returning the list of ids is good enough
19:21:14 <kgriffs> an alternative to returning the ids in the body would be something like
19:21:17 <flaper87> I can't think, right now, about a case where we'd need more info
19:21:33 <edsrzf> If the list is unordered, how can the client associate an ID with a message? Or are you saying you don't think that's necessary?
19:21:39 <flaper87> the idea of just returning the ids is to save bandwith
19:21:40 <kgriffs> Location: .../messages/id1,id2,id3
19:22:40 <flaper87> edsrzf: the list wouldn't be un-ordered but I don't think the association is an issue in that case
19:23:10 <bryansd> kgriffs: Then that would mean the GET endpoint would need to support that syntax, too, correct?
19:23:20 <flaper87> kgriffs: and will it be possible to call that url? I like that Location format and I've used it in other projects with good results
19:23:20 <kgriffs> bryansd: correct
19:23:44 <flaper87> bryansd is faster than me
19:23:58 <kgriffs> I think it would be interesting to use that syntax for bulk queries across multiple queues as well, but that's a discussion for another day
19:24:13 <flaper87> yep
19:25:37 <flaper87> so, ordered list of ids?
19:25:44 <kgriffs> #startvote return id's in header or body? Header, Body
19:25:44 <openstack> Begin voting on: return id's in header or body? Valid vote options are Header, Body.
19:25:45 <openstack> Vote using '#vote OPTION'. Only your last vote counts.
19:26:09 <flaper87> #vote Body
19:26:36 <kgriffs> #vote Header
19:26:49 <bryansd> #vote Header
19:27:10 <bryansd> (in the name of reducing payload)
19:27:33 * flaper87 assumes we all agreed on returning a list of ids
19:27:47 <bryansd> flaper87: agreed
19:28:20 <kgriffs> so, we will return a list of IDs, the question is do we do it in the body as a document or in the Location header
19:28:24 <flaper87> bryansd: mmh might be, but I think that headers are not meant to be used like that
19:28:50 <flaper87> they should contain parameters need for the communication but not the actual response
19:28:57 <kgriffs> well, if we were trying to return more than the id, I would agree
19:28:57 <flaper87> anyway, I'm ok with it as well
19:29:11 <flaper87> so, endvote?
19:29:29 <kgriffs> actually, it is Best Practice™ to return the location of the newly created resource.
19:29:37 <kgriffs> #endvote
19:29:38 <openstack> Voted on "return id's in header or body?" Results are
19:29:39 <openstack> Body (1): flaper87
19:29:40 <openstack> Header (2): bryansd, kgriffs
19:29:42 <flaper87> kgriffs: agreed on that
19:30:13 <flaper87> so, that would be a list of , separated ids ?
19:30:20 <kgriffs> I think we can list the IDs in the URI and if we ever decide we need to return more data than that, we can also add a body
19:30:29 <flaper87> that will need to be splited in the client side ?
19:30:39 <flaper87> kgriffs: +1
19:30:43 <flaper87> for teh uri stuff
19:30:55 <flaper87> kgriffs: place an #agree on that
19:31:02 <flaper87> :)
19:31:07 <kgriffs> i guess so. We could define a URI template that would allow the client to do know what to expect and parse it.
19:31:47 * flaper87 doesn't know how to do the f#@$@$ TM symbol
19:32:35 <bryansd> And with regards to ordering, we ARE guaranteeing messages are posted in the order received, correct? With the stipulation it's possible other clients may have inserted their own messages in the middle.
19:33:00 <kgriffs> #agree Response to Post Message will be the full path to the message or messages created
19:33:27 <flaper87> bryansd: we guarantee the order but not that the'll be sequential
19:33:27 <kgriffs> #agree Extend messages URI template to allow for multiple message_id's
19:34:10 <bryansd> flaper87: Nice verbiage :-)
19:34:28 * flaper87 deserves a pop-tart
19:34:50 <bryansd> So the list of IDs returned should be in the correct order as well
19:34:55 <kgriffs> bryansd: I imagine we can guarantee that the node will send the messages to storage in the order received, but I'm not sure that means a lot since messages can be posted in parallel.
19:35:37 <flaper87> kgriffs: bryansd yes, what really matters there is that we respect the order of the current requests
19:35:51 <kgriffs> although, that node should be able to return the list of ids to the client in the same order they were in the body of the POST
19:35:57 <flaper87> the rest is up to the god of the Race Conditions!
19:36:18 <kgriffs> OK, so I guess we need to specify that in the transport blueprint
19:36:27 <flaper87> kgriffs: +1
19:36:43 * flaper87 will add that to the base class as well
19:37:08 <kgriffs> flaper87: do you want to take care of the blueprint or shall I?
19:37:20 <flaper87> kgriffs: I'll take care of that as well
19:38:02 <kgriffs> #action flaper87 to update Transport blueprint and base class with info about returning ids in same order that messages are written to storage
19:38:07 * flaper87 ™ <- cool
19:38:33 * bryansd gives flaper87 a Pop-Tart™
19:38:55 <kgriffs> ok, moving on?
19:38:57 <flaper87> yes
19:39:08 <kgriffs> so, same topic, but different tangent
19:39:26 <kgriffs> Any objections to allowing per-message TTLs?
19:39:47 <kgriffs> And if the TTL is not specified, it defaults to the setting in the queue metadata?
19:40:07 <flaper87> that sounds good to me
19:40:11 <flaper87> so no objections so far
19:40:12 <bryansd> I was thinking about that. Just wondering if that's adding an extra layer of complexity to a v1.0 release. Any compelling reason we need it?
19:40:36 <kgriffs> mostly to be consistent with the way claims work
19:40:56 <flaper87> bryansd: thinking about how the current implementation is moving on, that'd be really easy to implement
19:40:57 <kgriffs> although, we could just say for both claims and queues, you set the global value and can't change it per-claim or per-message.
19:41:46 <flaper87> so, a few "implementative" words about that (just to be sure it makes sense to have it in v1)
19:42:49 <flaper87> In order to post a message we need to have the queue where it will be posted (kind of model like) which should (i guess it depends on the storage backend as well) contain the per-queue ttl already
19:43:05 <flaper87> that would allow us to manage per-message ttl when it is not specified
19:43:50 <flaper87> so, I guess what I'm trying to say is that at the end, we'll just manage per-message ttl and putting it in the queue meta or directly in the message just changes the level where the parameter is stored
19:43:54 <flaper87> does that make sense?
19:44:46 <flaper87> the element that expires is the message, not the queue
19:45:24 <flaper87> again, that could definitely vary when it comes to the final implementation (which involves the storage backend)
19:45:35 <flaper87> thoughts?
19:46:03 <kgriffs> it seems to me that assuming the same TTL for all messages in a given queue might lead choices in the storage driver implementation that make it hard to do different TTLs for each message later on.
19:47:02 <kgriffs> re storage model, I think "queues" are really just a metadata collection, indexed by queue name.
19:47:37 <flaper87> kgriffs: wow, you expressed that way simpler than what I did
19:47:45 <flaper87> kgriffs: +1, that's what I meant
19:47:48 <flaper87> :P
19:47:50 <kgriffs> sweet
19:48:43 <kgriffs> we need to be careful that we don't tightly couple the abstract notion of a "queue" to a physical resource that is expensive to create/manage, since we want to support gazillions of queues.
19:48:56 <kgriffs> anyway, that's a discussion for another day.
19:49:08 <flaper87> kgriffs: agreed
19:49:27 <bryansd> Okay, I was only concerned with getting to a minimum viable product. Sounds like it won't be too much of a concern.
19:49:44 <kgriffs> #topic API - Create Queue - Metadata Format
19:50:17 <kgriffs> so, couple of questions I had on this
19:50:37 <flaper87> shot
19:50:42 <kgriffs> first, do we allow arbitrary metadata with some space limit (say, 64K)
19:51:18 <kgriffs> second, do we say that only simple key-value's are allowed, or are nested docs OK?
19:52:59 <flaper87> so, first, I honestly don't think that arbitrary metadata is needed in the queue
19:53:43 <flaper87> mmhh, wait
19:53:56 <flaper87> no no, I guess it's not needed
19:54:30 <flaper87> Is there a case where a client could need to store arbitrary metadata to a queue for later use?
19:54:42 <bryansd> I guess I'm the Debbie Downer of features but I vote no. Agree with flaper87
19:55:09 <kgriffs> Well, metadata can be stored out-of-band by the app that's using the queue, I suppose. There may be a useful reason for it, but we can add it later if the need arises.
19:55:25 <bryansd> +1 for adding it later based on demand
19:55:43 <kgriffs> Do Swift containers allow arbitrary metadata?
19:55:55 * kgriffs can't remember
19:56:03 <flaper87> kgriffs: if there are use cases for that, then cool!!! +1 for adding it later
19:56:13 <flaper87> mmh, don't remember either
19:57:06 <flaper87> mhh, wait again
19:57:18 <flaper87> I think it could be useful
19:57:21 <flaper87> for example
19:57:39 <kgriffs> http://docs.rackspace.com/files/api/v1/cf-devguide/content/Update_Container_Metadata-d1e1900.html
19:58:10 <flaper87> thinking about SNS, if a message get's published to a specific queue and it has to be notified to X plugin based on queue metadata (filtering or whatever)
19:59:08 <flaper87> I now think it is useful. It isn't for the *actual* work the queue should do but it could be for other stuff
19:59:33 <kgriffs> I guess my point is, TTL is probably just the beginning of other stuff we want to store, so if we are building a mechanism for storing multiple config items, essentially, it is trivial to allow arbitrary metadata
19:59:51 <flaper87> indeed
20:00:29 <flaper87> I agree on allowing arbitrary metadata
20:00:36 <kgriffs> I guess, as long as we build a JSON schema that can allow some kind of nested docs or we just use X-headers like Swift
20:00:47 <flaper87> dunno if it is worth it to add a limit on the size of it
20:01:01 <flaper87> and I agree on allowing nested objects as well
20:01:39 <kgriffs> well, it could be really high, we just need to sanity-check it so someone isn't upload a few gigs worth of metadta
20:02:02 <flaper87> lol
20:02:23 <kgriffs> re nested objects, that may change the way the controller base classes are defined - maybe should take a single dict or something
20:02:34 * flaper87 thinks about someone saving a movie in a last_seen metadata field
20:03:04 <flaper87> kgriffs: why should it change it?
20:03:13 <flaper87> it all goes inside the same metadata field
20:03:15 <kgriffs> #agreed The tentative plan is to allow custom metadata, as long as it isn't much more work than just doing TTL.
20:03:24 <kgriffs> well, does metadata need to be a kwargs?
20:03:50 <flaper87> kind of, I think of metadata as a dictionary of key-object
20:04:03 <kgriffs> I guess you would decompose the top-level keys in the document.
20:04:37 <kgriffs> so, messages={"ttl":30}
20:04:41 <kgriffs> or something like that
20:04:54 <kgriffs> well, looks like we are about out of time.
20:05:19 <flaper87> lets discuss about this in the next meeting
20:06:44 <kgriffs> final thing
20:07:41 <kgriffs> looking at the etherpad, which document type do you guys prefer as the body of the PUT?
20:07:56 <kgriffs> (not type, schema)
20:09:09 * flaper87 looking
20:09:13 <bryansd> I like the nested messages -> ttl
20:09:37 <bryansd> That way if we have more messages-related metadata we throw it in that collection
20:09:49 <bryansd> And we could make another similar section for claims if need be
20:10:25 <flaper87> yeah, I guess that's kind of how metadata will be stored as well
20:10:48 <flaper87> something like "queues have a message field"
20:10:52 <flaper87> messages*
20:11:02 <flaper87> where all messages configs are stored
20:11:06 <kgriffs> OK, well, sleep on it and let me know in openstack-marconi
20:11:12 * kgriffs needs to run
20:11:15 <flaper87> kk
20:11:22 <kgriffs> thanks everyone
20:11:27 <bryansd> thanks!
20:11:34 * flaper87 thanks everyone
20:11:49 <kgriffs> #endmeeting