19:01:50 <kgriffs> #startmeeting marconi
19:01:51 <openstack> Meeting started Thu Feb  7 19:01:50 2013 UTC.  The chair is kgriffs. Information about MeetBot at http://wiki.debian.org/MeetBot.
19:01:52 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
19:01:54 <openstack> The meeting name has been set to 'marconi'
19:02:12 <kgriffs> #topic API
19:03:29 <kgriffs> So, as I mentioned in the meeting agenda email, I spent a lot of time cleaning up and simplifying the API spec based on feedback from the community as well as from some Rackspace folks.
19:03:50 <kgriffs> It's by no means final, but I wanted to get everyone's feedback on the draft as it now stands.
19:04:02 <kgriffs> http://wiki.openstack.org/marconi/specs/api/v1
19:04:34 <kgriffs> Anyone have something specific they want to bring up?
19:04:46 <treeder> reading...
19:04:54 <flaper87> o/
19:05:07 <flaper87> so, the api looks good so far
19:06:11 <flaper87> what I was wondering is whether we're planning to support just http requests or also add support for other channels like zmq (for example)? (might be a bit too early to talk about this but it would help us to define the api in a more consistent way)
19:06:12 <wkharold> agree, might help if resources were identified explicitly
19:06:29 <flaper87> I was wondering that because other channels might be faster for some cases
19:06:39 <flaper87> and would help to improve performance
19:08:11 <kgriffs> There's been a little bit of talk about doing web sockets or something, but I think it makes sense to consider the broader question of other protocols.
19:08:45 <kgriffs> Scaling out can get tricky with persistent connections, but it can be done.
19:10:20 <flaper87> kgriffs: indeed it can but, considering that we're defining it right now, it might be worth it to design it more pluggable from the beggining. I've no objections so far and I think it can be "translated" to something else easily
19:10:42 <treeder> kgriffs: should we go through the spec function by function?
19:10:48 <flaper87> I guess what I'm trying to say is that I agree with the api so far but I would definitely create a more pluggable architecture
19:11:55 <kgriffs> Agreed. In fact, I've alluded to that in the spec (scroll to the bottom):
19:11:56 <kgriffs> http://wiki.openstack.org/marconi/specs/grizzly
19:12:01 <treeder> flaper87: are you suggesting to not use http?
19:12:26 <kgriffs> However, I hadn't considered making the protocol itself pluggable. That's something that could be done without too much trouble.
19:12:40 <flaper87> treeder: not at all, I'm suggesting to use http and other channels
19:12:45 <flaper87> kgriffs: agreed!
19:14:19 <kgriffs> #action kgriffs to update spec/architecture plan to allow more than just HTTP
19:14:37 <flaper87> kgriffs: feel free to add me in that action as well
19:14:58 <kgriffs> Cool, thx.
19:15:38 <kgriffs> #action flaper87 to update spec/architecture plan to accommodate persistent transports such as ZMQ
19:15:54 <kgriffs> OK, getting back to the API
19:16:22 <kgriffs> #topic API - Media Types
19:16:36 <treeder> be nice if we could comment right in the wiki
19:16:45 <kgriffs> yeah
19:17:07 <kgriffs> I could try to paste it into etherpad
19:17:17 <flaper87> kgriffs: +1
19:18:26 <kgriffs> https://etherpad.openstack.org/queuing-api
19:18:57 <kgriffs> So, the eternal debate: XML, JSON
19:19:10 <kgriffs> Does 1.0 need to support XML?
19:19:20 * flaper87 loves JSON and hates xml
19:19:39 <treeder> json +10
19:19:53 <wkharold> flaper87: +1
19:20:29 * kgriffs senses mild enthusiasm for JSON
19:20:38 <treeder> kgriffs: would there be any particular reason to support it?
19:20:42 <flaper87> I would say we should got step-by-step and add first one of those (and by one of those I mean JSON) and think about creating a more consistent api
19:20:58 <flaper87> then we would be able to add support for other stuff like xml, msgpack or whatsoever
19:21:10 <kgriffs> Agreed.
19:21:21 <treeder> agreed
19:21:47 <wkharold> #agreed
19:22:12 <kgriffs> There was a push at the last summit to get better XML support (some projects don't offer it at all, or are buggy). However, the argument was that enterprises have these expensive appliances they've invested in that only do XML.
19:22:41 <flaper87> kgriffs: makes sense
19:22:44 <kgriffs> That being said, there's the argument for using a translator that takes in JSON and spits out XML.
19:23:11 <kgriffs> (ala Atom Nuke)
19:23:33 <kgriffs> Of course, I think it's OK to not target large enterprise with 1.0
19:23:42 <flaper87> kgriffs: hehe
19:24:08 <flaper87> kgriffs: translators could make sense, even though I'm afraid there'll be a real pain in the ***
19:24:18 <flaper87> anywho
19:24:20 <kgriffs> OK
19:24:33 <kgriffs> #agreed Start with JSON, add other media types later
19:25:30 <kgriffs> So, if you guys want to comment directly in etherpad and I can bubble those up as discussion topics, we can keep moving.
19:25:40 <treeder> ok
19:25:41 <kgriffs> (or just post on IRC)
19:25:45 <treeder> i'm going to go through each function
19:25:49 <cppcabrera> (I'm taking notes of the meeting. I can distribute them afterwards to those interested.)
19:25:50 <kgriffs> Sounds good
19:25:51 <treeder> i think that'll be most productive
19:26:05 <treeder> Create a qudeue
19:26:06 <treeder> queue
19:26:11 <treeder> I think that looks fine as is
19:26:20 <kgriffs> cppcabrera: Thx. meetbot does that too, but it would be good to compare notes.
19:26:33 <flaper87> +1
19:26:43 <treeder> cool
19:26:51 <treeder> Alright, Reconfigure a queue
19:27:02 <kgriffs> OK. I didn't have any other config other than TTL at the moment.
19:27:03 <treeder> I think this could just be changed to the parameters themselves
19:27:33 <treeder> instead of having ops, how about just if "ttl" is specified, then it changes the ttl?
19:27:47 <treeder> same as creating a queue
19:28:21 <kgriffs> If/when more options are available, I think we want standard JSON-Patch support. However, we can just allow all-or-nothing to start with.
19:28:33 <kgriffs> "op" is from the JSON Patch RFC
19:28:40 <treeder> oh
19:28:53 <treeder> does that mean one op per request though?
19:29:12 <kgriffs> no, I messed that up. It's supposed to be inside an array
19:30:22 <treeder> http://tools.ietf.org/html/draft-pbryan-json-patch-04
19:30:41 <kgriffs> yep. It's actually on revision 10 now
19:30:42 <kgriffs> https://tools.ietf.org/html/draft-ietf-appsawg-json-patch-10
19:31:16 <wkharold> is PATCH supported in the relevant HTTP servers?
19:31:21 <kgriffs> There was some talk about doing xml-patch as well, but not sure where that ended up
19:31:35 <kgriffs> Do you have one in mind?
19:31:44 <kgriffs> (server)
19:32:03 <treeder> probably the bigger problem is client libs supporting it
19:32:08 <wkharold> The usual suspects, Apache, etc
19:32:12 <kgriffs> oic
19:32:23 <kgriffs> Just wanted to make sure you weren't talking about IIS. :p
19:32:31 <flaper87> hahahaha
19:32:32 <wkharold> blech
19:32:36 <treeder> hehe
19:32:54 * flaper87 takes his shotgun
19:32:57 <kgriffs> Does anyone know about Apache and Nginx off the top of their heads.
19:33:17 <flaper87> cherokee ?
19:33:48 <wkharold> nope, hence the query
19:34:04 <kgriffs> Anyone want to volunteer to survey PATCH support?
19:34:33 <wkharold> I guess I should step up
19:34:39 <kgriffs> cool.
19:34:49 <flaper87> wkharold: thx
19:35:15 <treeder> wkharold: bet you wish you didn't ask that question huh?  ;)
19:35:20 <kgriffs> #action wkharold to check PATCH support in Apache, Nginx, Cherokee, etc.
19:35:33 <wkharold> someday I'll learn
19:36:04 <treeder> regarding json patch vs just a straight json object (same as PUT), i'd vote for straight up json object.
19:36:06 <treeder> much simpler
19:36:08 <kgriffs> wkharold: if you could send out an email to openstack-dev [marconi] with the results, or share in the next meeting that would be groovy.
19:36:38 <flaper87> treeder: +1
19:36:41 <kgriffs> Agreed, as long as the document isn't complex
19:36:49 <wkharold> treeder: modulo the size of the json
19:36:56 <treeder> i don't think we'll have any complex documents in this spec
19:37:02 <flaper87> that would make everything more consistent
19:37:06 <cppcabrera> kgriffs: Does the marconi project have a Trello board or something similar yet? This thought struck me as action items are being detailed.
19:37:18 <kgriffs> #agreed for simple document updates, don't use PATCH
19:37:26 * flaper87 is a big fan of consistency
19:37:49 <kgriffs> cppcabrera: Not a public one, but that can be easily remedied.
19:37:59 * treeder is a big fan of consistency too
19:38:08 <treeder> trello board: +1
19:38:55 <flaper87> trello board ++
19:39:21 <kgriffs> #action cppcabrera to hook us up with a public trello board
19:39:33 <kgriffs> Looks like we have a volunteer. :D
19:39:42 <cppcabrera> Will do. I'll start adding action items to it during the course of the meeting. :)
19:39:46 <treeder> awesome
19:40:16 <kgriffs> cppcabrera: At the end of the meeting, meetbot will summarize actions, but it would be good to watch for anything we miss
19:40:37 <cppcabrera> done - https://trello.com/board/openstack-marconi/511403287d138cd6200078e0
19:40:46 <treeder> alright, done with patching a queue then?
19:40:55 <kgriffs> #topic API - Delete a Queue
19:41:25 <treeder> cppcabrera: can you us to the board? i'm treeder
19:41:44 <treeder> thanks
19:41:51 <cppcabrera> treeder: np
19:42:05 <treeder> delete looks good to me
19:42:31 <flaper87> cppcabrera: flaper87
19:42:33 <flaper87> :)
19:42:40 <ametts-atl> cppcabrera: allanmetts
19:42:42 <flaper87> looks good to me as well
19:42:54 <treeder> kgriffs: notice you changed reconfigure to a PUT
19:43:05 <treeder> I think the http op should still be PATCH
19:43:08 <kgriffs> #agreed Delete a Queue is OK as-is
19:43:17 <kgriffs> mmm. Not sure about that.
19:43:31 <treeder> and here's example of why
19:43:42 <treeder> let's assume there's more than one parameter
19:43:44 <treeder> ttl and x
19:43:57 <treeder> say I just want to change the ttl
19:44:00 <treeder> then I just send in the ttl
19:44:05 <treeder> { "ttl": x}
19:44:31 <treeder> if i want to change both: {"ttl":y, "x": z}
19:44:45 <kgriffs> Well, in that case I'd rather use json-patch since that's on the standards track.
19:45:08 <kgriffs> But if there are only two attributes, it seems fine to require setting the entire thing
19:45:08 <treeder> you think that'll become a used standard?
19:45:12 <kgriffs> yes
19:45:15 <flaper87> yep
19:45:37 <treeder> rough
19:46:13 <treeder> well maybe we should use it then
19:46:22 <kgriffs> I know that there's a de-factor standard out there today to do what you describe, but eventually things will move to JSON-patch
19:46:32 <flaper87> wait
19:46:36 <kgriffs> IMHO
19:46:38 * flaper87 thinking
19:47:04 <kgriffs> Although, either way it's really only a few lines of code to implement.
19:47:40 <kgriffs> #topic API - json-patch vs. implicit patch
19:47:41 <flaper87> PATCH http method should be used for modifine *some* arguments of the document
19:48:00 <flaper87> and PUT for the whole document
19:48:03 <kgriffs> Correct. If you want to modify all of them, then it doesn't make sense.
19:48:26 <kgriffs> That's what I was saying about not needing it if there's only "ttl", for example.
19:48:37 <kgriffs> But we could always add support l8r
19:48:52 <wkharold> usually PUT updates by replacing the whole representation, even if only one thing changes
19:49:06 <kgriffs> good point. And you aren't really recreating the queue
19:49:13 <treeder> kgriffs: ya, there's high probability that more features will be added
19:49:40 <kgriffs> it's a fuzzy line. What's everyone's vote?
19:49:43 <treeder> and we'll want to keep this consistent across entire api
19:50:09 <kgriffs> yes, and currently claiming a message is done with PATCH
19:50:21 <flaper87> let's vote
19:50:22 <treeder> I vote PATCH, but not sure about the json-patch
19:50:34 <kgriffs> noted
19:50:39 <flaper87> options json-patch or straight json
19:51:09 <wkharold> straight json, PUT complete representation
19:51:13 <cppcabrera> +1 json-patch
19:51:16 <treeder> straight json
19:51:20 <flaper87> kgriffs: bot should handle votes
19:51:32 <flaper87> so it is logged in the meeting report
19:51:34 <kgriffs> oic
19:52:24 <kgriffs> hmmm. Is there a hashtag for that?
19:52:28 <kgriffs> I'm not seeing it
19:52:31 <kgriffs> http://meetbot.debian.net/Manual.html
19:52:39 <flaper87> #startvote JSON-PATCH? Yes, No
19:52:40 <openstack> Only the meeting chair may start a vote.
19:52:50 <flaper87> kgriffs: :)
19:53:07 * flaper87 shoots the stupid bot
19:53:07 <kgriffs> cool
19:53:23 <kgriffs> Ok, let's take it one at a time - first, PATCH vs. PUt
19:53:37 <kgriffs> #startvote PATCH vs. PUT? PATCH, PUT
19:53:38 <openstack> Begin voting on: PATCH vs. PUT? Valid vote options are PATCH, PUT.
19:53:39 <openstack> Vote using '#vote OPTION'. Only your last vote counts.
19:53:46 <treeder> #vote PATCH
19:53:48 <flaper87> #vote PATCH
19:53:49 <cppcabrera> #vote PATCH
19:53:52 <wkharold> #vote PUT
19:54:01 <kgriffs> #endvote
19:54:02 <openstack> Voted on "PATCH vs. PUT?" Results are
19:54:03 <openstack> PUT (1): wkharold
19:54:04 <openstack> PATCH (3): cppcabrera, treeder, flaper87
19:54:08 <flaper87> (perhaps we should have considered both?)
19:54:09 <flaper87> :P
19:54:15 <treeder> wkharold: we could have PUT too
19:54:26 <wkharold> np ... I'm a good loser
19:54:28 <treeder> ahh, ya, was just thinking the same flaper87
19:54:49 <kgriffs> #startvote Do PUT first, add PATCH later? Yes, No
19:54:50 <openstack> Begin voting on: Do PUT first, add PATCH later? Valid vote options are Yes, No.
19:54:51 <openstack> Vote using '#vote OPTION'. Only your last vote counts.
19:55:17 <wkharold> #vote Yes
19:55:23 <flaper87> #vote Yes
19:55:24 <treeder> my vote for that depends on what we decide on the next vote. ;)
19:55:29 <cppcabrera> #vote Yes
19:56:14 <kgriffs> #endvote
19:56:15 <openstack> Voted on "Do PUT first, add PATCH later?" Results are
19:56:16 <openstack> Yes (3): wkharold, cppcabrera, flaper87
19:56:20 <kgriffs> "The PUT method requests that the state of the target resource be
19:56:20 <kgriffs> created or replaced with the state defined by the representation
19:56:20 <kgriffs> enclosed in the request message payload."
19:56:26 * flaper87 likes IRC politics
19:56:26 <kgriffs> http://tools.ietf.org/html/draft-ietf-httpbis-p2-semantics-21#section-5.3.4
19:56:44 <treeder> alright, that makes it easy
19:56:47 <kgriffs> I'm fine with doing PUT
19:57:01 <cppcabrera> PUT is simpler to support than PATCH - makes sense to go after it first, IMO.
19:57:01 <treeder> until we get to something else that requires modifications
19:57:05 <kgriffs> RESTafarians may disagree, but it's close enough semantically
19:57:27 <treeder> can we skip to POST Message before GET?
19:57:30 <kgriffs> #agreed PUT first, PATCH later if needed
19:57:33 <flaper87> I stick with my vote, both should be supported. PUT first and then patch it if needed
19:57:46 <flaper87> w00t
19:57:48 <kgriffs> You read my mind. :D
19:58:01 <kgriffs> OK, we only have time for one more topic
19:58:15 <kgriffs> #topic API - POST message
19:58:17 <treeder> ok
19:58:51 <treeder> ok, one thing that isn't clear to me is what body format is
19:59:00 <wkharold> #info according to http://weblog.rubyonrails.org/2012/2/25/edge-rails-patch-is-the-new-primary-http-method-for-updates most servers grok PATCH
19:59:08 <treeder> looks like it's json in the spec
19:59:17 <kgriffs> wait a sec
19:59:28 <kgriffs> sorry, I think I forgot to hoist those up one level
19:59:35 <kgriffs> Let me try something in etherpad...
19:59:38 <treeder> I think it should be a string?
19:59:41 <treeder> ie: anything
19:59:50 <kgriffs> Oh, wait
20:00:40 <kgriffs> OK, let's talk about contents of body then we can jump back to whether "body" can be removed and the contents hoisted.
20:00:42 <flaper87> treeder: what do you mean? I'm missing something
20:00:52 <kgriffs> So, body can be anything JSON supports
20:01:01 <kgriffs> "body": "foo"
20:01:07 <kgriffs> Guess I should add more examples
20:01:48 <flaper87> kgriffs: mmh, not sure about that
20:01:53 <flaper87> so
20:02:47 <flaper87> Body content can be anything, agreed, but we should somehow "de-normalize" it to something less harmful
20:03:01 <wkharold> question, why POST to queue_name/messages why not just POST to queue_name?
20:03:03 <flaper87> for example, what happens if the content has non-ascii characters? or other characters?
20:03:29 <flaper87> what about having it serialized to b64 or something like that in order to *always* have a string
20:03:31 <flaper87> or whatever
20:03:32 <kgriffs> It would have to be valid JSON, and we could enforce UTF-8
20:03:33 <treeder> it should be a json safe string
20:03:39 <flaper87> celery does something similar
20:03:45 <kgriffs> I anticipate doing JSON validation
20:04:05 * flaper87 is way paranoid
20:04:08 <kgriffs> If a client submits something bogus, they get 400
20:04:28 <kgriffs> Storage backends can just store as an opaque string (no need to parse)
20:04:35 <cppcabrera> +1 for JSON validation
20:04:42 <flaper87> fair enough
20:04:49 <wkharold> ditto +1
20:04:59 <kgriffs> #agreed JSON validation required
20:05:14 <flaper87> and, we will support compressed messages?
20:05:23 <flaper87> maybe for the next meeting
20:05:23 <wkharold> still wondering about queue_name/messages ...
20:05:28 <kgriffs> If we ever add JSON-P support, we will have to get even more paranoid, like using the while(1) trick
20:05:46 * flaper87 shoots json-p in the legs
20:05:48 <treeder> seems to me that shoudl be encoded
20:05:49 <treeder> eg:
20:05:56 <wkharold> JSON-P old and broken, CORS new hotness
20:06:16 <kgriffs> #action kgriffs to Add more examples for body of messages
20:06:24 <treeder> "body": "{
20:06:24 <treeder> \"event\": \"BackupStarted\",
20:06:24 <treeder> \"backupId\": \"c378813c-3f0b-11e2-ad92-7823d2b0f3ce\"
20:06:24 <treeder> }"
20:06:45 <treeder> could be xml, could be anything
20:06:56 <kgriffs> Well, if clients want to do that, they are perfectly welcome.
20:07:15 <kgriffs> But I don't want to prevent people from just using straight JSON for readability or whatever
20:07:32 <wkharold> MIME?
20:07:58 <wkharold> for message body ...
20:09:20 <cppcabrera> 'application/json' makes sense for a json-encoded body.
20:09:21 <treeder> might need a mime type i guess if we support json structures
20:09:31 <kgriffs> Well, I don't think it's very hard to validate the body text and make sure there's nothing funky in there, and assuming you aren't constructing DB inserts by hand, I'm not worried about the security issue.
20:09:34 <treeder> json or text
20:09:45 <flaper87> I'd stick with application/json
20:10:25 <kgriffs> agreed
20:10:35 <flaper87> cool
20:10:40 <flaper87> gotta run
20:10:48 <flaper87> lets finish it :P
20:10:50 <kgriffs> If clients want to use something different, as long as they can UTF-8 encode into a string, go for it.
20:10:55 <kgriffs> OK
20:11:17 <treeder> guess it just makes more work on the backend
20:11:22 <kgriffs> Let me just touch real quick on why /messages
20:11:49 <wkharold> yes please
20:11:58 <kgriffs> Well, the backend needs to somehow validate whatever it's given anyway, IMHO, and re SQL injection or whatever, it can just be stored as an opaque blob.
20:12:39 <treeder> on the way back, it has to be marshalled back into json though
20:12:43 <kgriffs> The reason for the extra namespace is that it lets us add things like /actions and /stats
20:12:58 <kgriffs> Well, depends on how you are constructing the JSON
20:13:14 <treeder> i gotta run, good meeting guys.
20:13:15 <kgriffs> If you are using, e.g., dumps then yes, that's true
20:13:19 <kgriffs> k
20:13:23 <treeder> good progress!
20:13:25 <kgriffs> thx for the feedback
20:13:26 <cppcabrera> Take care, treeder.
20:13:30 <wkharold> OK ... if there are additional resources coming OK
20:13:36 <kgriffs> yeah
20:13:41 <kgriffs> it makes it consistent
20:14:13 <kgriffs> Once we take a full pass through the spec, we can loop back to the top and see how everything looks in context.
20:14:39 <flaper87> makes sense
20:14:41 <kgriffs> So, next week let's continue going down the list of each action in the spec
20:14:50 <flaper87> +1
20:14:54 <cppcabrera> +1
20:15:12 <ametts-atl> We switching back to weekly meetings?
20:15:16 <flaper87> great meeting guys
20:15:22 <flaper87> thx folks
20:15:25 <kgriffs> Related: Falcon is coming along nicely. Check it out and let me know what you think.
20:15:26 <kgriffs> https://github.com/racker/falcon
20:15:33 <kgriffs> Thanks everyone.
20:15:39 <cppcabrera> I'll keep the Trello up to date.
20:15:54 <kgriffs> #endmeeting