16:10:52 #startmeeting marconi-api-http 16:10:53 Meeting started Thu Oct 31 16:10:52 2013 UTC and is due to finish in 60 minutes. The chair is kgriffs. Information about MeetBot at http://wiki.debian.org/MeetBot. 16:10:54 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 16:10:56 The meeting name has been set to 'marconi_api_http' 16:10:58 #link https://wiki.openstack.org/wiki/Marconi/specs/api/next 16:11:28 #topic Return an href for deleting all claimed messages with a single call 16:12:00 Hmmm... 16:12:08 That's a strange kind of pattern. 16:12:13 It's almost like cancelling a claim. 16:12:18 somebody was asking about this. Currently I don't think the bulk delete takes a claim ID 16:12:22 Or grabbing a *right* to delete a set of messages. 16:12:36 I guess the idea is you would process N messages, then do the delete 16:12:40 but seems like an anti-pattern 16:12:47 ahh 16:13:13 I think that's an anti-pattern, given that it makes sense to grab one, delete one as it is completed. 16:13:15 Also 16:13:18 Crash conditions 16:13:26 If you processed 1-5 of 10, but then crashed... 16:13:33 How do others know you completed 1-5? 16:13:37 mmh, I'm not sure I like this idea 16:13:40 Anti-pattern rationale ^^ 16:13:53 I mean, why would you delete all claimed messages ? 16:13:53 since if the worker crashes in the middle, that is N-p work items that would have to be rolled back. 16:14:06 I'd rather release them if I don't want to work on them anymore 16:15:16 only reason to support this that I can see (and it is a thin reason at that) would be if you have tons of very tiny messages 16:15:28 like stats or something 16:15:43 but then, marconi is probably not the right tool for the job in that case? 16:16:07 i mean, the advantage there would be batching up the deletes so they happen faster 16:16:20 agreed 16:16:26 yeah. :/ 16:16:46 on the other-other hand, that could mostly be solved by the transport layer 16:17:03 zmq, spdy/http2.0, websockets 16:17:17 ok, so strike this one? 16:18:13 +1 16:18:20 for striking 16:18:26 striking +1 16:20:50 next ? 16:21:01 sorry, taking notes 16:21:23 ok, sorry, didn't mean to push! 16:21:26 #info do not support bulk delete of claimed messages 16:21:31 #topic Allow PUTing metadata when creating a queue 16:21:47 TBH, I can't remember where this came from 16:22:14 we used to do this, but changed 16:22:31 because we decided that mixing queue metadata and queue options in a single resource was ugly 16:22:38 I think the only advantage of that is to reduce the number of calls required to create a queue 16:22:40 so we wanted to make them their own resources 16:22:42 yup 16:22:49 flaper87: right 16:22:56 however, what you said was our last thought 16:23:05 making them independent resources 16:23:08 I suppose if we add, say, default message TTL we could have that directly on the queue resource 16:24:00 http put https://marconi.example.com/v2/queues default_ttl:=300 16:24:09 (or something like that) 16:25:02 you might also put queue flavor as well (choose a shard type) 16:25:16 anyway, I think we won't change this 16:25:26 I agree with you 16:25:27 but we might add default options and stuff 16:25:37 also, metadata != queue properties 16:25:42 something like default TTL - would that be a v1.1 or a v2 do you think? 16:25:47 flaper87: +1 16:25:53 so, which should we support during queue's creation? 16:25:57 that will confuse users 16:26:02 and amke the api inconsistent 16:26:09 so, totally with you there! Lets strike this one 16:26:15 ok. 16:26:22 strike + 1 16:26:57 I just added this one... 16:27:02 #topic Set queue options for defaults, such as message and claim TTL. 16:27:29 TBH, I'm not sure how useful that is, since the clients will just stick in whatever 16:27:41 only thing is I guess it would make the request body a tiny bit smaller 16:28:26 my only concern is that we'll have to have those properties available everytime a message gets in 16:28:46 yes, we can cache them, but it will still be a little bit of a perf hit 16:29:04 and it'll require even more resources to the final user 16:29:35 like 'take queue caching under consideration when you buy the RAM' 16:29:36 how about striking then, given clients can implement default behavior on their own 16:29:48 kgriffs: +1 to that 16:29:50 flaper87: true 16:29:55 kgriffs: +1 16:30:29 ok, let me make notes 16:30:56 #topic homedoc should return relative URIs or encode version in href-template 16:31:03 see also: https://bugs.launchpad.net/marconi/+bug/1245656 16:31:17 so, I view this as a typo in the spec and a bug in the implementation 16:31:59 pedants will insist the v1 API is frozen, but I think we should fix this 16:32:10 +1 for relative urls 16:32:29 and +1 for encoded version of href-template 16:32:33 and +1 for fixing this 16:33:09 ok. Any client that is trying to "follow it's nose", using hrefs from the home doc, is broken now anyway 16:33:09 I'm in favor of getting this fixed sooner rather than later. 16:33:23 Since the longer we leave it a broken state, the longer we'll encourage "bad" behavior. 16:33:50 flaper87: fwiw, I discovered that "relative URI" is a misleading term 16:34:00 kgriffs: what's the right one? 16:34:07 * flaper87 is about to learn something new 16:34:16 a "relative URI" as described by RFC really means "partial URI" 16:34:48 using my term there, a "partial URI" can have either an *absolute* and *relative* path 16:34:48 alcabrera: kgriffs in addition, I need this to work to implement that client :D 16:34:55 s/and/or 16:35:06 so both of these are "partial" URIs: 16:35:17 /queues/foo 16:35:19 flaper87: yeah, keep the client coming! ;) 16:35:22 queues/foo 16:35:43 formally defined and "relative URIs" 16:36:02 kgriffs: mmh, interesting! I ignored that. 16:36:03 s/and/as 16:36:36 so, when you see someone say "relative URI" it may or may not mean the *path* in the URI is relative! 16:36:44 Just read: "so, I view this as a typo in the spec and a bug in the implementation" 16:36:51 kgriffs should be a politician 16:36:56 #link http://www.ietf.org/rfc/rfc1808.txt 16:37:05 ametts: LOL 16:37:06 Now we can change the API any time we want! :) 16:37:35 ok so, any objections to fixing this in v1.0 ? 16:37:51 the only danger is someone may have written a client that works around the bug 16:37:56 and we will break their workaround 16:38:00 none from me 16:38:01 Not from me. I think sooner than later is better too. 16:38:03 kgriffs: we really need this fixed in v1.0 :) 16:38:24 any volunteers? 16:38:27 Now I'm doubly-sure. We don't want to incur the wrath of Malini. 16:38:28 v1.0 16:38:31 #link https://bugs.launchpad.net/marconi/+bug/1245656 16:38:42 fvollero: cpallares ^ ? 16:38:54 maybe you guys wanna take that one 16:39:08 cpallares: you're already working on s/queues:// bug, aren't you? 16:39:18 yes 16:39:27 #info fix href-template paths in home doc in v1.0; don't wait for 1.1 16:39:45 I think fvollero is not around but he was looking for a bug to fix 16:39:55 I'll ping him off-line and see if he wants to take it 16:40:02 otherwise, I guess I could take it 16:40:10 if he doesn't , I could! 16:40:15 cpallares: cool 16:40:17 It's just fixing the path right? 16:40:18 +1 16:40:31 ok, whoever it is just assign yourself 16:40:38 I will update the spec 16:40:54 should be easy; just remove the '/' prefix 16:41:17 oh, and write a test to ensure it can be combined 16:41:19 flaper87: i'm here 16:41:20 cpallares: yeah, you also have to build a car, an airplane and elaborate a new theory about the milky way 16:41:21 let me note that 16:41:38 flaper87: haha don't mock me, for a beginner it sometimes feel like that :P 16:41:55 cpallares: hihihi! 16:41:59 kgriffs: what abt /v1 in some of the href-templates? 16:42:08 fvollero: so, I remember you're looking for a bug to fix 16:42:18 there's one that wouldn't take much of your time 16:42:18 flaper87: yep 16:42:24 fvollero: wanna squash it ? 16:42:30 kgriffs: like this one '"http://docs.openstack-marconi.org/rel/claim": { 16:42:30 "href-template": "/v1/queues/{queue_name}/claims{?limit}", 16:42:30 ' 16:42:31 flaper87: ok.. .i'll give a try :) 16:42:43 fvollero: cool, assign yourself there! 16:42:51 and many thanks! 16:42:52 flaper87: i was backporting some stuff to puppet-ceilo 16:42:58 flaper87: let me finish it first 16:43:06 fvollero: yeah sure, no hurries 16:43:10 fvollero: thanks man! 16:43:21 kgriffs: Arrrgh :) I didn't look at it yet :) 16:43:50 fvollero: ametts will buy you some beers 16:43:55 fvollero: he told me that! 16:44:06 ok, moving on 16:44:08 flaper87: lol ! Fair enough! ametts thanks a bunch :) 16:44:17 #topic 204 vs. 200 + empty array 16:44:35 I'll swap beers for code any day 16:44:36 kgriffs: link for rhia? 16:44:49 link for the bug? 16:44:52 yeah 16:44:54 https://bugs.launchpad.net/marconi/+bug/1245656 16:44:55 kgriffs: stupid keyboard 16:45:10 * flaper87 is leaning towards 200 + empty list here 16:45:12 kgriffs: I was referring to 204 vs 200 + [] 16:45:21 #action fvollero to take care of bug #1245656 16:45:24 flaper87: well, depend 16:45:27 fvollero: oic 16:45:46 * ametts likes the empty list too. He even saw this technique in an OReilly book recently. 16:45:49 i don't have a link to a bug or anything, we are just walking through a list of feedback 16:45:54 ok 16:46:01 I'm going to step out of this meeting to get heads down on the remaining sharding unit test failures. 16:46:12 kgriffs: for what we're using 204 usually ? 16:46:21 Go alcabrera, go! 16:46:21 so, while 204 isn't necessarily incorrect, and saves a few bytes on the wire, it is inconsistent with other OpenStack APIs 16:46:37 and *may* make client implementations a tad bit easier 16:46:55 fvollero: if there are no messages, we return 204 currently 16:46:55 kgriffs: damn, you're faster than me 16:47:03 (Rather than an empty JSON array) 16:47:05 I was about to write the client thing 16:47:34 the other thing I like about the empty list is that it's more explicit 16:47:47 you see that and you know there are no messages 16:47:53 no need to check status code 16:47:57 etc 16:48:01 flaper87: agreed 16:48:08 kgriffs: gotcha :) 16:48:09 #link https://wiki.openstack.org/wiki/Marconi/specs/api/v1#List_Messages 16:48:15 current behavior ^^ 16:48:41 flaper87: usually no one should look at the API response :) 16:49:02 Zhihao Yuan proposed a change to openstack/marconi: feat(shard): queue listing from multiple sources https://review.openstack.org/53127 16:49:04 ok, any objections to doing this in v1.1 ? 16:49:11 a trivial rebase 16:49:17 fvollero: ish, it depends on what the user is doing 16:49:21 but I agree 16:49:29 kgriffs: no objections from me 16:49:33 i did not follow 16:49:42 what's the resolution 16:50:15 This will break existing clients. Will there be too much legacy code by the time we get 1.1. out? 16:50:41 zyuan: by "doing this" I mean switching to returning an empty array of messages and queues rather than 204 16:51:01 Or I guess that's what API versioning is for. 16:51:04 right 16:51:14 v1 will still be available 16:51:25 Okay. I'm good, then. 16:51:29 v1.0, I mean 16:51:29 i have no strong objection, but i don't see how can we benefit from this also 16:51:36 should we also do this when claiming messages? 16:51:45 if no messages to claim, return empty array? 16:51:52 it seems to be easier to use, but affect user's bandwidth 16:52:57 yeah, small tread-off betwen UX and bandwith, I guess 16:52:58 zyuan: well, user bandwidth in the order of 4 bytes 16:53:02 zyuan: true, but we can mitigate that somewhat by using msgpack media type 16:53:18 ok, let's do it 16:53:21 it'll give you more consistency, better API and easier integration 16:53:28 fvollero: more than that 16:53:33 I think we have rough consensus 16:53:38 fvollero: you need hrefs as well 16:53:49 (consensus on doing this in marconi) 16:53:50 otherwise it's even more incovient 16:54:07 kgriffs: +1 from me 16:54:58 #info return an empty array on listing options when no items are available, plan for v1.1 16:55:17 #topic Consistency around response envelope for /messages vs /messages?ids 16:56:07 this would simplify client implementations 16:56:13 and conceptually, I think it makes sense 16:56:31 because both operations operate on the same resource, so you would expect the same representation to come back in each case IMO 16:56:34 i see no need so "consistency" 16:56:38 for* 16:56:48 because they are jsut different 16:56:55 and we showed how they are different 16:58:07 to make them look furtherly different, we can change endpoints name. but not response. 16:58:36 I have a counterpoint to that argument 16:59:06 to support API extensions, we will need to wrap those json arrays in objects 16:59:25 i don't see a need for API extension on message bulk get 16:59:26 so you can have {"EXT-foo": {...}, "messages": []} 16:59:46 this is just an endpoint for... restful, only 16:59:59 zyuan: the point of extensions is to allow 3rd-parties to customize the API with things we didn't think about or feel is outside the scope of the core project 17:00:17 kgriffs: 3rd can already do that on message listing 17:00:32 i'm jsut saying there is no need for message bulk geting 17:00:33 my point is, if we do that, then it makes it easy to make the two response schemas consistent 17:00:34 the ids one 17:00:40 they are different 17:00:46 since all you need is a "links" with and empty list 17:00:50 so there is no need for "consistency" 17:01:07 zyuan: could you ellaborate on why they are different? 17:01:19 elaborate* 17:01:24 one is listing, one is get multiple 17:01:52 ?ids is not an API filtering the first one 17:02:03 it's just another completely different API 17:02:12 and pretty much useless in marconi picture 17:02:39 for the most of time you want listing, without ?ids 17:02:42 thanks, we're on the same page there 17:03:17 they unfortuantely overloaded on names 17:03:26 they're conceptually different, however, I can see some value in making the response consistent, although there's no need for consistency. 17:03:28 but we just don't have an HTTP method call LIST 17:04:00 just think this: 17:04:10 but I'm leaning towards to keeping them as they are 17:04:13 which "links" you want to put in ?ids response? 17:04:20 i can not think of any 17:04:40 next? prev? lol 17:05:21 yeah, I agree with you there 17:05:29 but we can at least have { messages:{}} 17:05:38 but we can at least have { messages:[]} 17:06:06 but that won't make much sense, I guess. 17:06:12 * flaper87 is thinking outloud 17:06:19 then that furtherly confuse people 17:06:30 you see, even we defined their semnatics 17:06:42 people come to us and say "why they are inconsistent" 17:06:49 and if you make them even more similiar 17:06:57 hehe 17:07:14 heh 17:07:14 "why ids does not accept limit=?" 17:07:37 the best way to make different things different is just to name them differently 17:07:52 otherwise, i'm ok with removing ?ids GET 17:08:34 if we ever get there, that'd be v2 material 17:08:39 removing ?ids, that is! 17:08:55 anyway, I think we shouldn't make these 2 responses consistent 17:09:00 kgriffs: thoughts ? 17:10:13 Alejandro Cabrera proposed a change to openstack/marconi: feat: connect sharding manager to control drivers https://review.openstack.org/54605 17:10:38 kgriffs, ametts, flaper87: There it is - shahrding manager core. ^^ 17:10:46 * alcabrera catches up to everything 17:10:56 alcabrera: wb, thanks! 17:12:23 phew, lots of consistency discussions. 17:12:38 * alcabrera waits for jenkins to sing 17:13:32 oops, I forgot to wrap a test in 'requires_mongodb' 17:21:43 flaper87: hmmm, I still think that querying the same resource should yield the same basic schema 17:21:49 even if it is just { 17:21:54 "messages": [] } 17:21:58 instead of just [] 17:22:23 having a "links": [] may no be needed 17:22:38 so my suggestion is to split them into two resourses 17:22:41 if we want to change the operation syntax so it is it's own resource, then they can be different 17:22:58 list move listing messages to /messages_list 17:23:05 like* 17:23:15 or remove ?ids 17:23:22 or move ?ids to another endpoint 17:23:38 or invent a concept of "message group" 17:24:00 lots of options; let's just plan on creating a new resource in v2 17:24:03 mmh, in any case, this seem to be a v2 change, we're changing completely the response of one of our calls. 17:24:11 kgriffs: T_T 17:24:18 LOL 17:24:58 flaper87: I was just about to ask you if you agreed! 17:25:08 :P 17:25:13 that's my answer 17:25:16 :D 17:25:16 OK, let's try to plow through a few more real quick 17:25:39 alcabrera is out tomorrow, so I wanted to get as far as we can today 17:25:47 #topic Response envelopes in general - don't return raw arrays (to allow for extensions) 17:26:01 I alluded to this earlier 17:26:28 when we just return [] there is no where to put a non-breaking extension document 17:26:44 so, we should be returning {"things":[]} instead 17:27:11 (or so the suggestion has been given) 17:27:53 alcabrera: ping 17:28:39 kgriffs: pong 17:28:51 * flaper87 will be out as well 17:28:54 thoughts: ^^ 17:28:55 it's holiday here 17:28:55 Alejandro Cabrera proposed a change to openstack/marconi: feat: connect sharding manager to control drivers https://review.openstack.org/54605 17:29:15 kgriffs: +1 for the [] -> {} 17:29:28 I agree with the extensibility argument 17:29:44 * flaper87 agress as well 17:29:53 ok 17:29:55 moving on 17:30:07 wait. 17:30:55 agrees 17:30:57 * 17:31:03 #info encapsulate arrays in objects to make resource representations extensible 17:31:25 #topic Return full URIs in location/content-location headers (rather than relative ones) 17:31:55 this tripped up Repose. I don't know if any other reverse proxies have a similar problem 17:31:56 hmmm 17:31:58 I don't have a strong opinion here and TBH, not sure if I know which one is correct 17:32:06 I'm with flaper87 17:32:27 ok... only downside is slightly larger responses 17:32:40 what's the overall preference there? 17:32:47 I mean, what do people do normally? 17:32:53 good question 17:33:30 partial URIs are being standardized in the latest HTTP 1.1 RFC draft 17:33:56 the goal of that effort is to formalize stuff that is already pretty common, iirc 17:34:06 I wonder what HTTP 2.0 will do in this case... 17:34:06 I know a lot of web servers handle them fine 17:34:33 other openstack projects return full URIs afaik 17:35:04 (Jenkins: Patch Set 7: Works for me) - re: sharding manager (w0000t) 17:35:05 if latest HTTP RFC uses partial URIs, I'd say we should use them and let not-standarized services burn in hell! 17:35:08 muahahahahahahahhaha 17:35:11 * alcabrera returns to topic 17:35:15 LOL 17:35:23 lol 17:35:49 client URI/HTTP libraries handle them fine, don't they? 17:36:23 * flaper87 calls python-request's phone number and asks that 17:36:40 not sure if it speaks english, though. :D 17:36:51 * flaper87 should have a webcam in his place 17:36:52 i mean, if I give you a "base" URI and a "relative" AKA "partial" URI, the library should just do the right thing 17:37:08 kgriffs: yeah, correct 17:37:23 I know python-request sticks to the RFC 17:37:23 I think by forcing absolute URIs, we impose a subtle constraint/difficulty. 17:37:32 That being 17:37:40 If someone wants to implement a client-side proxy 17:37:43 I'm not sure about this specific case, though! 17:37:47 They'd have to manually split content location each time. 17:37:56 Whereas if we stick to relative/partial 17:38:04 TBH, I don't know why Repose is trying to parse those headers in the first place 17:38:05 but hey, if that's what the standard says, I think we should stick to that 17:38:09 The client-side proxy can just use base + partial. 17:38:11 (rather than just pass-through) 17:38:31 * flaper87 votes for partial URIs 17:39:40 +1 for partials 17:40:00 ok, cool 17:40:15 I think repose has a bug or something to fix this anyway 17:40:40 and other proxies don't care AFAIK 17:40:51 Just a few more! 17:41:02 #info Continue to return partial URIs 17:41:03 kk 17:41:08 #topic Is content-location header really necessary? 17:41:17 I did some reading about this 17:41:58 it is really only helpful if you have alternate representations of a resource that can be accessed using a different path vs. specifying the media type in Accept 17:42:15 example: 17:42:49 client requests /queues with Accept: application/json 17:42:58 then the server could respond with 17:43:30 content-location: /queues.json 17:43:49 Personally, I think the whole blah.json thing is silly 17:44:04 so, I don't think we need to be setting this header 17:44:05 I don;t like that X.format, either. :x 17:45:02 ok, how about no longer setting that header in v1.1 17:45:10 save a few bytes, and it isn't useful 17:45:21 yeah 17:45:24 +1 17:45:27 +1 17:45:40 #topic Polling model is based on the assumption of a "streaming" interaction model. Is there a place for a "point in time" listing model, where you don't always get a "next" href? 17:45:52 This a big one, and would be for v2.0 anyway 17:45:58 so I'd like to discuss it later 17:46:31 #info revisit for v2 API 17:46:44 yeah and we'll also have a notification session during the design summit 17:46:53 I'd discuss this after the summit 17:47:24 +1 for postponed discussion. I'd say we need for thoughts on that. 17:47:27 also - v2.0 17:47:30 It can wait. :DF 17:47:32 :D 17:47:50 #topic Query parameters are usually considered to be "optional", and yet DELETE queues/my-queue/messages requires an "ids" parameter. Some options: 17:48:01 So, I think we can revisit this in v2 as well 17:48:25 hehe 17:48:34 "consistency" problem again 17:48:40 right 17:48:47 if user don't understand overloading 17:49:02 then we disambigulate them 17:49:07 right 17:49:14 I made a note to that affect (one of the options) 17:49:15 that's *the* way to solve problem like this 17:49:35 ok, that's it! 17:49:42 thanks for hanging in there everyone! 17:49:48 #endmeeting