Thursday, 2013-10-31

*** amitgandhi has joined #openstack-marconi00:16
*** kgriffs is now known as kgriffs_afk00:19
*** vkmc has joined #openstack-marconi00:26
*** vkmc has quit IRC00:26
*** vkmc has joined #openstack-marconi00:26
*** fifieldt_ has joined #openstack-marconi00:35
*** amitgandhi has quit IRC00:41
*** amitgandhi has joined #openstack-marconi00:42
*** amitgandhi has quit IRC00:46
*** reed has quit IRC01:02
*** nosnos has joined #openstack-marconi01:05
*** vkmc has quit IRC01:17
*** vkmc has joined #openstack-marconi01:18
*** vkmc has quit IRC01:52
*** oz_akan_ has joined #openstack-marconi02:41
*** amitgandhi has joined #openstack-marconi02:42
*** amitgandhi has quit IRC02:42
*** amitgandhi has joined #openstack-marconi02:43
*** amitgandhi has quit IRC02:47
*** amitgandhi has joined #openstack-marconi03:43
*** amitgandhi has quit IRC03:47
*** amitgandhi has joined #openstack-marconi03:53
*** amitgandhi has quit IRC03:58
*** fifieldt_ has quit IRC04:02
*** fifieldt has joined #openstack-marconi04:02
*** oz_akan_ has quit IRC04:06
*** nosnos has quit IRC04:17
*** nosnos_ has joined #openstack-marconi04:17
*** nosnos_ has quit IRC04:22
*** nosnos has joined #openstack-marconi04:23
*** nosnos has quit IRC04:26
*** nosnos has joined #openstack-marconi04:29
*** nosnos_ has joined #openstack-marconi04:33
*** nosnos has quit IRC04:37
*** amitgandhi has joined #openstack-marconi04:43
*** amitgandhi has joined #openstack-marconi04:44
*** amitgandhi has quit IRC04:49
*** nosnos_ has quit IRC05:28
*** nosnos has joined #openstack-marconi05:28
*** amitgandhi has joined #openstack-marconi05:45
*** amitgandhi has quit IRC05:49
*** amitgandhi has joined #openstack-marconi05:55
*** amitgandhi has quit IRC05:59
*** nosnos_ has joined #openstack-marconi06:06
*** nosnos has quit IRC06:08
*** amitgandhi has joined #openstack-marconi06:45
*** amitgandhi has quit IRC06:45
*** amitgandhi has joined #openstack-marconi06:45
*** amitgandhi has quit IRC06:50
*** fifieldt has quit IRC07:16
*** nosnos_ has quit IRC07:22
*** nosnos has joined #openstack-marconi07:22
*** flaper87|afk is now known as flaper8708:03
flaper87al-maisan: ping08:04
*** amitgandhi has joined #openstack-marconi08:46
*** amitgandhi has joined #openstack-marconi08:47
*** amitgandhi has quit IRC08:51
*** yassine has joined #openstack-marconi09:11
openstackgerritFlavio Percoco proposed a change to openstack/marconi: Isolate tests a bit more
*** amitgandhi has joined #openstack-marconi09:47
*** amitgandhi has quit IRC09:51
*** amitgandhi has joined #openstack-marconi10:47
*** amitgandhi has quit IRC10:47
*** amitgandhi has joined #openstack-marconi10:48
al-maisanflaper87: pong?10:51
*** nosnos has quit IRC10:51
*** amitgandhi has quit IRC10:52
flaper87al-maisan: hey :D10:52
flaper87al-maisan: I pingged you before reading your email10:52
al-maisanhello flaper87 :)10:52
flaper87al-maisan: thanks for that!10:52
al-maisanhope it helps10:52
al-maisanalso, if you'd like to discuss it we can have a call10:53
*** malini_afk is now known as malini11:20
*** flaper87 is now known as flaper87|afk11:30
*** vkmc has joined #openstack-marconi11:30
*** vkmc has quit IRC11:30
*** vkmc has joined #openstack-marconi11:30
*** tedross has joined #openstack-marconi11:35
*** amitgandhi has joined #openstack-marconi11:58
*** amitgandhi has quit IRC12:03
*** flaper87|afk is now known as flaper8712:41
*** oz_akan_ has joined #openstack-marconi12:47
*** oz_akan_ has joined #openstack-marconi12:48
openstackgerritAlejandro Cabrera proposed a change to openstack/marconi: feat: integrate shard storage with transport
openstackgerritAlejandro Cabrera proposed a change to openstack/marconi: feat: integrate shard storage with transport
*** mpanetta has joined #openstack-marconi13:15
*** amitgandhi has joined #openstack-marconi13:18
*** mpanetta has quit IRC13:24
*** mpanetta has joined #openstack-marconi13:24
*** mpanetta has quit IRC13:25
*** mpanetta has joined #openstack-marconi13:26
*** mpanetta_ has joined #openstack-marconi13:29
*** mpanetta has quit IRC13:29
*** tedross has quit IRC13:32
*** tedross has joined #openstack-marconi13:48
*** ayoung_ is now known as ayoung13:49
*** kgriffs_afk is now known as kgriffs13:55
*** malini is now known as malini_afk13:57
*** jcru has joined #openstack-marconi14:04
*** jergerber has joined #openstack-marconi14:15
*** amitgandhi has quit IRC14:19
*** alcabrera has joined #openstack-marconi14:26
alcabreraGood morning! :D14:26
alcabrerakgriffs, flaper87: o/14:30
flaper87alcabrera: kgriffs good morning guys14:31
* flaper87 has been investigating things for next week sessions14:31
alcabrerasweet. How goes prep for the summit? :D14:33
alcabreraflaper87, kgriffs: I rebased the transport + storage sharding patch this morning and addressed all feedback. I couldn't get functional tests in at the moment, but I've noted we need them.14:34
kgriffskk. Changes look good.14:37
*** cpallares has joined #openstack-marconi14:37
kgriffsflaper87: can you take another look and see if everything you noted was addressed?14:38
flaper87kgriffs: on it! ;)14:39
flaper87alcabrera: could you please add a bug for the missing functional tests ?14:45
flaper87I'm fine with those landing later!14:45
alcabreraflaper87: cool! I'll do that now.14:47
* flaper87 +2'd that patch14:47
* flaper87 gives alcabrera many many pop-tarts14:47
flaper87lot of them14:48
flaper87like, tons of trucks full of them14:48
alcabreraflaper87: done
alcabrerayaaaay, pop tarts!14:48
* alcabrera eats one, shares the rest with #openstack-marconi14:49
*** amitgandhi has joined #openstack-marconi14:50
openstackgerritA change was merged to openstack/marconi: feat: integrate shard storage with transport
*** tedross has quit IRC14:59
flaper87alcabrera: great work there! Way to go!15:00
alcabreraflaper87: ¬°gracias!15:01
flaper87kgriffs: if you've a chance, i'd love to hear your thoughts about the remaining client patches15:01
flaper87man, i've 3 patches in workinprogress, what's worng with me?15:02
* flaper87 needs to get more things done15:02
alcabreraflaper87: and there's always more to be done! ;)15:04
*** amitgandhi has quit IRC15:06
*** amitgandhi has joined #openstack-marconi15:07
*** amitgandhi has quit IRC15:12
*** amitgandhi has joined #openstack-marconi15:13
*** tedross has joined #openstack-marconi15:16
*** malini_afk is now known as malini15:35
*** reed has joined #openstack-marconi15:51
kgriffsflaper87, alcabrera: have a few minutes to discuss more API feedback?16:09
alcabrerakgriffs: yup16:09
kgriffs#startmeeting marconi-api-http16:10
openstackMeeting started Thu Oct 31 16:10:52 2013 UTC and is due to finish in 60 minutes.  The chair is kgriffs. Information about MeetBot at
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.16:10
openstackThe meeting name has been set to 'marconi_api_http'16:10
kgriffs#topic Return an href for deleting all claimed messages with a single call16:11
alcabreraThat's a strange kind of pattern.16:12
alcabreraIt's almost like cancelling a claim.16:12
kgriffssomebody was asking about this. Currently I don't think the bulk delete takes a claim ID16:12
alcabreraOr grabbing a *right* to delete a set of messages.16:12
kgriffsI guess the idea is you would process N messages, then do the delete16:12
kgriffsbut seems like an anti-pattern16:12
alcabreraI think that's an anti-pattern, given that it makes sense to grab one, delete one as it is completed.16:13
alcabreraCrash conditions16:13
alcabreraIf you processed 1-5 of 10, but then crashed...16:13
alcabreraHow do others know you completed 1-5?16:13
flaper87mmh, I'm not sure I like this idea16:13
alcabreraAnti-pattern rationale ^^16:13
flaper87I mean, why would you delete all claimed messages ?16:13
kgriffssince if the worker crashes in the middle, that is N-p work items that would have to be rolled back.16:13
flaper87I'd rather release them if I don't want to work on them anymore16:14
kgriffsonly reason to support this that I can see (and it is a thin reason at that) would be if you have tons of very tiny messages16:15
kgriffslike stats or something16:15
kgriffsbut then, marconi is probably not the right tool for the job in that case?16:15
kgriffsi mean, the advantage there would be batching up the deletes so they happen faster16:16
alcabrerayeah. :/16:16
kgriffson the other-other hand, that could mostly be solved by the transport layer16:16
kgriffszmq, spdy/http2.0, websockets16:17
kgriffsok, so strike this one?16:17
alcabrerafor striking16:18
flaper87striking +116:18
flaper87next ?16:20
kgriffssorry, taking notes16:21
flaper87ok, sorry, didn't mean to push!16:21
kgriffs#info do not support bulk delete of claimed messages16:21
kgriffs#topic Allow PUTing metadata when creating a queue16:21
kgriffsTBH, I can't remember where this came from16:21
kgriffswe used to do this, but changed16:22
kgriffsbecause we decided that mixing queue metadata and queue options in a single resource was ugly16:22
flaper87I think the only advantage of that is to reduce the number of calls required to create a queue16:22
kgriffsso we wanted to make them their own resources16:22
kgriffsflaper87: right16:22
flaper87however, what you said was our last thought16:22
flaper87making them independent resources16:23
kgriffsI suppose if we add, say, default message TTL we could have that directly on the queue resource16:23
kgriffshttp put default_ttl:=30016:24
kgriffs(or something like that)16:24
kgriffsyou might also put queue flavor as well (choose a shard type)16:25
kgriffsanyway, I think we won't change this16:25
flaper87I agree with you16:25
kgriffsbut we might add default options and stuff16:25
flaper87also, metadata != queue properties16:25
kgriffssomething like default TTL - would that be a v1.1 or a v2 do you think?16:25
kgriffsflaper87: +116:25
flaper87so, which should we support during queue's creation?16:25
flaper87that will confuse users16:25
*** whenry has quit IRC16:25
flaper87and amke the api inconsistent16:26
flaper87so, totally with you there! Lets strike this one16:26
alcabrerastrike + 116:26
kgriffsI just added this one...16:26
kgriffs#topic  Set queue options for defaults, such as message and claim TTL.16:27
kgriffsTBH, I'm not sure how useful that is, since the clients will just stick in whatever16:27
kgriffsonly thing is I guess it would make the request body a tiny bit smaller16:27
*** whenry has joined #openstack-marconi16:27
flaper87my only concern is that we'll have to have those properties available everytime a message gets in16:28
kgriffsyes, we can cache them, but it will still be a little bit of a perf hit16:28
flaper87and it'll require even more resources to the final user16:29
flaper87like 'take queue caching under consideration when you buy the RAM'16:29
kgriffshow about striking then, given clients can implement default behavior on their own16:29
alcabrerakgriffs: +1 to that16:29
kgriffsflaper87: true16:29
flaper87kgriffs: +116:29
kgriffsok, let me make notes16:30
kgriffs#topic homedoc should return relative URIs or encode version in href-template16:30
kgriffssee also:
kgriffsso, I view this as a typo in the spec and a bug in the implementation16:31
kgriffspedants will insist the v1 API is frozen, but I think we should fix this16:31
flaper87+1 for relative urls16:32
flaper87and +1 for encoded version of href-template16:32
flaper87and +1 for fixing this16:32
kgriffsok. Any client that is trying to "follow it's nose", using hrefs from the home doc, is broken now anyway16:33
alcabreraI'm in favor of getting this fixed sooner rather than later.16:33
alcabreraSince the longer we leave it a broken state, the longer we'll encourage "bad" behavior.16:33
kgriffsflaper87: fwiw, I discovered that "relative URI" is a misleading term16:33
flaper87kgriffs: what's the right one?16:34
* flaper87 is about to learn something new16:34
kgriffsa "relative URI" as described by RFC really means "partial URI"16:34
kgriffsusing my term there, a "partial URI" can have either an *absolute* and *relative* path16:34
flaper87alcabrera: kgriffs in addition, I need this to work to implement that client :D16:34
kgriffsso both of these are "partial" URIs:16:35
kgriffs /queues/foo16:35
alcabreraflaper87: yeah, keep the client coming! ;)16:35
kgriffsformally defined and "relative URIs"16:35
flaper87kgriffs: mmh, interesting! I ignored that.16:36
kgriffsso, when you see someone say "relative URI" it may or may not mean the *path* in the URI is relative!16:36
amettsJust read:  "so, I view this as a typo in the spec and a bug in the implementation"16:36
amettskgriffs should be a politician16:36
kgriffsametts: LOL16:37
amettsNow we can change the API any time we want! :)16:37
kgriffsok so, any objections to fixing this in v1.0 ?16:37
kgriffsthe only danger is someone may have written a client that works around the bug16:37
kgriffsand we will break their workaround16:37
alcabreranone from me16:38
amettsNot from me.  I think sooner than later is better too.16:38
malinikgriffs: we really need this fixed in v1.0 :)16:38
kgriffsany volunteers?16:38
amettsNow I'm doubly-sure.  We don't want to incur the wrath of Malini.16:38
flaper87fvollero: cpallares ^ ?16:38
flaper87maybe you guys wanna take that one16:38
flaper87cpallares: you're already working on s/queues:// bug, aren't you?16:39
kgriffs#info fix href-template paths in home doc in v1.0; don't wait for 1.116:39
flaper87I think fvollero is not around but he was looking for a bug to fix16:39
flaper87I'll ping him off-line and see if he wants to take it16:39
flaper87otherwise, I guess I could take it16:40
cpallaresif he doesn't , I could!16:40
flaper87cpallares: cool16:40
cpallaresIt's just fixing the path right?16:40
kgriffsok, whoever it is just assign yourself16:40
kgriffsI will update the spec16:40
kgriffsshould be easy; just remove the '/' prefix16:40
kgriffsoh, and write a test to ensure it can be combined16:41
fvolleroflaper87: i'm here16:41
flaper87cpallares: yeah, you also have to build a car, an airplane and elaborate a new theory about the milky way16:41
kgriffslet me note that16:41
cpallaresflaper87: haha don't mock me, for a beginner it sometimes feel like that :P16:41
flaper87cpallares: hihihi!16:41
malinikgriffs: what abt /v1 in some of the href-templates?16:41
flaper87fvollero: so, I remember you're looking for a bug to fix16:42
flaper87there's one that wouldn't take much of your time16:42
fvolleroflaper87: yep16:42
flaper87fvollero: wanna squash it ?16:42
malinikgriffs: like this one '"": {16:42
malini            "href-template": "/v1/queues/{queue_name}/claims{?limit}",16:42
fvolleroflaper87: ok.. .i'll give a try :)16:42
flaper87fvollero: cool, assign yourself there!16:42
flaper87and many thanks!16:42
fvolleroflaper87: i was backporting some stuff to puppet-ceilo16:42
fvolleroflaper87: let me finish it first16:42
flaper87fvollero: yeah sure, no hurries16:43
kgriffsfvollero:  thanks man!16:43
fvollerokgriffs: Arrrgh :) I didn't look at it yet :)16:43
flaper87fvollero: ametts will buy you some beers16:43
flaper87fvollero: he told me that!16:43
kgriffsok, moving on16:44
fvolleroflaper87: lol ! Fair enough! ametts thanks a bunch :)16:44
kgriffs#topic 204 vs. 200 + empty array16:44
amettsI'll swap beers for code any day16:44
fvollerokgriffs: link for rhia?16:44
kgriffslink for the bug?16:44
fvollerokgriffs: stupid keyboard16:44
* flaper87 is leaning towards 200 + empty list here16:45
fvollerokgriffs: I was referring to 204 vs 200 + []16:45
kgriffs#action fvollero to take care of bug #124565616:45
fvolleroflaper87: well, depend16:45
kgriffsfvollero: oic16:45
* ametts likes the empty list too. He even saw this technique in an OReilly book recently.16:45
kgriffsi don't have a link to a bug or anything, we are just walking through a list of feedback16:45
alcabreraI'm going to step out of this meeting to get heads down on the remaining sharding unit test failures.16:46
fvollerokgriffs: for what we're using 204 usually ?16:46
amettsGo alcabrera, go!16:46
kgriffsso, while 204 isn't necessarily incorrect, and saves a few bytes on the wire, it is inconsistent with other OpenStack APIs16:46
*** alcabrera is now known as alcabrera|brb16:46
kgriffsand *may* make client implementations a tad bit easier16:46
kgriffsfvollero: if there are no messages, we return 204 currently16:46
flaper87kgriffs: damn, you're faster than me16:46
kgriffs(Rather than an empty JSON array)16:47
flaper87I was about to write the client thing16:47
flaper87the other thing I like about the empty list is that it's more explicit16:47
flaper87you see that and you know there are no messages16:47
flaper87no need to check status code16:47
amettsflaper87: agreed16:48
fvollerokgriffs: gotcha :)16:48
kgriffscurrent behavior ^^16:48
*** oz_akan_ has quit IRC16:48
fvolleroflaper87: usually no one should look at the API response :)16:48
*** oz_akan_ has joined #openstack-marconi16:48
openstackgerritZhihao Yuan proposed a change to openstack/marconi: feat(shard): queue listing from multiple sources
kgriffsok, any objections to doing this in v1.1 ?16:49
zyuana trivial rebase16:49
flaper87fvollero: ish, it depends on what the user is doing16:49
flaper87but I agree16:49
flaper87kgriffs: no objections from me16:49
zyuani did not follow16:49
zyuanwhat's the resolution16:49
amettsThis will break existing clients.   Will there be too much legacy code by the time we get 1.1. out?16:50
kgriffszyuan: by "doing this" I mean switching to returning an empty array of messages and queues rather than 20416:50
amettsOr I guess that's what API versioning is for.16:51
kgriffsv1 will still be available16:51
amettsOkay.  I'm good, then.16:51
kgriffsv1.0, I mean16:51
zyuani have no strong objection, but i don't see how can we benefit from this also16:51
kgriffsshould we also do this when claiming messages?16:51
kgriffsif no messages to claim, return empty array?16:51
zyuanit seems to be easier to use, but affect user's bandwidth16:51
flaper87yeah, small tread-off betwen UX and bandwith, I guess16:52
fvollerozyuan: well, user bandwidth in the order of 4 bytes16:52
kgriffszyuan: true, but we can mitigate that somewhat by using msgpack media type16:53
kgriffsok, let's do it16:53
*** oz_akan_ has quit IRC16:53
flaper87it'll give you more consistency, better API and easier integration16:53
zyuanfvollero: more than that16:53
kgriffsI think we have rough consensus16:53
zyuanfvollero: you need hrefs as well16:53
kgriffs(consensus on doing this in marconi)16:53
zyuanotherwise it's even more incovient16:53
flaper87kgriffs: +1 from me16:54
kgriffs#info return an empty array on listing options when no items are available, plan for v1.116:54
kgriffs#topic Consistency around response envelope for /messages vs /messages?ids16:55
kgriffsthis would simplify client implementations16:56
kgriffsand conceptually, I think it makes sense16:56
kgriffsbecause both operations operate on the same resource, so you would expect the same representation to come back in each case IMO16:56
zyuani see no need so "consistency"16:56
zyuanbecause they are jsut different16:56
zyuanand we showed how they are different16:56
zyuanto make them look furtherly different, we can change endpoints name. but not response.16:58
kgriffsI have a counterpoint to that argument16:58
kgriffsto support API extensions, we will need to wrap those json arrays in objects16:59
zyuani don't see a need for API extension on message bulk get16:59
kgriffsso you can have {"EXT-foo": {...}, "messages": []}16:59
zyuanthis is just an endpoint for... restful, only16:59
kgriffszyuan: the point of extensions is to allow 3rd-parties to customize the API with things we didn't think about or feel is outside the scope of the core project16:59
zyuankgriffs: 3rd can already do that on message listing17:00
zyuani'm jsut saying there is no need for message bulk geting17:00
kgriffsmy point is, if we do that, then it makes it easy to make the two response schemas consistent17:00
zyuanthe ids one17:00
zyuanthey are different17:00
kgriffssince all you need is a "links" with and empty list17:00
zyuanso there is no need for "consistency"17:00
flaper87zyuan: could you ellaborate on why they are different?17:01
zyuanone is listing, one is get multiple17:01
zyuan?ids is not an API filtering the first one17:01
zyuanit's just another completely different API17:02
zyuanand pretty much useless in marconi picture17:02
zyuanfor the most of time you want listing, without ?ids17:02
flaper87thanks, we're on the same page there17:02
zyuanthey unfortuantely overloaded on names17:03
flaper87they're conceptually different, however, I can see some value in making the response consistent, although there's no need for consistency.17:03
zyuanbut we just don't have an HTTP method call LIST17:03
zyuanjust think this:17:04
flaper87but I'm leaning towards to keeping them as they are17:04
zyuanwhich "links" you want to put in ?ids response?17:04
zyuani can not think of any17:04
zyuannext? prev? lol17:04
flaper87yeah, I agree with you there17:05
flaper87but we can at least have { messages:{}}17:05
flaper87but we can at least have { messages:[]}17:05
*** oz_akan_ has joined #openstack-marconi17:05
flaper87but that won't make much sense, I guess.17:06
* flaper87 is thinking outloud17:06
zyuanthen that furtherly confuse people17:06
zyuanyou see, even we defined their semnatics17:06
zyuanpeople come to us and say "why they are inconsistent"17:06
zyuanand if you make them even more similiar17:06
zyuan"why ids does not accept limit=?"17:07
zyuanthe best way to make different things different is just to name them differently17:07
zyuanotherwise, i'm ok with removing ?ids GET17:07
flaper87if we ever get there, that'd be v2 material17:08
flaper87removing ?ids, that is!17:08
flaper87anyway, I think we shouldn't make these 2 responses consistent17:08
flaper87kgriffs: thoughts ?17:09
openstackgerritAlejandro Cabrera proposed a change to openstack/marconi: feat: connect sharding manager to control drivers
*** alcabrera|brb is now known as alcabrera17:10
alcabrerakgriffs, ametts, flaper87: There it is - shahrding manager core. ^^17:10
* alcabrera catches up to everything17:10
flaper87alcabrera: wb, thanks!17:10
alcabreraphew, lots of consistency discussions.17:12
* alcabrera waits for jenkins to sing17:12
alcabreraoops, I forgot to wrap a test in 'requires_mongodb'17:13
*** fvollero is now known as fvollero|gone17:19
kgriffsflaper87: hmmm, I still think that querying the same resource should yield the same basic schema17:21
kgriffseven if it is just {17:21
kgriffs"messages": [] }17:21
kgriffsinstead of just []17:21
kgriffshaving a "links": [] may no be needed17:22
zyuanso my suggestion is to split them into two resourses17:22
kgriffsif we want to change the operation syntax so it is it's own resource, then they can be different17:22
zyuanlist move listing messages to /messages_list17:22
zyuanor remove ?ids17:23
zyuanor move ?ids to another endpoint17:23
zyuanor invent a concept of "message group"17:23
kgriffslots of options; let's just plan on creating a new resource in v217:24
flaper87mmh, in any case, this seem to be a v2 change, we're changing completely the response of one of our calls.17:24
flaper87kgriffs: T_T17:24
kgriffsflaper87: I was just about to ask you if you agreed!17:24
flaper87that's my answer17:25
kgriffsOK, let's try to plow through a few more real quick17:25
kgriffsalcabrera is out tomorrow, so I wanted to get as far as we can today17:25
kgriffs#topic Response envelopes in general - don't return raw arrays (to allow for extensions)17:25
kgriffsI alluded to this earlier17:26
kgriffswhen we just return [] there is no where to put a non-breaking extension document17:26
kgriffsso, we should be returning {"things":[]} instead17:26
kgriffs(or so the suggestion has been given)17:27
kgriffsalcabrera: ping17:27
alcabrerakgriffs: pong17:28
* flaper87 will be out as well17:28
kgriffsthoughts: ^^17:28
flaper87it's holiday here17:28
openstackgerritAlejandro Cabrera proposed a change to openstack/marconi: feat: connect sharding manager to control drivers
alcabrerakgriffs: +1 for the [] -> {}17:29
alcabreraI agree with the extensibility argument17:29
* flaper87 agress as well17:29
kgriffsmoving on17:29
kgriffs#info encapsulate arrays in objects to make resource representations extensible17:31
kgriffs#topic Return full URIs in location/content-location headers (rather than relative ones)17:31
kgriffsthis tripped up Repose. I don't know if any other reverse proxies have a similar problem17:31
flaper87I don't have a strong opinion here and TBH, not sure if I know which one is correct17:31
alcabreraI'm with flaper8717:32
kgriffsok... only downside is slightly larger responses17:32
flaper87what's the overall preference there?17:32
flaper87I mean, what do people do normally?17:32
kgriffsgood question17:32
kgriffspartial URIs are being standardized in the latest HTTP 1.1 RFC draft17:33
kgriffsthe goal of that effort is to formalize stuff that is already pretty common, iirc17:33
alcabreraI wonder what HTTP 2.0 will do in this case...17:34
kgriffsI know a lot of web servers handle them fine17:34
kgriffsother openstack projects return full URIs afaik17:34
alcabrera(Jenkins: Patch Set 7: Works for me) - re: sharding manager (w0000t)17:35
flaper87if latest HTTP RFC uses partial URIs, I'd say we should use them and let not-standarized services burn in hell!17:35
* alcabrera returns to topic17:35
kgriffsclient URI/HTTP libraries handle them fine, don't they?17:35
* flaper87 calls python-request's phone number and asks that17:36
flaper87not sure if it speaks english, though. :D17:36
* flaper87 should have a webcam in his place17:36
kgriffsi mean, if I give you a "base" URI and a "relative" AKA "partial" URI, the library should just do the right thing17:36
flaper87kgriffs: yeah, correct17:37
flaper87I know python-request sticks to the RFC17:37
alcabreraI think by forcing absolute URIs, we impose a subtle constraint/difficulty.17:37
alcabreraThat being17:37
alcabreraIf someone wants to implement a client-side proxy17:37
flaper87I'm not sure about this specific case, though!17:37
alcabreraThey'd have to manually split content location each time.17:37
alcabreraWhereas if we stick to relative/partial17:37
kgriffsTBH, I don't know why Repose is trying to parse those headers in the first place17:38
flaper87but hey, if that's what the standard says, I think we should stick to that17:38
alcabreraThe client-side proxy can just use base + partial.17:38
kgriffs(rather than just pass-through)17:38
* flaper87 votes for partial URIs17:38
alcabrera+1 for partials17:39
kgriffsok, cool17:40
kgriffsI think repose has a bug or something to fix this anyway17:40
kgriffsand other proxies don't care AFAIK17:40
kgriffsJust a few more!17:40
kgriffs#info Continue to return partial URIs17:41
kgriffs#topic Is content-location header really necessary?17:41
kgriffsI did some reading about this17:41
kgriffsit is really only helpful if you have alternate representations of a resource that can be accessed using a different path vs. specifying the media type in Accept17:41
kgriffsclient requests /queues with Accept: application/json17:42
kgriffsthen the server could respond with17:42
kgriffscontent-location: /queues.json17:43
kgriffsPersonally, I think the whole blah.json thing is silly17:43
*** cpallares has quit IRC17:43
kgriffsso, I don't think we need to be setting this header17:44
alcabreraI don;t like that X.format, either. :x17:44
kgriffsok, how about no longer setting that header in v1.117:45
kgriffssave a few bytes, and it isn't useful17:45
kgriffs#topic Polling model is based on the assumption of a "streaming" interaction model. Is there a place for a "point in time" listing model, where you don't always get a "next" href?17:45
kgriffsThis a big one, and would be for v2.0 anyway17:45
kgriffsso I'd like to discuss it later17:45
kgriffs#info revisit for v2 API17:46
flaper87yeah and we'll also have a notification session during the design summit17:46
flaper87I'd discuss this after the summit17:46
alcabrera+1 for postponed discussion. I'd say we need for thoughts on that.17:47
alcabreraalso - v2.017:47
alcabreraIt can wait. :DF17:47
kgriffs#topic Query parameters are usually considered to be "optional", and yet DELETE queues/my-queue/messages requires an "ids" parameter. Some options:17:47
kgriffsSo, I think we can revisit this in v2 as well17:48
zyuan"consistency" problem again17:48
zyuanif user don't understand overloading17:48
zyuanthen we disambigulate them17:49
kgriffsI made a note to that affect (one of the options)17:49
zyuanthat's *the* way to solve problem like this17:49
kgriffsok, that's it!17:49
kgriffsthanks for hanging in there everyone!17:49
openstackMeeting ended Thu Oct 31 17:49:48 2013 UTC.  Information about MeetBot at . (v 0.1.4)17:49
openstackMinutes (text):
flaper87thanks everyone!17:50
alcabreraflaper87: thanks for hanging around pretty late to see this through! :D17:51
flaper87alcabrera: my pleasure!17:51
alcabreraI'll brb.17:52
kgriffsnotes are up:
*** yassine has quit IRC17:59
kgriffsalcabrera: green! Thanks!
alcabrerakgriffs: thanks for the notes!18:01
kgriffsI need to go grab some munchies. brb18:01
zyuanthanks :)18:01
zyuanthat's very comprehensive18:02
alcabrerazyuan: :D18:03
alcabreranote: it has a bug somewhere that's not being caught by the unit tests, so I'm investigating it further.18:04
alcabrerahowever, the structure is unlikely to change.18:04
alcabreraso you can use that to build up the list queues impl., zyuan. :)18:05
alcabrerakgriffs, zyuan: found the bug (L208)18:08
alcabreraI'm not setting up the dict -> config correctly18:08
alcabreraI stick all options in the group 'queues:drivers'18:08
alcabreraI should be doing 'queues:drivers:storage:{scheme}' for the driver-specific opts.18:09
alcabreraNow fixing.18:09
zyuanah ha ha ha18:09
flaper87gtg guys18:09
flaper87Have a good one!18:09
flaper87Good weekend to everyone!18:09
flaper87kgriffs: ametts have a safe flight!18:10
flaper87see you guys there!18:10
*** flaper87 is now known as flaper87|afk18:10
alcabreraflaper87|afk: enjoy your weekend~18:11
*** kgriffs is now known as kgriffs_afk18:12
*** briancline has quit IRC18:12
alcabreramore bugs are lurking18:21
*** oz_akan_ has quit IRC18:25
*** reed has quit IRC18:25
*** jergerber has quit IRC18:25
*** openstackgerrit has quit IRC18:25
*** ametts has quit IRC18:25
*** amitgandhi has quit IRC18:32
*** kgriffs_afk is now known as kgriffs18:32
*** ani has joined #openstack-marconi18:41
*** ani has quit IRC18:44
alcabrerazyuan: ping18:50
malinianybody else not seeing gerrit notifications in IRC?18:51
alcabreraI'm not.18:51
malinilooks like its broken18:51
alcabreramalini: ooohh, I see.18:53
alcabreraLook above ^^18:53
alcabrera"openstackgerrit has quit"18:53
alcabreraI hope it finds a great new job. :')18:53
kgriffsi think infra must be doing some maintenance today18:53
kgriffsI've been seeing openid weirdness too18:53
zyuanalcabrera: ?18:56
alcabrerazyuan: how goes the sharded queue listing?18:57
*** amitgandhi has joined #openstack-marconi18:57
zyuanalcabrera: replied in cloudqueues...18:57
*** briancline has joined #openstack-marconi18:59
alcabreramalini: +2'd (
alcabrerakgriffs: can you take a look at the tests above? They'll help us with sharding.19:01
malinithanks alcabrera..tht was fast!!19:01
kgriffsyeah, got to eat a burger real fast19:05
kgriffsmalini: re
kgriffsseems like patching a non-existing claim should return 404?19:15
maliniyou mean it doesnt currently return 404 ?19:16
*** ORerik has quit IRC19:16
maliniL243 is delete_non_existing_claim19:17
*** reed has joined #openstack-marconi19:18
*** jergerber has joined #openstack-marconi19:18
*** openstackgerrit has joined #openstack-marconi19:18
*** ametts has joined #openstack-marconi19:18
malinikgriffs: Am I missing something?19:19
kgriffsyou are testing that it returns 20419:21
kgriffsoh, nevermind19:21
kgriffswe decided that delete always "succeeds"19:21
kgriffsSomehow I got that mixed up with patch non-existing19:21
kgriffsthought that one was returning 20419:21
kgriffscarry on19:22
* kgriffs runs away19:22
malinican you +2 before running away ;)19:22
alcabrerakgriffs: ^ :)19:24
kgriffswhy delete there?19:26
maliniI added that to make sure the queue really does not exist, in case it exists in the db already19:27
maliniIn case somebody went ahead & wanted to create a does_not_exist queue :-P19:28
kgriffsyou could just use a UUId for the queue name19:28
kgriffsthen it will be, by definition, a unique queue that won't have a collision19:28
malinithe idea is that the queue will never exist19:29
maliniwe want a queue to be NOT there for these tests19:29
kgriffstrue, but if you are worried about someone else creating a queue with that name in the test DB then a UUID would avoid having to do a delete19:29
kgriffsand avoids the side-effect19:30
malinifair..I can update to reflect tht19:30
kgriffs(of deleting something someone else created)19:30
kgriffsyou can hard-code the uuid19:30
alcabrerakgriffs: gtk!19:31
kgriffsmalini: re "Get messages on non existing Queue."19:31
mpanetta_kgriffs: Is uuidgen too primitive? :P19:31
kgriffsthat returns 204 now, and we decided to wait and see if we get more complaints before deciding to change, right?19:31
malinikgriffs: correct19:32
kgriffsmpanetta_: that works too!19:32
*** ani has joined #openstack-marconi19:32
mpanetta_It is lacking a duck though19:33
kgriffstrue. true.19:33
kgriffsfwiw, ddg has all kind of things like that19:33
openstackgerritMalini Kamalambal proposed a change to openstack/marconi: Add Tests for non-existing resources
mpanetta_Ooo wolfram alpha19:35
maliniyayy..gerrit is back on irc19:35
mpanetta_Siris big brother19:35
malinikgriffs: tht one addresses ur comment19:35
kgriffsmalini: thanks! btw, looking at all those 404's, the 204's do look a bit out of place19:37
maliniwhich of those?19:37
kgriffsget messages, claim messages19:37
kgriffslet me make a note on that bug19:38
malinithts the current behavior..iirc we discussed this in our last/ or the one before meeting19:38
maliniwe decided to leave it as is19:38
malinibecause we wanted it to be as lazy as possible *-)19:39
kgriffsyeah, trouble is, then all of them should be 204 following that logic19:39
kgriffsor most of them19:39
kgriffswhich doesn't seem right either19:39
kgriffsunless we are going to auto-create when you GET /queues/my-queue19:39
malinitht wud confuse a lot of folks19:40
maliniwe dont want to create on gets , rt?19:40
kgriffsalthough... I guess that one could kind of be a noop if we auto-create when doing any create or update underneath the queue19:40
kgriffsmalini: it gets strange19:40
kgriffsmongo drivers allow you to create a db instance but I don't think the DB is actually created until you try to write something19:41
kgriffs(the first time)19:41
malinibut does it create a db when you try to read from it?19:41
kgriffsgood question, I suspect not - it just returns an empty cursor19:42
kgriffsbut I could be wrong19:42
malinitht maps to the 204 marconi currently returns19:42
kgriffsnow here is something interesting19:43
kgriffsJust noticed that self.client.get('/foo')19:43
kgriffsis treating '/foo' as a relative path19:43
kgriffsmalini: good point19:44
kgriffsi guess what I'm saying is if we are going to be lazy, we should be lazy all the way19:44
maliniyes..for creates & updates the lazy behavior makes sense19:45
malinibut it wud be weird to create on GET19:45
mpanetta_Creating a queue on get?19:46
kgriffsno, that is ok19:46
kgriffsi mean, it can return 204 and not create19:46
kgriffsbut then also /queues/foo/stats should return 204 as well19:47
openstackgerritZhihao Yuan proposed a change to openstack/marconi: feat(shard): queue listing from multiple sources
kgriffsmalini: anyway, _build_url should probably use uri to combine paths and then stuff that is passed to it should be relative paths, not absolute19:47
kgriffshttp.Client._build_url, that is19:48
kgriffsmalini: build failed on ur patch19:49
openstackgerritMalini Kamalambal proposed a change to openstack/marconi: Add Tests for non-existing resources
openstackgerritZhihao Yuan proposed a change to openstack/marconi: feat(shard): queue listing from multiple sources
malinikgriffs: just fixed it19:53
malinialcabrera: can you reapply ur +2 ?19:54
alcabreramalini: sure thing19:54
alcabreramalini: done!19:55
alcabreranow the last touch - I need to remove the uri from mongodb_options *if* a URI is already present in the oslo.config.cfg.ConfigOpts object passed in.19:56
alcabreraThis'll fix the annoying errors.19:56
mpanetta_which ones?19:57
kgriffsalcabrera: question19:58
kgriffswhere does the mongodb uri for the catalog itself come from?19:58
mpanetta_config file I hope19:58
mpanetta_Cause that is where I put it19:58
kgriffsare we overloading the option in the config file?19:58
kgriffsso, when sharding is disabled, that uri is for the messages DB19:59
mpanetta_I assume so19:59
kgriffswhen sharding is enabled, it is for the catalog?19:59
kgriffsalcabrera: ^^^19:59
alcabreraand... responding19:59
openstackgerritA change was merged to openstack/marconi: Add Tests for non-existing resources
alcabrera[queues:drivers:storage:mongodb].uri is used for both the data driver in a non-sharded context, and for the control driver in a sharded context.20:00
openstackgerritZhihao Yuan proposed a change to openstack/marconi: feat(shard): queue listing from multiple sources
alcabrerain a sharded context, the uri for the data driver comes from a catalogue lookup20:00
alcabrerakgriffs: so yes to your second question20:00
kgriffsis that documented in the sample conf, also in the description of the StrOpt?20:01
alcabreraNot yet - good thought!20:01
kgriffs...because I can see people getting confused!20:01
alcabrerayup - I was confused 'til I tackled it earlier.20:01
alcabreraAnd I was the one making these changes~!20:01
alcabreraalright, I've got things failing as expected locally.20:02
alcabreraThe solution isn't clean, but it's testable.20:02
*** jergerber has quit IRC20:05
*** jergerbe_ has joined #openstack-marconi20:05
alcabrerarunning past the tox gauntlet20:10
*** ametts has quit IRC20:11
*** ametts has joined #openstack-marconi20:12
openstackgerritAlejandro Cabrera proposed a change to openstack/marconi: feat: connect sharding manager to control drivers
alcabreraThat's where I had to make a kind of workaround20:30
kgriffsalcabrera: let me take a look20:33
*** ayoung has quit IRC20:41
kgriffsalcabrera: re
kgriffswhy is there a collision between the two ConfigOpts instances?20:43
kgriffsi mean, with uri?20:43
alcabreraLemme explain20:43
alcabreraThat ConfigOpt instance is passed down the chain: utils.load_storage_driver(conf) => mongodb.DataDriver.__init__(conf)20:44
alcabreraThat conf contains conf['queues:drivers:storage:mongodb'].uri20:44
alcabreraWhen the mongo.DataDriver initializes20:45
alcabreraIt also tries to register 'uri'20:45
alcabreraIn the same group.20:45
kgriffsoh, this has nothing to do with sharding.Catalog20:45
alcabreraOnly with20:46
alcabrerathe _init_shard portion of it20:46
kgriffs_init_driver is registering options20:47
kgriffsand so is datadriver.__init__20:48
kgriffswhay does _init_driver need to register storage_opts20:48
* kgriffs is looking20:48
alcabrerathat's the crux of the issue20:48
alcabrerakgriffs: I'm not sure how else to construct an oslo.config20:48
kgriffslet me see dict_to_conf20:49
kgriffsyou are using the defaults there20:49
*** jergerbe_ has quit IRC20:51
kgriffslet's see what happens if i set it, then register20:53
openstackgerritZhihao Yuan proposed a change to openstack/marconi: feat(shard): queue listing from multiple sources
alcabrerahow do I represent groups, I wonder...20:56
alcabreraI'd probably have to do something setattr20:56
alcabrerasomething like20:56 = <uri>20:56
*** ani has quit IRC20:57
*** ayoung has joined #openstack-marconi20:58
kgriffswhat about21:00
kgriffsin DaraDriver.__init__21:00
kgriffsjust check conf to see if sharding is enabled21:00
*** jergerber has joined #openstack-marconi21:00
kgriffsif it is, don't register any opts. period21:00
kgriffsstill hacky21:01
kgriffsbut, that way you don't run into problems with all the options21:01
kgriffsyou could have collisions with partions option, for example, right?21:01
kgriffsin _init_driver just set "sharding" in the general opts21:02
alcabreragood point21:03
alcabreraOf the mongodb options, 'uri' and 'database' are required, right?21:03
alcabrerakgriffs: ^21:04
alcabreraIf that's the case, then I can create database 'opt' on the fly if in a sharded context and it isn't provided.21:04
kgriffsuri is the only one without a default value21:05
kgriffs(according to MONGODB_OPTIONS)21:05
*** jergerber has quit IRC21:05
alcabrerabut 'database' won't be provided using that approach in a sharded context.21:06
kgriffsbtw, at some point we should find a way to verify the options that the admin is setting for a shard through the API.21:06
kgriffsre database21:06
kgriffsI'm confused21:06
*** tedross has quit IRC21:07
kgriffsI thought that basically the entire MONGODB_GROUP section was provided from the catalog, not INI21:07
kgriffsand the catalog itself would use uri and database from the INI21:07
alcabreragood point21:08
alcabrerathat's up to the admin21:08
alcabreraso if the admin fails to provide at least 'database' in options, there'll be 500s.21:08
alcabrerafor a mongodb shard21:08
kgriffsyou mean at least URI21:08
kgriffsdatabase defaults to "marconi"21:08
alcabrerasorry, I'm being unclear. :x21:09
alcabreraLet me clarify21:09
alcabreraIf we take the strategy of not registering options in a sharded context in mongodb.DataDriver, then 'database' won't be present in the configuration, not even with a default value.21:10
kgriffsyes, I see that now21:10
alcabreraAm I missing something?21:10
alcabreraAh, k.21:10
kgriffsno, I am just still wrapping my head around _init_driver21:10
kgriffsso, it doesn't insert defaults21:10
kgriffsbasically, we require the admin to set ALL options21:10
kgriffsno such thing as defaults for a shard entry21:11
*** ayoung has quit IRC21:11
kgriffsunless you want to splice them in by iterating over MONGODB_OPTIONS21:11
kgriffsbasically, for every missing thing, grab the default value from MONGODB_OPTIONS21:12
kgriffsdon't do that21:12
alcabreraIf I were to do that, I'd want to refactor... yeah...21:12
kgriffsnevermind again21:12
kgriffsyes, that would work21:12
alcabreraThere's a cool refactoring there.21:12
alcabrerasplit MONGODB_OPTIONS into REQUIRED and *BONUS*21:12
alcabreraWhere I'm at a loss for words on BONUS21:12
kgriffsi don't think you need to split21:13
alcabrerabonus being things like...21:13
kgriffsfor opt in MONGODB_OPTIONS:21:13
kgriffs    if opt not in shard_options:21:14
kgriffs        if opt.has_default():21:14
kgriffs            #inject default21:15
kgriffs        else:21:15
kgriffs            # Raise error21:15
kgriffssomething like that21:15
alcabrerathat looks like it would work.21:16
kgriffswhere shard_options = shard['options']21:16
alcabreraI'll experiment with it once I finish eliminating this batch of forwarding bugs21:16
*** ayoung has joined #openstack-marconi21:16
kgriffssounds like a plan21:16
kgriffsalcabrera: ETA on forwarding bugs?21:17
kgriffsI was about to rebase but will wait if you are close21:17
*** ayoung has quit IRC21:20
alcabrerakgriffs: ETA 2 minutes21:25
*** ayoung has joined #openstack-marconi21:25
alcabreraI tracked them all down.21:25
openstackgerritAlejandro Cabrera proposed a change to openstack/marconi: feat: connect sharding manager to control drivers
alcabrerampanetta_, kgriffs, malini, zyuan: There you go - a few more bugs down.21:27
kgriffsalcabrera: your on fire!
alcabreraIt is a bit warm in here...21:28
mpanetta_alcabrera: Thanks!21:28
* alcabrera burns21:28
alcabreraI'm starting to think that the forward mechanic was sneaky, sneaky.21:28
alcabreraGiven that each of these storage methods has rather different return behavior. :/21:29
kgriffsrebasing my cache stuff21:29
alcabrerakgriffs: w00t21:30
zyuan"A programmer has a problem, and says 'I'll solve this with object-oriented design. ' Now, a Programmer has a Problem..."21:30
kgriffsalcabrera: if it gets too ugly, we can just hard-code all the methods21:30
amettszyuan: lol  (it took me a few minutes to figure it out)21:30
alcabrerakgriffs: that's almost where I'm at. :P21:31
alcabreraI may have hard-coded all of them already.21:31
zyuan"A programmer has a problem.  He thought to himself, 'I know, I'll solve it with threads!' has Now problems. two he"21:32
alcabrerazyuan: lol21:33
openstackgerritAlejandro Cabrera proposed a change to openstack/marconi: feat: connect sharding manager to control drivers
*** mpanetta_ has quit IRC21:42
alcabrerakgriffs: I'm headed home. I'll be back on later to resolve the remaining three errors.21:42
alcabreradinner awaits. :D21:42
alcabreraSee you guys later. Thanks for all your help!21:42
*** alcabrera has quit IRC21:43
openstackgerritZhihao Yuan proposed a change to openstack/marconi: feat(shard): queue listing from multiple sources
amettskgriffs: Do you know the 30,000 picture of the status of python-marconiclient?   I was going to ask flaper87, but he got away before I got out of my meeting.  Do we know what works and what doesn't work?21:50
*** malini is now known as malini_afk22:09
*** fifieldt has joined #openstack-marconi22:15
kgriffsametts: I think all it does so far is CRUD for queues22:16
kgriffsnot messages22:16
kgriffslet me check22:16
kgriffsyeah, looks like messages are not implemented22:18
kgriffsgotta run22:18
*** amitgandhi has quit IRC22:18
amettsOk -- thanks.22:18
*** amitgandhi has joined #openstack-marconi22:18
kgriffsshould be pretty quick to add now that all the plumbing is in place22:18
*** amitgandhi has quit IRC22:20
*** amitgandhi has joined #openstack-marconi22:21
*** jcru has quit IRC22:22
*** amitgandhi has quit IRC22:26
*** kgriffs is now known as kgriffs_afk22:28
*** mpanetta has joined #openstack-marconi23:15
*** amitgandhi has joined #openstack-marconi23:17
*** mpanetta has quit IRC23:18
*** mpanetta has joined #openstack-marconi23:18
*** amitgandhi has quit IRC23:25
*** amitgandhi has joined #openstack-marconi23:26
*** malini_afk is now known as malini23:56

Generated by 2.14.0 by Marius Gedminas - find it at!