| *** amitgandhi has joined #openstack-marconi | 00:16 | |
| *** kgriffs is now known as kgriffs_afk | 00:19 | |
| *** vkmc has joined #openstack-marconi | 00:26 | |
| *** vkmc has quit IRC | 00:26 | |
| *** vkmc has joined #openstack-marconi | 00:26 | |
| *** fifieldt_ has joined #openstack-marconi | 00:35 | |
| *** amitgandhi has quit IRC | 00:41 | |
| *** amitgandhi has joined #openstack-marconi | 00:42 | |
| *** amitgandhi has quit IRC | 00:46 | |
| *** reed has quit IRC | 01:02 | |
| *** nosnos has joined #openstack-marconi | 01:05 | |
| *** vkmc has quit IRC | 01:17 | |
| *** vkmc has joined #openstack-marconi | 01:18 | |
| *** vkmc has quit IRC | 01:52 | |
| *** oz_akan_ has joined #openstack-marconi | 02:41 | |
| *** amitgandhi has joined #openstack-marconi | 02:42 | |
| *** amitgandhi has quit IRC | 02:42 | |
| *** amitgandhi has joined #openstack-marconi | 02:43 | |
| *** amitgandhi has quit IRC | 02:47 | |
| *** amitgandhi has joined #openstack-marconi | 03:43 | |
| *** amitgandhi has quit IRC | 03:47 | |
| *** amitgandhi has joined #openstack-marconi | 03:53 | |
| *** amitgandhi has quit IRC | 03:58 | |
| *** fifieldt_ has quit IRC | 04:02 | |
| *** fifieldt has joined #openstack-marconi | 04:02 | |
| *** oz_akan_ has quit IRC | 04:06 | |
| *** nosnos has quit IRC | 04:17 | |
| *** nosnos_ has joined #openstack-marconi | 04:17 | |
| *** nosnos_ has quit IRC | 04:22 | |
| *** nosnos has joined #openstack-marconi | 04:23 | |
| *** nosnos has quit IRC | 04:26 | |
| *** nosnos has joined #openstack-marconi | 04:29 | |
| *** nosnos_ has joined #openstack-marconi | 04:33 | |
| *** nosnos has quit IRC | 04:37 | |
| *** amitgandhi has joined #openstack-marconi | 04:43 | |
| *** amitgandhi has joined #openstack-marconi | 04:44 | |
| *** amitgandhi has quit IRC | 04:49 | |
| *** nosnos_ has quit IRC | 05:28 | |
| *** nosnos has joined #openstack-marconi | 05:28 | |
| *** amitgandhi has joined #openstack-marconi | 05:45 | |
| *** amitgandhi has quit IRC | 05:49 | |
| *** amitgandhi has joined #openstack-marconi | 05:55 | |
| *** amitgandhi has quit IRC | 05:59 | |
| *** nosnos_ has joined #openstack-marconi | 06:06 | |
| *** nosnos has quit IRC | 06:08 | |
| *** amitgandhi has joined #openstack-marconi | 06:45 | |
| *** amitgandhi has quit IRC | 06:45 | |
| *** amitgandhi has joined #openstack-marconi | 06:45 | |
| *** amitgandhi has quit IRC | 06:50 | |
| *** fifieldt has quit IRC | 07:16 | |
| *** nosnos_ has quit IRC | 07:22 | |
| *** nosnos has joined #openstack-marconi | 07:22 | |
| *** flaper87|afk is now known as flaper87 | 08:03 | |
| flaper87 | al-maisan: ping | 08:04 |
|---|---|---|
| *** amitgandhi has joined #openstack-marconi | 08:46 | |
| *** amitgandhi has joined #openstack-marconi | 08:47 | |
| *** amitgandhi has quit IRC | 08:51 | |
| *** yassine has joined #openstack-marconi | 09:11 | |
| openstackgerrit | Flavio Percoco proposed a change to openstack/marconi: Isolate tests a bit more https://review.openstack.org/53669 | 09:45 |
| *** amitgandhi has joined #openstack-marconi | 09:47 | |
| *** amitgandhi has quit IRC | 09:51 | |
| *** amitgandhi has joined #openstack-marconi | 10:47 | |
| *** amitgandhi has quit IRC | 10:47 | |
| *** amitgandhi has joined #openstack-marconi | 10:48 | |
| al-maisan | flaper87: pong? | 10:51 |
| *** nosnos has quit IRC | 10:51 | |
| *** amitgandhi has quit IRC | 10:52 | |
| flaper87 | al-maisan: hey :D | 10:52 |
| flaper87 | al-maisan: I pingged you before reading your email | 10:52 |
| flaper87 | :/ | 10:52 |
| al-maisan | hello flaper87 :) | 10:52 |
| flaper87 | al-maisan: thanks for that! | 10:52 |
| al-maisan | hope it helps | 10:52 |
| al-maisan | also, if you'd like to discuss it we can have a call | 10:53 |
| *** malini_afk is now known as malini | 11:20 | |
| *** flaper87 is now known as flaper87|afk | 11:30 | |
| *** vkmc has joined #openstack-marconi | 11:30 | |
| *** vkmc has quit IRC | 11:30 | |
| *** vkmc has joined #openstack-marconi | 11:30 | |
| *** tedross has joined #openstack-marconi | 11:35 | |
| *** amitgandhi has joined #openstack-marconi | 11:58 | |
| *** amitgandhi has quit IRC | 12:03 | |
| *** flaper87|afk is now known as flaper87 | 12:41 | |
| *** oz_akan_ has joined #openstack-marconi | 12:47 | |
| *** oz_akan_ has joined #openstack-marconi | 12:48 | |
| openstackgerrit | Alejandro Cabrera proposed a change to openstack/marconi: feat: integrate shard storage with transport https://review.openstack.org/50998 | 12:50 |
| openstackgerrit | Alejandro Cabrera proposed a change to openstack/marconi: feat: integrate shard storage with transport https://review.openstack.org/50998 | 13:13 |
| *** mpanetta has joined #openstack-marconi | 13:15 | |
| *** amitgandhi has joined #openstack-marconi | 13:18 | |
| *** mpanetta has quit IRC | 13:24 | |
| *** mpanetta has joined #openstack-marconi | 13:24 | |
| *** mpanetta has quit IRC | 13:25 | |
| *** mpanetta has joined #openstack-marconi | 13:26 | |
| *** mpanetta_ has joined #openstack-marconi | 13:29 | |
| *** mpanetta has quit IRC | 13:29 | |
| *** tedross has quit IRC | 13:32 | |
| *** tedross has joined #openstack-marconi | 13:48 | |
| *** ayoung_ is now known as ayoung | 13:49 | |
| *** kgriffs_afk is now known as kgriffs | 13:55 | |
| *** malini is now known as malini_afk | 13:57 | |
| *** jcru has joined #openstack-marconi | 14:04 | |
| *** jergerber has joined #openstack-marconi | 14:15 | |
| *** amitgandhi has quit IRC | 14:19 | |
| *** alcabrera has joined #openstack-marconi | 14:26 | |
| alcabrera | Good morning! :D | 14:26 |
| alcabrera | kgriffs, flaper87: o/ | 14:30 |
| kgriffs | yo | 14:31 |
| flaper87 | alcabrera: kgriffs good morning guys | 14:31 |
| * flaper87 has been investigating things for next week sessions | 14:31 | |
| alcabrera | sweet. How goes prep for the summit? :D | 14:33 |
| alcabrera | flaper87, kgriffs: I rebased the transport + storage sharding patch this morning and addressed all feedback. I couldn't get functional tests in at the moment, but I've noted we need them. | 14:34 |
| kgriffs | kk. Changes look good. | 14:37 |
| *** cpallares has joined #openstack-marconi | 14:37 | |
| kgriffs | flaper87: can you take another look and see if everything you noted was addressed? | 14:38 |
| kgriffs | https://review.openstack.org/#/c/50998 | 14:38 |
| flaper87 | kgriffs: on it! ;) | 14:39 |
| flaper87 | alcabrera: could you please add a bug for the missing functional tests ? | 14:45 |
| flaper87 | I'm fine with those landing later! | 14:45 |
| alcabrera | flaper87: cool! I'll do that now. | 14:47 |
| * flaper87 +2'd that patch | 14:47 | |
| * flaper87 gives alcabrera many many pop-tarts | 14:47 | |
| flaper87 | lot of them | 14:48 |
| flaper87 | like, tons of trucks full of them | 14:48 |
| alcabrera | flaper87: done https://bugs.launchpad.net/marconi/+bug/1246757 | 14:48 |
| alcabrera | :D | 14:48 |
| alcabrera | yaaaay, pop tarts! | 14:48 |
| * alcabrera eats one, shares the rest with #openstack-marconi | 14:49 | |
| *** amitgandhi has joined #openstack-marconi | 14:50 | |
| openstackgerrit | A change was merged to openstack/marconi: feat: integrate shard storage with transport https://review.openstack.org/50998 | 14:51 |
| alcabrera | w00t | 14:53 |
| *** tedross has quit IRC | 14:59 | |
| flaper87 | alcabrera: great work there! Way to go! | 15:00 |
| alcabrera | flaper87: ¡gracias! | 15:01 |
| flaper87 | kgriffs: if you've a chance, i'd love to hear your thoughts about the remaining client patches | 15:01 |
| flaper87 | man, i've 3 patches in workinprogress, what's worng with me? | 15:02 |
| * flaper87 needs to get more things done | 15:02 | |
| alcabrera | flaper87: and there's always more to be done! ;) | 15:04 |
| *** amitgandhi has quit IRC | 15:06 | |
| *** amitgandhi has joined #openstack-marconi | 15:07 | |
| *** amitgandhi has quit IRC | 15:12 | |
| *** amitgandhi has joined #openstack-marconi | 15:13 | |
| *** tedross has joined #openstack-marconi | 15:16 | |
| *** malini_afk is now known as malini | 15:35 | |
| *** reed has joined #openstack-marconi | 15:51 | |
| kgriffs | flaper87, alcabrera: have a few minutes to discuss more API feedback? | 16:09 |
| alcabrera | kgriffs: yup | 16:09 |
| flaper87 | yup | 16:10 |
| kgriffs | ok | 16:10 |
| kgriffs | https://wiki.openstack.org/wiki/Marconi/specs/api/next | 16:10 |
| kgriffs | #startmeeting marconi-api-http | 16:10 |
| openstack | Meeting started Thu Oct 31 16:10:52 2013 UTC and is due to finish in 60 minutes. The chair is kgriffs. Information about MeetBot at http://wiki.debian.org/MeetBot. | 16:10 |
| openstack | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 16:10 |
| openstack | The meeting name has been set to 'marconi_api_http' | 16:10 |
| kgriffs | #link https://wiki.openstack.org/wiki/Marconi/specs/api/next | 16:10 |
| kgriffs | #topic Return an href for deleting all claimed messages with a single call | 16:11 |
| alcabrera | Hmmm... | 16:12 |
| alcabrera | That's a strange kind of pattern. | 16:12 |
| alcabrera | It's almost like cancelling a claim. | 16:12 |
| kgriffs | somebody was asking about this. Currently I don't think the bulk delete takes a claim ID | 16:12 |
| alcabrera | Or grabbing a *right* to delete a set of messages. | 16:12 |
| kgriffs | I guess the idea is you would process N messages, then do the delete | 16:12 |
| kgriffs | but seems like an anti-pattern | 16:12 |
| alcabrera | ahh | 16:12 |
| alcabrera | I think that's an anti-pattern, given that it makes sense to grab one, delete one as it is completed. | 16:13 |
| alcabrera | Also | 16:13 |
| alcabrera | Crash conditions | 16:13 |
| alcabrera | If you processed 1-5 of 10, but then crashed... | 16:13 |
| alcabrera | How do others know you completed 1-5? | 16:13 |
| flaper87 | mmh, I'm not sure I like this idea | 16:13 |
| alcabrera | Anti-pattern rationale ^^ | 16:13 |
| flaper87 | I mean, why would you delete all claimed messages ? | 16:13 |
| kgriffs | since if the worker crashes in the middle, that is N-p work items that would have to be rolled back. | 16:13 |
| flaper87 | I'd rather release them if I don't want to work on them anymore | 16:14 |
| kgriffs | only reason to support this that I can see (and it is a thin reason at that) would be if you have tons of very tiny messages | 16:15 |
| kgriffs | like stats or something | 16:15 |
| kgriffs | but then, marconi is probably not the right tool for the job in that case? | 16:15 |
| kgriffs | i mean, the advantage there would be batching up the deletes so they happen faster | 16:16 |
| flaper87 | agreed | 16:16 |
| alcabrera | yeah. :/ | 16:16 |
| kgriffs | on the other-other hand, that could mostly be solved by the transport layer | 16:16 |
| kgriffs | zmq, spdy/http2.0, websockets | 16:17 |
| kgriffs | ok, so strike this one? | 16:17 |
| alcabrera | +1 | 16:18 |
| alcabrera | for striking | 16:18 |
| flaper87 | striking +1 | 16:18 |
| flaper87 | next ? | 16:20 |
| kgriffs | sorry, taking notes | 16:21 |
| flaper87 | ok, sorry, didn't mean to push! | 16:21 |
| kgriffs | #info do not support bulk delete of claimed messages | 16:21 |
| kgriffs | #topic Allow PUTing metadata when creating a queue | 16:21 |
| kgriffs | TBH, I can't remember where this came from | 16:21 |
| kgriffs | we used to do this, but changed | 16:22 |
| kgriffs | because we decided that mixing queue metadata and queue options in a single resource was ugly | 16:22 |
| flaper87 | I think the only advantage of that is to reduce the number of calls required to create a queue | 16:22 |
| kgriffs | so we wanted to make them their own resources | 16:22 |
| alcabrera | yup | 16:22 |
| kgriffs | flaper87: right | 16:22 |
| flaper87 | however, what you said was our last thought | 16:22 |
| flaper87 | making them independent resources | 16:23 |
| kgriffs | I suppose if we add, say, default message TTL we could have that directly on the queue resource | 16:23 |
| kgriffs | http put https://marconi.example.com/v2/queues default_ttl:=300 | 16:24 |
| kgriffs | (or something like that) | 16:24 |
| kgriffs | you might also put queue flavor as well (choose a shard type) | 16:25 |
| kgriffs | anyway, I think we won't change this | 16:25 |
| flaper87 | I agree with you | 16:25 |
| kgriffs | but we might add default options and stuff | 16:25 |
| flaper87 | also, metadata != queue properties | 16:25 |
| kgriffs | something like default TTL - would that be a v1.1 or a v2 do you think? | 16:25 |
| kgriffs | flaper87: +1 | 16:25 |
| flaper87 | so, which should we support during queue's creation? | 16:25 |
| flaper87 | that will confuse users | 16:25 |
| *** whenry has quit IRC | 16:25 | |
| flaper87 | and amke the api inconsistent | 16:26 |
| flaper87 | so, totally with you there! Lets strike this one | 16:26 |
| kgriffs | ok. | 16:26 |
| alcabrera | strike + 1 | 16:26 |
| kgriffs | I just added this one... | 16:26 |
| kgriffs | #topic Set queue options for defaults, such as message and claim TTL. | 16:27 |
| kgriffs | TBH, I'm not sure how useful that is, since the clients will just stick in whatever | 16:27 |
| kgriffs | only thing is I guess it would make the request body a tiny bit smaller | 16:27 |
| *** whenry has joined #openstack-marconi | 16:27 | |
| flaper87 | my only concern is that we'll have to have those properties available everytime a message gets in | 16:28 |
| kgriffs | yes, we can cache them, but it will still be a little bit of a perf hit | 16:28 |
| flaper87 | and it'll require even more resources to the final user | 16:29 |
| flaper87 | like 'take queue caching under consideration when you buy the RAM' | 16:29 |
| kgriffs | how about striking then, given clients can implement default behavior on their own | 16:29 |
| alcabrera | kgriffs: +1 to that | 16:29 |
| kgriffs | flaper87: true | 16:29 |
| flaper87 | kgriffs: +1 | 16:29 |
| kgriffs | ok, let me make notes | 16:30 |
| kgriffs | #topic homedoc should return relative URIs or encode version in href-template | 16:30 |
| kgriffs | see also: https://bugs.launchpad.net/marconi/+bug/1245656 | 16:31 |
| kgriffs | so, I view this as a typo in the spec and a bug in the implementation | 16:31 |
| kgriffs | pedants will insist the v1 API is frozen, but I think we should fix this | 16:31 |
| flaper87 | +1 for relative urls | 16:32 |
| flaper87 | and +1 for encoded version of href-template | 16:32 |
| flaper87 | and +1 for fixing this | 16:32 |
| kgriffs | ok. Any client that is trying to "follow it's nose", using hrefs from the home doc, is broken now anyway | 16:33 |
| alcabrera | I'm in favor of getting this fixed sooner rather than later. | 16:33 |
| alcabrera | Since the longer we leave it a broken state, the longer we'll encourage "bad" behavior. | 16:33 |
| kgriffs | flaper87: fwiw, I discovered that "relative URI" is a misleading term | 16:33 |
| flaper87 | kgriffs: what's the right one? | 16:34 |
| * flaper87 is about to learn something new | 16:34 | |
| kgriffs | a "relative URI" as described by RFC really means "partial URI" | 16:34 |
| kgriffs | using my term there, a "partial URI" can have either an *absolute* and *relative* path | 16:34 |
| flaper87 | alcabrera: kgriffs in addition, I need this to work to implement that client :D | 16:34 |
| kgriffs | s/and/or | 16:34 |
| kgriffs | so both of these are "partial" URIs: | 16:35 |
| kgriffs | /queues/foo | 16:35 |
| alcabrera | flaper87: yeah, keep the client coming! ;) | 16:35 |
| kgriffs | queues/foo | 16:35 |
| kgriffs | formally defined and "relative URIs" | 16:35 |
| flaper87 | kgriffs: mmh, interesting! I ignored that. | 16:36 |
| kgriffs | s/and/as | 16:36 |
| kgriffs | so, when you see someone say "relative URI" it may or may not mean the *path* in the URI is relative! | 16:36 |
| ametts | Just read: "so, I view this as a typo in the spec and a bug in the implementation" | 16:36 |
| ametts | kgriffs should be a politician | 16:36 |
| kgriffs | #link http://www.ietf.org/rfc/rfc1808.txt | 16:36 |
| kgriffs | ametts: LOL | 16:37 |
| ametts | Now we can change the API any time we want! :) | 16:37 |
| kgriffs | ok so, any objections to fixing this in v1.0 ? | 16:37 |
| kgriffs | the only danger is someone may have written a client that works around the bug | 16:37 |
| kgriffs | and we will break their workaround | 16:37 |
| alcabrera | none from me | 16:38 |
| ametts | Not from me. I think sooner than later is better too. | 16:38 |
| malini | kgriffs: we really need this fixed in v1.0 :) | 16:38 |
| kgriffs | any volunteers? | 16:38 |
| ametts | Now I'm doubly-sure. We don't want to incur the wrath of Malini. | 16:38 |
| flaper87 | v1.0 | 16:38 |
| kgriffs | #link https://bugs.launchpad.net/marconi/+bug/1245656 | 16:38 |
| flaper87 | fvollero: cpallares ^ ? | 16:38 |
| flaper87 | maybe you guys wanna take that one | 16:38 |
| flaper87 | cpallares: you're already working on s/queues:// bug, aren't you? | 16:39 |
| cpallares | yes | 16:39 |
| kgriffs | #info fix href-template paths in home doc in v1.0; don't wait for 1.1 | 16:39 |
| flaper87 | I think fvollero is not around but he was looking for a bug to fix | 16:39 |
| flaper87 | I'll ping him off-line and see if he wants to take it | 16:39 |
| flaper87 | otherwise, I guess I could take it | 16:40 |
| cpallares | if he doesn't , I could! | 16:40 |
| flaper87 | cpallares: cool | 16:40 |
| cpallares | It's just fixing the path right? | 16:40 |
| flaper87 | +1 | 16:40 |
| kgriffs | ok, whoever it is just assign yourself | 16:40 |
| kgriffs | I will update the spec | 16:40 |
| kgriffs | should be easy; just remove the '/' prefix | 16:40 |
| kgriffs | oh, and write a test to ensure it can be combined | 16:41 |
| fvollero | flaper87: i'm here | 16:41 |
| flaper87 | cpallares: yeah, you also have to build a car, an airplane and elaborate a new theory about the milky way | 16:41 |
| kgriffs | let me note that | 16:41 |
| cpallares | flaper87: haha don't mock me, for a beginner it sometimes feel like that :P | 16:41 |
| flaper87 | cpallares: hihihi! | 16:41 |
| malini | kgriffs: what abt /v1 in some of the href-templates? | 16:41 |
| flaper87 | fvollero: so, I remember you're looking for a bug to fix | 16:42 |
| flaper87 | there's one that wouldn't take much of your time | 16:42 |
| fvollero | flaper87: yep | 16:42 |
| flaper87 | fvollero: wanna squash it ? | 16:42 |
| malini | kgriffs: like this one '"http://docs.openstack-marconi.org/rel/claim": { | 16:42 |
| malini | "href-template": "/v1/queues/{queue_name}/claims{?limit}", | 16:42 |
| malini | ' | 16:42 |
| fvollero | flaper87: ok.. .i'll give a try :) | 16:42 |
| flaper87 | fvollero: cool, assign yourself there! | 16:42 |
| flaper87 | and many thanks! | 16:42 |
| fvollero | flaper87: i was backporting some stuff to puppet-ceilo | 16:42 |
| fvollero | flaper87: let me finish it first | 16:42 |
| flaper87 | fvollero: yeah sure, no hurries | 16:43 |
| kgriffs | fvollero: thanks man! | 16:43 |
| fvollero | kgriffs: Arrrgh :) I didn't look at it yet :) | 16:43 |
| flaper87 | fvollero: ametts will buy you some beers | 16:43 |
| flaper87 | fvollero: he told me that! | 16:43 |
| kgriffs | ok, moving on | 16:44 |
| fvollero | flaper87: lol ! Fair enough! ametts thanks a bunch :) | 16:44 |
| kgriffs | #topic 204 vs. 200 + empty array | 16:44 |
| ametts | I'll swap beers for code any day | 16:44 |
| fvollero | kgriffs: link for rhia? | 16:44 |
| kgriffs | link for the bug? | 16:44 |
| fvollero | yeah | 16:44 |
| kgriffs | https://bugs.launchpad.net/marconi/+bug/1245656 | 16:44 |
| fvollero | kgriffs: stupid keyboard | 16:44 |
| * flaper87 is leaning towards 200 + empty list here | 16:45 | |
| fvollero | kgriffs: I was referring to 204 vs 200 + [] | 16:45 |
| kgriffs | #action fvollero to take care of bug #1245656 | 16:45 |
| fvollero | flaper87: well, depend | 16:45 |
| kgriffs | fvollero: oic | 16:45 |
| * ametts likes the empty list too. He even saw this technique in an OReilly book recently. | 16:45 | |
| kgriffs | i don't have a link to a bug or anything, we are just walking through a list of feedback | 16:45 |
| fvollero | ok | 16:45 |
| alcabrera | I'm going to step out of this meeting to get heads down on the remaining sharding unit test failures. | 16:46 |
| fvollero | kgriffs: for what we're using 204 usually ? | 16:46 |
| ametts | Go alcabrera, go! | 16:46 |
| kgriffs | so, while 204 isn't necessarily incorrect, and saves a few bytes on the wire, it is inconsistent with other OpenStack APIs | 16:46 |
| *** alcabrera is now known as alcabrera|brb | 16:46 | |
| kgriffs | and *may* make client implementations a tad bit easier | 16:46 |
| kgriffs | fvollero: if there are no messages, we return 204 currently | 16:46 |
| flaper87 | kgriffs: damn, you're faster than me | 16:46 |
| kgriffs | (Rather than an empty JSON array) | 16:47 |
| flaper87 | I was about to write the client thing | 16:47 |
| flaper87 | the other thing I like about the empty list is that it's more explicit | 16:47 |
| flaper87 | you see that and you know there are no messages | 16:47 |
| flaper87 | no need to check status code | 16:47 |
| flaper87 | etc | 16:47 |
| ametts | flaper87: agreed | 16:48 |
| fvollero | kgriffs: gotcha :) | 16:48 |
| kgriffs | #link https://wiki.openstack.org/wiki/Marconi/specs/api/v1#List_Messages | 16:48 |
| kgriffs | current behavior ^^ | 16:48 |
| *** oz_akan_ has quit IRC | 16:48 | |
| fvollero | flaper87: usually no one should look at the API response :) | 16:48 |
| *** oz_akan_ has joined #openstack-marconi | 16:48 | |
| openstackgerrit | Zhihao Yuan proposed a change to openstack/marconi: feat(shard): queue listing from multiple sources https://review.openstack.org/53127 | 16:49 |
| kgriffs | ok, any objections to doing this in v1.1 ? | 16:49 |
| zyuan | a trivial rebase | 16:49 |
| flaper87 | fvollero: ish, it depends on what the user is doing | 16:49 |
| flaper87 | but I agree | 16:49 |
| flaper87 | kgriffs: no objections from me | 16:49 |
| zyuan | i did not follow | 16:49 |
| zyuan | what's the resolution | 16:49 |
| ametts | This will break existing clients. Will there be too much legacy code by the time we get 1.1. out? | 16:50 |
| kgriffs | zyuan: by "doing this" I mean switching to returning an empty array of messages and queues rather than 204 | 16:50 |
| ametts | Or I guess that's what API versioning is for. | 16:51 |
| kgriffs | right | 16:51 |
| kgriffs | v1 will still be available | 16:51 |
| ametts | Okay. I'm good, then. | 16:51 |
| kgriffs | v1.0, I mean | 16:51 |
| zyuan | i have no strong objection, but i don't see how can we benefit from this also | 16:51 |
| kgriffs | should we also do this when claiming messages? | 16:51 |
| kgriffs | if no messages to claim, return empty array? | 16:51 |
| zyuan | it seems to be easier to use, but affect user's bandwidth | 16:51 |
| flaper87 | yeah, small tread-off betwen UX and bandwith, I guess | 16:52 |
| fvollero | zyuan: well, user bandwidth in the order of 4 bytes | 16:52 |
| kgriffs | zyuan: true, but we can mitigate that somewhat by using msgpack media type | 16:53 |
| kgriffs | ok, let's do it | 16:53 |
| *** oz_akan_ has quit IRC | 16:53 | |
| flaper87 | it'll give you more consistency, better API and easier integration | 16:53 |
| zyuan | fvollero: more than that | 16:53 |
| kgriffs | I think we have rough consensus | 16:53 |
| zyuan | fvollero: you need hrefs as well | 16:53 |
| kgriffs | (consensus on doing this in marconi) | 16:53 |
| zyuan | otherwise it's even more incovient | 16:53 |
| flaper87 | kgriffs: +1 from me | 16:54 |
| kgriffs | #info return an empty array on listing options when no items are available, plan for v1.1 | 16:54 |
| kgriffs | #topic Consistency around response envelope for /messages vs /messages?ids | 16:55 |
| kgriffs | this would simplify client implementations | 16:56 |
| kgriffs | and conceptually, I think it makes sense | 16:56 |
| kgriffs | because both operations operate on the same resource, so you would expect the same representation to come back in each case IMO | 16:56 |
| zyuan | i see no need so "consistency" | 16:56 |
| zyuan | for* | 16:56 |
| zyuan | because they are jsut different | 16:56 |
| zyuan | and we showed how they are different | 16:56 |
| zyuan | to make them look furtherly different, we can change endpoints name. but not response. | 16:58 |
| kgriffs | I have a counterpoint to that argument | 16:58 |
| kgriffs | to support API extensions, we will need to wrap those json arrays in objects | 16:59 |
| zyuan | i don't see a need for API extension on message bulk get | 16:59 |
| kgriffs | so you can have {"EXT-foo": {...}, "messages": []} | 16:59 |
| zyuan | this is just an endpoint for... restful, only | 16:59 |
| kgriffs | zyuan: the point of extensions is to allow 3rd-parties to customize the API with things we didn't think about or feel is outside the scope of the core project | 16:59 |
| zyuan | kgriffs: 3rd can already do that on message listing | 17:00 |
| zyuan | i'm jsut saying there is no need for message bulk geting | 17:00 |
| kgriffs | my point is, if we do that, then it makes it easy to make the two response schemas consistent | 17:00 |
| zyuan | the ids one | 17:00 |
| zyuan | they are different | 17:00 |
| kgriffs | since all you need is a "links" with and empty list | 17:00 |
| zyuan | so there is no need for "consistency" | 17:00 |
| flaper87 | zyuan: could you ellaborate on why they are different? | 17:01 |
| flaper87 | elaborate* | 17:01 |
| zyuan | one is listing, one is get multiple | 17:01 |
| zyuan | ?ids is not an API filtering the first one | 17:01 |
| zyuan | it's just another completely different API | 17:02 |
| zyuan | and pretty much useless in marconi picture | 17:02 |
| zyuan | for the most of time you want listing, without ?ids | 17:02 |
| flaper87 | thanks, we're on the same page there | 17:02 |
| zyuan | they unfortuantely overloaded on names | 17:03 |
| flaper87 | they're conceptually different, however, I can see some value in making the response consistent, although there's no need for consistency. | 17:03 |
| zyuan | but we just don't have an HTTP method call LIST | 17:03 |
| zyuan | just think this: | 17:04 |
| flaper87 | but I'm leaning towards to keeping them as they are | 17:04 |
| zyuan | which "links" you want to put in ?ids response? | 17:04 |
| zyuan | i can not think of any | 17:04 |
| zyuan | next? prev? lol | 17:04 |
| flaper87 | yeah, I agree with you there | 17:05 |
| flaper87 | but we can at least have { messages:{}} | 17:05 |
| flaper87 | but we can at least have { messages:[]} | 17:05 |
| *** oz_akan_ has joined #openstack-marconi | 17:05 | |
| flaper87 | but that won't make much sense, I guess. | 17:06 |
| * flaper87 is thinking outloud | 17:06 | |
| zyuan | then that furtherly confuse people | 17:06 |
| zyuan | you see, even we defined their semnatics | 17:06 |
| zyuan | people come to us and say "why they are inconsistent" | 17:06 |
| zyuan | and if you make them even more similiar | 17:06 |
| zyuan | hehe | 17:06 |
| flaper87 | heh | 17:07 |
| zyuan | "why ids does not accept limit=?" | 17:07 |
| zyuan | the best way to make different things different is just to name them differently | 17:07 |
| zyuan | otherwise, i'm ok with removing ?ids GET | 17:07 |
| flaper87 | if we ever get there, that'd be v2 material | 17:08 |
| flaper87 | removing ?ids, that is! | 17:08 |
| flaper87 | anyway, I think we shouldn't make these 2 responses consistent | 17:08 |
| flaper87 | kgriffs: thoughts ? | 17:09 |
| openstackgerrit | Alejandro Cabrera proposed a change to openstack/marconi: feat: connect sharding manager to control drivers https://review.openstack.org/54605 | 17:10 |
| *** alcabrera|brb is now known as alcabrera | 17:10 | |
| alcabrera | kgriffs, ametts, flaper87: There it is - shahrding manager core. ^^ | 17:10 |
| * alcabrera catches up to everything | 17:10 | |
| flaper87 | alcabrera: wb, thanks! | 17:10 |
| alcabrera | phew, lots of consistency discussions. | 17:12 |
| * alcabrera waits for jenkins to sing | 17:12 | |
| alcabrera | oops, I forgot to wrap a test in 'requires_mongodb' | 17:13 |
| *** fvollero is now known as fvollero|gone | 17:19 | |
| kgriffs | flaper87: hmmm, I still think that querying the same resource should yield the same basic schema | 17:21 |
| kgriffs | even if it is just { | 17:21 |
| kgriffs | "messages": [] } | 17:21 |
| kgriffs | instead of just [] | 17:21 |
| kgriffs | having a "links": [] may no be needed | 17:22 |
| zyuan | so my suggestion is to split them into two resourses | 17:22 |
| kgriffs | if we want to change the operation syntax so it is it's own resource, then they can be different | 17:22 |
| zyuan | list move listing messages to /messages_list | 17:22 |
| zyuan | like* | 17:23 |
| zyuan | or remove ?ids | 17:23 |
| zyuan | or move ?ids to another endpoint | 17:23 |
| zyuan | or invent a concept of "message group" | 17:23 |
| kgriffs | lots of options; let's just plan on creating a new resource in v2 | 17:24 |
| flaper87 | mmh, in any case, this seem to be a v2 change, we're changing completely the response of one of our calls. | 17:24 |
| flaper87 | kgriffs: T_T | 17:24 |
| kgriffs | LOL | 17:24 |
| kgriffs | flaper87: I was just about to ask you if you agreed! | 17:24 |
| flaper87 | :P | 17:25 |
| flaper87 | that's my answer | 17:25 |
| flaper87 | :D | 17:25 |
| kgriffs | OK, let's try to plow through a few more real quick | 17:25 |
| kgriffs | alcabrera is out tomorrow, so I wanted to get as far as we can today | 17:25 |
| kgriffs | #topic Response envelopes in general - don't return raw arrays (to allow for extensions) | 17:25 |
| kgriffs | I alluded to this earlier | 17:26 |
| kgriffs | when we just return [] there is no where to put a non-breaking extension document | 17:26 |
| kgriffs | so, we should be returning {"things":[]} instead | 17:26 |
| kgriffs | (or so the suggestion has been given) | 17:27 |
| kgriffs | alcabrera: ping | 17:27 |
| alcabrera | kgriffs: pong | 17:28 |
| * flaper87 will be out as well | 17:28 | |
| kgriffs | thoughts: ^^ | 17:28 |
| flaper87 | it's holiday here | 17:28 |
| openstackgerrit | Alejandro Cabrera proposed a change to openstack/marconi: feat: connect sharding manager to control drivers https://review.openstack.org/54605 | 17:28 |
| alcabrera | kgriffs: +1 for the [] -> {} | 17:29 |
| alcabrera | I agree with the extensibility argument | 17:29 |
| * flaper87 agress as well | 17:29 | |
| kgriffs | ok | 17:29 |
| kgriffs | moving on | 17:29 |
| kgriffs | wait. | 17:30 |
| flaper87 | agrees | 17:30 |
| flaper87 | * | 17:30 |
| kgriffs | #info encapsulate arrays in objects to make resource representations extensible | 17:31 |
| kgriffs | #topic Return full URIs in location/content-location headers (rather than relative ones) | 17:31 |
| kgriffs | this tripped up Repose. I don't know if any other reverse proxies have a similar problem | 17:31 |
| alcabrera | hmmm | 17:31 |
| flaper87 | I don't have a strong opinion here and TBH, not sure if I know which one is correct | 17:31 |
| alcabrera | I'm with flaper87 | 17:32 |
| kgriffs | ok... only downside is slightly larger responses | 17:32 |
| flaper87 | what's the overall preference there? | 17:32 |
| flaper87 | I mean, what do people do normally? | 17:32 |
| kgriffs | good question | 17:32 |
| kgriffs | partial URIs are being standardized in the latest HTTP 1.1 RFC draft | 17:33 |
| kgriffs | the goal of that effort is to formalize stuff that is already pretty common, iirc | 17:33 |
| alcabrera | I wonder what HTTP 2.0 will do in this case... | 17:34 |
| kgriffs | I know a lot of web servers handle them fine | 17:34 |
| kgriffs | other openstack projects return full URIs afaik | 17:34 |
| alcabrera | (Jenkins: Patch Set 7: Works for me) - re: sharding manager (w0000t) | 17:35 |
| flaper87 | if latest HTTP RFC uses partial URIs, I'd say we should use them and let not-standarized services burn in hell! | 17:35 |
| flaper87 | muahahahahahahahhaha | 17:35 |
| * alcabrera returns to topic | 17:35 | |
| kgriffs | LOL | 17:35 |
| alcabrera | lol | 17:35 |
| kgriffs | client URI/HTTP libraries handle them fine, don't they? | 17:35 |
| * flaper87 calls python-request's phone number and asks that | 17:36 | |
| flaper87 | not sure if it speaks english, though. :D | 17:36 |
| * flaper87 should have a webcam in his place | 17:36 | |
| kgriffs | i mean, if I give you a "base" URI and a "relative" AKA "partial" URI, the library should just do the right thing | 17:36 |
| flaper87 | kgriffs: yeah, correct | 17:37 |
| flaper87 | I know python-request sticks to the RFC | 17:37 |
| alcabrera | I think by forcing absolute URIs, we impose a subtle constraint/difficulty. | 17:37 |
| alcabrera | That being | 17:37 |
| alcabrera | If someone wants to implement a client-side proxy | 17:37 |
| flaper87 | I'm not sure about this specific case, though! | 17:37 |
| alcabrera | They'd have to manually split content location each time. | 17:37 |
| alcabrera | Whereas if we stick to relative/partial | 17:37 |
| kgriffs | TBH, I don't know why Repose is trying to parse those headers in the first place | 17:38 |
| flaper87 | but hey, if that's what the standard says, I think we should stick to that | 17:38 |
| alcabrera | The client-side proxy can just use base + partial. | 17:38 |
| kgriffs | (rather than just pass-through) | 17:38 |
| * flaper87 votes for partial URIs | 17:38 | |
| alcabrera | +1 for partials | 17:39 |
| kgriffs | ok, cool | 17:40 |
| kgriffs | I think repose has a bug or something to fix this anyway | 17:40 |
| kgriffs | and other proxies don't care AFAIK | 17:40 |
| kgriffs | Just a few more! | 17:40 |
| kgriffs | #info Continue to return partial URIs | 17:41 |
| alcabrera | kk | 17:41 |
| kgriffs | #topic Is content-location header really necessary? | 17:41 |
| kgriffs | I did some reading about this | 17:41 |
| kgriffs | it is really only helpful if you have alternate representations of a resource that can be accessed using a different path vs. specifying the media type in Accept | 17:41 |
| kgriffs | example: | 17:42 |
| kgriffs | client requests /queues with Accept: application/json | 17:42 |
| kgriffs | then the server could respond with | 17:42 |
| kgriffs | content-location: /queues.json | 17:43 |
| kgriffs | Personally, I think the whole blah.json thing is silly | 17:43 |
| *** cpallares has quit IRC | 17:43 | |
| kgriffs | so, I don't think we need to be setting this header | 17:44 |
| alcabrera | I don;t like that X.format, either. :x | 17:44 |
| kgriffs | ok, how about no longer setting that header in v1.1 | 17:45 |
| kgriffs | save a few bytes, and it isn't useful | 17:45 |
| flaper87 | yeah | 17:45 |
| alcabrera | +1 | 17:45 |
| flaper87 | +1 | 17:45 |
| kgriffs | #topic Polling model is based on the assumption of a "streaming" interaction model. Is there a place for a "point in time" listing model, where you don't always get a "next" href? | 17:45 |
| kgriffs | This a big one, and would be for v2.0 anyway | 17:45 |
| kgriffs | so I'd like to discuss it later | 17:45 |
| kgriffs | #info revisit for v2 API | 17:46 |
| flaper87 | yeah and we'll also have a notification session during the design summit | 17:46 |
| flaper87 | I'd discuss this after the summit | 17:46 |
| alcabrera | +1 for postponed discussion. I'd say we need for thoughts on that. | 17:47 |
| alcabrera | also - v2.0 | 17:47 |
| alcabrera | It can wait. :DF | 17:47 |
| alcabrera | :D | 17:47 |
| kgriffs | #topic Query parameters are usually considered to be "optional", and yet DELETE queues/my-queue/messages requires an "ids" parameter. Some options: | 17:47 |
| kgriffs | So, I think we can revisit this in v2 as well | 17:48 |
| zyuan | hehe | 17:48 |
| zyuan | "consistency" problem again | 17:48 |
| kgriffs | right | 17:48 |
| zyuan | if user don't understand overloading | 17:48 |
| zyuan | then we disambigulate them | 17:49 |
| flaper87 | right | 17:49 |
| kgriffs | I made a note to that affect (one of the options) | 17:49 |
| zyuan | that's *the* way to solve problem like this | 17:49 |
| kgriffs | ok, that's it! | 17:49 |
| kgriffs | thanks for hanging in there everyone! | 17:49 |
| kgriffs | #endmeeting | 17:49 |
| openstack | Meeting ended Thu Oct 31 17:49:48 2013 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 17:49 |
| openstack | Minutes: http://eavesdrop.openstack.org/meetings/marconi_api_http/2013/marconi_api_http.2013-10-31-16.10.html | 17:49 |
| openstack | Minutes (text): http://eavesdrop.openstack.org/meetings/marconi_api_http/2013/marconi_api_http.2013-10-31-16.10.txt | 17:49 |
| openstack | Log: http://eavesdrop.openstack.org/meetings/marconi_api_http/2013/marconi_api_http.2013-10-31-16.10.log.html | 17:49 |
| flaper87 | awesome! | 17:50 |
| flaper87 | thanks everyone! | 17:50 |
| alcabrera | w00t | 17:50 |
| alcabrera | flaper87: thanks for hanging around pretty late to see this through! :D | 17:51 |
| flaper87 | alcabrera: my pleasure! | 17:51 |
| flaper87 | :D | 17:51 |
| alcabrera | I'll brb. | 17:52 |
| alcabrera | back | 17:54 |
| kgriffs | notes are up: https://wiki.openstack.org/wiki/Marconi/specs/api/next#HTTP | 17:59 |
| *** yassine has quit IRC | 17:59 | |
| kgriffs | alcabrera: green! Thanks! https://review.openstack.org/#/c/54605/ | 18:01 |
| alcabrera | kgriffs: thanks for the notes! | 18:01 |
| kgriffs | I need to go grab some munchies. brb | 18:01 |
| alcabrera | kk | 18:01 |
| zyuan | thanks :) | 18:01 |
| zyuan | that's very comprehensive | 18:02 |
| alcabrera | zyuan: :D | 18:03 |
| alcabrera | note: it has a bug somewhere that's not being caught by the unit tests, so I'm investigating it further. | 18:04 |
| alcabrera | however, the structure is unlikely to change. | 18:04 |
| alcabrera | so you can use that to build up the list queues impl., zyuan. :) | 18:05 |
| alcabrera | kgriffs, zyuan: found the bug https://review.openstack.org/#/c/54605/7/marconi/queues/storage/sharding.py (L208) | 18:08 |
| alcabrera | I'm not setting up the dict -> config correctly | 18:08 |
| alcabrera | I stick all options in the group 'queues:drivers' | 18:08 |
| alcabrera | I should be doing 'queues:drivers:storage:{scheme}' for the driver-specific opts. | 18:09 |
| alcabrera | Now fixing. | 18:09 |
| zyuan | ah ha ha ha | 18:09 |
| alcabrera | :D | 18:09 |
| flaper87 | gtg guys | 18:09 |
| flaper87 | Have a good one! | 18:09 |
| flaper87 | Good weekend to everyone! | 18:09 |
| flaper87 | kgriffs: ametts have a safe flight! | 18:10 |
| flaper87 | see you guys there! | 18:10 |
| *** flaper87 is now known as flaper87|afk | 18:10 | |
| alcabrera | flaper87|afk: enjoy your weekend~ | 18:11 |
| *** kgriffs is now known as kgriffs_afk | 18:12 | |
| *** briancline has quit IRC | 18:12 | |
| alcabrera | more bugs are lurking | 18:21 |
| *** oz_akan_ has quit IRC | 18:25 | |
| *** reed has quit IRC | 18:25 | |
| *** jergerber has quit IRC | 18:25 | |
| *** openstackgerrit has quit IRC | 18:25 | |
| *** ametts has quit IRC | 18:25 | |
| *** amitgandhi has quit IRC | 18:32 | |
| *** kgriffs_afk is now known as kgriffs | 18:32 | |
| *** ani has joined #openstack-marconi | 18:41 | |
| ani | \aniuskad | 18:41 |
| *** ani has quit IRC | 18:44 | |
| alcabrera | zyuan: ping | 18:50 |
| malini | anybody else not seeing gerrit notifications in IRC? | 18:51 |
| alcabrera | <---- | 18:51 |
| alcabrera | I'm not. | 18:51 |
| malini | looks like its broken | 18:51 |
| alcabrera | malini: ooohh, I see. | 18:53 |
| alcabrera | Look above ^^ | 18:53 |
| alcabrera | "openstackgerrit has quit" | 18:53 |
| alcabrera | I hope it finds a great new job. :') | 18:53 |
| malini | :D | 18:53 |
| kgriffs | i think infra must be doing some maintenance today | 18:53 |
| kgriffs | I've been seeing openid weirdness too | 18:53 |
| zyuan | alcabrera: ? | 18:56 |
| alcabrera | zyuan: how goes the sharded queue listing? | 18:57 |
| *** amitgandhi has joined #openstack-marconi | 18:57 | |
| zyuan | alcabrera: replied in cloudqueues... | 18:57 |
| alcabrera | thanks! | 18:58 |
| *** briancline has joined #openstack-marconi | 18:59 | |
| alcabrera | malini: +2'd (https://review.openstack.org/#/c/53691/) | 19:01 |
| alcabrera | kgriffs: can you take a look at the tests above? They'll help us with sharding. | 19:01 |
| malini | thanks alcabrera..tht was fast!! | 19:01 |
| kgriffs | yeah, got to eat a burger real fast | 19:05 |
| kgriffs | brb | 19:05 |
| kgriffs | malini: re https://review.openstack.org/#/c/53691/9/tests/functional/wsgi/v1/test_claims.py | 19:15 |
| kgriffs | seems like patching a non-existing claim should return 404? | 19:15 |
| malini | yes | 19:16 |
| kgriffs | bug? | 19:16 |
| malini | you mean it doesnt currently return 404 ? | 19:16 |
| *** ORerik has quit IRC | 19:16 | |
| kgriffs | L243 | 19:16 |
| malini | L243 is delete_non_existing_claim | 19:17 |
| *** reed has joined #openstack-marconi | 19:18 | |
| *** jergerber has joined #openstack-marconi | 19:18 | |
| *** openstackgerrit has joined #openstack-marconi | 19:18 | |
| *** ametts has joined #openstack-marconi | 19:18 | |
| malini | kgriffs: Am I missing something? | 19:19 |
| kgriffs | you are testing that it returns 204 | 19:21 |
| kgriffs | oh, nevermind | 19:21 |
| kgriffs | we decided that delete always "succeeds" | 19:21 |
| kgriffs | Somehow I got that mixed up with patch non-existing | 19:21 |
| kgriffs | thought that one was returning 204 | 19:21 |
| kgriffs | carry on | 19:22 |
| * kgriffs runs away | 19:22 | |
| malini | can you +2 before running away ;) | 19:22 |
| alcabrera | kgriffs: ^ :) | 19:24 |
| kgriffs | malini: https://review.openstack.org/#/c/53691/9/tests/functional/wsgi/v1/test_queues.py | 19:26 |
| kgriffs | L416 | 19:26 |
| kgriffs | why delete there? | 19:26 |
| malini | I added that to make sure the queue really does not exist, in case it exists in the db already | 19:27 |
| malini | In case somebody went ahead & wanted to create a does_not_exist queue :-P | 19:28 |
| kgriffs | you could just use a UUId for the queue name | 19:28 |
| kgriffs | then it will be, by definition, a unique queue that won't have a collision | 19:28 |
| malini | the idea is that the queue will never exist | 19:29 |
| malini | we want a queue to be NOT there for these tests | 19:29 |
| kgriffs | true, but if you are worried about someone else creating a queue with that name in the test DB then a UUID would avoid having to do a delete | 19:29 |
| kgriffs | and avoids the side-effect | 19:30 |
| malini | fair..I can update to reflect tht | 19:30 |
| kgriffs | (of deleting something someone else created) | 19:30 |
| kgriffs | you can hard-code the uuid | 19:30 |
| malini | ok | 19:30 |
| kgriffs | ea2b6dd4-4262-11e3-9b0d-6d82a24410d2 | 19:30 |
| kgriffs | protip: https://duckduckgo.com/?q=uuid | 19:30 |
| alcabrera | kgriffs: gtk! | 19:31 |
| kgriffs | malini: re "Get messages on non existing Queue." | 19:31 |
| mpanetta_ | kgriffs: Is uuidgen too primitive? :P | 19:31 |
| kgriffs | that returns 204 now, and we decided to wait and see if we get more complaints before deciding to change, right? | 19:31 |
| malini | kgriffs: correct | 19:32 |
| kgriffs | mpanetta_: that works too! | 19:32 |
| *** ani has joined #openstack-marconi | 19:32 | |
| mpanetta_ | It is lacking a duck though | 19:33 |
| kgriffs | true. true. | 19:33 |
| kgriffs | fwiw, ddg has all kind of things like that | 19:33 |
| kgriffs | e.g. | 19:34 |
| kgriffs | https://duckduckgo.com/?q=rand | 19:34 |
| kgriffs | https://duckduckgo.com/?q=sha256+foo1 | 19:34 |
| openstackgerrit | Malini Kamalambal proposed a change to openstack/marconi: Add Tests for non-existing resources https://review.openstack.org/53691 | 19:34 |
| kgriffs | https://duckduckgo.com/?q=distance+to+the+moon | 19:34 |
| kgriffs | :p | 19:34 |
| mpanetta_ | Ooo wolfram alpha | 19:35 |
| malini | yayy..gerrit is back on irc | 19:35 |
| mpanetta_ | Siris big brother | 19:35 |
| malini | kgriffs: tht one addresses ur comment | 19:35 |
| kgriffs | malini: thanks! btw, looking at all those 404's, the 204's do look a bit out of place | 19:37 |
| kgriffs | https://review.openstack.org/#/c/53691/10/tests/functional/wsgi/v1/test_queues.py | 19:37 |
| malini | which of those? | 19:37 |
| kgriffs | get messages, claim messages | 19:37 |
| kgriffs | let me make a note on that bug | 19:38 |
| malini | thts the current behavior..iirc we discussed this in our last/ or the one before meeting | 19:38 |
| malini | we decided to leave it as is | 19:38 |
| malini | because we wanted it to be as lazy as possible *-) | 19:39 |
| kgriffs | yeah, trouble is, then all of them should be 204 following that logic | 19:39 |
| kgriffs | or most of them | 19:39 |
| kgriffs | which doesn't seem right either | 19:39 |
| malini | yeah | 19:39 |
| kgriffs | unless we are going to auto-create when you GET /queues/my-queue | 19:39 |
| malini | tht wud confuse a lot of folks | 19:40 |
| malini | we dont want to create on gets , rt? | 19:40 |
| kgriffs | although... I guess that one could kind of be a noop if we auto-create when doing any create or update underneath the queue | 19:40 |
| kgriffs | malini: it gets strange | 19:40 |
| malini | :D | 19:40 |
| kgriffs | mongo drivers allow you to create a db instance but I don't think the DB is actually created until you try to write something | 19:41 |
| kgriffs | (the first time) | 19:41 |
| malini | but does it create a db when you try to read from it? | 19:41 |
| kgriffs | good question, I suspect not - it just returns an empty cursor | 19:42 |
| kgriffs | but I could be wrong | 19:42 |
| malini | tht maps to the 204 marconi currently returns | 19:42 |
| kgriffs | now here is something interesting | 19:43 |
| kgriffs | Just noticed that self.client.get('/foo') | 19:43 |
| kgriffs | is treating '/foo' as a relative path | 19:43 |
| kgriffs | malini: good point | 19:44 |
| kgriffs | i guess what I'm saying is if we are going to be lazy, we should be lazy all the way | 19:44 |
| mpanetta_ | kgriffs++ | 19:45 |
| mpanetta_ | heh | 19:45 |
| malini | yes..for creates & updates the lazy behavior makes sense | 19:45 |
| malini | but it wud be weird to create on GET | 19:45 |
| mpanetta_ | Creating a queue on get? | 19:46 |
| kgriffs | no, that is ok | 19:46 |
| kgriffs | i mean, it can return 204 and not create | 19:46 |
| kgriffs | but then also /queues/foo/stats should return 204 as well | 19:47 |
| openstackgerrit | Zhihao Yuan proposed a change to openstack/marconi: feat(shard): queue listing from multiple sources https://review.openstack.org/53127 | 19:47 |
| kgriffs | malini: anyway, _build_url should probably use uri to combine paths and then stuff that is passed to it should be relative paths, not absolute | 19:47 |
| kgriffs | http.Client._build_url, that is | 19:48 |
| kgriffs | malini: build failed on ur patch | 19:49 |
| openstackgerrit | Malini Kamalambal proposed a change to openstack/marconi: Add Tests for non-existing resources https://review.openstack.org/53691 | 19:50 |
| openstackgerrit | Zhihao Yuan proposed a change to openstack/marconi: feat(shard): queue listing from multiple sources https://review.openstack.org/53127 | 19:51 |
| malini | kgriffs: just fixed it | 19:53 |
| malini | alcabrera: can you reapply ur +2 ? | 19:54 |
| alcabrera | malini: sure thing | 19:54 |
| alcabrera | malini: done! | 19:55 |
| malini | thx!! | 19:56 |
| alcabrera | now the last touch - I need to remove the uri from mongodb_options *if* a URI is already present in the oslo.config.cfg.ConfigOpts object passed in. | 19:56 |
| alcabrera | This'll fix the annoying errors. | 19:56 |
| mpanetta_ | which ones? | 19:57 |
| kgriffs | alcabrera: question | 19:58 |
| kgriffs | where does the mongodb uri for the catalog itself come from? | 19:58 |
| mpanetta_ | config file I hope | 19:58 |
| mpanetta_ | Cause that is where I put it | 19:58 |
| kgriffs | are we overloading the option in the config file? | 19:58 |
| kgriffs | so, when sharding is disabled, that uri is for the messages DB | 19:59 |
| mpanetta_ | I assume so | 19:59 |
| kgriffs | when sharding is enabled, it is for the catalog? | 19:59 |
| kgriffs | alcabrera: ^^^ | 19:59 |
| alcabrera | reading | 19:59 |
| alcabrera | and... responding | 19:59 |
| openstackgerrit | A change was merged to openstack/marconi: Add Tests for non-existing resources https://review.openstack.org/53691 | 19:59 |
| alcabrera | [queues:drivers:storage:mongodb].uri is used for both the data driver in a non-sharded context, and for the control driver in a sharded context. | 20:00 |
| openstackgerrit | Zhihao Yuan proposed a change to openstack/marconi: feat(shard): queue listing from multiple sources https://review.openstack.org/53127 | 20:00 |
| alcabrera | in a sharded context, the uri for the data driver comes from a catalogue lookup | 20:00 |
| alcabrera | kgriffs: so yes to your second question | 20:00 |
| kgriffs | is that documented in the sample conf, also in the description of the StrOpt? | 20:01 |
| alcabrera | Not yet - good thought! | 20:01 |
| kgriffs | ...because I can see people getting confused! | 20:01 |
| alcabrera | yup - I was confused 'til I tackled it earlier. | 20:01 |
| alcabrera | And I was the one making these changes~! | 20:01 |
| kgriffs | heh | 20:01 |
| alcabrera | alright, I've got things failing as expected locally. | 20:02 |
| alcabrera | The solution isn't clean, but it's testable. | 20:02 |
| *** jergerber has quit IRC | 20:05 | |
| *** jergerbe_ has joined #openstack-marconi | 20:05 | |
| alcabrera | running past the tox gauntlet | 20:10 |
| *** ametts has quit IRC | 20:11 | |
| *** ametts has joined #openstack-marconi | 20:12 | |
| openstackgerrit | Alejandro Cabrera proposed a change to openstack/marconi: feat: connect sharding manager to control drivers https://review.openstack.org/54605 | 20:30 |
| alcabrera | kgriffs: https://review.openstack.org/#/c/54605/9/marconi/queues/storage/mongodb/driver.py | 20:30 |
| alcabrera | That's where I had to make a kind of workaround | 20:30 |
| kgriffs | alcabrera: let me take a look | 20:33 |
| *** ayoung has quit IRC | 20:41 | |
| kgriffs | alcabrera: re https://review.openstack.org/#/c/54605/9/marconi/queues/storage/sharding.py | 20:42 |
| kgriffs | L275 | 20:42 |
| alcabrera | cfg.ConfigOpts() | 20:43 |
| kgriffs | why is there a collision between the two ConfigOpts instances? | 20:43 |
| kgriffs | i mean, with uri? | 20:43 |
| alcabrera | Lemme explain | 20:43 |
| alcabrera | That ConfigOpt instance is passed down the chain: utils.load_storage_driver(conf) => mongodb.DataDriver.__init__(conf) | 20:44 |
| alcabrera | That conf contains conf['queues:drivers:storage:mongodb'].uri | 20:44 |
| alcabrera | So... | 20:44 |
| alcabrera | When the mongo.DataDriver initializes | 20:45 |
| alcabrera | It also tries to register 'uri' | 20:45 |
| alcabrera | In the same group. | 20:45 |
| kgriffs | oh, this has nothing to do with sharding.Catalog | 20:45 |
| kgriffs | ? | 20:45 |
| alcabrera | Only with | 20:46 |
| alcabrera | the _init_shard portion of it | 20:46 |
| kgriffs | wait | 20:46 |
| kgriffs | hmm | 20:46 |
| kgriffs | oic | 20:47 |
| kgriffs | _init_driver is registering options | 20:47 |
| alcabrera | yup | 20:47 |
| kgriffs | and so is datadriver.__init__ | 20:48 |
| kgriffs | whay does _init_driver need to register storage_opts | 20:48 |
| * kgriffs is looking | 20:48 | |
| alcabrera | that's the crux of the issue | 20:48 |
| alcabrera | kgriffs: I'm not sure how else to construct an oslo.config | 20:48 |
| alcabrera | thingy | 20:49 |
| kgriffs | let me see dict_to_conf | 20:49 |
| kgriffs | oic | 20:49 |
| kgriffs | you are using the defaults there | 20:49 |
| *** jergerbe_ has quit IRC | 20:51 | |
| kgriffs | alcabrera: http://paste.openstack.org/show/50334/ | 20:52 |
| alcabrera | hmmm | 20:52 |
| kgriffs | let's see what happens if i set it, then register | 20:53 |
| openstackgerrit | Zhihao Yuan proposed a change to openstack/marconi: feat(shard): queue listing from multiple sources https://review.openstack.org/53127 | 20:54 |
| kgriffs | alcabrera: http://paste.openstack.org/show/50337/ | 20:55 |
| alcabrera | hmmmm | 20:55 |
| alcabrera | how do I represent groups, I wonder... | 20:56 |
| alcabrera | I'd probably have to do something setattr | 20:56 |
| alcabrera | something like | 20:56 |
| alcabrera | conf.queues.drivers.storage.mongodb.uri = <uri> | 20:56 |
| *** ani has quit IRC | 20:57 | |
| *** ayoung has joined #openstack-marconi | 20:58 | |
| kgriffs | hmm | 20:59 |
| kgriffs | what about | 21:00 |
| kgriffs | in DaraDriver.__init__ | 21:00 |
| kgriffs | just check conf to see if sharding is enabled | 21:00 |
| *** jergerber has joined #openstack-marconi | 21:00 | |
| kgriffs | if it is, don't register any opts. period | 21:00 |
| kgriffs | still hacky | 21:01 |
| kgriffs | but, that way you don't run into problems with all the options | 21:01 |
| kgriffs | you could have collisions with partions option, for example, right? | 21:01 |
| kgriffs | in _init_driver just set "sharding" in the general opts | 21:02 |
| alcabrera | hmmm | 21:03 |
| alcabrera | good point | 21:03 |
| alcabrera | Of the mongodb options, 'uri' and 'database' are required, right? | 21:03 |
| alcabrera | kgriffs: ^ | 21:04 |
| alcabrera | If that's the case, then I can create database 'opt' on the fly if in a sharded context and it isn't provided. | 21:04 |
| kgriffs | uri is the only one without a default value | 21:05 |
| kgriffs | (according to MONGODB_OPTIONS) | 21:05 |
| *** jergerber has quit IRC | 21:05 | |
| alcabrera | but 'database' won't be provided using that approach in a sharded context. | 21:06 |
| kgriffs | btw, at some point we should find a way to verify the options that the admin is setting for a shard through the API. | 21:06 |
| alcabrera | +1 | 21:06 |
| kgriffs | re database | 21:06 |
| kgriffs | I'm confused | 21:06 |
| *** tedross has quit IRC | 21:07 | |
| kgriffs | I thought that basically the entire MONGODB_GROUP section was provided from the catalog, not INI | 21:07 |
| kgriffs | and the catalog itself would use uri and database from the INI | 21:07 |
| alcabrera | ahh | 21:08 |
| alcabrera | good point | 21:08 |
| alcabrera | that's up to the admin | 21:08 |
| alcabrera | yeah... | 21:08 |
| alcabrera | so if the admin fails to provide at least 'database' in options, there'll be 500s. | 21:08 |
| alcabrera | for a mongodb shard | 21:08 |
| kgriffs | you mean at least URI | 21:08 |
| kgriffs | database defaults to "marconi" | 21:08 |
| alcabrera | sorry, I'm being unclear. :x | 21:09 |
| alcabrera | Let me clarify | 21:09 |
| alcabrera | If we take the strategy of not registering options in a sharded context in mongodb.DataDriver, then 'database' won't be present in the configuration, not even with a default value. | 21:10 |
| kgriffs | yes, I see that now | 21:10 |
| alcabrera | Am I missing something? | 21:10 |
| alcabrera | Ah, k. | 21:10 |
| kgriffs | no, I am just still wrapping my head around _init_driver | 21:10 |
| kgriffs | so, it doesn't insert defaults | 21:10 |
| alcabrera | gotcha | 21:10 |
| kgriffs | basically, we require the admin to set ALL options | 21:10 |
| alcabrera | yup | 21:10 |
| kgriffs | no such thing as defaults for a shard entry | 21:11 |
| *** ayoung has quit IRC | 21:11 | |
| kgriffs | unless you want to splice them in by iterating over MONGODB_OPTIONS | 21:11 |
| kgriffs | basically, for every missing thing, grab the default value from MONGODB_OPTIONS | 21:12 |
| alcabrera | hmmm | 21:12 |
| kgriffs | oops | 21:12 |
| kgriffs | nevermind | 21:12 |
| kgriffs | don't do that | 21:12 |
| kgriffs | wait | 21:12 |
| alcabrera | If I were to do that, I'd want to refactor... yeah... | 21:12 |
| kgriffs | nevermind again | 21:12 |
| kgriffs | yes, that would work | 21:12 |
| alcabrera | There's a cool refactoring there. | 21:12 |
| alcabrera | split MONGODB_OPTIONS into REQUIRED and *BONUS* | 21:12 |
| alcabrera | Where I'm at a loss for words on BONUS | 21:12 |
| kgriffs | i don't think you need to split | 21:13 |
| mpanetta_ | bonus? | 21:13 |
| alcabrera | bonus being things like... | 21:13 |
| alcabrera | partitions | 21:13 |
| kgriffs | algorithm: | 21:13 |
| alcabrera | max_retry_jitter | 21:13 |
| alcabrera | etc | 21:13 |
| mpanetta_ | oic | 21:13 |
| kgriffs | for opt in MONGODB_OPTIONS: | 21:13 |
| kgriffs | if opt not in shard_options: | 21:14 |
| kgriffs | if opt.has_default(): | 21:14 |
| kgriffs | #inject default | 21:15 |
| kgriffs | else: | 21:15 |
| kgriffs | # Raise error | 21:15 |
| kgriffs | something like that | 21:15 |
| alcabrera | that looks like it would work. | 21:16 |
| kgriffs | where shard_options = shard['options'] | 21:16 |
| alcabrera | I'll experiment with it once I finish eliminating this batch of forwarding bugs | 21:16 |
| *** ayoung has joined #openstack-marconi | 21:16 | |
| kgriffs | sounds like a plan | 21:16 |
| kgriffs | alcabrera: ETA on forwarding bugs? | 21:17 |
| kgriffs | I was about to rebase but will wait if you are close | 21:17 |
| *** ayoung has quit IRC | 21:20 | |
| alcabrera | kgriffs: ETA 2 minutes | 21:25 |
| *** ayoung has joined #openstack-marconi | 21:25 | |
| alcabrera | I tracked them all down. | 21:25 |
| kgriffs | w00t | 21:26 |
| openstackgerrit | Alejandro Cabrera proposed a change to openstack/marconi: feat: connect sharding manager to control drivers https://review.openstack.org/54605 | 21:27 |
| alcabrera | mpanetta_, kgriffs, malini, zyuan: There you go - a few more bugs down. | 21:27 |
| kgriffs | alcabrera: your on fire! http://goo.gl/Y1JIRd | 21:27 |
| kgriffs | s/your/you're | 21:28 |
| alcabrera | hahaha | 21:28 |
| alcabrera | It is a bit warm in here... | 21:28 |
| mpanetta_ | alcabrera: Thanks! | 21:28 |
| * alcabrera burns | 21:28 | |
| alcabrera | I'm starting to think that the forward mechanic was sneaky, sneaky. | 21:28 |
| alcabrera | Given that each of these storage methods has rather different return behavior. :/ | 21:29 |
| kgriffs | rebasing my cache stuff | 21:29 |
| alcabrera | kgriffs: w00t | 21:30 |
| zyuan | "A programmer has a problem, and says 'I'll solve this with object-oriented design. ' Now, a Programmer has a Problem..." | 21:30 |
| kgriffs | alcabrera: if it gets too ugly, we can just hard-code all the methods | 21:30 |
| ametts | zyuan: lol (it took me a few minutes to figure it out) | 21:30 |
| alcabrera | kgriffs: that's almost where I'm at. :P | 21:31 |
| alcabrera | I may have hard-coded all of them already. | 21:31 |
| alcabrera | Maybe./ | 21:31 |
| kgriffs | LOL | 21:32 |
| zyuan | "A programmer has a problem. He thought to himself, 'I know, I'll solve it with threads!' has Now problems. two he" | 21:32 |
| alcabrera | zyuan: lol | 21:33 |
| openstackgerrit | Alejandro Cabrera proposed a change to openstack/marconi: feat: connect sharding manager to control drivers https://review.openstack.org/54605 | 21:35 |
| *** mpanetta_ has quit IRC | 21:42 | |
| alcabrera | kgriffs: I'm headed home. I'll be back on later to resolve the remaining three errors. | 21:42 |
| alcabrera | dinner awaits. :D | 21:42 |
| kgriffs | ciao! | 21:42 |
| alcabrera | See you guys later. Thanks for all your help! | 21:42 |
| *** alcabrera has quit IRC | 21:43 | |
| openstackgerrit | Zhihao Yuan proposed a change to openstack/marconi: feat(shard): queue listing from multiple sources https://review.openstack.org/53127 | 21:44 |
| ametts | kgriffs: Do you know the 30,000 picture of the status of python-marconiclient? I was going to ask flaper87, but he got away before I got out of my meeting. Do we know what works and what doesn't work? | 21:50 |
| *** malini is now known as malini_afk | 22:09 | |
| *** fifieldt has joined #openstack-marconi | 22:15 | |
| kgriffs | ametts: I think all it does so far is CRUD for queues | 22:16 |
| kgriffs | not messages | 22:16 |
| kgriffs | let me check | 22:16 |
| kgriffs | yeah, looks like messages are not implemented | 22:18 |
| kgriffs | gotta run | 22:18 |
| *** amitgandhi has quit IRC | 22:18 | |
| ametts | Ok -- thanks. | 22:18 |
| *** amitgandhi has joined #openstack-marconi | 22:18 | |
| kgriffs | should be pretty quick to add now that all the plumbing is in place | 22:18 |
| kgriffs | ttfn | 22:19 |
| ametts | cio | 22:19 |
| ametts | ciao | 22:19 |
| *** amitgandhi has quit IRC | 22:20 | |
| *** amitgandhi has joined #openstack-marconi | 22:21 | |
| *** jcru has quit IRC | 22:22 | |
| *** amitgandhi has quit IRC | 22:26 | |
| *** kgriffs is now known as kgriffs_afk | 22:28 | |
| *** mpanetta has joined #openstack-marconi | 23:15 | |
| *** amitgandhi has joined #openstack-marconi | 23:17 | |
| *** mpanetta has quit IRC | 23:18 | |
| *** mpanetta has joined #openstack-marconi | 23:18 | |
| *** amitgandhi has quit IRC | 23:25 | |
| *** amitgandhi has joined #openstack-marconi | 23:26 | |
| *** malini_afk is now known as malini | 23:56 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!