Friday, 2013-09-20

*** reed has quit IRC00:05
*** reed has joined #openstack-marconi00:06
*** malini1 has joined #openstack-marconi00:10
*** malini1 has quit IRC00:16
*** malini1 has joined #openstack-marconi00:16
*** amitgandhi has quit IRC00:22
*** nosnos has joined #openstack-marconi00:23
*** oz_akan_ has joined #openstack-marconi01:00
*** kgriffs_afk is now known as kgriffs01:13
*** cthulhup has joined #openstack-marconi01:26
*** cthulhup has quit IRC01:30
*** kgriffs is now known as kgriffs_afk01:36
*** reed has quit IRC01:42
*** malini1 has quit IRC02:13
*** kgriffs_afk is now known as kgriffs02:14
*** kgriffs is now known as kgriffs_afk02:14
*** kgriffs_afk is now known as kgriffs02:15
*** oz_akan_ has quit IRC02:32
*** oz_akan_ has joined #openstack-marconi02:32
*** oz_akan_ has quit IRC02:37
*** kgriffs is now known as kgriffs_afk02:47
*** reed has joined #openstack-marconi02:57
*** AyoungTheRed is now known as ayoung-ZzZzZz03:05
*** oz_akan_ has joined #openstack-marconi03:33
*** oz_akan_ has quit IRC03:41
*** reed has quit IRC05:11
al-maisanLooking at https://wiki.openstack.org/wiki/Marconi/specs/api/v1#Endpoints_Synopsis -- are the parameters prefixed by '?' optional?05:26
*** rektide has joined #openstack-marconi05:54
al-maisanhmm .. by looking at the request examples I am getting the impression that all GET URL params are optional ..06:41
*** cthulhup has joined #openstack-marconi06:51
*** cthulhup has quit IRC06:55
*** gordonsim has joined #openstack-marconi07:22
*** flaper87|afk is now known as flaper8707:34
flaper87al-maisan: ping07:56
al-maisanflaper87: pong08:11
flaper87al-maisan: yo, got your email08:11
al-maisanOK08:11
flaper87al-maisan: it's very weird that Jenkins +1 that review08:11
flaper87keystoneclient is indeed missing08:12
flaper87:/08:12
al-maisanI believe I figured out what params are mandatory/optional in the mean time08:12
flaper87stupid jenkins08:12
al-maisanflaper87: yeah .. I was just puzzled08:12
* flaper87 kicks jenkins legs08:12
al-maisan:-)08:12
flaper87al-maisan: did you put those params in an etherpad or something?08:12
* flaper87 is curious08:12
al-maisanI will be updating the branch in a minute08:13
al-maisanflaper87: http://paste.openstack.org/show/47300/08:13
openstackgerritMuharem Hrnjadovic proposed a change to openstack/python-marconiclient: This adds a transport layer request spec class.  The idea here is to be declarative i.e. specify *what* is desired. It's up to the respective transport to turn a request spec into an appropriate request.  https://review.openstack.org/4744308:26
flaper87al-maisan: here's a wiki page w.r.t commit message and some guidelines we follow in OpenStack. https://wiki.openstack.org/wiki/GitCommitMessages08:27
al-maisanflaper87: thanks! I am violating all kinds of rules here :-P08:28
al-maisanbiab08:29
flaper87al-maisan: :D, kk08:30
*** oz_akan_ has joined #openstack-marconi08:50
*** oz_akan_ has joined #openstack-marconi08:50
*** torgomatic has quit IRC08:55
*** torgomatic has joined #openstack-marconi08:55
al-maisanflaper87: do we care about python26 test failures (due to 'TestRequest' object has no attribute 'assertIs')08:58
al-maisan?08:58
flaper87al-maisan: yup, we should be using testtools for that. This aptch here introduces a TestBase class09:00
flaper87al-maisan: https://review.openstack.org/#/c/46288/09:00
flaper87I'd suggest you to make your current work depend on this patch here: https://review.openstack.org/#/c/47169/09:00
flaper87which also depends on the TestBase one09:00
flaper87the last patch cahnges Marconiclient's structure to reflect Marconi's09:00
al-maisanhow do I do that? " make your current work depend on this patch here"?09:01
flaper87to make your patch depend on that patch, you just need to rebase your local branch with the changes in that review09:01
flaper87hehehe09:01
al-maisanbranch from your branch?09:01
al-maisanabandon the patch set, branch from your branch and re-upload to reviews...?09:02
flaper87al-maisan: you can do git review -d PATCH_ID, that will create a new branch with the changes in the review you want to depend on09:02
al-maisanOK09:02
flaper87al-maisan: then you can cherry-pick your commit into that new branch09:02
flaper87and then git review09:02
flaper87git review will ask you if the dependency chain is correct before submitting it09:03
al-maisanflaper87: do I need to pass any special params to "git review"09:26
al-maisanalso, the rebase prior to "git review" should be with respect to master or your "change-marconi-structure" branch?09:27
al-maisanflaper87: this is what I see when trying a "git review" : http://paste.openstack.org/show/47301/09:28
* flaper87 looking09:31
flaper87no params needed09:31
flaper87al-maisan: that looks correct to me09:31
flaper87just write yes and you're done09:31
al-maisanso, I should just say 'yes' then?09:31
*** leizhang has joined #openstack-marconi09:32
flaper87btw, no '.' needed at the end of the commit 'title'09:32
al-maisanOK09:32
al-maisanI was afraid I'd create an avalanche of reviews again :-P09:32
flaper87al-maisan: yeah, also, if you use git review -y it'll submit everything without asking you09:32
flaper87al-maisan: LOL :D09:32
al-maisan:)09:32
al-maisanOK09:32
al-maisanthen09:32
flaper87al-maisan: it checks if there's a review for each commit's change-id09:32
al-maisancool09:33
* al-maisan pulls the trigger09:33
openstackgerritMuharem Hrnjadovic proposed a change to openstack/python-marconiclient: This adds a transport layer request spec class  https://review.openstack.org/4753809:33
al-maisanpheewwww .. only *one* review :-P09:33
flaper87al-maisan: COOOOOOL, thanks for contributing :D I'll take a look in a bit09:33
* flaper87 creates a new OS acronym09:34
al-maisanthanks!09:34
flaper87tlab09:34
flaper87take a look in a bit09:34
flaper87w000t09:34
al-maisannice one :)09:34
flaper87I admit it, I first wrote itlab but then I thought that apple may sue me if I dare to use an 'i' in front of anything, so....09:35
al-maisanhe-he :)09:35
al-maisanjenkins is happy :) good!09:36
* al-maisan takes a break09:38
al-maisansee ya later in the afternoon flaper87 !09:38
flaper87al-maisan: kk, thanks! ttyl09:38
openstackgerritMuharem Hrnjadovic proposed a change to openstack/python-marconiclient: This adds a transport layer request spec class  https://review.openstack.org/4753809:43
*** leizhang has quit IRC09:56
*** leizhang has joined #openstack-marconi09:56
*** nosnos has quit IRC10:17
*** oz_akan_ has quit IRC12:11
*** oz_akan_ has joined #openstack-marconi12:12
*** cthulhup has joined #openstack-marconi12:16
*** oz_akan_ has quit IRC12:16
*** cthulhup has quit IRC12:20
*** tedross has joined #openstack-marconi12:38
*** cppcabrera has joined #openstack-marconi12:40
cppcabreraMorning~ :D12:42
*** kgriffs_afk is now known as kgriffs12:44
flaper87cppcabrera: goood morning12:52
flaper87:D12:52
flaper87cppcabrera: how are you doing?12:52
flaper87cppcabrera: btw, didn't mean to be a PITA with the request of breaking that patch into several12:52
*** oz_akan_ has joined #openstack-marconi12:54
cppcabrerano worries, flaper87. I'm happy to help with making the review process easier. :)12:54
cppcabreraI'm doing pretty awesome this morning, though a little on the hungry side. How about you?12:55
flaper87I'm doing well, already caught up with email, bugs and some reviews. I'm going through the proxy patches now12:56
cppcabrerawoot! I've got some fixes to apply to the second one, namely, it parses the config file twice when running the test suites.12:58
flaper87oh, ok. I'll keep that in mind. I'm reviewing gozilla now12:58
cppcabreralol12:59
openstackgerritMuharem Hrnjadovic proposed a change to openstack/python-marconiclient: This adds a transport layer request spec class  https://review.openstack.org/4753813:00
al-maisanflaper87: this is the fix I mentioned ^13:00
*** amitgandhi has joined #openstack-marconi13:04
flaper87al-maisan: awesome, thanks! tlib :D13:05
flaper87btw guys, al-maisan is helping us out with marconiclient, we had a great brainstorm session yday13:05
flaper87w00000t13:05
kgriffsawesome sauce13:05
kgriffsflaper87: I'm reimplementing my changes to messages.py13:05
kgriffsnot sure where they went13:06
kgriffs:p13:06
al-maisan:)13:06
flaper87kgriffs: LOOOOOOOOL13:06
al-maisanuh-oh13:06
kgriffs#git-fail13:06
kgriffs#user-error13:06
kgriffsflaper87: gozilla? I thought we were writing the proxy in Python. ;)13:07
* cppcabrera almost caught up with morning email13:07
flaper87kgriffs: btw, not sure if you noticed the comment I left on the left side of the diff (in messages.py)13:07
al-maisankgriffs: take it easy -- git is a bit of a beast .. it's very easy to make errors :-P13:07
flaper87kgriffs: we are, it just ate Godzilla13:08
*** jergerber has joined #openstack-marconi13:09
al-maisanflaper87: I thought gozilla is a thinly veiled reference to rust ;-)13:09
flaper87not sure if it is godzilla that has a Python skin or it is python with godzilla's shape13:09
flaper87mmhh13:09
kgriffsgo-zilla13:09
flaper87LOOOOL13:09
kgriffspyzilla13:09
al-maisanyeah ;)13:09
kgriffs:[13:09
flaper87rustzilla13:09
al-maisanhmm ..13:10
kgriffsnow ur talkin!13:10
kgriffsflaper87: re that other comment, let me explain13:10
kgriffswhile bulk inserts are not atomic, they are inserted in order, and as soon as one fails, the rest are abandoned13:11
* flaper87 reads carefully O.O13:11
kgriffsin the case of a collision, it will always happen on the first message document in the list13:11
* kgriffs thinks about that to make sure it is correct13:11
kgriffsi'm pretty sure that is correct13:12
flaper87mmh, I think that's correct as well13:12
kgriffssince the counter is not incremented until batch A is in13:12
flaper87very well thought13:12
flaper87have you had some sleep lately?13:13
flaper87:D13:13
kgriffsbatch B cannot proceed until request A increments the counter13:13
kgriffsflaper87: heh, not enough!13:13
flaper87i can tell ^^13:13
flaper87aaaaaaanyway, I agree with you13:13
kgriffsFWIW, I tested this with mongo and indeed, thats the behavior13:13
flaper87it wasn't obvious to me when I reviewed the patch13:14
flaper87cool13:14
* flaper87 so happy13:14
kgriffsbut, I am glad that you are considering the patch carefully; I'm still worried that there's a race condition or something I missed.13:14
flaper87we can remove that regex-based error interpretation utility13:14
kgriffs+1 zillion13:14
kgriffslet me do that13:14
kgriffsflaper87: btw - https://review.openstack.org/#/c/46757/13:16
kgriffsflaper87: also - https://review.openstack.org/#/c/47482/13:17
flaper87kgriffs: +113:17
cppcabrerafalcon 0.1.7~13:17
kgriffsthat last one was driving me crazy yesterday13:17
* kgriffs had to fix it13:17
flaper87kgriffs: approved13:18
cppcabrera#python_gotchas13:18
kgriffsw00t13:18
kgriffsat first i thought i was experiencing a time-space anomaly. then I realized it was just Python.13:19
cppcabreraPython produces space time anomalies - this is now known.13:19
flaper87ROFL13:19
al-maisan:)13:20
*** malini1 has joined #openstack-marconi13:22
kgriffs"Our continuing mission: To boldly go where no developer has gone before!"13:23
*** malini1 has quit IRC13:23
kgriffscppcabrera: be sure to make an entry of this incident in the logs so future explorers know to steer clear of this part of the galaxy.13:23
* kgriffs jk13:24
* kgriffs shuts up and gets back to work13:24
*** malini has joined #openstack-marconi13:24
cppcabreralol13:24
cppcabreraWhich reminds me that I need to finish watching Star Trek: NG at some point. :P13:25
* kgriffs after that last joke, I think people are going to steer clear of this IRC channel13:25
cppcabreraThey13:25
cppcabrera'll return for the space-time anomalies.13:25
* cppcabrera goes to have breakfast13:26
cppcabrerabrb13:26
kgriffswell, if our project gets canceled, we can always look forward to Marconi: The Next Generation13:26
* kgriffs *really* needs to get more sleep13:27
cppcabreralol13:28
cppcabrerarelevant: http://goo.gl/ux78UI13:29
kgriffsdouble-relevant: http://pictr.io/pic/5768226071379683792_0_20768900.jpg13:31
kgriffsROFL13:32
cppcabrerahahaha13:33
cppcabrerasuch an interesting service, pictr.io. Love == incr_TTL, Hate == decr_TTL.13:34
* kgriffs loves this irc channel13:34
al-maisanwhere do you guys find all these pictures?13:39
al-maisan:)13:39
kgriffswe work for the NSA13:39
kgriffsyou wouldn't *believe* what we have access to13:39
al-maisanaaaaaaaaaaaaaah .. I understand now ;)13:39
al-maisanso, Marconi is NSA-scale then?13:40
kgriffsI13:42
kgriffsI13:42
kgriffsI can neither confirm nor deny that.13:42
flaper87LOOOOL at double-relevant13:42
flaper87lol, lol, lol, lol13:42
flaper87btw, isn't it hackday at RAX ?13:43
flaper87what are you guys hacking on?13:43
amitgandhithats next week13:43
amitgandhilast friday of every month13:43
flaper87ah, I thought it was every week13:43
flaper87ah ok13:43
flaper87ok13:43
* flaper87 STFU13:43
amitgandhiyou should join us13:43
amitgandhiguest racker haha13:43
* flaper87 stops prettending he knows RAX that well13:43
flaper87pretending*13:43
flaper87amitgandhi: I will work on a rust-marconiclient13:44
flaper87done, I said it!13:44
kgriffsw00t13:44
al-maisankgriffs: deny it the Linus way: http://linux.slashdot.org/story/13/09/19/0227238/linus-torvalds-admits-hes-been-asked-to-insert-backdoor-into-linux13:44
flaper87but please, make sure to remind that to me next friday, otherwise I'll start working on a brain-control algorithm13:45
* al-maisan starts an elixir-marconiclient :)13:45
cppcabreraall the marconiclients13:46
amitgandhirust looks interesting13:46
flaper87al-maisan: +113:46
cppcabrerathere will be no shortage13:46
cppcabreraof marconi flavords13:46
cppcabrera*flavors13:46
al-maisanyeah :)13:46
flaper87amitgandhi: rust is not just interesting, it is damn sexy13:46
cppcabrera+1 flaper8713:47
kgriffsal-maisan: Linus +113:47
al-maisanflaper87: never say that in front of your girl friend ;)13:47
cppcabrerawhich reminds me, I want to start hanging out in the rust IRC channel. :P13:47
flaper87al-maisan: I did, she just told me: As long as you're not talking about another girl, that's fine with me13:48
flaper87cppcabrera: irc.mozilla.something13:48
cppcabrera.org13:48
cppcabreraI just looked it up. :D13:48
flaper87cppcabrera: that :D13:48
*** russellb is now known as rustlebee13:48
cppcabreraI get a daily dose of haskell on freenode#haskell (it's so active!), might as well get a dose of rustlang. :)13:48
flaper87cppcabrera: rust's channel is very active13:49
al-maisanflaper87: pheww .. you were just plain lucky :)13:50
* al-maisan googles six.add_metaclass(abc.ABCMeta)13:51
cppcabrera+1 al-maisan13:54
al-maisancppcabrera: thank you very much !!13:54
*** oz_akan_ has quit IRC13:55
*** oz_akan_ has joined #openstack-marconi13:55
openstackgerritAlejandro Cabrera proposed a change to openstack/marconi: proxy: adds oslo.cache, config support, & stevedore  https://review.openstack.org/4747514:07
cppcabrerathat should fix the unit test config loading bug. ^^14:09
flaper87kgriffs: when you get a chance, could you pls take a look at marconiclient patches?14:20
*** ayoung-ZzZzZz is now known as ayoung14:21
*** ayoung has quit IRC14:21
al-maisanflaper87: in marconiclient/auth/keystone.py, line 91 you describe 'kwargs' as a param but it's a local dict introduced later in the method body14:23
flaper87al-maisan: erm, most likely left over from ceilometers code14:24
flaper87o.0 did I say that?14:24
flaper87:D14:24
flaper87al-maisan: I'll fix that14:24
flaper87al-maisan: pls, -1 the patch with a comment14:24
al-maisanlet me look through the rest14:24
al-maisanflaper87: can you please explain why authenticate() needs to be an abstract method?14:25
al-maisanbecause there is no meaningful default implementation?14:25
zyuanpls review https://review.openstack.org/#/c/45952/14:25
zyuanthis is done14:26
flaper87al-maisan: yes but, I was thinking about that this morning. I would like to have a noop implementation that just returns the transport obj we talked about14:26
zyuanflaper87: how to add dependecy to openstack?14:26
zyuanwe need pylru and mmh314:27
flaper87al-maisan: so, I guess we could either let it land like that and fix implement the noop implementation later or do it in that same patch14:27
flaper87zyuan: add it to global-requirements.txt in openstack/requirements14:27
flaper87github.com/openstack/requirements14:27
al-maisanflaper87: is `request` in marconiclient/auth/keystone.py, line 86 meant to be an instance of `transport.request.Spec` ?14:27
al-maisanI am just trying to understand how all these pieces will fit together..14:28
zyuanflaper87: where is gerrit of this?14:28
flaper87zyuan: https://review.openstack.org/#/q/status:open+project:openstack/requirements,n,z14:29
zyuanthanks14:29
flaper87al-maisan: yeah, me too :P14:29
flaper87al-maisan: so, I think I'll make that patch depend on yours14:30
flaper87I think we shouldn't pass a request there and instead, let the auth implementation return one14:31
flaper87I'm thinking about making that work pretty much as a middleware chain would14:32
flaper87al-maisan: ^14:32
flaper87auth[returns req-spec]->lower_api->transport14:32
flaper87or something like that14:32
flaper87al-maisan: mmh, actually, I think it might be worth passing the request spec14:34
flaper87to the auth method14:34
al-maisanyeah .. it's a pass through .. auth just adds the auth data14:34
al-maisanand then it goes to the transport14:35
flaper87al-maisan: plus, I think keystone's v3 will have RBAC as well, so it'll need to know what ops the user is trying to do14:35
flaper87ok14:35
al-maisanyup .. so passing the request spec to the auth layer make sense IMHO14:36
flaper87al-maisan: btw, how are you thinking to differentiate headers from query-string params?14:36
flaper87I think we should have 2 separate dicts for those14:36
al-maisanisn't that something the transport needs to sort out? What goes where?14:36
al-maisanthe request spec is general and does not know about headers14:37
flaper87al-maisan: mmh, don't think so, for example:14:37
flaper87if we have 2 different auth providers, they'll most likely use different headers parameters. If we let the transport to sort which param is a header and which is a query param, we'll have to map them somewhere14:38
flaper87does that make sense?14:38
flaper87I agree that the impl needs to be generic14:38
flaper87I mean, it should just care about params, content and endpoints14:39
flaper87but, I think having them in separate structures will make transport's implementation cleaner and easier14:39
al-maisanflaper87: is there a situation where we have more than one auth provider per transport? e.g. hhtp?14:39
kgriffsflaper87: question - http://paste.openstack.org/show/47318/14:39
al-maisan*http even14:39
kgriffsdo you think we can remove the $or there since c.e is always set/initialized on post message?14:40
flaper87al-maisan: there could be, I guess. But I'm thinking about the client having to talk to 2 different endpoints that use different auth providers.14:40
flaper87al-maisan: let me try to explain this better14:40
al-maisanwould the client talk to 2 different endpoints simultaneously?14:41
al-maisanflaper87: we can also do a hangout session if that helps..14:41
flaper87yeah, lets do that14:42
al-maisangive me 2 minutes14:42
flaper87kgriffs: looking14:43
zyuanhttps://review.openstack.org/#/c/47580/14:44
flaper87kgriffs: ah yes, I was looking at that yday while reviewing your patch14:44
flaper87I think we can get rid of that or14:44
flaper87kgriffs: is c.e set to now on post message?14:44
flaper87'now'14:45
al-maisanflaper87: https://plus.google.com/hangouts/_/ca7fe30c69c2c9cf70e7b193dc0d91d375bec5cb?hl=en14:45
flaper87zyuan: I think you'll need 2 separte patches for that14:45
al-maisanready whenever you are14:45
flaper87al-maisan: can you invite me through G+? I'd have to send the link to my ipad otherwise14:46
al-maisanOK14:46
* flaper87 can't G+ on his laptop14:46
kgriffsflaper87: c': {'id': None, 'e': now},14:46
kgriffsyes14:46
flaper87kgriffs: cool, I guess we can get rid of that $or14:46
kgriffsexcellent14:46
kgriffsI will do that in another patch14:47
* flaper87 smiles while imagining those queries going faster14:47
al-maisanThank God for G+ hangout :)14:56
flaper87yeaaahhh14:58
zyuanGoogle's religion?14:58
flaper87we figured the whole thing in 5 seconds14:58
openstackgerritKurt Griffiths proposed a change to openstack/marconi: perf(mongodb): Combine project and queue message fields  https://review.openstack.org/4725914:58
openstackgerritKurt Griffiths proposed a change to openstack/marconi: fix(mongodb): Marker generation creates a bottleneck under heavy load  https://review.openstack.org/4727214:58
al-maisanflaper87: yeah :)14:59
openstackgerritMuharem Hrnjadovic proposed a change to openstack/python-marconiclient: This adds a transport layer request spec class  https://review.openstack.org/4753815:01
al-maisanflaper87: done ^15:01
* al-maisan signs off15:02
flaper87al-maisan: danke!15:02
al-maisanflaper87: di nulla :)15:03
cppcabreraTime to do some reviews. :)15:04
*** ayoung has joined #openstack-marconi15:04
flaper87cppcabrera: ignore my auth patch for the client, I'll update it in a bit based on al-maisan changes15:04
cppcabrerakk, flaper87.15:05
cppcabreraI'll check out kgriff's patches first.15:05
flaper87cppcabrera: +115:05
flaper87http://www.pixelbeat.org/patches/coreutils/inbox_aug_2013.html#137929862215:07
*** reed has joined #openstack-marconi15:18
openstackgerritKurt Griffiths proposed a change to openstack/marconi: fix(mongodb): Remove $or clause used to claim messages  https://review.openstack.org/4759015:21
* flaper87 can see the smoke comming out kgriffs's keyboard15:25
kgriffs:D15:26
kgriffsflaper87: I am looking at http://goo.gl/aVGVtg15:27
kgriffsgrep for "if isinstance(value, datetime.datetime):"15:28
flaper87line 429 ?15:28
kgriffsat one point there used to be a bson.datetime15:28
kgriffsright15:28
cppcabrerakgriffs: all patches reviewed.15:28
cppcabreraThe marker simplification patch was -1'd because of a potential 'TypeError'.15:29
cppcabreraThe rest are +2s, with one awaiting benchmarking.15:29
kgriffsi was thinking, why not have bson.date class which can be constructed from a unix timestamp or default to current time, or constructed from a datetime.datetime15:29
kgriffsinternally, it would just store millis15:30
kgriffsusing calendar is lame and slow15:30
zyuancppcabrera: http://paste.openstack.org/show/47322/15:30
kgriffsimo15:30
kgriffsflaper87: thoughts?15:30
zyuancppcabrera: with that diff, tests pass (so far)15:30
kgriffscppcabrera: looking15:31
flaper87kgriffs: I agree with both the idea and calendar being lame and slow15:31
cppcabrerathanks, zyuan.15:32
cppcabreraI'll go fix up the patch. :)15:32
kgriffscppcabrera: nice catch there15:32
* kgriffs fixes that15:32
kgriffsbtw, i found a nice way to deal with dependent patches15:32
*** openstackgerrit has quit IRC15:33
kgriffsmake a change, commit15:33
*** openstackgerrit has joined #openstack-marconi15:33
kgriffsthen rebase, and reorder+squash into the correct commit15:33
kgriffsas long as you preserve the change ID, ur good15:33
cppcabreraI've also had good luck preserving the change ID. I go a step further and usually 'git review -d' the dependency and rebase off of that,15:34
openstackgerritKurt Griffiths proposed a change to openstack/marconi: perf(mongodb): Combine project and queue message fields  https://review.openstack.org/4725915:38
openstackgerritKurt Griffiths proposed a change to openstack/marconi: fix(mongodb): Marker generation creates a bottleneck under heavy load  https://review.openstack.org/4727215:38
kgriffscppcabrera: fixed those issues you noted ^^^15:39
cppcabrera+2'd15:40
* kgriffs cheers15:40
zyuankgriffs: irrc, '-' is allowed in queue name15:41
zyuanuse '/' instead15:41
kgriffshmm, good point15:42
kgriffshow about ':'15:42
flaper87kgriffs: +1 for :15:42
zyuan'/' is never be used15:42
zyuanallowed*15:42
flaper87lets use :: as Rust15:42
flaper87hahahah15:42
kgriffs::15:42
zyuan...15:42
flaper87just kiding15:42
zyuanproject/queue15:43
flaper87zyuan: mmh '...' is a bit long, though15:43
zyuanand just queue when project is None15:43
flaper87:D15:43
zyuanproject/queue and queue15:43
cppcabreralol15:43
cppcabrera:: is tempting because I already think of projects as namespaces.15:44
flaper87either work for me, as long as you don't pick 'win-php' namespace style, I'm good15:44
cppcabreraBut it's not worth the extra character, heh.15:44
zyuani jsut pick the char will *never* be used15:44
flaper87I guess / is fine15:44
zyuanno matter how we change API in the future15:44
zyuanand prject/queue and queue will never clush since neither can contains '/'15:46
cppcabreramarconi proxy is introducing a few config file additions, namely: [drivers:proxy], [drivers:proxy:storage:{type}], and [drivers:proxy:transport:{type}]. How do you guys feel about this?15:46
kgriffshmm15:46
flaper87zyuan: ah no? What if we start using unix-paths as queues names15:46
* flaper87 runs runs away15:46
zyuanflaper87: you can't15:46
flaper87zyuan: duuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuude, it's a joooooooooooooooooooooooooooooooooookeeee15:46
kgriffsi'm ok with just 'queue' when no project, but only if we think that folks won't mix single-tenant and multitenant modes15:46
zyuanwe have no way to add this "feature"15:46
flaper87:D15:47
kgriffsas in, all or nothing15:47
cppcabreraI'm sure we'll find all the edge cases when people *can't* stop using marconi. ;)15:47
flaper87mmh, but, wouldn't that mess up the "all queues for this project" query ?15:47
flaper87I'd use None/queue15:48
zyuancppcabrera: lengthy... you mentioned that you want a separated config file right?15:48
flaper87plus, what if a queue in tenant None has the same name of an existing project?15:48
zyuanNone/queue..............................15:48
kgriffsflaper87: good point, we want to be able to do a prefix query some day15:48
flaper87but also, what if a project is called None15:48
flaper87:P15:48
zyuani can create a project named "None"15:48
flaper87zyuan: I won :D15:48
zyuan...15:48
zyuanqueue along works15:49
kgriffshmm15:49
flaper87project/queue and /queue I guess15:49
cppcabrerazyuan: I'm considering a separate config file, but I'm not entirely sure how to trigger that.15:49
kgriffsif there is only ONE project (single tenant) then it is fine15:49
flaper87that would allow us to say "everything after the /"15:49
zyuanwhat if i creae a project named ""?15:49
zyuanthen "/queue" clush with "/queue"15:50
cppcabreraMy best guess is that it's determined when you do config.project('marconi_proxy')15:50
flaper87zyuan: project is not something we create15:50
zyuanflaper87: it can be anothing in test15:50
kgriffsbut if you allow project=None as well as project=1234 in the same DB, things aren't as nice15:50
flaper87but anyway, project "" == no project to me15:50
zyuanso i don't want our code rely on some external facts15:50
flaper87zyuan: agreed about not relying on external things.15:51
kgriffs_/my-queue15:51
zyuana project alled "_'15:51
kgriffs104854/bobs_queue15:51
kgriffsin practice, won't happen15:51
kgriffsproject IDs are numbers, uuids, stuff like that15:51
zyuan"$queue" does not depends on practice to work15:51
zyuanit just works and always works...15:52
flaper87to me no project is like project without name15:52
kgriffshmm15:52
flaper87and we can use projects id, btw15:53
kgriffsso to ask the question "what are all the global queues" we would have to say "give me everything that does not contain '/'15:53
kgriffsthe trouble with that in Mongo is you can't use an index15:53
flaper87kgriffs: nope15:53
flaper87prj/queue or /queue15:53
zyuani'm ok with "/queue"15:53
flaper87so, give me  ^/15:53
kgriffsflaper87: right, we would need the prefix there15:53
zyuanthere is actualy a bug in the current code which does not allow a project of ""15:54
zyuanbut it's arguable15:54
zyuani'm fine with "project/queue" and "/queue"15:54
kgriffssure, i just don't want to make schema decision that reduces our options later on15:54
kgriffsI kinda like ^/my-queue15:54
kgriffsto tell you the truth15:54
kgriffswe are picking nits now15:55
kgriffsmain thing is to use /15:55
kgriffsstand by15:55
zyuananother possibilty is to use15:55
zyuan/project/queue15:55
zyuanand /queue15:55
flaper87but then again, if there's a project called /, that query will fail, we need something like ^/\w15:55
zyuanthat looks more consistent15:55
kgriffsbtw, should i treat the empty string as the same as None?15:55
flaper87kgriffs: yes15:55
flaper87IMHO15:56
zyuanotherwise, project/queue and queue15:56
zyuankgriffs: yes and no15:56
flaper87I'm fine with either /prj/queue and /queue or prj/queue and /queue15:56
flaper87I agree with zyuan, /prj/queue is more consistent15:56
zyuankgriffs: i'm ok with that, but there might be some problem in our code15:56
zyuankgriffs: i'm None and "" stuff15:57
flaper87but, dunno if that makes the query more complex, lets make sure we still use the index with the regex15:57
zyuani mean...15:57
zyuanprj/queue is little bit better than /prj/queue; less similarity, more fair distribution15:57
zyuan(i'm going to use the name directly in multidb)15:58
zyuanfor same reason, 'queue' may works better than '/queue', but we might not care about single tenant case that much15:59
zyuanit's a relatively more "private15:59
zyuan" deploment.15:59
cppcabreraultimately, we're trying to (idx :: String -> String -> String; idx project queue = project ++ '/' ++ queue), so as long as the chosen separator character is in the set of disallowed charcaters, I think it's enough to just use that as a separator.15:59
cppcabrera'/' is one of the disallowed charcaters, IIRC, so that's cool with me.16:00
zyuanidx None queue = queue16:00
zyuanidx project queue = project ++ "/" ++ queue16:01
cppcabreraI've been using '.' while prototyping, fwiw.16:01
zyuananyway, we can vote16:01
flaper87'.' is forbidden by mongo16:01
cppcabreragtk, flaper8716:01
zyuani prefer to distinguish None and "" as much as I can16:02
zyuan:)16:02
kgriffshttps://etherpad.openstack.org/queuing-scratch16:05
kgriffsI still like ^16:05
cppcabrera /queues16:12
flaper87kgriffs: so, just to be clear, I'll create a bug to forbid empty-names for projects16:14
flaper87and you'll update your patch with what we discussed in the etherpad16:15
flaper87right?16:15
kgriffsflaper87: right16:16
flaper87kk16:16
flaper87danke16:16
flaper87doing it now16:16
kgriffsalso, maybe register a bp for single-tenant mode?16:17
flaper87not sure, I don't think we care much about that. it's a matter of whether project is being passed or not16:17
kgriffshmm16:17
flaper87do we need to distinguish that ?16:17
kgriffsidk what the WSGI driver does if Project-Id is missing16:18
kgriffsI think it complains16:18
zyuankgriffs: it does not16:18
flaper87kgriffs: nope, https://github.com/openstack/marconi/blob/master/marconi/queues/transport/wsgi/driver.py#L5416:19
kgriffsah, so it would pass through None16:19
zyuanyes16:19
zyuanand!16:19
kgriffsdo we have tests covering that?16:19
zyuanmongo driver allows ""16:19
zyuansqlite driver allows "" but clush with None16:20
zyuanthat's what i said "our impl does not have an consensus on this"16:20
flaper87https://bugs.launchpad.net/marconi/+bug/122824916:22
flaper87brb16:23
zyuanrequirements freze16:23
zyuanhehe16:23
malinihmm..what is a project name?16:25
openstackgerritAlejandro Cabrera proposed a change to openstack/marconi: proxy: memory storage driver + tests + fixes  https://review.openstack.org/4739916:26
kgriffsmalini: project ID16:26
cppcabrera^^ removes normalization at transport layer for proxy16:27
openstackgerritAlejandro Cabrera proposed a change to openstack/marconi: proxy: mongodb storage driver  https://review.openstack.org/4740216:27
openstackgerritAlejandro Cabrera proposed a change to openstack/marconi: proxy: adds oslo.cache, config support, & stevedore  https://review.openstack.org/4747516:27
cppcabrerarebase ^^16:28
openstackgerritKurt Griffiths proposed a change to openstack/marconi: perf(mongodb): Combine project and queue message fields  https://review.openstack.org/4725916:36
openstackgerritKurt Griffiths proposed a change to openstack/marconi: fix(mongodb): Marker generation creates a bottleneck under heavy load  https://review.openstack.org/4727216:36
cppcabrerareviewed, kgriffs: +2'd on both.16:40
cppcabreraFinding the small changes was made possible by two browser tabs and gerrit math. :P16:40
kgriffsflaper87: ping16:52
flaper87kgriffs: pong17:02
cppcabreraseeing enough pings and pongs, I start to think that we've got more or less an ad-hoc heartbeat system going in this channel.17:05
kgriffscommented: https://review.openstack.org/#/c/46757/17:09
kgriffsd00d, whenever i ask keystone questions, I just get crickets17:09
kgriffsi was redirected to #openstack-dev from #openstack-keystone17:10
kgriffsstill now luck17:10
kgriffsbooh17:10
kgriffsI'll try the mailing list17:11
cppcabreraI've almost got the proxy transport restructured to mirror marconi.queues17:13
cppcabreraI just finished duking it out with stevedore over my tiny errors. :D17:13
cppcabrerawoooo, I got it obeying the config file!17:14
cppcabreraLet's see what tox says...17:14
kgriffsflaper87, cppcabrera: thanks for reviewing my comment on that reqs patch17:16
cppcabreranp, kgriffs.17:16
cppcabrerawhenever you have time, proxy needs some love. :)17:17
cppcabreraI'm currently finalizing the last patch needed before the proxy is featureful enough to work with at scale.17:17
flaper87kgriffs: thank you for those lovely patches.17:18
zyuankgriffs: openstack requirements is frozen17:19
zyuandoes not mean we can't get the multidb patch merged?17:19
* flaper87 should stop talking about code as it was a human being17:19
zyuanif not, can we deploy it?17:19
kgriffsflaper87: :D17:19
flaper87zyuan: no req, no land :(17:20
kgriffsAlex_Gaynor is on the reqs team - maybe he can give you some direction on that?17:20
flaper87jenkins will fail if the requirements don't match17:20
flaper87so the patch can't land17:20
zyuanPatch Set 3: I would prefer that you didn't merge this17:20
zyuanWe're in a requirements freeze until development on icehouse opens up.17:20
zyuanhttp://lists.openstack.org/pipermail/openstack-dev/2013-September/015131.html17:20
zyuanhttps://review.openstack.org/#/c/47580/17:21
flaper87yeah, i know! I guess we'll have to wait until havana is released and Ith developemnt starts17:22
flaper87at least for that patch to land17:22
flaper87man, what's wrong with jenkins17:22
flaper87?17:22
flaper87T_T17:22
cppcabrerais jenkins grumping again?17:23
* cppcabrera checks emails17:23
flaper87yeah17:23
flaper87at least it's really slow17:23
cppcabreraAhh, I see.17:23
* cppcabrera checks -infra17:23
flaper87erm, and grumping, I guess17:24
flaper87:D17:24
cppcabrerazuul was restarted.17:24
cppcabreraAlso, I happened upon this...17:24
cppcabreraAdd Python 3.3 as Experimental for Oslo-Incubator  https://review.openstack.org/4717017:24
cppcabreraif oslo.incubator gets 3-ized, that's amazing news for the rest of OS.17:25
flaper87cppcabrera: there's been a lot of work in that area17:25
flaper87so, I wouldn't be surprissed if the gate is made voting soon.17:25
flaper87kgriffs: jenkins didn't like your patch :(17:26
flaper87brb17:26
cppcabreraflaper87: It looks like it's going to be made voting, but experimental.17:26
cppcabreraSo you'd have to leave a comment of 'check experimental' to trigger it.17:26
cppcabrerakgriffs: message_scope isn't defined yet in your markers patch, so jenkins is upset about this.17:29
zyuanflaper87: does that mean falcon>0.1.7 can't go as well17:31
zyuan?17:31
cppcabreraI suspect that is the case, zyuan. We may have to wait on falcon>=0.1.7 'til after Icehouse begins.17:36
zyuancppcabrera: hehe17:38
zyuancppcabrera: you saw this right? https://github.com/racker/falcon/issues/18517:38
cppcabrerayup, zyuan. :P17:39
zyuanthen what do you want to do with it?17:40
cppcabreraMy thoughts are to work with it as it is, atm.17:41
cppcabreramiddleware can be used to normalize headers, if necessary.17:41
zyuanthen proxy gives wrong results17:41
zyuanit can't, because they can conflict17:41
zyuanthey have different values17:41
zyuancppcabrera: try this17:42
zyuan'-'.join([s.capitalize() for s in a.split('-')])17:42
zyuanon requests.headers17:43
kgriffscppcabrera: crappies, my rebase went awry17:49
*** cthulhup has joined #openstack-marconi17:51
*** cthulhup has quit IRC17:51
*** cthulhup has joined #openstack-marconi17:52
*** reed has quit IRC17:56
openstackgerritZhihao Yuan proposed a change to openstack/marconi: test(proxy): fix proxy-transport tests  https://review.openstack.org/4761217:57
*** cthulhup has quit IRC17:58
amitgandhikgriffs: meeting18:09
openstackgerritKurt Griffiths proposed a change to openstack/marconi: perf(mongodb): Combine project and queue message fields  https://review.openstack.org/4725918:30
openstackgerritKurt Griffiths proposed a change to openstack/marconi: fix(mongodb): Marker generation creates a bottleneck under heavy load  https://review.openstack.org/4727218:30
cppcabrerare-reviewing, kgriffs18:31
kgriffsoz_akan_: can you take a look at this version patch?18:45
kgriffshttps://review.openstack.org/#/c/47459/18:45
kgriffs+1 if it does what you need18:45
openstackgerritZhihao Yuan proposed a change to openstack/marconi: test(proxy): fix proxy-transport tests  https://review.openstack.org/4761218:46
oz_akan_kgriffs: on the phone, will check after18:48
kgriffskk, thanks18:48
*** leizhang has quit IRC18:55
*** mpanetta has joined #openstack-marconi18:57
mpanettaQuiet...18:58
kgriffsToo quiet...19:01
cppcabrera:D19:01
ekarlsohow's progress marconi folks ?19:05
kgriffsyoyo19:06
cppcabrerait's fun times, ekarlso19:06
cppcabrerabusy as ever19:06
kgriffswe are finishing up the last major patches for a "1.0" release. There will still be minor bugs and stuff, but all the big stuff will be good to go.19:07
ekarlsowill there be other drivers then mongo btw ?19:07
* ekarlso stares wit hevil eyes at GPL19:07
kgriffsyes19:07
kgriffslooking like SQLAlchemy and Redis19:08
kgriffswon't use the SQLAlchemy ORM19:08
kgriffs(probably)19:08
kgriffs(fwiw)19:08
ekarlsowhat's the bad with the orm ? :p19:08
kgriffsperformance19:09
ekarlsoah19:09
kgriffsand honestly, there are only a handful of docs, so the extra boilerplate of defining objects doesn't seem justified19:10
kgriffsbut I can be persuaded otherwise19:10
* kgriffs likes brownies19:10
ekarlsowhat's the bad thing with sqlalchemy docs ?19:10
* kgriffs and pop-tarts19:10
kgriffssorry, not docs as in "documentation"19:10
kgriffsdoc as in dict or resource or document19:11
ekarlsok19:11
* kgriffs has document DBs on the brain19:11
ekarlsocassandra ?19:12
kgriffsif someone would like to write a 3rd-party driver for it, be my quest, but TBH it isn't well-suited to storing lots and lots of short-lived messages19:13
kgriffsCassandra slows down quite a bit when you keep deleting things19:13
zyuanbtw, ykaplan is going to write sqlalchemy storage right? progress?19:13
kgriffsekarlso: at least, that was Comcast's experience19:14
kgriffsthey wrote an SQS/SNS clone on top of Cassandra and ran into all kinds of problems19:14
kgriffs…but maybe there's a Cassandra hacker who could make it work19:15
* kgriffs isn't one19:15
ekarlsokgriffs: just that GPL is evil though19:16
kgriffsYou mean AGPL19:16
kgriffs?19:16
kgriffsguys: I gotta run for bit19:17
cppcabrerasee ya, kgriffs19:18
ekarlsokgriffs: yes :p19:18
kgriffsekarlso: i totally understand. definitely some angst there19:18
kgriffsttfn19:18
*** kgriffs is now known as kgriffs_afk19:19
*** malini is now known as malini_afk19:33
*** gordonsim has quit IRC19:45
*** kgriffs_afk is now known as kgriffs19:49
*** kgriffs is now known as kgriffs_afk19:58
zyuancppcabrera: http://paste.openstack.org/show/47340/19:59
*** kgriffs_afk is now known as kgriffs19:59
zyuanyou also need to set headers v1, queues, whenever you uses requests lib.20:00
zyuanyou can wrap the Captitalize code into a function somewhere20:00
*** kgriffs is now known as kgriffs_afk20:00
cppcabreraI'll give that a try in the next patch.20:03
cppcabrerafor now, I will save that patch in the marconi-proxy-issues etherpad.20:03
cppcabreraThanks, zyuan!20:03
*** amitgandhi has quit IRC20:07
zyuancppcabrera: if you are busy, i can make the patch (on gerrit)20:11
cppcabreraGo ahead, zyuan. I won't get to it today.20:20
openstackgerritZhihao Yuan proposed a change to openstack/marconi: fix(proxy): forward marconi headers  https://review.openstack.org/4762820:34
openstackgerritZhihao Yuan proposed a change to openstack/marconi: WIP: test(proxy): forwarded endpoints tested  https://review.openstack.org/4696320:55
*** malini_afk is now known as malini21:04
*** oz_akan_ has quit IRC21:09
*** EmilienM has quit IRC21:22
*** EmilienM has joined #openstack-marconi21:23
*** tedross has quit IRC21:30
*** ametts has quit IRC21:43
*** flaper87 is now known as flaper87|afk21:44
*** cppcabrera has quit IRC21:57
openstackgerritAlejandro Cabrera proposed a change to openstack/marconi: proxy: mirror structure of marconi queues + bootstrap  https://review.openstack.org/4764222:01
*** mpanetta has quit IRC22:06
*** openstack has quit IRC22:35
*** openstack has joined #openstack-marconi22:35
*** malini is now known as malini_afk22:36

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!