Monday, 2014-06-16

*** haomaiwang has quit IRC00:51
*** haomaiwang has joined #openstack-marconi00:54
*** nosnos has joined #openstack-marconi00:55
*** fifieldt__ is now known as fifieldt02:30
*** haomai___ has joined #openstack-marconi02:33
*** haomaiwang has quit IRC02:34
*** jeffreycoho has quit IRC03:20
*** nosnos has quit IRC03:36
*** nosnos has joined #openstack-marconi04:15
*** haomai___ has quit IRC04:44
*** haomaiwa_ has joined #openstack-marconi04:45
*** haomai___ has joined #openstack-marconi05:00
*** haomaiwa_ has quit IRC05:03
*** amalagon has joined #openstack-marconi06:16
*** haomai___ has quit IRC06:53
*** haomaiwang has joined #openstack-marconi06:53
*** jamiehannaford has joined #openstack-marconi06:56
*** AAzza_afk is now known as AAzza07:01
*** chandan_kumar has quit IRC07:05
*** haomaiw__ has joined #openstack-marconi07:08
*** haomaiwang has quit IRC07:10
*** haomaiw__ has quit IRC07:14
*** flaper87|afk is now known as flaper8707:24
*** ykaplan has joined #openstack-marconi08:18
*** seiflotfy has left #openstack-marconi09:12
*** AAzza is now known as AAzza_afk09:13
*** AAzza_afk is now known as AAzza09:13
*** davidhadas_ has quit IRC09:32
*** AAzza is now known as AAzza_afk09:34
*** ykaplan has quit IRC09:37
*** norman has joined #openstack-marconi09:41
openstackgerritOpenStack Proposal Bot proposed a change to openstack/marconi: Updated from global requirements  https://review.openstack.org/9903309:48
openstackgerritOpenStack Proposal Bot proposed a change to openstack/python-marconiclient: Updated from global requirements  https://review.openstack.org/9909309:52
*** openstackgerrit has quit IRC10:06
*** openstackgerrit has joined #openstack-marconi10:07
*** davidhadas has joined #openstack-marconi10:09
*** haomaiwa_ has joined #openstack-marconi10:13
*** haomaiwa_ has quit IRC10:23
*** haomaiwang has joined #openstack-marconi10:23
*** haomaiw__ has joined #openstack-marconi10:25
*** haomaiwang has quit IRC10:28
*** ykaplan has joined #openstack-marconi10:29
*** openstackgerrit has quit IRC10:35
*** openstackgerrit has joined #openstack-marconi10:36
*** vkmc has joined #openstack-marconi10:47
vkmcmorning all! o/10:50
*** davidhadas___ has joined #openstack-marconi11:03
*** davidhadas has quit IRC11:06
flaper87vkmc: morning :)11:06
vkmcflaper87, hey flaaaa :)11:08
flaper87vkmc: soooo, I gotta summarize the call we had on Friday11:09
flaper87I'll do that today11:09
vkmcflaper87, sure, whenever you have a moment I'm looking forward to it11:10
vkmcflaper87, probably you want to send it to the ML, so everyone is up to date11:10
flaper87the sooner I do it, the better. I'm starting to forget things from that call11:10
flaper87:P11:10
vkmclol11:11
vkmcthat's the bad part of video conferences... there is no log -.-11:11
flaper87add to that my really bad memory11:13
vkmchaha11:13
vkmcI also have a bad memory... like Nemo's Dory11:14
*** nosnos has quit IRC11:15
vkmcbbs11:44
*** norman has quit IRC12:19
*** norman has joined #openstack-marconi12:20
normanhi all, how can I own the new bugs? The 'assigned to' is not  editable12:23
*** davidhadas___ has quit IRC12:24
flaper87norman: you can assign it to yourself12:34
flaper87norman: what bug is it?12:34
*** sriram has joined #openstack-marconi12:37
normanflaper87: 1328720  about the pymongo's cert options12:39
flaper87bug 132872012:39
flaper87bug #132872012:39
flaper87stupid bot12:39
normanI refreshed my browser, it seems that it's editable again12:39
*** sriram has quit IRC12:40
*** sriram has joined #openstack-marconi12:40
normanthanks, it owned by me now -:)12:40
*** jmckind has joined #openstack-marconi12:43
flaper87norman: np12:50
*** haomaiw__ has quit IRC12:58
*** Obulpathi has joined #openstack-marconi13:26
*** Obulpathi has quit IRC13:26
*** Obulpathi has joined #openstack-marconi13:27
*** tonytan4ever has joined #openstack-marconi13:47
*** malini1 has joined #openstack-marconi13:52
*** balajiiyer has joined #openstack-marconi13:54
*** tonytan4ever has quit IRC14:21
malini1hola!!!14:23
malini1flaper87: can you please review this when you get a chance https://review.openstack.org/#/c/93295/ ?14:24
vkmchola malini1!14:27
malini1hello vkmc!!14:27
malini1how are you?14:27
vkmcall good and you? :)14:28
malini1good.for a change, it feels like I had a long enough weekend :D14:29
sriramnice!14:29
*** rwsu has joined #openstack-marconi14:29
sriramand hola! :P14:29
*** cpallares has joined #openstack-marconi14:29
*** kgriffs|afk is now known as kgriffs14:33
*** kgriffs is now known as kgriffs|afk14:34
*** kgriffs|afk is now known as kgriffs14:35
*** chandan_kumar has joined #openstack-marconi14:36
vkmchola sriram!14:36
cpallareshi vkmc :)14:39
vkmchi cpallares!14:43
cpallaresvkmc: how are you doing?14:44
vkmccpallares, all good around here and you?14:45
vkmcfreeeeeeeeezing14:45
vkmc>.>14:45
cpallaresvkmc: Oh? Too cold outside? It's too hot here >_<14:45
cpallaresvkmc: Maybe we can trade half the weather :P14:46
vkmccpallares, too cold for my taste yeah... I love the idea :P14:47
*** tonytan4ever has joined #openstack-marconi14:47
sriramkgriffs: ping14:54
kgriffsyo14:54
sriramhow are you today? :)14:55
sriramwrt https://review.openstack.org/#/c/91804/ , I liked flaper87's comments. That is how I had tried to implement it earlier.14:56
srirambut storage is not version aware, how do we go about doing it at the storage layer rather than at transport layer.14:56
sriramkgriffs: ^14:56
sriramit = queue create logic.14:56
kgriffslooking14:57
*** tonytan4ever has quit IRC15:03
*** tonytan4ever has joined #openstack-marconi15:04
*** tonytan4ever has quit IRC15:11
*** tonytan4ever has joined #openstack-marconi15:15
*** amitgandhi has joined #openstack-marconi15:16
kgriffssriram: I replied to flaper87's comments15:25
kgriffsTL;DR I'm having trouble understanding how the upsert approach would be any better15:25
sriramupsert would involve 2 queries instead of one insert, so it might be slower wrt to performance.15:26
flaper87I think the lazy-create is something we should leave to the storage layer. Storage drivers will do it in the best way possible for the driver15:27
*** tonytan4_ has joined #openstack-marconi15:27
flaper87I don't think checking existence will be faster because it still needs to send data from/to marconi to/from the storage15:27
flaper87instead, with the upsert, we'd send everything to the storage and let it handle everything using the index15:27
kgriffsflaper87: but you can cache the existence check15:28
flaper87IIRC, the previous version that used upsert had already taken care of the possible-counter-bug15:28
kgriffsflaper87: also, in the case of mongodb, does an upsert lock the db?15:28
flaper87yeah but if we're using redis, we still need to send the request to redis and get it back15:28
flaper87kgriffs: not during the check15:29
flaper87(AFAIK)15:29
flaper87kgriffs: and yes, I meant exactly that. Use upsert for v1.1 and check for existence in v1.015:29
kgriffshow do you avoid resetting the coutner?15:29
flaper87kgriffs: you would write the counter just when inserting the document15:30
flaper87(unless I'm missing something)15:30
sriramself._collection.update({'p_q': scoped_name, 'c': counter},{'$set': {'m': {}}}, upsert=True)15:30
sriramThis is what is in there before.15:30
kgriffsdoesn't an upsert insert a new document or update the existing? in the latter case, it would reset the counter15:30
kgriffswait15:31
kgriffsbeen too long since I've written upsert code15:31
*** tonytan4ever has quit IRC15:31
* kgriffs looks that up15:31
flaper87kgriffs: yes but the upsert query is: "{'p_q': scoped_name, 'c': counter}" where counter is a 0 counter15:31
flaper87(IIRC)15:31
* flaper87 needs to read the old patch-set again15:32
flaper87I know it worked15:32
flaper87:P15:32
kgriffshmm15:32
sriramyeah it worked, I tested it :)15:32
flaper87kgriffs: see that the query doesn't check just on the `p_q` field but `p_q` and counter15:33
kgriffsok. let me walk through this15:33
flaper87that's to force the insert to happen *just* when the queue doesn't exist15:33
kgriffsif the queue does not exist15:33
kgriffsyou will insert a queue with the given query15:33
kgriffs"{'p_q': scoped_name, 'c': counter}"15:33
kgriffsplus, setting metadata to an empty document, {}15:34
kgriffscorrect?15:34
flaper87there's still a bug in that query, because it basically unsets the metadata15:34
flaper87yeah15:34
kgriffsflaper87: yeah, I was going to ask about that15:34
sriramunsets the metadata?15:35
kgriffsif the queue does exist, it does not recreate it, but does reset the metadata15:35
sriramohh.15:35
flaper87sriram: the query sets metadata to {} regardless the .... you figured it out15:35
flaper87:P15:35
sriramI got it. :P15:35
kgriffsso... how do you fix it?15:37
kgriffsself._collection.update({'p_q': scoped_name, 'c': counter, 'm': {}},{}, upsert=True)15:37
flaper87kgriffs: that's one way. Another way is not setting the metadata15:37
flaper87it'll be set whenever is needed15:37
flaper87and it's a waste of space to keep it there, empty.15:38
kgriffsyeah15:38
flaper87Although that helps with updates paddings15:38
flaper87still, I think not setting it is better15:38
kgriffsso15:39
kgriffsself._collection.update({'p_q': scoped_name, 'c': counter},{}, upsert=True)15:39
kgriffsassuming most of the time the queue *does* exist15:39
flaper87yup15:39
kgriffswhich is faster, doing a cached check for whether the queue exists, or doing this upsert that turns into a noop if the thing exists?15:40
kgriffstwo things to check: 1. does update always lock the db15:40
kgriffs2. benchmark the two approaches15:40
kgriffs(if only we had a way to benchmark this...)15:40
kgriffsbenchmarking will depend on whether we are hitting a localhost cache or a central cache (i.e., memcached/redis)15:41
kgriffserrr15:41
kgriffssorry, conflated that15:41
kgriffsa localhost could be in-process (memory driver), memcached or redis (when those drivers land)15:41
flaper87yeah, the local memory will definitely be faster although I expect most deployments to use a central one15:42
flaper87s/central/remote/15:42
kgriffs...or a hybrid (similar to the way L1, L2 caches work on a CPU)15:42
kgriffs...but I digress15:42
kgriffsflaper87: what's the status on the memcached driver?15:42
flaper87kgriffs: bryan was working on it, I'm not sure if he kept doing so or just abandoned it.15:43
flaper87but there's a bigger thing going on15:43
flaper87kgriffs: https://review.openstack.org/#/c/97155/15:43
flaper87check that review out15:43
kgriffs"currently unused"15:44
kgriffswell, by integrated projects anyway. :p15:44
kgriffsflaper87: "Keystone has used..."15:45
kgriffsis keystone middleware using dogpile as well, or just the service?15:45
flaper87yeah, so, I've an on-going review there15:47
flaper87part of it is that it's being used15:47
flaper87that said, I'm all for writing a wrapper on top of dogpile15:47
flaper87that would bring more supported backends for caching15:47
flaper87kgriffs: I think the middleware uses dogpile as well15:48
flaper87brb15:48
*** ykaplan has quit IRC15:49
sriramok, one more thing. coming back to my intial question15:49
sriramif we move logic to storage layer, wont it affect v1 api as well?15:49
srirammaybe I'm missing something here.15:49
flaper87sriram: you have to keep the check for v1.0 *in the transport*15:50
flaper87*just for v1.0*15:50
flaper87v1.1 will rely on the storage layer15:50
kgriffssriram: it's sort of the inverse from what you are doing now15:51
kgriffsinstead of checking in v1.1, you check in v1.0 and raise an error if it does *not* exist15:52
kgriffsbut still, given that we still have to do a check, what does moving to an upsert buy us?15:52
sriramI see. I'm getting some clarity now. Sorry for the confusion15:53
kgriffsJust feels like we are moving things around and ending up with something equivalent, with the added possibility that the upsert may be slower (not proven, but possible)15:53
sriramI'm not sure how we would benchmark this to prove which is faster15:54
srirammaybe mongo's .explain() might help here.15:54
kgriffsmmm15:54
kgriffsyou could try both ways15:54
kgriffsoh, actually, we can't really benchmark this because our only option for caching right now is memory, since there are no other drivers for oslo.cache15:55
kgriffsblah15:55
kgriffsI think we should decide whether there is some design reason to change back to upsert15:56
sriramhmm15:56
kgriffsif it is actually equivalent, design-wise, to what you have now, then there is no point changing it again15:56
sriramgood point.15:56
kgriffsesp. given the performance risk, but even without that15:56
kgriffsconsider:15:56
kgriffsin both approaches you have to do a check for queue existence. in one approach, you do it in v1.0, while in the other you do it in v1.115:59
* kgriffs hopes he got that right15:59
sriramYes15:59
*** alcabrera|afk is now known as alcabrera15:59
kgriffsin one approach, you also modify create() to be an upsert rather than an insert16:00
sriramkgriffs: yes, that was the earlier approach16:00
kgriffswhat is the benefit?16:00
sriramwe let the storage layer handle this.16:00
srirammaybe some other backend has a really cool way of doing it16:01
sriramand that costs us way less in terms of time.16:01
sriramso its better to off load that work to the storage controller.16:01
sriramthose are the pros there.16:01
kgriffsI can't see how the storage controller would be able to be any *faster* than cache16:01
sriramUmm, yes.16:02
kgriffsit may be close, or equivalent, but not faster16:02
kgriffsI think the one argument that may make sense is that future API versions will not have to do the "exists" check16:02
kgriffswe just put it in v1.0 and subsequent APIs don't need to do it16:03
kgriffsbut then, we lose the opportunity for any possible performance gains by caching...unless we implement that in each driver16:03
kgriffshmmm16:03
sriramcaching every driver is additional hassle, and makes the code harder to read in some cases.16:04
kgriffsyeah16:04
*** tonytan4_ has quit IRC16:05
kgriffson the other hand, we will have to do the exists check in v1.1, 2.0, 2.116:05
sriramdo we decide this at the meeting tomorrow?16:05
kgriffsit still feels like a wash to me. No clear winner. So I would just go with what you have right now. Optimize for least amount of code change.16:06
sriramwe could have more people weigh in with their opinions.16:06
sriramAll right16:06
kgriffssriram: nah, I don't think this is that critical of a decision.16:06
flaper87the whole point was exactly that, I don't think future API versions will have to do the exists16:07
kgriffsIf Flavio has a super good reason why we should change more code than what you have currently, then do that. otherwise, leave as is16:07
kgriffsflaper87: like I said, I don't think the decision is super critical either way, just as long as we make sure we can still make things fast.16:08
srirambbl lunch16:08
kgriffswe can, in both ways, because we can cache at transport or driver layer if needed16:08
flaper87sure16:08
* sriram will think on this as he munches food16:08
flaper87anyway, I guess we can leave it as-is16:09
kgriffsI'm still curious about whether upsert takes a db lock16:09
flaper87we'll figure this out later16:09
flaper87I think the "query" part of it doesn't16:09
flaper87kgriffs: btw, We'll have the mongodb summit next week. If there's anything I should bring up there, let me know16:09
flaper87I'll also ask this16:09
kgriffskk16:10
*** jergerber has joined #openstack-marconi16:12
*** vkmc has quit IRC16:14
*** tonytan4ever has joined #openstack-marconi16:16
*** AAzza_afk is now known as AAzza16:40
*** balajiiyer has quit IRC16:41
*** tonytan4ever has quit IRC16:42
*** tonytan4ever has joined #openstack-marconi16:43
*** tonytan4ever has quit IRC16:48
*** balajiiyer has joined #openstack-marconi16:50
*** balajiiyer has quit IRC16:50
*** balajiiyer has joined #openstack-marconi16:50
openstackgerritSriram Madapusi Vasudevan proposed a change to openstack/marconi: Implement Lazy Create Queue in v1.1 API  https://review.openstack.org/9180416:57
*** vkmc has joined #openstack-marconi17:14
*** kgriffs is now known as kgriffs|afk17:17
*** balajiiyer has quit IRC17:21
*** jmckind has quit IRC17:25
*** jamiehannaford has quit IRC17:26
*** balajiiyer has joined #openstack-marconi17:27
vkmcflaper87, about AMQP, do we have a possible way to make it work or are we completely blocked?17:27
vkmcflaper87, (a preview would be great to kill the anxiety)17:29
*** reed has joined #openstack-marconi17:36
*** kgriffs|afk is now known as kgriffs17:38
*** kgriffs is now known as kgriffs|afk17:39
*** ChanServ changes topic to "OpenStack Queuing and Notification Service || Smile :D || Meetings every Tuesday @ 15:00 UTC || Wiki: https://wiki.openstack.org/wiki/Marconi || Paste: http://paste.openstack.org/ || Send messages and make some noise :D"17:39
*** tonytan4ever has joined #openstack-marconi17:40
*** malini2 has joined #openstack-marconi18:01
*** haomaiwang has joined #openstack-marconi18:02
*** malini1 has quit IRC18:04
*** alcabrera is now known as alcabrera|afk18:05
*** AAzza is now known as AAzza_afk18:17
*** AAzza_afk is now known as AAzza18:23
*** tonytan4ever has quit IRC18:32
*** tonytan4ever has joined #openstack-marconi18:33
*** tonytan4ever has quit IRC18:43
*** tonytan4ever has joined #openstack-marconi18:44
*** tonytan4_ has joined #openstack-marconi18:45
*** tonytan4_ has quit IRC18:47
*** tonytan4_ has joined #openstack-marconi18:48
*** tonytan4ever has quit IRC18:48
*** vkmc_ has joined #openstack-marconi18:53
*** vkmc has quit IRC18:56
*** vkmc_ is now known as vkmc18:56
*** ametts has joined #openstack-marconi19:02
*** AAzza is now known as AAzza_afk19:07
*** reed has quit IRC19:09
srirammalini: Where does the wiki for all the status codes for marconi live? I'm trying to find it.19:11
vkmcsriram, https://wiki.openstack.org/wiki/Marconi/specs/api/v1#HTTP_Response_Codes19:13
sriramthanks vkmc, I had been scrolling past that :P19:13
vkmchaha is too small19:14
vkmcnp19:14
*** reed has joined #openstack-marconi19:14
*** malini2 has quit IRC19:25
*** alcabrera|afk is now known as alcabrera19:28
*** cpallares has quit IRC19:38
*** ykaplan has joined #openstack-marconi19:38
*** jdprax has joined #openstack-marconi19:41
*** haomaiwang has quit IRC19:52
*** jamiehannaford has joined #openstack-marconi20:13
*** jdprax has quit IRC20:27
*** jdprax has joined #openstack-marconi20:27
*** jdprax has quit IRC20:27
*** sriram has quit IRC20:35
*** kgriffs|afk is now known as kgriffs20:48
kgriffsthere is one for v1.1 as well20:54
kgriffshttps://wiki.openstack.org/wiki/Marconi/specs/api/v1.1/errors20:55
kgriffsvkmc, sriram: ^^^20:55
vkmcthanks kgriffs!20:55
kgriffsmalini: when should I schedule this bug? https://bugs.launchpad.net/marconi/+bug/127333520:56
*** tonytan4_ has quit IRC21:05
vkmckgriffs, which is the status of this bp? https://blueprints.launchpad.net/python-marconiclient/+spec/cli21:07
vkmckgriffs, is it scheduled for K?21:07
*** Obulpathi has quit IRC21:09
kgriffslooking21:09
kgriffsah, OK21:13
kgriffsFlavio did some basic plumbing on that21:13
kgriffshttps://github.com/openstack/python-marconiclient/blob/master/marconiclient/queues/cli.py21:13
kgriffsI don't think it does much right now, though21:13
* vkmc clicks21:14
vkmccool21:14
kgriffshttps://review.openstack.org/#/c/87937/21:15
vkmcit really caught my attention... and since AMQP is a bit stalled right now I was looking for something to contribute with in parallel21:15
flwanghi guys when is this week meeting?21:16
*** tonytan4ever has joined #openstack-marconi21:16
kgriffsok, looks like it just does queue CRUD21:16
*** jamiehannaford has quit IRC21:16
vkmcyup :)21:17
kgriffsflwang: we haven't moved it yet, so it is at the same time. 1500 UTC. That's like, 3am for you or something?21:17
flwanggot it21:17
flwang3am for me, gosh :D21:17
kgriffsyeah, we'll have to move it. I'll bring it up in tomorrow's meeting. :p21:18
flwangI will read the meeting minutes21:18
vkmc:o too early and too late at the same time21:18
flwangvkmc: yep, I'm based in New Zealand21:19
vkmcflwang, yaay so nice :)21:20
flwangvkmc: sort of :D21:21
kgriffsvkmc: re the CLI, it would be good to sync with flaper87, but I don't see any reason why you couldn't lend a hand21:28
vkmckgriffs, k :) thanks21:30
*** tonytan4ever has quit IRC21:34
flwangkgriffs: when is the m-2 due date?21:38
kgriffshere's the full schedule21:39
kgriffshttps://wiki.openstack.org/wiki/Juno_Release_Schedule21:39
flwangkgriffs: cool, thanks21:39
vkmckgriffs, regarding that schedule... I was thinking that maybe I could check some of the others backend storages21:41
vkmcif that is a priority over the cli for Naav21:41
flwangwow, so Naav has been the official name, right?21:42
kgriffsyep, that was voted on by the team21:42
flwangby us or by TC? :)21:42
vkmcoh... spoiler alert :(21:42
kgriffsvkmc: re backends... did we determine that AMQP 1.0 can't work with even part of the 1.x API?21:43
flwangTBH, I love Marconi21:43
vkmckgriffs, I still have to chat with flaper87 about it21:43
kgriffsyeah, someone snagged the Marconi trademark... I was sworn to silence, but I'll just say it was rather disingenuous of the trademark holder21:44
kgriffsout of the short list the team brainstormed: "Our trademark counsel did a search and recommended Zaqar or Naav as the best potential names with low-to-moderate risk. Tamtam came up as a moderate risk based on TamTamy, but we could still likely pursue it because the software is quite distinguishable. Both Raven and Copper pose a higher risk, because they are registered to companies in dev / cloud spaces (Rave21:44
kgriffsnflow and Copper.io respectively)"21:44
*** balajiiyer has quit IRC21:45
flwangkgriffs: how to pronounce Naav?   is it like [na:vi] ?21:46
vkmcis like... dragging the a21:47
kgriffsyeah. "Na" is like the "kno" in "knot"21:48
kgriffsyou drag out the middle a bit (like slurring two notes together of the same value, if you are a musician)21:49
kgriffssomething like that, anyway. :p21:50
kgriffsvkmc: back on the topic of backend drivers21:51
kgriffsI've been discussing with marconi core reviewers what they want to do about that. It's a rather complicated question.21:52
vkmckgriffs, it is yeah21:52
vkmcmaybe we want to focus on redis and amqp right now?21:52
vkmcwe have to define the API unified model so we can go through a secure path21:53
kgriffsyeah. The API is the sticking point. Also the question of durability.21:53
kgriffsredis should be super fast (TBD), but is not durable21:53
kgriffsMongoDB is reasonably fast, durable, but AGPL so some people don't want to deploy it21:54
kgriffsAMQP 1.0 brokers are pretty fast, durable, but don't support the entire API21:55
kgriffsKafka is insanely fast, durable, but doesn't support the entire API or perhaps any of it21:55
vkmcwell.. redis and mongo are alike21:56
kgriffsAMQP is also nice because it gives operators another way to scale their brokers21:56
vkmcand amqp and kafka too..21:56
vkmcin a high level of abstraction of course21:57
kgriffsredis and mongo can support the entire API21:57
vkmcthey follow the same arch21:57
kgriffsAMQP and Kafka are more opinionated about the semantics21:57
kgriffsFlavio also mentioned Elasticsearch should be able to support the entire API21:57
kgriffsand we could just go all out and roll our own thing using LevelDB as a building block21:58
vkmcsounds good21:58
kgriffsI kind of feel like we need to either decide to just go with 1.x design for J and only work on things that can support all of it21:59
kgriffsor21:59
kgriffsreboot the project and design the API so it will be lovely for AMQP and Kafka21:59
kgriffsor21:59
vkmcthing is... why do we want to give support for brokers like amqp and kafka? just for efficiency reasons?21:59
kgriffstwo reasons22:00
kgriffsone is that Kafka has some impressive benchmarks, albeit those aren't going over HTTP22:00
kgriffsand using a compiled, highly-optimized language22:01
kgriffs:p22:01
vkmcand can't we achieve something similar by using other storage backends as elasticsearch?22:01
kgriffssecond reason is that Flavio was thinking we could help AMQP brokers scale out22:01
kgriffsvkmc: that's the million-dollar question22:02
kgriffswe don't really know how fast a more generic nosql backend can be22:02
vkmchehe we will need to do some benchmarks22:02
kgriffsor how fast Qpid or Kafka would be if put behind a python-based, REST interface22:02
kgriffsactually, there were three reasons22:03
kgriffslast one was that it has been posited that there is something inherently less efficient about the current design - that if we simplify it drastically, we could make it much faster.22:04
kgriffss/design/API22:04
kgriffsThat is probably true, at least for task distribution workloads22:04
vkmcyes that's true22:04
*** alcabrera is now known as alcabrera|afk22:05
kgriffsbut then again, it is hard to know for sure without benchmarks22:05
flaper87o/22:05
flaper87'sup folks22:05
vkmcwe have to figure out if the flexibility provided is important for naav users... if that pay off the reduced efficiency22:05
flaper87vkmc: so, spoler, it's quite possible that the result of this whole AMQP research ends up in being a no-go for the backend22:06
vkmcadjusting ourselves to strict queue semantics would make us to change the api... a lot22:06
flaper87however, having a amqp 1.0 transport still makes sense22:06
flaper87that said, I'll write everything on an etherpad as promissed and bring it up in tomorrow's meeting22:06
vkmcthanks flaper87 :)22:07
flaper87I believe there are some semplifications we can do in the API but even with those semplifications I don't think we'll be able to support AMQP backends22:07
*** amitgandhi has quit IRC22:07
vkmc:/22:07
flaper87vkmc: since you've done lot of research on the AMQP side, would you be interested in writing an AMQP 1.0 transport22:08
flaper87?22:08
flaper87Azure Queues has support for AMQP1.0 which is really interesting22:09
flaper87they reported a 20x times performance improvement after enabling it.22:09
flaper87Not saying the same will happen with Marconi22:09
vkmcsure I'd be glad :)22:09
vkmcI was looking into ElasticSearch as well22:10
flaper87but the AMQP 1.0 protocol puts marconi in the position to sit along side other amqp technologies22:10
vkmcbut maybe we want to stick with Redis storage and AMQP transport for now?22:10
flaper87vkmc: funny you said that. I started playing with it22:10
flaper87I mean, working on a storage for ES22:10
vkmcit looks cool22:10
flaper87I've used ES in the past already22:10
vkmchaha ok I won't take your toy away :)22:10
flaper87if you prefer to stick to storage technologies, I'm happy to leave that work to you22:10
flaper87vkmc: really, I wan't you to work on something you find interesting22:11
kgriffsBTW, not sure how much it matters, but one more data point: All the feedback we have gotten from Rackspace's Marconi deployment has never been "hey, we don't like the API" but "hey, I want to do like 10,000 req/sec, can you guys handle that?" I think we can do that today on optimized MongoDB hardware, but it may not be as cost-efficient as some other backend.22:11
flaper87since you started looking at it, it may probably make sense22:11
flaper87kgriffs: agreed22:11
kgriffsalso, there is the question of how much performance gain we get from using a non-HTTP protocol22:11
flaper87kgriffs: that's something I want  share w/ you tomorrow too22:12
flaper87re the API22:12
kgriffsok22:12
vkmcflaper87, whatever is best for Marconi :)22:12
flaper87TL;DR: I don't think it's bad22:12
flaper87vkmc: and for you ;)22:12
flaper87kgriffs: what do you think about the ES backend?22:13
vkmcflaper87, thx... I could continue with AMQP... this time in the transport side22:13
vkmcflaper87, so we can get something from the research done22:13
kgriffsi will say, however, that I've done a different service using python, over HTTP, that was pretty darn fast. Not Kafka-fast, but fast enough to make people do a double-take when I told them the benchmarks.22:13
flaper87vkmc: it's your call22:13
kgriffsthis is why we __neeeeeed__ benchmarks22:13
vkmck22:13
vkmcflaper87, should I drop the pseudo-amqp-storage-driver?22:14
flaper87kgriffs: I don't believe HTTP is "slow for the task"22:14
flaper87vkmc: yeah, lets stop that work there and change direction22:14
kgriffsflaper87: no, what you said is we should migrate everything to Go22:14
vkmc(y)22:14
flaper87don't trash it, though22:14
vkmcoh no it's in my repo22:14
* kgriffs trolling for fun and profit22:14
flaper87kgriffs: LIAR, I said rust22:14
flaper87vkmc: great22:15
flaper87vkmc: so, what do you pick? ES store or AMQP transport ?22:15
vkmcflaper87, AMQP transport :)22:15
flaper87vkmc: cool22:16
flaper87vkmc: so, some hints there22:16
flaper87vkmc: take a look at how Azure Queues implemented it22:16
flaper87weeeellllll22:16
vkmcnoted, thanks :)22:16
flaper87that was a fun thing to say given the fact that it's Microsoft22:16
flaper87I meant to say, read the API docs22:16
flaper87:D22:16
vkmcshh, we keep that under Marconi's rug22:16
vkmcof course22:17
flaper87vkmc: thanks22:18
flaper87a looooooooooooooooooooooooot22:18
vkmc:)22:18
kgriffsremember, great artists steal22:18
kgriffs...with a grain of salt22:19
vkmctotally haha22:19
vkmcoh and flaper87, what do you think about the cli right now?22:19
vkmcfor marconi client22:20
kgriffsspecifically: https://blueprints.launchpad.net/python-marconiclient/+spec/cli22:21
vkmcyes, sorry22:21
vkmcthanks kgriffs22:21
kgriffsp.s. - need some help triaging bugz22:21
kgriffshttps://bugs.launchpad.net/python-marconiclient22:21
flaper87vkmc: it's a WIP22:21
* kgriffs goes away again22:21
flaper87:D22:21
flaper87kgriffs: I'll triage them22:21
kgriffsexcellent22:21
flaper87vkmc: want to help with that?22:21
vkmcflaper87, yeah it would be cool!22:21
flaper87vkmc: cool I just added support for queues22:22
flaper87I was focusing on plumbing everything22:22
vkmcflaper87, kgriffs shared it with me... it looks great22:22
flaper87it should be pretty straightforward to add support for other things22:22
vkmcyes.. it's indeed really helpful to have it as an example22:22
flaper87great. Thanks, vkmc22:26
flaper87:)22:26
vkmcnp :)22:26
vkmckgriffs, flaper87 I just updated the AMQP bp... if there is something wrong let me know22:33
flaper87vkmc: ok22:34
flaper87thanks22:34
flaper87It can always be revisited in the future22:34
vkmcof course22:34
vkmcmaybe I should add it?22:34
flaper87sure22:34
*** jergerber has quit IRC22:35
vkmcdone22:35
flaper87great, thanks again vkmc22:36
flaper87ok, gtg now22:36
flaper87:)22:36
flaper87take care22:36
vkmcttfn flaper87 o/ you too!22:36
vkmcenjoy the rest of your evening22:36
*** flaper87 is now known as flaper87|afk22:37
*** kgriffs is now known as kgriffs|afk22:47
openstackgerritA change was merged to openstack/marconi: Sync from oslo-incubator  https://review.openstack.org/10009323:41

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!