Wednesday, 2014-06-18

*** cpallares has joined #openstack-marconi00:13
openstackgerritKurt Griffiths proposed a change to openstack/marconi: Update hacking version and fix new violations  https://review.openstack.org/10072700:13
*** mwagner_lap has joined #openstack-marconi00:14
kgriffsok, I'm out00:16
kgriffswe should be able to get that patch finalized tomorrow00:16
*** kgriffs is now known as kgriffs|afk00:17
vkmco/00:28
vkmctake care!00:28
openstackgerritOpenStack Proposal Bot proposed a change to openstack/marconi: Updated from global requirements  https://review.openstack.org/9903300:42
openstackgerritOpenStack Proposal Bot proposed a change to openstack/python-marconiclient: Updated from global requirements  https://review.openstack.org/9909300:47
*** oz_akan_ has joined #openstack-marconi00:50
*** rwsu has quit IRC00:58
*** flwang has quit IRC01:10
*** cpallares has quit IRC01:13
*** flwang has joined #openstack-marconi01:25
*** nosnos has joined #openstack-marconi01:31
openstackgerritFei Long Wang proposed a change to openstack/marconi: Make sure msgpack-python version >= 0.4.2  https://review.openstack.org/10074502:12
*** oz_akan_ has quit IRC02:13
*** oz_akan_ has joined #openstack-marconi02:14
openstackgerritpeng fei wang proposed a change to openstack/marconi: Expose pymongo's SSL cert options  https://review.openstack.org/10074602:15
normanall, I'd delivered my first review , and I'd tried to be careful , anyway still make some mistake that included some unrelated changes,  what can I do now?02:25
normandeliver a following changeset to revert the changes ?02:30
*** norman has quit IRC02:34
*** haomai___ has quit IRC02:36
*** oz_akan__ has joined #openstack-marconi02:41
*** oz_akan_ has quit IRC02:42
*** haomaiwa_ has joined #openstack-marconi02:45
openstackgerritPeng Fei Wang proposed a change to openstack/marconi: Expose pymongo's SSL cert options  https://review.openstack.org/10074602:48
*** oz_akan__ has quit IRC02:52
*** norman has joined #openstack-marconi02:53
normanhad amend the commit with other's help02:53
*** renlt has joined #openstack-marconi02:54
vkmcthanks norman!02:55
vkmcI'll review it soon :)02:56
normanvkmc: thanks a lot02:56
normanit make me a little nervous since it's my first one , haha02:56
normanvkmc:  if I found some typo in the code,  do I need to create a bug  to deliver the code ?03:00
openstackgerritFei Long Wang proposed a change to openstack/marconi: Make sure msgpack-python version >= 0.4.0  https://review.openstack.org/10074503:03
flwangnorman: did you mean found typo in Marconi repo?03:04
normanflwang:  yes03:04
flwangnorman: if so, you can just submit a ps with/without filing a bug03:04
normanok, thanks03:05
flwangnorman: thanks for working on Marconi/Naav03:05
*** nosnos has quit IRC03:27
vkmcnorman, you did great... just some minor nits, but the core looks good!03:30
normanvkmc:  thanks ,I will modify them soon03:31
normanvkmc:  you said Lines length in the commit messages should be >=72,  but it should be <=72, right?03:38
normango for lunch ....03:38
vkmcnorman, right :)03:39
vkmcthat's why I added the link03:39
vkmcI cannot trust myself at this hours03:39
*** oz_akan_ has joined #openstack-marconi03:50
vkmcok... so kgriffs|afk patchset is not merged yet and this one is passing https://review.openstack.org/#/c/99033/03:50
* vkmc lost03:50
*** oz_akan_ has quit IRC03:55
vkmcttfn o/03:58
*** vkmc has quit IRC03:59
*** nosnos has joined #openstack-marconi04:07
*** haomaiwa_ has quit IRC04:17
*** haomaiwang has joined #openstack-marconi04:18
*** haomai___ has joined #openstack-marconi04:27
*** haomaiwang has quit IRC04:31
*** oz_akan_ has joined #openstack-marconi04:51
*** oz_akan_ has quit IRC04:55
*** AAzza_afk is now known as AAzza05:21
*** reed has quit IRC05:38
*** AAzza is now known as AAzza_afk05:43
openstackgerritPeng Fei Wang proposed a change to openstack/marconi: Expose pymongo's SSL cert options  https://review.openstack.org/10074605:46
openstackgerritPeng Fei Wang proposed a change to openstack/marconi: Expose pymongo's SSL cert options  https://review.openstack.org/10074605:46
openstackgerritPeng Fei Wang proposed a change to openstack/marconi: Expose pymongo's SSL cert options  https://review.openstack.org/10074605:48
*** oz_akan_ has joined #openstack-marconi05:52
*** oz_akan_ has quit IRC05:56
openstackgerritPeng Fei Wang proposed a change to openstack/marconi: Expose pymongo's SSL cert options  https://review.openstack.org/10074605:58
openstackgerritPeng Fei Wang proposed a change to openstack/marconi: Expose pymongo's SSL cert options  https://review.openstack.org/10074606:01
*** alcabrera|afk is now known as alcabrera06:02
normanflwang:  Hi Feilong ,there?06:04
flwangnorman: hi06:04
normanflwang:  what do you mean for the comment : And as for the requirements change, you need to submit a separate patch against requirements project firstly.06:05
normanwhich project it is you mentioned06:05
*** dragondm has quit IRC06:05
flwangyou can take this https://review.openstack.org/#/c/100745/ as an example06:06
normanflwang:  Ok got it ,thanks06:08
normanand btw ,I'd fixed your comments about the whitespace06:08
*** dragondm has joined #openstack-marconi06:08
flwangnorman: ok, awesome06:09
*** openstackgerrit has quit IRC06:19
*** openstackgerrit has joined #openstack-marconi06:23
normanflwang:   PS for requirements project submitted06:34
flwangcool06:34
*** oz_akan_ has joined #openstack-marconi06:52
openstackgerritPeng Fei Wang proposed a change to openstack/marconi: Correct the rtype in the function docstrings  https://review.openstack.org/10079306:56
*** oz_akan_ has quit IRC06:57
*** alcabrera is now known as alcabrera|afk06:57
*** renlt has quit IRC06:58
*** renlt has joined #openstack-marconi07:13
*** chandan_kumar has quit IRC07:17
*** flaper87|afk is now known as flaper8707:33
*** jamiehannaford has joined #openstack-marconi07:50
*** oz_akan_ has joined #openstack-marconi07:53
*** oz_akan_ has quit IRC07:57
*** renlt has quit IRC08:06
*** renlt has joined #openstack-marconi08:06
*** renlt has quit IRC08:10
*** renlt has joined #openstack-marconi08:11
*** renlt has quit IRC08:11
*** renlt has joined #openstack-marconi08:11
*** stannie has joined #openstack-marconi08:12
*** renlt has quit IRC08:25
*** renlt has joined #openstack-marconi08:26
*** renlt has quit IRC08:27
*** renlt has joined #openstack-marconi08:28
*** haomai___ has quit IRC08:28
*** renlt has quit IRC08:32
*** renlt has joined #openstack-marconi08:32
*** oz_akan_ has joined #openstack-marconi08:54
flaper87flwang: ping08:55
*** oz_akan__ has joined #openstack-marconi08:56
*** oz_akan_ has quit IRC08:56
openstackgerritFlavio Percoco proposed a change to openstack/marconi: Rename shards to pool  https://review.openstack.org/9646308:57
*** oz_akan__ has quit IRC09:00
*** ykaplan has joined #openstack-marconi09:34
*** mkoderer has quit IRC09:42
openstackgerritA change was merged to openstack/marconi: Rename shards to pool  https://review.openstack.org/9646309:47
*** renlt has quit IRC09:48
*** oz_akan_ has joined #openstack-marconi09:56
*** oz_akan_ has quit IRC10:02
*** oz_akan_ has joined #openstack-marconi10:57
*** oz_akan_ has quit IRC11:01
openstackgerritFlavio Percoco proposed a change to openstack/marconi: Add flavors support to mongodb  https://review.openstack.org/9879311:44
openstackgerritFlavio Percoco proposed a change to openstack/marconi: Start scratching Queue Flavors api  https://review.openstack.org/9877711:44
*** oz_akan_ has joined #openstack-marconi11:58
*** oz_akan_ has quit IRC12:02
*** mwagner_lap has quit IRC12:10
openstackgerritFlavio Percoco proposed a change to openstack/marconi: Add flavors support to mongodb  https://review.openstack.org/9879312:11
*** vkmc has joined #openstack-marconi12:25
*** sriram has joined #openstack-marconi12:36
*** sriram has quit IRC12:36
*** sriram has joined #openstack-marconi12:37
*** sriram has quit IRC12:37
*** sriram has joined #openstack-marconi12:37
vkmcg'morning all!12:38
sriramgood morning vkmc :)12:38
vkmcsriram, how are you today? :)12:38
sriramTired, but doing good :)12:39
srirammarathon tennis session yesterday :P12:39
vkmcw00t! you mean it when you said you wanted to start exercising haha12:40
vkmcthats great12:40
sriramyeah, will start some training for 5k soon, need to get stamina up before that :)12:41
*** Obulpathi has joined #openstack-marconi12:51
*** Obulpathi has quit IRC12:52
*** jmckind has joined #openstack-marconi12:59
*** abettadapur has joined #openstack-marconi13:03
*** oz_akan_ has joined #openstack-marconi13:09
*** tjanczuk_ has quit IRC13:23
*** keith_newstadt has joined #openstack-marconi13:26
*** tjanczuk has joined #openstack-marconi13:29
abettadapurkgriffs13:32
abettadapurkgriffs|13:32
abettadapurkgriffs|afk: ping13:32
abettadapurthat typing thing....13:32
flaper87vkmc: sriram abettadapur gooooooooooooooooooooooooooooooooood morning13:36
flaper87it's so depressing when you ninja-approve a patch and it fails because of a merge conflict13:37
flaper87it breaks the power of that approval13:37
vkmcflaper87, heeeeeeeeey :)13:37
abettadapurgoodmorning :)13:37
vkmcabettadapur, good morning!13:38
abettadapurvkmc: how are you doing today?13:39
*** balajiiyer has joined #openstack-marconi13:40
vkmcabettadapur, a bit sleepy -.- but good, and you?13:41
sriramhey hey hey13:42
sriramgoood morning flaper87 :)13:42
abettadapurpretty good, but i can't wait for lunch13:42
abettadapurhard to work when you are hungry13:43
abettadapur:D13:43
flaper87sriram: good morning to you, sir! I heard you're doing excersice for both of us. Keep it up! :)13:44
*** malini1 has joined #openstack-marconi13:44
vkmctrue that!13:44
flaper87erm, exercise13:44
sriramflaper87: haha. :P13:44
*** AAzza_afk is now known as AAzza13:46
*** tonytan4ever has joined #openstack-marconi13:47
*** mwagner_lap has joined #openstack-marconi13:47
*** malini1 has quit IRC13:48
*** malini1 has joined #openstack-marconi13:48
abettadapurflaper87: i responded to your comment13:49
abettadapurdoes that make sense, or should it be done differently?13:49
*** alcabrera|afk is now known as alcabrera13:49
AAzzagood day all)13:50
alcabrerahey, all. o/13:50
sriramhey alcabrera good morning :)13:52
*** cpallares has joined #openstack-marconi13:52
vkmcmorning AAzza, alcabrera o/13:52
alcabreraI'm so sleepy today. I've gotten my sleep schedule flipped so that I'm waking up around midnight. >.>13:53
sriramthats bad :/13:53
flaper87abettadapur: I proposed 2 different options in reply to your comment. I'd prefer the later. Assume it'll be set and skip if not13:53
*** balajiiyer has quit IRC13:53
* flaper87 wishes he could say that13:53
*** balajiiyer has joined #openstack-marconi13:53
flaper87Waking up around midnight means I went to bed before midnight13:53
flaper87I don't remember the last time that happened13:54
flaper87but I'm happy there's responsible people like alcabrera that take care of having a good sleep schedule13:54
* flaper87 should learn how to do these things13:54
flaper87damnit13:54
alcabreralol13:55
sriramflaper87: I had installed a sleep monitoring app before, but I uninstalled it after it kept saying I'm not getting enough sleep :P13:55
*** Obulpathi has joined #openstack-marconi13:56
*** Obulpathi has quit IRC13:56
abettadapurflaper87: ok, I did it the second way. I didn't realize superclasses could access subclass attributes13:57
*** Obulpathi has joined #openstack-marconi13:57
AAzzai defend my master thesis) yahhooo)13:57
abettadapurpython continues to surprise me13:57
sriramAAzza: congratulations!!13:58
AAzzasriram: thanks)13:58
malini1AAzza: Congratulations!!13:59
vkmccongrats AAzza! :)14:00
alcabreranice, AAzza! :D14:01
*** nosnos has quit IRC14:03
openstackgerritA change was merged to openstack/marconi: Updated from global requirements  https://review.openstack.org/9903314:04
flaper87AAzza: CONGRATTTTTTTTSSSSSSSSSSSSSSSSSS!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!14:04
AAzzahahaha, thanks)14:05
malini1Tht is real serious congratulation!!14:06
malini1I will remember tht next time :)14:06
cpallaresAAzza: Congrats! :)14:10
openstackgerritOpenStack Proposal Bot proposed a change to openstack/marconi: Updated from global requirements  https://review.openstack.org/10091314:13
*** oz_akan_ has quit IRC14:19
*** oz_akan_ has joined #openstack-marconi14:19
*** amitgandhi has joined #openstack-marconi14:19
openstackgerritOpenStack Proposal Bot proposed a change to openstack/marconi: Updated from global requirements  https://review.openstack.org/10091314:20
*** haomaiwang has joined #openstack-marconi14:23
*** oz_akan_ has quit IRC14:24
*** tonytan4ever has quit IRC14:26
*** rwsu has joined #openstack-marconi14:29
*** oz_akan_ has joined #openstack-marconi14:31
*** balajiiyer has quit IRC14:31
*** AAzza_ has joined #openstack-marconi14:37
*** AAzza_ has left #openstack-marconi14:37
*** AAzza has quit IRC14:40
openstackgerritA change was merged to openstack/marconi: Make storage.utils.keyify py3k compatible  https://review.openstack.org/10008414:40
*** reed has joined #openstack-marconi14:45
*** tonytan4ever has joined #openstack-marconi14:46
*** balajiiyer has joined #openstack-marconi14:47
*** balajiiyer has quit IRC14:50
*** oz_akan_ has quit IRC14:50
*** balajiiyer has joined #openstack-marconi14:51
*** jergerber has joined #openstack-marconi15:04
*** kgriffs|afk is now known as kgriffs15:07
kgriffsrebasing...15:09
kgriffs:p15:09
flaper87kgriffs:  I hear you15:17
openstackgerritKurt Griffiths proposed a change to openstack/marconi: Update hacking version and fix new violations  https://review.openstack.org/10072715:22
*** oz_akan_ has joined #openstack-marconi15:22
kgriffssriram: i think we can use that benchmarking tool to do some localhost testing when working on perf changes, but we also need to have it running for every patch submitted to gerrit, on a dedicated benchmarking cluster15:25
kgriffsI think getting that in place should take priority over another iteration on the benchmarking tool15:26
kgriffsI was thinking of something like this:15:26
kgriffs1. get basic producer-consumer merged15:26
kgriffs2. set up performance gate15:27
kgriffs3. add to producer-consumer a measure of how long it takes, on average to post a message and then consumer it on the other hand (end-to-end clock)15:27
kgriffs4. add benchmarks for some other messaging patterns15:27
kgriffssriram: what do you think?15:27
sriramhey kgriffs, just saw the messages.15:28
srirammakes sense15:28
*** oz_akan_ has quit IRC15:35
*** mkoderer has joined #openstack-marconi15:42
*** malini1 has quit IRC15:44
*** cpallares has quit IRC15:45
*** cpallares1 has joined #openstack-marconi15:45
*** jmckind has quit IRC15:46
sriramkgriffs|afk: what do you mean by another iteration though? like adding oslo.config?15:46
flaper87sriram: kgriffs didn't we agree to call it `bench` ?15:50
sriramkgriffs: flaper87: sure, I can rename it. :)15:50
flaper87sriram: oh, I'm asking because I probably missed something15:51
flaper87.... and because I prefer bench MUAHAHAHHA15:51
flaper87(just kidding)15:51
*** haomaiwang has quit IRC15:51
sriramflaper87: no, I started out the patch before we had the conversation and I forgot about changing it. :P15:51
*** haomaiwang has joined #openstack-marconi15:52
flaper87malini: hey :)15:53
flaper87malini: http://lists.openstack.org/pipermail/openstack-infra/2014-June/001370.html15:53
flaper87are you subscribed to the os-infra mailing list?15:53
flaper87There's some talk to get f20 in with mongo >=2.415:53
flaper87and Eoghan mentioned Marconi as well15:53
flaper87If you can, it would be nice to chime in there too.15:54
flaper87otherwise, I'll try to do it15:54
srirambbl lunch15:57
sriramkgriffs, flaper87: we can discuss a bit more on marconi-bench when I'm back :)15:57
kgriffssriram: no, not oslo.config. I just mean just generally speaking adding more features to benchmarking15:57
sriramflaper87: thanks for the comments on the review.15:58
sriramkgriffs: gotcha15:58
flaper87np15:58
flaper87I think we should clean up the one we have without adding new features15:58
sriramI'll work on the suggestions and have another patch in.15:58
flaper87let it land and add more things later15:58
flaper87yeah, just what kgriffs said :P15:58
* flaper87 likes re-stating kgriffs15:58
sriram:P15:58
openstackgerritOpenStack Proposal Bot proposed a change to openstack/marconi: Updated from global requirements  https://review.openstack.org/10091316:07
*** haomaiwang has quit IRC16:08
*** balajiiyer has quit IRC16:11
*** tonytan4ever has quit IRC16:12
*** tonytan4ever has joined #openstack-marconi16:14
vkmcflaper87, saying hi AMQP style http://paste.openstack.org/show/84405/16:15
vkmcI'm digging into the transport layer16:15
flaper87vkmc: ohhh, AWESOME!16:16
flaper87vkmc: how's that going?16:17
flaper87any trouble so far?16:17
flaper87vkmc: did you look into Azure's documentation?16:17
*** chandan_kumar has joined #openstack-marconi16:18
vkmcflaper87, good so far... there are some headers we use that may require some thinking to add to amqp, so I'm checking that out16:18
vkmcflaper87, I did yeah!16:18
flaper87vkmc: anything interesting?16:19
flaper87I've no idea how they did it16:19
*** stannie has quit IRC16:20
*** amalagon has quit IRC16:22
vkmcflaper87, there is not much to take from there16:22
vkmcat least in this early stages16:22
vkmcwell... the interaction with the Service Bus using Proton gives us a good idea of how the clients would use it16:24
vkmcso I'm doing reverse engeneering to implement a similar behavior from Marconi :) while keeping our ways of course16:26
* kgriffs is setting up a new, hardened ZNC box16:27
kgriffsflaper87: I'm curious: any python-based IRC bouncers you know of?16:27
kgriffsor Rust-based. :D16:27
*** oz_akan_ has joined #openstack-marconi16:29
*** balajiiyer has joined #openstack-marconi16:29
vkmcs/engeneering/engineering16:29
flaper87kgriffs: lol @ rust based :P16:31
flaper87mmh, wasn't there something written using twisted ?16:31
kgriffsno idea16:31
*** balajiiyer1 has joined #openstack-marconi16:31
flaper87vkmc: re reverse engineering. That sounds perfect :P16:31
*** malini1 has joined #openstack-marconi16:31
flaper87let me know as soon as you've something up16:31
flaper87I'm curious16:32
keith_newstadtflaper87, kgriffs: available to talk about high throughput apis?16:32
kgriffsflaper87, vkmc: IronPython FTW!16:32
flaper87(plop)16:33
flaper87keith_newstadt: sure, kgriffs ?16:33
kgriffsyep16:33
*** balajiiyer has quit IRC16:33
vkmckgriffs, Python for .NET?16:33
* vkmc hides16:34
keith_newstadtas we discussed - we're building a high throughput message queue as input to stream processing16:35
keith_newstadtusing kafka16:35
keith_newstadtwe need a layer on top to enforce keystone auth and multitenancy, and are considering different types of streaming apis16:36
keith_newstadtbut we'd also like the functionality marconi provides16:36
keith_newstadtand would like to build it in a way that it would useful to the community16:37
keith_newstadtsounds like you all have been talking about similar requirements and scenarios16:37
keith_newstadtkgriffs: how would you want to start the discussion?16:37
flaper87keith_newstadt: I think we should first know what the requirements are in terms of throughput16:38
keith_newstadtbtw, we've identified engineers for this task.  not looking for a free ride.  :)16:38
flaper87it'd also be interesting to know what kind of semantics you guys are interested in16:39
flaper87is the current API good enough? (in terms of semantics)16:39
keith_newstadtthroughput requirements - we've got an internal customer that is currently doing about 400,000 messages/sec in their legacy, non-openstack environment16:39
*** amitgandhi has quit IRC16:39
keith_newstadtso we need to support that at least, and we expect the throughput from that customer to grow over time16:40
keith_newstadtas well as other customers to use the service as well16:40
flaper87keith_newstadt: how many nodes? (If you can share)16:40
flaper87I don't think that's a 1 node thing, is it?16:40
keith_newstadtyou mean messaging service / kafka nodes?  definitely not a 1 node thing16:41
flaper87keith_newstadt: yeah kafka nodes.16:41
keith_newstadtwe've got initial PoC that does about a 40,000 on three kafka VMs16:42
flaper87awesome, we're very interested in that kind of numbers16:42
keith_newstadtwe still need to do performance tuning and hardware tuning16:42
flaper87because we're currently working on benchmarking marconi16:42
keith_newstadtcool.  we'd be happy to collaborate on testing tools and methodologies as well16:43
flaper87awesome, awesome, awesome.16:43
flaper87now, what about the API?16:43
keith_newstadtas for api semantics, we need to support multiple producers and multiple consumers16:43
flaper87It's very different from Kafka, as it's right now.16:43
*** haomaiwa_ has joined #openstack-marconi16:43
keith_newstadtand consumers need to be able to pull from the queue at independent places/times16:44
flaper87ok, that we can do16:44
flaper87erm, Marconi16:44
flaper87:P16:44
keith_newstadtcool16:44
keith_newstadtregarding the api, we're primarily looking at how to get throughput and performance16:44
flaper87I think the first step would be doing a similar benchmark on Marconi16:44
keith_newstadte.g. open up a single connection and push many messages through it16:45
flaper87and see how close (<- me being optimistic) we are to the required performance16:45
flaper87Having a kafka store driver sounds... sexy :P16:45
keith_newstadtseems like it would be hard to be comparable with the overhead of a rest based api16:46
flaper87The thing is that I believe we'll have the same issues we had with AMQP16:46
keith_newstadtwhat issues did you have?16:46
flaper87keith_newstadt: mostly mapping the existing API with AMQP.16:46
flaper87keith_newstadt: I listed some here: http://blog.flaper87.com/post/53a09586d987d23f49c777bf/16:47
keith_newstadtcool, thanks, i'll read it offline16:47
flaper87TL;DR: Kafka API (and AMQP's) are stream-based16:47
flaper87it's quite hard to map the feed semantics of Marconi's API to them16:48
flaper87Note that I've just read papers and docs about kafka. I've never used it before.16:48
*** haomaiwa_ has quit IRC16:48
keith_newstadti might be oversimplifying, so set me right if i am...16:48
flaper87so, if I'm wrong, feel free to send me a gentle STFU and I will :P16:48
keith_newstadti was thinking that there would be a set of streaming apis (http streaming, long polling, web sockets...) that would live alongside the existing marconin ones16:49
keith_newstadtand no reason why you couldn't use both16:49
keith_newstadte.g. producer uses existing APIs, but a consumer gets them on a stream16:49
keith_newstadtor is your concern that it woudl be difficult to support the existing marconi apis on kafka?16:50
kgriffsI think we may be conflating two things16:50
kgriffsthere is the protocol and the semantics16:50
kgriffsthe 1.x API semantics are what may be difficult or impossible to map16:50
flaper87keith_newstadt: what kgriffs said16:51
flaper87keith_newstadt: we can support different transmission protocols, but the api semantics remain the same.16:51
keith_newstadtimpossible to map to streaming, or to kafka?16:51
keith_newstadtflaper87: your blog post might help bring me up to speed on this16:52
flaper87keith_newstadt: I highlighted some of the most crutial issues we had, there are definitely others. If they are not true for kafka, then lets talk about it :)16:53
flaper87keith_newstadt: it'd be impossible to map to a "pure-streamed" API.16:53
keith_newstadti see you discuss this in the post.  let me catch up, and we'll talk more again later.16:54
keith_newstadtlet's also say in touch on benchmarking16:54
keith_newstadtwhere are you at in that process?16:54
*** malini1 has quit IRC16:54
*** malini1 has joined #openstack-marconi16:55
flaper87keith_newstadt: https://review.openstack.org/#/c/98875/16:55
flaper87sriram: is working on that small tool. We'll start benchmarking prod/con16:55
flaper87with different loads16:55
flaper87I'm very interested in knowing what tests you guys did16:55
flaper87we can try to replicate them on marconi16:56
keith_newstadtso far our tests have been based on our specific customer data16:56
keith_newstadtwe're working on more generic tests that we can share16:56
keith_newstadtthey are also using the kafka protocol directly, which we'd like to move away from16:56
flaper87keith_newstadt: could some of your engineers take a look at that patch and share some thoughts?16:57
flaper87that'd be amazing16:57
keith_newstadtflaper87: i'll see what i can share from what we have16:57
keith_newstadtsure16:57
flaper87awesome16:57
flaper87so, I gotta run now but I'll be back in a bit16:57
keith_newstadtno problem.  they'll be interested to see it as well16:57
keith_newstadtme too.  you're on vacation for the next few days?16:58
flaper87keith_newstadt: yes^Wconference16:58
keith_newstadtcool.  have fun, will reach out when you're back.16:58
flaper87feel free to ping me on email/irc/phone/twitter/etc16:58
keith_newstadtwill do!16:59
flaper87I'll be on-line, just with limited internet access16:59
flaper87kk, gtg now... for realz16:59
*** balajiiyer1 has quit IRC17:03
*** amitgandhi has joined #openstack-marconi17:04
*** amitgandhi has quit IRC17:05
*** ykaplan has quit IRC17:16
*** tonytan4ever has quit IRC17:23
*** tonytan4ever has joined #openstack-marconi17:23
sriramflaper87: last we discussed, did we want bench to be a part of the marconi package?17:27
sriramkgriffs: ^17:28
kgriffsum17:28
kgriffsI don't remember17:29
kgriffslet's see17:29
*** balajiiyer has joined #openstack-marconi17:29
*** balajiiyer has quit IRC17:30
sriramlet me checkout the logs of the prior meeting.17:30
sriramI think we discussed it there.17:30
kgriffsi think it is simplest to say, just pip install marconi and you get marconi-server and marconi-bench17:30
kgriffsas a developer, that would be nice. I can just "pip install -e marconi" and then as I change stuff I can benchmark it17:31
sriramyes that is what I have it doing now, but I made bench a part of marconi's package.17:31
*** oz_akan_ has quit IRC17:31
kgriffsflaper87 may have had a good reason to make it a separate package, though still same repo/project17:31
sriramcool17:31
*** oz_akan_ has joined #openstack-marconi17:32
kgriffsflaper87: but I can't think of what the benefit is to make it separate17:32
*** ykaplan has joined #openstack-marconi17:33
*** amitgandhi has joined #openstack-marconi17:34
*** tonytan4ever has quit IRC17:34
*** tonytan4ever has joined #openstack-marconi17:35
*** rossk has joined #openstack-marconi17:43
*** haomaiwa_ has joined #openstack-marconi17:44
*** tonytan4ever has quit IRC17:45
*** tonytan4ever has joined #openstack-marconi17:45
*** malini1 has quit IRC17:48
*** haomaiwa_ has quit IRC17:49
*** malini1 has joined #openstack-marconi17:52
*** balajiiyer has joined #openstack-marconi18:18
tjanczukkeith_newstadt, flaper87: interesting kafka discussion. are you online?18:20
keith_newstadttjanczuk: i'm here18:22
flaper87sriram: kgriffs I wouldn't mind packaging it as part of marconi18:25
flaper87FWIW, you get redis-bench when you install redis18:25
flaper87so go for it18:26
sriramyeah, that was going to be my argument as well :P18:26
sriramflaper87: \m/18:26
flaper87keith_newstadt: tjanczuk: I am now but I'll have to go in a bit18:26
tjanczukkeith_newstadt: I am curious about your goals for Kafka in the context of Marconi. In particular, would you want to expose Kafka transport directly to apps, or put it behind the Marconi APIs?18:27
tjanczukkeith_newstadt: what is the best way to connect besides IRC here?18:27
keith_newstadttjanczuk: goals - high throughput as a feed in/out of Storm-based stream processing as a service18:28
keith_newstadttjanczuk: we are currently exposing kafka in our very early environment, but don't want to do that in production or as we take on more internal customers18:29
keith_newstadttjanczuk: we also need MQ as a service in general, outside of stream processing18:29
flaper87I don't think exposing kafka would be a good idea in this context18:29
keith_newstadtflaper87: 100% agree18:30
keith_newstadtwe also need auth, multi-tenancy, and TLS18:30
keith_newstadtand so using marconi is our first choice18:30
*** cpallares1 is now known as cpallares18:30
keith_newstadtwe just need to make sure we can get the throughput we need18:30
tjanczukkeith_newstadt: so you expect Marconi to provide a protocol abstraction over Kafka? Or do you expect Marconi to *replace* Kafka while offering the same perf charactestics?18:31
keith_newstadttjanczuk: email: keith_newstadt@symantec.com / twitter: @knewstadt18:31
keith_newstadttjanczuk: we expect to build or help build the capability and contribute it18:32
flaper87IIUC, any of those options work for keith_newstadt18:33
keith_newstadttjanczuk: marconi on top of kafka is my initial thought, but the requirements are primary over the implementation18:33
flaper87am I right?18:33
flaper87I mean, as long as you get the throughput you need, you're fine with marconi18:33
keith_newstadtflaper87: throughput, durability, and multiple consumers, yes18:34
flaper87(side note: Pls, keep me in the loop of any off-line discussion. Thanks)18:34
keith_newstadtflaper87: absolutely18:34
flaper87(side note2: I'd prefer to keep them all in the open, if possible)18:34
flaper87keith_newstadt: right18:34
flaper87keith_newstadt: I think Marconi can provide all that18:34
flaper87and well, as we said before, we should first benchmark it18:34
flaper87I mean, I've real doubts about its current performance18:35
flaper87but I'm sure that whatever the output is, it can be improved18:35
*** oz_akan_ has quit IRC18:35
flaper87that said, the kafka driver is a really sexy idea18:35
keith_newstadtflaper87: do you think the current api can do on the order of 10K req/sec?18:35
flaper87and we should add it to the evaluation of possible drivers18:35
keith_newstadtper node that is18:35
*** balajiiyer has quit IRC18:35
tjanczukI think it would be a challange for Marconi to provide perf on par with Kafka without a connection-oriented transport.18:35
keith_newstadti agree18:36
* flaper87 agrees18:36
flaper87I mean, not in the current state but I'd like to know where we are18:36
flaper87before doing any changes18:36
keith_newstadtwhat kinds of changes do you have in mind?18:36
flaper87Just to make sure the plans we make are oriented in the right direction18:37
tjanczukOn that note, I did some experiments with WebSockets on top of AMQP 0.9 at http://github.com/tjanczuk/narconi. The implementation is in Node not Python. I was able to get ~6000 msg/s out of a single connection.18:37
tjanczukWithout any tuning.18:37
flaper87keith_newstadt: A new transport with persistent connection is something we've discussed.18:38
flaper87tjanczuk: did you see my websocket hack?18:38
flaper87well, mine was on top of marconi18:38
tjanczukI saw it.18:38
flaper87anyway, that said, I think we should first see how we can tune the existing transport/API/storage and wok on something else in parallel18:38
tjanczukI did not make the cut for me because I wanted full fidelity with the connection-oriented nature of the backend. Which I expect an implementation on top Kafka would also need.18:38
keith_newstadttjanczuk, flaper87: we built a long polling solution that did around 14K/sec on a physical server, written in C18:39
flaper87What I'm trying to say is that I think our current implementation can do better, regardless the possibility of adding new transports18:39
flaper87transports/storage/etc18:39
keith_newstadtdo you have a sense of what you'd be shooting for yet?  if we think the upper end of the current implementation is good enough, then it's a good route to pursue18:40
keith_newstadtbut if we think it's low enough to warrant a streaming approach, perhaps it's something to pursue in parallel?18:40
tjanczukkeith_newstadt: what do you mean by "streaming approach"? connection oriented transport or message access semantics?18:41
keith_newstadtwould the durability requirement require a new engine regardless?18:41
*** kgriffs is now known as kgriffs|afk18:42
keith_newstadti'm thinking in terms of the api performance - so a single connection of some type, one authorization, tenancy enforcement, and then pushing the messages through18:43
flaper87keith_newstadt: durability is enforced by the storage layer. As of now, mongodb is our recommended solution for production environemnts and it provides durability18:43
tjanczukI am fully with you. Fully.18:43
*** tonytan4ever has quit IRC18:43
flaper87keith_newstadt: some history:18:43
tjanczukDo you care about web-friendliness of the protocol?18:43
*** tonytan4ever has joined #openstack-marconi18:43
flaper87Through the last year we've talked about websocket, raw tcp, amqp (still tpc) and zmq as possible transport drivers18:44
flaper87The one we've been leaning towards to is websocket because of it's web capabilities18:44
*** oz_akan_ has joined #openstack-marconi18:44
flaper87And it's possible to do websocket connections from libraries too.18:44
flaper87s/it's/its/18:45
flaper87s/it's/its web capabilities/18:45
flaper87zmq is out of discussion for now18:45
flaper87and there's an on-going research on the amqp side ( vkmc )18:45
flaper87but again, I think we need to becnhmark the current version and see where the bottleneck is18:46
flaper87and where we should be focusing efforts18:46
keith_newstadtflaper87: makes sense18:46
flaper87That said, I do think a persistent transmission protocol will happen in the future18:47
flaper87for sure18:47
keith_newstadttjanczuk: ideally web-friendly, but that's a p2 or p318:47
keith_newstadtflaper87: sounds encouraging18:47
keith_newstadtflaper87: let's continue the conversation when you're back, and how we can put our engineers to work on it18:48
tjanczukkeith_newstadt: Performance aside, does the semantics of the current Marconi HTTP APIs satisfy your requirements?18:48
*** balajiiyer has joined #openstack-marconi18:48
flaper87keith_newstadt: sure, we can also follow-up on email since the clock for juno is ticking really fast18:49
keith_newstadttjanczuk: let me put together more formal/clearer requirements and i'll share18:49
keith_newstadttjanczuk: i think so, but i will verify18:50
flaper87keith_newstadt: btw, thanks a lot! It was a great discussion and I'm happy to know you're interested in the project18:50
flaper87huge welcome to you and your team :)18:50
keith_newstadtflaper87: thanks :)18:51
*** AAzza_afk has joined #openstack-marconi18:53
*** AAzza_afk is now known as AAzza18:53
tjanczukkeith_newstadt: I would be curious about your thoughts on the HTTP/WebSocket API prototype at https://github.com/tjanczuk/narconi. With this POR I was trying to enable using connection oriented backends and optimize for performance level that I don't think can be achieved with standard HTTP. Some of the problems this POR solves are outlined in the AMQP 0.9 investigation (most of which I expect to apply to Kafka as well): http://lists.openstack.or18:58
*** openstackgerrit has quit IRC19:06
*** openstackgerrit_ has joined #openstack-marconi19:08
*** openstackgerrit_ is now known as openstackgerrit19:08
*** abettadapur has quit IRC19:09
*** tonytan4ever has quit IRC19:09
*** tonytan4ever has joined #openstack-marconi19:10
*** kgriffs|afk is now known as kgriffs19:18
*** jamiehannaford has quit IRC19:26
*** ykaplan has quit IRC19:29
*** balajiiyer has quit IRC19:29
*** jamiehannaford has joined #openstack-marconi19:39
flaper87kgriffs: hey, can I get your review on the queue flavors patches ?19:39
kgriffsthanks for the reminder. I need to do a bunch of reviews today19:40
keith_newstadttjanczuk: thanks for the link.  i will take a look19:41
flaper87keith_newstadt: thank you for the reviews :)19:41
flaper87erm, kgriffs ^19:43
flaper87:P19:43
*** malini1 has quit IRC19:43
* flaper87 should use 2 letters tab, instead of 1 letter tab19:43
* kgriffs stops playing with arch linux and goes to do some reviews19:44
*** cpallares has quit IRC19:45
flaper87kgriffs: archlinux rocks19:46
flaper87erm, I mean, fedora19:46
* flaper87 hides19:46
flaper87:P19:46
kgriffsLOL19:47
*** amitgandhi has quit IRC19:49
*** balajiiyer has joined #openstack-marconi19:51
*** balajiiyer has quit IRC19:51
*** balajiiyer has joined #openstack-marconi19:51
*** amitgandhi has joined #openstack-marconi19:53
*** Obulpath_ has joined #openstack-marconi19:57
*** Obulpathi has quit IRC19:57
*** AAzza is now known as AAzza_afk19:59
*** Obulpath_ has quit IRC20:05
*** Obulpathi has joined #openstack-marconi20:05
*** alcabrera is now known as alcabrera|afk20:07
sriramgunicorn 19.0.0 seems to be broken, cant serve marconi with it. 18.0 works though, is it the same for everyone?20:13
*** tonytan4ever has quit IRC20:20
kgriffsI haven't tried it20:32
kgriffswhat is the error?20:32
kgriffsflwang: can you sanity-check this an merge if it looks OK? https://review.openstack.org/#/c/100727/20:34
*** amalagon has joined #openstack-marconi20:34
*** tonytan4ever has joined #openstack-marconi20:35
*** sriram has quit IRC20:40
*** oz_akan__ has joined #openstack-marconi20:52
*** oz_akan__ has quit IRC20:52
*** oz_akan_ has quit IRC20:53
kgriffsmalini: or could you merge it? ^^^20:57
*** keith_newstadt has quit IRC21:00
*** mwagner_lap has quit IRC21:00
* kgriffs wonders where everyone is?21:00
kgriffsflwang: ping21:01
vkmc*crickets*21:02
flwangkgriffs: pong21:06
flwangI saw your patch, will take a look soon21:07
flwangkgriffs: wow, seems now you're really good at playing with whitespace21:10
flwangand blank lines :D21:10
kgriffsyep. all kinds of "fun".21:10
flwangkgriffs:  looks good for me, and I believe you're an expert on whitespace now21:12
flwangapproved21:12
* kgriffs puts on "Expert Whitespacer" badge21:12
kgriffsthanks!21:12
flwangkgriffs: btw, as for this patch https://review.openstack.org/#/c/100745/21:13
flwangkgriffs: we need to pin the msgpack version21:13
kgriffsoh21:13
flwangI have submitted a patch for requirements project and it's merged now21:13
kgriffsah, that was my next question. :D21:14
flwangok21:14
flwangshould we wait for the robot to sync it?21:14
flwangflwang: what's your question?21:14
kgriffsalready happening, looks like: https://review.openstack.org/#/c/100913/3/requirements.txt21:15
kgriffsso you can abandon https://review.openstack.org/#/c/100745/21:15
flwangcool21:15
kgriffsnice catch, thanks for taking care of that!21:16
flwangmy pleasure :)21:16
*** jergerber has quit IRC21:26
*** fifieldt has quit IRC21:27
*** tonytan4_ has joined #openstack-marconi21:32
openstackgerritA change was merged to openstack/marconi: Update hacking version and fix new violations  https://review.openstack.org/10072721:34
openstackgerritA change was merged to openstack/marconi: Updated from global requirements  https://review.openstack.org/10091321:34
*** tonytan4ever has quit IRC21:35
*** jamiehannaford has quit IRC21:37
*** fifieldt has joined #openstack-marconi21:40
*** balajiiyer has quit IRC21:52
*** tonytan4_ has quit IRC22:00
*** mkoderer has quit IRC22:02
*** Obulpathi has quit IRC22:02
*** oz_akan_ has joined #openstack-marconi22:06
*** amitgandhi has quit IRC22:10
kgriffsFYI: first draft of reviewer's guide, including proposed comment prefixes22:14
kgriffshttps://wiki.openstack.org/wiki/Reviewer_Guide_(Marconi)22:14
kgriffsthis is linked from the home page22:15
flwangin case you guys interested in http://www.redhat.com/enovance/?utm_content=buffer24e09&utm_medium=social&utm_source=plus.google.com&utm_campaign=buffer22:16
kgriffseeenteresting22:21
flwangkgriffs: Reviewing ang being reviewed   I think it should 'and', right?22:24
kgriffsyeah22:26
kgriffsnice nice catch22:26
flwangI have fixed it22:26
flwangjust wanna confirm22:26
kgriffsrock on22:26
flwangkgriffs:  btw, I like the prefixes you posted22:27
flwangkgriffs: though I need some time to be used to22:27
kgriffsthanks. I just brainstormed a starting list and we can refine them as we try them out22:27
flwangyep, it can make the review more efficient22:28
flwangas for the redhat+enovance, seem redhat is aim to be the giant of Openstack22:29
*** flaper87 is now known as flaper87|afk22:41
*** mwagner_lap has joined #openstack-marconi22:47
*** amitgandhi has joined #openstack-marconi23:21
*** amitgandhi has quit IRC23:25
*** kgriffs is now known as kgriffs|afk23:52

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!