Tuesday, 2014-03-25

*** whenry has joined #openstack-marconi00:07
*** openstack has joined #openstack-marconi00:20
*** flaper87 is now known as flaper87|afk00:26
*** whenry has quit IRC00:37
*** mwagner_lap has joined #openstack-marconi01:23
*** rwsu has quit IRC01:26
*** nosnos has joined #openstack-marconi01:29
*** rwsu has joined #openstack-marconi01:42
*** malini_afk is now known as malini01:54
*** haomaiwang has joined #openstack-marconi02:31
*** rossk has quit IRC02:33
*** malini is now known as malini_afk03:00
openstackgerritA change was merged to openstack/marconi: Fix functional config file instructions  https://review.openstack.org/7992503:27
*** haomaiw__ has joined #openstack-marconi03:28
*** nosnos has quit IRC03:29
*** haomaiwang has quit IRC03:32
*** chandankumar_ has joined #openstack-marconi03:33
*** chandankumar_ has quit IRC03:40
*** chandankumar_ has joined #openstack-marconi03:55
*** chandankumar_ has quit IRC04:23
*** nosnos has joined #openstack-marconi04:25
*** chandan_kumar has quit IRC05:48
*** prashanthr_ has quit IRC06:58
*** saju_m has joined #openstack-marconi07:30
*** prashanthr_ has joined #openstack-marconi07:47
prashanthr_flwang: Hi :)07:48
*** flaper87|afk is now known as flaper8707:51
*** saju_m has quit IRC08:50
*** saju_m has joined #openstack-marconi08:52
*** chandan_kumar has joined #openstack-marconi09:13
*** prashanthr_1 has joined #openstack-marconi09:20
*** mkoderer has joined #openstack-marconi09:21
*** ykaplan has joined #openstack-marconi09:25
*** saju_m has quit IRC09:38
*** nosnos has quit IRC09:39
*** saju_m has joined #openstack-marconi09:56
*** ykaplan has quit IRC10:42
*** saju_m has quit IRC11:07
*** saju_m has joined #openstack-marconi11:08
*** saju_m has quit IRC11:09
*** mwagner_lap has quit IRC11:12
*** saju_m has joined #openstack-marconi11:15
*** malini_afk is now known as malini11:17
*** ykaplan has joined #openstack-marconi11:36
*** saju_m has quit IRC11:38
*** saju_m has joined #openstack-marconi11:38
*** tedross has joined #openstack-marconi11:58
*** malini is now known as malini_afk12:09
*** oz_akan_ has joined #openstack-marconi12:21
flwangflaper87: ping12:29
*** sriram has joined #openstack-marconi12:34
*** sriram has quit IRC12:34
*** sriram has joined #openstack-marconi12:34
flaper87flwang: pong12:35
*** saju_m has quit IRC12:41
*** oz_akan_ has quit IRC12:44
*** oz_akan_ has joined #openstack-marconi12:44
flwangflaper87: i'm testing the client12:45
flaper87flwang: awesome! Thanks a lot for that!12:46
*** mwagner_lap has joined #openstack-marconi12:46
flwangflaper87: i'm wondering if we should add a sample against the keystone auth strategy mode12:46
flaper87flwang: sure, that would be amazing12:47
flwangflaper87: seems the client also supports a private conf file, is it?12:47
flaper87flwang: mmh, what do you mean?12:48
flaper87the client doesn't have support for config files12:48
*** seiflotfy_ has joined #openstack-marconi12:51
flwangflaper87: ok, i see.12:51
*** seiflotfy has quit IRC12:52
*** saju_m has joined #openstack-marconi12:55
flwangflaper87:  so for keystone mode, I can just pass the param like auth_opts={'os_auth_token': 'xxxx'}, right?12:56
flwangflaper87: ah, no12:57
*** saju_m has quit IRC12:59
flwangflaper87: would you mind briefing how the client side to talk with marconi server with the keystone mode?13:00
*** saju_m has joined #openstack-marconi13:00
*** saju_m has quit IRC13:04
*** saju_m has joined #openstack-marconi13:05
flaper87flwang: you're correct, to talk to keystone you need to pass the auth_strategy and the rest of the options in auth_opts13:08
*** saju_m has quit IRC13:09
*** saju_m has joined #openstack-marconi13:10
flwangflaper87: do you have a sample?13:10
flwangto construct the params13:10
flaper87flwang: erm, mmh, yeah, sure13:10
* flaper87 runs away13:10
* flaper87 hides from flwang13:10
* flaper87 is scared because he lied to flwang13:11
* flaper87 doesn't really have a sample13:11
flaper87flwang: https://github.com/openstack/python-marconiclient/blob/master/marconiclient/transport/request.py#L4513:11
flwangflaper87: haha13:11
flaper87flwang: we should have more tests about that13:12
flaper87flwang: this is all we have so far: https://github.com/openstack/python-marconiclient/tree/master/tests/auth13:12
flwangflaper87: i have reviewed both of them actually :D13:13
flaper87flwang: :( then my hands are empty :(13:13
flaper87flwang: you could write some samples and tests ;)13:14
flaper87flwang: https://review.openstack.org/#/c/82820/13:14
flwangflaper87: haha, ok, i see.13:14
flaper87flwang: I'm really sorry about that :/13:14
flwangflaper87: then we can backport them to client /sample13:14
flaper87flwang: yup13:15
flwangflaper87: good, let's rock on13:15
*** mpanetta has joined #openstack-marconi13:19
*** ChanServ changes topic to "the gerrit event stream is currently hung, blocking all testing. troubleshooting is in progress (next update at 14:00 utc)"13:21
*** amitgandhi has joined #openstack-marconi13:23
openstackgerritFlavio Percoco proposed a change to openstack/python-marconiclient: Move auth tests under the unit package  https://review.openstack.org/8282013:26
*** jcru has joined #openstack-marconi13:26
*** ChanServ changes topic to "OpenStack Queuing and Notification Service || Smile :D || Meetings every Tuesday @ 15:00 UTC || Wiki: https://wiki.openstack.org/wiki/Marconi || Paste: http://paste.openstack.org/ || Send messages and make some noise :D"13:29
*** ykaplan has quit IRC13:37
*** alcabrera has joined #openstack-marconi13:54
*** balajiiyer has joined #openstack-marconi13:54
alcabreraGood morning! :D13:54
*** ykaplan has joined #openstack-marconi13:55
flaper87alcabrera: GOOOOOOOOOOOOOOOOOOOOOOD morning13:55
alcabreraflaper87: hey! How goes? :)13:56
flaper87alcabrera: good... You ?13:56
flaper87more writing less coding :/13:57
flaper87btw, I forgot I have an internal presentation today and I won't be able to attend the meeting :(13:57
alcabreraI'm doing alright this morning. It may also be a day of more writing, less coding for me. :P13:57
alcabreraOh, alright.13:57
alcabreraI'll be attending, so feel free to ping me after wards, or...13:58
alcabrerais there any topic you'd like to see come up in today's meeting?13:58
flaper87I should learn that calendars are not just to create events but also to check when you have events13:58
alcabreraheh - write-only mediums? ;)13:58
flaper87fun part is that I know my memory sucks13:58
flaper87There's a part of me that refuses to admit it and use calendars13:59
flaper87anyway, I think we were planning to discuss a lot about what happened last week13:59
flaper87and prepare a small plan forward13:59
flwangflaper87: seems we need an 'auto-discovery' for https://github.com/openstack/python-marconiclient/blob/master/marconiclient/transport/request.py#L4513:59
flaper87I started writing Marocni's FAQ today13:59
flaper87alcabrera: flwang kgriffs_afk malini_afk https://etherpad.openstack.org/p/draft-marconi-faq13:59
flaper87Answering some of the questions that came up and adding a couple more that I've heard13:59
flaper87I'm planning to give a very quick walkthrough that FAQ on Thursday's hangout14:00
alcabreraflaper87: thanks for kickstarting that!14:00
flwangflaper87: to read the correct url/api from keystone service catalog14:00
flwangflaper87: it's a good start14:01
flwangflaper87:  the faq14:01
alcabrerare: calendaring14:02
alcabrerathat reminds me that I need to add a thing or two to my own14:02
flaper87alcabrera: YOU'RE WELCOME! >.>14:03
flaper87(that was re calendaring)14:03
*** alcabrera is now known as alcabrera|afk14:03
*** ykaplan has quit IRC14:06
*** malini_afk is now known as malini14:06
*** AnastasiosAndron has joined #openstack-marconi14:12
*** whenry has joined #openstack-marconi14:18
*** ykaplan has joined #openstack-marconi14:19
*** alcabrera|afk is now known as alcabrera14:21
*** oz_akan_ has quit IRC14:24
*** oz_akan_ has joined #openstack-marconi14:26
*** Kuwagata has joined #openstack-marconi14:29
*** cpallares has joined #openstack-marconi14:33
*** jmckind has joined #openstack-marconi14:35
openstackgerritSriram Madapusi Vasudevan proposed a change to openstack/marconi: fix() : introduce division import  https://review.openstack.org/8283814:45
sriramalcabrera,flaper87,flwang,malini: Could you guys look at the above patch? Thanks!14:47
alcabrerasriram: nice catch.14:47
* alcabrera had already looked at it14:47
sriramawesome! :)14:47
flwangsriram: will +2 after Mr. Jenkins says it's ok14:48
alcabreraI'll +2 after I finish writing an email. I hope jenkins beats me to the punch. :P14:48
maliniI hope I can beat alcabrera ;)14:48
alcabreramalini: oh yeah! +2 powers. :D14:48
malinijust kidding14:48
alcabreraLet's all join together and give sriram +1014:48
alcabrera(which amuses me, because +10 in binary -> +2)14:49
* sriram binary puns14:49
alcabrera10**10 -> 100  -- only in binary14:50
flwangyep, in the developer world, there are 10 kinds of people, one know what's binary, one doesn't :)14:50
sriramThat's the developer 10114:50
alcabrerasounds like a super hero group -- the developer *five*14:51
*** reed has joined #openstack-marconi14:51
*** oz_akan_ has quit IRC14:57
*** oz_akan_ has joined #openstack-marconi14:58
alcabreramarconi team meeting in 2 minutes14:58
*** kgriffs_afk is now known as kgriffs14:59
alcabrerakgriffs: o/15:00
malinimegan_w:  #openstack-meeting-alt15:01
kgriffsflaper87: meeting?15:01
alcabrerakgriffs: flaper87 is out, for now. Internal presentation.15:02
megan_wmalini: thanks :)15:02
*** saju_m has quit IRC15:13
mpanettaWee binary15:18
*** whenry has quit IRC15:20
*** vkmc has joined #openstack-marconi15:26
*** vkmc has quit IRC15:26
*** vkmc has joined #openstack-marconi15:26
*** ayoung has quit IRC15:37
*** ayoung has joined #openstack-marconi15:48
*** jergerber has joined #openstack-marconi15:54
kgriffsmegan_w: I guess we have to register for the marketing mtg?16:02
megan_wkgriffs: i'll take care of it16:02
megan_woh wow, its on a phone bridge16:03
kgriffsyeah. I didn't realize. Expected it to be IRC.16:04
kgriffsmegan_w: I think Niki Acosta @ Rackspace may be on that team16:04
megan_woh good, i can talk to her beforehand16:04
kgriffsmaybe you could reach out to here and see what we should do16:04
megan_wby the way, my registration is only good for one participant, apparentlhy16:05
megan_wso if you're going to join, you should register too16:05
kgriffsthe graduation requirement is: "Project should have engaged with marketing team to check suitable official name"16:05
*** alcabrera is now known as alcabrera|afk16:05
kgriffsmegan_w: yep, I registered16:05
kgriffsOK, wrt g+ hangouts...16:06
*** alcabrera|afk is now known as alcabrera16:07
kgriffsI think the things that most useful...16:07
kgriffs1. voice16:07
kgriffs2. show slides or virtual whiteboard16:07
kgriffs3. know who is talking, when16:07
kgriffs4. have minutes/notes16:08
alcabrerabecause we sacrifice logs for (4) w/ g+, we'll need a minutes-taker16:08
kgriffsg+ works well for smaller teams, but as we grow, I don't want to leave people out16:08
kgriffsalcabrera: yep!16:09
alcabreraI'd be happy to help with that.16:09
kgriffsrock on16:09
flaper87alcabrera: malini kgriffs re hangout. It will be a proper presentation about Marconi16:10
flaper87I'll just take that change to also go through the FAQ16:10
flaper87at least the most common question16:10
flaper87(not that it hasn't been done before)16:10
alcabreraflaper87: ¡gracias!16:11
kgriffsflaper87: ah, ok16:11
kgriffsflaper87: so, we were talking about doing a monthly checkpoint meeting with the team16:11
kgriffswe talk about how our community and development process is working, what we should continue and what we need to do better on16:12
flaper87kgriffs: sounds A.M.A.Z.I.N.G16:12
kgriffsand in part 2 we track where we are wrt graduation16:12
flaper87like really, we should do this monthly, invite people16:12
flaper87and even if it's just us, we'll talk about stuff that have happened16:13
flaper87and that will happen16:13
kgriffsflaper87: so, the question is, do we try to do this with video conferencing or stick to IRC16:13
kgriffs(video conferencing ala Hangouts, GoToMeetings, etc.)16:13
kgriffsmalini: ^^^16:14
flaper87we need a different way for people to reach out to us16:14
flaper87that's a good way community-wise16:14
kgriffsok. So, with G+ we are limited to 10 participants methinks16:14
flaper87PLUS, that will be recorded and people (or even us) can go back to it16:14
flaper87are we?16:15
flaper87mmh, didn't know that16:15
kgriffsflaper87: I think so. There may be a way for voice-only via telephone for more than 10 - there used to be, but last time I went looking I couldn't find that option.16:15
alcabrera"Hangouts are limited to 10 people at a time with video and audio and additional 10 users by phone."16:17
alcabrerakgriffs, flaper87: ^^16:17
flaper87I wonder how the Marconi hangout was setup :/16:18
oz_akan_kgriffs: flaper87 malini what is the vision statement for Marconi?16:18
kgriffsCommercial solutions allow higher numbers of users, but obviously aren't free16:19
kgriffsoz_akan_: I've tried to write that down in the first couple sections on our wiki, https://wiki.openstack.org/wiki/Marconi16:20
kgriffsbut, I think we've some work to do to make our key value proposition / mission crystal-clear16:21
kgriffsmegan_w: does Rackspace have any conferencing accounts we can leverage for Marconi meetings?16:22
flaper87kgriffs: I'd say community-understandable :D16:22
oz_akan_mission defines how to achieve a vision, I guess that first sentence under mission is vision16:22
megan_wkgriffs: i have a phone bridge16:22
megan_wwe also have webex, which *might* have video16:23
oz_akan_and "variety of distributed application messaging patterns in an efficient, scalable and highly-available manner" has a very big scope16:23
kgriffsI've had bad experiences with webex in the past. :(16:23
kgriffshas anyone tried GoToMeeting or another alternative?16:23
* flaper87 has had really bad experiences with webex16:23
oz_akan_kgriffs: should we try to define the scope better? with better I mean narrowing it16:24
kgriffsoz_akan_: agreed.16:24
mpanettawebex is horrible in my experience.  And it leaks like a sieve, not that it would matter for an OS project.16:25
oz_akan_does web-friendly mean REST?16:25
kgriffsThe mission statement is hard because it is supposed to be very brief. This was actually copied from the governance incubation YAML16:25
kgriffs(although I've now modified it slightly)16:26
kgriffsmaybe rename mission -> vision and summary -> mission ?16:26
kgriffsoz_akan_: more or less. I guess we could just say that16:27
oz_akan_vision is something like this: to provide web friendly api then mission statement is provide a web friendly api by using RESTful interface etc..16:27
oz_akan_so yes, mission statement describes how to achieve that desired vision16:27
oz_akan_and I think we need to identify the audience16:28
oz_akan_for marconi so we can decide easily what is in the scope of that vision, what is not16:28
kgriffs(made a couple tweaks if you want to refresh)16:28
oz_akan_and vision statement may change after achieved]16:28
oz_akan_that vision discards flavors16:29
oz_akan_we won't write it now, I am just trying to gain some interest in defining the scope of the project as early as possible16:30
oz_akan_I am scared of "backend" changes, rest vs amqp etc...16:31
oz_akan_we invested a year in mongodb and now it is all for nothing16:32
kgriffsnot all for nothing16:32
kgriffsthere are lots of people who have no problem running mongodb16:32
kgriffsbut there are also lots who would rather not, and need other options16:32
oz_akan_now we know what we could use what not, that is a good start and if we define who is our audience we can understand what is more important to provide16:32
kgriffs+1 for defining the audience16:33
oz_akan_kgriffs: we need to be able to move queues from one shard to another, queues on multiple shards etc. these will be different on different storage types16:33
oz_akan_so if mongodb loses focus, that is going to effect the only in production marconi implementation on earth16:34
oz_akan_if we keep mongodb as the primary, that seems to block us next time as well16:34
*** AnastasiosAndron has quit IRC16:36
*** balajiiyer has quit IRC16:38
kgriffsI started an etherpad. Let's brainstorm on there and discuss with flaper87 et al.16:43
kgriffsBefore the summit, I want to overhall our wiki site and have all this incorporated16:43
kgriffsalcabrera: ping16:44
kgriffsamitgandhi: ping16:45
kgriffsbleh. My internets are clogged16:46
kgriffsguys, I'm gonna move locations. Be back online in a bit16:46
*** ykaplan has quit IRC16:50
alcabrerakgriffs: pong16:52
* alcabrera catches up on mongodb discussion16:52
*** vkmc has quit IRC16:55
kgriffsalcabrera: I was just thinking about the pop messages thing16:57
oz_akan_kgriffs:  I added who are not our users, I guess it helps too16:58
kgriffshttp POST marconi.example.org/v1.1/queues/crunchy/claims?limit=10&pop=true16:59
kgriffsthat's another idea16:59
kgriffsoz_akan_: definitely!16:59
flwangoz_akan_: ping17:00
kgriffsgotta run for a little while. be back l8r17:00
flwangoz_akan_: except the operation doc you contributed, is there any doc can be shared? now i'm working on setting up Marconi server with production starndard17:01
alcabrerakgriffs: I like that better, associating pop w/ claims17:01
alcabrerakgriffs: o/17:02
oz_akan_flwang: I don't have anything else at the moment17:04
oz_akan_flwang: at some point we may have saltstack formulas17:04
flwangoz_akan_: so you're using saltstack to deploy Marconi?17:04
flwangoz_akan_: how to handle the scale issue?17:05
oz_akan_yes we use saltstack17:05
flwangoz_akan_: so based on RAX policy, can the salkstack formulas be shared?17:06
oz_akan_it could be shared if we didn't have anything specific to RAX which we have17:07
*** rossk has joined #openstack-marconi17:07
oz_akan_that is why I was saying at some point we can open them and that point is based on the time we can invest to separate the formulas17:07
flwangoz_akan_: my question is how do you guys monitor the performance of Marconi and deploy more to handle the rquest spike?17:07
oz_akan_at this point we deploy more than we need and won't have to act for spikes17:08
flwangoz_akan_: got it, thanks17:08
oz_akan_scaling is not that granular17:08
flwangoz_akan_: so are you willing to monitor the performance/metering marconi?17:08
*** balajiiyer has joined #openstack-marconi17:09
oz_akan_we will probably make some changes in the way we deployed it17:09
oz_akan_to be able to scale more granulary17:09
*** kgriffs is now known as kgriffs_afk17:10
flwangI'm going to contribute Ceilometer so as to leverage Heat to get better scale?17:10
flwangoz_akan_: any comments?17:10
*** balajiiyer has quit IRC17:13
oz_akan_I am not sure if we need anything other than  saltstack to scale automatically17:14
flwangoz_akan_: ok17:15
flwangoz_akan_: thanks17:15
oz_akan_I guess you meant scaling automatically we could use heat right/17:16
flwangoz_akan_: exactly17:19
*** balajiiyer has joined #openstack-marconi17:19
*** balajiiyer has left #openstack-marconi17:29
*** balajiiyer has joined #openstack-marconi17:29
amitgandhikgriffs_afk: you pinged?17:31
*** flaper87 is now known as flaper87|afk17:41
alcabreraGetting close: https://github.com/linsomniac/python-memcached/pull/26 (Python 3.3 support)17:42
amitgandhi+1 on using ceilometer and heat to scale automatically17:55
amitgandhiit gives us better openstack integration17:56
* amitgandhi has caught up on thread17:56
amitgandhire google hangouts for the retro's why not do the "hangouts on air" so that everyone can at least watch if not participate directly17:57
amitgandhikgriffs_afk: ^17:57
*** alcabrera is now known as alcabrera|afk18:00
*** balajiiyer has left #openstack-marconi18:01
*** kgriffs_afk is now known as kgriffs18:02
kgriffsamitgandhi: that could work18:05
kgriffsre hangouts on air18:05
kgriffsflaper87|afk: ^^^18:05
kgriffsamitgandhi: was wondering about your thoughts on adding a pop param to the claims resource instead of the messages resource18:06
*** amitgandhi has quit IRC18:09
*** amitgandhi has joined #openstack-marconi18:09
*** oz_akan_ has quit IRC18:21
*** oz_akan__ has joined #openstack-marconi18:34
*** oz_akan__ has quit IRC18:37
*** amitgandhi has quit IRC18:39
*** oz_akan_ has joined #openstack-marconi18:40
kgriffsfolks, I am going to sign off for a bit so I can focus on teh codez. ttfn!18:49
*** kgriffs is now known as kgriffs_afk18:50
*** malini is now known as malini_afk19:16
*** openstackgerrit has quit IRC19:18
*** openstackgerrit has joined #openstack-marconi19:18
*** alcabrera|afk is now known as alcabrera19:19
*** balajiiyer has joined #openstack-marconi19:19
*** balajiiyer has quit IRC19:19
*** balajiiyer has joined #openstack-marconi19:19
*** kgriffs_afk is now known as kgriffs19:21
*** amitgandhi has joined #openstack-marconi19:24
*** amitgandhi has quit IRC19:24
*** amitgandhi has joined #openstack-marconi19:25
*** balajiiyer has left #openstack-marconi19:25
*** oz_akan_ has quit IRC19:27
*** oz_akan_ has joined #openstack-marconi19:27
*** balajiiyer has joined #openstack-marconi19:33
*** tjanczuk has joined #openstack-marconi19:41
tjanczukHello folks, anyone there? I was wondering if there have been any performance measurements of Marconi done, and if the results are available somewhere I can look?19:42
kgriffstjanczuk: hi!19:43
tjanczukHello again.19:43
kgriffstjanczuk: balajiiyer and malini_afk have done some tests and I've asked them to post some data on the wiki. I don't have an ETA on it yet, because the team has been in the midst of Summit and Juno planning, but feel free to ping them19:44
tjanczukThanks. Is there a good way to ping them other than through here?19:45
kgriffstjanczuk: do you have specific requirements in mind?19:45
kgriffstjanczuk: go ahead and send an email to [marconi] on the mailing list and I will get them to reply19:45
kgriffstjanczuk: it would be great if in that email you could mention a range of numbers you are after19:47
kgriffsfor example, posting X number of messages per second or whatever19:47
tjanczukI am generally trying to set my expectations in terms of latency and throughput as a function of the size of the deployment, number of queues, number of producers/consumers, type of backend, size of backend cluster etc.. Trying to understand how Marconi would do in a large multi-tenant deployment.19:47
kgriffstjanczuk: the reason I ask is we want to have some concrete targets as we work on Marconi19:47
kgriffstjanczuk: ok, that is some good background you could include in your email as well.19:48
kgriffsif you have certain variables in mind, please mention those as well19:48
kgriffssuch as number of queues, size of cluster, etc.19:48
tjanczukSounds good. Let me move this to the mailing list.19:48
kgriffscool, thanks!19:49
tjanczukBy the way, I just learned I am going to the summit in Atlanta, so I hope we will have a chance to meet in person.19:53
alcabreratjanczuk: awesome! Hope to see you there. :)19:55
sriramalcabrera, kgriffs : Could you have a look at the patch, Jenkins seems to have +1'd it.  https://review.openstack.org/#/c/82838/20:02
alcabrerasriram: +220:03
sriramalcabrera: Thanks.20:04
alcabreraGood night, everyone. :)20:07
*** alcabrera has quit IRC20:08
openstackgerritA change was merged to openstack/marconi: fix() : introduce division import  https://review.openstack.org/8283820:11
*** tjanczuk has quit IRC20:15
*** sriram has quit IRC20:22
*** vkmc has joined #openstack-marconi20:24
*** vkmc has quit IRC20:24
*** vkmc has joined #openstack-marconi20:24
*** tjanczuk has joined #openstack-marconi20:38
kgriffsbalajiiyer: ping21:05
amitgandhikgriffs: re pop parameter21:09
amitgandhiso you would claim a message with pop, which hides the message from all others, and deletes it from the queue?21:09
amitgandhiall in one step?21:09
kgriffsamitgandhi: correct.21:09
amitgandhiso claim semantic is maintained, and a subsequent delete call is now unecessary21:09
kgriffsyou can compare that to adding a pop query param to messages resource21:10
kgriffsamitgandhi: right, although the only loose end21:10
kgriffsis you have a claim created that isn't really necessary21:10
kgriffsactually, i think claims aren't implemented as separate records in mongo, so that is ok21:10
amitgandhiif you pop the message, and crash before processing it, you lose the message right (since no claim ttl)21:10
kgriffsbut they exist in SQL21:10
amitgandhidoes pop infer FIFO?21:11
kgriffsamitgandhi: that is correct, you lose it. That is the tradeoff, but for some kinds of events that is OK (or so one user told me)21:11
kgriffsamitgandhi: pop is identical to claiming, then deleting all messages before the claim expires. messages are claimed in FIFO order21:11
* kgriffs checks21:12
amitgandhikgriffs: i bought that up since we were considering dropping FIFO guarantee21:12
amitgandhikgriffs: i dont think i understood the loose end about claims existing21:13
*** tedross has quit IRC21:17
*** balajiiyer has quit IRC21:23
*** Alex_Gaynor has quit IRC21:39
*** Alex_Gaynor has joined #openstack-marconi21:41
*** oz_akan_ has quit IRC21:45
*** russellb has quit IRC21:47
*** metral has quit IRC21:47
*** russellb has joined #openstack-marconi21:48
*** mwagner_lap has quit IRC21:49
*** metral has joined #openstack-marconi21:49
kgriffsamitgandhi: oh, just that conceptually you are creating a claim, deleting it's messages, but leaving the claim around21:50
kgriffs(if you add the pop=true param)21:50
amitgandhihow does the ttl on it work today (ie how does it get GC when the message TTL expires?)21:51
amitgandhican we not code the sql driver to delete the message on pop()21:51
*** malini_afk is now known as malini21:52
*** cpallares has left #openstack-marconi21:53
*** mpanetta has quit IRC21:54
kgriffsso, for the mongo driver we just tag each claimed message with a claim ID and TTL, so there is no separate claim resource. We just filter out "claimed" messages on the fly by checking current timestamp against the claim ttl21:56
malinitjanczuk: ping21:56
kgriffsfor sql, it is in another table and I haven't looked at that code for a while...21:56
tjanczukmalini: hello21:56
kgriffsbut I know messages themselves are deleted on the fly during any message GET operation - not ideal, obviously, and so we had a work item to create a garbage collector21:57
malinitjanczuk: hello..Thank you for your interest in Marconi benchmarks21:57
tjanczukI am all ears21:58
kgriffsmalini: please also reply on the ML for posterity. :)21:58
kgriffsamitgandhi:  anyway, if we aren't planning on supporting sqla driver in production deployments, then that optimization is less important21:58
malinitjanczuk:I have some tsung xmls & set up instruction to run the benchmarks. Are you interested in doing some tests with those?21:58
amitgandhikgriffs: agreed21:59
malinikgriffs: sure..I just wanted to chk if tjanczuk wanted our test scripts21:59
kgriffsamitgandhi: wrt FIFO, I think we may still want to offer it, but it would be optional21:59
malinimainly to make sure if he wants additional operations etc21:59
kgriffsmalini: cool21:59
*** jmckind has quit IRC22:00
tjanczukmalini: at some point I may want to use them for private testing; in the meantime I was hoping you would have some numbers to share.22:00
malinitjanczuk: I blew away the tsung servers with the reports after the benchmarks22:01
maliniBut we can re-run them & share in the ML22:01
malinihopefully within the next 2-3 days22:02
tjanczukThat would be very helpful. Do you recall any numbers off the top of your head? Ballparks?22:03
maliniwe were able to reach ~ 2000 rps, using 4 webservers22:08
maliniThe latency was in the milliseconds range…But I dont remmeber accurately22:09
*** jcru has quit IRC22:15
amitgandhikgriffs: my updates are up on the etherpad for the vision/mission22:15
amitgandhiim heading out now22:16
kgriffstjanczuk: fwiw, we are planning to start benchmarking marconi with a cron job or something and charting the results over time. Performance will be a big focus during Juno.22:16
kgriffsamitgandhi: thanks!22:16
kgriffstjanczuk: we are also looking for independent tests to put up on the wiki22:16
*** amitgandhi has quit IRC22:17
tjanczukkgriffs: sounds good22:17
tjanczukmalini: re 2000 rps, was it publishing throughput, or end to end? Was it using Mongo as the storage or comething else? How was storage scaled?22:18
tjanczukAlso, how much performance optimization work had been done on Marconi to date? Are bottlenecks understood?22:20
malinitjanczuk: We did the tests with external cluster generating the load . We had a tsung cluster generating traffic. We used Mongo as the backend22:21
maliniWe used Marconi's sharding to scale. But oz_akan will be the right person to give you more info on how we scaled the DB22:22
malinitjanczuk: We went through a round of perf optimizations last November (tht was when we did the tests)22:23
tjanczukWhat was the bottleneck in the test? CPU? Memory? Network?22:23
tjanczukAnd at which layer? Transport or DB?22:23
maliniIt was probably at the DB22:23
maliniBut I will let flaper87|afk or oz_akan to answer that22:24
*** balajiiyer has joined #openstack-marconi22:24
malinitjanczuk: These are good questions.Can you append that to your email in the ML ?22:24
maliniWe will attempt to answer all of those along with benchmark numbers from a new test22:25
tjanczukWill do, thanks. Do you remember what the structure of the test was? One queue or multiple queues, how many publishers and subscribers per queue, message size?22:26
malinitjanczuk: multiple queues..22:26
tjanczukAlso when you say "latency in millisecond range", what was included in that timing?22:26
tjanczukWere messages produced and consumed one at a time or in bulk?22:27
*** balajiiyer has quit IRC22:28
malinitjanczuk: Let me just update the github repo with the  tsung config file22:28
maliniThat'll answer most of your questions22:28
tjanczukGreat, can you post a link back here once you do please?22:29
maliniOn further thoughts..I will do that..But IRC discussion is easier to understand than an XML22:29
maliniUpdating the repo now22:29
*** Kuwagata has quit IRC22:30
*** oz_akan_ has joined #openstack-marconi22:31
*** jergerber has quit IRC22:44
*** oz_akan_ has quit IRC22:45
malinitjanczuk: See https://github.com/rackerlabs/csi-marconi22:49
maliniLet me know if you are interested in additional scenarios beyond what we currently have in tsung.xml22:51
tjanczukThanks, I need some time to process this. Once you re-run the tests (in a couple of days?), how are you going to share the results?22:55
malinitjanczuk: mailing list22:56
maliniwe can chat further in the ML/ here if you need more clarifications22:56
*** mwagner_lap has joined #openstack-marconi23:02
tjanczukGreat, thanks.23:02
*** malini has left #openstack-marconi23:04
*** whenry has joined #openstack-marconi23:50
*** mwagner_ has joined #openstack-marconi23:54

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!