Monday, 2014-03-24

*** oz_akan__ has joined #openstack-marconi01:07
*** oz_akan__ has quit IRC01:11
*** nosnos has joined #openstack-marconi01:36
*** flwang has quit IRC01:39
*** oz_akan_ has joined #openstack-marconi02:07
*** oz_akan_ has quit IRC02:12
*** flwang has joined #openstack-marconi02:18
*** arnaud has joined #openstack-marconi02:48
*** arnaud has quit IRC02:52
*** arnaud has joined #openstack-marconi02:53
*** arnaud has quit IRC03:05
*** arnaud has joined #openstack-marconi03:06
*** oz_akan_ has joined #openstack-marconi03:08
*** oz_akan_ has quit IRC03:13
*** ekarlso has quit IRC03:48
*** ekarlso has joined #openstack-marconi03:48
*** oz_akan_ has joined #openstack-marconi04:09
*** oz_akan_ has quit IRC04:14
*** arnaud has quit IRC04:16
*** arnaud has joined #openstack-marconi04:58
*** chandankumar has quit IRC05:14
*** nosnos_ has joined #openstack-marconi05:16
*** nosnos has quit IRC05:19
*** chandan_kumar has joined #openstack-marconi05:25
prashanthr_hi flwang,05:51
flwangprashanthr_: hi05:51
prashanthr_I was able to fix the bug :
prashanthr_The functional tests run fine on my local machine, but on Jenkins the errors I am observing are not related to my code change.05:54
prashanthr_My patch URL :
flwangprashanthr_: ok, I will take a look at it after a session in 30 minutes05:55
flwangthanks for letting me know the patch :)05:55
prashanthr_flwang: Sure , thank you ! You are welcome ! :)05:56
*** flwang has quit IRC06:01
*** flwang has joined #openstack-marconi06:09
*** nosnos has joined #openstack-marconi06:19
*** nosnos_ has quit IRC06:22
*** saju_m has joined #openstack-marconi06:35
*** arnaud has quit IRC06:57
*** saju_m has quit IRC06:59
*** saju_m has joined #openstack-marconi07:12
*** saju_m has quit IRC07:17
*** saju_m has joined #openstack-marconi07:17
*** seiflotfy has quit IRC07:19
*** saju_m has quit IRC07:22
*** saju_m has joined #openstack-marconi07:23
*** seiflotfy has joined #openstack-marconi07:24
*** flwang has quit IRC07:28
*** saju_m has quit IRC07:30
*** saju_m has joined #openstack-marconi07:31
*** flwang has joined #openstack-marconi07:33
*** flaper87|afk is now known as flaper8707:40
*** jamie_h has joined #openstack-marconi08:34
*** renlt has joined #openstack-marconi08:42
*** prashanthr_1 has joined #openstack-marconi08:53
*** arnaud__ has joined #openstack-marconi08:57
*** jamie_h has quit IRC08:58
*** jamie_h_ has joined #openstack-marconi08:59
*** jamie_h_ has quit IRC09:00
*** arnaud__ has quit IRC09:02
*** prashanthr_1 has quit IRC09:14
*** flwang has quit IRC09:20
*** renlt has quit IRC09:35
*** njirap has joined #openstack-marconi09:39
*** ykaplan has joined #openstack-marconi09:40
*** AAzza has joined #openstack-marconi09:53
*** flwang has joined #openstack-marconi10:14
flwangflaper87: morning10:20
flaper87flwang: hey hey10:23
*** mkoderer has joined #openstack-marconi10:23
flwangflaper87: how are you? good weekend?10:23
flwangflaper87: seems we have a lot items on our Juno to-do list10:24
flwangwill we touch them in tomorrow weekly meeting?10:24
flwangI would like to  see what I can help10:24
flaper87flwang: yeah, that's a rough list, we need to make it shorter and prioritize10:25
flaper87it'd be helpful if you could go through the proposed bps in that list and add some comments10:26
flwangflaper87: will do, it's on my today to-do list :D10:26
flwangbtw, I think we need to also confirm the final to-do list is the expectation from the TC/community POV10:27
flwangflaper87: in other words, we need to interlock with those guys who has concerns now to make sure they are on the same page for all the next stages10:28
flaper87flwang: yes and no, I think we need the TC feedback on the plan but we also need to run most of those things through the community10:28
flwangflaper87: I know i'm saying some rubbishs10:28
flaper87the work we've to do is bigger that, we'll need to split it in several areas (we are kinda doing this in the etherpad)10:29
flaper87there are some technical things that we've to take care of and other community aspects that we need to clear up10:29
flwangflaper87: yep10:29
flwangflaper87: okay, so i'm a new comer, I'm standing by to take some tasks :)10:30
flaper87flwang: btw, if you get a chance, could you help me testing the client library?10:36
flwangfor claims?10:36
flaper87Testing some scenarios should be enough10:36
flaper87like, queue's create, deletes, messages post, delete, claims etc10:37
flaper87flwang: also, it'd be really cool if you could give it a try with keystone10:37
flaper87we'd like to release a new version of the client library10:37
flwangflaper87: any deadline?10:38
flwangdue date10:38
flaper87tomorrow :P10:38
flaper87nah, just kidding10:38
flaper87so, I'd really, really, really like to release the new version tomorrow or Wednesday10:38
flaper87the sooner the better10:38
flwangflaper87: i see. I can start to do it from today, but I can' guarantee to complete by tomorrow :D but I want to promise complete it by Wed, is it acceptable?10:40
flwangbut you know, now it's my night, so most of the work I will start from my tomorrow10:41
flaper87flwang: yeah, don't worry! Just take your time and do what you can10:45
flaper87flwang: I'll be testing it too10:45
flaper87I just wanted someone that double checks my: "It Works" :D10:46
flwangflaper87: ah, sounds good.10:46
flwangflaper87: I will interlock with you to make sure you're not lonely :D10:47
* flaper87 doesn't feel lonely anymore10:47
*** arnaud__ has joined #openstack-marconi10:58
*** arnaud__ has quit IRC11:03
*** nosnos has quit IRC11:23
*** chandankumar_ has joined #openstack-marconi11:29
njirapmarconi ping.. anyone11:31
njirapi dont have MARCONI_TESTS_CONFIGS_DIR in my os environ where is it?11:34
*** prashanthr_1 has joined #openstack-marconi11:36
*** tedross has joined #openstack-marconi11:37
AAzzanjirap: hmm... i'm new here, but i think you could set it via export MARCONI_TESTS_CONFIGS_DIR=/somedir/with/config11:40
njirapAAzza i cant see the dir anywhere even the marconi github repo doest have it11:41
AAzzanjirap: the default dir is tests/etc as far as i can see (in tox.ini)11:43
njirapAazza ok lemmie try it11:44
njirapAazza i have it now.. thanks11:47
*** nosnos has joined #openstack-marconi11:54
flaper87fwiw, MARCONI_TESTS_CONFIGS_DIR is optional :)12:10
*** chandankumar_ has quit IRC12:13
njirapflaperf87 ... am running line by line in python shell.. am tryna understand the functional tests.. so its kinda compulsory for me :)12:31
*** jeremyh28 has joined #openstack-marconi12:46
*** arnaud__ has joined #openstack-marconi12:59
*** mpanetta has joined #openstack-marconi13:00
*** amitgandhi has joined #openstack-marconi13:03
*** arnaud__ has quit IRC13:04
*** amitgandhi has joined #openstack-marconi13:04
*** mpanetta has quit IRC13:05
*** mpanetta has joined #openstack-marconi13:06
*** mpanetta has quit IRC13:07
*** mpanetta has joined #openstack-marconi13:07
*** oz_akan_ has joined #openstack-marconi13:07
*** mwagner_lap has quit IRC13:09
*** rustlebee is now known as russellb13:23
*** nosnos has quit IRC13:24
*** nosnos has joined #openstack-marconi13:25
*** nosnos has quit IRC13:29
*** jcru has joined #openstack-marconi13:32
*** jeremyh28 has quit IRC13:43
*** balajiiyer has joined #openstack-marconi13:48
*** amitgandhi has quit IRC13:49
*** nosnos has joined #openstack-marconi13:58
*** nosnos has quit IRC14:00
*** nosnos has joined #openstack-marconi14:01
*** amitgandhi has joined #openstack-marconi14:02
*** nosnos has quit IRC14:05
*** nosnos has joined #openstack-marconi14:11
*** saikrishna has joined #openstack-marconi14:11
*** nosnos has quit IRC14:14
*** rwsu has joined #openstack-marconi14:17
*** Kuwagata has joined #openstack-marconi14:21
*** anamalagon has quit IRC14:28
*** alcabrera has joined #openstack-marconi14:35
alcabreraGood morning! :D14:35
flaper87alcabrera: hey heyyy14:37
flaper87how are you doing?14:37
alcabreraI'm feeling soooo laggy today, flaper87. Happy, though.14:37
alcabreraflaper87: how about you?14:38
*** saju_m has quit IRC14:39
*** mwagner_lap has joined #openstack-marconi14:43
*** malini_afk is now known as malini14:44
flaper87alcabrera: happy that you're happy :D14:46
flaper87alcabrera: I'm feeling great, hacking on marconi's client a bit and waiting for the US to wake up :D14:47
malinihello !!14:47
flaper87malini: 'sup girl???? how are you doing?14:52
alcabrerahehe, I'm still not quite awake. :P14:53
alcabrerabut sweet - marconiclient improvements14:53
*** cpallares has joined #openstack-marconi14:53
maliniflaper87: good!!14:53
alcabreramalini: good morning!14:53
malinialcabrera: must be something in not quite awake either14:53
cpallaresmalini, alcabrera, flaper87: Good morning14:53
malinigood morning cpallares!!14:53
flaper87alcabrera: malini in case you are curious :D14:53
flaper87cpallares: hey hey14:54
maliniflaper87: that is awesome!14:54
alcabreracpallares: good morning. :)14:56
jbernardflaper87: what exactly does "ninja approval" mean?14:56
maliniflaper87, flwang: I dont know how easy it is to add client tests..But we'll have tests for those in Tempest.14:56
maliniIts called CLI tests & we havent started anything for tht yet14:57
maliniBut we can start on tht sooner if it'll help14:57
alcabrerajbernard: it's when there's been only one +2 on a review and it gets approved.14:57
alcabrerasometimes, it's out of emergency needs14:57
jbernardahh, i see14:57
alcabreraother times, it's because other +2s were given earlier but was lost between patch sets, and the changes were minor.14:57
alcabrerathat's the gist of it. :)14:58
flaper87jbernard: it's approving like a ninja14:58
alcabreraoh yeah - congrats, malini! You're core now, right? :D14:58
flaper87jbernard: dude, haven't you ever seen ninja turtles ?14:58
alcabreraflaper87: in that case, all ninja approvals require pizza.14:59
alcabrerapreferably supremely chessy pizza14:59
flaper87alcabrera: fuck yeah!14:59
flwangmanili: are you working on adding marconi client test cases for Tempest?14:59
*** kgriffs_afk is now known as kgriffs15:00
maliniflaper87: not yet..but we can get started on tht15:00
alcabrerakgriffs: o/15:00
malinisorry..tht was for flwang15:00
*** arnaud__ has joined #openstack-marconi15:00
kgriffsand...I'm back. ZNC gotchya when adding other servers.15:00
maliniflwang: Somebody else has shown interest in working on Marconi Tempest integration15:00
flaper87malini: you're now officially a marconi-core member, you've the power for making contributors sad or happy by -2ing and +2ing their patches15:01
flwangflaper87: +1 :D15:01
maliniwoohoo..I have the POWER ;)15:01
flwangmalini: sounds cool15:01
jbernardmalini: also, i prefer to have all of my patches ninja approved15:01
flwangmalini: you're the superman now15:01
jbernardmalini: just fyi ;)15:01
malinijbernard: Lets talk about the deal offline ;)15:01
jbernardmalini: great, i will send pizza15:01
maliniOn a serious note, Thank you all! I  will do my best to prove your opinions about me true :)15:02
maliniI look forward to working on the code beyond tests.15:03
flaper87malini: you've already done that. KEep up with the great work!15:03
maliniThanks flaper87!15:03
flaper87jbernard: so, package ?15:03
jbernardflaper87: im reviewing other fedora packages now15:04
jbernardflaper87: this is a requirement (i must be sponsored into the pacakging group before marconi can be approved)15:04
*** cpallares1 has joined #openstack-marconi15:04
cpallares1yay malini congrats!15:04
*** cpallares1 is now known as cpallare15:04
malinithank you cpallares !!15:04
jbernardflaper87: at least, that is my understanding current15:04
*** arnaud__ has quit IRC15:05
flaper87jbernard: erm, what? (plop)15:05
*** cpallare is now known as cpallares_15:05
jbernardflaper87: also, i suppose we'll need the client as well15:05
jbernardflaper87: see the last comment on the bug15:05
flaper87jbernard: yeah but lets sort the server out first, I don't think you need to review other packages to get sponsored15:05
flaper87who's your sponsor?15:05
jbernardflaper87: carlthered15:06
*** jmckind has joined #openstack-marconi15:06
flaper87jbernard: mmh, maybe he has different requirements15:06
flaper87kgriffs: FYI :)15:06
flaper87jbernard: yeah, he has different requirements, then!15:07
jbernardflaper87: no worries, i will update the ticket today15:07
jbernardflaper87: if the packages proves to be in good shape, then i think it won't take much effort to approve15:08
*** cpallares has quit IRC15:08
jbernardflaper87: if you have time to test it, that would be awesome15:08
jbernardflaper87: i can fix any issues in parallel15:08
flaper87jbernard: I will, thanks for your work there, like really thanks!15:09
jbernardflaper87: no problem15:09
kgriffsflaper87: thanks!15:09
*** cpallares has joined #openstack-marconi15:10
malinianybody seen the MagnetoDB discussions in the ML?15:12
kgriffsflaper87, malini: BTW, I spoke with Randall from Heat last Friday and he is going to propose a feature at the ATL that uses Marconi to surface events to end users15:13
kgriffsHe's been wanting to do this for a while, FWIW.15:13
*** cpallares_ has quit IRC15:13
kgriffsso it isn't like this is an artificial use case15:13
flaper87kgriffs: that's useful for Horizon too15:14
amitgandhithere had better be a Hotlanta analogy in there somewhere15:14
alcabrerasweet, re: Heat15:14
flaper87and there's (or will be) a session proposed by one of the horizon cores15:14
amitgandhii wouldnt mind seeing jay-atl1 do something similar for horizon15:14
amitgandhihe's worked on similar types of stuff for an internal product here15:15
alcabreraoh yeah - the hands on: w/ marconi session was approved as an *alternate*15:15
malinialcabrera: what does *alternate* mean?15:15
*** cpallares1 has joined #openstack-marconi15:15
alcabreraif someone decides to not show up and talk, our talk will get a slot15:15
kgriffsflaper87: I was thinking maybe to do a lightning talk on Falcon15:16
alcabreraor could get a slot15:16
kgriffswith this outline15:16
kgriffs1. What it is15:16
alcabreradepending on priority as an alternate15:16
flaper87malini: a pessimist would say: "You ain't gonna make it but we'll let you figure that out yourself"15:16
kgriffs2. Why it is complimentary to other frameworks15:16
kgriffs3. Try it and tell me what you like and don't like15:16
*** cpallares has quit IRC15:17
kgriffsBasically, I need to, in a public forum, say what I've been telling teams all along who have been evaluating Pecan and Falcon15:17
*** cpallares1 is now known as cpallares15:17
*** mwagner_lap has quit IRC15:17
kgriffs"try both and decide which tool works best for you; there are a lot of variables, and I am not going to tell you which one to choose."15:18
mpanettaamitgandhi: *Hot*lanta? We are having a low around 29 tomorrow :P15:18
flaper87kgriffs: sounds good to me!15:18
flaper87we should also have an unconference about Marconi15:18
amitgandhimpanetta: cos we are waiting for someone to bring the heat ;-)15:18
flaper87I'm not taking the talk rejection as our final no15:18
kgriffsflaper87: I don't think there are unconferences anymore15:18
mpanettaamitgandhi: haha true15:18
flaper87kgriffs: yeah, the changed the name to something different but there's still a slot for random talking, IIRC15:19
kgriffsI heard mentions about some parallel series of 15-minute talks or something. Can't remember what that was called. Is that what you were thinking?15:19
amitgandhithe hallways are big at the Georgia Conference Center.  Worst case stand on a chair and start talking lol15:20
flaper87kgriffs: yeah, so this is my plan (cc alcabrera )15:20
alcabrerawe will conquer the hall way track15:21
alcabreranone shall resist the allure of poptarts15:21
alcabrerathey shall queue for them15:21
alcabreraand we will give them the Message15:21
flaper87the 3 of us should submit a one of those 15min lightning talks, request that they're put sequentially. Then, the first will introduce marconi, the second will talk about its internals and the third about deploying it15:21
kgriffsflaper87: do you have info on where to submit those talks? I only heard a brief mention in the climate teem meeting last week15:22
kgriffs(they also got the shaft on talks)15:22
flaper87I think that's not open yet15:22
maliniIt'll be good to have oz_akan_ talk about his real world Marconi deployment experience15:23
kgriffsflaper87: kk, can you watch for that and let us know ASAP so we can submit? I think you and Oz should definitely submit talks. Then one more from one of [me, malini, alcabrera]15:24
kgriffsI don't mind doing one, but I want people to see new faces so they get the idea this Marconi thing isn't just a Flavio-Kurt show15:24
maliniWe need to communicate that we are production ready15:25
oz_akan_malini: and we are in production really15:25
maliniGiven the skeptical emails we had abt performance & tests15:25
malinioz_akan_: exactly, that is what we need to communicate15:25
amitgandhiyeh if oz can talk about his experiences and if we can talk about our real customers15:26
amitgandhimegan_w: should be able to help with the customer part15:26
mpanettamarconi: drop dead easy to deploy. :)15:26
*** mwagner_lap has joined #openstack-marconi15:27
amitgandhiand show real performance metrics from a production deployment (both realtime, as well as stress tests)15:27
maliniamitgandhi: +115:27
*** oz_akan_ has quit IRC15:28
kgriffspls. add notes to the etherpad on this stuff ^^^15:28
amitgandhimessage should be: "Its easy to deploy, we did it, and look at this cool graphs"15:28
kgriffsLet me add a section for the summit15:28
kgriffsamitgandhi: +1 for pictures15:28
*** oz_akan_ has joined #openstack-marconi15:28
kgriffsI think we should have tons of pictures in our talks, on our wiki, etc.15:28
kgriffsnot just graphs, obviously15:28
amitgandhi"fluffy bunnies?"15:29
kgriffswhatever works. :)15:29
kgriffsamitgandhi: fluffy bunnies with pancakes on their heads15:29
amitgandhikgriffs: i want a tshirt this time ;-)15:29
alcabreraoh yeah, t-shirts15:30
oz_akan_performance is very relative, as I understand people are trying to compare this to rabbitmq and alike so won't be impressive for those. I think scalability and ease of management is more remarkable with marconi15:30
mpanettakgriffs: I want your marconi shirt, speaking of :P15:30
mpanettaWhy do people keep comparing marconi to rabbit?15:31
amitgandhioz_akan_: i think performance comparisons need to be made with similar players (sqs, ironio, azure service bus, etc)15:31
oz_akan_and for some it is still not clear whey people need marconi15:31
malinimayvbe its more realistic to compare our performance with SQS/ ?15:31
mpanettaI think so15:31
amitgandhiand customer use cases (how are our production customers using marconi.)15:31
maliniI am sure amitgandhi was looking over my shoulders ;)15:31
amitgandhimalini: no, i just read your mind, and can type faster ;-P15:32
alcabrerathe relative comparison would draw the most attendees, I feel. kestrel/rabbitmq/ as compared to marconi15:32 and sqs would make the most sense15:32
alcabrerait depends, oz_akan_. Is marconi also interested in competing against self-hosted queuing solutions?15:34
maliniWe should also start sharing these numbers in the ML more often, not just when we are ready for grad review ;)15:34
alcabreramalini: +2/A15:35
alcabrerathat A comes with pizza - because it's so ninja15:35
megan_ware you guys still wanting a customer testimony?15:35
malinimegan_w: Yes !15:36
malini& FYI
maliniWe should start paying attention to this15:36
alcabreramalini: ah, sweet15:37
alcabreramagnetodb <=> dynamodb15:38
alcabrerayeah, that's definitely compelling15:38
maliniwe should pitch our needs to the MagnetoDB team15:38
alcabreragiven how much good I've heard of cassandra15:38
amitgandhiso why not just use cassandra?15:38
kgriffsamitgandhi: see my note on the etherpad about cassandra15:39
kgriffsamitgandhi: cassandra is a really bad platform when you have high data churn15:39
*** cpallares1 has joined #openstack-marconi15:39
oz_akan_cassandra is not good with frequent data deletion15:39
amitgandhidue to replication15:39
oz_akan_hmm, magnetoDB is API for Cassandre15:40
oz_akan_so no go15:40
amitgandhiso anyway, my ref above was why build magnetodb based on dynamodb when thats how cassandra came about.  Why no use cassandra instead of forking magnetodb?15:40
kgriffsamitgandhi: not sure about because of replication, but certainly because it is a column store iirc15:40
oz_akan_and few like DynomoDB15:40
kgriffsamitgandhi: oic15:40
kgriffsthought you meant why not use cassandra for Marconi vs. MagnetoDB15:41
oz_akan_this is wh15:41
amitgandhithat was another question, answer helped.15:41
oz_akan_though good to read the comments15:41
oz_akan_i guess best approach would be try and see15:41
amitgandhiApache Cassandra looks very suitable for that case due to having tunable consistency, easy scalability, key-value data model, ring topology and other features what gives us predictable high performance and fault tolerance. Cassandra data model perfectly fits MagnetoDB needs.15:41
flaper87not sure what we're discussing here?15:42
amitgandhiflaper87: magnetodb topic15:42
amitgandhii think we sidebar'd into it =P15:42
kgriffspersonally, I think Redis is our best bet for a non-AGPL data store. Maybe Riak. Other things aren't as battle-tested or you only get the good stuff in the commercial version (i.e., velocityDB)15:42
oz_akan_we need something like redis and a ring algortihm15:42
*** cpallares has quit IRC15:43
*** cpallares1 is now known as cpallares15:43
mpanettaJez, how many DB's are there?!15:43
flaper87TBH, I think we should focus on Redis and amqp based store now15:43
alcabrerampanetta: no less than 2015:43
mpanettaWow.  That seems excessive heh15:44
flaper87We also have to think about what's already being used in the community (although I think this deserves a separate time and discussion)15:44
alcabreraflaper87: +115:45
kgriffsflaper87: I would like for one of us to submit a patch to keystone middleware to use oslo cache15:45
kgriffsthen we can say "use redis for cache and for Marconi"15:45
kgriffsoh, and we need oslo cache to support Redis too. :p15:45
oz_akan_i think it is wrong approach15:45
flaper87yeah :P15:45
oz_akan_we should say redis because it is really good for marconi15:45
oz_akan_not because others use15:45
flaper87oz_akan_: trust me, that's how we think but there's also a community impact that we need to consider15:46
kgriffsoz_akan_:  I agree, but I want all the pieces to be there and let people come to the above realization if they are concerned about having to manage "yet another DB"15:46
flaper87it's not that A because N people use it15:46
flaper87it's A because A is good *AND* other people use it15:46
oz_akan_marconi has to be the best messaging service in the world, and any operator would like to provide the best service and would be willing to maintain it even if it means something to learn15:46
kgriffsoz_akan_: I like the way you think15:46
flaper87oz_akan_: that's the goal15:47
oz_akan_flaper87: yes AND approach if good to have but not something to start with15:47
oz_akan_if = is15:47
*** jeremyh28 has joined #openstack-marconi15:48
oz_akan_marconi's problem, I think is, the willingness to support everything15:48
amitgandhialso everyones perception of good performance and scale is different depending on scale of deployment and intended usecases15:48
kgriffsnobody complains when they have to deploy Redis in order to use Sensu15:48
flaper87marconi won't support everything15:48
kgriffsbecause they *want* sensu15:48
flaper87but it shouldn't forbid other things to be implemented15:49
oz_akan_flaper87: it would, mongodb, sqlalchemy, redis...15:49
flaper87oz_akan_: that's not everything, we've talked about this several times15:49
oz_akan_now it won't as mongodb has license issue, no one likes RDMS..15:49
flaper87we agreed to add 2 more stores15:49
flaper87the rest can live elsewhere15:49
oz_akan_so that I believe makes it complicated15:50
kgriffswith all the work we have coming up on backend stuff, benchmarking, docs, etc. I think we had better stick with HTTP transport for this cycle. My $0.0215:50
oz_akan_we need to choose what marconi is good for, so expectations on performance and scalability will be determined more realistically15:50
kgriffsplus, our talk on supporting other transports makes people confused15:51
oz_akan_kgriffs: +115:51
amitgandhii think an amqp transport/backend will confuse ppl more that we are trying to replace rabbit15:51
oz_akan_I think we shall stay with http until we have the best messaging/queueing service out there over http15:51
kgriffsalso, consider that HTTP 2.0 is just around the corner15:52
kgriffsoz_akan_: +115:52
kgriffsamitgandhi: could be15:52
kgriffsflaper87: what do you think?15:52
flaper87amitgandhi: I disagree, if you put Marconi on top of an amqp store, it should be clearer that they don't go side-by-side15:52
kgriffsflaper87: can you elaborate?15:52
*** arnaud has joined #openstack-marconi15:53
oz_akan_amitgandhi: if we position marconi as a RESTful queuing service (which it was), that can scale, expectations will be similar to SQS. if we say it will support amqp this and that, then race is against messaging software that are installed on 1-2 servers and most in memory15:54
oz_akan_amqp people are looking for something else15:55
*** whenry has quit IRC15:55
flaper87What I mean is that people is concern about 1) Marconi replacing other brokers, 2) marconi not being as fast as rabbit. If we go and say: "Hey, you don't have to drop rabbit, you can deploy marocni on top of it and create several rabbit shards for different purposes"15:55
flaper87it should be clearer that we don't it to be a drop-in replacement but to be a tool that can play nice with the amqp technology that people have already installed15:55
oz_akan_flaper87: if you have a rabbit cluster, why would you replace it with marconi?15:55
flaper87oz_akan_: again, you *don't* have to replace it15:56
amitgandhialso the http adds latency so it will be slower15:56
flaper87plus, rabbit has serious clustering issues which is one of the things we want to fix15:56
flaper87amitgandhi: +115:56
kgriffswhat about the bridge idea15:56
amitgandhieither way i feel redis is the higher priority15:57
kgriffsa more interesting use case to me is this:15:57
kgriffs"I have an existing thing that talks AMQP. I don't want to rip it out over night (or maybe ever). Can Marconi be bridged to AMQP?"15:57
amitgandhikgriffs: why would they switch to marconi?15:57
flaper87amitgandhi: why wouldn't they? Isn't marconi the best messaging service ever ?15:58
*** AnastasiosAndron has joined #openstack-marconi15:58
*** arnaud has quit IRC15:58
kgriffsamitgandhi: I don't think they would switch, they would say, I want AMQP to do this internal X thing, but I want Marconi to be the public face, when I need to communicate with users"15:58
kgriffsor, they may just switch if Marconi's performance is good enough, which for many apps it is15:59
kgriffspeople like to think they need WAAAAAY more than they actually do.15:59
oz_akan_I see this happening in real life, 1- only cloud vendors are installing / managing marconi 2- others consume it as a service 3- only reason someone switches to marconi is to get queuing as  a service15:59
flaper87I really don't see the huge problem here. We shouldn't limit the things marconi can do as far as they  don't require us to break the API15:59
kgriffsI agree, the API is key16:00
oz_akan_4- marconi will always be slower due to http and being over internet 5- so it's audience is different than hard core rabbit users16:00
flaper87We've been focusing on having a nice API since the very beginning16:00
amitgandhiflaper87: +1 but it comes back down to time and energy maintaining backends16:00
flaper87an API that is common through *existing* technologies16:00
kgriffsswift doesn't say "gosh, you can't use X for your backend"16:00
kgriffsbut they don't compromise their API to support X, either16:00
flaper87The main goal has always been to not reinvent the wheel16:00
oz_akan_I think, at this point, deciding on what we don't want to do is more critical than what we want to16:00
oz_akan_flaper87: who do think will be installing marconi?16:01
oz_akan_I don't see anyone other than cloud vendors and some larger companies that provide private clouds16:02
amitgandhii feel it should be "who do we think will be using marconi" - lets think about the customer's and what they want to use it for, rather than the person installing the software on their cloud16:02
flaper87oz_akan_: whoever needs a simple, stable, fast messaging service. We could add to this that people using marconi will likely have multi-tenancy requirements, and they will also likely use multiple stores16:02
flaper87oz_akan_: why not? I could setup marconi for itnernal services16:02
amitgandhicustomers could be app developers using it from their phone apps, or it could be infra apps sending messages around16:03
oz_akan_flaper87: that wouldn't be marconi, that would be single something16:03
kgriffsflaper87: do you think Marconi should consider supporting a MagnetoDB driver?16:03
flaper87kgriffs: I think it should, the question is: SHould this driver be maintained by marconi cores - meaning, live in the code base- ?16:03
oz_akan_installing marconi is not simple, it has sharding, catalog, different layers.16:03
*** reed has joined #openstack-marconi16:04
flaper87oz_akan_: that's still marconi16:04
kgriffsoz_akan_: I think a scrappy startup might run marconi themselves if they have certain needs, like they want to install custom message pipeline filters16:04
flaper87I think we won't ever know all the uses cases marconi can / cannot be good for16:04
oz_akan_that is not marconi is intended for16:04
oz_akan_a startup will run rabbitmq16:04
flaper87really, I think it's wrong to waste time on that topic16:04
oz_akan_to save money as it will require less hardware16:04
*** cpallares1 has joined #openstack-marconi16:04
oz_akan_a startup will use marconi as a service16:04
amitgandhiwe should build something that can be used in different ways and let customers surprize us16:05
oz_akan_it won't even install though16:05
flaper87I could simply say: Take the list of people using SQS and, they can all use marconi16:05
oz_akan_flaper87: +116:05
kgriffsflaper87: would you mind going to the next magnetoDB mtg or something and asking them if they would consider supporting something other than cassandra for a backend? I'm concerned about Marconi using it due to this, which was learned the hard way by Comcast with their SQS clone:
oz_akan_no one, other than vendors, will install marconi, lets see a year after graduation to see how many active installations it will have16:06
oz_akan_none of the SQS and users will ever install marconi even if they choose to use it16:06
flaper87oz_akan_: but they can switch and use RAX's queuing service ;) :D16:06
oz_akan_they use sqs and because they choose not to manage it16:06
oz_akan_flaper87: yes, :)16:07
oz_akan_I mean we need to aim big marconi installations16:07
oz_akan_that can scale, support thousands of customers16:07
*** cpallares has quit IRC16:07
flaper87kgriffs: yeah, I can. I still think we have room for 2 more stores in marconi's code base: 1) Redis 2) amqp based16:07
flaper87the rest can live in stackforge16:07
flaper87or GH16:07
flaper87or whatsoever16:07
amitgandhican we ditch sqla16:07
flaper87amitgandhi: we can, although it's useful for the gate and development16:08
kgriffsflaper87: agreed. maybe a third, but that would be the absolute limit16:08
flaper87the sqlite driver is like the best thing when working on marconi16:08
kgriffsflaper87: for example, a magnetodb driver would live in the magnetodb contrib or something16:09
flaper87we always said that sqlite *shouldn't* be used in production16:09
flaper87we can say the same thing about sqla16:09
flaper87which is exactly the way I feel16:09
amitgandhiso keep sqla+sqlite, and ditch sqla+mysql16:09
amitgandhior at that point it doesnt matter16:09
flaper87kgriffs: yeah16:09
kgriffsflaper87: yeah, maybe say it is there for mostly SQLite and definitely not for production16:10
*** prashanthr_1 has left #openstack-marconi16:10
flaper87the good thing is that the redis store is almost done (cc alcabrera )16:10
flaper87so, we'll likely get 2 stores doone for Juno16:11
*** cpallares has joined #openstack-marconi16:11
kgriffsI would love to have flavors done too16:11
alcabreraflaper87: the redis store feels pretty hacky atm. There are many bugs and is terrifyingly slow, given I went simple and careless with it. :P16:11
kgriffsso people realize that "one size doesn't fit all"16:11
alcabrerawe've much interest from students wanting to make redis/marconi happen through GSoC16:12
alcabreras/and is/and it's/16:12
*** cpallares1 has quit IRC16:12
flaper87kgriffs: +1 for flavors16:12
flaper87we should definitely have that session16:12
kgriffsalcabrera: ok. I think we should be willing to relax the FIFO guarantee or whatever if that makes the driver do less work and be blazing fast16:12
alcabrerakgriffs: +116:12
amitgandhiare our customers expecting FIFO (ie using it, and its critical to them?)16:13
flaper87kgriffs: alcabrera relazing FIFO is part of the flavors work, IMHO16:13
oz_akan_yes to flavors16:14
kgriffsPeople want the ability to say "I want queue X with A durability and B performance and C guarantee16:14
oz_akan_kgriffs: we don't really know what people want right?16:14
kgriffs2-3 flavors, that hit the sweet spots on that spectrum16:14
oz_akan_I think people want best of everything16:14
flaper87the good thing is that with marconi they can pretty much get the best of everything as long as it's supported by the API16:15
kgriffsoz_akan_: yes, but given that is impossible, I've heard at least two people say they would be happy with a couple coarse-grained options that gives them the freedom to choose their tradeoffs16:15
oz_akan_I think flavor support shouldn't include backend change, if possible, than great16:15
alcabrerathat's the key to making flavors successful: there has to be a clear mapping from flavor -> parameter tweaking (perf, fifo, durability are interesting params)16:15
oz_akan_kgriffs: I think it is possible16:15
kgriffsoz_akan_: that may be possible, depends on the tradeoff16:16
flaper87I've always said that deploying and using marconi is like playing with lego bricks. You deploy / use what you need16:16
oz_akan_after SSD's  :)16:16
kgriffsoz_akan_: for instance, we could have a mongo driver that doesn't guarantee FIFO16:16
kgriffsand even has lower durability - doesn't wait for a majority of replicas to ack the write, for instance16:16
amitgandhido we need to provide flavors from everyone for everything or is 80% coverage good enough?16:17
kgriffsso flavors needs to be only loosely coupled to driver16:17
oz_akan_I dream we manage replication of data and we have redis (something alike) as backend so we can manage durability vs performance etc.16:17
oz_akan_and that is decided during queue creation16:17
* amitgandhi hopes we dont build replication ourselves16:17
kgriffsamitgandhi: I don't think we need every possible combination of variables.16:17
kgriffspeople with special needs are the kind of people more attracted to private cloud anyway, IMO16:18
oz_akan_kgriffs: that is right16:18
alcabreraamitgandhi: if we do, then we've failed - regarding build our own replication. :P16:18
amitgandhiprivate cloud still equals openstack cloud16:18
kgriffsthe operator can choose some common flavors16:18
oz_akan_public cloud is about standardization16:18
kgriffsit is up to them16:18
flaper87I think people could create their own flavors too16:18
kgriffs(up to them how many/what kinds of flavors)16:18
*** njirap has quit IRC16:18
flaper87I see the storage driver providing a set of supported features16:18
flaper87like mongodb exposes a list of: [durable, performance, database]16:19
oz_akan_about flavors, I think we can be really open minded and then evaluate how much of it makes sense16:19
flaper87redis: [freaking_fast, in-memory, etc]16:19
flaper87then we can combine those features and create flavors out of them16:19
oz_akan_again, I think we shall have only one backend16:19
oz_akan_flavors should also be just because of the hardware16:20
amitgandhii think code can support multiple, deployment only deploys one backend16:20
oz_akan_should = could16:20
oz_akan_we circle around16:20
flaper87oz_akan_: +1 about considering hardware16:20
oz_akan_I will write a looong e-mail about tit16:20
amitgandhioz_akan_: spam filters might block that email ;-)16:21
oz_akan_I will do a blog post then16:21
*** saikrishna has quit IRC16:22
kgriffsoz_akan_: agreed, a driver can be configured to enable/disable certain guarantees, for example16:22
kgriffssame driver, configured differently. You could also pass it in to the driver on every call, but that would be less performant methinks16:23
oz_akan_kgriffs: a shard might only support one flavor if it is installation time configuration16:23
oz_akan_and we shall call flavors as flavios :) sounds more artistic16:23
kgriffsflaper87: flavors can be tricky with topics16:24
kgriffsoz_akan_: lol16:24
alcabreraoz_akan_: heh. :P16:24
flaper87kgriffs: mmh, why?16:25
kgriffsre topics vs. queues, I think some flavor variables are going to be more easily implemented for topics than others16:25
kgriffsflaper87: first off, if I have a message assigned to two topics, what flavor is the message? well, messages now are associated with a flavor, not a topic. So now, how do I query for messages that have both topic X and topic Y16:26
flaper87btw, guys, March 27th, Marconi hangout. I'll take this opportunity go through some of the points in our FAQ16:26
kgriffsI have to query all backends - I don't know what cluster to look at since I don't know in advance what flavor the messages might be16:27
flaper87kgriffs: the message may have 2 flavors16:27
flaper87ah wait16:27
flaper87so yeah16:27
kgriffsflaper87: hangout was 1400 UTC?16:28
flaper87basically, when we thought about flavors the first time, we said that the relation queue-flavor was 1:1, I think the same applies for topics16:28
flaper87we can come up with really nice syntax to do this16:28
flaper87kgriffs: yeah, let me double check16:28
flaper87there'll be a default flavor if none is specified, but we could also get some sort of [(topic, flavor)] kind of thing16:29
kgriffsso, if I associate 2 topics with 1 message, I would basically duplicate the message to two different backends (potentially, depending on how flavors are implemented in driver land)16:29
flaper87something to consider in the session16:29
flaper87yeah, basically16:30
*** amitgandhi is now known as amitgandhi_lunch16:30
flaper87which IMHO, is fair16:30
flaper87they are 2 different topics16:30
kgriffsand if I ask for messages that are in topic A and topic B, the service would have to find out which messages are in both backends16:30
flaper87kgriffs: not needed, IMHO16:30
kgriffsor we include metadata in both backends, so we just query one and we know where else it is16:31
kgriffsoh, wait16:31
flaper87If I (a consumer) am listening on 2 different topics, I may get a message duplicated16:31
kgriffsflaper87: what if I want to do this:16:31
flaper87that's fair, that would happen in rabbit too, FWIW16:31
*** cpallares1 has joined #openstack-marconi16:31
kgriffstopic a: AppUserID16:31
kgriffstopic b: UsersMachineAgentID16:32
kgriffsi want to broadcast all messages to agents for that ID, so I just send to topic a16:32
kgriffsbut if I want to sent to a single agent, I would put both topics on a message16:32
kgriffshmmm. I guess this isn't the same sort of use of topics that you are proposing16:33
*** cpallares has quit IRC16:33
kgriffsthis would be more like tags16:33
flaper87kgriffs: exactly16:33
flaper87that would be tagging the message16:33
flaper87which we can have as well16:33
kgriffsso, one would be able to post to two topics, but only list/claim messages from *one* topic per request?16:34
flaper87(which is how message brokers work too)16:35
flaper87I mean, you have separate channels16:35
kgriffsI was just conflating topics and tags again16:35
kgriffsI am convinced now that you can't simply create one affordance that covers both topic and tag semantics16:37
*** rossk has joined #openstack-marconi16:37
flaper87kgriffs: haha, yeah, they're pretty separate things16:37
kgriffsso, yes, it would make sense to add tags as a separate thing later16:37
flaper87topics are key for routing messages16:37
flaper87tags are a way to add more attributes to a message. Well, tagging it :P16:37
flaper87anyway, very good discussions16:37
flaper87damn, I love this channel16:38
kgriffsflaper87: BTW - I started adding notes to for summit planning16:42
kgriffsI left it in the same etherpad for now since it is related to graduation planning16:42
*** cpallares has joined #openstack-marconi16:45
*** jeremyh28 has quit IRC16:45
flaper87kgriffs: sounds good, I'll go through them in a bit16:46
*** cpallares1 has quit IRC16:48
*** cpallares1 has joined #openstack-marconi16:55
*** cpallares has quit IRC16:58
*** ykaplan has quit IRC16:58
*** AnastasiosAndron has quit IRC16:59
*** ykaplan has joined #openstack-marconi17:02
*** balajiiyer has quit IRC17:02
*** cpallares1 is now known as cpallares17:06
kgriffscpallares: o/17:13
kgriffssorry for me taking so long to review your falcon patch17:13
kgriffsI just found a couple minor nits17:13
cpallareskgriffs: o/17:14
cpallareskgriffs: thanks for reviewing it17:14
kgriffscpallares: you are going to be in Atlanta for the summit, right?17:15
flaper87cpallares: say no, say no17:15
flaper87for your own good, say no17:15
* flaper87 hides17:15
cpallaresflaper87, kgriffs: I don't know yet, but I should know soon.17:16
kgriffsflaper87: no good, I have friends in the NSA17:16
kgriffscpallares: kk17:16
flaper87kgriffs: I'm your friend in the NSA...17:16
kgriffsbecause... I was thinking about setting up some hardware hacking schtuff at our project pod17:16
flaper87I shouldn't have said that17:16
cpallareshaha flaper87, very high training you at the NSA nowadays.17:17
* kgriffs was wondering what his real name was17:17
cpallareskgriffs, flaper87: I actually heard Jacob Appelbaum give a talk this weekend. Over the internet, of course.17:18
kgriffs"Frogs in the head"17:20
flaper87sebasmagri: :P17:21
kgriffsI like this person's sense of humor.17:21
flaper87kgriffs: he's sebasmagri :D17:21
*** balajiiyer has joined #openstack-marconi17:22
kgriffsalcabrera, malini: could use your eyes on this -
mpanettaOh that is awesome17:22
alcabreracuteness_in_the_head(req, resp)17:24
alcabrerafluffiness_in_the_head(req, resp):17:24
alcabrerayou know this is a great patch17:24
*** amitgandhi_lunch is now known as amitgandhi17:24
alcabrerait speaks to me as a human, and some of my fundamental concerns17:24
kgriffscan you folks just sanity check this patch for me? I am about ready to merge it as-is.17:24
alcabrerabut technically speaking17:25
kgriffsIt's so dang cute17:25
alcabreraI'll look ot over now17:25
* malini checking now as well17:25
cpallareshaha aww alcabrera17:25
alcabreraself.assertEqual('fluffy', self.srmock.headers_dict['X-Fluffiness']) -- very important17:26
alcabrerakgriffs, sebasmagri: +1 on this patch17:27
sebasmagrithanks folks!17:27
malinikgriffs: LGTM17:30
*** mkoderer has quit IRC17:31
*** ykaplan has quit IRC17:38
flaper87brb, I'm going to get some gummy bears17:39
flaper87and some food too17:39
flaper87but gummy bears first17:39
alcabreraflaper87: w00t17:39
alcabrerayes of course. ;)17:39
kgriffsoz_akan_, flaper87: interesting article on microservices. Relevant when you talk about the trend towards more heterogeneous operating environments -
*** cpallares1 has joined #openstack-marconi17:41
kgriffs(via @alcabrera)17:42
*** cpallares has quit IRC17:42
alcabrera"The ultimate 'microservice' is a pure function." - via @pchiusano17:46
*** reed has quit IRC17:47
*** cpallares has joined #openstack-marconi17:47
oz_akan_thanks kgriffs I will read it17:47
*** cpallares1 has quit IRC17:48
*** kgriffs is now known as kgriffs_afk17:53
*** cpallares1 has joined #openstack-marconi18:00
*** cpallares has quit IRC18:03
*** jmckind has quit IRC18:09
*** jmckind has joined #openstack-marconi18:10
*** cpallares has joined #openstack-marconi18:22
*** cpallares1 has quit IRC18:23
*** haomaiwang has quit IRC18:27
*** cpallares1 has joined #openstack-marconi18:31
*** cpallares has quit IRC18:33
*** cpallares1 has quit IRC18:50
*** cpallares has joined #openstack-marconi19:06
*** balajiiyer1 has joined #openstack-marconi19:16
*** cpallares1 has joined #openstack-marconi19:18
*** malini is now known as malini_afk19:18
*** balajiiyer has quit IRC19:19
*** cpallares has quit IRC19:19
*** kgriffs_afk is now known as kgriffs19:21
*** cpallares1 has quit IRC19:24
*** cpallares has joined #openstack-marconi19:24
*** balajiiyer1 has quit IRC19:46
*** cpallares1 has joined #openstack-marconi19:47
*** cpallares has quit IRC19:49
*** cpallares1 has quit IRC19:52
*** cpallares has joined #openstack-marconi19:53
*** cpallares has joined #openstack-marconi19:53
*** balajiiyer has joined #openstack-marconi19:58
*** AAzza has quit IRC20:01
*** tedross has quit IRC20:06
alcabreraTake care, everyone. :)20:30
*** alcabrera has quit IRC20:30
*** mwagner_lap has quit IRC20:31
*** cpallares has quit IRC20:32
*** tedross has joined #openstack-marconi21:09
*** fifieldt has quit IRC21:30
*** balajiiyer has quit IRC21:41
*** amitgandhi has quit IRC21:43
*** amitgandhi has joined #openstack-marconi21:43
*** tedross has quit IRC21:45
*** fifieldt has joined #openstack-marconi21:46
*** oz_akan_ has quit IRC21:48
*** oz_akan__ has joined #openstack-marconi21:48
*** mpanetta has quit IRC21:55
*** amitgandhi has quit IRC22:00
*** Kuwagata has quit IRC22:05
*** oz_akan__ has quit IRC22:07
*** jcru has quit IRC22:23
*** jmckind has quit IRC22:42
*** circ-user-Kcb7C has joined #openstack-marconi22:46
*** circ-user-Kcb7C has left #openstack-marconi22:46
*** tjanczuk has joined #openstack-marconi22:47
*** tjanczuk has left #openstack-marconi22:54
*** kgriffs is now known as kgriffs_afk22:55

Generated by 2.14.0 by Marius Gedminas - find it at!