Monday, 2013-12-16

*** rongze has joined #openstack-marconi00:31
*** rongze has quit IRC00:37
*** nosnos has joined #openstack-marconi01:05
*** ayoung has joined #openstack-marconi01:37
*** sudorandom has left #openstack-marconi02:05
*** oz_akan_ has joined #openstack-marconi02:06
*** rongze has joined #openstack-marconi02:23
*** sudorandom has joined #openstack-marconi02:29
*** rongze_ has joined #openstack-marconi02:31
*** rongze has quit IRC02:33
*** oz_akan_ has quit IRC03:26
*** jergerber has joined #openstack-marconi03:51
*** chandankumar has joined #openstack-marconi03:56
*** jergerber has quit IRC03:57
*** rongze_ has quit IRC04:44
*** haomaiwang has joined #openstack-marconi05:09
*** haomaiwang has quit IRC05:14
*** haomaiwang has joined #openstack-marconi05:18
*** haomaiwang has quit IRC05:23
*** flwang has joined #openstack-marconi05:31
*** nosnos_ has joined #openstack-marconi05:34
*** rongze has joined #openstack-marconi05:35
*** nosnos_ has quit IRC05:35
*** nosnos_ has joined #openstack-marconi05:36
*** nosnos has quit IRC05:37
*** rongze has quit IRC05:41
*** haomaiwang has joined #openstack-marconi05:49
*** rongze has joined #openstack-marconi06:08
*** nosnos has joined #openstack-marconi06:59
*** nosnos_ has quit IRC06:59
*** flaper87|afk is now known as flaper8708:12
flwangflaper87: ping08:21
flaper87flwang: pong08:21
flwangflaper87: some stupid question coming08:21
flwangflaper87: how to configure marconi to distinguish queues with tenant or user?08:22
flaper87flwang: mmh, what do you mean?08:22
flwangflaper87: you know, the common queue url is
flwangso is it possible that two different tenant/user have the same name queues? or they can't?08:24
flwangI saw it somewhere, but i can't remember it08:25
flaper87flwang: ah yeah, they can. You just need to create the queue from a different project (tenant)08:28
flaper87there's a header X-PROJECT-ID08:28
flwangflaper87: so is there any specific design consideration using header instead of a patch within url? you know, SQS using the user id with in URL08:30
flaper87flwang: We, as OpenStack, used to have the tenant in the URL but we're now moving away from that and putting it in the headers08:32
flaper87plus, it was renamed from tenant to project08:33
flaper87I think there's an email thread about that08:33
flwangyou mean from url to headers?08:33
*** Abhishek_ has joined #openstack-marconi08:34
flaper87flwang: yup08:35
flwangflaper87: ok, I will search it.08:36
flwangflaper87: #2, is the Email service in the scope of Marconi?08:36
flaper87there you go08:36
flaper87flwang: that sounds like something that could be implemented as a notification plugin08:37
flaper87we've talked about that :)08:37
flwangflaper87: I joined that brainsotrom meeting. so based on my understanding, Marconi notification sounds like the SNS of AWS, it's a Pub/Sub way, right?08:39
flaper87flwang: Marconi is both. It's queues and notification for openstack08:39
flwangflaper87: i know08:39
flaper87wherease AWS has 2 separate services08:39
flaper87flwang: I meant to say that Marconi is SQS and SNS08:40
flaper87so, you're right :D08:40
flwangso you mean Marocni is combine the SQS, SNS, SES all in one, right?08:40
flaper87flwang: well, not SES08:41
flwangin other words, we have a pub/sub framework, so we just extend it to support SNS and SES, is it?08:41
flaper87flwang: exactly08:41
flwangflaper87: so that's what i'm asking08:41
flwangcan Marconi cover the SES functions?08:41
flaper87It may be split in the future but as for now, it's all in one!08:41
flwangor is that our goal?08:42
flaper87flwang: I think it can - I'm not an SES expert - but I'm not sure we want to officially support that08:42
flaper87Marconi's goal is to provide queuing and notification - SQS and SNS - without adding unnecessary complexity08:43
flwangflaper87: got it08:43
flwangflaper87: 3rd question coming08:43
*** ykaplan has joined #openstack-marconi08:43
flaper87flwang: shoot08:44
flwangflaper87: you know, there are only a few patches waiting for review, so why?  less contributor? or we have created everything :D08:44
flaper87flwang: The team is still small, we're working on getting more people. Recently Cindy joined us as part of the OPW intership program and ykaplan has been contributing as well.08:48
flaper87flwang: there are many things to do in Marconi, we're far from feature complete and optimized08:48
flwangflaper87: i see. I'm eager to contribute more, but I think need more time to warm-up08:49
flaper87flwang: please, feel free to ask anything you need08:53
flaper87there are many areas you can contribute to :)08:53
flaper87And there are also many bugs we just triaged last week08:53
flwangflaper87: yep, it would be nice if you can help figure out. and I really want to join the Marconi session in the next summit :)08:54
flwangflaper87: will go through the bug list and find some low-hanging-fruit again :D and meanwhile, pls let me know if there is any more complex bug I can fix08:57
flaper87flwang: sure thing! Thanks flwang!08:57
flwangflaper87: thanks, more questions will be coming soon :D08:58
*** yassine has joined #openstack-marconi09:06
*** Abhishek_ has quit IRC09:15
*** Abhishek_ has joined #openstack-marconi09:24
*** flwang has quit IRC09:41
*** ykaplan has quit IRC09:45
*** ykaplan has joined #openstack-marconi09:45
*** haomaiwang has quit IRC09:52
*** briancline has quit IRC09:52
*** haomaiwang has joined #openstack-marconi09:54
*** briancline has joined #openstack-marconi09:54
*** Abhishek_ has quit IRC09:58
*** rongze has quit IRC10:09
*** yassine has quit IRC10:38
*** rongze has joined #openstack-marconi10:40
*** rongze has quit IRC10:45
*** ykaplan has quit IRC10:53
*** ykaplan has joined #openstack-marconi11:56
*** yassine has joined #openstack-marconi12:05
*** Abhishek_ has joined #openstack-marconi12:24
*** Abhishek_ has quit IRC12:41
*** rongze has joined #openstack-marconi12:41
*** thomasem has joined #openstack-marconi12:43
*** rongze has quit IRC12:46
*** nosnos has quit IRC12:47
*** Abhishek has joined #openstack-marconi13:15
*** oz_akan_ has joined #openstack-marconi13:25
*** oz_akan_ has quit IRC13:26
*** oz_akan_ has joined #openstack-marconi13:26
*** jamieh has joined #openstack-marconi13:35
*** Abhishek has quit IRC13:44
*** cpallares has joined #openstack-marconi13:58
*** rongze has joined #openstack-marconi14:12
*** thomasm_ has joined #openstack-marconi14:15
*** thomasm_ has left #openstack-marconi14:17
*** thomasm has joined #openstack-marconi14:19
*** thomasm is now known as Guest838514:20
*** rustlebee is now known as russellb14:22
*** notmyname_ has joined #openstack-marconi14:22
*** cpallares has quit IRC14:23
*** thomasem has quit IRC14:23
*** notmyname has quit IRC14:23
*** notmyname_ is now known as notmyname14:23
*** Abhishek_ has joined #openstack-marconi14:24
*** jcru has joined #openstack-marconi14:24
*** mpanetta has joined #openstack-marconi14:29
*** openstackgerrit has quit IRC14:31
*** openstackgerrit has joined #openstack-marconi14:31
*** cpallares has joined #openstack-marconi14:33
openstackgerritYeela Kaplan proposed a change to openstack/marconi: Implement tables and unit tests for sqlalchemy storage backend
*** malini_afk is now known as malini14:43
*** Abhishek_ has quit IRC14:43
*** alcabrera has joined #openstack-marconi14:50
alcabreraGood morning!14:51
alcabrerampanetta: hey! How was your weekend? :)14:59
mpanettaNot too bad, went to see the new Hobbit movie15:00
alcabreraHow was it?15:00
mpanettaPretty good :)15:01
mpanettameeting time...15:01
*** whenry has joined #openstack-marconi15:08
*** mpanetta_ has joined #openstack-marconi15:11
*** mpanetta has quit IRC15:14
*** Alex_Gaynor has quit IRC15:14
*** Alex_Gaynor has joined #openstack-marconi15:14
*** Alex_Gaynor has quit IRC15:14
*** Alex_Gaynor has joined #openstack-marconi15:14
maliniflaper87: ping15:15
*** Alex_Gaynor has quit IRC15:15
*** Alex_Gaynor has joined #openstack-marconi15:15
*** flwang has joined #openstack-marconi15:19
*** flwang has quit IRC15:22
*** kgriffs_afk is now known as kgriffs15:24
*** Alex_Gaynor_ has joined #openstack-marconi15:26
*** flwang has joined #openstack-marconi15:26
*** sudorandom_ has joined #openstack-marconi15:26
*** Alex_Gaynor has quit IRC15:27
*** sudorandom has quit IRC15:27
*** sudorandom_ is now known as sudorandom15:27
*** Alex_Gaynor_ is now known as Alex_Gaynor15:27
*** Guest8385 is now known as thomasem15:31
*** zyuan has joined #openstack-marconi15:35
*** zyuan has joined #openstack-marconi15:36
*** jergerber has joined #openstack-marconi15:37
*** jergerber has quit IRC15:37
*** jergerber has joined #openstack-marconi15:37
*** jergerber has quit IRC15:39
*** haomaiwang has quit IRC15:39
*** briancline has quit IRC15:39
*** haomaiwang has joined #openstack-marconi15:39
*** Abhishek has joined #openstack-marconi15:44
*** briancline has joined #openstack-marconi15:44
*** haomaiwa_ has joined #openstack-marconi15:46
flaper87malini: pong15:50
flaper87alcabrera: mpanetta_ good morning15:50
* flaper87 was in a call15:50
maliniflaper87: how are you?15:50
* flaper87 went to see The Hobbit15:50
flaper87imax + 3D15:51
malinitht sounds cool :)15:51
alcabreraflaper87: yo! :)15:51
*** torgomatic_ has joined #openstack-marconi15:52
*** cpallares1 has joined #openstack-marconi15:52
flaper87alcabrera: 'sup buddy! How was your weekend ?15:52
flaper87malini: yeah! It was reaaally nice!15:52
flaper87I was like wooowing the whole time15:52
*** Abhishek_ has joined #openstack-marconi15:53
maliniflaper87: I have a question for you on marconi-devstack15:53
alcabreraIt was relaxing~ I got to write a bunch and spend plenty of time with family, flaper87! :)15:54
maliniflaper87: Are you using the default marconi config with auth turned off ?15:54
*** cpallares has quit IRC15:54
*** torgomatic has quit IRC15:54
*** torgomatic_ is now known as torgomatic15:55
*** Abhishek has quit IRC15:56
*** haomaiwang has quit IRC15:56
flaper87alcabrera: awesome, I'm glad to heard that!16:00
flaper87malini: mmh, nope, devstack turns auth on16:00
flaper87malini: any issues?16:01
*** flaper87 is now known as flaper87|call16:02
maliniI dont think its validating the auth token, but cud be me16:03
maliniI'll chk again16:03
*** kgriffs has quit IRC16:04
*** flwang has quit IRC16:04
*** kgriffs has joined #openstack-marconi16:05
*** ChanServ sets mode: +o kgriffs16:05
*** haomaiwang has joined #openstack-marconi16:08
*** chandankumar has quit IRC16:17
*** haomaiwa_ has quit IRC16:18
*** dragondm has quit IRC16:18
*** briancline has quit IRC16:18
*** al-maisan has quit IRC16:18
*** apmelton1 has quit IRC16:18
*** russell_h has quit IRC16:18
*** lakspace has quit IRC16:18
*** mpanetta_ has quit IRC16:18
*** metral has quit IRC16:18
*** briancline has joined #openstack-marconi16:21
*** mpanetta has joined #openstack-marconi16:21
*** al-maisan has joined #openstack-marconi16:21
*** apmelton1 has joined #openstack-marconi16:21
*** metral has joined #openstack-marconi16:21
*** dragondm has joined #openstack-marconi16:21
*** russell_h has joined #openstack-marconi16:21
*** lakspace has joined #openstack-marconi16:21
*** rwsu_ has joined #openstack-marconi16:26
*** jergerber has joined #openstack-marconi16:26
*** rwsu has quit IRC16:27
*** rektide_ has quit IRC16:27
*** rektide has joined #openstack-marconi16:27
*** thomasem has quit IRC16:37
*** ametts has joined #openstack-marconi16:42
*** rektide has quit IRC16:45
*** rektide has joined #openstack-marconi16:46
*** flaper87|call is now known as flaper8716:56
*** cpallares1 is now known as cpallares16:57
*** thomasem has joined #openstack-marconi16:58
flaper87cpallares: HELLOOOOO!16:59
flaper87cpallares: how are you doing? :)16:59
cpallaresflaper87: I'm good :) how are you?16:59
cpallareshow was your weekend?16:59
flaper87cpallares: it was good. I went to see The Hobbit :)17:00
flaper87what about yours?17:00
cpallaresflaper87: Did you like it? I want to see it!17:00
cpallaresIt was good and relaxing :)17:00
*** haomaiwang has quit IRC17:01
*** ykaplan has quit IRC17:02
flaper87cpallares: it was good. I did have couple of "WTF?" moments but I don't want to spoiler anything17:04
cpallaresflaper87: oooh i need to see it now. I liked the first one and I really enjoyed the book when i read it years ago17:05
mpanettaflaper87: You went to see the Hobbit as well?17:07
* flaper87 ties his hands17:07
flaper87mpanetta: yeaaah!17:07
*** yassine has quit IRC17:11
kgriffsmalini: ping17:13
kgriffsflaper87: anything you want to add to the agenda?17:13
flaper87kgriffs: not this time around.17:14
kgriffsflaper87: different topic17:15
kgriffsover is solum-land there's a patch under review that adds some functional tests17:15
kgriffsbut, host/port is hard-coded17:15
kgriffsIn marconi we have a conf file for that stuff17:16
cpallaresspeaking of the meeting someone should change the meeting time in the irc description :D17:16
kgriffscpallares: lol17:16
kgriffsgood catch17:16
cpallareskgriffs: o/17:16
cpallareshow are you?17:17
kgriffsflaper87: is there any reason you could think of for solum *not* following marconi's lead here?17:17
kgriffscpallares: I'm groovy. How are you?17:17
cpallaresi'm groovy myself, kgriffs :)17:18
*** kgriffs changes topic to "OpenStack Queuing and Notification Service || Smile :D || Meetings every Tuesday @ 15:00 UTC || Wiki: ttps:// || Paste: || Send messages and make some noise :D"17:18
flaper87kgriffs: I can definitely tell you why we're using a config file. We wanted to keep tests configurable in order to test different scenarios with the same installation of marconi. The idea there is to also have a way to have a base test and make it point to different scenarios17:19
flaper87OTOH, I don't think Solum needs this17:19
kgriffsflaper87: kk, that makes sense17:20
flaper87and the last scenario should have been deployment17:20
flaper87but you understood what I meant :D17:21
kgriffsright. :D17:21
kgriffsmalini: could you help review this patch?
kgriffsflaper87: ^^^ Solum patch I was talking about17:24
*** alcabrera is now known as alcabrera|afk17:26
malinikgriffs: sure17:27
kgriffsmalini: thanks!17:31
*** flaper87 is now known as flaper87|afk17:41
*** jamieh has quit IRC18:15
*** neosergey has joined #openstack-marconi18:28
*** neosergey has quit IRC18:30
*** rongze has quit IRC18:41
*** primochka has joined #openstack-marconi18:56
*** primochka has quit IRC19:03
*** Abhishek_ has quit IRC19:36
*** balajiiyer has joined #openstack-marconi19:48
*** malini is now known as malini_afk20:00
*** malini_afk is now known as malini20:01
*** reed has joined #openstack-marconi20:23
*** malini is now known as malini_afk20:25
*** alcabrera|afk is now known as alcabrera20:47
*** openstackgerrit has quit IRC21:08
*** openstackgerrit has joined #openstack-marconi21:08
alcabrerakgriffs: pong21:26
alcabrerahey, hey21:26
kgriffsalcabrera: yo21:26
kgriffsso, got a little research to share21:26
kgriffsand get feedback on21:26
alcabreraalright - I'm ready to listen.21:27
kgriffsI've been playing with NTPD21:28
kgriffsso, I am about to test synchronization between different boxes21:30
kgriffsbut first, I used those functions to check granularity of time.time()21:30
kgriffsturns out that with over 500,000 requests/sec you still get unique timestamps21:30
mpanettaamazing heh21:31
kgriffsthe clock granularity hovers around 1 microsecond21:31
kgriffsso, that's not going to be a problem for marconi if we want to use timestamps21:31
mpanettawow I thought IEEE1588 was supposed to do that, I guess NTP can too :)21:31
kgriffsi am about to test clock drift between boxes21:32
alcabreraso what that tells me, is that given marconi on a single machine (or some oracle), we can scale up to 500K rps while relying solely on time.time()21:32
alcabreralet's see what time drift tells us21:32
kgriffstrue, we could do it with an oracle21:34
kgriffsanybody remember the option for forcing ntpd to converge faster?21:35
alcabrerakgriffs: seems to be -x21:36
alcabreraI looked that up just now.21:36
mpanettakgriffs: if NTP can't do what you want, check out
kgriffsmpanetta: interesting, I'll check it out!21:52
mpanettaIEEE1588 is high res time stamping21:53
alcabrerampanetta, kgriffs: I found this interesting comparison of NTP vs. PTP:
alcabrerathe high-resolution aspect seems to be identical for ntp vs. ptp, but the communication style is a little different21:55
kgriffsalcabrera: btw, I was just thinking, if we go the oracle route, we only need to be collision-resistant for a single queue21:55
alcabrera(e.g., sub-microsecond resolution)21:55
mpanettawell I knew about the second part, not the first.21:55
kgriffsso, you could have a pool of oracles and shard on queue name21:55
kgriffsthen, you use a clock21:55
kgriffsand you don't have to worry about clock drift21:55
kgriffsjust, ensuring that no single queue can overrun a single oracle21:56
kgriffsbut, it would be even cooler to not have an oracle21:56
mpanettaWe should get our overlords to install atomic clocks at all the datacenters21:56
alcabrerait sounds like it would work as long as no single queue was servicing more than 1 worker at a time (since that has a nondeterministic order, anyway) and as long as no oracle was servicing more than 500K rps21:57
alcabreraactually, the entire notion of monotonically increasing markers just made me think of something I just watched21:58
alcabreraLattices for distributed systems - e.g., how to guarantee consistency in a distributed context as long as the operations in question are monotonically increasing21:58
kgriffsalcabrera: darn; timekeeper costs $$$21:59
alcabreraI wasn't able to absorb most of it this run, but I wonder what if it's applicable to the marker generation problem for FIFO queues...21:59
alcabreralet's see about this timekeeper21:59
kgriffsalcabrera: interesting. is that a CS paper?22:00
alcabrerakgriffs: It's both a CS paper and a RICON West 2013 talk22:01
alcabreratimekeeper boasts some impressive features, mostly time-smoothing and performance points22:02
alcabrerakgriffs: also, CRDTs (Conflict-Free Replicated Data Types) - one of the fundamental ideas behind Riak (DB):,
* alcabrera is on a reading, watching, learning day22:09
*** balajiiyer has quit IRC22:10
kgriffsah yes22:10
kgriffscrazy stuff22:10
*** jergerber has quit IRC22:19
*** oz_akan_ has quit IRC22:25
mpanettaI love watching you 2 converse, thanks for the NTP vs PTP link alcabrera22:30
alcabrerampanetta: no problem. :)22:33
mpanettaIt puts to bed some of the misconceptions I had about NTP22:34
alcabreraI didn't even realize PTP existed 'til today.22:35
mpanettaAh, yeah it has been around for a few years22:35
mpanettaMostly for embedded and industrial spaces22:36
alcabreraWhenever I'm presented with a new technology or protocol, particularly as an alternative, I'm off to the Googles to compare, constrast for about 5 minutes.22:36
* mpanetta has a lot of interest in those spaces22:36
mpanettaCool :)22:36
mpanettaGood thing to do.22:36
alcabrerampanetta: I have noted your interest in the embedded space. :D22:36
alcabreraWhich reminds me...22:36
*** thomasem has quit IRC22:36
alcabreraHave you heard of the Parallela chip?22:37
mpanettaYep :)  Very interesting22:37
mpanettaNot sure how far they got tho22:37
alcabreraseems to be a hit22:37
kgriffsalcabrera: I think we need to fix this for icehouse22:37
mpanettaThey had a ickstarter last year I think?  Maybe earler this year,22:37
alcabreramy newsfeeds have been a-buzz with talk of it22:37
kgriffsit is a pretty critical ops thing22:37
* mpanetta goes to search22:37
alcabreraThere's even plans to get a 1024-core chip out next year22:37
alcabrerakgriffs: lessee...22:37
kgriffsI upped the priority22:37
kgriffsametts: ^^^22:38
kgriffswe will need to get this in Rackspace's Q1 roadmap, if possible22:38
kgriffsif not, then perhaps some of our new contributors can help out22:38
mpanettaOh yeah, now I remember.  I wish they allowed more access to the Zynq IO...  Such a nice chip the Zynq.22:38
mpanettaalcabrera: already there :)22:39
amettskgriffs:  This  is what you're talking about?
alcabrerakgriffs: So the keystone auth middleware is injected in such a way that bastion won't work...?22:39
kgriffsametts: yes22:39
mpanettakgriffs: isn't that what the bastion was for?22:39
kgriffsalcabrera: no, sorry it is a little confusing22:39
kgriffsthe bastion can work but you have to do your own app.py22:40
kgriffsa couple things22:40
kgriffsfirst, health needs to be callable out of the box even when auth_strategy is enabled22:40
kgriffs(without auth, but only if NOT coming from the LB, like eom.bastion does)22:41
kgriffsi mean, it should be a list of restricted things22:41
kgriffsand we default to including health in that list22:41
alcabrerawould this be support for restricted routes in marconi?22:41
kgriffsalcabrera: the second part is less important, I guess, but we need a way to insert middleware between keystone and the app without having to write your own app.py22:41
kgriffsalcabrera: right22:41
kgriffsalcabrera: but, I don't think Rackspace could use it because it would assume use of auth_strategy22:42
kgriffswhich currently precludes using post-auth middleware22:42
*** jergerber has joined #openstack-marconi22:43
alcabreraI see the need for this, and I agree with the priority,'22:49
alcabrerakgriffs: ^22:49
kgriffskk, thanks!22:49
alcabreraflaper87|afk, and cpallares' API patch work that labels certain endpoints as 'admin': True seems complementary to this effort22:50
*** flaper87|afk is now known as flaper8722:50
flaper87alcabrera: awesome! Thanks!22:50
flaper87alcabrera: btw, ykaplan submitted a new version of her patch, could you take a lok there too?22:51
flaper87kgriffs: ^^^^22:51
alcabrerawill do. I'll do it tomorrow morning, flaper87. :)22:51
flaper87kgriffs: did you read in #falconframework the case where I think N route 1 controller could make sense22:51
flaper87alcabrera: sounds good :)22:51
flaper87thanks a lot!22:51
alcabreraflaper87, kgriffs: This particular concept (CRDTs: has eaten my brain for the rest of the night.22:52
* flaper87 wonders what kgriffs is doing with ntpd22:53
* flaper87 is afraid he doesn't want to know22:53
mpanettaSecret squirly things22:54
* flaper87 just bought this:
alcabreraflaper87: so that's what that Tweet was about!22:55
alcabreraGood times. :D22:55
flaper87alcabrera: yeaaah!!!22:56
mpanettaNice lense22:56
*** mpanetta has quit IRC22:56
flaper87that lens stole my heart last July. Today it stole my money too.22:56
kgriffsflaper87: wrt the N:122:57
kgriffsyes, I saw it22:58
kgriffsare you proposing to have One Big Controller?22:58
kgriffsrather than resource-based?22:58
flaper87kgriffs: and you didn't say anything??? :0 you just broke my heart22:58
kgriffsflaper87: heh... can't remember why - it may have been pulled away just then to a mtg or something22:58
* kgriffs still loves flaper8722:58
flaper87kgriffs: it's not actually 1 big controller, it's just a small controller that will call the API layer which will then actually process the request22:59
* flaper87 feels better now22:59
flaper87I'd like transports to be stupid logic wise22:59
flaper87and be fucking smart protocol wise22:59
kgriffsbut all requests would be funneled through a single class?22:59
* kgriffs thinks about that23:00
kgriffsflaper87: were you hoping to auto-generate the transports?23:06
flaper87kgriffs: yup :D23:06
kgriffsso, off the top of my head, the WSGI transport has the following functions currently23:08
kgriffs1. input validation23:08
kgriffs2. protocol translation (http request to storage controller request, and then the response back out23:08
kgriffs3. error translation - e.g., convert DoesNotExist to 404 Not Found23:09
kgriffssome input validation may be more efficiently done before passing stuff down the stack23:10
kgriffserror translation will be tricky to automate - you will need a way to configure the mapping, but also include metadata about the error, in case it needs to be returned in the diagnostic section of the response body23:10
kgriffsfor the error translation, you will basically end up writing a DSL that may be more complex than just writing the Python23:11
kgriffswhat was you plan to tackle that stuff?23:12
flaper87kgriffs: I was thinking about moving most of that to the API layer23:16
flaper87things like Exception translation will need to stay in the transport layer, though23:16
kgriffsflaper87: right23:16
kgriffsso, I think it makes sense to push down some logic into the API layer23:16
kgriffsbut, I don't know that it is a great idea to auto-generate the transport23:16
kgriffstaking HTTP for example23:17
flaper87oh wait, by autogenerate the transport I mean register endpoints dynamically23:17
flaper87not the controller itself23:17
kgriffsjust to be clear on terminology23:17
kgriffswe have routes, http resources, storage controllers23:18
flaper87yeah, that's in marconi, I was just talking about falcon routes and controllers23:18
kgriffswhat I meant was the http resource classes should stay hand-coded so we can make sure exceptions are translated appropriately and optimize certain validations as needed23:18
kgriffsflaper87: falcon doesn't have controllers23:18
kgriffsthose are resources (ala REST)23:19
flaper87kgriffs: erm, sorry, I used the wrong term T_T23:19
kgriffsdjango/rails I imagine use the term "controllers"23:19
flaper87kgriffs: re hand-coded http resources, yes. We're on the same page there23:19
flaper87What I want to happen dynamicall is the routes registration23:19
kgriffswhat does that buy us?23:19
flaper87and I think that we can have just 1 controller that will fit them all.23:20
kgriffsflaper87: I'd prefer having separate resource classes rather than lumping everything into one big class23:20
flaper87kgriffs: the fact that the API layer will tell the transport what are the available endpoints23:20
kgriffssmells like poor separation of conccerns23:20
kgriffsflaper87: but the transport has to know in advance what those are in order to implement the translation logic between HTTP land and storage/API land23:21
flaper87kgriffs: no need, the transport needs to translate from HTTP to a common request object that the API layer knows how to use23:22
flaper87transport -> api layer -> storage23:22
kgriffsbut doesn't the http resource still need to some out of band knowledge, e.g., about error types that must be hand-coded?23:22
flaper87kgriffs: yeah, that's the bit that needs to be resolved.23:23
flaper87I was thinking that if we have a good separation of exceptions, we can easily map 1 exception to 1 http code23:23
flaper87I'm not convinced, though23:24
kgriffsassuming we have a mapping engine that is smart enough to pull out error metadata and put it into the error body diagnostic info23:24
kgriffsI'm still not sold that this is saving us much work23:25
kgriffsdynamically generating the routes means having to dynamically generate the translation between protocol and API semantics23:25
kgriffsthat's a tall order23:26
kgriffsI think that we may get there eventually, but the first step would be to push down as much logic as we can, down the stack23:26
flaper87I don't want to generate the translation between the protocol and the API dynamically23:26
kgriffsflaper87: I think you will have to if you generate the routes23:26
kgriffsgenerating the routes implies generating the responders to those routes, or just assuming they exist23:27
flaper87I'm assuming they exist23:27
kgriffsand if you are just assuming they exist, you might as well just define some "register()" hook on the transport that the bootstrap calls and it does it's own add_route things23:27
flaper87that's why I thought about using 1 responder for all the routes23:27
flaper87kgriffs: that's exactly what I'm saying23:28
flaper87damnit, I failed again at making my point!23:28
flaper87I think we just got mixed up with the terms23:28
flaper87or I did23:28
kgriffsso, let me continue with my misunderstanding23:28
kgriffsand then you can correct me23:28
kgriffsso, I was envisioning that the API would read the schema and generate a bunch of add_route calls to a single resource class23:29
kgriffsi guess it would assume the appropriate responders already exist on the resource class23:29
kgriffsnow, at this point, I am wondering how this saves us devs any work?23:30
kgriffsseems like just about as hard to write that add_route generator as it is to just write out a list of add_route calls23:30
*** balajiiyer has joined #openstack-marconi23:30
kgriffs(by hand)23:30
kgriffsthat correspond to the responders that I already wrote by hand23:30
kgriffsI mean, we are talking about 15 minutes of dev time here for each new API to wire up the routes23:31
* kgriffs is done with that train of thought23:31
flaper87well, my proposal was to use N routes and 1 responder! I don't think we need more than one responder. I admit I haven't thought much about the error translation.23:32
kgriffshaving a single resource vs. multiple resources doesn't really matter so much; I think the crux of the issue is whether we can auto-generate the resources.23:33
flaper87that said, the benefit it gives use is that the transport won't need to worry about the API version. it just needs to translate the incoming request to a MarconiRequest that the API layer will then process23:33
kgriffs(responder == on_get or on_post)23:33
flaper87I don't think we should auto-generate them, that's too much magic that I'd prefer avoiding23:33
kgriffswe could also have a simply proxy that eats everything and spits it out the other end, fwiw23:33
flaper87(HTTP -> MarconiRequest) -> API -> Storage23:34
flaper87Transport = (HTTP -> MarconiRequest)23:34
kgriffsso, I like the idea of that flow there, pushing common logic down into API23:35
kgriffsI think that would be our first iteration23:35
kgriffsthen, we can look at what is left over in the transports and see how much of that can be pushed elsewhere or automated23:35
kgriffsif we end up with zero hand-rolled code in the transport, yay!23:35
*** oz_akan_ has joined #openstack-marconi23:36
kgriffsbut I don't think we can make that determination at this juncture23:36
flaper87wAt this point, I've put lot of thoughts about this. I think we'll get there.23:36
flaper87and I really want to get there23:36
flaper87As said before, I would like transports to be stupid logic wise and smart protocol wise23:37
kgriffsok, but we also need to not compromise on performance and user experience to get there23:37
flaper87that will help us with avoiding backward incompatible changes between the transport and the api, it'll make transport forward compatible etc23:38
kgriffssure, it sounds really nice, don't get me wrong!23:38
flaper87kgriffs: I don't think it'll compromise UX. I agree about the performance concern, though23:38
kgriffsI just want to take it iteratively23:38
kgriffstake a step, experiment, get results, feed that into the next cycle23:38
flaper87kgriffs: yeah, and please, doubt about whatever idea comes to my non-human mind. :)23:39
flaper87and I agree, we should get there iteratively23:39
kgriffsI think that a lot of things will be easier to reason about once we put the API layer in place and start pushing logic down into it23:39
flaper87We, as in cpallares and cpallares and cpallares, are starting to work on the reousrces needed by the API layer. That is, schema, response objects, request objects etc23:40
kgriffsflaper87: just to confirm23:40
*** oz_akan_ has quit IRC23:40
kgriffsthe idea is sort of create a domain-specific WSGI23:40
kgriffswhere we normalize everything to standard request and response objects23:40
kgriffssounds pretty awesome23:41
kgriffsand something that may be nice to do as a standalone library - i mean, parts of it23:41
*** oz_akan_ has joined #openstack-marconi23:41
kgriffsmaybe even a PEP. :p23:41
flaper87Btw, I had another idea but this time is related to messages and storage. I'll share it tomorrow. I'll go and eat a book now23:42
flaper87kgriffs: yeah, that'd be awesome23:42
flaper87also, if I stay here 1 more second, I'll end up buying another lens23:42
kgriffsflaper87: would it be weird to turn this idea on it's head and translate from X transport to WSGI?23:42
kgriffsI guess that would be slow23:42
alcabreraflaper87: +1 for eating books23:43
kgriffsflaper87: LOL23:43
amettsflaper87: Dude... don't you ever sleep?23:43
flaper87kgriffs: that's kinda of what the websocket transport does, it's a bit slow!23:43
flaper87ametts: nope :D23:43
flaper87it's one of the benefits of not being human23:43
kgriffsflaper87: right, I remember that now - there is uwsgi as well23:43
kgriffsanyway, just want to make sure we aren't in the throes of NIH syndrome23:44
flaper87lol @ NIH23:45
flaper87kgriffs: btw, latest cache patch is up:
flaper87not sure why py27 is failing. "It works on my laptop"23:50
kgriffslol, thanks - I noticed that and have it on my todo list to review23:51
*** oz_akan_ has quit IRC23:51
flaper87I don't get it:
flaper87anyway, I'll figure that out tomorrow!23:57
flaper87by guys, have a good one!23:57
* flaper87 back to books23:57
*** flaper87 is now known as flaper87|afk23:58
alcabreraI'm out for the night, as well.23:58
alcabreraTake care, guys. :)23:59
*** alcabrera has quit IRC23:59

Generated by 2.14.0 by Marius Gedminas - find it at!