Tuesday, 2013-10-15

*** tedross has quit IRC00:07
*** reed has quit IRC00:15
*** Chat2315 has joined #openstack-marconi00:39
*** nosnos has joined #openstack-marconi01:05
*** kgriffs_afk is now known as kgriffs02:06
*** malini_afk is now known as malini02:07
*** kgriffs is now known as kgriffs_afk03:26
*** kgriffs_afk is now known as kgriffs03:27
*** kgriffs is now known as kgriffs_afk03:34
jburkhartis marconi currently closed source, or am I just looking in the wrong place? I'm getting a 404 from github on the link provided under Resources on the wiki -- (https://github.com/stackforge/marconi)04:02
jburkharthmm, looks like there was a git repo available at https://review.openstack.org/openstack/marconi -- you might want to update the wiki Resources section to reflect the move04:05
*** reed has joined #openstack-marconi04:14
*** reed has quit IRC04:25
*** dafter has joined #openstack-marconi07:38
*** dafter has quit IRC07:38
*** dafter has joined #openstack-marconi07:38
*** flaper87|afk is now known as flaper8707:45
*** ykaplan has joined #openstack-marconi07:46
flaper87jburkhart: it's open source github.com/openstack/marconi07:46
*** yassine has joined #openstack-marconi07:57
openstackgerritA change was merged to openstack/marconi: chore: Remove GC cruft from storage driver base class  https://review.openstack.org/5101608:08
*** tvb|afk has joined #openstack-marconi08:15
*** dafter has quit IRC08:19
flaper87fvollero|gone: come back, now!08:43
*** tvb|afk has quit IRC08:49
*** ykaplan has quit IRC08:50
*** openstackgerrit has quit IRC09:38
*** dafter has joined #openstack-marconi09:42
*** dafter has quit IRC09:43
*** tvb|afk has joined #openstack-marconi09:49
*** flaper87 is now known as flaper87|afk10:11
*** nosnos has quit IRC10:28
*** nosnos has joined #openstack-marconi10:28
*** nosnos has quit IRC10:33
*** tvb|afk has quit IRC10:37
*** dafter has joined #openstack-marconi10:37
*** fifieldt_ has quit IRC10:39
*** tvb|afk has joined #openstack-marconi10:42
*** tvb|afk has quit IRC10:42
*** tvb|afk has joined #openstack-marconi10:42
*** dafter has quit IRC10:43
*** fifieldt has joined #openstack-marconi10:45
*** tvb|afk has quit IRC11:01
*** dafter has joined #openstack-marconi11:02
*** dafter has quit IRC11:02
*** dafter has joined #openstack-marconi11:02
*** tvb|afk has joined #openstack-marconi11:28
*** dafter_ has joined #openstack-marconi11:29
*** dafter_ has quit IRC11:29
*** dafter_ has joined #openstack-marconi11:30
*** dafter has quit IRC11:32
*** dafter_ has quit IRC11:32
*** tvb|afk has quit IRC11:33
*** tedross has joined #openstack-marconi11:38
*** flaper87|afk is now known as flaper8711:49
*** dafter has joined #openstack-marconi11:52
*** tvb|afk has joined #openstack-marconi11:57
*** tvb|afk has quit IRC11:57
*** tvb|afk has joined #openstack-marconi11:57
*** tvb|afk has quit IRC11:57
*** tvb|afk has joined #openstack-marconi11:57
*** dafter has quit IRC12:00
*** yassine has quit IRC12:04
*** flaper87 is now known as flaper87|afk12:08
*** yassine has joined #openstack-marconi12:12
*** flaper87|afk is now known as flaper8712:16
*** mpanetta has joined #openstack-marconi12:16
*** mpanetta has quit IRC12:20
*** mpanetta has joined #openstack-marconi12:23
*** ayoung has quit IRC12:41
*** oz_akan_ has joined #openstack-marconi12:55
*** fifieldt has quit IRC12:56
*** oz_akan_ has quit IRC12:56
*** oz_akan_ has joined #openstack-marconi12:57
*** openstackgerrit has joined #openstack-marconi13:02
*** tvb|afk has quit IRC13:04
*** fvollero|gone is now known as fvollero13:05
fvolleroflaper87: son qui13:05
*** dafter has joined #openstack-marconi13:07
flaper87fvollero: yoooo!!!13:07
flaper87fvollero: how ya doing?13:07
fvolleroflaper87: not bad, emanuela is here, i'll talk with her later and give your damn cocosette13:08
flaper87fvollero: hahaha, thanks buddy!!!!13:08
*** amitgandhi has joined #openstack-marconi13:09
flaper87fvollero: I wanted to know how is the ES thing going and whether you wanted to take a look over the client patches13:09
flaper87if you've some time13:09
fvolleroflaper87: still trying to fix packstack && puppet :( it's mandatory to have the support by thrusday, so I hope to finish sooner13:11
*** dafter has quit IRC13:15
*** dafter has joined #openstack-marconi13:15
*** tvb|afk has joined #openstack-marconi13:18
*** dafter has quit IRC13:20
*** yassine has quit IRC13:23
*** yassine has joined #openstack-marconi13:23
flaper87am I still alive?13:27
amettsflaper87: Either that, or you figured out how to connect to IRC from beyond the grave...13:32
flaper87ametts: hahahah, most likely the second one!13:33
flaper87If there's something I'll take with me from this world, it is internet!13:33
*** jcru has joined #openstack-marconi13:43
*** amitgandhi has quit IRC13:49
*** kgriffs_afk is now known as kgriffs13:59
*** ayoung_ has joined #openstack-marconi14:05
*** kgriffs is now known as kgriffs_afk14:08
openstackgerritFlavio Percoco proposed a change to openstack/python-marconiclient: Add list of required fields to the API definition  https://review.openstack.org/5185014:09
*** amitgandhi has joined #openstack-marconi14:10
*** ayoung_ is now known as ayoung14:12
flaper87knock ?14:41
flaper87where's everybody?14:41
flaper87zyuan: hey! :D14:42
zyuaninternet hahahh14:43
zyuanan internet without any people is also boring14:44
flaper87zyuan: indeed! I couldn't agree more!14:53
*** alcabrera has joined #openstack-marconi14:53
flaper87it's like being alone between so many things14:54
flaper87alcabrera: GOOOOD MORNING!!!14:54
alcabreraflaper87: hey! :D14:54
flaper87alcabrera: how are you doing?14:57
alcabreraa bit sluggish. It's been slow going for me this week with a lot going on at home. @_@14:58
alcabreraBetween one of my cats hurting her leg, having a doctor's appointment on a Monday, and a few other things, it's been a little hard to stay focused. :P14:59
*** reed has joined #openstack-marconi15:01
flaper87alcabrera: aahh, I'm sorry about your cat :(15:02
flaper87Yeah, kind of the same here! :/15:03
alcabrerathe little one's getting better, thankfully. She hobbles a bit, but less so every day. :)15:04
alcabreraflaper87: it's just one of those weeks! I hope your's gets better, too.15:04
openstackgerritA change was merged to openstack/marconi: Follow hacking rules about import  https://review.openstack.org/5030815:04
flaper87w000t, 2 patches merged today!15:05
alcabreraI've been swimming in Haskell in my free time. Monads finally clicked this weekend. :P15:05
alcabrerazyuan: maybe I'll put together an introduction to haskell tech talk at some point. :D15:07
zyuan(i guess seldom people care.... -_-)15:08
alcabrerait's all about finding a way to make it relevant. :)15:08
zyuan(can't think of any ... -_-)15:08
zyuanit IS relevant to the programming world, but15:09
zyuannot every parts15:09
zyuanof the world15:09
zyuanflaper87: does redhat have tech talk about haskell?15:11
* flaper87 likes haskell :D15:11
flaper87zyuan: yup, we've tech talks about anything we want to talk about15:11
flaper87There was one about Go recently15:11
*** reed has quit IRC15:11
alcabreraflaper87: awesome! We're trying to grow the tech talk program here. It's very flexible, and we can talk about anything, but getting participation up is challenging. :)15:18
alcabreraNow I'm going to sit back, grab a pencil. and go through the latest changes in marconi-sharding.15:18
openstackgerritMalini Kamalambal proposed a change to openstack/marconi: Validation for messages returned by queue/stats  https://review.openstack.org/5099515:19
flaper87alcabrera: hehehehehe15:25
*** malini is now known as malini_afk15:30
*** malini_afk is now known as malini15:30
*** yassine has quit IRC15:46
openstackgerritFlavio Percoco proposed a change to openstack/python-marconiclient: Implement queue's API methods  https://review.openstack.org/5063815:53
openstackgerritFlavio Percoco proposed a change to openstack/python-marconiclient: Add list of required fields to the API definition  https://review.openstack.org/5185015:53
openstackgerritMalini Kamalambal proposed a change to openstack/marconi: Validation for messages returned by queue/stats  https://review.openstack.org/5099516:09
* flaper87 is working on the message's API (client-side)16:11
flaper87alcabrera: btw, I think there are 2 new patches waiting for you16:12
*** tvb|afk has quit IRC16:16
alcabreraflaper87: nice!16:27
alcabreraI just finished up lunch - forgot to set my afk. :P16:28
zyuanthere are a lot https://review.openstack.org/#/q/status:open+project:openstack/marconi,n,z16:28
alcabrerazyuan: so many things to review. :P16:30
flaper87... and code on! :D16:30
flaper87anyway, zyuan is right. We've got to get that list smaller16:31
alcabreraLet's see if I can help that along before I branch off of kgriffs_afk's latest patch and continue the sharding effort.16:35
alcabreraHeadphones time - this will require music.16:35
flaper87alcabrera: :D16:39
flaper87alcabrera: enjoy!16:39
flaper87alcabrera: Do you use spotify?16:39
* flaper87 is happy to announce that it's now possible to post and list messages using the client library16:39
alcabreraflaper87: no, though I've heard of spotify. I mostly use audacious player locally and listen to game OSTs. :)16:40
alcabreraflaper87: awesome! Client lib is *winning*. :D16:40
alcabreraflaper87: https://review.openstack.org/#/c/50638/ (-1)16:58
alcabreraLooking good, though!16:58
alcabreraI'm breaking that review into parts to do it justice.16:58
alcabrera700 lines is at my limit. :P16:59
flaper87alcabrera: awesome, THANKS!16:59
flaper87alcabrera: yeah, I just couldn't split it! :/16:59
flaper87I guess I could split the next one in lower-level / higher-level16:59
alcabreraflaper87: no worries on that one - patch splitting is difficult, esp. as we cover new ground. I understand. :)17:00
flaper87but I think it doesn't show how the API would be used and it may hide some possible issues in the design17:00
alcabrerayeah, that's the hard part.17:00
alcabreraLossless splitting17:00
alcabreraWe need the context.17:00
flaper87alcabrera: awesome, thanks for understanding! I'll take a look at your comments, address them and then we can iterate again17:00
alcabreraIt might be best to do multi-part reviews for the big ones.17:01
alcabreraThe running theory on review limits seem to discuss single session limitations (400-600 lines over a period of 30 minutes), but say nothing about deliberate multi-part reviews, AFAIK.17:01
alcabreraflaper87: cool! I'm sure openstackgerrit will let me know when your next iteration is in. ;)17:02
alcabreraBack to reviews. :)17:02
*** jdaggett has joined #openstack-marconi17:04
flaper87alcabrera: great comments, thanks a lot!17:04
alcabreraflaper87: np. :)17:06
*** flaper87 is now known as flaper87|afk17:07
alcabreraflaper87: Thoughts on https://review.openstack.org/#/c/51083/1 ? (pip install vs. python setup.py develop)17:07
alcabreraflaper87|afk: just missed you, hahaha17:07
openstackgerritA change was merged to openstack/python-marconiclient: Make the request object API aware  https://review.openstack.org/4978717:12
openstackgerritMalini Kamalambal proposed a change to openstack/marconi: Validation for messages returned by queue/stats  https://review.openstack.org/5099517:14
openstackgerritMalini Kamalambal proposed a change to openstack/marconi: Validation for messages returned by queue/stats  https://review.openstack.org/5099517:24
*** jdaggett has quit IRC17:26
*** tedross has quit IRC17:27
*** dafter has joined #openstack-marconi17:27
*** jdaggett has joined #openstack-marconi17:30
*** dafter has quit IRC17:32
*** openstackgerrit has quit IRC17:37
*** openstackgerrit has joined #openstack-marconi17:38
*** kgriffs_afk is now known as kgriffs17:39
alcabrerakgriffs: o/17:42
alcabreraThanks for all your work on the sharding details. I've found a place where I can pick up and continue, to start filling in those gaps.17:42
kgriffsI've been in meetings in San Antonio all morning, but should be done for the day now. :p17:43
alcabreranever enough meetings, ya know. ;)17:43
*** tedross has joined #openstack-marconi17:43
alcabreraspeaking of which, I do believe we have another meeting in just 15 minutes, heh.17:44
*** reed has joined #openstack-marconi17:44
alcabrerazyuan: responded else where. :)17:45
*** mpanetta has quit IRC17:53
*** kgriffs is now known as kgriffs_afk17:54
*** alcabrera is now known as alcabrera|afk17:57
amettskgriffs:  Time for the queues meeting. :D18:00
*** kgriffs_afk is now known as kgriffs18:00
openstackgerritKurt Griffiths proposed a change to openstack/marconi: feat: Storage sharding foundation  https://review.openstack.org/5043718:40
openstackgerritKurt Griffiths proposed a change to openstack/marconi: fix(queues): Global config used everywhere  https://review.openstack.org/5170518:40
openstackgerritKurt Griffiths proposed a change to openstack/marconi: Setup storage pipeline in the boostrap instead of driver base  https://review.openstack.org/5104918:40
*** fvollero is now known as fvollero|gone18:45
zyuanalcabrera|afk: can you log some work items about shading on some etherpad?18:46
*** alcabrera|afk is now known as alcabrera18:56
alcabrerazyuan: sure thing18:57
alcabreraI need to analyze the next points and then I'll find a means to log those tasks.18:58
alcabreraPOST /v1/queues/sharding_tasks/messages < all_the_things18:59
zyuanneed some filter18:59
zyuankgriffs: ping19:10
kgriffsalcabrera: wanna sync up on sharding?19:13
zyuankgriffs: what's the point of having both per-message limit and total limit?19:13
openstackgerritMalini Kamalambal proposed a change to openstack/marconi: Validation for messages returned by queue/stats  https://review.openstack.org/5099519:13
zyuani think we discussed the question before, but since we concluded that server alwasy have request limits, i want to revisit this problem19:14
*** tedross has quit IRC19:15
*** jdaggett has quit IRC19:15
alcabrerakgriffs: almost ready - I'm taking some notes on missing pieces.19:20
alcabrerakgriffs: and yes! :)19:21
alcabrerakgriffs: alright, I've got a reasonable understanding at this point.19:22
kgriffsso, thoughts on plugging into the catalog class?19:23
kgriffshow much work is involved there, what kind of work?19:23
alcabreraThere are two Catalogue classes now - one from the vantage of the admin API and one from the vantage of sharding.19:24
alcabreraMost of the plugging is pretty straight-forward, given that an instance of the storage controller used for the admin API catalogue is given to the sharding Catalog class.19:24
alcabrerafor example, register/deregister map to .create/.delete19:25
alcabrera.lookup maps to .get19:25
kgriffsok, so you will pass in the admin api catalog instance to sharding.catalog19:25
alcabrerayup - that's the plan.19:26
alcabrerawell, hmm...19:26
kgriffsthe other thing you will need to do is manufacture a ConfigOpts instance to pass to stevedore when it instantiates drivers for each shard19:26
alcabrerathat needs one more step19:26
alcabrerayeah, there's that, too.19:27
alcabreraThe other step I thought of is that the catalogue from the proxy needs to be ported to use the new semantics.19:27
alcabrerasemantics + schema19:27
kgriffsis the admin catalog thingy standalone, or coupled to the admin API?19:28
alcabrerathe storage is standalone19:28
alcabrerastorage driver19:28
kgriffsis the storage driver what you are passing to sharding.Catalog?19:28
kgriffsdoes the storage driver handle caching?19:29
alcabrerathat was previously handled at the transport layer19:29
kgriffsso, would caching happen in the new catalog class?19:29
kgriffsis caching needed anywhere else?19:30
alcabrerayeah, that seems like the place to handle that.19:30
alcabreraAs far as being needed anywhere else...19:30
alcabreragiven that we've moved most of the new logic to the storage layer for sharding, we shouldn't need to make any changes at the transport layer19:30
alcabreraSo caching should be the responsibility of the sharding driver19:31
alcabreraThat includes invalidation on deregister19:31
*** tedross has joined #openstack-marconi19:31
alcabreraSo there might need to be additional logic in place to handle invalidations whenever a queue is deleted.19:31
alcabreraThough it seems like the deregister method should be able to handle that - as long as it's called every time a delete queue is issued.19:32
kgriffsiirc, the queue controller in the sharding module always calls deregister when a queue is deleted19:33
alcabreracool. That should make things easy on that front.19:34
alcabreraSo I've got some four tasks that need to be addressed.19:34
alcabreraEach should have enough context to be independently reviewed.19:35
alcabreraThose tasks are: (feel free to add to that list)19:35
alcabrera1. Implement sharding controller's queue's list method (query each shard and merge)19:35
alcabrera2. Port/use the proxy catalogue storage driver to be used by sharding.catalog19:35
alcabrera3. Fill in the blanks for sharding.catalog19:36
alcabrera4. Add caching for sharding.catalog19:36
alcabrerazyuan, kgriffs: That's the gist of what's left (unit testing included at each step(19:36
kgriffsok, makes sense19:37
kgriffsand just to be clear, let's put off working on the REST admin API until the above is done and tested19:37
alcabreraadmin API is fleshed out enough19:37
kgriffsthat way malini can get to work sooner. :p19:38
alcabreraI'm going to start working on (2). I'll post an update on that by the end of the day today, because I suspect I might be able to use it *as-is*. If that's the case, I intend to pick up (3).19:38
alcabreraThat leaves (1) and (4).19:39
alcabrera(1) requires an instance of the shard storage controller.19:40
zyuan(1) is queue listing?19:40
alcabreraDoing (1) efficiently will be quite the feat, since the optimal case let's us use something like eventlet to perform those queries async-style.19:40
alcabrerazyuan: yes19:41
alcabreraIn the worst case, I'd say implement it sync on the first pass, and come back later to try to add async.19:41
zyuaninterface to query each db?19:41
zyuanin gerrit?19:42
alcabrerazyuan: https://review.openstack.org/#/c/50815/19:42
alcabrerathat's the shard controller patch, dependent on split_queues_api -> storage_interface19:42
kgriffsgive me 15 min to get those patches green before basing off them19:42
alcabrerakgriffs: sure thing. :P19:42
alcabrerazyuan: in the patch following the one above (https://review.openstack.org/#/c/50998/4), I changed 'location' -> 'uri'.19:44
alcabrera#50998 is the last in the line for the admin API patches, and represents the latest state of the storage controllers for sharding.19:44
alcabrerazyuan, kgriffs: I'm going to head home and continue working from there. You should see me online again in about 30 minutes.19:45
*** alcabrera has quit IRC19:45
*** jcru has quit IRC20:01
*** jdaggett has joined #openstack-marconi20:03
*** alcabrera has joined #openstack-marconi20:03
*** jcru has joined #openstack-marconi20:05
*** jcru has quit IRC20:05
openstackgerritKurt Griffiths proposed a change to openstack/marconi: feat: Storage sharding foundation  https://review.openstack.org/5043720:06
openstackgerritKurt Griffiths proposed a change to openstack/marconi: fix(queues): Global config used everywhere  https://review.openstack.org/5170520:06
*** jcru has joined #openstack-marconi20:06
*** kgriffs is now known as kgriffs_afk20:08
openstackgerritZhihao Yuan proposed a change to openstack/marconi: WIP: feat(api): define object size not JSON chars  https://review.openstack.org/5192120:12
alcabrerazyuan: pretty cool. Recursive definitions tend to read very elegantly. (object_size)20:18
openstackgerritZhihao Yuan proposed a change to openstack/marconi: WIP: feat(api): define object size not JSON chars  https://review.openstack.org/5192120:18
*** kgriffs_afk is now known as kgriffs20:19
zyuanalcabrera: that's why i say this is easy to implement even on general purpose clients20:20
*** jdaggett has quit IRC20:21
*** russell_h has joined #openstack-marconi20:26
*** jdaggett has joined #openstack-marconi20:34
kgriffszyuan: before you get too far on that sizing patch it may be a good idea to talk with flaper87 about your idea. Also we should engage Rackspace's DRG since they have the perspective of the client library author20:34
kgriffsdeveloper relations group20:35
kgriffsthey contribute to a lot of openstack client libs20:35
zyuani'm curious what openstack does now20:36
kgriffsthat would also be a good thing to research. :D20:36
zyuanhow the size is defined20:36
kgriffsswift object payloads are blobs20:37
alcabrerakgriffs, zyuan, flaper87|afk: I'm working on this (https://etherpad.openstack.org/p/sharding-merge-strategy) before proceeding with implementing things. I want the big picture of our current status to make it easier to merge/review/iterate. There's a lot of pieces and I want to avoid us redoing work.20:37
kgriffsevery other api is a management api20:37
kgriffswe are unique in that we have structured data going through our HTTP data API20:38
kgriffsbut, we may find some ideas out there20:38
kgriffsanyway, my point is I am willing to bet this is going to be a contentious question and it would be prudent to gather more feedback before proceeding. TBH, I am still not entirely comfortable with the proposal myself.20:39
openstackgerritMalini Kamalambal proposed a change to openstack/marconi: Validation for messages returned by queue/stats  https://review.openstack.org/5099520:41
zyuanwhile i usually prefer to develop something to ebd disputation... anyway, it's just an experimental thing so far20:42
openstackgerritZhihao Yuan proposed a change to openstack/marconi: WIP: feat(api): define object size not JSON chars  https://review.openstack.org/5192120:42
zyuanrebase ^20:42
zyuankgriffs: "ends disputation" i mean, so far, we have no idea how to uniform JSON size and msgpack size and whatever may appears in the future.20:43
zyuanother than this...20:44
kgriffsoh, sure20:44
kgriffsprototyping can definitely aid the discussion20:44
kgriffsI just want to make sure this idea is fully discussed20:44
alcabreraI like that you're prototyping the idea, zyuan. I think that makes this discussion much easoer to have come team meeting time.20:46
alcabreraalright, I've gathered the big picture on what's going on in marconi-sharding.20:46
zyuannext monday, hopefully20:46
alcabreraI've found two points that would benefit from some discussion, since they might be blockers or rebase-hell points.20:47
alcabreraall the details are fleshed out on that etherpad. :_20:47
kgriffs1.Merge the pipeline enhancements.20:48
kgriffsthose will have to be rebased on top of my latest patchsets20:48
kgriffsat least, the pipeline one20:49
malinihmmm..I am at  the verge of pulling my hair out..+o(20:50
maliniI am trying to get the extra validation for stats in20:50
maliniBut it keeps failing due to some weird time issue.20:50
maliniMy message create_time is sometimes ahead of 'now' by a few ms20:51
kgriffsso, you are creating messages in the future?20:52
kgriffswow. how did you manage that?20:52
*** jdaggett has quit IRC20:52
maliniwish I knew :D20:52
*** jdaggett has joined #openstack-marconi20:52
kgriffsif we could replicate it reliably, we could get a massive boost in performance. ;)20:52
maliniI wud do better things in future, if I cud do tht ;)20:53
kgriffsseriously tho20:54
kgriffssmells like rounding20:54
maliniI am using the timeutils module to get 'now'20:54
malinieg. for error http://logs.openstack.org/95/50995/8/check/gate-marconi-python26/1f00f01/console.html20:54
maliniCreated time 2013-10-15 20:43:14, Now 2013-10-15 20:43:13.91411720:55
kgriffsas a datetime, looks like20:55
kgriffsso, utcnow()20:55
kgriffsshould just return datetime.datetime.utcnow()20:56
*** jdaggett has quit IRC20:57
zyuankgriffs: ?20:57
zyuan(what does killer app mean?  i installed Chrome just in order to use Google Presentation)20:57
kgriffsmalini: I'm looking20:58
malinithanks kgriffs20:58
*** jcru_ has joined #openstack-marconi21:04
kgriffsmalini: looks like isotime should just truncate the time, not round up21:05
kgriffsso, there goes that theory21:05
kgriffsit may be a clock resolution thing21:05
zyuankgriffs: why?21:05
kgriffswhy what?21:05
zyuansometimes user expect different rounding21:05
malinikgriffs: I was wondering if we are dealing with multiple servers somewhere21:06
zyuanthe problem we met is now() is round up21:06
kgriffszyuan: I was just checking to see if timeutils was rounding up to the nearest second anywhere21:06
zyuanbut what if user want a time of past?21:06
maliniI do the post message before getting the 'now'..So the now shud be > created_time21:06
*** jcru has quit IRC21:07
malinilike this one Created time 2013-10-15 20:43:14, Now 2013-10-15 20:43:13.91411721:07
zyuanwhat i want to say is, either way may make sense21:07
kgriffsthis would be with the SQLite driver?21:07
maliniI cannot get a repro locally :(21:07
maliniaah..I didnt try with SQLite21:07
kgriffshow does that driver calculate now?21:07
maliniLet me try tht21:07
kgriffsmaybe it is slightly different than the way python calculates now21:07
alcabrerait would be the sqlite driver - it's the only one that could run on Jenkins.21:07
malinigood point21:08
alcabrerakgriffs: yeah, it uses JulianDate, or something like that.21:08
*** jcru_ has quit IRC21:08
alcabreraThat's a little different thac utcnow(), afaik.21:08
*** jcru has joined #openstack-marconi21:09
zyuani did no rounding, just default, then truncate it and return21:09
kgriffswell, since you can't be sure how the driver is calculating, I guess you need a fudge factor in the test?21:09
alcabreraso... one approach is to use rounding to our advantage. For those tests, we say that a result passes if a floating point time when truncated yields a value that is >= 0.21:09
zyuanso the number after . may be accumulated21:09
zyuanbut overall it should be correct21:09
alcabreramalini: could you implement the above strategy for the test? ^^21:10
alcabreradelta = int(...)21:10
maliniI cud..let me try tht21:11
kgriffsalcabrera: #3 on the etherpad can be done without waiting for flavio's patches, right?21:12
kgriffsI just want to do as much as we can without waiting on those21:12
kgriffsno. 3, as in "fill in the blanks for sharding.ctalog"21:13
alcabrerakgriffs: yup. I'm going to be working on that between tonight and tomorrow.21:13
* ametts realizes that "between tonight and tomorrow" is the stroke of midnight. alcabrera is really fast.21:14
zyuanOAO QAQ ToT21:15
alcabreraametts: though, I did say "working". "completing" is another story. ;)21:15
amettsGood point.  Maybe that's your sneaky way of saying "I'm not going to work on this at all." :)21:16
alcabreraAlternatively, I could borrow malini's technique and bring in work from the future.21:16
alcabreraWe could be done right now. :D21:16
malinialcabrera: I cud lend you my time machine ;)21:16
zyuanthe one which can only go back21:17
maliniuh oh..we wud be stuck at proxy then..21:17
zyuan(i guess you haven't watched Steins;Gate, hehe)21:18
malinianyways wrt my future bug, I cud repro it with sqlite & it went away when I enabled ntp..wonder if jenkins might not have ntp :-S21:19
zyuanomgsh ntp....21:20
russell_his it intended that GET /v1/queues returns 204 if there are no queues?21:24
alcabreramalini: it's possible. Good to hear that you were able to repro.21:24
zyuanrussell_h: yes21:24
russell_has opposed to an empty list response?21:24
russell_hthat seems unfriendly to clients21:24
zyuanrussell_h: oppose?21:24
zyuanrussell_h: where?21:24
russell_hI mean, if I'm making a client, now I have to handle two totally different successful resposnes21:25
zyuani mean, where21:25
alcabrerarussell_h: yes, GET /v1/queues => 204 when there are no queues.21:25
alcabrerathat's intentional.21:25
russell_hwhat's the rationale behind it?21:25
zyuanlisting messages also give you 20421:25
zyuanwhen there is no more message yet21:26
alcabrerain part, to be able to make decisions about how to proceed in processing a request without needing to investigate the body. kgriffs can share some more rationale.21:26
alcabrera*request -> response21:26
russell_hIMO, parsing the body is easier than not21:26
zyuanrussell_h: you'd better to look at response code first21:27
zyuanand very carefully21:27
zyuanmarconi is very very precise on using response code21:27
russell_hthis is going to make building clients unnecessarily complicated21:27
russell_hnow I need 3 branches to handle 2 cases21:27
zyuani don't think a client can servive without handling 40421:28
alcabrerawhy not use a dictionary lookup instead?21:28
alcabrera{200: process_body(), 204: continue(), 400: bad_request_handler(), ...}?21:28
zyuanso i don't think a client is worth to exist without understanding the design of an API, especially the API is totally RESTful-confirming21:28
russell_hbut why build an API if not to be friendly to clients?21:29
russell_hlike if I wanted to write a lot of code I'd just queue things in mongo directly21:29
zyuanit's something hard to define, "client friendly"21:29
russell_hthe whole point of an API is to provide friendly abstractions21:29
alcabrerarussell_h: you have a point there, and it's something that's hard to nail.21:29
zyuansome APIs embeds error inside response body21:30
zyuansome APIs all return 20021:30
zyuansome APIs returns something not JSON at all for security21:30
zyuanwhich is "client-friendly"?21:30
alcabreraI am starting to like the idea of a default representation for things like listing queues and messages - that allows the unification of the handling on the client-side, except in error scenarios.21:30
alcabrerawe've frozen the marconi v1.0 API, but keep bringing the feedback. Things will likely change to make life easier for everyone come v1.1 (maybe?) and definitely by v2.0.21:32
kgriffsjust catching up here21:32
kgriffsrussell_h: we of course want to be friendly to clients21:33
russell_hyeah, I might have been… hyperbolous21:33
*** oz_akan_ has quit IRC21:33
russell_hif thats a word21:33
kgriffsthe 204 thing comes up a lot and we are looking into it21:33
russell_hprops for making marconi easy to run though21:36
russell_htook like 3 minutes to get it going21:36
russell_h(of which 2 was spent screwing with virtualenv)21:36
kgriffsI think switching over to returning an empty array is probably going to be OK. 204 may save a few bytes on the wire, but it is starting to sound more trouble than it's worth. The abstract "correctness" is debatable - arguments on both sides. Which tells me it doesn't really matter. Just pick the one that is easiest to code against.21:37
alcabrerakgriffs: agreed21:37
kgriffsanyway, that is my thinking at the moment21:37
russell_hfwiw, I agree21:38
russell_hI'm about to sound like a ruby dev, but a few bytes one way or another don't matter much to me21:38
russell_hif they did I probably wouldn't use http at all21:38
kgriffsbtw, this change would have to wait for a v2 api i guess unless the community wants to OK media type versionioning21:39
kgriffsthat being said I always anticipated v2 coming close on the heals of v1 because we were bound to screw things up the first time around. :p21:39
kgriffsby close, I don't mean "1-2 months"21:40
kgriffsbut more like 6-12 months21:40
alcabreraonward, towards marconi v27!21:40
zyuanalcabrera: i'm drinking Mist21:40
kgriffsIn spite of our best efforts to vet with the community, some things still end up broken.21:40
kgriffs(you can't know everything in advance)21:41
kgriffs(or keep track of everything all the time)21:41
russell_htrue story21:41
*** kgriffs is now known as kgriffs_afk21:41
zyuani against changing 204 to 200, because client implementers will find that they have to handle [] case then21:42
zyuanthe turth is, we currently give "non-empty" guarentee21:42
russell_his [] a special case though?21:42
zyuanwhich means, the document you get always have the 'next' link21:42
*** kgriffs_afk is now known as kgriffs21:43
zyuanthen here comes the question: do you expect the empty response also has "next" link?21:43
russell_hlike what would you do with an empty list that you wouldn't do with a non-empty list?21:43
zyuanwhen you got an 204 or empty list, client are suppose to try later21:43
zyuanbecause listing is a continuous operation21:43
* kgriffs made it through the temporal distortion field21:43
alcabrerazyuan: I'd expect it to have the same structure. {'links': {}, 'messages': []}21:43
* kgriffs is back21:44
* alcabrera notices how fast kgriffs is21:44
zyuanalcabrera: yes, then it's not just a few bytes, it's "some" bytes21:44
zyuanand "some" is duplicated for each, say second21:44
russell_hbut if I get a non-empty list, I'm almost certainly just going to iterate the list, then poll again21:44
zyuanwhat i'm saying, we have to handle some special case, unless we send "some" bytes21:45
zyuanand i don't think it worth21:45
zyuanand 204 is the cheapest choise to say "client, retry!"21:45
russell_hthats fair21:46
russell_hI don't think the tradeoff is worth it, but I can live either way21:46
russell_heven if the empty response were 512 bytes every second (it should be like 5% of that) we're takling… 40MB/day?21:47
*** tedross has quit IRC21:47
alcabreranumbers are fun to think about.21:48
alcabrerait's all about the target scale, at that point21:48
alcabrerarequests per second21:48
zyuanit also uses server resource, like CPU, like power...21:48
zyuanok, i don't know21:48
russell_hheh :)21:48
alcabreraI'm in agreement about the trade off - it's a matter of efficiency vs. productivity. It is easier to handle a uniform representation. :)21:49
alcabreraSo assuming that somebody is serving a million requests per second using a marconi deployment, then the wasted bandwidth per day amounts to about ~2TB, also assuming that all of those requests are for listing queues.21:51
alcabreraIt's all about trade-offs. :P21:51
alcabrera+1 for *not* using HTTP in such a scenario21:51
kgriffsrussell_h: let us know what other ways we can improve the API.21:51
* kgriffs loves feedback21:52
kgriffs(as you come across them)21:52
russell_hwill do21:52
kgriffsrock on21:52
alcabrerarussell_h: the README needs help, so I'm really happy to hear you got marconi up and running despite that. :D21:52
kgriffsI know flaper87 is also very open to ideas21:53
kgriffshe is based in Italy, tho, so you'll have to catch him earlier in the day. :p21:53
* kgriffs goes back to reviewing patches21:54
* alcabrera returns to porting the catalogue21:54
zyuanrussell_h: flaper87|afk is developing python client; feel free to ping him21:57
russell_hnice, will do21:57
alcabrerakgriffs, zyuan: Yeah, there's going to be some porting effort. The proxy catalogue implemented entries in terms of {project, queue, partition_name, extraneous_fields, ...}. I need to update those assumptions to remove the extraneous fields and maintain a mapping from project + queue => shard identifier.21:58
kgriffsyou might want to use that combining function so you end up with a single field, rather than two fields, "project" and "id"21:59
zyuanwe already have the function21:59
alcabreraget_scoped_something_or_other, iirc22:00
kgriffsI guess then scope_queue_name would need to move up a level to storage.utils22:00
alcabreraI'll address those two in the porting patch.22:00
alcabrerabefore I head out for the night, how do you feel about the proposed merging strategy, kgriffs? I'll bring it for discussion with flaper87|afk tomorrow morning to get some more input, too.22:01
alcabrera**bring it up for...22:02
kgriffssorry, did that get updated? Seems like we identified some places where we didn't have to wait to merge flaper8722:02
kgriffs's pipeline patches22:02
alcabreralemme double check22:02
alcabreraI think I updated it.22:02
alcabreraThe merging strategy is pretty midterm, though.22:02
alcabreraI'm thinking out to22:02
kgriffsI actually gotta run22:03
alcabrera"things I'd like to see done by Monday"22:03
kgriffswe can sync up on things in the morning22:03
alcabreracool, cool22:03
kgriffsI should be online a lot earlier than today22:03
alcabrerame, too.22:03
*** alcabrera has quit IRC22:05
*** kgriffs is now known as kgriffs_afk22:06
*** amitgandhi has quit IRC22:28
*** ayoung has quit IRC22:29
*** oz_akan_ has joined #openstack-marconi22:45
*** oz_akan_ has quit IRC22:49
*** fifieldt has joined #openstack-marconi22:56
*** jcru has quit IRC23:14
*** amitgandhi has joined #openstack-marconi23:34
*** amitgandhi has quit IRC23:41
*** amitgandhi has joined #openstack-marconi23:41

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!