| *** tedross has quit IRC | 00:07 | |
| *** reed has quit IRC | 00:15 | |
| *** Chat2315 has joined #openstack-marconi | 00:39 | |
| *** nosnos has joined #openstack-marconi | 01:05 | |
| *** kgriffs_afk is now known as kgriffs | 02:06 | |
| *** malini_afk is now known as malini | 02:07 | |
| *** kgriffs is now known as kgriffs_afk | 03:26 | |
| *** kgriffs_afk is now known as kgriffs | 03:27 | |
| *** kgriffs is now known as kgriffs_afk | 03:34 | |
| jburkhart | is marconi currently closed source, or am I just looking in the wrong place? I'm getting a 404 from github on the link provided under Resources on the wiki -- (https://github.com/stackforge/marconi) | 04:02 |
|---|---|---|
| jburkhart | hmm, looks like there was a git repo available at https://review.openstack.org/openstack/marconi -- you might want to update the wiki Resources section to reflect the move | 04:05 |
| *** reed has joined #openstack-marconi | 04:14 | |
| *** reed has quit IRC | 04:25 | |
| *** dafter has joined #openstack-marconi | 07:38 | |
| *** dafter has quit IRC | 07:38 | |
| *** dafter has joined #openstack-marconi | 07:38 | |
| *** flaper87|afk is now known as flaper87 | 07:45 | |
| *** ykaplan has joined #openstack-marconi | 07:46 | |
| flaper87 | jburkhart: it's open source github.com/openstack/marconi | 07:46 |
| *** yassine has joined #openstack-marconi | 07:57 | |
| openstackgerrit | A change was merged to openstack/marconi: chore: Remove GC cruft from storage driver base class https://review.openstack.org/51016 | 08:08 |
| *** tvb|afk has joined #openstack-marconi | 08:15 | |
| *** dafter has quit IRC | 08:19 | |
| flaper87 | fvollero|gone: come back, now! | 08:43 |
| flaper87 | :D | 08:43 |
| *** tvb|afk has quit IRC | 08:49 | |
| *** ykaplan has quit IRC | 08:50 | |
| *** openstackgerrit has quit IRC | 09:38 | |
| *** dafter has joined #openstack-marconi | 09:42 | |
| *** dafter has quit IRC | 09:43 | |
| *** tvb|afk has joined #openstack-marconi | 09:49 | |
| *** flaper87 is now known as flaper87|afk | 10:11 | |
| *** nosnos has quit IRC | 10:28 | |
| *** nosnos has joined #openstack-marconi | 10:28 | |
| *** nosnos has quit IRC | 10:33 | |
| *** tvb|afk has quit IRC | 10:37 | |
| *** dafter has joined #openstack-marconi | 10:37 | |
| *** fifieldt_ has quit IRC | 10:39 | |
| *** tvb|afk has joined #openstack-marconi | 10:42 | |
| *** tvb|afk has quit IRC | 10:42 | |
| *** tvb|afk has joined #openstack-marconi | 10:42 | |
| *** dafter has quit IRC | 10:43 | |
| *** fifieldt has joined #openstack-marconi | 10:45 | |
| *** tvb|afk has quit IRC | 11:01 | |
| *** dafter has joined #openstack-marconi | 11:02 | |
| *** dafter has quit IRC | 11:02 | |
| *** dafter has joined #openstack-marconi | 11:02 | |
| *** tvb|afk has joined #openstack-marconi | 11:28 | |
| *** dafter_ has joined #openstack-marconi | 11:29 | |
| *** dafter_ has quit IRC | 11:29 | |
| *** dafter_ has joined #openstack-marconi | 11:30 | |
| *** dafter has quit IRC | 11:32 | |
| *** dafter_ has quit IRC | 11:32 | |
| *** tvb|afk has quit IRC | 11:33 | |
| *** tedross has joined #openstack-marconi | 11:38 | |
| *** flaper87|afk is now known as flaper87 | 11:49 | |
| *** dafter has joined #openstack-marconi | 11:52 | |
| *** tvb|afk has joined #openstack-marconi | 11:57 | |
| *** tvb|afk has quit IRC | 11:57 | |
| *** tvb|afk has joined #openstack-marconi | 11:57 | |
| *** tvb|afk has quit IRC | 11:57 | |
| *** tvb|afk has joined #openstack-marconi | 11:57 | |
| *** dafter has quit IRC | 12:00 | |
| *** yassine has quit IRC | 12:04 | |
| *** flaper87 is now known as flaper87|afk | 12:08 | |
| *** yassine has joined #openstack-marconi | 12:12 | |
| *** flaper87|afk is now known as flaper87 | 12:16 | |
| *** mpanetta has joined #openstack-marconi | 12:16 | |
| *** mpanetta has quit IRC | 12:20 | |
| *** mpanetta has joined #openstack-marconi | 12:23 | |
| *** ayoung has quit IRC | 12:41 | |
| *** oz_akan_ has joined #openstack-marconi | 12:55 | |
| *** fifieldt has quit IRC | 12:56 | |
| *** oz_akan_ has quit IRC | 12:56 | |
| *** oz_akan_ has joined #openstack-marconi | 12:57 | |
| *** openstackgerrit has joined #openstack-marconi | 13:02 | |
| *** tvb|afk has quit IRC | 13:04 | |
| *** fvollero|gone is now known as fvollero | 13:05 | |
| fvollero | flaper87: son qui | 13:05 |
| *** dafter has joined #openstack-marconi | 13:07 | |
| flaper87 | fvollero: yoooo!!! | 13:07 |
| flaper87 | fvollero: how ya doing? | 13:07 |
| fvollero | flaper87: not bad, emanuela is here, i'll talk with her later and give your damn cocosette | 13:08 |
| flaper87 | fvollero: hahaha, thanks buddy!!!! | 13:08 |
| *** amitgandhi has joined #openstack-marconi | 13:09 | |
| flaper87 | fvollero: I wanted to know how is the ES thing going and whether you wanted to take a look over the client patches | 13:09 |
| flaper87 | if you've some time | 13:09 |
| fvollero | flaper87: still trying to fix packstack && puppet :( it's mandatory to have the support by thrusday, so I hope to finish sooner | 13:11 |
| *** dafter has quit IRC | 13:15 | |
| *** dafter has joined #openstack-marconi | 13:15 | |
| *** tvb|afk has joined #openstack-marconi | 13:18 | |
| *** dafter has quit IRC | 13:20 | |
| *** yassine has quit IRC | 13:23 | |
| *** yassine has joined #openstack-marconi | 13:23 | |
| flaper87 | am I still alive? | 13:27 |
| ametts | flaper87: Either that, or you figured out how to connect to IRC from beyond the grave... | 13:32 |
| flaper87 | ametts: hahahah, most likely the second one! | 13:33 |
| flaper87 | :D | 13:33 |
| flaper87 | If there's something I'll take with me from this world, it is internet! | 13:33 |
| ametts | :) | 13:34 |
| *** jcru has joined #openstack-marconi | 13:43 | |
| *** amitgandhi has quit IRC | 13:49 | |
| *** kgriffs_afk is now known as kgriffs | 13:59 | |
| *** ayoung_ has joined #openstack-marconi | 14:05 | |
| *** kgriffs is now known as kgriffs_afk | 14:08 | |
| openstackgerrit | Flavio Percoco proposed a change to openstack/python-marconiclient: Add list of required fields to the API definition https://review.openstack.org/51850 | 14:09 |
| *** amitgandhi has joined #openstack-marconi | 14:10 | |
| *** ayoung_ is now known as ayoung | 14:12 | |
| flaper87 | knock ? | 14:41 |
| flaper87 | where's everybody? | 14:41 |
| zyuan | ? | 14:42 |
| flaper87 | zyuan: hey! :D | 14:42 |
| zyuan | sup | 14:43 |
| zyuan | internet hahahh | 14:43 |
| zyuan | an internet without any people is also boring | 14:44 |
| flaper87 | zyuan: indeed! I couldn't agree more! | 14:53 |
| *** alcabrera has joined #openstack-marconi | 14:53 | |
| flaper87 | it's like being alone between so many things | 14:54 |
| flaper87 | :D | 14:54 |
| flaper87 | alcabrera: GOOOOD MORNING!!! | 14:54 |
| alcabrera | flaper87: hey! :D | 14:54 |
| flaper87 | alcabrera: how are you doing? | 14:57 |
| alcabrera | a bit sluggish. It's been slow going for me this week with a lot going on at home. @_@ | 14:58 |
| alcabrera | Between one of my cats hurting her leg, having a doctor's appointment on a Monday, and a few other things, it's been a little hard to stay focused. :P | 14:59 |
| *** reed has joined #openstack-marconi | 15:01 | |
| flaper87 | alcabrera: aahh, I'm sorry about your cat :( | 15:02 |
| flaper87 | Yeah, kind of the same here! :/ | 15:03 |
| alcabrera | the little one's getting better, thankfully. She hobbles a bit, but less so every day. :) | 15:04 |
| alcabrera | flaper87: it's just one of those weeks! I hope your's gets better, too. | 15:04 |
| openstackgerrit | A change was merged to openstack/marconi: Follow hacking rules about import https://review.openstack.org/50308 | 15:04 |
| flaper87 | w000t, 2 patches merged today! | 15:05 |
| alcabrera | hehe | 15:05 |
| alcabrera | I've been swimming in Haskell in my free time. Monads finally clicked this weekend. :P | 15:05 |
| zyuan | wuu | 15:06 |
| alcabrera | zyuan: maybe I'll put together an introduction to haskell tech talk at some point. :D | 15:07 |
| zyuan | (i guess seldom people care.... -_-) | 15:08 |
| alcabrera | lol | 15:08 |
| alcabrera | it's all about finding a way to make it relevant. :) | 15:08 |
| zyuan | (can't think of any ... -_-) | 15:08 |
| zyuan | it IS relevant to the programming world, but | 15:09 |
| zyuan | not every parts | 15:09 |
| zyuan | of the world | 15:09 |
| zyuan | flaper87: does redhat have tech talk about haskell? | 15:11 |
| * flaper87 likes haskell :D | 15:11 | |
| flaper87 | zyuan: yup, we've tech talks about anything we want to talk about | 15:11 |
| flaper87 | There was one about Go recently | 15:11 |
| *** reed has quit IRC | 15:11 | |
| alcabrera | flaper87: awesome! We're trying to grow the tech talk program here. It's very flexible, and we can talk about anything, but getting participation up is challenging. :) | 15:18 |
| alcabrera | Now I'm going to sit back, grab a pencil. and go through the latest changes in marconi-sharding. | 15:18 |
| openstackgerrit | Malini Kamalambal proposed a change to openstack/marconi: Validation for messages returned by queue/stats https://review.openstack.org/50995 | 15:19 |
| flaper87 | alcabrera: hehehehehe | 15:25 |
| *** malini is now known as malini_afk | 15:30 | |
| *** malini_afk is now known as malini | 15:30 | |
| *** yassine has quit IRC | 15:46 | |
| openstackgerrit | Flavio Percoco proposed a change to openstack/python-marconiclient: Implement queue's API methods https://review.openstack.org/50638 | 15:53 |
| openstackgerrit | Flavio Percoco proposed a change to openstack/python-marconiclient: Add list of required fields to the API definition https://review.openstack.org/51850 | 15:53 |
| openstackgerrit | Malini Kamalambal proposed a change to openstack/marconi: Validation for messages returned by queue/stats https://review.openstack.org/50995 | 16:09 |
| * flaper87 is working on the message's API (client-side) | 16:11 | |
| flaper87 | alcabrera: btw, I think there are 2 new patches waiting for you | 16:12 |
| *** tvb|afk has quit IRC | 16:16 | |
| alcabrera | flaper87: nice! | 16:27 |
| alcabrera | I just finished up lunch - forgot to set my afk. :P | 16:28 |
| zyuan | there are a lot https://review.openstack.org/#/q/status:open+project:openstack/marconi,n,z | 16:28 |
| alcabrera | zyuan: so many things to review. :P | 16:30 |
| flaper87 | ... and code on! :D | 16:30 |
| flaper87 | anyway, zyuan is right. We've got to get that list smaller | 16:31 |
| alcabrera | agreed. | 16:35 |
| alcabrera | Let's see if I can help that along before I branch off of kgriffs_afk's latest patch and continue the sharding effort. | 16:35 |
| alcabrera | Headphones time - this will require music. | 16:35 |
| flaper87 | alcabrera: :D | 16:39 |
| flaper87 | alcabrera: enjoy! | 16:39 |
| flaper87 | alcabrera: Do you use spotify? | 16:39 |
| * flaper87 is happy to announce that it's now possible to post and list messages using the client library | 16:39 | |
| alcabrera | flaper87: no, though I've heard of spotify. I mostly use audacious player locally and listen to game OSTs. :) | 16:40 |
| alcabrera | flaper87: awesome! Client lib is *winning*. :D | 16:40 |
| alcabrera | flaper87: https://review.openstack.org/#/c/50638/ (-1) | 16:58 |
| alcabrera | Looking good, though! | 16:58 |
| alcabrera | I'm breaking that review into parts to do it justice. | 16:58 |
| alcabrera | 700 lines is at my limit. :P | 16:59 |
| flaper87 | alcabrera: awesome, THANKS! | 16:59 |
| flaper87 | alcabrera: yeah, I just couldn't split it! :/ | 16:59 |
| flaper87 | I guess I could split the next one in lower-level / higher-level | 16:59 |
| alcabrera | flaper87: no worries on that one - patch splitting is difficult, esp. as we cover new ground. I understand. :) | 17:00 |
| flaper87 | but I think it doesn't show how the API would be used and it may hide some possible issues in the design | 17:00 |
| alcabrera | yeah, that's the hard part. | 17:00 |
| alcabrera | Lossless splitting | 17:00 |
| alcabrera | We need the context. | 17:00 |
| flaper87 | alcabrera: awesome, thanks for understanding! I'll take a look at your comments, address them and then we can iterate again | 17:00 |
| alcabrera | It might be best to do multi-part reviews for the big ones. | 17:01 |
| alcabrera | The running theory on review limits seem to discuss single session limitations (400-600 lines over a period of 30 minutes), but say nothing about deliberate multi-part reviews, AFAIK. | 17:01 |
| alcabrera | flaper87: cool! I'm sure openstackgerrit will let me know when your next iteration is in. ;) | 17:02 |
| alcabrera | Back to reviews. :) | 17:02 |
| *** jdaggett has joined #openstack-marconi | 17:04 | |
| flaper87 | alcabrera: great comments, thanks a lot! | 17:04 |
| alcabrera | flaper87: np. :) | 17:06 |
| *** flaper87 is now known as flaper87|afk | 17:07 | |
| alcabrera | flaper87: Thoughts on https://review.openstack.org/#/c/51083/1 ? (pip install vs. python setup.py develop) | 17:07 |
| alcabrera | flaper87|afk: just missed you, hahaha | 17:07 |
| openstackgerrit | A change was merged to openstack/python-marconiclient: Make the request object API aware https://review.openstack.org/49787 | 17:12 |
| openstackgerrit | Malini Kamalambal proposed a change to openstack/marconi: Validation for messages returned by queue/stats https://review.openstack.org/50995 | 17:14 |
| openstackgerrit | Malini Kamalambal proposed a change to openstack/marconi: Validation for messages returned by queue/stats https://review.openstack.org/50995 | 17:24 |
| *** jdaggett has quit IRC | 17:26 | |
| *** tedross has quit IRC | 17:27 | |
| *** dafter has joined #openstack-marconi | 17:27 | |
| *** jdaggett has joined #openstack-marconi | 17:30 | |
| *** dafter has quit IRC | 17:32 | |
| *** openstackgerrit has quit IRC | 17:37 | |
| *** openstackgerrit has joined #openstack-marconi | 17:38 | |
| *** kgriffs_afk is now known as kgriffs | 17:39 | |
| alcabrera | kgriffs: o/ | 17:42 |
| kgriffs | yo | 17:42 |
| alcabrera | Thanks for all your work on the sharding details. I've found a place where I can pick up and continue, to start filling in those gaps. | 17:42 |
| kgriffs | I've been in meetings in San Antonio all morning, but should be done for the day now. :p | 17:43 |
| alcabrera | heh | 17:43 |
| alcabrera | never enough meetings, ya know. ;) | 17:43 |
| *** tedross has joined #openstack-marconi | 17:43 | |
| alcabrera | speaking of which, I do believe we have another meeting in just 15 minutes, heh. | 17:44 |
| *** reed has joined #openstack-marconi | 17:44 | |
| zyuan | where? | 17:44 |
| alcabrera | zyuan: responded else where. :) | 17:45 |
| *** mpanetta has quit IRC | 17:53 | |
| *** kgriffs is now known as kgriffs_afk | 17:54 | |
| *** alcabrera is now known as alcabrera|afk | 17:57 | |
| ametts | kgriffs: Time for the queues meeting. :D | 18:00 |
| *** kgriffs_afk is now known as kgriffs | 18:00 | |
| openstackgerrit | Kurt Griffiths proposed a change to openstack/marconi: feat: Storage sharding foundation https://review.openstack.org/50437 | 18:40 |
| openstackgerrit | Kurt Griffiths proposed a change to openstack/marconi: fix(queues): Global config used everywhere https://review.openstack.org/51705 | 18:40 |
| openstackgerrit | Kurt Griffiths proposed a change to openstack/marconi: Setup storage pipeline in the boostrap instead of driver base https://review.openstack.org/51049 | 18:40 |
| *** fvollero is now known as fvollero|gone | 18:45 | |
| zyuan | alcabrera|afk: can you log some work items about shading on some etherpad? | 18:46 |
| *** alcabrera|afk is now known as alcabrera | 18:56 | |
| alcabrera | zyuan: sure thing | 18:57 |
| alcabrera | I need to analyze the next points and then I'll find a means to log those tasks. | 18:58 |
| alcabrera | POST /v1/queues/sharding_tasks/messages < all_the_things | 18:59 |
| zyuan | lol | 18:59 |
| zyuan | need some filter | 18:59 |
| zyuan | kgriffs: ping | 19:10 |
| kgriffs | here | 19:13 |
| kgriffs | alcabrera: wanna sync up on sharding? | 19:13 |
| zyuan | kgriffs: what's the point of having both per-message limit and total limit? | 19:13 |
| openstackgerrit | Malini Kamalambal proposed a change to openstack/marconi: Validation for messages returned by queue/stats https://review.openstack.org/50995 | 19:13 |
| zyuan | i think we discussed the question before, but since we concluded that server alwasy have request limits, i want to revisit this problem | 19:14 |
| *** tedross has quit IRC | 19:15 | |
| *** jdaggett has quit IRC | 19:15 | |
| alcabrera | kgriffs: almost ready - I'm taking some notes on missing pieces. | 19:20 |
| alcabrera | kgriffs: and yes! :) | 19:21 |
| kgriffs | ok | 19:22 |
| alcabrera | kgriffs: alright, I've got a reasonable understanding at this point. | 19:22 |
| alcabrera | ready | 19:22 |
| kgriffs | so, thoughts on plugging into the catalog class? | 19:23 |
| kgriffs | how much work is involved there, what kind of work? | 19:23 |
| alcabrera | There are two Catalogue classes now - one from the vantage of the admin API and one from the vantage of sharding. | 19:24 |
| alcabrera | Most of the plugging is pretty straight-forward, given that an instance of the storage controller used for the admin API catalogue is given to the sharding Catalog class. | 19:24 |
| alcabrera | for example, register/deregister map to .create/.delete | 19:25 |
| alcabrera | .lookup maps to .get | 19:25 |
| kgriffs | ok, so you will pass in the admin api catalog instance to sharding.catalog | 19:25 |
| alcabrera | yup - that's the plan. | 19:26 |
| alcabrera | well, hmm... | 19:26 |
| kgriffs | the other thing you will need to do is manufacture a ConfigOpts instance to pass to stevedore when it instantiates drivers for each shard | 19:26 |
| alcabrera | that needs one more step | 19:26 |
| alcabrera | yeah, there's that, too. | 19:27 |
| alcabrera | The other step I thought of is that the catalogue from the proxy needs to be ported to use the new semantics. | 19:27 |
| alcabrera | semantics + schema | 19:27 |
| alcabrera | hmmm | 19:27 |
| kgriffs | is the admin catalog thingy standalone, or coupled to the admin API? | 19:28 |
| alcabrera | the storage is standalone | 19:28 |
| alcabrera | storage driver | 19:28 |
| kgriffs | is the storage driver what you are passing to sharding.Catalog? | 19:28 |
| alcabrera | yes | 19:28 |
| kgriffs | does the storage driver handle caching? | 19:29 |
| alcabrera | no | 19:29 |
| alcabrera | that was previously handled at the transport layer | 19:29 |
| kgriffs | so, would caching happen in the new catalog class? | 19:29 |
| kgriffs | is caching needed anywhere else? | 19:30 |
| alcabrera | yeah, that seems like the place to handle that. | 19:30 |
| alcabrera | As far as being needed anywhere else... | 19:30 |
| alcabrera | given that we've moved most of the new logic to the storage layer for sharding, we shouldn't need to make any changes at the transport layer | 19:30 |
| alcabrera | So caching should be the responsibility of the sharding driver | 19:31 |
| alcabrera | That includes invalidation on deregister | 19:31 |
| alcabrera | Hmmm... | 19:31 |
| *** tedross has joined #openstack-marconi | 19:31 | |
| alcabrera | So there might need to be additional logic in place to handle invalidations whenever a queue is deleted. | 19:31 |
| alcabrera | Maybe | 19:31 |
| alcabrera | Though it seems like the deregister method should be able to handle that - as long as it's called every time a delete queue is issued. | 19:32 |
| kgriffs | iirc, the queue controller in the sharding module always calls deregister when a queue is deleted | 19:33 |
| alcabrera | cool. That should make things easy on that front. | 19:34 |
| alcabrera | So I've got some four tasks that need to be addressed. | 19:34 |
| alcabrera | Each should have enough context to be independently reviewed. | 19:35 |
| alcabrera | Those tasks are: (feel free to add to that list) | 19:35 |
| alcabrera | 1. Implement sharding controller's queue's list method (query each shard and merge) | 19:35 |
| alcabrera | 2. Port/use the proxy catalogue storage driver to be used by sharding.catalog | 19:35 |
| alcabrera | 3. Fill in the blanks for sharding.catalog | 19:36 |
| alcabrera | 4. Add caching for sharding.catalog | 19:36 |
| alcabrera | zyuan, kgriffs: That's the gist of what's left (unit testing included at each step( | 19:36 |
| alcabrera | )) | 19:36 |
| kgriffs | ok, makes sense | 19:37 |
| kgriffs | and just to be clear, let's put off working on the REST admin API until the above is done and tested | 19:37 |
| alcabrera | +1 | 19:37 |
| alcabrera | admin API is fleshed out enough | 19:37 |
| kgriffs | that way malini can get to work sooner. :p | 19:38 |
| alcabrera | yeeesss | 19:38 |
| alcabrera | I'm going to start working on (2). I'll post an update on that by the end of the day today, because I suspect I might be able to use it *as-is*. If that's the case, I intend to pick up (3). | 19:38 |
| alcabrera | That leaves (1) and (4). | 19:39 |
| alcabrera | (1) requires an instance of the shard storage controller. | 19:40 |
| zyuan | (1) is queue listing? | 19:40 |
| alcabrera | Doing (1) efficiently will be quite the feat, since the optimal case let's us use something like eventlet to perform those queries async-style. | 19:40 |
| alcabrera | zyuan: yes | 19:41 |
| alcabrera | In the worst case, I'd say implement it sync on the first pass, and come back later to try to add async. | 19:41 |
| zyuan | interface to query each db? | 19:41 |
| zyuan | in gerrit? | 19:42 |
| alcabrera | zyuan: https://review.openstack.org/#/c/50815/ | 19:42 |
| kgriffs | ok | 19:42 |
| alcabrera | that's the shard controller patch, dependent on split_queues_api -> storage_interface | 19:42 |
| kgriffs | give me 15 min to get those patches green before basing off them | 19:42 |
| kgriffs | :p | 19:42 |
| alcabrera | kgriffs: sure thing. :P | 19:42 |
| alcabrera | zyuan: in the patch following the one above (https://review.openstack.org/#/c/50998/4), I changed 'location' -> 'uri'. | 19:44 |
| alcabrera | #50998 is the last in the line for the admin API patches, and represents the latest state of the storage controllers for sharding. | 19:44 |
| alcabrera | zyuan, kgriffs: I'm going to head home and continue working from there. You should see me online again in about 30 minutes. | 19:45 |
| zyuan | ok. | 19:45 |
| alcabrera | o/ | 19:45 |
| *** alcabrera has quit IRC | 19:45 | |
| *** jcru has quit IRC | 20:01 | |
| *** jdaggett has joined #openstack-marconi | 20:03 | |
| *** alcabrera has joined #openstack-marconi | 20:03 | |
| *** jcru has joined #openstack-marconi | 20:05 | |
| *** jcru has quit IRC | 20:05 | |
| openstackgerrit | Kurt Griffiths proposed a change to openstack/marconi: feat: Storage sharding foundation https://review.openstack.org/50437 | 20:06 |
| openstackgerrit | Kurt Griffiths proposed a change to openstack/marconi: fix(queues): Global config used everywhere https://review.openstack.org/51705 | 20:06 |
| *** jcru has joined #openstack-marconi | 20:06 | |
| alcabrera | o/ | 20:07 |
| *** kgriffs is now known as kgriffs_afk | 20:08 | |
| openstackgerrit | Zhihao Yuan proposed a change to openstack/marconi: WIP: feat(api): define object size not JSON chars https://review.openstack.org/51921 | 20:12 |
| alcabrera | zyuan: pretty cool. Recursive definitions tend to read very elegantly. (object_size) | 20:18 |
| openstackgerrit | Zhihao Yuan proposed a change to openstack/marconi: WIP: feat(api): define object size not JSON chars https://review.openstack.org/51921 | 20:18 |
| *** kgriffs_afk is now known as kgriffs | 20:19 | |
| zyuan | alcabrera: that's why i say this is easy to implement even on general purpose clients | 20:20 |
| *** jdaggett has quit IRC | 20:21 | |
| *** russell_h has joined #openstack-marconi | 20:26 | |
| *** jdaggett has joined #openstack-marconi | 20:34 | |
| kgriffs | zyuan: before you get too far on that sizing patch it may be a good idea to talk with flaper87 about your idea. Also we should engage Rackspace's DRG since they have the perspective of the client library author | 20:34 |
| zyuan | DRG? | 20:35 |
| kgriffs | developer relations group | 20:35 |
| kgriffs | they contribute to a lot of openstack client libs | 20:35 |
| zyuan | i'm curious what openstack does now | 20:36 |
| kgriffs | that would also be a good thing to research. :D | 20:36 |
| zyuan | how the size is defined | 20:36 |
| kgriffs | swift object payloads are blobs | 20:37 |
| alcabrera | kgriffs, zyuan, flaper87|afk: I'm working on this (https://etherpad.openstack.org/p/sharding-merge-strategy) before proceeding with implementing things. I want the big picture of our current status to make it easier to merge/review/iterate. There's a lot of pieces and I want to avoid us redoing work. | 20:37 |
| kgriffs | every other api is a management api | 20:37 |
| kgriffs | we are unique in that we have structured data going through our HTTP data API | 20:38 |
| kgriffs | but, we may find some ideas out there | 20:38 |
| kgriffs | anyway, my point is I am willing to bet this is going to be a contentious question and it would be prudent to gather more feedback before proceeding. TBH, I am still not entirely comfortable with the proposal myself. | 20:39 |
| openstackgerrit | Malini Kamalambal proposed a change to openstack/marconi: Validation for messages returned by queue/stats https://review.openstack.org/50995 | 20:41 |
| zyuan | while i usually prefer to develop something to ebd disputation... anyway, it's just an experimental thing so far | 20:42 |
| openstackgerrit | Zhihao Yuan proposed a change to openstack/marconi: WIP: feat(api): define object size not JSON chars https://review.openstack.org/51921 | 20:42 |
| zyuan | rebase ^ | 20:42 |
| zyuan | kgriffs: "ends disputation" i mean, so far, we have no idea how to uniform JSON size and msgpack size and whatever may appears in the future. | 20:43 |
| zyuan | other than this... | 20:44 |
| zyuan | unify* | 20:44 |
| kgriffs | oh, sure | 20:44 |
| kgriffs | prototyping can definitely aid the discussion | 20:44 |
| kgriffs | I just want to make sure this idea is fully discussed | 20:44 |
| zyuan | yes | 20:45 |
| alcabrera | I like that you're prototyping the idea, zyuan. I think that makes this discussion much easoer to have come team meeting time. | 20:46 |
| alcabrera | *easier | 20:46 |
| alcabrera | alright, I've gathered the big picture on what's going on in marconi-sharding. | 20:46 |
| zyuan | next monday, hopefully | 20:46 |
| alcabrera | I've found two points that would benefit from some discussion, since they might be blockers or rebase-hell points. | 20:47 |
| alcabrera | all the details are fleshed out on that etherpad. :_ | 20:47 |
| alcabrera | :) | 20:47 |
| kgriffs | 1.Merge the pipeline enhancements. | 20:48 |
| kgriffs | those will have to be rebased on top of my latest patchsets | 20:48 |
| kgriffs | at least, the pipeline one | 20:49 |
| malini | hmmm..I am at the verge of pulling my hair out..+o( | 20:50 |
| malini | I am trying to get the extra validation for stats in | 20:50 |
| malini | But it keeps failing due to some weird time issue. | 20:50 |
| malini | My message create_time is sometimes ahead of 'now' by a few ms | 20:51 |
| alcabrera | hmmm | 20:52 |
| kgriffs | so, you are creating messages in the future? | 20:52 |
| kgriffs | wow. how did you manage that? | 20:52 |
| *** jdaggett has quit IRC | 20:52 | |
| malini | wish I knew :D | 20:52 |
| *** jdaggett has joined #openstack-marconi | 20:52 | |
| kgriffs | if we could replicate it reliably, we could get a massive boost in performance. ;) | 20:52 |
| malini | I wud do better things in future, if I cud do tht ;) | 20:53 |
| alcabrera | lol | 20:53 |
| kgriffs | seriously tho | 20:54 |
| kgriffs | smells like rounding | 20:54 |
| malini | I am using the timeutils module to get 'now' | 20:54 |
| malini | eg. for error http://logs.openstack.org/95/50995/8/check/gate-marconi-python26/1f00f01/console.html | 20:54 |
| malini | Created time 2013-10-15 20:43:14, Now 2013-10-15 20:43:13.914117 | 20:55 |
| kgriffs | as a datetime, looks like | 20:55 |
| kgriffs | so, utcnow() | 20:55 |
| kgriffs | should just return datetime.datetime.utcnow() | 20:56 |
| *** jdaggett has quit IRC | 20:57 | |
| zyuan | kgriffs: ? | 20:57 |
| zyuan | (what does killer app mean? i installed Chrome just in order to use Google Presentation) | 20:57 |
| kgriffs | malini: I'm looking | 20:58 |
| malini | thanks kgriffs | 20:58 |
| *** jcru_ has joined #openstack-marconi | 21:04 | |
| kgriffs | malini: looks like isotime should just truncate the time, not round up | 21:05 |
| kgriffs | so, there goes that theory | 21:05 |
| kgriffs | it may be a clock resolution thing | 21:05 |
| zyuan | kgriffs: why? | 21:05 |
| kgriffs | why what? | 21:05 |
| zyuan | sometimes user expect different rounding | 21:05 |
| malini | kgriffs: I was wondering if we are dealing with multiple servers somewhere | 21:06 |
| zyuan | the problem we met is now() is round up | 21:06 |
| kgriffs | zyuan: I was just checking to see if timeutils was rounding up to the nearest second anywhere | 21:06 |
| zyuan | but what if user want a time of past? | 21:06 |
| malini | I do the post message before getting the 'now'..So the now shud be > created_time | 21:06 |
| *** jcru has quit IRC | 21:07 | |
| malini | like this one Created time 2013-10-15 20:43:14, Now 2013-10-15 20:43:13.914117 | 21:07 |
| zyuan | what i want to say is, either way may make sense | 21:07 |
| kgriffs | this would be with the SQLite driver? | 21:07 |
| malini | I cannot get a repro locally :( | 21:07 |
| malini | aah..I didnt try with SQLite | 21:07 |
| kgriffs | how does that driver calculate now? | 21:07 |
| malini | Let me try tht | 21:07 |
| kgriffs | maybe it is slightly different than the way python calculates now | 21:07 |
| alcabrera | it would be the sqlite driver - it's the only one that could run on Jenkins. | 21:07 |
| malini | good point | 21:08 |
| alcabrera | kgriffs: yeah, it uses JulianDate, or something like that. | 21:08 |
| *** jcru_ has quit IRC | 21:08 | |
| alcabrera | That's a little different thac utcnow(), afaik. | 21:08 |
| zyuan | yes | 21:08 |
| alcabrera | *than | 21:08 |
| *** jcru has joined #openstack-marconi | 21:09 | |
| kgriffs | ah. | 21:09 |
| zyuan | i did no rounding, just default, then truncate it and return | 21:09 |
| kgriffs | well, since you can't be sure how the driver is calculating, I guess you need a fudge factor in the test? | 21:09 |
| alcabrera | so... one approach is to use rounding to our advantage. For those tests, we say that a result passes if a floating point time when truncated yields a value that is >= 0. | 21:09 |
| zyuan | so the number after . may be accumulated | 21:09 |
| zyuan | but overall it should be correct | 21:09 |
| alcabrera | malini: could you implement the above strategy for the test? ^^ | 21:10 |
| alcabrera | delta = int(...) | 21:10 |
| malini | I cud..let me try tht | 21:11 |
| kgriffs | alcabrera: #3 on the etherpad can be done without waiting for flavio's patches, right? | 21:12 |
| kgriffs | I just want to do as much as we can without waiting on those | 21:12 |
| kgriffs | no. 3, as in "fill in the blanks for sharding.ctalog" | 21:13 |
| alcabrera | kgriffs: yup. I'm going to be working on that between tonight and tomorrow. | 21:13 |
| * ametts realizes that "between tonight and tomorrow" is the stroke of midnight. alcabrera is really fast. | 21:14 | |
| zyuan | OAO QAQ ToT | 21:15 |
| alcabrera | ametts: though, I did say "working". "completing" is another story. ;) | 21:15 |
| ametts | Good point. Maybe that's your sneaky way of saying "I'm not going to work on this at all." :) | 21:16 |
| alcabrera | Alternatively, I could borrow malini's technique and bring in work from the future. | 21:16 |
| alcabrera | We could be done right now. :D | 21:16 |
| alcabrera | lol | 21:16 |
| malini | alcabrera: I cud lend you my time machine ;) | 21:16 |
| kgriffs | heh | 21:16 |
| zyuan | the one which can only go back | 21:17 |
| malini | uh oh..we wud be stuck at proxy then.. | 21:17 |
| zyuan | (i guess you haven't watched Steins;Gate, hehe) | 21:18 |
| malini | anyways wrt my future bug, I cud repro it with sqlite & it went away when I enabled ntp..wonder if jenkins might not have ntp :-S | 21:19 |
| zyuan | omgsh ntp.... | 21:20 |
| russell_h | is it intended that GET /v1/queues returns 204 if there are no queues? | 21:24 |
| alcabrera | malini: it's possible. Good to hear that you were able to repro. | 21:24 |
| zyuan | russell_h: yes | 21:24 |
| russell_h | as opposed to an empty list response? | 21:24 |
| russell_h | that seems unfriendly to clients | 21:24 |
| zyuan | russell_h: oppose? | 21:24 |
| zyuan | russell_h: where? | 21:24 |
| russell_h | I mean, if I'm making a client, now I have to handle two totally different successful resposnes | 21:25 |
| zyuan | i mean, where | 21:25 |
| alcabrera | russell_h: yes, GET /v1/queues => 204 when there are no queues. | 21:25 |
| alcabrera | that's intentional. | 21:25 |
| russell_h | what's the rationale behind it? | 21:25 |
| zyuan | listing messages also give you 204 | 21:25 |
| zyuan | when there is no more message yet | 21:26 |
| alcabrera | in part, to be able to make decisions about how to proceed in processing a request without needing to investigate the body. kgriffs can share some more rationale. | 21:26 |
| alcabrera | *request -> response | 21:26 |
| russell_h | IMO, parsing the body is easier than not | 21:26 |
| zyuan | russell_h: you'd better to look at response code first | 21:27 |
| zyuan | and very carefully | 21:27 |
| zyuan | marconi is very very precise on using response code | 21:27 |
| russell_h | crap | 21:27 |
| zyuan | :) | 21:27 |
| russell_h | this is going to make building clients unnecessarily complicated | 21:27 |
| russell_h | now I need 3 branches to handle 2 cases | 21:27 |
| zyuan | i don't think a client can servive without handling 404 | 21:28 |
| alcabrera | why not use a dictionary lookup instead? | 21:28 |
| alcabrera | {200: process_body(), 204: continue(), 400: bad_request_handler(), ...}? | 21:28 |
| zyuan | so i don't think a client is worth to exist without understanding the design of an API, especially the API is totally RESTful-confirming | 21:28 |
| russell_h | sure... | 21:28 |
| russell_h | but why build an API if not to be friendly to clients? | 21:29 |
| russell_h | like if I wanted to write a lot of code I'd just queue things in mongo directly | 21:29 |
| zyuan | it's something hard to define, "client friendly" | 21:29 |
| russell_h | the whole point of an API is to provide friendly abstractions | 21:29 |
| alcabrera | russell_h: you have a point there, and it's something that's hard to nail. | 21:29 |
| zyuan | some APIs embeds error inside response body | 21:30 |
| zyuan | some APIs all return 200 | 21:30 |
| zyuan | some APIs returns something not JSON at all for security | 21:30 |
| zyuan | which is "client-friendly"? | 21:30 |
| alcabrera | I am starting to like the idea of a default representation for things like listing queues and messages - that allows the unification of the handling on the client-side, except in error scenarios. | 21:30 |
| alcabrera | we've frozen the marconi v1.0 API, but keep bringing the feedback. Things will likely change to make life easier for everyone come v1.1 (maybe?) and definitely by v2.0. | 21:32 |
| kgriffs | just catching up here | 21:32 |
| kgriffs | russell_h: we of course want to be friendly to clients | 21:33 |
| russell_h | yeah, I might have been… hyperbolous | 21:33 |
| *** oz_akan_ has quit IRC | 21:33 | |
| russell_h | if thats a word | 21:33 |
| kgriffs | the 204 thing comes up a lot and we are looking into it | 21:33 |
| russell_h | props for making marconi easy to run though | 21:36 |
| russell_h | took like 3 minutes to get it going | 21:36 |
| alcabrera | sweet | 21:36 |
| russell_h | (of which 2 was spent screwing with virtualenv) | 21:36 |
| kgriffs | I think switching over to returning an empty array is probably going to be OK. 204 may save a few bytes on the wire, but it is starting to sound more trouble than it's worth. The abstract "correctness" is debatable - arguments on both sides. Which tells me it doesn't really matter. Just pick the one that is easiest to code against. | 21:37 |
| alcabrera | kgriffs: agreed | 21:37 |
| kgriffs | anyway, that is my thinking at the moment | 21:37 |
| russell_h | fwiw, I agree | 21:38 |
| russell_h | I'm about to sound like a ruby dev, but a few bytes one way or another don't matter much to me | 21:38 |
| russell_h | if they did I probably wouldn't use http at all | 21:38 |
| kgriffs | btw, this change would have to wait for a v2 api i guess unless the community wants to OK media type versionioning | 21:39 |
| kgriffs | lol | 21:39 |
| kgriffs | yeah | 21:39 |
| kgriffs | that being said I always anticipated v2 coming close on the heals of v1 because we were bound to screw things up the first time around. :p | 21:39 |
| kgriffs | by close, I don't mean "1-2 months" | 21:40 |
| kgriffs | but more like 6-12 months | 21:40 |
| alcabrera | onward, towards marconi v27! | 21:40 |
| zyuan | alcabrera: i'm drinking Mist | 21:40 |
| kgriffs | In spite of our best efforts to vet with the community, some things still end up broken. | 21:40 |
| kgriffs | (you can't know everything in advance) | 21:41 |
| kgriffs | (or keep track of everything all the time) | 21:41 |
| russell_h | true story | 21:41 |
| *** kgriffs is now known as kgriffs_afk | 21:41 | |
| zyuan | i against changing 204 to 200, because client implementers will find that they have to handle [] case then | 21:42 |
| zyuan | the turth is, we currently give "non-empty" guarentee | 21:42 |
| russell_h | is [] a special case though? | 21:42 |
| zyuan | which means, the document you get always have the 'next' link | 21:42 |
| *** kgriffs_afk is now known as kgriffs | 21:43 | |
| zyuan | then here comes the question: do you expect the empty response also has "next" link? | 21:43 |
| russell_h | like what would you do with an empty list that you wouldn't do with a non-empty list? | 21:43 |
| zyuan | when you got an 204 or empty list, client are suppose to try later | 21:43 |
| zyuan | because listing is a continuous operation | 21:43 |
| * kgriffs made it through the temporal distortion field | 21:43 | |
| alcabrera | zyuan: I'd expect it to have the same structure. {'links': {}, 'messages': []} | 21:43 |
| * kgriffs is back | 21:44 | |
| * alcabrera notices how fast kgriffs is | 21:44 | |
| zyuan | alcabrera: yes, then it's not just a few bytes, it's "some" bytes | 21:44 |
| zyuan | and "some" is duplicated for each, say second | 21:44 |
| russell_h | but if I get a non-empty list, I'm almost certainly just going to iterate the list, then poll again | 21:44 |
| zyuan | what i'm saying, we have to handle some special case, unless we send "some" bytes | 21:45 |
| zyuan | and i don't think it worth | 21:45 |
| zyuan | and 204 is the cheapest choise to say "client, retry!" | 21:45 |
| russell_h | thats fair | 21:46 |
| russell_h | I don't think the tradeoff is worth it, but I can live either way | 21:46 |
| russell_h | even if the empty response were 512 bytes every second (it should be like 5% of that) we're takling… 40MB/day? | 21:47 |
| *** tedross has quit IRC | 21:47 | |
| alcabrera | hmmm... | 21:48 |
| alcabrera | numbers are fun to think about. | 21:48 |
| alcabrera | it's all about the target scale, at that point | 21:48 |
| alcabrera | requests per second | 21:48 |
| zyuan | it also uses server resource, like CPU, like power... | 21:48 |
| zyuan | ok, i don't know | 21:48 |
| russell_h | heh :) | 21:48 |
| alcabrera | I'm in agreement about the trade off - it's a matter of efficiency vs. productivity. It is easier to handle a uniform representation. :) | 21:49 |
| alcabrera | So assuming that somebody is serving a million requests per second using a marconi deployment, then the wasted bandwidth per day amounts to about ~2TB, also assuming that all of those requests are for listing queues. | 21:51 |
| alcabrera | It's all about trade-offs. :P | 21:51 |
| alcabrera | haha | 21:51 |
| alcabrera | +1 for *not* using HTTP in such a scenario | 21:51 |
| kgriffs | russell_h: let us know what other ways we can improve the API. | 21:51 |
| * kgriffs loves feedback | 21:52 | |
| kgriffs | (as you come across them) | 21:52 |
| russell_h | will do | 21:52 |
| kgriffs | rock on | 21:52 |
| alcabrera | awesome! | 21:52 |
| alcabrera | russell_h: the README needs help, so I'm really happy to hear you got marconi up and running despite that. :D | 21:52 |
| kgriffs | I know flaper87 is also very open to ideas | 21:53 |
| kgriffs | he is based in Italy, tho, so you'll have to catch him earlier in the day. :p | 21:53 |
| kgriffs | aaaanyway | 21:54 |
| * kgriffs goes back to reviewing patches | 21:54 | |
| * alcabrera returns to porting the catalogue | 21:54 | |
| zyuan | russell_h: flaper87|afk is developing python client; feel free to ping him | 21:57 |
| russell_h | nice, will do | 21:57 |
| alcabrera | kgriffs, zyuan: Yeah, there's going to be some porting effort. The proxy catalogue implemented entries in terms of {project, queue, partition_name, extraneous_fields, ...}. I need to update those assumptions to remove the extraneous fields and maintain a mapping from project + queue => shard identifier. | 21:58 |
| kgriffs | ok | 21:59 |
| kgriffs | you might want to use that combining function so you end up with a single field, rather than two fields, "project" and "id" | 21:59 |
| alcabrera | +1 | 21:59 |
| zyuan | we already have the function | 21:59 |
| alcabrera | get_scoped_something_or_other, iirc | 22:00 |
| kgriffs | I guess then scope_queue_name would need to move up a level to storage.utils | 22:00 |
| kgriffs | also | 22:00 |
| kgriffs | descope_queue_name | 22:00 |
| alcabrera | I'll address those two in the porting patch. | 22:00 |
| alcabrera | before I head out for the night, how do you feel about the proposed merging strategy, kgriffs? I'll bring it for discussion with flaper87|afk tomorrow morning to get some more input, too. | 22:01 |
| alcabrera | **bring it up for... | 22:02 |
| kgriffs | sorry, did that get updated? Seems like we identified some places where we didn't have to wait to merge flaper87 | 22:02 |
| kgriffs | 's pipeline patches | 22:02 |
| alcabrera | lemme double check | 22:02 |
| alcabrera | I think I updated it. | 22:02 |
| alcabrera | The merging strategy is pretty midterm, though. | 22:02 |
| alcabrera | I'm thinking out to | 22:02 |
| kgriffs | I actually gotta run | 22:03 |
| alcabrera | "things I'd like to see done by Monday" | 22:03 |
| kgriffs | we can sync up on things in the morning | 22:03 |
| alcabrera | cool, cool | 22:03 |
| kgriffs | I should be online a lot earlier than today | 22:03 |
| alcabrera | me, too. | 22:03 |
| kgriffs | k | 22:03 |
| alcabrera | o/ | 22:03 |
| kgriffs | cheers! | 22:03 |
| *** alcabrera has quit IRC | 22:05 | |
| *** kgriffs is now known as kgriffs_afk | 22:06 | |
| *** amitgandhi has quit IRC | 22:28 | |
| *** ayoung has quit IRC | 22:29 | |
| *** oz_akan_ has joined #openstack-marconi | 22:45 | |
| *** oz_akan_ has quit IRC | 22:49 | |
| *** fifieldt has joined #openstack-marconi | 22:56 | |
| *** jcru has quit IRC | 23:14 | |
| *** amitgandhi has joined #openstack-marconi | 23:34 | |
| *** amitgandhi has quit IRC | 23:41 | |
| *** amitgandhi has joined #openstack-marconi | 23:41 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!