Thursday, 2013-10-10

*** alcabrera has joined #openstack-marconi00:34
openstackgerritKurt Griffiths proposed a change to openstack/marconi: fix(mongo): Negative age reported in stats  https://review.openstack.org/5079000:35
alcabrerakgriffs: o/00:38
alcabreraI make it in just in time to catch the bug fix.00:38
kgriffsheh00:39
kgriffsindeed00:39
alcabrerakgriffs: +2'd - this mistake happened before, same fix strategy as last time. Flip the things!00:39
kgriffsyep. I thought we fixed all the places, but I guess one got missed00:39
alcabrerayup, the stats endpoint. Ah well, nice catch. Always good to see a bug getting closed.00:40
alcabrerakgriffs: any developments while I was away?00:41
*** amitgandhi has joined #openstack-marconi00:44
*** fifieldt has quit IRC00:52
*** nosnos has joined #openstack-marconi00:59
*** amitgandhi has quit IRC01:26
kgriffsalcabrera: Not much to speak of. I ended up having to do email and stuff and didn't get to updating the sharding patch. I will have to try to get that done later tonight or first thing in the morning.01:28
alcabrerakgriffs: ah, alright. Email does tend to pile up, even within a period of a few hours. :P01:43
alcabreraI'm working on the mongodb storage driver for the shards patch, so there's still time. I'm curious to see how the foundational code evolves. :)01:46
*** kgriffs is now known as kgriffs_afk01:50
*** reed has quit IRC01:58
*** fifieldt has joined #openstack-marconi02:04
*** alcabrera has quit IRC03:24
*** kgriffs_afk is now known as kgriffs03:25
openstackgerritAlejandro Cabrera proposed a change to openstack/marconi: feat: shards mongodb driver + tests  https://review.openstack.org/5081503:42
*** kgriffs is now known as kgriffs_afk04:00
*** flaper87|afk is now known as flaper8707:16
*** reed has joined #openstack-marconi07:26
openstackgerritA change was merged to openstack/marconi: fix(mongo): Negative age reported in stats  https://review.openstack.org/5079007:34
*** dafter has joined #openstack-marconi07:37
flaper87yo yo! Sup' world?07:45
*** ykaplan has joined #openstack-marconi07:53
*** yassine has joined #openstack-marconi07:57
*** dafter has quit IRC08:09
*** jarret has quit IRC08:26
*** nosnos has quit IRC09:06
*** ykaplan has quit IRC09:15
*** reed has quit IRC09:15
*** dafter has joined #openstack-marconi09:20
*** dafter has quit IRC09:25
*** dafter has joined #openstack-marconi10:05
*** alcabrera has joined #openstack-marconi10:13
alcabreraflaper87: o/10:14
alcabreraMorning. :)10:14
flaper87alcabrera: HELLOOOO!!!10:14
alcabreraw00t10:14
flaper87alcabrera: Man, you woke up early today10:14
flaper87:D10:14
alcabreralol, yup. Yesterday, too. In order to make this storage-0sharding thing work asap, I've been doing ~10 hours a day, and getting up this early works best for my wife. :)10:15
alcabrera(first one in the office, yeah!)10:15
flaper87alcabrera: I call that a Success Story10:15
flaper87:D10:15
alcabrerahahaha10:16
alcabreraflaper87: so we're going to continue with the openstack review process, and I agree with Kurt - we'll miss out on some really cool ideas and some subtle details if we don't all pull together. :D10:17
alcabreraThe storage pipeline idea is one of those awesome ideas that is already changing what we might have done apart.10:17
flaper87alcabrera: I agree with that as well10:18
flaper87alcabrera: and guess what?10:18
flaper87alcabrera: (I mean, I agree with continue with the review process)10:18
alcabreraflaper87: I'm guessing - the POC is done. ;o10:19
openstackgerritFlavio Percoco proposed a change to openstack/marconi: Implement common pipeline class  https://review.openstack.org/5088910:22
openstackgerritFlavio Percoco proposed a change to openstack/marconi: Use the pipeline for the storage layer  https://review.openstack.org/5089010:22
flaper87alcabrera: not just the POC ^^^^^^10:22
flaper87alcabrera: dude, first time you get here before I've had lunch!!!!10:22
flaper87:D10:22
flaper87Take a look there and drop comments. There may be some docstrings missing as well10:23
flaper87but that's the genera;, working, idea10:23
flaper87general*10:23
alcabreraflaper87: oooohhh10:24
alcabreraYup, I'm diving right in!10:24
alcabreraI addressed the last question you had on the multi-update proxy patch. Turns out I wasn't looking closely ennough. :P10:24
alcabrera*enough10:24
alcabreraMind if I address that particular refactoring in a later patch (fields = {...})? I've got it generalized in the last patch I submitted (shards: mongo storage driver)10:25
flaper87alcabrera: Sounds like a good plan! Thanks for taking care of that!10:26
flaper87alcabrera: I seriously doubted of what I was seeing10:26
alcabreranp. Generalizing is fun. :D10:26
flaper87I was, "mmh, they look very similar"10:26
alcabreralol10:26
flaper87:P10:27
flaper87alcabrera: +2'd10:27
alcabreraflaper87: woot!10:27
flaper87no no no, I don't like py26 functional tests not being deterministic!!! I've been in that hell before!!!10:28
flaper87I hate those random failures10:28
flaper87alcabrera: anyway, Jenkins is happy with the pipeline changes, let me know!10:29
flaper87I'll have lunch now!10:29
flaper87bbib10:29
alcabrerakk, enjoy lunch!10:29
alcabreraflaper87: James. Bond. :P10:34
*** ykaplan has joined #openstack-marconi10:45
*** dafter has quit IRC11:25
flaper87alcabrera: :P11:47
* flaper87 is back11:47
openstackgerritAlejandro Cabrera proposed a change to openstack/marconi: feat: split queues API into public/admin  https://review.openstack.org/5070111:48
flaper87alcabrera: not sure I understand your comment here: https://review.openstack.org/#/c/50890/1/marconi/queues/storage/pipeline.py11:50
flaper87the first one11:50
flaper87I mean, pipeline is already implemented in common/pipeline.py11:50
flaper87ahhhhh11:50
flaper87you mean that storage_pipeline can be used for the proxy as well11:51
flaper87alcabrera: Did I understand correctly?11:51
alcabreraflaper87: yup. :D11:52
alcabreraThe storage_pipeline idea is pretty generic, and plays nice with any storage layers we use in the future.11:52
flaper87alcabrera: yeah, totally agree. I'll rename the function to get_storage_pipeline, though11:53
flaper87so we can have get_transport_pipeline in case we need it11:53
*** tedross has joined #openstack-marconi11:54
openstackgerritAlejandro Cabrera proposed a change to openstack/marconi: feat: add shard management resource  https://review.openstack.org/5070211:54
alcabreraflaper87: good idea.11:54
flaper87alcabrera: what are your impressions so far?11:55
alcabrerahmmm...11:55
alcabreraI like the idea. I'll have stronger impressions once I can see what a configuration file entry will look like. :)11:55
alcabreraThat's the part where I'm drawing a blank.11:56
alcabreraBecause the notion of a list of functions/objects as a pipeline makes perfect sense.11:56
alcabrera(I think I broke the dependency chain in the two patches above, hmmm...)11:56
* alcabrera experiences the dangers of rebasing from time to time11:56
flaper87alcabrera: I'm writing the docstring right now11:57
openstackgerritAlejandro Cabrera proposed a change to openstack/marconi: feat: storage sharding schema for marconi-queues  https://review.openstack.org/5045611:59
openstackgerritAlejandro Cabrera proposed a change to openstack/marconi: feat: split queues API into public/admin  https://review.openstack.org/5070111:59
alcabreraflaper87: awesome@11:59
alcabrera!!11:59
openstackalcabrera: Error: "!" is not a valid command.11:59
openstackgerritAlejandro Cabrera proposed a change to openstack/marconi: feat: add shard management resource  https://review.openstack.org/5070212:00
flaper87alcabrera: btw, I was also thinking! Since we now have separate files for marconi-* services, I don't think it makes sense to have the config params prefixed with *:something12:02
flaper87I feel like we already discussed this, didn't we?12:03
flaper87anyway, I'm thinking about removing (queues:|proxy:) from all our configu groups12:03
flaper87instead of having queues:storage:blah we'd have storage:blah12:03
flaper87which makes having get_storage_pipeline as a common function easier12:04
alcabreraflaper87: +112:04
alcabreralemme find the bug report... :)12:04
alcabreraflaper87: https://bugs.launchpad.net/marconi/+bug/123166912:04
alcabreraYeah, we discussed it before.12:04
flaper87ahhh, we did discussed this then12:05
flaper87:D12:05
alcabreraOr, at least, I remember putting together this bug report as a result. :P12:05
flaper87cool beans12:05
openstackgerritAlejandro Cabrera proposed a change to openstack/marconi: feat: shards storage controller interface  https://review.openstack.org/5072112:05
*** dafter has joined #openstack-marconi12:05
*** dafter has quit IRC12:05
*** dafter has joined #openstack-marconi12:05
openstackgerritAlejandro Cabrera proposed a change to openstack/marconi: feat: shards mongodb driver + tests  https://review.openstack.org/5081512:08
alcabreraflaper87: woot! Rebasing complete after I reviewed my own patches and found little errors. ;)12:09
alcabreraI wonder how helpful it is as a general practice in a review pipeline to have the author of a patch review their patches the day after they submit? At the very least, it might ease review by others a little down the line as the author creates a tour of a particular patch, but there's the possibility of introducing bias into the review if it isn't done carefully (maybe?).12:10
flaper87alcabrera: AWESOME, I'll submit this with the docstring, review your stuff and then heads down for the rest of the day working on my PyconIE presentation12:10
flaper87gotta finish it12:11
alcabreraflaper87: oh yeah! Good luck with that! It's (PyconIE) is going to be awesome, and I hope the videos of the talks are released soon after. :D12:11
*** dafter has quit IRC12:29
openstackgerritFlavio Percoco proposed a change to openstack/marconi: Implement common pipeline class  https://review.openstack.org/5088912:30
openstackgerritFlavio Percoco proposed a change to openstack/marconi: Use the pipeline for the storage layer  https://review.openstack.org/5089012:30
flaper87alcabrera: docstring ^12:30
alcabreraflaper87: +2 for the first patch, and now... :)12:32
alcabreraflaper87: oooohh, this docstring is awesome.12:33
* alcabrera keeps reading12:33
flaper87alcabrera: glad you like it. I'm sure it's full of grammatical errors :D12:34
alcabreraflaper87: there's a few. :P I'll point them out in review comments! This patch is ready for a +2 from me after the docstring is cleaned up, but yeah...! I really like this idea.12:35
flaper87alcabrera: cooooooool, thanks! :)12:40
alcabreraflaper87: all set. The suggestions have been submitted. :D12:41
flaper87alcabrera: awesome, let me address those so it's ready to be reviewed by kgriffs_afk when he's around12:42
alcabreracool, cool. :)12:43
openstackgerritFlavio Percoco proposed a change to openstack/marconi: Use the pipeline for the storage layer  https://review.openstack.org/5089012:47
alcabreraflaper87: +2'd :D12:52
flaper87alcabrera: thank you, Sir! :D12:52
alcabreraI've cleaned up my last 5 storage-sharding patches, so time allowing, they're ready for you. :)12:53
flaper87alcabrera: looking12:54
flaper87alcabrera: not sure this patch should depend on the schema one: https://review.openstack.org/#/c/50701/412:56
flaper87alcabrera: can I ask what your working copy looks like?12:56
flaper87I mean, are you working on a single branch ?12:56
alcabreraflaper87: Hmm...12:58
alcabreraHow to put it...12:58
flaper87:P12:58
flaper87try me!12:58
alcabreraI suppose I made them dependent because I expected to use it very soon, heh. :)12:58
alcabreraWhich I do - in the patch right after, where I implement the shards_resource.12:58
alcabreraBut I see your point. ;)12:58
flaper87hahaha12:59
flaper87I mean, I'm ok with that12:59
flaper87what always worries me about these dependencies is that -1 one will block the whole queue12:59
alcabreraflaper87: agreed. It's happened to me before.12:59
alcabreraHeh. :P12:59
flaper87and as I mentioned yday, I'm not sure the transport is the right place for this schema to live in12:59
alcabrerahmmm13:00
flaper87or is it?13:00
alcabreraIt's a good thought to explore.13:00
alcabreraWhere else could it live?13:00
flaper87Isn't this schema related to the sharding thing ?13:00
alcabreraYup. I considered doing marconi/*/transport/schemas/shards.py13:01
alcabreraSince I'm 95% certain that the schemas will be applied at the transport layer.13:01
flaper87I was thinking about marconi/queues/storage/shards/13:01
flaper87and there we could put the schemas used for sharding, the shard manager and all sharding related code13:02
flaper87or marconi/queues/shard13:02
flaper87so we can keep it separte from the storage13:02
flaper87and still be used by it13:02
alcabrerahmmm13:02
flaper87or actually, by both, if needed.13:02
flaper87Although, I'd prefer to keep the transport completely storage / shard agnostic13:03
alcabreraif they might be used by both, the place to put them would be marconi/common/schemas/*13:03
alcabrerasomething like that.13:03
flaper87by both I meant transport and storage13:03
alcabreraAnd +1 for keeping storage shard-agnostic. That's what the pipeline is for. :)13:03
flaper87not exactly queues and proxy13:03
*** oz_akan_ has joined #openstack-marconi13:03
alcabreraerr, transport.13:03
alcabreraflaper87: I thought that's what you meant - storage vs. transport.13:04
flaper87TBH, if the pipeline idea makes sense and we all like how those 2 patches work, I'd love to see the shard code using the pipeline right away13:05
alcabreraI like the approach of putting them in marconi/commmon/schemas/* because then I can do something like 'marconi.common.schemas.shard.create'. :)13:05
flaper87that will isolate changes in a single package and feature13:05
alcabreraflaper87: +1. kgriffs_afk just arrived in the office, too, so we'll be able to make progress on getting his shard storage foundation idea based off of pipelines, or something like that.13:06
alcabrerathoughts for moving schema.py -> marconi/common/schemas/shards.py?13:06
flaper87alcabrera: I think I'm missing something about this schema but yeah, marconi/common/schemas/* makes sense to me as well13:06
*** mpanetta has joined #openstack-marconi13:06
flaper87alcabrera: d000d, I'm slow...13:06
flaper87:D13:06
alcabreralol13:07
alcabrerasweet13:07
alcabreraalready, I'll make that move shortly.13:07
flaper87awesome, lemme know and I'll +2 that patch13:07
*** mpanetta has quit IRC13:08
*** mpanetta has joined #openstack-marconi13:09
*** kgriffs_afk is now known as kgriffs13:13
*** amitgandhi has joined #openstack-marconi13:13
flaper87kgriffs: morning13:14
kgriffso/13:14
kgriffsI was just reading the log13:14
*** mpanetta has quit IRC13:14
*** mpanetta has joined #openstack-marconi13:14
kgriffsso, I think the pipeline thing makes sense, but I would like to get the shard patch in and then circle back when the pipeline is ready13:15
kgriffsflaper87: have you already started working on the pipeline?13:15
flaper87kgriffs: https://review.openstack.org/#/q/status:open+project:openstack/marconi+branch:master+topic:storage-pipeline,n,z13:15
flaper87kgriffs: yup :)13:15
flaper87it already works13:15
flaper87and as a POC, I migrated both storage backends to it13:16
alcabreraflaper87 has done an awesome job rolling this out so quickly.13:16
alcabreraI'm almost done updating my schema patch with the suggestions given.13:17
kgriffseeeeeenteresting13:17
openstackgerritAlejandro Cabrera proposed a change to openstack/marconi: feat: storage sharding schema for marconi-queues  https://review.openstack.org/5045613:18
alcabreraflaper87: ready. ^^13:18
alcabreraI'll add the deps on in a moment. :P13:18
openstackgerritAlejandro Cabrera proposed a change to openstack/marconi: feat: split queues API into public/admin  https://review.openstack.org/5070113:19
alcabreraflaper87: dependency on schema removed for this patch. ^^13:20
*** fifieldt has quit IRC13:20
*** ykaplan has quit IRC13:21
kgriffsflaper87: so, I was just looking over those pipeline patches13:22
flaper87alcabrera: COOL, I'll take a look!13:23
flaper87kgriffs: shoot13:23
kgriffsThere doesn't seem any way to influence the flow of the pipe13:23
flaper87If you return something != None, the pipe will stop and return13:24
kgriffsthe shard manager needs to be able to pick one of N storage controllers to pass control to13:24
kgriffsbasically...13:25
flaper87kgriffs: yup, the shard manager can have an instance of each controller and then pick the storage, call the final method from that storage and return13:25
flaper87kgriffs: that stops the pipeline13:25
kgriffsi guess I was thinking it would be nice if the pipeline handled that13:26
openstackgerritAlejandro Cabrera proposed a change to openstack/marconi: feat: storage sharding schema for marconi-queues  https://review.openstack.org/5045613:27
flaper87Yeah, I had that in mind as well, thing is that chaining returned values into the next call requires to change the method signature13:27
*** oz_akan_ has quit IRC13:27
flaper87the other thing is that the pipeline need to be agnostic of what the pipes return and do13:28
*** oz_akan_ has joined #openstack-marconi13:28
flaper87unless we subclass it and make it storage specific13:28
kgriffsfor reference: http://paste.openstack.org/show/48210/13:29
* kgriffs thinking13:30
flaper87Yeah, exactly13:30
* kgriffs smells smoke13:30
mpanettakgriffs: You should be careful wit that13:30
flaper87I thought about implementing that by checking if the returned value was an instance of Pipe13:30
*** malini_afk is now known as malini13:30
alcabreramalini, mpanetta: o/13:31
flaper87if ShardManager returns a Pipeline instance, then the pipeline will consume that instead of going forward with the current pipeline13:31
flaper87mpanetta: LOL13:31
flaper87mpanetta: morning13:31
mpanettaMorning people :)13:31
flaper87that's the cleanest way to implement "Pipeline hijacking"13:32
kgriffsIt would be nice if the individual controllers didn't have to know/care about the pipe13:32
flaper87kgriffs: exactly13:32
flaper87with the current implementation they don't13:32
kgriffsreferring to the ascii art...13:32
flaper87you just have to implement a __getattr__ on the ShardManager13:32
flaper87make that method call the lookup13:32
flaper87get the controller instance13:32
flaper87and getattr(controller, method)(*args, **kwargs)13:33
flaper87the returned value will stop the pipeline execution13:33
*** ykaplan has joined #openstack-marconi13:33
flaper87and the controller instance (which is at the end of the pipeline) won't be called13:33
kgriffsyep.13:33
alcabreraa little bit of magic for hijacking things.13:34
flaper87TBH, I'm a bit worried about depending on return values != None13:34
flaper87but couldn't think of a better way to do it13:35
alcabreraflaper87: I feel like it should be the other way around.13:35
alcabreraPossibly using an enum13:35
* flaper87 notices that alcabrera is an Enum fan13:35
flaper87:D13:35
alcabrerapipeline.Continue, pipeline.Exit, etc.13:35
alcabreraMake a decision based off of that ^13:35
flaper87alcabrera: yeah, but that requires making the pipes "pipeline-aware"13:35
flaper87which they are not13:36
alcabrerahmmm... true.13:36
flaper87or shouldn't be13:36
alcabreragood point13:36
* flaper87 once implemented a workflow engine that was also BPMN20 compliant13:37
*** dafter has joined #openstack-marconi13:37
alcabreraflaper87: I remember reading about pipelines somewhere else over the past 3 days. I realize now that this might be relevant to what we're trying to solve, though I'm not sure how: http://hackage.haskell.org/package/pipes-4.0.0/docs/Pipes-Core.html13:38
alcabrera* not sure how (yet)13:38
alcabreraooohhh, I think I found something.13:40
alcabreraThe idea behind this library decouples the pipe-connection mechanism from the transformer in the pipe.13:40
openstackgerritAlejandro Cabrera proposed a change to openstack/marconi: feat: split queues API into public/admin  https://review.openstack.org/5070113:43
openstackgerritAlejandro Cabrera proposed a change to openstack/marconi: feat: add shard management resource  https://review.openstack.org/5070213:43
alcabrerahmmm... trying to figure out multiple dependencies. :/13:44
alcabreraAlmost got it.13:44
alcabrerabah, it doesn't seem like gerrit supports multiple deps. :/13:47
alcabreraflaper87: ^^13:47
*** jraim_ has joined #openstack-marconi13:47
flaper87alcabrera: mmh, it does13:48
flaper87:/13:48
flaper87wait, what do you mean exactly?13:48
flaper87kgriffs: so, thoughts?13:49
kgriffsI was just thinking, we could define the notion of a pipeline meta-stage that would return a callable to the pipeline that the pipeline would then actually call, but that seems like cleverness for the sake of being clever. Might as well just have the pipeline stage call the method itself and appear externally to behave like any other stage.13:49
kgriffsanyway13:50
kgriffsa couple things13:50
alcabreraI want the resource patch to depend on both the 'split API' patch and the 'schema' patch without the 'split API' patch depending on the 'schema' patch. :x13:50
kgriffs1. Seems like __getattr__ is slow-ish, but I can't remember if it is enough to worry about13:50
flaper871. Not sure how slow it is, TBH. But that's the way to put the pipeline in the middle without making the transport pipline-aware13:52
kgriffs2. I was actually already planning to make a psuedo "ShardDriver" that just picks the right storage instance behind the scenes and fowards the call to it. That work seems orthogonal to the pipeline work13:52
flaper872. Awesome!!!13:52
kgriffsflaper87: Well, you could make pipeline "controller" classes that implement the various controller interfaces and then just forward things on.13:54
kgriffsOR13:54
kgriffswith some python magic13:54
kgriffsyou could...13:54
alcabrerakgriffs: getattr() seems to add about 80ns based on a few local experiments. (Python 2.7.5 and 3.3.2)13:55
flaper87kgriffs: yeah, I meant, without changing current code13:55
kgriffsdynamically generate those proxy controllers13:55
flaper87kgriffs: well, we could make it call getattr just once13:55
flaper87if at least one of the pipes implement it, we can set it as an attribute of the current pipeline instance13:56
flaper87and that will reduce the cost of __getattr__ if that method is called again13:56
kgriffsah, since the next time python will find a method on Pipeline with the name, so won't have to fall back to __getattr__13:57
flaper87we could also cache the pipes that actually implement those methods and reduce the pipeline for each method13:57
flaper87kgriffs: yup13:57
flaper87so for method "create" just pipe 1 and 3 implement it, next time getattr won't be called on pipe 213:58
kgriffs+113:58
flaper87I'll implement those optimizations in a separate patch!13:58
flaper87if you guys like current implementation, lets get it merge13:58
flaper87so we can base the shard work on that13:58
flaper87(if we all agree)13:59
alcabreraI'm in favor.13:59
kgriffsFWIW, I just did a quick benchmark14:00
* flaper87 hides14:00
alcabrerawoot, data!14:00
kgriffslet me paste that14:00
openstackgerritAlejandro Cabrera proposed a change to openstack/marconi: feat: add shard management resource  https://review.openstack.org/5070214:00
kgriffshttp://paste.openstack.org/show/48211/14:01
openstackgerritAlejandro Cabrera proposed a change to openstack/marconi: feat: shards storage controller interface  https://review.openstack.org/5072114:01
kgriffsI think that perf difference will actually be noticeable when serving lots of concurrent requests, so I'm in favor of the lazy caching idea.14:03
alcabrera__getattr__ gets pretty expensive. :P14:03
alcabreraI was comparing getattr(obj, 'name') to obj.name in my local experiments.14:03
zyuanmight appears to be more expensive in pypy14:03
flaper87kgriffs: yeah, agree, I think setting those methods as attributes of the pipeline instace might make sense14:03
zyuanbecause pypy focuses on compile time optimization14:04
openstackgerritAlejandro Cabrera proposed a change to openstack/marconi: feat: shards storage controller interface  https://review.openstack.org/5072114:04
flaper87we could actually override __getattribute__ which is *ALWAYS* called14:05
flaper87and memoize that method instead14:05
* flaper87 wonders what the other 241 lines of kgriffs's iPython history have14:06
alcabreraflaper87: the secrets of the universe. :P14:06
flaper87LOL14:06
kgriffsok14:06
kgriffsso I just timed this14:07
kgriffshttp://paste.openstack.org/show/48213/14:07
openstackgerritAlejandro Cabrera proposed a change to openstack/marconi: feat: shards mongodb driver + tests  https://review.openstack.org/5081514:07
kgriffsand it was super fast again14:07
kgriffsso, I think that optimization is the way to go14:07
flaper87kgriffs: cool!14:07
flaper87so, I'll add that in the next patch14:07
kgriffskk14:07
flaper87if you guys agree14:07
flaper87:D14:07
kgriffssounds like a plan14:07
flaper87kgriffs: awesome!14:07
alcabreraflaper87: go for it! :D14:08
flaper87kgriffs: btw, if you get a chance today, could you take a look at client's patches ?14:08
alcabreraI just finished my rebasing stream, woot.14:08
flaper87alcabrera: ^ (I think there's still 1 waiting for you)14:08
alcabreraflaper87: yup, queue's API methods. :)14:08
alcabreraI thought I had them all, heh.14:08
flaper87alcabrera: hehe, that one may need some iterations but it shows how the whole thing works, FINALLY14:09
kgriffsflaper87: sure, but things are pretty crazy today here, so it may not be until later today or tomorrow morning. I'll get to them ASAP.14:09
flaper87kgriffs: no worries, that works as well. I don't think I'll be able to do much on that tomorrow anyway14:09
flaper87but a general thought / comment about the API would be nice14:09
flaper87if there's something to fix, I can tackle it down on Monday's morning14:10
alcabreraflaper87: I'm happy to see the system coming together! I'll give the last part review when I find some time. :)14:10
flaper87anyway, way to go guys!! This whole Shard + Pipeline thing looks great14:10
alcabreraflaper87: could I get your +2 on schemas?14:10
flaper87alcabrera: sure you can! Thanks14:10
alcabrerawoot14:11
alcabreraI feel like today, I finally understood the secrets of the gerrit-rebase cycle.14:12
flaper87alcabrera: it's about nutella, isn't it?14:12
alcabreraWith long dependency chains, one just has to *really* know what they're doing between 'git review' and... and, yes, nutella. :)14:12
* alcabrera gives flaper87 gerritella14:12
flaper87LOOL14:13
kgriffsflaper87: can you add a test for calling a method that has *both* args and kwargs?14:16
kgriffshttps://review.openstack.org/#/c/50889/2/tests/unit/common/test_pipeline.py14:16
openstackgerritMalini Kamalambal proposed a change to openstack/marconi: Tests for Client-ID validation  https://review.openstack.org/5092814:16
flaper87kgriffs: mmh, damn, I thought I did! Yup, lemme do that14:17
openstackgerritMalini Kamalambal proposed a change to openstack/marconi: Tests for Client-ID validation  https://review.openstack.org/5092814:18
openstackgerritFlavio Percoco proposed a change to openstack/marconi: Implement common pipeline class  https://review.openstack.org/5088914:20
openstackgerritFlavio Percoco proposed a change to openstack/marconi: Use the pipeline for the storage layer  https://review.openstack.org/5089014:20
flaper87kgriffs: alcabrera ther you go ^14:20
alcabreraflaper87: +2 on  the first one, and...14:24
alcabreraI was already +2 on the second one. :P14:24
alcabrerabrb14:25
flaper87alcabrera: thanks14:25
openstackgerritMalini Kamalambal proposed a change to openstack/marconi: Tests for Client-ID validation  https://review.openstack.org/5092814:27
alcabreramalini: fast! +2. :)14:28
malinithanks alcabrera!!14:28
kgriffscome on jenkins...14:29
flaper87kgriffs: :P14:29
flaper87kgriffs: both patches got +114:30
flaper87from jenkins14:30
kgriffsrock on14:31
alcabrerawoot14:31
* alcabrera is looking forward to merges14:31
kgriffsflaper87: https://review.openstack.org/#/c/50890/4/marconi/common/pipeline.py14:35
kgriffswould there be any benefit to using stevedore to load the pipes?14:35
kgriffsBTW, the words "pipe" and "pipeline" are easily confused14:36
kgriffswe may want to talk about "stages" or something like that to be more clear14:36
kgriffs"filters" is a common term as well14:36
flaper87kgriffs: I'll rename those to stages in another patch, if that sounds good!14:37
kgriffssure14:37
flaper87kgriffs: I thought about using stevedore there14:37
flaper87but I then thought it was a bit overengineered for this task14:37
flaper87I may be wrong, though14:38
kgriffsi was just thinking that for get_storage_pipeline it would make sure things implement the storage driver interface, right?14:38
flaper87actually, maybe using stevedore makes more sense too. By using entrypoints, we can let those filters live elsewhere14:39
kgriffsyeah, that too.14:39
kgriffs:D14:39
flaper87kgriffs: cool, I'll do that as well. :D14:40
*** jraim_ has quit IRC14:40
kgriffsok14:40
flaper87kgriffs: can I do it in a bit? I gotta finish this slides14:40
kgriffssure, no problem14:40
flaper87or maybe in a separate patch as the other changes14:40
flaper87so you can start integrating the shard thing with this14:41
kgriffsit can be a follow-up patch, sounds good14:41
alcabrera+1 for follow up patch. Get those PyconIE slides done. :P14:41
flaper87kgriffs: alcabrera awesome, thanks guys!!!14:41
* flaper87 really wanted to help with the sharding part somehow, this is it! :D14:42
alcabrera:)14:42
kgriffsalcabrera: I feel like "location" isn't quite the right term, not sure why...14:45
kgriffshttps://review.openstack.org/#/c/50456/7/marconi/common/schemas/shards.py14:46
kgriffshow about just calling it "uri"14:46
kgriffsor…something :p14:46
alcabrerakgriffs: 'uri' works for me. I'll take note of that and update it in the current patch.14:46
kgriffsok14:47
kgriffsthanks!14:47
alcabrerakgriffs: https://etherpad.openstack.org/storage-sharding-issues14:47
alcabrera:D14:47
alcabreranp14:47
kgriffsok, I'm off to do some major surgery on my sharding patch14:47
alcabrerakgriffs: +2 approve maybe on the schema patch? ;)14:48
*** jraim_ has joined #openstack-marconi14:49
kgriffsalcabrera: Sure. Since we're friends, I'll give you a deal. Only $25.14:50
kgriffs(per patch)14:50
flaper87kgriffs: d00d, sooo cheap. $25 / LOC14:51
alcabreraMaaaan14:51
alcabreraAlright, here you go...14:52
* alcabrera gives kgriffs $25e-13 / LOC14:52
openstackgerritA change was merged to openstack/marconi: Implement common pipeline class  https://review.openstack.org/5088914:52
openstackgerritA change was merged to openstack/marconi: Use the pipeline for the storage layer  https://review.openstack.org/5089014:52
openstackgerritA change was merged to openstack/marconi: Tests for Client-ID validation  https://review.openstack.org/5092814:59
openstackgerritA change was merged to openstack/marconi: feat: storage sharding schema for marconi-queues  https://review.openstack.org/5045615:20
*** Alex_Gaynor has quit IRC15:31
*** jcru has joined #openstack-marconi15:37
*** Alex_Gaynor has joined #openstack-marconi15:40
kgriffsflaper87: I think we will need to hoist the pipe creation up a layer, out of storage/base.py15:48
kgriffsthe problem is this:15:48
kgriffsthe shard manager will load N drivers, one per partition, each with a unique connection string (uri)15:49
kgriffsbut when each of those loads up15:49
kgriffsit will create it's own pipeline15:49
kgriffswhich will then load in the shard manager15:49
kgriffswhich will then load drivers15:49
kgriffswhich will then load the pipelline15:49
openstackgerritZhihao Yuan proposed a change to openstack/marconi: feat(mongo): starts to use aggregation for stats  https://review.openstack.org/5096915:49
kgriffswhich will then load shard manager15:50
kgriffs...15:50
kgriffs:p15:50
* flaper87 (plop)15:50
flaper87you completely lost me there :P15:50
kgriffsok, let me try again. :D15:50
flaper87the shard manager shouldn't create the pipeline, the pipeline is create by the driver15:50
flaper87ohhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh15:51
kgriffsright15:51
flaper87that means, I know what you mean15:51
flaper87:D15:51
kgriffsheh15:51
flaper87but15:51
flaper87(silence in my brain)15:52
kgriffshow about constructing the pipeline in the bootstrap or something?15:52
flaper87so, you're basically saying. Let's have a ShardManager that creates as many storage.driver instances as needed15:52
flaper87and gets controllers from there15:52
flaper87by calling lookup15:52
kgriffsyeah15:53
flaper87sounds good to me15:53
kgriffsthat way the drivers don't know they are being sharded15:53
flaper87kgriffs: I thought about the bootstrap thing but I didn't do it because I didn't think about this issue. I thought that being able to controll the controllers isntanciation was enough15:53
kgriffsand you can mix-and-match different driver types if you want15:54
flaper87but now, it sounds like a better option15:54
kgriffsok15:54
flaper87kgriffs:  +1 sounds good!15:54
kgriffsdid you want to make that change?15:54
kgriffsor I can do it, doesn't matter to me15:55
flaper87kgriffs: if you've some extra time to work on that, it'd be great. Otherwise, I'll take it15:55
* flaper87 is stuck with the slides15:55
flaper87mmh15:56
flaper871 thing15:56
flaper87how will you know what controller it is targetting ?15:57
flaper87I mean, if you've just 1 pipeline15:57
*** dafter has quit IRC15:57
flaper87ah, nevermind15:57
flaper87this is what happens when you start working on presentations and stop coding!15:58
* flaper87 STFU15:58
kgriffsheh15:58
kgriffsi guess the pipeline will end up looking like15:58
* alcabrera catches up, goes back to writing unit tests15:58
amettshttp://www.youtube.com/watch?v=bWGtjqv19ZA15:58
kgriffsametts: LOL15:59
kgriffs==> Stage X ==> Controller16:00
kgriffsbut then if you insert shard manager in the pipeline config, after Stage X,16:00
kgriffs==> Stage X ==> Shard Manager [==> Controller]16:00
kgriffswhere, the last stage never gets executed, actually16:01
flaper87yeah, exactly16:01
flaper87but, you'll have 1 pipeline per controller anyway16:01
*** yassine has quit IRC16:01
flaper87ametts: LOOOOOL16:02
kgriffsyeah, the bootstrap will create 1 pipeline per controller16:02
alcabreraametts: hahaha, nice.16:02
*** mpanetta has quit IRC16:02
*** dafter has joined #openstack-marconi16:02
*** mpanetta has joined #openstack-marconi16:03
alcabrerabrb16:04
*** mpanetta_ has joined #openstack-marconi16:05
*** mpanetta_ has quit IRC16:06
*** dafter has quit IRC16:06
openstackgerritZhihao Yuan proposed a change to openstack/marconi: feat(mongo): starts to use aggregation for stats  https://review.openstack.org/5096916:06
*** mpanetta_ has joined #openstack-marconi16:06
*** dafter has joined #openstack-marconi16:06
*** mpanetta has quit IRC16:07
*** jraim_ has quit IRC16:08
*** alcabrera has quit IRC16:09
*** jraim_ has joined #openstack-marconi16:11
*** ykaplan has quit IRC16:18
openstackgerritChad Lung proposed a change to openstack/marconi: Updated Marconi ReadMe file with correct git repo address  https://review.openstack.org/5097616:20
*** flaper87 is now known as flaper87|afk16:20
*** ykaplan has joined #openstack-marconi16:25
*** alcabrera has joined #openstack-marconi16:25
*** dafter has quit IRC16:42
*** dafter has joined #openstack-marconi17:26
*** dafter has quit IRC17:42
openstackgerritAlejandro Cabrera proposed a change to openstack/marconi: feat: split queues API into public/admin  https://review.openstack.org/5070117:42
openstackgerritAlejandro Cabrera proposed a change to openstack/marconi: feat: add shard management resource  https://review.openstack.org/5070217:43
*** ykaplan has quit IRC17:44
openstackgerritAlejandro Cabrera proposed a change to openstack/marconi: feat: shards storage controller interface  https://review.openstack.org/5072117:45
openstackgerritAlejandro Cabrera proposed a change to openstack/marconi: feat: shards mongodb driver + tests  https://review.openstack.org/5081517:47
openstackgerritMalini Kamalambal proposed a change to openstack/marconi: Validation for age returned by queue/stats  https://review.openstack.org/5099517:52
*** jraim_ has quit IRC18:06
openstackgerritAlejandro Cabrera proposed a change to openstack/marconi: feat: integrate shard storage with transport  https://review.openstack.org/5099818:07
openstackgerritMalini Kamalambal proposed a change to openstack/marconi: Validation for age returned by queue/stats  https://review.openstack.org/5099518:14
*** mpanetta has joined #openstack-marconi18:14
*** mpanetta_ has quit IRC18:16
*** mpanetta has quit IRC18:16
*** mpanetta has joined #openstack-marconi18:16
*** jraim_ has joined #openstack-marconi18:21
openstackgerritKurt Griffiths proposed a change to openstack/marconi: WIP: AutoConnect retry logic  https://review.openstack.org/5100618:21
*** dafter has joined #openstack-marconi18:22
*** dafter has quit IRC18:22
*** dafter has joined #openstack-marconi18:22
*** mpanetta has quit IRC18:26
*** mpanetta has joined #openstack-marconi18:26
*** acabrera has joined #openstack-marconi18:34
*** alcabrera has quit IRC18:38
openstackgerritZhihao Yuan proposed a change to openstack/marconi: feat(mongo): starts to use aggregation for stats  https://review.openstack.org/5096918:49
*** mpanetta has quit IRC18:54
*** ykaplan has joined #openstack-marconi18:59
openstackgerritKurt Griffiths proposed a change to openstack/marconi: chore: Remove GC cruft from storage driver base class  https://review.openstack.org/5101619:14
acabreraI'm heading home. See you guys later. :)19:55
*** acabrera has quit IRC19:56
*** malini is now known as malini_afk19:57
openstackgerritKurt Griffiths proposed a change to openstack/marconi: chore: Remove GC cruft from storage driver base class  https://review.openstack.org/5101620:03
openstackgerritA change was merged to openstack/marconi: Updated Marconi ReadMe file with correct git repo address  https://review.openstack.org/5097620:09
*** jdaggett1 has joined #openstack-marconi20:17
*** jdaggett1 has quit IRC20:21
*** jdaggett1 has joined #openstack-marconi20:21
*** vkmc has joined #openstack-marconi20:29
*** vkmc has quit IRC20:29
*** vkmc has joined #openstack-marconi20:29
*** jraim_ has quit IRC20:36
openstackgerritKurt Griffiths proposed a change to openstack/marconi: Setup storage pipeline in the boostrap instead of driver base  https://review.openstack.org/5104920:43
openstackgerritKurt Griffiths proposed a change to openstack/marconi: Setup storage pipeline in the boostrap instead of driver base  https://review.openstack.org/5104920:45
*** oz_akan_ has quit IRC21:02
*** etoews1 has joined #openstack-marconi21:14
etoews1in the service catalog, what is the "type" for marconi? is it "queues"?21:21
kgriffsWe discussed that in our last team mtg. Nothing is final, but it will be either queues or queueing21:25
kgriffsprobably the latter21:25
*** whenry has joined #openstack-marconi21:25
kgriffsfor example21:26
kgriffsqueueing21:26
kgriffshttps://review.openstack.org/#/c/47999/1/files/keystone_data.sh21:26
kgriffsetoews1: does that help?21:26
*** jraim_ has joined #openstack-marconi21:28
etoews1kgriffs: yes it does. so the type is "queuing".21:30
etoews1do you have a feeling for when the devstack impl will be done?21:31
kgriffsnot sure. Flavio (flaper87) was working on that21:34
kgriffsI forgot to ask him for an update at the last team mtg21:34
etoews1k. thx for the pointer.21:35
*** jcru has quit IRC21:57
*** dafter has quit IRC22:05
*** dafter has joined #openstack-marconi22:05
*** dafter has quit IRC22:10
*** etoews1 has quit IRC22:16
*** tedross has quit IRC22:18
*** jraim_ has quit IRC22:19
*** jraim_ has joined #openstack-marconi22:27
*** ykaplan has quit IRC22:30
*** amitgandhi has quit IRC22:34
*** jdaggett1 has quit IRC22:45
*** jraim_ has quit IRC22:56
*** jraim_ has joined #openstack-marconi22:59
*** jraim_ has quit IRC22:59
*** metral has joined #openstack-marconi23:34
metraltrying to setup marconi locally with mongo on a clean ubuntu 12.04 vm. upon doing $ python setup.py develop, im getting the following error: TypeError: dist must be a Distribution instance. there are also warnings such as: UserWarning: Unknown distribution option: 'pbr' and UserWarning: Unknown distribution option: 'setup_requires'. anyone care to help?23:35
vkmcHey metral, you should try with virtualenv23:39
metralhey vkmc - thx for the help & reply23:39
metrali am using virtualenv23:39
vkmcOh, great23:39
vkmcWell, some time ago I had the same problem and I solved it with virtualenv... but if you are already using it, it should be for another reason I'm not aware of :/23:40
metralcould be user error ;), so ill try again from scratch just to make sure23:40
metral;)23:40
vkmcOk :) If you cannot solve it, probably you will find more people around tomorrow at 10-11am UTC23:41
metralsounds good! appreciate the info23:42
vkmcNp, let us know what happens23:42
metralwill do23:44
metraltried from scratch on a clean vm with virtualenv done & source/bin/activated before i did $python setup.py develop and i still get the error: TypeError: dist must be a Distribution instance23:49
kgriffshmm23:54
kgriffsI haven't ran into that issue myself, but seems like python-setuptools is out of date or something23:54
kgriffsIf found this, not sure if it will help you (scroll down to the bottom): http://code.doughellmann.com/virtualenvwrapper/issue/199/packaging-problem-unable-to-use23:54

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!