Thursday, 2013-10-24

*** whenry has quit IRC00:08
*** vkmc has quit IRC00:14
*** whenry has joined #openstack-marconi00:19
*** reed has quit IRC00:33
*** nosnos has joined #openstack-marconi00:43
*** amitgandhi has joined #openstack-marconi00:59
*** amitgandhi has quit IRC01:03
*** amitgandhi has joined #openstack-marconi01:18
*** whenry has quit IRC01:38
*** malini_afk is now known as malini01:40
*** cthulhup has quit IRC01:50
*** asalkeld_lunch is now known as asalkeld01:57
*** amitgand_ has joined #openstack-marconi02:08
*** amitgand_ has quit IRC02:12
*** amitgand_ has joined #openstack-marconi02:13
*** amitgandhi has quit IRC02:19
*** amitgand_ has quit IRC02:28
*** amitgandhi has joined #openstack-marconi02:29
*** amitgandhi1 has joined #openstack-marconi02:30
*** amitgandhi1 has quit IRC02:31
*** amit has joined #openstack-marconi02:38
*** amit is now known as Guest3174702:39
*** amitgandhi has quit IRC02:42
*** Guest31747 has quit IRC02:49
*** amitgandhi has joined #openstack-marconi02:50
*** malini is now known as malini_afk02:52
*** amitgandhi has quit IRC02:54
*** jergerber has joined #openstack-marconi03:24
*** jergerber has quit IRC03:26
*** amitgandhi has joined #openstack-marconi03:50
*** whenry has joined #openstack-marconi03:59
*** amitgandhi has quit IRC04:09
*** amitgandhi has joined #openstack-marconi04:34
*** amit has joined #openstack-marconi04:37
*** amitgandhi has quit IRC04:37
*** amit is now known as Guest4336104:37
*** Guest43361 has quit IRC04:54
*** whenry has quit IRC05:05
*** whenry has joined #openstack-marconi05:36
*** amitgandhi has joined #openstack-marconi05:49
*** amit has joined #openstack-marconi05:50
*** amitgandhi has quit IRC05:50
*** amit is now known as Guest9956905:51
openstackgerritChristian Berendt proposed a change to openstack/python-marconiclient: change assertEquals to assertEqual  https://review.openstack.org/5351205:55
*** Guest99569 has quit IRC05:55
openstackgerritChristian Berendt proposed a change to openstack/python-marconiclient: change assert_ to assertTrue  https://review.openstack.org/5352606:09
*** amitgandhi has joined #openstack-marconi06:51
*** amitgandhi has quit IRC06:55
*** whenry has quit IRC07:03
*** flaper87|afk is now known as flaper8707:29
*** yassine has joined #openstack-marconi07:44
*** yassine has quit IRC07:50
*** yassine has joined #openstack-marconi07:50
*** yassine has quit IRC07:58
*** yassine has joined #openstack-marconi08:13
*** amitgandhi has joined #openstack-marconi08:53
*** amitgandhi has quit IRC08:57
*** amitgandhi has joined #openstack-marconi09:53
*** amitgandhi has quit IRC09:58
*** flaper87 is now known as flaper87|afk10:50
*** amitgandhi has joined #openstack-marconi10:54
*** amitgandhi has quit IRC10:59
*** nosnos has quit IRC11:12
*** nosnos has joined #openstack-marconi11:12
*** nosnos has quit IRC11:17
*** briancline has quit IRC11:19
*** briancline has joined #openstack-marconi11:20
*** oz_akan_ has joined #openstack-marconi11:25
*** tedross has joined #openstack-marconi11:37
*** flaper87|afk is now known as flaper8711:44
*** oz_akan_ has quit IRC11:50
*** amitgandhi has joined #openstack-marconi11:55
*** amitgandhi has quit IRC11:59
*** alcabrera has joined #openstack-marconi12:35
alcabreraGood morning!12:35
flaper87alcabrera: gooood morning12:45
flaper87:)12:45
* flaper87 will get to review stuff in a bit!12:46
flaper87just got rid of the backlog + other stuff12:46
alcabreraflaper87: :D12:49
alcabrerathe backlog builds up too quickly. :P12:49
alcabrerawelcome back, in any case! I hope your time with family was awesome. :)12:49
flaper87alcabrera: it was, I really enjoyed it!12:50
alcabreraGlad to hear it. :)12:54
*** amitgandhi has joined #openstack-marconi12:56
*** amitgandhi has quit IRC13:01
*** oz_akan_ has joined #openstack-marconi13:03
*** oz_akan_ has quit IRC13:06
*** oz_akan_ has joined #openstack-marconi13:07
*** amitgandhi has joined #openstack-marconi13:10
*** amitgandhi has quit IRC13:14
alcabrera[offtopic] Compiling GHC (trunk) is a great way to keep warm in the winter.13:19
*** mpanetta has joined #openstack-marconi13:25
flaper87LOOOOOOL13:27
flaper87alcabrera: Shouldn't admin_mode be a CLI param? https://review.openstack.org/#/c/50701/12/marconi/queues/bootstrap.py13:30
alcabreraflaper87: how do I set it as such? :x13:31
flaper87alcabrera: LOL at your comment here: https://review.openstack.org/#/c/50701/12/marconi/queues/transport/wsgi/admin/driver.py13:31
flaper87I was writing my comment and then though: "Lets first read Alej's comment below"13:31
flaper87:P13:32
flaper87(re inheriting)13:32
alcabrera"Notice that this leads to a double-dose of health when running in admin mode" :P13:32
alcabreraMy patches are the healthiest, dontcha know? ;P13:32
flaper87LOL13:32
flaper87re cli params13:32
flaper87The param should be moved into a separte list of params and register_cli_opts should be called instead of register_opts13:33
flaper87I'm fine with doing it in a separte patch13:33
flaper87no need to rebase this one13:33
alcabreraahh, I'll file a bug for that, in that case. Gtk!13:33
alcabreraOnce sharding is done enough, I'm going to have a wonderful bug hunting spree. >:)13:34
alcabreraflaper87: https://bugs.launchpad.net/marconi/+bug/1244228 (filed)13:35
jdpraxIs there going to be a pop tart bounty for bugs?13:36
mpanettaOoo13:36
flaper87jdprax: of course!!!!!13:37
flaper87alcabrera: +1 thanks13:37
alcabrerajdprax: ooohh, good point. :D13:39
*** jcru has joined #openstack-marconi13:41
*** amitgandhi has joined #openstack-marconi13:42
*** ametts has joined #openstack-marconi13:42
*** amitgandhi has quit IRC13:43
*** amitgandhi has joined #openstack-marconi13:44
*** malini_afk is now known as malini13:51
flaper87alcabrera: +2 the first one13:53
alcabreraflaper87: awesome!13:54
openstackgerritA change was merged to openstack/marconi: feat: split queues API into public/admin  https://review.openstack.org/5070113:54
alcabrerawooot13:54
alcabreraI'd been rebasing that one for about a week. :P13:54
flaper87alcabrera: is this meant to go under admin or public? https://review.openstack.org/#/c/50702/14/marconi/queues/transport/wsgi/shards.py14:09
alcabreralessee...14:09
alcabreraadmin14:10
alcabrerashard registration via shards.Resource and catalogue management via catalogue.Resource (other patch) are the current set of admin functions for the queues API. :)14:11
alcabreraflaper87: ^^14:11
flaper87alcabrera: awesome, I thought so, I needed your confirmation!14:11
* flaper87 has been publishing Marconi changes here: https://thoughtstreams.io/flaper87/marconi-progress-and-updates/14:11
flaper87dunno if anyone is actually reading that stuff14:11
alcabrerait's hard to share, I've found. :/14:12
alcabreraThe stream I've found most reliable is Twitter land + links to a blog in the Tweet.14:12
alcabreraI suppose one could ThoughtStream, then post a link to a stream in Twitter.14:12
flaper87yeah! What I'm liking about tstream is that I can just write short / long random ideas / thoughts of things and keep them contextualized14:14
flaper87I've a marconi stream, for example14:14
alcabreraflaper87: https://twitter.com/cppcabrera/status/39337998175922176014:14
flaper87:D Awesome!14:15
alcabrera:)14:15
* ametts flaper87's thoughtstream to his news reader14:15
flaper87it'd love to co-post there14:15
flaper87it'd be awesome if we could co-post there14:15
alcabreraI like to play Twitter bingo - coming as close to 140 characters as possible in a single Tweet. If I get exactly 140 characters, it's a *bingo*. :D14:16
alcabreraflaper87: I'm all for it.14:16
alcabreraIIRC, they have some shared stream feature.14:16
flaper87alcabrera: do they?14:16
* flaper87 looks14:16
alcabreraflaper87: I haven't been able to figure it out. :/14:24
alcabreraOooohh14:26
flaper87me neither!14:26
alcabreraI think I figured it out!14:26
alcabrera"Combined streams"14:26
flaper87how how how how?14:26
alcabrerahttps://thoughtstreams.io/combined/marconi-progress-and-updates/14:26
flaper87mmh14:26
alcabreraAll streams with the same title appear in "combined"14:26
flaper87SWWWWWWWWWWWEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEETTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTT14:27
alcabrera#win14:27
flaper87besides the lack of documentation, this is great!14:27
alcabreraYup.  :)14:27
alcabreraAlright, so if I keep that tab open all day: thought streams away!14:27
flaper87done, tweeted that stof!14:28
amettsHey... this one doesn't have an atom feed.  How am I supposed to add it to my news reader?14:28
flaper87stuff14:28
flaper87ametts: F5 ?14:28
flaper87:P14:28
flaper87damn, that sucks!14:28
flaper87:/14:28
alcabreraHmm...14:29
flaper87that seems something not hard to add, I'll ping jtauber on twitter14:29
alcabreraAwww, I tried to URL hack the feature, haha14:29
alcabrerahttps://thoughtstreams.io/combined/marconi-progress-and-updates.atom14:29
alcabrerafailed ^^14:29
flaper87alcabrera: hahahah, I tried that as well :P14:30
flaper87I bet eldarion guys are loughing looking at our failed attempts14:31
flaper87(as if they don't have anything better to do)14:31
alcabreralol yeesss14:31
flaper87alcabrera: ametts https://twitter.com/flaper87/status/39338393241191628814:32
flaper87comment here :)14:32
alcabreraflaper87: nice!14:32
flaper87alcabrera: +2 the next one14:35
flaper87kgriffs_afk: knock knock14:35
flaper87kgriffs_afk: can we get the client patches in?14:35
* flaper87 needs to write some code now14:35
alcabrerayay!14:36
alcabreraI'll update the ported catalogue patchset now. kgriffs_afk had some great feedback in it.14:37
alcabreraflaper87: make the code awesome. :D14:37
alcabreraflaper87: I found something *really* cool. That comment you were telling me - :returns: http - 200 | 404 - made me think to see what sphinx thought.14:40
alcabreraLook what I found: http://pythonhosted.org/sphinxcontrib-httpdomain/14:40
alcabrera:query sort: one of ``hit``, ``created-at``14:40
alcabrera:reqheader Accept: the response content type depends on ....14:40
alcabrera:resheader Content-Type: ...14:41
alcabrera:statuscode 200: no error14:41
alcabreraetc.14:41
alcabreraand it's actively maintained!14:41
flaper87alcabrera: AWESOME! That sounds really nice!14:41
alcabreraLatest release: 1.2.0 Released on October 19, 2013.14:41
flaper87cool cool, my vote goes to stick to that14:42
flaper87:D14:42
alcabrerayeeessss14:42
alcabreraI'm going to check with infra.14:42
alcabreraSee if we have such a thing installed.14:42
flaper87alcabrera: I'd suggest checking global-requirements.txt first14:44
flaper87and see if it's already being used ofr if there's somethign similar being used by other projects14:44
alcabreragood point.14:44
alcabreraflaper87: the search yielded fruit: global-requirements:141: sphinxcontrib-httpdomain14:45
alcabrerayaaaaay, this find is *awesome*14:45
* alcabrera loves sphinx more today than yesterday.14:45
flaper87alcabrera: awesome, that's even better!14:47
flaper87I wonder what Marconi's docs look like as for today, I may run the build later and see what comes out of it14:48
alcabreraThat'd be cool. Let me know. :D14:50
alcabreraanywaaaay, with that excitement noted and learned, I'll continue working on the catalogue port. :P14:55
*** jergerber has joined #openstack-marconi14:57
*** whenry has joined #openstack-marconi14:59
*** kgriffs_afk is now known as kgriffs15:09
alcabrerakgriffs: o/15:10
kgriffshey15:17
*** etoews1 has joined #openstack-marconi15:19
alcabrerakgriffs: how go things? :)15:20
etoews1when listing messages i've noticed that the response always includes a "links" with a "rel": "next" even if the next page is empty.15:21
etoews1this means clients always have to make one extra call to know that they've hit the end of the messages.15:22
kgriffswell, have you really hit the "end"?15:23
kgriffsI mean, a producer could always post another message15:23
etoews1you mean post another message between the time it takes to get a response and requesting the next page. right?15:24
kgriffsyes15:24
kgriffsalso...15:24
kgriffsThe original thought was that clients would be constantly polling for new messages; there isn't a sense of ever being an end15:25
etoews1i see what you mean but then isn't there a pretty good chance if someone starts listing messages they wind up in an "infinite" loop.15:25
etoews1so list is more like stream.15:26
kgriffsyeah15:26
kgriffsthat's a good way to put it15:26
kgriffsthat was the original idea, like i said, but i'm happy to discuss other interaction models15:26
etoews1ya, i can definitely see the use case for streaming. i'm sure there's a use case for a "point in time" listing but nothing springs to mind.15:28
kgriffsLet me add a note to our API feedback page and we can keep noodling on it.15:28
*** reed has joined #openstack-marconi15:29
kgriffsadded a note at the bottom here: https://wiki.openstack.org/wiki/Marconi/specs/api/next#HTTP15:29
etoews1ya, gotta think about this one a bit.15:29
etoews1thx.15:30
kgriffsthanks for the feedback!15:30
kgriffsI can't let you guys have all the fun. https://thoughtstreams.io/combined/marconi-progress-and-updates/15:30
alcabrerakgriffs: :D15:31
*** whenry has quit IRC15:31
alcabreraetoews1: thanks for the feedback! I'm pretty fond of the streaming model, personally, but more ideas can reveal an even better model!15:31
jdpraxIs an exiting feature something that makes users want to use somthing else? ;-)15:32
alcabrerajdprax: lol - possibly after someone figures out it doesn't quite work. :P15:32
etoews1alcabrera: the streaming model makes perfect sense. i'd expect this to be the majority use case. just wondering (aloud) if there are other use cases users might need.15:33
alcabrerawondering aloud - we need more of that. :)15:42
kgriffsalcabrera: re https://review.openstack.org/#/c/52160/4/marconi/queues/storage/base.py15:42
alcabreraflaper87: I heard you're having some strange unit test failures. I ran into some while developing the admin API branch. Could you tell me what you're seeing?15:43
kgriffsHow about explicitly listing out the different kwargs and giving them default values of "None"?15:43
alcabrerakgriffs: that'd work, esp. since I'm not expecting this storage end point to end up with too many args.15:43
kgriffsThe child class could still do **kwargs if it wanted, but at least this way you have the interface self-documenting15:43
alcabreras/end point/controller15:43
alcabrera+115:43
openstackgerritFlavio Percoco proposed a change to openstack/marconi: Isolate tests a bit more  https://review.openstack.org/5366915:44
kgriffsalcabrera: re https://review.openstack.org/#/c/52160/4/marconi/queues/storage/exceptions.py15:44
openstackgerritA change was merged to openstack/marconi: feat: add shard management resource  https://review.openstack.org/5070215:44
openstackgerritFlavio Percoco proposed a change to openstack/marconi: Isolate tests a bit more  https://review.openstack.org/5366915:45
kgriffsI see what you are saying. ShardDoesNotExist is ok to keep as a separate exception15:45
kgriffsbut, "entry" is still a bit vague IMO15:45
alcabrerakgriffs: agreed. I'm at a loss for an alternate name. :/15:45
flaper87alcabrera: indeed!15:45
kgriffs(considering this is in a common storage exceptions module)15:45
*** vkmc has joined #openstack-marconi15:45
*** yassine has quit IRC15:46
* kgriffs is thinking15:46
flaper87back!15:46
alcabreraflaper87: the particular error I kept running into was that every unit.queues.transport.wsgi test was using Faulty storage. I linked it to the setattrs in the FaultyTestBase.15:47
flaper87alcabrera: hehe, EXACTLY15:47
alcabreraflaper87: I fixed it in the final patch of the admin API series.15:47
alcabrera:D15:47
alcabreraFixed - by promoting faulty to a first class driver in setup.cfg15:47
flaper87alcabrera: cooool, I just pushed a patch to isolate mongodb tests a bit more15:47
alcabreraI spent 4 hours on this yesterday, so I didn't want you to suffer the same fate. :P15:47
flaper87alcabrera: ok, did you remove the monkey patch thing in the TestFaulty ?15:48
alcabreraflaper87: yup!15:48
flaper87alcabrera: awesome!15:48
flaper87alcabrera: thanks15:48
alcabreraflaper87: https://review.openstack.org/#/c/50998/7/tests/unit/queues/transport/wsgi/base.py15:48
alcabreraaaand: https://review.openstack.org/#/c/50998/7/setup.cfg15:48
flaper87alcabrera: cool15:48
flaper87kgriffs: good morning15:48
flaper87:)15:48
kgriffso/15:49
kgriffsgood afternoon!15:49
flaper87kgriffs: how are you doing?15:50
kgriffsThe solum guys keep trying to steal me away from Marconi. :p15:50
flaper87kgriffs: don't you dare!15:50
alcabreralol15:50
alcabreraproject solum - lots of excitement.15:50
kgriffsnah, I will be with Marconi for a while yet. Looks like I will start helping them though as well.15:50
kgriffsaaaanyway15:51
alcabrerano wonder you've been so busy, kgriffs! How do you find the time? :P15:51
kgriffsflaper87: glad to have you back, and to hear you had a nice break15:51
flaper87kgriffs: thanks :D15:51
flaper87kgriffs: hey, I was wondering if we could get client patches merged today15:52
kgriffsheaven knows, I could use a break some time as well. Things have been to busy to take one for a while.15:52
flaper87they've been there for a while and are feeling lonely15:52
flaper87:P15:52
kgriffsflaper87: I've been trying to get to those all week. Hopefully today!15:52
flaper87kgriffs: thanks!!15:53
kgriffsalcabrera: ShardNotMapped ?15:54
alcabrerakgriffs: perfect - I'll use that. Thanks!15:54
kgriffsactually15:54
kgriffsQueueNotMapped15:54
kgriffswe are mapping queues to shards15:54
alcabreraYeah, the gears were turning, but didn't catch up to my excitement. :P15:54
kgriffsnot the other way around?15:54
alcabrera(project, queue) => shard15:55
alcabreraCatalogue ^^15:55
alcabreraQueueNotMapped works better.15:55
kgriffskk15:55
alcabreraI'm considering still having it inherit from DoesNotExist - thoughts? (kgriffs)15:56
alcabreraIt's a form of MappingDoesNotExist15:56
flaper87why do we need to split the storage into admin / public ?15:57
flaper87TBH, I don't like that idea much!15:57
kgriffsalcabrera: that's fine to still inherit15:57
flaper87IMHO, the admin / public abstraction belongs to the API / transport and not to the storage15:58
alcabreraflaper87: good question - I'm not entirely sure on this one.15:58
alcabreraIt actually got a little messy towards the end of the admin API implementation because of the distinction.15:59
alcabreraWhat does admin storage mean in the context of: sharding? pipeline drivers?15:59
alcabreraSo that made me a little reluctant about how to approach that, thinking back to that.15:59
flaper87admin means nothing in the context of storage driver - including sharding.16:00
alcabreraI'll happy to remove that distinction from the patches, and leave storage as storage.16:00
alcabrera*I'm16:00
alcabrerakgriffs: thoughts>16:00
alcabrera?16:00
flaper87The only reason wy that would make sense is because we're actually splitting the databases16:01
flaper87and that could also be addressed with the pipeline16:01
flaper87instead of splitting them16:01
openstackgerritChad Lung proposed a change to openstack/marconi: Update the Marconi readme to have more detailed and correct installation instructions.  https://review.openstack.org/5367316:02
*** kgriffs has left #openstack-marconi16:05
*** kgriffs has joined #openstack-marconi16:05
*** ChanServ sets mode: +o kgriffs16:05
alcabreraI'll be right back.16:05
alcabreraI have a meeting. :)16:05
alcabreraShould be about 30m.16:06
*** alcabrera is now known as alcabrera|afk16:06
kgriffsflaper87: if we say that the storage driver for messages also manages the shard catalog, we would need to add another config section for the DB where the catalog resides16:08
kgriffsor, I suppose, we could overload the existing config, since shard DB configs come from the catalog16:08
kgriffsbut that might get confusing16:08
kgriffsin one mode, the config means "location to store messages" and in the other "location to store the catalog"16:09
kgriffssecond consideration:16:09
kgriffsyou won't be able to mix catalog and message storage backends, e.g. MySQL for catalog and MongoDB for messages16:09
kgriffsnot sure that is a big deal, just calling it out for discussion16:10
flaper87but, wouldn't it be better to have a ShardsController and a CatalogController instead? That would follow current API lines16:11
* flaper87 is taking a look at the patch while discussing this16:14
kgriffshmm16:14
flaper87for example, I'd expect the ShardController to be loaded if sharding is enabled16:17
flaper87and I'd expect the admin endpoint to call `admin` like operations on that controller16:18
kgriffsright16:18
flaper87we could also have a ShardDriver if that makes it easier to isolate sharding16:18
flaper87but loading it shouldn't depend on the admin_mode being True16:18
kgriffsso, marconi would just always load ShardDriver16:19
kgriffsit just may not be used?16:19
kgriffsif the controllers are instantiated lazily16:20
kgriffsthen that seems ok.16:20
flaper87kgriffs: exactly16:20
kgriffsI'm still thinking about whether we need a standalone shard driver16:20
flaper87kgriffs: I'd rather have a controller, TBH16:21
flaper87the only benefit I see of having a separate driver is that we can actually control the database, like putting the shard stuff in a separte db16:21
flaper87but, I'm not sure I like that idea16:21
kgriffswell, the shard stuff needs to be in a "global" db that sits outside any shard cluster16:22
flaper87agreed16:23
kgriffsbut the question is, does that "global" db need to be a different backend flavor?16:24
kgriffshere's an idea:16:24
kgriffshttp://paste.openstack.org/show/49522/16:26
kgriffsif sharding is enabled, then the storage driver is responsible for initializing those controllers however it wants, based on config options16:27
kgriffssome drivers could even accept the name of a separate "ShardDriver" type to be loaded via stevedore16:28
flaper87I don't think that's necessary (re separate db)16:28
flaper87I think using the database specified under `database` is good16:28
kgriffswe could add that functionality later if people want to write custom shard drivers16:28
flaper87+116:28
kgriffshttp://paste.openstack.org/show/49523/16:29
kgriffsI'm trying to decide if it is worth having catalog_uri16:29
kgriffsI'm leaning toward just overloading uri and database16:30
kgriffsand documenting that in sharding mode, those options refer to the catalog DB16:30
kgriffswhile shard configs actually come from the catalog, not the INI16:31
*** alcabrera|afk is now known as alcabrera16:32
alcabreraLet's see all I missed.16:32
alcabreraSeems like things have been busy since I left. :P16:33
flaper87kgriffs: mmh, it kinda makes sense to me!16:33
* flaper87 is giving it more thoughts16:33
flaper87btw guys, I16:33
flaper87damn it16:33
flaper87btw guys, I'm proposing marconi for Outreach Program for Women16:33
kgriffscool.16:34
* alcabrera reads up on that ^^16:34
alcabreraflaper87: +10!16:34
alcabreraI love it! :D16:35
alcabrerayup - I missed a lot on the sharding discussion. Interesting points.16:36
alcabreraI was off onboarding someone on Marconi. :)16:36
alcabreraSo I'm +1 for dropping the notion of admin storage driver.16:37
flaper87alcabrera: w000t16:37
alcabreraI'm not sure on the shard driver issue - I'll need to think on that a bit.16:37
kgriffs?16:37
kgriffsI thought you just said you were +116:38
alcabreraLemme clarify, my apologies. :P16:38
kgriffsdo you mean, how to configure16:38
alcabreraI'm +1 for dropping the distinction between admin and public for storage drivers. That's the domain of the transport API.16:39
alcabreraI'm unsure about whether a separate sharding storage driver is necessary, and I'm not sure about the overloaded 'uri' field.16:39
kgriffsoic16:40
kgriffsmy thought was YAGNI - wait until someone asks for a separate storage driver16:40
* alcabrera thinks while eating tuna salad16:40
kgriffssorry, I meant separate sharding driver16:40
kgriffs(that the storage driver would use for the catalog)16:41
kgriffsI guess it isn't a sharding driver so much as a catalog driver16:41
kgriffsbut whatever16:41
alcabrerayeah, I see what you're saying.16:41
alcabrerathe catalogue driver did handle the register/deregister aspects. It just needs to be plugged into the queues transport + caching if sharding is enabled.16:42
kgriffsthat driver was more of a meta-driver that sits in front of the real driver16:43
*** amitgandhi has quit IRC16:43
kgriffshmm16:44
kgriffsseems like we need to reconcile the architecture here16:45
* kgriffs gets out whiteboard16:45
alcabrera+1 for whiteboards16:45
kgriffsbrb16:54
flaper87guys, I'm heading out now! TTYL16:57
alcabreraflaper87: take care! Thanks for your review help today. :D16:59
flaper87alcabrera: thanks! :D17:00
*** cpallares has joined #openstack-marconi17:00
flaper87cpallares: welcome :D17:00
cpallares:)17:00
alcabreracpallares: hey hey! :D17:00
flaper87Guys, cpallares picked Marconi as her OPW project :)17:00
alcabreraw00t!17:00
amettsOPW == ?17:01
cpallaresHello, #openstack-marconi!17:01
cpallaresoutreach program for women17:01
alcabreraametts: https://wiki.gnome.org/OutreachProgramForWomen17:01
amettsAh.17:01
openstackgerritMalini Kamalambal proposed a change to openstack/marconi: Add Tests for non-existing resources  https://review.openstack.org/5369117:03
flaper87cpallares: btw, There's one project in the OPW/Ideas (websocket for Marconi) but there are other really cool things to do in case you don't find that one *Very interesting*17:03
maliniyayy..More WOMEN!!! 8-)17:04
flaper87ok, heading out now!17:04
alcabreraflaper87: go! Enjoy your time away from work. ;)17:04
*** flaper87 is now known as flaper87|afk17:05
cpallaresby flaper87!17:05
cpallaresand thank you!17:05
cpallares*bye17:05
maliniwelcome cpallares!17:06
cpallareshi malini!17:06
cpallaresthank you17:06
malinigreat to have you here !17:06
alcabreraI'll be back. :)17:08
*** alcabrera is now known as alcabrera|afk17:08
kgriffscpallares: welcome!17:09
cpallaresso I'm looking at the marconi launchpad and it suggests to go here https://github.com/stackforge/marconi17:10
cpallaresbut that's a dead link17:10
cpallaresalso thank kgriffs17:10
maliniits here https://github.com/openstack/marconi17:10
kgriffsalcabrera|afk, flaper87|afk: http://goo.gl/wOvZXy17:11
kgriffsalcabrera|afk: ping me when you get back.17:12
cpallaresthanks malini!17:13
openstackgerritMalini Kamalambal proposed a change to openstack/marconi: Add Tests for non-existing resources  https://review.openstack.org/5369117:19
*** kgriffs is now known as kgriffs_afk17:22
*** amitgandhi has joined #openstack-marconi17:29
cpallaresmalini: should I install the mongoDB in a vm with DevStack?17:38
cpallaresor can I install it on my computer?17:38
*** etoews1 has quit IRC17:46
*** kgriffs_afk is now known as kgriffs17:54
*** alcabrera|afk is now known as alcabrera17:59
alcabreracpallares: hey!17:59
alcabreraYou can insatll mongodb locally and marconi locally. It's how I usually develop on it.18:00
alcabrera*install18:00
alcabrerakgriffs: ping18:00
cpallaresawesome, thanks.18:00
*** oz_akan_ has quit IRC18:03
alcabrerakgriffs: nice whiteboard!18:03
*** oz_akan_ has joined #openstack-marconi18:03
*** oz_akan_ has quit IRC18:07
*** oz_akan_ has joined #openstack-marconi18:08
asalkeldhi, is the marconi  client function atm?18:16
asalkeldI want to write a proof of concept based on marconi, but I see a pile of patches for the client still pending18:17
asalkeldI just want to create a q, send/recv messages18:17
maliniasalkeld: we dont have the client yet..But if you would like cURL reqs for those, I can give them18:18
asalkeldok, that's fine18:18
asalkeldmalini, I appreciate it thanks18:19
malinilet me grab those for you18:19
alcabreraasalkeld: the client grows a little each day, mostly thanks to flaper87|afk's efforts.18:20
asalkeldit's *really* needed18:20
alcabreraWe're at the point right now (in master) where one can make a connection to a marconi server (possibly behind keystone auth) and request queues, IIRC.18:20
alcabreraThat still leaves the part where one would query messages, post messages, handle claims...18:21
*** etoews1 has joined #openstack-marconi18:21
asalkeldso you can't send/recv messages yet?18:22
alcabreranot yet. :/18:23
alcabreraTo test locally, I usually use httpie18:23
alcabreraSo I can do things like...18:23
alcabrerahttp post localhost:8001/v1/queues/my_queue/messages < m.json18:23
alcabrerawhere m.json contains my message bodies18:23
alcabrera[{ttl': 120,18:24
alcabreraoops18:24
asalkeldso the server support message send/recv?18:24
alcabrerayep18:24
asalkeldphew18:24
alcabrerathe server is fully functional18:24
alcabreraAPI v1.0 is implemented18:24
asalkeldok, cool18:24
alcabreraall the way18:24
alcabreraJust the client needs work - sorry asbout the confusion.18:24
alcabrera*about18:24
alcabreram.json: [{"ttl": 120, "body": {"awesome": "data"}}, {"ttl": 1000, "body": {"tacos": "and_more"}}]18:25
maliniasalkeld: I have some curl reqs here http://paste.openstack.org/show/49540/18:25
asalkeldooo, that's rocking - thanks malini18:25
alcabreramalini: nice! That's a thorough curl repo. :D18:25
alcabreraI need to make one for httpie... :)18:25
alcabrerademonqueue. <318:26
alcabrerawait...18:26
alcabreraI found a typo. :P18:26
maliniWe shud have a docs directory in our repo & add the cURL, httpie stuff there18:26
alcabreramost of the queues are demoqueue - there's at least one demo*n*queue.18:26
*** kgriffs is now known as kgriffs_afk18:26
alcabreramalini: +118:26
malinithts a demon ;)18:26
alcabreralol18:26
alcabreraasalkeld: if the marconi server you're testing is localhost (easy to bring one up), you won't need the X-Auth-Token header unless you're also running it against some auth endpoint.18:27
asalkeldi need it to work across machines18:28
*** kgriffs_afk is now known as kgriffs18:28
asalkeldI want to demo using marconi to order the application of configuration on vms using Heat18:29
alcabrerakgriffs: ping part 218:29
alcabrerabrb18:30
kgriffs+1 for docs18:30
kgriffshttps://blueprints.launchpad.net/marconi/+spec/rtd18:30
*** whenry has joined #openstack-marconi18:31
*** amitgandhi has quit IRC18:31
zyuankgriffs: a sample implementatuib about /health which uses alive(): https://review.openstack.org/#/c/53457/18:31
alcabrerakgriffs: did you know that sphinx supports the amazing httpdomain? (wrt to rtd)18:31
kgriffsalcabrera: saw that earlier - good find!18:31
*** amitgandhi has joined #openstack-marconi18:31
*** alcabrera is now known as alcabrera|meet18:32
*** alcabrera|meet is now known as alcabrera18:39
*** etoews1 has quit IRC18:39
*** etoews1 has joined #openstack-marconi18:39
openstackgerritAngus Salkeld proposed a change to openstack/python-marconiclient: Fix some links in the README  https://review.openstack.org/5371018:44
openstackgerritAngus Salkeld proposed a change to openstack/python-marconiclient: Add a status message to the README  https://review.openstack.org/5371118:44
alcabreraasalkeld: Thanks! I'll check those out soon. :D18:45
asalkeldsure, no rush18:47
*** relaxdiego has joined #openstack-marconi18:53
zyuanalcabrera: which middleware our deployment currently using? eom?18:55
alcabreraasalkeld: just one nit on the second pull request, but then that gets a +2 from me.18:56
alcabreraerr, patchset.18:56
alcabrerarepo terminologies get messy at times.18:56
asalkeldlooking18:56
alcabrerakgriffs: ping18:59
openstackgerritAngus Salkeld proposed a change to openstack/python-marconiclient: Add a status message to the README  https://review.openstack.org/5371119:04
kgriffsalcabrera: pong19:05
alcabreraawesome19:06
alcabreralet's talk sharding19:06
kgriffsok19:06
kgriffsg+ for screen share, but we can keep chatting in here19:06
alcabreracool19:06
kgriffs#startmeeting sharding19:07
openstackMeeting started Thu Oct 24 19:07:00 2013 UTC and is due to finish in 60 minutes.  The chair is kgriffs. Information about MeetBot at http://wiki.debian.org/MeetBot.19:07
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.19:07
openstackThe meeting name has been set to 'sharding'19:07
alcabreraad-hoc meeting - that's cool.19:07
kgriffsamitgandhi: ur invited19:08
kgriffs:p19:08
kgriffsok19:09
*** malini is now known as malini_afk19:09
kgriffsso here we have what I think was my original concept19:09
alcabreraalright19:09
kgriffshttps://plus.google.com/hangouts/_/e74ecfded42b35093aa9dad06e4af44c7824ea2a?hl=en19:09
kgriffs#topic planes19:10
*** whenry has quit IRC19:10
alcabrerathis is a really pretty diagram.19:10
kgriffs:D19:10
alcabrera:)19:10
kgriffsso, I realize this is different from what we discussing earlier today19:11
alcabrerahmmm19:11
kgriffsI just wanted to get my mental model down from what I was thinking when i did my sharding patch19:11
alcabrerathat looks a lot like what I was envisioning.19:11
kgriffsso, a few things about that19:12
alcabrerawhere the transport speaks with the /v1/shards for registration, and then the sharding.X is implemented in terms of the ShardController/CatalogueController19:12
kgriffsfirst, how distinct do we want to make the control and data planes?19:12
alcabreraI want to make it so that if sharding is removed, pipeline fastforwards to mongodb.Driver19:13
alcabreraso...19:13
kgriffswell, that is taken care of in the current implementation19:14
alcabrerahmm...19:14
alcabrerayeah19:14
kgriffsin the bootstrap, it will either use sharding.Driver or a real driver19:14
kgriffsdepending on config19:14
alcabrerawhat came to mind when you asked about 'distinct' above?19:15
alcabreraI'm drawing a blank19:15
kgriffsoh, so do we consider both of those part of the same API, or not?19:15
alcabreraah, I see.19:15
alcabrerait doesn't make sense to have the sharding admin API enabled if sharding isn't enabled. :/19:16
alcabreraI hadn't considered that.19:16
kgriffsetherpad19:17
alcabreraYup19:17
kgriffshttps://etherpad.openstack.org/p/queuing-scratch19:17
kgriffslet me jot down api things19:17
*** ametts has quit IRC19:17
alcabrerak19:17
kgriffsalcabrera: ping19:29
kgriffsok. so the idea is to keep a single API19:29
alcabrerayup19:30
kgriffsand enable extra routes when sharding is enabled19:30
alcabrerawith conditionally enabled routes19:30
alcabreradepending on admin_mode and sharding19:30
kgriffsand those routes should require admin role.19:30
kgriffsah, admin mode19:30
kgriffsso you would have to enable both19:30
alcabrerayup. :)19:30
kgriffsI guess that sets us up for other admin things that will always be on19:31
kgriffse.g., /storage/stats19:31
kgriffsalthough19:31
kgriffswhy not always have them on, just let RBAC take care of it?19:31
alcabrerathat sounds better19:31
alcabreraexcept19:31
alcabrerafor the case of registering shards19:31
alcabrera/v1/shards/{shard}19:31
alcabreraThat should be firewalled/holywalled/watev19:31
kgriffsdoesn't that require an admin token?19:32
kgriffshmmm19:32
kgriffsmaybe safer to keep the admin mode after all19:32
*** reed has quit IRC19:32
alcabrerayup19:32
alcabreraI can see */stats being limited by RBAC19:32
kgriffsthen you can protect people with auth as well as network19:32
*** asalkeld is now known as asalkeld_breakfa19:33
*** asalkeld_breakfa is now known as asalkeld_afk19:33
alcabreraBut I remember oz_akan_'s words about proxy-dev.19:33
kgriffswell, we ought to just do all or nothing imo19:33
kgriffsso, keep admin mode flag for now19:33
kgriffsand only turn on control plane there19:33
kgriffsmaybe change to "enable_control_plane" or something19:34
alcabrerahmmmm19:34
alcabreramaybe!19:34
kgriffsi guess admin mode is ok too19:34
kgriffsanyway19:34
alcabreraenable_control_plane is another good name19:34
kgriffsthat is effectively what it would do19:34
alcabreraa little more clear maybe19:34
alcabrerayeah19:34
kgriffsok, last thing to think about is whether to have that separate "Catalog Driver"19:35
*** reed has joined #openstack-marconi19:35
kgriffslet's noodle on that and sync up later19:35
alcabreracool19:35
alcabreraI'll continue updating the admin API patches wrt to this discussion and pending review comments.19:35
alcabreraLet's hold on that etherpad19:35
alcabreraSounds like BP gold19:35
alcabrerasee ya later, kgriffs!19:35
oz_akan_sorry, quick Q: do we change the uri based on the shard and do we return the uri with shard information in to the user?19:35
oz_akan_alcabrera: you might reply19:36
alcabreraoz_akan_: let's talk offline on that. :)19:36
oz_akan_k19:36
*** alcabrera is now known as alcabrera|afk19:36
kgriffs#link http://goo.gl/BiCgnw19:37
kgriffs#endmeeting19:43
openstackMeeting ended Thu Oct 24 19:43:23 2013 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)19:43
openstackMinutes:        http://eavesdrop.openstack.org/meetings/sharding/2013/sharding.2013-10-24-19.07.html19:43
openstackMinutes (text): http://eavesdrop.openstack.org/meetings/sharding/2013/sharding.2013-10-24-19.07.txt19:43
openstackLog:            http://eavesdrop.openstack.org/meetings/sharding/2013/sharding.2013-10-24-19.07.log.html19:43
openstackgerritChad Lung proposed a change to openstack/marconi: Update the Marconi readme to have more detailed and correct installation instructions.  https://review.openstack.org/5367319:52
*** vkmc has quit IRC20:02
*** vkmc has joined #openstack-marconi20:03
*** vkmc has quit IRC20:03
*** vkmc has joined #openstack-marconi20:03
*** amitgandhi has quit IRC20:07
*** amitgandhi has joined #openstack-marconi20:08
openstackgerritChad Lung proposed a change to openstack/marconi: Update the Marconi readme with more detailed installation instructions  https://review.openstack.org/5367320:11
*** amitgandhi has quit IRC20:12
alcabrera|afkI'm out for the night, guys.20:21
*** alcabrera|afk is now known as alcabrera20:21
alcabreraTake care!20:21
*** alcabrera has quit IRC20:21
*** relaxdiego has left #openstack-marconi20:28
*** asalkeld_afk is now known as asalkeld20:35
*** amitgandhi has joined #openstack-marconi20:46
kgriffsupdated diagram: http://grab.by/rsoI20:48
*** amitgandhi has quit IRC20:57
*** amitgandhi has joined #openstack-marconi20:58
*** amitgandhi has quit IRC21:01
*** amitgandhi has joined #openstack-marconi21:02
cpallareshello! I'm installing mongoDB and it's asking me to give it a directory. Is the directory ~/.marconi or is it /data/db/?21:04
zyuanjust regular mongodb's; could be /data/db21:07
kgriffscpallares: yeah, I think of mongo as something that can be shared between different apps on my box when I'm developing, so I just keep the db files in a central location21:13
cpallareszyuan, kgriffs: ok thanks!21:14
kgriffsrock on21:17
zyuankgriffs: so, any further ideas about the json size checking?  i can implement it in either way, just i guess openstack won't accept it.21:18
cpallaresmmm... when I try to run marconi-server, I get this error http://pastebin.com/E6DTckpG21:30
zyuankgriffs: ^^ why we import pbr.packaging in version.py ?21:33
*** ayoung has quit IRC21:34
zyuancpallares: you may missed some dependency.  can you open a python intepreter and try import pbr; import pbr.packaging?21:34
cpallaresImportError: No module named pbr21:35
cpallareszyuan: is this a module I can download?21:36
zyuanthis is listed in requirements.txt ; so if you installed marconi through pip, pbr should has been there21:36
zyuanhttps://review.openstack.org/#/c/53673/3/README.rst21:36
zyuanwe are updating the installation instructions21:36
cpallaresokay, no worries, it installed through pip thank you zyuan21:37
cpallaresah nvm, i still get an error http://pastebin.com/YUQsCPex21:40
zyuancpallares: see this https://review.openstack.org/#/c/53673/3/README.rst21:42
zyuanyou need to .conf files; when you defined some path there. make sure they exist21:43
zyuanmarconi-server the command is for testing only, so you can just use an per-user config file and run it as a user (of course, port will be different)21:44
cpallareshow do i set up a per-user conf file?21:51
openstackgerritChad Lung proposed a change to openstack/marconi: Update the Marconi readme with more detailed installation instructions  https://review.openstack.org/5367321:52
openstackgerritChad Lung proposed a change to openstack/marconi: Update the Marconi readme with more detailed installation instructions  https://review.openstack.org/5367321:53
*** tedross has quit IRC22:04
*** mpanetta has quit IRC22:06
*** etoews1 has quit IRC22:07
*** oz_akan_ has quit IRC22:24
*** amitgandhi has quit IRC22:26
*** amitgandhi has joined #openstack-marconi22:27
*** amitgandhi has quit IRC22:32
*** whenry has joined #openstack-marconi22:42
*** jergerber has quit IRC22:47
*** whenry has quit IRC22:55
*** whenry has joined #openstack-marconi22:57
*** kgriffs is now known as kgriffs_afk23:00
*** whenry has quit IRC23:06
*** jcru has quit IRC23:16
*** etoews1 has joined #openstack-marconi23:23
*** amitgandhi has joined #openstack-marconi23:27
*** amitgandhi has quit IRC23:34
*** amitgandhi has joined #openstack-marconi23:38
*** amitgandhi has quit IRC23:42
*** amitgandhi has joined #openstack-marconi23:54
*** etoews1 has quit IRC23:59

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!