16:04:07 <kgriffs> #startmeeting marconi
16:04:08 <openstack> Meeting started Mon Oct 28 16:04:07 2013 UTC and is due to finish in 60 minutes.  The chair is kgriffs. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:04:09 <alcabrera> o/
16:04:10 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
16:04:12 <openstack> The meeting name has been set to 'marconi'
16:04:17 <flaper87> \o/
16:04:22 <flaper87> <o/ \o>
16:04:28 <kgriffs> /o/
16:04:29 * flaper87 bitboxes
16:04:34 <kgriffs> \o\
16:04:38 <kgriffs> \o/
16:04:56 <kgriffs> #link https://wiki.openstack.org/wiki/Meetings/Marconi#Agenda
16:05:02 <malini> o/
16:05:28 <flaper87> cp16net: welcome :)
16:05:42 <kgriffs> #topic revi3w @CtioN$ fr0m l@57 7IM3
16:05:54 <flaper87> (plop)
16:05:57 <alcabrera> flaper87: you have *all* the actions.
16:05:59 <alcabrera> :D
16:06:02 <flaper87> damnit!
16:06:03 <kgriffs> #link http://eavesdrop.openstack.org/meetings/marconi/2013/marconi.2013-10-21-16.03.html
16:06:17 <kgriffs> looks like flaper87 is the only one doing any work these days
16:06:19 <kgriffs> ;)
16:06:27 <flaper87> so, I didn't do much on the Heat area because I had to focus more on the summit and API things.
16:06:32 <flaper87> #link https://etherpad.openstack.org/p/cross-transport-api-spec
16:06:36 <flaper87> that's re API spec
16:07:03 * kgriffs reads
16:07:08 <flaper87> If you want to take a look to the design, just open the PDF
16:07:27 <flaper87> which contains some diagrams and very high-colored definition
16:07:30 <flaper87> :P
16:07:50 <flaper87> anyway, It still needs some writing but the bases are already written there
16:08:35 <cp16net> flaper87: hello :-P
16:08:57 <flaper87> cp16net: hey :)
16:09:03 <alcabrera> Herein, an API sounds like a mapping of routes to resources.
16:09:19 <alcabrera> so each transport can serve up a series of resources according to a particular map.
16:09:25 <alcabrera> That's how I understand it. :)
16:10:41 <flaper87> alcabrera: yeah, that's basically it
16:10:58 <flaper87> there are some thing I want it to guarantee, though:
16:11:12 <flaper87> I want that API to be easy to extend, version and validate
16:11:20 <flaper87> and it obviously has to be cross-transport
16:11:42 <kgriffs> re extensions
16:11:51 <alcabrera> the extend part is going to be interesting.
16:11:52 <kgriffs> will those receive the stevedore treatment?
16:13:09 <flaper87> I was thinking, yes!
16:13:32 <flaper87> That way users may choose to enable them or not
16:13:42 <kgriffs> ok, good. It sure beats having to maintain a marconi fork with custom patches
16:14:01 <alcabrera> +1
16:14:25 <alcabrera> sharding is a sort of first API extension./
16:14:29 <kgriffs> people can then load whatever cute, stevedorable extensions they like without hacking the code
16:14:38 <flaper87> cool! I think I'll be able to work something out for our next meeting, which will be after the summit
16:15:14 <kgriffs> and you are planning to have the endpoints serve up the schema itself so clients can use it?
16:15:15 <alcabrera> a cool demo of getting that machination together would a /stats all the things API extension loaded hrough stevedore and implemented in a separate repop.
16:15:18 <alcabrera> *repo
16:15:22 <kgriffs> s/endpoints/transports
16:15:28 <alcabrera> **would be a...
16:15:35 <kgriffs> alcabrera: yeah
16:15:48 <kgriffs> or, like, a statsd-a-nator
16:15:48 <flaper87> kgriffs: yeah, that too!
16:15:56 <kgriffs> ok, rock on
16:16:02 <kgriffs> anything else on this topic?
16:16:06 <flaper87> not from mee!
16:16:23 <kgriffs> #action flaper87 to firm up cross-transport-api-spec
16:16:42 <kgriffs> alcabrera: want to investigate heat?
16:17:01 <alcabrera> lemme see...
16:17:06 <alcabrera> not this go around.
16:17:08 <kgriffs> sometime in the next 2 weeks
16:17:11 <alcabrera> My eyes are still deep into sharding.
16:17:17 <alcabrera> hmmm
16:17:22 <alcabrera> next week, I'll do it. :D
16:17:25 <kgriffs> ok
16:17:25 <alcabrera> Yeah, sign me up.
16:17:27 <flaper87> alcabrera: THANKS!
16:17:32 <flaper87> that would be really cool
16:17:36 <alcabrera> I've been curious about Heat for some time.
16:17:47 <flaper87> I'm sorry for not working on that
16:17:52 <kgriffs> #action alcabrera to research ways we can use heat to deploy marconi itself, as well as provision queues, whatever
16:18:00 <alcabrera> flaper87: no worries. :)
16:18:09 <kgriffs> no worries, we need to spread the love
16:18:11 * alcabrera got an action this time
16:18:16 <kgriffs> can't have flaper87 doing *all* the work
16:18:20 <kgriffs> (just most of it)
16:18:20 <alcabrera> hehe
16:18:22 <kgriffs> ;)
16:18:25 <alcabrera> :D
16:18:34 <zyuan> i'd like to work on msgpack media type support
16:18:55 <kgriffs> #topic Upd@73$ on 5H@rDinG
16:19:01 <flaper87> LOOL
16:19:08 <flaper87> alcabrera: go go go go go go go!
16:19:11 <alcabrera> kkk
16:19:13 <alcabrera> soooo
16:19:15 <kgriffs> zyuan: that would be cool, let's discuss in a bit
16:19:28 <alcabrera> It's been some fierce rebasing last week.
16:19:36 <alcabrera> I think most of the core ideas have settled into place.
16:20:01 <alcabrera> Some of the biggest updates include a distinction between control storage drivers and data storage drivers being made, which cleaned up a lot of the abstarction.
16:20:16 <alcabrera> The other side of the fence was in the transport layer, where we have an admin API and a public API.
16:20:34 <alcabrera> The admin is truly a *super* API, meaning, an admin instance can do all a public instance can do (and more).
16:20:57 <alcabrera> That more, in the case of the sharding API, is being able to register shards and investigate the state of the project/queue mapping catalogue.
16:21:12 <alcabrera> Other news - patches on that admin API are starting to get merged in.
16:21:33 <alcabrera> The remaining patches are mostly reviewed, and quite nearly ready for merging.
16:21:36 <alcabrera> After that...
16:21:56 <alcabrera> It's a matter of hooking the control layer of storage up to queues.
16:21:59 <alcabrera> Maybe add some caching.
16:22:01 <alcabrera> Test it.
16:22:03 <alcabrera> Play with it.
16:22:05 <alcabrera> And watch it zoom.
16:22:08 <alcabrera> It's going to be fun.
16:22:12 <kgriffs> w00t
16:22:20 <kgriffs> thanks for all the hard work
16:22:22 <flaper87> awesome, awesome, awesome WORK!
16:22:25 <kgriffs> it's been quite a dance
16:22:27 <kgriffs> http://ln-s.net/:MsM
16:22:34 <alcabrera> quite the dance. indeed. :D
16:22:40 <alcabrera> We even have a thoughtstream tpo show for it.
16:22:45 <flaper87> seriously, i know it's been kinda painful this whole proxy / sharding thing
16:22:49 <alcabrera> #link https://thoughtstreams.io/combined/marconi-progress-and-updates/
16:23:02 <kgriffs> yeah
16:23:26 <alcabrera> sometimes finding what works can be a little painful. I think we're on the right path.
16:23:28 <alcabrera> Experiments.
16:23:30 <alcabrera> For science!
16:23:31 <kgriffs> lesson learned: draw more pretty pictures, do more prototypes
16:23:50 <kgriffs> science indeed. :)
16:24:07 <kgriffs> ok, so lets rally around this feature and get it merged
16:24:11 <kgriffs> and tested
16:24:16 <kgriffs> and awesome-i-fied
16:24:27 <alcabrera> +1
16:24:30 <flaper87> +2
16:24:34 <kgriffs> anything else before we move on?
16:24:49 <alcabrera> I think the rest we can figure out as we go over at #marconi
16:25:01 <alcabrera> I vote: next topic
16:25:16 <kgriffs> #topic UPD@73s oN 8Ug$
16:25:33 <kgriffs> malini: ?
16:25:47 <malini> We need to discuss a couple
16:25:57 <kgriffs> #link http://goo.gl/OlfzxC
16:26:08 <malini> https://bugs.launchpad.net/marconi/+bug/1243752
16:26:39 <malini> zyuan: You had some thoughts on this , rt?
16:26:48 <flaper87> #link  https://bugs.launchpad.net/marconi/+bug/1243752
16:27:24 <malini> zyuan here?
16:27:30 <zyuan> yes
16:27:40 <zyuan> i agree that 204 is not intuiitive
16:27:53 <zyuan> but 404 requires more work
16:28:17 <zyuan> as shown by one of kurt's commit
16:28:20 <alcabrera> To make that 404 efficient, it would take a little bit of caching, IMO.
16:28:35 <kgriffs> yes
16:28:43 <kgriffs> i don't want to do it without oslo.cache
16:28:47 <zyuan> i don't think that's nessarsary; user can try head a queue too
16:29:03 <kgriffs> btw, using oslow.cache for checking whether queue exists will speed up the other place(s) we do it already
16:30:02 <zyuan> for the places we place the checking
16:30:04 <flaper87> kgriffs: +1 that's something we have in the queue of: 'to cache' things
16:30:37 <zyuan> cache may result in more race conditions
16:30:37 <zyuan> like, put messages into another queue...
16:30:53 <zyuan> temp file problem
16:31:36 <kgriffs> also, once I get around to doing the LRU driver that acts like a CPU's L1+L2 hierarchy, it will be insanely fast.
16:32:07 <alcabrera> it's the eventual consistency problem. We'd have to decide what consistency guarantees we want to uphold, and where it matters.
16:32:10 <alcabrera> zyuan: ^^
16:32:11 <kgriffs> so, the race condition results in a 204 - like, a queue was deleted right after we checked for existence
16:32:23 <zyuan> cache is fast, i'm saying it may not be c correct
16:32:30 <kgriffs> and in that case, we move forward, assuming the cache is there
16:32:43 <kgriffs> thing is, that doesn't seem to be any worse that what happens now
16:33:10 <zyuan> hard to say...
16:33:48 <zyuan> and,
16:33:50 <kgriffs> except, i guess we might set the expectation for clients that if they get 204 the queue must exist
16:34:10 <kgriffs> we would have to say "if you get a 204, the queue is empty, or may have just been deleted"
16:34:10 <zyuan> i think db brings eventual consistency not for nothing
16:34:12 <kgriffs> or something
16:34:26 <zyuan> i don't think it's something *we* can try to workaround
16:34:47 <zyuan> if we can, why db does not?
16:34:59 <kgriffs> zyuan: true, if the data store is eventually consistent, the "exists" check isn't guaranteed anyway
16:35:28 <kgriffs> so the bug is more like, should we make a best effort to check?
16:35:40 <kgriffs> flaper87: what do u think?
16:35:59 <flaper87> sorry, stupid thing got stucked!
16:36:03 <zyuan> this "problen" is rare
16:36:23 <malini> zyuan: agrre with the rare part
16:36:32 <zyuan> we assume queues are not deleted very often
16:36:44 <zyuan> if so, then what we "best effect" do is just
16:36:55 <zyuan> transfer to one genrentee to another
16:37:01 <kgriffs> clarification: the race condition is rare, or the use case of querying a non-existing queue?
16:37:14 <zyuan> i don't see the benefit
16:37:25 <zyuan> use case is rare as well as the race condition
16:37:27 <malini> kgriffs: the use- case
16:37:42 <flaper87> TBH, I'd like us, at some point, to treat queues as lazy resources. Meaning, create them when a message is post (for example)
16:37:54 <flaper87> that doesn't fix the issue we're discussing, though.
16:37:57 <kgriffs> would we create them on a GET as well?
16:37:58 <zyuan> race condition... actually i don't know. but i hope cache don't make the situation worse
16:37:59 <alcabrera> hmmm
16:38:09 <malini> kgriffs: :D
16:38:17 <zyuan> .. you know, it's like we are fixing something db can not fix
16:38:19 <flaper87> kgriffs: nope
16:38:47 <kgriffs> ok, here is what I suggest
16:39:00 <kgriffs> wait until we get another complaint or two
16:39:03 <alcabrera> I'm not sure about create on write, but that's one to discuss for another time, flaper87. Interesting thought, though. :)
16:39:09 <kgriffs> if we don't hear anything, then we can assume nobody cares
16:39:20 <flaper87> kgriffs: +1
16:39:22 <alcabrera> kgriffs: sounds fair enough to me.
16:39:39 <alcabrera> It's trippy for new users, but may not be a dire issue.
16:39:41 <flaper87> alcabrera: yeah, TBH, taken from mongodb and how it treats collections
16:39:44 <malini> how do we keep track -so move the bug to a lower priority ?
16:39:49 <zyuan> since users just started to try (to break) marconi, i expect such complain will come to us soon
16:39:57 <flaper87> they don't exist until something is written into it
16:40:12 <flaper87> zyuan: agreed
16:40:42 <flaper87> malini: I think we could close the bug as wontfix and re-open it later if necessary
16:40:43 <zyuan> anyway
16:40:50 <flaper87> or invalid
16:40:54 <flaper87> or something like that
16:40:57 <kgriffs> flaper87: FWIW, the lazy create is how RSE worked (marconi's predecessor)
16:41:14 <malini> flaper87: problem is we are not going to remember this discussion when somebody else complains :(
16:41:17 <kgriffs> you never created or deleted queues
16:41:40 <kgriffs> I'll put a note on the bug
16:41:43 <flaper87> TBH, I think that's how it should be
16:41:47 <flaper87> kgriffs: ^
16:41:48 <alcabrera> malini: it would have to be closed with rationale - good point.
16:41:57 <zyuan> haha
16:41:59 <flaper87> it may not be relevant for some storage but it is for others
16:42:32 <flaper87> kgriffs: well, I'd say delete yes
16:42:43 <flaper87> but we could definitely make creation easier
16:42:51 <kgriffs> in RSE, queues didn't exist as a standalone resource
16:42:56 <kgriffs> so there was nothing to delete
16:43:01 <kgriffs> a "queue" was just a namespace
16:43:04 <kgriffs> aaaanyway
16:43:09 <zyuan> itis related to both mongo and sql
16:43:13 <flaper87> kgriffs: yeah, by delete a queue I mean: A way to remove all those messages
16:43:26 <kgriffs> malini: any other bugs to bring up?
16:43:31 <zyuan> sql cares about relation between sets
16:43:37 <kgriffs> flaper87: gotchya
16:43:42 <zyuan> if queue sets is empty, intersaction is also empty
16:43:46 <zyuan> silently
16:43:57 <malini> no more bugs to discuss..But we have a lot to set priority on
16:44:03 <flaper87> malini: agreed, but if we don't we'll keep a bug around that may create some confussion
16:44:14 <flaper87> I'm happy with either!
16:44:29 <kgriffs> malini: let's triage bugs in a breakout after this
16:44:33 <malini> ok
16:44:56 <alcabrera> +1
16:44:59 <zyuan> i would suggest if the compaint comes to us again, discuss whether we need to update the doc
16:45:15 <alcabrera> next up - triaging BPs?
16:45:55 <malini> flaper87: we need a way to keep track of user feedback -  this does not fit into marconi v2 suggestion wiki.
16:46:12 <zyuan> yea
16:46:18 <flaper87> malini: yeah, I wonder what other projects are doing here
16:46:31 <zyuan> issues?
16:46:35 <alcabrera> malini: +1
16:46:36 <flaper87> will send an email later.
16:46:38 <zyuan> github ones?
16:47:12 <flaper87> We should have a general suggestion wiki then
16:47:14 <kgriffs> ok, let's triage blueprints in a breakout as well
16:47:32 <zyuan> wow, i love this BP, encrypt user data
16:47:39 <alcabrera> kk
16:47:40 <zyuan> alought it's very old
16:48:01 <alcabrera> Review API v1 feedback?
16:48:16 <kgriffs> #topic r3vi3w @PI v1 f33D8@ck
16:48:34 <kgriffs> #link https://wiki.openstack.org/wiki/Marconi/specs/api/next
16:48:52 <kgriffs> So, as you may know, I've been collecting feedback from users on this page
16:49:35 <zyuan> top to down?
16:49:39 <alcabrera> yup - good work, kgriffs! This is a good start to tracking v1.1 ideas.
16:49:50 <flaper87> kgriffs: +2
16:50:11 <kgriffs> sure, let me just give a quick background on these and we can start thinking about which ones we want to do and stuff
16:50:12 <flaper87> There's a session where we'll be reviewing them, isn't there?
16:50:16 <flaper87> or something like that
16:51:04 <kgriffs> kind of
16:51:22 <kgriffs> i think we want to pull in a few things for a v1.1 api bump in icehouse
16:51:27 <kgriffs> # link http://icehousedesignsummit.sched.org/event/72e05db820e4e7ca8eebf19a1260de64
16:51:29 <kgriffs> #link http://icehousedesignsummit.sched.org/event/72e05db820e4e7ca8eebf19a1260de64
16:51:57 <kgriffs> but, we will need to do an unconference or something before then to get it down to a small list of proposed API changes
16:52:13 <flaper87> totally!
16:52:19 <kgriffs> ok
16:52:25 <flaper87> We should review them all together and then go to the session
16:52:30 <kgriffs> +1
16:52:33 <kgriffs> so, real quick
16:52:38 <flaper87> shoot
16:52:45 <kgriffs> auto-gen client UUID if not given
16:53:10 <kgriffs> this was proposed by a user who said it would be nice to have in two cases
16:53:29 <kgriffs> first, when doing simple job queueing with marconi, in which case client ID isn't really needed
16:53:47 <zyuan> oh
16:53:55 <kgriffs> second, when client isn't using a queue as a duplex pipe
16:54:23 <kgriffs> that being said, I think always having client-uuid could help surface interesting stats
16:54:29 <zyuan> what does 2 mean?
16:54:39 <kgriffs> and also could be logged to facilitate ops/support for users
16:55:13 <kgriffs> 2 means, i could just have a "send" queue and a separate "receive" queue and then I wouldn't have the message echo problem
16:55:20 <flaper87> mmhh
16:55:34 <flaper87> I'd expect people to use library clients
16:55:38 <kgriffs> IMO, however, having echo cancellation can really simplify system architecture
16:55:44 <flaper87> and the library could generate the uuid
16:55:50 <kgriffs> yep
16:55:56 <kgriffs> ok, not a lot of time...
16:55:58 <kgriffs> moving on
16:56:01 <kgriffs> actually
16:56:18 <kgriffs> so, next item is related, so I will skip
16:56:18 <flaper87> what if we take 30mins everyday this week to discuss these itesm
16:56:23 <flaper87> or something like that?
16:56:36 <kgriffs> next item someone said it was a pain for them to make a UUID but I think they were just using curl or something
16:56:53 <kgriffs> and they couldn't give me a good example of an alternative ID a client might want to use
16:56:53 <alcabrera> flaper87: +1
16:56:55 <flaper87> I think we would benefit of several really small meetings this week.
16:57:02 <kgriffs> flaper87: sounds good
16:57:05 <kgriffs> lots to do
16:57:05 <zyuan> haha, +1
16:57:08 <alcabrera> I'm in favor of small meetings on this subject.
16:57:11 <kgriffs> so, 11 each day?
16:57:19 <flaper87> 11 EDT ?
16:57:20 <alcabrera> Also, same goes for BPs triaging.
16:57:21 <kgriffs> sorry
16:57:23 <kgriffs> 1600 UTC
16:57:31 <flaper87> sounds good!
16:57:35 <alcabrera> woot
16:57:39 <kgriffs> but in #openstack-marconi
16:57:43 <flaper87> yup
16:57:44 <alcabrera> 11EDT works wonderfully for me.
16:57:47 <kgriffs> kk
16:58:00 <kgriffs> alcabrera: I think it is actually 12 your time
16:58:00 <flaper87> Open discussion ?
16:58:01 <zyuan> ok
16:58:21 <kgriffs> https://duckduckgo.com/?q=1600+utc+in+edt
16:58:34 <alcabrera> kgriffs: still works. :D
16:58:35 <kgriffs> #topic open discussion
16:58:39 <flaper87> o/
16:58:51 <alcabrera> zyuan: you wanted to bring up msgpack?
16:58:57 <zyuan> yes
16:59:04 <flaper87> zyuan: you go
16:59:11 <flaper87> mine is just a small comment
16:59:12 <zyuan> not sure how deep though
16:59:28 <zyuan> (i mean, not sure wehther need to change falcon)
17:01:10 <kgriffs> zyuan: error message responses - I have been meaning to make those hookable, so apps can customize the response
17:01:21 <zyuan> kgriffs: exactly...
17:01:41 <zyuan> that's what i mean "how deep"
17:01:42 <zyuan> i'll think about this
17:01:50 <kgriffs> but anyway, I guess we need to balance working on that with other stuff. I think msgpack support could be a potential for v1.1 API
17:02:15 <flaper87> perhaps once we add support for zmq as well
17:02:23 <kgriffs> but, let's make sure we have sharding and high-priority bugs covered first
17:02:40 <zyuan> msgpack does not break 1.0 API, afaics
17:03:06 <flaper87> zyuan: agreed!
17:03:14 <kgriffs> zyuan: we could decide to sneak it it but not document it until 1.1
17:03:26 <kgriffs> client maintainers just want to feel like the API is stable
17:03:31 <zyuan> +0.8
17:03:42 <zyuan> +1
17:03:43 <kgriffs> i think this could also be a good POC for extensions
17:03:53 <zyuan> yea
17:03:59 <zyuan> on bugs...
17:04:18 <zyuan> continue on openstack-marconi?
17:04:23 <kgriffs> mmm
17:04:29 <flaper87> just one small comment before we wrap up
17:04:30 <kgriffs> I just realized I need to run
17:04:46 <alcabrera> flaper87: shoot
17:05:13 <flaper87> as you know next week is the Summit and we'll attend and present our very super cool and young API v1, perhaps with some parts of the client working
17:05:23 <flaper87> now, we need to impress people a bit
17:05:47 <flaper87> so, if you guys have ideas, thoughts or things we could bring up
17:05:59 <flaper87> for an un-conference, sprint or whatever and get more people interested
17:06:26 <flaper87> please, lets work on that
17:06:36 <alcabrera> flaper87: +1
17:06:46 <flaper87> it's a great opportunity to make our team bigger
17:06:51 <kgriffs> +!
17:06:54 <kgriffs> +1
17:06:58 <alcabrera> agreed. Let's make our summit efforts spetacular!
17:07:10 * flaper87 STFU now!
17:07:14 <kgriffs> generally, kudos to flaper87 for his work in building the marconi community
17:07:37 * flaper87 blushes
17:07:51 <kgriffs> i like the sprint idea
17:07:59 <kgriffs> that will help too, and give us a chance to pass out poptarts
17:08:09 <flaper87> kgriffs: awesome, I was thinking some time on Wednesday or Thursday
17:08:16 <kgriffs> we can also just invite people to come hang out and ask questions, give feedback
17:08:24 <flaper87> to make people curious and help them think on new ideas to bring up during the sessions
17:08:41 <kgriffs> maybe we just grab an unconference slot for Q&A or something
17:08:49 <kgriffs> ah, that
17:08:52 <kgriffs> what you said. :D
17:09:00 <kgriffs> hmmm. We need t-shirts
17:09:02 <alcabrera> announce an o'reilly book: Marconi Up and Running. ;)
17:09:07 <alcabrera> hehehe
17:09:08 <kgriffs> LOL
17:09:08 <flaper87> kgriffs: LOL
17:09:20 <kgriffs> hmmm. we need a logo
17:09:26 <flaper87> +1 t-shirts
17:09:32 <alcabrera> +1 t-shirts - also, I want one.
17:09:42 <kgriffs> #action kgriffs to see about t-shirts
17:09:46 <alcabrera> yes
17:09:58 <kgriffs> ok, any last words before we end the mtg?
17:10:03 <alcabrera> #info Marconi needs a logo and t-shirts
17:10:05 <malini> bye-bye
17:10:11 <alcabrera> o/
17:10:18 <kgriffs> Liv3 long, anD Pr0$P3R
17:10:22 <kgriffs> #endmeeting