*** rnirmal has quit IRC | 00:15 | |
*** vipul is now known as vipul|away | 00:20 | |
*** robertmyers has joined #openstack-meeting-alt | 00:25 | |
*** vipul|away is now known as vipul | 00:28 | |
*** juice has quit IRC | 00:38 | |
*** kaganos has quit IRC | 00:38 | |
*** juice has joined #openstack-meeting-alt | 00:54 | |
*** esp has joined #openstack-meeting-alt | 01:00 | |
*** esp has left #openstack-meeting-alt | 01:07 | |
*** bdpayne has quit IRC | 01:28 | |
*** amyt has quit IRC | 01:47 | |
*** kaganos has joined #openstack-meeting-alt | 02:54 | |
*** jog0 has joined #openstack-meeting-alt | 02:58 | |
*** grapex has joined #openstack-meeting-alt | 03:03 | |
*** jog0 has quit IRC | 03:36 | |
*** kaganos has quit IRC | 04:09 | |
*** robertmy_ has joined #openstack-meeting-alt | 04:18 | |
*** robertmyers has quit IRC | 04:20 | |
*** juice has quit IRC | 04:29 | |
*** amyt has joined #openstack-meeting-alt | 04:32 | |
*** grapex has quit IRC | 04:55 | |
*** juice has joined #openstack-meeting-alt | 05:01 | |
*** kaganos has joined #openstack-meeting-alt | 05:36 | |
*** kaganos has quit IRC | 05:56 | |
*** juice has quit IRC | 06:35 | |
*** juice has joined #openstack-meeting-alt | 06:38 | |
*** juice has quit IRC | 07:29 | |
*** juice has joined #openstack-meeting-alt | 07:35 | |
*** juice has quit IRC | 07:57 | |
*** juice has joined #openstack-meeting-alt | 07:58 | |
*** juice has quit IRC | 08:36 | |
*** amyt has quit IRC | 13:38 | |
*** robertmy_ has quit IRC | 13:51 | |
*** jog0 has joined #openstack-meeting-alt | 14:33 | |
*** grapex has joined #openstack-meeting-alt | 14:34 | |
*** grapex has joined #openstack-meeting-alt | 14:34 | |
*** robertmyers has joined #openstack-meeting-alt | 14:48 | |
*** robertmyers has joined #openstack-meeting-alt | 14:48 | |
*** amyt has joined #openstack-meeting-alt | 14:54 | |
*** rnirmal has joined #openstack-meeting-alt | 14:56 | |
*** djohnstone has joined #openstack-meeting-alt | 15:06 | |
*** jog0 has left #openstack-meeting-alt | 15:08 | |
*** jcru has joined #openstack-meeting-alt | 15:09 | |
*** cp16net is now known as cp16net|away | 15:19 | |
*** juice has joined #openstack-meeting-alt | 15:21 | |
*** jcru is now known as jcru|away | 15:26 | |
*** juice has quit IRC | 15:33 | |
*** esp1 has joined #openstack-meeting-alt | 15:59 | |
*** esp1 has left #openstack-meeting-alt | 15:59 | |
*** juice has joined #openstack-meeting-alt | 16:04 | |
*** cp16net|away is now known as cp16net | 16:16 | |
*** jcru|away is now known as jcru | 16:16 | |
*** juice has quit IRC | 16:19 | |
*** kmansel has quit IRC | 17:05 | |
*** kaganos has joined #openstack-meeting-alt | 17:06 | |
*** juice has joined #openstack-meeting-alt | 17:20 | |
*** bdpayne has joined #openstack-meeting-alt | 17:28 | |
*** esp1 has joined #openstack-meeting-alt | 17:52 | |
*** esp1 has left #openstack-meeting-alt | 17:52 | |
*** bdpayne has quit IRC | 18:29 | |
*** bdpayne has joined #openstack-meeting-alt | 18:30 | |
*** edsrzf has joined #openstack-meeting-alt | 18:50 | |
*** kgriffs has joined #openstack-meeting-alt | 18:52 | |
*** treeder has joined #openstack-meeting-alt | 18:57 | |
*** chad has joined #openstack-meeting-alt | 18:58 | |
*** chad is now known as carimura | 18:58 | |
*** carimura has quit IRC | 18:59 | |
*** chad has joined #openstack-meeting-alt | 18:59 | |
*** chad is now known as carimura | 19:00 | |
*** kgriffs1 has joined #openstack-meeting-alt | 19:04 | |
kgriffs1 | #startmeeting | 19:04 |
---|---|---|
openstack | kgriffs1: Error: A meeting name is required, e.g., '#startmeeting Marketing Committee' | 19:04 |
kgriffs1 | sorry guys. My IRC client is being lame. | 19:05 |
kgriffs1 | #startmeeting Marconi | 19:05 |
openstack | Meeting started Thu Jan 24 19:05:11 2013 UTC. The chair is kgriffs1. Information about MeetBot at http://wiki.debian.org/MeetBot. | 19:05 |
openstack | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 19:05 |
*** openstack changes topic to " (Meeting topic: Marconi)" | 19:05 | |
openstack | The meeting name has been set to 'marconi' | 19:05 |
kgriffs1 | #topic Review last week's actions | 19:05 |
*** openstack changes topic to "Review last week's actions (Meeting topic: Marconi)" | 19:05 | |
*** robertmyers has left #openstack-meeting-alt | 19:05 | |
kgriffs1 | #info kgriffs to kick the tires on Pecan | 19:05 |
kgriffs1 | So, I did play around a bit with Pecan. | 19:06 |
*** kgriffs has quit IRC | 19:06 | |
kgriffs1 | It has a lot of cool stuff, but is a bit slow for my tastes. | 19:06 |
kgriffs1 | We are doing a hack day today in my office and some people are playing with various web frameworks, including Falcon. | 19:07 |
kgriffs1 | I'm hoping to get some good feedback from everyone to see what they like and don't like about each framework, to feed into the discussion around what we should use for Marconi. | 19:08 |
treeder | with a tagline like that, i'm surprised. ;) | 19:08 |
kgriffs1 | At this point, Falcon is still a bit of a science project, so the jury's still out. | 19:09 |
kgriffs1 | heh | 19:09 |
treeder | referring to Pecan's tagline | 19:09 |
kgriffs1 | exactly. | 19:10 |
kgriffs1 | I did a fairly simple benchmark where I set a body, read a single query parameter, and also have a param as part of the URI path. Pecan and Flask were about the same speed, while Bottle was several times faster per request, and Falcon was about twice as fast as Bottle. | 19:11 |
kgriffs1 | Granted, Falcon does less than the other frameworks, but at least you don't have to pay for what you don't use. Or something like that. | 19:11 |
kgriffs1 | We'll have to see how things go with more realistic tests. | 19:11 |
kgriffs1 | Anyway, that something simmering on the back-burner while we figure out the API. | 19:12 |
treeder | Can you post link to Falcon? | 19:12 |
treeder | and what was yours again? | 19:12 |
kgriffs1 | http://github.com/racker/falcon | 19:12 |
kgriffs1 | BTW, the benchmark was hitting the WSGI callable directly, not going through any webserver. | 19:13 |
kgriffs1 | We're talking about a performance range of 2 thousand req/sec vs 30 for Falcon on my MBP. | 19:14 |
kgriffs1 | Kinda crazy. I just don't want the framework to be the bottleneck in terms of latency. | 19:15 |
treeder | ya, that's pretty significant | 19:15 |
kgriffs1 | #topic Review the g1 milestone | 19:17 |
*** openstack changes topic to "Review the g1 milestone (Meeting topic: Marconi)" | 19:17 | |
kgriffs1 | So, I just wanted to take a minute and review what our first milestone looks like. It has "g" in the name for "Grizzly" - I'm hoping to get at least a few of these done before the next summit. | 19:18 |
kgriffs1 | https://launchpad.net/marconi/+milestone/grizzly-1 | 19:18 |
*** SlickNik has left #openstack-meeting-alt | 19:18 | |
kgriffs1 | #info Set up Marconi wiki, repo, Launchpad, etc. | 19:18 |
kgriffs1 | That's pretty much all done at this point. The Wiki could use some refactoring, but it's there. | 19:19 |
kgriffs1 | #info Define v1.0 functional and non-functional requirements | 19:19 |
kgriffs1 | I've got a rough draft of this that is mostly based on the meeting we had last summit to kick off the project. | 19:20 |
kgriffs1 | http://wiki.openstack.org/marconi/specs/grizzly | 19:20 |
kgriffs1 | In a future meeting we should refine it, but it's probably best to get some code out there rather than spend too much time writing specs. | 19:20 |
*** cp16net is now known as cp16net|away | 19:21 | |
treeder | +1 | 19:21 |
treeder | 1) define MVP API 2) code | 19:21 |
kgriffs1 | #info Define v1.0 API | 19:21 |
kgriffs1 | +1 | 19:21 |
kgriffs1 | So, the API draft is out there but has a bunch of TBD on it that I'm hoping we can work through pretty quickly. We started last week, and will keep going today. | 19:23 |
kgriffs1 | #info Create a skeleton implementation and test suite | 19:23 |
kgriffs1 | So, this will probably go into the next milestone. I'm thinking we will just close out g1 once we have the API (mostly) figured out. | 19:24 |
kgriffs1 | (also I've got some interviews coming up in the next couple weeks to hire some full-time devs) | 19:25 |
kgriffs1 | #action kgriffs1 to move coding to g2 | 19:25 |
kgriffs1 | Any questions/comments before we move on? | 19:26 |
kgriffs1 | #topic Decide whether to support tags or some other fanout mechanism, or just stick with N-queue approach (for v1.0 at least) | 19:27 |
*** openstack changes topic to "Decide whether to support tags or some other fanout mechanism, or just stick with N-queue approach (for v1.0 at least) (Meeting topic: Marconi)" | 19:27 | |
kgriffs1 | So, what do you guys think is the way to go for v1.0? | 19:27 |
treeder | what's N-queue approach? | 19:28 |
edsrzf | I think that's your preferred approach | 19:28 |
kgriffs1 | What I mean by that is to, say, use two queues rather than nested namespaces. | 19:29 |
treeder | ahh, ya | 19:29 |
kgriffs1 | Also, not having tags means all filtering happens either by using multiple queues or doing it in the client firehose-style. | 19:29 |
treeder | N-queue approach is definitely the simplest | 19:29 |
kgriffs1 | +1 | 19:30 |
treeder | I also think we should have concrete queues regardless of whether tags are added on | 19:30 |
kgriffs1 | #agreed Have concrete queues | 19:30 |
treeder | So as a base point, I think the N-queues approach is best | 19:30 |
treeder | fanout and tags can be added on top | 19:30 |
kgriffs1 | Any objections? | 19:31 |
*** rnirmal has quit IRC | 19:33 | |
kgriffs1 | #agreed No tags or fanout for v1 | 19:34 |
treeder | brb | 19:34 |
kgriffs1 | OK | 19:34 |
kgriffs1 | #topic Consider splitting the API into two namespaces, one for work queuing and one for eventing | 19:36 |
*** openstack changes topic to "Consider splitting the API into two namespaces, one for work queuing and one for eventing (Meeting topic: Marconi)" | 19:36 | |
kgriffs1 | A few Pros off the top of my head: | 19:37 |
kgriffs1 | #info Pro: Allows for different scaling architectures, storage backends, etc. | 19:37 |
kgriffs1 | #info Pro: Simplifies semantics for the two major use cases | 19:37 |
kgriffs1 | #info Con: Remove affordances - clients won't be able to mix/match semantics (we would be effectively removing features) | 19:38 |
kgriffs1 | Thoughts? | 19:38 |
kgriffs1 | By "eventing", I mean notifications and RPC style communication | 19:39 |
kgriffs1 | By work queuing, I mean semantics like those provided by IronMQ and SQS. | 19:40 |
edsrzf | So are we talking about separate semantics, or separate namespaces? | 19:42 |
kgriffs1 | The API spec as currently written lets you do all of the above under a single namespace. The way the client interacts with a given queue determines the contract, essentially. | 19:43 |
kgriffs1 | http://wiki.openstack.org/marconi/specs/api/v1 | 19:43 |
kgriffs1 | So, I could have a queue with work items or metering data documents in it, and have consumers pulling those off in a way that hides those documents from other consumers (similar to the way SQS works). | 19:44 |
kgriffs1 | But an observer could peek at the messages on the queue if they wanted to - it would be up to the developer to ensure the observer is passive. | 19:45 |
*** vipul is now known as vipul|away | 19:45 | |
*** vipul|away is now known as vipul | 19:45 | |
*** vipul is now known as vipul|away | 19:46 | |
kgriffs1 | The alternative would be to basically have two different queue types and clients aren't able to mix-and-match semantics. | 19:46 |
treeder | what's the alternative way? | 19:47 |
kgriffs1 | I guess you could even have two completely different services for each type. | 19:47 |
treeder | a consumer could take a message and it wouldn't be hidden? | 19:47 |
kgriffs1 | They couldn't take a message per say, but they could peek and promise to only be a passive observer. | 19:48 |
treeder | right, so that's essentially just a different function/endpoint | 19:50 |
*** cp16net|away is now known as cp16net | 19:50 | |
treeder | GET vs PEEK | 19:50 |
treeder | peek doesn't reserve the message | 19:50 |
kgriffs1 | as written, the spec would allow PEEKing at messages that are reserved AKA hidden/locked. | 19:51 |
treeder | where is that in the spec? | 19:51 |
treeder | can't find it | 19:51 |
kgriffs1 | sorry, it's a ways down. Search for "Lock Messages" | 19:52 |
kgriffs1 | http://wiki.openstack.org/marconi/specs/api/v1 | 19:52 |
treeder | oh, so GET messages is like peek | 19:52 |
treeder | and lock is more like a get? | 19:53 |
kgriffs1 | yeah. I was attempting to keep state on the client as much as possible. | 19:53 |
treeder | in terms of SQS/beanstalk/IronMQ terms | 19:53 |
kgriffs1 | right. But GET isn't really supposed to change state. | 19:53 |
*** bdpayne has quit IRC | 19:53 | |
*** juice has quit IRC | 19:53 | |
kgriffs1 | I jut had a thought. The body of the post could be a list of queues to pull from. That reduces load on the server. | 19:55 |
kgriffs1 | Although that means we would probably want a batch GET across multiple queues, and that could get weird. | 19:56 |
*** jcru has quit IRC | 19:59 | |
kgriffs1 | SQS gives you back some kind of transaction ID or something for each batch of messages, right? | 19:59 |
kgriffs1 | (it's been a little while since I looked at it) | 19:59 |
kgriffs1 | (looking at docs) | 19:59 |
*** djohnstone_ has joined #openstack-meeting-alt | 20:00 | |
kgriffs1 | oic. | 20:00 |
kgriffs1 | ReceiptHandle | 20:00 |
kgriffs1 | It's per message | 20:00 |
treeder | don't think there's a transaction | 20:00 |
treeder | ya | 20:00 |
kgriffs1 | What does IronMQ do? | 20:00 |
treeder | in terms of? | 20:02 |
treeder | lock vs peek? | 20:02 |
treeder | very similar to SQS | 20:02 |
treeder | GET messages locks/reserves messages for the timeout period | 20:02 |
treeder | we also have a /peek endpoint | 20:02 |
kgriffs1 | no, I mean, is their some kind of handle for, e.g., renewing the timeout? | 20:02 |
*** djohnstone has quit IRC | 20:02 | |
treeder | each message has an id | 20:02 |
kgriffs1 | # IronMQ has a /peek endpoint | 20:03 |
treeder | we have touch/release for renewing the lock | 20:03 |
treeder | or releaseing the lock | 20:03 |
kgriffs1 | oic | 20:03 |
treeder | releasing | 20:03 |
*** djohnstone has joined #openstack-meeting-alt | 20:03 | |
kgriffs1 | Do you think releasing a lock manually is something Marconi should have? | 20:03 |
treeder | probably | 20:03 |
kgriffs1 | #IronMQ has touch/release for renewing the lock | 20:04 |
treeder | use case is: "i got a message that I can't deal with right now so I'll put it back on the queue for someone else" | 20:04 |
kgriffs1 | oic | 20:04 |
treeder | and you can release with a delay too meaning it won't be available for a while | 20:04 |
treeder | just as you can post a message with a delay | 20:04 |
*** djohnstone_ has quit IRC | 20:05 | |
kgriffs1 | So it seems like doing the locks per-message is better than as a batch? | 20:05 |
treeder | touch resets the timeout | 20:05 |
treeder | hmmmm | 20:05 |
kgriffs1 | or does it matter? | 20:05 |
treeder | i don't know if there's any sane way to batch them? | 20:05 |
kgriffs1 | I guess you just assign the same lock ID to a group of messages. | 20:06 |
kgriffs1 | (not normalized) | 20:06 |
treeder | ok | 20:06 |
treeder | so if I take 100 messages, I have to deal with them all, no matter what? | 20:06 |
treeder | like I can't fail on one message | 20:07 |
treeder | without affecting the whole batch | 20:07 |
kgriffs1 | Well, if you fail on one and keep going, that one will eventually time out and go back into the pot | 20:07 |
treeder | just thinking out loud | 20:07 |
kgriffs1 | or you could manually release it | 20:07 |
treeder | if the lock is on a batch though | 20:07 |
treeder | wouldn't the whole batch have to go back on the pot | 20:07 |
treeder | ? | 20:07 |
kgriffs1 | oic. You would keep renewing until you are done with the entire batch, in the meantime another worker could have been picking up the ones you failed/skipped. | 20:08 |
treeder | eg: | 20:09 |
treeder | server X grabs 100 messages | 20:09 |
treeder | server crashes after processing 50 messages | 20:09 |
treeder | lock expires and all 100 messages would have to go back on? | 20:09 |
kgriffs1 | no, the server would presumably have deleted the 50 messages before crashing | 20:10 |
treeder | ahh, right | 20:10 |
kgriffs1 | (by ID, not lock) | 20:10 |
treeder | got it, so it deletes by message id? | 20:10 |
kgriffs1 | yeah | 20:10 |
treeder | ok | 20:11 |
kgriffs1 | maybe also have to include the lock id for safety | 20:11 |
treeder | you say tags in the spec | 20:11 |
kgriffs1 | yeah | 20:11 |
treeder | is a tag a message id too? | 20:11 |
kgriffs1 | no, that's a holdover from the idea to not use concrete queues | 20:11 |
kgriffs1 | I need update it | 20:11 |
kgriffs1 | so, would be... | 20:11 |
kgriffs1 | POST {base_url}/messages/{queue}/locks{?limit} | 20:12 |
kgriffs1 | or something like that | 20:12 |
*** djohnstone has left #openstack-meeting-alt | 20:12 | |
kgriffs1 | oops | 20:12 |
kgriffs1 | POST {base_url}/{queue}/locks{?tags,limit} | 20:13 |
kgriffs1 | anyway, you get the idea - it will take some refinement | 20:13 |
kgriffs1 | dang. Sorry - left tags in again. /me blushes | 20:13 |
*** esp1 has joined #openstack-meeting-alt | 20:13 | |
kgriffs1 | So, I'm thinking that per-message receipt handles or whatever you call them is a little more flexible. | 20:15 |
treeder | ya, I think so | 20:16 |
treeder | message id's to keep it simple. ;) | 20:16 |
kgriffs1 | #agreed use per-message lock/receipt IDs or just message IDs directly | 20:16 |
kgriffs1 | I guess that's an argument for having separate queue types or namespaces or whatever. | 20:17 |
*** esp1 has quit IRC | 20:18 | |
kgriffs1 | On the other hand, having to provide a handle of some sort when deleting a message (in addition to it's ID) does mitigate a race condition between a lock expiring and a message being deleted. | 20:18 |
treeder | handle to a lock? | 20:19 |
kgriffs1 | Another option would be to require clients put a UUID string in the request (possibly User-Agent header), then when a client gets a batch, the messages are tagged with that UUID so only they can delete them if they are still locked. | 20:20 |
kgriffs1 | yeah | 20:20 |
kgriffs1 | #action kgriffs1 to update API spec to remove tags, add concrete queues | 20:21 |
treeder | need to think about that batch lock thing | 20:22 |
kgriffs1 | OK | 20:22 |
kgriffs1 | Well, the meeting is running pretty long | 20:22 |
kgriffs1 | maybe we should sleep on the locking as well as the question of whether to have a unified queue type/namespace. | 20:22 |
treeder | ya, well we're making progress so that's good. | 20:22 |
kgriffs1 | definitely. | 20:23 |
treeder | alright, I have to run to another meeting, talk to you next week. | 20:23 |
kgriffs1 | sounds like a plan | 20:23 |
treeder | maybe we should get a marconi mailing list going or something? | 20:24 |
kgriffs1 | Well, the way OpenStack seems to like to do it is put everything in openstack-dev with a tag in the subject line, i.e., [marconi]. | 20:25 |
kgriffs1 | #agreed Start some discussion threads on the mailing list | 20:25 |
kgriffs1 | appreciate everyone's help. | 20:25 |
kgriffs1 | #action everyone think about locking semantics | 20:26 |
kgriffs1 | #action everyone think about / discuss the implications of a unified queue type/namespace and come up with a recommendation for next mtg. | 20:27 |
kgriffs1 | #endmeeting | 20:27 |
*** openstack changes topic to "OpenStack meetings (alternate) || Development in #openstack-dev || Help in #openstack" | 20:27 | |
openstack | Meeting ended Thu Jan 24 20:27:32 2013 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 20:27 |
openstack | Minutes: http://eavesdrop.openstack.org/meetings/marconi/2013/marconi.2013-01-24-19.05.html | 20:27 |
openstack | Minutes (text): http://eavesdrop.openstack.org/meetings/marconi/2013/marconi.2013-01-24-19.05.txt | 20:27 |
openstack | Log: http://eavesdrop.openstack.org/meetings/marconi/2013/marconi.2013-01-24-19.05.log.html | 20:27 |
*** treeder has quit IRC | 20:29 | |
*** kgriffs1 has quit IRC | 20:35 | |
*** kgriffs has joined #openstack-meeting-alt | 20:35 | |
*** vipul|away is now known as vipul | 20:35 | |
*** carimura has quit IRC | 20:37 | |
*** esp1 has joined #openstack-meeting-alt | 20:38 | |
*** juice has joined #openstack-meeting-alt | 20:38 | |
*** chad has joined #openstack-meeting-alt | 20:41 | |
*** chad is now known as Guest13348 | 20:41 | |
*** carimura has joined #openstack-meeting-alt | 20:41 | |
*** esp1 has left #openstack-meeting-alt | 20:43 | |
*** cp16net is now known as cp16net|away | 20:44 | |
*** cp16net|away is now known as cp16net | 20:50 | |
*** grapex has quit IRC | 20:59 | |
*** bdpayne has joined #openstack-meeting-alt | 21:09 | |
*** jcru has joined #openstack-meeting-alt | 21:16 | |
*** edsrzf has left #openstack-meeting-alt | 21:28 | |
*** carimura has quit IRC | 21:34 | |
*** chad has joined #openstack-meeting-alt | 21:45 | |
*** chad is now known as Guest92828 | 21:45 | |
*** treeder has joined #openstack-meeting-alt | 21:46 | |
*** kgriffs has quit IRC | 22:16 | |
*** Guest92828 has quit IRC | 22:33 | |
*** treeder has quit IRC | 22:33 | |
*** kaganos has quit IRC | 22:45 | |
*** jcru is now known as jcru|away | 22:51 | |
*** juice has quit IRC | 22:55 | |
*** juice has joined #openstack-meeting-alt | 22:58 | |
*** juice has quit IRC | 23:01 | |
*** juice has joined #openstack-meeting-alt | 23:05 | |
*** jcru|away is now known as jcru | 23:08 | |
*** jcru has quit IRC | 23:20 | |
*** juice has quit IRC | 23:24 | |
*** juice has joined #openstack-meeting-alt | 23:26 | |
*** juice has quit IRC | 23:26 | |
*** juice has joined #openstack-meeting-alt | 23:27 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!