| *** kaganos has quit IRC | 00:11 | |
| *** kaganos has joined #openstack-meeting-alt | 00:13 | |
| *** kaganos has quit IRC | 00:18 | |
| *** amyt has quit IRC | 00:20 | |
| *** esp has joined #openstack-meeting-alt | 00:23 | |
| *** esp has left #openstack-meeting-alt | 00:23 | |
| *** cp16net is now known as cp16net|away | 02:07 | |
| *** cp16net|away is now known as cp16net | 02:07 | |
| *** robertmyers has joined #openstack-meeting-alt | 02:14 | |
| *** cp16net is now known as cp16net|away | 02:41 | |
| *** cp16net|away is now known as cp16net | 02:43 | |
| *** cp16net is now known as cp16net|away | 03:21 | |
| *** cp16net|away is now known as cp16net | 03:24 | |
| *** juice has quit IRC | 03:25 | |
| *** cp16net is now known as cp16net|away | 03:51 | |
| *** cp16net|away is now known as cp16net | 03:56 | |
| *** juice has joined #openstack-meeting-alt | 04:00 | |
| *** kaganos has joined #openstack-meeting-alt | 04:14 | |
| *** robertmyers has quit IRC | 04:43 | |
| *** robertmyers has joined #openstack-meeting-alt | 04:44 | |
| *** robertmyers has quit IRC | 04:48 | |
| *** jcooley is now known as jcooley|away | 05:31 | |
| *** jcooley|away is now known as jcooley | 06:40 | |
| *** kaganos has quit IRC | 06:46 | |
| *** jcooley is now known as jcooley|away | 07:22 | |
| *** cp16net is now known as cp16net|away | 07:41 | |
| *** jcooley|away is now known as jcooley | 11:22 | |
| *** jcooley is now known as jcooley|away | 11:32 | |
| *** jcooley|away is now known as jcooley | 13:22 | |
| *** jcooley is now known as jcooley|away | 13:32 | |
| *** jog0 has joined #openstack-meeting-alt | 13:56 | |
| *** jcooley|away is now known as jcooley | 14:20 | |
| *** jcru has joined #openstack-meeting-alt | 14:21 | |
| *** jcooley is now known as jcooley|away | 14:22 | |
| *** jcooley|away is now known as jcooley | 14:22 | |
| *** robertmyers has joined #openstack-meeting-alt | 14:25 | |
| *** jcooley is now known as jcooley|away | 14:34 | |
| *** jcru is now known as jcru|away | 14:59 | |
| *** jcru|away is now known as jcru | 15:06 | |
| *** amyt has joined #openstack-meeting-alt | 15:10 | |
| *** djohnstone has joined #openstack-meeting-alt | 15:13 | |
| *** jcooley|away is now known as jcooley | 15:14 | |
| *** jcooley is now known as jcooley|away | 15:17 | |
| *** ametts-atl has joined #openstack-meeting-alt | 15:34 | |
| *** ametts-atl has left #openstack-meeting-alt | 15:36 | |
| *** jcru is now known as jcru|away | 15:46 | |
| *** cp16net|away is now known as cp16net | 15:56 | |
| *** jcru|away is now known as jcru | 16:13 | |
| *** cp16net is now known as cp16net|away | 16:23 | |
| *** nithiwat has joined #openstack-meeting-alt | 16:38 | |
| *** nithiwat has quit IRC | 16:38 | |
| *** djohnstone_ has joined #openstack-meeting-alt | 16:40 | |
| *** djohnstone has quit IRC | 16:40 | |
| *** djohnstone_ is now known as djohnstone | 16:40 | |
| *** amyt has quit IRC | 17:04 | |
| *** amyt has joined #openstack-meeting-alt | 17:04 | |
| *** amyt has quit IRC | 17:09 | |
| *** jcru is now known as jcru|away | 17:11 | |
| *** amyt has joined #openstack-meeting-alt | 17:17 | |
| *** kaganos has joined #openstack-meeting-alt | 17:18 | |
| *** amyt has quit IRC | 17:22 | |
| *** amyt has joined #openstack-meeting-alt | 17:22 | |
| *** jcooley|away is now known as jcooley | 17:30 | |
| *** esp1 has joined #openstack-meeting-alt | 17:51 | |
| *** esp1 has left #openstack-meeting-alt | 17:58 | |
| *** kaganos has quit IRC | 18:11 | |
| *** jcooley is now known as jcooley|away | 18:24 | |
| *** jcooley|away is now known as jcooley | 18:28 | |
| *** kaganos has joined #openstack-meeting-alt | 18:29 | |
| *** jcooley is now known as jcooley|away | 18:32 | |
| *** jcooley|away is now known as jcooley | 18:32 | |
| *** jamie-rax has joined #openstack-meeting-alt | 18:38 | |
| *** jcru|away is now known as jcru | 18:42 | |
| *** ChadL has joined #openstack-meeting-alt | 18:47 | |
| *** kgriffs has joined #openstack-meeting-alt | 18:49 | |
| *** jamie-racklanta has joined #openstack-meeting-alt | 18:50 | |
| *** jamie-rax has quit IRC | 18:50 | |
| *** jhopper has joined #openstack-meeting-alt | 18:52 | |
| *** edsrzf has joined #openstack-meeting-alt | 18:54 | |
| *** treeder has joined #openstack-meeting-alt | 18:56 | |
| *** nithiwat has joined #openstack-meeting-alt | 18:56 | |
| *** oz_ has joined #openstack-meeting-alt | 19:00 | |
| kgriffs | #startmeeting | 19:00 |
|---|---|---|
| openstack | kgriffs: Error: A meeting name is required, e.g., '#startmeeting Marketing Committee' | 19:00 |
| oz_ | ih | 19:01 |
| kgriffs | #startmeeting Marconi Project Team | 19:01 |
| oz_ | hi | 19:01 |
| openstack | Meeting started Thu Jan 17 19:01:15 2013 UTC. The chair is kgriffs. Information about MeetBot at http://wiki.debian.org/MeetBot. | 19:01 |
| openstack | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 19:01 |
| *** openstack changes topic to " (Meeting topic: Marconi Project Team)" | 19:01 | |
| openstack | The meeting name has been set to 'marconi_project_team' | 19:01 |
| kgriffs | #topic introductions | 19:01 |
| *** openstack changes topic to "introductions (Meeting topic: Marconi Project Team)" | 19:01 | |
| kgriffs | OK, folks, let's get this party started | 19:01 |
| *** ametts-atl has joined #openstack-meeting-alt | 19:01 | |
| kgriffs | Since this is a new project and involves some folks new to OpenStack, I thought it would be good to start out with a few introductions from the core team and anyone else who's here to talk about Marconi. | 19:02 |
| *** edsrzf has quit IRC | 19:03 | |
| kgriffs | First off, I'm Kurt Griffiths with Rackspace in the Atlanta office. I developed the notification bus for our Cloud Backup product. | 19:03 |
| *** oz_ is now known as oz_akan | 19:03 | |
| kgriffs | treeder: you want to go next? | 19:04 |
| treeder | sure | 19:04 |
| *** Guest88946 has joined #openstack-meeting-alt | 19:05 | |
| treeder | I'm Travis Reeder, CTO of Iron.io, in SF. We built a cloud message queue called IronMQ. Excited to be involved in this project to help define the cloud MQ standard. | 19:05 |
| *** edsrzf has joined #openstack-meeting-alt | 19:06 | |
| kgriffs | (Travis is on the core team for Marconi) | 19:06 |
| kgriffs | Cool. Anyone else want to say hi? Let's give it a couple minutes and move on to the next topic. | 19:06 |
| edsrzf | I'm Evan Shaw from Iron.io. | 19:07 |
| jhopper | I'm John Hopper, engineer over in the Cloud Integration group. We're working on a unified logging component for OpenStack and considering the potential applications of Marconi, we're very interested | 19:07 |
| jhopper | Rackspace, San Antonio^ | 19:08 |
| *** Guest88946 is now known as carimura | 19:08 | |
| carimura | Hey all. Chad Arimura, Iron.io, in SF with Travis. | 19:08 |
| jamie-racklanta | I'm Jamie Painter in Rackspace Atlanta (Kurt's colleague). Looking forward to contributing to Marconi. | 19:09 |
| nithiwat | Hi, I am Nithiwat from Rackspace's Blacksburg office. We use a lot of messaging here and are interested in Marconi project. | 19:09 |
| ChadL | I'm Chad Lung, Rackspace Software Developer most recently on Atom Hopper (atomhopper.org) and helping out John Hopper on the Logging as a Service effort now. I'm in San Antonio | 19:09 |
| oz_akan | Hi, I am Oz Akan from Rackspace, Atlanta | 19:09 |
| ametts-atl | Allan Metts, also from Rackspace Atlanta. Happy to see such a great turnout! | 19:09 |
| kgriffs | #topic Answer any general questions about the project | 19:09 |
| *** openstack changes topic to "Answer any general questions about the project (Meeting topic: Marconi Project Team)" | 19:09 | |
| kgriffs | Fire away, and I'll do my best to answer (or nudge one of my colleagues to answer). | 19:10 |
| jhopper | Can you define the initial use cases for the system? The target user audience, etc... | 19:10 |
| *** jog0 has left #openstack-meeting-alt | 19:11 | |
| kgriffs | The high-level use cases revolve around providing a message bus that web app developers can use when deploying on OpenStack cloud(s)... | 19:14 |
| kgriffs | Two main use cases that I see are: | 19:14 |
| kgriffs | 1. Notifications and RPC between backend systems and user agents through the Internet | 19:15 |
| kgriffs | 2. Distributed job queuing (producer-consumer) | 19:16 |
| jhopper | Gotcha. Thank you, that makes it more clear to me | 19:16 |
| kgriffs | treeder: Anything to add based on your experience? | 19:17 |
| ChadL | I like the idea of using a "Home Document" versus WADL, are the other OpenStack projects using home documents since the RFC is pretty new? Or is this just something Marconi will have out of the box? | 19:17 |
| treeder | there's quite a few, mind if i post a link to a blog post with top 10 uses? | 19:17 |
| treeder | don't want to be spammy | 19:17 |
| kgriffs | sounds good | 19:17 |
| jhopper | That'd be great | 19:18 |
| treeder | http://blog.iron.io/2012/12/top-10-uses-for-message-queue.html | 19:18 |
| kgriffs | #info http://blog.iron.io/2012/12/top-10-uses-for-message-queue.html | 19:18 |
| treeder | There's also this one which gets a bit deeper into some things: http://blog.iron.io/2013/01/common-actions-on-messages-in-queue.html | 19:18 |
| kgriffs | #info http://blog.iron.io/2013/01/common-actions-on-messages-in-queue.html | 19:18 |
| kgriffs | cool, thanks | 19:18 |
| treeder | the worker pattern is probably the most common though | 19:19 |
| treeder | one system puts messages on the queue | 19:19 |
| jhopper | Makes sense - so this is targeting a much wider distribution model than the current queue implementations already in use (RabbitMQ), is that a safe assumption? | 19:20 |
| treeder | another system (could be many different machines) pulls them off to process them | 19:20 |
| treeder | I think so | 19:20 |
| kgriffs | chadl: The idea for a home document came from mnot and I've been trying to follow his guidance. Thoughts on WADL vs. home documents? | 19:20 |
| jhopper | I prefer home documents - they're more concise and easier to parse and process in web-native platforms | 19:20 |
| treeder | For instance, one interesting thing we're seeing since it's a cloud queue that supports webhoooks | 19:20 |
| treeder | is integration between third party systems. | 19:20 |
| jhopper | That would be unique, especially when looking at the Mobile market | 19:21 |
| treeder | ie: service X (stripe, github, or whatever) posts messages to a queue via their webhook support | 19:21 |
| treeder | then you can deal with them later to process them with workers, etc. | 19:21 |
| treeder | rabbit wouldn't work for those scenarios | 19:22 |
| kgriffs | #agreed Stick with home document | 19:22 |
| jhopper | How will the bus handle the potentially vastly different latencies consumers will have. For example: mobile clients may have a much higher per-message latency for platform reasons than my VM hosted in an adjacent VLAN | 19:22 |
| kgriffs | Do you think mobile clients would use the service directly or via some kind of mobile push bridge? | 19:25 |
| treeder | do you mean the latency in terms of the mobile users experience? or how the mq would deal with it technically? | 19:25 |
| kgriffs | I guess their web browsers will do it via JavaScript if they load up a web app. | 19:25 |
| treeder | kgriffs: the mobile thing is a bit tricky. We'd like them to be able to post directly, the hardest part of that is authentication though | 19:26 |
| jhopper | The bridge would make sense. I was curious to know if there's a distinction between retention and replication of events made available to consumers that might drain their queue infrequently. The bridge is probably a more ideal solution since they're two different use-case domains | 19:26 |
| *** clarkb1 has joined #openstack-meeting-alt | 19:27 | |
| kgriffs | #info Need to figure out auth to allow posting directly from mobile | 19:27 |
| treeder | regarding authentication, the mobile app developer would have to embed their authentication tokens into the client | 19:27 |
| treeder | even BaaS haven't figured that out yet | 19:28 |
| *** clarkb has quit IRC | 19:28 | |
| kgriffs | Interesting. We should take that up in depth in a future meeting | 19:28 |
| kgriffs | #action kgriffs add mobile app auth discussion to a future mtg agenda | 19:29 |
| *** clarkb1 is now known as clarkb | 19:30 | |
| jhopper | Are there any throughput/scaling targets? I've been following some of your blog posts but I'm curious to know if there's a workload range defined | 19:31 |
| kgriffs | re the latency question, the combination of customizable message TTLs and the ability to query stats (messages per queue, also trending) would allow app developers to tune for different scenarios (even dynamically) | 19:31 |
| *** vipul is now known as vipul|away | 19:31 | |
| *** vipul|away is now known as vipul | 19:31 | |
| *** nithiwat has quit IRC | 19:32 | |
| *** kaganos has quit IRC | 19:32 | |
| kgriffs | Right now I've just got some SWAGs re throughput and scaling targets | 19:33 |
| jhopper | Haha, fair enough - just curious. I know development is in the early stages | 19:33 |
| ChadL | AMQP support or not? | 19:34 |
| kgriffs | For example, looking at 50ms max turnaround per request (hopefully will be more like 10-20ms, but we're dealing with Python here, so no promises) | 19:34 |
| treeder | -1 | 19:34 |
| treeder | ;) | 19:34 |
| jhopper | In the logging service there's a REST component that's loosely defined for downstream event dissemination - the REST interface described by this system seems to fit our needs quite well, hence the curiosity on throughput | 19:35 |
| kgriffs | I like the idea of keeping Marconi RESTful (to use an abused term), and creating bridges to other systems that may be more appropriate for certain problem domains. | 19:35 |
| treeder | that -1 was for amqp support | 19:36 |
| jhopper | I agree with that. AMQP was designed for certain reasons | 19:36 |
| treeder | jhopper: is 50ms max response time good enough for your logging use case? | 19:36 |
| kgriffs | re throughput, it isn't unreasonable to target millions of tps for a 10-12 box cluster | 19:37 |
| jhopper | that's basically 2000 serial requests a second. I'd need to know payload size and how that impacts request/response latency but I don't see us outstripping your performance. We're looking at a sustained output of 10k+ messages a second however. | 19:38 |
| *** nithiwat has joined #openstack-meeting-alt | 19:38 | |
| jhopper | Our numbers are based roughly on Nova's API logging output | 19:39 |
| jhopper | Well, Nova in general | 19:39 |
| kgriffs | #info 10k+ messages/sec | 19:40 |
| ChadL | What kind of networking libs are you planning to use? Twisted? Eventlet? Tornado? etc. | 19:40 |
| kgriffs | Good to know. We should keep that in mind and see how little hardware we can get away with deploying to support that. | 19:40 |
| jhopper | What I'm really interested in is how Marconi provides us with a uniform way of transmitting messages. Celiometer currently pulls a lot of information right out of RabbitMQ - Marconi may be a more ideal, decoupled solution. Even more so if we treat it as the log event distribution mechanism (used in metering, billing, monitoring, etc...) | 19:42 |
| kgriffs | This is very tentative, but it's looking like gevent with monkey-patched sockets for async requests to auth/storage. I don't think we'll need to do any disk I/O directly from the app (other than error logging). | 19:42 |
| kgriffs | jhopper: Good point. The Ceilometer team has talked about some frustrations they have had in working with the current RabbitMQ-based RPC mechanism | 19:43 |
| jhopper | Indeed. We would like to position our solution as the feeding tube of Ceilometer and Marconi would drastically simplify that | 19:44 |
| kgriffs | re library, it will be WSGI. If people want to do push, they can add that as another layer on their own. | 19:44 |
| kgriffs | I don't want to necessarily position Marconi as a direct competitor to things like RabbitMQ, but if there are use cases where it makes sense to use it, I don't see any reason to stop people either. | 19:46 |
| kgriffs | P.S. - as for web frameworks, I'm working on one built specifically for these kinds of projects: simple, fast, focused on APIs only (vs. web apps). Preliminary benchmarks show it's about 10x faster than Flask, but that number will probably come down when I do some more realistic benchmarks. | 19:49 |
| *** vipul is now known as vipul|away | 19:49 | |
| kgriffs | Latency is very important and I'm also interested in efficient use of hardware. | 19:49 |
| jhopper | Interesting. Is the framework going to be Python 3 compatible? | 19:50 |
| jhopper | Part of my concern with the current dependency on RabbitMQ for Ceilometer is that it requires the Ceilometer instance to be hosted locally to that AMQP instance. This means that having a unified global view requires another aggregation system (another Celiometer perhaps) - not ideal in my opinion. | 19:51 |
| kgriffs | Yes, I've done some preliminary work on Python 3 compat, but there is more to do. Once the library stabilizes, that will be next on the todo list. | 19:51 |
| kgriffs | https://github.com/racker/falcon | 19:51 |
| jhopper | I'll take a look at it. We're using Pecan at the moment to design our REST interfaces. | 19:51 |
| kgriffs | interesting - definitely less than ideal | 19:51 |
| kgriffs | Doug & Co. were touting that at the summit - I will run it through it's paces as well. | 19:52 |
| kgriffs | (AMQP situation is less than ideal, not Pecan) | 19:52 |
| treeder | kgirffs: looks nice | 19:53 |
| kgriffs | Yeah, that's the trick with things like AMQP and DDS | 19:53 |
| jhopper | Check. I find Pecan a little under-developed still but usable and better than Flask. The models in it are pretty straight forward and the framework is tiny. | 19:53 |
| jhopper | I'd love to hear your prospective as well as see how the framework behaves when put into the grinder. | 19:54 |
| jhopper | ^ in the future, that is | 19:54 |
| kgriffs | treeder: Thanks. I sort of got strong-armed into doing this in python, so I'm using just about every trick in the book to get performance up. | 19:54 |
| kgriffs | jhopper: Sure thing. Pecan is another framework more targeted at web apps rather than just APIs, but that doesn't mean you can't do both with it. | 19:54 |
| treeder | jhopper: quick question, for your use case, is it ok to lose some messages in a failure scenario? | 19:54 |
| kgriffs | # info https://github.com/racker/falcon | 19:55 |
| treeder | or do you need guarantees that all messages are safe | 19:55 |
| treeder | ? | 19:55 |
| kgriffs | #info https://github.com/racker/falcon | 19:55 |
| jhopper | Since many of the events will be utilized for alerts and usage, redundancy and message durability are a must | 19:56 |
| kgriffs | #action kgriffs Kick the tires on Pecan | 19:56 |
| kgriffs | Do you require cross-DC replication? | 19:56 |
| treeder | ok, so in your rabbit install, you are clustering it? | 19:56 |
| treeder | and using persistence? | 19:56 |
| ChadL | Is there a plan down the road to have a web ui peering into Marconi, kind of like how RabbitMQ has its own? | 19:56 |
| jhopper | Cross DC replication is not required but would be ideal. I don't know many of the specifics on the RabbitMQ cluster itself but there is message durability implied but not necessarily flushing | 19:57 |
| kgriffs | ChadL: Good idea. Noone has talked about a standalone admin surface | 19:57 |
| oz_akan | about durability, AWS says that even though it won't happen often, some messages in SQS might get lost | 19:58 |
| jhopper | Well, now that I think about it, cross DC responsibilities may be better satisfied by a datastore | 19:58 |
| *** djohnstone_ has joined #openstack-meeting-alt | 19:58 | |
| kgriffs | ChadL: I assume a Horizon plugin is in the future, but it would be nice to have something standalone as well. | 19:58 |
| *** djohnstone_ has quit IRC | 19:58 | |
| oz_akan | so it might be very difficult to have a fully redundant web scale queue service | 19:58 |
| kgriffs | #action kgriffs Add admin web UI to list of future features | 19:59 |
| *** djohnstone_ has joined #openstack-meeting-alt | 19:59 | |
| kgriffs | You can replicate across DCs but it slows things down. What do you think about clients being able to specify message durability? If they want something super-durable but slower throughput, they can choose that. | 20:00 |
| *** djohnstone has quit IRC | 20:00 | |
| *** djohnstone_ is now known as djohnstone | 20:00 | |
| *** joe5081 has joined #openstack-meeting-alt | 20:01 | |
| kgriffs | #topic winding down the discussion | 20:01 |
| *** openstack changes topic to "winding down the discussion (Meeting topic: Marconi Project Team)" | 20:01 | |
| kgriffs | Looks like we are running short on time | 20:01 |
| kgriffs | I will shift the remaining agenda items to future meetings. | 20:02 |
| jhopper | That would be interesting. It might make things more flexible if the client can determine if durability is important to it. I'd like to explore that more but I like the concept. | 20:02 |
| jhopper | Cool | 20:02 |
| treeder | kgriffs: can we talk about the spec a bit? | 20:02 |
| kgriffs | Sure. I just didn't want to keep people around if they've got other meetings, but if some folks wanna stick around a bit, I don't mind. | 20:03 |
| kgriffs | The spec in general, or the API? | 20:03 |
| *** esp1 has joined #openstack-meeting-alt | 20:04 | |
| treeder | well both I suppose | 20:04 |
| *** robertmyers has quit IRC | 20:05 | |
| kgriffs | #topic Discuss the spec rough draft | 20:05 |
| *** openstack changes topic to "Discuss the spec rough draft (Meeting topic: Marconi Project Team)" | 20:05 | |
| kgriffs | #info http://wiki.openstack.org/marconi/specs/grizzly | 20:05 |
| *** jcru has quit IRC | 20:05 | |
| treeder | two things I'm interested in discussing | 20:05 |
| treeder | tags | 20:05 |
| treeder | vs a concrete queue name | 20:05 |
| kgriffs | #info API blueprint - http://wiki.openstack.org/marconi/specs/api/v1 | 20:06 |
| treeder | and the options to Get Messages | 20:06 |
| treeder | GET {base_url}/messages{?tags,ts,limit,sort,audit,echo} | 20:06 |
| kgriffs | OK. Let's take them one at a time | 20:06 |
| kgriffs | So, tags and/or concrete queue name | 20:07 |
| kgriffs | #topic tags and/or concrete queues | 20:07 |
| *** openstack changes topic to "tags and/or concrete queues (Meeting topic: Marconi Project Team)" | 20:07 | |
| *** jcooley is now known as jcooley|away | 20:07 | |
| kgriffs | So, I know we discussed previously in a G+ hangout that having queue names offers some benefits. | 20:08 |
| treeder | ya | 20:08 |
| *** esp1 has quit IRC | 20:08 | |
| treeder | so a few of those benefits | 20:08 |
| kgriffs | While we *could* do everything with tags, it isn't very pragmatic | 20:08 |
| *** robertmyers has joined #openstack-meeting-alt | 20:08 | |
| treeder | well just to reiterate, tags essentially feels like the user has 1 single massive queue where each message is tagged | 20:09 |
| treeder | with one or more tags | 20:09 |
| jhopper | tags will come with a hefty performance hit | 20:09 |
| jhopper | and replication hit | 20:09 |
| treeder | yep, for sure | 20:09 |
| jhopper | I love them, don't get me wrong | 20:09 |
| *** robertmyers has quit IRC | 20:09 | |
| treeder | so 1) scaling with tags would be very hard | 20:10 |
| *** robertmyers has joined #openstack-meeting-alt | 20:10 | |
| treeder | 2) sharing queues - ie: different rights/access to queues for different users/parties | 20:10 |
| treeder | i suppose you could assign rights to tags, but tags feel more ad-hoc | 20:11 |
| jhopper | They're really powerful for implying event types and interest - if Marconi had a poll-push model then they might make more sense in that you could actively filter. However I don't think removing the construct is a good idea either - tags are powerful descriptors. It might be enough to just have them as part of the event schema and let downstream poller decide interest | 20:11 |
| kgriffs | I agree, there's an impedance mismatch with the mental model | 20:11 |
| jhopper | Sharing queues makes sense to me but authentication and authorization would have to be defined first before visiting what it means to host access for a shared queue | 20:12 |
| jhopper | If AuthN/Z are offloaded by a proxy then permissions at the queue level might not matter at all? | 20:13 |
| treeder | guess it depends how fine grained the auth can be | 20:13 |
| jhopper | This is true | 20:14 |
| kgriffs | Seems like auth would be more expensive if it were based on a tag set | 20:14 |
| *** nithiwat has quit IRC | 20:14 | |
| treeder | kgriffs: yes, it would be | 20:14 |
| jhopper | If the bus doesn't really do any kind of tag introspection then using tags for permissions (while interesting) would require additional effort | 20:14 |
| kgriffs | What do you guys think about having concrete queues as well as some limited number of tags purely for filtering? | 20:15 |
| treeder | if you can assign tokens to a particular queue, then you can give that token to a third party, embed it in a mobile app, or whatever and be sure it can only access messages in that queue | 20:15 |
| kgriffs | that's an interesting concept | 20:15 |
| jhopper | Indeed | 20:15 |
| kgriffs | generate an API token that is tied to a specific app and a specific queue | 20:15 |
| treeder | the idea of a global queue scares me | 20:15 |
| treeder | ;) | 20:15 |
| treeder | ya, exactly | 20:15 |
| kgriffs | #action kgriffs Add generating an API token that is tied to a specific app and a specific queue to the feature proposals list | 20:16 |
| jhopper | Having concrete queues makes sense to me. If you want to do tag comprehension for filtering input into the queue, I think it would be interesting but that model also opens up a lot of extra questions. | 20:16 |
| treeder | jhopper: agree | 20:16 |
| treeder | although, for first version, I would highly recommend no tagging | 20:17 |
| jhopper | Can a queue impart tags to messages published to it? | 20:17 |
| kgriffs | The sticking point is that we need it for Cloud Backup and I was hoping to use that as a guinea pig | 20:17 |
| jhopper | Ah, I see. | 20:18 |
| treeder | you need tags? | 20:18 |
| kgriffs | just one | 20:18 |
| treeder | can you explain the use case? | 20:18 |
| kgriffs | Sure. | 20:18 |
| kgriffs | If the control panel wants to communicate with a backup agent installed on someone's server, it uses RSE... | 20:19 |
| kgriffs | For example... | 20:19 |
| *** vipul|away is now known as vipul | 20:20 | |
| *** grapex has joined #openstack-meeting-alt | 20:20 | |
| kgriffs | …actually, I'm jumping ahead of myself, but I think it would be possible to do what Cloud Backup needs by allocating two queues per agent. The downside is you are putting 2x the load on the servers and using 2x bandwidth, etc. | 20:21 |
| kgriffs | …so back to my example | 20:21 |
| kgriffs | When a customer logs in, the control panel checks for recent heartbeat events | 20:21 |
| treeder | i think I get it already, you want some sort of fanout? | 20:21 |
| treeder | one message in, two consumers? | 20:22 |
| kgriffs | yeah, you probably know where I'm going | 20:22 |
| treeder | so we tell people to use two queues, just like you said | 20:22 |
| kgriffs | when someone logs in, the CP checks for heartbeat events | 20:22 |
| kgriffs | it does this at an account-wide level | 20:22 |
| kgriffs | There are other scenarios where the CP wants to talk just to a particular agent | 20:23 |
| kgriffs | anyway, depending on what's going on you've got fanout in one direction or the other. | 20:23 |
| treeder | right | 20:23 |
| treeder | would using multiple queues work just as well? | 20:24 |
| treeder | besides the fact you'd have to post multiple times | 20:24 |
| kgriffs | Event types are all multiplexed across a single queue, essentially, and the queues are only used to namespace by account and agent | 20:24 |
| jhopper | It might be expensive since you could potentially have a queue for each host-agent of which there may be hundreds | 20:24 |
| kgriffs | Yes, you could, it just doubles the load, also for polling. | 20:24 |
| jhopper | I see, so you'd use tags to demux the queue? | 20:25 |
| kgriffs | there are actually going to be hundreds of thousands of agents | 20:25 |
| *** joe5081 is now known as joe5081|away | 20:25 | |
| kgriffs | Possibly, but I think the real benefit is to add one or two nested namespaces in order to provide an affordance than can be leveraged for stuff like fanout. | 20:26 |
| kgriffs | In any case, we could do v1.0 without tags and folks could use multiple queues, but I think it makes sense to do it at some point. | 20:26 |
| jhopper | I ran into this thought game when it came to AtomNuke - This type of filtering, in my opinion, should probably be part of the responsibility of the poller or an intermediate bridge. | 20:27 |
| kgriffs | Of course, once the ability is there, who knows how people will leverage it? People do surprising things. | 20:27 |
| jhopper | But again, there's a lot of +/- to the argument. That's a fair question too, kgriffs | 20:27 |
| kgriffs | jhopper: food for thought. | 20:27 |
| kgriffs | Probably not the poller; consider a web app doing the filtering. JavaScript isn't exactly a speed demon. | 20:28 |
| treeder | kgriffs: for your tagged example | 20:28 |
| *** esp1 has joined #openstack-meeting-alt | 20:28 | |
| jhopper | Well, an intermediate bridge - maybe not the end client but rather a client from the queue's perspective | 20:29 |
| kgriffs | But maybe a proxy approach. Although depending on the storage drive, it may be more efficient to just do it as part of the query, assuming a finite number of tags (say, 2-3) | 20:29 |
| treeder | if agent A takes a message how does it even get to the two consumers? | 20:29 |
| jhopper | That's a good question too. Hrm. | 20:30 |
| treeder | you'd have to have two messages anyways | 20:30 |
| kgriffs | Not sure I follow - if the agent gets a message, it *is* the consumer, right? | 20:30 |
| treeder | i thought you wanted one message to get to two consumers? | 20:30 |
| kgriffs | oic | 20:31 |
| kgriffs | Well, the scenario is the CP wants to broadcast an event to all agents that Bob is running under his cloud account. | 20:31 |
| kgriffs | So, agents need to listen on the broadcast queue, as well as another queue for RPC type stuff. | 20:32 |
| kgriffs | Unless you can do a tag | 20:32 |
| treeder | ok | 20:32 |
| treeder | so even in that scenario, I think two queues is probably better than tags anyways | 20:32 |
| treeder | unless you wanted to ensure that EVERY tag gets a message somehow | 20:33 |
| treeder | but that's adding some serious complexity | 20:33 |
| jhopper | Tags imply a lot of routing logic | 20:33 |
| *** robertmy_ has joined #openstack-meeting-alt | 20:33 | |
| treeder | and delivery logic | 20:33 |
| jhopper | I think that kgriffs' case makes sense, it's very useful in my opinion - just hard to implement. | 20:33 |
| kgriffs | In RSE we basically have the notion of a single tag AKA subqueue | 20:34 |
| jhopper | Well, delivery logic would be simply placing a message reference into the relevant queue | 20:34 |
| kgriffs | It works pretty well. Things would get more complicated with N tags | 20:34 |
| jhopper | This is true, unlimited taxonomies produce only headaches | 20:34 |
| *** robertmyers has quit IRC | 20:34 | |
| kgriffs | I guess you need the ability to say "give me everything that has no tags and give me everything just with this tag set" in a single query | 20:35 |
| *** esp1 has left #openstack-meeting-alt | 20:35 | |
| *** cp16net|away is now known as cp16net | 20:36 | |
| kgriffs | so, I don't query /foo?tag=bar AND /foo in separate requests. Otherwise might as well just use two queues | 20:36 |
| treeder | maybe fanout support would be good? | 20:39 |
| kgriffs | Going the other way, if an agent were to post to /account with tag=agent, the CP could decide whether it wants to listen to all events (in the case of a dashboard page) and just get everything regardless of tags for /account, or if you are looking at a single agent, it would then request /foo?tag=agent | 20:40 |
| *** esp1 has joined #openstack-meeting-alt | 20:40 | |
| treeder | eg: | 20:41 |
| treeder | hmmm, need to think about that a bit more | 20:41 |
| treeder | the broadcast use case is interesting | 20:42 |
| treeder | we have that with our Push Queues, but not for pull queues | 20:42 |
| *** joe5081|away is now known as joe5081 | 20:42 | |
| treeder | can we continue that discussion next week? | 20:43 |
| kgriffs | Sounds good. We've been add it for quite a while now. :D | 20:43 |
| jhopper | I'd like that - there's a lot to think about | 20:43 |
| kgriffs | add => at | 20:43 |
| treeder | good first meeting though! | 20:44 |
| kgriffs | Definitely. I wasn't sure what to expect but I think we're off to a good start | 20:44 |
| treeder | ya, definitely | 20:44 |
| jhopper | I'm excited. This all sounds really promising and I like the API. | 20:44 |
| jhopper | good stuff guys! | 20:44 |
| ChadL | yeah the API looks great so far | 20:44 |
| *** esp1 has left #openstack-meeting-alt | 20:45 | |
| kgriffs | OK. let's sleep on some of these things we've been discussing and follow up next week. | 20:45 |
| treeder | alright, cheers guys, good meeting you all and talk to you next week. | 20:46 |
| jhopper | take care | 20:46 |
| kgriffs | #endmeeting | 20:46 |
| *** openstack changes topic to "OpenStack meetings (alternate) || Development in #openstack-dev || Help in #openstack" | 20:46 | |
| openstack | Meeting ended Thu Jan 17 20:46:43 2013 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 20:46 |
| openstack | Minutes: http://eavesdrop.openstack.org/meetings/marconi_project_team/2013/marconi_project_team.2013-01-17-19.01.html | 20:46 |
| openstack | Minutes (text): http://eavesdrop.openstack.org/meetings/marconi_project_team/2013/marconi_project_team.2013-01-17-19.01.txt | 20:46 |
| kgriffs | ttfn | 20:46 |
| openstack | Log: http://eavesdrop.openstack.org/meetings/marconi_project_team/2013/marconi_project_team.2013-01-17-19.01.log.html | 20:46 |
| *** ChadL has left #openstack-meeting-alt | 20:47 | |
| *** ametts-atl has left #openstack-meeting-alt | 20:47 | |
| *** ametts-atl has joined #openstack-meeting-alt | 20:48 | |
| *** ametts-atl has left #openstack-meeting-alt | 20:48 | |
| *** oz_akan has quit IRC | 20:54 | |
| *** kaganos has joined #openstack-meeting-alt | 20:57 | |
| *** cp16net is now known as cp16net|away | 20:58 | |
| *** joe5081 has quit IRC | 21:06 | |
| *** jcru has joined #openstack-meeting-alt | 21:07 | |
| *** robertmy_ has quit IRC | 21:10 | |
| *** grapex has quit IRC | 21:10 | |
| *** grapex has joined #openstack-meeting-alt | 21:11 | |
| *** robertmyers has joined #openstack-meeting-alt | 21:11 | |
| *** edsrzf has left #openstack-meeting-alt | 21:12 | |
| *** juice has quit IRC | 21:14 | |
| *** treeder has quit IRC | 21:16 | |
| *** grapex has quit IRC | 21:20 | |
| *** grapex has joined #openstack-meeting-alt | 21:20 | |
| *** carimura has quit IRC | 21:35 | |
| *** robertmyers has left #openstack-meeting-alt | 21:38 | |
| *** juice has joined #openstack-meeting-alt | 21:40 | |
| *** jamie-racklanta has quit IRC | 21:51 | |
| *** kgriffs has left #openstack-meeting-alt | 22:00 | |
| *** jhopper has quit IRC | 22:04 | |
| *** jcooley|away is now known as jcooley | 22:46 | |
| *** kaganos has quit IRC | 22:47 | |
| *** jcooley is now known as jcooley|away | 22:59 | |
| *** djohnstone has quit IRC | 23:10 | |
| *** esp has joined #openstack-meeting-alt | 23:10 | |
| *** kaganos has joined #openstack-meeting-alt | 23:12 | |
| *** jcru has quit IRC | 23:13 | |
| *** jog0 has joined #openstack-meeting-alt | 23:16 | |
| *** jcooley|away is now known as jcooley | 23:17 | |
| *** jog0 has quit IRC | 23:19 | |
| *** jcooley is now known as jcooley|away | 23:27 | |
| *** jcooley|away is now known as jcooley | 23:29 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!