19:05:31 <kgriffs> #startmeeting marconi
19:05:32 <openstack> Meeting started Thu Jun 27 19:05:31 2013 UTC.  The chair is kgriffs. Information about MeetBot at http://wiki.debian.org/MeetBot.
19:05:33 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
19:05:35 <openstack> The meeting name has been set to 'marconi'
19:05:47 <kgriffs> #link https://wiki.openstack.org/wiki/Meetings/Marconi#Agenda
19:06:01 <kgriffs> So, let's real quick take a look at actions from last time
19:06:10 <kgriffs> #link http://eavesdrop.openstack.org/meetings/marconi/2013/marconi.2013-06-20-19.06.html
19:06:32 <kgriffs> hmm, I thought there were more actions
19:06:38 <flaper87> LOL
19:06:46 <kgriffs> well, since malini isn't here, I guess I can't ask her about that wiki page
19:06:47 <kgriffs> :p
19:06:58 <flaper87> I'm afraid, I'm in bad shape wrt the AutoReconnect patch. Will take care of that next week
19:07:06 <ametts> "2:29:02 PM) malini: I registered a client BP for swift integration , per our last meeting"
19:07:09 <kgriffs> in other news, megan_w has been hard at work on a draft incubation appllication
19:07:13 <flaper87> wait, next week is EuroPython T_T, anyway, I'll do my best
19:07:26 <kgriffs> flaper87: kk
19:07:40 <kgriffs> yeah, we will need that patch in the next few weeks
19:08:04 <flaper87> yup, yup. Will get that done
19:08:08 <kgriffs> cool
19:08:25 <kgriffs> let's spend a minute talking about incubation - I thought it would show up in the action items from last week
19:08:30 <kgriffs> #topic incubation
19:08:45 <kgriffs> megan_w: would you like to give us an update?
19:09:06 <megan_w> so the draft of the incubation application is ready for a wider audience to review
19:09:27 <megan_w> kgriffs: do you happen to have a copy we can review outside of our team wiki?
19:09:40 <flaper87> megan_w: hey, nice to ircmeet you
19:09:56 <megan_w> flaper87: ditto, welcome back
19:10:19 <megan_w> i think we should try to get the application submitted in the next few days
19:10:31 <kgriffs> megan_w: it would be nice to have it on the openstack wiki
19:10:47 <megan_w> agreed.  not sure if I have permissions to edit that page
19:11:09 <flaper87> re incubation: I'm afraid having more contributors is a must for being incubated, which means H-2 seems hard
19:11:20 <flaper87> megan_w: you just need a Launchpad account
19:11:31 <flaper87> the wiki uses LP openid
19:12:32 <megan_w> ok, i'll see if I can get it there now..
19:14:08 <kgriffs> well, I guess we can send the draft application to the TC and ask them what they would like to see from a contributor standpoint beyond what we already have.
19:14:14 <megan_w> sorry, i'm getting an invalid token error when trying to login.  i may have to do this offline
19:14:39 <kgriffs> ok, no worries
19:14:49 <kgriffs> right now i guess we just need to decide our next step
19:15:49 <megan_w> so once we all approve the doc, we just need to submit to TC, right?
19:15:58 <flaper87> I'd say we should keep working on our application and make marconi rock solid
19:16:18 <kgriffs> well, let's get it on the wiki
19:16:36 <flaper87> once we have both, we can send the application to the TC team and see what happens
19:16:49 <kgriffs> and then solicit some more contributors
19:16:54 <flaper87> but, AFAIK, Designate didn't get incubated because the team is not ready yet
19:17:01 <ametts> flaper87:  "more contributors" = more people, or more than Redhat and Rackspace as organizations?
19:17:03 <flaper87> or at least, that was one of the reasons
19:17:11 <flaper87> ametts: more people
19:17:44 <ametts> We could probably put 2-3 more people on it here.  Is that enought?
19:17:54 <ametts> megan_w can start coding....
19:17:55 <ametts> :D
19:18:16 <megan_w> oh yes, that'll work for sure :)
19:18:57 <flaper87> hehe
19:19:05 <kgriffs> well, I think we can probably pick up a couple more contributors and get the service pretty solid in 3-4 weeks from now
19:19:14 <flaper87> Dunno if that's enough, I'd say yes since there's not a min number
19:19:25 <flaper87> kgriffs: sounds great!
19:19:40 <flaper87> hopefully, I'll be able to do the same
19:20:39 <flaper87> next topic ?
19:21:03 <kgriffs> A few things in our favor: we've been developing this in the open since the beginning, and many people are already familiar with the project. Also, we've already done a "Request for Comments" and got fairly positive feedback from a couple TC members. Plus we will have TONs of tests to prove our code thanks to Malini.
19:21:10 <kgriffs> anyway, I digress.
19:21:11 <kgriffs> :D
19:21:44 <flaper87> kgriffs: Agreed with that. Plus, we're completely integrated with OpenStack's tools
19:21:49 <flaper87> so, that's another super +1
19:21:51 <kgriffs> #agreed Be ready to apply for incubation in 3-4 weeks. Rock solid code, more contributors.
19:22:28 <kgriffs> #action megan_w to put incubation application on wiki
19:22:41 <kgriffs> #topic input validation
19:22:41 <megan_w> sounds good
19:23:07 <kgriffs> so, a few things here
19:23:35 <kgriffs> first, we discussed via email and irc the proposal to make limits configurable
19:23:45 <flaper87> +1
19:23:52 <kgriffs> so, stuff like max message size, min and max TTL, etc.
19:23:55 <kgriffs> any objections?
19:24:16 <kgriffs> ok, cool
19:24:41 <flaper87> ametts: bryansd thoughts ?
19:24:46 <kgriffs> as far as what to use as defaults, let's just make an educated guess and we can refine as we get more data on real usage
19:25:27 <kgriffs> ametts: where did you end up on the validation middleware thingy?
19:26:06 <flaper87> kgriffs: my thinking about the default value is: "Lets not copy {PUT_COMPANY_NAME_HERE}, instead, we should keep those values low and fast"
19:26:18 <flaper87> and by fast I mean optimized
19:26:28 <flaper87> dunno which of those terms is more ambiguous
19:26:29 <ametts> I have a middleware layer stubbed out that pulls configuration from a [drivers:middleware:validation] group in config.
19:27:11 <ametts> Haven't gotten back to monkey_patching the controller calls, but I'm optimistic about what looks like a meeting free day tomorrow and a slow holiday week next week.
19:27:20 <kgriffs> when do you think you'll have a patch ready for review?
19:27:34 <ametts> Let's say Monday.
19:28:41 <kgriffs> #action ametts to finish up the first draft for input validation
19:28:49 <kgriffs> cool, that would be great
19:28:57 <flaper87> ametts: HA, you're not going to run away from that action!
19:29:34 <kgriffs> re defaults, I agree that we shouldn't just blindly copy what [REDACTED] does.
19:30:19 <flaper87> kgriffs: what about keeping 64 and 20 ?
19:30:24 <megan_w> i think its fair to acknowledge that those who already have products out are learning what customers demand, though
19:30:32 <kgriffs> megan_w: can you do some digging on max message's/request (both GET and POST), as well as max message size and min/max TTL for a message?
19:31:01 <kgriffs> I agree, but we want to understand the use cases as well
19:31:12 <flaper87> megan_w: agreed, but those are up to the deployer anyway, right?
19:31:19 <megan_w> kgriffs: yes, i'll dig into that
19:31:52 <flaper87> I mean, the thought is more like: This values work better for the server but you can adjust them to what your clients demand
19:32:29 <megan_w> right.  the default should definitely be our recommended usage
19:32:37 <kgriffs> yeah, it seams like we will find a natural sweet spot for those values once we get real usage data
19:32:48 <flaper87> agreed!
19:33:32 <kgriffs> but we also need to be smart about the way we pipe data around, i.e., streaming vs. buffering as much as possible so we don't just give up the ghost the first time someone throws a big pile of messages at us.
19:33:57 <kgriffs> so, let's just start with some sane defaults and adjust later
19:34:04 <flaper87> agreed
19:34:14 <kgriffs> (based on customer and operational research)
19:34:18 <megan_w> agreed
19:35:00 <kgriffs> also, keep in mind that we haven't done hardly any tuning so performance numbers are going to change a lot (hopefully) over the next few weeks.
19:35:11 <kgriffs> ok, anything else on that topic?
19:35:33 <flaper87> nope from me
19:35:57 <kgriffs> #topic stats blueprint
19:36:02 <kgriffs> #link https://blueprints.launchpad.net/marconi/+spec/ops-stats
19:36:28 * ametts is going to nag oz_akan to join the meeting
19:36:35 <kgriffs> So, this came out of a discussion I was having with megan_w and oz_akan the other day
19:37:28 <kgriffs> the idea is we need a basic set of stats for operational monitoring, capacity planning, and such
19:37:45 <kgriffs> we might also expose a subset of these through our API to the user
19:38:15 <megan_w> kgriffs: i would think things like messages/sec and queue stats would be helpful for users to see
19:38:17 <flaper87> agreed, have you guys put some thoughts around it?
19:38:22 <oz_akan_> hi
19:38:31 <flaper87> oz_akan_: -1 pop-tart for being late
19:38:41 <oz_akan_> :/
19:38:46 * flaper87 eats oz_akan_ pop-tart
19:38:55 <ametts> So we're all agreed that oz_akan is going to implement that major change?
19:39:00 <kgriffs> yep.
19:39:05 <oz_akan_> oh, what is it?
19:39:17 <flaper87> oz_akan_: you'll have to read the log
19:39:22 <flaper87> there's an action for ya!
19:39:26 <kgriffs> it should only take him 6-8 weeks if he doesn't sleep more than 4 hours a night
19:39:39 <flaper87> kgriffs: you're always so optimistic
19:39:52 <kgriffs> lol
19:39:57 <kgriffs> so, back on topic
19:39:58 <oz_akan_> ok so 4 weeks, if I don't sleep
19:40:04 <kgriffs> megan came up with a list of metrics
19:40:30 <ametts> kgriffs:  no real names -- she's megan_w :)
19:40:49 <megan_w> haha, incognito
19:40:51 <oz_akan_> new metrics and  ?
19:41:03 <kgriffs> megan_w: do you think I should add all of those to the blueprint, or just the most important ones you starred?
19:41:20 <kgriffs> oz_akan_: https://blueprints.launchpad.net/marconi/+spec/ops-stats
19:41:26 <megan_w> kgriffs: is the blueprint supposed to be set in stone, or up for discussion?
19:41:32 <kgriffs> it's a living document
19:41:33 <megan_w> if the latter, let's put them all there
19:41:39 <flaper87> kgriffs: add the ones that are supposed to be implemented in that blueprint
19:41:45 <kgriffs> actually, we should just make a wiki page and link it
19:41:49 <flaper87> we can create another one for the missing ones
19:41:53 <kgriffs> kk
19:41:57 <kgriffs> I can do that
19:41:58 <flaper87> or add work items
19:42:01 <flaper87> that should work
19:42:06 <kgriffs> #action kgriffs to add metrics to blueprint
19:42:14 <flaper87> btw, have you guys thought about how that is going to be implemented?
19:42:18 <kgriffs> ok, the other thing on that topic is how to report stats
19:42:32 <kgriffs> statsd, counters in the DB, a combination?
19:42:39 <kgriffs> heh
19:42:54 <kgriffs> thanks segue dude!
19:43:01 <oz_akan_> about stats collector, should it be part of the driver ?
19:43:05 <flaper87> we should also check ceilometer. IIRC, it doesn't support statsd
19:43:18 <megan_w> btw, incubation app now on wiki (sorry for going off topic)
19:43:20 <flaper87> or at least it didn't
19:43:30 <kgriffs> megan_w: excellent, thanks!
19:43:33 <flaper87> megan_w: +1
19:43:42 * flaper87 gives megan_w an apple pie [$]
19:43:43 <kgriffs> can you pipe ceilometer output to graphite?
19:43:48 <torgomatic> statsd has some really nice properties IMHO
19:43:57 <torgomatic> like it doesn't hose your app when the collector is down :)
19:44:04 <kgriffs> :D
19:44:07 <flaper87> lol
19:44:13 <flaper87> put an action on me
19:44:19 <flaper87> I'll dig into what ceilo supports
19:44:24 <flaper87> and how it integrates with statsd
19:44:41 <kgriffs> #action flaper87 to find out if ceilometer can be used for operational stats
19:44:53 <kgriffs> #action flaper87 to investigate statsd integration with ceilometer
19:44:58 <flaper87> awesome
19:45:02 <flaper87> dude, take it easy
19:45:04 <flaper87> :P
19:45:09 <kgriffs> I suppose we could always just make a statsd collector/bridge for ceilometer
19:45:23 <flaper87> kgriffs: if ceilo supports statsd, we're done!
19:45:25 <kgriffs> the nice thing about ceilometer is then we are all set up for billing
19:45:40 <kgriffs> flaper87: and if it doesn't we can contribute a lovely patch. :D
19:45:50 <flaper87> kgriffs: I'd prefer ceilo to be optional
19:45:56 <flaper87> but be supported
19:46:04 <kgriffs> sure, that makes sense
19:46:16 <flaper87> (that will be another question when applying for incubation)
19:46:21 <kgriffs> I was thinking the stats collector would be a driver
19:46:37 <kgriffs> so folks can plug in different drivers depending on their needs
19:46:49 <oz_akan_> that sits above storage driver?
19:46:56 <kgriffs> sort of on the side
19:47:12 <kgriffs> transport and storage would both need a reference to it, nicht?
19:47:12 <oz_akan_> which side?
19:47:13 <oz_akan_> :)
19:47:48 <kgriffs> you know, right next to the mashed potatoes and cole slaw
19:47:57 <kgriffs> aaaaaanyway
19:47:57 <oz_akan_> I was thinking like pipes, request goes through stats collector to storage..
19:48:04 <kgriffs> oic
19:48:06 <kgriffs> not a bad idea
19:48:14 <flaper87> IMHO, it's kind of a middleware
19:48:20 <flaper87> or something between the transport and the storage
19:48:32 <kgriffs> yeah, that sounds great
19:48:47 <kgriffs> let's get another contributor to build that
19:49:02 <kgriffs> (more contributors!)
19:49:14 <torgomatic> FWIW, Swift has a proxy_logging middleware that does a bunch of stuff, but there's also stats calls sprinkled throughout the code for stuff not in the request path
19:49:22 <torgomatic> and it works well for us
19:49:40 <kgriffs> hmm, that's good to know
19:50:01 <kgriffs> We can start with capturing things on the request path
19:50:03 <torgomatic> dunno how much stuff Marconi has that's not in a request path, though
19:50:10 <kgriffs> and add more comprehensive hooks down the road
19:50:23 <kgriffs> good question; we'd have to dig in and see
19:50:58 <kgriffs> #agreed use a middleware approach for metering/stats initially
19:51:07 <kgriffs> ok, we'll have to be careful it stays performant
19:51:16 <kgriffs> (which is the other reason I like statsd, btw)
19:51:24 <kgriffs> ok, anything else on that topic?
19:52:03 <kgriffs> #topic message size and messages/req
19:52:08 <kgriffs> we touched on this earlier.
19:52:25 <kgriffs> basically, [REDACTED] upped their max message size to 256 KB
19:52:46 <kgriffs> and some Rackers were concerned that Marconi's initial 4 K limit was too low
19:53:42 <kgriffs> we could go up to 10 MB for all I care, but I don't want users to abuse Marconi for use cases that are better suited to swift, and Marconi does some buffering at the moment, so memory usage is a concern (although we will be fixing this soon, hopefully)
19:53:42 <megan_w> so-and-so had a limit of 64 and upped it.  They've actually upped it a few times now
19:53:48 <megan_w> i think they started at 8k
19:54:33 <kgriffs> so, one thought I had was for really large payloads the client could automagically store the message bodies in a swift container, using a temp uri and auto-expiration or something
19:54:41 <megan_w> once we have swift as a storage option, this may be less of an issue
19:54:55 <megan_w> once = "if"
19:55:04 <kgriffs> but that aside, we need a good understanding of what users really need
19:55:34 <kgriffs> seems like there should be a sweet spot in there somewhere - large enough to get work done, but not too large that you are abusing the transport
19:55:54 <kgriffs> flaper87: thoughts?
19:55:55 <flaper87> I think 6k and 20messages is good
19:55:56 <oz_akan_> I don't really see why one would need more than a few KBs
19:56:02 <flaper87> 64k
19:56:04 <flaper87> sorry
19:56:10 <oz_akan_> probably it would be a wrong use case
19:56:31 <torgomatic> how big are Keystone PKI tokens? I can imagine people wanting to fit one or two of those in there
19:57:11 <flaper87> torgomatic: you rekon? That doesn't sound good :(
19:57:21 <ametts> Some of the RAX guys had some use cases where 4K was definitely too small, and 256K felt "right".  megan_w --> do you remember any of those?
19:58:17 <megan_w> i don't think they said anything specific, just that use cases existed.  maybe something about xml objects?
19:58:29 <torgomatic> flaper87: dunno; seems like with bearer tokens, you might want to jam one into the queued message so that your message processors can keep doing authenticated stuff
19:59:25 <flaper87> torgomatic: might make sense in some cases. It doesn't feel right, though
19:59:35 <flaper87> we should dig more in that topic
19:59:39 <kgriffs> yeah
19:59:41 <torgomatic> so, like, Service A gets a request w/a token, then sticks (token + other stuff) in Marconi; processor takes (token + other stuff), does work based on stuff, then makes request to Service Slowpoke with token
19:59:46 <torgomatic> just a thought, though
20:00:06 <flaper87> torgomatic: fair one, thanks for that!
20:00:29 <torgomatic> so I guess I can't justify the "or two" part of "one or two tokens" :)
20:00:31 <oz_akan_> queue is not for storing data, it is for storing pointers to data, so...
20:01:22 <flaper87> oz_akan_: fair point! But in some cases data passes through it.
20:01:32 <flaper87> oz_akan_: btw, that's why I think 64k is good enough
20:01:39 <flaper87> but again, we should investigate more
20:01:46 <ametts> We should also consider things like thumbnail images, which can be a few-dozen kilobytes.  That would be inconvenient to transmit as pointers.
20:01:50 <flaper87> kgriffs: we ran out of time
20:01:56 <kgriffs> oz_akan_: right, if we make it seamless to store payloads in swift then I bet users won't care so much about max size
20:02:00 <oz_akan_> 256KB is SQS size, that must have a valid reason, which I can't understand
20:02:07 <kgriffs> kk
20:02:22 <oz_akan_> kgriffs: +1
20:02:26 <kgriffs> well, let's keep noodling on that and do some research
20:02:30 <flaper87> oz_akan_: or not? What if it's just marketing and more machines ?
20:02:38 <flaper87> lets dig more
20:02:57 <oz_akan_> flaper87: still valid reasons for someone :)
20:03:07 <kgriffs> #agreed need more data before we know what max message size to target
20:03:10 <flaper87> oz_akan_: fair enough :P
20:03:28 <kgriffs> ok folks, anything else for today?
20:03:33 * flaper87 likes these meetings
20:03:46 <kgriffs> #topic open discussion
20:03:50 <flaper87> kgriffs: I got some thoughts, but I'll catch you guys later on #openstack-marconi
20:03:55 <kgriffs> kk
20:04:06 <kgriffs> thanks folks
20:04:08 <kgriffs> #endmeeting