16:00:13 <dachary> #startmeeting
16:00:14 <openstack> Meeting started Thu Apr 26 16:00:13 2012 UTC.  The chair is dachary. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:00:15 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic.
16:00:24 <dachary> #link http://wiki.openstack.org/Meetings/MeteringAgenda
16:01:12 <dachary> nijaba: will join later, he is held up by a last minute interview
16:01:45 <dachary> #topic Agree on the project objectives
16:02:27 <dachary> nijaba: proposed "The project aims to deliver a unique point of contact for billing  systems to aquire all counters they need to establish customer billing,  accross all current and future OpenStack components. The delivery of  counters must be tracable and auditable, the counters must be easily  extensible to support new projects, and agents doing data collections  should be independent of the overall system"
16:02:50 <dhellmann> That seems like a good start.
16:03:07 <dhellmann> I'm not sure how the term "counter" is being used, though.
16:03:34 <dhellmann> For example, I may want to list on the bill all of the floating IPs a tenant is using, and how long they have had each of them (total and since the last billing cycle).
16:03:43 <dhellmann> So just knowing the number of floating IPs isn't enough.
16:03:56 <dhellmann> Is that part of what you mean for how counters will work?
16:04:43 <dachary> I see a counter as "something that is billable" indeed
16:04:54 <dhellmann> maybe the floating IP counter measures hours/minutes per IP just as we do for compute for instance?
16:05:09 <dhellmann> I'm just trying to make sure I understand the framing
16:05:11 <dachary> for IPs they tend to have a fixed cost but the cost may vary
16:05:36 <dhellmann> well, it's fixed over a period of time, right? so if I allocate one and then deallocate it later, I am only billed for the time I'm actually using it
16:05:49 <dachary> yes
16:05:57 <dhellmann> granted a tenant will tend to hold on to an IP for a long period of time
16:06:08 <dachary> If you lease an IP at OVH it's not the same as if you least it from Hetzner for instance
16:06:42 <dhellmann> sorry, are those providers?
16:06:52 <dachary> yes
16:06:57 <dhellmann> got it
16:07:14 <dhellmann> so maybe my objection is just to the definition of that particular counter, and not to the idea of counters in general
16:07:15 <dachary> I should say : "someone from whom you lease a number of IP"
16:07:42 <dhellmann> that feels like a detail we can work out later
16:07:43 <dachary> right
16:07:52 <dachary> the level of detail is tricky
16:07:57 <dachary> for instance regarding IPs
16:08:10 <dachary> IPv6 transit is free sometimes
16:08:29 <dachary> therefore, although it's transit external to the OpenStack cluster, you may want to *not* bill it
16:08:47 <dachary> and the dinstinction between "extrenal" traffic and "internal" traffic may not be enough
16:09:00 <dhellmann> that's an issue for the billing calculation, not the metering, though, right? I may still want to show the customer how much IPv6 traffic they had at $0 so they know how much value they are getting :-)
16:09:05 <dachary> in which case you may want to add more counters but that may prove unpractical
16:10:00 <dachary> the metering should provide dinstinctive counters at least. In the proposed table the external network traffic does not distinguish IPv6 and IPv4
16:10:17 <dhellmann> I agree, we need to allow for separate counters for those
16:11:01 <dhellmann> I assume we will also need to have the counter be per object (per instance, per network, per VIF, etc.) and not global
16:11:01 <dachary> The database schema meeting will be very much about the granularity of these "counters" or whatever agregated value is deemed useful to keep and expose.
16:11:10 <dhellmann> cool, I can wait for that :-)
16:11:40 <dhellmann> I don't see anything in the objective statement that I disagree with, then
16:11:50 <dachary> ok.
16:12:07 <dhellmann> the short version is: We are collecting data for another system to use to calculate the bill for a tenant
16:12:28 <dhellmann> we aren't going to try to do the actually billing
16:12:29 <dachary> That's also my understanding.
16:12:41 <dhellmann> good, we are on the same page, then
16:12:45 <dachary> I've read carefully
16:12:47 <dachary> #link https://lists.launchpad.net/openstack/msg10334.html
16:13:09 <dachary> and the part about the existing system in nova
16:13:18 <dachary> #link http://wiki.openstack.org/SystemUsageData
16:13:25 <dachary> kept me thinking for a while.
16:13:58 <dhellmann> I have not had a chance to review that, yet
16:14:15 <dachary> It actually makes the metering implemetation easier because Dragon is apparently willing to store and account for a lot of counters.
16:15:21 <dachary> So, instead of harvesting data on its own, the metering agent could query nova on the node itself. And getting the data (I/O for the root device for instance) could be done within the context of nova instead of outside nova.
16:16:19 <dachary> Dragon made it clear that he wishes other OpenStack components also implement a similar metering logic. But it's not his focus and I don't think there is anything yet in swift.
16:16:23 <dhellmann> that seems like a good approach
16:16:28 <dhellmann> how much of that is already implemented?
16:16:29 <dachary> And I did not research what Quantum plans.
16:16:40 <dachary> that => http://wiki.openstack.org/SystemUsageData ?
16:16:52 <dhellmann> yes, sorry
16:16:59 <dachary> all of it.
16:17:03 <dhellmann> oh, wow
16:17:14 <dachary> It's a good source of data.
16:17:32 <dhellmann> oh, these are events not table definitions
16:17:37 <dachary> Yes.
16:17:39 <dhellmann> yeah, so we still need to monitor the events and log them
16:17:44 * dhellmann is reading as we go
16:18:42 <chmouel> dachary: I think the swiftstack were talking about an integrated version for metering swift (merging syn's https://github.com/pandemicsyn/swift-informant)
16:19:20 <dachary> #link https://github.com/pandemicsyn/swift-informant
16:19:24 <dhellmann> yes, this matches what I had been expecting to need to build: something that watches for events and logs them, possibly after post-processing to get data not in the event but needed for billing (like the definition of an instance type) and then a query API on top of it for the tool that extracts the data and aggregates it
16:20:29 <dachary> chmouel: ok :-)
16:20:34 <dachary> Agents running on OpenStack nodes harvest data. The data from the existing agents is collected using a message queue. The data collected is made available thru APIs.
16:20:49 <dachary> ^ this is my onliner to describe the metering architecture.
16:20:49 <uvirtbot> dachary: Error: "this" is not a valid command.
16:20:58 <dachary> ?
16:21:08 <dachary> hahaha I woke up a bot without knowing ;-)
16:21:13 <dhellmann> :-)
16:21:38 <dachary> anyone has a say about the project objectives before we move to the next topic ?
16:22:01 <dhellmann> +1 to adopt those objectives
16:22:13 <dachary> +1
16:22:29 <dachary> #topic Agree on a project name
16:23:08 <dachary> Metering and FillBill are proposed
16:23:24 <dhellmann> Ceilometer
16:23:34 <dhellmann> (an instrument for measuring cloud cover)
16:23:58 * nijaba_tab waves. sorry for being late
16:24:13 <dachary> nijaba_tab: we just agreed on project objectives
16:24:21 <dachary> and chosing a name
16:24:29 <dachary> Ceilometer
16:24:37 <dachary> was proposed by dhellmann
16:24:51 <dachary> I actually like Ceilometer
16:24:55 <nijaba_tab> great, thanks
16:25:07 <dhellmann> some other ideas on: http://en.wikipedia.org/wiki/Meteorological_instrumentation
16:25:07 <dachary> nipper has been proposed on the list but I still prefer Ceilometer
16:25:38 <nijaba_tab> Ceilometer sounds cool
16:25:52 <jayahn> +1 for ceilometer
16:25:53 <dhellmann> Radiosondes may be easier to say out loud
16:26:06 <dachary> +1 for ceilometer
16:26:10 * littleidea wants something without 'meter' in the name
16:26:36 <dhellmann> "A radiosonde (Sonde is French for probe) is a unit for use in weather balloons that measures various atmospheric parameters and transmits them to a fixed receiver."
16:26:38 <Aswad_> +1 for ceilometer
16:26:48 <dachary> littleidea: when I apt-cache search meter I'd like it to show ;-)
16:27:08 <littleidea> dachary: easily solved
16:27:16 <nijaba_tab> I had put fillbill on the wiki.... ;)
16:28:26 <dachary> unless there is another proposal shall we vote ?
16:28:32 <littleidea> does the name need to be decided in this meeting?
16:29:09 <nijaba_tab> littleidea: yes, this is blocking us to create the project on lp, have a ml, etc...
16:29:14 <dachary> littleidea: that's the idea, yes. And then it will be used to create projects, domains, lists etc.
16:29:15 <dhellmann> I think the idea is to decide and get to work on implementation, because we need a name before the infrastructure can be set up
16:29:52 <dachary> Votes for "Metering" : (say +0 or +1)
16:29:57 <dachary> +0
16:30:03 <dhellmann> +0
16:30:07 <littleidea> +0
16:30:10 <jayahn> +0
16:30:15 <Aswad_> +0
16:30:17 <nijaba_tab> +0
16:30:34 <dachary> Votes for "FillBill" : (say +0 or +1)
16:30:39 <dhellmann> +0
16:30:40 <dachary> +0
16:30:44 <jayahn> +0
16:30:44 <Aswad_> +0
16:30:49 <nijaba_tab> +1 just because I proposed it
16:30:59 <Aswad_> :)
16:31:05 <dachary> Votes for "Nipper" : (say +0 or +1)
16:31:15 <dachary> +0
16:31:22 <dhellmann> +0
16:31:25 <nijaba_tab> +0
16:31:31 <Aswad_> +0
16:31:35 <jayahn> +0
16:31:55 <dachary> Votes for "Ceilometer" : (say +0 or +1)
16:31:57 <dachary> +1
16:32:01 <dhellmann> +1
16:32:02 <Aswad_> +1
16:32:07 <jayahn> +1
16:32:22 <zul> +1
16:32:26 <nijaba_tab> +1
16:32:28 <dhellmann> please also record a +1 for me for "radiosonde" when that comes up, I have to leave.
16:32:38 <dachary> k
16:32:52 <dachary> Votes for "radiosondes" : (say +0 or +1)
16:32:55 <dachary> +0
16:33:07 <dachary> dhellmann says : +1
16:33:18 <Aswad_> +0
16:33:18 <littleidea> +1
16:33:49 <nijaba_tab> +0
16:33:49 <littleidea> and remember kids, apt-cache search matches descriptions
16:34:01 <nijaba_tab> true
16:34:07 <flacoste> +0
16:34:15 <dachary> littleidea: yes but *many* descriptions match meter ;-)
16:34:18 <jayahn> +1
16:34:51 <littleidea> dachary: your point is searching for meter is useless anyway?
16:35:03 <dachary> That's all I have. We have a winner I think : "ceilometer".
16:35:26 <mnewby> apologies, what is being named?
16:35:38 <dachary> mnewby: http://wiki.openstack.org/Meetings/MeteringAgenda
16:35:47 <littleidea> a project to meter openstack services
16:35:49 <mnewby> danke :)
16:36:07 <Aswad_> danke!
16:36:17 <dachary> Unless someone objects, I'll move on to the next topic.
16:36:59 <dachary> #topic Agree on a project decision roadmap
16:37:07 <nijaba_tab> yay for ceilometer
16:37:52 <nijaba_tab> so loic and I made a proposal for this on the agenda
16:38:06 <nijaba_tab> loic=dachary
16:38:54 <nijaba_tab> I think these decision are a bit fundamental for the project, so I think having a week to consider each one is not too much
16:39:09 <nijaba_tab> what do you think?
16:39:43 * nijaba_tab blames 3G for the lag
16:39:44 <flacoste> where will discussion around these happens?
16:39:56 <flacoste> on a mailing list, or on IRC during themeeting?
16:40:08 <flacoste> would probably be good to discuss on a list before the meeting, no?
16:40:09 <nijaba_tab> on the ml, during the week before the meeting, then validation on irc meeting?
16:40:20 <dachary> It would make sense to discuss before the meeting, I think.
16:41:08 <dachary> flacoste: so that the meeting is not about thinking of solutions but validating the solutions already proposed. Come to an understanding based on the work done during the week.
16:41:18 <flacoste> agreed
16:41:35 <flacoste> where would that happen? the general openstack list? or a openstack-metering list?
16:42:17 <nijaba_tab> openstack-ceilometer list
16:42:18 <dachary> I'd be tempted to start on the general mailing list because a *lot* of people have shown interest in the topic.
16:42:46 <nijaba_tab> let's cross post at the beginning?
16:42:51 <dachary> ok
16:43:05 <dachary> so that it can move to the specialized list of it's too much spam.
16:43:20 <nijaba_tab> +1 for me
16:43:20 <dachary> However, these discussions are time boxed. 5 topics. 5 weeks.
16:43:30 <nijaba_tab> yes!
16:43:45 <dachary> After that all is decided. It's a fairly simple project. Which is what I like about it ;-)
16:43:50 <nijaba_tab> the first one, for the next meeting, is which db to use
16:44:21 <jayahn> (sorry if i miss this kind of discussion on the beginnig. but..) in the decision roadmap, i would like to see a kind of requirment gathering about what to count, especially for network traffic metering.
16:44:22 <nijaba_tab> simple, until you hit production
16:44:42 <littleidea> I think it should be on the main list
16:44:59 <jayahn> i agree that it should be on the main list.
16:45:07 <dachary> jayahn: agree and I think that's part of the "schema choice".
16:45:16 <jayahn> i see.
16:45:18 <dachary> s/agree/agreed/
16:45:40 <dachary> it's the most difficult part in my opinion
16:45:55 <dachary> decided what to account for, what to aggregate or not.
16:46:12 <littleidea> does it have to be tied to a db?
16:46:17 <nijaba_tab> jayahn, there is a proposal on the blueprint.  Feel free to add to it
16:46:18 <nijaba_tab> on of the main idea is that we do not lock on counters, but on data model
16:46:18 <nijaba_tab> so that anyone can add an agent to gather another counter that was not part of the original plan
16:46:18 <nijaba_tab> and each counter gathering is optional
16:47:08 <nijaba_tab> the schema should be flexible enough not to prevent adding counters
16:47:46 <dachary> nijaba_tab: while this is an open ended definition, metering will define counters by default (the one in the table for instance). I think the core of the discussion will be about these counters.
16:48:20 <nijaba_tab> ok, let's get back on topic.  We have 3 proposal for discussion to take place
16:48:23 <dachary> The default list of counters should be useful to most people and make sense to most usages. It's key to have a consensus on using the metering API.
16:49:30 <dachary> I propose that we modify "Schema choice" for "Schema choice and counter definitions"
16:49:40 <nijaba_tab> 1/On the main mailing list
16:49:40 <nijaba_tab> 2/ on the ceilometer one
16:49:40 <nijaba_tab> 3/on both using cross posting
16:49:40 <nijaba_tab> Who votes for 1?
16:49:51 <flacoste> +1
16:49:51 <littleidea> 1
16:49:54 <jayahn> 1
16:50:01 <dachary> 3
16:50:14 <Aswad_> 1
16:50:42 <dachary> I'll handle the votes because nijaba_tab lags ;-)
16:51:01 * nijaba_tab does lagg badly
16:51:08 <nijaba_tab> thanks dachary
16:51:22 <flacoste> dachary: i like your proposal (adding counter definitions to the schema topic)
16:51:37 <flacoste> dachary: but i think we should do that one before commiting to a database
16:51:37 <dachary> #agreed discussions are On the main mailing list
16:51:38 * nijaba_tab votes for 3
16:51:39 <jayahn> i also like adding counter definitions to the schema.
16:52:00 <flacoste> as the schema and counters we need is more likely to inform the db choice
16:52:06 <flacoste> then limiting our schema based on the db
16:52:17 <flacoste> s/then/rather than/
16:52:17 <dachary> flacoste: yes
16:52:42 <nijaba_tab> So the proposal is to a/ change the order between 1 and 2, b) to add counters to shcema.  works for me
16:53:10 <dachary> #agreed meeting 2 is : Schema choice and counter definitions
16:53:32 <dachary> meeting 3 message queue choice
16:54:19 <flacoste> or meeting 3 database choice?
16:54:25 <dachary> it's about chosing AMQP over RabbitMQ over ... etc
16:55:08 <dachary> flacoste: I'm confused ? Meeting 1 is database choice or did I miss something ?
16:55:11 <littleidea> dachary: RabbitMQ is an implementation of AMQP
16:55:37 <nijaba_tab> dachary: we should invert 1 and 2
16:55:42 <dachary> ok
16:56:17 <flacoste> ah, i was counting from this current meeting, but i guess we are using 0, traditional off by one error :-)
16:56:28 <dachary> :-D
16:56:39 <dachary> I agree that database choice should come after
16:56:48 <dachary> littleidea: right ... :-D
16:57:12 <nijaba_tab> so that we know th schema before choosing the db
16:57:12 <nijaba_tab> c vs pascal...
16:57:21 <jayahn> does openstack already have
16:57:33 <jayahn> .. oops.. mistyping.
16:57:37 <littleidea> Why do we need to choose a DB?
16:58:01 <dachary> littleidea: we need to chose a storage system, definitely
16:58:08 <nijaba_tab> littleidea: because of the massive amount of data we are going to collect
16:58:18 <littleidea> seems like what we should define is a service interface and mechanisms for gathering the information
16:58:38 <flacoste> littleidea: maybe the storage should be pluggable, but let's save that for the DB discussion :-)
16:58:38 <dachary> Current state: 1 schema and counter definitions, 2 storage system, 3 message queue choice, 4 API message format, 5, external API definition
16:58:44 <littleidea> nijaba_tab: then clearly the db should be cassandra
16:58:51 <dachary> flacoste: ok
16:59:14 <dachary> littleidea: I think that's exactly what the second meeting should be about, indeed ;-)
16:59:25 <nijaba_tab> yes, but we could be limited by the type of db, so the reference implementation choice does matter
16:59:33 <flacoste> dachary: does it make sense to discuss 4 and 5 after 1? because again, the message queue and storage might restrict these?
16:59:34 <dachary> Current state: 1 schema and counter definitions, 2 chose a plugable storage system, 3 message queue choice, 4 API message format, 5, external API definition
17:00:14 * dachary thinking
17:00:18 <jaypipes> dachary, LinuxJed, nijaba_tab: heya, the QA meeting starts now... in this channel :)
17:00:34 <flacoste> we overran
17:00:43 <littleidea> to the extent schema is about defining the data we want to collect, that seems right, if schema is about creating tables in a relational model, I think we're going about this the wrong way
17:00:46 <jaypipes> flacoste: no big deal :)
17:00:46 <LinuxJedi> jaypipes: ??
17:00:51 <vladimir3p> jaypipes: I thought that we have a volume meeting at 10am PST
17:00:51 <dachary> jaypipes: we'll cut short sorry about that
17:01:07 <dachary> #endmeeting