16:00:16 <dachary> #startmeeting
16:00:16 <openstack> Meeting started Thu May  3 16:00:16 2012 UTC.  The chair is dachary. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:00:17 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
16:00:26 <dachary> #chair nijaba dachary
16:00:26 <dachary> #meetingname ceilometer
16:00:27 <openstack> Current chairs: dachary nijaba
16:00:28 <openstack> The meeting name has been set to 'ceilometer'
16:00:36 <dachary> #topic actions from previous meetings
16:00:36 <dachary> #info creation of the ceilometer project
16:00:36 <dachary> #https://launchpad.net/ceilometer
16:00:36 <dachary> #info The repository for the ceilometer project has been created
16:00:36 <dachary> #link https://github.com/stackforge/ceilometer
16:00:37 <dachary> #info and the first commit was successfully reviewed and merged today https://review.stackforge.org/#/c/25/
16:01:03 <dachary> #topic meeting organisation
16:01:03 <dachary> #info This is 1/5 meetings to decide the architecture of the Metering project https://launchpad.net/ceilometer
16:01:03 <dachary> #info Today's focus is on the definition of the counters / meters and the associated schema for the storage
16:01:03 <dachary> #info It is the conclusion of the discussions held on the mailing list and the goal is to make a final choice that will then be implemented.
16:01:03 <dachary> #info The meeting is time boxed and there will not be enough time to introduce inovative ideas and research for solutions.
16:01:03 <dachary> #info The debate will be about the pro and cons of the options already discussed on the mailing list.
16:01:03 <dachary> #link https://lists.launchpad.net/openstack/msg10810.html
16:01:21 <dachary> any comments on the meeting organisation ?
16:01:31 <flacoste_ipod> No
16:01:42 <nijaba> ditto
16:01:46 <Aswad_> no
16:01:52 <zinux> No
16:01:52 <dhellmann> sounds good to me
16:02:00 <jd___> nop
16:02:08 <dachary> ok :-) Now the real thing:
16:02:10 <dachary> #topic counter definitions
16:02:10 <dachary> #info Proposed http://wiki.openstack.org/EfficientMetering#Counters
16:03:04 <nijaba> note: this list is not a locked list, but a list of the counters we will want to have in a first version
16:03:36 <nijaba> hoping that if we can satisfy this set, we will be able to add new ones easily later
16:03:46 <dachary> I'm happy with the counters as they are. I've crossed check them with sales and it has all we need for billing ;-)
16:04:02 <dachary> I'm not sure they will fit in what HP has done but I have no input from them so far.
16:04:23 <nijaba> any other comments?
16:04:54 <dhellmann> for the network counters, are we still talking about per-network or did we move to per IP/VIF?
16:05:10 <dachary> we moved per IP/VIF
16:05:11 <jd___> FWIW, I've added a list of possible counter to add for Swift
16:05:35 <dhellmann> dachary, thanks, that's what we need
16:06:02 <dhellmann> this list looks good, and if we plan to allow it to be extended that should take care of anything I don't know we need, yet
16:06:04 <dachary> dhellmann: the IP shows in the database schema (the ID of the resource)
16:06:24 <dachary> dhellmann: before that the database schema did not have an id
16:06:26 <dhellmann> as the unique id of the resource for the counter?
16:06:33 <dachary> yes
16:06:35 <dhellmann> got it
16:06:49 * dachary checking the counters again
16:07:21 <dhellmann> the note for net_float still talks about "number of floating IPs" but we're actually counting time per IP, right?
16:07:23 * nijaba does not see reference to the IP/VIF chjoice
16:07:37 <jd___> I'm not sure we can fetch net_*_int directly, I think we will have net_int and net_out, and we can probably grab net_ext_* and deduce net_int_* from that
16:07:47 <dachary> jd___: added:
16:07:51 <dachary> * Number of object in Swift
16:07:51 <dachary> * Number of containers in Swift
16:07:51 <dachary> * Number of GET/HEAD/PUT/POST requests in Swift
16:08:30 <nijaba> sounds good to me, but should be included in the table
16:08:56 <dachary> dhellmann: yes
16:09:14 <dhellmann> for quantum there will be L3 devices (logical or hardware) so we (DreamHost) will want to count those and possibly bill for them
16:09:18 <dachary> #action dachary fix the note for net_float still talks about "number of floating IPs"
16:09:42 <jd___> dhellmann: do there's no int/ext distinction directly on that?
16:09:48 <jd___> s/do/so/
16:10:10 <dhellmann> we want to account for the router, separately from the traffic it passes
16:10:11 <dachary> #action jd___ include Number of object in Swift, Number of containers in Swift, Number of GET/HEAD/PUT/POST requests in Swift in the table
16:10:35 <dhellmann> we don't know yet what we will charge for, but if the tenant can create a bunch of routers we want to make sure we could charge for them in the future
16:11:33 <nijaba> dhellmann: do you see anyhting in the checma that could prevent that?
16:11:42 <nijaba> s/checma/schema/
16:11:57 <flacoste_ipod> I think this is just more counters?
16:11:59 <dhellmann> nijaba, no, I think we can handle it with the schema
16:12:00 <dhellmann> right
16:12:41 <flacoste_ipod> Or is it exisiting counter with a different resource ids?
16:12:46 <dhellmann> dachary, do we care about the # of objects or just the # of containers? (I'm not that familiar with swift)
16:12:49 <nijaba> so I think we should be good, then.  will just need to work with quantum on having an "agent" that sends the stuff to our queue
16:13:26 <dhellmann> a counter for routers would look like c1 but refer to the router id instead of the instance id
16:13:26 <dachary> there will be a lot of change when quantum, cinder are integrated properly. And having a uniform structure for counters is a big plus to evolve quickly without complex schema change. I'm concerned about how http://wiki.openstack.org/SystemUsageData will evolve because the structure of the information is not uniform.
16:13:28 <nijaba> dhellmann: as all counters should be made optional to collect, I don't think it matters what each of us care about
16:13:42 <dhellmann> nijaba, agreed
16:13:44 <dachary> dhellmann: we care about both because they impact performances / replication.
16:14:01 <dhellmann> dachary, makes sense
16:14:35 <dachary> dhellmann: if someone makes a huge number of small objects, we may want to charge a little more. Compared to creating a single large object.
16:14:59 <dhellmann> dachary, sure, that makes sense
16:15:26 <dhellmann> so count # of objects per container?
16:16:24 <dhellmann> with the idea that containers with large # objects may cost more than containers with small # of objects
16:16:28 <dachary> dhellmann: good question. I'm not sure it makes a difference. For a given tenant I kind of assume the total number of container + the total number of objects matters.
16:16:41 <dhellmann> or you might just care about # objects for an account, which can be handled in aggregation logic
16:17:15 <flacoste_ipod> Did we settle the "account" définition?
16:17:16 <dachary> dhellmann: there 1 level of hierarchy and I don't think there is a penalty when you have 10000 objects on 100 containers versus 10000 objects in 1 container.
16:17:29 <nijaba> flacoste_ipod: nope
16:17:32 <dhellmann> right, but if you collect the data with the relationship intact you can ignore the relationship later. if you don't include the relationship, you can't discover it later
16:17:46 <nijaba> dhellmann: agreed
16:17:48 <dhellmann> what is the question about account? isn't account == tenant?
16:18:09 <jd___> dhellmann: for now it's really easy to retrieve total number of containers and objects for a tenant (1 request), but retrieving the number of objects per containers is probably harder from what I know
16:18:25 <nijaba> dhellmann: well, one question that we could ask is should we have a source + account, instaead of just account
16:18:33 <dhellmann> ok, maybe we do the simple thing for now and implement another counter later if we need to
16:18:38 <nijaba> so that in the future we could extend ot other id stores
16:18:47 <dhellmann> what is "source"?
16:19:01 <nijaba> for example, if we wanted to meter a paas on top that would use something else than keystone
16:19:05 <flacoste_ipod> Does tenant include the project?
16:20:00 <dachary> I'm not sure how to encode the relationship container / object
16:20:02 <flacoste_ipod> we basically want a user/project tuple
16:21:02 <nijaba> dachary: use the id to specify the countainer for the object count?
16:21:20 <dachary> nijaba: ok
16:21:44 <dachary> #action dachary add note about the fact that the resource_id for the object count is the container_id
16:22:16 <nijaba> any comment about transforming account_id into project/account tupple?
16:22:26 <dachary> nijaba: you propose that we add a "source" to the schema ?
16:22:46 <dachary> I agree
16:22:50 <dhellmann> flacoste_ipod, tenant is the same thing as project, right?
16:23:06 <nijaba> dachary: either that, or use one field as a composite key
16:23:13 <flacoste> dhellmann: you tell me, i'm not intimate with the keystone data model :-0
16:23:14 <jd___> dachary: so that means we need to fetch object count per container?
16:23:26 <dhellmann> nijaba, -1 to composite keys, let's just add both fields
16:23:37 <nijaba> dhellmann: I tend to agree with you
16:23:43 <dachary> jd___: yes. That won't be cheap.
16:23:47 <dhellmann> flacoste_ipod, I think the terms mean the same thing
16:23:54 <jd___> dachary: indeed.
16:23:56 <flacoste> then we are fine
16:24:27 <dachary> flacoste: it's the same indeed tenant == project
16:24:34 <dhellmann> dachary, maybe we can convince swift to give us a new API for that?
16:24:40 <nijaba> the change should apply to both account and event records then
16:24:43 <dachary> dhellmann: yes
16:25:03 <dachary> I think we're moving ahead, should we conclude this topic and agree on the counters before we move to the schema ?
16:25:11 <dachary> I propose:
16:25:14 <dhellmann> nijaba, which change applies to both records?
16:25:34 <nijaba> dhellmann: the 2 fields instead o just account_id
16:25:35 <dachary> http://wiki.openstack.org/EfficientMetering#Counters is agreed on, provided the actions listed above are carried out.
16:26:00 <nijaba> dachary: I thnk we miss an action to record the VIF/IP choice
16:26:12 <dhellmann> yeah, we need to make sure that is clear somewhere
16:26:26 <dachary> dhellmann: I'm indeed convinced that we need to talk to each component to agree on ways to extract metering information. swift, nova, quantum etc.
16:27:10 <dachary> nijaba: this belongs to the schema, doesn't it ? It's the resource_id of the net* counters. Or am I missing something ?
16:27:46 <nijaba> dachary: I think it needs to be in net counters definition
16:27:58 <dachary> ok.
16:28:04 <dhellmann> didn't the counters table get an id column at some point, or was that on the mailing list?
16:28:06 <dachary> secondary is the IP ?
16:28:18 <dachary> secondary is the IP for net_out_ext for instance ?
16:28:32 <nijaba> sounds good to me.  dhellmann?
16:28:46 <dachary> dhellmann: I added a resource_id at some point but then it was move to the schema because all counters need that.
16:29:05 <dachary> nijaba: what would be the resource_id for a net_out_ext counter then ?
16:29:19 <dhellmann> dachary, we should document what the resource id for each counter is meant to be, though, right?
16:29:27 <dachary> dhellmann: right
16:29:59 <dhellmann> ok, I didn't see that in the "storage" section. Is there another place the schema is defined?
16:30:33 <dachary> #action jd___ document the resource_id for each counter
16:30:46 <dachary> dhellmann: resource_id ( the unique ID of a resource, for instance the IP, the nova instance id, the glance image id etc. )
16:30:50 <nijaba> dachary: good point.  I think we have a small misalignment between schema and counters that we need to fix
16:30:55 <dachary> in http://wiki.openstack.org/EfficientMetering#Storage
16:31:03 <dhellmann> dachary, yeah, I took those as examples rather than hard documentation
16:31:29 <dhellmann> when we're done, what I would like is something that describes the general table schema and then something that says for each counter exactly what goes in the fields of that table
16:31:38 <nijaba> dachary: and we need to be able to record secondary field counters in the schema too
16:31:40 <dhellmann> I think we're close to that with the existing tables
16:31:49 <jd___> doesn't schema miss resource_type too?
16:32:04 <dhellmann> the resource type is implied by the counter id, isn't it?
16:32:22 <nijaba> dhellmann: yes
16:32:53 <dhellmann> dachary, back to your question about secondary id being IP, was that for all network counters?
16:32:56 <jd___> and the counter id is in the schema ? (can't see it)
16:33:08 <dhellmann> counter_type?
16:33:21 <dachary> dhellmann: yes
16:33:25 <jd___> dhellmann: ah ok :)
16:33:26 <dhellmann> (btw, I like mark's suggestion of calling them "meters" instead of "counters")
16:33:27 <dachary> #action jd___  describes the general table schema and then something that says for each counter exactly what goes in the fields of that table and show how secondary field counters are recorded in the in the schema too
16:33:42 <dachary> dhellmann: me too
16:34:34 <nijaba> dachary: nice action summary
16:35:00 <dachary> I think we agree on the basics and on the fact that the documentation needs to be clarified
16:35:08 <dhellmann> yes
16:35:21 * dhellmann likes progress
16:35:33 <dachary> I should have left the "resource_id" column in the counters section where there is a description for each counter.
16:35:42 <nijaba> I think everyone here does, or we would not bother ;)
16:35:48 <dhellmann> :-)
16:35:56 <dachary> do we agree to s/counter/meter/ ?
16:36:00 <dhellmann> +1
16:36:07 <nijaba> +1
16:36:07 * dachary +1
16:36:36 <flacoste> +1
16:36:41 <Aswad_> +1
16:36:50 <zinux> +1
16:37:11 <dachary> #agreed s/counter/meter/
16:37:38 <dachary> http://wiki.openstack.org/EfficientMetering#Counters is agreed on, provided the actions listed above are carried out. ?
16:37:43 <dhellmann> +1
16:37:54 <nijaba> +1
16:38:00 <Aswad_> +1
16:38:18 <flacoste> +1
16:38:52 <dachary> #agree http://wiki.openstack.org/EfficientMetering#Counters is agreed on, provided the actions listed above are carried out. ?
16:39:05 <dachary> #topic schema definition
16:39:05 <dachary> #info Proposed http://wiki.openstack.org/EfficientMetering#Storage
16:39:20 <dachary> I realize it needs cleanup.
16:39:36 <dachary> I was entirely focused on the part describing the fields necessary for the meters
16:39:44 <dachary> (s/counter/meter/ in effect ;-)
16:39:54 * nijaba brings back the requirement for a tupple instead of just "account_id"
16:40:13 <flacoste> it seems that account_id should be renamed to tenant_id
16:40:21 <flacoste> and that contains both account and project
16:40:31 <flacoste> as tenant_id is a keystone identifier really
16:40:31 <dhellmann> flacoste, is account the same as user?
16:40:35 <flacoste> which encapsulates both
16:40:41 * dhellmann is confused by shifting terminology
16:40:42 <dachary> #action jd___ clarify / document the fact that the secondary field or the resource_id field carries the IP/VIF for the meters related to network so that they can be "per IP"
16:40:46 <flacoste> dhellmann: indeed
16:40:54 <flacoste> dhellmann: yes, in my mind account == user
16:40:59 <flacoste> we should use keystone terminoly
16:41:02 <flacoste> and i shouuld learn it :-)
16:41:21 <whitt> we are using tenant at the lowest level
16:41:31 <dhellmann> ok. I'm not sure which is the current terminology, but for now let's talk in terms of "tenant" and "user" and then agree to figure out the right values when we write the docs
16:41:39 <flacoste> +1
16:41:49 <nijaba> +1
16:42:10 <dhellmann> I agree that we should have both tenant and user. I don't know why we would want to have them in a single field instead of separately, though. Can you elaborate flacoste?
16:42:35 <dachary> so the fields should be source (paas, iaas, etc.), user_id, tenant_id, resource_id, counter_type, counter_volume, counter_duration, counter_datetime, secondary type, message_signature, message_id ?
16:42:38 <zinux> The name tenants will perhaps disappear for projects
16:43:06 <dachary> zinux: you're correct
16:43:16 * dhellmann thought it was the other way around, but ok
16:43:18 <dachary> so the fields should be source (paas, iaas, etc.), user_id, project_id, resource_id, counter_type, counter_volume, counter_duration, counter_datetime, secondary type, message_signature, message_id ?
16:43:28 <dhellmann> dachary, where does the value for source come from?
16:43:32 * dachary confused now
16:43:34 <dachary> :-D
16:43:40 <dhellmann> is it in the event?
16:43:52 <dhellmann> dachary, join the club :-)
16:44:05 <nijaba> source is set by the agent, based on what it uses for auth
16:44:27 <dachary> nijaba: suggested it. I'm not sure how it set but I can see the value.
16:45:00 <nijaba> so we can later add agents for projects outside of openstack, but yet billable by the same owner
16:45:09 <dhellmann> nijaba, that assumes separate agents though, right? I thought we were going to try monitoring notifications to start
16:45:30 <dhellmann> ah, so "source" for the event monitoring may just be "internal" or something?
16:45:30 <dachary> that allows separate agents
16:46:06 <nijaba> dhellmann: the concept of agents is fuzzy, as they can both be sepaate or intergrated.  who cares, as long as they speak the same queue API language?
16:46:27 <nijaba> dachary: no, for openstack project, it should be keystone
16:46:42 <dachary> hum
16:46:43 <dhellmann> it's always easier to add something than to remove it, so I want to make sure version 1 only includes stuff we're going to use out of the gate
16:47:02 <dhellmann> we also need to be able to document the expected value(s) for each field, so I want to make sure I understand that
16:47:20 <dhellmann> I agree that a "source" or "agent" field may provide value, FWIW
16:47:23 <flacoste> dhellmann: please bear with me, i apologize for being keystone illeterate, i thought a tenant encapsulates both project and user, if that's not the case, then yeah, we want two fields
16:47:23 <dachary> me too.
16:47:24 <nijaba> dhellmann: so expected value for all existing counters should be keystone
16:47:44 <dachary> tenant == project
16:47:51 <dachary> only one term replaces the other
16:47:54 <flacoste> then yes, we want both tenant and user
16:47:56 <dachary> and indeed we need both
16:48:01 <dhellmann> flacoste, a user may have access to more than one tenant but a tenant owns things
16:48:02 <dachary> flacoste: yes :-D
16:48:11 <flacoste> so that we can bill projects and/or users depending on our business model
16:48:23 <flacoste> sorry for the confusion
16:48:31 <flacoste> tenant + user is what we need
16:48:33 <nijaba> np... better all agree now...
16:48:59 <dachary> nijaba: I trust your vision when you say that encoding the authentication source (a URL maybe ? ) is a way to distinguish the global set of counters. And maybe aggregate them for billing purposes without mixing them.
16:49:10 <nijaba> dachary: yep
16:49:45 <dhellmann> so the source is the *authentication* source?
16:49:51 <dachary> nijaba: would you agree to explain the rationale in the document and discuss it afterwards on the list ?
16:49:57 <dhellmann> not the thing that triggered the operation that is going to be costing the customer $?
16:50:16 <dhellmann> dachary, good plan, we're running out of time
16:50:21 <nijaba> ok
16:50:26 <dachary> #action nijaba describe the source field rationale and use case, initiate a thread to validate its use.
16:50:49 <dachary> so the fields should be source (to be discussed), user_id, project_id, resource_id, counter_type, counter_volume, counter_duration, counter_datetime, secondary type, message_signature, message_id ?
16:51:02 <dachary> are we missing something ?
16:51:24 * nijaba does not think so
16:51:46 <dhellmann> +1
16:51:53 <nijaba> +1
16:52:03 <zinux> +1
16:52:09 * dhellmann may want to bike shed the name of "secondary type" on the list
16:52:20 <Aswad_> +1
16:52:22 * dachary +1
16:52:36 <jd___> payload? :)
16:52:41 <dachary> dhellmann: I don't like it either
16:52:50 <dachary> jd___: +1 on payload
16:52:55 <dhellmann> ok, let's table that discussion for now, though
16:53:01 <dachary> ok
16:53:04 <dhellmann> 8 more minutes, right?
16:53:08 <nijaba> right
16:53:09 <dhellmann> what else is on the agenda?
16:53:19 <nijaba> that were the 2 main items
16:53:20 <dachary> #agree the fields should be source (to be discussed), user_id, project_id, resource_id, counter_type, counter_volume, counter_duration, counter_datetime, secondary type / payload, message_signature, message_id ?
16:53:47 <dhellmann> someone needs to propose a formula for calculating the message signature
16:53:48 <dachary> #action jd___ update the documentation
16:54:10 <dachary> #action jd___ document the resource_id for each counter
16:54:28 <dachary> dhellmann: yes ;-)
16:54:43 <jd___> dhellmann: gpg ;)
16:54:50 <nijaba> dhellmann: should that be part of the message API discussion? (24 may)
16:54:51 <dachary> #topic discuss API assumptions
16:54:59 <dhellmann> nijaba, that sounds like a good idea
16:55:04 <dachary> hum
16:55:06 <dhellmann> I propose hmac
16:55:18 <dachary> I feel like we should move to the last topic
16:55:19 <dachary> #topic open discussion
16:55:29 <dachary> there won't be room for anything else really ;-)
16:55:42 <dachary> dhellmann: +1 on hmac
16:56:22 <dachary> jd___: could you tell us about your plans to bootstrap the software ?
16:56:43 * nijaba proposes to set the signature algo on the discusion to be agreed on may 24
16:56:47 <Aswad_> are we storing the past data too? should there be a time stamp for each recorded data - just a thought
16:56:53 <dachary> nijaba: ok
16:57:07 <dhellmann> nijaba, +1
16:57:25 <nijaba> Aswad_: counter_datetime?
16:57:31 <dachary> there is a timestamp already counter_datetime
16:57:56 <jd___> dachary: since there's no decision made on everything, I'll start with things that are agreed on, like an agent getting nova compute creation events, swift middlewares
16:57:58 <Aswad_> ah i missed it
16:58:01 * dhellmann probably wants to bike shed that column name, too :-)
16:58:08 <dachary> does anyone have any advice on how to approach the project leads with regard to introducing metering code ? swift ?
16:58:24 <nijaba> dachary: ml sounds best?
16:58:40 * dachary taking advantage of the last 90 secs ;-)
16:58:43 <dachary> mouhahahah
16:59:09 <dhellmann> right, we should send a concrete proposal (we want to record X for project Y, and will need to change it in Z way)
16:59:14 <dachary> thank you everyone, I'm glad we came to an understanding :-D
16:59:34 <Aswad_> :) thanks !
16:59:46 <nijaba> thanks everyone
16:59:59 <dhellmann> jd__, are you going to be working on a prototype in the new repo or separately on github?
17:00:03 <dachary> jd___: +1
17:00:05 <dachary> #endmeeting