15:01:11 <jd__> #startmeeting ceilometer
15:01:13 <openstack> Meeting started Thu Dec 12 15:01:11 2013 UTC and is due to finish in 60 minutes.  The chair is jd__. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:01:14 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
15:01:16 <openstack> The meeting name has been set to 'ceilometer'
15:01:20 <dragondm> o/
15:01:20 <sileht> o/
15:01:20 <thomasem> o/
15:01:23 <herndon> o/
15:01:29 <ildikov> o/
15:01:37 <eglynn> o/
15:01:46 <Alexei_987> \o/
15:02:03 <sandy__> o/
15:02:04 <gordc> o/
15:02:04 <eglynn> jd__: I just update the agenda
15:02:08 <llu-laptop> o/
15:02:08 <eglynn> #link https://wiki.openstack.org/wiki/Meetings/Ceilometer#Agenda
15:02:27 <eglynn> (... was last week's previously)
15:02:31 <dperaza> o/
15:02:37 <nealph> o/
15:02:42 <nprivalova> o/
15:03:23 <jd__> #topic Models validation (Alexei_987)
15:03:38 <jd__> go ahead Alexei_987
15:03:40 <Alexei_987> I have a patch with a draft proposal https://review.openstack.org/#/c/60885/
15:04:20 <Alexei_987> dhellmann says that that's should not happen cause we already have validation in WSME + possibly nova-objects
15:04:59 <Alexei_987> my reasoning is the following: storage/models is the middleware between different application layers - API, storage
15:05:08 <Alexei_987> this is the exact place validation should be
15:05:19 <eglynn> from the gerrit comment trail doesn't sound dhellmann is ruling your approach out
15:05:33 <Alexei_987> we discussed it a little bit in chat yesterday
15:05:50 <eglynn> ... more just taking the conversative "let's see if anything we have already can do the job" approach, no?
15:06:08 <dragondm> Yah, I looked at that earlier. Using __slots__ for that is an interesting idea (I've done the same for an ORM I wrote awhile back for some internal Rackspace stuff), but yah, I think we just need to agree on a direction here.
15:06:23 <dperaza> Alexei_987: maybe a blueprint linked to patch explaining the validation benefits will help
15:06:57 <eglynn> Alexei_987: did dhellmann come down more firmly against your approach in your IRC discussion?
15:06:57 <dperaza> that way we can all pitch in help with review
15:07:00 <Alexei_987> ok in such case I'll have to create a comparison of existing validation scenarios
15:07:23 <eglynn> Alexei_987: that sounds reasonable to me
15:07:24 <Alexei_987> to create an overview of possible options
15:07:56 <jd__> eglynn: you got this right
15:08:11 <Alexei_987> ok I guess we all agreed on this
15:08:12 <jd__> the point dhellmann raised and I agree, is that we shouldn't write another set of code of does validation
15:08:33 <jd__> we should leverage something else, that something else being to be determined, I'm working on that
15:09:02 <Alexei_987> yeah but still I insist that we should use slots and have the validation in models
15:09:24 <Alexei_987> this will allow us to remove duplication of logic in API and storage
15:09:34 <Alexei_987> related to query processing
15:10:01 * dhellmann apologizes for arriving late
15:10:11 * jd__ fires dhellmann
15:10:35 <jd__> Alexei_987: again, we're not against validating, we're against writing code that does the validation
15:10:42 <dperaza> Alexei_987: how will that look at the API layer
15:11:08 <dhellmann> the issue I have with the patch is not *where* the validation is being done, it is with the fact that it creates another framework for declaring classes, types, and validators
15:11:10 <Alexei_987> on API layer we'll reuse models to do the validation of incoming parameters
15:11:15 <dperaza> is there work needed on API to catch exception and raise the right http error code
15:11:32 <Alexei_987> dhellmann: it's not a framework. it's a tiny piece of code that does it
15:11:40 <dperaza> or will that show as 500 on client side
15:12:00 <Alexei_987> dperaza: it will catch exception and fire a 400 code with description from exc message
15:12:08 <dhellmann> Alexei_987: that "tiny bit of code" is completely different from the 2-3 other ways we do the same thing elsewhere, so it doesn't matter how big it is
15:12:34 <Alexei_987> true but we don't have nova objects :)
15:12:42 <Alexei_987> and WSME validation won't do on models layer
15:12:43 <dperaza> Alexei_987: will this validate metadata too
15:13:16 <Alexei_987> dperaza: it depends on our needs. it can be extended to do different kind of things
15:13:35 <Alexei_987> my main goals - simple type validation, referential integrity control for storage
15:13:45 <dperaza> right now metadata validation is a gap I see
15:14:15 <dperaza> I think it is bacause metadata is meant to be free form, but some folks see different
15:14:37 <dhellmann> we have to be very careful about the validation we do, so we don't introduce "rules" that prevent us from collecting data that is "true" even if it looks like it might be bad
15:15:02 <dhellmann> the storage models are used to save the data that comes directly from notifications, and we can't necessarily restrict what the other services are telling us
15:15:30 <dhellmann> that's why we focused most of the validation on the models in the API layer, because those come from users, not services
15:15:38 <dperaza> dhellmann: I lean to agree with that
15:15:54 <Alexei_987> dhellmann: I cannot agree with the last one
15:15:59 <dhellmann> the same applies to metadata -- we don't know what rules to enforce about metadata
15:16:10 <Alexei_987> dhellmann: API layer is not the place to validate incoming data
15:16:23 <dhellmann> we tried to create a completely agnostic system that would store whatever metadata was coming to us
15:16:50 <dhellmann> Alexei_987: we *CANNOT* reject data from notifications just because it doesn't look right to us -- we don't make the rules for what goes into notifications
15:17:02 <thomasem> Alexei_987, Ceilometer isn't the source of truth in this whole architecture. If we start putting rules around what a notification should look like, we may lose data that we actually want, even if it's incorrect.
15:17:02 <Alexei_987> dhellmann: I'm not telling to reject it
15:17:13 <dperaza> so we have not way to distiguish from "bad" queries and querries that simply does not match
15:17:14 <dhellmann> thomasem: exactly my point
15:17:19 <Alexei_987> we don't have to validate free for meta
15:17:25 <dperaza> what i hear is that this is by design
15:17:26 <thomasem> The source of truth is each component pushing the notifications, and that's the only communication we have.
15:17:31 <Alexei_987> but we need to make sure that basic data is valid
15:17:35 <Alexei_987> timestamp, etc.
15:17:51 <thomasem> Alexei_987, What would we do if the timestamp is incorrectly formatted?
15:17:58 <thomasem> from, say, a Glance notification?
15:18:06 <eglynn> and if it fails validation, discard?
15:18:11 <eglynn> thomasem: snap!
15:18:15 <nprivalova> fix it :)
15:18:19 <Alexei_987> for timestamp - replace with current datetime
15:18:28 <thomasem> And scramble the event stream?
15:18:32 <thomasem> So we can't recreate for audits?
15:18:52 <sileht> /60
15:18:53 <dragondm> File a bug w/ the source project, methinks..
15:18:53 <dperaza> https://github.com/openstack/ceilometer/blob/master/ceilometer/api/controllers/v2.py#L343
15:19:18 <dperaza> that one validates queries comming from client
15:19:21 <dhellmann> no, replacing the timestamp would completely invalidate any audit trail
15:19:29 <thomasem> What if we have a request that sends 10 notifications, and one of those timestamps happens to be poorly formatted and we replace with utcnow()? Then we can't ensure the integrity of that. We lost the integrity in trying to validate something we're not responsible for.
15:19:30 <eglynn> better to preserve even the apparent wrongness AND file a bug with the originating service?
15:19:46 <dhellmann> eglynn: yes
15:19:47 <thomasem> ^^
15:19:50 <Alexei_987> well we can't do that
15:19:59 <Alexei_987> for example MySQL needs to have valid input
15:20:15 <dragondm> True, if the timestamp is garbage, we might not event be able to store the data.
15:20:17 <dperaza> I do think that validation on V2 API should be move to more common code
15:20:18 <Alexei_987> so it has to receive valid date
15:20:49 <thomasem> Alexei_987, That's actually a problem in and of itself. We're enforcing relationships and datatypes on things we're not responsible for the relationship or datatype of.
15:21:10 <thomasem> It's not our data. It's Nova's, Glance's, Swift's.
15:21:25 <thomasem> We're just persisting it for auditing, metering, monitoring...
15:21:28 <Alexei_987> again - we should only validate essential fields that are required by ceilometer and we may not validate free form data
15:21:29 <thomasem> And hoping it's correct
15:21:29 <nprivalova> btw, if timestamp is incorrect Ceilometer will not be able to take this Sample into account, right?
15:22:02 <eglynn> yeap, the timestamp is garbage, the sample can't be aggregated
15:22:12 <thomasem> That's true
15:22:38 <dragondm> (not to mention it'd be a violation of the spec such as it is, on a notification. )
15:22:39 <Alexei_987> the same goes if we'll create reference id's to missing resources
15:22:49 <nprivalova> so we put it into db but actually do not look at it in most cases
15:22:50 <thomasem> dragondm, yep, that's true too hehe
15:23:00 <dhellmann> we will have a lot of resource ids in our database that don't exist anywhere else in openstack
15:23:13 <dhellmann> objects are deleted all the time, but we still need to record their history
15:23:31 <dhellmann> and we designed the schema to let us record data about resources that are not even in openstack at all
15:23:42 <Alexei_987> ok agreed
15:23:49 <jd__> good point dhellmann
15:23:53 <dragondm> Yup.
15:24:08 <dhellmann> what's the adage about being liberal in what you accept but conservative in what you send?
15:24:27 <eglynn> BTW what's motivating this validation requirement?
15:24:32 <eglynn> i.e. do we have a history of garbage fields being emitted by the OS services in real notifications?
15:24:45 <dhellmann> the best way to ensure good data in the notifications, including our own format, is to provide a library that's easy to use to generate them
15:24:54 <sandy__> by this logic though, validation should occur in the schema of the notification generation and all downstream services should assume them to be gospel (or share the same validation code as the generator)
15:24:57 <eglynn> or is it more of a theorectical what-if scenario?
15:25:06 <dhellmann> oslo.messaging (or oslo rpc) includes that for standard notifications, and jd__'s work on notification schemas will take it even further
15:25:21 <thomasem> Yeah
15:25:27 <dperaza> I agree with a validation library
15:25:38 <dhellmann> eglynn: good question
15:25:51 <dhellmann> Alexei_987: have you seen bad data in production?
15:25:58 <sandy__> we have
15:26:02 <thomasem> yeah
15:26:09 <dhellmann> which services generated the notifications?
15:26:11 <Alexei_987> no.. my motivation is improving internal code logic
15:26:18 <Alexei_987> and reducing duplication
15:26:24 <sandy__> glance, sending a list instead of a dict payload
15:26:27 <Alexei_987> so only code improvement
15:26:29 <sandy__> for error events
15:26:55 <dhellmann> sandy__: a list of …?
15:27:01 <dhellmann> error messages?
15:27:05 <dragondm> sandy__: We filed a glance bug on that?
15:27:06 <apmelton1> dhellmann: sandy__: it was actually a string
15:27:08 <nprivalova> oh, I have so many questions about Glance nnotifications today…
15:27:18 <apmelton1> it was just a string message about the error
15:27:22 <sandy__> dhellmann, things ... just the consumer always assumed payloads are dicts, so get() failed
15:27:24 <dhellmann> apmelton1: ah
15:27:36 <sileht> glance doesn't use oslo incubator for rpc/notification so ....
15:27:38 <sandy__> apmelton1, sorry, thanks ... same problem
15:27:44 <dhellmann> sandy__: ok, that's a bug in ceilometer, I think
15:27:49 <dragondm> payloads *should* always be dicts :-P
15:27:55 <apmelton1> sileht: I think they just switched to oslo messaging
15:27:56 <dhellmann> because we don't get to dictate the notification body or payload format
15:28:00 <dhellmann> dragondm: I'd like that, too. :-)
15:28:00 <sandy__> dragondm, yeah, until they're not
15:28:04 <thomasem> dragondm, optimism, I like it.
15:28:09 <sileht> apmelton1, good news
15:28:13 <apmelton1> definitely
15:28:16 <dhellmann> sileht: yet, right? :-)
15:28:20 <nprivalova> sileht, apmelton1, yes, they did
15:28:29 <dhellmann> apmelton1: oh, good news indeed
15:28:38 <nprivalova> no :)
15:28:41 <sileht> dhellmann, I have fight and fail to get oslo-incubator in glance for havana
15:29:03 <sandy__> so, I guess my only concern with this discussion is similar to one we always have in openstack ...
15:29:09 <sandy__> "I need X"
15:29:10 <dperaza> btw I think there is work schedule on glance to support oslo notification
15:29:22 <sandy__> "well, we're working on a thing to handle X, somewhat"
15:29:23 <dperaza> trying to find the link
15:29:27 <jd__> Alexei_987: does that cover enough of your initial question?
15:29:31 <dragondm> dhellmann: we don't get to dictate what keys are  in the payload. It should be a dict. We don't have much of a spec for notifications, but that's in what we have.
15:29:33 <sandy__> "but it's not ready and will be a big change"
15:29:44 <sandy__> "so, wait until it's ready and use it"
15:29:47 <Alexei_987> well I think we are agreed that validation is needed :)
15:29:55 <dhellmann> dragondm: it sounds like that will be fixed in glance in I, then?
15:29:56 <sandy__> ... meanwhile, innovation suffers and nothing gets done
15:29:57 <Alexei_987> but not agreed about how to do it
15:30:03 <nprivalova> dperaza, please! We are digging the new Glance code but cannot make notification work
15:30:25 <Alexei_987> but again I'll make a blueprint for this stuff
15:30:31 <Alexei_987> so partially covered
15:30:38 <dperaza> hold on
15:30:50 <sandy__> we're left, sitting at the end of pipe, waiting for a library to come out ... vs, solving the problem, moving on and refactoring commonality later
15:31:14 <sandy__> we're so afraid of duplication that movement forward suffers
15:31:16 <thomasem> Alexei_987, It's in contracts. Each component must use X notification format, and that's all we can rely on. Payloads are relatively freeform, but the envelope (things like 'generated') we need to be able to rely on. If we can't trust these components, then we'll spend all day validating and slow the system to a halt at scale.
15:31:27 <dhellmann> sandy__: I'm not sure what you're talking about here. I asked Alexei_987 to use an existing library instead of making a new one, and to not enforce rules that are wrong by rejecting data at the wrong level.
15:31:32 <dperaza> wow
15:31:33 <dperaza> https://bugs.launchpad.net/glance/+bug/1020749
15:31:34 <uvirtbot> Launchpad bug 1020749 in glance "Use Openstack-Common notifier" [Wishlist,Won't fix]
15:31:37 <dperaza> won't fix
15:31:54 <dperaza> that sucks
15:32:03 <thomasem> youch
15:32:04 <nprivalova> oh no...
15:32:28 <dperaza> what about that last comment though
15:32:29 <eglynn> won't because it already uses olso.messaging, no?
15:32:30 <jd__> they moved to oslo.messaging AFAIK, no?
15:32:34 <sileht> yes
15:32:34 <gordc> https://review.openstack.org/#/c/57678/
15:32:36 <sileht> I have checked
15:32:37 * jd__ high fives eglynn
15:32:38 <thomasem> Alexei_987, But that's just my thoughts on it. :)
15:32:48 <eglynn> jd__: :)
15:32:48 <thomasem> sentiment, more or less
15:33:07 * Alexei_987 sigh
15:33:10 <dhellmann> dperaza: if you look, it is "wont fix" because the dev wants to use oslo.messaging instead of the incubator version of the code, which is the right approach now that it is released
15:33:19 <nprivalova> jd__, the did! I saw this code in master. today :) but...
15:33:20 <Alexei_987> openstack contribution can be such a pain sometimes
15:33:26 <jd__> I think we are diverging so I'll move to the next topic
15:33:33 <jd__> #topic First stable/hava release immenent (eglynn)
15:33:59 <eglynn> just a quick heads-up that the first realse off stable is about to fly
15:34:01 <dperaza> ok and 57678 is merged so can we assume glance does support oslo notifications now?
15:34:01 <eglynn> https://launchpad.net/ceilometer/+milestone/2013.2.1
15:34:26 <eglynn> ... and a plea to get more folks involved in stable-maint
15:34:26 <sandy__> dhellmann, and that's fine, so long as the library does what Alexei_987 needs ... otherwise it can turn into yak shaving to do something relatively simple.
15:34:44 <nprivalova> dperaza, I cannot prove it works
15:35:10 <dhellmann> eglynn: what's the status with that sphinx issue? did that have any impact on us?
15:35:12 <dperaza> nprivalova: sounds like a bug
15:35:13 <nprivalova> dperaza, old code works ok but when switch to master no notification comes
15:35:54 <eglynn> dhellmann: which sphinx issue do you mean?
15:36:07 <eglynn> ... in any case 2013.2.1 is well frozen at this stage
15:36:08 <sandy__> nprivalova, the new notification system uses entry points, so the configs have to change
15:36:10 <nprivalova> dperaza, I'm trying to talk to glance guys in irc but no answer yet
15:36:15 <dhellmann> wasn't there a problem with a new release of sphinx breaking builds on stable branches?
15:36:23 <eglynn> ... so I don't think it'll be getting in
15:36:23 <dragondm> was that the version issue ?
15:36:25 <dhellmann> folks, please, we've changed topics
15:36:32 <sileht> dhellmann, the new 1.2 version
15:36:37 <dhellmann> right ^^
15:36:52 <dhellmann> eglynn: if you don't know about it, I assume it didn't break our stable builds :-)
15:36:54 <llu-laptop> https://launchpad.net/bugs/1259511, is this the sphinx issue dhellman mentioned?
15:36:55 <uvirtbot> Launchpad bug 1259511 in openstack-ci "sphinx docs gating jobs are broken (source_dir must be a directory)" [Critical,Fix released]
15:37:02 <dhellmann> yes
15:37:04 <dperaza> nprivalova: if you open a bug I can help you get to glance folks
15:37:05 <dhellmann> thanks, llu-laptop
15:37:07 <dragondm> Yah, I saw the requirements change for that.
15:39:18 <nprivalova> sandy__, if you have an example or a link to docs about new configs please share
15:39:28 <llu-laptop> This seems only affect test-requirement
15:39:46 <eglynn> llu-laptop: yeap, just about to say that
15:39:58 <jd__> eglynn: what's required to get involved in stable-maint?
15:40:01 <eglynn> will look into it further after this meeting
15:40:30 <eglynn> jd__: preparedness to spread the backporting religion ;)
15:40:50 <eglynn> jd__: nah, it's mainly just reviewing patches proposed to stable/havana
15:41:01 <jd__> eglynn: ok :)
15:41:16 <eglynn> jd__: and trawling the bugs fixed on master for backporting candidates
15:41:26 <sandy__> nprivalova, sec
15:41:38 <jd__> make sense
15:41:39 <eglynn> and that second task is the one I'd like push onto the original bug fixer
15:41:58 <jd__> eglynn: we should imaginate an automatic system to help with that I think
15:41:59 <eglynn> if poss, to at least tag good backporting candidates
15:42:36 <eglynn> jd__: yeah, that would be cool ... once the fix lands on master, some prompt or cue to consider for backporting
15:43:04 <jd__> eglynn: yes, not sure how it could be done, but let's keep that in mind, it'd even serve all projects
15:43:05 <sandy__> nprivalova, same flags, but names are now entry points: https://github.com/openstack/oslo.messaging/blob/master/setup.cfg#L45-L50
15:43:20 <eglynn> yep, it's a neat idea
15:43:32 <rajdeep> resolution to sphinx issue https://github.com/openstack/nova/blob/master/tox.ini#L9
15:43:38 <jd__> anything else on that matter eglynn or should I move on?
15:43:51 <eglynn> jd__: fire away!
15:43:57 <jd__> #topic Release python-ceilometerclient?
15:44:08 <eglynn> no need that I'm aware of
15:44:22 <jd__> cool then
15:44:30 <jd__> #topic Open discussion
15:44:47 <dperaza> I have something realted to validation
15:44:48 <dragondm> ^ Not til' the event api lands, anyway.
15:44:54 <dperaza> trying to handle this bug
15:45:04 <rajdeep> i had a question on integrating vCenter integration into ceilometer
15:45:05 <llu-laptop> eglynn: one question for you, about https://bugs.launchpad.net/ceilometer/+bug/1260176, which method you think I should take?
15:45:06 <uvirtbot> Launchpad bug 1260176 in ceilometer "different behavior of alarm creation on different DB backend" [Undecided,New]
15:45:10 <dperaza> https://bugs.launchpad.net/ceilometer/+bug/1245367
15:45:13 <uvirtbot> Launchpad bug 1245367 in ceilometer "Doing non-existing metadata property filter can not get expected error prompts" [Medium,Triaged]
15:45:33 <eglynn> llu-laptop: looking
15:45:33 <rajdeep> we are trying to map the measurements between ceilometer and vsphere
15:46:48 <dperaza> my take on 1245367 is that there is no way to differentiate from a query with bad metadata fild and a query with valid field that simply does not match
15:46:58 <rajdeep> Question : what is the guideline to add hypervisor specific metrics
15:47:06 <ityaptin> jd__, I've question about change  https://review.openstack.org/#/c/59214/.
15:47:16 <dperaza> I gather that is as design in ceilometer
15:47:27 <eglynn> llu-laptop: so are the foreign constraints actually serving any real purpose if we ensure that they're always satisified by creating the user and project IDs if absent?
15:47:28 <Alexei_987> dperaza: I'm afraid that we cannot do much here. since there is no way to determine is meta filter valid
15:47:29 <dperaza> in both cases they return 200
15:47:40 <gordc> llu-laptop:  i added a comment to that bug
15:47:42 <nprivalova> and I'm looking for one core reviewer more for https://review.openstack.org/#/c/52670/
15:47:56 <jd__> ityaptin: shoot
15:48:06 <eglynn> llu-laptop: I don't think a constraint that we ensure is always met really adds any value TBH, or?
15:48:09 <jd__> rajdeep: do you have an example?
15:48:40 <dperaza> Alexei_987 so you agree with me on do nothing here?
15:49:02 <rajdeep> yes cpu.usagemhz.average metric in vsphere
15:49:10 <jd__> dhellmann: I think you should take a look at https://review.openstack.org/#/c/52670/ for nprivalova
15:49:13 <rajdeep> there is not corresponding metric in nova
15:49:31 <llu-laptop> eglynn: agree
15:49:31 <jd__> rajdeep: just create a new meter for that then
15:49:38 * dhellmann dutifully clicks on the link from jd__
15:49:43 <rajdeep> another example power.energy.summation
15:49:53 <jd__> dhellmann: I re-hire you then! :)
15:49:55 <thomasem> dperaza, Alexei_987: I suppose that's an unfortunate case for arbitrary metadata items, huh? :(
15:49:56 <eglynn> llu-laptop: coo, so method 1 looks more sensible to me
15:49:56 <gordc> dperaza: i think it's working as designed. that 500 error is wrong though i think.
15:50:05 <rajdeep> ok and for metrics which are non existing in vsphere i am assuming we leave them blank
15:50:16 <dhellmann> jd__: woo!
15:50:18 <Alexei_987> thomasem: yes
15:50:22 <eglynn> llu-laptop: I'll add a comment on LP to that effect
15:50:23 <dperaza> I think 500 is wrong too gordc
15:50:35 <jd__> 500 is always wrong, isn't it?
15:50:40 <dperaza> I added a note that what i see for no-metadata fields is a 400
15:50:45 <gordc> jd__: comments on https://bugs.launchpad.net/ceilometer/+bug/1245367 (since it's marked as a backport potential)
15:50:48 <uvirtbot> Launchpad bug 1245367 in ceilometer "Doing non-existing metadata property filter can not get expected error prompts" [Medium,Triaged]
15:50:53 <dperaza> jd_: +1
15:50:54 <llu-laptop> eglynn: should it be method 1+2? because we need to create for un-exsiting project_id/user_id
15:51:40 <eglynn> llu-laptop: won't we create when we see an actual sample with the previously unknown project/user ID?
15:51:50 <rajdeep> -->For metrics which are non existing in vsphere i am assuming we leave them blank, please confirm
15:51:50 <eglynn> llu-laptop: (as opposed to when we create an alarm)
15:52:03 <Alexei_987> jd__: a proposal for the next meeting is to think about the following - we should change data structure for MongoDB in such a way that we don't have do aggregation for realtime queries
15:52:05 <thomasem> dperaza, Do we spit back out the query for human intervention?
15:52:19 <thomasem> Since that's something we can't really assess, a human can.
15:52:36 <jd__> llu-laptop: eglynn: I hope I'm not saying something wrong, but in the case where we would envision to split the sample and alarm database storage, dropping the foreign key makes even more sense
15:52:47 <thomasem> "Nothing found for <query>, are you sure this is correct?"
15:53:24 <jd__> Alexei_987: I don't think it's a subject for a _meeting_, sounds like you want to write a blueprint + wiki page spec and discuss that on the mailing list
15:53:28 <gordc> quick q, anyone know status of alembic migrations? we ever going to move to them? we might need to decide how to handle our existing alembic migrations.
15:53:31 <ityaptin> It's required run unit tests with Postgresql environment like https://review.openstack.org/#/c/59489/. I want add unit tests with postgresql with dependence on your change request and suggest my fix without these tests.
15:53:34 <Alexei_987> jd__: Ok
15:53:57 <ityaptin> _jd It's required run unit tests with Postgresql environment like https://review.openstack.org/#/c/59489/. I want add unit tests with postgresql with dependence on your change request and suggest my fix without these tests.
15:54:04 <jd__> gordc: I think we lost everybody on that, you can be my hero if you take this over
15:54:05 <eglynn> jd__: yeap, it would be more flexible that way (... tho' I'm not sure such a split is on the cards, at least in the short to medium term, or?)
15:54:10 <ityaptin> jd__ It's required run unit tests with Postgresql environment like https://review.openstack.org/#/c/59489/. I want add unit tests with postgresql with dependence on your change request and suggest my fix without these tests.
15:54:19 <dperaza> thomasem: do you mean in the response
15:54:28 <gordc> jd__: ok i'll post on mailing list to get a good direction.
15:54:30 <llu-laptop> eglynn: for sample, we create new user/project id if they're not not existed. Should we do this for alarm creation too? and drop the foreign key constaits?
15:54:38 <dperaza> even if query is bad it looks like not matching query
15:54:47 <herndon> gordc I'm happy to help out with that
15:54:58 <hvp> have a question on https://blueprints.launchpad.net/ceilometer/+spec/central-agent-improvement
15:55:13 <thomasem> dperaza, Sure. So, it's just an idea, but returning the query back to the client will allow it to be assessed.
15:55:15 <jd__> eglynn: just thinking out loud, but as anyway user are just id, there even shouldn't be a user table anyway so I think dropping the key and the table is the way to go
15:55:19 <sileht> llu-laptop, mongo doesn't have this constraint
15:55:20 <gordc> herndon: cool. let's discuss it.
15:55:24 <jd__> eglynn: I meant the foreign key reference
15:55:55 <dperaza> but the client just build the query, right
15:55:57 <eglynn> a-ha I see
15:56:14 <dperaza> not sure we gain much sending back same query
15:56:26 <dperaza> all we can do is what we do now
15:56:31 <jd__> hvp: ?
15:56:42 <hvp> Is this blueprint also considering a multi-datacenter active/active central agent polling multiple sources (openstack endpoints) ?
15:56:50 <thomasem> dperaza, Agreed. Sorry, I was just thinking of another alternative to validation. :)
15:57:40 <dperaza> right, there is not match it is up to client to then check is a good query before trying again then
15:57:59 <nealph> hvp: IIRC, the summit conversation did include that aspect.
15:58:14 <hvp> we have multiple openstack endpoints in different data centers and having a unified ceilometer infrastructure would be ideal
15:58:25 <hvp> nealph, sorry i joined late
15:58:35 <eglynn> llu-laptop: ok, is the reason we ensure the user & project ID exist on alarm creation is just to fullfil the foreign key constraint, or because we feel "it's the right thing to do"?
15:58:36 <thomasem> Okay. I think it'd be awesome for us to call out that limitation in our documentation. That we can't validate arbitrary fields like that.
15:58:52 <thomasem> Then clients can approach with a more accurate expectation.
15:58:58 <dperaza> thomasem: I will handle that
15:59:02 <eglynn> llu-laptop: and if we're keeping the logic to ensure the user & project ID, exist ... why drop the FK constraint?
15:59:10 <dperaza> good sugestion
15:59:16 <jd__> hvp: nop, that'd be another blueprint I think
15:59:19 <thomasem> dperaza, Thank you! :)
15:59:40 <jd__> eglynn: llu-laptop: drop it! drop it all!
15:59:42 <nealph> jd__: certainly the task distribution part of the BP is related?
15:59:45 <sileht> eglynn, the mongodb backend doesn't ensure that
15:59:50 <jd__> nealph: that'd be all I think
15:59:56 <hvp> jd_, so it will still have some dependencies on the central agent improvements
16:00:06 <jd__> hvp: like nealph said :)
16:00:08 <eglynn> sileht: yep, so say we change mongo to do what sqlalchemy does
16:00:27 <jd__> that'd be terrible! :)
16:00:28 <eglynn> sileht: ... in that case, why also drop the FK constraint?
16:00:34 <llu-laptop> eglynn: Aah, you're right. Seems there is no need to make sure the user/project_id existing in alarms perspective?
16:00:51 <jd__> it's time to wrap up guys
16:01:11 <nprivalova> jd__, please take a look on ityaptin's question later
16:01:16 <sileht> eglynn, the question is what is the purpose of project/user storage API, samples only or everything in ceilometer ?
16:01:22 <eglynn> llu-laptop: that's what I'm thinking, so either drop the auto-create AND the FK constraint, or else get mongodb to auto-create also
16:01:29 * dhellmann goes back to his sick-bed
16:01:46 <nealph> hvp: in our experience, active/active is possible with some massaging of configs. curious if you have specific issues....ping me offline.
16:02:14 <hvp> thx nealph.. will do
16:02:32 <jd__> nprivalova: thanks I missed it
16:02:51 <jd__> ityaptin: I'll answer on #openstack-ceilometer
16:02:59 <jd__> #endmeeting