Monday, 2014-01-13

*** openstack has joined #openstack-ceilometer13:37
*** gordc has joined #openstack-ceilometer13:39
nprivalovagordc, hi!13:39
gordcnprivalova: hi, whatsup?13:40
nprivalovagordc, I'd like to discuss . I've uploaded a new variant, without changes in flatten_metadata. Could you please take a look? I have -1 from Jenkins but it's because I faced with broken server13:41
gordcnprivalova: sure. i'll take a look at it.13:42
nprivalovagordc, I've changed tests. Not sure it's acceptable but I don't see any other variants :(13:44
*** jdob has joined #openstack-ceilometer13:49
*** eglynn has joined #openstack-ceilometer13:50
*** prad_ has joined #openstack-ceilometer13:51
*** jmckind has joined #openstack-ceilometer13:52
gordcnprivalova: did you drop lists from test because the results from hbase will be different from results in other backends? i don't think we want that.13:54
nprivalovagordc, not only because hbase. I don't understand why lists should be ignored actually. Looks like the code was written first and then test appeared13:56
nprivalovagordc, but yes, only hbase returns lists13:56
gordcnprivalova: yeah, i don't know why lists are ignored either. i would think lists should be there.13:58
gordcnprivalova: i forgot how _flatten_metadata method works, is it formatting the results the returned by db or is it formatting something else?13:59
nprivalovagordc, As I see from the code _flatten_metadata happens in 2 scenarios: 1. return data from db 2. during POST request from user14:01
nprivalovagordc, so e.g. Mongo contains entries with list but when user ask "show me data" lists are not displayed14:02
gordcnprivalova: and if hbase contains entries with lists, does it not filter out the lists in the same way as mongo?14:04
nprivalovagordc, the thing is that hbase cannot understand that list is actually the list. Everything is just strings. So when I get entry from HBase the code "if type(v) not in set([list, set]))" doesn't work because type(v) is actually a string. These lines are from flatten_metadata method, I don't want to add ugly ifs there14:07
nprivalova"these lines" is if type(v) not in set([list, set]))14:09
gordcnprivalova: got it. in hbase, the data has already been flattened when it is stored so flatten_metadata doesn't filter out list when it is run.14:11
nprivalovagordc, yep14:11
nprivalovagordc, I get one more idea. I tried to use a different method in flatten_metadata. but maybe it will be ok to change  only this line "if type(v) not in set([list, set]))" => "if type(v) not in set([set]))"14:13
gordcnprivalova: i think it's best to find out why we're dropping lists to begin with... if there's a reason for that, then we can ask why we're even storing the data if we're dropping it anyways.14:13
gordcdhellmann: i don't recall why we drop lists in api, any ideas?14:16
nprivalovagordc, previously it was "if type(v) not in set([list, dict, set]))"14:19
nprivalovagordc, dict was erased by 149e4d614:21
gordcnprivalova: that's weird. what was it flattening if it was ignoring dict?14:21
nprivalovagordc, take a look on left part
gordcnprivalova: whoa, that's pretty recent.lol14:26
jd__sileht: AFAIU, there's still not receive part for oslo.messaging until your patches are merged right?14:27
gordcnprivalova: i'm still not sure why we're ignoring lists. i think we should do the same thing like we did for dict and stop ignoring lists as well.14:27
nprivalovagordc, :) . Let me try to run tests without ignoring lists14:29
silehtjd__, yes14:30
jd__oslo.messaging depressed me14:31
dhellmanngordc: "drop lists"?14:38
nprivalovajd__, hold on, everything will be fine! :)14:39
jd__nprivalova: :-)14:40
nprivalovadhellmann, yes, during flatten_matadata14:40
dhellmannnprivalova: I don't know what that phrase means, I don't have enough context14:40
nprivalovadhellmann, sorry,  we're discussing again14:41
gordcdhellmann: specifically.14:42
dhellmanngordc: originally jd__ (I think) didn't want to worry about arbitrarily nested metadata, so we only took the top level values14:43
dhellmanngordc: I guess at some point we decided we needed to recurse :-)14:43
jd__that rings a bell indeed14:43
dhellmannnprivalova: can you show that your version of _flatten_metadata() and the old version produce the same results for the same inputs? because if you can, then we can just remove one of the functions it is calling (dict_to_keyval or recursive_keypairs)14:44
dhellmannnprivalova: nevermind, I see your comment on the changeset now14:44
gordcnprivalova: i assume you have requirement to recurse further to include lists.14:44
gordcdhellmann, jd__: i think we should probably include lists as well... if we're storing it in db, might as well show it.14:45
dhellmanngordc: we have to be careful here, because this function is used to process both inputs and outputs14:46
nprivalovadhellmann, yep, I think I should not change recursive_keypairs => dict_to_keyval but I can remove "if type(v) not in set([list, set]))"14:46
jd__that or we rewrite notifications to have nothing nested14:46
dhellmanngordc: we can't change the output format (so the key names need to stay the same) but we can change the content14:46
jd__since I'm writing on validation right now it's a good idea to let me know that14:47
dhellmannjd__: I'd like that, but I'm not sure it's possible (I haven't examined every notification payload to see)14:47
dhellmannjd__: it would also break backwards compat with other notification subscribers14:47
jd__everything's possible! :)14:47
jd__true that14:47
jd__honestly at some point I wonder if we wouldn't be better with a new notification API14:48
gordcdhellmann: right. when i originally +2, both the methods were near identical minus how lists are represented (which get dropped anyways). i think there was a change to recursive_keypairs() recently so i'm not sure if they're similar still.14:48
jd__with all the changes we would like to make and the backward compat it's going to be very frustrating14:48
dhellmanngordc: we need to get rid of one of those functions, we shouldn't have 2 slightly different ways to serialize data like that14:48
jd__(just saying that so I can point at that IRC log in a few months)14:48
gordcdhellmann: agreed. i'll open bug.14:49
dhellmannjd__: we'll have to version the notifications, and keep the old notifier around for people to use until they can update their receivers14:49
jd__dhellmann: even the routing of messages is crappy IMHO, so I think they could live side by side14:50
dhellmannjd__: yep14:50
jd__but well, one battle at a time14:50
*** coolsvap has joined #openstack-ceilometer14:54
nprivalovadhellmann, gordc, it will be hard to remove one function. dict_to_keyval is used even in migrations and recursive_keypairs are commonly used too. And I suspect that recursive_keypairs contais a bug: I guess separator should be added to reursive call14:54
dhellmannnprivalova: if dict_to_keyval is used somewhere else and we can't remove it, let's move it closer to where it's needed and make it a private function14:55
dhellmannnprivalova: and you're right about that bug14:55
dhellmannnprivalova: while you're in there fixing things, that sorting stuff on line 45 doesn't seem to do what the comment implies -- it should use an OrderedDict14:57
nprivalovadhellmann, ok, I will take care of it14:58
dhellmannnprivalova: thanks!15:01
nprivalovadhellmann, gordc, jd__ so I have 3 items: 1. get rid of dict_to_keyval (or make it as private as possible) 2. fix the bug about separator (will file it now) 3. Fix flatten_metadata by including lists (in HBase commit)  And after all of above backward compatibility should be kept...15:05
gordcnprivalova: seems like a good summary to me. #1 can be tracked here: bug #126861815:06
*** sandywalsh has joined #openstack-ceilometer15:09
nprivalovagordc, thank you! may I assign to me?15:10
gordcnprivalova: go for it. :)15:10
nprivalovaok :)15:10
nprivalovaeglynn, hi! Please be aware of our plan above. Because I know that you worked on metadata representation15:13
eglynnnprivalova: thanks for the heads-up, can you annotate the LP bug with a quick TL;DR summary?15:17
*** tongli has joined #openstack-ceilometer15:19
nprivalovaeglynn, I will add summary to bug #1268618, ok? because each of items above have their own bugs, but #1268618 is the most embracing one15:20
eglynnnprivalova: sounds like a plan, thank you!15:20
*** coolsvap has quit IRC15:21
*** absubram_ has joined #openstack-ceilometer15:31
openstackgerritJulien Danjou proposed a change to openstack/ceilometer: Update oslo
*** coolsvap has joined #openstack-ceilometer15:42
openstackgerritNadya Privalova proposed a change to openstack/ceilometer: Insertion in HBase should be fixed
*** ityaptin has joined #openstack-ceilometer15:54
*** SergeyLukjanov has quit IRC15:57
ityaptinjd__: Hi! Please have a look blueprint
*** ruhe is now known as ruhe_16:01
*** Dakoolest has joined #openstack-ceilometer16:01
*** ruhe_ is now known as ruhe16:01
*** ityaptin has quit IRC16:04
Dakoolesti have a few questions regarding data fetching via ceilometer rest api16:36
Dakoolestcan anyone answer them?16:36
Dakoolestwell, in any case i'll just write a few in here :)16:37
Dakoolestif i fetch a list of meters i would like to identify the resource they meter16:38
Dakoolestdoes that mean i need to make a separate request on /resource/ with the resource_id i got in the meter or is there a way to avoid another request?16:38
*** sandywalsh has quit IRC16:39
*** sandywalsh has joined #openstack-ceilometer16:39
*** ruhe is now known as ruhe_16:42
eglynnDakoolest: well, meters are per-resource to begin with16:46
eglynnDakoolest: (i.e. cpu_util for instance1 will appear as a separate entry in the meter-list to cpu_util for instance2)16:47
*** ruhe_ has quit IRC16:47
*** gordc1 has joined #openstack-ceilometer16:50
*** SergeyLukjanov has joined #openstack-ceilometer16:50
Dakoolesteglynn: true, but my real issue is once I retrieve a list of meters all i know about a meter's resource is it's ID. Now, if I have 100 meters does that mean i need to make 100 requests to /resource/ to identify all resources (or list all resources, then match them on my end, which could take even longer)?16:50
*** gordc has quit IRC16:51
Dakoolesteglynn: (I start by listing meters because my goal is to visualise metering data for each meter)16:51
eglynnDakoolest: sadly yes, as things currently stand ...16:52
eglynnDakoolest: ... unless there's come other unifying aspect for those resources (e.g. all being associated with the same tenant for example)16:52
eglynnDakoolest: ... but there are move afoot to support more complex queries in icehouse16:52
eglynnDakoolest: ... e.g. combining query constraints with logical OR as opposed to AND16:52
eglynns/come other/some other/16:53
eglynn... I want my money back, that "miracle" cure for dyslexia never worked ;)16:54
ildikovDakoolest: the short details of the future complex query support is here:
Dakoolesteglynn: about the unifying aspect... if all meters were from the same tenant what would be the best way to use that? list resources, with a project_id query so I end up with a smaller batch of resources to match on my end?16:57
gibiDakoolest: what kind of data do you need for a meter that can be accessed through the resource_id?16:57
eglynnDakoolest: ... exactomundo :)16:57
Dakoolestalright, thanks for the help peeps :)16:58
Dakoolestgibi: I am trying to visualise metering data for different meters. For example, I can get statistics for a meter and using groupby tey can be grouped by resource. In that case I need some resource data (at least a display name or something) so I can show the user data grouped by resource17:01
gibiDakoolest: I see. So you need some resource metadata in the statistics query.17:03
gibiDakoolest: we haven't started planning the API for the complex statistcs query but we will think about this use case during the plannig17:04
Dakoolestgibi: excellent :)17:04
*** herndon_ has joined #openstack-ceilometer17:09
herndon_eglynn: could I get you to review this one more time? gordon had a couple of new suggestions that I cleaned up:
eglynnherndon_: sure, no prob, on it now ...17:10
herndon_thank you :)17:11
*** tongli has quit IRC17:26
*** herndon_ has quit IRC17:27
*** herndon_ has joined #openstack-ceilometer17:29
*** nadya_ has joined #openstack-ceilometer17:37
*** nadya_ has quit IRC17:41
*** tongli has joined #openstack-ceilometer17:53
*** tong_ has joined #openstack-ceilometer17:54
*** tongli has quit IRC17:54
*** tong_ has quit IRC17:55
*** tong_ has joined #openstack-ceilometer17:55
openstackgerritIldiko Vancsa proposed a change to openstack/ceilometer: Implements field validation for complex query functionality
openstackgerritIldiko Vancsa proposed a change to openstack/ceilometer: Implements complex query functionality for alarms
openstackgerritIldiko Vancsa proposed a change to openstack/ceilometer: Implements complex query functionality for alarm history
openstackgerritIldiko Vancsa proposed a change to openstack/ceilometer: Implements complex query functionality for samples
*** Dakoolest has quit IRC18:15
*** boris-42 has quit IRC18:20
*** nadya_ has joined #openstack-ceilometer18:23
*** eglynn has quit IRC18:24
*** ruhe has joined #openstack-ceilometer18:30
*** s2r2 has quit IRC18:40
*** sayali_ has quit IRC18:40
*** s2r2 has joined #openstack-ceilometer18:40
*** sayali_ has joined #openstack-ceilometer18:40
*** herndon_ has quit IRC18:44
*** dhellmann is now known as dhellmann_18:45
openstackgerritA change was merged to openstack/python-ceilometerclient: Support the Event API
nadya_ERROR: InvocationError: '/home/jenkins/workspace/gate-ceilometer-python26/tools/config/" on gating :(18:51
*** jmckind has quit IRC18:53
*** tong_ has quit IRC18:54
*** herndon_ has joined #openstack-ceilometer18:56
*** tongli has joined #openstack-ceilometer18:57
*** elextro_ has joined #openstack-ceilometer18:57
ildikovnadya_: we've just faced with the same problem :(18:58
ildikovnadya_: four times actually :S18:58
openstackgerritIldiko Vancsa proposed a change to openstack/ceilometer: Implements field validation for complex query functionality
elextro_Does anybody know a quick way to remove deleted resources from the ceilometer DB? Or do I pretty much have to script that?19:01
elextro_What if I truncate the entire meter table? :)19:02
nadya_elextro_, what db do you use?19:04
nadya_elextro_:  if it's acceptable for you you may just remove tables19:05
elextro_yeah that's no problem for me. right now I have over 1,000,000 rows in my meter table and it's making ceilo run really slow :/ I'm going to try removing the tables to see if that does anything.19:07
elextro_Hopefully it doesn't break openstack XD19:07
nadya_elextro_: there is a 'ceilometer-db-sync' command (hope spelling is right)19:07
thomasemelextro_, If it's just a test environment and you want to clear the DB, you can just DROP DATABASE ceilometer; CREATE DATABASE ceilometer; then on your ceilometer host ceilometer-dbsync.19:07
*** yassine has quit IRC19:08
thomasemI did that for my DB load testing for events a good 20 or so times. :P19:08
elextro_^^^ That's exactly what I was looking for :) Thanks @nadya @thomasem19:08
*** yassine has joined #openstack-ceilometer19:08
*** yassine has quit IRC19:09
*** yassine has joined #openstack-ceilometer19:09
nadya_ildikov, hope somebody will repair it by tomorrow...19:09
*** yassine has quit IRC19:10
thomasemelextro_, You bet!19:10
elextro_I assume I have to wait for the data to come through the pipeline before the meters come back up, correct?19:11
elextro_Currently have the interval set to 60sec19:11
elextro_they just came up19:12
*** yassine has joined #openstack-ceilometer19:12
*** yassine has quit IRC19:12
*** yassine has joined #openstack-ceilometer19:12
*** yassine has quit IRC19:13
ildikovnadya_: I have no better idea for now, than hoping this19:13
*** yassine has joined #openstack-ceilometer19:14
*** yassine has quit IRC19:14
gibinadya_: There is already a bug reported for the problem
gibibut it seems they only fixing it for nova19:17
gibiso we will propose a fix for ceilometer soon19:23
*** ruhe is now known as ruhe_19:23
openstackgerritIldiko Vancsa proposed a change to openstack/ceilometer: Update ceilometer.conf.sample
ildikovnadya_: I've just uploaded the change, which fixes that error19:29
*** boris-42 has joined #openstack-ceilometer19:30
*** openstackgerrit has quit IRC19:31
*** openstackgerrit has joined #openstack-ceilometer19:31
*** ruhe_ is now known as ruhe19:36
*** ruhe has quit IRC19:36
*** ildikov has quit IRC19:46
*** tong_ has joined #openstack-ceilometer19:47
*** tongli has quit IRC19:50
*** dhellmann_ is now known as dhellmann20:00
*** sayali_ has quit IRC20:03
*** boris-42 has quit IRC20:05
*** boris-42 has joined #openstack-ceilometer20:08
*** jmckind has joined #openstack-ceilometer20:09
*** ildikov_ has joined #openstack-ceilometer20:09
nadya_ildikov_, are you here?20:31
*** nadya_ has quit IRC20:35
ildikov_nprivalova: ping20:48
*** jaypipes has quit IRC20:58
*** nadya_ has joined #openstack-ceilometer21:05
*** nadya_ has quit IRC21:14
*** ok_delta has joined #openstack-ceilometer21:14
*** tong_ has quit IRC21:19
*** SergeyLukjanov has quit IRC21:37
*** herndon_ has quit IRC21:38
*** herndon_ has joined #openstack-ceilometer21:46
*** ok_delta has quit IRC21:49
*** jdob has quit IRC22:02
*** jmckind has quit IRC22:40
*** thomasem has quit IRC22:41
*** flwang has quit IRC23:03
*** jmckind has joined #openstack-ceilometer23:14
*** jmckind has quit IRC23:14
*** jaypipes has joined #openstack-ceilometer23:14
*** prad_ has left #openstack-ceilometer23:20
*** openstackgerrit has quit IRC23:24
*** openstackgerrit has joined #openstack-ceilometer23:24
*** flwang has joined #openstack-ceilometer23:39
*** herndon_ has quit IRC23:51

Generated by 2.14.0 by Marius Gedminas - find it at!