| *** openstack has joined #openstack-ceilometer | 13:37 | |
| *** gordc has joined #openstack-ceilometer | 13:39 | |
| nprivalova | gordc, hi! | 13:39 |
|---|---|---|
| gordc | nprivalova: hi, whatsup? | 13:40 |
| nprivalova | gordc, I'd like to discuss https://review.openstack.org/#/c/52670/ . I've uploaded a new variant, without changes in flatten_metadata. Could you please take a look? I have -1 from Jenkins but it's because I faced with broken server | 13:41 |
| gordc | nprivalova: sure. i'll take a look at it. | 13:42 |
| nprivalova | gordc, I've changed tests. Not sure it's acceptable but I don't see any other variants :( | 13:44 |
| *** jdob has joined #openstack-ceilometer | 13:49 | |
| *** eglynn has joined #openstack-ceilometer | 13:50 | |
| *** prad_ has joined #openstack-ceilometer | 13:51 | |
| *** jmckind has joined #openstack-ceilometer | 13:52 | |
| gordc | nprivalova: did you drop lists from test because the results from hbase will be different from results in other backends? i don't think we want that. | 13:54 |
| nprivalova | gordc, not only because hbase. I don't understand why lists should be ignored actually. Looks like the code was written first and then test appeared | 13:56 |
| nprivalova | gordc, but yes, only hbase returns lists | 13:56 |
| gordc | nprivalova: yeah, i don't know why lists are ignored either. i would think lists should be there. | 13:58 |
| gordc | nprivalova: i forgot how _flatten_metadata method works, is it formatting the results the returned by db or is it formatting something else? | 13:59 |
| nprivalova | gordc, As I see from the code _flatten_metadata happens in 2 scenarios: 1. return data from db 2. during POST request from user | 14:01 |
| nprivalova | gordc, so e.g. Mongo contains entries with list but when user ask "show me data" lists are not displayed | 14:02 |
| gordc | nprivalova: and if hbase contains entries with lists, does it not filter out the lists in the same way as mongo? | 14:04 |
| nprivalova | gordc, the thing is that hbase cannot understand that list is actually the list. Everything is just strings. So when I get entry from HBase the code "if type(v) not in set([list, set]))" doesn't work because type(v) is actually a string. These lines are from flatten_metadata method, I don't want to add ugly ifs there | 14:07 |
| nprivalova | "these lines" is if type(v) not in set([list, set])) | 14:09 |
| gordc | nprivalova: got it. in hbase, the data has already been flattened when it is stored so flatten_metadata doesn't filter out list when it is run. | 14:11 |
| nprivalova | gordc, yep | 14:11 |
| nprivalova | gordc, I get one more idea. I tried to use a different method in flatten_metadata. but maybe it will be ok to change only this line "if type(v) not in set([list, set]))" => "if type(v) not in set([set]))" | 14:13 |
| gordc | nprivalova: i think it's best to find out why we're dropping lists to begin with... if there's a reason for that, then we can ask why we're even storing the data if we're dropping it anyways. | 14:13 |
| gordc | dhellmann: i don't recall why we drop lists in api, any ideas? | 14:16 |
| nprivalova | gordc, previously it was "if type(v) not in set([list, dict, set]))" | 14:19 |
| nprivalova | gordc, dict was erased by 149e4d6 | 14:21 |
| gordc | nprivalova: that's weird. what was it flattening if it was ignoring dict? | 14:21 |
| nprivalova | gordc, take a look on left part https://review.openstack.org/#/c/38597/10/ceilometer/api/controllers/v2.py | 14:23 |
| gordc | nprivalova: whoa, that's pretty recent.lol | 14:26 |
| jd__ | sileht: AFAIU, there's still not receive part for oslo.messaging until your patches are merged right? | 14:27 |
| gordc | nprivalova: i'm still not sure why we're ignoring lists. i think we should do the same thing like we did for dict and stop ignoring lists as well. | 14:27 |
| nprivalova | gordc, :) . Let me try to run tests without ignoring lists | 14:29 |
| sileht | jd__, yes | 14:30 |
| jd__ | sigh | 14:31 |
| jd__ | oslo.messaging depressed me | 14:31 |
| dhellmann | gordc: "drop lists"? | 14:38 |
| nprivalova | jd__, hold on, everything will be fine! :) | 14:39 |
| jd__ | nprivalova: :-) | 14:40 |
| nprivalova | dhellmann, yes, during flatten_matadata | 14:40 |
| dhellmann | nprivalova: I don't know what that phrase means, I don't have enough context | 14:40 |
| nprivalova | dhellmann, sorry, we're discussing https://review.openstack.org/#/c/52670/7/ceilometer/api/controllers/v2.py again | 14:41 |
| gordc | dhellmann: https://github.com/openstack/ceilometer/blob/master/ceilometer/api/controllers/v2.py#L515 specifically. | 14:42 |
| dhellmann | gordc: originally jd__ (I think) didn't want to worry about arbitrarily nested metadata, so we only took the top level values | 14:43 |
| dhellmann | gordc: I guess at some point we decided we needed to recurse :-) | 14:43 |
| jd__ | that rings a bell indeed | 14:43 |
| dhellmann | nprivalova: can you show that your version of _flatten_metadata() and the old version produce the same results for the same inputs? because if you can, then we can just remove one of the functions it is calling (dict_to_keyval or recursive_keypairs) | 14:44 |
| dhellmann | nprivalova: nevermind, I see your comment on the changeset now | 14:44 |
| gordc | nprivalova: i assume you have requirement to recurse further to include lists. | 14:44 |
| gordc | dhellmann, jd__: i think we should probably include lists as well... if we're storing it in db, might as well show it. | 14:45 |
| dhellmann | gordc: we have to be careful here, because this function is used to process both inputs and outputs | 14:46 |
| nprivalova | dhellmann, yep, I think I should not change recursive_keypairs => dict_to_keyval but I can remove "if type(v) not in set([list, set]))" | 14:46 |
| jd__ | that or we rewrite notifications to have nothing nested | 14:46 |
| dhellmann | gordc: we can't change the output format (so the key names need to stay the same) but we can change the content | 14:46 |
| jd__ | since I'm writing on validation right now it's a good idea to let me know that | 14:47 |
| dhellmann | jd__: I'd like that, but I'm not sure it's possible (I haven't examined every notification payload to see) | 14:47 |
| dhellmann | jd__: it would also break backwards compat with other notification subscribers | 14:47 |
| jd__ | everything's possible! :) | 14:47 |
| jd__ | true that | 14:47 |
| jd__ | honestly at some point I wonder if we wouldn't be better with a new notification API | 14:48 |
| gordc | dhellmann: right. when i originally +2, both the methods were near identical minus how lists are represented (which get dropped anyways). i think there was a change to recursive_keypairs() recently so i'm not sure if they're similar still. | 14:48 |
| jd__ | with all the changes we would like to make and the backward compat it's going to be very frustrating | 14:48 |
| dhellmann | gordc: we need to get rid of one of those functions, we shouldn't have 2 slightly different ways to serialize data like that | 14:48 |
| jd__ | (just saying that so I can point at that IRC log in a few months) | 14:48 |
| gordc | dhellmann: agreed. i'll open bug. | 14:49 |
| dhellmann | jd__: we'll have to version the notifications, and keep the old notifier around for people to use until they can update their receivers | 14:49 |
| jd__ | dhellmann: even the routing of messages is crappy IMHO, so I think they could live side by side | 14:50 |
| dhellmann | jd__: yep | 14:50 |
| jd__ | but well, one battle at a time | 14:50 |
| *** coolsvap has joined #openstack-ceilometer | 14:54 | |
| nprivalova | dhellmann, gordc, it will be hard to remove one function. dict_to_keyval is used even in migrations and recursive_keypairs are commonly used too. And I suspect that recursive_keypairs contais a bug: https://github.com/openstack/ceilometer/blob/master/ceilometer/utils.py#L33 I guess separator should be added to reursive call | 14:54 |
| dhellmann | nprivalova: if dict_to_keyval is used somewhere else and we can't remove it, let's move it closer to where it's needed and make it a private function | 14:55 |
| dhellmann | nprivalova: and you're right about that bug | 14:55 |
| dhellmann | nprivalova: while you're in there fixing things, that sorting stuff on line 45 doesn't seem to do what the comment implies -- it should use an OrderedDict | 14:57 |
| nprivalova | dhellmann, ok, I will take care of it | 14:58 |
| dhellmann | nprivalova: thanks! | 15:01 |
| nprivalova | dhellmann, gordc, jd__ so I have 3 items: 1. get rid of dict_to_keyval (or make it as private as possible) 2. fix the bug about separator (will file it now) 3. Fix flatten_metadata by including lists (in HBase commit) And after all of above backward compatibility should be kept... | 15:05 |
| gordc | nprivalova: seems like a good summary to me. #1 can be tracked here: bug #1268618 | 15:06 |
| *** sandywalsh has joined #openstack-ceilometer | 15:09 | |
| nprivalova | gordc, thank you! may I assign to me? | 15:10 |
| gordc | nprivalova: go for it. :) | 15:10 |
| nprivalova | ok :) | 15:10 |
| nprivalova | eglynn, hi! Please be aware of our plan above. Because I know that you worked on metadata representation | 15:13 |
| eglynn | nprivalova: thanks for the heads-up, can you annotate the LP bug with a quick TL;DR summary? | 15:17 |
| *** tongli has joined #openstack-ceilometer | 15:19 | |
| nprivalova | eglynn, I will add summary to bug #1268618, ok? because each of items above have their own bugs, but #1268618 is the most embracing one | 15:20 |
| eglynn | nprivalova: sounds like a plan, thank you! | 15:20 |
| *** coolsvap has quit IRC | 15:21 | |
| *** absubram_ has joined #openstack-ceilometer | 15:31 | |
| openstackgerrit | Julien Danjou proposed a change to openstack/ceilometer: Update oslo https://review.openstack.org/65661 | 15:34 |
| *** coolsvap has joined #openstack-ceilometer | 15:42 | |
| openstackgerrit | Nadya Privalova proposed a change to openstack/ceilometer: Insertion in HBase should be fixed https://review.openstack.org/52670 | 15:51 |
| *** ityaptin has joined #openstack-ceilometer | 15:54 | |
| *** SergeyLukjanov has quit IRC | 15:57 | |
| ityaptin | jd__: Hi! Please have a look blueprint https://blueprints.launchpad.net/ceilometer/+spec/run-all-pollsters-on-demand. | 16:00 |
| *** ruhe is now known as ruhe_ | 16:01 | |
| *** Dakoolest has joined #openstack-ceilometer | 16:01 | |
| *** ruhe_ is now known as ruhe | 16:01 | |
| *** ityaptin has quit IRC | 16:04 | |
| Dakoolest | i have a few questions regarding data fetching via ceilometer rest api | 16:36 |
| Dakoolest | can anyone answer them? | 16:36 |
| Dakoolest | well, in any case i'll just write a few in here :) | 16:37 |
| Dakoolest | if i fetch a list of meters i would like to identify the resource they meter | 16:38 |
| Dakoolest | does that mean i need to make a separate request on /resource/ with the resource_id i got in the meter or is there a way to avoid another request? | 16:38 |
| *** sandywalsh has quit IRC | 16:39 | |
| *** sandywalsh has joined #openstack-ceilometer | 16:39 | |
| *** ruhe is now known as ruhe_ | 16:42 | |
| eglynn | Dakoolest: well, meters are per-resource to begin with | 16:46 |
| eglynn | Dakoolest: (i.e. cpu_util for instance1 will appear as a separate entry in the meter-list to cpu_util for instance2) | 16:47 |
| *** ruhe_ has quit IRC | 16:47 | |
| *** gordc1 has joined #openstack-ceilometer | 16:50 | |
| *** SergeyLukjanov has joined #openstack-ceilometer | 16:50 | |
| Dakoolest | eglynn: true, but my real issue is once I retrieve a list of meters all i know about a meter's resource is it's ID. Now, if I have 100 meters does that mean i need to make 100 requests to /resource/ to identify all resources (or list all resources, then match them on my end, which could take even longer)? | 16:50 |
| *** gordc has quit IRC | 16:51 | |
| Dakoolest | eglynn: (I start by listing meters because my goal is to visualise metering data for each meter) | 16:51 |
| eglynn | Dakoolest: sadly yes, as things currently stand ... | 16:52 |
| eglynn | Dakoolest: ... unless there's come other unifying aspect for those resources (e.g. all being associated with the same tenant for example) | 16:52 |
| eglynn | Dakoolest: ... but there are move afoot to support more complex queries in icehouse | 16:52 |
| eglynn | Dakoolest: ... e.g. combining query constraints with logical OR as opposed to AND | 16:52 |
| eglynn | s/come other/some other/ | 16:53 |
| eglynn | ... I want my money back, that "miracle" cure for dyslexia never worked ;) | 16:54 |
| ildikov | Dakoolest: the short details of the future complex query support is here: https://wiki.openstack.org/wiki/Ceilometer/ComplexFilterExpressionsInAPIQueries | 16:54 |
| Dakoolest | eglynn: about the unifying aspect... if all meters were from the same tenant what would be the best way to use that? list resources, with a project_id query so I end up with a smaller batch of resources to match on my end? | 16:57 |
| gibi | Dakoolest: what kind of data do you need for a meter that can be accessed through the resource_id? | 16:57 |
| eglynn | Dakoolest: ... exactomundo :) | 16:57 |
| Dakoolest | alright, thanks for the help peeps :) | 16:58 |
| eglynn | np! | 16:58 |
| Dakoolest | gibi: I am trying to visualise metering data for different meters. For example, I can get statistics for a meter and using groupby tey can be grouped by resource. In that case I need some resource data (at least a display name or something) so I can show the user data grouped by resource | 17:01 |
| gibi | Dakoolest: I see. So you need some resource metadata in the statistics query. | 17:03 |
| gibi | Dakoolest: we haven't started planning the API for the complex statistcs query but we will think about this use case during the plannig | 17:04 |
| Dakoolest | gibi: excellent :) | 17:04 |
| *** herndon_ has joined #openstack-ceilometer | 17:09 | |
| herndon_ | eglynn: could I get you to review this one more time? gordon had a couple of new suggestions that I cleaned up: https://review.openstack.org/#/c/54582/ | 17:10 |
| eglynn | herndon_: sure, no prob, on it now ... | 17:10 |
| herndon_ | thank you :) | 17:11 |
| *** tongli has quit IRC | 17:26 | |
| *** herndon_ has quit IRC | 17:27 | |
| *** herndon_ has joined #openstack-ceilometer | 17:29 | |
| *** nadya_ has joined #openstack-ceilometer | 17:37 | |
| *** nadya_ has quit IRC | 17:41 | |
| *** tongli has joined #openstack-ceilometer | 17:53 | |
| *** tong_ has joined #openstack-ceilometer | 17:54 | |
| *** tongli has quit IRC | 17:54 | |
| *** tong_ has quit IRC | 17:55 | |
| *** tong_ has joined #openstack-ceilometer | 17:55 | |
| openstackgerrit | Ildiko Vancsa proposed a change to openstack/ceilometer: Implements field validation for complex query functionality https://review.openstack.org/65302 | 18:00 |
| openstackgerrit | Ildiko Vancsa proposed a change to openstack/ceilometer: Implements complex query functionality for alarms https://review.openstack.org/62158 | 18:00 |
| openstackgerrit | Ildiko Vancsa proposed a change to openstack/ceilometer: Implements complex query functionality for alarm history https://review.openstack.org/62159 | 18:00 |
| openstackgerrit | Ildiko Vancsa proposed a change to openstack/ceilometer: Implements complex query functionality for samples https://review.openstack.org/62157 | 18:00 |
| *** Dakoolest has quit IRC | 18:15 | |
| *** boris-42 has quit IRC | 18:20 | |
| *** nadya_ has joined #openstack-ceilometer | 18:23 | |
| *** eglynn has quit IRC | 18:24 | |
| *** ruhe has joined #openstack-ceilometer | 18:30 | |
| *** s2r2 has quit IRC | 18:40 | |
| *** sayali_ has quit IRC | 18:40 | |
| *** s2r2 has joined #openstack-ceilometer | 18:40 | |
| *** sayali_ has joined #openstack-ceilometer | 18:40 | |
| *** herndon_ has quit IRC | 18:44 | |
| *** dhellmann is now known as dhellmann_ | 18:45 | |
| openstackgerrit | A change was merged to openstack/python-ceilometerclient: Support the Event API https://review.openstack.org/54582 | 18:51 |
| nadya_ | ERROR: InvocationError: '/home/jenkins/workspace/gate-ceilometer-python26/tools/config/check_uptodate.sh" on gating :( | 18:51 |
| *** jmckind has quit IRC | 18:53 | |
| *** tong_ has quit IRC | 18:54 | |
| *** herndon_ has joined #openstack-ceilometer | 18:56 | |
| *** tongli has joined #openstack-ceilometer | 18:57 | |
| *** elextro_ has joined #openstack-ceilometer | 18:57 | |
| ildikov | nadya_: we've just faced with the same problem :( | 18:58 |
| ildikov | nadya_: four times actually :S | 18:58 |
| openstackgerrit | Ildiko Vancsa proposed a change to openstack/ceilometer: Implements field validation for complex query functionality https://review.openstack.org/65302 | 18:58 |
| elextro_ | Does anybody know a quick way to remove deleted resources from the ceilometer DB? Or do I pretty much have to script that? | 19:01 |
| elextro_ | What if I truncate the entire meter table? :) | 19:02 |
| nadya_ | elextro_, what db do you use? | 19:04 |
| elextro_ | mysql | 19:04 |
| nadya_ | elextro_: if it's acceptable for you you may just remove tables | 19:05 |
| elextro_ | yeah that's no problem for me. right now I have over 1,000,000 rows in my meter table and it's making ceilo run really slow :/ I'm going to try removing the tables to see if that does anything. | 19:07 |
| elextro_ | Hopefully it doesn't break openstack XD | 19:07 |
| nadya_ | elextro_: there is a 'ceilometer-db-sync' command (hope spelling is right) | 19:07 |
| thomasem | elextro_, If it's just a test environment and you want to clear the DB, you can just DROP DATABASE ceilometer; CREATE DATABASE ceilometer; then on your ceilometer host ceilometer-dbsync. | 19:07 |
| *** yassine has quit IRC | 19:08 | |
| thomasem | I did that for my DB load testing for events a good 20 or so times. :P | 19:08 |
| elextro_ | ^^^ That's exactly what I was looking for :) Thanks @nadya @thomasem | 19:08 |
| *** yassine has joined #openstack-ceilometer | 19:08 | |
| *** yassine has quit IRC | 19:09 | |
| *** yassine has joined #openstack-ceilometer | 19:09 | |
| nadya_ | ildikov, hope somebody will repair it by tomorrow... | 19:09 |
| *** yassine has quit IRC | 19:10 | |
| thomasem | elextro_, You bet! | 19:10 |
| elextro_ | I assume I have to wait for the data to come through the pipeline before the meters come back up, correct? | 19:11 |
| elextro_ | Currently have the interval set to 60sec | 19:11 |
| elextro_ | yup.. | 19:12 |
| elextro_ | they just came up | 19:12 |
| *** yassine has joined #openstack-ceilometer | 19:12 | |
| *** yassine has quit IRC | 19:12 | |
| thomasem | yeppers | 19:12 |
| *** yassine has joined #openstack-ceilometer | 19:12 | |
| *** yassine has quit IRC | 19:13 | |
| ildikov | nadya_: I have no better idea for now, than hoping this | 19:13 |
| *** yassine has joined #openstack-ceilometer | 19:14 | |
| *** yassine has quit IRC | 19:14 | |
| gibi | nadya_: There is already a bug reported for the problem https://bugs.launchpad.net/nova/+bug/1268614 | 19:17 |
| gibi | but it seems they only fixing it for nova | 19:17 |
| gibi | so we will propose a fix for ceilometer soon | 19:23 |
| *** ruhe is now known as ruhe_ | 19:23 | |
| openstackgerrit | Ildiko Vancsa proposed a change to openstack/ceilometer: Update ceilometer.conf.sample https://review.openstack.org/66402 | 19:27 |
| ildikov | nadya_: I've just uploaded the change, which fixes that error | 19:29 |
| *** boris-42 has joined #openstack-ceilometer | 19:30 | |
| *** openstackgerrit has quit IRC | 19:31 | |
| *** openstackgerrit has joined #openstack-ceilometer | 19:31 | |
| *** ruhe_ is now known as ruhe | 19:36 | |
| *** ruhe has quit IRC | 19:36 | |
| *** ildikov has quit IRC | 19:46 | |
| *** tong_ has joined #openstack-ceilometer | 19:47 | |
| *** tongli has quit IRC | 19:50 | |
| *** dhellmann_ is now known as dhellmann | 20:00 | |
| *** sayali_ has quit IRC | 20:03 | |
| *** boris-42 has quit IRC | 20:05 | |
| *** boris-42 has joined #openstack-ceilometer | 20:08 | |
| *** jmckind has joined #openstack-ceilometer | 20:09 | |
| *** ildikov_ has joined #openstack-ceilometer | 20:09 | |
| nadya_ | ildikov_, are you here? | 20:31 |
| *** nadya_ has quit IRC | 20:35 | |
| ildikov_ | nprivalova: ping | 20:48 |
| *** jaypipes has quit IRC | 20:58 | |
| *** nadya_ has joined #openstack-ceilometer | 21:05 | |
| *** nadya_ has quit IRC | 21:14 | |
| *** ok_delta has joined #openstack-ceilometer | 21:14 | |
| *** tong_ has quit IRC | 21:19 | |
| *** SergeyLukjanov has quit IRC | 21:37 | |
| *** herndon_ has quit IRC | 21:38 | |
| *** herndon_ has joined #openstack-ceilometer | 21:46 | |
| *** ok_delta has quit IRC | 21:49 | |
| *** jdob has quit IRC | 22:02 | |
| *** jmckind has quit IRC | 22:40 | |
| *** thomasem has quit IRC | 22:41 | |
| *** flwang has quit IRC | 23:03 | |
| *** jmckind has joined #openstack-ceilometer | 23:14 | |
| *** jmckind has quit IRC | 23:14 | |
| *** jaypipes has joined #openstack-ceilometer | 23:14 | |
| *** prad_ has left #openstack-ceilometer | 23:20 | |
| *** openstackgerrit has quit IRC | 23:24 | |
| *** openstackgerrit has joined #openstack-ceilometer | 23:24 | |
| *** flwang has joined #openstack-ceilometer | 23:39 | |
| *** herndon_ has quit IRC | 23:51 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!