| *** openstack has joined #openstack-ceilometer | 00:02 | |
| *** eglynn-afk has quit IRC | 00:21 | |
| openstackgerrit | Doug Hellmann proposed a change to openstack/ceilometer: Use stevedore's make_test_instance https://review.openstack.org/69491 | 00:32 |
|---|---|---|
| openstackgerrit | A change was merged to openstack/ceilometer: Correct spelling of logger for dispatcher.file https://review.openstack.org/68854 | 00:36 |
| *** xianghui has joined #openstack-ceilometer | 00:53 | |
| *** xmltok has quit IRC | 00:56 | |
| openstackgerrit | ZhiQiang Fan proposed a change to openstack/python-ceilometerclient: Remove unused import for print_function https://review.openstack.org/69518 | 01:03 |
| *** _ruhe is now known as ruhe | 01:16 | |
| openstackgerrit | Yuuichi Fujioka proposed a change to openstack/ceilometer: fix column name to Unit from Volume https://review.openstack.org/69524 | 01:17 |
| openstackgerrit | Yuuichi Fujioka proposed a change to openstack/ceilometer: Implements monitoring-network https://review.openstack.org/60473 | 01:20 |
| *** gordc has quit IRC | 01:31 | |
| *** yfujioka has joined #openstack-ceilometer | 02:02 | |
| *** ruhe is now known as _ruhe | 02:03 | |
| *** urulama has quit IRC | 02:12 | |
| *** rwsu has quit IRC | 03:18 | |
| *** xianghui has quit IRC | 03:27 | |
| openstackgerrit | Jia Dong proposed a change to openstack/python-ceilometerclient: Modify ceilometer client cmd line help info https://review.openstack.org/63984 | 03:51 |
| *** sayali_ has joined #openstack-ceilometer | 04:10 | |
| *** sayali has joined #openstack-ceilometer | 04:10 | |
| openstackgerrit | A change was merged to openstack/ceilometer: Use stevedore's make_test_instance https://review.openstack.org/69491 | 04:13 |
| *** sayali_ has quit IRC | 04:13 | |
| *** sayali_ has joined #openstack-ceilometer | 04:16 | |
| *** sayali_ has quit IRC | 04:21 | |
| *** sayali_ has joined #openstack-ceilometer | 04:22 | |
| *** ildikov_ has quit IRC | 05:57 | |
| openstackgerrit | Jenkins proposed a change to openstack/ceilometer: Imported Translations from Transifex https://review.openstack.org/62808 | 06:02 |
| *** sayali_ has quit IRC | 06:17 | |
| *** sayali_ has joined #openstack-ceilometer | 06:21 | |
| *** xianghui has joined #openstack-ceilometer | 06:41 | |
| *** eglynn-afk has joined #openstack-ceilometer | 06:57 | |
| *** flwang has quit IRC | 07:04 | |
| *** xianghui has quit IRC | 07:10 | |
| *** xianghui has joined #openstack-ceilometer | 07:13 | |
| openstackgerrit | Jenkins proposed a change to openstack/ceilometer: Updated from global requirements https://review.openstack.org/68237 | 07:16 |
| openstackgerrit | liusheng proposed a change to openstack/python-ceilometerclient: Reduce redundant parameter of some commands in CLI https://review.openstack.org/66776 | 07:24 |
| openstackgerrit | liusheng proposed a change to openstack/python-ceilometerclient: Reduce redundant parameter of some commands in CLI https://review.openstack.org/66776 | 07:27 |
| openstackgerrit | liusheng proposed a change to openstack/python-ceilometerclient: Reduce redundant parameter of some commands in CLI https://review.openstack.org/66776 | 07:28 |
| *** flwang has joined #openstack-ceilometer | 07:34 | |
| *** ildikov_ has joined #openstack-ceilometer | 08:05 | |
| *** yfujioka has quit IRC | 08:07 | |
| openstackgerrit | A change was merged to openstack/ceilometer: Clean .gitignore https://review.openstack.org/69366 | 08:15 |
| *** eglynn-afk has quit IRC | 08:17 | |
| *** eglynn-afk has joined #openstack-ceilometer | 08:20 | |
| *** s2r2 has quit IRC | 08:20 | |
| *** yassine has joined #openstack-ceilometer | 08:23 | |
| *** s2r2 has joined #openstack-ceilometer | 08:26 | |
| openstackgerrit | ZhiQiang Fan proposed a change to openstack/ceilometer: Remove unnecessary code from alarm test https://review.openstack.org/69557 | 08:31 |
| *** xianghui has quit IRC | 08:35 | |
| *** _nadya_ has joined #openstack-ceilometer | 08:42 | |
| *** _ruhe is now known as ruhe | 08:51 | |
| *** tian has joined #openstack-ceilometer | 08:52 | |
| tian | hi, anyone can help me ? how to use the function notify in ceilometer.computer.nova_notifier.py ? | 08:55 |
| *** _nadya_ has quit IRC | 08:55 | |
| *** eglynn-afk is now known as eglynn | 08:59 | |
| *** lsmola_ has joined #openstack-ceilometer | 09:08 | |
| *** SergeyLukjanov_ is now known as SergeyLukjanov | 09:08 | |
| *** _nadya_ has joined #openstack-ceilometer | 09:13 | |
| *** jsuchome has joined #openstack-ceilometer | 09:20 | |
| *** ruhe is now known as _ruhe | 09:20 | |
| openstackgerrit | A change was merged to openstack/python-ceilometerclient: Raise traceback on error when using CLI and -debug https://review.openstack.org/69236 | 09:25 |
| openstackgerrit | A change was merged to openstack/python-ceilometerclient: Update client to display data type of traits https://review.openstack.org/67224 | 09:25 |
| *** Alexei_987 has joined #openstack-ceilometer | 09:31 | |
| *** _nadya_ has quit IRC | 09:40 | |
| *** _nadya_ has joined #openstack-ceilometer | 09:48 | |
| *** _nadya_ has quit IRC | 09:52 | |
| openstackgerrit | Alexei Kornienko proposed a change to openstack/ceilometer: Refactored session access https://review.openstack.org/67850 | 09:54 |
| *** lsmola_ has quit IRC | 09:58 | |
| *** flwang has quit IRC | 10:03 | |
| *** ityaptin has quit IRC | 10:11 | |
| *** lsmola_ has joined #openstack-ceilometer | 10:13 | |
| *** lsmola_ has quit IRC | 10:43 | |
| *** flwang has joined #openstack-ceilometer | 10:46 | |
| *** lsmola_ has joined #openstack-ceilometer | 10:56 | |
| *** boris-42 has quit IRC | 11:11 | |
| *** SergeyLukjanov is now known as SergeyLukjanov_a | 11:19 | |
| *** SergeyLukjanov_a is now known as SergeyLukjanov | 11:19 | |
| openstackgerrit | Ildiko Vancsa proposed a change to openstack/ceilometer: Add documentation for pipeline configuration https://review.openstack.org/69350 | 11:20 |
| openstackgerrit | Shuangtai Tian proposed a change to openstack/ceilometer: Use flavorid instead of id in the nova_notifier https://review.openstack.org/69582 | 12:00 |
| *** boris-42 has joined #openstack-ceilometer | 12:31 | |
| openstackgerrit | A change was merged to openstack/ceilometer: Remove unnecessary code from alarm test https://review.openstack.org/69557 | 12:42 |
| openstackgerrit | A change was merged to openstack/ceilometer: use common code for migrations https://review.openstack.org/69459 | 12:42 |
| *** _ruhe is now known as ruhe | 12:51 | |
| *** jdob has joined #openstack-ceilometer | 13:04 | |
| *** flwang has quit IRC | 13:10 | |
| *** dhellmann_ is now known as dhellmann | 13:16 | |
| *** flwang has joined #openstack-ceilometer | 13:33 | |
| *** dhellmann is now known as dhellmann_ | 13:36 | |
| *** dhellmann_ is now known as dhellmann | 13:37 | |
| gibi | lsmola_: are you around? | 13:44 |
| lsmola_ | gibi: yes, hello | 13:44 |
| gibi | lsmola_: hello! We did some thinking with ildikov_ about the instance statistic query you need | 13:45 |
| *** prad has joined #openstack-ceilometer | 13:45 | |
| ildikov_ | lsmola_: hi | 13:45 |
| lsmola_ | ildikov_: hello | 13:45 |
| lsmola_ | gibi: ildikov_ have you figured it out ? :-) | 13:45 |
| gibi | lsmola_: we made some progress... | 13:46 |
| gibi | :) | 13:46 |
| lsmola_ | gibi: seems like it will be quite a complex query | 13:46 |
| lsmola_ | :-) | 13:46 |
| gibi | lsmola_: I think in sql you would need something like the following | 13:46 |
| ildikov_ | lsmola_: we tried to figure out, how could we solve your problem and we also tried to figure out a generic way, to achieve this | 13:46 |
| gibi | SELECT count(*), project_id FROM ( SELECT DISTINCT resource_id, project_id FROM Meter) GROUP BY project_id | 13:46 |
| gibi | lsmola_: I think this would give you the number of resources that was active at a given period | 13:47 |
| gibi | grouped by project_id | 13:47 |
| lsmola_ | gibi: ok | 13:47 |
| lsmola_ | gibi: but how to run it for each period? | 13:47 |
| gibi | the current old query handles period as spearate querys in a for loop :) | 13:48 |
| lsmola_ | gibi: oh god :-) | 13:48 |
| gibi | looping over the time window with the given period | 13:48 |
| Alexei_987 | jd__: question about eventlet. Recently a patch landed that disabled it. Are we going to use this moment to enable py3 tests for ceilometer? | 13:48 |
| lsmola_ | gibi: though i think I saw something like this | 13:48 |
| jd__ | Alexei_987: I wish but it's not enough yet | 13:48 |
| jd__ | the main event loop still depends on eventlet | 13:49 |
| gibi | the reason is that sql time handling is dialect specific | 13:49 |
| lsmola_ | gibi: I know elastic search has feature called facets for this kind of query | 13:49 |
| Alexei_987 | jd__: ah I see | 13:49 |
| *** viktors has joined #openstack-ceilometer | 13:49 | |
| lsmola_ | gibi: also in postgre, you could do something like that with WINDOW I suppose | 13:49 |
| lsmola_ | gibi: not sure about what SQL alchemy supports though | 13:50 |
| lsmola_ | gibi: and not sure what mongo has | 13:50 |
| gibi | lsmola_: I'm also not sure Mysql has WINDOW :) | 13:50 |
| gibi | so I think we have to keep the looping to be compatible with multiple SQL dialect | 13:51 |
| lsmola_ | gibi: would be nice to have it in one query, cause the overview pages will shoot a lot of queries | 13:51 |
| lsmola_ | gibi: that sounds pretty evil to me :-) | 13:51 |
| lsmola_ | gibi: it will be sooo slow, nobody will actually use it :-( | 13:51 |
| gibi | it is, the other way would be to have different implementation for different SQL dialect | 13:51 |
| lsmola_ | gibi: well ceilometer supports only mysql and postgree AFAIK | 13:52 |
| lsmola_ | gibi: or more? | 13:52 |
| lsmola_ | gibi: for queries like this, SQL alchemy is just not good enough | 13:52 |
| gibi | lsmola_: good question :) I would think that it can suppot whatever SQL engine that SQL Alchemy supports | 13:53 |
| lsmola_ | gibi: I think I saw they do the aggregation in the loop also | 13:53 |
| lsmola_ | gibi: i believe Ceilometer has only those two in doc | 13:53 |
| *** JCxMLnblFl has joined #openstack-ceilometer | 13:54 | |
| *** JCxMLnblFl has left #openstack-ceilometer | 13:54 | |
| gibi | lsmola_: I'm not sure which aggregation you refer to. Anyhow the period handling is on side of the problem | 13:54 |
| gibi | lsmola_: the other is that for your query we need subqueries | 13:54 |
| gibi | lsmola_: allowing generic support for subqueries makes the API very complex | 13:55 |
| lsmola_ | gibi: yeah that is true | 13:55 |
| lsmola_ | gibi: well the other option would be to expose a new meter | 13:56 |
| gibi | lsmola_: for example you might want to define a WHERE clause (a filter expression) at every subquery separately | 13:56 |
| lsmola_ | gibi: yeah | 13:57 |
| gibi | lsmola_: separate meter is something we have to think about but it sounds a good idea as it would hide the complexity behind the descriptive name of the new meter | 13:57 |
| ildikov_ | lsmola_: I haven't found a reference in sqlalchemy that it would have support for the period | 13:58 |
| openstackgerrit | A change was merged to openstack/ceilometer: Updated from global requirements https://review.openstack.org/68237 | 13:58 |
| lsmola_ | ildikov_: yeah you would probably have to implement it with WINDOW | 13:59 |
| ildikov_ | lsmola_: we had a kind of similar idea, like this new meter, we could have an aggregated meter, or something like, we just weren't sure if it will be accepted as a solution for your need | 13:59 |
| *** gordc has joined #openstack-ceilometer | 13:59 | |
| lsmola_ | ildikov_: does it support window? | 13:59 |
| lsmola_ | ildikov_: the prepared aggregated meter could work | 14:00 |
| lsmola_ | ildikov_: at least for project overview page | 14:00 |
| lsmola_ | ildikov_: so having samples e.g. total_nodes, total_nodes_up, percentage of nodes up for project | 14:02 |
| lsmola_ | ildikov_: then i could show it in timeline any way i want | 14:02 |
| gibi | lsmola_: postgres supports WINDOW but sqlalchemy is not. :/ | 14:03 |
| *** eglynn is now known as eglynn-lunch | 14:03 | |
| ildikov_ | lsmola_: I could not find WINDOW in sqlalchemy, but I'm not an expert | 14:03 |
| lsmola_ | ildikov_: yeah it probably doesn't support it | 14:03 |
| gibi | lsmola_: total_nodes meter is possible to do with the current data but total_nodes_up needs extra data to be collected as the VM status is not inculded in the metadata of the "instance" meter | 14:04 |
| lsmola_ | gibi: I believe it has value 1 when it is up, and 0 when suspended | 14:05 |
| lsmola_ | gibi: not sure though | 14:05 |
| gibi | basically the total_nodes would be total_allocated_nodes as the current "instance" meter has value 1 if it is ACTIVE, PAUSED, STOPPED and there os no sample with 0 value | 14:05 |
| ildikov_ | lsmola_: I'm not against of the aggregated meters idea for those values that requires subqueries for count it, as to support the generation of queries like the one you need is a complex problem | 14:05 |
| lsmola_ | gibi: I was planning to write Documentation of the meters | 14:05 |
| lsmola_ | gibi: cause nobody know how they act :-) | 14:06 |
| lsmola_ | gibi: hm, ok | 14:06 |
| ildikov_ | lsmola_: we've tried before ping you, just to be sure :) | 14:06 |
| gibi | lsmola_: we just tried. There is no 0 reported for stopped VMs | 14:06 |
| gibi | lsmola_: I think we can write a blueprint for this new meters to be added and let's see how the other thinks about this direction | 14:07 |
| lsmola_ | gibi: i thought I saw 0 :-) | 14:07 |
| ildikov_ | lsmola_: we had the instance meter, I do not think that we missed any configuration, which would change the content of that meter | 14:07 |
| lsmola_ | gibi: ok | 14:07 |
| lsmola_ | gibi: i will put there links to overview pages wireframes | 14:08 |
| lsmola_ | gibi: so we can see what everything we need | 14:08 |
| gibi | lsmola_: I shortly checked you pdf, but I think I will do a deeper look in it | 14:08 |
| lsmola_ | ildikov_: yeah, suspending of machine probably restart some meter | 14:09 |
| lsmola_ | ildikov_: what measures uptime? | 14:09 |
| ildikov_ | lsmola_: that would be good, if we would have everything in place that is needed | 14:09 |
| lsmola_ | ildikov_: I believe we have blueprint for admin and project overview pages, there should be wireframes for it | 14:09 |
| gordc | dhellmann: do you know if there are any settings i need to set to get oslo.rootwrap to work? i keep getting an import error "from oslo.rootwrap.cmd import main\nImportError: No module named rootwrap.cmd" when running devstack. | 14:10 |
| lsmola_ | ildikov_: and we are preparing some more wireframes | 14:10 |
| ildikov_ | lsmola_:I do not think that we currently measure uptime | 14:10 |
| ildikov_ | lsmola_:ok, I will check that bp | 14:11 |
| lsmola_ | gibi: somebody deleted list of nova meters :-o http://docs.openstack.org/developer/ceilometer/measurements.html | 14:11 |
| ildikov_ | lsmola_:will the additional wireframs will be linked there too | 14:11 |
| ildikov_ | lsmola_:? | 14:11 |
| lsmola_ | ildikov_: yes, it should be there | 14:11 |
| ildikov_ | lsmola_:yes, eglynn had a not sphinx compatible change in that table, the fix is up for review already :) | 14:12 |
| *** prad has quit IRC | 14:12 | |
| lsmola_ | ildikov_: ok | 14:12 |
| gibi | lsmola_: thanks for the info, we will try to draft a bp for this new meters | 14:17 |
| lsmola_ | gibi: ildikov_ excellent, thank you :-) | 14:18 |
| ildikov_ | lsmola_:in that table there is instance and instance:<type> meters and the description is duration for both, but it is not duration, the text is not correct there | 14:18 |
| lsmola_ | ildikov_: there is some duration counter somewhere, as cumulative meter | 14:19 |
| *** CephFan1 has joined #openstack-ceilometer | 14:21 | |
| gibi | lsmola_: there is cpu meter that measures cpu time in nanoseconds | 14:21 |
| CephFan1 | Does Ceilometer have Ceph Object storage integration? I keep seeing this error in central manager Account HEAD failed:http://192.168.0.100:80/v1/AUTH_9cb8481b89725a5aqvr14b203e4551x0 400 Bad Reques | 14:21 |
| lsmola_ | gibi: yeah, that is probably it | 14:21 |
| lsmola_ | gibi: not sure how it is related, but the counter is being restarted after suspend | 14:22 |
| ildikov_ | lsmola_: I saw cumulative counters only for cpu and disk and network I/O rate meters | 14:22 |
| lsmola_ | gibi: so getting toatl uptime is pretty hard when you suspend it in the middle | 14:22 |
| gibi | lsmola_: yes, I saw that same that cpu counter is reset if the VM suspended so you are right is is not easy to calculate total uptime if that is required | 14:24 |
| ildikov_ | lsmola_: I think there is a need periodically for having uptime meter, maybe it would be good to have an uptime and total uptime to handle somehow when the instance and/or the hypervisor is down | 14:24 |
| ildikov_ | lsmola_: I do not know how we can get the info from libvirt, I need to check what can be done to have an uptime meter | 14:25 |
| lsmola_ | ildikov_: yeah as I heard it is just showing libvirt counter, which is being restarted | 14:27 |
| openstackgerrit | Ilya Tyaptin proposed a change to openstack/ceilometer: Add table prefix for unit tests with hbase https://review.openstack.org/68698 | 14:30 |
| *** sayali_ has quit IRC | 14:30 | |
| ildikov_ | lsmola_: we need to investigate how we can gather the required information and I will also check if there is any bug or bp registered for this uptime topic | 14:31 |
| openstackgerrit | A change was merged to openstack/ceilometer: Use explicit http error code for api v2 https://review.openstack.org/68775 | 14:43 |
| *** RelayChatInfo has joined #openstack-ceilometer | 14:47 | |
| *** RelayChatInfo has left #openstack-ceilometer | 14:47 | |
| openstackgerrit | A change was merged to openstack/python-ceilometerclient: Remove unused import for print_function https://review.openstack.org/69518 | 14:51 |
| *** prad has joined #openstack-ceilometer | 14:55 | |
| *** ruhe is now known as _ruhe | 14:56 | |
| *** jmckind has joined #openstack-ceilometer | 15:05 | |
| *** urulama has joined #openstack-ceilometer | 15:06 | |
| *** eglynn-lunch is now known as eglynn-call | 15:07 | |
| ildikov_ | lsmola_:I've found a blueprint for VM state: https://blueprints.launchpad.net/ceilometer/+spec/state-meter | 15:12 |
| *** rwsu has joined #openstack-ceilometer | 15:12 | |
| *** _ruhe is now known as ruhe | 15:30 | |
| gibi | lsmola_: just checked the cpu meter but that is the calculated time that the VM used not wall clock time. So if the VM loads the cpu 50% for a minute then that meter reports 30 sec cpu time | 15:33 |
| lsmola_ | gibi: interesting | 15:33 |
| lsmola_ | gibi: do you think you could put those facts into doc somewhere? :-) | 15:34 |
| gibi | lsmola_: that is a fair request :) | 15:35 |
| eglynn-call | gibi: yes, the cpu meter measure cumulative CPU time actually used by an instance (not the wall-clock duration for which the CPU is associated with an instance) | 15:36 |
| gibi | eglynn-call: you can help with the documentation :) | 15:37 |
| ildikov_ | eglynn-call: will you available for a short statistics talk today? :) | 15:38 |
| eglynn-call | gibi: some new documentation, or just clarifying what's already described in https://github.com/openstack/ceilometer/blob/master/doc/source/measurements.rst ? | 15:41 |
| gibi | eglynn-call: I think that needs clarification as it was not clear for me at first. I had to check libvirt to see | 15:43 |
| gibi | I will propose a patch for it | 15:43 |
| eglynn-call | ildikov_: I'm on a call right now, but if you want to take the stats question off-line to an email I'll try to get to it before EoD | 15:43 |
| eglynn-call | gibi: cool enough, put me down as a reviewer pls | 15:43 |
| gibi | eglynn-call: OK, I will do | 15:44 |
| eglynn-call | gibi: thank you sir! | 15:44 |
| ildikov_ | eglynn-call:I can send some initials via mail, but it would be better, if we could have an online more-or-less brainstorming session about this topic, if it is ok for you | 15:48 |
| ildikov_ | eglynn-call: we had a discussion today with gibi about distinct and some generic way of creating statistics, but we haven't reached a consensus yet | 15:50 |
| eglynn-call | ildikov_: k, IRC discussion at circa 4:30 work for you? | 15:51 |
| ildikov_ | eglynn-call: it's good for us, thanks | 15:53 |
| *** boris-42 has quit IRC | 16:07 | |
| *** SergeyLukjanov is now known as SergeyLukjanov_ | 16:09 | |
| openstackgerrit | Balazs Gibizer proposed a change to openstack/ceilometer: Enhance the documentation of the cpu measurement https://review.openstack.org/69637 | 16:10 |
| *** eglynn-call is now known as eglynn | 16:12 | |
| eglynn | ildikov_, gibi: k, small digression first ... | 16:16 |
| eglynn | so currently we have say 'GET /v2/meters/instance/statistics?period=p' | 16:17 |
| eglynn | which returns (sum, max, min, avg, count) | 16:17 |
| eglynn | for the 'instance' meter, aggregated for each period p | 16:17 |
| gibi | eglynn: go ahead :) | 16:17 |
| eglynn | yep, so we get the *same* set of aggregate function applied each time | 16:18 |
| eglynn | my idea was to allow individual aggregate functions to be selected | 16:18 |
| eglynn | *and* for these functions to be parameterized | 16:18 |
| eglynn | so for example ... | 16:18 |
| eglynn | GET /v2/meters/instance/statistics?aggregate-by=stddev&period=p | 16:18 |
| eglynn | to force the computation of stddev instead of the normal (sum, max, min, avg, count) | 16:19 |
| eglynn | then to extend this to allow the aggregate function itself be parameterized | 16:19 |
| eglynn | say for example ... | 16:19 |
| eglynn | GET /v2/meters/instance/statistics?aggregate-by=quantile&aggregate-on=0.99&period=p | 16:20 |
| jd__ | does anyone recall how we handle NotImplemented from the storage driver in the API? | 16:20 |
| eglynn | to allow the 99th percentile to calculated | 16:20 |
| eglynn | jd__: doesn't it just throw the NotImplemented back to the caller? | 16:20 |
| eglynn | ... so what's the relevance of these paremeterized aggregates? | 16:21 |
| eglynn | ... well it occurred to me that *distinct* could itself be though of as an aggregate function | 16:22 |
| gibi | distint in SQL is equvivalent to a groupby without agggregation function | 16:22 |
| gibi | s/distint/distinct/ | 16:23 |
| eglynn | ... where the aggregate-on is the attribute we're counting the distinct values for | 16:23 |
| eglynn | gibi: in sqlalchemy, distinct can be counted via func.count(distinct(Model.attribute)), no? | 16:24 |
| eglynn | so say ... | 16:24 |
| eglynn | GET /v2/meters/instance/statistics?aggregate-by=distinct&aggregate-on=resource_id&period=p | 16:24 |
| eglynn | would map to ... | 16:24 |
| eglynn | func.count(distinct(models.Meter.resource_id)).label('distinct') | 16:24 |
| eglynn | at or around line ... https://github.com/openstack/ceilometer/blob/master/ceilometer/storage/impl_sqlalchemy.py#L564 | 16:25 |
| eglynn | (instead of the usual func.avg, func.sum, func.max ... etc.) | 16:25 |
| eglynn | for horizon per-tenant grouping, we'd have ... | 16:25 |
| eglynn | GET /v2/meters/instance/statistics?aggregate-by=distinct&aggregate-on=resource_id&period=p&groupby=project_id | 16:26 |
| eglynn | gibi: ... does that make sense? | 16:26 |
| jd__ | eglynn: well I got a 500 in _one_ test because of that it seems, not sure why | 16:27 |
| eglynn | I'm no SQL expert, but it doesn't seem any worse than the current aggregation cost in the sqlalchemy driver | 16:27 |
| gibi | eglynn: well you made a trick here I guess | 16:28 |
| eglynn | jd__: that's strange, don't the scenario tests have a strategy for swallow the not NotImplementedErrors and skipping those tests? | 16:28 |
| eglynn | *swallowing | 16:28 |
| gibi | eglynn: I tend to agree that it would work | 16:29 |
| eglynn | gibi: cool ... I was about to ask whether a "trick" was a good thing or a bad thing ;) | 16:29 |
| gibi | eglynn: I don't know if this trick with the distinct and group by is generic enough | 16:30 |
| eglynn | gibi: have you a usecase in mind? | 16:30 |
| eglynn | (that it wouldn't address) | 16:30 |
| jd__ | eglynn: I thought so but I'm lost in the code :) | 16:31 |
| *** ruhe is now known as _ruhe | 16:31 | |
| eglynn | jd__: ... I know the feeling | 16:31 |
| gibi | eglynn: In general we might need to group by on multiple fields | 16:31 |
| gibi | eglynn: for example | 16:31 |
| eglynn | gibi: ... would that not still work? | 16:32 |
| eglynn | gibi: ... i.e. we could still groupby on both foo and bar while counting distinct snafus? | 16:33 |
| gibi | eglynn: let's assume we want to calculate an avg(counter_volume) group by resource_id and then a sum() group by project_id | 16:33 |
| eglynn | in separate queries, or? | 16:33 |
| gibi | in one query. | 16:34 |
| * gibi try to make a real example... | 16:34 | |
| openstackgerrit | Ilya Tyaptin proposed a change to openstack/ceilometer: Skip unit tests with mongo or db2 when environment variables aren't set https://review.openstack.org/69644 | 16:34 |
| eglynn | gibi: well as things stand, surely groupby is per-query? | 16:35 |
| eglynn | gibi: the ability to conflate different groupbys in a single query seems to me like a different problem | 16:36 |
| gibi | let's assume we have a proper memory meter that reports actual memory usage of VMs | 16:36 |
| eglynn | k... maybe even like a hammer looking for a nail ;) | 16:37 |
| gibi | and we need a chart with project level memory usage | 16:37 |
| eglynn | yep | 16:37 |
| gibi | then we first need to calc an average group by resource_id on a period then a sum group by project id | 16:38 |
| openstackgerrit | Ilya Tyaptin proposed a change to openstack/ceilometer: Skip unit tests with mongo or db2 when environment variables aren't set https://review.openstack.org/69644 | 16:38 |
| gibi | if we want to do it in one query then we need a subquery | 16:38 |
| eglynn | gibi: sure, but sounds like 3 separate queries is the most natural way of doing it though | 16:38 |
| * jd__ found his problem | 16:39 | |
| eglynn | gibi: ... i.e. I'm not sure I get why the conflation into a single query would be important? | 16:39 |
| eglynn | jd__: \o/ ... enlighten us | 16:39 |
| openstackgerrit | Julien Danjou proposed a change to openstack/ceilometer: samples: fix test case status code check https://review.openstack.org/69648 | 16:40 |
| openstackgerrit | Julien Danjou proposed a change to openstack/ceilometer: Use swift master https://review.openstack.org/68150 | 16:40 |
| openstackgerrit | Julien Danjou proposed a change to openstack/ceilometer: storage: bases of a Cassandra driver https://review.openstack.org/62779 | 16:40 |
| jd__ | eglynn: https://review.openstack.org/69648 | 16:40 |
| gibi | eglynn: actually it is only important on API level as in SQLAlchemy we can have subqueries for sure | 16:40 |
| gibi | (and in mongo we can have multiple map-reduce) | 16:40 |
| eglynn | jd__: a-ha | 16:41 |
| gibi | eglynn: your API proposal is good in a sense that it does not allow too complex things like multiple group bys | 16:41 |
| gibi | eglynn: and it is problematic in the same sense :) as it does not allow querying the mem usage on project level because distinct is not enough there | 16:42 |
| eglynn | gibi: k, so if I understand you right, you're thinking that different groupbys are *possible* to do in the storage drivers (via sub-queries, multiple M-Rs, ...) | 16:43 |
| eglynn | gibi: ... but does that make them desirable to support in the API? | 16:43 |
| gibi | eglynn: yes, I'm affraid of the API complexity | 16:44 |
| eglynn | gibi: me too :) | 16:44 |
| gibi | eglynn: I mean if we allow multiple group bys, but on the other hand multiple group by is needed by horizon for the mem usage, or the disk usage statistics | 16:44 |
| eglynn | gibi: ... also I don't think I fully understand your mem usage example | 16:45 |
| gibi | eglynn: here is the overview page plane for horizon http://people.redhat.com/~lsurette/OpenStack/Horizon%20Admin%20Overview%20Pages_2.0.pdf | 16:45 |
| gibi | eglynn: ther you see something like network usage calculated on project level | 16:46 |
| gibi | to do that you have to first aggregate the average on resource level on a period then summ it on project level | 16:46 |
| gibi | and for that we need two group bys and that is something that opens the road to a complex AP | 16:47 |
| gibi | s/AP/API/ | 16:47 |
| gibi | If we allow two groupbys then we basically allow subquery and then then we end up supporting filtering on subquery level | 16:49 |
| eglynn | gibi: I'm still confused about that example | 16:50 |
| eglynn | gibi: why would make sense to sum the averages? | 16:50 |
| eglynn | gibi: why not just sum the underlying raw data? | 16:50 |
| eglynn | gibi: wouldn't you get the same answer either way? | 16:51 |
| openstackgerrit | Julien Danjou proposed a change to openstack/ceilometer: samples: fix test case status code check https://review.openstack.org/69648 | 16:51 |
| openstackgerrit | Julien Danjou proposed a change to openstack/ceilometer: storage: bases of a Cassandra driver https://review.openstack.org/62779 | 16:51 |
| gibi | you have two VMs that report every minute the average network usage in B/s. | 16:53 |
| gibi | (the two VM is in the same project) | 16:53 |
| gibi | and now we need to draw a chart that show the average network usage on project level and in one hour period | 16:54 |
| eglynn | gibi: k, when you said average I thought you meant the average aggregate function applied to some sample data | 16:54 |
| eglynn | gibi: (not a meter that captures a rate per second) | 16:54 |
| openstackgerrit | Ilya Tyaptin proposed a change to openstack/ceilometer: Replace non-ascii symbols in docs https://review.openstack.org/69658 | 16:55 |
| gibi | eglynn: ok. :) so we both need to make average on a longer time period than the sample are reported and in the same time sum up the VM in the same project | 16:55 |
| gibi | s/VM/VMs/ | 16:56 |
| gibi | eglynn: I don't know if it is clear enough. :) | 17:01 |
| eglynn | gibi: I'm struggling to understand by the averaging per-VM is even required | 17:02 |
| eglynn | gibi: i.e. why you can't just average over the entire project and multiply that by the number of VMs? | 17:02 |
| eglynn | s/struggling to understand by/struggling to understand *why*/ | 17:05 |
| *** boris-42 has joined #openstack-ceilometer | 17:07 | |
| *** flwang has quit IRC | 17:10 | |
| gibi | eglynn: you are right we can rearrange the things to a sum and a multiplication. Do we want that the user express the query in this way or we need to translate it to this less complex from? | 17:11 |
| * gibi needed to do some basic math. :) | 17:11 | |
| eglynn | gibi: well my preference would be to only give the user as much hammer as they really need | 17:12 |
| eglynn | gibi: (as opposed to exposing a lot of complexity in API, even if an equivalent query can be expressed more simply from the get-go) | 17:12 |
| openstackgerrit | Pradeep Kilambi proposed a change to openstack/ceilometer: Fix docs on what an instance meter represents https://review.openstack.org/66746 | 17:13 |
| gibi | eglynn: I agree that complexity is a bad thing. | 17:14 |
| *** _ruhe is now known as ruhe | 17:15 | |
| gibi | eglynn: lsmola_ needs this instance statistics that can be solved with distinct and one group by, he also needs the network statistics I used as an example above | 17:15 |
| gibi | in that usecase your API proposal with distinct is not enough | 17:15 |
| jaypipes | jd__: question for you... looking at your cassandra patches, I didn't know that Cassandra depended on swift... is that the case? | 17:17 |
| gibi | we either need to allow multiple groupby or allow to express the multiplication | 17:17 |
| gibi | or the client needs to do two separate queries and then do the multiplication on the client side | 17:17 |
| eglynn | gibi: ... /me is not seeing that as unreasonable burden on the client TBH | 17:18 |
| gibi | eglynn: OK. Understood. So only the instance statistics is special due to the fact that that meter only reports when a VM exists | 17:23 |
| gibi | therefore in this case we cannot move the problem to the client side in a same way as in the network statistics case | 17:23 |
| openstackgerrit | Victor Sergeyev proposed a change to openstack/ceilometer: WIP: Add test for check sync models and migrations https://review.openstack.org/69674 | 17:25 |
| gibi | eglynn: a bit different question. When you introduce the distinct in the query then what changes needed to be done on the Statistics object used in the API as a return value for the query? | 17:26 |
| eglynn | gibi: yeah, the distinct instance isn't suitable for moving to the client side as there wouldn't be much prior distillation of the data within the API layer | 17:26 |
| eglynn | gibi: yeah, we probably need a generic name-value pair on the Statistics API representation also | 17:27 |
| *** ruhe is now known as _ruhe | 17:28 | |
| eglynn | gibi: (as that's currently hardcoded with the current set of aggregate labels) | 17:28 |
| jd__ | jaypipes: no | 17:28 |
| gibi | eglynn: Yes, I agree that we need to change that part to something more generic. I guess the parameterised aggregation functions also needs some extra care there | 17:29 |
| openstackgerrit | Pradeep Kilambi proposed a change to openstack/ceilometer: Fix measurement docs around various meters to represent Existance instead of Duration https://review.openstack.org/69675 | 17:29 |
| eglynn | gibi: yep, that's a good point ... overall though are we ad idem that distinct as a new statistics aggregate function is a workable idea? | 17:30 |
| jaypipes | jd__: is there any reason the cassandra patch is dependent on the swift git master patch? | 17:33 |
| jaypipes | jd__: sorry, when I see a dependent patch in gerrit, I believe that the patches are related... | 17:34 |
| *** yassine has quit IRC | 17:34 | |
| jd__ | jaypipes: yeah no relation, just the result of my incremental bug fixing in that branch | 17:35 |
| jaypipes | jd__: k, no worries, was just checking with you. thx. | 17:35 |
| gibi | eglynn: I think distinct will work for this specific usecase. However it only works in this case because we can assume that one resource is always belongs to onyl one project | 17:35 |
| jaypipes | jd__: figured easier to ask you in IRC than have you get mad at me for asking a question in review ;) | 17:35 |
| jd__ | lol | 17:36 |
| *** sayali has joined #openstack-ceilometer | 17:36 | |
| *** SergeyLukjanov_ is now known as SergeyLukjanov | 17:36 | |
| jd__ | I never get mad | 17:36 |
| jaypipes | :) | 17:36 |
| gibi | eglynn: I don't know that if we allow distinct in general then what kind of other assumptions we need to make | 17:37 |
| *** viktors has left #openstack-ceilometer | 17:38 | |
| *** Alexei_987 has quit IRC | 17:38 | |
| eglynn | gibi: I think distinct should be allowed in general, but probably most useful in practice for grouped-by queries | 17:39 |
| eglynn | gibi: gotta drop off shortly (... kids to feed) | 17:39 |
| eglynn | gibi: let's chat further tmrw if you any more doubts on that approach | 17:39 |
| gibi | eglynn: OK. Thanks for the discussion. Let's continue it tomorrow | 17:40 |
| eglynn | gibi: cool | 17:41 |
| *** eglynn has quit IRC | 17:54 | |
| *** ildikov_ has quit IRC | 18:04 | |
| *** xmltok has joined #openstack-ceilometer | 18:11 | |
| *** kwhitney has left #openstack-ceilometer | 18:18 | |
| openstackgerrit | A change was merged to openstack/python-ceilometerclient: Modify ceilometer client cmd line help info https://review.openstack.org/63984 | 18:19 |
| *** Alexei_987 has joined #openstack-ceilometer | 18:33 | |
| openstackgerrit | Pradeep Kilambi proposed a change to openstack/ceilometer: Fix measurement docs to correctly represent Existance meters https://review.openstack.org/69675 | 18:52 |
| *** ildikov_ has joined #openstack-ceilometer | 18:58 | |
| *** dperaza has joined #openstack-ceilometer | 19:00 | |
| *** jsuchome has quit IRC | 19:33 | |
| *** _ruhe is now known as ruhe | 19:34 | |
| *** rwsu has quit IRC | 19:36 | |
| *** rwsu has joined #openstack-ceilometer | 19:40 | |
| *** hewbrocca has joined #openstack-ceilometer | 19:41 | |
| *** hewbrocca has left #openstack-ceilometer | 19:41 | |
| *** tongli has joined #openstack-ceilometer | 20:17 | |
| *** ruhe is now known as _ruhe | 20:54 | |
| *** _ruhe is now known as ruhe | 21:00 | |
| *** yassine has joined #openstack-ceilometer | 21:12 | |
| *** urulama has quit IRC | 21:23 | |
| *** CephFan1 has quit IRC | 21:30 | |
| *** tongli has quit IRC | 21:44 | |
| *** dperaza has quit IRC | 21:50 | |
| *** SergeyLukjanov is now known as SergeyLukjanov_ | 21:59 | |
| *** ryanpetrello has joined #openstack-ceilometer | 22:32 | |
| *** jdob has quit IRC | 22:33 | |
| dhellmann | does anyone have any idea why I see "ImportError: No module named middleware.proxy_logging" when running the tests from trunk? | 22:51 |
| dhellmann | swift is installed in my .tox/py27 directory but it does not include a middleware package | 22:52 |
| dhellmann | the import line is actually "from swift.common.middleware.proxy_logging import InputProxy" | 22:52 |
| dhellmann | jd__, eglynn: ^^ | 22:53 |
| jd__ | yes | 22:55 |
| jd__ | dhellmann: you need this: https://review.openstack.org/#/c/68150/ | 22:55 |
| *** prad has quit IRC | 22:55 | |
| dhellmann | jd__: looks good, shall I approve it? | 22:55 |
| jd__ | dhellmann: I think so yeah | 22:56 |
| dhellmann | oh, you submitted it, that's why you haven't approved it | 22:56 |
| jd__ | :-) | 22:56 |
| * dhellmann is starting to get crossed eyes from reviewing code | 22:56 | |
| dhellmann | jd__: +2 a | 22:56 |
| jd__ | dhellmann: while you're around, any idea what I need to do to have a new pbr released? | 22:57 |
| dhellmann | technically I can do it, but I would want to check with mordred first | 22:58 |
| jd__ | dhellmann: then, would you be kind enough to check with him once https://review.openstack.org/#/c/63236/ is merged? | 22:58 |
| jd__ | I would then be able to go forward with Oslo portage | 22:59 |
| dhellmann | jd__: sure | 22:59 |
| jd__ | \o/ | 22:59 |
| dhellmann | actually, I'm not 100% sure I have that ACL, but I can check | 22:59 |
| jd__ | at least I hope we don't have a SPOF on that, as mordred hasn't be really responsive recently | 23:00 |
| dhellmann | we can get it fixed if we do have that | 23:01 |
| *** openstack has joined #openstack-ceilometer | 23:04 | |
| *** ruhe is now known as _ruhe | 23:10 | |
| openstackgerrit | Yuuichi Fujioka proposed a change to openstack/ceilometer: fix column name and alignment https://review.openstack.org/69524 | 23:54 |
| *** dperaza has joined #openstack-ceilometer | 23:56 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!