| *** liuhanxi has joined #openstack-telemetry | 00:02 | |
| *** zhurong has quit IRC | 00:06 | |
| *** chlong has quit IRC | 00:16 | |
| *** thorst has joined #openstack-telemetry | 00:21 | |
| *** thorst_ has joined #openstack-telemetry | 00:37 | |
| *** thorst has quit IRC | 00:39 | |
| *** liuhanxi has quit IRC | 00:40 | |
| openstackgerrit | xiaozhuangqing proposed openstack/ceilometer: Remove unnecessary confirm prompt https://review.openstack.org/362829 | 00:58 |
|---|---|---|
| openstackgerrit | fengchaoyang proposed openstack/gnocchi: Check whether the resource type attribute min is less than or equal to max https://review.openstack.org/361604 | 01:00 |
| *** zhurong has joined #openstack-telemetry | 01:13 | |
| *** donghao has joined #openstack-telemetry | 01:17 | |
| *** donghao has quit IRC | 01:20 | |
| *** donghao has joined #openstack-telemetry | 01:24 | |
| *** fguillot has quit IRC | 01:27 | |
| *** oneapm has joined #openstack-telemetry | 01:28 | |
| *** fguillot has joined #openstack-telemetry | 01:30 | |
| *** liuhanxi has joined #openstack-telemetry | 01:35 | |
| *** thorst_ has quit IRC | 01:38 | |
| *** thorst has joined #openstack-telemetry | 01:38 | |
| *** chlong has joined #openstack-telemetry | 01:40 | |
| *** thorst has quit IRC | 01:46 | |
| *** oneapm has quit IRC | 01:54 | |
| *** zhangguoqing has joined #openstack-telemetry | 01:57 | |
| *** vishwanathj has quit IRC | 01:59 | |
| *** zhurong_ has joined #openstack-telemetry | 02:00 | |
| *** zhurong has quit IRC | 02:02 | |
| *** Kevin_Zheng has joined #openstack-telemetry | 02:13 | |
| *** diogogmt has quit IRC | 02:19 | |
| *** fguillot has quit IRC | 02:30 | |
| *** thorst has joined #openstack-telemetry | 02:45 | |
| openstackgerrit | gecong proposed openstack/panko: Add missing %s in print message https://review.openstack.org/363361 | 02:47 |
| *** thorst has quit IRC | 02:52 | |
| *** donghao has quit IRC | 03:12 | |
| *** donghao has joined #openstack-telemetry | 03:14 | |
| *** dduvall has quit IRC | 03:23 | |
| *** dduvall has joined #openstack-telemetry | 03:23 | |
| *** adriant has quit IRC | 03:41 | |
| *** diogogmt has joined #openstack-telemetry | 03:44 | |
| *** thorst has joined #openstack-telemetry | 03:50 | |
| *** thorst has quit IRC | 03:57 | |
| *** donghao has quit IRC | 03:58 | |
| *** achatterjee has joined #openstack-telemetry | 04:30 | |
| *** links has joined #openstack-telemetry | 04:36 | |
| *** chlong has quit IRC | 04:38 | |
| *** chlong has joined #openstack-telemetry | 04:39 | |
| *** oneapm has joined #openstack-telemetry | 04:40 | |
| *** chlong has quit IRC | 04:45 | |
| *** thorst has joined #openstack-telemetry | 04:54 | |
| *** chlong has joined #openstack-telemetry | 04:57 | |
| *** donghao has joined #openstack-telemetry | 04:59 | |
| *** thorst has quit IRC | 05:01 | |
| *** donghao has quit IRC | 05:04 | |
| *** yprokule has joined #openstack-telemetry | 05:08 | |
| *** diogogmt has quit IRC | 05:17 | |
| *** _nadya_ has joined #openstack-telemetry | 05:54 | |
| *** donghao has joined #openstack-telemetry | 05:56 | |
| *** _nadya_ has quit IRC | 05:59 | |
| *** thorst has joined #openstack-telemetry | 06:00 | |
| *** donghao has quit IRC | 06:05 | |
| *** thorst has quit IRC | 06:07 | |
| *** pcaruana has joined #openstack-telemetry | 06:15 | |
| *** rcernin has joined #openstack-telemetry | 06:21 | |
| *** sheel has joined #openstack-telemetry | 06:24 | |
| openstackgerrit | Nguyen Phuong An proposed openstack/ceilometer: Config logABug feature for Ceilometer api-ref https://review.openstack.org/363430 | 06:57 |
| openstackgerrit | XueFeng Liu proposed openstack/ceilometer: Fix string interpolation in log https://review.openstack.org/355422 | 06:57 |
| *** thorst has joined #openstack-telemetry | 07:05 | |
| *** tesseract- has joined #openstack-telemetry | 07:08 | |
| *** zhangguoqing has quit IRC | 07:10 | |
| *** thorst has quit IRC | 07:12 | |
| *** shardy has joined #openstack-telemetry | 07:12 | |
| *** zhangguoqing has joined #openstack-telemetry | 07:17 | |
| openstackgerrit | Merged openstack/ceilometer: remove needless brackets https://review.openstack.org/362697 | 07:27 |
| *** ljxiash has joined #openstack-telemetry | 07:30 | |
| *** shardy has quit IRC | 07:39 | |
| *** shardy has joined #openstack-telemetry | 07:40 | |
| *** _nadya_ has joined #openstack-telemetry | 07:43 | |
| openstackgerrit | Merged openstack/ceilometer: The debug that network.incoming/outgoing.packets print is not right https://review.openstack.org/356333 | 07:45 |
| *** zhurong_ has quit IRC | 07:45 | |
| *** zhurong has joined #openstack-telemetry | 07:48 | |
| openstackgerrit | Wenyan Zhang proposed openstack/ceilometer: correct input params in get_samples https://review.openstack.org/363460 | 07:53 |
| *** cdent has joined #openstack-telemetry | 07:56 | |
| openstackgerrit | Julien Danjou proposed openstack/gnocchi: storage: re-store/write read-only splits https://review.openstack.org/362892 | 07:58 |
| openstackgerrit | Julien Danjou proposed openstack/gnocchi: carbonara: optimize uncompressed serialization https://review.openstack.org/358314 | 07:58 |
| openstackgerrit | Julien Danjou proposed openstack/gnocchi: carbonara: compress non padded timeseries https://review.openstack.org/358313 | 07:58 |
| openstackgerrit | Julien Danjou proposed openstack/gnocchi: carbonara: use calcsize rather than hard coded value https://review.openstack.org/358312 | 07:58 |
| *** r-mibu has quit IRC | 08:02 | |
| *** chlong has quit IRC | 08:06 | |
| *** thorst has joined #openstack-telemetry | 08:09 | |
| *** kelv has quit IRC | 08:10 | |
| *** ljxiash has quit IRC | 08:12 | |
| *** thorst has quit IRC | 08:17 | |
| *** ljxiash has joined #openstack-telemetry | 08:21 | |
| *** kong has quit IRC | 08:22 | |
| *** kong has joined #openstack-telemetry | 08:23 | |
| *** liuhanxi has quit IRC | 08:24 | |
| *** r-mibu has joined #openstack-telemetry | 08:33 | |
| *** ossifrath has quit IRC | 08:33 | |
| *** ossifrath has joined #openstack-telemetry | 08:36 | |
| openstackgerrit | Julien Danjou proposed openstack/gnocchi: carbonara: optimize uncompressed serialization https://review.openstack.org/358314 | 08:52 |
| openstackgerrit | Julien Danjou proposed openstack/gnocchi: carbonara: compress non padded timeseries https://review.openstack.org/358313 | 08:52 |
| openstackgerrit | Julien Danjou proposed openstack/gnocchi: carbonara: Timeserie.aggregate https://review.openstack.org/363492 | 08:52 |
| openstackgerrit | fengchaoyang proposed openstack/gnocchi: Check whether the resource type attribute min is less than or equal to max https://review.openstack.org/361604 | 08:52 |
| *** nijaba has quit IRC | 09:05 | |
| *** nijaba has joined #openstack-telemetry | 09:07 | |
| *** nijaba has quit IRC | 09:07 | |
| *** nijaba has joined #openstack-telemetry | 09:07 | |
| *** liuhanxi has joined #openstack-telemetry | 09:13 | |
| *** thorst has joined #openstack-telemetry | 09:14 | |
| *** thorst has quit IRC | 09:22 | |
| *** ljxiash has quit IRC | 09:44 | |
| *** zhurong has quit IRC | 10:01 | |
| *** oneapm has quit IRC | 10:12 | |
| *** thorst has joined #openstack-telemetry | 10:20 | |
| *** thorst has quit IRC | 10:26 | |
| *** shardy is now known as shardy_lunch | 11:24 | |
| *** zhangguoqing has quit IRC | 11:37 | |
| *** thorst has joined #openstack-telemetry | 11:43 | |
| *** vishwanathj has joined #openstack-telemetry | 11:44 | |
| *** larainema has quit IRC | 11:58 | |
| *** Kevin_Zheng has quit IRC | 11:59 | |
| *** larainema has joined #openstack-telemetry | 12:02 | |
| *** Kevin_Zheng has joined #openstack-telemetry | 12:04 | |
| *** shardy_lunch is now known as shardy | 12:15 | |
| *** sheel has quit IRC | 12:16 | |
| *** pradk has joined #openstack-telemetry | 12:20 | |
| *** gordc has joined #openstack-telemetry | 12:27 | |
| openstackgerrit | xiaozhuangqing proposed openstack/ceilometer: agentbase: miss a blank in info https://review.openstack.org/363619 | 12:29 |
| *** openstackgerrit has quit IRC | 12:34 | |
| *** openstackgerrit has joined #openstack-telemetry | 12:35 | |
| *** eglynn has joined #openstack-telemetry | 12:42 | |
| *** zhurong has joined #openstack-telemetry | 12:44 | |
| *** jefrite has quit IRC | 12:51 | |
| *** jefrite has joined #openstack-telemetry | 12:56 | |
| *** pradk has quit IRC | 13:01 | |
| openstackgerrit | Julien Danjou proposed openstack/gnocchi: carbonara: Timeserie.aggregate https://review.openstack.org/363492 | 13:05 |
| openstackgerrit | Julien Danjou proposed openstack/gnocchi: storage: re-store/write read-only splits https://review.openstack.org/362892 | 13:05 |
| openstackgerrit | Julien Danjou proposed openstack/gnocchi: carbonara: optimize uncompressed serialization https://review.openstack.org/358314 | 13:05 |
| openstackgerrit | Julien Danjou proposed openstack/gnocchi: carbonara: compress non padded timeseries https://review.openstack.org/358313 | 13:05 |
| openstackgerrit | Julien Danjou proposed openstack/gnocchi: carbonara: use calcsize rather than hard coded value https://review.openstack.org/358312 | 13:05 |
| *** fguillot has joined #openstack-telemetry | 13:11 | |
| *** liuhanxi has quit IRC | 13:23 | |
| *** paw has quit IRC | 13:28 | |
| *** ljxiash has joined #openstack-telemetry | 13:29 | |
| gordc | jd__: i had no idea you could just define a granularity and no timespan/points. | 13:31 |
| gordc | i'll start testing out the branch now. | 13:31 |
| *** eglynn has quit IRC | 13:31 | |
| *** ljxiash has quit IRC | 13:34 | |
| *** liuhanxi has joined #openstack-telemetry | 13:47 | |
| *** liuhanxi has quit IRC | 13:47 | |
| *** liuhanxi has joined #openstack-telemetry | 13:47 | |
| *** paw has joined #openstack-telemetry | 13:51 | |
| *** ljxiash has joined #openstack-telemetry | 13:53 | |
| *** c64cosmin_ has joined #openstack-telemetry | 13:57 | |
| *** links has quit IRC | 14:00 | |
| *** pradk has joined #openstack-telemetry | 14:02 | |
| *** c64cosmin_ has quit IRC | 14:02 | |
| *** zhurong has quit IRC | 14:03 | |
| *** zhurong has joined #openstack-telemetry | 14:03 | |
| *** cdent_ has joined #openstack-telemetry | 14:05 | |
| *** cdent has quit IRC | 14:06 | |
| *** cdent_ has quit IRC | 14:07 | |
| *** cdent has joined #openstack-telemetry | 14:07 | |
| *** liamji has joined #openstack-telemetry | 14:21 | |
| *** rbak has joined #openstack-telemetry | 14:23 | |
| *** _mbelanger has joined #openstack-telemetry | 14:25 | |
| *** xiaozhuangqing has quit IRC | 14:25 | |
| *** xiaozhuangqing has joined #openstack-telemetry | 14:26 | |
| *** donghao has joined #openstack-telemetry | 14:29 | |
| *** nicodemus_ has joined #openstack-telemetry | 14:32 | |
| *** KrishR has joined #openstack-telemetry | 14:45 | |
| openstackgerrit | Mehdi Abaakouk (sileht) proposed openstack/gnocchi: CRUD type attributes https://review.openstack.org/363737 | 15:03 |
| openstackgerrit | Mehdi Abaakouk (sileht) proposed openstack/gnocchi: CRUD resource type attributes (add) https://review.openstack.org/363737 | 15:09 |
| *** pshedimb has joined #openstack-telemetry | 15:10 | |
| *** KrishR1 has joined #openstack-telemetry | 15:10 | |
| openstackgerrit | Mehdi Abaakouk (sileht) proposed openstack/gnocchi: CRUD resource type attributes (add) https://review.openstack.org/363737 | 15:10 |
| *** zhurong has quit IRC | 15:11 | |
| *** KrishR has quit IRC | 15:11 | |
| *** KrishR1 has quit IRC | 15:12 | |
| *** diogogmt has joined #openstack-telemetry | 15:14 | |
| *** KrishR has joined #openstack-telemetry | 15:16 | |
| *** pshedimb has quit IRC | 15:19 | |
| *** pshedimb has joined #openstack-telemetry | 15:21 | |
| *** pcaruana has quit IRC | 15:22 | |
| *** rcernin has quit IRC | 15:22 | |
| *** liuhanxi has quit IRC | 15:22 | |
| openstackgerrit | Mehdi Abaakouk (sileht) proposed openstack/gnocchi: CRUD resource type attributes (add) https://review.openstack.org/363737 | 15:24 |
| *** ljxiash has quit IRC | 15:29 | |
| *** tesseract- has quit IRC | 15:36 | |
| *** nicodemus_ has quit IRC | 15:40 | |
| *** nicodemus_ has joined #openstack-telemetry | 15:40 | |
| *** _nadya_ has quit IRC | 15:46 | |
| *** rcernin has joined #openstack-telemetry | 15:54 | |
| *** pshedimb has quit IRC | 15:56 | |
| *** sheel has joined #openstack-telemetry | 15:57 | |
| *** pshedimb has joined #openstack-telemetry | 15:57 | |
| gordc | jd__: i think the compression stuff isn't getting triggered. i have the exact same size in disk space. | 15:58 |
| openstackgerrit | Mehdi Abaakouk (sileht) proposed openstack/gnocchi: CRUD resource type attributes (add) https://review.openstack.org/363737 | 15:58 |
| openstackgerrit | Mehdi Abaakouk (sileht) proposed openstack/gnocchi: CRUD resource type attributes (remove) https://review.openstack.org/363770 | 15:58 |
| jd__ | gordc: I have unit test that prove that work :( | 15:58 |
| jd__ | gordc: any hint? | 15:59 |
| jd__ | gordc: are you time serie big enough, do they have enough splits? | 15:59 |
| jd__ | because if they are all of 1 split it's not goint to change anything | 15:59 |
| gordc | no idea, taking a look right now. | 15:59 |
| EmilienM | so "it worked in unit tests" now replaces "it worked in devstack" ? | 15:59 |
| EmilienM | :-) /me runs | 15:59 |
| jd__ | EmilienM: yes :D | 15:59 |
| gordc | i have one series that is up to 144000opints | 15:59 |
| jd__ | it's better it's reproducible | 16:00 |
| gordc | and i'm putting in 7200 points | 16:00 |
| gordc | i'm assuming there should be at least one split | 16:00 |
| jd__ | gordc: 7200 consecutively aggregated points? | 16:00 |
| jd__ | yep | 16:00 |
| gordc | yeah | 16:00 |
| jd__ | but not sure 2 splits is going to make a huge diff | 16:00 |
| gordc | probably not. but it should be something? looknig at ceph logs, it's the exact same footprint over entire set | 16:01 |
| jd__ | are your data values completely random? | 16:02 |
| jd__ | IIRC worst case is 8 bytes vs 9 bytes without compression so… | 16:02 |
| jd__ | if it's random… it might be avery small diff | 16:02 |
| gordc | not really. i'm actually just posting 720 real points and then offsetting the same 720 points over and over | 16:03 |
| jd__ | if you can read one of the old split directly you'll be fixed though :) | 16:04 |
| *** ddaskal has joined #openstack-telemetry | 16:05 | |
| gordc | looking now | 16:06 |
| *** ildikov has quit IRC | 16:06 | |
| openstackgerrit | Mehdi Abaakouk (sileht) proposed openstack/gnocchi: CRUD resource type attributes (add) https://review.openstack.org/363737 | 16:10 |
| openstackgerrit | Mehdi Abaakouk (sileht) proposed openstack/gnocchi: CRUD resource type attributes (remove) https://review.openstack.org/363770 | 16:10 |
| *** links has joined #openstack-telemetry | 16:11 | |
| openstackgerrit | Mehdi Abaakouk (sileht) proposed openstack/gnocchi: CRUD resource type attributes (add) https://review.openstack.org/363737 | 16:16 |
| openstackgerrit | Mehdi Abaakouk (sileht) proposed openstack/gnocchi: CRUD resource type attributes (remove) https://review.openstack.org/363770 | 16:16 |
| gordc | jd__: it has compressed format... i still don't know how it's the exact same size in ceph though. | 16:17 |
| gordc | i'll keep looking. | 16:17 |
| jd__ | ah, that's interesting lol | 16:18 |
| gordc | from performance pov, at quick glance it looks like it's running at roughly the same speed so doesn't seem like compression is negative impact. | 16:18 |
| gordc | i didn't really dig into it though. | 16:18 |
| openstackgerrit | Mehdi Abaakouk (sileht) proposed openstack/gnocchi: CRUD resource type attributes (add) https://review.openstack.org/363737 | 16:22 |
| openstackgerrit | Mehdi Abaakouk (sileht) proposed openstack/gnocchi: CRUD resource type attributes (remove) https://review.openstack.org/363770 | 16:22 |
| openstackgerrit | Mehdi Abaakouk (sileht) proposed openstack/gnocchi: CRUD resource type attributes (add) https://review.openstack.org/363737 | 16:22 |
| openstackgerrit | Mehdi Abaakouk (sileht) proposed openstack/gnocchi: CRUD resource type attributes (remove) https://review.openstack.org/363770 | 16:22 |
| *** donghao has quit IRC | 16:27 | |
| *** zenpac has joined #openstack-telemetry | 16:28 | |
| zenpac | I'm on Mikata and we've created a dispatcher to send events over to a monitoring system. It was working on older versions of OopenStack but now its not sending data. | 16:29 |
| openstackgerrit | Milan Potdar proposed openstack/ceilometer: Nit: Aligning the content. https://review.openstack.org/363794 | 16:34 |
| gordc | zenpac: um... i don't recall anything changing. | 16:36 |
| gordc | zenpac: we added batch notificaitons | 16:36 |
| gordc | zenpac: can't really help other than that since it's not opensource :P | 16:36 |
| zenpac | We're trying to send messages via AMQP.. What driver in neutron.conf should i list? | 16:37 |
| *** pshedimb has quit IRC | 16:37 | |
| zenpac | ceilometer is proprietary? | 16:37 |
| gordc | zenpac: there already is a notifier publisher. | 16:37 |
| *** pshedimb has joined #openstack-telemetry | 16:38 | |
| gordc | zenpac: no, but i have no idea what this dispatcher you created is | 16:38 |
| gordc | the notifier publisher just dumps msg to queue for something to pick up | 16:38 |
| zenpac | I got a warning from neutron-server saying that "Option "notification_driver" from group "DEFAULT" is deprecated. Use option "driver" from group "oslo_messaging_notifications" | 16:39 |
| zenpac | But its not set, not sure what it should be. | 16:39 |
| gordc | http://docs.openstack.org/developer/ceilometer/install/manual.html#neutron | 16:40 |
| zenpac | https://github.com/zenoss/ceilometer_zenoss | 16:42 |
| zenpac | gordc: Can you see that repo? | 16:43 |
| *** _nadya_ has joined #openstack-telemetry | 16:43 | |
| gordc | yes | 16:43 |
| *** fguillot_ has joined #openstack-telemetry | 16:44 | |
| *** _nadya_ has quit IRC | 16:44 | |
| *** _nadya_ has joined #openstack-telemetry | 16:52 | |
| gordc | jd__: did you try compress with just lz4 and no double delta encoding? | 16:59 |
| jd__ | gordc: you mean the current serialization formaet? | 17:00 |
| gordc | right | 17:01 |
| jd__ | yes I did | 17:01 |
| jd__ | why? | 17:01 |
| *** yprokule has quit IRC | 17:01 | |
| gordc | the compress object i have is 32400B and there should only be 2900 points in it. | 17:01 |
| jd__ | that sounds weird | 17:03 |
| jd__ | is this the actual file size or what Ceph reports? | 17:03 |
| gordc | size when i get file from ceph to local | 17:03 |
| gordc | 32400 is perfectly 3600points (9B each) | 17:04 |
| jd__ | so it's not compressed? | 17:04 |
| jd__ | it's 2900 points with padding I imagine | 17:04 |
| jd__ | the compressed format has 'c' has the first byte of the file if you want to check | 17:05 |
| jd__ | s/has/as/ | 17:05 |
| gordc | right. i think it should be 2900 because the uncompress obj is 2880 points long | 17:05 |
| gordc | yeah, my compressed object starts with 'c' | 17:05 |
| jd__ | and it's 32400 bytes long | 17:05 |
| jd__ | ? | 17:05 |
| gordc | and i'm sending 5780 points so i assume the remaining 2900 is in the compress obj | 17:05 |
| gordc | yeah, compressed obj is 32400B | 17:06 |
| gordc | uncmpressed is 25920B | 17:06 |
| jd__ | I think I'm lost | 17:06 |
| jd__ | 32400 for a compressed object sounds impossible | 17:06 |
| gordc | agreed | 17:06 |
| jd__ | send me the file I'll take a look | 17:07 |
| gordc | sure | 17:07 |
| sileht | Is the gnocchi gate broken by the pymysql thing ? | 17:09 |
| jd__ | yes sileht | 17:09 |
| gordc | jd__: sent you the file. | 17:10 |
| marcin1234 | guys do you know how to speed up a creation of a "resource" in Gnocchi? | 17:10 |
| jd__ | marcin1234: how slow it is? | 17:11 |
| marcin1234 | I have interval of 5 minutes on computes, it takes up to 20 minutes | 17:11 |
| marcin1234 | this leads to messages that I am collecting measures without predecesor | 17:12 |
| jd__ | gordc: lol this is not even a valid data | 17:14 |
| jd__ | gordc: I think we miss something here | 17:14 |
| gordc | going to lunch. marcin1234 the actual resource creation should take very little time. less than a second. something is wrong. | 17:14 |
| jd__ | there might be a subtility and a different between real Ceph and our fake code | 17:14 |
| gordc | yeah, that's what i have. all i see is the 'c' so i assume it's compressed.lol | 17:15 |
| jd__ | marcin1234: so it seems the problem is between Nova and Ceilometer, not Gnocchi | 17:15 |
| jd__ | gordc: yeah, it looks like it has been rewritten but the rest of the data stayed | 17:15 |
| jd__ | gordc: like… maybe write() with offset= does not truncate… sileht? | 17:16 |
| gordc | truncate only writes from offset to however long your data is | 17:17 |
| gordc | s/truncate/write/ | 17:17 |
| gordc | so it leavse stuff... | 17:17 |
| gordc | bbl | 17:17 |
| marcin1234 | ok | 17:17 |
| jd__ | gordc: if that's the case, that' the source of the problem – our fake driver does not do that | 17:17 |
| jd__ | I would just like to be sure it's the case as it's a difference with a standard file system | 17:18 |
| jd__ | which is ok but surprising somehow - maybe I don't know | 17:18 |
| *** fguillot_ has quit IRC | 17:18 | |
| jd__ | it's not a file system | 17:18 |
| * jd__ looks at sileht | 17:18 | |
| *** nicodemus_ has quit IRC | 17:20 | |
| *** nicodemus_ has joined #openstack-telemetry | 17:21 | |
| openstackgerrit | Milan Potdar proposed openstack/ceilometer: Nit: Aligning the content. https://review.openstack.org/363794 | 17:22 |
| *** _nadya_ has quit IRC | 17:26 | |
| sileht | jd__, indead that doesn't truncate | 17:34 |
| *** sergio_nubeliu has joined #openstack-telemetry | 17:38 | |
| nicodemus_ | hello | 17:38 |
| *** pcaruana has joined #openstack-telemetry | 17:39 | |
| nicodemus_ | using current python-gnocchiclient, is it possible to list all current resources? That is, resources that do not have an ended_at date? | 17:39 |
| *** pshedimb has quit IRC | 17:39 | |
| marcin1234 | jd: so I measured exactly by looking at time stamps, from the time my VM was created (nova.instances) to time resource with corresponding original_resource_id was created (gnocchi.resource), it has 15 minutes14 seconds | 17:44 |
| marcin1234 | delay | 17:44 |
| *** rwsu has quit IRC | 17:44 | |
| sileht | nicodemus_, you can use the keyword null as value I think, that's works on server side, perhaps the client does the same | 17:54 |
| *** _nadya_ has joined #openstack-telemetry | 17:55 | |
| nicodemus_ | sileht, would that be with the --sort option from the client? Or doing a resource search for ended_at not null? | 17:57 |
| sileht | the search | 17:57 |
| nicodemus_ | sileht, got it. Thanks! | 17:58 |
| sileht | nicodemus_, I don't remenber how but yiu can sort with on top or on bottom by passing a special sort direction | 17:59 |
| sileht | *sort the null values* | 18:00 |
| jd__ | sileht: haha, ok | 18:00 |
| jd__ | sileht: any trick to truncate? | 18:00 |
| jd__ | sileht: have to remove the object? | 18:00 |
| sileht | jd__, ioctx.trunc(name, len) or writeop.truncate(len) | 18:01 |
| *** liamji has quit IRC | 18:02 | |
| *** diogogmt has quit IRC | 18:02 | |
| *** thumpba has joined #openstack-telemetry | 18:05 | |
| *** Kevin_Zheng has quit IRC | 18:06 | |
| *** larainema has quit IRC | 18:08 | |
| sileht | nicodemus_, for example: sort=revision_end:asc-nullslast | 18:10 |
| jd__ | sileht: awesome, thanks | 18:11 |
| jd__ | i'll fix that tomorrow | 18:11 |
| *** Kevin_Zheng has joined #openstack-telemetry | 18:11 | |
| *** larainema has joined #openstack-telemetry | 18:11 | |
| zenpac | I get this type of traceback: http://paste.debian.net/798638/ when we use our dispatcher. | 18:16 |
| gordc | marcin1234: is the data coming from ceilometer? do you have a massive backlog on metering.sample topic of MQ? | 18:17 |
| gordc | zenpac: sorry, we don't actually use eventlet in ceilometer. we probably dropped it in Liberty? | 18:18 |
| gordc | i would suggest you try modifying/contributing to notifier publisher, you shouldn't need collector/dispatcher if you are just publishing to rabbit. | 18:18 |
| marcin1234 | I do not have backlog on metering.sample | 18:19 |
| marcin1234 | what I do have is in ceilometer log "dropping sample with no predecessor" | 18:19 |
| marcin1234 | during that 15+ minutes of delay | 18:20 |
| nicodemus_ | marcin1234, in my case I have configured 1200 seconds for each source in /etc/ceilometer/pipeline.yaml. Sometimes I launch an instance and, since it's going to take up to 20 minutes for the next ceilometer-agent-compute to post to gnocchi... the resource takes a while to actually appear | 18:22 |
| nicodemus_ | could that be the case? | 18:22 |
| *** shardy is now known as shardy_afk | 18:24 | |
| gordc | marcin1234: that log is unrelated. it will happen on new instances and if you restart your notification agent since our transformations are done in memory | 18:24 |
| *** shardy_afk has quit IRC | 18:25 | |
| gordc | nicodemus_: is there a reason you poll every 20min? | 18:25 |
| gordc | too much load on Nova api? | 18:25 |
| nicodemus_ | gordc, that 20min is for billing purposes and is using a little old gnocchi deploy, a more aggressive polling interval resulted in too many measures for metricd to cope with | 18:26 |
| nicodemus_ | using current code I'm polling every 5 mins | 18:27 |
| *** _nadya_ has quit IRC | 18:28 | |
| gordc | nicodemus_: kk, if you have free envirnoment i would actually try https://review.openstack.org/#/c/275741/ | 18:29 |
| gordc | in theory, it improves the batching (if you dispatch straight from notification agent rather than use collector | 18:29 |
| gordc | it doesn't improve metricd though. that is only done through v3 | 18:30 |
| nicodemus_ | gordc, that looks promising... I intend to move from collector shortly | 18:32 |
| nicodemus_ | I'll give it a try | 18:32 |
| gordc | cool cool | 18:33 |
| marcin1234 | nicodemus: what does it meant it takes a while? it take 21 minutes to appear in gnocchi table or more like an hour? | 18:33 |
| nicodemus_ | marcin1234, maybe it appears one minute after the instance was created, maybe it appears 19 minutes after | 18:34 |
| marcin1234 | I need to speed up resource creation on gnocchi, somehow | 18:34 |
| nicodemus_ | never took more than 20 mins though | 18:35 |
| *** _nadya_ has joined #openstack-telemetry | 18:35 | |
| gordc | marcin1234: have you tried creating a dummy resource? like just use gnocchi resource create abcxyz' | 18:35 |
| gordc | that command takes 15mins? | 18:35 |
| marcin1234 | gordc: no that works immediately | 18:36 |
| *** pshedimb has joined #openstack-telemetry | 18:36 | |
| gordc | marcin1234: oh. then it's probably either your pipeline period is very long. | 18:38 |
| gordc | marcin1234: or notification agent is backed up like crazy | 18:38 |
| marcin1234 | gordc: it is 5 minutes | 18:38 |
| gordc | marcin1234: if your metering.sample queue is empty and that resource create command is instant than it's something up with ceilometer. | 18:38 |
| marcin1234 | I can see those "dropping sample with no predecessor" because there is data coming from ceompute, through collector, so I am collecting measures, for resource that does not exist yet | 18:39 |
| gordc | if you do rabbitmqctl list_queues are there queues that are very large | 18:39 |
| marcin1234 | gordc: how can it be with ceilometer? Ceilometer collector is pushing the data to Gnocchi | 18:39 |
| marcin1234 | it is just Gnocchi that is not creating resource from some reason | 18:40 |
| gordc | the "dropping sample with no predecessor" should not be on collector end. it's a notification agent message relating to transfomrations | 18:40 |
| EmilienM | gordc: you know rabbit right? ;) | 18:40 |
| gordc | it has nothing to do with gnocchi. | 18:40 |
| gordc | EmilienM: i eat rabbit? | 18:40 |
| gordc | so do you. you're french :P | 18:40 |
| EmilienM | gordc: I remember your awesome talk about rabbitmq in austin | 18:41 |
| gordc | hahahah! | 18:41 |
| marcin1234 | queues are all "0" | 18:41 |
| gordc | oh yeah, google it | 18:41 |
| marcin1234 | gordc: so based on what Gnocchi decides to create resource in its mysql table? | 18:41 |
| gordc | marcin1234: gnocchi creates a resource if you tell it to create a resource. | 18:42 |
| gordc | marcin1234: from ceilometer+gnocchi pov, ceilometer will try to send measures to gnocchi, and if gnocchi returns saying, 'i don't know this resource', ceilometer will tell it to create resource and send measrues again | 18:43 |
| gordc | this is the message you are getting: https://github.com/openstack/ceilometer/blob/67739c2278dba6197e141715faa3d6eb047170a7/ceilometer/transformer/conversions.py#L98 | 18:43 |
| marcin1234 | gordc: ok, in case when I build new VM, when Gnocchi is "asked" to create resource? | 18:43 |
| gordc | it has no relation to storage or gnocchi. | 18:43 |
| *** _nadya_ has quit IRC | 18:44 | |
| nicodemus_ | marcin1234, perhaps you could try to do a packet capture on the gnocchi API, and see exactly when the POST to create the resource arrives to gnocchi | 18:44 |
| gordc | marcin1234: it depends. basically when ceilometer gets a datapoint eitehr from notification or polling, it will end up at collector and then it interacts with gnocchi | 18:44 |
| marcin1234 | gordc: got it, so I gues my ceilometer does not ask to create resource, or it does but after long ass time | 18:44 |
| nicodemus_ | if when the POST arrives the resource appears on gnocchi, then gnocchi is not taking time to create it | 18:45 |
| gordc | when you say gnocchi doesn't have resource, do you mean it doesn't have any measures or it doesn't even know about resource at all? | 18:45 |
| gordc | if the latter, than it means ceilometer is getting no data at all | 18:46 |
| gordc | considering you have 0 items in your queues | 18:46 |
| gordc | that basically means ceilometer is idle | 18:46 |
| marcin1234 | gordc: it does not have resource in gnocchi.resource with original_resource_id from my nova uuid | 18:46 |
| *** dduvall has quit IRC | 18:46 | |
| marcin1234 | gordc: ceilometer is getting data, it is just those queses are very quickly consumed | 18:47 |
| *** diogogmt has joined #openstack-telemetry | 18:48 | |
| marcin1234 | I can see on graph in metering.sample | 18:48 |
| gordc | marcin1234: if it's not in ceilometer it's gotta be in gnocchi. and if it's not in either, than you should have errors everywhere. | 18:48 |
| marcin1234 | I can see measure being collected from compute, going to rabbit, being consumed by ceilometer collector | 18:48 |
| *** thumpba has quit IRC | 18:49 | |
| marcin1234 | gordc: there are no errors | 18:49 |
| *** thumpba has joined #openstack-telemetry | 18:49 | |
| gordc | marcin1234: maybe change your dispatcher from gnocchi to something else temporarily and validate the data is actually getting that far? | 18:50 |
| marcin1234 | gordc: okay I will test, and tcpdump this | 18:50 |
| marcin1234 | I just did another test, and again 22 minutes.... | 18:50 |
| gordc | i don't know. from data flow pov, your data appears to be straight up gone. i can't explain this 15min delay, with no backlog and no errors | 18:50 |
| gordc | marcin1234: alternatively, just add logs to gnocchi dispatcher in ceilometer. and see how when it gets data | 18:52 |
| gordc | add log here: https://github.com/openstack/ceilometer/blob/master/ceilometer/dispatcher/gnocchi.py#L326 | 18:53 |
| gordc | i guess another check is how long is it between instance create and when you see 'dropping sample...' log in notification agent? | 18:55 |
| *** sheel has quit IRC | 18:56 | |
| *** diogogmt has quit IRC | 18:57 | |
| *** pshedimb has quit IRC | 18:59 | |
| *** ildikov has joined #openstack-telemetry | 19:01 | |
| nicodemus_ | I have a question regarding ceilometer-agent-notification: I'm using a rabbitmq vhost for each service (nova, neutron, etc). If I want to use https://review.openstack.org/#/c/217017/ I guess I'd need to use a ceilometer.conf with the vhost & credentials for Nova, since it needs to listen on the notifications.info queue, is that correct? | 19:05 |
| gordc | nicodemus_: how are you doing this currently? you define multiple urls to listen to. https://github.com/openstack/ceilometer/blob/master/ceilometer/notification.py#L64-L74 | 19:08 |
| nicodemus_ | gordc, cool. I didn't know that configuration option. I'll give it a try, thanks! | 19:10 |
| *** links has quit IRC | 19:13 | |
| marcin1234 | gordc: dropping sample shows up 2-3 minutes after VM is created | 19:22 |
| marcin1234 | what does not make sense, is once resource is created, it works great with no delay | 19:23 |
| gordc | marcin1234: 2-3minutes seems pretty long still. i would add long to dispatcher. | 19:26 |
| gordc | it doesn't make any sense there's a 20 min delay between notification agent and collector. | 19:27 |
| gordc | only other thing i can think of is you have batching enabled and the timeout is extremely high. | 19:27 |
| gordc | s/long/log | 19:32 |
| *** _nadya_ has joined #openstack-telemetry | 19:34 | |
| marcin1234 | ok | 19:35 |
| marcin1234 | thx gordc: I apprecite your insight | 19:35 |
| gordc | np. just for reference, the data flow is polling agent -> notification agent -> collector -> gnocchi. (assuming you have liberty+ | 19:38 |
| gordc | i'd add logging in those services to see where the delay is. your empty queues are very curiuos though. they should be exploding if it's a ceilometer issue. | 19:39 |
| zenpac | gordc: Does that mean that our use of eventlet in our dispatcher is the problem? | 19:44 |
| zenpac | or part or it? | 19:45 |
| gordc | zenpac: oh, i didn't really dive into your code. sorry. was just pointing out we don't use it so i don't know how they interact. | 19:50 |
| zenpac | ok.. Thanks. | 19:50 |
| openstackgerrit | Merged openstack/ceilometer: Nit: Aligning the content. https://review.openstack.org/363794 | 19:53 |
| *** KrishR has quit IRC | 19:56 | |
| *** KrishR has joined #openstack-telemetry | 19:59 | |
| *** _nadya_ has quit IRC | 20:03 | |
| paw | gordc: Hi there, sorry for the pestering, for the gerrit review on the IP version ( https://review.openstack.org/363737), I was wondering if you had any additional comments? thanks in advance! | 20:06 |
| gordc | paw: is that the right patch? | 20:09 |
| gordc | nm. found it. | 20:09 |
| marcin1234 | gordc: you are on the money: VM created: 19:49:33, gnocchi.py#L326 executed at 20:05:17, matching with resource created at date | 20:11 |
| gordc | paw: done. thanks for following up | 20:11 |
| paw | gordc: thanks for the review! | 20:11 |
| *** kelv has joined #openstack-telemetry | 20:12 | |
| gordc | marcin1234: what's the time diff between dispatcher log and the 'dropping sample...' log in notification agent. | 20:12 |
| gordc | if there's a massive gap, then something is up with MQ | 20:13 |
| *** pshedimb has joined #openstack-telemetry | 20:14 | |
| *** diogogmt has joined #openstack-telemetry | 20:14 | |
| marcin1234 | so there is a big gap. 15 minutes | 20:18 |
| *** pshedimb has quit IRC | 20:18 | |
| marcin1234 | so here are all timings: VM created: 19:49:33, first "Dropping sample" 19:50:18, gnocchi.py#L326 executed at 20:05:17, | 20:18 |
| marcin1234 | gordc: yes I am on mitaka | 20:20 |
| *** paw has quit IRC | 20:22 | |
| gordc | i have no idea why you have that delay tbh. literally the only step between dropping sample is to publish sample to queue, and collector picks it up and you get that L326 log | 20:22 |
| gordc | my only other suggestion is to turn off collector and see how long it takes you to see it on queue. | 20:22 |
| gordc | there is no reason that there's a delay between notification agent and collector when polling agent -> notification agent uses the same service. | 20:23 |
| marcin1234 | gordc: which queue we are talking about metering.sample or notifications.info ? | 20:23 |
| gordc | metering.sample | 20:23 |
| marcin1234 | gordc: what is default "batch_timeout" for collector? I am thinking this has something to do | 20:25 |
| *** cdent has quit IRC | 20:27 | |
| gordc | indefinite | 20:27 |
| gordc | marcin1234: but batch_size is 1. so it should trigger right away | 20:28 |
| gordc | unless you changed batch_size | 20:28 |
| marcin1234 | changed it to 1 | 20:29 |
| marcin1234 | from unset | 20:29 |
| gordc | it should default to 1 if you don't have it set | 20:32 |
| openstackgerrit | Merged openstack/ceilometermiddleware: Add release note for background thread notifier https://review.openstack.org/363168 | 20:44 |
| *** _mbelanger has quit IRC | 20:56 | |
| *** ddaskal_ has joined #openstack-telemetry | 20:56 | |
| *** ddaskal__ has joined #openstack-telemetry | 20:57 | |
| *** pshedimb has joined #openstack-telemetry | 20:58 | |
| *** ddaskal has quit IRC | 20:59 | |
| *** rwsu has joined #openstack-telemetry | 21:00 | |
| *** ddaskal_ has quit IRC | 21:01 | |
| *** sergio_nubeliu has quit IRC | 21:03 | |
| *** paw has joined #openstack-telemetry | 21:08 | |
| marcin1234 | gordc: what process is consuming notifications.info and publishing them to metering.sample? It looks like it is "agent-notification" correct? | 21:20 |
| gordc | right | 21:20 |
| marcin1234 | w | 21:21 |
| marcin1234 | sorry, hit enter by an accident | 21:22 |
| *** pshedimb has quit IRC | 21:22 | |
| *** pshedimb has joined #openstack-telemetry | 21:24 | |
| *** fguillot_ has joined #openstack-telemetry | 21:27 | |
| *** ddaskal has joined #openstack-telemetry | 21:27 | |
| *** ddaskal__ has quit IRC | 21:30 | |
| marcin1234 | gordc: so is it possible that agent-notification is consuming this message, and it takes 15-20 minutes to publish it to metering.sample ? | 21:30 |
| gordc | nope. especially considering the 'dropping sample...' message is one of the last steps in processing. | 21:33 |
| gordc | the other step after is to push onto queue. | 21:33 |
| *** fguillot_ has quit IRC | 21:33 | |
| gordc | https://github.com/openstack/ceilometer/blob/master/ceilometer/publisher/messaging.py#L128 this is actually last step | 21:36 |
| *** thorst has quit IRC | 21:38 | |
| *** rwsu has quit IRC | 21:38 | |
| *** rwsu has joined #openstack-telemetry | 21:39 | |
| *** nicodemus_ has quit IRC | 21:40 | |
| *** thumpba has quit IRC | 21:41 | |
| marcin1234 | gordc: this is my understanding how it works: 1. Vm is created 2. ceilometer.compute.pollsters.cpu measures in my example cpu utilization and publishes it to RMQ notifications.info 3. notification-agent consumes it and publishes it to metering.sample 4. ceilometer-collector consumes it and dispatches it to Gnocchi | 21:47 |
| *** diogogmt has quit IRC | 21:48 | |
| gordc | correct. | 21:59 |
| gordc | nova will also send notifications (if enabled) and will go to step 3 | 22:00 |
| *** cdent has joined #openstack-telemetry | 22:00 | |
| gordc | http://docs.openstack.org/developer/ceilometer/architecture.html#high-level-architecture | 22:01 |
| *** cdent has quit IRC | 22:05 | |
| *** fguillot has quit IRC | 22:07 | |
| *** fguillot has joined #openstack-telemetry | 22:09 | |
| *** adriant has joined #openstack-telemetry | 22:11 | |
| *** gordc has quit IRC | 22:14 | |
| *** KrishR has quit IRC | 22:42 | |
| *** zhurong has joined #openstack-telemetry | 22:47 | |
| *** zhurong has quit IRC | 22:52 | |
| *** rbak has quit IRC | 22:52 | |
| *** pradk has quit IRC | 23:03 | |
| *** ddaskal has quit IRC | 23:05 | |
| *** ddaskal has joined #openstack-telemetry | 23:06 | |
| *** ddaskal has quit IRC | 23:10 | |
| *** pshedimb has quit IRC | 23:11 | |
| *** cdent has joined #openstack-telemetry | 23:22 | |
| *** cdent has quit IRC | 23:29 | |
| *** fguillot has quit IRC | 23:31 | |
| openstackgerrit | Merged openstack/ceilometer: Config logABug feature for Ceilometer api-ref https://review.openstack.org/363430 | 23:43 |
| *** diogogmt has joined #openstack-telemetry | 23:46 | |
| *** fguillot has joined #openstack-telemetry | 23:48 | |
| *** donghao has joined #openstack-telemetry | 23:54 | |
| marcin1234 | gordc: so... I found the solution, it works properly if I have only 1 notification-agent running on one controller, and it will only create a resource, during second time it collects measure from compute | 23:56 |
| *** rwsu has quit IRC | 23:59 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!