| *** edmondsw has joined #openstack-telemetry | 00:05 | |
| *** edmondsw has quit IRC | 00:10 | |
| *** AlexeyAbashkin has joined #openstack-telemetry | 00:13 | |
| *** AlexeyAbashkin has quit IRC | 00:17 | |
| *** itlinux has quit IRC | 00:26 | |
| *** edmondsw has joined #openstack-telemetry | 00:49 | |
| *** edmondsw has quit IRC | 00:52 | |
| *** edmondsw has joined #openstack-telemetry | 00:52 | |
| *** tovin07_ has joined #openstack-telemetry | 00:55 | |
| *** Guest18917 has quit IRC | 01:11 | |
| *** yassine has joined #openstack-telemetry | 01:11 | |
| *** yassine has quit IRC | 01:11 | |
| *** yassine has joined #openstack-telemetry | 01:12 | |
| *** AlexeyAbashkin has joined #openstack-telemetry | 01:12 | |
| *** yassine is now known as Guest25442 | 01:12 | |
| *** AlexeyAbashkin has quit IRC | 01:16 | |
| *** daidv has quit IRC | 01:18 | |
| *** daidv has joined #openstack-telemetry | 01:19 | |
| *** edmondsw has quit IRC | 01:20 | |
| *** edmondsw has joined #openstack-telemetry | 01:20 | |
| *** edmondsw has quit IRC | 01:25 | |
| *** dave-mccowan has joined #openstack-telemetry | 01:31 | |
| openstackgerrit | Merged openstack/ceilometer master: pluggable notification agent https://review.openstack.org/517715 | 01:33 |
|---|---|---|
| *** rwsu has joined #openstack-telemetry | 02:06 | |
| *** gmann_afk is now known as gmann | 02:25 | |
| *** itlinux has joined #openstack-telemetry | 02:38 | |
| *** edmondsw has joined #openstack-telemetry | 02:55 | |
| *** edmondsw has quit IRC | 03:00 | |
| *** dave-mccowan has quit IRC | 03:10 | |
| *** links has joined #openstack-telemetry | 03:19 | |
| *** harlowja has quit IRC | 03:22 | |
| openstackgerrit | Merged openstack/ceilometer master: queues per manager rather than per pipeline https://review.openstack.org/517734 | 03:57 |
| *** itlinux has quit IRC | 03:58 | |
| *** harlowja has joined #openstack-telemetry | 04:01 | |
| *** psachin has joined #openstack-telemetry | 04:11 | |
| *** itlinux has joined #openstack-telemetry | 04:12 | |
| openstackgerrit | Merged openstack/ceilometer master: always declare partitioning variables https://review.openstack.org/519401 | 04:12 |
| *** edmondsw has joined #openstack-telemetry | 04:44 | |
| *** itlinux has quit IRC | 04:44 | |
| *** gkadam has joined #openstack-telemetry | 04:46 | |
| *** itlinux has joined #openstack-telemetry | 04:47 | |
| *** edmondsw has quit IRC | 04:48 | |
| *** maestropandy has joined #openstack-telemetry | 05:03 | |
| *** edmondsw has joined #openstack-telemetry | 05:29 | |
| *** itlinux has quit IRC | 05:31 | |
| *** edmondsw has quit IRC | 05:33 | |
| *** pcaruana has joined #openstack-telemetry | 05:39 | |
| *** edmondsw has joined #openstack-telemetry | 05:48 | |
| *** harlowja has quit IRC | 05:52 | |
| *** edmondsw has quit IRC | 05:52 | |
| *** pcaruana has quit IRC | 06:13 | |
| *** maestropandy has quit IRC | 06:19 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/aodh master: Imported Translations from Zanata https://review.openstack.org/521722 | 06:21 |
| *** maestropandy has joined #openstack-telemetry | 06:23 | |
| *** edmondsw has joined #openstack-telemetry | 06:29 | |
| *** edmondsw has quit IRC | 06:33 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/ceilometer master: Imported Translations from Zanata https://review.openstack.org/521726 | 06:35 |
| *** tianqing has joined #openstack-telemetry | 06:40 | |
| tianqing | hello.. | 06:41 |
| tianqing | does gnocchi supports udp? | 06:41 |
| tianqing | then ceilometer notification agent -> udp - gnocchi -> gnocchi backend... | 06:41 |
| tianqing | I think we should add udp supports.... | 06:42 |
| tianqing | any one ideas? | 06:42 |
| *** edmondsw has joined #openstack-telemetry | 06:48 | |
| *** edmondsw has quit IRC | 06:53 | |
| *** lhx_ has joined #openstack-telemetry | 06:54 | |
| *** rcernin has quit IRC | 06:59 | |
| *** iranzo has joined #openstack-telemetry | 07:03 | |
| *** edmondsw has joined #openstack-telemetry | 07:08 | |
| tianqing | ???? | 07:10 |
| tianqing | any one? | 07:10 |
| *** spectr has joined #openstack-telemetry | 07:11 | |
| *** spectr has quit IRC | 07:12 | |
| *** edmondsw has quit IRC | 07:12 | |
| tianqing | any one? | 07:17 |
| *** gkadam has quit IRC | 07:20 | |
| *** edmondsw has joined #openstack-telemetry | 07:23 | |
| *** sanchitmalhotra has joined #openstack-telemetry | 07:27 | |
| *** pcaruana has joined #openstack-telemetry | 07:30 | |
| *** edmondsw has quit IRC | 07:33 | |
| *** edmondsw has joined #openstack-telemetry | 07:48 | |
| *** shardy has joined #openstack-telemetry | 07:49 | |
| *** maestropandy has quit IRC | 07:50 | |
| *** edmondsw has quit IRC | 07:52 | |
| *** yprokule has joined #openstack-telemetry | 08:07 | |
| *** edmondsw has joined #openstack-telemetry | 08:08 | |
| *** edmondsw has quit IRC | 08:13 | |
| *** hoonetorg has quit IRC | 08:18 | |
| *** tesseract has joined #openstack-telemetry | 08:23 | |
| tianqing | does gnocchi support udp to get measures from ceilometer? | 08:25 |
| tianqing | how? | 08:25 |
| *** edmondsw has joined #openstack-telemetry | 08:28 | |
| *** edmondsw has quit IRC | 08:33 | |
| *** hoonetorg has joined #openstack-telemetry | 08:34 | |
| jd_ | tianqing: everybody's sleeping you know at that time :) | 08:35 |
| tianqing | .... | 08:42 |
| tianqing | oh... | 08:42 |
| tianqing | hello jd... | 08:42 |
| tianqing | ceilometer sends measures to gnocchi use http... | 08:43 |
| tianqing | gnocchi has statd dameon.. | 08:43 |
| tianqing | but it must be in statd protocol... | 08:43 |
| tianqing | Does the ceilometer can direct send to statd daemon of gnocchi? | 08:43 |
| *** edmondsw has joined #openstack-telemetry | 08:49 | |
| *** maestropandy has joined #openstack-telemetry | 08:49 | |
| *** edmondsw has quit IRC | 08:53 | |
| *** gkadam has joined #openstack-telemetry | 09:01 | |
| *** maestropandy has quit IRC | 09:26 | |
| *** maestropandy has joined #openstack-telemetry | 09:27 | |
| *** zhangguoqing has joined #openstack-telemetry | 09:46 | |
| *** rcernin has joined #openstack-telemetry | 10:01 | |
| *** maestropandy has quit IRC | 10:04 | |
| *** daidv has quit IRC | 10:05 | |
| *** maestropandy has joined #openstack-telemetry | 10:16 | |
| *** tovin07_ has quit IRC | 10:23 | |
| *** gmann is now known as gmann_afk | 10:34 | |
| *** edmondsw has joined #openstack-telemetry | 10:48 | |
| *** maestropandy has quit IRC | 10:50 | |
| *** edmondsw has quit IRC | 10:52 | |
| *** maestropandy has joined #openstack-telemetry | 10:59 | |
| *** maestropandy1 has joined #openstack-telemetry | 11:03 | |
| *** maestropandy2 has joined #openstack-telemetry | 11:04 | |
| *** maestropandy has quit IRC | 11:05 | |
| *** tianqing has quit IRC | 11:06 | |
| *** maestropandy1 has quit IRC | 11:07 | |
| *** sanchitmalhotra has quit IRC | 11:08 | |
| *** edmondsw has joined #openstack-telemetry | 11:08 | |
| *** edmondsw has quit IRC | 11:12 | |
| *** gokhan has quit IRC | 11:15 | |
| *** AlexeyAbashkin has joined #openstack-telemetry | 11:24 | |
| *** edmondsw has joined #openstack-telemetry | 11:28 | |
| *** gokhan has joined #openstack-telemetry | 11:29 | |
| *** edmondsw has quit IRC | 11:32 | |
| *** edmondsw has joined #openstack-telemetry | 11:48 | |
| openstackgerrit | Julien Danjou proposed openstack/ceilometer master: fix ceilometer-compute invoke libvirt exception error https://review.openstack.org/517588 | 11:48 |
| *** edmondsw has quit IRC | 11:52 | |
| openstackgerrit | Merged openstack/aodh master: Replace jsonutils by ujson https://review.openstack.org/516212 | 12:20 |
| *** gordc has joined #openstack-telemetry | 12:38 | |
| *** edmondsw has joined #openstack-telemetry | 12:48 | |
| *** pradk has joined #openstack-telemetry | 12:53 | |
| openstackgerrit | gordon chung proposed openstack/ceilometer stable/pike: always declare partitioning variables https://review.openstack.org/521842 | 12:57 |
| *** lhx_ has quit IRC | 13:01 | |
| *** dave-mccowan has joined #openstack-telemetry | 13:11 | |
| *** zhangguoqing has quit IRC | 13:13 | |
| gordc | if anyone wants to tackle a (possible) threading issue: https://bugs.launchpad.net/ceilometer/+bug/1729865 | 13:14 |
| openstack | Launchpad bug 1729865 in Ceilometer "partitioning queues not cleared" [Critical,Triaged] - Assigned to gordon chung (chungg) | 13:14 |
| *** dave-mcc_ has joined #openstack-telemetry | 13:17 | |
| jd_ | this seems scary | 13:17 |
| *** dave-mccowan has quit IRC | 13:17 | |
| jd_ | gordc: any chance to simplify the spaghetti? | 13:17 |
| gordc | jd_: what's the spaghetti? | 13:17 |
| gordc | i should mention, all my debugging is on oslo.messaging side right now. | 13:18 |
| jd_ | all the code around this in Ceilometer makes me think of spaghetti code | 13:18 |
| jd_ | whaaat a bug in oslo.messaging | 13:18 |
| * jd_ stares at sileht | 13:18 | |
| gordc | (i think) | 13:18 |
| gordc | i can make the random gate failures disappear by dropping the batch_timeout down... but that's just hiding the issue. | 13:19 |
| sileht | gordc, the patch you send to oslo.messaging haven't fix the issue ? | 13:21 |
| gordc | sileht: no :( i hoped it would | 13:21 |
| gordc | i think it just ended up being a refactor.lol | 13:21 |
| *** shardy is now known as shardy_lunch | 13:22 | |
| gordc | i imagine it has to do with with the batch_timeout. i often see the 'idle' time ~= a multiple of batch_timeout. | 13:23 |
| sileht | oslo.messaging gates have 43 jobs only two passes pep8 and docs ... | 13:24 |
| sileht | gordc, how did you test you oslo.msg patch ? | 13:25 |
| gordc | lol. my test patch? yeah i never understood why my log messages would break oslo.messaging check. | 13:25 |
| *** maestropandy2 has quit IRC | 13:26 | |
| gordc | sileht: i basically added logging to where oslo.messaging submits messages to be dispatched (after 'consume' has batched messages from queue) | 13:27 |
| gordc | and then i added logging to when it's actually handled by dispatch code (dispatched to endpoints) | 13:27 |
| gordc | usually it gets submited to be dispatched and then dispatched relatively quickly. | 13:28 |
| gordc | but on random runs, it'll pause for 5-6s before dispatch code triggers... but nothing is happening/logged in those 6s (according to notification-agent). | 13:29 |
| *** zhangguoqing has joined #openstack-telemetry | 13:29 | |
| *** lhx_ has joined #openstack-telemetry | 13:30 | |
| *** rcernin has quit IRC | 13:30 | |
| gordc | the ordering is what confuses me to. it submits batchA to empty executor, waits, and then when it starts, the executor works on batchB, batchC, ...(batches just submitted) then batchA. | 13:37 |
| sileht | gordc, looks like the issue is how concurrent.futures schedule jobs, no ? | 13:41 |
| sileht | gordc, for whatever reason it decides to do batchA last even it submitted first | 13:42 |
| gordc | sileht: i think so? although we're using futurist in oslo.messaging. i tried switching to concurrent.futures but it did the same thing | 13:42 |
| sileht | gordc, futurist use futures underthehodd | 13:43 |
| gordc | sileht: yeah... if you look at log, i have WHHHHHHAT(executor queue size) as the log for when it gets submitted | 13:43 |
| gordc | but when batchB get submitted 6s later, it shows the queue is empty even though batchA hasn't triggered dispatch code yet | 13:44 |
| gordc | oh. i thought futurist was some custom thing we were doing. | 13:45 |
| *** shardy_lunch is now known as shardy | 13:45 | |
| jd_ | no it's just a layer above which adds a few nifty features | 13:45 |
| jd_ | I don't think there is any ordering promise done by concurrent.futures | 13:46 |
| gordc | well there is a queue... so i would hope a queue maintains order. | 13:49 |
| sileht | We use threads so we can't promise order too | 13:50 |
| jd_ | if you have multiple threads pulling from the queue, yeah | 13:53 |
| gordc | i should mention notification agent sets oslo.messaging to 1 thread. but it's really only 1 thread to guarantee it recovers from scenario where all notification agents go down. | 13:53 |
| gordc | so i guess in theory, we could change that as it's only single threaded to handle scenario where you didn't do HA properly or something catastrophic happened | 13:54 |
| sileht | gordc, ordering in onlyuseful for transformation, if you just want to do rate | 13:56 |
| sileht | gordc, we can now use Gnocchi to do this, and it doesn't have this issue if points are received within the backwindow | 13:56 |
| gordc | sileht: yeah. the pipeline partitioning stuff is really only for transform... but i don't think we can tell everyone to use gnocchi :( | 13:58 |
| sileht | preserving ordering is something to think end-to-end | 13:59 |
| sileht | the dump way is you consume samples with one worker/thread | 14:00 |
| sileht | the complicate way, you make the system aware of timestamps attached to samples | 14:00 |
| gordc | well we have dumb way right now. :) | 14:00 |
| jd_ | gordc: we can | 14:00 |
| jd_ | we already kinda do anyway :) | 14:01 |
| gordc | but multithreading isn't a big deal because all our metrics happen on specific cadence so there's always a gap between matching samples | 14:02 |
| gordc | the only time they are batched near each other is if the notification agent died. | 14:03 |
| gordc | still don't understand why we see what we see in gate, but we could make it multithreaded | 14:04 |
| sileht | gordc, I don't known if that the issue, but I found, in batch mode, we ack message one by one, and wait between 0.01 seconds to 1 seconds between each | 14:17 |
| sileht | here we loop to ack messages: https://github.com/openstack/oslo.messaging/blob/master/oslo_messaging/notify/listener.py#L203-L210 | 14:17 |
| sileht | ack are done like this: https://github.com/openstack/oslo.messaging/blob/master/oslo_messaging/_drivers/amqpdriver.py#L188 | 14:17 |
| sileht | then we block here: https://github.com/openstack/oslo.messaging/blob/master/oslo_messaging/_drivers/amqpdriver.py#L104 | 14:18 |
| sileht | because only the main thread is allowed to ack messages | 14:18 |
| sileht | now we have to way process() to be trigger here: https://github.com/openstack/oslo.messaging/blob/master/oslo_messaging/_drivers/amqpdriver.py#L286-L303 | 14:19 |
| sileht | way/wait | 14:19 |
| gordc | sileht: yeah i noticed that too. rabbitmq would say i have items on queue but notification agent would be idle because i think the ack never used rabbitmq | 14:19 |
| sileht | gordc, by default list_queues don't show messages taken by oslo.messaging but don't yet ack | 14:20 |
| sileht | gordc, I have make a comment in one of your patch to show up this information | 14:20 |
| gordc | in a patch that was merged? | 14:21 |
| gordc | never realised only main thread could ack. all the threads created made me confused.lol | 14:22 |
| *** zhangguoqing has quit IRC | 14:24 | |
| sileht | gordc, neither kombu or py-amqp are thread-safe | 14:26 |
| sileht | gordc, so oslo.messaging have to deal with that, but because the lib have been built on top of eventlet, and people working with eventlet have no idea of "concurrency" mean | 14:27 |
| sileht | gordc, we just add tape to make oslo.messaging thread-safe | 14:27 |
| sileht | infortunatly, this have been done with queue and lock | 14:28 |
| sileht | making all threads competing each others | 14:28 |
| sileht | the funny thing is that ack message are small enough, that's submitting them in incorrect order work without ssl enabled | 14:29 |
| sileht | gordc, if I could rewrite oslo.messaging I will just creates a pool of connection and attach a thread to a pool | 14:30 |
| sileht | gordc, like sqlalchemy does ... | 14:30 |
| gordc | just create olso.messaging.good. | 14:31 |
| gordc | let all the eventlet projects use this hacky one | 14:31 |
| sileht | honestly I would just use one of the rabbitmq python driver in the project directly | 14:32 |
| jd_ | [15:31:08] <gordc>just create olso.messaging.good. | 14:33 |
| jd_ | lol gordc | 14:33 |
| jd_ | let's do also.messaging | 14:33 |
| gordc | better.simpler.messaging? :) | 14:34 |
| sileht | damn, no more battery I have to get out of my sofa | 14:34 |
| sileht | brb | 14:34 |
| jd_ | life is a bitch | 14:36 |
| jd_ | you need a 220V sofa sileht | 14:36 |
| *** larainema has quit IRC | 14:39 | |
| *** iranzo has quit IRC | 14:40 | |
| *** iranzo has joined #openstack-telemetry | 14:41 | |
| *** iranzo has quit IRC | 14:41 | |
| *** iranzo has joined #openstack-telemetry | 14:41 | |
| *** links has quit IRC | 14:47 | |
| *** weshay is now known as weshay_pto | 14:55 | |
| openstackgerrit | Julien Danjou proposed openstack/ceilometer stable/pike: fix ceilometer-compute invoke libvirt exception error https://review.openstack.org/521872 | 14:58 |
| openstackgerrit | Merged openstack/ceilometer master: fix ceilometer-compute invoke libvirt exception error https://review.openstack.org/517588 | 15:46 |
| openstackgerrit | gordon chung proposed openstack/ceilometer stable/ocata: update install docs https://review.openstack.org/518731 | 15:55 |
| *** links has joined #openstack-telemetry | 16:04 | |
| openstackgerrit | Julien Danjou proposed openstack/ceilometer stable/pike: fix ceilometer-compute invoke libvirt exception error https://review.openstack.org/521872 | 16:05 |
| *** itlinux has joined #openstack-telemetry | 16:10 | |
| *** links has quit IRC | 16:12 | |
| *** edmondsw has quit IRC | 16:24 | |
| *** edmondsw has joined #openstack-telemetry | 16:25 | |
| *** AlexeyAbashkin has quit IRC | 16:28 | |
| *** edmondsw has quit IRC | 16:29 | |
| *** psachin has quit IRC | 16:29 | |
| *** iranzo has quit IRC | 16:44 | |
| openstackgerrit | Julien Danjou proposed openstack/ceilometer master: polling: run polling tasks immediately on start https://review.openstack.org/521930 | 16:46 |
| *** openstackstatus has quit IRC | 16:58 | |
| *** openstackstatus has joined #openstack-telemetry | 16:59 | |
| *** ChanServ sets mode: +v openstackstatus | 16:59 | |
| *** flwang1 has joined #openstack-telemetry | 17:09 | |
| *** tesseract has quit IRC | 17:09 | |
| *** purplerbot has quit IRC | 17:18 | |
| *** purplerbot has joined #openstack-telemetry | 17:18 | |
| *** gkadam has quit IRC | 17:26 | |
| *** vint_bra has joined #openstack-telemetry | 17:39 | |
| *** pcaruana has quit IRC | 17:48 | |
| *** yprokule has quit IRC | 18:13 | |
| *** flwang1 has quit IRC | 18:24 | |
| *** shardy has quit IRC | 18:26 | |
| *** AlexeyAbashkin has joined #openstack-telemetry | 18:40 | |
| *** AlexeyAbashkin has quit IRC | 18:44 | |
| *** lhx_ has quit IRC | 18:47 | |
| *** itlinux has quit IRC | 18:57 | |
| *** itlinux has joined #openstack-telemetry | 19:03 | |
| EmilienM | jd_: hey, can you please approve https://review.openstack.org/#/c/521284/ ? | 19:06 |
| gordc | done. | 19:09 |
| *** hoonetorg has quit IRC | 19:09 | |
| EmilienM | gordc: thx | 19:09 |
| *** hoonetorg has joined #openstack-telemetry | 19:10 | |
| jd_ | EmilienM: you're welcome | 19:11 |
| EmilienM | jd_: you're too slow. | 19:11 |
| *** AlexeyAbashkin has joined #openstack-telemetry | 19:11 | |
| jd_ | blame RATP | 19:12 |
| EmilienM | jd_: gordc is faster, admit it | 19:15 |
| *** AlexeyAbashkin has quit IRC | 19:16 | |
| jd_ | EmilienM: ok :( | 19:17 |
| openstackgerrit | gordon chung proposed openstack/ceilometer master: test why pipeline queues filling https://review.openstack.org/517066 | 19:54 |
| *** AlexeyAbashkin has joined #openstack-telemetry | 20:11 | |
| *** AlexeyAbashkin has quit IRC | 20:15 | |
| *** flwang1 has joined #openstack-telemetry | 20:39 | |
| *** sbezverk has quit IRC | 20:42 | |
| *** Guest25442 has quit IRC | 21:01 | |
| openstackgerrit | Merged openstack/ceilometer master: Imported Translations from Zanata https://review.openstack.org/521726 | 21:03 |
| *** sbezverk has joined #openstack-telemetry | 21:11 | |
| *** vishwanathj has joined #openstack-telemetry | 21:23 | |
| *** gordc has quit IRC | 21:38 | |
| *** dave-mcc_ has quit IRC | 21:47 | |
| *** threestrands has joined #openstack-telemetry | 21:48 | |
| *** rcernin has joined #openstack-telemetry | 21:50 | |
| *** vint_bra has quit IRC | 22:40 | |
| *** itlinux has quit IRC | 23:05 | |
| *** AlexeyAbashkin has joined #openstack-telemetry | 23:12 | |
| *** AlexeyAbashkin has quit IRC | 23:16 | |
| *** gmann_afk is now known as gmann | 23:28 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!