| *** yuntongjin has joined #openstack-ceilometer | 00:17 | |
| *** larainema has joined #openstack-ceilometer | 00:34 | |
| *** robot has joined #openstack-ceilometer | 00:49 | |
| *** robot is now known as zqfan | 00:49 | |
| *** zqfan has quit IRC | 01:19 | |
| *** nadya has joined #openstack-ceilometer | 01:26 | |
| *** nadya has quit IRC | 01:31 | |
| *** liusheng has joined #openstack-ceilometer | 01:35 | |
| *** yfujioka has joined #openstack-ceilometer | 01:36 | |
| *** julim has quit IRC | 01:39 | |
| *** ljxiash has quit IRC | 02:10 | |
| *** zqfan has joined #openstack-ceilometer | 03:01 | |
| *** larainema has quit IRC | 03:02 | |
| *** ddieterly has quit IRC | 03:10 | |
| *** nadya has joined #openstack-ceilometer | 03:27 | |
| *** nadya has quit IRC | 03:31 | |
| *** eglynn_ has quit IRC | 03:34 | |
| *** links has joined #openstack-ceilometer | 03:41 | |
| *** larainema has joined #openstack-ceilometer | 03:47 | |
| tonyb | stable team and @cores: Please look at: https://review.openstack.org/#/c/235207/ that should unblock the stable/liberty branch for ceilometer. | 04:06 |
|---|---|---|
| *** ddieterly has joined #openstack-ceilometer | 04:10 | |
| *** ddieterly has quit IRC | 04:15 | |
| *** jwcroppe_ has quit IRC | 04:20 | |
| *** jwcroppe has joined #openstack-ceilometer | 04:26 | |
| *** yuntongjin has quit IRC | 04:26 | |
| *** yprokule has joined #openstack-ceilometer | 04:33 | |
| *** yfujioka_ has joined #openstack-ceilometer | 04:41 | |
| *** yfujioka has quit IRC | 04:44 | |
| *** yfujioka_ has quit IRC | 04:44 | |
| *** yfujioka has joined #openstack-ceilometer | 04:44 | |
| *** yuntongjin has joined #openstack-ceilometer | 05:10 | |
| *** ddieterly has joined #openstack-ceilometer | 05:11 | |
| *** yuntongjin1 has joined #openstack-ceilometer | 05:15 | |
| *** ddieterly has quit IRC | 05:16 | |
| *** yuntongjin has quit IRC | 05:17 | |
| openstackgerrit | Lianhao Lu proposed openstack/gnocchi: Use ListOpt for default_aggregation_methods option https://review.openstack.org/235269 | 05:23 |
| *** nadya has joined #openstack-ceilometer | 05:27 | |
| *** nadya has quit IRC | 05:32 | |
| *** shardy_a1k has joined #openstack-ceilometer | 05:52 | |
| *** shardy_afk has quit IRC | 05:54 | |
| *** shardy_a1k has quit IRC | 05:57 | |
| *** shardy_afk has joined #openstack-ceilometer | 05:58 | |
| *** r-mibu has joined #openstack-ceilometer | 06:02 | |
| *** ddieterly has joined #openstack-ceilometer | 06:12 | |
| *** lsmola_ has joined #openstack-ceilometer | 06:16 | |
| *** ddieterly has quit IRC | 06:17 | |
| *** boris-42 has joined #openstack-ceilometer | 06:28 | |
| *** yasu has joined #openstack-ceilometer | 06:45 | |
| *** yasu has quit IRC | 06:58 | |
| *** yasu__ has joined #openstack-ceilometer | 06:58 | |
| *** nadya has joined #openstack-ceilometer | 07:12 | |
| *** ddieterly has joined #openstack-ceilometer | 07:13 | |
| *** ddieterly has quit IRC | 07:18 | |
| *** safchain has joined #openstack-ceilometer | 07:26 | |
| *** shardy_afk is now known as shardy | 07:36 | |
| *** nadya has quit IRC | 07:42 | |
| openstackgerrit | ZhiQiang Fan proposed openstack/python-ceilometerclient: fix typos in docstring https://review.openstack.org/236835 | 07:56 |
| *** Ala has joined #openstack-ceilometer | 08:02 | |
| *** deepthi has joined #openstack-ceilometer | 08:11 | |
| *** ddieterly has joined #openstack-ceilometer | 08:14 | |
| *** yassine has joined #openstack-ceilometer | 08:17 | |
| *** yassine has quit IRC | 08:17 | |
| *** yassine has joined #openstack-ceilometer | 08:17 | |
| *** ddieterly has quit IRC | 08:18 | |
| *** larainema has quit IRC | 08:22 | |
| *** shardy_ has joined #openstack-ceilometer | 08:25 | |
| *** Liam has joined #openstack-ceilometer | 08:25 | |
| *** shardy has quit IRC | 08:26 | |
| *** shardy_ has quit IRC | 08:30 | |
| *** akakios has joined #openstack-ceilometer | 08:31 | |
| *** shardy has joined #openstack-ceilometer | 08:31 | |
| *** nadya has joined #openstack-ceilometer | 08:34 | |
| openstackgerrit | Merged openstack/ceilometer: Updated from global requirements https://review.openstack.org/235939 | 08:40 |
| *** yasu__ has quit IRC | 08:41 | |
| *** akakios has quit IRC | 08:43 | |
| *** eglynn_ has joined #openstack-ceilometer | 08:45 | |
| *** eglynn_ has quit IRC | 08:46 | |
| *** atty has joined #openstack-ceilometer | 08:47 | |
| zqfan | is it possible to notify in this room when a new bug is created? | 08:53 |
| *** cdent has joined #openstack-ceilometer | 08:57 | |
| *** Liam has quit IRC | 09:02 | |
| *** Liam has joined #openstack-ceilometer | 09:04 | |
| *** openstack has joined #openstack-ceilometer | 09:17 | |
| *** ddieterly has quit IRC | 09:19 | |
| *** iberezovskiy has joined #openstack-ceilometer | 09:21 | |
| openstackgerrit | ZhiQiang Fan proposed openstack/ceilometer: reject post sample via direct while gnocchi is enabled https://review.openstack.org/236856 | 09:21 |
| openstackgerrit | ZhiQiang Fan proposed openstack/ceilometer: reject post sample via direct while gnocchi is enabled https://review.openstack.org/236856 | 09:25 |
| *** Liam has quit IRC | 09:43 | |
| *** liusheng has quit IRC | 09:44 | |
| *** liusheng has joined #openstack-ceilometer | 09:44 | |
| *** nadya has quit IRC | 09:53 | |
| *** r-mibu has quit IRC | 09:53 | |
| *** yassine has quit IRC | 09:54 | |
| *** larainema has joined #openstack-ceilometer | 09:56 | |
| *** r-mibu has joined #openstack-ceilometer | 09:59 | |
| *** yasu has joined #openstack-ceilometer | 10:13 | |
| *** ddieterly has joined #openstack-ceilometer | 10:16 | |
| *** ddieterly has quit IRC | 10:21 | |
| openstackgerrit | ZhiQiang Fan proposed openstack/ceilometer: Reject posting sample with direct=true if Gnocchi is enabled https://review.openstack.org/236856 | 10:23 |
| *** cdent has quit IRC | 10:24 | |
| *** akakios has joined #openstack-ceilometer | 10:27 | |
| *** akakios has quit IRC | 10:38 | |
| *** nadya has joined #openstack-ceilometer | 10:39 | |
| *** weihan has joined #openstack-ceilometer | 10:39 | |
| *** eglynn has joined #openstack-ceilometer | 10:41 | |
| *** cdent has joined #openstack-ceilometer | 10:52 | |
| *** khushbu_ has joined #openstack-ceilometer | 11:03 | |
| *** akakios has joined #openstack-ceilometer | 11:06 | |
| *** ddieterly has joined #openstack-ceilometer | 11:17 | |
| *** ddieterl_ has joined #openstack-ceilometer | 11:20 | |
| *** ddieterly has quit IRC | 11:21 | |
| *** gordc has joined #openstack-ceilometer | 11:21 | |
| cdent | gordc: are you still ill because I sure as hell am and it is your fault. | 11:30 |
| gordc | cdent: mind or matter | 11:30 |
| * cdent surrenders | 11:31 | |
| gordc | i'm not stronger than ever. | 11:31 |
| gordc | now* | 11:31 |
| gordc | also lazier than ever. | 11:31 |
| cdent | I think you were right the first time | 11:32 |
| gordc | shaddup | 11:32 |
| * cdent puts up his dukes | 11:33 | |
| gordc | i'm staying clear. i'm not getting sick again | 11:33 |
| * cdent will be bringing some unique uk germs to tokyo, for sharing | 11:34 | |
| *** Ala has quit IRC | 11:35 | |
| gordc | cdent: i think they will force you to wear a mask | 11:35 |
| gordc | you can pretend you're a doctor. | 11:35 |
| cdent | gee that's swell | 11:35 |
| akakios | Hi all, I am having a problem getting ceilometer to retrieve ceph/rados metrics. I have followed the steps in the Ceph Object Storage section (http://docs.openstack.org/admin-guide-cloud/telemetry-measurements.html#ceph-object-storage) and adjusted the ceilometer.conf & pipeline.yaml (http://pastebin.com/g9MGR5y6). I have also restarted the ceilometer agent (and other ceilometer services in the controller) but the rados metrics st | 11:38 |
| akakios | ill won't show up. Nova & glance metrics are collected and rados has been setup as the object store. Querying the swift API also works correctly. Any ideas on what step/s I may have missed? I don't see any rados or object store related errors in the logs. Environment is running on Openstack Kilo and Ceph 0.93. | 11:38 |
| gordc | akakios: you see any errors in your central agent logs? | 11:46 |
| akakios | gordc: the errors I've seen are related to authentication leading to the nova client | 11:48 |
| *** krotscheck_ is now known as krotscheck | 11:48 | |
| akakios | gordc: unfortunately, environment is being reinstalled so I can't provide the exact errors | 11:48 |
| gordc | akakios: hmm. you can try opening a bug but i'm not sure we have enough info to go on (maybe someone else has same issue and has more detail or a fix) | 11:50 |
| akakios | gordc: okay, going to give it another shot and see if it will get through this time | 11:52 |
| akakios | gordc: but so far, I haven't missed any steps? | 11:52 |
| gordc | akakios: i don't use ceph personally but it's entirely possible there's a gap in the documentation. feel free to open a bug to track it | 11:55 |
| gordc | might take a while to get to it since a lot of the devs are prepping for summit | 11:55 |
| akakios | gordc: okay, thanks for the help | 11:55 |
| *** thorst has joined #openstack-ceilometer | 12:01 | |
| *** larainema has quit IRC | 12:02 | |
| gordc | cdent: i think we need to equivalent https://github.com/openstack/ceilometer/blob/master/devstack/settings#L43-L45 for ceilometermiddleware | 12:03 |
| gordc | should we be adding it to ceilometer repo or somewhere else? | 12:03 |
| * cdent looks | 12:04 | |
| gordc | we have similar 'not installed correctly' errors as ceilometerclient http://logs.openstack.org/00/224600/6/check/gate-tempest-dsvm-neutron-src-ceilometermiddleware/35d5853/logs/devstacklog.txt.gz#_2015-10-16_01_28_02_044 | 12:05 |
| cdent | lib/swift | 12:05 |
| gordc | ack | 12:05 |
| cdent | at the moment in devstack, all ceilo-middleware stuff is (and should be) in lib/swift | 12:05 |
| cdent | so I'm guessing that's an oversight | 12:06 |
| gordc | hmmm. seems to be there https://github.com/openstack-dev/devstack/blob/master/stackrc#L449 | 12:09 |
| gordc | maybe that's not the issue. | 12:09 |
| *** yasu has quit IRC | 12:11 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/ceilometer: Updated from global requirements https://review.openstack.org/236940 | 12:17 |
| *** ljxiash has joined #openstack-ceilometer | 12:25 | |
| cdent | jd__: I haven't got any brilliant insights on what do about this issue in the gnocchi gate | 12:25 |
| *** weihan_ has joined #openstack-ceilometer | 12:25 | |
| cdent | the only thing I can think of is adding some more swift jobs (with other indexers) as experimental and try them to see if the results are any different | 12:25 |
| *** weihan_ has quit IRC | 12:26 | |
| *** weihan has quit IRC | 12:26 | |
| *** weihan has joined #openstack-ceilometer | 12:26 | |
| *** weihan has quit IRC | 12:27 | |
| cdent | I diddled around with trying to replicate things locally but haven't had much success | 12:27 |
| *** yasu has joined #openstack-ceilometer | 12:27 | |
| *** weihan has joined #openstack-ceilometer | 12:27 | |
| *** edmondsw has joined #openstack-ceilometer | 12:29 | |
| *** weihan has quit IRC | 12:30 | |
| jd__ | cdent: :/ | 12:31 |
| *** weihan has joined #openstack-ceilometer | 12:31 | |
| *** weihan has quit IRC | 12:34 | |
| * cdent is trying one more local investigation | 12:36 | |
| *** atty has quit IRC | 12:36 | |
| *** weihan has joined #openstack-ceilometer | 12:37 | |
| *** weihan has quit IRC | 12:37 | |
| *** weihan has joined #openstack-ceilometer | 12:38 | |
| cdent | jd__: got it | 12:41 |
| cdent | not a solution | 12:41 |
| *** dikonoor has joined #openstack-ceilometer | 12:41 | |
| cdent | but I've replicated the failure | 12:41 |
| cdent | "die 387 'gnocchi-api did not start'" is the problem, yeah? | 12:41 |
| *** yasu has quit IRC | 12:42 | |
| dikonoor | gordc: hi gordc | 12:43 |
| dikonoor | gordc: could you take a look at https://bugs.launchpad.net/ceilometer/+bug/1506717 when you get a chance | 12:43 |
| openstack | Launchpad bug 1506717 in Ceilometer "Ceilometer GET events throws deadlock or timeout errors" [Undecided,New] | 12:43 |
| *** khushbu_ has quit IRC | 12:46 | |
| *** weihan has quit IRC | 12:46 | |
| *** weihan has joined #openstack-ceilometer | 12:47 | |
| *** trown|outttypeww is now known as trown | 12:47 | |
| gordc | dikonoor: isolation_level is causing issues in db2? | 12:52 |
| dikonoor | gordc: yes..it works fine if no isolation level is set | 12:54 |
| dikonoor | gordc: which was the case before | 12:55 |
| *** ljxiash has quit IRC | 12:55 | |
| gordc | dikonoor: what's the isolation level normally in db2 | 12:56 |
| gordc | the issue is there are two queries to build event list | 12:56 |
| gordc | if you don't set isolation_level to repeatable read, both queries (may) work against different datasets | 12:57 |
| *** ljxiash has joined #openstack-ceilometer | 12:57 | |
| *** yassine__ has joined #openstack-ceilometer | 12:58 | |
| gordc | the alternative is to leave default isolation levels for all db, and just toss any differences from second query and assume it's all good. | 12:58 |
| *** yprokule has quit IRC | 13:00 | |
| *** yprokule has joined #openstack-ceilometer | 13:01 | |
| *** larainema has joined #openstack-ceilometer | 13:01 | |
| openstackgerrit | Merged openstack/ceilometer: make script under tools directory executable https://review.openstack.org/236695 | 13:02 |
| openstackgerrit | Merged openstack/ceilometer: Reject posting sample with direct=true if Gnocchi is enabled https://review.openstack.org/236856 | 13:02 |
| openstackgerrit | Merged openstack/ceilometer: Added the README.rst in devstack folder https://review.openstack.org/236176 | 13:02 |
| *** ddieterl_ has quit IRC | 13:03 | |
| *** yuntongjin has joined #openstack-ceilometer | 13:04 | |
| dikonoor | gordc: Not sure of the default isolation level in db2 . Also, I don't know if this problem occurs with other databases (but the documentation implies that it could). why do we do two queries to get events? | 13:16 |
| *** Ala has joined #openstack-ceilometer | 13:17 | |
| gordc | one to get events, the other to get traits. | 13:17 |
| gordc | last i tried, i couldn't come up with one single query to do that (as i couldn't handle scenario with events with no traits) | 13:18 |
| gordc | also, i refuse to use orm relationships becasue it's slow as molasses. | 13:19 |
| *** yuntongjin1 has joined #openstack-ceilometer | 13:23 | |
| jd__ | cdent: yeah | 13:24 |
| jd__ | cdent: it times-out because of swift, etc… | 13:24 |
| jd__ | cdent: you have something interesting? | 13:24 |
| cdent | jd__: well I have a local.conf that causes the same issue, so it iought to become a bit easier to debug | 13:24 |
| jd__ | completely | 13:25 |
| *** yuntongjin has quit IRC | 13:25 | |
| jd__ | lucky you | 13:25 |
| cdent | however I'm not yet any closer to understanding what's going on | 13:25 |
| cdent | I agree with you that it seems likely an eventlet related thing | 13:26 |
| dikonoor | gordc: ok..I don't see this problem all the time..but once I see this..I have to wait for few minutes for this to be resolved.. :-| | 13:26 |
| *** julim has joined #openstack-ceilometer | 13:27 | |
| gordc | dikonoor: i'm not against dropping it. mysql already has repeatable read as default. the only thing is that if we don't set isolation level, we need to make sure the results are consistent still. | 13:28 |
| *** akakios has quit IRC | 13:28 | |
| dikonoor | gordc: hmm..agreed | 13:28 |
| *** dan-t has joined #openstack-ceilometer | 13:29 | |
| *** ddieterly has joined #openstack-ceilometer | 13:32 | |
| cdent | jd__: can you help me understand the flow of things a bit better? | 13:35 |
| cdent | If I make any request to the gnocchi-api server after it starts up it appears to do something with the keystone client service_catalog and then does a PUT /v1/AUTH_<some key> to swift. What is that PUT? Why is it happening? | 13:36 |
| *** lawrancejing has joined #openstack-ceilometer | 13:38 | |
| *** ddieterl_ has joined #openstack-ceilometer | 13:40 | |
| zqfan | hi, good evening, I have a patch need your reviews: https://review.openstack.org/235826 , this patch aims at improving performance for query-sample API based on mysql backend, previously it will time out (10 minutes) even specify limit on a test data set which only contains 500K samples, with this patch, it can response in around 10 seconds | 13:43 |
| *** links has quit IRC | 13:44 | |
| *** ddieterly has quit IRC | 13:44 | |
| jd__ | cdent: it's the initial container creation to store new measures | 13:44 |
| jd__ | cdent: that's the call that is blocking gnocchi entirely, the API and metricd | 13:44 |
| jd__ | it never returns :( | 13:44 |
| cdent | yeah, I see what | 13:44 |
| cdent | It just wasnt' clear what its purpose was | 13:44 |
| jd__ | is it reproducible with a small Python script? | 13:45 |
| cdent | I'm narrowing my local.conf down to its bare minimum so I can share it with you | 13:45 |
| jd__ | which would indicate Gnocchi has nothing to do with it | 13:45 |
| jd__ | ok cool | 13:45 |
| *** flwang has joined #openstack-ceilometer | 13:46 | |
| flwang | gordc: ping | 13:46 |
| *** exploreshaifali has joined #openstack-ceilometer | 13:47 | |
| cdent | jd__: here's a relatively fast and small local.conf that shows the problem: https://tank.peermore.com/_/c96bacfd-a54a-41d4-9bbf-2f81df431867 | 13:49 |
| cdent | jd__: with that setup the swift client blocks when trying to use it without gnocchi being involved at all | 13:51 |
| * cdent tries again without gnocchi running anywhere | 13:52 | |
| jd__ | pretty cool | 13:52 |
| jd__ | I hope we can say we have a so good test coverage we find bugs people miss -_- | 13:52 |
| jd__ | I'm trying your local.conf | 13:52 |
| *** ityaptin has joined #openstack-ceilometer | 13:53 | |
| gordc | flwang: hey, whatsup? | 13:54 |
| flwang | gordc: jd__: did you see this https://mitakadesignsummit.sched.org/event/eb08d0da3f041647aa0053ee031d58ef ? | 13:55 |
| gordc | flwang: i do now | 13:55 |
| cdent | jd__: :( If I use that same local.conf but with no gnocchi around, swift works just fine... | 13:55 |
| *** weihan has quit IRC | 13:55 | |
| gordc | flwang: it's at the same time as ceilometer work session though | 13:56 |
| flwang | zaqar team and horizon team are working on a real time table update topic | 13:56 |
| *** yprokule has quit IRC | 13:56 | |
| flwang | gordc: i will talk with mrunge to see if the time can be changed | 13:56 |
| flwang | gordc: in a short, zaqar team would like to have a zaqar publisher, just like kafka | 13:57 |
| gordc | flwang: ah, the stuff vkmc was playing with | 13:57 |
| *** yprokule has joined #openstack-ceilometer | 13:58 | |
| gordc | flwang: not sure what exactly is being planned but i can try joining if it doesn't overlap with ceilometer sessions | 13:59 |
| vkmc | o/ | 14:01 |
| *** flwang1 has joined #openstack-ceilometer | 14:01 | |
| vkmc | gordc, that incredible piece of code you meant, right? | 14:02 |
| flwang1 | gordc: just wanna get your comments about adding a publisher for zaqar in ceilometer | 14:02 |
| flwang1 | vkmc: hey | 14:02 |
| gordc | vkmc: lol. sure. | 14:02 |
| vkmc | flwang, gordc, hey :) | 14:02 |
| *** flwang has quit IRC | 14:02 | |
| jd__ | cdent: interesting! | 14:05 |
| gordc | flwang1: i'm not super familiar with zaqar but how is it different from the other oslo.messaging drivers? or they completely unrelated? | 14:05 |
| jd__ | cdent: a diff in pip freeze maybe? I don't think gnocchi configures swift in a different way | 14:06 |
| flwang1 | gordc: they're different | 14:06 |
| cdent | jd__: something like that yeah | 14:06 |
| flwang1 | the oslo.messaging drivers, like rabbitmq, are most like a lib, but zaqar is multi tenancy messaging system | 14:07 |
| * cdent makes a diff | 14:07 | |
| flwang1 | gordc: for example, when publish the message to zaqar, you need to know the project id to post the messages | 14:07 |
| gordc | flwang1: the project_id of the target? | 14:09 |
| flwang1 | gordc: yep | 14:09 |
| *** KrishR has joined #openstack-ceilometer | 14:09 | |
| gordc | flwang1: i see. | 14:10 |
| gordc | flwang1: i don't really have any issue with having it internally in ceilometer repo or externally. | 14:10 |
| flwang1 | gordc: the current problem of horizon is it has to poll forever to get the latest object status | 14:11 |
| flwang1 | for example, when user is viewing the instances page | 14:11 |
| cdent | jd__: no diff | 14:11 |
| flwang1 | horizon has to poll nova to get the latest status for all the instnaces per 2.5 sec | 14:12 |
| *** yuntongjin1 has quit IRC | 14:12 | |
| flwang1 | it's really bad | 14:12 |
| flwang1 | so horizon team is trying to leverage ceilometer and zaqar to figure out a good way to fix it | 14:12 |
| gordc | flwang1: you mean the the pie graphs on the main page? | 14:13 |
| flwang1 | gordc: the instance ovewview table | 14:13 |
| flwang1 | on horizon menu /project/instances | 14:13 |
| jd__ | cdent: hihihihi O_O | 14:14 |
| cdent | eh? | 14:14 |
| flwang1 | gordc: and if there is no ceilometer or zaqar deployed, then horizon will use current way to poll | 14:14 |
| gordc | flwang1: this page? http://hispavirt.com/wp-content/uploads/2013/09/horizonopenstack.png | 14:15 |
| jd__ | cdent: i'm just getting crazy | 14:15 |
| cdent | darn, I thought you had uncovered the secret... | 14:15 |
| flwang1 | gordc: no, Instances | 14:15 |
| flwang1 | not overview | 14:15 |
| jd__ | cdent: I can reproduce it now, same as you | 14:16 |
| *** rbak has joined #openstack-ceilometer | 14:16 | |
| cdent | I guess that's progress | 14:16 |
| jd__ | cdent: I'm gonna try to comment part of gnocchi devstack plugin.sh to see what part makes that | 14:16 |
| gordc | flwang1: got it | 14:16 |
| gordc | yeah, i wasn't aware that page was polling constantly. | 14:16 |
| cdent | I suppose a next step is to instrument the swift-proxy server to see what it is doing when it is blocking | 14:16 |
| * cdent tries that | 14:17 | |
| flwang1 | gordc: if there is no objection, i will post a spec in ceilometer-spec for you guys review | 14:17 |
| flwang1 | gordc: and i will talk with horizon guys to see if we can change the time | 14:17 |
| flwang1 | i mean the session time | 14:17 |
| gordc | flwang1: sure sounds fine. | 14:18 |
| gordc | flwang1: i should point out we also have a ceilometer+horizon slot https://mitakadesignsummit.sched.org/event/4c0d8ce4c14e3e49a5aa18228975ee05 | 14:18 |
| flwang1 | for visualizing data, IIRC? | 14:19 |
| gordc | right | 14:19 |
| flwang1 | gordc: cool, that's an option | 14:20 |
| gordc | not exactly related to what you're trying to accomplish but basically we're trying to discuss how to present ceilometer data in a useful manner | 14:20 |
| gordc | (because based on feedback, it's not useful currently) | 14:20 |
| *** shardy_ has joined #openstack-ceilometer | 14:20 | |
| flwang1 | gordc: yep, i think it's good topic | 14:21 |
| flwang1 | gordc: i do have some suggestions for that | 14:21 |
| *** shardy has quit IRC | 14:22 | |
| flwang1 | gordc: thank you so much for your comments | 14:22 |
| gordc | flwang1: cool cool. feel free to add notes to etherpad | 14:23 |
| openstackgerrit | Wen Zhi Yu proposed openstack/ceilometer: Unify the timestamp of polled data https://review.openstack.org/232976 | 14:23 |
| flwang1 | gordc: will do :) | 14:24 |
| *** alexpilotti has quit IRC | 14:26 | |
| *** shardy_ has quit IRC | 14:26 | |
| *** rakhi_ has joined #openstack-ceilometer | 14:26 | |
| *** links has joined #openstack-ceilometer | 14:26 | |
| *** shardy has joined #openstack-ceilometer | 14:27 | |
| *** links has quit IRC | 14:35 | |
| *** weihan has joined #openstack-ceilometer | 14:37 | |
| *** weihan has quit IRC | 14:40 | |
| *** weihan has joined #openstack-ceilometer | 14:41 | |
| jd__ | gordc: do we know if anyone from Horizon is going to show up? | 14:45 |
| cdent | jd__: slightly more data (you may already have this datapoint): If you kill the swift-proxy server and start it back up again subsequent queries do not block | 14:45 |
| jd__ | gordc: because I'm sure we all have good ideas but if nobody cares to implement them, it's just a waste of time | 14:45 |
| jd__ | cdent: oh? no I did not notice that, very strange | 14:46 |
| jd__ | cdent: me I just learnt that the problem lies in init_gnocchi | 14:46 |
| gordc | jd__: i believe david-lyle has made his team aware? | 14:46 |
| cdent | jd__: how'd you determine that? | 14:47 |
| gordc | jd__: i don't know if the exact right people will come but we can always re-raise in ML | 14:47 |
| jd__ | cdent: I enabled Gnocchi plugin, then # out everything – swift worked | 14:48 |
| cdent | oh I see what you mean | 14:49 |
| jd__ | cdent: then I uncommented line by line and waited to find the culprit = swift does not work anymore | 14:49 |
| jd__ | init_gnocchi does very few things | 14:49 |
| jd__ | I'll comment line by line and see | 14:49 |
| jd__ | dat suspense | 14:50 |
| * cdent votes for auth cache dir | 14:50 | |
| jd__ | devstackbet.com | 14:51 |
| jd__ | cdent: I guess you lost | 14:53 |
| cdent | okay, but if you've found it we all win and cookies all round | 14:53 |
| cdent | oh hey, I just had a successful run | 14:54 |
| cdent | I changed that curl call | 14:54 |
| cdent | to behave slightly differently | 14:54 |
| cdent | and now it works | 14:54 |
| jd__ | ?? | 14:56 |
| cdent | hmmm | 14:56 |
| cdent | one moment please | 14:56 |
| jd__ | recreate_database gnocchi utf8 | 14:56 |
| jd__ | that's the line apparently | 14:56 |
| * jd__ rechecks | 14:56 | |
| * cdent rechecks his thing | 14:56 | |
| jd__ | I really can't see how this line can make things not work so might be something else | 14:57 |
| cdent | my theory is that this line isn't good enough: https://github.com/openstack/gnocchi/blob/master/devstack/plugin.sh#L386 | 14:58 |
| cdent | I'm adding a --max-time 5 | 14:58 |
| cdent | so that it will fail waiting for a response | 14:58 |
| cdent | otherwise it never actually retries | 14:58 |
| * cdent waits | 15:00 | |
| jd__ | hmmm so it's just a slowness issue? | 15:00 |
| cdent | it doesn't seem to want to work this time, still waiting | 15:00 |
| cdent | yeah, didn't work that time, not sure what's different | 15:00 |
| * cdent sighs | 15:00 | |
| * cdent tries again | 15:02 | |
| jd__ | epic debug… | 15:03 |
| *** deepthi has quit IRC | 15:04 | |
| *** rakhi_ has quit IRC | 15:05 | |
| *** larainema_ has joined #openstack-ceilometer | 15:05 | |
| cdent | jd__: is there any chance the database recreate is throwing postgres into confusion which would then block the world? | 15:06 |
| jd__ | cdent: I don't know, I'm retrying to confirm this is init_gnocchi failing and where | 15:07 |
| jd__ | one of my test got mangled because of a bad unstack I think | 15:07 |
| * jd__ wishes devstack was fast | 15:07 | |
| cdent | yeah, unstack is not super reliable | 15:07 |
| cdent | patches accepted! ;) | 15:07 |
| jd__ | wait | 15:09 |
| jd__ | why is my devstack trying to run keystone-all | 15:09 |
| jd__ | ENABLE_HTTPD_MOD_WSGI_SERVICES ah ok | 15:09 |
| jd__ | is this on the gate too? | 15:09 |
| cdent | gate has mod wsgi false for gnocchi but true for keystone if I recall correctly | 15:10 |
| jd__ | no gnocchi uses wsgi too actually | 15:11 |
| cdent | we turned if off a while ago didn't we? | 15:11 |
| cdent | or was it that we just set the port? | 15:11 |
| openstackgerrit | ZhiQiang Fan proposed openstack/ceilometer: avoid generate temporary table when query samples https://review.openstack.org/235826 | 15:12 |
| * cdent prepares to try some other permutations | 15:13 | |
| cdent | (I got distracted from that, that was actually my original goal) | 15:13 |
| jd__ | so my current problem is the following | 15:14 |
| jd__ | 2015-10-19 15:12:44.397 TRACE keystone error: [Errno 98] Address already in use | 15:14 |
| jd__ | keystone starts somewhere in background | 15:14 |
| *** ityaptin has quit IRC | 15:14 | |
| jd__ | and screen shows that in the one that is ran in screen | 15:14 |
| jd__ | so there's like 2 keystone-all running | 15:14 |
| jd__ | I don't get why | 15:14 |
| cdent | that sounds like a bad unstack, or a left over keystone in /etc/{httpd,apache2}/conf.d | 15:15 |
| jd__ | apache conf is clean, and a I unstacked and killed everything | 15:15 |
| jd__ | and I was able to reproduce it | 15:15 |
| * jd__ investigates | 15:15 | |
| * cdent doesn't seem to have that problem | 15:16 | |
| jd__ | :/ | 15:19 |
| jd__ | Lost in Bash | 15:20 |
| jd__ | an OpenStack movie | 15:20 |
| *** dikonoor has quit IRC | 15:22 | |
| vkmc | jd__, want to see that | 15:22 |
| * cdent keeps on stackin' | 15:24 | |
| * jd__ too | 15:24 | |
| jd__ | that could be a song | 15:24 |
| cdent | jd__: you probably already guess this, but postgres/mysql makes no difference, same problem in both | 15:24 |
| jd__ | ♭ these scripts are made for stackin' ᖱ | 15:25 |
| cdent | and that's what I'm gonna do | 15:25 |
| cdent | one of these days these scripts are gonna stack all over you | 15:25 |
| cdent | yep, that'll do | 15:25 |
| jd__ | :)) | 15:26 |
| cdent | \o/ | 15:32 |
| cdent | jd__: try this: https://tank.peermore.com/_/d8d33dca-9f57-42bf-8cd4-98dc19378d7f | 15:32 |
| cdent | from what I can tell gnocchi-metricd is starting up to talk to swift and making a bunch of bad connections, causing the proxy server to run out of available stuff | 15:33 |
| cdent | moving it to the end prevents that | 15:33 |
| cdent | but is not enough | 15:33 |
| cdent | the swift proxy server isn't quite ready when the curl is done so we have to wait for it but the previous retry was not the right kind: it would only timeout once because it was blocking | 15:34 |
| cdent | the --max-time change makes it so the curl will fail and then properly retry | 15:34 |
| cdent | I've made that work twice in a row now so it is at least a move in the right direction | 15:34 |
| cdent | seem like there may be an issue in metricd's retry logic or connection pool handling? or swift is just lame at startup time | 15:35 |
| cdent | try that, lemme know, I'll brb | 15:35 |
| openstackgerrit | Julien Danjou proposed openstack/gnocchi: devstack: remove unused utf8 argument https://review.openstack.org/237048 | 15:35 |
| jd__ | cdent: trying! | 15:38 |
| *** lawrancejing has quit IRC | 15:45 | |
| jd__ | cdent: first run worked | 15:49 |
| cdent | zomg | 15:49 |
| jd__ | I really don't understand what your patch changes | 15:49 |
| jd__ | it should not change anything to have metricd started first | 15:49 |
| cdent | if you look at the gnocchi-metricd logs in failed runs you may see that it has a lot of failed auth errors | 15:50 |
| cdent | or just failed connections, can't remember | 15:50 |
| cdent | whatever it is, there are a lot of them | 15:50 |
| cdent | does metricd rememer to do the same initial container creation that the api server does? | 15:51 |
| cdent | s/rememer/remember/ | 15:51 |
| jd__ | yup | 15:53 |
| * jd__ goes watch logs | 15:53 | |
| jd__ | and what does it fail if swift is running anyway | 15:53 |
| jd__ | s/what/why/ | 15:53 |
| cdent | My guess is that there is an initial slowness in s-proxy that metricd is not dealing with, and is instead just doing more work, somehow, this then aggrevates swift even more so that by the time we get to testing if the api server is up things are hosed | 15:54 |
| cdent | it's quite hard to figure out the how/why of it all :( | 15:54 |
| jd__ | yeah, sounds like a race condition on Swift side | 15:58 |
| jd__ | cdent: you send a patch? | 15:58 |
| cdent | I will yeah, sure, just a sec | 15:58 |
| jd__ | my second devstack run succeeded | 15:58 |
| *** safchain has quit IRC | 16:00 | |
| openstackgerrit | Chris Dent proposed openstack/gnocchi: Make sure that swift doesn't block gnocchi startup https://review.openstack.org/237060 | 16:03 |
| cdent | we'll see how that goes | 16:03 |
| jd__ | ah Gerrit still 500 with Unicode in comments, sigh | 16:06 |
| jd__ | . | 16:06 |
| jd__ | l. | 16:06 |
| * cdent checks what year it is | 16:07 | |
| *** shardy_ has joined #openstack-ceilometer | 16:08 | |
| *** shardy has quit IRC | 16:10 | |
| openstackgerrit | Dong Ma proposed openstack/ceilometer: Added CORS support to Ceilometer https://review.openstack.org/224464 | 16:11 |
| *** shardy_ has quit IRC | 16:14 | |
| *** shardy has joined #openstack-ceilometer | 16:15 | |
| *** vishwana_ has joined #openstack-ceilometer | 16:26 | |
| *** trown|brb has joined #openstack-ceilometer | 16:28 | |
| *** Ala has quit IRC | 16:28 | |
| *** vishwanathj has quit IRC | 16:29 | |
| *** chmouel has quit IRC | 16:29 | |
| *** trown has quit IRC | 16:29 | |
| cdent | jd__: it worked in the gate too. have a good flight! | 16:32 |
| *** weihan has quit IRC | 16:33 | |
| *** chmouel has joined #openstack-ceilometer | 16:34 | |
| *** trown|brb is now known as trown | 16:34 | |
| *** davidlenwell_ is now known as davidlenwell | 16:36 | |
| jd__ | cdent: thanks! | 16:36 |
| *** exploreshaifali has quit IRC | 16:45 | |
| *** yprokule has quit IRC | 16:49 | |
| *** Ephur has joined #openstack-ceilometer | 16:50 | |
| *** yassine__ has quit IRC | 16:51 | |
| *** openstackgerrit has quit IRC | 17:01 | |
| *** openstackgerrit has joined #openstack-ceilometer | 17:01 | |
| *** lsmola_ has quit IRC | 17:04 | |
| *** jwcroppe_ has joined #openstack-ceilometer | 17:23 | |
| *** jwcroppe has quit IRC | 17:24 | |
| *** iberezovskiy has quit IRC | 17:26 | |
| *** dan-t has quit IRC | 17:26 | |
| *** jwcroppe has joined #openstack-ceilometer | 17:41 | |
| *** jwcroppe_ has quit IRC | 17:44 | |
| openstackgerrit | Merged openstack/gnocchi: Make sure that swift doesn't block gnocchi startup https://review.openstack.org/237060 | 18:08 |
| *** cdent has quit IRC | 18:21 | |
| *** nadya has quit IRC | 18:21 | |
| *** devlaps has joined #openstack-ceilometer | 18:22 | |
| *** dan-t has joined #openstack-ceilometer | 18:22 | |
| *** shardy is now known as shardy_afk | 18:30 | |
| *** eglynn has quit IRC | 18:32 | |
| *** eglynn has joined #openstack-ceilometer | 18:32 | |
| ildikov | gordc: hey, are you around? | 18:37 |
| gordc | ildikov: yep | 18:39 |
| *** khushbu_ has joined #openstack-ceilometer | 18:43 | |
| ildikov | gordc: maybe I asked it already, but maybe not :) | 18:44 |
| ildikov | gordc: I have a patch for Aodh which contains the config options | 18:45 |
| ildikov | but it has additional items added by the script which generates the docs | 18:45 |
| *** Guest3686 is now known as mgagne | 18:45 | |
| *** mgagne has joined #openstack-ceilometer | 18:45 | |
| ildikov | the items are coming from oslomiddleware | 18:45 |
| ildikov | do you have any idea what dependency can bring them in, which is not present when we generate the config file? | 18:46 |
| ildikov | gordc: ^^ | 18:46 |
| gordc | aodh has additional items? | 18:47 |
| ildikov | gordc: so my issue is that the config file I generate with tox contains less config options than the list what is generated by the doctools script | 18:47 |
| ildikov | gordc: which would not be a problem if we would not generate the config ref with scripts, which means that if I delete it now by hands it will not solve my issue long term | 18:48 |
| ildikov | gordc: I'm just out of ideas regarding where the dependenies can go wrong and where the oslomiddleware options can come | 18:49 |
| ildikov | gordc: sorry for the dumb question/topic :) | 18:49 |
| *** larainema_ has quit IRC | 18:50 | |
| gordc | ildikov: did you add it here? https://github.com/openstack/aodh/blob/master/generate-config-file.sh | 18:50 |
| *** larainema_ has joined #openstack-ceilometer | 18:50 | |
| gordc | ildikov: to be honest, my mind is shut down so i'm not really sure i know what you're asking.lol | 18:50 |
| ildikov | gordc: lol :) | 18:51 |
| openstackgerrit | Merged openstack/python-gnocchiclient: Creates better exceptions for http code 409 https://review.openstack.org/230396 | 18:51 |
| ildikov | gordc: I have this patch: https://review.openstack.org/#/c/222931/8 | 18:51 |
| ildikov | gordc: which is not merged, because the list contains additional items compared to the actual config file of Aodh | 18:52 |
| ildikov | gordc: but the option list is generated by a script, that reads the code and of course does some magic... | 18:52 |
| ildikov | gordc: and I ran out of ideas that which part of the magic went wrong :) | 18:53 |
| gordc | where's the script? | 18:53 |
| *** julim has quit IRC | 18:53 | |
| *** nadya has joined #openstack-ceilometer | 18:53 | |
| ildikov | gordc: script: https://github.com/openstack/openstack-doc-tools/blob/master/autogenerate_config_docs/autohelp.py | 18:53 |
| ildikov | gordc: hook for Aodh: https://github.com/openstack/openstack-doc-tools/blob/master/autogenerate_config_docs/hooks.py#L21 | 18:54 |
| *** julim has joined #openstack-ceilometer | 18:56 | |
| gordc | hmm.. so we can't do the exact same thing as we did for ceilometer? | 18:56 |
| gordc | i have no idea what that script does. | 18:57 |
| *** boris-42 has quit IRC | 18:58 | |
| gordc | ildikov: i think it's pretty obvious i have no idea how that works | 18:59 |
| ildikov | gordc: it differs from Ceilometer because the script uses the global CONF object, which we removed from Aodh :( | 19:00 |
| gordc | i see. | 19:01 |
| ildikov | gordc: I wonder how much that would hurt to leave the very few extra options in the docco | 19:01 |
| ildikov | gordc: I'm really not sure how many people reads it as a novel, I mean from the first line to the last | 19:01 |
| ildikov | gordc: I hate the idea, but don't have any better :S | 19:02 |
| ildikov | gordc: and I already asked the person, who wrote that script, which I only slightly modified... | 19:02 |
| gordc | ildikov: i think it's ok to have more (if they're still valid) | 19:04 |
| ildikov | gordc: maybe I will ask the guys to merge the patch and report a bug about the issue | 19:05 |
| ildikov | gordc: the options are valid, just not for Aodh or at least we don't use them, so I assume they don't do any harm | 19:06 |
| gordc | ildikov: yeah that migh tbe easiest. | 19:06 |
| gordc | i've no idea how that script works. | 19:06 |
| ildikov | gordc: yeah, in this detail me neither :( | 19:08 |
| ildikov | gordc: thanks for your time! | 19:10 |
| gordc | ildikov: is it just the CORS stuff? | 19:13 |
| ildikov | gordc: oslo_middleware/max_request_body_size and oslo_middleware/secure_proxy_ssl_header IIRC | 19:16 |
| ildikov | gordc: I mean these are not in the sample config file we generate | 19:16 |
| gordc | seems like keystone has it in their's as well | 19:22 |
| gordc | ildikov: maybe it's just a limitation of tool | 19:22 |
| gordc | i'd probably just leave it | 19:22 |
| ildikov | you mean the docco or the keystone config file itself where they have it? | 19:26 |
| *** eglynn has quit IRC | 19:29 | |
| gordc | it's in both for keystone.. but i don't think max_request_body_size is actually used anywhere | 19:32 |
| *** KrishR has quit IRC | 19:37 | |
| *** eglynn has joined #openstack-ceilometer | 19:41 | |
| *** KrishR has joined #openstack-ceilometer | 19:41 | |
| *** khushbu_ has quit IRC | 19:49 | |
| *** exploreshaifali has joined #openstack-ceilometer | 19:50 | |
| *** eglynn has quit IRC | 20:07 | |
| *** eglynn has joined #openstack-ceilometer | 20:07 | |
| *** nadya has quit IRC | 20:14 | |
| ildikov | gordc: I'm convinced :) | 20:14 |
| openstackgerrit | Chris Dent proposed openstack/gnocchi: rest: deserialize directly with file descriptor https://review.openstack.org/234601 | 20:25 |
| *** larainema_ has quit IRC | 20:28 | |
| *** larainema_ has joined #openstack-ceilometer | 20:29 | |
| openstackgerrit | Chris Dent proposed openstack/gnocchi: TEST: Try a longer timeout on waiting for swift https://review.openstack.org/237193 | 20:30 |
| *** Ephur has quit IRC | 20:31 | |
| *** gordc has quit IRC | 20:38 | |
| *** thorst has quit IRC | 20:40 | |
| *** Ephur has joined #openstack-ceilometer | 20:49 | |
| *** julim has quit IRC | 21:04 | |
| *** trown is now known as trown|outttypeww | 21:06 | |
| *** eglynn has quit IRC | 21:15 | |
| *** jwcroppe has quit IRC | 21:21 | |
| *** larainema_ has quit IRC | 21:31 | |
| *** larainema has quit IRC | 21:32 | |
| *** dan-t has quit IRC | 21:47 | |
| *** exploreshaifali has quit IRC | 22:14 | |
| *** ddieterl_ has quit IRC | 22:16 | |
| *** flwang1 has quit IRC | 22:29 | |
| *** thorst has joined #openstack-ceilometer | 22:34 | |
| *** ddieterly has joined #openstack-ceilometer | 22:45 | |
| *** KrishR has quit IRC | 22:53 | |
| *** rbak has quit IRC | 22:59 | |
| openstackgerrit | ZhiQiang Fan proposed openstack/ceilometer: avoid generate temporary table when query samples https://review.openstack.org/235826 | 23:10 |
| *** lawrancejing has joined #openstack-ceilometer | 23:16 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/ceilometer: Updated from global requirements https://review.openstack.org/236940 | 23:26 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/ceilometermiddleware: Updated from global requirements https://review.openstack.org/224600 | 23:26 |
| *** ddieterly has quit IRC | 23:35 | |
| *** ddieterly has joined #openstack-ceilometer | 23:38 | |
| *** ddieterly has quit IRC | 23:42 | |
| *** ddieterly has joined #openstack-ceilometer | 23:43 | |
| *** ddieterl_ has joined #openstack-ceilometer | 23:49 | |
| *** ddieterly has quit IRC | 23:52 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!
