| *** _cjones_ has quit IRC | 00:21 | |
| *** _cjones_ has joined #openstack-telemetry | 00:22 | |
| *** _cjones_ has quit IRC | 00:26 | |
| *** thorst has joined #openstack-telemetry | 00:31 | |
| *** thorst has quit IRC | 00:36 | |
| *** thorst has joined #openstack-telemetry | 00:41 | |
| *** gongysh has joined #openstack-telemetry | 00:45 | |
| *** ppiela has quit IRC | 00:45 | |
| *** gongysh has quit IRC | 00:49 | |
| *** iceyao has joined #openstack-telemetry | 00:50 | |
| *** iceyao has quit IRC | 00:54 | |
| *** thorst has quit IRC | 01:01 | |
| *** catintheroof has joined #openstack-telemetry | 01:04 | |
| *** sanfern has quit IRC | 01:36 | |
| *** iceyao has joined #openstack-telemetry | 01:43 | |
| *** catintheroof has quit IRC | 02:01 | |
| flwang | jd_: any luck you're around? | 02:31 |
|---|---|---|
| flwang | or any core member of telemetry team around? | 02:31 |
| flwang | i have a question about panko | 02:31 |
| flwang | about the publisher | 02:31 |
| flwang | i can see only a database.py under panko/publisher | 02:32 |
| flwang | but i can see many publisher driver under ceilometer/publisher/ | 02:32 |
| flwang | are they 'same' publisher? | 02:32 |
| *** rbak has joined #openstack-telemetry | 02:40 | |
| *** sanfern has joined #openstack-telemetry | 02:50 | |
| openstackgerrit | Hanxi Liu proposed openstack/ceilometer master: Support loading multiple meter definition files https://review.openstack.org/429590 | 02:54 |
| *** rbak has quit IRC | 02:59 | |
| *** gongysh has joined #openstack-telemetry | 03:04 | |
| *** dave-mccowan has joined #openstack-telemetry | 03:06 | |
| *** links has joined #openstack-telemetry | 03:08 | |
| *** gongysh has quit IRC | 03:16 | |
| *** thorst has joined #openstack-telemetry | 03:50 | |
| *** thorst has quit IRC | 03:50 | |
| *** rcernin has quit IRC | 03:50 | |
| *** sudipto has joined #openstack-telemetry | 04:10 | |
| *** sudipto_ has joined #openstack-telemetry | 04:10 | |
| *** dave-mccowan has quit IRC | 04:16 | |
| *** Gautam has joined #openstack-telemetry | 04:29 | |
| *** sudipto_ has quit IRC | 04:30 | |
| *** sudipto has quit IRC | 04:30 | |
| *** thorst has joined #openstack-telemetry | 04:32 | |
| *** thorst has quit IRC | 04:32 | |
| *** iceyao has quit IRC | 04:32 | |
| *** nadya has joined #openstack-telemetry | 04:51 | |
| *** nadya has quit IRC | 04:52 | |
| *** thorst has joined #openstack-telemetry | 05:03 | |
| *** iceyao has joined #openstack-telemetry | 05:13 | |
| *** thorst has quit IRC | 05:16 | |
| *** iceyao has quit IRC | 05:18 | |
| *** iceyao has joined #openstack-telemetry | 05:18 | |
| *** adriant_ has quit IRC | 05:20 | |
| *** Jack_Iv has joined #openstack-telemetry | 05:46 | |
| *** narasimha_SV has joined #openstack-telemetry | 06:27 | |
| *** Jack_Iv has quit IRC | 06:41 | |
| *** Jack_Iv has joined #openstack-telemetry | 06:42 | |
| *** Jack_Iv_ has joined #openstack-telemetry | 06:46 | |
| *** Jack_Iv has quit IRC | 06:47 | |
| *** zhurong has joined #openstack-telemetry | 06:48 | |
| *** zhurong has quit IRC | 06:53 | |
| *** sudipto has joined #openstack-telemetry | 06:56 | |
| *** sudipto_ has joined #openstack-telemetry | 06:56 | |
| *** sudipto_ has quit IRC | 07:00 | |
| *** sudipto has quit IRC | 07:00 | |
| *** rcernin has joined #openstack-telemetry | 07:17 | |
| *** tesseract has joined #openstack-telemetry | 07:18 | |
| *** iceyao has quit IRC | 07:21 | |
| *** Jack_Iv_ has quit IRC | 07:33 | |
| *** Jack_Iv has joined #openstack-telemetry | 07:34 | |
| *** Jack_Iv has quit IRC | 07:35 | |
| *** Jack_Iv has joined #openstack-telemetry | 07:35 | |
| *** gongysh has joined #openstack-telemetry | 07:41 | |
| *** iceyao has joined #openstack-telemetry | 07:45 | |
| *** nadya has joined #openstack-telemetry | 07:45 | |
| *** yprokule has joined #openstack-telemetry | 07:50 | |
| *** lhx__ has joined #openstack-telemetry | 07:55 | |
| *** Jack_Iv has quit IRC | 08:00 | |
| *** Jack_Iv has joined #openstack-telemetry | 08:01 | |
| jd_ | flwang: yes | 08:01 |
| *** shardy has joined #openstack-telemetry | 08:04 | |
| *** Jack_Iv has quit IRC | 08:06 | |
| openstackgerrit | Hanxi Liu proposed openstack/ceilometer master: Support loading multiple meter definition files https://review.openstack.org/429590 | 08:12 |
| *** pcaruana has joined #openstack-telemetry | 08:13 | |
| jd_ | sileht: so has nova fixed our gate? lol | 08:20 |
| sileht | jd_, maybe I haven't check yet | 08:21 |
| jd_ | i still see failure but i did not dig | 08:22 |
| sileht | jd_, no more nova ERROR at least | 08:22 |
| *** yprokule_ has joined #openstack-telemetry | 08:22 | |
| jd_ | ok cool | 08:22 |
| sileht | jd_, some are failling due to pscygo2 lib issue | 08:22 |
| jd_ | hm and tooz? | 08:24 |
| jd_ | recheck should be enough | 08:24 |
| *** yprokule has quit IRC | 08:24 | |
| *** yprokule_ is now known as yprokule | 08:24 | |
| sileht | jd_, yes | 08:24 |
| sileht | jd_, the integration job that fail is maybe due to the oslo.messaging threading issue | 08:26 |
| sileht | jd_, I will ask a release of oslo.messaging for pike with the fix | 08:26 |
| *** danpawlik has left #openstack-telemetry | 08:27 | |
| jd_ | wtf is that http://logs.openstack.org/23/439823/1/check/gate-ceilometer-dsvm-tempest-plugin-mongodb-ubuntu-xenial/0238a02/console.html#_2017-03-01_22_11_37_891989 | 08:27 |
| jd_ | it's in the ocata branch https://review.openstack.org/#/c/439823 | 08:28 |
| jd_ | sileht: is this your copy of tempest we need to cherry pick? | 08:28 |
| * sileht trows tempest by the windows, following by a dinosaur on top on it, to ensure it does not survive | 08:29 | |
| *** Jack_Iv has joined #openstack-telemetry | 08:31 | |
| sileht | jd_, I will backport https://review.openstack.org/#/c/431698/ | 08:40 |
| sileht | jd_, https://review.openstack.org/440212 | 08:43 |
| sileht | the openstackstatus bot is dead ? | 08:43 |
| sileht | wakeup openstackgerrit | 08:43 |
| *** sshnaidm|off is now known as sshnaidm | 08:46 | |
| *** yprokule_ has joined #openstack-telemetry | 09:03 | |
| *** shardy has quit IRC | 09:04 | |
| *** shardy has joined #openstack-telemetry | 09:04 | |
| *** yprokule has quit IRC | 09:04 | |
| *** yprokule_ is now known as yprokule | 09:04 | |
| *** eglynn has joined #openstack-telemetry | 09:11 | |
| *** amoralej|off is now known as amoralej | 09:13 | |
| *** thorst has joined #openstack-telemetry | 09:13 | |
| *** thorst has quit IRC | 09:18 | |
| *** nadya has quit IRC | 10:01 | |
| *** gongysh has quit IRC | 10:27 | |
| openstackgerrit | Hanxi Liu proposed openstack/ceilometer master: Support loading multiple meter definition files https://review.openstack.org/429590 | 10:28 |
| *** links has quit IRC | 10:31 | |
| *** cdent has joined #openstack-telemetry | 10:33 | |
| *** iceyao has quit IRC | 10:39 | |
| jd_ | sileht: looks like integration gate is really broken | 10:39 |
| jd_ | gtg | 10:39 |
| *** nadya has joined #openstack-telemetry | 10:40 | |
| *** sanfern has quit IRC | 10:44 | |
| *** links has joined #openstack-telemetry | 10:48 | |
| openstackgerrit | Mehdi Abaakouk (sileht) proposed openstack/ceilometer master: prepare future tempest breakage https://review.openstack.org/439414 | 10:49 |
| openstackgerrit | Mehdi Abaakouk (sileht) proposed openstack/ceilometer master: Don't run ceilometer-upgrade on unconfigured db https://review.openstack.org/439627 | 10:49 |
| *** sanfern has joined #openstack-telemetry | 10:54 | |
| *** Gautam_ has joined #openstack-telemetry | 10:57 | |
| *** shardy has quit IRC | 11:00 | |
| *** Gautam_ has quit IRC | 11:00 | |
| *** Gautam_ has joined #openstack-telemetry | 11:00 | |
| *** Gautam has quit IRC | 11:01 | |
| *** shardy has joined #openstack-telemetry | 11:01 | |
| *** Gautam has joined #openstack-telemetry | 11:01 | |
| *** Gautam has quit IRC | 11:05 | |
| openstackgerrit | Mehdi Abaakouk (sileht) proposed openstack/ceilometer master: prepare future tempest breakage https://review.openstack.org/439414 | 11:07 |
| openstackgerrit | Mehdi Abaakouk (sileht) proposed openstack/ceilometer master: Don't run ceilometer-upgrade on unconfigured db https://review.openstack.org/439627 | 11:07 |
| *** Jack_Iv has quit IRC | 11:13 | |
| *** Jack_Iv has joined #openstack-telemetry | 11:13 | |
| *** thorst has joined #openstack-telemetry | 11:14 | |
| *** sanfern has quit IRC | 11:15 | |
| *** nadya_ has joined #openstack-telemetry | 11:18 | |
| *** nadya has quit IRC | 11:18 | |
| *** Jack_Iv has quit IRC | 11:18 | |
| *** thorst has quit IRC | 11:19 | |
| *** Gautam has joined #openstack-telemetry | 11:30 | |
| *** Gautam has quit IRC | 11:34 | |
| *** eglynn has quit IRC | 11:43 | |
| lhx__ | seems issue related to oslo.messaging, but not sure | 11:46 |
| *** links has quit IRC | 11:51 | |
| *** yassine has quit IRC | 11:52 | |
| *** yassine has joined #openstack-telemetry | 11:58 | |
| *** narasimha_SV has quit IRC | 12:02 | |
| *** links has joined #openstack-telemetry | 12:04 | |
| *** gkadam has joined #openstack-telemetry | 12:13 | |
| *** shardy is now known as shardy_lunch | 12:21 | |
| *** thorst has joined #openstack-telemetry | 12:45 | |
| *** gkadam has quit IRC | 12:46 | |
| *** catintheroof has joined #openstack-telemetry | 12:56 | |
| *** shardy_lunch is now known as shardy | 12:58 | |
| *** sanfern has joined #openstack-telemetry | 13:02 | |
| *** zhangguoqing has joined #openstack-telemetry | 13:05 | |
| *** nadya_ has quit IRC | 13:08 | |
| *** Gautam has joined #openstack-telemetry | 13:09 | |
| *** lhx__ has quit IRC | 13:18 | |
| jd_ | what how? | 13:19 |
| jd_ | sileht: clue? | 13:19 |
| sileht | jd_, a link ? | 13:21 |
| jd_ | sileht: https://review.openstack.org/#/c/439156/2 | 13:21 |
| jd_ | all integration jobs are dead | 13:21 |
| sileht | jd_, Looks like nova returns an empty server list...d | 13:22 |
| jd_ | sileht: indeed, but why … | 13:22 |
| jd_ | is there really no server running? | 13:22 |
| sileht | jd_, the devstack patch yesterday should have fixed this | 13:22 |
| jd_ | is this a plot against telemetry? | 13:23 |
| sileht | jd_, yes because gnocchi have the metrics and heat checks creation of resources and it returns OK | 13:23 |
| sileht | jd_, even "openstack server list" is empty... | 13:24 |
| jd_ | is CloudStack still a thing? asking for a friend | 13:24 |
| openstackgerrit | Mehdi Abaakouk (sileht) proposed openstack/ceilometer master: DNM: test https://review.openstack.org/440554 | 13:30 |
| *** cdent has quit IRC | 13:31 | |
| *** gordc has joined #openstack-telemetry | 13:33 | |
| sileht | jd_, openstack server list is empty, but openstack server show XXXX works ... | 13:35 |
| *** chlong_ has joined #openstack-telemetry | 13:40 | |
| jd_ | sileht: hilarious | 13:49 |
| sileht | jd_, I'm writting a ML mail | 13:50 |
| jd_ | sigh | 13:50 |
| sileht | jd_, I think their break something with policies or something | 13:50 |
| sileht | jd_, because we are the only project that create instances as admin | 13:50 |
| sileht | jd_, and the policy is perhaps confused | 13:50 |
| *** aolwas has quit IRC | 13:51 | |
| sileht | jd_, if https://review.openstack.org/440554 pass, that will confirm my though | 13:51 |
| jd_ | ok | 13:52 |
| jd_ | fingers crossed | 13:52 |
| sileht | jd_, I have seen some patches about converting policy yaml files in code | 13:52 |
| sileht | perhaps related, we will see | 13:53 |
| *** lhx__ has joined #openstack-telemetry | 13:53 | |
| * sileht just quits nova 'git log', after seeing 'Merge "vomiting"' | 13:56 | |
| jd_ | LOL | 13:59 |
| *** nadya has joined #openstack-telemetry | 14:02 | |
| *** cdent has joined #openstack-telemetry | 14:03 | |
| openstackgerrit | Rico Lin proposed openstack/python-pankoclient master: [Fix gate]Update test requirement https://review.openstack.org/440574 | 14:06 |
| *** Jack_Iv has joined #openstack-telemetry | 14:07 | |
| *** fguillot has joined #openstack-telemetry | 14:12 | |
| *** links has quit IRC | 14:12 | |
| sileht | jd_, funny DEBUG message: | 14:15 |
| sileht | nova.osapi_compute.wsgi.server [req-e44f5adf-519c-4b52-ad8a-845cd5a0f30a admin admin] (11780) timed out ('104.130.199.12', 32778) process_request /usr/local/lib/python2.7/dist-packages/eventlet/wsgi.py:724 | 14:15 |
| jd_ | did you plug your hard drive to the software? | 14:15 |
| sileht | it occurs when we list instances with gabbi | 14:16 |
| sileht | while we have : | 14:16 |
| sileht | (11780) accepted ('104.130.199.12', 32778) server /usr/local/lib/python2.7/dist-packages/eventlet/wsgi.py:868 | 14:16 |
| sileht | when the instances are list via Ceilometer pollsters | 14:16 |
| cdent | sileht: is that gabbi with wsgi-intercept or live? | 14:17 |
| *** nadya has quit IRC | 14:20 | |
| sileht | cdent, live | 14:20 |
| cdent | I guess I should have known that since you're talking to nova there. | 14:21 |
| * cdent needs to wake up | 14:21 | |
| sileht | https://github.com/openstack/ceilometer/blob/master/ceilometer/tests/integration/gabbi/gabbits-live/autoscaling.yaml | 14:21 |
| * jd_ slaps cdent | 14:22 | |
| jd_ | always glad to help | 14:22 |
| cdent | ow! I have a very sunburnt face. | 14:22 |
| jd_ | too much beach | 14:22 |
| cdent | yeah, too much beach with too much wind so I didn't realize I was roasting | 14:22 |
| openstackgerrit | Béla Vancsics proposed openstack/ceilometer master: Remove unused variable https://review.openstack.org/439471 | 14:23 |
| *** ppiela has joined #openstack-telemetry | 14:26 | |
| openstackgerrit | Béla Vancsics proposed openstack/ceilometer master: Remove unused parameter https://review.openstack.org/439469 | 14:32 |
| EmilienM | I just noticed tripleo jobs don't run anymore in telemetry project | 14:33 |
| EmilienM | someone removed them? | 14:33 |
| EmilienM | I think I found why, I'm working on it | 14:34 |
| *** jmlowe has quit IRC | 14:46 | |
| sileht | jd_, in fact the call works: | 14:55 |
| sileht | http://logs.openstack.org/56/439156/2/check/gate-telemetry-dsvm-integration-gnocchi-ubuntu-xenial/d4a6c69/logs/screen-n-api.txt.gz#_2017-03-02_09_16_25_647 | 14:55 |
| sileht | and some minutes later it doesn't: | 14:55 |
| sileht | http://logs.openstack.org/56/439156/2/check/gate-telemetry-dsvm-integration-gnocchi-ubuntu-xenial/d4a6c69/logs/screen-n-api.txt.gz#_2017-03-02_09_16_44_349 | 14:55 |
| sileht | len 323 is when the list is empty | 14:56 |
| jd_ | sileht: -_- | 14:58 |
| jd_ | sileht: the database has been erased in the meantime :P | 14:58 |
| sileht | jd_, we delete an instance | 14:59 |
| sileht | jd_, I see only one deletion on nova-compute, so we don't erase everything :) | 14:59 |
| jd_ | a good old DELETE with a missing WHERE :P | 14:59 |
| sileht | ahah | 14:59 |
| *** jmlowe has joined #openstack-telemetry | 15:17 | |
| *** Gautam has quit IRC | 15:20 | |
| *** rbak has joined #openstack-telemetry | 15:21 | |
| *** dave-mccowan has joined #openstack-telemetry | 15:26 | |
| *** cdent has quit IRC | 15:27 | |
| sileht | jd_, nova cores are looking around a global var called CELLS for our issue | 15:28 |
| *** zhangguoqing has quit IRC | 15:28 | |
| * sileht opens the pop corn | 15:28 | |
| sileht | nova have 7 years and their still don't get why global are wrong... | 15:29 |
| jd_ | lol | 15:31 |
| jd_ | you still need parents at 7 | 15:31 |
| sileht | lol | 15:31 |
| sileht | jd_, the good new is mriedem is working hard for us, thx to him | 15:35 |
| jd_ | yeah i'm glad they're taking that seriously :) | 15:36 |
| sileht | jd_, the issue around using global seems to be confirmed: https://bugs.launchpad.net/nova/+bug/1669473/comments/1 | 15:44 |
| openstack | Launchpad bug 1669473 in OpenStack Compute (nova) "openstack server list returns an empty list" [Undecided,New] | 15:44 |
| sileht | jd_, they have to add more debug to nova to continue, unfortunatly | 15:44 |
| *** nicodemus_ has joined #openstack-telemetry | 15:45 | |
| jd_ | :( | 15:45 |
| nicodemus_ | hello | 15:45 |
| jd_ | hi nicodemus_ | 15:45 |
| nicodemus_ | I have a question regarding a feature we're thinking here, and wanted to ask you guys to see what you think | 15:46 |
| nicodemus_ | If I add custom metadata to a certain instance, how can that be updated in the resource in an automatic fashion? | 15:48 |
| sileht | 26 | 15:48 |
| sileht | oups | 15:48 |
| sileht | jd_, concurrency 101, doesn't lock to protect concurency update of global: https://github.com/openstack/nova/blob/master/nova/compute/api.py#L2524-L2528 | 15:51 |
| sileht | doesn't/don't use | 15:51 |
| sileht | jd_, one more nova bug root cause discover by telemetry \o/ | 15:53 |
| nicodemus_ | To elaborate: now we can create out custom resource types, and then map our custom metadata in gnocchi_resources.yaml. If tomorrow a sample contains some new extra metadata, unless we update by hand the resource-type add it on gnocchi_resources.yaml that extra metadata is ignored. Is there any way/any plan to allow adding this new metadata dynamically? | 15:55 |
| *** cdent has joined #openstack-telemetry | 15:57 | |
| gordc | nicodemus_: i don't think we want that. seems you want document storage capabilities if you want to blindly capture metadata... i don't think that's gnocchi's usecase | 15:58 |
| gordc | (my opinion) | 15:58 |
| *** rcernin has quit IRC | 15:59 | |
| *** yprokule has quit IRC | 16:00 | |
| *** chlong_ has quit IRC | 16:01 | |
| *** joadavis_ has joined #openstack-telemetry | 16:01 | |
| *** chlong__ has joined #openstack-telemetry | 16:01 | |
| *** sergio_ has joined #openstack-telemetry | 16:01 | |
| *** sergio_ is now known as Guest17596 | 16:01 | |
| gordc | jd_: sileht: was there a reason we made 'delete metric' async? | 16:04 |
| nicodemus_ | gordc: our thinking behind my inquiry is that maybe some instances have extra metadata, for example each project adds tags to instances like the type of application its running | 16:07 |
| nicodemus_ | later on, a user wishes to aggregate metrics from all instances that are running eg. a database, for that we need to use the tag | 16:08 |
| sileht | gordc, why not :p | 16:09 |
| nicodemus_ | as it is now, we should update the resource-type and edit the gnocchi_Resources.yaml by hand | 16:09 |
| gordc | sileht: :) | 16:09 |
| sileht | gordc, releasing the lock and compute another metric | 16:09 |
| gordc | sileht: i want to move this code somewhere esle https://github.com/openstack/gnocchi/blob/master/gnocchi/storage/_carbonara.py#L440-L454 | 16:10 |
| nicodemus_ | that is nowadays a normal behavior, something that can be done in gnocchi (and we are doing it currently) | 16:10 |
| gordc | nicodemus_: yeah, so it seems like you want somethin like how we capture server_group right now. it's a user_metadata field that may be added to nova instances by heat | 16:11 |
| *** nadya has joined #openstack-telemetry | 16:11 | |
| openstackgerrit | Mehdi Abaakouk (sileht) proposed openstack/gnocchi master: Nova testing https://review.openstack.org/440657 | 16:11 |
| gordc | nicodemus_: my hesitation is once you start allowing free capturing of all metadata, people will just stop being explicit about what they capture because it's easier, and then they run into issue of db is now too big because it's capturing a lot of noise. | 16:12 |
| *** rbak_ has joined #openstack-telemetry | 16:12 | |
| *** rbak has quit IRC | 16:12 | |
| gordc | it's not the only issue but it was one of the issues why old legacy ceilometer grew to be an issue. | 16:13 |
| nicodemus_ | what we wonder now is if that could be automatic, like allowing a certain limit of extra keys but those extra keys to be added "automatically" to the resource_type and not having to be something fixed and/or something that requires manual intervention | 16:13 |
| nicodemus_ | yeah, allowing anyone to add any amount of extra keys is overkill | 16:13 |
| nicodemus_ | but perhaps adding some configuration option in order to limit the amount of extra keys | 16:14 |
| gordc | nicodemus_: possibly. but then you have scenario where you have no idea which keys will get accepted... just that whatever keys are accepted were the first x | 16:15 |
| nicodemus_ | gordc, you think that's something undesirable? Because if you want to control specifically what metadata to allow, you can always put extra_keys=false ans just accept the metadata defined in gnocchi_resources.yaml | 16:17 |
| sileht | gordc, go ahead | 16:17 |
| nicodemus_ | we're thinking this as a way to allow some extra (let's say "random") metadata without needing to update everything by hand | 16:18 |
| gordc | nicodemus_: i think it's not safe if you have a limit... you won't really know what metadata you're capturing unless you are explicit. | 16:21 |
| gordc | nicodemus_: understood that it's easier... i don't think it's safe though. *shrugs* | 16:22 |
| *** nadya has quit IRC | 16:22 | |
| gordc | if the limit is too small, you might not get what you want. if too big, your db is collecting junk... i don't know. it's too difficult for me to think about :) | 16:24 |
| nicodemus_ | gordc, that's the thing. The metadata that I care is all explicit in gnocchi_resources.yaml. The extra metadata is giving the user the possibility to add some tags | 16:26 |
| nicodemus_ | the possibility and the responsibility of managing his or her extra keys, so if the user knows that it has a 5 extra keys limit, if he added 6 and is missing one, he or she has to remove some old key for the new one to show up | 16:27 |
| gordc | nicodemus_: so you'd put metadata you want in gnocchi_resources.yaml? you just don't want to update the resource-type manually? | 16:28 |
| nicodemus_ | and then he or she could later group the resources by whatever criteria they want | 16:28 |
| gordc | if you remove one from gnocchi_resources.yaml, does that mean it'd get deleted from resource-type? | 16:29 |
| jd_ | sileht: I am going to ask that we merge Telemetry and QA then | 16:30 |
| sileht | jd_, I will add a new mandatory pep8 checks '/global.*/' -> failure | 16:31 |
| jd_ | gordc: yeah it's async on purpose, why? | 16:32 |
| jd_ | sileht: cfg.CONF? :p | 16:32 |
| gordc | jd_: i want to move this https://github.com/openstack/gnocchi/blob/master/gnocchi/storage/_carbonara.py#L440-L454 | 16:32 |
| jd_ | nicodemus_: you can have "metadata" if you just accept to add a string(256) as a field as not ever do any query with it… then sure | 16:33 |
| gordc | because if we create multiple bucket, i might know which bucket they will be in. | 16:33 |
| jd_ | gordc: nicodemus_ : however i think a nice feature would be to have ceilometer change/create the resource types in Gnocchi by reading what's in gnocchi_resources.yaml :) | 16:34 |
| nicodemus_ | the idea is to have a resource-type with 10 metadata items that are all explicit in gnocchi_resources.yaml, metadata items that we need and we define as required=true in the resource-type. Besides that, if a user wishes to add some other type of metadata (let's say app_type) to the instances on his project (because he later wants to get all metrics from the apache webservers), that extra metadata will be ignored as it is | 16:34 |
| nicodemus_ | . For that metadata to appear in the resource, we need to add it by hand to the gnocchi_Resources.yaml and to the resource-type. | 16:34 |
| gordc | jd_: does the resource-type delete attributes when removed from gnocchi_resources? | 16:34 |
| jd_ | gordc: I'm not following what the problem is :( can you give me an example? | 16:35 |
| jd_ | gordc: well it does not and that's the operator to do it :p | 16:35 |
| gordc | jd_: which problem? metadata or buckets? ) | 16:35 |
| jd_ | gordc: buckets lol | 16:35 |
| jd_ | nicodemus_: well then just add an extra "metadata" field and let the user put whatever it wants, now? | 16:35 |
| gordc | :) | 16:35 |
| jd_ | s/now/no/ | 16:35 |
| jd_ | this channel is almost always silent and suddenly there are 3 conversations at the same time | 16:36 |
| jd_ | get a schedule ffs | 16:36 |
| jd_ | :P | 16:36 |
| nicodemus_ | jd_, heheh sorry about that | 16:36 |
| nicodemus_ | jd_, yeah | 16:36 |
| gordc | jd_: in the code i linked, if the metric is removed from indexer, i cannot figure out which bucket the metric's unprocessed measures are in. | 16:37 |
| nicodemus_ | but it happening without having to alter it by hand | 16:37 |
| jd_ | gordc: it's not removed from the indexer, it's "marked for removal" so you just have to hash the metric id to get the bucket, no? | 16:37 |
| gordc | if i move the code to delete unprocessed measures when metric is removed from indexer, it might keep getting scheduled until cleanup process runs | 16:38 |
| gordc | yes, but the cleanup process is different from scheduler process. | 16:38 |
| gordc | so in theory, it could cleanup, and then schedule unprocessed | 16:38 |
| jd_ | oh you can do that to I don't think it's a problem | 16:39 |
| nicodemus_ | suddenly a user ads a key-value to an instance, ceilometer picks it up and produces a sample with this extra key-value. the dispatcher sees that extra_keys=true in the config and extra_leys_limit=10 is not reached yet, the dispatcher updates the resource-type with the new metadata (with required=false) and pushes it | 16:39 |
| nicodemus_ | does that make any sense? | 16:39 |
| gordc | jd_: yeah, that's what i was thinking. i guess there's an issue if cleanup doesn't run for a long time, then we might have a bunch of unprocessed measures that keep getting scheduled and skipped. | 16:40 |
| jd_ | gordc: I think this code predates the metricd job partitinong we have right now | 16:40 |
| jd_ | gordc: yeah, I don't think it's nor will be a big deal | 16:40 |
| gordc | jd_: kk, i'll move it and we can fix it if someone asks 'wtf'. | 16:41 |
| jd_ | nicodemus_: as long as these keys/values are stored in a single field, I don't see a problem even right now | 16:41 |
| jd_ | nicodemus_: the problem is if you want to create a custom field on resource type for each key key… | 16:41 |
| jd_ | nicodemus_: you'd even need different resource types | 16:41 |
| *** jd_ has left #openstack-telemetry | 16:42 | |
| *** jd_ has joined #openstack-telemetry | 16:42 | |
| jd_ | gordc: 👍 | 16:42 |
| *** pcaruana has quit IRC | 16:42 | |
| nicodemus_ | jd_, yeah, we'd imagine creating the "instance" resource-type for project A, a separate "instance" resource-type for project B and so on | 16:45 |
| nicodemus_ | so that each project has its own extra-keys not shared with snother resource-type for another project | 16:45 |
| jd_ | this does not sound like a good design | 16:45 |
| jd_ | I think what'd be more interesting nicodemus_ is to explain why in the first place you want to do that, i.e. what the root problem is, what do you want to solve | 16:46 |
| jd_ | which is at least is not clear to me right now :) | 16:46 |
| jd_ | so what are theses key/values metadata for? | 16:47 |
| *** _cjones_ has joined #openstack-telemetry | 16:49 | |
| *** _cjones_ has quit IRC | 16:49 | |
| nicodemus_ | jd_, the scenario we're thinking is the following: you have two different openstack deployments independent from one another. The user from deploy A launches each instance with "app_type" metadata (that is not defined in gnocchi_resources.yaml nor in the resource_type. The user from deploy B launches instances with some other, different metadata (let's say "day of the week" just to say something). | 16:50 |
| *** _cjones_ has joined #openstack-telemetry | 16:50 | |
| nicodemus_ | each customer makes use of their extended attributes for whatever reason, and user A wishes to get all metrics from resources that have app_type=webserver | 16:51 |
| nicodemus_ | next week, user A suddenly wishes to add another key, let's say "distro" that refers to ubuntu/centos/slack/etc | 16:52 |
| nicodemus_ | each of these extended attributes /extra keys, we would need to add to the resource-type and to the gnocchi_resources.yaml file | 16:52 |
| *** jmlowe has quit IRC | 16:53 | |
| nicodemus_ | I think in this scenario, it would be better just to alow user A to define whatever extra key he wants, and simply accept it from gnocchi's side | 16:53 |
| nicodemus_ | with a configurable limit | 16:53 |
| jd_ | you again describes a solution but not the problem you want to solve :) | 16:53 |
| jd_ | I still don't know why you need these key/values | 16:53 |
| *** jmlowe has joined #openstack-telemetry | 16:54 | |
| jd_ | *what* are you trying to solve? :) | 16:54 |
| jd_ | now *how* | 16:54 |
| nicodemus_ | the user needs it. The user wishes to see all metrics from webservers | 16:54 |
| jd_ | so the user has to store a list of server that are webservers ;) | 16:55 |
| nicodemus_ | how can he get all webservers? by grouping together all resources that have the "webservers" app_type key/value in the resource | 16:55 |
| jd_ | nicodemus_: how can the user identify a "webserver" in Nova? | 16:57 |
| nicodemus_ | because the instance has a tag added by the user when he launched it | 16:57 |
| nicodemus_ | if not, suppose the user is using some other cloud provider that is not openstack-based and he wishes to push metrics to gnocchi | 16:59 |
| openstackgerrit | gordon chung proposed openstack/gnocchi master: delete unprocessed measures on expunge https://review.openstack.org/440688 | 17:00 |
| openstackgerrit | Darren Hague proposed openstack/ceilometer master: Enable Basic and certificate authentication for http publisher & dispatcher targets https://review.openstack.org/440704 | 17:07 |
| openstackgerrit | Julien Danjou proposed openstack/gnocchi master: carbonara: remove misleading comment https://review.openstack.org/440707 | 17:09 |
| jd_ | nicodemus_: so nova has server tag? I should look into that | 17:09 |
| jd_ | nicodemus_: ok so Nova tags are just keys, not keys/values, right? | 17:11 |
| nicodemus_ | jd_, yes | 17:11 |
| nicodemus_ | perhaps I misspoke when I said key/value | 17:11 |
| nicodemus_ | its just tags | 17:12 |
| jd_ | yeah it's a bit different indeed | 17:12 |
| jd_ | because the list of tag can be translated to e.g. "webserver,production,foobar" | 17:12 |
| *** dave-mcc_ has joined #openstack-telemetry | 17:12 | |
| jd_ | so then you can store that as a string in Gnocchi and use it for requesting the resources | 17:12 |
| jd_ | you use store ",webserver,production,foobar," and then do a search with "like %,webserver,%" for example | 17:13 |
| *** dave-mccowan has quit IRC | 17:13 | |
| jd_ | and it works :) | 17:13 |
| nicodemus_ | but each tag should be translated to a separate metadata item in the resource, if not... how can we aggregate the resources? | 17:14 |
| jd_ | POST /v1/aggregation/resource/instance/metric/cpu.util with body: {"like": {"%,webserver,%"}} | 17:15 |
| jd_ | IIUC what you want | 17:15 |
| jd_ | POST /v1/aggregation/resource/instance/metric/cpu.util with body: {"like": {"tags": {"%,webserver,%"}}} | 17:15 |
| *** dave-mccowan has joined #openstack-telemetry | 17:15 | |
| jd_ | sorry :) | 17:15 |
| jd_ | we could also have something more robust with a type "array of string" but that would only work with PostgreSQL unfortunately | 17:16 |
| jd_ | and I know many folks like this good ol' MySQL | 17:16 |
| nicodemus_ | jd_, me for example :) | 17:16 |
| *** dave-mcc_ has quit IRC | 17:17 | |
| jd_ | yeah so forget about that, you sneaky little MySQL user! | 17:17 |
| jd_ | :P | 17:17 |
| jd_ | but the like trick should work | 17:17 |
| jd_ | how does that sound? | 17:17 |
| jd_ | this could be implemented upstream in Ceilometer I think | 17:18 |
| jd_ | if Nova gives us the list of tags, we can add it to gnocchi_resources.yaml | 17:18 |
| nicodemus_ | that could work... though if those were only tags, there'd be no group_by | 17:23 |
| *** nadya has joined #openstack-telemetry | 17:23 | |
| nicodemus_ | I mean, the resource will have a collection of tags, but I cannot know from all those tags which one is the app_type, which tag is the production_status, etc | 17:23 |
| jd_ | nicodemus_: ah right | 17:23 |
| *** aolwas has joined #openstack-telemetry | 17:23 | |
| *** lhx__ has quit IRC | 17:27 | |
| jd_ | nicodemus_: it's really an interesting challenge but it gets down to be limited by MySQL TBH | 17:36 |
| *** tesseract has quit IRC | 17:52 | |
| nicodemus_ | jd_, ok... we'll think it through a bit more | 17:52 |
| nicodemus_ | thanks jd_ & gordc ! | 17:52 |
| *** rbak_ has quit IRC | 17:58 | |
| catintheroof | jd_: to use gnocchi file backend, do i have some special "pip" requirement ? | 18:19 |
| stevelle | nicodemus_: is that really how tags should work anyway? | 18:20 |
| nicodemus_ | stevelle, the use cases we've been discussing included such considerations | 18:21 |
| *** nadya has quit IRC | 18:21 | |
| *** shardy has quit IRC | 18:21 | |
| nicodemus_ | meaning what would happen if one day we start gathering samples from a source where the resources should contain its tags or associated key/values | 18:22 |
| stevelle | I would assume tags would be named so you don't need labels or keys to them e.g. tags="credits_app,is_production,financial_group" | 18:22 |
| stevelle | so just having support for tags would do it | 18:22 |
| stevelle | that said, I have to also wonder what gnocchi should do when tags change on a resource | 18:24 |
| stevelle | last writer wins would probably be good enough, but should validate | 18:25 |
| stevelle | tags behave different from key/value, maybe implement as a different case? | 18:26 |
| *** amoralej is now known as amoralej|off | 18:44 | |
| nicodemus_ | stevelle, got sidetracked. The tags case shouldn't be difficul, because I can allow on all resource_types n extra metadata called "tags", and in there I can write the list of all tags the resource has associated | 18:54 |
| nicodemus_ | difficul/difficult | 18:54 |
| nicodemus_ | a different story would be if I wanted to add each tag as a key/value | 18:55 |
| *** _cjones_ has quit IRC | 19:01 | |
| *** _cjones_ has joined #openstack-telemetry | 19:01 | |
| *** _cjones_ has quit IRC | 19:06 | |
| *** ddyer2 has quit IRC | 19:06 | |
| *** thorst has quit IRC | 19:09 | |
| *** thorst has joined #openstack-telemetry | 19:10 | |
| *** ddyer has joined #openstack-telemetry | 19:11 | |
| openstackgerrit | Matt Riedemann proposed openstack/gnocchi master: Nova testing https://review.openstack.org/440657 | 19:11 |
| *** thorst has quit IRC | 19:14 | |
| *** sshnaidm is now known as sshnaidm|off | 19:16 | |
| *** cdent has quit IRC | 19:36 | |
| *** rbak_ has joined #openstack-telemetry | 19:47 | |
| openstackgerrit | gordon chung proposed openstack/gnocchi master: delete unprocessed measures on expunge https://review.openstack.org/440688 | 19:51 |
| *** rbak_ has quit IRC | 20:04 | |
| *** yassine has quit IRC | 20:06 | |
| *** narasimha_SV has joined #openstack-telemetry | 20:15 | |
| narasimha_SV | i am using ceilometer collectd plugin | 20:15 |
| narasimha_SV | as collectd starts I am getting metrics like this | 20:16 |
| narasimha_SV | http://paste.openstack.org/show/601182/ | 20:16 |
| narasimha_SV | it is missing resource id | 20:16 |
| narasimha_SV | is this the way it is expected ? | 20:16 |
| nicodemus_ | jd_, gordc, as a follow-up to our earlier talk: I was checking the key/value vs tag approach, and be it amazon, openstack and even azure you can add key/values to an instance (not simply a tag) | 20:25 |
| nicodemus_ | I was also looking at some history and found this oldie: https://bugs.launchpad.net/gnocchi/+bug/1513623 | 20:27 |
| openstack | Launchpad bug 1513623 in Gnocchi "dynamic metadata on gnocchi resources" [Wishlist,Fix released] - Assigned to Mehdi Abaakouk (sileht) | 20:27 |
| *** narasimha_SV has quit IRC | 20:28 | |
| nicodemus_ | our idea behind that one bug was to replicate what ceilometer did with mongodb, allowing the gnocchi dispatcher to update a certain resource-type when the sample has more key/values than the ones in gnocchi_resources.yaml | 20:30 |
| gordc | nicodemus_: i wasn't reading what you/jd_ were typing earlier. | 20:33 |
| gordc | nicodemus_: i'm pretty sure the 'dynamic' part of that bug was being allowed to modify resource type after creation. now to make gnocchi arbitrarily create attributes. | 20:34 |
| *** thorst has joined #openstack-telemetry | 20:35 | |
| gordc | i'll be honest, i'm probably not going to add your request since i've my concerns (and i don't need it personally). | 20:36 |
| gordc | could you not create a script/service that adds it to the resource using existing methods? so when you add a k/v to instance, it'll add manually trigger add in gnocchi? | 20:37 |
| nicodemus_ | gordc, in short: I wanted to discuss this possibility with you and see if you considered something worthy to add upstream. My guess is we could do some hacking on the dispatcher so that this happens without manual intervention | 20:38 |
| flwang | gordc: jd_: question for panko | 20:39 |
| flwang | (15:31:42) flwang: i have a question about panko | 20:39 |
| flwang | (15:31:51) flwang: about the publisher | 20:39 |
| flwang | (15:32:11) flwang: i can see only a database.py under panko/publisher | 20:39 |
| flwang | (15:32:26) flwang: but i can see many publisher driver under ceilometer/publisher/ | 20:39 |
| flwang | (15:32:45) flwang: are they 'same' publisher? | 20:39 |
| gordc | brb, i have to make a call. | 20:39 |
| *** narasimha_SV has joined #openstack-telemetry | 20:47 | |
| gordc | nicodemus_: yeah, i think it's better to have something trigger an attribute add... instead of just creating some super ability in gnocchi itself. | 20:48 |
| gordc | nicodemus_: maybe the other folks have different opinion | 20:48 |
| narasimha_SV | http://paste.openstack.org/show/601182/ will collectd-ceilometer-plugin metrics are expected as this | 20:48 |
| narasimha_SV | because resource_id is missing here | 20:48 |
| gordc | flwang: sort of. the publisher in panko is basically ceilometer publish to panko db | 20:49 |
| gordc | the other publishers in ceilometer have nothign to do with panko (ie. notifier) so that's why you don't see it in panko | 20:50 |
| gordc | narasimha_SV: https://julien.danjou.info/blog/2017/gnocchi-collectd-setup | 20:50 |
| nicodemus_ | gordc, I believe gnocchi does have the super ability that allows through the API to update a resource-type... the extra lines of code should be in ceilometer (more specifically in the dispatcher) | 20:51 |
| gordc | well you can add it yes. but you have to explicitly add attributes. | 20:52 |
| nicodemus_ | I'll dig a little into it, I hope it's not that hard | 20:52 |
| gordc | i'll let you find out if its hard or not :) | 20:53 |
| *** yassine has joined #openstack-telemetry | 20:54 | |
| nicodemus_ | thanks gordc ! If I return crying some days later, it proved too hard for my skills :) | 20:54 |
| gordc | lol best of luck | 20:54 |
| nicodemus_ | ty | 20:54 |
| *** yassine has quit IRC | 20:58 | |
| *** adriant has joined #openstack-telemetry | 20:59 | |
| *** gordc has quit IRC | 21:05 | |
| *** Guest17596 has quit IRC | 21:07 | |
| *** gordc has joined #openstack-telemetry | 21:09 | |
| *** gordc has quit IRC | 21:09 | |
| *** tlian has joined #openstack-telemetry | 21:20 | |
| *** yassine has joined #openstack-telemetry | 21:21 | |
| *** narasimha_SV has quit IRC | 21:27 | |
| *** yassine has quit IRC | 21:28 | |
| *** tlian has quit IRC | 21:29 | |
| openstackgerrit | Emilien Macchi proposed openstack/ceilometer master: CI test - never merge https://review.openstack.org/440820 | 21:36 |
| catintheroof | jd_: to use gnocchi file backend, do i have some special "pip" requirement ? | 21:37 |
| *** nicodemus_ has quit IRC | 21:40 | |
| *** yassine has joined #openstack-telemetry | 21:43 | |
| *** dave-mccowan has quit IRC | 21:44 | |
| *** yassine has quit IRC | 21:44 | |
| *** gordc has joined #openstack-telemetry | 21:46 | |
| *** Jack_Iv has quit IRC | 21:50 | |
| *** Jack_Iv has joined #openstack-telemetry | 21:51 | |
| *** jmlowe has quit IRC | 21:52 | |
| jd_ | gordc: well the problem with nicodemus thing is also the perf IMHO… key/value in SQL… | 21:53 |
| gordc | jd_: yeah, that's also why i'm not doing it :) | 21:54 |
| *** anteaya has quit IRC | 21:54 | |
| jd_ | i like that tag idea but it won't scale | 21:54 |
| *** Jack_Iv has quit IRC | 21:55 | |
| * gordc didn't read what you folks were talking about | 21:56 | |
| *** fguillot has quit IRC | 22:05 | |
| *** anteaya has joined #openstack-telemetry | 22:07 | |
| *** thorst has quit IRC | 22:08 | |
| *** jmlowe has joined #openstack-telemetry | 22:11 | |
| *** thorst has joined #openstack-telemetry | 22:36 | |
| *** catintheroof has quit IRC | 22:40 | |
| *** thorst has quit IRC | 22:41 | |
| *** gordc has quit IRC | 23:11 | |
| *** thorst has joined #openstack-telemetry | 23:17 | |
| *** oomichi has quit IRC | 23:18 | |
| *** thorst has quit IRC | 23:19 | |
| *** oomichi has joined #openstack-telemetry | 23:21 | |
| *** yassine has joined #openstack-telemetry | 23:40 | |
| *** Jack_Iv has joined #openstack-telemetry | 23:52 | |
| *** _cjones_ has joined #openstack-telemetry | 23:55 | |
| *** _cjones_ has quit IRC | 23:55 | |
| *** _cjones_ has joined #openstack-telemetry | 23:55 | |
| *** Jack_Iv has quit IRC | 23:57 | |
| *** yassine has quit IRC | 23:59 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!