| *** rbak has quit IRC | 00:04 | |
| *** gordc has quit IRC | 00:06 | |
| *** tongli has quit IRC | 00:20 | |
| *** ljxiash has joined #openstack-telemetry | 00:28 | |
| *** ljxiash has quit IRC | 00:32 | |
| *** ljxiash has joined #openstack-telemetry | 00:36 | |
| *** ljxiash has quit IRC | 00:38 | |
| *** ljxiash has joined #openstack-telemetry | 00:39 | |
| *** ljxiash has quit IRC | 00:43 | |
| *** ljxiash has joined #openstack-telemetry | 00:52 | |
| *** ljxiash has quit IRC | 00:54 | |
| *** cheneydc has joined #openstack-telemetry | 00:55 | |
| *** mgarza has quit IRC | 01:10 | |
| *** ljxiash has joined #openstack-telemetry | 01:18 | |
| *** Liuqing has joined #openstack-telemetry | 01:34 | |
| *** david-lyle has quit IRC | 01:36 | |
| *** diogogmt has quit IRC | 01:48 | |
| *** guillaume_ has quit IRC | 02:03 | |
| *** prashantD_ has quit IRC | 02:08 | |
| *** diogogmt has joined #openstack-telemetry | 02:58 | |
| *** pradk has quit IRC | 02:59 | |
| *** pradk has joined #openstack-telemetry | 03:12 | |
| *** achatterjee has joined #openstack-telemetry | 03:17 | |
| openstackgerrit | Merged openstack/ceilometer: enable notification agent partitioning https://review.openstack.org/267808 | 03:21 |
|---|---|---|
| *** ljxiash has quit IRC | 04:07 | |
| *** david-lyle has joined #openstack-telemetry | 04:25 | |
| *** david-lyle has quit IRC | 04:25 | |
| *** mragupat has joined #openstack-telemetry | 04:33 | |
| *** ljxiash has joined #openstack-telemetry | 04:35 | |
| *** david-lyle has joined #openstack-telemetry | 04:38 | |
| *** guillaume_ has joined #openstack-telemetry | 05:04 | |
| *** ljxiash has quit IRC | 05:07 | |
| *** ljxiash has joined #openstack-telemetry | 05:08 | |
| *** ljxiash has quit IRC | 05:10 | |
| *** ljxiash has joined #openstack-telemetry | 05:10 | |
| *** vishwanathj has joined #openstack-telemetry | 05:15 | |
| *** vishwanathj has quit IRC | 05:16 | |
| *** yprokule has joined #openstack-telemetry | 05:30 | |
| *** vishwanathj has joined #openstack-telemetry | 05:33 | |
| *** vishwanathj has quit IRC | 05:34 | |
| *** gtt116_ has joined #openstack-telemetry | 06:13 | |
| *** gtt116__ has quit IRC | 06:16 | |
| *** vishwanathj has joined #openstack-telemetry | 06:20 | |
| *** vishwanathj has quit IRC | 06:21 | |
| *** gtt116_ has quit IRC | 06:23 | |
| *** gtt116 has joined #openstack-telemetry | 06:24 | |
| *** mragupat has quit IRC | 06:28 | |
| *** ifat_afek has joined #openstack-telemetry | 06:32 | |
| zigo | sileht: jd__: I have issues building gnocchiclient 2.1.0: http://paste.debian.net/370311/ | 06:55 |
| *** prashantD_ has joined #openstack-telemetry | 06:56 | |
| openstackgerrit | Ren Qiaowei proposed openstack/ceilometer: xenapi: support the session when xenserver is master https://review.openstack.org/215393 | 07:12 |
| *** guillaume_ has left #openstack-telemetry | 07:14 | |
| *** belmoreira has joined #openstack-telemetry | 07:15 | |
| *** guillaume_ has joined #openstack-telemetry | 07:15 | |
| *** rcernin has joined #openstack-telemetry | 07:28 | |
| *** thorst_ has joined #openstack-telemetry | 07:28 | |
| *** chlong_zzz is now known as chlong | 07:31 | |
| *** thorst_ has quit IRC | 07:32 | |
| *** prashantD_ has quit IRC | 07:42 | |
| *** cdent has joined #openstack-telemetry | 08:00 | |
| *** guillaume_ has quit IRC | 08:02 | |
| *** guillaume_ has joined #openstack-telemetry | 08:08 | |
| *** guillaume_ has quit IRC | 08:10 | |
| *** guillaume_ has joined #openstack-telemetry | 08:10 | |
| *** guillaume_ has quit IRC | 08:10 | |
| *** cdent has quit IRC | 08:20 | |
| *** shardy has joined #openstack-telemetry | 08:29 | |
| *** ifat_afek_ has joined #openstack-telemetry | 08:32 | |
| *** ifat_afek has quit IRC | 08:33 | |
| openstackgerrit | Xia Linjuan proposed openstack/python-aodhclient: Fix aodh client fails when command with the arg --time-constraint https://review.openstack.org/272946 | 08:46 |
| *** safchain has joined #openstack-telemetry | 08:48 | |
| *** jaypipes has joined #openstack-telemetry | 08:49 | |
| *** mattyw has joined #openstack-telemetry | 08:52 | |
| *** cdent has joined #openstack-telemetry | 08:56 | |
| *** thorst_ has joined #openstack-telemetry | 09:08 | |
| *** yassine__ has joined #openstack-telemetry | 09:20 | |
| *** efoley has joined #openstack-telemetry | 09:26 | |
| openstackgerrit | Mehdi Abaakouk (sileht) proposed openstack/gnocchi: Add some resource types tests https://review.openstack.org/270419 | 09:28 |
| openstackgerrit | Mehdi Abaakouk (sileht) proposed openstack/gnocchi: Implements resource attribute int and float https://review.openstack.org/270091 | 09:28 |
| openstackgerrit | Mehdi Abaakouk (sileht) proposed openstack/gnocchi: Implements resource attribute bool https://review.openstack.org/270418 | 09:28 |
| openstackgerrit | Mehdi Abaakouk (sileht) proposed openstack/gnocchi: Implements resource attribute uuid https://review.openstack.org/270090 | 09:28 |
| openstackgerrit | Mehdi Abaakouk (sileht) proposed openstack/gnocchi: Don't create Ceilometer resource types by default. https://review.openstack.org/270322 | 09:28 |
| openstackgerrit | Mehdi Abaakouk (sileht) proposed openstack/gnocchi: Move legacy Ceilometer resource into indexer. https://review.openstack.org/270266 | 09:28 |
| openstackgerrit | Mehdi Abaakouk (sileht) proposed openstack/gnocchi: Move resource type into their own sql table https://review.openstack.org/269843 | 09:28 |
| openstackgerrit | Mehdi Abaakouk (sileht) proposed openstack/gnocchi: Implements resource type CRUD. https://review.openstack.org/269844 | 09:28 |
| openstackgerrit | Mehdi Abaakouk (sileht) proposed openstack/gnocchi: Implements resource attribute string https://review.openstack.org/269888 | 09:28 |
| *** Liuqing has quit IRC | 09:33 | |
| *** permalac has joined #openstack-telemetry | 09:35 | |
| *** ifat_afek_ has quit IRC | 09:48 | |
| *** r-mibu has quit IRC | 09:49 | |
| *** r-mibu has joined #openstack-telemetry | 10:00 | |
| *** cheneydc has quit IRC | 10:05 | |
| openstackgerrit | Mehdi Abaakouk (sileht) proposed openstack/gnocchi: Remove custom api-paste.ini for testing https://review.openstack.org/267409 | 10:07 |
| openstackgerrit | Mehdi Abaakouk (sileht) proposed openstack/gnocchi: Renable lint for WebTest https://review.openstack.org/268989 | 10:07 |
| openstackgerrit | Mehdi Abaakouk (sileht) proposed openstack/gnocchi: Bypass the auth when listing Gnocchi versions https://review.openstack.org/266301 | 10:07 |
| *** _nadya_ has quit IRC | 10:30 | |
| *** _nadya_ has joined #openstack-telemetry | 10:30 | |
| jd__ | zigo: don't run the functional tests if gnocchi is not running | 10:36 |
| zigo | jd__: How is this related to "AttributeError: 'NoneType' object has no attribute 'endswith'" ? | 10:38 |
| zigo | (I agree, and will fix it though...) | 10:39 |
| jd__ | yes | 10:41 |
| zigo | jd__: It's looking like there's *no* unit tests at all... :/ | 10:41 |
| zigo | That's not nice at all... :( | 10:42 |
| *** ifat_afek has joined #openstack-telemetry | 10:42 | |
| zigo | jd__: Also, if you remember, I'm waiting on gnocchi 1.3.4 for Mitaka (so I could run gnocchi to run gnocchiclient tests... :P) | 10:44 |
| jd__ | I know, i'm working on it | 10:44 |
| zigo | Cheers ! | 10:45 |
| zigo | jd__: I've lied, there's *ONE* unit test ! :) | 10:50 |
| jd__ | yeah most of the tests are functionnals | 10:51 |
| *** efoley has quit IRC | 10:52 | |
| *** efoley has joined #openstack-telemetry | 10:58 | |
| openstackgerrit | Julien Danjou proposed openstack/gnocchi: Bump tooz version in requirements https://review.openstack.org/272740 | 11:01 |
| *** jd__ has quit IRC | 11:23 | |
| *** jd__ has joined #openstack-telemetry | 11:28 | |
| *** mattyw has quit IRC | 11:31 | |
| openstackgerrit | Xia Linjuan proposed openstack/python-aodhclient: Fix aodh client fails when command with the arg --time-constraint https://review.openstack.org/272946 | 11:43 |
| *** pcaruana has joined #openstack-telemetry | 11:44 | |
| *** jkraj has joined #openstack-telemetry | 11:48 | |
| openstackgerrit | Lianhao Lu proposed openstack/gnocchi: Added original_resource_id field into resource https://review.openstack.org/273008 | 12:08 |
| *** cheneydc has joined #openstack-telemetry | 12:17 | |
| *** mattyw has joined #openstack-telemetry | 12:20 | |
| *** efoley has quit IRC | 12:21 | |
| *** leitan has joined #openstack-telemetry | 12:24 | |
| *** efoley has joined #openstack-telemetry | 12:25 | |
| *** gordc has joined #openstack-telemetry | 12:29 | |
| openstackgerrit | Michael Krotscheck proposed openstack/aodh: Added CORS support to Aodh https://review.openstack.org/265342 | 12:33 |
| openstackgerrit | Michael Krotscheck proposed openstack/aodh: Use paste.ini in gabbi tests https://review.openstack.org/265330 | 12:33 |
| *** pcaruana has quit IRC | 12:37 | |
| openstackgerrit | Zi Lian Ji proposed openstack/python-aodhclient: Add some unit test cases https://review.openstack.org/273021 | 12:55 |
| openstackgerrit | Merged openstack/gnocchi: Bump tooz version in requirements https://review.openstack.org/272740 | 13:00 |
| zigo | jd__: How come gnocchi uses overtest, but it's not in global-requirements? Am I missing something? | 13:04 |
| zigo | jd__: And do you think I need it in Debian? | 13:06 |
| gordc | zigo: its used only to set up the functional test environment. | 13:07 |
| zigo | Right, I have read the README.rst | 13:08 |
| *** david-lyle has quit IRC | 13:08 | |
| *** chlong has quit IRC | 13:15 | |
| *** ifat_afek has quit IRC | 13:19 | |
| *** ifat_afek has joined #openstack-telemetry | 13:20 | |
| *** ljxiash has quit IRC | 13:23 | |
| *** _nadya_ has quit IRC | 13:27 | |
| *** chlong has joined #openstack-telemetry | 13:28 | |
| *** _nadya_ has joined #openstack-telemetry | 13:30 | |
| *** Ephur has joined #openstack-telemetry | 13:31 | |
| jd__ | zigo: we don't use global-requirements, and you only need it if you want to run the tests | 13:31 |
| jd__ | yeah you can't best RTFM | 13:32 |
| *** yassine__ has quit IRC | 13:38 | |
| openstackgerrit | gordon chung proposed openstack/ceilometer: rewriting history https://review.openstack.org/273049 | 13:44 |
| *** yassine__ has joined #openstack-telemetry | 13:45 | |
| *** julim has joined #openstack-telemetry | 13:48 | |
| openstackgerrit | Merged openstack/ceilometermiddleware: Updated from global requirements https://review.openstack.org/272781 | 13:53 |
| *** jkraj has quit IRC | 13:59 | |
| *** ljxiash has joined #openstack-telemetry | 14:01 | |
| *** diogogmt has quit IRC | 14:01 | |
| *** julim_ has joined #openstack-telemetry | 14:02 | |
| *** diogogmt has joined #openstack-telemetry | 14:03 | |
| *** ljxiash has quit IRC | 14:05 | |
| *** julim has quit IRC | 14:05 | |
| *** david-lyle has joined #openstack-telemetry | 14:17 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/ceilometer: Updated from global requirements https://review.openstack.org/272780 | 14:19 |
| *** diogogmt has quit IRC | 14:21 | |
| *** ifat_afek_ has joined #openstack-telemetry | 14:27 | |
| *** ifat_afek has quit IRC | 14:27 | |
| *** liusheng has quit IRC | 14:29 | |
| *** liusheng has joined #openstack-telemetry | 14:30 | |
| *** achatterjee has quit IRC | 14:34 | |
| *** ifat_afek has joined #openstack-telemetry | 14:35 | |
| *** ifat_afek_ has quit IRC | 14:38 | |
| openstackgerrit | gordon chung proposed openstack/ceilometer: rolling upgrades https://review.openstack.org/273081 | 14:43 |
| *** david-lyle has quit IRC | 14:44 | |
| *** thorst_ has quit IRC | 14:54 | |
| *** pradk_ has joined #openstack-telemetry | 14:56 | |
| *** rbak has joined #openstack-telemetry | 14:57 | |
| *** thorst_ has joined #openstack-telemetry | 14:59 | |
| *** ifat_afek has quit IRC | 14:59 | |
| *** thorst_ has quit IRC | 15:01 | |
| *** ifat_afek has joined #openstack-telemetry | 15:05 | |
| *** thorst_ has joined #openstack-telemetry | 15:14 | |
| *** dan-t has left #openstack-telemetry | 15:16 | |
| *** nicodemus_ has joined #openstack-telemetry | 15:16 | |
| *** dant has joined #openstack-telemetry | 15:17 | |
| *** dant has left #openstack-telemetry | 15:17 | |
| *** dant has joined #openstack-telemetry | 15:17 | |
| *** dant has quit IRC | 15:18 | |
| *** datravis has joined #openstack-telemetry | 15:18 | |
| pradk_ | jd__, i see https://review.openstack.org/#/c/272680/ is merged, do we have to do anything to get the tag bumped in master? i still see 1.3.1 | 15:19 |
| nicodemus_ | hello sileht, do you have a moment? | 15:25 |
| zigo | jd__: When I try to build the latest commit of Gnocchi, I get so many unit test errors of this kind: http://paste.openstack.org/show/485158/ | 15:25 |
| zigo | Any idea what's going on? | 15:25 |
| sileht | nicodemus_, sure | 15:27 |
| zigo | sileht: ^ | 15:28 |
| nicodemus_ | I have a question regarding https://review.openstack.org/#/c/272650/ | 15:28 |
| nicodemus_ | I pulled gnocchi master and applied the patch | 15:28 |
| sileht | nicodemus_, I'm guessing that stopping the process won't work ? | 15:29 |
| nicodemus_ | but I don't quite understand if the multiple threads that were being opened were on the host running gnocchi-metricd or in the ceph backend | 15:30 |
| nicodemus_ | I check the open thredas on two metricd hosts, with and without the patch, but the thread # is the same on both | 15:30 |
| jd__ | pradk_: release something, so 1.3.4 should be merged as soon as I tag it | 15:30 |
| sileht | zigo, I have no idea why you get that | 15:31 |
| zigo | :/ | 15:31 |
| jd__ | zigo: you need to set GNOCCHI_INDEXER_URL before running the tests | 15:31 |
| pradk_ | jd__, cool, thx | 15:31 |
| zigo | jd__: Ah, thanks, but what should I put in there? | 15:31 |
| *** vishwanathj has joined #openstack-telemetry | 15:32 | |
| sileht | nicodemus_, did get the message "Enable rados.run_in_thread monkeypatching" in your log ? | 15:32 |
| zigo | A MySQL db DSN ? | 15:32 |
| jd__ | zigo: check setup-test-env.sh for a clue; best thing is to put the URL of your indexer :p postgresql:// or something | 15:32 |
| nicodemus_ | yes, on the one patched it's the first message | 15:32 |
| zigo | jd__: Does this replaces GNOCCHI_TEST_INDEXER_UR ? | 15:33 |
| jd__ | zigo: yes | 15:33 |
| zigo | Ah... :P | 15:33 |
| zigo | jd__: You're doing this kind of jokes just on purpose to make my life harder, right? :) | 15:34 |
| openstackgerrit | Julien Danjou proposed openstack/gnocchi: storage: autoconfigure coordination_url https://review.openstack.org/270907 | 15:34 |
| jd__ | zigo: if only :) | 15:34 |
| *** mragupat has joined #openstack-telemetry | 15:35 | |
| sileht | nicodemus_, I will retry to test it tomorrow | 15:35 |
| jd__ | sileht: does that improve the performance when listing xattr? | 15:36 |
| zigo | Everything magically works now ... :) | 15:36 |
| * zigo cross fingers for the end of the unit test runs to go well... | 15:36 | |
| sileht | jd__, no idea yet | 15:39 |
| sileht | bbl | 15:39 |
| jd__ | sileht: hum so why is it useful for then? | 15:44 |
| zigo | Gosh... the setup.cfg is *NOT* to install configuration files, please don't do that !!! | 15:44 |
| zigo | E: python-gnocchi: FSSTND-dir-in-usr usr/etc/ | 15:44 |
| jd__ | zigo: right, that does not look the right dir | 15:46 |
| jd__ | i'll fix taht | 15:46 |
| zigo | jd__: Just don't attempt to use PBR to install config files, that's simply broken. | 15:46 |
| zigo | jd__: Monty Tailor says that any attempt to do that is a bug. | 15:46 |
| zigo | gnocchi_1.3.3+2016.01.27.git.e1339d77a9-1_amd64.build <--- No unit test errors, I'll push that to Experimental ! :) | 15:47 |
| zigo | \o/ | 15:47 |
| jd__ | if Monty says so then… | 15:47 |
| jd__ | i'll fix nonetheless :) | 15:47 |
| *** _nadya_ has quit IRC | 15:47 | |
| *** ifat_afek_ has joined #openstack-telemetry | 15:49 | |
| *** ifat_afek has quit IRC | 15:49 | |
| zigo | jd__: The thing is, there's no way to fix it. Either you add a /, then it's broken in the gate, either you don't, and then config files are going to /usr/etc when building the package. | 15:49 |
| zigo | IMO, the right way to fix it, is to teach PBR to be smarter... | 15:50 |
| jd__ | how 30 years of autotools did that, I wonder then | 15:50 |
| jd__ | zigo: surely | 15:50 |
| zigo | Yup. | 15:50 |
| *** mgarza_ has joined #openstack-telemetry | 16:03 | |
| nicodemus_ | sileht: just an FYI... this error is showing up on the metricd hosts that don't use threading: http://paste.openstack.org/show/485163/ | 16:08 |
| *** ljxiash has joined #openstack-telemetry | 16:09 | |
| *** rcernin has quit IRC | 16:09 | |
| *** datravis has quit IRC | 16:11 | |
| *** david-lyle has joined #openstack-telemetry | 16:14 | |
| *** datravis has joined #openstack-telemetry | 16:18 | |
| *** yprokule has quit IRC | 16:20 | |
| *** cheneydc has quit IRC | 16:25 | |
| *** ifat_afek_ has quit IRC | 16:25 | |
| *** ifat_afek has joined #openstack-telemetry | 16:25 | |
| *** ljxiash has quit IRC | 16:27 | |
| *** ifat_afek_ has joined #openstack-telemetry | 16:29 | |
| *** ifat_afek has quit IRC | 16:29 | |
| openstackgerrit | gordon chung proposed openstack/ceilometer: raise coordination error if not registered https://review.openstack.org/268292 | 16:31 |
| *** ifat_afek has joined #openstack-telemetry | 16:36 | |
| *** efoley has quit IRC | 16:37 | |
| *** belmoreira has quit IRC | 16:38 | |
| *** david-lyle has quit IRC | 16:38 | |
| *** ifat_afek_ has quit IRC | 16:40 | |
| *** mattyw has quit IRC | 16:40 | |
| *** david-lyle has joined #openstack-telemetry | 16:40 | |
| *** prashantD_ has joined #openstack-telemetry | 16:48 | |
| *** ifat_afek has quit IRC | 16:52 | |
| *** diogogmt has joined #openstack-telemetry | 16:56 | |
| *** jwcroppe has quit IRC | 17:03 | |
| *** jwcroppe_ has joined #openstack-telemetry | 17:05 | |
| *** jaypipes has quit IRC | 17:07 | |
| *** thorst_ has quit IRC | 17:08 | |
| *** thorst_ has joined #openstack-telemetry | 17:09 | |
| *** vishwanathj has quit IRC | 17:11 | |
| *** thorst_ has quit IRC | 17:13 | |
| *** cdent has quit IRC | 17:14 | |
| *** guillaume_ has joined #openstack-telemetry | 17:17 | |
| *** thorst_ has joined #openstack-telemetry | 17:17 | |
| *** mattyw has joined #openstack-telemetry | 17:21 | |
| *** thorst_ has quit IRC | 17:22 | |
| *** permalac has quit IRC | 17:22 | |
| zigo | jd__: Do I really need to have gnocchi build-depends on python-ceilometer? | 17:26 |
| *** shardy has quit IRC | 17:26 | |
| zigo | Or is this requirement gone in trunk? | 17:26 |
| gordc | zigo: gnocchi should not need ceilometer or it's client | 17:27 |
| zigo | gordc: Even for unit testing? | 17:28 |
| zigo | Cool. | 17:28 |
| zigo | This breaks a circular build-depends ! :) | 17:28 |
| gordc | hm... i can't imaging why gnocchi depends on ceilometer for unit/functional tests... we only have it for integration tests between heat+gnocchi+ceilometer+aodh+nova | 17:29 |
| zigo | Ok. | 17:29 |
| zigo | Good to know. | 17:29 |
| EmilienM | gordc: feel free to comment https://etherpad.openstack.org/p/puppet-openstack-integration-scenarios | 17:33 |
| EmilienM | we plan to have a full telemetry scenario | 17:33 |
| *** thorst_ has joined #openstack-telemetry | 17:37 | |
| gordc | EmilienM: cool stuff. | 17:37 |
| *** thorst__ has joined #openstack-telemetry | 17:38 | |
| *** thorst_ has quit IRC | 17:41 | |
| *** thorst__ has quit IRC | 17:43 | |
| *** _nadya_ has joined #openstack-telemetry | 17:47 | |
| guillaume_ | Hello, I have a question about radosgw-admin and ceilometer. I'm trying to give a user the right to request usage of other ussers. Do you know how to do that ? I already try to give caps * (usage,users,buckets,zone and metadata) and full-control on the subuser. I'm trying to plug ceilometer of openstack to radosgw-admin. The connection work, but I only see usage for the user I created instead of all the users. | 17:48 |
| gordc | guillaume_: you can try to change your policy.json file. but in openstack in general, RBAC policies controls what results you see. | 17:50 |
| gordc | generally, admins have access to all details from their project and general users only have access to their own data. | 17:51 |
| *** _nadya_ has quit IRC | 17:51 | |
| guillaume_ | ok, but if this is the case I should be able to see all usage in CLI with radosgw-admin right? I use radosgw-admin --uid=ceilometer usage show | 17:56 |
| gordc | guillaume_: huh? are you trying to use ceph as object store or trying to get the meters relating to ceph from ceilometer? | 17:57 |
| gordc | guillaume_: sorry, i'm stepping out for lunch. bbl. | 17:58 |
| jd__ | zigo: gordc: we used to need Ceilometer to test the dispatcher that was in Gnocchi but now it is in Ceilometer so no nee | 18:00 |
| jd__ | d | 18:00 |
| guillaume_ | i'm tryoing to get usage of our object-storage who is in ceph | 18:00 |
| zigo | Ah, so I didn't dream! :) | 18:00 |
| zigo | Thanks for confirming. | 18:00 |
| *** rcernin has joined #openstack-telemetry | 18:00 | |
| *** mattyw has quit IRC | 18:09 | |
| *** safchain has quit IRC | 18:25 | |
| *** ljxiash has joined #openstack-telemetry | 18:39 | |
| *** ljxiash has quit IRC | 18:43 | |
| *** tongli has joined #openstack-telemetry | 19:10 | |
| *** boris-42 has quit IRC | 19:13 | |
| *** pece has joined #openstack-telemetry | 19:38 | |
| *** pece has quit IRC | 19:40 | |
| *** cdent has joined #openstack-telemetry | 19:50 | |
| *** _nadya_ has joined #openstack-telemetry | 19:55 | |
| *** barra204 has quit IRC | 20:06 | |
| *** guillaume_ has quit IRC | 20:08 | |
| *** leitan has quit IRC | 20:16 | |
| *** julim_ has quit IRC | 20:17 | |
| *** guillaume_ has joined #openstack-telemetry | 20:23 | |
| *** yassine__ has quit IRC | 20:25 | |
| *** leitan has joined #openstack-telemetry | 20:30 | |
| *** flwang has joined #openstack-telemetry | 20:41 | |
| flwang | gordc: ping is there any way to skip specific tenants metrics? | 20:42 |
| *** vishwanathj has joined #openstack-telemetry | 20:42 | |
| *** _nadya_ has quit IRC | 20:42 | |
| gordc | flwang: not through configuration. | 20:44 |
| gordc | flwang: https://blueprints.launchpad.net/ceilometer/+spec/project-based-data-collection-configuration | 20:44 |
| gordc | depending on backend you could create some ttl rule. | 20:45 |
| *** cdent has quit IRC | 20:47 | |
| flwang | gordc: yep, that's our pain point | 20:54 |
| flwang | the mongodb size is increasing very fast | 20:55 |
| flwang | so we're trying to skip some internal/test tenants' metrics | 20:55 |
| gordc | flwang: you should probably be using gnocchi -- unless you have a use case for having full-fidelity data | 20:57 |
| gordc | flwang: if you aren't storing data for auditing, you really shouldn't be using ceilometer's db | 20:58 |
| gordc | flwang: if you must use ceilometer db, sql is a much smaller footprint. | 20:58 |
| gordc | that said, it requires knowledge on how to scale sql... same as mongodb. | 20:59 |
| EmilienM | gordc: do you do consulting? | 20:59 |
| EmilienM | I need your help | 20:59 |
| gordc | EmilienM: 250/hr? | 20:59 |
| EmilienM | our Puppet CI deploys now gnocchi/ceilometer/aodh | 20:59 |
| EmilienM | gordc: ok I'll expense | 20:59 |
| gordc | EmilienM: perfect | 20:59 |
| EmilienM | gordc: can you look http://logs.openstack.org/98/272298/7/check/gate-puppet-openstack-integration-scenario001-tempest-dsvm-centos7/7beeb72/logs/etc/ | 20:59 |
| gordc | EmilienM: what am i looking at? | 21:00 |
| EmilienM | gordc: conf files | 21:00 |
| EmilienM | to make sure ceilometer is using gnocchi for meter_dispatcher | 21:01 |
| gordc | ok...? | 21:01 |
| gordc | oh | 21:01 |
| gordc | you don't need this http://logs.openstack.org/98/272298/7/check/gate-puppet-openstack-integration-scenario001-tempest-dsvm-centos7/7beeb72/logs/etc/ceilometer/policy.json.txt.gz | 21:01 |
| gordc | if you have gnocchi | 21:01 |
| EmilienM | it's provided by packaging bro | 21:01 |
| EmilienM | I don't even configure it | 21:01 |
| gordc | unless you are using events? | 21:01 |
| EmilienM | let me check | 21:02 |
| gordc | EmilienM: you do... you defined one. | 21:03 |
| gordc | although you turn events off...sooo | 21:03 |
| gordc | you have a db with nothing | 21:03 |
| *** julim has joined #openstack-telemetry | 21:03 | |
| EmilienM | gordc: http://logs.openstack.org/98/272298/7/check/gate-puppet-openstack-integration-scenario001-tempest-dsvm-centos7/7beeb72/console.html#_2016-01-27_19_57_09_940 | 21:03 |
| EmilienM | see the tests I'm running | 21:03 |
| EmilienM | I have some telemetry stuffs | 21:04 |
| gordc | mitaka or liberty? | 21:04 |
| EmilienM | mitaka trunk | 21:04 |
| gordc | you shuld have a cache. | 21:04 |
| EmilienM | how? | 21:04 |
| EmilienM | gordc: I don't see anything in gnocchi api logs: http://logs.openstack.org/98/272298/7/check/gate-puppet-openstack-integration-scenario001-tempest-dsvm-centos7/7beeb72/logs/apache/ | 21:05 |
| EmilienM | is it normal? | 21:05 |
| flwang | gordc: we're using mongo for production | 21:06 |
| flwang | and our billing program is depending on ceilometer/mongo | 21:07 |
| gordc | EmilienM: your collector for some reason does not use gnocchi http://logs.openstack.org/98/272298/7/check/gate-puppet-openstack-integration-scenario001-tempest-dsvm-centos7/7beeb72/logs/ceilometer/collector.txt.gz | 21:07 |
| EmilienM | gordc: is it a question? | 21:08 |
| EmilienM | I have [dispatcher_gnocchi] in ceilometer.conf | 21:08 |
| EmilienM | http://logs.openstack.org/98/272298/7/check/gate-puppet-openstack-integration-scenario001-tempest-dsvm-centos7/7beeb72/logs/ceilometer/collector.txt.gz#_2016-01-27_19_38_21_625 | 21:09 |
| EmilienM | err | 21:09 |
| EmilienM | meter_dispatchers = ['database'] | 21:09 |
| gordc | flwang: i don't know what to suggest. you'll need to either scale mongo up to handle size, or set a higher ttl and offload data. | 21:09 |
| gordc | flwang: ceilometer's data stores everything and mongo backend specifically stores it even more verbose see: nosql. | 21:10 |
| gordc | flwang: https://bugs.launchpad.net/ceilometer/+bug/1527231 | 21:10 |
| openstack | Launchpad bug 1527231 in Ceilometer "The meter model in mongodb backend should be light weight" [Undecided,Incomplete] - Assigned to Liusheng (liusheng) | 21:10 |
| gordc | EmilienM: yeah | 21:11 |
| EmilienM | wait meter_dispatchers is empty in ceilo.conf | 21:11 |
| EmilienM | oh I found why | 21:12 |
| flwang | gordc: i see, thank you for the feedback | 21:12 |
| EmilienM | err no | 21:12 |
| gordc | EmilienM: this is mine http://paste.openstack.org/show/485192/ | 21:13 |
| gordc | flwang: yeah, sorry can't help. i'm really just trying to encourage people to not use mongo ;) | 21:14 |
| EmilienM | gordc: ok.. I'l debug from here | 21:14 |
| EmilienM | thanks | 21:14 |
| EmilienM | gordc: I'll send you poutine | 21:14 |
| *** thorst_ has joined #openstack-telemetry | 21:14 | |
| gordc | EmilienM: i'm not drunk | 21:14 |
| flwang | gordc: oh, interesting... what's the story? | 21:16 |
| *** rcernin has quit IRC | 21:16 | |
| gordc | flwang: about me drunk or about mongodb?lol | 21:17 |
| flwang | mongodb | 21:17 |
| flwang | and drunk later :) | 21:17 |
| EmilienM | meter_dispatcher=gnocchi | 21:18 |
| EmilienM | event_dispatcher=database | 21:18 |
| EmilienM | http://logs.openstack.org/98/272298/7/check/gate-puppet-openstack-integration-scenario001-tempest-dsvm-centos7/7beeb72/logs/etc/ceilometer/ceilometer.conf.txt.gz | 21:18 |
| EmilienM | gordc: ^ | 21:18 |
| EmilienM | wtf | 21:18 |
| *** thorst_ has quit IRC | 21:18 | |
| gordc | flwang: lol yeah, i'm just trying to promote the gnocchi. there are a lot of inefficiencies with the ceilomter api model itself so i'm not sure how much it's worth to attempt to optimise the db as is. | 21:19 |
| EmilienM | gordc: I use file backend FYI | 21:19 |
| gordc | flwang: the idea is if you want responsiveness, use gnocchi. if you want full-fidelity data but don't care about responsiveness, use existing ceilometer db | 21:20 |
| gordc | EmilienM: as you want. | 21:20 |
| EmilienM | gordc: yeah but it does not load the dispatcher | 21:20 |
| EmilienM | my config is good, I set meter_dispatchers = gnocchi | 21:21 |
| gordc | file dispatcher? | 21:21 |
| EmilienM | oh wait | 21:21 |
| EmilienM | it's meter_dispatchers = gnocchi or meter_dispatcher = gnocchi ? | 21:21 |
| gordc | firs tone | 21:21 |
| EmilienM | ... | 21:21 |
| gordc | first one | 21:21 |
| EmilienM | facepalm | 21:21 |
| EmilienM | I did wrong in puppet-gnocchi | 21:21 |
| gordc | i round up. this was an hour. | 21:22 |
| gordc | 250 | 21:22 |
| EmilienM | can I pay in beers man? | 21:22 |
| flwang | gordc: i see. so the question is, is gnocchi ready for production? | 21:23 |
| gordc | flwang: *shrugs* we need guinea pigs.lol | 21:24 |
| gordc | i would say, the biggest performacne changes are in (as of this monday) | 21:25 |
| flwang | gordc: good to know | 21:25 |
| gordc | there is an additional large change in dynamic resources but it's not critical to ceilometer+gnocchi integration | 21:25 |
| EmilienM | gordc: https://review.openstack.org/273244 | 21:26 |
| gordc | i would consider dynamic resources more a ease of use feature. | 21:26 |
| jd__ | nicodemus_ and mnaser are doing good deployments and good feedback, so we're starting to be pretty confident, flwang | 21:26 |
| flwang | jd__: hey man, long time no see | 21:26 |
| jd__ | ^^ | 21:26 |
| flwang | jd__: thanks for the feedback | 21:26 |
| jd__ | at least with Ceph as a backend :) | 21:27 |
| flwang | jd__: i may need to talk with our operations team to see if they're happy to do the migration | 21:27 |
| jd__ | let me know if they have any question | 21:27 |
| nicodemus_ | Ceph as backend and millions of measures to process :) | 21:28 |
| gordc | flwang: there's no existing migration path (possible but not something we will expend resources on unless required) | 21:28 |
| flwang | gordc: wow, no migration path | 21:28 |
| gordc | nicodemus_: i'm quoting you on slide. | 21:28 |
| mnaser | im just going to chime in here but there's quite a bit of work to be done in terms of batching work amongst workers before gnocchi can scale out | 21:29 |
| gordc | flwang: the models are fundamentally different. that said, it's entirely possibly to just have a tool pull from ceilometer api, and input into gnocchi api. | 21:29 |
| mnaser | with a large # of items in queue, the current system struggles more with scale (spinning to do work that's already done), poor use of resources | 21:30 |
| flwang | yep, i understand. just pretend i'm really surprise :D | 21:30 |
| jd__ | mnaser: you got numbers? | 21:30 |
| mnaser | 2x servers with 24 cores (2.93ghz each) dedicated for gnocchi and it couldn't keep up unfortunately | 21:30 |
| mnaser | not right away, but if you ask nicodemus_ to grep the amount of "Skipped" lines with debug one, it's going to be a *huge* number | 21:31 |
| jd__ | i would love to have a more precise picture of the situation yeah | 21:31 |
| nicodemus_ | my case is 4x instances with 4vcpus and 16gb ram (each) | 21:32 |
| jd__ | how many measures per second are being sent, etc | 21:32 |
| mnaser | i dont have vpn access to the site where we have that deployment right now, but nicodemus_ could try seeing if he has debug on | 21:32 |
| jd__ | ok :) | 21:32 |
| jd__ | we also have landed a large optimization yesterday so that might be worth a try | 21:32 |
| gordc | mnaser: stable/1.3? | 21:33 |
| jd__ | I think they are using master of a few days ago | 21:34 |
| nicodemus_ | right now all my metricd are stopped until ceph finishes rebalancing the gnocchi pool (changed the size), when done I'll leave them working during the night and see if I can get some performance numbers | 21:34 |
| jd__ | nicodemus_: awesome! | 21:34 |
| nicodemus_ | I'm using master pulled today. | 21:34 |
| jd__ | nicodemus_: i may ask you for a small sample of log (e.g. like 5 minutes of all your metricd for the same timerange) to see how they behave | 21:35 |
| mnaser | i think at the time we went back to master and not much changed, ive been busy with a new customer past few days so im not sure | 21:35 |
| jd__ | we changed the Carbonara archive storage scheme | 21:36 |
| nicodemus_ | jd__: no problem! I'll gather them tonight | 21:36 |
| gordc | mnaser: ack. | 21:36 |
| gordc | nicodemus_: you resolve your problem from yesterday? | 21:36 |
| mnaser | to be honest, there are a few things that are concerning in how gnocchi processes data | 21:37 |
| mnaser | and it all comes from the workers sharing the same assigned workload | 21:37 |
| mnaser | but i feel this is a very difficult problem to solve | 21:38 |
| jd__ | nothing we don't know how to solve | 21:38 |
| nicodemus_ | gordc: since metricd is very taxing on our ceph cluster, today I couldn't let all four work for too long, so the issue from yesterday is still inconclusive | 21:39 |
| mnaser | well this type of thing has historically been resolved by things like message queues.. but this will limit the scalability of gnocchi a lot imho | 21:39 |
| jd__ | my problem is that I don't see the problem yet (i.e. I barely have any info on it) | 21:39 |
| *** ljxiash has joined #openstack-telemetry | 21:40 | |
| gordc | mq is one way yes or you can just bucketise and divide accordingly. | 21:40 |
| *** guillaume_ has quit IRC | 21:40 | |
| mnaser | every cycle, each worker requests the list of all metrics to be processed and starts going one-by-one through them. if teh list of metrics is 5000 and you have 10 workers, the 1st worker will grab the 1st, the 2nd will try to grab 1st, see it's done, skip to 2nd.. when you have 48 workers, you basically have 48 metrics being ignored before you process one | 21:40 |
| mnaser | and that number could increase if your cpus are stronger, so yeah as jd__ said.. i think ceilometer did something to distribute load using the locking mechanism | 21:41 |
| mnaser | sorry, gordc ^ | 21:41 |
| jd__ | which is not a problem and only true on cold start | 21:41 |
| jd__ | then no worker should be in sync and they should grab randomly metrics | 21:42 |
| gordc | mnaser: yeah, it'd affect mq as well. anytime your backlog whether on mq or current method is huge, it's going to be a bad time. | 21:42 |
| jd__ | an easy optimization is to grab smaller batches of metric, not all though | 21:42 |
| mnaser | jd__: there is obviously advantages and disadvantages to each | 21:43 |
| mnaser | i mean, i have had some ideas but they depend on external software (redis) so i dont know if that's easy to bring in | 21:43 |
| mnaser | or thinking about it, it might not be required. just introduce concept of buckets and lock them when a worker is processing them | 21:44 |
| *** ljxiash has quit IRC | 21:44 | |
| mnaser | bucket-ing process can happen when the metrics come in | 21:44 |
| jd__ | they are tons of solution with pros and cons | 21:44 |
| jd__ | as I said, until I have more information it's hard to determine which trade offs we should do :) | 21:45 |
| * gordc puts away dice | 21:45 | |
| gordc | we want logic? ok. :( | 21:45 |
| *** guillaume_ has joined #openstack-telemetry | 21:52 | |
| nicodemus_ | jd__: besides from the 5-minute logs from the 4 metricd, is there any other info that you would like me to gather? | 21:54 |
| *** nicodemus_ has quit IRC | 22:04 | |
| *** nicodemus_ has joined #openstack-telemetry | 22:05 | |
| *** leitan has quit IRC | 22:13 | |
| *** AndChat|224721 has joined #openstack-telemetry | 22:14 | |
| *** nicodemus_ has quit IRC | 22:17 | |
| EmilienM | gordc: http://logs.openstack.org/98/272298/8/check/gate-puppet-openstack-integration-scenario001-tempest-dsvm-centos7/4ab76c8/logs/ceilometer/collector.txt.gz#_2016-01-27_21_56_51_891 | 22:21 |
| openstackgerrit | gordon chung proposed openstack/ceilometer: rewriting history https://review.openstack.org/273049 | 22:22 |
| gordc | EmilienM: you need to create a default starting archive policy | 22:22 |
| gordc | https://github.com/openstack/gnocchi/blob/master/devstack/plugin.sh#L396-L400 | 22:23 |
| gordc | EmilienM: although this has been asked so often i wonder if we should just not include one in gnocchi by default. | 22:23 |
| EmilienM | gordc: http://www.nooooooooooooooo.com/ | 22:24 |
| gordc | EmilienM: haha. | 22:24 |
| gordc | i just watched that movie | 22:24 |
| gordc | first time... it was... meh | 22:25 |
| EmilienM | gordc: so I need to run those commands??? | 22:26 |
| EmilienM | the 4 commands? | 22:26 |
| *** AndChat|224721 has quit IRC | 22:30 | |
| openstackgerrit | gordon chung proposed openstack/ceilometer: rolling upgrades https://review.openstack.org/273081 | 22:31 |
| *** nicodemus_ has joined #openstack-telemetry | 22:31 | |
| EmilienM | gordc: is it mandatory to run this? | 22:33 |
| EmilienM | can't I skip it? | 22:33 |
| EmilienM | don't you guys provide default policies? | 22:34 |
| gordc | EmilienM: did i not just say 'i wonder if we should?' | 22:34 |
| gordc | EmilienM: dammit bro. | 22:34 |
| EmilienM | yes you should | 22:34 |
| gordc | you should | 22:34 |
| EmilienM | I'll hack something on puppet in the meantime | 22:35 |
| gordc | creating a wishlist bug | 22:35 |
| EmilienM | gordc: like a christmas list? | 22:36 |
| gordc | EmilienM: leave | 22:36 |
| *** AndChat|224721 has joined #openstack-telemetry | 22:38 | |
| *** thorst_ has joined #openstack-telemetry | 22:41 | |
| *** nicodemus_ has quit IRC | 22:41 | |
| openstackgerrit | gordon chung proposed openstack/ceilometer: raise coordination error if not registered https://review.openstack.org/268292 | 22:50 |
| *** AndChat|224721 has quit IRC | 22:53 | |
| *** nicodemus_ has joined #openstack-telemetry | 22:53 | |
| *** AndChat|224721 has joined #openstack-telemetry | 22:58 | |
| *** AndChat|224721 has quit IRC | 22:59 | |
| *** AndChat|224721 has joined #openstack-telemetry | 22:59 | |
| *** AndChat|224721 has quit IRC | 23:00 | |
| *** AndChat|224721 has joined #openstack-telemetry | 23:00 | |
| *** nicodemus_ has quit IRC | 23:00 | |
| *** AndChat|224721 has quit IRC | 23:03 | |
| *** harlowja has quit IRC | 23:03 | |
| *** harlowja has joined #openstack-telemetry | 23:04 | |
| *** nicodemus_ has joined #openstack-telemetry | 23:05 | |
| *** davidlenwell has quit IRC | 23:06 | |
| *** davidlenwell has joined #openstack-telemetry | 23:08 | |
| *** chlong has quit IRC | 23:18 | |
| *** mragupat has quit IRC | 23:22 | |
| *** chlong has joined #openstack-telemetry | 23:30 | |
| *** rbak has quit IRC | 23:46 | |
| *** pradk_ has quit IRC | 23:59 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!