| *** harlowja has joined #openstack-telemetry | 00:06 | |
| *** gordc has quit IRC | 00:15 | |
| openstackgerrit | Merged openstack/gnocchi: rest: reject / as resource id and metric name https://review.openstack.org/419009 | 00:25 |
|---|---|---|
| *** thorst has joined #openstack-telemetry | 00:33 | |
| *** thorst has quit IRC | 00:34 | |
| *** zhurong has joined #openstack-telemetry | 00:56 | |
| *** zhurong has quit IRC | 00:56 | |
| *** zhurong has joined #openstack-telemetry | 00:56 | |
| *** chopmann has quit IRC | 01:02 | |
| *** adriant has quit IRC | 01:18 | |
| *** dave-mccowan has quit IRC | 01:26 | |
| *** r-mibu has quit IRC | 01:27 | |
| *** stevemar has quit IRC | 01:27 | |
| *** basilAB has joined #openstack-telemetry | 01:28 | |
| *** r-mibu has joined #openstack-telemetry | 01:29 | |
| *** stevemar has joined #openstack-telemetry | 01:29 | |
| *** thorst has joined #openstack-telemetry | 01:35 | |
| *** hfu has joined #openstack-telemetry | 01:35 | |
| *** thorst has quit IRC | 01:40 | |
| *** thorst has joined #openstack-telemetry | 01:50 | |
| *** thorst has quit IRC | 01:50 | |
| openstackgerrit | qin.jiang proposed openstack/ceilometer: Extract the same code https://review.openstack.org/421619 | 01:57 |
| *** lhx__ has joined #openstack-telemetry | 02:01 | |
| *** lhx__ has quit IRC | 02:03 | |
| *** lhx__ has joined #openstack-telemetry | 02:04 | |
| openstackgerrit | Merged openstack/gnocchi: mysql: fix timestamp upgrade https://review.openstack.org/419909 | 02:09 |
| *** thorst has joined #openstack-telemetry | 02:30 | |
| *** thorst has quit IRC | 02:30 | |
| *** harlowja has quit IRC | 02:36 | |
| *** davidlenwell_ has joined #openstack-telemetry | 02:38 | |
| openstackgerrit | qin.jiang proposed openstack/ceilometer: Extract the same code https://review.openstack.org/421619 | 02:47 |
| *** thorst has joined #openstack-telemetry | 02:56 | |
| *** thorst has quit IRC | 02:56 | |
| *** zhurong has quit IRC | 03:00 | |
| *** zhurong has joined #openstack-telemetry | 03:02 | |
| *** davidlenwell_ has quit IRC | 03:06 | |
| openstackgerrit | Jeremy Liu proposed openstack/panko: Update requirements https://review.openstack.org/421642 | 03:18 |
| *** dave-mccowan has joined #openstack-telemetry | 03:18 | |
| *** gongysh has joined #openstack-telemetry | 03:20 | |
| openstackgerrit | Hanxi Liu proposed openstack/ceilometer: Ship YAML file to /usr/share https://review.openstack.org/412309 | 03:22 |
| *** dave-mccowan has quit IRC | 03:23 | |
| *** zhurong has quit IRC | 03:23 | |
| *** thorst has joined #openstack-telemetry | 03:27 | |
| *** thorst has quit IRC | 03:33 | |
| *** sheel has joined #openstack-telemetry | 03:34 | |
| *** rwsu has quit IRC | 03:37 | |
| *** sudipto_ has joined #openstack-telemetry | 03:50 | |
| *** sudipto has joined #openstack-telemetry | 03:50 | |
| *** links has joined #openstack-telemetry | 04:02 | |
| *** r-mibu has quit IRC | 04:03 | |
| *** r-mibu has joined #openstack-telemetry | 04:03 | |
| *** sudipto has quit IRC | 04:14 | |
| *** sudipto_ has quit IRC | 04:14 | |
| *** tlian has quit IRC | 04:25 | |
| *** sudipto_ has joined #openstack-telemetry | 04:35 | |
| *** sudipto has joined #openstack-telemetry | 04:35 | |
| *** sudipto_ has quit IRC | 05:10 | |
| *** sudipto has quit IRC | 05:10 | |
| *** sudipto has joined #openstack-telemetry | 05:12 | |
| *** sudipto_ has joined #openstack-telemetry | 05:13 | |
| *** thorst has joined #openstack-telemetry | 05:29 | |
| *** nadya has joined #openstack-telemetry | 05:31 | |
| *** thorst has quit IRC | 05:34 | |
| *** donghao has joined #openstack-telemetry | 05:35 | |
| *** donghao has quit IRC | 05:39 | |
| *** nadya has quit IRC | 05:47 | |
| *** Jack_Iv has joined #openstack-telemetry | 05:57 | |
| *** Jack_Iv_ has joined #openstack-telemetry | 05:58 | |
| *** Jack_Iv has quit IRC | 06:02 | |
| *** Jack_Iv_ has quit IRC | 06:09 | |
| openstackgerrit | Mehdi Abaakouk (sileht) proposed openstack/gnocchi: carbonara: handle timestamps from struct with numpy https://review.openstack.org/421570 | 06:52 |
| openstackgerrit | Mehdi Abaakouk (sileht) proposed openstack/gnocchi: carbonara: prepare datetime for pandas.to_datetime() https://review.openstack.org/421208 | 06:52 |
| openstackgerrit | Mehdi Abaakouk (sileht) proposed openstack/gnocchi: carbonara: remove a pandas.iteritems() https://review.openstack.org/421516 | 06:52 |
| openstackgerrit | Mehdi Abaakouk (sileht) proposed openstack/gnocchi: carbonara: handle timestamps from struct with numpy https://review.openstack.org/421570 | 06:53 |
| openstackgerrit | Mehdi Abaakouk (sileht) proposed openstack/gnocchi: carbonara: prepare datetime for pandas.to_datetime() https://review.openstack.org/421208 | 06:53 |
| openstackgerrit | Mehdi Abaakouk (sileht) proposed openstack/gnocchi: carbonara: remove a pandas.iteritems() https://review.openstack.org/421516 | 06:53 |
| *** gongysh has quit IRC | 07:00 | |
| *** dschultz has quit IRC | 07:03 | |
| openstackgerrit | Alexey Weyl proposed openstack/aodh: Add userdata field to all alarms https://review.openstack.org/420409 | 07:12 |
| *** bapalm has quit IRC | 07:16 | |
| *** bapalm has joined #openstack-telemetry | 07:18 | |
| *** tesseract has joined #openstack-telemetry | 07:25 | |
| *** tesseract has quit IRC | 07:25 | |
| *** tesseract has joined #openstack-telemetry | 07:26 | |
| *** thorst has joined #openstack-telemetry | 07:30 | |
| openstackgerrit | Mehdi Abaakouk (sileht) proposed openstack/gnocchi: carbonara: Don't call clean_ts() when unserialize https://review.openstack.org/421714 | 07:35 |
| *** thorst has quit IRC | 07:35 | |
| *** zhurong has joined #openstack-telemetry | 07:37 | |
| openstackgerrit | Mehdi Abaakouk (sileht) proposed openstack/gnocchi: carbonara: Don't call clean_ts() when unserialize https://review.openstack.org/421714 | 07:38 |
| *** zhurong has quit IRC | 07:46 | |
| *** thorst has joined #openstack-telemetry | 08:00 | |
| *** thorst has quit IRC | 08:00 | |
| *** yprokule has joined #openstack-telemetry | 08:04 | |
| *** shardy has joined #openstack-telemetry | 08:12 | |
| *** gongysh has joined #openstack-telemetry | 08:14 | |
| openstackgerrit | Jeremy Liu proposed openstack/panko: Optimize policy engine initialization https://review.openstack.org/421736 | 08:35 |
| *** gongysh has quit IRC | 08:46 | |
| *** nadya has joined #openstack-telemetry | 09:08 | |
| *** dschultz has joined #openstack-telemetry | 09:09 | |
| *** dschultz has quit IRC | 09:16 | |
| openstackgerrit | Hanxi Liu proposed openstack/ceilometer: Deprecate collector https://review.openstack.org/413920 | 09:34 |
| *** yassine has joined #openstack-telemetry | 09:36 | |
| *** yassine is now known as Guest62565 | 09:37 | |
| openstackgerrit | Mehdi Abaakouk (sileht) proposed openstack/gnocchi: carbonara: use numpy for serialization https://review.openstack.org/421766 | 09:40 |
| openstackgerrit | Mehdi Abaakouk (sileht) proposed openstack/gnocchi: carbonara: use numpy for serialization https://review.openstack.org/421766 | 09:41 |
| openstackgerrit | Mehdi Abaakouk (sileht) proposed openstack/gnocchi: carbonara: handle timestamps from struct with numpy https://review.openstack.org/421570 | 09:41 |
| openstackgerrit | Mehdi Abaakouk (sileht) proposed openstack/gnocchi: carbonara: Don't call clean_ts() when unserialize https://review.openstack.org/421714 | 09:41 |
| *** pcaruana has joined #openstack-telemetry | 09:41 | |
| sileht | jd__, before: http://paste.openstack.org/show/595291/ after:http://paste.openstack.org/show/595333/ | 09:44 |
| sileht | jd__, I have one more optimisation, remove the struck.unpack by reading the payload directly with numpy | 09:44 |
| *** gongysh has joined #openstack-telemetry | 09:46 | |
| sileht | numpy powered ! | 09:46 |
| *** andymccr has joined #openstack-telemetry | 09:58 | |
| *** thorst has joined #openstack-telemetry | 10:01 | |
| *** nadya has quit IRC | 10:02 | |
| *** hfu has quit IRC | 10:04 | |
| *** larainema has quit IRC | 10:06 | |
| *** thorst has quit IRC | 10:06 | |
| *** gongysh has quit IRC | 10:11 | |
| *** vint_bra has joined #openstack-telemetry | 10:24 | |
| jd__ | sileht: awesome | 10:29 |
| jd__ | sileht: i did not know you could use numpy to read struct that's cool | 10:29 |
| sileht | jd__, I have almost finish the read stuff | 10:29 |
| jd__ | sileht: it would be interesting to see if you have a better number of points/metrics handled by metricd in the end | 10:29 |
| jd__ | i guess yes :) | 10:29 |
| sileht | jd__, hop numbers without struct.unpack: http://paste.openstack.org/show/595345/ | 10:33 |
| sileht | a huge gain again :) | 10:33 |
| jd__ | akrzos is gonna be happy | 10:35 |
| jd__ | we really need to test that in our next benchmark :) | 10:35 |
| sileht | jd__, I will retry to replace the tetaneutral rrdcached by gnocchi and see if it win now :) | 10:38 |
| *** pcaruana has quit IRC | 10:38 | |
| *** vint_bra has quit IRC | 10:38 | |
| *** Jack_Iv has joined #openstack-telemetry | 10:38 | |
| sileht | jd__, you misread my number, factor is x15 - x20 | 10:40 |
| * sileht -> lunch time | 10:40 | |
| jd__ | sileht: damn lol | 10:40 |
| * jd__ fixes it | 10:41 | |
| sileht | 20MB/s vs 380MB/s | 10:41 |
| sileht | 38MB/s vs 115MB/s | 10:41 |
| *** nadya has joined #openstack-telemetry | 10:47 | |
| *** nadya has quit IRC | 10:49 | |
| *** nadya has joined #openstack-telemetry | 10:49 | |
| *** cdent has joined #openstack-telemetry | 10:58 | |
| *** lhx__ has quit IRC | 11:10 | |
| *** sudipto_ has quit IRC | 11:25 | |
| *** sudipto has quit IRC | 11:25 | |
| openstackgerrit | qin.jiang proposed openstack/ceilometer: Simplify code of test_complex_query https://review.openstack.org/421845 | 11:36 |
| *** lhx_ has joined #openstack-telemetry | 11:43 | |
| *** pcaruana has joined #openstack-telemetry | 11:56 | |
| *** thorst has joined #openstack-telemetry | 12:02 | |
| *** Jack_Iv has quit IRC | 12:04 | |
| *** thorst has quit IRC | 12:06 | |
| openstackgerrit | Merged openstack/aodh: modernise gabbi usage https://review.openstack.org/420958 | 12:11 |
| *** leitan has joined #openstack-telemetry | 12:23 | |
| *** shardy is now known as shardy_lunch | 12:29 | |
| *** dave-mccowan has joined #openstack-telemetry | 12:34 | |
| *** sudipto_ has joined #openstack-telemetry | 12:44 | |
| *** sudipto has joined #openstack-telemetry | 12:44 | |
| *** links has quit IRC | 12:46 | |
| *** thorst has joined #openstack-telemetry | 12:51 | |
| *** zhurong has joined #openstack-telemetry | 12:53 | |
| *** Guest62565 has quit IRC | 13:06 | |
| *** Guest62565 has joined #openstack-telemetry | 13:09 | |
| *** zhurong has quit IRC | 13:11 | |
| *** zhurong has joined #openstack-telemetry | 13:12 | |
| *** zhurong has quit IRC | 13:13 | |
| *** vint_bra has joined #openstack-telemetry | 13:14 | |
| *** dschultz has joined #openstack-telemetry | 13:15 | |
| *** dschultz has quit IRC | 13:19 | |
| *** shardy_lunch is now known as shardy | 13:21 | |
| openstackgerrit | Mehdi Abaakouk (sileht) proposed openstack/gnocchi: carbonara: use numpy for serialization https://review.openstack.org/421766 | 13:27 |
| openstackgerrit | Mehdi Abaakouk (sileht) proposed openstack/gnocchi: carbonara: handle timestamps from struct with numpy https://review.openstack.org/421570 | 13:27 |
| openstackgerrit | Mehdi Abaakouk (sileht) proposed openstack/gnocchi: carbonara: prepare datetime for pandas.to_datetime() https://review.openstack.org/421208 | 13:27 |
| openstackgerrit | Mehdi Abaakouk (sileht) proposed openstack/gnocchi: carbonara: Don't call clean_ts() when unserialize https://review.openstack.org/421714 | 13:27 |
| openstackgerrit | Mehdi Abaakouk (sileht) proposed openstack/gnocchi: carbonara: remove a pandas.iteritems() https://review.openstack.org/421516 | 13:27 |
| openstackgerrit | Mehdi Abaakouk (sileht) proposed openstack/gnocchi: indexer: fix migration script https://review.openstack.org/421900 | 13:27 |
| openstackgerrit | Mehdi Abaakouk (sileht) proposed openstack/gnocchi: carbonara: use numpy for unserialization https://review.openstack.org/421901 | 13:27 |
| *** zhurong has joined #openstack-telemetry | 13:32 | |
| *** vint_bra1 has joined #openstack-telemetry | 13:32 | |
| *** vint_bra1 has quit IRC | 13:32 | |
| *** vint_bra has quit IRC | 13:32 | |
| *** vint_bra1 has joined #openstack-telemetry | 13:32 | |
| *** fc__ has quit IRC | 13:37 | |
| *** fc__ has joined #openstack-telemetry | 13:37 | |
| *** pradk has joined #openstack-telemetry | 13:39 | |
| *** catintheroof has joined #openstack-telemetry | 13:45 | |
| *** larainema has joined #openstack-telemetry | 13:53 | |
| sileht | jd__, on my ttnn setup, the metricd logs now always show: locked during 0.00 seconds | 13:59 |
| sileht | it's not yet fully loaded, the API currently creates all missing resources, once this is finish, the load on metricd should increase | 14:00 |
| *** eglynn has joined #openstack-telemetry | 14:04 | |
| *** donghao has joined #openstack-telemetry | 14:05 | |
| *** donghao has quit IRC | 14:09 | |
| *** pradk has quit IRC | 14:12 | |
| *** tlian has joined #openstack-telemetry | 14:14 | |
| lhx_ | gordc, could you help review this? | 14:27 |
| lhx_ | sileht, could you? https://review.openstack.org/#/c/413920/ | 14:28 |
| *** shardy has quit IRC | 14:30 | |
| *** shardy has joined #openstack-telemetry | 14:31 | |
| *** fguillot has joined #openstack-telemetry | 14:33 | |
| *** Jack_Iv has joined #openstack-telemetry | 14:36 | |
| *** donghao has joined #openstack-telemetry | 14:39 | |
| openstackgerrit | Hanxi Liu proposed openstack/ceilometer: Deprecate collector https://review.openstack.org/413920 | 14:44 |
| *** fguillot has quit IRC | 14:48 | |
| jd__ | sileht: so? :) | 14:54 |
| *** gongysh has joined #openstack-telemetry | 14:54 | |
| *** links has joined #openstack-telemetry | 14:56 | |
| *** links has quit IRC | 14:56 | |
| *** rbak has joined #openstack-telemetry | 14:58 | |
| *** Jack_Iv has quit IRC | 15:00 | |
| sileht | jd__, still not enought :( | 15:00 |
| *** leitan has quit IRC | 15:02 | |
| jd__ | sileht: damn so long to create resources? | 15:03 |
| sileht | jd__, still not enough quick to process data generated by mynagios | 15:04 |
| jd__ | ah there's still lag? but is there less? | 15:04 |
| sileht | jd__, it looks to process file more quickly, but we don't have any useful number | 15:08 |
| jd__ | sileht: why don't you have numbers? | 15:09 |
| *** llu has quit IRC | 15:11 | |
| *** openstack has joined #openstack-telemetry | 15:19 | |
| *** jd__ has joined #openstack-telemetry | 15:19 | |
| *** fc__ has joined #openstack-telemetry | 15:19 | |
| *** sheeprine has joined #openstack-telemetry | 15:19 | |
| jd__ | cdent: btw I don't recall if I replied again about gabbi, but if we use newer feature than 1.7 we should fix that version number then :( | 15:19 |
| sileht | jd__, I writting a script to compare master and my branch with a 10 minutes run | 15:20 |
| jd__ | wahou | 15:21 |
| * cdent looks at release notes of gabbi | 15:21 | |
| *** mgagne has joined #openstack-telemetry | 15:22 | |
| *** sudipto has joined #openstack-telemetry | 15:22 | |
| *** Taytay has joined #openstack-telemetry | 15:22 | |
| *** mgagne has quit IRC | 15:23 | |
| cdent | 1.19.1 came about because of integration of telemetry stuff with puppet/tripleo http://gabbi.readthedocs.io/en/latest/release.html#id13 | 15:23 |
| *** mgagne has joined #openstack-telemetry | 15:23 | |
| *** mgagne is now known as Guest58531 | 15:23 | |
| cdent | and gnocchi uses LAST_URL which was in 1.19.0 | 15:23 |
| cdent | jd__: so I would think at least 1.19.1 | 15:26 |
| *** leitan has quit IRC | 15:26 | |
| jd__ | cdent: do u want to send a patch or should I? | 15:26 |
| jd__ | is this only in Aodh? | 15:26 |
| *** kong has joined #openstack-telemetry | 15:26 | |
| cdent | I aodh/panko/ceilo are probably fine for older, it is gnocchi that would need to be up higher than 1.7 | 15:28 |
| cdent | i haven't got time to do it right now | 15:28 |
| openstackgerrit | Julien Danjou proposed openstack/gnocchi: rest,indexer: handle ResourceUUID conversion in the REST API https://review.openstack.org/413016 | 15:29 |
| openstackgerrit | Julien Danjou proposed openstack/gnocchi: sqlalchemy: use a list rather than if/elif to convert type in queries https://review.openstack.org/413015 | 15:29 |
| openstackgerrit | Julien Danjou proposed openstack/gnocchi: utils: allow ResourceUUID to convert UUID https://review.openstack.org/413014 | 15:29 |
| openstackgerrit | Julien Danjou proposed openstack/gnocchi: Remove workaround to upgrade from 2.2.0 https://review.openstack.org/419538 | 15:29 |
| *** Tamayo has quit IRC | 15:30 | |
| *** Tamayo has joined #openstack-telemetry | 15:32 | |
| *** gordc has joined #openstack-telemetry | 15:36 | |
| *** sheel has quit IRC | 15:37 | |
| *** david-lyle has joined #openstack-telemetry | 15:47 | |
| *** nadya has quit IRC | 15:51 | |
| *** Jack_Iv has joined #openstack-telemetry | 16:01 | |
| *** larainema has quit IRC | 16:06 | |
| *** gongysh has quit IRC | 16:23 | |
| openstackgerrit | Hanxi Liu proposed openstack/ceilometer: Deprecate collector https://review.openstack.org/413920 | 16:25 |
| *** efoley has joined #openstack-telemetry | 16:26 | |
| *** vint_bra has joined #openstack-telemetry | 16:35 | |
| gordc | sileht: when you have time, can you take a look.: https://review.openstack.org/#/c/333129/ | 16:36 |
| *** vint_bra1 has quit IRC | 16:37 | |
| *** harlowja has joined #openstack-telemetry | 16:39 | |
| *** lhx_ has quit IRC | 16:40 | |
| *** vint_bra1 has joined #openstack-telemetry | 16:42 | |
| sileht | gordc, done | 16:43 |
| *** ddyer has quit IRC | 16:44 | |
| *** vint_bra has quit IRC | 16:45 | |
| gordc | sileht: thanks | 16:45 |
| gordc | sileht: you can go back to flipping table re:kafka | 16:46 |
| *** sheel has joined #openstack-telemetry | 16:48 | |
| sileht | gordc, lol | 16:48 |
| *** ddyer has joined #openstack-telemetry | 16:48 | |
| sileht | gordc, I'm waiting for requirements team mail | 16:48 |
| *** liusheng has quit IRC | 16:49 | |
| sileht | gordc, this situation is a joke | 16:49 |
| *** liusheng has joined #openstack-telemetry | 16:50 | |
| gordc | sileht: i like " We really do need to | 16:50 |
| gordc | investigate migrating" | 16:50 |
| gordc | isn't that what you asked 2 months ago | 16:50 |
| sileht | yes... | 16:50 |
| gordc | sigh | 16:51 |
| sileht | gordc, the 2x memory is just obvious, no need to do any tests to get that... | 16:51 |
| sileht | gordc, kafka use a thread to send message and use a queue where it copy all your messages to process | 16:52 |
| sileht | gordc, because their use the old 'SYNC' API, | 16:52 |
| sileht | gordc, the memory is freed only when the message is removed from the queue | 16:52 |
| sileht | gordc, while with the async one, you can return to your code and python will free your message | 16:53 |
| gordc | sileht: why did they need sync again? | 16:53 |
| sileht | gordc, their mix 'ensure delivery' with 'need of sync' | 16:54 |
| sileht | with async you can just register callback when the message is delivered | 16:54 |
| gordc | yeah, that's what i thought as well | 16:55 |
| sileht | I think their just don't have time to work on monasca | 16:55 |
| * gordc thinks so too. | 16:55 | |
| *** jmlowe has joined #openstack-telemetry | 16:58 | |
| jmlowe | gordc: Hey, you remember that bug where partially converted aggregates would cause the metricd to choke and then we did a work around to ignore that? I think that finally came back to haunt me | 16:59 |
| gordc | :( | 17:00 |
| jmlowe | I've got 6M+ objects in my gnocchi ceph pool and anytime I turn on metricd my ceph starts throwing slow ops and marking osd's down because they are too heavily loaded | 17:00 |
| jmlowe | this took about 15-20 min 'rados ls -p gnocchi |grep measure |wc -l | 17:01 |
| jmlowe | 3482025' | 17:01 |
| jmlowe | which doesn't seem right | 17:01 |
| jmlowe | so what do you think? | 17:01 |
| * gordc trying to remember what the object names are in ceph | 17:02 | |
| jmlowe | stuff like this gnocchi_d0d21771-2a56-4148-9317-ac3b73fd8916_1451520000.0_median_3600.0_v3 | 17:05 |
| jmlowe | measure_5c661af5-8329-4587-b1de-4fcc5bd2b582_f0ee88f2-e38b-450a-8f5f-8698770704be_20170117_21:43:06 | 17:05 |
| gordc | yeah, got it. so measure is just unprocessed stuff | 17:05 |
| sileht | gordc, read the joe comment here: https://review.openstack.org/#/c/420579/ | 17:05 |
| gordc | what's the metricd errors you get? | 17:05 |
| *** sudipto has quit IRC | 17:07 | |
| *** sudipto_ has quit IRC | 17:07 | |
| gordc | jmlowe: ^ | 17:08 |
| gordc | sileht: i commented on ML. i don't think it makes sense to even argue with "it doesn't meet our perforamnce requirements". i feel like there's always going to be an excuse. | 17:09 |
| jmlowe | nothing really, just hangs | 17:09 |
| gordc | jmlowe: oh? when you run in debug you get nothing? | 17:10 |
| *** dschultz has joined #openstack-telemetry | 17:10 | |
| gordc | i wonder if ceph has some limit on omap values? | 17:10 |
| gordc | sileht: ^ you know if ceph has a 3Million value limit? | 17:11 |
| jmlowe | I get lots of tooz, looking for other stuff in the spam | 17:11 |
| gordc | don't think that should be block reads though | 17:11 |
| sileht | gordc, everything have a limit | 17:12 |
| gordc | jmlowe: maybe stop sending metrics to it and kill api? i've never actually gotten that ceph error before | 17:12 |
| gordc | sileht: does your patience have a limit? :P | 17:12 |
| jmlowe | 2017-01-18 12:11:41.608 89801 DEBUG gnocchi.storage._carbonara [-] Computed new metric 0033f33a-e753-46a3-ae95-fff19b95f61d with 6 new measures in 0.51 seconds (4204 points/s, 283 measures/s) process_new_measures /usr/lib/python2.7/site-packages/gnocchi/storage/_carbonara.py:589 | 17:13 |
| jmlowe | so it's processing, I went way down to 4 metricd workers | 17:13 |
| jmlowe | I'll probably need to turn my ceilometer way way down, to the one metric I really need | 17:14 |
| gordc | 4 workers? that won't keep up. | 17:15 |
| gordc | especially with the default archive policies... and because we didn't really fix the scheduling logic. | 17:16 |
| stevelle | is it workers holding ceph up or ceph holding workers up? | 17:16 |
| gordc | so it only starts to do stuff when you turn down # of workers? | 17:17 |
| jmlowe | what I'm thinking is that ceph is choking on the large number of objects when it goes to list | 17:18 |
| jd__ | gordc: if the perf are bad they should just go ahead and work upstream and fix it… | 17:18 |
| jmlowe | that hangs the workers, which then don't keep up, new objects keep coming | 17:18 |
| gordc | jd__: kafka? | 17:18 |
| jd__ | gordc: yes sorry | 17:19 |
| jmlowe | larger number of workers causes ceph osd's to fall over | 17:19 |
| gordc | jd__: yeah, i'm trying to be diplomatic... i'm not good at taht. | 17:20 |
| openstackgerrit | Merged openstack/panko: Update requirements https://review.openstack.org/421642 | 17:21 |
| *** nadya has joined #openstack-telemetry | 17:21 | |
| jd__ | gordc: oh that mail was not so bad :) | 17:21 |
| gordc | jmlowe: i don't think large number of objects should be issue in v3. we only list first x amount regardless | 17:21 |
| gordc | jd__: i removed all the f bombs | 17:21 |
| *** tesseract has quit IRC | 17:21 | |
| sileht | gordc, I have answered to the thread | 17:22 |
| jd__ | gordc: lol | 17:22 |
| gordc | sileht: i'm sad you didn't add f bombs.lol | 17:24 |
| gordc | jmlowe: what you mention workers do you mean you are deploying multiple metricd agents or a single agent but setting workers value in conf? | 17:24 |
| sileht | jd__, I finished my testing with my branch in 10 minutes on my setup I handle ~10000 more measures | 17:25 |
| jmlowe | was 48 workers each on two agents | 17:25 |
| sileht | jd__, 33006 vs 41414 now | 17:25 |
| gordc | sileht: your magic numpy stuff? | 17:26 |
| sileht | gordc, yes | 17:26 |
| gordc | nice. | 17:27 |
| gordc | jmlowe: how often is it scheduling metrics? | 17:28 |
| gordc | metric_processing_delay | 17:28 |
| jd__ | sileht: so a good 25%, nice | 17:28 |
| jd__ | sileht: what's still slow? I/O? | 17:29 |
| gordc | jmlowe: 48 workers * 2 means it's scheduling ~1500 objects which isn't terrible. | 17:29 |
| gordc | although i imagine it could be more since scheduling is wack and it starves things | 17:30 |
| sileht | jd__, still cpu bound | 17:32 |
| sileht | jd__, I use ~ 7% of IO of my ssd | 17:32 |
| jd__ | sileht: gordc: i could not resist to jump in that thread :D | 17:33 |
| *** harlowja has quit IRC | 17:33 | |
| jd__ | sileht: damn | 17:33 |
| *** Jack_Iv has quit IRC | 17:33 | |
| jd__ | sileht: we need a new lead to optimize | 17:33 |
| sileht | jd__, thx for the support :p | 17:33 |
| gordc | there's too much math. build your own pandas? | 17:37 |
| *** david-lyle is now known as bailing-wire | 17:37 | |
| *** nadya has quit IRC | 17:37 | |
| jmlowe | got distracted, donuts showed up | 17:39 |
| jmlowe | didn't really want one but felt obligated | 17:39 |
| *** ddyer has quit IRC | 17:39 | |
| * gordc thinks whether he should get donuts or lunch | 17:40 | |
| gordc | jd__: when you tagging all the projects? | 17:40 |
| jd__ | gordc: probably at rc1, except for Gnocchi that I'd like to push around m3 | 17:41 |
| jd__ | so next week | 17:41 |
| jd__ | gordc: donuts :( get real food dude | 17:41 |
| jd__ | gordc: I think going from pandas to numpy might be faster but… it might be hard | 17:42 |
| jd__ | (ETOOLAZY) | 17:42 |
| jd__ | i'm waiting for profiling data from sileht | 17:42 |
| *** ddyer has joined #openstack-telemetry | 17:43 | |
| gordc | jd__: but donuts are a lot closer. and i'm lazy. | 17:43 |
| *** catintheroof has quit IRC | 17:44 | |
| gordc | jd__: i didn't realise so much weird stuff with pandas. | 17:44 |
| jd__ | gordc: i'm sure there's UberEATS or something around | 17:44 |
| *** catintheroof has joined #openstack-telemetry | 17:44 | |
| *** catintheroof has quit IRC | 17:44 | |
| sileht | jd__, tomorrow I will profile the whole processing method in metricd instead of just carbonara | 17:45 |
| jd__ | there might be interesting stuff | 17:45 |
| *** catintheroof has joined #openstack-telemetry | 17:45 | |
| jd__ | a few time.sleep() to remove maybe | 17:45 |
| gordc | jd__: i'm not in city right now... also, i did the math and i feel bad for ubereats ppl. they losing money.lol | 17:45 |
| * jd__ . o O (WAIT WHAT) | 17:45 | |
| jd__ | gordc: enjoy 'til they run dry :P | 17:46 |
| gordc | lol | 17:46 |
| sileht | jd__, I have used vmprof that someone pointed yesterday: http://vmprof.com/#/2a7b9a9b7ebbe2bdc8e768ac06d196b0 | 17:46 |
| *** bailing-wire has quit IRC | 17:46 | |
| gordc | sileht: i think i deleted my profiles from but not sure how relevant they are after all your changes. | 17:47 |
| jd__ | http://vmprof.com/#/ has been spammed by anonymous working on gnocchi i wonder who that is | 17:47 |
| sileht | the page yesterday when I have discover than some pandas method are just slow: http://vmprof.com/#/92884f01017281fc0458ecdab72d7c2f | 17:48 |
| sileht | bbl | 17:49 |
| *** catintheroof has quit IRC | 17:50 | |
| jmlowe | gah, I'm going to have to pick this up in a bit, probably best to not troubleshoot during a live demo especially when I'm in the same room | 17:50 |
| jmlowe | the donuts eliminated all my yolo | 17:51 |
| jd__ | sad | 17:52 |
| gordc | sileht: when i did profile, the (un)serialize methods weren't that much compared to other stuff | 17:52 |
| gordc | after the io methods, it was split and from_group_serie that was next highest. | 17:53 |
| *** efoley has quit IRC | 18:09 | |
| *** shardy has quit IRC | 18:15 | |
| *** shardy has joined #openstack-telemetry | 18:16 | |
| *** yprokule has quit IRC | 18:22 | |
| *** fguillot has joined #openstack-telemetry | 18:45 | |
| *** shardy is now known as shardy_afk | 18:52 | |
| jmlowe | what do you think about this message "2017-01-18 14:05:13.796 135338 DEBUG gnocchi.storage._carbonara [-] Metric 008988b2-a598-442c-b100-86de0f7bcd70 locked during 154.61 seconds process_new_measures /usr/lib/python2.7/site-packages/gnocchi/storage/_carbonara.py:595" | 19:07 |
| jmlowe | that seems really slow | 19:07 |
| gordc | that does... usually it's in milliseconds. | 19:08 |
| gordc | maybe it's ceph configuration? my write throughput dropped over time because my filestore configuration options weren't great (or did not match the io requirements of gnoccchi) | 19:10 |
| *** nadya has joined #openstack-telemetry | 19:14 | |
| *** cdent has quit IRC | 19:14 | |
| *** eglynn has quit IRC | 19:31 | |
| *** Jack_Iv has joined #openstack-telemetry | 19:32 | |
| EmilienM | jd__: ++ | 19:42 |
| *** jmlowe1 has joined #openstack-telemetry | 19:56 | |
| *** jmlowe has quit IRC | 19:56 | |
| *** bailing-wire has joined #openstack-telemetry | 20:00 | |
| *** bailing-wire is now known as david-lyle | 20:02 | |
| jd__ | jmlowe1: this is way too slow to be true indeed | 20:08 |
| *** Guest62565 has quit IRC | 20:12 | |
| *** nadya has quit IRC | 20:13 | |
| *** Guest62565 has joined #openstack-telemetry | 20:16 | |
| jmlowe1 | even with one worker, when I start metricd I wind up with this in ceph "2 requests are blocked > 32 sec" | 20:16 |
| openstackgerrit | Julien Danjou proposed openstack/aodh: Move policy.json out of etc https://review.openstack.org/422220 | 20:20 |
| gordc | jmlowe1: what's your ceph health look like? | 20:22 |
| gordc | this seems specific to ceph... i vaguely remember seenig that but i was playing around with original profiling. | 20:23 |
| jmlowe1 | HEALTH_WARN 2 requests are blocked > 32 sec | 20:24 |
| *** adriant has joined #openstack-telemetry | 20:44 | |
| *** jmlowe1 has quit IRC | 20:45 | |
| *** ryanpetrello has left #openstack-telemetry | 20:46 | |
| *** jmlowe has joined #openstack-telemetry | 20:47 | |
| *** Jack_Iv has quit IRC | 21:10 | |
| *** Jack_Iv has joined #openstack-telemetry | 21:10 | |
| openstackgerrit | gordon chung proposed openstack/ceilometer: Add support of refereshing the resource info in local cache https://review.openstack.org/333129 | 21:12 |
| *** Jack_Iv has quit IRC | 21:14 | |
| *** dschultz has quit IRC | 21:38 | |
| *** vint_bra1 has quit IRC | 21:49 | |
| *** thorst has quit IRC | 22:07 | |
| openstackgerrit | Merged openstack/gnocchi: opts: list entry points with pkg_resources rather than stevedore https://review.openstack.org/419962 | 22:09 |
| *** fguillot has quit IRC | 22:11 | |
| *** dave-mccowan has quit IRC | 22:28 | |
| *** Guest62565 has quit IRC | 22:29 | |
| *** david-lyle has quit IRC | 22:32 | |
| *** jmlowe has quit IRC | 22:35 | |
| *** dschultz has joined #openstack-telemetry | 22:38 | |
| *** thorst has joined #openstack-telemetry | 22:39 | |
| *** david-lyle has joined #openstack-telemetry | 22:40 | |
| *** thorst has quit IRC | 22:41 | |
| openstackgerrit | Merged openstack/ceilometer: Add support of refereshing the resource info in local cache https://review.openstack.org/333129 | 22:43 |
| openstackgerrit | Merged openstack/python-ceilometerclient: Adding default project and domain if nothing is specified https://review.openstack.org/408421 | 22:53 |
| *** dschultz has quit IRC | 23:00 | |
| *** ddyer has quit IRC | 23:24 | |
| *** sheel has quit IRC | 23:24 | |
| *** stevemar has quit IRC | 23:24 | |
| *** ddyer has joined #openstack-telemetry | 23:24 | |
| *** sheel has joined #openstack-telemetry | 23:24 | |
| *** stevemar has joined #openstack-telemetry | 23:24 | |
| *** larainema has joined #openstack-telemetry | 23:34 | |
| *** DinaBelova has quit IRC | 23:34 | |
| *** jp_ has joined #openstack-telemetry | 23:35 | |
| *** DinaBelova has joined #openstack-telemetry | 23:35 | |
| *** aignatov has joined #openstack-telemetry | 23:35 | |
| *** pradk has quit IRC | 23:42 | |
| *** DinaBelova has quit IRC | 23:57 | |
| *** aignatov has quit IRC | 23:57 | |
| *** aignatov has joined #openstack-telemetry | 23:57 | |
| *** DinaBelova has joined #openstack-telemetry | 23:57 | |
| *** DinaBelova has quit IRC | 23:59 | |
| *** aignatov has quit IRC | 23:59 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!