Wednesday, 2016-03-16

« Tuesday, 2016-03-15 Index Thursday, 2016-03-17 »
*** yuntongjin has joined #openstack-telemetry00:17
*** openstack has joined #openstack-telemetry00:26
*** sheeprine has quit IRC00:30
*** yuntongjin1 has joined #openstack-telemetry00:32
*** sheeprine has joined #openstack-telemetry00:34
*** yuntongjin has quit IRC00:34
*** yuntongjin has joined #openstack-telemetry00:40
*** yuntongjin1 has quit IRC00:42
*** yuntongjin1 has joined #openstack-telemetry00:43
*** yuntongjin has quit IRC00:45
*** afazekas has quit IRC00:53
*** dave-mccowan has joined #openstack-telemetry00:57
*** afazekas has joined #openstack-telemetry00:59
*** david-lyle has joined #openstack-telemetry01:06
*** boris-42 has quit IRC01:24
*** ljxiash has joined #openstack-telemetry01:25
*** afazekas has quit IRC01:25
*** afazekas has joined #openstack-telemetry01:28
*** ljxiash has quit IRC01:35
*** ljxiash has joined #openstack-telemetry01:36
*** vishwanathj has joined #openstack-telemetry01:37
*** ljxiash has quit IRC01:40
*** mdih has joined #openstack-telemetry01:58
*** mdih has quit IRC02:06
*** ljxiash has joined #openstack-telemetry02:12
*** mdih has joined #openstack-telemetry02:12
*** mdih has quit IRC02:13
*** mdih has joined #openstack-telemetry02:13
*** liusheng has quit IRC02:16
*** caisan has joined #openstack-telemetry02:16
*** liusheng has joined #openstack-telemetry02:16
*** thorst_ has quit IRC02:18
*** mdih has quit IRC02:18
openstackgerritZhenyu Zheng proposed openstack/python-aodhclient: Add name support for show, update and delete  https://review.openstack.org/29230502:22
openstackgerritliusheng proposed openstack/python-aodhclient: Add --filter to "alarm list"  https://review.openstack.org/29223302:26
*** mdih has joined #openstack-telemetry02:31
*** liamji has joined #openstack-telemetry02:32
openstackgerritliusheng proposed openstack/python-aodhclient: Add --filter to "alarm list"  https://review.openstack.org/29223302:49
*** dave-mccowan has quit IRC03:17
*** thorst has joined #openstack-telemetry03:19
*** thorst has quit IRC03:27
*** yuntongjin1 has quit IRC03:28
*** ljxiash has quit IRC03:38
*** vishwanathj has quit IRC04:05
*** links has joined #openstack-telemetry04:10
*** flwang1 has quit IRC04:16
*** thorst has joined #openstack-telemetry04:25
*** thorst has quit IRC04:32
*** ljxiash has joined #openstack-telemetry05:09
*** zqfan has joined #openstack-telemetry05:10
*** ljxiash has quit IRC05:10
*** ljxiash has joined #openstack-telemetry05:10
*** boris-42 has joined #openstack-telemetry05:13
*** thorst has joined #openstack-telemetry05:30
*** vishwanathj has joined #openstack-telemetry05:31
*** vishwanathj has quit IRC05:31
*** thorst has quit IRC05:37
*** david-lyle has quit IRC05:45
*** david-lyle has joined #openstack-telemetry05:46
*** vishwanathj has joined #openstack-telemetry05:48
*** vishwanathj has quit IRC05:51
*** _nadya_ has joined #openstack-telemetry06:00
*** flwang1 has joined #openstack-telemetry06:01
*** _nadya_ has quit IRC06:01
openstackgerritOpenStack Proposal Bot proposed openstack/ceilometer: Imported Translations from Zanata  https://review.openstack.org/29271506:07
*** flwang1 has quit IRC06:09
*** vishwanathj has joined #openstack-telemetry06:34
*** vishwanathj is now known as vishwanathj_zzz06:35
*** thorst has joined #openstack-telemetry06:35
*** thorst has quit IRC06:42
*** yprokule has joined #openstack-telemetry07:05
*** flwang1 has joined #openstack-telemetry07:13
*** tesseract has joined #openstack-telemetry07:16
*** tesseract is now known as Guest6496707:17
*** _nadya_ has joined #openstack-telemetry07:19
*** _nadya_ has quit IRC07:21
*** Guest64967 has quit IRC07:21
*** ljxiash has quit IRC07:30
*** flwang1 has quit IRC07:30
*** ljxiash has joined #openstack-telemetry07:31
*** ljxiash has quit IRC07:32
openstackgerritMehdi Abaakouk (sileht) proposed openstack/gnocchi: Add some resource types tests  https://review.openstack.org/27041907:35
openstackgerritMehdi Abaakouk (sileht) proposed openstack/gnocchi: Implements resource attribute number  https://review.openstack.org/27009107:35
openstackgerritMehdi Abaakouk (sileht) proposed openstack/gnocchi: Implements resource attribute bool  https://review.openstack.org/27041807:35
openstackgerritMehdi Abaakouk (sileht) proposed openstack/gnocchi: Implements resource attribute uuid  https://review.openstack.org/27009007:35
openstackgerritMehdi Abaakouk (sileht) proposed openstack/gnocchi: Don't create Ceilometer resource types by default.  https://review.openstack.org/27032207:35
openstackgerritMehdi Abaakouk (sileht) proposed openstack/gnocchi: Move legacy Ceilometer resource into indexer.  https://review.openstack.org/27026607:35
openstackgerritMehdi Abaakouk (sileht) proposed openstack/gnocchi: Implements resource attribute string  https://review.openstack.org/26988807:35
openstackgerritMehdi Abaakouk (sileht) proposed openstack/gnocchi: Retry to delete resource types tables on deadlock  https://review.openstack.org/29277607:35
*** ljxiash has joined #openstack-telemetry07:36
*** tesseract- has joined #openstack-telemetry07:36
*** thorst has joined #openstack-telemetry07:40
*** chlong has quit IRC07:42
*** thorst has quit IRC07:48
openstackgerritMehdi Abaakouk (sileht) proposed openstack/aodh: devstack: allow uwsgi deployments  https://review.openstack.org/29224507:49
openstackgerritMehdi Abaakouk (sileht) proposed openstack/gnocchi: devstack: allow uwsgi deployments  https://review.openstack.org/29207707:50
*** jwcroppe has quit IRC07:57
*** belmoreira has joined #openstack-telemetry07:58
*** jwcroppe has joined #openstack-telemetry07:58
openstackgerritZhenyu Zheng proposed openstack/python-aodhclient: Add --query to alarm list  https://review.openstack.org/29154108:02
*** jwcroppe has quit IRC08:02
openstackgerritZhenyu Zheng proposed openstack/python-aodhclient: Add --query to alarm list  https://review.openstack.org/29154108:13
openstackgerritZhenyu Zheng proposed openstack/python-aodhclient: Add --query to alarm list  https://review.openstack.org/29154108:18
*** permalac has joined #openstack-telemetry08:20
*** kbyrne has joined #openstack-telemetry08:22
*** cdent has joined #openstack-telemetry08:26
*** ig0r_ has quit IRC08:39
*** hparekh_ has quit IRC08:42
*** hparekh has joined #openstack-telemetry08:44
*** thorst has joined #openstack-telemetry08:46
*** thorst has quit IRC08:52
*** safchain has joined #openstack-telemetry08:54
openstackgerritZhenyu Zheng proposed openstack/python-aodhclient: Add --query to alarm list  https://review.openstack.org/29154108:55
*** ljxiash has quit IRC08:58
*** ljxiash has joined #openstack-telemetry09:05
*** nijaba has quit IRC09:05
openstackgerritZhenyu Zheng proposed openstack/python-aodhclient: Add --query to alarm list  https://review.openstack.org/29154109:10
*** ljxiash has quit IRC09:10
*** nijaba has joined #openstack-telemetry09:11
*** nijaba has quit IRC09:11
*** nijaba has joined #openstack-telemetry09:11
*** ljxiash has joined #openstack-telemetry09:35
*** yassine__ has joined #openstack-telemetry09:36
*** _nadya_ has joined #openstack-telemetry09:37
*** ljxiash has quit IRC09:40
openstackgerritMerged openstack/ceilometer: change dns and trove notifications to declarative  https://review.openstack.org/29213609:42
*** thorst has joined #openstack-telemetry09:50
*** openstackgerrit has quit IRC09:53
*** openstackgerrit_ is now known as openstackgerrit09:53
*** openstackgerrit has quit IRC09:53
*** openstackgerrit_ has joined #openstack-telemetry09:54
*** openstackgerrit_ is now known as openstackgerrit09:54
*** openstackgerrit_ has joined #openstack-telemetry09:55
*** openstackgerrit has quit IRC09:55
*** openstackgerrit has joined #openstack-telemetry09:56
*** thorst has quit IRC09:57
*** lsmola has joined #openstack-telemetry10:05
*** fwdit has joined #openstack-telemetry10:21
*** caisan has quit IRC10:37
*** ljxiash has joined #openstack-telemetry10:51
*** thorst has joined #openstack-telemetry10:55
*** ljxiash has quit IRC10:56
*** pece has joined #openstack-telemetry10:59
*** flwang1 has joined #openstack-telemetry10:59
*** thorst has quit IRC11:02
*** mdih has quit IRC11:12
*** openstackgerrit has quit IRC11:18
*** openstackgerrit has joined #openstack-telemetry11:19
*** liamji has quit IRC11:27
openstackgerritzhaozhilong proposed openstack/gnocchi: Add a decorate for noauth function  https://review.openstack.org/29338111:31
*** thorst has joined #openstack-telemetry11:35
*** cdent has quit IRC11:43
*** gordc has joined #openstack-telemetry11:44
*** dave-mccowan has joined #openstack-telemetry11:52
*** pece has quit IRC12:01
*** pece has joined #openstack-telemetry12:09
*** cdent has joined #openstack-telemetry12:10
*** pece has quit IRC12:10
jd__sileht: https://bugs.launchpad.net/gnocchi/+bug/1557724 if you did not notice12:12
openstackLaunchpad bug 1557724 in Gnocchi "gnocchi-metricd causes slow queries on ceph pool" [Medium,Triaged]12:12
*** pece has joined #openstack-telemetry12:13
*** pcaruana has quit IRC12:28
openstackgerritlitong01 proposed openstack/ceilometer: Add http publisher  https://review.openstack.org/29062412:28
*** boris-42 has quit IRC12:34
*** fwdit has quit IRC12:39
*** fwdit has joined #openstack-telemetry12:39
*** pradk has joined #openstack-telemetry12:51
*** rcernin has joined #openstack-telemetry13:03
openstackgerritMerged openstack/python-aodhclient: Don't support "alarm search" and mandatory --type in "alarm list"  https://review.openstack.org/28395813:07
openstackgerritMerged openstack/python-aodhclient: Add --query to alarm list  https://review.openstack.org/29154113:07
*** jwcroppe has joined #openstack-telemetry13:08
*** peristeri has joined #openstack-telemetry13:08
*** kro_sinus_hell is now known as krot_sinus_purga13:15
*** krot_sinus_purga is now known as kro_sniffles13:15
*** ljxiash has joined #openstack-telemetry13:22
*** ig0r_ has joined #openstack-telemetry13:24
*** _nadya_ has quit IRC13:25
*** ljxiash has quit IRC13:26
*** _nadya_ has joined #openstack-telemetry13:29
openstackgerritzhangguoqing proposed openstack/ceilometer: Use assertIn for test  https://review.openstack.org/29342813:32
cdentI'm surprised you didn't burn me out gordc13:32
cdentthat would have been irony at its finest13:32
gordccdent: i don't understand irony13:32
gordcalthough i should've now that you mention it13:32
cdentgordc: watch "reality bites", learn all about irony13:33
cdenthttps://www.youtube.com/watch?v=7GM--22zOlw13:35
gordci can't remember if i've seen this.13:35
gordcthe title sounds like some 'jersey shore' type show13:36
gordci don't know the uk equivalent.13:36
cdenthere its geordi shore13:37
cdentsame things baically13:37
openstackgerritzhangguoqing proposed openstack/aodh: Use assertIn for test  https://review.openstack.org/29343913:40
*** shardy has joined #openstack-telemetry13:42
*** zqfan has quit IRC13:52
*** rbak has joined #openstack-telemetry13:56
openstackgerritgordon chung proposed openstack/ceilometer: core status cleanup  https://review.openstack.org/29344814:03
gordcp!hist14:03
gordcp!hist14:03
gordccdent: i'm assuming what i just typed isn't private...14:04
cdentI can see you saying p!hist14:04
cdentbut now the responses14:04
cdentnot14:04
gordccool cool14:04
*** mgagne has quit IRC14:07
*** mgagne has joined #openstack-telemetry14:07
*** alextricity25 has quit IRC14:07
*** alextricity25 has joined #openstack-telemetry14:08
*** KrishR has joined #openstack-telemetry14:13
openstackgerritzhangguoqing proposed openstack/ceilometer: Use assertIn and assertNotIn  https://review.openstack.org/29342814:18
openstackgerritzhangguoqing proposed openstack/aodh: Use assertIn and assertNotIn for test  https://review.openstack.org/29343914:21
openstackgerritgordon chung proposed openstack/aodh: cleanup core list  https://review.openstack.org/29346314:25
*** nicodemus_ has joined #openstack-telemetry14:29
*** _nadya_ has quit IRC14:41
*** _nadya_ has joined #openstack-telemetry14:45
nicodemus_hello gordc15:06
nicodemus_do you have a minute?15:06
*** david-lyle has quit IRC15:15
*** david-lyle has joined #openstack-telemetry15:15
*** purplerbot has quit IRC15:16
*** purplerbot has joined #openstack-telemetry15:17
*** david-lyle_ has joined #openstack-telemetry15:34
*** david-lyle has quit IRC15:34
*** vishwanathj_zzz is now known as vishwanathj15:38
*** jroll has quit IRC15:41
*** jroll has joined #openstack-telemetry15:41
*** jroll has quit IRC15:41
*** jroll has joined #openstack-telemetry15:41
*** yprokule has quit IRC15:47
*** cdent has quit IRC16:05
*** belmoreira has quit IRC16:11
*** david-lyle_ is now known as david-lyle16:20
*** fwdit has quit IRC16:30
*** cdent has joined #openstack-telemetry16:30
*** ljxiash has joined #openstack-telemetry16:36
*** links has quit IRC16:38
gordcnicodemus_: sorry, was moving stuff. whatsup?16:46
*** flwang1 has quit IRC16:49
nicodemus_no problem gordc, wanted to ask you a quick question if you have a moment16:49
gordcsure.16:51
gordcnicodemus_: going to wait out this rain before going to lunch anyways16:51
nicodemus_if the gnocchi backlog is large, does it make sense for metricd to take longer to process measures?16:54
gordcnicodemus_: i think it more depends no the granularity of your policies. but even then, it should cap out since we time split the measures.16:57
gordcdo you know which metric is taking a long time? or is it all of them taking longer?16:58
nicodemus_I have now a 100k backlog and, on average, all of them take longer compared to a 50k backlog17:00
gordchmm.. this is for ceph right?17:02
nicodemus_some take 300 secs, I'm guessing because of that worker waiting to acquire a rw lock on the measure object. But those are the least, the average for 100k backlog is around 10 secs17:02
nicodemus_yes, for a 22 osd, 4-node ceph cluster17:02
gordci see. if you enable the extremely verbose 'retrieve, write, expire' logs. can you check to see if anything sticks out as slow?17:04
*** _nadya_ has quit IRC17:07
nicodemus_gordc, let me check17:10
nicodemus_gordc, there is one very slow retrieve17:16
nicodemus_http://paste.openstack.org/show/490761/17:17
nicodemus_4 lines before the end, one retrieve took 153 secs17:17
stevellethe retrieve and store each are outliers there17:19
gordci see. i imagine it's probably that locking issue that you mentioned17:20
gordcit doesn't seem to be consistent with other granularities of similar/larger size or related to the aggregation type17:21
nicodemus_here is another case: http://paste.openstack.org/show/490763/17:22
nicodemus_where the delay is for a different object (granularity & aggregation method)17:23
nicodemus_these are extreme cases, not all metrics take that much time... but on average, I think the larger the backlog the longer it takes to process measures17:25
stevelleone of those outliers will heavily skew not only the average, but the flow rate17:25
gordcstevelle: are you folks still in planning/configuration stage for setting up testing? (i'm assuming you're working with llu to test some performance stuff)17:27
stevellegordc: yes, we were not going to do ceph for our first tests though17:27
gordcstevelle: cool cool. that's fine. i was hoping to do some testing myself but lab is getting delayed constantly.17:28
gordcnicodemus_: is this with 2 * 24 workers?17:28
stevellelabs are always 2 months or more behind the plan. :((17:28
nicodemus_gordc, yes. Plus those 2 are the 2 dedicated servers (instance metricd are slower)17:29
gordcstevelle: yeah, unfortuantely. i'm at "or more" stage.17:30
gordcnicodemus_: so you have 4 separate metricd services?17:31
nicodemus_gordc, in an effort not to get behing in measure processing, I have now 2 HW servers w/24 CPU cores and 2 instances w/16 vCPU17:32
nicodemus_all of them have 24 workers each17:32
nicodemus_those logs are with the 2 servers only17:32
gordcnicodemus_: didn't it perform 'better' when you disabled the two instance metricd services yesterday?17:33
nicodemus_another thing I see is that metricd cannot scale horizontally beyond a certain point... adding more metricd workers causes overall measure processing to slow down17:33
nicodemus_yeah, that's why I left it with the 2 servers now17:33
stevelleI don't know a lot yet but I would suspect that this is a ceph saturation, not metricd17:33
nicodemus_ceph does complain about slow queries: https://bugs.launchpad.net/gnocchi/+bug/155772417:34
openstackLaunchpad bug 1557724 in Gnocchi "gnocchi-metricd causes slow queries on ceph pool" [Medium,Triaged]17:34
gordcstevelle: yeah, that's what i was thinking -- we're just exhaustnig the number of connections.17:35
stevelleI'm not sure if it's connections or something else, but I don't know much about ceph really17:36
stevellewhat is the block_size being used here?17:36
gordcyeah, sileht is the ceph expert among us but i don't believe he's online currently.17:36
gordcnicodemus_: did you set a aggregation_workers_number value?17:37
nicodemus_gordc, yes, all metricd hosts have aggregation_workers_number = 217:40
nicodemus_as far as network goes, I can say there's no link saturation (installed a cacti a week ago to check that)17:41
gordcoh. i'd maybe bump that up to the number of aggregates you're calculating. but that will probably lead to more parallel connections to ceph.17:41
gordcmaybe try dropping metricd workers and increasing aggregate workers (disclaimer: i'm just guessing right now)17:43
gordcbiab. going to grab some lunch.17:43
nicodemus_it's worth the try :)17:44
stevellenicodemus_: did you do similar tests with file storage earlier in Mitaka?17:44
*** _nadya_ has joined #openstack-telemetry17:45
nicodemus_stevelle, only for a brief time17:45
nicodemus_not nearly as extensive as with ceph17:45
*** vishwanathj is now known as vishwanathj_chow17:45
stevellenicodemus_: that's reasonable. did you publish any data from that publicly?17:45
stevelleI'd like to include a review of that in our build spec17:46
nicodemus_I'd have to look, one test I remember doing was using file as backend, and setting file_basepath to a rbd volume17:48
*** rcernin has quit IRC17:53
stevellenicodemus_: back to your current test, if the resource contention is as I suspect the size of the backlog won't make matters worse, beyond the factor of a coefficient, once you have all of the metricd workers in your pool processing blocks but the outliers are already 100x to 1000x which means that coefficient could be large17:54
*** safchain has quit IRC17:55
nicodemus_stevelle, I see. I'm still looking for the 'file' log17:57
*** _nadya_ has quit IRC18:02
nicodemus_One thing I remember, is that using 'file' backend there was a lock directory in the basepath18:02
nicodemus_and there were separate locks for different measure_ objects18:02
stevelleright18:06
*** _nadya_ has joined #openstack-telemetry18:13
*** tesseract- has quit IRC18:19
*** gordc_ has joined #openstack-telemetry18:24
*** cdent has quit IRC18:40
*** _nadya_ has quit IRC18:45
*** gordc_ has quit IRC18:53
*** pece has quit IRC18:56
*** vishwanathj_chow is now known as vishwanathj19:01
*** harlowja has joined #openstack-telemetry19:04
*** cdent has joined #openstack-telemetry19:13
*** pece has joined #openstack-telemetry19:25
*** kro_sniffles is now known as kro_mnckn_doc19:27
openstackgerritgordon chung proposed openstack/ceilometer: add rc1 release notes  https://review.openstack.org/29367119:32
*** _nadya_ has joined #openstack-telemetry19:41
*** harlowja has quit IRC19:46
*** ig0r_ has quit IRC19:55
*** _nadya_ has quit IRC20:09
*** _nadya_ has joined #openstack-telemetry20:14
*** _nadya_ has quit IRC20:15
*** cdent has quit IRC20:15
*** marcin12345 has joined #openstack-telemetry20:16
marcin12345Has anyone have Ceilometer configured with Gnocchi that uses Swift as its storage? I cant make it work.20:17
gordcmarcin12345: what's the issue?20:22
gordcit works for me on single node (although swiftclient has an annoying bug which makes it seem like it's exploding)20:23
*** diogogmt has joined #openstack-telemetry20:33
nicodemus_marcin12345, this config worked for us: http://paste.openstack.org/show/490795/20:45
nicodemus_stevelle, I can't find for the life of me the 'file' storage backend log20:57
stevellenicodemus_: thanks for looking20:57
gordcstevelle: just curious, you helping to build the 1000node intel+rax cluster?20:58
nicodemus_if you need, next week I'll deploy a small gnocchi lab to play with, I could configure fie and gather some performance numbers20:59
stevellegordc: that's already built, but it's using the OSA deployment tooling20:59
*** safchain has joined #openstack-telemetry20:59
gordcstevelle: random qustion, you wouldn't know what RAID level is used on the racks would you?21:00
stevellegordc: if you like, I'll find out21:00
stevellenicodemus_: we can just figure it out again :)21:00
*** thorst has quit IRC21:01
gordcstevelle: if it's easy.21:01
gordcnicodemus_: do you know what RAID level you have?21:01
stevellegordc: I'm pretty sure there is a doc that was passed around somewhere around here21:01
gordcstevelle: yeah. i vaguely remember seeing RAID 10 as what RAX uses but it's possible i'm pulling that out of nowhere.21:02
*** thorst has joined #openstack-telemetry21:03
nicodemus_gordc, I don't think I follow... you mean the RAID in our servers?21:03
gordcnicodemus_: right. (unrelated to anything you've been asking)21:04
nicodemus_gordc, we use two SATADOM configured on RAID 121:05
gordcnicodemus_: ok, cool cool.21:06
*** thorst has quit IRC21:07
stevellegordc: found the answer, and unsurprisingly it's complicated :)21:18
gordcstevelle: yay21:18
gordcstevelle: i think i'm just going to roll some dice.21:19
*** kro_mnckn_doc has quit IRC21:19
stevelleobv swift doesn't need fancy RAID21:19
*** anteaya has quit IRC21:19
stevellebut the simple answer is that RAID10 is a good place to start for non-cinder/swift21:20
gordccool cool. that seems reasonable. might play around with that or RAID50.21:21
gordcthanks for checking21:21
*** nicodemus_ has quit IRC21:22
*** shardy has quit IRC21:28
*** pradk has quit IRC21:32
*** lsmola has quit IRC21:32
*** lsmola has joined #openstack-telemetry21:45
*** peristeri has quit IRC21:48
*** permalac has quit IRC21:50
*** gordc has quit IRC22:18
*** r-mibu has quit IRC22:20
*** r-mibu has joined #openstack-telemetry22:20
*** KrishR has quit IRC22:29
*** chlong has joined #openstack-telemetry22:32
*** safchain has quit IRC22:36
*** rbak has quit IRC22:54
*** jwcroppe has quit IRC22:59
*** jwcroppe has joined #openstack-telemetry22:59
*** jwcroppe has quit IRC23:04
*** vishwanathj has quit IRC23:06
*** yassine__ has quit IRC23:08
*** krotscheck has joined #openstack-telemetry23:11
*** anteaya has joined #openstack-telemetry23:21
*** diogogmt has quit IRC23:38
*** chlong has quit IRC23:41
*** pece has quit IRC23:44
*** jwcroppe has joined #openstack-telemetry23:44
« Tuesday, 2016-03-15 Index Thursday, 2016-03-17 »

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!