Tuesday, 2017-07-04

« Monday, 2017-07-03 Index Wednesday, 2017-07-05 »
*** dave-mccowan has joined #openstack-telemetry00:18
*** jmlowe has joined #openstack-telemetry00:20
*** zhangguoqing has joined #openstack-telemetry00:33
*** zhurong has joined #openstack-telemetry00:37
*** tovin07_ has joined #openstack-telemetry00:54
*** catintheroof has quit IRC00:55
*** zhangguoqing has quit IRC00:59
*** dave-mccowan has quit IRC01:03
*** zhangguoqing has joined #openstack-telemetry01:18
*** thorst has joined #openstack-telemetry01:50
*** thorst has quit IRC01:57
*** lhx__ has joined #openstack-telemetry02:03
*** thorst has joined #openstack-telemetry02:16
*** thorst has quit IRC02:20
*** swamireddy has quit IRC02:26
openstackgerritHangdong Zhang proposed openstack/ceilometer master: Add some unit test cases  https://review.openstack.org/48000302:27
openstackgerritHuachao Mao proposed openstack/ceilometer master: Fix typo for additional device support in intel node manger  https://review.openstack.org/48000903:15
*** catintheroof has joined #openstack-telemetry03:48
*** catintheroof has quit IRC03:55
*** nicodemus_ has quit IRC04:05
*** links has joined #openstack-telemetry04:11
*** psachin has joined #openstack-telemetry04:18
*** thorst has joined #openstack-telemetry04:21
*** thorst has quit IRC04:26
*** rcernin has joined #openstack-telemetry06:04
*** thorst has joined #openstack-telemetry06:22
*** thorst has quit IRC06:27
*** flg_ has joined #openstack-telemetry06:29
*** rcernin has quit IRC06:31
*** rcernin has joined #openstack-telemetry06:34
*** rcernin has quit IRC06:34
*** rcernin has joined #openstack-telemetry06:34
*** shardy has joined #openstack-telemetry06:54
*** pcaruana has joined #openstack-telemetry06:55
*** kzaitsev_ws has joined #openstack-telemetry06:56
*** yprokule has joined #openstack-telemetry06:58
*** tesseract has joined #openstack-telemetry07:14
openstackgerritDeepthi V V proposed openstack/ceilometer master: Project_id for SDN controller meters  https://review.openstack.org/48004007:25
openstackgerritsudhir agarwal proposed openstack/ceilometer master: Replaces uuid.uuid4 with uuidutils.generate_uuid()  https://review.openstack.org/48004407:35
*** efoley has joined #openstack-telemetry08:16
*** thorst has joined #openstack-telemetry08:23
*** thorst has quit IRC08:27
*** kzaitsev_ws has quit IRC08:49
openstackgerritJulien Danjou proposed openstack/ceilometer master: api: remove unused code  https://review.openstack.org/48008009:08
openstackgerritDeepthi V V proposed openstack/ceilometer master: Project_id for SDN controller meters  https://review.openstack.org/48004010:00
*** tovin07_ has quit IRC10:14
*** tovin07 has joined #openstack-telemetry10:16
*** tovin07 has quit IRC10:17
*** thorst has joined #openstack-telemetry10:24
*** thorst has quit IRC10:26
*** flg_ has quit IRC10:37
*** afazekas is now known as afazekas|away11:43
*** links has quit IRC11:53
*** thorst has joined #openstack-telemetry11:56
*** thorst has quit IRC11:57
*** links has joined #openstack-telemetry12:10
*** thorst has joined #openstack-telemetry12:21
*** links has quit IRC12:21
*** thorst has quit IRC12:26
*** thorst has joined #openstack-telemetry12:27
*** links has joined #openstack-telemetry12:40
*** thorst has quit IRC12:48
*** thorst has joined #openstack-telemetry12:48
*** links has quit IRC12:52
*** zhangguoqing has quit IRC12:57
*** pcaruana has quit IRC12:59
*** zhangguoqing has joined #openstack-telemetry12:59
*** zhurong has quit IRC13:03
*** zhangguoqing has quit IRC13:11
*** zhangguoqing has joined #openstack-telemetry13:11
*** zhangguoqing has quit IRC13:15
*** thorst has quit IRC13:22
*** lhx__ has quit IRC13:27
*** pcaruana has joined #openstack-telemetry13:36
*** links has joined #openstack-telemetry13:36
*** catintheroof has joined #openstack-telemetry13:39
*** pcaruana has quit IRC13:45
*** pcaruana has joined #openstack-telemetry13:50
*** nicodemus_ has joined #openstack-telemetry13:57
*** aignatov has quit IRC14:03
*** aignatov has joined #openstack-telemetry14:06
*** lhx__ has joined #openstack-telemetry14:19
*** r-daneel has joined #openstack-telemetry14:20
lhx__jd_, could help review the series of patches? :)14:24
lhx__https://review.openstack.org/#/c/477361/14:24
jd_sure I just pressed +214:25
jd_:D14:25
lhx__jd_, thanks, btw, related patches need to be reviewed ;)14:27
*** links has quit IRC14:59
*** links has joined #openstack-telemetry15:11
*** thorst has joined #openstack-telemetry15:23
*** thorst has quit IRC15:28
*** psachin has quit IRC15:29
*** chlong_ has quit IRC15:48
*** yprokule has quit IRC15:56
*** gordc has joined #openstack-telemetry15:56
*** rcernin has quit IRC15:58
*** links has quit IRC16:54
*** arahal_ has joined #openstack-telemetry17:00
*** r-daneel has quit IRC17:04
*** tesseract has quit IRC17:17
*** lhx__ has quit IRC17:23
*** efoley has quit IRC17:30
*** shardy has quit IRC17:40
*** arahal_ has quit IRC17:55
openstackgerritMerged openstack/ceilometer master: Fix typo for additional device support in intel node manger  https://review.openstack.org/48000917:57
*** flg_ has joined #openstack-telemetry18:29
EmilienMjd_: just in case you're still around, I have this error with latest gnocchi: http://paste.openstack.org/show/614404/18:57
EmilienMjd_: logs can be found on https://logs.rdoproject.org/00/7400/1/experimental/gate-weirdo-dlrn-master-puppet-scenario001/Zcae923839c784a4bb143f48cec3de4c0/weirdo-project/logs/18:57
EmilienMgordc: ^18:57
gordcmeh18:59
EmilienMlol18:59
EmilienMgordc: you're canadian, you have to work today :P18:59
gordcU-S-A! U-S-A!19:00
EmilienMlol19:00
EmilienMgordc: do you run tempest/gabbi in gnocchi gate in github?19:02
EmilienMat least you don't run gnocchi.tempest.scenario.GnocchiGabbiTest.test_live for sure19:03
gordcnope19:03
gordctravis sucks if you don't pay19:03
EmilienMlol19:03
EmilienMwhy didn't you stay on openstack infra then19:03
EmilienMit's free19:03
EmilienMnice upgrade folks! :P19:03
gordcyeah, openstack infra is a lot better.19:04
EmilienManyway, this timeouts makes our ci bad19:04
EmilienMerr, this auth issue I mean19:04
gordcis this master?19:04
gordcor gnocchi release19:04
EmilienMon a852437007129afe846de8a402a8c0600c344e8219:09
gordchmm... thats 4.0.0... which is why we use in ceilometer gate19:12
gordcEmilienM: did this pass before?19:15
EmilienMgordc: yeah19:16
EmilienMmaybe it's random failure but still19:17
gordci see you have metric_cleanup_delay set to 50019:17
EmilienMis it too short?19:17
gordc300*19:17
gordcno it's long. basically, there's a chance the test will take upto 300s to pass19:17
gordcit is strange i don't see 'expunging...' log in https://logs.rdoproject.org/00/7400/1/experimental/gate-weirdo-dlrn-master-puppet-scenario001/Zcae923839c784a4bb143f48cec3de4c0/weirdo-project/logs/gnocchi/metricd.txt.gz19:18
gordcyou have a log from a test that passed?19:18
EmilienMgordc: I can find it after lunch19:19
gordcEmilienM: i would lower that cleanup value... the janitor also might defer cleanup if something else locks what it needs.19:21
gordcso it might be up to 300s and even longer if it defers19:22
openstackgerritMerged openstack/ceilometer master: Migrate telemetry system architecture  https://review.openstack.org/47736119:22
jd_indeed19:36
EmilienMjd_, gordc: we don't configure metric_cleanup_delay - so we rely on upstream gnocchi20:07
EmilienMdefault is 30020:08
EmilienMhttps://github.com/gnocchixyz/gnocchi/blob/051593c2d908aad597b42161aedd0d976c81a944/gnocchi/opts.py#L12720:08
EmilienMshould I override it with a lower value?20:12
jd_EmilienM: that or increase your timeout I'd say20:12
jd_but clearly I'd set a lower value :)20:12
EmilienMjd_: why providing a default value that doesn't work well when running tempest?20:13
EmilienMshouldn't we rather change the default value in upstream gnocchi?20:13
jd_EmilienM: err we target default for production, not testing? :p20:13
EmilienMwhat is the difference here?20:14
jd_people barely ever delete metrics in real, so 5 minutes is way enough20:15
EmilienMjd_: ok so iiuc, some tests in our env take more than 5 min?20:17
jd_yes20:17
jd_what's your typical timeout EmilienM ?20:17
*** openstackgerrit has quit IRC20:18
EmilienMjd_: config is here: https://logs.rdoproject.org/00/7400/1/experimental/gate-weirdo-dlrn-master-puppet-scenario001/Zcae923839c784a4bb143f48cec3de4c0/weirdo-project/logs/etc/gnocchi/gnocchi.conf.txt.gz20:20
jd_EmilienM: no I mean your timeout since we were discussing a timeout?20:21
jd_I don't see the log though20:21
EmilienMjd_: also, I'm not sure it's a timeout http://paste.openstack.org/show/614404/20:21
jd_the log you pasted talks about 40120:21
EmilienMyeah20:21
jd_yeah why are we talking about a timeout lol?20:21
EmilienMgordc talked about timeouts, not me :D20:21
* EmilienM blames gordc20:21
jd_fair enough, let's blame gordc20:21
EmilienMhttps://logs.rdoproject.org/00/7400/1/experimental/gate-weirdo-dlrn-master-puppet-scenario001/Zcae923839c784a4bb143f48cec3de4c0/weirdo-project/logs/gnocchi/app.txt.gz20:22
EmilienM2017-07-04 17:25:21.816 1831 ERROR pecan.core [-] Controller 'get' defined does not support content_type 'None'. Supported type(s): ['application/json']20:22
EmilienMmaybe that20:22
jd_yeah i'm not worry, that's just because we test that and pecan is too verbose20:22
EmilienMok20:23
jd_EmilienM: where is the full tempest log?20:23
EmilienMgimme a sec20:23
EmilienMjd_: https://logs.rdoproject.org/00/7400/1/experimental/gate-weirdo-dlrn-master-puppet-scenario001/Zcae923839c784a4bb143f48cec3de4c0/weirdo-project/logs/tempest.txt.gz20:24
gordci think it tries so long that keystone doesn't like it.20:24
gordchttps://logs.rdoproject.org/00/7400/1/experimental/gate-weirdo-dlrn-master-puppet-scenario001/Zcae923839c784a4bb143f48cec3de4c0/weirdo-project/logs/apache/gnocchi_wsgi_access_ssl.txt.gz20:24
gordcsearch for "/v1/metric?id=cb039950-95dd"20:24
gordci'm guessing the token expires20:25
jd_that's verbose20:25
gordcbut change the value regardless. in production we don't care when it gets deleted, just that it gets deleted eventually.20:25
jd_gordc: lol20:25
EmilienMscenario001 use uuid token20:26
EmilienMand they don't expire I believe20:26
jd_indeed it goes from 200 to 40120:26
jd_first ::1 - - [04/Jul/2017:17:29:22 +0000] "GET /v1/metric?id=cb039950-95dd-47b4-8714-5b6957413755 HTTP/1.1" 200 649 "-" "gabbi/1.33.0 (Python urllib3)"20:26
jd_first 401 ::1 - - [04/Jul/2017:17:35:20 +0000] "GET /v1/metric?id=cb039950-95dd-47b4-8714-5b6957413755&status=delete HTTP/1.1" 401 114 "-" "gabbi/1.33.0 (Python urllib3)"20:26
jd_so they expire after 6 minutes?20:27
gordcthat is a weird expiry limit if that's the case.lol20:27
jd_yeah20:27
jd_or maybe it's 10 but it did nothing the first 4 minutes?20:27
gordcmaybe... maybe the entire gabbi test is run on same token20:28
EmilienMlet me check20:28
gordcbut it does seem like there are calls after 401s that work20:28
EmilienMtoken expire after 600 seconds20:28
jd_defaults:20:29
jd_    request_headers:20:29
jd_        x-auth-token: $ENVIRON['GNOCCHI_SERVICE_TOKEN']20:29
jd_        authorization: $ENVIRON['GNOCCHI_AUTHORIZATION']20:29
EmilienM(not sure the token/expiration parameter works with UUID)20:29
jd_ 43         token = self.os_admin.auth_provider.get_token()20:29
jd_ 60         os.environ["GNOCCHI_SERVICE_TOKEN"] = token20:29
jd_so yes only one token is used for the whole set of test20:29
EmilienMwe set token_expiration to 600 so we can test fernet rotations in our CI20:30
jd_or at least one token per scenario I think20:30
EmilienMlet me check in keystone if token_expiration is actually used when using UUID backend20:30
jd_600 that's 10 minutes so yeah that makes more sense20:30
*** flg_ has quit IRC20:31
jd_EmilienM: so I think gordc had a point about that timeout, he's smarter than us :(20:31
jd_makes the test faster by reducing the delay20:31
gordci am very smart20:31
EmilienMdon't say that please20:31
jd_for FYI we do run these tests upstream, but without Keystone, only with basic auth20:31
jd_so we don't have that token issue20:31
gordcEmilienM: say it!~20:32
jd_bbl20:32
EmilienMjd_, gordc  : thx20:32
gordci think this is the first call of test suite: [04/Jul/2017:17:25:20 +0000] "POST /v1/archive_policy HTTP/1.1" 415 98 "-" "gabbi/1.33.0 (Python urllib3)"20:32
gordc~10min20:32
gordci'm going to go buy myself a trophy now20:32
EmilienMso what value should we put?20:33
EmilienMfor metric_cleanup_delay20:33
gordc30s maybe?20:33
EmilienMmhh ok20:33
gordci think pifpaf uses 1s.lol20:33
EmilienMI see20:34
EmilienMso we had a race where sometimes metrics were cleaned after a token renewal20:34
EmilienMweird20:34
gordcEmilienM: it did try to delete metric 2 minutes before 401: 2017-07-04 17:33:20.575 1473 DEBUG gnocchi.storage._carbonara [-] Deleting metric cb039950-95dd-47b4-8714-5b695741375520:35
gordci think it's just that it got deferred which caused it to have to wait anther 300s cycle20:36
gordchttps://logs.rdoproject.org/00/7400/1/experimental/gate-weirdo-dlrn-master-puppet-scenario001/Zcae923839c784a4bb143f48cec3de4c0/weirdo-project/logs/gnocchi/metricd.txt.gz20:36
gordcor there's a bug... who knows.20:37
EmilienMgordc: you, since you're smart.20:37
gordcthanks. i was waiting for you to say it20:37
gordci can sleep easy now.20:38
EmilienMlol20:38
EmilienMgordc: https://review.openstack.org/#/q/topic:metric_cleanup_delay can you quickly have a look?20:48
gordcEmilienM: 10s works for me.20:52
*** catintheroof has quit IRC20:53
gordcif that fails, then there's a bug.... or you have very bad luck the cleanup cycle somehow sync with locking20:53
EmilienMok20:54
*** nicodemus_ has quit IRC21:08
*** gordc has quit IRC21:48
*** ddyer has quit IRC22:18
*** ddyer has joined #openstack-telemetry22:21
*** loptalica has joined #openstack-telemetry22:30
*** thorst has joined #openstack-telemetry22:33
*** thorst has quit IRC22:34
loptalicahi... does ceilometer collects cpu usage (percentage) of the vm ?  I do not mean how much percentage of the compute host one VM uses (that would be cpu_util meter) but the value that would "top" display when run inside VM OS22:36
*** nijaba has quit IRC22:42
*** nijaba has joined #openstack-telemetry22:50
*** vint_bra has quit IRC22:54
*** baffle has quit IRC23:05
*** yassine has joined #openstack-telemetry23:09
*** nijaba has quit IRC23:16
*** nijaba has joined #openstack-telemetry23:17
*** vint_bra has joined #openstack-telemetry23:41
*** thorst has joined #openstack-telemetry23:43
*** thorst has quit IRC23:43
*** vint_bra has quit IRC23:51
*** vint_bra has joined #openstack-telemetry23:57
« Monday, 2017-07-03 Index Wednesday, 2017-07-05 »

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!