Wednesday, 2014-08-27

*** Qiming has joined #openstack-ceilometer00:09
*** adam_g has joined #openstack-ceilometer00:10
*** Qiming has quit IRC00:14
adam_gany weirdness going on around the switch to oslo.utils?  running into grenade issues atm:
*** joannah has joined #openstack-ceilometer00:47
*** sbfox has joined #openstack-ceilometer00:48
*** joannah has quit IRC00:51
*** joannah has joined #openstack-ceilometer00:52
*** joannah has quit IRC00:53
*** joannah has joined #openstack-ceilometer00:54
*** alexpilotti has quit IRC00:56
*** sbfox has quit IRC00:56
*** joannah has quit IRC00:58
*** alexpilotti has joined #openstack-ceilometer01:02
*** joannah has joined #openstack-ceilometer01:05
*** alexpilotti has quit IRC01:06
*** joannah has quit IRC01:06
*** joannah has joined #openstack-ceilometer01:08
*** joannah has quit IRC01:09
*** Qiming has joined #openstack-ceilometer01:10
*** renlt has joined #openstack-ceilometer01:12
*** _cjones_ has quit IRC01:16
*** _cjones_ has joined #openstack-ceilometer01:17
*** _cjones_ has quit IRC01:21
*** sbfox has joined #openstack-ceilometer01:28
*** shakamunyi has quit IRC01:29
*** mitz_ has joined #openstack-ceilometer01:31
*** mitz has quit IRC01:33
*** shakamunyi has joined #openstack-ceilometer01:37
*** yasukun has joined #openstack-ceilometer01:37
*** nosnos has joined #openstack-ceilometer01:56
*** _cjones_ has joined #openstack-ceilometer02:06
*** rbak__ has joined #openstack-ceilometer02:19
*** fnaval has joined #openstack-ceilometer02:20
*** rbak_ has quit IRC02:21
*** _cjones_ has quit IRC02:23
*** renlt has quit IRC02:27
*** yasukun has quit IRC02:43
*** yasukun has joined #openstack-ceilometer02:43
*** harlowja is now known as harlowja_away02:44
*** changbl has joined #openstack-ceilometer03:12
*** stevelle has quit IRC03:19
*** sdake_ has quit IRC03:20
*** nosnos has quit IRC03:23
*** nosnos has joined #openstack-ceilometer03:24
*** stevelle has joined #openstack-ceilometer03:27
*** nosnos has quit IRC03:28
*** fnaval has quit IRC03:29
*** harlowja_away has quit IRC03:29
*** fnaval has joined #openstack-ceilometer03:30
*** yasukun has quit IRC03:36
*** yasukun has joined #openstack-ceilometer03:37
*** yasukun has quit IRC03:38
*** yasukun has joined #openstack-ceilometer03:38
*** nsaje has quit IRC03:50
*** nsaje has joined #openstack-ceilometer03:51
*** _cjones_ has joined #openstack-ceilometer03:51
*** nosnos has joined #openstack-ceilometer03:53
*** _cjones_ has quit IRC03:56
*** stevelle has quit IRC04:05
*** fnaval has quit IRC04:17
*** chuck_ has quit IRC04:49
*** zhiyan has joined #openstack-ceilometer04:51
*** _cjones_ has joined #openstack-ceilometer04:53
*** ildikov_ has quit IRC04:57
*** _cjones_ has quit IRC04:57
*** adriant has quit IRC05:16
*** idegtiarov has joined #openstack-ceilometer05:16
*** yasukun has quit IRC05:30
*** yasukun has joined #openstack-ceilometer05:31
nsajeDinaBelova: o/ ping me when you're here05:42
*** yatin has joined #openstack-ceilometer05:42
*** _cjones_ has joined #openstack-ceilometer05:53
*** _cjones_ has quit IRC05:58
*** k4n0 has joined #openstack-ceilometer05:59
*** ildikov_ has joined #openstack-ceilometer05:59
*** asalkeld has quit IRC06:17
*** Longgeek has joined #openstack-ceilometer06:20
*** asalkeld has joined #openstack-ceilometer06:21
*** Qiming has quit IRC06:27
*** atan8 has joined #openstack-ceilometer06:34
*** atan8 has quit IRC06:50
*** nosnos has quit IRC07:04
*** nosnos_ has joined #openstack-ceilometer07:04
*** Qiming has joined #openstack-ceilometer07:18
*** arezmerita has joined #openstack-ceilometer07:45
*** Qiming has quit IRC07:52
*** Qiming has joined #openstack-ceilometer07:52
*** _cjones_ has joined #openstack-ceilometer07:55
*** _cjones_ has quit IRC07:59
*** shakayumi has joined #openstack-ceilometer08:01
*** llu_office has joined #openstack-ceilometer08:03
*** nsaje_ has joined #openstack-ceilometer08:03
*** gpocente1 has joined #openstack-ceilometer08:05
*** _nadya_ has joined #openstack-ceilometer08:05
*** yatin has quit IRC08:05
*** jd__` has joined #openstack-ceilometer08:08
*** dhellmann_ has joined #openstack-ceilometer08:08
*** SergeyLukjanov2 has joined #openstack-ceilometer08:09
*** ruhe2 has joined #openstack-ceilometer08:09
*** Daviey_ has joined #openstack-ceilometer08:09
*** nsaje has quit IRC08:10
*** shakamunyi has quit IRC08:10
*** sbfox has quit IRC08:10
*** jd__ has quit IRC08:10
*** Daviey has quit IRC08:10
*** llu_linux has quit IRC08:10
*** gpocentek has quit IRC08:10
*** ruhe has quit IRC08:10
*** vissborg has quit IRC08:10
*** dhellmann has quit IRC08:10
*** SergeyLukjanov has quit IRC08:10
*** SergeyLukjanov2 is now known as SergeyLukjanov08:10
*** Daviey_ is now known as Daviey08:10
*** ruhe2 is now known as ruhe08:10
*** dhellmann_ has quit IRC08:12
*** jd__` is now known as jd__08:12
*** dhellmann has joined #openstack-ceilometer08:12
*** Longgeek has quit IRC08:20
*** Longgeek has joined #openstack-ceilometer08:21
*** vissborg has joined #openstack-ceilometer08:23
*** idegtiarov has quit IRC08:24
*** idegtiarov has joined #openstack-ceilometer08:25
*** alexpilotti has joined #openstack-ceilometer08:25
*** Longgeek has quit IRC08:25
*** gpocente1 is now known as gpocentek08:28
*** gpocentek is now known as Guest3210808:28
*** Guest32108 is now known as gpocentek08:30
*** gpocentek has quit IRC08:30
*** gpocentek has joined #openstack-ceilometer08:30
*** _nadya_ has quit IRC08:37
*** shakayumi has quit IRC08:40
*** Longgeek has joined #openstack-ceilometer08:43
*** inc0 has joined #openstack-ceilometer08:44
inc0good morning guys08:44
*** nosnos_ has quit IRC08:45
*** nosnos has joined #openstack-ceilometer08:46
*** nosnos has quit IRC08:50
nsaje_good morning!08:50
*** isviridov|away is now known as isviridov08:51
*** _cjones_ has joined #openstack-ceilometer08:56
*** _cjones_ has quit IRC09:00
*** idegtiarov has left #openstack-ceilometer09:03
*** nosnos has joined #openstack-ceilometer09:11
*** idegtiarov has joined #openstack-ceilometer09:13
*** ccrouch1 has quit IRC09:28
nsaje_DinaBelova: earth calling Dina :-)09:31
Qimingcannot start ceilometer-alarm-evaluator in devstack: AttributeError: 'module' object has no attribute 'import_object'09:36
nsaje_Qiming: i think I was a bug report regarding that this morning09:36
Qimingnsaje_, any quick workaround ?09:37
Qimingnsaje_, I mean by changing package version or configuration options?09:37
nsaje_Qiming: let me see09:37
Qimingnsaje_, thanks!!!09:38
silehtnsaje_, Qiming someone have tried a fix /1
silehtnsaje_, adam_g seems that oslo.utils have moved a public method to "protected"09:39
silehtQiming, ^09:39
silehtjd__, ^09:39
nsaje_why the heck did they put that into protected... can't just go around removing public methods09:42
nsaje_sileht: what if we put a temporary ignore in pep config and use the protected method? Because fixing this in oslo.utils will require a new release, which is anything but fast09:43
silehtnsaje_, public exists09:44
silehtI have updated the review
nsaje_sileht: great, ok then09:45
silehtjd__, eglynn-office , ildikov_ ^09:45
nsaje_sileht: I still don't like api changes though :)09:45
*** Longgeek has quit IRC09:50
* sileht is waiting coredev until the end of my lunch break, before unblocking gate itself09:52
*** _cjones_ has joined #openstack-ceilometer09:56
*** _cjones_ has quit IRC10:01
idegtiarovnsaje_: o/10:10
nsaje_idegtiarov: hey!10:11
idegtiarovnsaje_: we have completed HA testing with two CA and got from them the same notifications10:11
idegtiarovnsaje_: just duplicates10:12
nsaje_idegtiarov: I've been waiting for one of you guys to give me some more input!10:12
nsaje_idegtiarov: I can't reproduce that in devstack10:12
nsaje_idegtiarov: we'll need to work togethet to debug it10:12
idegtiarovnsaje_: with pleaser10:13
*** Qiming has quit IRC10:13
nsaje_did you set up any custom logging in the code? otherwise I'll send you a patch to apply the central-agent-ha branch10:14
nsaje_(or I could actually put the debug statements in the upstream tree as well)10:14
idegtiarovnsaje_: we have made some improvements in tooz zookiper driver10:15
nsaje_idegtiarov: good, but I'll need debug output from coordination code10:15
*** Longgeek has joined #openstack-ceilometer10:16
idegtiarovnsaje_:  in coordination part we only  add suffix for resource_id depends on CA10:17
nsaje_what suffix?10:17
idegtiarovnsaje_: to see from what CA we get meters  )10:18
nsaje_ah, ok10:18
nsaje_you save that in the sample? so you have that in publishing code actually?10:19
idegtiarovnsaje_: yep10:19
*** inc0_ has joined #openstack-ceilometer10:19
idegtiarovnsaje_: I think would be grate if you send your new patch in upstream10:21
nsaje_idegtiarov: working on it as we speak10:21
idegtiarovnsaje_:  :)10:22
inc0_can I have review please?:)
*** inc0 has quit IRC10:22
silehtinc0_, done10:30
nsaje_idegtiarov: new patch up, adds debugging output10:31
nsaje_idegtiarov: can you guys check what groups are being joined and what group members are being reported?10:32
*** Longgeek has quit IRC10:32
inc0_sileht, about your comment, yes, it will generate 2 notifs because there are two notifs. I believe thats exactly how its supposed to work...10:32
silehtinc0_, this plugins doesn't record event of heat, but how many stack have been update/create/deleted10:33
idegtiarovnsaje_:  ok! we are going to do that10:33
*** flwang1 has joined #openstack-ceilometer10:33
*** yasukun has quit IRC10:34
*** yasukun has joined #openstack-ceilometer10:34
*** flwang1 has quit IRC10:34
*** flwang1 has joined #openstack-ceilometer10:35
inc0_sileht, shouldn't there be logic for stack couting elsewhere? I mean in my opinion we should record all events in ceilo, and then filter only .end ones when getting stack count. Some other metrics may for example count stack in progress, and then recording just .end actions won't be enough10:36
silehtinc0_, I think the bug is invalid, if you want to track with ceilometer in how many time a stack has been created, you need to configure the event part of ceilometer and use the event API10:36
ildikov_sileht: nsaje: as the author pointed out that patch does not solve the issue it seems10:37
DinaBelovansaje_, o/10:37
nsaje_inc0_: as sileht pointed out, notification plugins don't emit events, but samples. Events are handled separately in the event notification plugin10:37
ildikov_sileht: nsaje: gate log: 2014-08-27 10:33:03.414 | *** Not Whitelisted *** 2014-08-27 10:20:37.681 24055 CRITICAL ceilometer [-] ImportError: Class ceilometer.alarm.service.SingletonAlarmService cannot be found10:38
* DinaBelova just came to the office10:38
DinaBelovawedding things to do :)10:38
nsaje_DinaBelova: how can you have time to get married if central agent scaling isn't working?! :-)10:38
nsaje_DinaBelova: I talked to idegtiarov regarding that, so we're progressing10:39
DinaBelovansaje_, hehe, when we were planning the party I did not know it won't work :)10:39
DinaBelovansaje_, yeah, /me reading10:39
silehtildikov_, my bad, the new API import module, not class ...10:39
DinaBelovansaje_, btw, we have devstack as well10:39
inc0_nsaje_, sileht we don't want to have sample with create.start?10:39
DinaBelovait is just multinode with additional services10:40
nsaje_DinaBelova: I'm guessing multinode is the culprit here10:40
nsaje_DinaBelova: I can't reproduce it single node10:40
DinaBelovansaje_, hm, even when you're testing the central agent itself?10:40
DinaBelovanot the compute one?10:40
ildikov_sileht: yeap, I've just checked that code now10:41
nsaje_DinaBelova: no, but it's literally the same code path10:41
nsaje_DinaBelova: well, we'll know more when we get debug output from the latest patch10:41
nsaje_inc0_: the notification plugin you modified creates a sample named 'stack.create'. with your modifications, two of those samples would be created for each one creation of a stack10:42
DinaBelovansaje_, so the compute ones (for the instances and the meters to collect from them) share the load okay?10:42
*** zul has joined #openstack-ceilometer10:43
nsaje_DinaBelova: yep10:43
nsaje_DinaBelova: I'm guessing there's a problem with coordination backend, somehow not getting group membership info10:43
DinaBelovansaje_, a-ha!10:43
inc0_nsaje_, is that a bad thing? I mean, when we count stacks we can count only stack.create.end samples, but shouldn't we also recore create.start ones?10:44
nsaje_DinaBelova: not that, if there was an error connecting to the backend, it wouldn't duplicate samples. I'm guessing the agent receives info from tooz that he's the only member. But let's wait for the logs!10:44
DinaBelovansaje_, yeah, let us find the issue10:44
DinaBelovansaje_, sorry, that might take some time10:44
DinaBelovamultynode devstack is not really stable thing :D10:45
nsaje_inc0_: We have a meter that counts the number of stacks created. We want to create a sample when a stack is created. That is true when we receive a stack.create.end notification, right?10:45
nsaje_inc0_: what you're talking about are events -
inc0_yes, maybe then change meter itself?:)10:46
nsaje_inc0_: there's a distinction between meters and events :-) meters measure some quantity, events represent events10:46
nsaje_inc0_: do we want to know how many times a stack has begun to get created? Perhaps we do, I don't know, but it doesn't make much sense to me :)10:47
inc0_nsaje_, yes, I know, but what notification plugin does is record sample. So if we are interested only in .end samples, maybe we should filter out that on meter side10:47
inc0_nsaje_, we may want to know how many stacks are in progress:)10:48
inc0_or failed10:48
inc0_still, thats something meter should solve, not notification plugin imho10:49
nsaje_inc0_: again,
*** zul has quit IRC10:51
inc0_nsaje_, ok, well then I'll check event part of it then10:52
inc0_if events are recorded10:52
nsaje_ildikov_: I was in a docco mood so I revamped the python-ceilometerclient docs:
nsaje_ildikov_: current state doesn't really reflect well on us10:55
*** ifarkas has quit IRC10:55
*** _cjones_ has joined #openstack-ceilometer10:57
*** ccrouch has joined #openstack-ceilometer10:57
*** Longgeek has joined #openstack-ceilometer11:00
*** _cjones_ has quit IRC11:01
*** ifarkas has joined #openstack-ceilometer11:02
*** zul has joined #openstack-ceilometer11:04
*** Qiming has joined #openstack-ceilometer11:07
*** dhellmann has quit IRC11:10
*** cdent has joined #openstack-ceilometer11:10
*** zul has quit IRC11:11
*** dhellmann has joined #openstack-ceilometer11:13
*** zul has joined #openstack-ceilometer11:14
*** ifarkas has quit IRC11:14
*** cdent has quit IRC11:15
*** cdent has joined #openstack-ceilometer11:16
*** nosnos has quit IRC11:20
*** nosnos has joined #openstack-ceilometer11:20
*** nosnos has quit IRC11:25
*** ifarkas has joined #openstack-ceilometer11:27
nsaje_DinaBelova, idegtiarov: how's the testing going?11:28
*** flwang1 has quit IRC11:34
*** flwang2 has joined #openstack-ceilometer11:35
silehtjd__, nsaje_, ildikov_ I have proposed an other approch for the import_utils issue, the API method have disapear because import_utils shouldn't be used for this purpose:
jd__sileht: +211:41
jd__exactly what it should be11:41
*** yassine has joined #openstack-ceilometer11:49
nsaje_sileht: good idea11:53
silehtnsaje_, just the time you remove all of that with the new tooz evaluator :p11:55
nsaje_sileht: yeah, hopefuly at least :-) that's why I said just put the hack in in the old patch, but this is better ofcourse11:55
*** yasukun has quit IRC12:00
idegtiarovnsaje_: we are in progress12:00
*** idegtiarov has left #openstack-ceilometer12:01
*** idegtiarov has joined #openstack-ceilometer12:01
*** yasukun has joined #openstack-ceilometer12:03
*** yasukun has quit IRC12:08
*** asalkeld has quit IRC12:21
*** inc0 has joined #openstack-ceilometer12:27
* cdent goes home12:28
*** cdent has quit IRC12:28
*** inc0_ has quit IRC12:29
*** inc0__ has joined #openstack-ceilometer12:29
*** inc0 has quit IRC12:32
ildikov_nsaje_: I saw the docco, just didn't have time for making a proper review yet12:33
nsaje_ildikov_: no rush, just letting you know :-)12:34
*** inc0__ has quit IRC12:34
nsaje_DinaBelova, idegtiarov: if I'm not here when you finish, please e-mail me some logs from different agents, or at least check that they are joining the same groups and reporting the same membership set12:36
*** flwang2 has quit IRC12:37
*** gordc has joined #openstack-ceilometer12:38
idegtiarovnsaje_:  now CA on our controller node do not set log notification about join any group at all12:39
idegtiarovnsaje_:  jn conpute node we have almost all logs, continue testing12:39
nsaje_so, compute node joins groups, central doesn't?12:40
idegtiarovnsaje_: yes. it is12:40
nsaje_what is your pipeline.yml?12:41
nsaje_idegtiarov: ^12:41
idegtiarovnsaje_:  pipeline.yaml is the same on both nods12:42
nsaje_can you paste it somewhere?12:42
idegtiarovnsaje_: yes, just a sec12:42
nsaje_idegtiarov: ok, I see the problem12:48
idegtiarovnsaje_:  I'm looking12:49
nsaje_idegtiarov: you don't actually use central agent :-) your pipeline config only uses compute pollsters12:49
nsaje_idegtiarov: for example '' is polled by ceilometer.compute.pollsters.disk:ReadBytesPollster12:50
nsaje_idegtiarov: ok, I see you do have storage samples, checking that, gimme a sec12:52
idegtiarovnsaje_:  so if we want polling 'storage.objects.*'   it would be done by compute.pollster not central.agent?12:53
nsaje_idegtiarov: nope, I missed that12:53
idegtiarovnsaje_: yep12:53
idegtiarovnsaje_:  actually we are testing our swift service12:53
nsaje_idegtiarov: yeah, so: swift pollster doesn't support partitioning! That's it!12:55
nsaje_idegtiarov: explanation: it doesn't use a discovery plugin, it should, I'll submit a patch for it12:55
nsaje_idegtiarov: I have no idea why it wasn't modified when discovery plugins were introduced12:55
idegtiarovnsaje_: I have the same thoughts :-)12:56
nsaje_idegtiarov: so can you guys check something else, like neutron?12:56
*** julim has joined #openstack-ceilometer12:57
idegtiarovnsaje_:  after finish with swift we will try something else of course12:57
nsaje_idegtiarov: ok, because partitioning won't work with swift12:58
*** rbowen has joined #openstack-ceilometer12:58
Qiminghi, any hint on this error? TypeError: 'unicode' object is not callable12:58
nsaje_idegtiarov: until it's migrated to the new discovery model12:58
*** _cjones_ has joined #openstack-ceilometer12:58
Qimingcontext:  File "/opt/stack/python-ceilometerclient/ceilometerclient/", line 186, in _do_authenticate12:58
idegtiarovnsaje_:  get it! ok we will see what service we can try testing next12:59
nsaje_one that has discovery: 'xxx' specified in
nsaje_idegtiarov: or you see a in its folder in the source12:59
idegtiarovnsaje_:  ok. thank you13:00
nsaje_idegtiarov: leaving for lunch now, bbl13:01
idegtiarovnsaje_:  see you13:02
*** jasondotstar has joined #openstack-ceilometer13:02
*** _cjones_ has quit IRC13:04
*** yassine has quit IRC13:06
DinaBelovasome not the logs, but some result here13:07
DinaBelovansaje_, btw - what gordc was testing?13:07
DinaBelovawe used swift for central agent13:09
DinaBelovayou used instances for the compute ones13:09
*** cdent has joined #openstack-ceilometer13:11
gordcDinaBelova: i don't need to say anything right? ^^13:12
DinaBelovagordc, hehe, you're online :)13:13
DinaBelovaI just forgot you're now :)13:13
DinaBelovanow *here*13:13
gordcyep... sort of... still waking up13:13
DinaBelovagordc, what meters and rsources did you use to poll?13:13
DinaBelovahehe, good morning :)13:13
gordcDinaBelova: i used standard pipeline.13:13
gordcso basically everything13:14
gordcwith neutron enabled13:14
DinaBelovaa-ha, so neutron meters were duplicated as well?13:14
DinaBelovaif so, nsaje_'s idea that the issue is with the discoveries is wrong13:15
DinaBelovaas neutron has the default discovery13:15
DinaBelovaif it's not the lbaas or vpnaas13:15
DinaBelovagordc, neutron meters were duplicated as well?13:17
gordcuh... i should try again. i was mostly just going based on fact that both central agent logs had the same log output13:17
gordcie. 'Polling compute' on both... yada yada yada13:17
DinaBelovaa-ha, so the compute meters were duplicated as well,13:18
DinaBelovanot only the swift ones13:18
DinaBelovagordc, you see, nsaje_ proposed the idea for the bug reason - that if it's not the default discovery, it won't work13:19
DinaBelovabut compute meters have default one13:19
gordcDinaBelova: we use discovery in compute meters? haven't looked at that in a while.13:23
DinaBelovagordc, it's 'default one'13:23
DinaBelovathat's why nsaje_ says, his patch will work for them13:23
DinaBelovafor the swift we need to set some written discovery - like "discovery: smth" in the pipeline13:24
gordcDinaBelova: hmm.. maybe it's an issue with logging then because i for sure saw duplicate logs for compute meters13:24
DinaBelovahe thought, that the issue we had here with idegtiarov is that we're polling the swift13:24
gordcDinaBelova: i'll give it a try again now13:24
DinaBelovagordc, you might saw them in the compute agent logs?13:25
DinaBelovaa-ha, no, really13:25
DinaBelovawe have meter that is actually doing 'nova list'13:25
DinaBelovaso the central agent will poll it itself13:25
DinaBelovagordc, as you might see in the - second central agent even did not joined the group13:26
DinaBelovadue to the logs13:26
gordchmm.. interesting. i wonder if there's a way for me to check this in memcached13:27
gordcDinaBelova: i'll play around with it but i think there's some issues to be worked out13:27
*** _nadya_ has joined #openstack-ceilometer13:28
DinaBelovagordc, we checked it using last patch set by nsaje_13:28
DinaBelovahe added some debug messages13:28
gordcoh cool. i'll pull that in13:28
*** rbak__ has quit IRC13:31
*** _nadya_ has quit IRC13:34
*** itisit has joined #openstack-ceilometer13:36
*** shakamunyi has joined #openstack-ceilometer13:41
*** Dafna is now known as Dafna_away13:57
*** shakamunyi has quit IRC13:58
*** rbak has joined #openstack-ceilometer14:06
*** ildikov_ has quit IRC14:09
*** joesavak has joined #openstack-ceilometer14:11
*** thomasem has joined #openstack-ceilometer14:11
*** shakamunyi has joined #openstack-ceilometer14:12
*** thomasem has quit IRC14:12
*** thomasem has joined #openstack-ceilometer14:16
gordcnsaje_, DinaBelova: in a meeting... but it seems like it's working right now using memcached14:19
Qiming DEBUG ceilometer.compute.pollsters.cpu [-] Obtaining CPU Util is not implemented for LibvirtInspector from (pid=9761) get_samples /opt/stack/ceilometer/ceilometer/compute/pollsters/
Qiminganybody please help me understand this? ^^^14:21
*** shakamunyi has quit IRC14:22
*** shakamunyi has joined #openstack-ceilometer14:23
*** Dafna_away is now known as Dafna14:27
*** tongli has joined #openstack-ceilometer14:30
*** yassine has joined #openstack-ceilometer14:31
*** ifarkas_ has joined #openstack-ceilometer14:36
*** ifarkas has quit IRC14:38
*** sdake_ has joined #openstack-ceilometer14:38
*** shakamunyi has quit IRC14:40
*** agrebennikov has joined #openstack-ceilometer14:47
*** agrebennikov has quit IRC14:47
*** agrebennikov has joined #openstack-ceilometer14:48
*** agrebennikov has joined #openstack-ceilometer14:49
agrebennikovhi all, does anyone know how to set up ceilometer so that it put samples from instance Only if the instance is running?14:50
DinaBelovagordc, ok, have a good meeting :) may you please ensure that both compute and network meters are not duplicated and the load is shared please14:52
*** renatoarmani has joined #openstack-ceilometer14:52
*** _nadya_ has joined #openstack-ceilometer14:53
*** openstackgerrit has joined #openstack-ceilometer14:54
openstackgerritJulien Danjou proposed a change to openstack/ceilometer: Switch to oslo.serialization
gordcQiming:... it's not implemented in cpu util (i suppose libvirt does not provide this info)14:55
gordcin libvirt*14:55
Qiminggordc, thanks for the reply14:55
gordcQiming: there is a cpu_util meter though which i believe we calculate based on cputime14:56
*** Longgeek has quit IRC14:56
Qiminggordc, strange enough, I waited a few more minutes, then I am getting cpu_util now14:56
gordcQiming: yes, it's a derived meter... you need multiple 'cpu' meter samples before you get cpu_util samlpes14:58
Qiminggordc, I believe it is a transient error.  after I modified pipeline yaml by changing 600 to 60 and restarted all ceilometer services, this problem is gone14:58
Qiminggordc, I was suspecting that my upgrade RHEL rpms are introducing some version mismatch problem, but it is not, :)14:58
gordcQiming: nah, it's probably becaues you didn't wait long enough... you'll only cpu_util meters after second poll... so in 600 internal case, you'd need to wait 10 mins14:59
Qiminggordc, got it.14:59
*** _cjones_ has joined #openstack-ceilometer15:00
gordcagrebennikov: i'm not sure we have that functionality. we just get all instances on the host15:00
gordcagrebennikov: might be a feature to propose15:00
gordcDinaBelova: i'll give it a try. lbaas/vpnaas stuff is throwing a bunch of errors15:01
*** tburnes has joined #openstack-ceilometer15:03
*** tburnes has quit IRC15:04
*** _cjones_ has quit IRC15:05
DinaBelovagordc, do you have them installed? /me still did not try them >_<15:06
gordcDinaBelova: no... but it'll try to find them even if you don't and that's why i'm getting error15:07
DinaBelovahehe, okay, thanks :D15:07
DinaBelovareally we're seeing quite strange thing right now15:07
DinaBelovaif we'll connect to the zk15:08
DinaBelovawe'll see no processed in the groups [compute-global, central-global]15:08
DinaBelovathat's kind of thing that makes we suspicious about tooz zk driver15:09
idegtiarovgordc:  we see that both CA join the same group, but steel polling the same resource, as a result we have duplicates of the data15:10
*** joesavak has quit IRC15:10
*** joesavak has joined #openstack-ceilometer15:11
*** Titilambert has quit IRC15:16
openstackgerritgordon chung proposed a change to openstack/ceilometer: update requirements
gordcDinaBelova: yeah seems like a tooz issue15:18
*** ildikov has joined #openstack-ceilometer15:18
*** IvanBerezovskiy has left #openstack-ceilometer15:18
*** Taytay has joined #openstack-ceilometer15:18
gordcidegtiarov: what's CA? central or compute?15:18
*** Daviey has quit IRC15:18
*** Daviey_ has joined #openstack-ceilometer15:19
*** Daviey_ is now known as Daviey15:19
*** erecio has joined #openstack-ceilometer15:22
*** erecio has quit IRC15:22
*** agrebennikov has quit IRC15:22
*** ruhe has quit IRC15:22
*** gpocentek has quit IRC15:22
*** promulo_ has quit IRC15:22
*** sdake has quit IRC15:22
*** ityaptin has quit IRC15:22
DinaBelovagordc, finally what's your verdict about memcached as a backend?15:22
DinaBelovawas it ok?15:22
*** arezmerita has quit IRC15:22
DinaBelovashould we search the problem in the common code or in the driver itself?15:22
idegtiarovgordc: both central agents15:22
*** Longgeek has joined #openstack-ceilometer15:22
*** nealph__ has quit IRC15:22
gordcDinaBelova: i get dups for compute agent15:22
*** erecio has joined #openstack-ceilometer15:22
*** agrebennikov has joined #openstack-ceilometer15:22
*** ruhe has joined #openstack-ceilometer15:22
*** gpocentek has joined #openstack-ceilometer15:22
*** promulo_ has joined #openstack-ceilometer15:22
*** sdake has joined #openstack-ceilometer15:22
*** ityaptin has joined #openstack-ceilometer15:22
*** Longgeek has quit IRC15:22
*** erecio has quit IRC15:23
*** erecio has joined #openstack-ceilometer15:23
*** nealph__ has joined #openstack-ceilometer15:23
DinaBelovagordc, okay, so that means discovery idea is inconsistent :)15:24
DinaBelovaokay :)15:24
DinaBelovaso here we are - it's smth wrong with tooz15:24
DinaBelovagordc, may you check in memcached directly how are the agents managed?15:24
DinaBelovaI'm not sure how that looks like here15:24
DinaBelovabut smth like we're done for ZK - went there directly and took a loook15:24
*** jergerber has joined #openstack-ceilometer15:24
agrebennikovhey, is it possible to do anything but calculations with transformers?  I mean if I got the status of the instance from the resource (metadata)... Is it possible to put it into the sample?15:25
*** stevelle has joined #openstack-ceilometer15:30
*** Dieterbe_ is now known as Dieterbe15:30
*** Dieterbe has quit IRC15:30
*** Dieterbe has joined #openstack-ceilometer15:30
*** thomasem has quit IRC15:35
*** dragondm has quit IRC15:37
*** openstack has joined #openstack-ceilometer16:18
openstackgerritOpenStack Proposal Bot proposed a change to openstack/ceilometer: Updated from global requirements
DinaBelovagordc, afair you tried this with neutron16:21
DinaBelovawe're turned on ip.floating meter16:21
DinaBelovacreated it16:21
DinaBelovaand not meter data was there :)16:21
DinaBelovanot notification, nothing16:21
DinaBelovaso it looks like wer'e having smth wrong with neutron as well16:22
DinaBelovaas it should use default discovery16:22
DinaBelovaand nothing was there actually16:22
*** _nadya_ has quit IRC16:22
nsajeDinaBelova: we need integration tests right now :-)16:23
DinaBelovaso in the discover method it were nothing in self.default_discovery16:24
gordcDinaBelova: are we suppose to have two groups? central-global and central-<hostname>?16:26
nsajegordc: not really, I'll fix that, but thats not what's causing the issues16:28
gordcnsaje: i see yeah, just looked stranged... thanks for tracking this16:29
*** ildikov_ has joined #openstack-ceilometer16:30
*** sbfox has joined #openstack-ceilometer16:31
*** ildikov has quit IRC16:33
*** sdake_ has quit IRC16:34
*** tburnes has joined #openstack-ceilometer16:34
DinaBelovawe're having [compute-global, central-global] groups16:35
DinaBelovagordc ^^16:36
DinaBelovaand that's not the hostname16:36
DinaBelovawe're having compute-2 as a name16:36
DinaBelovaand controller16:36
*** tburnes has quit IRC16:37
*** Q^3 has joined #openstack-ceilometer16:37
*** Q^3 has quit IRC16:37
gordcDinaBelova: but you have two central and two compute groups though right? i think that's what nsaje mentioned.16:37
DinaBelovagordc, ehm, no16:38
nsajegordc: that's only if you enable partition king for compute16:38
DinaBelovajust one central16:38
DinaBelovaand one compute16:38
nsajeLOL, autocorrect, partitioning16:38
gordcnsaje: :)16:38
DinaBelovaa-ha, yeah16:38
gordccool cool. yeah i have partitioning enabled16:38
DinaBelovawe're having no compue partitioning16:38
gordcgoing to head to lunch. thanks for debugging folks -- don't stay too late. i'll track messages when i get back16:40
DinaBelovagordc, nsaje - do you know what's up with the dicsovery?! we're having no default discovery for the ip.floating, image and image.size meters!16:40
DinaBelovagordc, ok16:40
DinaBelovathey are all 'both' - notifications + polling16:40
*** idegtiarov has quit IRC16:42
DinaBelovaa-a-and no meters from neutron at all16:42
nsajeDinaBelova Do you have them defined in the pipeline config? Do they exist in the code?16:43
DinaBelovaand there are some from glance (but I guess they are from the notificaitons)16:43
DinaBelovawe've set them in the pipeline16:43
DinaBelovawell, let's about image ones16:43
DinaBelovathey are defined in the pipeline16:43
DinaBelovaI see them in the meter-list16:43
DinaBelovabut in the pdb in the discover method16:44
DinaBelovathere were no default discovery16:44
DinaBelovawhile the polling16:44
*** alexpilotti has quit IRC16:44
DinaBelovaglance has the default dicovery16:44
nsajeNo no don't look at default discovery16:45
nsajeIf you have pdb in, you need to look at the second call16:46
nsajeFirst call is default discovery16:46
nsajeSecond call is plugin discovery16:46
DinaBelovaone moment16:46
DinaBelovaso what's the reason I'm not i  the get_my_subset method?16:48
DinaBelovathere is pdb there16:50
nsajeDinaBelova: good question, I don't know from here16:52
DinaBelovaa-ha, I see the reason - for the plugin discovery it'll be {"images": []}16:52
DinaBelovano discovery actually16:52
DinaBelovaa-ha, no, it were {"images": [{"status": "active", .... there as well - but no get_my_subset method call16:53
nsajeBreak in the poll_and_publish method16:55
nsajeAnd follow from there, until the list of resources is passed to the pollsters16:55
nsajeDinaBelova: ^16:55
DinaBelovauno momento16:55
*** yassine has quit IRC16:56
DinaBelovansaje, argh, I need to go home16:57
DinaBelovalet me try to copy all the logs16:57
DinaBelovato the paste16:57
cdentDinaBelova, nsaje: Just wanted to confirm that in a devstack, using memcached, and with a couple of instances spun up, I can get the partitioned compute-agent to work properly16:58
cdentBut only after fixing some typos in the LOG.debug lines recently added16:58
DinaBelovacdent - multinode devstack?16:59
cdentNot yet, no, this is just a simple setup.16:59
cdentnsaje: If you're on your phone do you mind if I patch the logging fixes so that they are there?17:00
nsajecdent: go ahead17:00
cdentroger that, will do17:00
openstackgerritChris Dent proposed a change to openstack/ceilometer: Central agent work-load partitioning
cdentthat makes the iterable, the members and the results visiable int he log17:04
*** harlowja has joined #openstack-ceilometer17:04
DinaBelovansaje, gordc, cdent - logs from all the ceilo on the controller (more or less fresh close to the end - take a look on the timestamps)17:06
DinaBelovathese logs are for the two days actually17:06
nsajeDinaBelova: thanks! Will take a look tomorrow17:07
DinaBelovabtw, funny moment - now I see the errors in the alarm evaluation - I did not look on it while the testing17:07
DinaBelovabut there are traces in the common log17:08
DinaBelovansaje, np17:09
*** pradk has joined #openstack-ceilometer17:12
*** jsavak has joined #openstack-ceilometer17:14
*** joesavak has quit IRC17:16
*** sbfox has quit IRC17:36
*** sdake_ has joined #openstack-ceilometer17:36
*** _nadya_ has joined #openstack-ceilometer17:40
*** _nadya_ has quit IRC17:42
*** sbfox has joined #openstack-ceilometer17:42
*** joesavak has joined #openstack-ceilometer17:44
*** jsavak has quit IRC17:47
*** sbfox has quit IRC17:55
*** _nadya_ has joined #openstack-ceilometer17:58
*** _nadya_ has quit IRC18:04
*** rbak has quit IRC18:07
*** sdake_ has quit IRC18:09
*** sbfox has joined #openstack-ceilometer18:10
*** changbl has quit IRC18:26
*** rbak has joined #openstack-ceilometer18:27
*** changbl has joined #openstack-ceilometer18:32
*** zigo has quit IRC18:43
*** dmsimard is now known as dmsimard_away18:44
*** dmsimard_away is now known as dmsimard18:45
*** harlowja has quit IRC18:46
*** zigo has joined #openstack-ceilometer18:46
*** harlowja has joined #openstack-ceilometer18:46
*** _nadya_ has joined #openstack-ceilometer18:49
*** htruta has joined #openstack-ceilometer18:56
*** sdake_ has joined #openstack-ceilometer19:03
*** sbfox has quit IRC19:10
*** sbfox has joined #openstack-ceilometer19:13
gordccdent, nsaje: compute partitioning works for me as well on memcached... hashring doesn't really split that well but i assume that's because i only have a few nodes.19:18
gordccentral agent doesn't work for me since there's no default discovery19:18
gordcfor reference, this is my pipeline.yaml (it has no discovery configured:
*** jsavak has joined #openstack-ceilometer19:22
*** sbfox has quit IRC19:22
*** r-daneel has joined #openstack-ceilometer19:24
*** joesavak has quit IRC19:24
*** joesavak has joined #openstack-ceilometer19:25
*** jsavak has quit IRC19:28
*** _nadya_ has quit IRC19:33
*** cdent_ has joined #openstack-ceilometer19:34
*** cdent has quit IRC19:36
*** cdent_ is now known as cdent19:36
*** cdent has quit IRC19:47
openstackgerritEric Pendergrass proposed a change to openstack/python-ceilometerclient: Verify alarm found before modifying
*** cdent has joined #openstack-ceilometer19:49
*** cdent has quit IRC19:49
*** cdent has joined #openstack-ceilometer19:57
*** flwang1 has joined #openstack-ceilometer19:57
*** xmltok has joined #openstack-ceilometer19:59
*** erecio has quit IRC20:07
*** flwang1 has quit IRC20:12
*** nsaje has quit IRC20:17
*** ccrouch has quit IRC20:20
*** sdake_ has quit IRC20:31
cdentgordc: did you have any leads on why HASHSEED impacts WSME?20:32
*** dmsimard is now known as dmsimard_away20:33
*** dmsimard_away is now known as dmsimard20:33
*** tongli has quit IRC20:35
*** julim has quit IRC20:36
gordccdent: nah i gave up...20:36
gordcagain something not reproducible locally20:36
cdentHow did you even figure it out?20:36
gordcthe errors we were getting in ceilometer were all timed to sphinxext code that exists in wsme20:37
gordcwhen i ran tests there. i started fixing issues but i'd post it to gerrit and jenkins would failed20:37
gordcand then i started to run rechecks on the other patches and they all failed.20:37
*** ildikov has joined #openstack-ceilometer20:37
cdentwas there any clue to it being hashseed, or just the fact that it was intermittent?20:38
gordcwell i added a hashseed patch to wsme and it's the only patch passing... but that's pretty much the extent of my debugging20:38
gordcgot to late.20:39
gordci think it was a friday.lol20:39
*** rodrigods has joined #openstack-ceilometer20:40
gordci guess i should probably look at that again since gate is broken20:41
*** ildikov_ has quit IRC20:41
cdentEverything is broken :)20:41
gordc:) i guess that's what happens when you merge a 3500line patch20:42
cdentboom! what was that?20:42
cdentI had started looking back at the hashseed stuff, because I did the original fix.20:42
gordcwant to look at the wsme issue? figured it's kind of late for you20:43
*** dekozo has joined #openstack-ceilometer20:43
cdentIt is getting late for me, but I'm planning to skip out tomorrow morning to do some other stuff so wanted to see if I could get something useful done this evening (other than trying to push existing stuff through)20:45
gordccool cool. have at it if you'd like20:45
cdentsuggestion on how to get it down to its narrowest bit?20:46
gordcdidn't get that far.20:46
gordcthat's my patch... it fails pretty regularly with a division by 0 error i think20:47
cdentOkay, I'll start there, as long as your happy with me jumping in there?20:47
gordci can take a look at it later if anything. i haven't started much debugging so no worries20:48
cdentI'll see if I can push it along some and get back to you.20:49
gordccdent: sounds good20:49
*** asalkeld has joined #openstack-ceilometer20:58
cdentI've pushed a new patch gordc which fixes the tests that is not good in the face of a random HASHSEED21:10
cdentBut what I don't understand is how broken tests in WSME breaks the ceilo docs job?21:10
cdentOr is there a longer chain of issues here?21:11
*** jsavak has joined #openstack-ceilometer21:11
*** joesavak has quit IRC21:14
cdentOr to put it another way, gordc,  what does your commit message have to do with the things you changed?21:19
*** yassine has joined #openstack-ceilometer21:20
*** terriyu has joined #openstack-ceilometer21:20
*** sbfox has joined #openstack-ceilometer21:27
*** jasondotstar has quit IRC21:28
*** sdake_ has joined #openstack-ceilometer21:33
*** sdake_ has quit IRC21:33
*** sdake_ has joined #openstack-ceilometer21:33
*** jsavak has quit IRC22:04
openstackgerritSrinivas Sakhamuri proposed a change to openstack/ceilometer: Get meters query performance improvements using index.
*** nsaje_ has quit IRC22:06
cdentI think I must be done for the night.22:08
* cdent waves22:08
*** cdent has quit IRC22:08
*** gordc has quit IRC22:08
*** rbowen has quit IRC22:12
*** pradk has quit IRC22:12
*** harlowja_ has joined #openstack-ceilometer22:13
*** harlowja_ has quit IRC22:15
*** harlowja_ has joined #openstack-ceilometer22:15
*** jergerber has quit IRC22:15
*** harlowja has quit IRC22:16
*** sbfox has quit IRC22:19
openstackgerritSrinivas Sakhamuri proposed a change to openstack/ceilometer: Get meters query performance improvements using index
openstackgerritEric Pendergrass proposed a change to openstack/python-ceilometerclient: Verify alarm found before modifying
*** Daviey has quit IRC22:31
*** sbfox has joined #openstack-ceilometer22:32
*** sdake_ has quit IRC22:32
*** Daviey has joined #openstack-ceilometer22:42
*** yassine has quit IRC22:53
*** stevelle has quit IRC23:02
*** asalkeld has quit IRC23:12
*** itisit has quit IRC23:12
*** stevelle has joined #openstack-ceilometer23:13
*** dmsimard is now known as dmsimard_away23:14
openstackgerritPhil Neal proposed a change to openstack/ceilometer: Document the standard for PaaS service notifications
*** sbfox has quit IRC23:23
*** asalkeld has joined #openstack-ceilometer23:24
*** ccrouch has joined #openstack-ceilometer23:25
*** ccrouch has quit IRC23:37
*** ccrouch1 has joined #openstack-ceilometer23:40
*** rbowen has joined #openstack-ceilometer23:41
*** sdake_ has joined #openstack-ceilometer23:45
*** sdake_ has quit IRC23:53
*** sdake_ has joined #openstack-ceilometer23:55

Generated by 2.14.0 by Marius Gedminas - find it at!