Thursday, 2015-07-16

*** ddieterly has joined #openstack-ceilometer00:04
*** ddieterly has quit IRC00:19
*** primechuck has quit IRC00:20
*** ddieterly has joined #openstack-ceilometer00:26
*** _cjones_ has quit IRC00:29
*** _nadya_ has joined #openstack-ceilometer00:32
*** _nadya_ has quit IRC00:37
*** ddieterly has quit IRC00:41
*** ddieterly has joined #openstack-ceilometer00:44
*** primechuck has joined #openstack-ceilometer00:50
*** Longgeek has joined #openstack-ceilometer01:00
*** primechuck has quit IRC01:10
*** liusheng has joined #openstack-ceilometer01:12
*** openstack has joined #openstack-ceilometer01:23
*** jruano has quit IRC01:26
*** htruta_ has joined #openstack-ceilometer01:34
*** jdandrea has quit IRC01:38
*** primechuck has joined #openstack-ceilometer01:44
*** jruano has joined #openstack-ceilometer01:48
*** ankita_wagh has joined #openstack-ceilometer01:59
*** ankita_wagh has quit IRC02:13
*** ankita_wagh has joined #openstack-ceilometer02:13
*** ankita_wagh has quit IRC02:14
*** ankita_wagh has joined #openstack-ceilometer02:15
openstackgerritRen Qiaowei proposed openstack/ceilometer: Add necessary executable permission  https://review.openstack.org/20235302:16
*** Longgeek has joined #openstack-ceilometer02:30
*** dikonoor has joined #openstack-ceilometer02:35
*** Longgeek has quit IRC02:35
*** julim has joined #openstack-ceilometer02:52
*** cmyster has quit IRC02:54
*** htruta_ has quit IRC03:04
*** primechuck has quit IRC03:17
*** dikonoor has quit IRC03:26
*** deepthi has joined #openstack-ceilometer03:35
*** serverascode has quit IRC03:45
*** zhiyan has quit IRC03:45
*** rex_lee has quit IRC03:46
*** h00327910_ has quit IRC03:46
*** boris-42 has quit IRC03:46
*** zhiyan has joined #openstack-ceilometer03:54
*** serverascode has joined #openstack-ceilometer03:58
*** jruano has quit IRC03:59
*** jruano has joined #openstack-ceilometer03:59
*** rex_lee has joined #openstack-ceilometer04:02
*** boris-42 has joined #openstack-ceilometer04:07
*** h00327910_ has joined #openstack-ceilometer04:11
*** belmoreira has joined #openstack-ceilometer04:15
*** jwcroppe has joined #openstack-ceilometer04:16
*** Longgeek has joined #openstack-ceilometer04:17
*** Longgeek has quit IRC04:25
*** Longgeek has joined #openstack-ceilometer04:25
*** belmoreira has quit IRC04:28
*** yatin has joined #openstack-ceilometer04:34
*** yprokule has joined #openstack-ceilometer04:45
*** ddieterly has quit IRC04:51
*** _nadya_ has joined #openstack-ceilometer04:56
*** _nadya_ has quit IRC04:57
*** _nadya_ has joined #openstack-ceilometer04:57
*** _nadya_ has quit IRC05:02
*** cmyster has joined #openstack-ceilometer05:03
*** cmyster has quit IRC05:03
*** cmyster has joined #openstack-ceilometer05:03
*** cmyster has quit IRC05:03
*** _nadya_ has joined #openstack-ceilometer05:14
openstackgerritRyota MIBU proposed openstack/aodh: Refactor alarm scenario tests (TestAlarmsHistory)  https://review.openstack.org/20196605:21
*** hans__ has joined #openstack-ceilometer05:23
*** hans_ has quit IRC05:26
*** jruano has quit IRC05:35
openstackgerritRohit Jaiswal proposed openstack/ceilometer: add Trove(DBaaS) events  https://review.openstack.org/20074605:35
*** _nadya_ has quit IRC05:38
*** _nadya_ has joined #openstack-ceilometer05:38
*** _nadya_ has quit IRC05:42
*** vishwana_ has joined #openstack-ceilometer05:43
*** vishwanathj has quit IRC05:47
*** ddieterly has joined #openstack-ceilometer05:52
*** ddieterly has quit IRC05:56
*** lakshmiS has joined #openstack-ceilometer06:03
lakshmiSquestion for ceilometer team. In Openstack searchlight project we are consuming glance notifications. when ceilometer is running we dont seem to get the notifications even though we are using a different pool name in oslo messaging?06:04
*** ildikov has quit IRC06:18
*** belmoreira has joined #openstack-ceilometer06:25
*** cmyster has joined #openstack-ceilometer06:27
*** Longgeek has quit IRC06:51
*** Longgeek has joined #openstack-ceilometer06:52
*** ddieterly has joined #openstack-ceilometer06:52
*** ddieterly has quit IRC06:57
*** ankita_wagh has quit IRC06:58
*** ankita_wagh has joined #openstack-ceilometer06:59
*** shardy has joined #openstack-ceilometer07:03
*** ankita_wagh has quit IRC07:03
*** ildikov has joined #openstack-ceilometer07:04
*** ankita_wagh has joined #openstack-ceilometer07:08
*** ankita_wagh has quit IRC07:16
*** _nadya_ has joined #openstack-ceilometer07:25
*** lan has quit IRC07:33
*** nijaba has quit IRC07:43
*** nijaba has joined #openstack-ceilometer07:44
*** nijaba has quit IRC07:44
*** nijaba has joined #openstack-ceilometer07:44
openstackgerritChris Dent proposed openstack/ceilometer: [WIP] Attempt to use grenade plugin with devstack plugin  https://review.openstack.org/20220307:46
*** Ala has joined #openstack-ceilometer07:49
*** ddieterly has joined #openstack-ceilometer07:53
*** lakshmiS has quit IRC07:53
*** Ala has quit IRC07:55
*** cdent has joined #openstack-ceilometer07:57
*** nsaje_ has joined #openstack-ceilometer07:57
*** ddieterly has quit IRC07:58
*** nsaje has quit IRC07:59
*** Ala has joined #openstack-ceilometer08:03
openstackgerritMehdi Abaakouk (sileht) proposed openstack/aodh: Use the right sqla type for JSONEncodedDict  https://review.openstack.org/20047608:05
openstackgerritMehdi Abaakouk (sileht) proposed openstack/aodh: api: Add location alarm creation  https://review.openstack.org/20242708:07
openstackgerritMehdi Abaakouk (sileht) proposed openstack/aodh: Add devstack support  https://review.openstack.org/20046708:07
*** safchain has joined #openstack-ceilometer08:12
openstackgerritMerged openstack/aodh: Use the right sqla type for JSONEncodedDict  https://review.openstack.org/20047608:25
*** boris-42 has quit IRC08:32
*** cdent has quit IRC08:33
*** cdent has joined #openstack-ceilometer08:33
*** yassine has joined #openstack-ceilometer08:40
*** exploreshaifali has joined #openstack-ceilometer08:41
*** exploreshaifali has quit IRC08:46
*** belmoreira has quit IRC08:51
*** ddieterly has joined #openstack-ceilometer08:54
*** ddieterly has quit IRC08:58
*** belmoreira has joined #openstack-ceilometer09:05
*** yatin has quit IRC09:06
openstackgerritliusheng proposed openstack/aodh: (WIP) Allow aodh directly acessing to its storage  https://review.openstack.org/20245409:16
*** hans__ has quit IRC09:18
*** yatin has joined #openstack-ceilometer09:34
*** Beetnik has joined #openstack-ceilometer09:41
*** jaosorior has joined #openstack-ceilometer09:41
*** deepthi has quit IRC09:50
*** ddieterly has joined #openstack-ceilometer09:55
*** ddieterly has quit IRC09:59
_nadya_sileht, cdent: hi folks, it's stuck a bit, could you please take a look https://review.openstack.org/#/c/199504/ . All your comments are fixed10:02
*** claudiub has joined #openstack-ceilometer10:03
sileht_nadya_, done10:03
_nadya_sileht: thanks!10:03
* sileht thought that was merged :p10:04
* cdent looks10:05
cdentit is now10:06
cdent(well, soon)10:06
*** afazekas has joined #openstack-ceilometer10:10
*** liusheng has quit IRC10:13
*** liusheng has joined #openstack-ceilometer10:13
openstackgerritMerged openstack/aodh: Make GnocchiThreshold evaluator derived from ThresholdEvaluator  https://review.openstack.org/19950410:14
openstackgerritMerged openstack/aodh: Refactor alarm scenario tests (TestAlarmsBase)  https://review.openstack.org/20141610:20
cdent_nadya_, sileht: can you guys push this, it will help with all the ceilo+gnocchi exploring going on this week: https://review.openstack.org/#/c/202245/10:28
cdentthanks10:31
cdentsileht: this one too: https://review.openstack.org/#/c/202259/10:34
*** afazekas has quit IRC10:35
cdentand this one: https://review.openstack.org/#/c/202191/10:35
silehtcdent, have you seen this one: https://bugs.launchpad.net/gnocchi/+bug/147254610:36
openstackLaunchpad bug 1472546 in Gnocchi "devstack: GNOCCHI_USE_MOD_WSGI no longer work by default" [Undecided,New]10:36
cdenthuh, I haven't had the problem, I wonder what I've done differently10:37
silehtcdent, I wonder what is the plan of infra to have everything on port 8010:38
silehtcdent, about logging in general10:38
silehtbecause one virtual host can only have on accesslog and one errorlog10:39
cdentyou mean how will people find their stuff in those logs? Presumably by the prefix in the request URL?10:40
* cdent doesn't know10:40
silehtcdent, devstack prepare a screen with one log per tab10:40
cdentyou mean the tail_log stuff?10:40
silehtyep10:41
silehtcdent, with one virtual host nothing seems exists in devstack to split the only log created with everything inside10:41
cdenttail log could presumably take an argument which is the prefix name (which is supposed to be the keystone endpoint) and do a grep during the tail10:42
cdentIt's not something I've put anything though into, I never run my devstack's with screen, is far too much overhead and noise10:42
*** exploreshaifali has joined #openstack-ceilometer10:43
cdentthat's probably why I didn't see that bug you've linked above10:43
silehtit's not only for screen, gate generates logs with tail_log too10:43
cdentwhy "generate logs" when they are already there?10:44
cdentI mean, at the end just copy the original log10:44
silehtsure10:45
silehtI just think devstack in not ready yet to run in that mode (everything on port 80) in gate10:46
cdentthat's probably true, and it's fine: the gate expectation is that all settings should be made explicitly and no defaults used10:46
cdentso if we need to make our gate settings more explicit, we should do that10:47
*** afazekas has joined #openstack-ceilometer10:48
cdentsileht: anyway, the reason for setting the wsgi stuff to default to /metric etc and non-port now, for individuals doing testing, is so that the issues that switching to it in real life will be revealed more quickly\10:53
cdentit's already shown a fair few assumptions10:53
cdents/few/few bad/10:53
*** ddieterly has joined #openstack-ceilometer10:55
openstackgerritChris Dent proposed openstack/gnocchi: Tail the correct apache log files in devstack  https://review.openstack.org/20249810:59
*** ddieterly has quit IRC11:00
cdentsileht: ^11:00
cdentI'm testing it now (with USE_SCREEN true)11:00
silehtcdent, I  would prefer the old behavior if GNOCCHI_SERVICE_PORT is set11:01
silehtcdent, and fallback to the catch in all case11:01
cdentthat is the old behavior isn't it?11:01
* sileht use often MOD_WSGI11:01
silehtcdent, oh yes sorry, misreading11:02
openstackgerritMehdi Abaakouk (sileht) proposed openstack/ceilometer: Move gnocchi resources definition in yaml file  https://review.openstack.org/20250011:05
*** cmyster has left #openstack-ceilometer11:07
openstackgerritChris Dent proposed openstack/ceilometer: [WIP] Attempt to use grenade plugin with devstack plugin  https://review.openstack.org/20220311:12
*** belmoreira has quit IRC11:15
openstackgerritMehdi Abaakouk (sileht) proposed openstack/ceilometer: Move gnocchi resources definition in yaml file  https://review.openstack.org/20250011:27
openstackgerritMehdi Abaakouk (sileht) proposed openstack/ceilometer: gnocchi: Remove useless resources patching  https://review.openstack.org/20250711:27
*** _nadya_ has quit IRC11:31
*** cmyster has joined #openstack-ceilometer11:31
*** _nadya_ has joined #openstack-ceilometer11:32
*** _nadya_ has quit IRC11:36
openstackgerritAleksey Ripinen proposed openstack/ceilometer: Batch recording metering data to gnocchi  https://review.openstack.org/20251111:38
*** shardy_ has joined #openstack-ceilometer11:39
*** shardy has quit IRC11:39
*** belmoreira has joined #openstack-ceilometer11:40
openstackgerritAleksey Ripinen proposed openstack/ceilometer: Batch recording metering data to gnocchi  https://review.openstack.org/20251111:41
*** shardy_ has quit IRC11:44
*** shardy has joined #openstack-ceilometer11:45
*** thorst has joined #openstack-ceilometer11:47
openstackgerritMerged openstack/gnocchi: Use a settings file for the devstack plugin  https://review.openstack.org/20219111:50
openstackgerritMerged openstack/gnocchi: Cleanup handling of metricd and its logs in devstack  https://review.openstack.org/20225911:51
cdentone more gnocchi+ceilometer+devstack related change: https://review.openstack.org/#/c/202192/11:51
*** Longgeek has quit IRC11:54
*** ddieterly has joined #openstack-ceilometer11:56
silehtcdent, if the ceilometer plugin is enabled after the gnocchi one you can still have the issue11:57
silehtfg11:57
silehtoups11:57
silehtcdent, perhaps something to add to doc/source/install/development.rst11:58
*** ddieterly has quit IRC12:01
*** Longgeek has joined #openstack-ceilometer12:05
*** prad has joined #openstack-ceilometer12:06
cdentthe collector problem sileht ? yeah, I'm not entirely sure what the real solution is12:11
cdentthere's been some talk about ways to manage dependencies between plugins but it hasnt' been solved yet12:12
silehtcdent, just documentation I think for now12:12
*** ddieterly has joined #openstack-ceilometer12:15
*** yatin has quit IRC12:20
*** gordc has joined #openstack-ceilometer12:24
*** pradk has joined #openstack-ceilometer12:27
*** yatin has joined #openstack-ceilometer12:29
gordcjd__: you get a chance to read Angus' email about aodh+heat?12:32
gordcit seems we need to get a client out there for the split to work properly... not sure if we want to have ceilometer redirecting to aodh as another solution.12:33
gordcildikov: cdent: ^^ you guys have opinion on best way to proceed?12:36
gordchttp://lists.openstack.org/pipermail/openstack-dev/2015-July/069684.html12:37
ildikovby keeping the old code deprecated and having support in the client to hide the situation we kind of should be ok I guess12:37
openstackgerritMerged openstack/gnocchi: Tail the correct apache log files in devstack  https://review.openstack.org/20249812:37
cdentgordc: this is kind of the same problem as "what do we do with ceilometerclient when gnocchi is turned on"12:38
ildikovcdent: what to do with the meters/samples part of the v2 API in general I would say12:39
cdentalarming is a bit easier to deal with because of the way endpoint handling works12:39
gordccdent: true.12:39
cdentbut it is still going to require some work12:39
gordcis the idea that we want to code ceilometerclient to point to the new services?12:40
cdentthe frictionless thing to do is setup in-gate ceilo and aodh so they both occupy the same effective endpoint with two different wsgi apps12:40
ildikovdoes heat depend on the client only?12:40
gordcor have a hard break and tell them to use new aodhclient12:40
ildikovas much as I remember the discussion with julien on the thread, I think we ended up with the option to hide this change in the client12:40
ildikovwill we have new aodh client? :)12:40
gordcildikov: i would think so. unless we want ceilometerclient to be a super client which wraps all our sub libs12:41
gordcwhich i'm indifferent to... it seems that's the least amount of breakage.12:42
ildikovgordc: yeap, I know, I just didn't remember it from the roadmap, but I don't really sleep much these days so it might be a reason for my poor memory12:42
gordccdent: does your suggestion still allow for same client to be used?12:43
ildikov... and lazyness to check12:43
gordcildikov: i don't think we discussed client much.12:43
cdentgordc: in theory, yes, but I wouldn't want to bet everything on it without more exploration12:43
ildikovgordc: I cannot remember too much discussion on it either12:44
gordcthe reason i asked is because liusheng offered to help out on the ceilometer+aodh gaps. and he was sort of wondering where to start.12:44
ildikovgordc: is it decided whether we keep the old code for Liberty or not?12:45
gordcildikov: the old code will sit as is for liberty12:45
gordcwe will remove in M*12:45
ildikovgordc: ok, good, I wasn't 100% sure we finally decided it12:46
gordcildikov: nope. that was one thing we definitely made decision on.12:46
gordci need to look at client code to see if it's ok to just continue using ceilometerclient12:46
ildikovas for the gaps I think we need to figure out the client question first to see how to move on12:47
gordci think that's the best solution for consumers12:47
gordcildikov: agreed.12:47
cdentgordc: I think there's two ways to go on the client12:48
ildikovalthough I'm still concerned with the long enough deprecation period, but it's kinda my job in the group to be the worrying party12:48
cdentone is to leave it alone and assume the server side changes I suggested12:48
*** shardy_ has joined #openstack-ceilometer12:48
cdentthe other is to make it aware of a new type of endpoint for anything to do with alarms12:48
cdentit ought to be possible to make that backward compatible: if able to find alarm endpoint then use it, otherwise use the old one...12:48
cdentetc12:48
*** shardy has quit IRC12:49
* gordc looking12:49
gordccdent: i'm trying out your pollster patch.12:50
gordcgood: it looks like it's drop cpu usage from compute agent12:51
*** yatin has quit IRC12:51
gordcside-effect: notification agent is consuming a massive amount of memory now12:51
cdentyeah, that's kind of expected12:51
cdentbecause now it has to have caches for everything and everyone12:51
cdentwhich is why you had to write that new spec12:52
cdentdoes the old compute-agent (without my changes) consume lots of memory in an allinone situation?12:52
cdentI haven't had a chance to actively play with it because gnocchi12:53
openstackgerritMehdi Abaakouk (sileht) proposed openstack/ceilometer: Move gnocchi resources definition in yaml file  https://review.openstack.org/20250012:53
ildikovcdent: I was also thinking about that backward compatible version for the client as falling back to the old one, but it only makes sene if we keep the old one for a while maybe12:53
*** shardy_ has quit IRC12:53
cdentildikov: what we do wasn't really the concern I was worried about there12:53
cdentmore about the same client being pointed at different clouds12:53
cdentsome clouds will be old12:53
openstackgerritMehdi Abaakouk (sileht) proposed openstack/ceilometer: Move gnocchi resources definition in yaml file  https://review.openstack.org/20250012:53
gordccdent: not really. the cpu usage would spike a lot more during polling cycle but i can't remember it ever jumping much in memory12:53
cdentthen maybe we are just tickling a bug in the notification agent that was lying latent12:54
*** shardy has joined #openstack-ceilometer12:54
ildikovcdent: yeap, right, I had that option in mind too, so agreed12:54
cdentbecause in an a single node situation nothing should really have changed12:54
gordccdent: i have notification using GB of memory right now12:54
cdentossum12:55
gordcweird thing is, that collector is consuming just as much memory12:55
cdenthmmm12:55
cdentyou sure something else isn't going on?12:55
gordcso it might be an oslo.messaging thing? i don't think anything happens on collecot12:55
cdentOpportunities12:56
cdent12:56
*** jruano has joined #openstack-ceilometer12:56
gordcsileht: ^^12:56
silehtgordc, I known that nova can create a payload of many megabytes12:57
*** _nadya_ has joined #openstack-ceilometer12:57
cdentgordc: I'm on a long slide into despair on finding and fixing little tiny problems in ceilo+gnocchi+devstack, once I rise out of that I can git the pollster thing a closer look but it might be days12:57
gordcsileht: so just for reference, i'm using oslo.messagin with harlowja patch to fix ack issue.12:57
cdentgordc: MAYBE WE SHOULD SWITCH BACK TO SINGLE SAMPLE PER NOTIFICATION :)12:57
gordcsileht: i'm polling every 10s12:57
cdentsexy12:58
gordcon 8VM.12:58
silehtgordc, yep I have added unit tests in its patch12:58
silehtgordc, me too12:58
*** thorst has quit IRC12:58
gordccdent: did you performance test?12:59
gordccdent: i will hear none of it, if you didn't.12:59
cdentno, I'm letting you performance test13:00
gordcperformance test == watch top/htop manually like a chimp for a few minutes.13:00
cdentand clearly memory performacne is crap13:00
gordccdent: just for reference, i did the same thing with existing master and collector will increase just the same.13:01
cdentthat sound like a bug we could find13:01
cdentbut the pollster does _not_ grow?13:01
cdent(on master)13:02
gordccdent: not that i noticed... that said, i didn't test master that long... not recently13:03
cdentwe have so much to do13:03
cdenthowever13:03
cdentall of this discovery is great13:03
cdenti love it when a plan comes together13:03
*** ddieterly has quit IRC13:03
cdentdoes "collector gets freaking huge" count as a bug?13:04
gordcsure why not13:04
cdentossum13:04
*** thorst has joined #openstack-ceilometer13:04
gordccdent: are you actually creating that? i have a copy of top for my env13:06
cdentI'm not, I was passively aggressively prompting you to do so13:06
gordckk13:07
gordci'm going to run master for 12hrs and see if it's any different13:07
*** pauldix has joined #openstack-ceilometer13:09
*** edmondsw has joined #openstack-ceilometer13:28
*** primechuck has joined #openstack-ceilometer13:28
cdentgordc: how much ram does your machine have?13:31
cdent(the one on which you are seeing the potential memory leak)13:31
gordccurrent one. 32GB13:31
cdentoh hey, I'm getting it too, 21.9g13:32
cdent(this is with gnocchi paired in as well)13:33
cdentgordc: have you restarted your rabbit lately?13:33
gordc... 21g on one service or all?13:33
gordccdent:13:33
gordccdent: nope13:33
cdentone service!13:33
cdentthat's just the collector13:33
gordc... how much memory do you have!??!:?!13:33
cdent3213:34
gordcoh.13:34
gordcare you using one worker?13:34
cdentfor the collector, yeah, at the moment13:34
gordcor just doing the math for me.13:34
gordcyikes. i know when i comparing mongo vs sql the mongo backend used less memory but way more cpu13:35
gordcright now i'm using sql as my backend when i have GBs of memory usage13:35
cdentthe reason I asked about rabbit is because if it hasn't been cleared out since doing gnocchi-related stuff it might be causing issues13:35
cdentI'm not sure of devstack tidies rabbit up or not13:35
gordcwhat does gnoochi do that requires me to restart rabbit?13:36
gordci haven't restarted since i tried to flood gnocchi with meters... it didn't work that well.13:36
cdentgnocchi causes vast amount of messages over the bus to the collector for the sake of the dispatcher and then sends some more after13:37
gordcbut if the queue is empty it should be fine right?13:37
cdentand then if you haven't got the messaging patch in place they all get requeued13:37
cdentgordc: yeah, I think so13:37
cdentthat was the real question: has your queue been confirmed as clean13:37
gordcok, yeah i cleared all the queues.. i did a test where i queued a few thousand records across 3 queues. and let mongo, sql, and gnocchi try to store them13:38
openstackgerritChris Dent proposed openstack/ceilometer: Store and restore the xtrace option in devstack plugin  https://review.openstack.org/20256113:39
gordcmongo does really well13:39
gordcsql needs more than 2 workers.13:39
openstackgerritChris Dent proposed openstack/gnocchi: In devstack use our own install redis command  https://review.openstack.org/20256313:40
gordcgnocchi i'm pretty sure there's something wrong. i'm hoping it's the dispatcher13:40
cdents/something/lots of things/13:41
cdentbut finding bugs is good13:41
cdentit means we get to fix thigns13:41
silehtgordc, do you use swift as backend ?13:41
*** prad has quit IRC13:41
gordci don't know what the bug is unfortunately. didn't get time/know enough to debug.13:41
gordcsileht: yeah. i turned off ceilometermiddleware13:42
gordcsileht: i tried with file backend too13:42
silehtgordc, is it better with file ?13:42
gordcit was about the same (maybe a little better)13:42
gordcfile is better.13:42
* cdent starts a ceph run13:42
*** prad has joined #openstack-ceilometer13:42
silehtgordc, I have increase the number of workers for swift:13:43
sileht[[post-config|${SWIFT_CONF_DIR}/proxy-server.conf]]13:43
sileht[DEFAULT]13:43
silehtworkers = 4013:43
gordci said 'something wrong' because one time gnocchi+swift cleared a queue really really quick (near mongo speed) but i think it's because it just dropped all the messages.13:43
gordcsileht: what's the normal default?13:44
silehtone I guess13:44
gordclol13:44
silehtone http connections at a time is a joke13:44
gordci can try that change over the weekend.13:44
gordcsileht: yeah. i thought swift in devstack created a lot of workers. i'll try again and see if it improves.13:45
gordcsileht: is there a way to just use generic resource and not build resource specific ones?13:46
gordcor do i need to hack dispatcher.13:46
silehtgordc, hacking :)13:46
*** jdandrea has joined #openstack-ceilometer13:46
gordcsileht: kk. i'll try that if 40 workers don't work.13:46
silehtgordc, I'm currently review the workflow to reduce the number of patch for a resource13:47
gordcsileht: on gnocchi side or ceilometer?13:48
silehtgordc, ceilometer side13:48
gordccool cool.13:48
*** exploreshaifali has quit IRC13:48
gordcjust for reference, right now i'm sending 30-50 messages/second over 10s polling cycle.13:49
silehtbut to be really efficient we need to put a maximun number of smaples into one message13:49
cdentthere's a patch in progress for that13:49
gordcsileht: you mean batch them right?13:49
cdenthttps://review.openstack.org/#/c/202511/13:49
silehtgordc, yes on sender side should be easier I guess13:49
cdent(btw, ceph fails to work with Error connecting to cluster: PermissionError, so I presumably have no idea what I'm doing, will skip ceph for now)13:50
silehtcdent, oh something like that but this patch afraid me13:50
silehtdoes record_batch_metering_data exists in the collector ?13:50
cdentyeah, I was commenting on  the quality of the patch, just the existence thereof13:50
silehtI have missed something ?13:50
cdents/was/wasn't/13:51
cdentI've barely read it, just saw it flash by13:51
gordcsileht: it doesn't exist.13:51
*** pradk has quit IRC13:52
silehtthe current method already batch by resource_id and counter_name, rewrite the whole workflow look wierd...13:52
gordci don13:52
gordci dont know where the batching is happening in that patch. iirc, we process it one at a time at the collector.13:52
cdentwell, if we don't like that person's way, we probably ought to stop them sooner than later13:54
gordcthere's this bp from a while ago: https://blueprints.launchpad.net/ceilometer/+spec/bulk-message-handling13:55
gordccdent: you made me the bad guy13:57
silehtI comment13:57
cdentyou know you wanted it13:57
gordcyeah13:57
gordcsileht: on the patch or bp?13:57
silehtgordc, on the patch13:57
gordcsileht: cool cool.13:58
cdentgordc, sileht: what fix do I need to o.messging14:01
cdentso that I'm not getting the REQUEUED related errors?14:02
silehtcdent, https://review.openstack.org/#/c/202202/14:02
cdentthanks14:02
silehtcdent, don't look at it, just apply, otherwise you will stop doing python ;p14:02
cdentha!14:02
cdenttoo late14:02
*** Kennan has quit IRC14:02
*** Kennan has joined #openstack-ceilometer14:03
* cdent cries14:03
gordcyeah. i already commented that it could be a one line change... but it is what it is.lol14:03
*** rbak has joined #openstack-ceilometer14:06
cdentso my collector is going to 5g within 30s14:08
cdentand growing by a gig about every 5 s14:08
* cdent smells a bad loop14:09
*** claudiub has quit IRC14:15
cdentit did that because as soon as it starts up all the noise on the message bus comes in all at once and we grow without bound14:16
cdentif we start the collector after a rabbit restart, then it ismoderably happy14:16
cdentsileht: if the eventlet executor is being used with get_notification_listener will that just continue to accept messages no matter how many it has going at once?14:19
silehtcdent, they are a limitation14:19
gordccdent: i was going to say same thing before i got pulled aside14:19
gordci grow about 100MB/10min14:20
silehtcdent, config: rpc_thread_pool_size=6414:20
gordcthis is with master+sql14:20
silehtso the collector process 64 messages per listener per collector14:20
cdentbrb14:22
gordcsileht: so it'll grab 64messages, process each one individual (record_metering_data) and then go back and grab another 64?14:22
silehtgordc, this is a threads pool each message processing go to a thread14:22
silehtgordc, when a process finish the thread return to the pool14:23
silehtgordc, and an other can be processed14:23
gordcah i see.14:23
silehtyour should have ~ 64 messages per collector processed at the same times14:23
gordcsileht: kk that makes sense now.14:23
openstackgerritMehdi Abaakouk (sileht) proposed openstack/ceilometer: Move gnocchi resources definition in yaml file  https://review.openstack.org/20250014:23
openstackgerritMehdi Abaakouk (sileht) proposed openstack/ceilometer: gnocchi: reduce the number of patch to gnocchi API  https://review.openstack.org/20258914:23
openstackgerritAleksey Ripinen proposed openstack/ceilometer: Add complex query for statistic requests  https://review.openstack.org/18833314:26
openstackgerritMehdi Abaakouk (sileht) proposed openstack/ceilometer: gnocchi: reduce the number of patch to gnocchi API  https://review.openstack.org/20258914:26
openstackgerritMehdi Abaakouk (sileht) proposed openstack/ceilometer: Move gnocchi resources definition in yaml file  https://review.openstack.org/20250014:26
openstackgerritMehdi Abaakouk (sileht) proposed openstack/ceilometer: gnocchi: reduce the number of patch to gnocchi API  https://review.openstack.org/20258914:27
silehtgordc, cdent I have thought about using event to create/update resource but I unsure that will really works in pratice14:28
silehtI don't think we can have when the server_group of a heat stack is updated for example14:28
cdentyeah, it would be nice to be able to do but it is not clear if it is reliable14:29
sileht(without updating the instance resource of each notifications, because in this case we have the same perf than the current workflow)14:29
cdents/reliable/universal/14:29
gordcsileht: yeah i think that relates back to building compute metadata from event.14:29
gordcsileht: the reason i asked about just using generic resource was to see how much the patching affected performance.14:30
gordcjust personally, i think it'd be good to remove all the metadata stuff from gnocchi. but that makes gnocchi sort of reliate on events in ceilometer.14:31
gordccdent: if we build events using your pollster-no-transform stuff. does that make it more reliable?14:32
gordcprobably not for heat i guess since we don't poll that.14:32
silehtgordc, but for heat and autoscaling, we need to aggregates measures of all instances in same server_group14:32
cdentyeah, gordc, that's the exact problem14:32
cdentwe dont' have control over the informaiton14:32
gordccdent: yeah. or the order we process it (assuming race)14:33
silehtgordc, the problem is that we have nowhere an id to identify a autoscaling group, just a name in instance user_metadata14:34
gordcsileht: is that an issue for ceilometer or heat?14:35
silehtfor ceilometer14:36
gordchow do we (properly) track something that doesn't have an id?14:36
silehtgordc, remove the word 'properly' and that works :p14:37
gordc:) i'm not sure that's how it works14:37
*** shardy_ has joined #openstack-ceilometer14:38
silehtgordc, we don't track it, we track metrics of an instance and heat creates alarms triggered on an aggregation on multiple instances (by filtering with metadata)14:38
silehtthat why metadata exists in gnocchi14:38
*** shardy has quit IRC14:39
silehtabout your memory usage issue, this is something that occurs with oslo.msg 1.16.X too ?14:40
cdentsileht: I haven't had a chance to try yet14:42
gordcsileht: yeah. i guess my idea was that it'd need to do metadata filtering from event side... but there's a lot of work to synchronise the ids from meter and event db.14:42
gordci didn't start looking at memory until 1.17.x14:42
cdentfrom what i can tell it only happens when a whole huge pile of stuff is in the queue14:43
cdentwhich is not super easy to set up unless there's that bug in place14:43
gordci'll try with 1.1614:43
*** shardy_ has quit IRC14:43
silehtcdent, removes the gnocchi/swift filter it will fill the queue quickly ;)14:44
*** shardy has joined #openstack-ceilometer14:44
gordccdent: it can happen if you're doing upgrade but leaving pollsters up to continue meter collection?14:44
gordcsileht: :) you'll be in thousands within seconds.14:44
cdentthat seems like a likely scenario, gordc14:45
cdentas long as you are polling fast14:45
gordccdent: yeah... in my env i'm doing 10s and even if i start fresh, memory will increase... so i think it's pretty much just based on how consistently you're processing data.14:46
gordci will try with 1.16 after meeting14:47
* gordc switches polling to 600s. will see if it affects collector memory over meeting14:55
*** eglynn has joined #openstack-ceilometer14:55
*** _nadya_ has quit IRC14:55
*** _nadya_ has joined #openstack-ceilometer14:55
*** Longgeek has quit IRC14:58
*** _nadya_ has quit IRC14:59
*** Longgeek has joined #openstack-ceilometer15:00
*** _nadya_ has joined #openstack-ceilometer15:06
*** Longgeek has quit IRC15:07
openstackgerritMehdi Abaakouk (sileht) proposed openstack/ceilometer: gnocchi: reduce the number of patch to gnocchi API  https://review.openstack.org/20258915:13
openstackgerritMehdi Abaakouk (sileht) proposed openstack/ceilometer: Move gnocchi resources definition in yaml file  https://review.openstack.org/20250015:13
*** belmoreira has quit IRC15:13
*** afazekas has quit IRC15:20
cdentseen this yet sileht? I'm getting it now too, went to report the bug and found it was already there but brand new: https://bugs.launchpad.net/gnocchi/+bug/147532915:22
openstackLaunchpad bug 1475329 in Gnocchi "Exception in gnocchi-metricd: ValueError: cannot reindex from a duplicate axis" [Undecided,New]15:22
openstackgerritMehdi Abaakouk (sileht) proposed openstack/aodh: api: Add location alarm creation  https://review.openstack.org/20242715:23
openstackgerritMehdi Abaakouk (sileht) proposed openstack/aodh: Add devstack support  https://review.openstack.org/20046715:23
silehtcdent, that looks bad15:24
cdentyeah, once it is there, the datastore is hosed15:24
cdentyou can still get measures out, at least for some metrics15:25
cdentbut stuff won't go in15:25
*** rjaiswal has joined #openstack-ceilometer15:28
*** ig0r_ has quit IRC15:40
*** _nadya_ has quit IRC15:45
*** shardy_ has joined #openstack-ceilometer15:45
*** _nadya_ has joined #openstack-ceilometer15:45
*** vishwanathj has joined #openstack-ceilometer15:45
*** shardy has quit IRC15:46
*** vishwana_ has quit IRC15:47
*** _nadya_ has quit IRC15:50
*** shardy_ has quit IRC15:50
*** shardy has joined #openstack-ceilometer15:51
gordccdent: collector still increasing with 600s polling15:56
gordcabout 15mb in an hour15:56
silehtgordc, for the record I have started this one day https://review.openstack.org/#/c/202666/ to allow configure pools on sender side, and provide the delivery assurance for all pools, not only the default 'notifications' one15:56
gordcsileht: cool cool. we need to replicate you so you can do more work. :)15:57
silehtgordc, unsure doing more unfinished work help a lot ;)15:58
openstackgerritIlya Tyaptin proposed openstack/gnocchi: InfluxDB storage driver  https://review.openstack.org/16540715:58
*** ddaskal has joined #openstack-ceilometer15:59
*** ccrouch has left #openstack-ceilometer16:03
gordclol16:04
openstackgerritMerged openstack/ceilometer: Set conf.gnocchi_dispatcher.url explicitly in tests  https://review.openstack.org/20224516:06
cdentgordc: if you could report your discoveries with that memory stuff somewhere, that would be handy, I may be back later, but gotta scoot now16:08
* cdent waves16:08
*** cdent has quit IRC16:08
*** thorst has quit IRC16:18
*** edmondsw has quit IRC16:26
*** _cjones_ has joined #openstack-ceilometer16:30
*** ankita_wagh has joined #openstack-ceilometer16:33
*** rwsu has quit IRC16:44
*** safchain has quit IRC16:49
*** thorst has joined #openstack-ceilometer16:52
openstackgerritOpenStack Proposal Bot proposed openstack/ceilometer: Updated from global requirements  https://review.openstack.org/20065916:59
*** pasquier-s has quit IRC17:01
*** tuvenen has quit IRC17:03
*** ildikov has quit IRC17:05
gordcnothing exciting at meeting. just unveiled roadmap and directing resources to ceilometer+aodh and ceilometer+gnocchi work17:06
gordcprad^17:06
*** pasquier-s has joined #openstack-ceilometer17:08
*** tuvenen has joined #openstack-ceilometer17:11
*** lexholden has joined #openstack-ceilometer17:12
*** tuvenen has quit IRC17:15
*** pasquier-s has quit IRC17:15
*** pasquier-s_ has joined #openstack-ceilometer17:22
*** tuvenen has joined #openstack-ceilometer17:23
gordcprad: regarding https://review.openstack.org/#/c/197633 do you know when you'll have subsequent patches?17:24
pradgordc, working on them as we speak actually.. should have them out probably tonight/tomorrow17:31
gordcprad: cool cool. i have a comment but not critical.17:32
pradgordc, deprecating the code/tests is taking time17:32
pradgordc, i'll have multiple ones after this17:32
gordcprad: "rm <filepath>"17:33
pradgordc, cool, i'll address them in the subsequent ones if its not critical17:33
gordcjust burn it all.17:33
*** exploreshaifali has joined #openstack-ceilometer17:34
gordcprad: are you working on volume plugin support next or deprecating? wondering if there's a split somehwere to make things faster.17:35
pradgordc, hehe, yea the instance ones are a bit tricky17:35
pradgordc, was thinking of deprecating whats already converted as next17:35
gordctbh, i don't think the tests are doing naything. just verifying the processor works against some madeup notification.17:35
pradgordc, then we'll be done with one round and i can look into the plugin support17:35
pradgordc, yea i'm more excited about deprecating the handlers themselves17:36
pradgordc, tests i'll just nuke em17:36
gordcprad: ok. let me know if you run into issues. trying to make sure we have as much functionality in for liberty (all hopefully)17:36
pradgordc, and disabling them from the entrypoints etc17:36
pradgordc, yep will do17:37
pradgordc, if we can get the current one in, that gets us the main framework17:37
pradgordc, converting meters is just laborious17:38
pradgordc, and perhaps we can do that as we go and get as much done as possible for liberty17:38
gordcprad: or all. ;)17:38
pradgordc, we have enough time i think17:38
gordcprad: kk17:39
gordcprad: do you know how long jasonamyers is on holiday?17:39
pradgordc, donno, i was looking to ping him on the resource-metadata-caching to see where he was17:40
gordcyeah. i'll check by next week. that's one thing i really want to benchmark17:41
gordcor give us time to implement broader planb17:41
pradyea17:42
*** yprokule has quit IRC17:43
*** htruta_ has joined #openstack-ceilometer17:46
openstackgerritDoug Hellmann proposed openstack/ceilometermiddleware: Drop use of 'oslo' namespace package  https://review.openstack.org/20272017:47
*** edmondsw has joined #openstack-ceilometer17:49
*** edmondsw has quit IRC17:51
*** edmondsw has joined #openstack-ceilometer17:51
*** shardy_ has joined #openstack-ceilometer17:53
dhellmanngordc: it would be great to get ^^ merged quickly this week so it doesn't hold up the oslo.messaging release17:53
*** shardy has quit IRC17:54
gordcdhellmann: +A'd :)17:54
dhellmanngordc: sweet, thanks17:54
gordcdhellmann: np17:54
*** henriquetruta has joined #openstack-ceilometer17:57
*** henriquetruta has quit IRC17:58
*** shardy_ has quit IRC17:58
*** shardy has joined #openstack-ceilometer17:59
*** Kennan has quit IRC18:02
*** Kennan has joined #openstack-ceilometer18:03
*** Ala has quit IRC18:07
*** htruta_ has quit IRC18:18
*** Ephur has joined #openstack-ceilometer18:19
*** ildikov has joined #openstack-ceilometer18:22
*** promulo_ has joined #openstack-ceilometer18:30
*** yassine has quit IRC18:34
*** nsaje__ has joined #openstack-ceilometer18:42
*** nsaje_ has quit IRC18:43
*** lexholden has quit IRC18:44
*** jaosorior has quit IRC18:46
*** _nadya_ has joined #openstack-ceilometer18:46
*** lexholden has joined #openstack-ceilometer18:46
*** nsaje___ has joined #openstack-ceilometer18:48
*** lexholden has quit IRC18:48
*** _nadya_ has quit IRC18:50
*** nsaje__ has quit IRC18:51
*** nsaje____ has joined #openstack-ceilometer18:51
*** nsaje___ has quit IRC18:53
openstackgerritMerged openstack/ceilometermiddleware: Updated from global requirements  https://review.openstack.org/20133518:53
*** _nadya_ has joined #openstack-ceilometer19:01
*** rbrooker has joined #openstack-ceilometer19:01
*** _nadya_ has quit IRC19:02
*** _nadya_ has joined #openstack-ceilometer19:02
*** htruta_ has joined #openstack-ceilometer19:04
openstackgerritMerged openstack/ceilometer: Declarative meters support  https://review.openstack.org/19763319:07
ityaptingordc: Hi! Could you take a look into patch with elasticsearch ttl? https://review.openstack.org/#/c/200573/19:10
*** ankita_wagh has quit IRC19:13
*** ankita_wagh has joined #openstack-ceilometer19:15
*** belmoreira has joined #openstack-ceilometer19:18
*** htruta_ has quit IRC19:21
gordcityaptin: sure. i will try later today. trying to memory profile a bug currently.19:23
ityaptingordc: With ceilometer-collector?19:24
gordcityaptin: yeah19:25
gordcor it's somewhere in new oslo.messaging19:25
gordcwhen i use 1.16.0 it's fine.19:25
gordc1.17.1, it'll explode.19:25
ityaptingordc: Hmm, it look like oslo.messaging fault)19:32
gordcityaptin: yep. just don't know what. something is being held. it's something obvious because it grows really really quickly19:33
*** ankita_w_ has joined #openstack-ceilometer19:36
*** ankita_wagh has quit IRC19:36
ityaptingordc: i started nightly test with same scenario as described, but comment all dispatcher rpc in ceilometer collector. If mem doesn't grow up when ceilometer-collector is more guilty than we think :)19:40
gordcityaptin: cool cool. let us know if you notice anything19:47
*** harlowja has joined #openstack-ceilometer19:49
*** harlowja_ has quit IRC19:49
*** harlowja has quit IRC19:52
*** harlowja has joined #openstack-ceilometer19:53
*** _nadya_ has quit IRC19:57
*** _nadya_ has joined #openstack-ceilometer19:57
*** lexholden has joined #openstack-ceilometer20:01
*** _nadya_ has quit IRC20:02
*** exploreshaifali has quit IRC20:04
*** belmoreira has quit IRC20:11
pradgordc, so when we deprecate the old ones, we need to move the cfg opts for exchange control somewhere .. i was thinking i'll just add them to the meter notification handler itself. Is there a better place?20:13
gordcprad: if you have meter handler enabled it'll be loaded anyways20:17
gordcright now it just tries to load same target over and over again which is why i added a check20:18
pradgordc, but we need to add the cfg opts for each exchange? otherwise oslo_config will give a no such option20:18
gordcah the options.20:19
gordcyeah, just add them to meter or notification agent.20:19
pradyea20:19
gordci think it's better at notification agent20:19
gordcwe shouldn't really be defining targets in meters tbh.20:19
*** pauldix has quit IRC20:22
openstackgerritRohit Jaiswal proposed openstack/ceilometer: Handles dns.domain.exists event in Ceilometer  https://review.openstack.org/20277220:26
*** Ephur has quit IRC20:27
openstackgerritRohit Jaiswal proposed openstack/ceilometer: Handles dns.domain.exists event in Ceilometer  https://review.openstack.org/20277220:33
openstackgerritRohit Jaiswal proposed openstack/ceilometer: Fixes querying alarm history with severity field  https://review.openstack.org/20023820:37
*** lexholden has quit IRC21:07
*** rjaiswal has quit IRC21:14
*** gordc has quit IRC21:33
*** boris-42 has joined #openstack-ceilometer21:42
*** gordc has joined #openstack-ceilometer21:42
*** edmondsw has quit IRC21:45
*** shardy has quit IRC22:00
*** jruano has quit IRC22:04
*** jruano has joined #openstack-ceilometer22:08
*** ddaskal has quit IRC22:10
*** primechuck has quit IRC22:13
*** primechuck has joined #openstack-ceilometer22:15
*** primechuck has quit IRC22:16
*** thorst has quit IRC22:21
*** gordc has quit IRC22:26
*** ankita_wagh has joined #openstack-ceilometer22:30
*** ankita_w_ has quit IRC22:34
*** cing has joined #openstack-ceilometer22:35
*** thorst has joined #openstack-ceilometer22:54
*** thorst has quit IRC22:56
*** _nadya_ has joined #openstack-ceilometer22:58
*** _nadya_ has quit IRC23:02
*** rbak has quit IRC23:02
*** Beetnik has quit IRC23:10
*** Beetnik has joined #openstack-ceilometer23:13
openstackgerritOpenStack Proposal Bot proposed openstack/ceilometer: Updated from global requirements  https://review.openstack.org/20065923:14
*** zigo has quit IRC23:21
*** ankita_wagh has quit IRC23:21
*** ankita_wagh has joined #openstack-ceilometer23:22
*** zigo has joined #openstack-ceilometer23:22
*** cing has quit IRC23:23
*** lexholden has joined #openstack-ceilometer23:29
*** openstackstatus has joined #openstack-ceilometer23:33
*** ChanServ sets mode: +v openstackstatus23:33
*** ddieterly has joined #openstack-ceilometer23:40
*** barra204 has joined #openstack-ceilometer23:54

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!