15:00:34 #startmeeting ceilometer 15:00:34 Meeting started Thu Jul 16 15:00:34 2015 UTC and is due to finish in 60 minutes. The chair is gordc. Information about MeetBot at http://wiki.debian.org/MeetBot. 15:00:35 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 15:00:37 The meeting name has been set to 'ceilometer' 15:00:44 o/ 15:00:44 o/ 15:00:49 o/ 15:01:05 o/ 15:01:18 o/ 15:01:46 i'll give it a minute. 15:02:02 i think we have a few people out on holiday/meeting today 15:02:39 hi 15:03:04 Howdy y'all. 15:03:11 cool. let's call this quorum. 15:03:30 so start off. thanks for all those who attended last week's midcycle. 15:03:53 not ideal timing for americas but considering our developer base, it is what it is. 15:04:02 thanks to cdent and prad for organising 15:04:24 * cdent plays a little violin for poor gordc 15:04:26 yeap thanks! 15:04:27 yep, cdent++ prad++ 15:05:10 we'll revisit the topic later but if you have input/feedback, feel free to bring it up during open discussion (or email me if you want it to remain private because you dont want to air dirty laundry) 15:05:41 first topic is a result of midcycle 15:05:46 #topic recurring: roadmap items (new/old/blockers) https://wiki.openstack.org/wiki/Ceilometer/RoadMap 15:05:59 we now have a 'formal' roadmap 15:06:27 nice 15:06:42 basically it lists all open priority items, some open items that could be targetted to current or next cycle 15:07:02 and some longer term goals which have been tossed around (by devs/users) 15:07:04 huge +1 for having this 15:07:11 on the gnocchi roadmap, do we need a "what to do about v2 API coexistence" item? 15:07:27 the longer term goals have no been discussed and will need to go through review process 15:07:54 eglynn: is that a Gnocchi roadmap? 15:08:03 eglynn: yes we can add that. we discussed v3 api at the midcycle 15:08:13 eglynn: or can we consider momre as v2 deprecation roadmap? 15:08:21 ildikov: sorry, I mean the gnocchi list on the general roadmap 15:08:40 but we didn't have a conclusive decision on whether we needed it just yet as it is only gnocchi that represents the change in representation 15:08:43 eglynn: a-ha, ok 15:09:31 we decided it's something to start thinking about... but right now gnocchi api will remain separate. 15:09:53 gordc: one simple approach might be to disable the meters/samples/statistics endpoint in v2 API if gnochi dispatcher is enabled 15:10:33 ... i.e. return a fault saying "don't use me, use the gnochhi-api instead to retrieve metric data" 15:10:35 yeap, but we need proper docco for that as it is quite big API change... 15:11:42 eglynn: that sounds like a easy solution... if the conf file passed to api is pointing to gnocchi as dispatcher? 15:12:20 that makes sense gordc 15:12:28 eglynn: i don't think the api currently is aware if gnocchi is enabled... all the normal setup is done. it would need to look at dispatcher value i think (which it doesn't currently) 15:12:33 it is safer than just checking for gnocchi's endpoint (as that's not conclusive) 15:12:33 gordc, cdent: yep, or even have a separate config option to enable that stricter behavior 15:12:44 yep 15:13:09 I think checking the dispatcher value is more bullet proof 15:13:53 just wondering if there's issues if there are multiple collectors... with different dispatchers 15:14:29 so there's use case to publish datapoint to different targets so it's handled differently... osmething to look into. 15:14:39 alternatively you could have two separate pipelines in wsgi 15:14:51 like pipeline = request_id authtoken api-v2-server 15:14:59 vs. pipeline = request_id authtoken api-v3-server 15:15:14 so you decide what API to enable 15:15:52 multiple dispatchers I guess could be handled via a separate config option to turn on the stricter behavior 15:16:45 fabiog: yeah that's true. can it point to an external api? if api-v3-server was gnocchi? 15:17:29 gordc: I think so, you will need a stub in the Ceilo code but then you can import the Gnocchi API in there 15:17:43 gordc: or make calls 15:18:11 fabiog: kk. something to look into 15:18:38 we should probably target this for liberty if we intend on promoting gnocchi as our usable 'default' driver. 15:19:15 gordc: strong +1 on that targeting 15:19:25 gordc: +1 15:19:37 #action add workitem to either disable metering apis when gnocchi is enabled or create wsgi pipeline 15:20:22 so i think one thing we haven't figured out was how to manage the road map... anyone can add stuff so how do we ensure people are aware of updates. 15:20:51 gordc: people can subscribe to email notifications for updates on that page, if they want 15:21:26 cdent: ah cool. i think i just didn't subscribe. 15:21:37 gordc: another item could be the aodh client question regarding targets there 15:21:50 non-issue then 15:22:31 ildikov: right. liusheng offered to start working on the aodh+ceilometer integration work 15:23:01 devstakc plugin is ready: https://review.openstack.org/#/c/200467/ 15:23:07 gordc: cool 15:23:15 i believe the opinion from ceilometer users (ie. heat) is that we keep the functionality in ceilometerclient 15:23:46 which is probably best from usability pov. 15:24:06 i didn't get a chance to see what work is actually required to do that. 15:24:36 or if it's possible... cdent mentioned it could be an endpoint change to aodh and fall back to ceilometer if not? 15:25:20 but i think for aodhclient specifically. that might be something for next cycle? unless we can't integrate it in to ceilometerclient for some reason. 15:26:21 i really don't have a break down what work items are required to be honest so anyone with better view can comment. 15:27:47 i'll check with liusheng to see what he discovers when he starts working on it i guess. 15:27:57 sounds like a good plan 15:28:04 we try to predict we'll just be wrong 15:28:54 cool cool. any other items regarding roadmap? feel free to reassign priority if you think somehting should be bumped up (and there's a resource we can attach to it) 15:30:50 #topic recurring: ceilometerclient release? 15:31:11 i'm tracking nothing. if someone needs a new client shout now 15:31:47 i should mention the release would probably be next week since release team isn't fond of thurs/fri releases 15:31:58 #topic recurring: Gnocchi status 15:32:13 awesome devstack+gnocchi pile on this week 15:32:16 lots of issues discovered 15:32:20 some fixed 15:32:22 still some to go 15:32:26 it's almost but not quite fun 15:32:31 lol 15:32:56 the latest concerning thing is a bug in carbonara: https://bugs.launchpad.net/gnocchi/+bug/1475329 15:32:56 Launchpad bug 1475329 in Gnocchi "Exception in gnocchi-metricd: ValueError: cannot reindex from a duplicate axis" [Undecided,New] 15:33:03 i told product to start looking at it. i'll check next week to see if they've disocvered anything 15:33:14 which came up after running with the file backend for a short while 15:33:30 that looks scary. i don't think i saw that. 15:33:38 I have proposed to move the samples2resources/metrics convertion to yaml like event and notification: https://review.openstack.org/#/c/202500/ 15:33:40 there have been a lot of changes to the devstack plugin in the last 24 hours so if anyone is testing with devstack, probably worth updating regularly 15:33:58 202500 is awesome 15:34:33 sileht: agreed. the hardcoded resources were really really bothering me 15:34:42 And I have reduced to only 1 resource patch per message received too: https://review.openstack.org/#/c/202589/ 15:35:07 next perf improvement in the ceilometer-notifier to batch samples into one messages 15:35:16 sileht: awesome news. good to see we're targeting performance rather than features. 15:35:33 death to features! 15:35:40 are we done with features in gnocchi? aside from influxdb driver 15:35:43 I want to see everything work in gate ! 15:35:45 sileht: batching, coolness :) 15:36:25 i think we should freeze features in gnocchi. assuming there isn't a massive gap somewhere. 15:36:28 I'm not sure batching is the priority, perhaps I will take that when more gate/functionnal testing have been done 15:36:58 sileht: yeah gating probably takes priority... then performance 15:37:23 +1 to that 15:38:48 i'll sync with jd__ to confirm there's no massive gaps in gnocchi and if not we can do feature freeze. 15:38:52 for gnocchi. 15:40:11 for reviewers, we should probably start focusing on the same. i think the only non-performance related feature we have outstanding is alarm work but that's in aodh. 15:40:35 any other comments on gnocchi? 15:41:15 #topic Open discussion 15:41:39 does anyone have oslo.messaging 1.16.x + devstack installed? 15:42:24 just for reference, https://bugs.launchpad.net/ceilometer/+bug/1475307 15:42:24 Launchpad bug 1475307 in Ceilometer "collector has possible memory leak" [Critical,New] 15:42:28 Have a question for ceilometer team. I am core commiter on Openstack searchlight project and we consume notifications from glance but when ceilometer is running, searchlight listener doesn't recieve those notifications. 15:42:41 We have to stop ceilometer for searchlight to receive new notifications. Searchlight project has a patchset to include a pool name: https://review.openstack.org/#/c/202392/ 15:42:46 Can ceilometer do similarly or any other suggestions? 15:42:58 lakshmiS: you need to configure glance to emit on two topics IIRC 15:43:05 lakshmiS: ^ 15:43:10 eglynn: that seems unfair 15:43:22 cdent: life is unfair :) 15:43:26 can we not use different pool names: atleast that's what the oslo messaging document says 15:43:28 you're basically putting a dependency on a source for their to be multiple listeners 15:43:34 s/their/there/ 15:43:53 whereas what we'd like in a distributed system is for there to be N potential listeners without needing to change the source 15:43:55 lakshmiS: there was an issue with that oslo.messaging 15:43:58 * cdent looks at sileht 15:44:10 cdent: +1 15:44:12 that should just work 15:44:16 cdent: yeap, it's certainly not pretty, I just think that's the current reality 15:44:25 * cdent does not aspire to reality ;) 15:44:34 sileht: a-ha, k ... good to know 15:44:44 I tried changing ceilometer listener code to have a pool name but it stops working afer a while 15:44:56 is there anyone from ceilometer team I can discuss this offline? 15:44:58 sileht: wasn't there an issue with pools, something about it not being created at right time so we'd lose messages? 15:45:33 kmartin,you need to take a look to your rabbitmq queues and find those prefixed with searchlight-listener 15:45:39 lakshmiS: we're all over at #openstack-ceilometer (more euro based) but there's a few of us (me) in americas tz 15:46:28 gordc, yes but you just lost message between the client side restart and the first server side start, that doesn't occurs offen 15:46:31 gordc: sileht: can you point me to the issue you had previously with oslo.messaging pools? 15:46:54 lakshmiS, it not a issue, this is a historical behavior :p 15:46:59 ok 15:47:10 unsure it's cleanly documented 15:47:27 so the solution/alternative is to have multiple topics for notification.info from glance? 15:47:36 lakshmiS: post your issue over at #openstack-ceilometer and someone will reply. if not, we're sleeping and ML is good. 15:47:59 gordc: i wil do ML since I already did on IRC with no response 15:48:06 lakshmiS: that is the alternative yes... that's typically what is done. (ie. stacktach) 15:48:16 gordc: thx 15:49:58 sileht: for the above, do ceilometer and searchlight need to create pool or just one? the problem right is we're both using hte same default pool right? 15:50:32 gordc, I think the safer ways is to emit the notification twice on the other side 15:50:55 sileht: that would be a change on each team(glance, nova) etc right? 15:51:08 lakshmiS, yes a configuration change 15:51:23 gordc, this method ensure that no notifications will be missed 15:51:39 sileht: got it. that's what i understood. 15:51:54 sileht: do you have any previous reviews doint this cofig change ? 15:52:00 s/doint/doing 15:52:10 lakshmiS: its either or i guess. pro/con either way. 15:52:26 lakshmiS: let me dig up stacktach docs, they might reference it 15:52:34 thanks gordc: 15:53:30 anything other topics? 15:53:52 lakshmiS: i'll send you link over in openstack-ceilometer unless you have another preference 15:54:34 30s countdown 15:55:11 gordc: please send it to lakshmi.sampath@hp.com if possbile too 15:55:16 lakshmiS: kk 15:55:25 ending meeting. thanks folks 15:55:28 #endmeeting