15:01:24 <eglynn> #startmeeting ceilometer
15:01:24 <openstack> Meeting started Thu Oct 30 15:01:24 2014 UTC and is due to finish in 60 minutes.  The chair is eglynn. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:01:25 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
15:01:27 <openstack> The meeting name has been set to 'ceilometer'
15:01:32 <fabiog> o/
15:01:37 <DinaBelova> \o
15:01:56 <eglynn> last outing before summit :)
15:01:56 <llu-laptop> o/
15:01:58 <gordc> o/
15:02:01 <nealph> o/
15:02:05 <ityaptin> Hi!
15:02:33 <eglynn> #topic gnocchi/TSDaaS status
15:03:07 <eglynn> DinaBelova, ityaptin: do you guys want to give the latest on the in-process dispatcher PoC?
15:03:18 <eglynn> ... i.e. the threading issues encountered
15:04:28 <DinaBelova> eglynn, yeah, for sure
15:04:49 <_elena_> hi)
15:04:57 <DinaBelova> so ityaptin and I are working on some draft POC of how ceilo + gnocchi via dispatcher without gnocchi api might work
15:05:12 <DinaBelova> so basically all POCs are loaded to the review
15:05:17 <DinaBelova> ityaptin - please publish links
15:05:34 <eglynn> #link https://review.openstack.org/#/c/131482/
15:05:40 <DinaBelova> eglynn, thanks!
15:06:02 <DinaBelova> so we're facing issues with direct dispatcher + TS storage
15:06:18 <DinaBelova> something goes wrong with the threading.local thing....
15:06:22 <DinaBelova> let me find a link
15:06:42 <DinaBelova> #link http://paste.openstack.org/show/126961/
15:06:46 <eglynn> so we use the threading.local for the context object, or?
15:07:08 <DinaBelova> eglynn, it's used in the one lib (numexpr) - that's pandas dependency actually
15:07:31 <DinaBelova> and it seems like the only place we might grab issues from is this pandas -> numexpr chain...
15:07:54 <DinaBelova> the only thing that makes me kind of suspecious - were do we grab this thing from while OpenTSDB usage?
15:08:01 <DinaBelova> \as basically we don
15:08:11 <ityaptin> opentsdb and swift dispatcher with/without using api: https://www.dropbox.com/s/3lpmh8wegl3vcyk/opentsdb-without-api.html?dl=0
15:08:13 <DinaBelova> don't seem to use pandas or whatever here
15:08:40 <eglynn> DinaBelova: is that because of the way the storage driver is being loaded *directly* in the dispatcher?
15:08:49 <DinaBelova> eglynn, results ityaptin has mentioned - are without real ceilometer usage, just syntethic dispatcher test
15:09:01 <eglynn> DinaBelova: ... and also being initialized in the usual way driven by config
15:09:10 <DinaBelova> eglynn, nope, with the stevedore...
15:09:15 <DinaBelova> direct load makes no issues
15:09:38 <eglynn> DinaBelova: for context, see my comment on https://review.openstack.org/#/c/131482/3/gnocchi/ceilometer/dispatcher.py
15:10:11 <DinaBelova> eglynn, one moment, yeah
15:10:39 <DinaBelova> eglynn, yeah, I got your point
15:10:52 <DinaBelova> as said, we've tryied get_driver thing on the lab..
15:11:00 <DinaBelova> and there was the issue I described...
15:11:13 <DinaBelova> direct thing you see in the review is working actially...
15:11:25 <eglynn> DinaBelova: could a second get_driver call also be occuring from here https://github.com/stackforge/gnocchi/blob/master/gnocchi/rest/app.py#L85
15:11:41 <DinaBelova> eglynn, we've removed importing rest locally as well....
15:11:42 <eglynn> DinaBelova: ... which causes the *default* swift driver to be loaded as well?
15:12:08 <DinaBelova> eglynn, hm, do you mean while the app is starting?
15:12:19 <DinaBelova> hm, yeah, interesting, point, let us check it!
15:12:24 <DinaBelova> eglynn, thank you sir!
15:12:41 <eglynn> cool, thanks!
15:13:07 <DinaBelova> eglynn, we'll emulate 'normal' working process with some hacks (these direct usage, etc) - to grab the performance results for the ceilo + gnocchi + opentsdb
15:13:25 <DinaBelova> eglynn, so we're supposing with collector's threading, etc. results will be nice
15:13:25 <eglynn> DinaBelova: so apart from this issue with threading, the performance of the direct approach is looking reasonable?
15:13:37 <DinaBelova> eglynn, yes, for sure
15:13:50 <DinaBelova> eglynn, Pecan's API is slowing this this to-o much
15:14:08 <DinaBelova> you may take a look on the results ityaptin has published earlier here
15:14:16 <eglynn> DinaBelova: reasonable for both carbonara/swift and opentsdb, or just tested for the latter?
15:14:37 <DinaBelova> eglynn, take a look on https://www.dropbox.com/s/3lpmh8wegl3vcyk/opentsdb-without-api.html?dl=0
15:14:46 <DinaBelova> there are results for the swift + carbonara there as well
15:15:07 <DinaBelova> and non-api makes things ~ x12 times faster for any backend thing
15:15:20 <DinaBelova> although for the swift + carbonara there is impl issue, I guess
15:15:36 <DinaBelova> with the measurements writing to the smae entity
15:16:06 <DinaBelova> as you see this /|/ pattern - | is actually when new resource + new entity was created
15:16:45 * eglynn wonders what the x-axis represents in those graphs ... labelled as "index"
15:17:11 <ityaptin> eglynn: Number of batch that is sent
15:17:42 <DinaBelova> ityaptin, as usually, you're having nice namings :D
15:18:16 <fabiog> ityaptin: these batches are sent sequentially or in parallel?
15:18:38 <ityaptin> fabiog: Sequentially, in one thread
15:18:48 <eglynn> ityaptin: so is this batching realistic? ... in the sense that we rarely see a so many samples for the same meter+resource all together?
15:19:23 <fabiog> ityaptin: are you running Pecan as separate app or using Apache?
15:19:32 <DinaBelova> eglynn, this was just to emulate kind of load, to have realistic results we're trying to have ceilo + gnocchi + no - api actually working
15:19:35 <eglynn> ityaptin: by "batch" do you mean a bunch of measures sent to the same entity in a single call?
15:19:41 <DinaBelova> eglynn, and facing some problems :)
15:20:22 <ityaptin> eglynn: yeah, it's 100 samples with same resource_id and counter_name
15:20:56 <eglynn> ityaptin: writen in a sequence of calls, or in a single call to the storage object?
15:21:33 <ityaptin> fabiog: When we tested gnocchi with api, we starts gnocchi-api cli script. i think it's run pecan how separate app
15:22:40 <fabiog> ityaptin: right, running it under Apache will improve perf significantly especially if u run several processes
15:22:58 <ityaptin> eglynn: It is one call of dispatcher.record_metering_data. But in dispatcher we call storage, indexer or api more or equal than one time.
15:23:13 <fabiog> ityaptin: nevertheless, the difference is too big 4s vs 0,7s for 100 samples
15:23:39 <fabiog> eglynn: I am wondering if we get rid of Pecan in Ceilometer we may fly :-)
15:24:06 <DinaBelova> fabiog, hehe
15:24:19 <gordc> fabiog: just curious. but with what?
15:24:26 <eglynn> fabiog: well I guess in general we need some API layer, i.e. this in-process approach is a special case for the collector
15:24:58 <eglynn> ityaptin: so storage.add_measures() is called with with a *single* measure each time?
15:25:09 <eglynn> (... as opposed to multiple measures in a single call to add_measures?)
15:25:10 <ityaptin> fabiog: In dispatcher with api, we send 1-3 api post request for every sample. May it affect so strong to processing time?
15:25:33 <fabiog> gordc: Keystone is still usign webOb and it is working well, also you can define routers and controllers in the way you want
15:25:43 <eglynn> ... so basically I'm wondering if we're really batching here at all?
15:26:27 <eglynn> perhaps we should take this discussion offline and include jd__ also?
15:26:32 <gordc> fabiog: i see... yeah i'm just interest because not really a huge fan of our api implmentation (and i've never looked at improving it)
15:26:53 <fabiog> ityaptin: Pecan is basically a single process with a round-robin like threading
15:27:32 <fabiog> ityaptin: so if you have 5 requests in parallel are acted upon in sequence
15:27:44 <DinaBelova> eglynn, well, yes probably + we'll have summit Ceilometer+Gnocchi integraitons session to finalize moments like this :)
15:27:47 <fabiog> ityaptin: so if one takes a long time it will impact all the others
15:28:05 <ityaptin> eglynn: How I'm understand - no. Maybe I made a mistake earlier.  We group samples by resource_id in dispatcher and work with it.
15:28:32 <ityaptin> fabiog: In this phrase I make a mistake "we send 1-3 api post request for every sample".
15:28:54 <eglynn> OK, I think we need to separate two orthongonal issues here ... (a) pecan sucking and (b) the problems seen with direct dispatch to gnocchi PoC
15:29:06 <eglynn> (a) is definitely a good topic for longer term discussion
15:29:08 <ityaptin> we group samples by resource_id in dispatcher. And send all measures with current res_id
15:29:18 <eglynn> (b) we need to solve for next week if poss
15:29:31 <DinaBelova> eglynn, yes, indeed
15:30:15 <fabiog> ityaptin: so you are batching measures that have the same resource_id, in batches of 100 each, is that correct? It sounds reasonable to me
15:31:06 <eglynn> ityaptin, fabiog: the way the gnocchi API is consructed, IIUC only writes to the same entity (not just resource) can be batched
15:31:20 <DinaBelova> eglynn, yes, that's true
15:31:47 <eglynn> (so we need a match on *both* resource_id and meter_name for multiple samples received at about the same time)
15:31:57 <ityaptin> fabiog: yes, we send batch with 100 samples.
15:31:57 <eglynn> this rarely happens in practice
15:32:24 <DinaBelova> eglynn, yes, usually we're having probably one meter_name, but resource IDs different
15:32:41 <ityaptin> eglynn, fabiog: I'll run this test with different resources in batch
15:32:59 <DinaBelova> ityaptin, well, gnocchi API won't batch them actually I suppose
15:33:00 <eglynn> the only time I can think of in practice is when we get a tight seuqnce of compute.instance.start/update notifications during an instance boot
15:33:08 <DinaBelova> ityaptin, as it's not designed for it
15:33:17 <eglynn> for polled meters, such bunching would never happen
15:33:24 <DinaBelova> eglynn, about ~8 notificaitons, actually
15:33:37 <eglynn> DinaBelova: but relatively rare
15:33:45 <DinaBelova> eglynn, yes, for sure
15:33:47 <ityaptin> DinaBelova: dispatcher is groups samples by resource_id and process it.
15:34:11 <DinaBelova> ityaptin, a-ha, ok
15:34:31 <eglynn> ityaptin: yep, but I think it's being optimistic there in that logic
15:34:36 <eglynn> sileht can confirm
15:35:04 <eglynn> but IIUC that group_by logic would rarely yield more than one sample of the same resource_id and same meter_name
15:35:43 <DinaBelova> eglynn, at least in real life - for sure
15:35:52 <DinaBelova> with all these different samples appearing
15:36:38 <eglynn> DinaBelova: yep, exactly ... and it's the real life case we need to optimize the profiling for
15:37:10 <eglynn> anyhoo, let's take the discussion off-line, sounds like we need some input from jd__ & sileht
15:37:16 <DinaBelova> eglynn, ++
15:37:36 <eglynn> #topic kilo summit planning
15:38:00 <eglynn> fabiog asked for a list of contributor meetup topics last time
15:38:06 <eglynn> #link http://bit.ly/kilo-ceilometer-contributor-meetup
15:38:30 <fabiog> eglynn: Thanks!
15:38:57 <eglynn> that's pretty much the left-over topics from the scheduled design track ... if anyone want to add further topics, please feel free
15:39:11 <llu-laptop> eglynn: could you give the real URL? bit.ly is blocked in china
15:39:25 <eglynn> llu-laptop: https://docs.google.com/spreadsheets/d/1DbL0ZR9yvgOIIgegMN65SesKtR2_BD4tz5Kpg2S934A/edit
15:39:34 <eglynn> llu-laptop: sorry about that!
15:39:55 <llu-laptop> eglynn: thx. no, sorry for the government :(
15:40:01 <gordc> fabiog: i'm looking at something similar to notification filter topic... ie. event pipeline
15:40:18 <gordc> fabiog: will sync up with you on that.
15:40:47 <eglynn> gordc: cool, that sounds interesting!
15:40:53 <fabiog> gordc: that's exactly what I want to do, have the same concept for the event pipeline duplicated for notification pipeline
15:41:24 <gordc> eglynn: fabiog: yeah, i think there's some existing rax stacktach bps that we could restart
15:41:36 <fabiog> gordoc: let's talk. are you going to be at the summit?
15:41:48 <gordc> fabiog: yep.
15:42:01 <fabiog> eglynn: should we have a quick list of people that are sure to go among the team?
15:43:33 <eglynn> that I know of ... fabiog, jd__, sileht, DinaBelova, ildikov, gordc, llu-laptop, nsaje, eglynn
15:43:46 <eglynn> pradk
15:44:03 <DinaBelova> looks like something close to real, yeah
15:44:16 <fabiog> eglynn: good bunch ;-)
15:44:42 <eglynn> so the Friday meetup sounds like exactly the right place for such emergent discussions
15:45:07 <eglynn> since we have the whole day, without the pressure of time-boxing
15:45:47 <eglynn> one question that came up earlier in the week on http://kilodesignsummit.sched.org/overview/type/ceilometer
15:46:25 <eglynn> since the cross-project track is now including a notifications session also http://kilodesignsummit.sched.org/event/0b6e7e23da3d4fdcdc5cec0777b92c6a
15:46:39 <eglynn> i.e. whether we need a ceilo-specific one as well
15:46:58 <DinaBelova> eglynn, yeah, I've seen this as well, already mathced I'll got there...
15:47:10 <DinaBelova> probably we don't need our own session here
15:47:35 <DinaBelova> and we may give it to something else left in our gTable? some lucky looser? :)
15:47:54 <eglynn> well jd__ will be leading both sessions, he reckoned the ceilo-specific one would give us a chance to discuss in more detail
15:48:12 <DinaBelova> eglynn, well, this will work as well :)
15:48:22 <eglynn> TBH it would have been better if the corss-project track was finalized by the TC at the same time as the project tracks
15:52:08 <eglynn> anyone else got anything on summit to discuss?
15:52:54 <DinaBelova> eglynn, nope from my side
15:53:24 <eglynn> #topic Open discussion
15:54:37 <fabiog> eglynn: I am working on preparing the mid-cycle meetup, can you please tell me when historically happened in the fall cycle? Late Jan?
15:55:12 <eglynn> fabiog: great question! ... we've actually only ever one ceilo midcycle (Paris last July)
15:55:42 <DinaBelova> fabiog, hehe, great news :) where actually? :)
15:55:55 <fabiog> eglynn: HP is happy to host it this time and I even can provide a choice of venues
15:56:11 <fabiog> DinaBelova: Sunnyvale, CA or Galway, IR
15:56:22 <fabiog> Ireland
15:56:37 <eglynn> fabiog: excellent! ... I think late Jan / early Feb would be close to the midpoint
15:57:10 <fabiog> eglynn: let's finalize the dates at the summit, so I can start the organization machine ;-)
15:57:42 <eglynn> fabiog: cool, thanks!
15:57:59 <fabiog> eglynn: my pleasure
15:58:08 <eglynn> fabiog: BTW note the agenda item for the joint TC/board meeting
15:58:11 <eglynn> #link https://wiki.openstack.org/wiki/Governance/Foundation/2Nov2014JointMeeting
15:58:24 <eglynn> "mid cycle meetups - is there a way to reduce travel strain?" ... LOL :)
15:58:41 <eglynn> someone's travel budget is getting strained methinks!
15:59:32 <fabiog> eglynn: that's why I asked my management to propose an European location as well ... I guess you can drive to Galway :-)
15:59:50 <eglynn> fabiog: I can cycle to Galway if needed :)
16:00:04 <russellb> meetup in galway?  ooh, sounds fun
16:00:13 <fabiog> eglynn: there you go
16:00:21 <eglynn> russellb: yeap, /me is warming to that idea! :)
16:00:51 <fabiog> eglynn: I guess you can stop from time to time for some "dark fuel" ... Guinness
16:01:53 <eglynn> fabiog: oh yes, indeed, fueling will be required :) ... the local craft beer is good http://www.galwaybaybrewery.com/
16:02:04 <eglynn> right-o, we're outta time
16:02:11 <eglynn> thanks folks for a productive meeting!
16:02:23 <gordc> laters
16:02:33 <eglynn> ... and happy travels for those attending summit
16:02:38 <fabiog> ciao
16:02:40 <eglynn> #endmeeting ceilometer