15:02:50 <dougsz> #startmeeting monasca
15:02:51 <openstack> Meeting started Wed Oct  3 15:02:50 2018 UTC and is due to finish in 60 minutes.  The chair is dougsz. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:02:52 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
15:02:54 <openstack> The meeting name has been set to 'monasca'
15:03:05 <dougsz> Just me and you today  joadavis?
15:03:21 <joadavis> Possibly. :)
15:03:40 <dougsz> Just as well, don't have much time today unfortunately :(
15:03:54 <dougsz> #topic Bug Board
15:04:01 <dougsz> https://storyboard.openstack.org/#!/board/114
15:04:20 <dougsz> Witek has made the Bugs board
15:04:49 <joadavis> yeah, we triaged some at the PTG, good to see organized
15:05:08 <dougsz> Nice :) Looks like we don't have any new bugs in there.
15:05:18 <dougsz> Although I see it needs to be manually updated.
15:05:18 <joadavis> nope.
15:05:34 <joadavis> Well, the one at the bottom of Triaged is new this week
15:05:39 <joadavis> The OOM bug
15:05:56 <dougsz> Ah yes, just had a quick look at that
15:06:07 <joadavis> I proposed one approach (turn all devstack logging down to INFO), but maybe not the preferred approach
15:06:24 <joadavis> There is another patch from Witek that sets up the oslo logging in a better way
15:07:11 <joadavis> I don't think the kafka debug logging is useful, and it is very noisy. but if just changing the log format is enough we can leave it alone
15:08:09 <dougsz> Yeah - that's been my experience too. I don't think I've ever needed to drop to debug level for Kafka
15:09:14 <joadavis> for now, the blacklisting of the log took care of the immediate problem
15:09:35 <dougsz> Yeah, I guess we'll need to undo that once we move forwards.
15:10:17 <joadavis> If you have a preference, please comment on the bug or in Witek's review
15:11:34 <dougsz> ok, will do
15:11:47 <dougsz> I wonder if we should use both
15:12:39 <dougsz> I need to verify the change actually reduced the log file size, hopefully will have some time next week.
15:12:51 <joadavis> My commit was a little heavy handed - I tried to set the default for everything to INFO.  But it was just to get a conversation.  If we can turn just kafka to INFO I think we should in addition to the format change
15:13:24 <dougsz> Yeah, that sounds sensible to me.
15:13:43 <joadavis> I need to try Witek's change - maybe rebuild my devstack with it
15:13:53 <dougsz> Likewise
15:14:33 <joadavis> Any other bugs we should discuss?  I know that the monasca-agent-ip one is being worked by one of my colleagues, Ritesh
15:14:59 <joadavis> I should move one of the two related bugs there to the in progress collumn
15:15:36 <dougsz> Yeah - I filed a trivial one, will add it to the board now.
15:15:37 <dougsz> https://storyboard.openstack.org/#!/story/2003931
15:17:13 <joadavis> huh, haven't seen that. But I don't use the InfluxDB version much
15:17:46 <dougsz> Yeah - I think it's a corner case. I noticed it when InfluxDB wasn't up.
15:18:06 <dougsz> It's a low priority as it doesn't actually break anything.
15:18:19 <joadavis> ah. yes, in general I think there are still a few error cases where we can improve handling
15:18:53 <dougsz> Yeah, just need more hours in the day!
15:19:05 <dougsz> Any other bugs?
15:19:10 <joadavis> I've been spending some time on cases where we get metrics with value: None sent by the monasca-agent. It has taken far to long to track them all down. :P
15:19:31 <joadavis> other than that, nothing else from me
15:19:39 <dougsz> sounds painful!
15:20:14 <dougsz> Ok, cool
15:20:24 <dougsz> #topic Spike at Neutron API hits by monasca agent
15:20:29 <joadavis> One case I think was from the hpssacli library changing name to ssacli. The stuff remaining seems to be a timeout on the http check on keystone, which baffles me.
15:20:40 <joadavis> Yeah, do you know about this Spike item?
15:20:57 <pandy_> hi
15:20:59 <dougsz> I think this is pandy's one
15:21:06 <joadavis> ah
15:21:38 <dougsz> pandy_: Is this following on from last week?
15:21:48 <pandy_> I have asked this question last meeting also, Monasca-agent plugin (libvirt) hittin neutron api very frequently
15:21:51 <pandy_> Yes
15:22:37 <joadavis> are all the hits to libvirt coming from one monasca-agent instance?
15:22:55 <dougsz> I think pandy has 500 hypervisors running it?
15:23:18 <pandy_> in our contrail node, checked loads too much spike
15:23:36 <pandy_> created by monasca agent which is interval of one minute
15:23:47 <pandy_> yes, we have 500 hypervisors
15:23:58 <pandy_> we are running on 500 hypervisors
15:24:38 <dougsz> I'm not sure what sort of scale people are running Monasca at, but that's pretty big.
15:24:48 <pandy_> here my agent.yaml file http://paste.openstack.org/show/731389/ libvirt.yaml http://paste.openstack.org/show/731390/
15:26:33 <pandy_> Let me know what are the parameters in libvirt is hitting neutron API & also getting empty value to port_cache in instance_cache  http://paste.openstack.org/show/731390/
15:27:17 <dougsz> Sorry pandy_ I don't use that plugin yet.
15:27:20 <pandy_> Even sometimes, kafka fails to listen leaders when doing scaling in large setup
15:27:53 <dougsz> I'm sorry, I have to drop from the meeting now.
15:28:08 <pandy_> okay dougx, let me know who can help out
15:29:10 <dougsz> I think you are probably going to need to submit a patch - I haven't heard or anyone else hitting this issue.
15:29:17 <dougsz> s/or/of
15:29:32 <dougsz> All you can do is ask in monasca IRC
15:29:40 <pandy_> sure, dougz
15:29:46 <dougsz> Please move discussion to there.
15:29:51 <joadavis> Unfortunately I'm not much help there either.
15:29:52 <pandy_> okay
15:29:56 <pandy_> :)
15:30:16 <dougsz> ok, thanks all and enjoy the rest of the week
15:30:18 <dougsz> #endmeeting