15:00:15 <witek> #startmeeting monasca
15:00:16 <openstack> Meeting started Wed Jul  3 15:00:15 2019 UTC and is due to finish in 60 minutes.  The chair is witek. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:00:17 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
15:00:20 <openstack> The meeting name has been set to 'monasca'
15:00:35 <witek> hello everyone
15:00:43 <Wasaac> o/
15:00:49 <joadavis> o/
15:00:49 <kaiokmo> \o
15:00:51 <koji_n> hi
15:01:03 <hosanai> hi
15:01:15 <witek> the agenda as usual:
15:01:19 <witek> https://etherpad.openstack.org/p/monasca-team-meeting-agenda
15:01:38 <Dobroslaw> hi
15:01:57 <witek> #topic Batching messages for Kafka
15:02:10 <witek> I don't see Doug
15:02:18 <witek> he proposed this change:
15:02:23 <witek> https://review.opendev.org/667931
15:02:56 <witek> the purpose is split the list of messages if the batch is too big
15:03:11 <witek> hi dougsz
15:03:17 <dougsz> sorry i'm late, hi
15:03:27 <witek> I've just introduced your change https://review.opendev.org/667931
15:03:35 <dougsz> ah, thanks
15:03:47 <witek> please take over
15:04:15 <dougsz> So we saw this problem with Monasca Agent with large buffers
15:04:32 <dougsz> When they were disconnected from the Monasca API due to a network reconfigure
15:04:45 <dougsz> They buffered up a lot of data
15:05:03 <dougsz> Then when the API was available, they made posts of large amounts of data
15:05:22 <dougsz> Monasca API tried to write this to Kafka, which failed due to the chunk being to large
15:05:34 <dougsz> There are a series of example errors in the Story
15:05:50 <dougsz> The fix simply writes a post in chunks
15:06:03 <dougsz> The chunks consist of a fixed number of metrics
15:06:23 <dougsz> It's not trivial to calculate the size of the metrics, as would be written to Kafka
15:06:47 <dougsz> We've tested in production - we could handle 20GB metric posts with this change.
15:07:01 <dougsz> That's it really
15:07:08 <witek> have you checked performance impact?
15:07:14 <joadavis> seems like an effective approach
15:08:21 <dougsz> I havent measured it, but I think the bottle neck is still in the monasca persister
15:08:30 <dougsz> we wrote about 300GB of metrics in a couple of hours to Kafka
15:08:32 <joadavis> good question - what is the 'normal' batch size
15:08:51 <dougsz> and then the persister spent the next few days writing them to InfluxDB
15:08:58 <witek> :)
15:09:30 <joadavis> lol
15:09:37 <dougsz> joadavis: yeah - it depends on your max write size to kafka
15:09:44 <kaiokmo> :)
15:10:06 <witek> the default value of message.max.bytes is ca. 1 MB
15:10:33 <dougsz> yeah, that's correct, I think roughly the default for this change should be about 0.5MB writes
15:11:36 <dougsz> Anyway, it was good proof that the architecture works. Kafka absorbed the massive spike in metrics very easily.
15:11:51 <witek> I just noticed that the initialization of `batch_size` variable could probably be moved to a better place
15:12:02 <dougsz> ah yes, I can do that
15:12:27 <witek> ok, thanks for the update
15:12:42 <dougsz> ok, thanks, will update asap
15:12:44 <witek> folks, please leave your comments in review if you have any thoughts
15:13:07 <witek> #topic Python 3.7
15:13:39 <witek> there has been a series of changes recently updating our CI jobs for Python 3.7
15:13:54 <witek> most of merged well, but there are some exceptions
15:14:04 <witek> which I wanted to bring to attention
15:14:10 <witek> https://review.opendev.org/667060
15:14:16 <witek> https://review.opendev.org/667058
15:14:38 <witek> the last one:
15:14:44 <witek> https://review.opendev.org/#/c/657829/
15:14:56 <witek> is general Python 3 support for monasca-agent
15:15:26 <joadavis> and I noticed in reviewing the other changes that the monasca-agent change could be using the -train job
15:16:02 <witek> yes, I haven't checked the other repos, some might need update as well
15:16:30 <witek> so if anyone looks for a task to pick up, help very welcome
15:16:35 <joadavis> I don't think all the monasca repos are on the list that the goal team is working on
15:16:50 <joadavis> so checking through them ourselves is a good idea
15:17:29 <witek> hm, do you know which list they're using?
15:18:07 <joadavis> I don't, I just looked at the list of changes with the python3-train branch
15:18:17 <joadavis> s/branch/topic/
15:19:04 <joadavis> we could ask in openstack-discuss
15:19:12 <witek> ok, I'll try to find out if they have some etherpad for tracking this
15:19:39 <witek> #topic monasca-notification config update
15:20:08 <witek> I've picked oslo.config for monasca-notification in DevStack again
15:20:16 <witek> and it finally seems to work
15:20:39 <joadavis> :thumbs-up:
15:20:47 <witek> we needed a bugfix from Isaac:
15:20:51 <witek> https://review.opendev.org/663960
15:21:08 <witek> and set the default enabled plugins:
15:21:16 <witek> https://review.opendev.org/668508
15:21:52 <witek> with them DevStack can be updated:
15:21:57 <witek> https://review.opendev.org/633194
15:22:32 <witek> these changes were long overdue
15:23:04 <witek> any comments?
15:23:36 <witek> #topic mid-cycle meeting
15:24:18 <witek> I though it might be a good idea to have an audio meeting in the middle of the cycle
15:24:24 <witek> to sync up on progress
15:25:02 <witek> what do you think?
15:25:13 <witek> we could do it around milestone-2
15:25:29 <witek> I don't expect it to be a long meeting
15:26:45 <Wasaac> No objections
15:26:57 <joadavis> I like the idea
15:27:14 <Dobroslaw> fine
15:27:44 <witek> cool, milestone-2 is in the week Jul 22 - Jul 26
15:28:30 <witek> I'd suggest to set up a one hour slot, we could extend it to two if needed
15:28:52 <witek> do we want to use the same time as this meeting, or better make a doodle list?
15:28:57 <dougsz> +1 for audio meeting
15:30:20 <witek> I'll send the email to the mailing list and set up a doodle with time proposals
15:30:37 <witek> I think that'll be fair for everyone
15:31:31 <witek> #topic other business
15:31:48 <witek> any other topics?
15:32:39 <witek> if not, we can finish earlier
15:33:27 <witek> oh, the deadline for Shanghai has been extended if anyone is interested
15:33:45 <witek> thanks for joining
15:33:49 <witek> see you next week
15:33:50 <joadavis> Thanks all
15:34:05 <koji_n> thanks
15:34:22 <hosanai> thx
15:34:31 <witek> #endmeeting