*** bobh has joined #openstack-monasca | 01:53 | |
*** mbindlish has joined #openstack-monasca | 02:24 | |
*** bobh has quit IRC | 02:47 | |
*** toabctl has quit IRC | 04:31 | |
*** pcaruana has joined #openstack-monasca | 05:06 | |
*** joadavis has quit IRC | 05:24 | |
*** joadavis has joined #openstack-monasca | 05:25 | |
*** joadavis has quit IRC | 05:27 | |
*** witek has joined #openstack-monasca | 06:36 | |
*** chaconpiza has joined #openstack-monasca | 07:09 | |
*** toabctl has joined #openstack-monasca | 07:30 | |
*** dougsz has joined #openstack-monasca | 08:08 | |
*** trident has quit IRC | 08:08 | |
*** trident has joined #openstack-monasca | 08:09 | |
*** ivve has joined #openstack-monasca | 08:34 | |
openstackgerrit | Witold Bedyk proposed openstack/monasca-tempest-plugin master: Use unicode literals in test_metrics https://review.opendev.org/663927 | 08:43 |
---|---|---|
openstackgerrit | Witold Bedyk proposed openstack/monasca-tempest-plugin master: Use unicode literals in test_metrics https://review.opendev.org/663927 | 08:57 |
*** sum12 has joined #openstack-monasca | 09:10 | |
*** witek has left #openstack-monasca | 09:39 | |
*** witek has joined #openstack-monasca | 09:39 | |
openstackgerrit | Adrian Czarnecki proposed openstack/monasca-api master: [WIP] Merge log-api and api https://review.opendev.org/651249 | 09:44 |
*** Wasaac has joined #openstack-monasca | 09:57 | |
openstackgerrit | Witold Bedyk proposed openstack/monasca-api master: Upgrade Storm to 1.2.2 https://review.opendev.org/658371 | 11:26 |
*** Wasaac has quit IRC | 11:32 | |
*** Wasaac has joined #openstack-monasca | 11:33 | |
*** Wasaac has quit IRC | 11:38 | |
*** elod has quit IRC | 12:11 | |
*** Wasaac has joined #openstack-monasca | 12:11 | |
witek | chaconpiza: https://review.opendev.org/663927 | 12:12 |
chaconpiza | it pass all checks | 12:16 |
witek | yes, finally | 12:16 |
chaconpiza | Do you have an idea of which thing is creating the random behavior? | 12:17 |
*** Wasaac has quit IRC | 12:18 | |
witek | you mean randomly failing tempest tests? | 12:18 |
chaconpiza | yes | 12:20 |
witek | I think it's resources issue | 12:21 |
witek | the node where the tests are run has 8GB RAM | 12:22 |
witek | http://lists.openstack.org/pipermail/openstack-discuss/2019-July/007493.html | 12:23 |
*** Wasaac has joined #openstack-monasca | 12:23 | |
openstackgerrit | Witold Bedyk proposed openstack/monasca-api master: Set topology.max.spout.pending for Apache Storm https://review.opendev.org/668887 | 12:25 |
*** elod has joined #openstack-monasca | 12:48 | |
*** elod_ has joined #openstack-monasca | 12:58 | |
*** elod has quit IRC | 12:58 | |
*** elod_ is now known as elod | 12:59 | |
*** elod has quit IRC | 13:06 | |
*** elod has joined #openstack-monasca | 13:07 | |
*** dougsz has quit IRC | 13:18 | |
*** dougsz has joined #openstack-monasca | 13:18 | |
*** dougsz has quit IRC | 13:22 | |
*** Wasaac has quit IRC | 13:43 | |
*** joadavis has joined #openstack-monasca | 13:43 | |
*** Wasaac_ has joined #openstack-monasca | 13:43 | |
*** mbindlish has quit IRC | 13:44 | |
*** Wasaac_ has quit IRC | 13:58 | |
*** ivve has quit IRC | 14:14 | |
*** sgrasley has joined #openstack-monasca | 14:17 | |
*** Wasaac has joined #openstack-monasca | 14:46 | |
*** koji_n has joined #openstack-monasca | 14:53 | |
*** hosanai has joined #openstack-monasca | 14:56 | |
witek | Courtesy Monasca meeting reminder in #openstack-monasca: witek, jayahn,iurygregory,ezpz,igorn,haad,sc,joadavis, akiraY,tobiajo,dougsz,fouadben, amofakhar, haru5ny (haruki),kaiokmo,pandiyan,charana,guilhermesp,chaconpiza,toabctl,hosanai,Wasaac | 14:58 |
witek | #startmeeting monasca | 15:00 |
openstack | Meeting started Wed Jul 3 15:00:15 2019 UTC and is due to finish in 60 minutes. The chair is witek. Information about MeetBot at http://wiki.debian.org/MeetBot. | 15:00 |
openstack | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 15:00 |
*** openstack changes topic to " (Meeting topic: monasca)" | 15:00 | |
openstack | The meeting name has been set to 'monasca' | 15:00 |
witek | hello everyone | 15:00 |
Wasaac | o/ | 15:00 |
joadavis | o/ | 15:00 |
kaiokmo | \o | 15:00 |
koji_n | hi | 15:00 |
hosanai | hi | 15:01 |
witek | the agenda as usual: | 15:01 |
witek | https://etherpad.openstack.org/p/monasca-team-meeting-agenda | 15:01 |
Dobroslaw | hi | 15:01 |
witek | #topic Batching messages for Kafka | 15:01 |
*** openstack changes topic to "Batching messages for Kafka (Meeting topic: monasca)" | 15:01 | |
witek | I don't see Doug | 15:02 |
witek | he proposed this change: | 15:02 |
witek | https://review.opendev.org/667931 | 15:02 |
*** haru5ny has joined #openstack-monasca | 15:02 | |
*** dougsz has joined #openstack-monasca | 15:02 | |
witek | the purpose is split the list of messages if the batch is too big | 15:02 |
witek | hi dougsz | 15:03 |
dougsz | sorry i'm late, hi | 15:03 |
witek | I've just introduced your change https://review.opendev.org/667931 | 15:03 |
dougsz | ah, thanks | 15:03 |
witek | please take over | 15:03 |
dougsz | So we saw this problem with Monasca Agent with large buffers | 15:04 |
dougsz | When they were disconnected from the Monasca API due to a network reconfigure | 15:04 |
dougsz | They buffered up a lot of data | 15:04 |
dougsz | Then when the API was available, they made posts of large amounts of data | 15:05 |
dougsz | Monasca API tried to write this to Kafka, which failed due to the chunk being to large | 15:05 |
dougsz | There are a series of example errors in the Story | 15:05 |
*** bobh has joined #openstack-monasca | 15:05 | |
dougsz | The fix simply writes a post in chunks | 15:05 |
dougsz | The chunks consist of a fixed number of metrics | 15:06 |
dougsz | It's not trivial to calculate the size of the metrics, as would be written to Kafka | 15:06 |
dougsz | We've tested in production - we could handle 20GB metric posts with this change. | 15:06 |
dougsz | That's it really | 15:07 |
witek | have you checked performance impact? | 15:07 |
joadavis | seems like an effective approach | 15:07 |
dougsz | I havent measured it, but I think the bottle neck is still in the monasca persister | 15:08 |
dougsz | we wrote about 300GB of metrics in a couple of hours to Kafka | 15:08 |
joadavis | good question - what is the 'normal' batch size | 15:08 |
dougsz | and then the persister spent the next few days writing them to InfluxDB | 15:08 |
witek | :) | 15:08 |
joadavis | lol | 15:09 |
dougsz | joadavis: yeah - it depends on your max write size to kafka | 15:09 |
kaiokmo | :) | 15:09 |
witek | the default value of message.max.bytes is ca. 1 MB | 15:10 |
dougsz | yeah, that's correct, I think roughly the default for this change should be about 0.5MB writes | 15:10 |
dougsz | Anyway, it was good proof that the architecture works. Kafka absorbed the massive spike in metrics very easily. | 15:11 |
witek | I just noticed that the initialization of `batch_size` variable could probably be moved to a better place | 15:11 |
dougsz | ah yes, I can do that | 15:12 |
witek | ok, thanks for the update | 15:12 |
dougsz | ok, thanks, will update asap | 15:12 |
witek | folks, please leave your comments in review if you have any thoughts | 15:12 |
witek | #topic Python 3.7 | 15:13 |
*** openstack changes topic to "Python 3.7 (Meeting topic: monasca)" | 15:13 | |
witek | there has been a series of changes recently updating our CI jobs for Python 3.7 | 15:13 |
witek | most of merged well, but there are some exceptions | 15:13 |
witek | which I wanted to bring to attention | 15:14 |
witek | https://review.opendev.org/667060 | 15:14 |
witek | https://review.opendev.org/667058 | 15:14 |
witek | the last one: | 15:14 |
witek | https://review.opendev.org/#/c/657829/ | 15:14 |
witek | is general Python 3 support for monasca-agent | 15:14 |
joadavis | and I noticed in reviewing the other changes that the monasca-agent change could be using the -train job | 15:15 |
witek | yes, I haven't checked the other repos, some might need update as well | 15:16 |
witek | so if anyone looks for a task to pick up, help very welcome | 15:16 |
joadavis | I don't think all the monasca repos are on the list that the goal team is working on | 15:16 |
joadavis | so checking through them ourselves is a good idea | 15:16 |
witek | hm, do you know which list they're using? | 15:17 |
joadavis | I don't, I just looked at the list of changes with the python3-train branch | 15:18 |
joadavis | s/branch/topic/ | 15:18 |
joadavis | we could ask in openstack-discuss | 15:19 |
witek | ok, I'll try to find out if they have some etherpad for tracking this | 15:19 |
witek | #topic monasca-notification config update | 15:19 |
*** openstack changes topic to "monasca-notification config update (Meeting topic: monasca)" | 15:19 | |
witek | I've picked oslo.config for monasca-notification in DevStack again | 15:20 |
witek | and it finally seems to work | 15:20 |
joadavis | :thumbs-up: | 15:20 |
witek | we needed a bugfix from Isaac: | 15:20 |
witek | https://review.opendev.org/663960 | 15:20 |
witek | and set the default enabled plugins: | 15:21 |
witek | https://review.opendev.org/668508 | 15:21 |
witek | with them DevStack can be updated: | 15:21 |
witek | https://review.opendev.org/633194 | 15:21 |
witek | these changes were long overdue | 15:22 |
witek | any comments? | 15:23 |
witek | #topic mid-cycle meeting | 15:23 |
*** openstack changes topic to "mid-cycle meeting (Meeting topic: monasca)" | 15:23 | |
witek | I though it might be a good idea to have an audio meeting in the middle of the cycle | 15:24 |
witek | to sync up on progress | 15:24 |
witek | what do you think? | 15:25 |
witek | we could do it around milestone-2 | 15:25 |
witek | I don't expect it to be a long meeting | 15:25 |
Wasaac | No objections | 15:26 |
joadavis | I like the idea | 15:26 |
Dobroslaw | fine | 15:27 |
witek | cool, milestone-2 is in the week Jul 22 - Jul 26 | 15:27 |
witek | I'd suggest to set up a one hour slot, we could extend it to two if needed | 15:28 |
witek | do we want to use the same time as this meeting, or better make a doodle list? | 15:28 |
dougsz | +1 for audio meeting | 15:28 |
witek | I'll send the email to the mailing list and set up a doodle with time proposals | 15:30 |
witek | I think that'll be fair for everyone | 15:30 |
witek | #topic other business | 15:31 |
*** openstack changes topic to "other business (Meeting topic: monasca)" | 15:31 | |
witek | any other topics? | 15:31 |
witek | if not, we can finish earlier | 15:32 |
witek | oh, the deadline for Shanghai has been extended if anyone is interested | 15:33 |
witek | thanks for joining | 15:33 |
witek | see you next week | 15:33 |
joadavis | Thanks all | 15:33 |
koji_n | thanks | 15:34 |
hosanai | thx | 15:34 |
witek | #endmeeting | 15:34 |
*** openstack changes topic to "OpenStack Monitoring as a Service | https://wiki.openstack.org/wiki/Monasca" | 15:34 | |
openstack | Meeting ended Wed Jul 3 15:34:31 2019 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 15:34 |
openstack | Minutes: http://eavesdrop.openstack.org/meetings/monasca/2019/monasca.2019-07-03-15.00.html | 15:34 |
openstack | Minutes (text): http://eavesdrop.openstack.org/meetings/monasca/2019/monasca.2019-07-03-15.00.txt | 15:34 |
openstack | Log: http://eavesdrop.openstack.org/meetings/monasca/2019/monasca.2019-07-03-15.00.log.html | 15:34 |
*** hosanai has quit IRC | 15:34 | |
*** koji_n has quit IRC | 15:34 | |
*** haru5ny has quit IRC | 15:53 | |
*** bobh has quit IRC | 15:58 | |
*** Wasaac has quit IRC | 16:17 | |
*** Wasaac has joined #openstack-monasca | 16:21 | |
*** Wasaac has quit IRC | 16:25 | |
*** bobh has joined #openstack-monasca | 16:33 | |
*** bobh has quit IRC | 16:38 | |
openstackgerrit | Merged openstack/monasca-notification master: Fix notification type plugin install loop https://review.opendev.org/663960 | 16:50 |
*** dougsz has quit IRC | 16:58 | |
*** openstackgerrit has quit IRC | 17:04 | |
*** witek has quit IRC | 17:04 | |
*** elod has quit IRC | 17:05 | |
*** elod has joined #openstack-monasca | 17:07 | |
*** trident has quit IRC | 20:43 | |
*** trident has joined #openstack-monasca | 20:45 | |
*** pcaruana has quit IRC | 21:16 | |
*** bobh has joined #openstack-monasca | 22:31 | |
*** bobh has quit IRC | 23:32 | |
*** bobh has joined #openstack-monasca | 23:33 | |
*** bobh has quit IRC | 23:37 | |
*** bobh has joined #openstack-monasca | 23:48 | |
*** bobh has quit IRC | 23:57 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!