| *** tridde has joined #openstack-monasca | 02:45 | |
| *** trident has quit IRC | 02:46 | |
| *** pcaruana has joined #openstack-monasca | 07:39 | |
| *** sum12 has quit IRC | 08:11 | |
| *** witek has joined #openstack-monasca | 08:13 | |
| *** spsurya has joined #openstack-monasca | 09:25 | |
| *** dougsz has joined #openstack-monasca | 09:30 | |
| *** sum12 has joined #openstack-monasca | 09:45 | |
| witek | adriancz, Dobroslaw_away: could I please get reviews on https://review.opendev.org/#/q/topic:story_2006059+status:open ? | 10:08 |
|---|---|---|
| adriancz | Done : ) | 10:10 |
| witek | merci | 10:13 |
| openstackgerrit | Merged openstack/monasca-common master: Allow passing config options to Kafka producer https://review.opendev.org/694738 | 10:48 |
| openstackgerrit | Merged openstack/monasca-api master: Correct log tempest name https://review.opendev.org/698024 | 12:30 |
| *** spsurya has quit IRC | 12:35 | |
| openstackgerrit | Witold Bedyk proposed openstack/monasca-api master: [ussuri][goal] Drop python 2.7 support and testing https://review.opendev.org/694586 | 12:51 |
| openstackgerrit | Merged openstack/monasca-api master: Set maximum buffer size for Kafka producer https://review.opendev.org/694740 | 12:58 |
| openstackgerrit | Witold Bedyk proposed openstack/monasca-api master: [ussuri][goal] Drop python 2.7 support and testing https://review.opendev.org/694586 | 12:59 |
| witek | Courtesy Monasca meeting reminder in #openstack-monasca: witek, jayahn,iurygregory,ezpz,igorn,haad,sc,joadavis, akiraY,tobiajo,dougsz,fouadben, amofakhar, haru5ny (haruki),kaiokmo,pandiyan,guilhermesp,chaconpiza,hosanai,Wasaac,brtknr | 12:59 |
| witek | #startmeeting monasca | 13:00 |
| openstack | Meeting started Tue Dec 10 13:00:49 2019 UTC and is due to finish in 60 minutes. The chair is witek. Information about MeetBot at http://wiki.debian.org/MeetBot. | 13:00 |
| openstack | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 13:00 |
| *** openstack changes topic to " (Meeting topic: monasca)" | 13:00 | |
| openstack | The meeting name has been set to 'monasca' | 13:00 |
| witek | hello everyone | 13:01 |
| witek | anyone around? | 13:05 |
| chaconpiza | Hi, sorry for coming late. | 13:08 |
| witek | hi Martin | 13:09 |
| chaconpiza | Dobek took a day-off | 13:09 |
| witek | let's start then | 13:09 |
| chaconpiza | sure | 13:09 |
| witek | agenda: | 13:09 |
| witek | https://etherpad.openstack.org/p/monasca-team-meeting-agenda | 13:09 |
| witek | #topic Promethues plugin update | 13:10 |
| *** openstack changes topic to "Promethues plugin update (Meeting topic: monasca)" | 13:10 | |
| witek | https://github.com/stackhpc/stackhpc-monasca-agent-plugins | 13:10 |
| witek | dougsz has published his extension for Prometheus plugin | 13:11 |
| witek | I went through the readme today | 13:11 |
| dougsz | hey, sorry i'm late | 13:11 |
| witek | hi Doug | 13:11 |
| witek | nice work | 13:11 |
| dougsz | ah, thanks, just a test really at the moment, but it seems feasible | 13:12 |
| dougsz | the main motivation was to use the Ceph Prometheus endpoint to replace the Monasca Agent Ceph collector | 13:12 |
| dougsz | but should hopefully work for everything | 13:13 |
| dougsz | I was wondering about the best approach to merge it into Monasca Agent | 13:13 |
| witek | right, Ceph has added Prometheus instrumentation | 13:13 |
| dougsz | The current Prometheus Monasca Agent plugin is heavily geared to k8s | 13:14 |
| dougsz | which is probably not what a lot of people want | 13:14 |
| witek | but it also allows static configuration, right? | 13:14 |
| dougsz | yeah, in a very basic sense | 13:14 |
| dougsz | I almost wonder if it should be the prometheus-k8s plugin | 13:15 |
| dougsz | and then have a vanilla prometheus plugin like the prototype | 13:15 |
| dougsz | Alternative is to have all functionality in one plugin | 13:15 |
| witek | is there that many conflicting points? | 13:16 |
| dougsz | There is some whitelisting capability but only for the k8s thing | 13:16 |
| dougsz | which would need unifying | 13:16 |
| witek | I see | 13:17 |
| dougsz | `monasca.io/whitelist` | 13:17 |
| dougsz | slightly weird naming | 13:17 |
| dougsz | and the metrics types thing, is for k8s | 13:19 |
| dougsz | but could be useful for general endpoints | 13:19 |
| witek | I thought, `auto_detect_endpoints` could control if we use static endpoints configuration or K8s auto detection | 13:19 |
| dougsz | yeah - I mean it works | 13:19 |
| dougsz | but it seems like a chunk of the k8s specific config needs to be pulled down and made available to the non-k8s bit | 13:20 |
| dougsz | and you have to worry about backwards compatible option naming etc. | 13:20 |
| dougsz | hence my thoughts about making a new plugin | 13:21 |
| witek | so you'd prefer to rename the existing one to prometheus-k8s, and publish the prototype based as prometheus? | 13:21 |
| dougsz | something like that | 13:22 |
| dougsz | probably best not to rename the existing one though for backwards compatibility | 13:22 |
| witek | fine for me, if that's much easier to implement | 13:23 |
| dougsz | thanks - it's much easier for me too, given limited time | 13:23 |
| witek | the plugin name is not that important after all, we could document both options in the same section and point to two different configuration files | 13:25 |
| dougsz | that's true - I can push something to that effect | 13:26 |
| witek | chaconpiza: any thoughts? | 13:26 |
| chaconpiza | to be in the context... the goal of this new plugin is to handle the new version of Ceph? | 13:27 |
| witek | new Ceph versions offer Prometheus instrumentation, so yes, we can monitor Ceph with this new Prometheus plugin | 13:28 |
| witek | but also any other Prometheus endpoints | 13:28 |
| dougsz | chaconpiza: Ceph was the motivation (because the existing plugin parses the Ceph CLI which changes from release to release) | 13:29 |
| chaconpiza | yes, I remember. | 13:29 |
| dougsz | but yeah, as witek says, to make it easier to use Prometheus endpoints in general without running another Monasca service to compute rates etc. | 13:29 |
| chaconpiza | then it sound good like a long term solution | 13:30 |
| witek | I'm a little worried about performance with larger setups, where we'd like to scrape several endpoints and define multiple aggregations | 13:31 |
| witek | in the long term aggregation on Monasca server would be better | 13:31 |
| witek | but that's more work, so I'm happy with this plugin | 13:32 |
| dougsz | One advantage of this is that the whitelist can greatly reduce the amount of data going into Monasca | 13:33 |
| witek | right, that's very useful | 13:33 |
| dougsz | Many Prometheus endpoints produce vast amounts of data at scale (eg. Ceph, 1M scrapes for 10 node cluster) | 13:33 |
| dougsz | But I agree, people may want to the agent to be lightweight and shift the compute burden to the central Monasca deployment | 13:34 |
| dougsz | *want the | 13:34 |
| witek | do you remember this component? | 13:34 |
| witek | https://github.com/monasca/monasca-aggregator | 13:34 |
| dougsz | yeah | 13:35 |
| witek | we could implement it scalable with Faust | 13:35 |
| dougsz | That's a good idea | 13:36 |
| witek | so many nice things we can do :) | 13:36 |
| witek | I was reading through the doc of your Prom plugin | 13:37 |
| witek | the `counter` section is somewhat missleading | 13:37 |
| dougsz | any feedback much appreciated :) | 13:38 |
| witek | will you be proposing it upstream in the next time? | 13:38 |
| dougsz | yeah, I will do that, probably the best place to dicuss | 13:39 |
| witek | very nice, thanks a lot! | 13:39 |
| witek | can we move on? | 13:39 |
| dougsz | please do, thanks | 13:39 |
| witek | #topic review | 13:40 |
| *** openstack changes topic to "review (Meeting topic: monasca)" | 13:40 | |
| witek | we've made some progress on reviews this week | 13:40 |
| witek | the merging of DevStack plugin landed | 13:40 |
| witek | also updating ELK change has been updated | 13:41 |
| witek | here our board: | 13:41 |
| witek | https://storyboard.openstack.org/#!/board/190 | 13:41 |
| witek | I started looking at periodic notifications | 13:42 |
| witek | https://storyboard.openstack.org/#!/story/2006837 | 13:42 |
| witek | the changes have been up for much too long already | 13:42 |
| witek | other one needing attention is IPv6 support: | 13:43 |
| witek | https://review.opendev.org/673274 | 13:43 |
| witek | Adrian has submitted new change deleting the old plugin from monasca-log-api: | 13:43 |
| witek | https://review.opendev.org/690527 | 13:43 |
| witek | do you have some more reviews you'd like to mention? | 13:44 |
| dougsz | None from me | 13:45 |
| witek | #topic new bugs | 13:45 |
| *** openstack changes topic to "new bugs (Meeting topic: monasca)" | 13:45 | |
| witek | we have one new bug report this week | 13:45 |
| witek | https://storyboard.openstack.org/#!/story/2006984 | 13:45 |
| witek | it's about upgrading the DB schema | 13:45 |
| dougsz | yes, that's mine, I will push a fix soon | 13:46 |
| witek | nice, thank dougsz | 13:46 |
| dougsz | I should have spotted that really - I think the alembic step just needs to query existing plugins which are configured and skip deleting the associated types | 13:46 |
| dougsz | Worth knowing about if anyone is upgrading anytime soon | 13:47 |
| witek | that's from Queens to any other version? | 13:48 |
| dougsz | yeah, we did Queens -> Rocky -> Stein and hit it then | 13:48 |
| witek | do you know at which step? | 13:48 |
| witek | I mean -> R or -> S | 13:49 |
| dougsz | I *think* it is in Rocky where we got rid of built in notification types | 13:49 |
| dougsz | No stein actually | 13:50 |
| witek | ok, thanks, we're upgrading to Rocky but I think we didn't hit it | 13:51 |
| witek | #topic AOB | 13:52 |
| *** openstack changes topic to "AOB (Meeting topic: monasca)" | 13:52 | |
| chaconpiza | We had in a pre-production-env suddenly a big increase of memory consuption from Influxdb 1.3.4 | 13:52 |
| chaconpiza | We noticed that the monasca-metric-agent was wrongly configured in the url of Nova for the metric "http_status". | 13:52 |
| chaconpiza | So it was producing the metrics with a long string in the "value_meta" besides of having "1" as the metric "value". | 13:53 |
| chaconpiza | we are wondering whether Influxdb has troubles to process a big amount of points with this "long value_meta" | 13:53 |
| dougsz | chaconpiza: Issue with the detection plugin not configuring the URL correctly? | 13:53 |
| chaconpiza | In our devs machines the detection plugin works well and it end up with correct URL for keystone, nova, cinder, etc | 13:54 |
| dougsz | value_meta limit is ~2kb right? the main issue I have seen is with two many unique dimensions | 13:54 |
| chaconpiza | I am not sure how the pre-production team got it | 13:54 |
| dougsz | *s/two/too | 13:54 |
| chaconpiza | because of the cardinality? | 13:54 |
| dougsz | yeah | 13:55 |
| chaconpiza | We are simulating the metric-agent with: https://github.com/monasca/monasca-perf/blob/master/scale_perf | 13:55 |
| chaconpiza | agent_simulator.py but setting a long string in the "value_meta". | 13:55 |
| chaconpiza | in order to reproduce the issue | 13:55 |
| chaconpiza | we will keep you informed in case we can break influxdb because of big "value_meta" | 13:58 |
| dougsz | thanks, it's probably worth investigating InfluxDB 1.7 as 1.3 has a security issue where dimension values are leaked across tenants | 13:58 |
| witek | I think it would be better to improve the auto-detection script to configure the agent as needed | 13:59 |
| chaconpiza | so far all metric-agent's configuration were fixed manually and restarted | 14:00 |
| chaconpiza | in pre-prod a single tenant is being used | 14:00 |
| witek | OK, please keep us updated | 14:01 |
| witek | the time is over | 14:01 |
| witek | thanks for the discussions | 14:01 |
| dougsz | thanks all, bye | 14:01 |
| chaconpiza | thanks | 14:01 |
| witek | thanks, bye | 14:01 |
| witek | #endmeeting | 14:01 |
| *** openstack changes topic to "OpenStack Monitoring as a Service | https://wiki.openstack.org/wiki/Monasca" | 14:01 | |
| openstack | Meeting ended Tue Dec 10 14:01:55 2019 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 14:01 |
| openstack | Minutes: http://eavesdrop.openstack.org/meetings/monasca/2019/monasca.2019-12-10-13.00.html | 14:01 |
| openstack | Minutes (text): http://eavesdrop.openstack.org/meetings/monasca/2019/monasca.2019-12-10-13.00.txt | 14:01 |
| openstack | Log: http://eavesdrop.openstack.org/meetings/monasca/2019/monasca.2019-12-10-13.00.log.html | 14:02 |
| *** pcaruana has quit IRC | 14:36 | |
| *** pcaruana has joined #openstack-monasca | 15:01 | |
| openstackgerrit | Adrian Czarnecki proposed openstack/monasca-api master: Update nodejs and npm packages https://review.opendev.org/695459 | 15:56 |
| *** witek has quit IRC | 17:23 | |
| openstackgerrit | Merged openstack/monasca-agent master: Enable running unit tests in py37 environment https://review.opendev.org/696038 | 17:27 |
| openstackgerrit | Merged openstack/monasca-api master: [ussuri][goal] Drop python 2.7 support and testing https://review.opendev.org/694586 | 17:37 |
| *** dougsz has quit IRC | 17:58 | |
| *** adriancz has quit IRC | 18:11 | |
| *** gmann is now known as gmann_afk | 19:40 | |
| *** pcaruana has quit IRC | 20:16 | |
| openstackgerrit | Merged openstack/monasca-common master: Enable running unit tests in py37 environment https://review.opendev.org/696042 | 20:18 |
| *** gmann_afk is now known as gmann | 23:06 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!