| *** jawad_axd has joined #openstack-monasca | 00:34 | |
| *** jawad_axd has quit IRC | 00:38 | |
| *** jawad_axd has joined #openstack-monasca | 01:15 | |
| *** jawad_axd has quit IRC | 01:19 | |
| *** jawad_axd has joined #openstack-monasca | 01:36 | |
| *** jawad_axd has quit IRC | 01:41 | |
| *** jawad_axd has joined #openstack-monasca | 01:57 | |
| *** jawad_axd has quit IRC | 02:01 | |
| *** mbindlish has joined #openstack-monasca | 03:03 | |
| *** vishalmanchanda has joined #openstack-monasca | 05:58 | |
| *** witek has joined #openstack-monasca | 06:22 | |
| *** jawad_axd has joined #openstack-monasca | 07:22 | |
| *** dougsz has joined #openstack-monasca | 07:37 | |
| *** tosky has joined #openstack-monasca | 07:40 | |
| *** k_mouza has joined #openstack-monasca | 09:32 | |
| openstackgerrit | kuang congxian proposed openstack/monasca-api master: Remove six.add_metaclass https://review.opendev.org/753269 | 09:34 |
|---|---|---|
| openstackgerrit | kuang congxian proposed openstack/monasca-api master: Remove six.PY2 and six.PY3 https://review.opendev.org/753274 | 09:52 |
| *** adriancz has joined #openstack-monasca | 10:02 | |
| *** k_mouza has quit IRC | 10:35 | |
| witek | adriancz, dougsz: could we get monasca-transform deprecation merged, please? | 10:44 |
| witek | https://review.opendev.org/751984 | 10:44 |
| *** k_mouza has joined #openstack-monasca | 10:44 | |
| adriancz | done | 10:45 |
| witek | thanks | 10:45 |
| openstackgerrit | Merged openstack/monasca-transform master: Remove project content on master branch https://review.opendev.org/751984 | 10:45 |
| *** k_mouza has quit IRC | 10:48 | |
| *** k_mouza has joined #openstack-monasca | 10:48 | |
| openstackgerrit | Witold Bedyk proposed openstack/monasca-analytics master: Disable Zuul jobs https://review.opendev.org/753288 | 11:17 |
| openstackgerrit | Witold Bedyk proposed openstack/monasca-analytics master: Add Monasca-analytics use-cases with Monasca https://review.opendev.org/656917 | 11:18 |
| openstackgerrit | Witold Bedyk proposed openstack/monasca-analytics master: WIP Add fard disks failure detection example https://review.opendev.org/692676 | 11:19 |
| *** bandorf has joined #openstack-monasca | 12:59 | |
| witek | hello everyone | 13:00 |
| bandorf | hello | 13:00 |
| witek | hi Matthias | 13:00 |
| witek | do we have any topics to discuss? | 13:01 |
| witek | the agenda for today is empty | 13:01 |
| bandorf | From my site, currently not. Did Adrian talk to you? | 13:01 |
| witek | yes, but hasn't mentioned any topics | 13:02 |
| bandorf | I'm currently mainly working on internal org-topics. Thus, no monasca-related topics | 13:02 |
| bandorf | from my site | 13:03 |
| witek | no worries, anyone else around? | 13:03 |
| bandorf | Let me ping Adrian | 13:03 |
| tosky | now that monasca-transform content has been cleaned, there are no jobs based on legacy-dsvm-base in master and (soon) victoria | 13:04 |
| adriancz | hi | 13:04 |
| tosky | there are still 3 other legacy jobs, which don't depend on devstack-gate, so less likely to suddendly break, but they should be ported at some point | 13:05 |
| witek | these are Java build and publish jobs, right? | 13:05 |
| tosky | yep | 13:05 |
| tosky | legacy-monasca-common-maven-build, legacy-monasca-persister-maven-build and legacy-monasca-persister-maven-build, last time I checked | 13:06 |
| witek | also legacy-monasca-thresh-maven-build | 13:06 |
| tosky | ups, right, wrong copy-and-paste | 13:07 |
| tosky | I copied twice the same | 13:07 |
| witek | we should drop legacy-monasca-persister-maven-build, we've deprecated Java implementation in monasca-persister | 13:07 |
| witek | let me propose a change for that | 13:08 |
| tosky | thanks! | 13:09 |
| tosky | as I already said, from my point of view (the goal's point of view) as long as the legacy jobs disappear it's fine; either because they are ported, or because they are removed | 13:10 |
| witek | does the goal include only DevStack based jobs, or any legacy Zuul jobs? | 13:10 |
| witek | I know we should migrate anyway, it's just for info | 13:11 |
| *** k_mouza has quit IRC | 13:12 | |
| witek | on the deprecation track: there are two documentation changes in monasca-analytics repo which I would like to merge before retiring the repo | 13:13 |
| witek | I wasn't able to merge them because of CI failing, so proposed disabling Zuul jobs | 13:13 |
| witek | https://review.opendev.org/753288 | 13:14 |
| *** k_mouza has joined #openstack-monasca | 13:14 | |
| witek | adriancz: could we please merge? ^^ | 13:14 |
| adriancz | ok, i will review that | 13:15 |
| adriancz | i cant merge this | 13:15 |
| witek | thanks | 13:15 |
| witek | ups | 13:15 |
| tosky | witek: so, the goal is about all legacy jobs, but the jobs which depends on devstack-gate are more relevant, because they *will* break for sure | 13:15 |
| adriancz | im not core reviewer : ( | 13:15 |
| witek | thanks tosky | 13:16 |
| witek | adriancz: indeed, the core list is very short | 13:16 |
| tosky | so you don't stricly need to rush, but having native jobs usually means less maintenance burden | 13:16 |
| tosky | it's for your own good | 13:16 |
| witek | tosky: do you know if there are any other Java related jobs existing? | 13:17 |
| tosky | witek: other jobs which deal with Java in other repositories? Uhm, I don't know | 13:18 |
| tosky | you may check if there are java-based base jobs in opendev.org/zuul/zuul-jobs.git, and if that fails, well, there is codesearch :) | 13:19 |
| witek | yep, thanks | 13:19 |
| witek | any other topics for today? | 13:20 |
| witek | if not, thanks and see you next week | 13:22 |
| bandorf | Bye, everyody | 13:22 |
| adriancz | Thanks | 13:23 |
| openstackgerrit | Merged openstack/monasca-analytics master: Disable Zuul jobs https://review.opendev.org/753288 | 13:23 |
| *** bandorf has quit IRC | 13:30 | |
| *** CBR09 has joined #openstack-monasca | 13:32 | |
| CBR09 | hi witek, are you around here | 13:34 |
| witek | yes | 13:34 |
| CBR09 | I've a question about monasca performance | 13:35 |
| CBR09 | I wonder about performance of monasca in large-scale, especially in validating token between monasca-api and keystone. | 13:35 |
| CBR09 | is there any benchmark performance of monasca and keystone ? ( I saw some presentation but it has very few informantion) | 13:35 |
| witek | it works as any other OpenStack service | 13:36 |
| witek | normally memcached is configured to cache the tokens and speed up | 13:36 |
| CBR09 | yea, I've setup memcached for caching | 13:36 |
| CBR09 | but I've still got low performance | 13:37 |
| CBR09 | around 2000 request/s | 13:37 |
| CBR09 | with 16 core 32G ram | 13:37 |
| CBR09 | I tried increase from 16 to 32core, but it's still around 2k request/s | 13:37 |
| witek | what's your setup? how many Monasca and Keystone nodes do you have? | 13:38 |
| CBR09 | I don't know what component is bottoneck | 13:38 |
| CBR09 | I've tried on one node with 16 cores | 13:38 |
| CBR09 | and using docker-compose to bootstrap monasca | 13:38 |
| CBR09 | and setup mecached to caching + fernet token for keystone | 13:39 |
| witek | is Keystone a bottleneck or Persister? | 13:40 |
| CBR09 | I don't know the bottleneck is monasca-api or keystone | 13:41 |
| CBR09 | how can I identify that ? | 13:41 |
| CBR09 | I don't think the persister here, because of posting metric is relevant to monasca-api and keystone | 13:43 |
| witek | how do you measure the performance? | 13:44 |
| CBR09 | I've wrote a go program (using goroutine) and send request to monasca-api as fast as possible | 13:45 |
| CBR09 | and calc avg of request/s | 13:46 |
| witek | does running multiple instances change the result? | 13:46 |
| CBR09 | I didn't try it, but assumpsion running with 3 instances | 13:49 |
| CBR09 | and x3 request/s, my total is 6k request/s, it's still low | 13:49 |
| CBR09 | I think with a node 16 core - 32g ram, it should handle more than 2k request/s | 13:50 |
| witek | but it will indicate if the bottleneck is the API or your test program | 13:50 |
| CBR09 | yes, I agree with that, it will indicate the bottleneck is the api or my client | 13:52 |
| CBR09 | ah in the performance test, I saw the strange thing | 13:54 |
| CBR09 | i've setup the flag: legacy_kafka_client_enabled=false | 13:55 |
| CBR09 | to using new confluent-kafka to improving performance | 13:55 |
| CBR09 | but it decrease performance than before, I don't know why | 13:56 |
| witek | you could try setting log.message.format.version=0.9.0.0 in Kafka's server.properties | 14:02 |
| CBR09 | my kafka server is 2.4, and config log.message.format.version=0.9.0.0 into kafka server.properties? | 14:04 |
| witek | that's because monasca-thresh still works with the old message format, or you could just skip deploying monasca-thresh | 14:08 |
| CBR09 | for now I just focus on request/s when posting metrics to monasca-api | 14:09 |
| CBR09 | and I don't know why new confluent kafka client cause decrease the performance | 14:10 |
| witek | seems strange to me as well, would be good to eliminate all other possible error sources | 14:17 |
| witek | try to run only API and Kafka and see how the results change | 14:17 |
| CBR09 | what do you mean try to run only api and kafka ? | 14:18 |
| CBR09 | do I need stop monasca-thres, persister and influxdb | 14:19 |
| witek | monasca-api and Kafka, without persister and thresh | 14:19 |
| witek | correct | 14:19 |
| CBR09 | yea, I will give it a try | 14:20 |
| CBR09 | ah is there any way to disable keystone auth when posting metrics ? | 14:24 |
| witek | no, but I think it would be a useful feature (for testing) | 14:25 |
| CBR09 | yes | 14:33 |
| dougsz | sorry, was head down, CBR09, have you looked at per process CPU load? | 14:35 |
| dougsz | is your go program single threaded and maxing out a core? | 14:36 |
| *** k_mouza has quit IRC | 14:36 | |
| CBR09 | my go program using 8 core and around 300 go-routine run on 8 core | 14:37 |
| dougsz | well, probably not that then :D | 14:38 |
| CBR09 | I've tunning keystone worker + monasca guniorn worker and it just increase max to 2k5: D | 14:38 |
| *** k_mouza has joined #openstack-monasca | 14:43 | |
| CBR09 | why not dougsz: D | 14:45 |
| dougsz | So 2.5k req/s with a 100 metrics in each is 250k metrics / second. That is close to the limit of InfluxDB. | 14:51 |
| CBR09 | ah, my go program just send 1 metrics in each request, so it mean 2.5k request/s ~ 2.5k metrics/s | 14:52 |
| dougsz | Understood, just thinking about real world use cases. | 14:53 |
| CBR09 | oh, I saw, thank for that point | 14:54 |
| CBR09 | can you share me some link about limit of influxdb | 14:54 |
| CBR09 | on number of metrics/s | 14:54 |
| dougsz | https://docs.influxdata.com/influxdb/v1.8/guides/hardware_sizing/#single-node-or-cluster | 14:55 |
| dougsz | 750k metrics/s claimed there | 14:56 |
| dougsz | I've found to get to ~100k metrics/s you need to run quite a few persisters | 14:56 |
| dougsz | Oh yeah, how many partitions do you have in Kafka? | 14:56 |
| *** jawad_axd has quit IRC | 14:56 | |
| dougsz | on the metrics topic? | 14:57 |
| CBR09 | let's me check | 14:57 |
| CBR09 | but if we close to the limit of InfluxDB, we need to implement more time series backend or something like load balancing | 14:59 |
| CBR09 | so we still need more performance when posting metrics to monasca-api : D | 14:59 |
| *** mbindlish has quit IRC | 15:00 | |
| CBR09 | if just 2k5 req/s, it can be accept with metrics, but for logs I think it very low | 15:01 |
| dougsz | How many nodes are you hoping to monitor? | 15:01 |
| CBR09 | just anwser how many partitions on metric topic: metrics:64:1, 64 partitions | 15:01 |
| dougsz | We batch logs in Kolla Ansible and post every 30 seconds or so, so there is a lot of capacity | 15:02 |
| dougsz | ok, sounds like you have plenty of partitions for sharing the work out | 15:03 |
| dougsz | Was wondering if you just had 1 or something | 15:03 |
| dougsz | And I assume you are using Fernet tokens? | 15:04 |
| dougsz | (not UUID) | 15:04 |
| CBR09 | How many nodes are you hoping to monitor? => we want to monitor as many as possible node, around 10k request/s | 15:04 |
| CBR09 | yes | 15:04 |
| CBR09 | I'm using fernet token and memcache for caching | 15:04 |
| dougsz | Ok, that is good | 15:05 |
| dougsz | And which processes are pinning cores? Sorry if i missed it. | 15:06 |
| dougsz | or rather maxing out cores | 15:06 |
| CBR09 | we don't pinning any processes to core, how can I do that ? | 15:07 |
| CBR09 | We batch logs in Kolla Ansible and post every 30 seconds or so, so there is a lot of capacity => we have a log rate ~ 40-50k req/s, so it's impossible to using monasca | 15:09 |
| CBR09 | when using kafka behind monasca+keystone, I saw the performance decrease too much | 15:10 |
| dougsz | Wow. Sorry, above I mean which processes are using 100% CPU. I guess you see Keystone? | 15:20 |
| dougsz | You should be able to self monitor to graph it | 15:20 |
| CBR09 | right now I using htop to view cpu usage and saw monasca-api gunicorn consume more cpu | 15:24 |
| CBR09 | I will setup self monitor to graph it | 15:26 |
| CBR09 | to see clearly about cpu usage of keystone and monasca-api | 15:26 |
| dougsz | Sounds good. You've compared against Apache/wsgi in Kolla? | 15:28 |
| CBR09 | I using docker-compose to bootraping monasca | 15:29 |
| CBR09 | and saw keystone run with uwsgi | 15:29 |
| CBR09 | any recommend about that ? | 15:29 |
| CBR09 | ah i saw many logs in monasca-api | 15:29 |
| CBR09 | 2020-09-22 15:28:23,936.936 42 WARNING urllib3.connectionpool [req-a3ea5df2-d554-4e94-b707-5d7cb3872419 559fa14ea23345a19b7f08885bd6c9eb d419ea8d4b544e2cb7dec129ff241b77 - default default] Connection pool is full, discarding connection: keystone: queue.Full | 15:29 |
| dougsz | ah, I wondered how gunicorn compares to wsgi in kolla for the API, same for Keystone | 15:30 |
| dougsz | Hmm, wonder if you hit max connections with memcached? | 15:31 |
| dougsz | We saw something like that for Neutron | 15:31 |
| CBR09 | ah I don't benchmark python web server like wsgi and gunicorn | 15:32 |
| CBR09 | but I searched the blogs | 15:32 |
| CBR09 | https://www.appdynamics.com/blog/engineering/a-performance-analysis-of-python-wsgi-servers-part-2/ | 15:32 |
| dougsz | yeah, saw that, no time to read carefully, but wondered if that was the consensus | 15:34 |
| CBR09 | Hmm, wonder if you hit max connections with memcached? => let me check | 15:34 |
| CBR09 | if keystone is the bottleneck, next time we will try run keystone behind apache mod_wsgi because uwsgi's performance is worst like above blog | 15:36 |
| dougsz | https://pastebin.com/4BLYKwKL | 15:36 |
| dougsz | You could try adding that setting in Monasca API? | 15:36 |
| dougsz | If you are seeing an issue with using all memcached connections | 15:37 |
| CBR09 | thank you, I will try it, for the above errors | 15:37 |
| dougsz | np | 15:38 |
| CBR09 | I saw memcache logs, too many open file | 15:38 |
| CBR09 | I will increase that limit and setting your config | 15:38 |
| dougsz | Interesting | 15:39 |
| *** witek has quit IRC | 15:56 | |
| *** irclogbot_0 has quit IRC | 16:08 | |
| *** irclogbot_2 has joined #openstack-monasca | 16:09 | |
| CBR09 | that's config just only work on python 2.x : ( | 16:17 |
| dougsz | CBR09: interesting | 16:21 |
| dougsz | For Neutron on Centos 8 + py3 it was fine, despite the warning | 16:21 |
| dougsz | You see it cause a failure in the logs? | 16:22 |
| CBR09 | really?, i will try it = )), I just see warning in config sample file | 16:22 |
| dougsz | yeah, we didn't fully investigate why the warning note is no longer valid, but it certainly seems to work on py3 | 16:23 |
| *** k_mouza has quit IRC | 16:25 | |
| CBR09 | thank you, i will try it | 16:27 |
| dougsz | CBR09: Thanks, please share results, I am interested :) | 16:31 |
| CBR09 | tomorrow i will share results with you, in my country now is mid night : )) | 16:32 |
| CBR09 | i guest i need get some sleep | 16:33 |
| dougsz | sleep is good! bye for now | 16:33 |
| CBR09 | thank you for your help, i will share soon with you | 16:33 |
| CBR09 | bye | 16:33 |
| *** dougsz has quit IRC | 16:33 | |
| *** k_mouza has joined #openstack-monasca | 16:34 | |
| *** CBR09 has quit IRC | 16:35 | |
| *** k_mouza has quit IRC | 16:38 | |
| *** k_mouza has joined #openstack-monasca | 16:42 | |
| *** k_mouza has quit IRC | 16:46 | |
| *** k_mouza has joined #openstack-monasca | 16:51 | |
| *** k_mouza has quit IRC | 17:11 | |
| *** k_mouza has joined #openstack-monasca | 17:18 | |
| *** k_mouza has quit IRC | 17:23 | |
| *** k_mouza has joined #openstack-monasca | 17:37 | |
| *** k_mouza has quit IRC | 17:41 | |
| *** k_mouza has joined #openstack-monasca | 17:46 | |
| *** k_mouza has quit IRC | 17:51 | |
| *** vishalmanchanda has quit IRC | 18:58 | |
| *** tosky_ has joined #openstack-monasca | 21:09 | |
| *** k_mouza has joined #openstack-monasca | 21:09 | |
| *** tosky is now known as Guest33200 | 21:10 | |
| *** tosky_ is now known as tosky | 21:10 | |
| *** k_mouza has quit IRC | 21:14 | |
| *** gmann has quit IRC | 22:04 | |
| *** adriancz has quit IRC | 22:04 | |
| *** gmann has joined #openstack-monasca | 22:06 | |
| *** adriancz has joined #openstack-monasca | 22:06 | |
| *** tosky has quit IRC | 22:48 | |
| *** gokhani has quit IRC | 23:09 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!