| openstackgerrit | Koji Nakazono proposed openstack/monasca-notification master: Fix slack notification https://review.openstack.org/467877 | 00:04 |
|---|---|---|
| openstackgerrit | Koji Nakazono proposed openstack/monasca-notification master: Fix slack notification https://review.openstack.org/467877 | 01:11 |
| openstackgerrit | Koji Nakazono proposed openstack/monasca-notification master: Fix slack notification https://review.openstack.org/467877 | 01:22 |
| *** vishwana_ has joined #openstack-monasca | 03:19 | |
| *** vishwana_ has quit IRC | 03:19 | |
| *** vishwanathj has quit IRC | 03:22 | |
| *** nicolasbock has joined #openstack-monasca | 04:13 | |
| *** mahen8 has joined #openstack-monasca | 05:36 | |
| mahen8 | pip install monasca-common gives errors: https://pastebin.com/vXFgw3H5 | 05:44 |
| mahen8 | I'm trying to install monasca following the article https://www.shaunos.com/643.html | 05:51 |
| arturb | mahen: do you have python-devel installed?? | 06:05 |
| arturb | python-dev | 06:05 |
| arturb | depending on which system you are installing | 06:05 |
| openstackgerrit | Koji Nakazono proposed openstack/monasca-notification master: Fix slack notification https://review.openstack.org/467877 | 06:18 |
| *** ukaynar has quit IRC | 06:36 | |
| openstackgerrit | Artur Basiak proposed openstack/monasca-ui master: Fix monasca-ui forms https://review.openstack.org/486926 | 07:55 |
| *** efoley has joined #openstack-monasca | 08:00 | |
| openstackgerrit | Artur Basiak proposed openstack/monasca-ui master: Fix monasca-ui forms https://review.openstack.org/486926 | 08:09 |
| openstackgerrit | Artur Basiak proposed openstack/monasca-ui master: Fix monasca-ui forms https://review.openstack.org/486926 | 08:09 |
| openstackgerrit | Artur Basiak proposed openstack/monasca-ui master: Fix monasca-ui forms https://review.openstack.org/486926 | 09:00 |
| *** mgoddard_ has joined #openstack-monasca | 09:03 | |
| openstackgerrit | Artur Basiak proposed openstack/monasca-ui master: Fix monasca-ui forms https://review.openstack.org/486926 | 09:30 |
| Neptu | hej | 09:45 |
| Neptu | i wrote an small stress test for monasca-api to test monasca throwput and i get around 4-5k messages/s with a 1c single node monasca-api | 09:46 |
| Neptu | i saw the cpu load for the monasca api with to 100% on that vm | 09:46 |
| Neptu | so i wonder if you have a metal thinking about that monasca-api can handle around 4k messages/second per core | 09:47 |
| Neptu | anyone has experience on this? | 09:47 |
| *** kornicameister has quit IRC | 09:56 | |
| *** witek has quit IRC | 10:00 | |
| *** openstackgerrit has quit IRC | 10:17 | |
| *** witek has joined #openstack-monasca | 11:05 | |
| mahen8 | arturb: python-dev is already present | 11:14 |
| mahen8 | what could be the reasons so far, that no one made a documentation for direct installation of monasca on OS | 11:20 |
| *** openstackgerrit has joined #openstack-monasca | 12:05 | |
| openstackgerrit | Artur Basiak proposed openstack/monasca-ui master: [WIP] Fix monasca-ui forms https://review.openstack.org/486926 | 12:05 |
| openstackgerrit | Artur Basiak proposed openstack/monasca-ui master: [WIP] Fix monasca-ui forms https://review.openstack.org/486926 | 12:19 |
| *** MVenesio has joined #openstack-monasca | 12:28 | |
| openstackgerrit | Artur Basiak proposed openstack/monasca-ui master: [WIP] Fix monasca-ui forms https://review.openstack.org/486926 | 12:30 |
| openstackgerrit | Artur Basiak proposed openstack/monasca-ui master: [WIP] Fix monasca-ui forms https://review.openstack.org/486926 | 12:41 |
| openstackgerrit | Artur Basiak proposed openstack/monasca-ui master: [WIP] Fix monasca-ui forms https://review.openstack.org/486926 | 12:49 |
| openstackgerrit | Artur Basiak proposed openstack/monasca-ui master: [WIP] Fix monasca-ui forms https://review.openstack.org/486926 | 12:59 |
| *** bobh has joined #openstack-monasca | 13:39 | |
| *** rbrndt has joined #openstack-monasca | 14:09 | |
| *** efoley has quit IRC | 14:15 | |
| *** efoley has joined #openstack-monasca | 14:16 | |
| openstackgerrit | Kaiyan Sheng proposed openstack/monasca-notification master: Modify Notification Engine to allow inhibit, silence, and group https://review.openstack.org/434537 | 14:46 |
| *** ukaynar has joined #openstack-monasca | 14:55 | |
| openstackgerrit | Kaiyan Sheng proposed openstack/python-monascaclient master: Support CLI for group, silence, and inhibit rule https://review.openstack.org/466114 | 14:57 |
| openstackgerrit | Kaiyan Sheng proposed openstack/python-monascaclient master: Support CLI for group, silence, and inhibit rule https://review.openstack.org/466114 | 15:22 |
| *** ukaynar has quit IRC | 15:24 | |
| *** vishwanathj has joined #openstack-monasca | 15:24 | |
| *** ukaynar has joined #openstack-monasca | 15:24 | |
| *** vishwanathj has quit IRC | 15:36 | |
| *** vishwanathj has joined #openstack-monasca | 15:37 | |
| Neptu | hej | 15:41 |
| Neptu | im stress testing monasca | 15:42 |
| Neptu | and i see something wrong happends when i push more than 4000 messages per second | 15:42 |
| *** MVenesio has quit IRC | 15:42 | |
| rbrndt | Neptu: can you be more specific? what environment are you using? which components? | 15:46 |
| Neptu | rbrndt: well we came into same conclusion in the office we are using monasca_perf with little code adapting and I wrote a c++ software that smacks into the cpu given the token... and in both cases we have a 4000 messages/s barrier | 16:06 |
| Neptu | i gave the monasca vm 8 cores | 16:06 |
| Neptu | and modified the amount of workers and the default 2000 to 8000 | 16:07 |
| Neptu | and i get that vm 100% up on each of the 8 cores | 16:07 |
| Neptu | but then something interesting happens | 16:07 |
| Neptu | and the cpu goes down to 3-5% | 16:07 |
| Neptu | and then up again | 16:08 |
| Neptu | my co-worker saw some problems with influx going down and the persistors | 16:08 |
| rbrndt | What is your environment? | 16:08 |
| rbrndt | are you trying to push this all through devstack? | 16:08 |
| Neptu | we have developed an decoupled your enviroment to pieces... so we have aprox 10 vms with each of the components and a vagrant+ansible script to get it up on our datacenter | 16:09 |
| Neptu | we have individually create clusters on this case i have a 3 node kafka | 16:10 |
| Neptu | 3 node zookeeper | 16:10 |
| Neptu | 1 node api | 16:10 |
| rbrndt | so which piece is maxing out? | 16:10 |
| Neptu | im setting up grafana at the moment | 16:10 |
| Neptu | with a nice dashboard to tell me that | 16:10 |
| Neptu | but is intereting we hit same conclusion both persons | 16:11 |
| Neptu | in my opinion seems to be kafka... but that sounds strange | 16:11 |
| Neptu | maybe there is a replica problem | 16:11 |
| Neptu | i just wanted to mention this because maybe i could benefit from your experience | 16:11 |
| rbrndt | Running each piece separately and clustered for kafka, etc. is the intending run state for monasca, you shouldn't be seeing a limit until much higher | 16:12 |
| rbrndt | If kafka is maxing out, then you may want to double check it's configuration | 16:12 |
| openstackgerrit | Kaiyan Sheng proposed openstack/monasca-notification master: Modify Notification Engine to allow inhibit, silence, and group https://review.openstack.org/434537 | 16:12 |
| rbrndt | kafka should never max during normal operation, even in our stress testing we never managed to push kafka to the limits | 16:13 |
| *** rbrndt has quit IRC | 16:16 | |
| Neptu | rbrndt: yep we did neigther | 16:16 |
| *** rbrndt has joined #openstack-monasca | 16:16 | |
| Neptu | rbrndt: we did neigher push kafka to limit should be a replica test | 16:16 |
| Neptu | what im curious is the api how much messages/s on a single core | 16:17 |
| Neptu | is python so should not be high | 16:17 |
| Neptu | but still need to check | 16:17 |
| rbrndt | we've pushing the system as a whole to hundreds of thousands | 16:17 |
| Neptu | with a loadbalancer | 16:17 |
| rbrndt | trying to run the API on a single core is something we never tested. The api is intended to scale as necessary to support the load | 16:18 |
| rbrndt | The general scale bottleneck that I am aware of is the persister/influx | 16:18 |
| Neptu | yes yes but is an addon how many cores i need to process half a million messages | 16:18 |
| rbrndt | but like I said, we've still managed to reach over 100,000/s | 16:18 |
| Neptu | influx showed 2M on the stress test | 16:19 |
| rbrndt | are you running clustered influx? | 16:19 |
| Neptu | i have not tested individually the persistors | 16:19 |
| Neptu | nop | 16:19 |
| Neptu | 2M one machine | 16:19 |
| Neptu | but big machine | 16:19 |
| Neptu | do not remember but a lot of cores | 16:19 |
| Neptu | we have some report somewhere | 16:19 |
| rbrndt | We would love to see your setup and data, we haven't really had a good opportunity to do comprehensive testing on each component | 16:20 |
| Neptu | we are using prometheus to monitor the cluster | 16:20 |
| Neptu | i need to setup some dashboards and see where the bottleneck is | 16:20 |
| rbrndt | The api should be capable of a lot of messages per second, but we've always been limited by the persister first, so I don't know exactly how many | 16:20 |
| Neptu | rbrndt: we had some time to do that | 16:20 |
| Neptu | ok i see | 16:21 |
| Neptu | problem of the persistor in theory is one persistor per replica | 16:21 |
| Neptu | so depending on how much you divide the data | 16:21 |
| Neptu | the more persistors you can fit | 16:21 |
| Neptu | ... | 16:21 |
| rbrndt | we've seen a drop-off in efficiency of more than 16 persisters | 16:22 |
| Neptu | mmmm | 16:22 |
| Neptu | we have 2-3 right now | 16:22 |
| rbrndt | somewhere between 12-16 is what we found to be best | 16:22 |
| Neptu | we are just tunning the performance test of the whole | 16:22 |
| Neptu | but i mean if we manage to get 2M messages in we will theoretically hit the influx wall | 16:22 |
| Neptu | which is probably the bigest one | 16:22 |
| Neptu | after that all the database need to be replaced | 16:23 |
| Neptu | so long story | 16:23 |
| rbrndt | Well, I would be very interested in your results, and I can definitely try to answer any questions you have about our own testing. | 16:24 |
| Neptu | is good | 16:25 |
| Neptu | i was a little worried my c++ sofrare hit the wall with 4000 messages per second | 16:25 |
| Neptu | but now i understand is something else | 16:26 |
| Neptu | in any case i find it gonna setup grafana | 16:26 |
| Neptu | for prometheus | 16:26 |
| Neptu | we intend to use monasca to monitor monsca | 16:26 |
| Neptu | but not for now ;) | 16:27 |
| *** mgoddard_ has quit IRC | 16:47 | |
| *** andreaf has quit IRC | 18:01 | |
| *** andreaf has joined #openstack-monasca | 18:09 | |
| *** nicolasbock has quit IRC | 19:52 | |
| *** efoley has quit IRC | 21:14 | |
| openstackgerrit | Merged openstack/monasca-ceilometer master: Create a package for monasca-ceilometer https://review.openstack.org/484925 | 21:23 |
| *** bobh has quit IRC | 22:41 | |
| *** rbrndt has quit IRC | 22:58 | |
| openstackgerrit | Kaiyan Sheng proposed openstack/python-monascaclient master: Support CLI for group, silence, and inhibit rule https://review.openstack.org/466114 | 23:18 |
| openstackgerrit | Kaiyan Sheng proposed openstack/monasca-notification master: Modify Notification Engine to allow inhibit, silence, and group https://review.openstack.org/434537 | 23:22 |
| *** vishwanathj has quit IRC | 23:37 | |
| *** bobh has joined #openstack-monasca | 23:37 | |
| *** vishwanathj has joined #openstack-monasca | 23:37 | |
| *** bobh has quit IRC | 23:53 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!