| *** timothyb89 has quit IRC | 01:14 | |
| *** timothyb89 has joined #openstack-monasca | 01:14 | |
| *** jayahn has joined #openstack-monasca | 01:16 | |
| *** bobh has quit IRC | 01:59 | |
| *** jayahn has quit IRC | 02:08 | |
| *** zhurong has joined #openstack-monasca | 02:33 | |
| *** bobh has joined #openstack-monasca | 02:45 | |
| *** bobh has quit IRC | 02:49 | |
| *** zhurong has quit IRC | 03:15 | |
| *** zenirc369 has joined #openstack-monasca | 04:22 | |
| *** zhurong has joined #openstack-monasca | 05:07 | |
| *** nadya has joined #openstack-monasca | 05:15 | |
| *** nadya has quit IRC | 05:21 | |
| *** zhurong has quit IRC | 05:52 | |
| *** nadya has joined #openstack-monasca | 06:14 | |
| *** zenirc369 has quit IRC | 06:16 | |
| *** zhurong has joined #openstack-monasca | 06:25 | |
| *** zhurong has quit IRC | 06:33 | |
| *** nadya has quit IRC | 06:45 | |
| *** nadya has joined #openstack-monasca | 07:18 | |
| *** pcaruana has joined #openstack-monasca | 07:19 | |
| *** nadya has quit IRC | 07:24 | |
| *** zhurong has joined #openstack-monasca | 07:27 | |
| *** efoley has joined #openstack-monasca | 07:39 | |
| *** zhurong has quit IRC | 08:08 | |
| *** kei_yama has quit IRC | 08:41 | |
| *** efoley has quit IRC | 08:46 | |
| *** nadya has joined #openstack-monasca | 09:12 | |
| *** zhurong has joined #openstack-monasca | 09:22 | |
| *** zhurong has quit IRC | 09:37 | |
| openstackgerrit | Artur Basiak proposed openstack/monasca-api master: Set WSGI_MODE to mod_wsgi https://review.openstack.org/460030 | 09:55 |
|---|---|---|
| *** nadya has quit IRC | 10:01 | |
| *** efoley has joined #openstack-monasca | 10:12 | |
| *** nadya has joined #openstack-monasca | 10:32 | |
| *** nadya has quit IRC | 10:37 | |
| *** witek has quit IRC | 10:53 | |
| *** nadya has joined #openstack-monasca | 10:54 | |
| *** witek has joined #openstack-monasca | 11:09 | |
| *** efoley has quit IRC | 11:48 | |
| openstackgerrit | Artur Basiak proposed openstack/monasca-api master: Set WSGI_MODE to mod_wsgi https://review.openstack.org/460030 | 11:52 |
| *** dtardivel has joined #openstack-monasca | 12:16 | |
| *** efoley has joined #openstack-monasca | 12:19 | |
| *** efoley_ has joined #openstack-monasca | 12:21 | |
| *** efoley has quit IRC | 12:24 | |
| *** tongli has joined #openstack-monasca | 13:20 | |
| *** bobh has joined #openstack-monasca | 13:42 | |
| *** bobh has quit IRC | 14:01 | |
| *** rbrndt has joined #openstack-monasca | 14:03 | |
| *** jayahn has joined #openstack-monasca | 14:08 | |
| *** bobh has joined #openstack-monasca | 14:27 | |
| openstackgerrit | Kaiyan Sheng proposed openstack/monasca-agent master: Add feature to define namespace annotations https://review.openstack.org/448895 | 14:58 |
| *** tongli has quit IRC | 15:00 | |
| *** rmelero has joined #openstack-monasca | 15:27 | |
| *** tung_doan has joined #openstack-monasca | 15:36 | |
| *** Neptu has joined #openstack-monasca | 15:39 | |
| Neptu | Hej | 15:39 |
| Neptu | hellow to everyone | 15:39 |
| rbrndt | Welcome | 15:40 |
| Neptu | thanks | 15:53 |
| Neptu | little cozy place | 15:54 |
| Neptu | anyway is getting late and need get kids going | 15:55 |
| Neptu | was a good meeting lets talk tomorrow | 15:55 |
| Neptu | what is the normal schedule of the channel european time or US time?? | 15:55 |
| rbrndt | I'm here US time, not sure what its like before or after that | 15:55 |
| Neptu | hahaha | 15:55 |
| rbrndt | :) | 15:55 |
| Neptu | then i do not bother to ask you anything before coffe time | 15:56 |
| rbrndt | no worries, I'm not a coffee drinker :D | 15:56 |
| Neptu | me neigther | 15:56 |
| Neptu | before going do you have some benchmarking reports | 15:57 |
| Neptu | to understand a bit what is a normal setup when it comes to api nodes... kafka nodes and influx nodes | 15:57 |
| rbrndt | I do, but not handy. I've been working on other things outside monasca for a while :( | 15:57 |
| Neptu | ok | 15:58 |
| rbrndt | I'll put together something for you today though | 15:58 |
| Neptu | no stress i just drafting the whole thing in my head and I need to understand where the bottleneck start | 15:59 |
| openstackgerrit | Johannes Grassler proposed openstack/monasca-kibana-plugin master: Use GET for Keystone health check https://review.openstack.org/460195 | 15:59 |
| rbrndt | We really want something other than influx now that they've gone paid only for clustering | 16:00 |
| Neptu | yes | 16:00 |
| Neptu | i mean we are 3, we got an interesting CV today from a python dev | 16:01 |
| Neptu | so this will grow | 16:01 |
| rbrndt | That sound awesome, we would love to welcome some new people to monasca | 16:01 |
| Neptu | i knwo other some banks have tested monasca and they have gone away from it here in stockholm due influx | 16:02 |
| Neptu | but i mean we are all developers here | 16:02 |
| Neptu | so we should be able to put something together | 16:02 |
| Neptu | quickly | 16:02 |
| rbrndt | Really? that's interesting. we don't have a lot of info about monasca outside our sphere in openstack | 16:03 |
| Neptu | still we are building ansible scripts to pop up cluster enviroments on all key components | 16:03 |
| rbrndt | We had some sensible, but it's long out of date now. | 16:03 |
| rbrndt | s/sensible/ansible | 16:03 |
| Neptu | the idea is to have something to benchmark | 16:03 |
| Neptu | so we are finalizing soon with the script to have x nodes on openstack cluster of kafka zookeper influx and stuff | 16:04 |
| Neptu | but i see a problem with customers in influx not clustering | 16:04 |
| Neptu | people not like to pay | 16:04 |
| rbrndt | Most of our benchmarks have been on three nodes of each API, Kafka, and Vertica. Influx went closed on clustering before we ramped up testing | 16:05 |
| *** nadya has quit IRC | 16:05 | |
| rbrndt | We do have some testing on single node influx though | 16:05 |
| Neptu | but vertica is aswell pay licence | 16:05 |
| *** nadya has joined #openstack-monasca | 16:05 | |
| Neptu | yep 900k which looks good | 16:05 |
| rbrndt | yeah, which is why it is slowly phasing out as well | 16:05 |
| *** nadya has quit IRC | 16:05 | |
| Neptu | yep | 16:05 |
| Neptu | but i mean is the beauty of scaling | 16:06 |
| Neptu | so i need to deploy this on large clusters | 16:06 |
| rbrndt | That has been our strategy | 16:06 |
| Neptu | so scalability is a must | 16:06 |
| Neptu | yep | 16:06 |
| Neptu | im curios about what you said about not compatible datamodel | 16:07 |
| rbrndt | The simple version is cassandra expects you to search on a single aspect of your data, which doesn't work well for slicing by our dimensions | 16:07 |
| Neptu | i will clean my desk before end of the week and will start looking more closely into influx and how stores data | 16:08 |
| rbrndt | that plus handling the size limits of their expected chunk sizes is a problem | 16:08 |
| Neptu | now im getting into something i donno but influxdb what is the chunck of data size | 16:08 |
| Neptu | ?? | 16:08 |
| rbrndt | Can't recall the specific number, but influx handles the tags (dimensions) very well, so we haven't had to worry. Cassandra required us to handle some of that ourselvevs | 16:09 |
| Neptu | mmm | 16:09 |
| Neptu | because casanda is not metric specific | 16:09 |
| rbrndt | More or less, yeah | 16:09 |
| Neptu | i see the problem | 16:10 |
| Neptu | well i need to know more about many things before have a clever conversation about that | 16:10 |
| Neptu | so | 16:10 |
| Neptu | first things first | 16:10 |
| Neptu | ah one last thing before go | 16:11 |
| Neptu | you you have any dummy client that introduces random metrics into the system | 16:11 |
| Neptu | to see it all moving | 16:11 |
| Neptu | and maybe stress it a bit | 16:11 |
| rbrndt | We do as a matter of fact. let me grab a link | 16:12 |
| Neptu | have the agent is good but faking the imput at the moment is helpfull | 16:12 |
| rbrndt | https://github.com/hpcloud-mon/monasca-perf/blob/master/scale_perf/agent_simulator.py | 16:12 |
| rbrndt | this is set up to simulate a number of agents submitting metrics | 16:12 |
| Neptu | awesome | 16:12 |
| Neptu | exactly what i was looking for | 16:13 |
| rbrndt | it has been modified a bit recently to work for containers, but the basic concept is the same | 16:13 |
| Neptu | hehehe still stuck on vagrant & ansible | 16:13 |
| Neptu | vms are my really still | 16:14 |
| rbrndt | I haven't brought up a devstack in weeks, I'm sure it would fail :/ | 16:14 |
| Neptu | it fails yes | 16:14 |
| Neptu | from 1 month ago | 16:14 |
| rbrndt | It was fragile all the time | 16:14 |
| Neptu | the stack command is crazy | 16:15 |
| Neptu | anyone from monasca going to the boston submit? | 16:15 |
| rbrndt | If it helps, we have a vagrant setup in our monasca-api repo. Perhaps you already found that though | 16:15 |
| rbrndt | Roland, the PTL is planning to be there | 16:15 |
| Neptu | yes i saw it but the provisioner is bash i wrote my own in ansible for my conveninnce | 16:16 |
| Neptu | ok my product owner will be there too | 16:16 |
| rbrndt | yeah, devstack is kinda mired in bash | 16:16 |
| Neptu | yep yep | 16:16 |
| Neptu | anyway | 16:17 |
| Neptu | i will maybe have to read some docker then | 16:17 |
| rbrndt | we do have a docker compose that works pretty well | 16:17 |
| Neptu | but the big issue i see is the deployment is gonna be mostly done on openstack vps | 16:17 |
| Neptu | and im not aware you can spin containers | 16:17 |
| Neptu | maybe someone solve that already | 16:17 |
| rbrndt | yeah, openstack is playing catchup on that | 16:17 |
| *** jayahn has quit IRC | 16:18 | |
| Neptu | soon then | 16:18 |
| Neptu | is important to "spin up" the whole cluster at once | 16:18 |
| Neptu | so you can provision all nodes in one go | 16:18 |
| rbrndt | yeah | 16:18 |
| Neptu | im developer and lazy devops | 16:18 |
| Neptu | so my lazy side tells me is a good idea | 16:19 |
| rbrndt | less itime on managing setup = more time on code | 16:19 |
| Neptu | well i have scripts now to spin a sizeable cluster of kafka and influx | 16:20 |
| Neptu | it does the graphana aswell | 16:20 |
| Neptu | missing the monasca api nodes | 16:20 |
| Neptu | soon i will have an script to rule them all | 16:20 |
| rbrndt | That would be excellent | 16:21 |
| Neptu | yes the idea is to start decoupling the components from the monasca-devstack install | 16:21 |
| Neptu | so at some point we can replace the components | 16:22 |
| Neptu | i think is the clever aproach | 16:22 |
| Neptu | but now im talking too much and i need to get home | 16:22 |
| Neptu | hehehe | 16:22 |
| rbrndt | heh, alright don't let me keep you | 16:22 |
| Neptu | nice meeting you | 16:22 |
| Neptu | my pleasure | 16:22 |
| rbrndt | You as well | 16:22 |
| tung_doan | rbrndt: long time no see. I hope you are still doing well! | 16:24 |
| *** rbak has quit IRC | 16:25 | |
| tung_doan | rbrndt: if you have time pls help me look into this Heat auto-scaling template: https://pastebin.com/x6h35sJr | 16:25 |
| tung_doan | rbrndt: I tried too many times but it seems not to work | 16:25 |
| rbrndt | tung_doan, I'd be happy to take e look, though I don't know much about heat | 16:26 |
| tung_doan | rbrndt: btw, i had a blueprint in openstack/tacker which leverage monasca: https://review.openstack.org/#/c/415216/ | 16:26 |
| *** rbak has joined #openstack-monasca | 16:26 | |
| tung_doan | rbrndt: np.. I really appreciate your enthusiasm | 16:29 |
| *** rbak has quit IRC | 16:30 | |
| *** rbak has joined #openstack-monasca | 16:30 | |
| *** jamielennox is now known as jamielennox|away | 16:31 | |
| rbrndt | If i'm reading this right, you have a very narrow window in which nothing should happen | 16:31 |
| rbrndt | between 5% and 10% | 16:32 |
| rbrndt | tung_doan: what if the behavior that you are seeing which is incorrect? | 16:35 |
| rbrndt | s/if/is | 16:35 |
| tung_doan | rbrndt: becasuse i am not in admin domain.. therefore, i used mysql to show alarm infomation | 16:36 |
| tung_doan | rbrndt: in fact, alarm definition, alarm-notification method worked well | 16:37 |
| tung_doan | rbrndt: however, i did not see any metric | 16:37 |
| rbrndt | hmm | 16:38 |
| rbrndt | So, if you query monasca, you don't see any metrics? | 16:38 |
| tung_doan | rbrndt: right | 16:39 |
| rbrndt | The easiest way to figure out this problem would be to query from the admin perspective, but if you don't have that... | 16:39 |
| rbrndt | You said you have access to mysql? do you by chance have access to influx as well? | 16:40 |
| tung_doan | rbrndt: not sure what's going wrong here | 16:40 |
| tung_doan | rbrndt: how to access to influx? why we need to do this? | 16:40 |
| *** rbak_ has joined #openstack-monasca | 16:40 | |
| rbrndt | My first guess is that metrics are all on the admin project and the alarm definition and notification are on a different project | 16:41 |
| tung_doan | rbrndt: good suggestion!!! | 16:41 |
| tung_doan | rbrndt: I had the same feeling | 16:41 |
| rbrndt | If you don't have access to admin, then influx is the only place to verify that | 16:41 |
| rbrndt | Alternatively, you may want to verify that the resources you are trying to autoscale are on the same project as your alarm definitions | 16:42 |
| *** rbak has quit IRC | 16:42 | |
| rbrndt | Just to clarify, do you see zero metrics in monasca? or just not the ones you need? | 16:42 |
| *** rbak has joined #openstack-monasca | 16:43 | |
| tung_doan | rbrndt: absolutely empty.. i did not see anything in mysql | 16:43 |
| rbrndt | Metrics won't be in mysql, only alarm definitions | 16:44 |
| tung_doan | rbrndt: i already tried many times before | 16:44 |
| rbrndt | but if you don't see any metrics under the deifnitions/alarms then they aren't arriving | 16:44 |
| *** rbak__ has joined #openstack-monasca | 16:44 | |
| rbrndt | Maybe a good question to ask someone with Heat experience, is 'how are the monasca alarm definitions created?' because if they are all on the admin tenant, then they should have access to all metrics | 16:45 |
| *** rbak_ has quit IRC | 16:45 | |
| tung_doan | rbrndt: I mean this one: https://pastebin.com/SNvG8znE | 16:46 |
| rbrndt | right | 16:47 |
| tung_doan | rbrndt: "alarm_definition" field worked well | 16:47 |
| tung_doan | rbrndt: but "alarm_metric" is empty :( | 16:47 |
| rbrndt | Right, that would imply the alarm hasn't seen any metrics to match yet. | 16:47 |
| tung_doan | rbrndt: +2 | 16:47 |
| rbrndt | Can you access monasca from its CLI? | 16:48 |
| *** rbak has quit IRC | 16:48 | |
| tung_doan | rbrndt: tried many times before but it seems monasca just works for admin | 16:48 |
| rbrndt | ok, so in your setup you don't have another user or something | 16:49 |
| rbrndt | Right...hmm | 16:49 |
| rbrndt | It's going to be very hard to verify anything without being able to check what metrics exist | 16:49 |
| tung_doan | rbrndt: I suspect that maybe I miss some monasca configuration in Heat template | 16:50 |
| rbrndt | We know the alarm doesn't see them, but that doesn't rule out a mispelling or anything | 16:50 |
| rbrndt | with my limited understanding, I don't see anything out of place | 16:50 |
| rbrndt | perhaps it's not getting the scale group right? | 16:51 |
| rbrndt | template: avg(cpu.utilization_perc{scale_group=scale_group_id}) > 10 times | 16:51 |
| rbrndt | 1 | 16:51 |
| rbrndt | how does it fill in that 'scale_group_id'? | 16:51 |
| tung_doan | rbrndt: right.. i stressed VMs many time but scaling-out did not work | 16:52 |
| tung_doan | rbrndt: it will get input from "params: {scale_group_id: SG1}" | 16:52 |
| tung_doan | rbrndt: I think it is just matching metadata, right? | 16:53 |
| rbrndt | Alright, well that's the extent of my knowledge on Heat. I would suggest looking at the alarm definitions (the expressions should be stored in mysql) to make sure that look right | 16:53 |
| rbrndt | Other than that, verifying the metrics are arriving as expected | 16:53 |
| tung_doan | rbrndt: In my test, VMs were also attached the same metadata "SG1" | 16:54 |
| *** rbak__ has quit IRC | 16:55 | |
| rbrndt | I'm not sure off the top of my head how that would translate into our metrics, but I would trust the Heat folks got it working correctly | 16:55 |
| rbrndt | Else, they would be working on it right now | 16:55 |
| tung_doan | rbrndt: you mean that metric is not set in monasca, right? | 16:55 |
| rbrndt | Yeah, the metrics may be missing the scale_group dimensions or just not arriving at all | 16:56 |
| tung_doan | rbrndt: it is so weird because no error happends | 16:56 |
| rbrndt | Well, missing metrics doesn't trigger anything in monasca | 16:56 |
| rbrndt | we allow you to create alarm definitions for things that don't exist | 16:56 |
| rbrndt | Not sure what Heat does for that | 16:56 |
| tung_doan | rbrndt: agree. | 16:56 |
| tung_doan | I will ask Heat folks for more information | 16:57 |
| tung_doan | rbrndt: thanks for your clarification. you help me confirm the problem. thanks | 16:58 |
| rbrndt | Let me know if you have any more questions for me :) | 16:58 |
| tung_doan | rbrndt: great.. really appreciate that. Wish you have a good day :) I might remind you later. thanks | 16:59 |
| *** kornica has quit IRC | 17:23 | |
| *** pcaruana has quit IRC | 17:40 | |
| *** efoley_ has quit IRC | 17:56 | |
| iurygregory | Hello, anyone have any idea on how i would be able to retrive more than 10K of metrics from monasca? I've try monasca measurement-list cpu.percent -2880 --dimensions hostname:computenode4 --merge_metrics --offset 11000 but its not working, without offset i get only 10K but i need more | 18:00 |
| iurygregory | The error I get is HTTPException code=500 message={ | 18:00 |
| iurygregory | "title": "The repository was unable to process your request", | 18:00 |
| iurygregory | "description": "invalid operation: time and *influxql.StringLiteral are not compatible" | 18:00 |
| iurygregory | } | 18:00 |
| *** nadya has joined #openstack-monasca | 18:05 | |
| *** nadya has quit IRC | 18:07 | |
| openstackgerrit | Kaiyan Sheng proposed openstack/monasca-agent master: Add Dimensions into White List https://review.openstack.org/459021 | 18:10 |
| *** nadya has joined #openstack-monasca | 18:25 | |
| rbrndt | iurygregory: the api has a configured limit of 10,000 to avoid overwhelming the database. If you want to change this, be warned you may see very long query times. | 18:41 |
| rbrndt | Also, offset is based on time for metric data | 18:41 |
| iurygregory | rbrndt, tks o/ | 18:41 |
| *** nadya has quit IRC | 18:43 | |
| iurygregory | rbrndt, the limit is set in configuration file? | 18:48 |
| rbrndt | It should be | 18:48 |
| *** rbak has joined #openstack-monasca | 18:55 | |
| iurygregory | would be http://git.openstack.org/cgit/openstack/monasca-api/tree/monasca_api/common/repositories/constants.py ? | 18:59 |
| *** dtardivel has quit IRC | 19:00 | |
| rbrndt | That's the default | 19:09 |
| rbrndt | I apologize, I may have led you astray. It appears the python API does not make that value configurable. | 19:17 |
| rbrndt | iurygregory: the value you found may be the only way to change that limit for the moment | 19:17 |
| iurygregory | rbrndt, tks o/ | 19:17 |
| *** nicolasbock has joined #openstack-monasca | 19:20 | |
| nicolasbock | Hi! | 19:20 |
| nicolasbock | How do I enable the tempest tests for monasca? | 19:20 |
| nicolasbock | I mean, what do I need to add to tempest.conf to enable them? | 19:21 |
| rbrndt | nicolasbock I'm sure I understand your question | 19:28 |
| rbrndt | Tempest.conf doesn't define which tests are run, as far as I know | 19:29 |
| nicolasbock | rbrndt, ok | 19:29 |
| nicolasbock | rbrndt, I don't know much about how tempest works | 19:29 |
| nicolasbock | which is why I guess my question is a bit confused :) | 19:29 |
| nicolasbock | I would like the monasca tests to run | 19:29 |
| nicolasbock | when I run | 19:29 |
| nicolasbock | tempest run --smoke | 19:29 |
| rbrndt | Me either, so hopefully we aren't too far off. I can point you at our devstack setup (which also drives the gate) | 19:29 |
| nicolasbock | ok | 19:29 |
| rbrndt | https://github.com/openstack/monasca-api/blob/master/devstack/post_test_hook.sh | 19:30 |
| nicolasbock | great, thanks! | 19:30 |
| nicolasbock | I'll check that out | 19:30 |
| rbrndt | If you have further questions, I might suggest asking in the infra channel how tempest is meant to be run | 19:31 |
| nicolasbock | ok | 19:31 |
| nicolasbock | that's a good idae | 19:31 |
| nicolasbock | I'll read up on the devstack script first | 19:31 |
| nicolasbock | thanks! | 19:31 |
| rbrndt | np | 19:32 |
| *** jamielennox|away is now known as jamielennox | 19:52 | |
| *** rbrndt has quit IRC | 21:02 | |
| *** rbrndt has joined #openstack-monasca | 21:04 | |
| openstackgerrit | Kaiyan Sheng proposed openstack/monasca-agent master: Add feature to report persistent volume capacity https://review.openstack.org/460317 | 21:14 |
| openstackgerrit | Kaiyan Sheng proposed openstack/monasca-agent master: Add feature to report persistent volume capacity https://review.openstack.org/460317 | 21:52 |
| openstackgerrit | Tomasz Trębski proposed openstack/monasca-kibana-plugin master: [WIP] Adjusting to uwsgi deployed keystone https://review.openstack.org/459267 | 22:01 |
| *** rbak has quit IRC | 22:15 | |
| openstackgerrit | Kaiyan Sheng proposed openstack/monasca-agent master: Add feature to report persistent volume capacity https://review.openstack.org/460317 | 22:23 |
| *** bobh has quit IRC | 22:38 | |
| *** rbrndt has quit IRC | 22:52 | |
| *** Fdaisuke_ has quit IRC | 23:36 | |
| *** kei_yama has joined #openstack-monasca | 23:40 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!