15:29:51 <DinaBelova> #startmeeting Performance Team
15:29:52 <openstack> Meeting started Tue Apr  4 15:29:51 2017 UTC and is due to finish in 60 minutes.  The chair is DinaBelova. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:29:53 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
15:29:56 <openstack> The meeting name has been set to 'performance_team'
15:30:02 <DinaBelova> hello folks!
15:30:20 <rcherrueau> o/
15:30:41 <DinaBelova> rcherrueau akrzos o/
15:30:47 <DinaBelova> I have a strange feeling we won't have too many people today :)
15:31:25 <DinaBelova> rcherrueau let's wait for a few moments :)
15:31:33 <rcherrueau> yep
15:31:34 <DinaBelova> it might be a chance we're alone today :D
15:31:40 <rcherrueau> :D
15:32:34 <tovin07> o/
15:32:41 <DinaBelova> tovin07 o/
15:32:52 <DinaBelova> ok, so there are three of us :D me, tovin07 and rcherrueau :D
15:33:01 <tovin07> ýe
15:33:06 <tovin07> ýeyes
15:33:06 <rcherrueau> tovin07: o/
15:33:14 <tovin07> rcherrueau: hi
15:33:25 <DinaBelova> I think we need to get started (no action items from last meeting, so we can jump to current progress)
15:33:33 <DinaBelova> #topic Current progress on the planned tests
15:33:44 <DinaBelova> rcherrueau the floor is yours sir :)
15:33:46 <DinaBelova> any news?
15:34:19 <rcherrueau> This weeks I spent most of time implementing a new feature in Enos that lets you define a set of OpenStack characteristics
15:34:42 <rcherrueau> eg, different number of compute nodes: 10, 100, 1000; network delay between groups of nodes: 50ms, 100ms ; topology of services location ...
15:35:02 <rcherrueau> The idea, then, is to automatically build all possible configurations by sweeping on theses characteristics and run Enos with each configuration
15:35:20 <DinaBelova> rcherrueau btw, is it possible to add to perf docs the methodology you're using to get the testing with network delays?
15:35:26 <rcherrueau> ie, first run with 10 computes and a network delay of 50ms, second run with 100 computes and a network delay of 50 ms, ..., sixth run with 1000 computes and a network delay of 100 ms.
15:35:43 <DinaBelova> I mean, I know you're going to do that anyway for summit
15:35:52 <rcherrueau> DinaBelova: oh yes, can you put an action for that?
15:35:58 <DinaBelova> but I wonder when you're planning to add this to the docs
15:35:59 <DinaBelova> sure
15:36:41 <DinaBelova> #action rcherrueau add OpenStack testing under networking delays (e.g. multisite deployment) methodology to performance docs
15:36:42 <DinaBelova> thanks
15:36:59 <rcherrueau> With this new feature it let us easily define a large set of configurations and let Enos do the performance evaluation in the background.
15:37:00 <DinaBelova> rcherrueau as for the feature - got your point
15:37:14 <DinaBelova> I believe for several tests it's going to be quite convinient
15:37:22 <rcherrueau> yes
15:37:40 <rcherrueau> And that's all on my side
15:37:41 <DinaBelova> anything else to share?
15:37:42 <DinaBelova> ok
15:37:45 <DinaBelova> ok, from my side
15:37:58 <DinaBelova> several test plans / tets results have been uploaded on review
15:38:22 <DinaBelova> #info ETCD health in k8s clouds under load https://review.openstack.org/451892
15:38:41 <DinaBelova> #info kube-proxy performance evaluation https://review.openstack.org/452257
15:38:54 <DinaBelova> #info Ceph RBD performance testing https://review.openstack.org/452246
15:39:11 <DinaBelova> #info containerized OpenStack upgrades testing https://review.openstack.org/451419
15:39:30 <DinaBelova> there were other changes as well, but I mentioned them last meeting
15:40:03 <DinaBelova> and I think that's all for now from my side :)
15:40:06 <rcherrueau> DinaBelova: Did you find something interesting? I mean if you compare with previous tests outside of container
15:40:41 <DinaBelova> rcherrueau I'd say we need to dig deeper to the upgrades if that will be possible next testing cycle
15:40:45 <DinaBelova> and the reliability
15:40:56 <DinaBelova> I believe k8s as a platform can provide better results
15:41:01 <DinaBelova> than the ones we've seen
15:41:18 <DinaBelova> in terms of control plane availability during those operations
15:41:34 <rcherrueau> DinaBelova: OK got it
15:42:10 <DinaBelova> rcherrueau although that will take some additional effort - we need to understand what the impact on our schedule and tasks that will be
15:42:42 <DinaBelova> there might be a chance our team will be switched a bit more to some internal tasks in future (no clarity so far)
15:42:55 <DinaBelova> so further tasks discussion is quite hot right now :D
15:43:08 <rcherrueau> DinaBelova: ack
15:43:22 <openstackgerrit> David Burnazyan proposed openstack/performance-docs master: Ceph RBD test results  https://review.openstack.org/452246
15:43:29 <openstackgerrit> Merged openstack/performance-docs master: Indicating the location tests directory in oslo_debug_helper  https://review.openstack.org/443530
15:43:41 <DinaBelova> and as only Inria and Mirantis are presented today (in terms of testing :D) we may go further :D
15:43:47 <DinaBelova> #topic Open Discussion
15:43:47 <rcherrueau> DinaBelova: I will take a look later, but could you explain a little bit how your perform your upgrade testing?
15:43:57 <DinaBelova> rcherrueau sure
15:44:02 <rcherrueau> how you*
15:44:04 <tovin07> this week
15:44:16 <tovin07> I continue to work on some OSprofiler patch
15:44:38 <tovin07> however, there is no other comment on SQL/function result patch
15:44:41 <DinaBelova> tovin07 link please :)
15:44:57 * tovin07 looking
15:45:14 <DinaBelova> rcherrueau ^^ I believe you guys promised to take a look on SQL/function result patch
15:45:37 <rcherrueau> tovin07: I take a look and wanna try it, but my osprofiler still not works when I go with a kolla-deploymnet :(
15:45:45 <tovin07> #link https://review.openstack.org/#/c/450072/
15:46:13 <tovin07> rcherrueau: the issue with ceilometer?
15:46:48 <rcherrueau> tovin07: However I look at the example you provide into your commit message and it's exactly what I need!
15:47:05 <rama_y__> Hello, I would appreciate taking a look at this patch: https://review.openstack.org/#/c/358142/
15:47:35 <tovin07> rama_y: long time no see :D
15:47:41 <rcherrueau> tovin07: Actually, last week you advice me to look into RMQ queues.
15:47:43 <rama_y__> I was able to get trace output sometime back; now, I am not getting the trace with the same code.
15:47:56 <rama_y__> Hi tovin07
15:48:05 <rama_y__> Hi DinaBelova
15:48:21 <DinaBelova> rcherrueau as for the upgrades. We were testing major upgrade from Mitaka to Newton using fuel-ccp. Overall process is described as "use new OpenStack services image and redeploy needed containers". At the same time we were running rally tests to check the control plane availablity. Right now we're doing to the same with shaker to check data plane (some issues with heat observed)
15:48:24 <tovin07> rama_y__: do you know what may be wrong?
15:48:27 <rcherrueau> tovin07: After inspecting this, it appears that there are no messages related to OSProfiler in the queue.
15:48:41 <DinaBelova> rama_y__ o/
15:48:42 <DinaBelova> nice to see you around againt
15:48:56 <rama_y__> I think ceilometer
15:49:31 <rcherrueau> tovin07: So, I set up manually the value of `messaging` into `[profiler]` to `rabbitmq://` but still nothing
15:49:32 <tovin07> rama_y__: did you try with other drivers? such as: redis or mongodb
15:49:49 <tovin07> rcherrueau: rabbitmq:// will not work
15:49:49 <rama_y__> tovin07: no
15:50:37 * rcherrueau look into its configuration file
15:50:41 <tovin07> rcherrueau: messaging:// means that OSprofiler will use default message queue that define to work with oslo.messaging
15:51:03 <rcherrueau> tovin07: sorry I wanna say: `connection_string = messaging://`
15:51:10 <tovin07> rama_y__: let give them (redis, mongodb) a try and figure out why
15:51:29 <tovin07> rcherrueau: default connection_string is set to messaging://
15:51:38 <rcherrueau> tovin07: However, I figured out that there is no osprofiler filter in the keystong-paste.ini file  during a kolla-ansible deployment
15:51:48 <tovin07> Oh
15:51:58 <DinaBelova> rcherrueau ok, so if with messaging:// there is nothing in queues, it means the error is earlier in the stack. the most common case is some misconfiguration of the profiler on the OpenStack service-level
15:51:58 <rcherrueau> tovin07: Maybe this is the root of my problem?
15:52:09 <tovin07> yes
15:52:18 <DinaBelova> rcherrueau it surely might be :) check all those files pleasef
15:52:41 <DinaBelova> for all services you'd like to have profiling from
15:52:47 <rcherrueau> It's ok for nova, glance, ... but not for keystone
15:52:49 <DinaBelova> missing of osprofiler there mean no WSGI middlware will be added -> no profiling
15:53:35 <tovin07> which version of keystone that you use when kolla deploy it?
15:53:50 <rcherrueau> tovin07: master
15:54:05 <rcherrueau> but kolla override this file
15:54:10 <tovin07> oh
15:54:43 <tovin07> so, there’s a problem with kolla in this case
15:54:59 <tovin07> kolla should keep osprofiler filter
15:55:28 <rcherrueau> But still no sure this is the root of my problem because nova api-paste.ini, glance, ... are correctly set
15:56:23 <rcherrueau> Anyway I can change it and perform a new deployment, then see if it works or not.
15:56:30 <tovin07> did you try to get a trace with nova or glance client?
15:57:15 <rcherrueau> tovin07: yes, not directly with nova client but something like `openstack server create ... --os-profile SECRET_KEY`
15:57:57 <tovin07> that command is correct, as I see
15:58:10 <rcherrueau> DinaBelova: Sorry, maybe this is not the right time and place to discuss about that.
15:58:22 <DinaBelova> rcherrueau it's ok :)
15:58:39 <tovin07> rcherrueau: it’s ok :D
15:58:42 <DinaBelova> we're here exactly for those discussions :)
15:59:02 <tovin07> rama_y__: do you have any (more) information about your case?
16:00:19 <tovin07> rcherrueau: you can change driver to redis or mongodb to debug/trouble-shoot it easier
16:00:56 <rcherrueau> tovin07: maybe you should proceed with patches you wanna talk about at start of open discussion. I will try to update the keystone-paste.ini and see what is happening. I will also try with mongodb
16:00:58 <rama_y__> tovin07, no; I will retry the testing and will let you know.
16:01:08 <rcherrueau> tovin07: DinaBelova: thanks for your help
16:01:17 <DinaBelova> rcherrueau no problem sir
16:01:20 <DinaBelova> you're welcome
16:01:21 <tovin07> because osprofiler flow with those drivers is simpler than flow of ceilometer case
16:01:29 <tovin07> rcherrueau: no problem at all
16:01:52 <rama_y__> Thanks tovin07, DinaBelova
16:01:53 <DinaBelova> ok, anything else to cover today?
16:02:00 <DinaBelova> rama_y__ you're welcome
16:02:02 <tovin07> rcherrueau: about sql/function result
16:02:07 <rcherrueau> tovin07: should I change the value of `connection_string` if I go with mongodb?
16:02:15 <tovin07> the demo fits your need
16:02:33 <tovin07> rcherrueau: yes, change it to something like: mongodb://localhost:27017
16:02:43 <tovin07> change localhost to your mongodb IP
16:02:52 <rcherrueau> tovin07: OK thanks.
16:03:06 <tovin07> and for redis: redis://localhost:6379
16:04:02 <rcherrueau> OK got it
16:04:24 <tovin07> rama_y__: you should recheck the configuration files, check the rabbitmq profiler.info queue, check mysql in panko schema
16:04:41 <DinaBelova> rcherrueau tovin07 rama_y__ - ok, it looks like we went through the progress and issues
16:04:52 <tovin07> If you see any trace info in it, it means that osprofiler worked
16:05:11 <rcherrueau> tovin07: And then, if I want the result with the osprofiler cli, I guess I have to link mongodb also?
16:05:19 <rama_y__> Ok, thanks.
16:05:40 <tovin07> rcherrueau: yes, add —connection-string
16:05:45 <tovin07> DinaBelova: :D
16:05:52 <rcherrueau> tovin07: OK thanks a lot.
16:05:56 <DinaBelova> tovin07 yeah, I see :D
16:06:04 <DinaBelova> tovin07 all of the details :)
16:06:34 <tovin07> at the end
16:06:42 <DinaBelova> tovin07 yes sir?
16:07:17 <DinaBelova> tovin07 anything else?
16:07:18 <tovin07> rcherrueau: could you review some infor in my patch about sql/function result? (closely)
16:07:35 <tovin07> then I will update and make that patch better
16:07:51 <tovin07> DinaBelova: end for now :D
16:08:05 <DinaBelova> rcherrueau I may add and action item on you :D
16:08:11 <DinaBelova> for the review :D
16:08:20 <DinaBelova> so next time you'll have to get it reviewed :D
16:08:30 <rcherrueau> tovin07: yes I can, but do not expect review on How you implement it, because I don't know how osprofiler works under the hood
16:09:15 <tovin07> rcherrueau: so, let look at the representation of results
16:09:16 <DinaBelova> rcherrueau true, but it'll be cool to get it reviewed anyway - inria was requesting this feature :)
16:09:17 <rcherrueau> tovin07: I can comment the output. Is it OK for you
16:09:23 <tovin07> Yes
16:09:27 <DinaBelova> #action rcherrueau review tovin07 's osprofiler patch with sql/function result https://review.openstack.org/#/c/450072/
16:09:46 <rcherrueau> OK Good
16:10:03 <DinaBelova> ok, thank you folks for participating
16:10:06 <DinaBelova> see you next week
16:10:09 <DinaBelova> bye!
16:10:11 <tovin07> bye
16:10:12 <rcherrueau> bye
16:10:14 <DinaBelova> #endmeeting