16:00:09 <mlavalle> #startmeeting neutron_performance
16:00:10 <openstack> Meeting started Mon Sep 23 16:00:09 2019 UTC and is due to finish in 60 minutes.  The chair is mlavalle. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:00:11 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
16:00:13 <openstack> The meeting name has been set to 'neutron_performance'
16:00:36 <bcafarel> hello!
16:00:43 <rubasov> hi!
16:00:51 <jrbalderrama> hello
16:00:57 <mlavalle> hi there!
16:02:30 <mlavalle> ok, that is probably enough wait
16:02:36 <mlavalle> #topic Updates
16:02:49 <mlavalle> are any updates from the past two weeks?
16:03:35 <rubasov> sorry, I don't have any
16:03:42 <mlavalle> that's cool
16:03:57 <mlavalle> We appreciatte all you have done so far
16:03:58 <bcafarel> only "following someone's poc review" here
16:04:56 <mlavalle> ok, so some of you are aware that over the past few days I continued working with https://review.opendev.org/#/c/678438/
16:05:45 <mlavalle> before I explain what I did, please see if you can download https://storage.gra1.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_e2f/678438/12/check/neutron-rally-task/e2f821c/controller/logs/screen-q-svc.txt.gz
16:06:09 <mlavalle> it is going to take some time
16:07:35 <mlavalle> here's what I did in the last revision of this patch:
16:08:24 <mlavalle> 1) The profiling becomes configurable with options in the default section of neutron.conf
16:09:23 <mlavalle> I added an option named "enable_code_profiling"
16:10:14 <mlavalle> when that is true, profiling will take place in the case of those methods decorated "profile"
16:10:49 <mlavalle> currently just create_port and update_port in the ml2 plugin
16:11:17 <mlavalle> 2) I enabled the profiling for the rally job
16:12:41 <mlavalle> 3) I add the result of each profiled call to the log file so anybody investigating performance only needs to look at the neutron server log in the rally job
16:13:07 <mlavalle> that is why I asked you to download ^^^^
16:14:13 <njohnston_> wow, it is taking a while
16:14:44 <jrbalderrama> 624M !
16:15:06 <mlavalle> 4) At the beginning of each profile report in the log file I also log the corresponding osprofiler id like the one shown here: https://storage.gra1.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_e2f/678438/12/check/neutron-rally-task/e2f821c/results/report.html.gz#/NeutronNetworks.create_and_bind_ports/output
16:15:27 <bcafarel> get it in a shell, it will leave it compressed (42M gz)
16:16:16 <mlavalle> that way we will be able to correlate the cProfile output with the osprofiler output
16:16:18 <rubasov> and a decent distro will decompress it on the fly if you open the gz with less
16:17:01 <njohnston_> or zcat | less, zgrep is also helpful if you know what you're looking for
16:17:15 <mlavalle> if you have downloaded the file search for "cumtime"
16:17:58 <mlavalle> and you will find the first table of profiling results
16:18:09 <mlavalle> does this all make sense?
16:19:00 <bcafarel> so that's the os-profiler table output?
16:19:27 <mlavalle> no if you find cumtime, you will see the cProfiler resutl
16:19:33 <rubasov> and we'll have one table like that for each call of create_port and update_port?
16:19:43 <mlavalle> correct rubasov
16:20:23 <mlavalle> and at the top of each table, the corresponding osprofiler id should be logged as well
16:20:33 <mlavalle> so we can correlate both views
16:20:38 <mlavalle> makes sense?
16:21:10 <njohnston_> +1
16:22:32 <mlavalle> the problem we had with the osprofiler data is that it was difficult to correlate with specific sections of the code
16:23:18 <rubasov> but now we can add the decorator wherever we want it, cool
16:23:39 <mlavalle> with this approach now we can see the osprofiler trace and relate it to the cProfile output and see sections of the code are doing what, who is calling who and who is being called by who
16:25:26 <mlavalle> rubasov: yes, I could remove the decorators in the patch from the ml2 plugin and merge the patch in its current form
16:25:55 <mlavalle> if we do that, the prospective performance investigator would follow these steps:
16:26:15 <mlavalle> 1) decide what portion of the code she / he wants to analyze
16:26:53 <mlavalle> 2) create a DNM patch adding the decorator to those methdos relevant to the investigation
16:27:17 <mlavalle> 3) See the results in the neutron server log of the rally job
16:28:03 <mlavalle> 4) correlate with the osprofiler output. I am looging both the osprofiler parent id and the trace id, so we all can go at whatever level of granularity we want
16:28:36 <mlavalle> I will even add a section to the devref to explain the process
16:28:39 <mlavalle> makes sense?
16:29:06 <bcafarel> I was about to say, noting these steps in dev doc would be great :)
16:31:02 <njohnston_> I have places where I could make use of this right now
16:31:08 <njohnston_> very awesome work
16:31:22 <rubasov> makes perfect sense
16:31:45 <mlavalle> yeah I know you are working on performance, njohnston_ . I've been thinking of you all along
16:32:01 <mlavalle> there is another approach though....
16:32:25 <mlavalle> not exclusive with the previous one. In fact, they can be complementary
16:33:25 <mlavalle> here it goes:
16:33:45 <mlavalle> 1) I can decorate methods like this https://github.com/openstack/neutron/blob/master/neutron/api/v2/base.py#L372
16:34:08 <mlavalle> in the base API controller. I can decorate index, show, create, update and delete
16:34:44 <mlavalle> that way we will have data and all the API calls that we exercise in the rally job for each patch
16:35:10 <mlavalle> 2) I can also decorate the loop methos in the agents
16:35:22 <mlavalle> which are also generating osprofiler traces
16:36:00 <mlavalle> and if, during a performance investigation, we need more granularity, we can go with the previous approach
16:36:12 <mlavalle> makes sense?
16:40:41 <njohnston_> yep, makes sense!
16:40:59 <mlavalle> ok, here's the plan for the next two weeks
16:41:36 <mlavalle> 1) I will experiment decorating the base controller. If infra can take it, that is the way we will go
16:41:52 <mlavalle> 2) I will decorate the agent loop methods
16:42:12 <mlavalle> 3) I will add a section to devref
16:43:47 <mlavalle> 4) after merging this code (after the freeze is over) I will start analyzing data for the create / bind port combo. My goal is to have data and findings to discuss in Shanghai
16:44:17 <mlavalle> and hoepfully you can leverage this by then, njohnston_, so we can also discuss your findings
16:44:32 <njohnston_> yes definitely
16:45:15 <mlavalle> rubasov: with the work you are doing, we will be able to correlate at the agent level as well
16:45:41 <rubasov> cool
16:45:54 <mlavalle> ok, that is the end of my update
16:46:03 <rubasov> will find some time to get the code in better shape
16:46:09 <mlavalle> anything else we should discuss today?
16:49:20 <mlavalle> ok, thanks for attending
16:49:25 <mlavalle> have a great week
16:49:30 <bcafarel> same!
16:49:45 <rubasov> you too
16:50:49 <mlavalle> #endmeeting