15:30:03 <DinaBelova> #startmeeting Performance Team
15:30:04 <openstack> Meeting started Tue Jan  9 15:30:03 2018 UTC and is due to finish in 60 minutes.  The chair is DinaBelova. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:30:06 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
15:30:08 <openstack> The meeting name has been set to 'performance_team'
15:30:27 <DinaBelova> okay, let's see if someone has survived the holiday season
15:31:52 <DinaBelova> tovin07 o/
15:33:00 <DinaBelova> ilyashakhat o/
15:34:34 <tovin07> o/
15:34:56 <DinaBelova> wow, I see a survivor :)
15:35:01 <DinaBelova> tovin07 I have a feeling we'll be alone today
15:35:02 <DinaBelova> :D
15:35:28 <tovin07> I saw that ilyashakhat publish a patch a few minutes ago in cinder
15:35:36 <tovin07> hope that he can join us
15:36:17 <DinaBelova> okay, let's wait for a few more moments
15:37:17 <ilyashakhat> hi!
15:37:20 <ilyashakhat> I'm here
15:37:24 <DinaBelova> yay!
15:37:31 <DinaBelova> good to see both of you :)
15:37:35 <DinaBelova> okay, let's proceed
15:37:43 <DinaBelova> no action items for a while
15:37:50 <DinaBelova> so switching to the current status
15:37:55 <DinaBelova> #topic Current status
15:38:06 <DinaBelova> tovin07 how are you doing sir? anything to share?
15:38:26 <tovin07> yes
15:38:28 <tovin07> a little bit
15:39:10 <tovin07> when testing ilyashakhat's patch https://review.openstack.org/#/c/519664/
15:39:29 <tovin07> I found an issue with OSProfiler in Nova
15:39:46 <tovin07> http://paste.openstack.org/show/641262/
15:40:28 <tovin07> and it is related to this change (from very long time ago) https://review.openstack.org/#/c/254703/20/nova/utils.py
15:41:02 <tovin07> line 1174 - 1175 from `spawn` function
15:41:08 <ilyashakhat> I also so this trace couple of times..
15:41:37 <DinaBelova> it's interesting :) I can swear that 11 months ago it used to work :D
15:41:45 <tovin07> yes
15:41:53 <ilyashakhat> for me the issue happens rarely
15:42:13 <DinaBelova> tovin07 ilyashakhat it's quite interesting
15:43:02 <DinaBelova> as obviously what the lines you've tovin07 mentioned is doing is just initiating profiler if not context with it is in place
15:43:07 <tovin07> as I see here, because we re-initialize osprofiler when it is already initialized will clear current profiler object
15:43:31 <tovin07> --> the self._name queue will be empty
15:43:40 <DinaBelova> oh, yeah, it might be the reason
15:43:57 <tovin07> I think we should have `a check` here
15:44:04 <DinaBelova> tovin07 can you provide a patch?
15:44:12 <DinaBelova> it looks like minor one
15:44:12 <tovin07> yes, I will
15:44:24 <DinaBelova> cool
15:44:26 <DinaBelova> thank you
15:44:51 <ilyashakhat> why this re-initialization is needed?
15:45:33 <DinaBelova> ilyashakhat frankly speaking I'm not sure - patch ir really old and I do not remember the details
15:45:51 <tovin07> Boris Pavlovic
15:45:51 <tovin07> Jan 19, 2016
15:45:51 <tovin07> Patch Set 19: Code-Review-1
15:45:51 <tovin07> We need to handle green threads as well, like here:
15:45:51 <tovin07> https://github.com/boris-42/nova/commit/6f81fbc45db8ed02c723e534f4a0780845e4d1a8
15:45:59 <tovin07> that's the reason
15:46:25 <DinaBelova> additionally iyour link is 404 not found to me :)
15:46:29 <tovin07> after this comment, DinaBelova adds these code to the utils.py file
15:46:38 <tovin07> yes, it 404 for now
15:46:59 <tovin07> I cannot find any cache of that link on the web too
15:47:04 <DinaBelova> yeah, I se this comment
15:47:36 <DinaBelova> I guess it was not accurate change introduced occasionally without a check
15:48:19 <DinaBelova> thanks tovin07 for finding it
15:48:37 <ilyashakhat> btw, profiler state is defined as threading.local() - does it mean to stay per green thread too?
15:49:17 <ilyashakhat> (that may be a reason why re-initialization is needed)
15:49:49 <DinaBelova> it was supposed to be so
15:51:42 <DinaBelova> tovin07 anything else to add?
15:51:46 <ilyashakhat> so the issue in Nova may occur when there is already some profiling data and then a new thread is forked
15:51:59 <DinaBelova> ilyashakhat yes, it looks like this
15:52:04 <ilyashakhat> so we need to properly 'fork' profiler state too
15:52:13 <DinaBelova> and this is happening not too often
15:52:28 <DinaBelova> that's why we do not see it too much
15:52:39 <ilyashakhat> yeah...
15:52:55 <ilyashakhat> let's move on
15:52:58 <DinaBelova> ack
15:53:04 <tovin07> ah
15:53:12 <DinaBelova> tovin07 anything else to share?
15:53:41 <tovin07> I see this happen with `openstack server create` and some other commands
15:54:30 <DinaBelova> tovin07 ack
15:54:34 <DinaBelova> ilyashakhat let's proceed with you :) anything to share with us? how did your holidays go?
15:55:08 <ilyashakhat> holidays went well :) so I don't have much to share ;)
15:55:18 <DinaBelova> ilyashakhat :D
15:55:24 <DinaBelova> that's good too :D
15:55:26 <ilyashakhat> I've posted a patch to Cinder to fix initialization of WSGI
15:55:46 <ilyashakhat> similar patch in Nova still hangs
15:55:48 <tovin07> yes
15:56:01 <tovin07> i tested your cinder patch too
15:56:02 <DinaBelova> yeah, I see it
15:56:17 <DinaBelova> nova patch is quite old...
15:56:29 <DinaBelova> well, Nova review queue sometimes may go crazy
15:56:48 <tovin07> however, I see that traces from cinder are quite short
15:56:56 <DinaBelova> although for 2 lines patch it looks a bit overkill
15:57:07 <tovin07> DinaBelova, yes
15:57:33 <ilyashakhat> tovin07, yes, I see only wsgi events -- looks like there's some more issue
15:58:57 <DinaBelova> it looks like new year brought more gifts than expected :)
15:58:58 <DinaBelova> more bugs :D
15:59:05 <DinaBelova> from my side nothing specific to share
15:59:21 <tovin07> I will reach nova cores in nova channel to get another +2 and A+1 for ilyashakhat patch
15:59:30 <DinaBelova> tovin07 thanks
15:59:32 <ilyashakhat> thanks!
15:59:37 <DinaBelova> #topic Open Discussion
16:00:02 <DinaBelova> okay, so tovin07 ilyashakhat - our brave holiday survivors - anything to share in addition?
16:00:06 <DinaBelova> :)
16:00:11 <ilyashakhat> I have :)
16:00:15 <DinaBelova> ilyashakhat shoot
16:00:27 <ilyashakhat> I'm going to finish https://review.openstack.org/#/c/521385/
16:00:50 <tovin07> thanks ilyashakhat
16:01:00 <DinaBelova> yaaay
16:01:01 <ilyashakhat> so we will have `trace list` command
16:01:07 <DinaBelova> it's good to hear
16:01:22 <ilyashakhat> DinaBelova, do we have any deadline to release and be included into Q?
16:01:42 <ilyashakhat> (if it's not too late already)
16:02:06 <DinaBelova> well, we did not have too many changes
16:02:08 <tovin07> usually no, but osprofiler is under oslo umbrella
16:02:18 <DinaBelova> yeah
16:02:30 <tovin07> therefore, it will depend on oslo release cycle
16:03:05 <ilyashakhat> Final non-client library release deadline: January 18
16:03:18 <DinaBelova> ilyashakhat so we have some time
16:03:20 <DinaBelova> a bit but still
16:03:25 <ilyashakhat> ok
16:04:21 <DinaBelova> thanks ilyashakhat
16:04:26 <DinaBelova> anything else?
16:05:03 <tovin07> nope
16:05:03 <tovin07> :D
16:05:06 <DinaBelova> it looks like nope :) if so, thanks tovin07  and ilyashakhat for showing up :)
16:05:06 <ilyashakhat> nope
16:05:14 <DinaBelova> have a nice evening!
16:05:17 <DinaBelova> #endmeeting