05:30:39 <dtruong> #startmeeting senlin
05:30:40 <openstack> Meeting started Fri Sep 14 05:30:39 2018 UTC and is due to finish in 60 minutes.  The chair is dtruong. Information about MeetBot at http://wiki.debian.org/MeetBot.
05:30:41 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
05:30:43 <openstack> The meeting name has been set to 'senlin'
05:31:04 <dtruong> anybody here for the meeting?
05:31:44 <fabian_> Hi，I'm on line.
05:32:13 <dtruong> hi chenyb4
05:32:21 <chenyb4> hi, dtruong
05:33:16 <dtruong> ok, let's get started
05:33:20 <dtruong> #topic announcements
05:34:06 <Qiming> hi
05:34:20 <dtruong> i have send out nominations for 2 new core members in the dev mailing list
05:34:25 <dtruong> #link http://lists.openstack.org/pipermail/openstack-dev/2018-September/134521.html
05:34:42 <dtruong> if you haven't voted yet, you have until monday morning to do so
05:34:44 <dtruong> hi qiming
05:35:34 <dtruong> i also send out a nomination for chenyb4 to join stable maintainers:
05:35:39 <dtruong> #link http://lists.openstack.org/pipermail/openstack-dev/2018-September/134519.html
05:35:56 <dtruong> again voting is open until monday for current stable cores
05:36:37 <dtruong> other news is that PTG is ongoing this week for other projects
05:37:06 <dtruong> i saw a mention of senlin in the K8s SIG etherpad:
05:37:09 <dtruong> #link https://etherpad.openstack.org/p/sig-k8s-2018-denver-ptg
05:38:25 <dtruong> from the notes it appears that they agreed on a common library for nova, heat and senlin
05:38:42 <dtruong> for autoscaling k8s
05:39:24 <dtruong> i'll keep an eye out for any more news on that
05:39:44 <Qiming> that is weird
05:39:53 <Qiming> why a common lib is needed?
05:41:30 <dtruong> not sure.  maybe they want to abstract the service which is performing the autoscaling away from k8s or magnum
05:41:52 <dtruong> i'll try to join the k8s SIG IRC meetings and find out more
05:42:21 <Qiming> that would be great
05:42:31 <dtruong> actually, i think they use slack not irc
05:43:05 <dtruong> yea, i'll keep everyone updated if i find out more
05:43:19 <chenyb4> dtruong, you need use Slack join sig
05:44:35 <dtruong> that's right.  i did join the slack channel before, but i rarely log in because there is a lot of discussions going on
05:45:02 <Qiming> just took a quick glance over the code here: https://github.com/kubernetes/autoscaler/pull/1226/files
05:45:57 <Qiming> It is in pretty early stage
05:46:13 <Qiming> and it is more about interfacing heat from k8s
05:47:17 <dtruong> yea, this one directly uses heat
05:47:53 <dtruong> we'll have to see how the common library fits in with this
05:48:55 <dtruong> ok, let's move on to next topic
05:49:03 <dtruong> #topic blueprint status
05:49:51 <dtruong> i have created a blueprint for fail fast on locked resources
05:49:54 <dtruong> #link https://blueprints.launchpad.net/senlin/+spec/fail-fast-locked-resource
05:50:22 <dtruong> the spec is here: https://review.openstack.org/#/c/598345/
05:50:44 <dtruong> qiming, the spec is already merged but did you have any comments on it?
05:50:53 <Qiming> what's the reason to skip the retry?
05:51:52 <Qiming> ah ... I see
05:51:59 <Qiming> reading the problem description now
05:52:02 <dtruong> two main reasons.  we have seen problems when a lot of API requests are being made
05:52:09 <Qiming> makes sense
05:52:11 <dtruong> and the engine keeps retrying
05:52:22 <Qiming> leave the choice to users again
05:52:52 <dtruong> yes, because we have seen the engine spin at 100% for 1 hour trying to process all the requests on the same cluster
05:53:06 <dtruong> *100% CPU usage
05:53:15 <Qiming> 409 error code is okay
05:53:42 <dtruong> also, i checked AWS autoscaling implementation and it does the same thing
05:53:55 <dtruong> it will return error code if the ASG is already in use
05:54:14 <Qiming> the original assumption was that locks will be freed up pretty soon
05:54:46 <Qiming> our design idea was mostly from JVM lock implementation
05:55:30 <dtruong> we will still keep the retry implemention because it is still possible that 2 actions can arrive at the same time
05:55:54 <dtruong> but this blueprint will prevent hundreds of request queuing up
05:56:35 <Qiming> okay, just wanted to clarify where we are from. Little bit hints here: https://www.ibm.com/developerworks/community/blogs/738b7897-cd38-4f24-9f05-48dd69116837/entry/jvm_under_the_hood_locking_in_ibm_j9_vm2?lang=en
05:57:16 <Qiming> in JVM, J9 specifically, there are three layers of locks for monitors. The inner most one is a spin loop
05:57:30 <Qiming> we borrowed that idea into senlin lock implementation
05:57:50 <Qiming> that doesn't mean the idea applies well to web services
05:58:04 <Qiming> so .. no objections from me regarding the BP
05:58:16 <dtruong> ok, thanks.
05:58:46 <dtruong> btw, i do think the lock implemention is not bad and still useful because we can have multiple engines running
05:59:44 <dtruong> the locking will be handle to the concurrency between two engines operating on the same cluster
06:00:48 <dtruong> ok, the next blueprint is multiple detection modes
06:00:52 <dtruong> #link https://blueprints.launchpad.net/senlin/+spec/multiple-detection-modes
06:01:10 <dtruong> #link https://review.openstack.org/#/c/601471/
06:01:33 <dtruong> please look over it when you get a chance
06:02:21 <dtruong> chenyb4: for the other items you added in https://etherpad.openstack.org/p/senlin-stein-workitems
06:02:29 <dtruong> can you create blueprints when you have time?
06:02:46 <chenyb4> ok
06:02:56 <dtruong> thanks
06:03:59 <dtruong> #topic stein goal: python 3
06:04:21 <dtruong> this is on-going.
06:04:52 <dtruong> a few patches for the tox environments were proposed by the python 3 champions and merged.
06:05:10 <Qiming> okay ...
06:05:12 <dtruong> any other updates on that topic, chenyb4
06:05:50 <Qiming> I believe when we started senlin we aimed to support python3 and we aimed to support keystone v3 only
06:06:46 <dtruong> i believe senlin runs fine with python 3.  all the unit tests and functional tests pass in python 3 environment
06:07:11 <Qiming> cool
06:07:15 <dtruong> so there shouldn't be much work for us to complete this goal
06:07:29 <Qiming> good to know
06:08:08 <chenyb4> yes, most work in python3 was finish.
06:08:37 <dtruong> ok, thanks
06:09:20 <dtruong> the other stein community goal is upgrade checks
06:09:31 <dtruong> #topic stein goal upgrade checks
06:09:35 <dtruong> #link https://governance.openstack.org/tc/goals/stein/upgrade-checkers.html
06:10:09 <dtruong> we still need a volunteer for this. if anyone is interested, please let me know
06:10:44 <dtruong> i probably also will send out an email to the dev mailing list to see if anyone is intestered on working on this
06:11:48 <chenyb4> I am watching, but I have not started working yet.
06:12:16 <dtruong> on the upgrade checkers?
06:12:18 <Qiming> chenyb4, don't be shy when getting your hands dirty
06:12:29 <Qiming> we all do
06:12:32 <Qiming> :D
06:13:32 <chenyb4> ok,
06:13:32 <chenyb4> I am looking at the implementation code of nova.
06:13:57 <Qiming> just checked https://github.com/gophercloud/gophercloud/tree/master/openstack/clustering/v1, it looks clustering support is all in
06:14:06 <Qiming> amazing
06:14:28 <dtruong> oh, we (blizzard) added that support in gophercloud
06:14:44 <dtruong> we also plan on adding terraform provider support for senlin
06:15:06 <eandersson> o/
06:15:24 <Qiming> \o/
06:15:28 <dtruong> o/
06:16:44 <dtruong> for the upgrade checks, the only potential problem that i can think of this upgrades to policies
06:16:56 <Qiming> I'm working on an internal workflow orchestrator recently
06:17:24 <dtruong> for example if we go from health policy v1.0 to v1.1 and the schema changes between the two versions
06:17:29 <Qiming> this orchestrator currently understands some VMware language
06:18:03 <Qiming> as the next step, I'm gonna teach it how to speak terraform, how to speak k8s etc.
06:18:48 <dtruong> cool. terraform is a common tool for cloud operators
06:18:51 <Qiming> I believe we have some version checks in place
06:19:51 <Qiming> the framework has room for improvement
06:20:22 <Qiming> I was thinking whether we can use oslo.versionedobject to manage the versioning of profiles and policies
06:21:05 <Qiming> if we do that, we can get rid of the current schema parsing and version checking etc.
06:21:38 <Qiming> it means we will be managing api requests/responses and resource representations all using the same technology
06:21:48 <Qiming> for versioning
06:22:14 <dtruong> that's not a bad idea
06:22:33 <eandersson> I like that
06:22:59 <dtruong> it's a pretty big change but it would simplify the senlin code a lot
06:23:49 <dtruong> ok, let me put it down as an investigation item
06:24:14 <dtruong> #action investigate using oslo.versionedobject for profiles and policies
06:24:43 <dtruong> ok, we have a few minutes left
06:24:46 <dtruong> #topic reviews
06:25:18 <eandersson> Lets get the python 3.x test merged
06:25:18 <dtruong> anybody have open patch sets that you would like to get looked at?
06:25:24 <eandersson> *fix
06:25:28 <eandersson> for rocky
06:25:51 <eandersson> #link https://review.openstack.org/#/c/600199/
06:26:41 <dtruong> looks like we got +2 from qiming, so i will go ahead and add +1 workflow
06:27:34 <eandersson> sweet
06:27:45 <dtruong> ok, if there no other reviews, we can open the floor for discussion
06:27:48 <dtruong> #topic open discussion
06:28:14 <dtruong> anybody have anything they would like to discuss?
06:28:30 <chenyb4> nothing
06:29:21 <dtruong> ok, i'll end the meeting then.
06:29:27 <dtruong> thanks everybody for attending
06:29:31 <Qiming> thanks guys
06:29:37 <chenyb4> thaks
06:29:47 <dtruong> #endmeeting