13:00:43 <yanyanhu> #startmeeting senlin
13:00:44 <openstack> Meeting started Tue Jul  5 13:00:43 2016 UTC and is due to finish in 60 minutes.  The chair is yanyanhu. Information about MeetBot at http://wiki.debian.org/MeetBot.
13:00:45 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
13:00:47 <openstack> The meeting name has been set to 'senlin'
13:00:52 <yanyanhu> hi
13:00:56 <elynn> Hi!
13:00:57 <Qiming> hi
13:01:07 <haiwei_> hi
13:01:08 <yanyanhu> hi :)
13:01:11 <yanyanhu> Qiming, you are here
13:01:16 <Qiming> seems so
13:01:21 <Qiming> watching
13:01:21 <yanyanhu> :P
13:01:34 <yanyanhu> free to hold the meeting?
13:01:45 <Qiming> pls go ahead
13:01:49 <yanyanhu> ok
13:01:53 <yanyanhu> https://wiki.openstack.org/wiki/Meetings/SenlinAgenda#Weekly_Senlin_.28Clustering.29_meeting
13:02:03 <yanyanhu> plz feel free to add items to agenda
13:02:13 <yanyanhu> #topic newton workitem
13:02:29 <yanyanhu> ok, lets go through newton workitem first
13:02:30 <yanyanhu> https://etherpad.openstack.org/p/senlin-newton-workitems
13:02:41 <yanyanhu> no new progress in testing I think
13:02:52 <yanyanhu> but the functional test migration has been done
13:03:15 <lixinhui_> hi
13:03:53 <yanyanhu> so will remove this item from etherpad
13:03:54 <yanyanhu> I guess also no progress in performance test
13:03:54 <yanyanhu> so lets skip it
13:04:01 <yanyanhu> hi, lixinhui_
13:04:15 <yanyanhu> next item is about HA
13:04:35 <yanyanhu> hi, lixinhui_, so you mentioned the fencing part has been done last week?
13:04:43 <lixinhui_> yes
13:04:56 <lixinhui_> manually set up
13:05:21 <lixinhui_> need to consider how to automatically enable the scenario
13:05:29 * Qiming raises his thumb, both ...
13:05:44 <lixinhui_> and the bug of nuetron-lbaas...
13:05:45 <yanyanhu> Great. Since we have proposed a topic about end-to-end HA solution based on Senlin HA to summit, we may need to finish some basic support to build a PoC for it
13:05:53 <lixinhui_> still need some time to get review
13:06:00 <yanyanhu> lixinhui_, yes, I saw you have assigned the bug to yourself :)
13:06:06 <yanyanhu> yes
13:06:30 <yanyanhu> hi, Qiming, any idea about this work item
13:06:43 <yanyanhu> or something we need to pay attention
13:06:43 <Qiming> it is fine
13:06:58 <Qiming> we may need a full story
13:07:16 <Qiming> even though we still have something missing
13:07:19 <yanyanhu> yes, full story is very important, not only for a demo
13:07:39 <yanyanhu> https://blueprints.launchpad.net/senlin/+spec/support-health-management-customization
13:07:46 <yanyanhu> so this etherpad is about HA design
13:08:08 <yanyanhu> https://etherpad.openstack.org/p/senlin-ha-recover
13:09:06 <yanyanhu> something is still missing
13:09:19 <yanyanhu> especially about the health policy
13:09:37 <Qiming> we can have a basic policy poc before summit
13:09:48 <yanyanhu> yes
13:09:57 <yanyanhu> that's a basic goal I think
13:10:16 <yanyanhu> not very difficult I guess
13:10:18 <Qiming> a broader topic may involve mistral
13:10:32 <Qiming> which should be discussed in section 4.1
13:10:39 <yanyanhu> to control the workflow of node recovery?
13:11:04 <Qiming> yes
13:11:29 <Qiming> there might be some out-of-band resources that need to be taken care of
13:11:37 <yanyanhu> I see.
13:11:43 <lixinhui_> agree
13:12:22 <yanyanhu> another item I'm still not clear is about application/service deployed inside VM
13:12:42 <yanyanhu> will senlin handle their failure as well?
13:13:00 <yanyanhu> by watching lbaas member status e.g.
13:13:20 <lixinhui_> I think so
13:13:37 <lixinhui_> we should provide listening to the event of member status change
13:13:58 <yanyanhu> it is possible to support app HA partially
13:14:16 <lixinhui_> if the lbaas service is created by senlin
13:14:32 <zzxwill> For all different kinds of applications/services?
13:14:40 <Qiming> not an easy job, but we, as always, can do our best for failure detection
13:14:59 <lixinhui_> difficult for all ... I think
13:15:00 <yanyanhu> sorry, just dropped
13:15:18 <haiwei_> detect  application's failure?
13:15:19 <Qiming> another thought on this -- maybe a thread in parallel --- is to setup guest level Linux HA cluster
13:15:30 <zzxwill> Is there a layer controlled by senlin which can detect their status?
13:16:00 <Qiming> suppose an application has been protected using Linux HA before, migrating them to Senlin cluster should be an easy job
13:16:28 <Qiming> zzxwill, a generic solution for application failure detection is almost impossible
13:17:11 <yanyanhu> so maybe we build a full HA story with typical use case and then figure out how senlin support it?
13:17:25 <Qiming> yes
13:17:58 <zzxwill> Thanks. lixinhui and Qiming.
13:18:03 <yanyanhu> if there are some gap, we know what part is missing in Senlin's HA mechanism
13:18:04 <Qiming> for example, in some domains, openais has been planted into the application code, so that reliable failure detection becomes a possibility
13:18:19 <yanyanhu> Qiming, yes
13:19:14 <yanyanhu> actually, I think whether senlin can support the fail over of an app also depends on how app expose their user interface
13:19:22 <Qiming> we can start with some wireframe diagram
13:20:52 <yanyanhu> yes. we need more detailed design and complete the proposal which is just a draft now
13:21:01 <Qiming> based on such a draft we can identify where it makes sense for senlin to play a role, and where it makes sense to reuse/integrate with existing technology/solutions
13:21:13 <yanyanhu> agree
13:21:23 <yanyanhu> to decide our scope
13:21:37 <Qiming> we have quite some piece technologies to integrate, to massage ...
13:21:40 <yanyanhu> and then we can start more concrete work
13:21:50 <yanyanhu> and decide their priority
13:22:01 <yanyanhu> yes, actually I just try to add zaqar driver to senlin
13:22:06 <yanyanhu> but met lots of problem
13:22:31 <yanyanhu> will talk about this issue a bit more later
13:22:34 <Qiming> right, so ... comment on the etherpad would be very helpful: https://etherpad.openstack.org/p/senlin-ha-recover
13:22:47 <Qiming> ok
13:23:17 <yanyanhu> Qiming, will you be free for a call this week or next week before tuesday?
13:23:29 <Qiming> yes, I think so
13:23:49 <yanyanhu> great, then I will arrange a call for some discussion about this topic
13:24:28 <Qiming> pls involve everyone who feels interested in this
13:24:29 <yanyanhu> ok, anything else about HA?
13:24:33 <yanyanhu> sure
13:24:39 <yanyanhu> will send out the invitation
13:24:41 <zzxwill> Great.
13:24:58 <Qiming> the listener implementation has some flaws
13:25:00 <yanyanhu> #action yanyanhu send out invitation about the call for HA topic discussion
13:25:16 <yanyanhu> ok, lets move on
13:25:17 <Qiming> I was seeing some exceptions thrown from oslo.messaging occasionally
13:25:26 <Qiming> will dig more into it
13:25:30 <yanyanhu> #topic profile for docker
13:25:58 <yanyanhu> hi, haiwei_ , any new progress about it?
13:26:08 <haiwei_> I have no progress on container jobs, this week
13:26:21 <yanyanhu> big network latency here...
13:26:29 <haiwei_> honestly, I am a little confused what to do the next
13:26:31 <yanyanhu> Qiming, great, thanks :)
13:26:40 <yanyanhu> haiwei_, yes?
13:27:33 <haiwei_> the network is not good to me, either
13:27:47 <Qiming> pour your confusion here, haiwei_
13:28:48 <haiwei_> I have added container create/delete functions to Senlin, maybe container_list , and after that what should we do?
13:28:58 <haiwei_> to support storage and network?
13:29:23 <Qiming> think from a user's perspective
13:30:18 <haiwei_> or support cluster actions for container first?
13:30:53 <Qiming> either way
13:31:29 <yanyanhu> This is really a big topic I think. Maybe we first finish the work we have done as PoC in last summit?
13:31:36 <Qiming> supporting cluster actions doesn't sound a great challenge, IIUC
13:31:49 <yanyanhu> including the docker driver/profile and a basic placement policy for it
13:32:20 <Qiming> however, if we are running some workloads, we will find that we will need to provision storage/network for some basic services
13:33:09 <Qiming> right, "scheduling" is another topic to work on
13:34:01 <haiwei_> I am still concern about the container's host when thinking about cluster actions, because things are different when containers are in one host or different hosts
13:34:28 <Qiming> that is not senlin's scope
13:34:45 <Qiming> we will just leverage whatever existing technology to provision the network
13:35:40 <yanyanhu> oh, BTW, in latest docker release, more and more functionality has been integrated into the docker-engine, including the orchestration
13:35:44 <Qiming> the difference should be masked from users
13:35:49 <Qiming> because ... they don't care
13:35:59 <yanyanhu> this is not a bad news I think
13:36:27 <yanyanhu> that means relying on less 3rd service/tools to support container cluster
13:36:32 <yanyanhu> imo
13:37:04 <yanyanhu> s/3rd/3rd part
13:37:14 <haiwei_> docker will use docker swarm to support container cluster I think
13:37:32 <yanyanhu> haiwei_, in latest 1.12, swarm has been the history :)
13:37:40 <yanyanhu> it is part of docker engine now I think
13:37:40 <zzxwill> Yes, I heard it from your WeChat.
13:37:44 <Qiming> so ... my general feeling is ... sometimes we are too easy to be brain washed
13:37:47 <haiwei_> ok, really
13:38:01 <lixinhui_> I am not a docket kong
13:38:02 <yanyanhu> haiwei_, yes, they call it docker orchestration
13:38:11 <Qiming> to understand the reality, we have to try it out
13:38:14 <lixinhui_> but feel interested at network part
13:38:18 <yanyanhu> Qiming, yes
13:38:29 <lixinhui_> will you leverge kuryr for that?
13:38:44 <yanyanhu> so my feeling is their are in fast progress
13:39:05 <Qiming> setup swarm and use it, and see if it works as expected
13:39:07 <yanyanhu> lixinhui_, you mean senlin?
13:39:15 <lixinhui_> yes
13:39:19 <lixinhui_> yanyanhu
13:39:33 <lixinhui_> orchestration part is very complicated
13:39:37 <yanyanhu> Qiming, announcement is different from reality some times :P
13:39:39 <Qiming> everyone is moving fast, because this is a brutal, crazy world
13:39:52 <lixinhui_> I am trying to understand the driver part firstly
13:40:04 <Qiming> some companies are very good at "generalizing" things
13:40:19 <lixinhui_> for example
13:40:30 <yanyanhu> lixinhui_, based on my understanding, docker network is still in preliminary mode, compared with SDN support in openstack
13:40:33 <lixinhui_> deploy a lbaas in container
13:40:45 <Qiming> better test and verify it using your own machine
13:40:49 <lixinhui_> can senlin automatically fo that?
13:40:50 <yanyanhu> its overlay network or some 3rd part tools support
13:41:39 <yanyanhu> lixinhui_, lb in side container?
13:41:44 <yanyanhu> or lb for container cluster
13:42:26 <lixinhui_> the the one
13:42:35 <lixinhui_> the later one
13:42:48 <yanyanhu> I'm not sure about the former one. for the latter one, it is a basic requirement I guess?
13:43:01 <lixinhui_> do not know
13:43:04 <lixinhui_> at least
13:43:04 <Qiming> yes, will definitely need it
13:43:12 <lixinhui_> network need to be done
13:43:14 <yanyanhu> they need to support it, otherwise, it is useless :P
13:43:59 <yanyanhu> so we may need more investigation here
13:44:15 <yanyanhu> for what docker-engine can support now
13:44:24 <yanyanhu> and what it doesn't
13:44:27 <lixinhui_> I do not think so
13:44:40 <haiwei_> yes, yanyanhu
13:44:41 <lixinhui_> since floating ip is created by neutron in this way
13:44:57 <lixinhui_> how for applications or container know that?
13:45:05 <yanyanhu> after we better understand it, we can make the correct decision :)
13:45:48 <yanyanhu> lixinhui_, if you mean network integration with openstack for docker, I'm not clear about the progress...
13:46:03 <yanyanhu> but I feel it is not an easy job
13:46:15 <Qiming> for networking stuff, we already have kuryr working on that, so we don't have to reinvent a thing (hopefully)
13:46:40 <Qiming> if we don't know kuryr, then we should learn it, instead of just imagining it
13:46:41 <yanyanhu> at least not as easy as integrating nova with neutron :P
13:46:48 <yanyanhu> Qiming, yes
13:47:02 <yanyanhu> this is a homework have to do
13:47:08 <Qiming> yes
13:47:31 <lixinhui_> forgive me if driver layer is not ready
13:47:35 <Qiming> when we have found something missing from kuryr, we propose a work item for that team/project
13:47:36 <yanyanhu> I'm trying to build a docker1.12 env and try it
13:47:36 <elynn> I think it's almost about container cluster, not about the IaaS cluster for container, we can proceed it and assume that network is done.
13:47:43 <lixinhui_> orchestration is more complicated to discuss
13:47:48 <yanyanhu> hope can understand its latest release better
13:48:03 <elynn> We should focus how do we use senlin to deploy docker application on senlin cluster cluster.
13:48:39 <Qiming> right, 'orchestration' may mean a thousand different things from different group, to different people
13:48:40 <elynn> openstack have magnum to deploy the IaaS cluster for cluster.
13:49:04 <elynn> for docker cluster
13:49:20 <yanyanhu> kinda agree with elynn
13:49:49 <Qiming> em ... magnum is not about clustering
13:50:03 <yanyanhu> yes, it isn't
13:50:16 <Qiming> IIUC, it is always about glueing a COE onto openstack
13:50:39 <yanyanhu> just feel maybe focusing on docker app cluster deployment  is a better idea?
13:50:48 <yanyanhu> and more useful
13:51:02 <yanyanhu> for end user
13:51:19 <Qiming> from end user's perspective, that is true, I think
13:51:40 <elynn> That's what I mean yanyanhu :)
13:51:41 <haiwei_> yanyanhu, not really understand you, build a docker app cluster is different from building a docker cluster?
13:51:48 <yanyanhu> yes, since pure docker instance(s) is not that useful I feel
13:52:16 <yanyanhu> haiwei_, it is the same if you treat docker instance deployment as app deployment
13:52:33 <haiwei_> when you start a container, you use an docker image, and in the image the app is installed, right?
13:52:34 <yanyanhu> since the reason user deploy docker is for deploying app I think
13:52:41 <yanyanhu> haiwei_, yes
13:52:50 <Qiming> yes? what is the difference?
13:52:52 <yanyanhu> so a docker cluster is actually an app cluster
13:53:01 <haiwei_> so, I think it is the same thing
13:53:17 <yanyanhu> so docker orchestration is actually service orchestration I feel?
13:53:19 <yanyanhu> haiwei_, yes
13:53:27 <haiwei_> I think so
13:53:29 <Qiming> why are we deploying pure docker instances?
13:53:39 <yanyanhu> just want to clarify which layer we are focusing on :)
13:53:53 <Qiming> it is only about wording
13:53:59 <yanyanhu> Qiming, yes :)
13:54:01 <Qiming> any technology difference?
13:54:31 <Qiming> I am not a big fan of strict layering
13:54:36 <Qiming> it means nothing to end users
13:54:40 <Qiming> they don't care
13:54:52 <yanyanhu> yea
13:54:55 <yanyanhu> so I think we are on the same page now for this topic :)
13:54:57 <Qiming> is this service a paas, or a iaas?
13:55:04 <Qiming> who cares?
13:55:46 <yanyanhu> ok, last 5 minutes
13:55:55 <haiwei_> since we import containers to senlin, we need to think about how to make containers fit for senlin's architecture
13:56:07 <yanyanhu> haiwei_, +1
13:56:19 <yanyanhu> after we finish our homework :)
13:56:36 <Qiming> yes, we can transparently adapt senlin's architecture for this purpose, provided that we are not breaking existing users
13:56:41 <yanyanhu> will learn more about in coming weeks
13:56:49 <yanyanhu> and lets have further discussion about it later
13:57:24 <yanyanhu> ok, anything else about this issue?
13:57:25 <lixinhui_> cool
13:57:30 <haiwei_> ok, I will express my thoughts by code
13:57:32 <Qiming> +2, got to jump into the pool an learn swimming that way
13:57:46 <yanyanhu> oops, big latency again...
13:57:53 <Qiming> instead of reading a thousand books on swimming, staying away from the pool
13:58:08 <yanyanhu> Qiming, +2
13:58:20 <yanyanhu> trying it is the best way to learn it in most cases
13:58:22 <haiwei_> I went to pool last sunday, and got burnt
13:58:24 <elynn> it's always a controversial  topic, better done then always discuss. Just do IT ;)
13:58:28 <yanyanhu> haiwei_, :P
13:58:32 <Qiming> LOL
13:58:34 <yanyanhu> yea
13:58:40 <yanyanhu> ok, last 2 minutes
13:58:43 <yanyanhu> https://etherpad.openstack.org/p/senlin-proposal-barcelona-summit
13:58:46 <Qiming> btw, forwarded you (yanyanhu) an email from a user
13:58:55 <yanyanhu> Qiming, ok
13:58:56 <Qiming> they are seeing some NULL respose from senlin API
13:59:05 <Qiming> which is impossible in theory
13:59:06 <yanyanhu> ok, will check it
13:59:11 <Qiming> thx
13:59:17 <yanyanhu> no problem
13:59:18 <yanyanhu> https://etherpad.openstack.org/p/senlin-proposal-barcelona-summit
13:59:26 <yanyanhu> plz add concrete idea to this etherpad
13:59:37 <yanyanhu> it is for our topic proposal for coming summit
13:59:53 <yanyanhu> we may have to finish the proposal before this weekend
14:00:02 <Qiming> ok
14:00:16 <yanyanhu> since next Wednesday is the deadline
14:00:24 <yanyanhu> ok, time is over
14:00:36 <yanyanhu> lets move back to senlin channel
14:00:40 <yanyanhu> thanks you guys for joining
14:00:48 <yanyanhu> #endmeeting