03:00:02 <hongbin> #startmeeting zun
03:00:03 <openstack> Meeting started Tue Jul 19 03:00:02 2016 UTC and is due to finish in 60 minutes.  The chair is hongbin. Information about MeetBot at http://wiki.debian.org/MeetBot.
03:00:04 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
03:00:07 <openstack> The meeting name has been set to 'zun'
03:00:09 <hongbin> #link https://wiki.openstack.org/wiki/Zun#Agenda_for_2016-07-19_0300_UTC Today's agenda
03:00:14 <hongbin> #topic Roll Call
03:00:20 <Namrata> O/
03:00:28 <yanyanhu> o/
03:00:30 <shubhams__> o/
03:00:32 <mkrai> Madhuri Kumari
03:00:53 <eliqiao> o/
03:01:18 <sudipto> o/
03:01:25 <kragniz> o/
03:01:31 <Qiming> o/
03:01:33 <flwang1> o/
03:01:45 <hongbin> Thanks for joining the meeting Namrata yanyanhu shubhams__ mkrai eliqiao sudipto kragniz Qiming flwang1
03:01:53 <hongbin> #topic Announcements
03:02:00 <hongbin> I have no announcement
03:02:06 <hongbin> Anyone has?
03:02:26 <hongbin> #topic Review Action Items
03:02:32 <hongbin> 1. hongbin create an etherpad for the COE API design (Done)
03:02:39 <hongbin> #link https://etherpad.openstack.org/p/zun-coe-service-api
03:03:00 <hongbin> We can work on the etherpad as a homework
03:03:07 <hongbin> 2. mkrai to create an etherpad for nova-integration (Done)
03:03:13 <hongbin> #link https://etherpad.openstack.org/p/zun-containers-nova-integration
03:03:28 <mkrai> I have written just my view point there.
03:03:42 <hongbin> ack
03:03:49 <hongbin> 3. shubhams to create a bp for managing state of containers
03:04:03 <shubhams__> Created a bp
03:04:04 <hongbin> shubhams__: you have any update?
03:04:31 <shubhams__> I have created a bp and working on the same
03:04:48 <hongbin> shubhams__: do you have a link to the BP?
03:04:53 <shubhams__> #link https://blueprints.launchpad.net/zun/+spec/container-state-management-poc
03:05:09 <hongbin> shubhams__: thanks :)
03:05:17 <hongbin> status: Done
03:05:23 <hongbin> 4. madhuri Add points why we want to integrate with nova on BP whiteboard
03:05:34 <hongbin> Done
03:05:42 <mkrai> Ok I have done that in etherpad itself
03:05:50 <hongbin> yes
03:05:54 <hongbin> #topic Runtimes API design
03:06:19 <hongbin> mkrai: your stage :)
03:06:28 <mkrai> Ok so I have written the basic APIs
03:06:49 <mkrai> Now I am working on docker controller patch to incorporate them all
03:07:08 <mkrai> I don't think we need spec for same
03:07:25 <hongbin> No, I don't think so
03:07:28 <yanyanhu> mkrai, you mean the API controller?
03:07:40 <mkrai> Yes
03:07:43 <yanyanhu> I see
03:07:56 <mkrai> #link https://review.openstack.org/#/c/328444/
03:08:16 <mkrai> If anyone has any concern they can comment on the patch
03:08:55 <mkrai> hongbin, thats all
03:08:55 <eliqiao> mkrai: there should be a related etherpad ?
03:09:00 <mkrai> Yes
03:09:11 <eliqiao> mkrai: can you please link it on the patch?
03:09:21 <eliqiao> for easy reviewing
03:09:23 <mkrai> Yes sure I will do that
03:09:32 <mkrai> eliqiao, good point :)
03:09:37 <eliqiao> mkrai: thanks :)
03:09:44 <mkrai> #link https://etherpad.openstack.org/p/zun-containers-service-api-spec
03:09:51 <mkrai> Above is the link for API spec
03:10:07 <hongbin> Thanks mkrai
03:10:12 <hongbin> Next topic
03:10:14 <hongbin> #topic Nova integration
03:10:22 <hongbin> #link https://blueprints.launchpad.net/zun/+spec/nova-integration The BP
03:10:27 <hongbin> #link https://etherpad.openstack.org/p/zun-containers-nova-integration The etherpad
03:10:55 <mkrai> hongbin, In last team meeting there were some concern on integrating with nova
03:11:08 <hongbin> I saw the log of last meeting, it looks there are concerns on this direction
03:11:11 <mkrai> Do we have strong point on doing that?
03:11:19 <mkrai> Yes
03:11:30 <hongbin> yanyanhu: Qiming : ^^
03:11:50 <mkrai> sudipto, ...
03:11:57 <Qiming> I think it mainly falls on the nova-docker use case
03:12:00 <yanyanhu> :)
03:13:12 <hongbin> silent...
03:13:20 <yanyanhu> not sure whether there is use case,  but from unified compute resource perspective, we may need it
03:13:34 <Qiming> there are use cases we want to support where people want to treat containers as light-weight virtual machines
03:14:15 <sudipto> Qiming, i think that's a very good use case. I have always thought of the nova integration being more seemless for an isolated/secure container workflow.
03:14:51 <sudipto> per the ML discussion, it does seem like nova-docker has a few consumers.
03:15:23 <mkrai> No one is maintaining it I guess
03:15:24 <Qiming> yep, that's my perception too
03:15:40 <Qiming> dims proposed to retire nova-docker
03:15:46 <sudipto> my only thing is - we should focus on the core of zun before the integration work - so that we have something solid out first. But if the team wants to do nova integration first, i have no problem.
03:16:21 <hongbin> Well, the rule of open source is who want it, who is the one to do it :)
03:16:28 <Qiming> if there is no obvious conflict, we may want to try pursue these two goals in parallel
03:16:48 <Qiming> agreed, hongbin
03:16:51 <hongbin> If nobody interrest to take the task, then it means nobody think it is useful, we don't need to worry about that anymore
03:17:29 <Qiming> I believe there are many variants out there, :)
03:17:30 <yanyanhu> at least not an urgent need I think if nova-docker will still be available in short future
03:17:31 <hongbin> Who is taking hte nova integration bp?
03:18:12 <Namrata_> I want to
03:18:23 <hongbin> Namrata_: go ahead
03:18:35 <Namrata_> okay thanks
03:18:36 <Qiming> thanks, Namrata_, pls let me know if helps needed
03:18:49 <mkrai> Thanks Namrata_
03:18:55 <Namrata_> okay
03:19:20 <hongbin> sudipto: If there is a volunteer, it looks we can do both in parallel.
03:19:32 <sudipto> hongbin sure.
03:19:58 <hongbin> Any thing else to discuss about nova integration?
03:20:01 <mkrai> Cool now we have an agreement :)
03:20:15 <hongbin> :)
03:20:32 <hongbin> Namrata_: Do you have idea about the design? implementation?
03:20:37 <eliqiao> my concern is how to test that
03:20:59 <eliqiao> Nova now doesn't support out-of-tree driver.
03:21:02 <Namrata_> currently i am going through ironic driver
03:21:24 <hongbin> Namrata_: Yes, that is a good start
03:21:49 <hongbin> eliqiao: I guess we can setup a pipeline in the gate?
03:22:24 <Qiming> nova always believe it is the god <-- okay, not funny
03:22:48 <hongbin> :)
03:22:55 <mkrai> :D
03:23:32 <eliqiao> haha. nova is the father.
03:23:33 <yanyanhu> hongbin, you mean a test job with nova pre-setup?
03:24:03 <hongbin> yanyanhu: Yes, it should be as easy as a normal devstack setup
03:24:12 <yanyanhu> and install Zun nova driver when building job
03:24:15 <mkrai> eliqiao, do they plan to take out ironic driver also?
03:24:27 <yanyanhu> hongbin, yes, it will work I think
03:24:46 <eliqiao> mkrai: no, I meant nova doesn't want to support virt-driver which is out of nova tree.
03:25:08 <hongbin> eliqiao: No, we support the out-of-tree driver
03:25:37 <yanyanhu> eliqiao, you mean the virt-driver interface will no longer be supported by nova?
03:25:47 <hongbin> eliqiao: We have the driver in our tree, have a devstack plugin to set it up, and test it in gate
03:26:09 <mkrai> yes and I guess that is more preferrable
03:26:15 <mkrai> It lies in our tree
03:26:20 <mkrai> And we manage it well
03:27:02 <Qiming> right, we are not talking about having nova team to support the container driver in zun
03:27:14 <eliqiao> hongbin: yes, I know that, but for nova loading our virt-driver, I am not sure if it's a easy work.
03:27:18 <Qiming> zun team can do it, no?
03:27:20 <eliqiao> hongbin: check https://review.openstack.org/#/c/309504/
03:27:40 <yanyanhu> Qiming, +1, for users who have requirement, they can get it from Zun repo
03:27:42 <eliqiao> yanyanhu ^^ check the link
03:28:16 <hongbin> eliqiao: We can figure out how to do that later.
03:28:33 <eliqiao> hongbin: okay, just checked, it's reverted.
03:28:39 <hongbin> eliqiao: No need to bother the technical details at the beginning IMO
03:29:00 <eliqiao> okay, I have no concern now. it's okay to go.
03:29:26 <hongbin> eliqiao: Thanks for pointing it out though
03:29:31 <yanyanhu> eliqiao, thanks for link. This change makes thing complicated I think...
03:29:37 <Qiming> great, that stupid patch is reverted
03:30:00 <eliqiao> yanyanhu: Qiming :( the reverted patch abandoned :(
03:30:23 <eliqiao> https://review.openstack.org/#/c/310255/
03:30:28 <yanyanhu> sigh...
03:31:10 <hongbin> There must be a way to hack it in my feelings
03:31:20 <hongbin> Especially we control the devstack
03:31:25 <Qiming> "The out of tree drivers just need to tweak their packaging to make this work."
03:31:26 <yanyanhu> so that will become Nova's decision: whether there is a Nova driver for Zun
03:31:33 <Qiming> from Matt in the review comment
03:31:39 <Qiming> so that is the 'workaround'
03:32:48 <hongbin> Any more comment?
03:33:07 <hongbin> OK. Advance topic
03:33:10 <hongbin> #topic Re-consider RabbitMQ. How about using key/value store (i.e. etcd) for passing messages
03:33:37 <hongbin> I guess this was discussed in the last meeting as well?
03:33:48 <shubhams__> For this, work is in progress ..
03:33:51 <mkrai> Not in detail
03:34:15 <shubhams__> but from what I have looked upon, it looks like that etcd is better option
03:34:34 <eliqiao> shubhams__: reason for etcd?
03:35:00 <shubhams__> Because of scalability and its ability to handle cases like failures
03:35:32 <eliqiao> shubhams__: hmm... I am okay to try new stuff, but does OpenStack has such oslo lib to using etcd ?
03:35:47 <shubhams__> Since container can live a short life , so we need to select a solution that handles node failures well
03:35:58 <shubhams__> eliqiao : I need to check that
03:36:07 <eliqiao> shubhams__: cool, thanks.
03:36:15 <mkrai> Isn't there any openstack project already using it?
03:36:27 <hongbin> taskflow
03:36:31 <eliqiao> mkrai: good question, that also I want to ask.
03:36:41 <eliqiao> ah.. good example.
03:37:07 <mkrai> Ok
03:37:09 <shubhams__> hongbin : you mean taskflow uses etcd?
03:37:09 <hongbin> Frankly, all COEs are using key/value store
03:37:31 <hongbin> shubhams__: yes, (or zookeeper)
03:37:33 <eliqiao> hongbin: yeah. we are more like a COE
03:37:51 <shubhams__> hongbin : you are right , kubernetes uses etcd while docker supports consul and etcd as well
03:38:28 <hongbin> Then, it raises another question
03:38:36 <hongbin> do we need the rabbitmq?
03:38:36 <mkrai> hongbin,  Ok so we can aim on using taskflow also?
03:38:40 <sudipto> i think the choice depends on what we want to establish from the data stores as well. Do you imagine running a etcd of zun's and then another etcd running for kubernetes?
03:38:45 <sudipto> (for example)
03:38:47 <eliqiao> so we plan to develop a db driver to support etcd/consul as such key/value store?
03:38:54 <hongbin> mkrai: I am not sure about using taskflow or not
03:39:11 <yanyanhu> or Tooz?
03:39:12 <yanyanhu> http://docs.openstack.org/developer/tooz/
03:39:19 <sudipto> so the question is more on the lines of - what do you want to store in etcd - a full fledged container state or something that is a proxy to the actual COE DB.
03:39:37 <yanyanhu> lock management, leadership election, group membership
03:39:47 <hongbin> sudipto: There are two things
03:39:50 <eliqiao> yanyanhu: cool, which project using that?
03:40:00 <yanyanhu> eliqiao, heat has plan I know
03:40:06 <hongbin> sudipto: the Zun itself and the referenced COE
03:40:06 <yanyanhu> and also Senlin has plan :)
03:40:14 <yanyanhu> to use it manage distributed lock
03:40:16 <eliqiao> cool yanyanhu
03:40:27 <hongbin> sudipto: We use etcd in the referenced COE, but in Zun
03:40:42 <Qiming> a little bit confused ... not sure I understand the question
03:40:44 <yanyanhu> it can support different kinds of backend, including zookeeper and etcd(as announced)
03:41:02 <Qiming> the topic is: " Re-consider RabbitMQ. How about using key/value store (i.e. etcd) for passing messages"
03:41:06 <yanyanhu> it will at least :)
03:41:20 <hongbin> Qiming: which part is confusing?
03:41:32 <Qiming> tooz cannot do message passing (aka. rpc), IIRC
03:42:02 <hongbin> Qiming: Here is how it works. Node A watch the etcd, Nova B write the etcd, Node A get notified
03:42:43 <Qiming> that is etcd's main use case
03:43:02 <hongbin> Qiming: Nova A watch K, Nova B write message to K, Nova A get message from K
03:43:22 <Qiming> so it is now treated as a (functional) superset of message queue?
03:43:41 <hongbin> I guess yes
03:43:49 <hongbin> That is how k8s works
03:44:04 <Qiming> okay, if that is true, it answers my question
03:45:10 <hongbin> However, etcd is just an option
03:45:22 <hongbin> I don't know if it fits into Zun or not
03:45:28 <hongbin> Another option is rabbitmq
03:45:51 <shubhams__> yes . so far we have 4 options : RabbitMQ, etcd, taskflow and tooz
03:45:56 <yanyanhu> hongbin, need some investigation here I think. Tooz's scope is wider I feel
03:46:12 <hongbin> yanyanhu: sure
03:46:31 <Qiming> tooz seems not desgined to be a message passing venue I think
03:46:35 <mkrai> rabbitmq is just for message passing. Isn't it? We can't store state there
03:46:48 <Qiming> though it provides a better abstraction of DLM, group memberhips management etc
03:47:06 <hongbin> mkrai: rabbimq for message passing, etcd for storage
03:47:55 <hongbin> It seems we need a volunteer to investigate?
03:48:11 <Qiming> oh, cool, etcd is written in go
03:48:23 <yanyanhu> hongbin, I can spend some time on Tooz I think
03:48:35 <hongbin> yanyanhu: thx
03:48:43 <yanyanhu> but as Qiming said, it is not for messaging I feel
03:49:05 <hongbin> ack
03:49:16 <eliqiao> also we can investigate how taskflow using etcd?
03:49:20 <hongbin> OK. I will take the task
03:49:31 <shubhams__> I can check etcd
03:49:32 <hongbin> #action hongbin investigate an option for message passing
03:49:34 <mkrai> hongbin, I think its better to write our requirement on bp
03:49:40 <mkrai> shubhams__, ^
03:49:45 <hongbin> shubhams__: sure. Feel free to join me
03:50:21 <hongbin> mkrai: what kind of requirement in your mind?
03:50:29 <yanyanhu> agree mkrai, better have a bp or etherpad to record the investigation result
03:50:50 <mkrai> yanyanhu, We do have one
03:51:16 <mkrai> hongbin, One is store containers state
03:51:31 <yanyanhu> great, will log result on it
03:52:12 <mkrai> hongbin, And message passing. I have one question here
03:52:35 <yanyanhu> https://blueprints.launchpad.net/zun/+spec/container-state-management-poc
03:52:41 <yanyanhu> hi, mkrai , this one?
03:53:03 <hongbin> This is a BP
03:53:03 <mkrai> Message passing above is between which nodes?
03:53:12 <mkrai> yes yanyanhu
03:53:15 <yanyanhu> got it
03:53:19 <hongbin> mkrai: between components
03:53:24 <mkrai> Where our containers are running?
03:53:41 <hongbin> mkrai: api <-> conductor? <-> agent
03:54:07 <mkrai> Ok got it
03:54:07 <hongbin> container is running on zun-compute (an agent)
03:54:13 <yanyanhu> just added an etherpad for it
03:54:14 <yanyanhu> https://etherpad.openstack.org/p/container-state-management
03:54:16 <mkrai> But do we want to reconsider it?
03:54:19 <yanyanhu> added link to bp
03:54:30 <hongbin> Nice
03:54:34 <mkrai> Is there any performance overhead or anything?
03:55:01 <mkrai> rabbitmq is used in all projects for message passing
03:55:04 <hongbin> The more components we have, the more overhead on the message queue
03:55:51 <hongbin> #topic Open Discussion
03:56:03 <yanyanhu> sorry, please use this one: https://etherpad.openstack.org/p/zun-container-state-management
03:56:06 <yanyanhu> add zun- prefix
03:56:12 <hongbin> k
03:56:31 <hongbin> mkrai: Yes, but we are developing a referenced COE now
03:57:16 <hongbin> mkrai: I doubt if rabbitmq still fit into a COE? as they fit into the OpenStack project?
03:57:30 <hongbin> mkrai: Frankly, I don't know.
03:57:48 <mkrai> hongbin, Sorry for confusion but I think rabbitmq part is independent of etcd
03:58:32 <mkrai> Where comes rabbitmq in picture for COE?
03:58:46 <mkrai> COE has its own etcd which we don't need to worry
03:59:05 <hongbin> mkrai: My argument is rabbitmq is universal in openstack
03:59:14 <hongbin> mkrai: but it is not used in any COE
03:59:34 <mkrai> Yes
03:59:53 <mkrai> Ok I think we should move over to #zun
04:00:01 <mkrai> Thanks hongbin
04:00:06 <hongbin> Yes, the team meeting is end
04:00:14 <hongbin> All. Thanks for joining the meeting
04:00:16 <yanyanhu> thanks
04:00:17 <hongbin> #endmeeting