15:31:53 <apuimedo> #startmeeting k8s integration
15:31:54 <openstack> Meeting started Wed Feb  3 15:31:53 2016 UTC and is due to finish in 60 minutes.  The chair is apuimedo. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:31:55 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
15:31:57 <openstack> The meeting name has been set to 'k8s_integration'
15:32:01 <apuimedo> who's here for the meeting?
15:32:09 <banix> o/
15:32:24 <gsagie_> hi
15:33:19 <apuimedo> irena said she'd be late
15:33:28 <apuimedo> fkautz: you there?
15:34:16 <irenab> hi
15:34:27 <gsagie_> irena!!!! :)
15:34:29 <apuimedo> irenab: is not that late :P
15:34:32 <irenab> almost on time :-)
15:34:58 <irenab> gsagie: also glad to see you :-) !
15:35:01 <apuimedo> well, I was hoping there would be more people, but we have a nice cozy assemble
15:35:07 <apuimedo> *assembly
15:35:56 <apuimedo> alright, let's start
15:36:08 <banix> salv-orlando: you here by any chance?
15:36:14 <irenab> shall we start from use cases?
15:36:35 <apuimedo> mmm
15:36:43 <apuimedo> I forgot where we left off
15:36:56 <apuimedo> ah yes
15:37:14 <banix> we wrer in a forrest and looking for a way out?
15:37:18 <apuimedo> option T vs option T+F vs option F
15:37:26 <apuimedo> banix: look left
15:37:39 <banix> always the right direction
15:37:44 <irenab> lets focus on what we will support and then how?
15:37:54 <banix> T was?
15:38:28 <gsagie_> translation to libnetwork
15:38:42 <banix> and F?
15:38:44 <gsagie_> and the other just to implement the CNI
15:38:51 <gsagie_> two different solutions
15:38:55 <banix> thx
15:38:58 <gsagie_> integrate with Kubernetes
15:39:04 <irenab> I thought we already decided to drop T
15:39:20 <gsagie_> its good with me
15:39:37 <apuimedo> irenab: no, we did not
15:39:41 <gsagie_> the only reason was that someone (i think Matt)
15:39:48 <gsagie_> wanted to try and work on it in parallel
15:39:49 <apuimedo> Mike
15:39:52 <gsagie_> Mike yeah
15:39:55 <apuimedo> Mike Spreitzer
15:40:26 <apuimedo> though I think there's a big consensus in reducing it to option t+f vs option F
15:40:39 <apuimedo> I vote to reduce it to that
15:40:58 <apuimedo> do you all agree?
15:41:01 <banix> apuimedo: what is exactly T+F?
15:41:02 <gsagie_> ok
15:41:17 <irenab> I already thought we agreed on this :-)
15:41:25 <banix> seems you all left me in the forrest :)
15:41:30 <gsagie_> banix: work on them in parallel together, i guess apuimedo things that translation to lib network is something we can do pretty quick
15:41:34 <gsagie_> for fast term goal
15:41:41 <gsagie_> thinks
15:41:42 <banix> ok
15:41:44 <irenab> banix: looks I am in the forest too, maybe the different one
15:41:57 <banix> so long term solution we think is F. Right?
15:42:01 <banix> irenab: :)
15:42:12 <gsagie_> i only remember it from our last meeting, and i wasn't concentrating most of the time :)
15:42:45 <gsagie_> i think since Mike is not here, we don't focus on this because there is no one that is actually going to devote time to do it, apuimedo: do you have anyone for this?
15:42:52 <apuimedo> banix: yes
15:42:55 <gsagie_> atlas not focus in this meeting
15:43:03 <gsagie_> atleast
15:43:08 <banix> just confused about reducing to T+F which is more like expanding to T+F
15:43:20 <apuimedo> gsagie_: well, I haven't dropped completely doing some sort of T
15:43:32 <apuimedo> while we work on F
15:43:49 <apuimedo> for me it would be, have somebody working on the CNI driver
15:44:08 <apuimedo> somebody working on the etcd component that sends commands to Neutron
15:44:09 <irenab> apuimedo: Can we please focus on WHAT before HOW?
15:44:30 <apuimedo> and, while we refactor, we can translate to libnetwork
15:44:37 <apuimedo> irenab: you got me
15:44:42 <apuimedo> :-D
15:44:57 <irenab> I think its easier to move with how :-)
15:45:37 <apuimedo> ok
15:45:41 <apuimedo> #topic HOW
15:45:51 <apuimedo> alright
15:46:06 <apuimedo> my proposal is that we do two components
15:46:21 <banix> i think ireana meant after we are done with WHAT
15:46:34 <apuimedo> oh right
15:46:42 <irenab> banix: I do beleive we are in the same forrest :-)
15:46:44 <apuimedo> #topic WHAT!!!
15:47:04 <apuimedo> I'm sorry for the confusion
15:47:08 <irenab> Shall we first support the “flat”, i.e. current k8s networking mode or we want to go forward with Network Policy proposal?
15:47:34 <apuimedo> F should start directly with the policy proposal
15:47:57 <irenab> do we want to have annotation/label based expressions supported?
15:48:23 <irenab> I think its probably matter of what k8s version we want to support
15:48:45 <irenab> the proposal will probably land in 1.2
15:48:48 <gsagie_> yes
15:49:00 <apuimedo> 1.2 is the target I think
15:49:00 <gsagie_> we want something that query etcd and translate to Kuryr
15:49:03 <irenab> gsagie: yes on what part?
15:49:41 <gsagie_> yes we want to have annotation/label based expressions supported, which means we define special labels/annotations for certain configuration in Neutron right?
15:50:11 <irenab> sounds good to me
15:50:15 <apuimedo> gsagie_: I think extra label support is a plus, but not an immediate requisite
15:50:51 <apuimedo> but yeah... I'd like to have it to be able to specify lb settings and other stuff
15:50:53 <gsagie_> ok, so what features currently in Kubernetes we want to support? we want the basic stuff of creating the networks right?
15:50:59 <gsagie_> and the lb replacing kube-proxy
15:51:12 <apuimedo> gsagie_: yes, that's the minimum
15:51:18 <banix> +1
15:51:21 <apuimedo> LB for replication
15:51:28 <gsagie_> for services
15:51:39 <irenab> if we want to support Policy, it may require SG
15:51:50 <apuimedo> FIP for load balancer mode
15:51:54 <irenab> so its connectivity + ACL
15:51:55 <gsagie_> okie, so the CNI already gives us the first part, for the services configuration we need to listen to etcd?
15:52:23 <irenab> seems we may need the listener to support both services and policies
15:52:30 <gsagie_> which is services VIPs and north-south LB
15:52:31 <banix> yes we need SG but that would be ok
15:52:34 <apuimedo> gsagie_: but CNI, as it is used in k8s now, IIRC only tells you to join the network that exists in the worker node
15:53:08 <apuimedo> we need it to be able to get network per tier for example
15:53:55 <irenab> apuimedo: I think this is something that needs to be concluded from the models
15:54:05 <apuimedo> exactly
15:54:12 <gsagie_> apuimedo: okie, so it seems to me that the challenge here will be to sync between the information we read from etcd and the CNI which needs to do the binding part
15:54:17 <apuimedo> so another "What" we need is
15:54:26 <apuimedo> gsagie_: exactly, you erad my mind
15:54:32 <apuimedo> *read
15:54:53 <apuimedo> * find a way for having the entities created by the watcher in the cni driver
15:54:59 <apuimedo> so that they can be used
15:55:07 <irenab> so we may even think of once understanding the model and doing some operations for neutron, add annotations after the actions to be processed by CNI
15:55:09 <apuimedo> I believe they have a similar issue in k8s-net-sig
15:55:23 <apuimedo> they were considering pushing the data down all the way to the cni
15:55:36 <apuimedo> but I'm not sure if it's going to happen nor if it is the right thing
15:55:48 <gsagie_> we can always read the etcd DB from our CNI implementation
15:55:51 <irenab> apuimedo: initially we may add details via annotations in scope of the watcher
15:55:56 <gsagie_> the problem is what happens first
15:56:10 <apuimedo> gsagie_: I'm not sure how operators would feel about that, but it is a good option
15:56:39 <irenab> gsagie: I hope we can avoid this
15:57:04 <apuimedo> irenab: me too
15:57:17 <irenab> CNI plugin should be not k8s specifc, at least ecventualy
15:57:27 <apuimedo> I'd want the cni driver to be able to do with as little as possible
15:57:38 <apuimedo> so that ideally
15:57:54 <apuimedo> it would only learn about the ip in the worst case
15:58:07 <apuimedo> or uuid/vlan(for nested) in the best base
15:58:11 <apuimedo> *case
15:58:14 <irenab> apuimedo:  there is both network and IPAM drivers
15:58:27 <apuimedo> I would like to do the ipam in the watcher
15:58:38 <apuimedo> to speed up
15:58:38 <gsagie_> one option is that we keep the CNI operations in Kuryr in a queue and wait until the watcher added the correct port in neutron and only then do the binding
15:58:56 <apuimedo> I would be overjoyed if the worker nodes don't need to have access to the neutron api
15:59:06 <apuimedo> and they can make do with the mq
15:59:23 <irenab> mq for what?
15:59:29 <banix> gsagie_: why? for applying acl?
16:00:15 <gsagie_> banix: we have a watcher that listen to etcd, it gets all the relevant information (IPAM/SG/correct network) from etcd and then call the Neutron API from the master
16:00:22 <gsagie_> then the CNI driver only do the binding part
16:00:24 <apuimedo> irenab: mq for the ovs neutron agent
16:00:42 <gsagie_> connect the neutron port to the container namespace
16:00:51 <irenab> apuimedo: this is backend dependent
16:01:18 <apuimedo> irenab: the point is
16:01:38 <apuimedo> If possible, the worker nodes shouldn't need to be on the neutron management network
16:02:07 <gsagie_> so we basically have something that integrate with Kubernetes "API" by listening to the etcd DB (can later use labels and annotations) and configure Neutron, then the end points in the nodes only do the binding
16:02:08 <irenab> apuimedo:  at least not talk to neutron API
16:02:14 <apuimedo> we have it in plain kuryr libnetwork because there is no orchestration we can check (well, with cluster store we could correct it, but that is another topic)
16:02:20 <gsagie_> apuimedo, irenab: what do you think about that?
16:02:37 <apuimedo> gsagie_: that's the goal, yes
16:02:47 <irenab> gsagie_: I agree
16:03:00 <gsagie_> apuimedo: okie, so i think we have defined it
16:03:09 <gsagie_> irenab
16:03:17 <apuimedo> gsagie_: thanks for synthesizing it
16:03:20 <irenab> so we are ok with How while discussing what :-)
16:03:38 <gsagie_> i think we also agree on the what
16:03:39 <apuimedo> the rest of the what is
16:03:49 <gsagie_> just how to model to Neutron
16:03:51 <gsagie_> networks/routers
16:03:52 <apuimedo> gsagie_: take a stab at synthesizing it
16:03:54 <apuimedo> :P
16:03:58 <apuimedo> you are on a roll
16:04:03 <gsagie_> heh
16:04:33 <gsagie_> i think we need to do the simplest thing at first stage just to build the "infrastructure" for the different components
16:04:44 <irenab> yes, and probably to define the lables/annotation we may expect as part of the k8s model entities
16:04:58 <gsagie_> so i guess you want to model it-> service=network
16:05:16 <irenab> what about namespaces isolation option?
16:05:21 <apuimedo> gsagie_: yes, that's the simplest and more common use case we can support
16:05:22 <gsagie_> and then add routing between them, at first between all
16:05:54 <irenab> we may consider pods (RC) to be on the same network regardless the service
16:06:03 <apuimedo> irenab: I'm tempted to say that namespaces is something that I want to ignore for the time being
16:06:19 <irenab> I think in some simple examples there are only Pods
16:06:21 <apuimedo> in the sense that we would not isolate namespaces of the same tenant
16:06:30 <apuimedo> but I'm not 80% on this
16:06:37 <irenab> apuimedo: do we have multi-tenancy?
16:06:39 <apuimedo> so I can be easily convinced
16:06:53 <apuimedo> irenab: sort of xD
16:07:01 <banix> well tenants don’t go across namespaces. no?
16:07:04 <apuimedo> but now it relies on one host per tenant
16:07:13 <gsagie_> multi tenancy is just how you connect the relevant networks to a router
16:07:16 <gsagie_> i think
16:07:18 <banix> nottalking about network namespaces
16:07:22 <apuimedo> banix: no. exactly. But tenants could have several namespaces
16:07:37 <irenab> k8s namespaces
16:07:43 <banix> wll are you talking about k8s namespace
16:07:45 <banix> ok
16:08:17 <apuimedo> I am talking to k8s namespaces, yes
16:09:11 <apuimedo> irenab: I think that for multitenancy we will have to rely on the multitenancy that they are adding to k8s
16:09:14 <irenab> so back to What, I think we should define scenarious to support, i.e. assuming all k8s namespace are not isolated
16:09:29 <apuimedo> and somehow bundle authorization there
16:09:46 <irenab> apuimedo: do you have any details on what is the k8s plan for multi-tenancy?
16:09:50 <apuimedo> so that in etcd we can get the auth of the tenant that is associated to the service
16:10:04 <apuimedo> and try to perform the neutron action with their auth
16:10:11 <apuimedo> irenab: not in hand, no
16:10:18 <irenab> apuimedo: this can be great, but we will need to manage tenants in keystone?
16:10:38 <apuimedo> irenab: I was thinking something simpler
16:10:52 <apuimedo> that somehow the creds would end up in etcd
16:10:58 <irenab> assuming no multi-tenancy should be ok for now, agreee?
16:11:11 <apuimedo> yes. without mulit-tenancy we should be ok
16:12:13 <irenab> gsagie_: banix ?
16:12:17 <apuimedo> hey, does any of you remember if it was possible to create an arbitrary token for a user in keystone?
16:12:24 <banix> +1
16:12:44 <banix> apuimedo: what do you mean by arbitrary?
16:12:46 <gsagie_> i am fine with it, i don't see the challenge if etcd support the namespace, we only need the "watcher" thread to have the various different tenants
16:12:50 <gsagie_> credentials
16:12:55 <gsagie_> unless i didnt get something
16:13:06 <gsagie_> but i am not too familiar with the k8 namespaces yet
16:13:13 <gsagie_> but we can work without them at first
16:13:18 <irenab> I think we need to see how to reflect this in neutron
16:13:19 <apuimedo> I meant like oath token like k8s uses
16:13:36 <gsagie_> irenab: why this can't be different OS tenants?
16:13:42 <gsagie_> different Neutron tenants
16:13:54 <banix> a tenant can create tokens at will but i am probably missing your point
16:13:56 <irenab> gsagie_: who defines them and when?
16:14:10 <apuimedo> oh
16:14:14 <apuimedo> hey guys
16:14:19 <apuimedo> and gals
16:14:26 <apuimedo> k8s supports keystone auth
16:14:28 <irenab> lets handle multi-tenancy for next phase
16:14:33 <apuimedo> we may just require it
16:14:37 <apuimedo> https://github.com/kubernetes/kubernetes/blob/master/docs/admin/authentication.md
16:14:43 <irenab> apuimedo: cool
16:15:09 <apuimedo> I'm not sure how big a handicap it will be for operators to accept keystone
16:15:18 <apuimedo> but I'm sure big operators should be content with it
16:15:34 <apuimedo> I'm betting openshift uses it
16:15:37 <apuimedo> so we could say
16:15:54 <apuimedo> if k8s keystone integration is configured it is multitenant, otherwise it is single tenant
16:15:57 <apuimedo> for starters
16:16:08 <apuimedo> #link https://github.com/kubernetes/kubernetes/blob/master/docs/admin/authentication.md
16:16:13 <irenab> apuimedo: I think we need to verify the k8s approach for app multi-tenancy
16:16:20 <gsagie_> sorry need to go all :) but i think the first start of the work is defined, so we can start coding and defining in parallel the next bits
16:16:30 <irenab> apuimedo: this maybe related to the whole cluster
16:16:48 <gsagie_> if it integrates with keystone that would be great
16:16:57 <apuimedo> #info
16:17:02 <apuimedo> #info we basically have something that integrate with Kubernetes "API" by listening to the etcd DB (can later use labels and annotations) and configure Neutron, then the end points in the nodes only do the binding
16:17:36 <gsagie_> yes challenge, we need to be able to keep the binding until neutron configuration has ended
16:18:12 <apuimedo> gsagie_: banix: irenab: what about we create a blueprint with what we discussed
16:18:23 <apuimedo> and as soon as it matures a bit we put it on a spec
16:18:28 <apuimedo> completely WIP
16:18:30 <banix> sounds good
16:18:31 <gsagie_> agreed!
16:18:31 <irenab> gsagie_: we need to check contrail approach, I think they have similar components
16:18:45 <apuimedo> or we can go straight for wip spec if you see fit
16:18:57 <irenab> apuimedo: I will try to arrange initial draft version
16:19:02 <apuimedo> as it seems people comment more on spec
16:19:15 <apuimedo> irena: you just earned an extra cheese delivery
16:19:17 <apuimedo> :-)
16:19:21 <apuimedo> thanks!
16:19:24 <irenab> apuimedo: :-)
16:19:31 <apuimedo> #topic other
16:19:31 <irenab> the last one was very good
16:19:40 <apuimedo> that was extra :-)
16:19:42 <apuimedo> anything else?
16:20:00 <gsagie_> :)
16:20:20 <gsagie_> somebody look at what contrail are doing
16:20:24 <gsagie_> i am fine doing it
16:20:27 <apuimedo> I'm delivering catalan cheese straight from the farm :-)
16:20:46 <apuimedo> gsagie_: if you can take a look at the etcd watcher, that'd be great
16:20:58 <irenab> gsagie_:banix you better go for the cheese than T-bone
16:21:14 <banix> :)
16:21:36 <apuimedo> alright, so let's close up
16:21:45 <apuimedo> #action irena to submit WIP spec
16:21:56 <apuimedo> #action gsagie to look at contrail watcher
16:22:02 <apuimedo> #endmeeting