18:01:03 <daneyon_> #startmeeting container-networking
18:01:04 <openstack> Meeting started Thu Dec 10 18:01:03 2015 UTC and is due to finish in 60 minutes.  The chair is daneyon_. Information about MeetBot at http://wiki.debian.org/MeetBot.
18:01:05 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
18:01:08 <openstack> The meeting name has been set to 'container_networking'
18:01:14 <daneyon_> Agenda
18:01:18 <daneyon_> #link https://wiki.openstack.org/wiki/Meetings/Containers#Agenda
18:01:33 <daneyon_> I'll pause a minute for everyone to review the agenda
18:01:49 <hongbin> o/
18:01:49 <daneyon_> #topic roll call
18:02:08 <dane> o/
18:02:09 <Tango> o/
18:02:37 <daneyon_> Thank you hongbin Tango dane for attending the meeting.
18:02:45 <daneyon_> #topic Flannel host-gw patch
18:02:50 <daneyon_> #link https://review.openstack.org/#/c/241866/
18:03:04 <daneyon_> Tango I'll defer to you
18:03:17 <daneyon_> Doesn't seem like much has happened with the patch
18:03:20 <Tango> So I tried a few things
18:03:39 <daneyon_> Can you share a few details?
18:03:50 <Tango> The patch that Gus added didn't work
18:04:14 <Tango> I tried turning off port security to allow the node to route traffic
18:04:39 <Tango> I can see that the packet does get rerouted correctly to the target host
18:05:05 <Tango> However once on the target host, the packet could not get to its destination
18:05:38 <Tango> I notice that the flannel0 TUN interface is not created in the host
18:06:01 <Tango> I am not sure if this is how it should work, or it's an error
18:06:26 <Tango> I sent email to Angus but have not heard back from him.
18:06:41 <daneyon_> Tango have you tried pinging the flannel or coreos irc channel?
18:06:54 <Tango> Tried a few times pinging on IRC, but the time difference is tricky
18:06:57 <daneyon_> I have never tried the host-gw mode
18:07:29 <daneyon_> but it should work since L2 is the only requirement.
18:07:37 <Tango> I am thinking about trying to create the flannel0 interface manually, seems like we need this for the flanneld to pick up the packet
18:07:59 <Tango> Another thing to try is a later flannel version
18:08:23 <daneyon_> I'll take a look through the flannel code too. If I see anything that pops out at me, I'll ping you on irc.
18:08:32 <Tango> I couldn't build with the package from kojipkg, but I can just overwrite the binary
18:08:33 <daneyon_> Thx for working on this.
18:08:57 <Tango> A little slow going, I haven't been able to put a lot of time on it.  Too many other things going on.
18:09:22 <daneyon_> Tango, If I recall correctly, our flannel version is super old. I think it's a good idea to update flannel no matter what.
18:09:32 <daneyon_> then at leat the vxlan bug will be fixed ;-)
18:09:42 <daneyon_> until we can run flannel in a container
18:09:52 <daneyon_> Understandable
18:09:56 <daneyon_> I know the feeling
18:09:59 <Tango> Yeah.  I think I almost get the fedora image without atomic to work, so this would be easier
18:10:24 <daneyon_> Agreed.. Atomic is a PITA ;-)
18:10:33 <Tango> Yep :)
18:10:58 <daneyon_> #topic Build Fedora Atomic with DIB
18:10:59 <dane> daneyon: What is the vxlan bug?
18:11:04 <daneyon_> #link https://blueprints.launchpad.net/magnum/+spec/fedora-atomic-image-build
18:11:18 <daneyon_> Yolanda added this topic to the agenda
18:11:29 <Tango> dane: the vxlan backend doesn't work on Flannel 0.5.0
18:11:44 <dane> Tango: Thanks!
18:11:54 <daneyon_> yolanda are you in attendance?
18:12:30 <daneyon_> Seems like Yolanda is not here.
18:12:39 <daneyon_> I will ping her offline
18:12:56 <daneyon_> Maybe we'll address this another time
18:13:21 <daneyon_> Does anyone else know about this BP?
18:14:04 <daneyon_> It's been a while since I used diskimagebuilder, but i do remember it being somewhat painful.
18:14:09 <Tango> I only had a discussion with her when she kicked off the BP
18:14:17 <daneyon_> Anyone else familiar with DIB?
18:14:23 <Tango> I have been using it
18:14:35 <Tango> to build fedora, ubunu images
18:14:55 <Tango> we have a few elements to build ironic and mesos images
18:15:00 <daneyon_> has is worked well for you?
18:15:04 <daneyon_> Tango
18:15:30 <Tango> It seems OK
18:15:44 <Tango> Documentation can be a little sparse
18:16:11 <hongbin> AFAIK, the ironic kubernetes image was built by Dib
18:16:33 <hongbin> It is a standard fedora image, not atomic
18:16:49 <hongbin> with several elements added to it. Works very well
18:16:53 <Tango> right
18:17:01 <greghaynes> Hello
18:17:04 <daneyon_> The coreos ISO is 200+ MB... I think we need to focus on coreos and put atomic on the backburner
18:17:12 <greghaynes> I know a thing or two about dib if you all want any info
18:17:27 <greghaynes> right now yolanda is working on adding lvm support to dib which is a requirement for atomic AIUI
18:17:47 <greghaynes> The documentation shouldn't be *that* sparce
18:17:53 <Tango> greghaynes: ah, here a question:  for the install-package element, can you specify a version othe package?
18:17:57 <daneyon_> greghaynes thanks for the add'l info
18:18:29 <Tango> greghaynes: Like if I want docker 1.9.1 and not 1.8.2
18:18:34 <greghaynes> Tango: There are many ways to install packages, that one doesn't really but you could either install the package using standard tools or add that to pkg-map
18:18:48 <daneyon_> greghaynes Do you happen to know why yolanda added #link https://blueprints.launchpad.net/magnum/+spec/fedora-atomic-image-build to today's agenda?
18:18:56 <greghaynes> As in - there is no reason you cant just use apt-get or yum to install whatever you want
18:19:20 <greghaynes> daneyon_: She has a goal of using DIB to generate the atomic images so we can greatly decrease their size, among other things
18:19:30 <greghaynes> daneyon_: I dont know of any specific topics she wanted to discuss
18:19:46 <greghaynes> Here are teh DIB docs FWIW http://docs.openstack.org/developer/diskimage-builder/
18:19:48 <daneyon_> greghaynes can you expand on the among other things?
18:20:28 <greghaynes> daneyon_: Sure - a *lot* of projects use DIB in testing for image creation - infra also uses it for their testenv creation. As a result we have a lot of knowledge around using it to make images as part of our testing pipeline
18:20:44 <greghaynes> it also is a lot more efficient because we have a lot of caching set up to make dib fast when run in our testenv's
18:21:04 <Tango> I like it :)
18:21:34 <daneyon_> greghaynes I see. Thanks.
18:21:53 <greghaynes> np
18:21:54 <daneyon_> greghaynes has anyone used DIB with coreos?
18:22:06 <greghaynes> I am not sure what that means - do you mean to make a coreos image?
18:22:12 <daneyon_> yes
18:22:35 <greghaynes> Not that I am aware of, but the tool itself is pretty general purpose - adding new 'distros' tends to be really easy
18:22:50 <daneyon_> OK
18:23:20 <Tango> greghaynes: How about the ubuntu-minimal or fedora-mininal elements?
18:23:29 <greghaynes> Tango: How about them? :)
18:23:52 <greghaynes> That is just a distinction of whether we use debootstrap or a cloud-image (regular ubuntu element)
18:23:53 <daneyon_> I had Tango create an image for me ~6 weeks ago for adding a new network-driver... We needed too add some DIR's and the ovs pkg to Atomic
18:24:05 <Tango> greghaynes: I tried those but kept getting errors. I am guessing I have to include other dependent elements?
18:24:07 <daneyon_> I have yet to test it b/c I have been working on a few other things
18:24:14 <daneyon_> I will be going back to it soon.
18:24:26 <greghaynes> Tango: You shouldnt have to, if you let me see the error I could probably help/fix
18:24:32 <daneyon_> I wonder if I should use DIB instead
18:24:39 * greghaynes would highly recommend
18:24:54 <Tango> That's what Yolanda BP is about
18:25:05 <Tango> The current process is manual
18:25:07 <daneyon_> greghaynes Is their an ETA on when yolanda's work will be ready to test?
18:25:31 <clarkb> there are system package requirements on the build host for -minimal elements
18:25:32 <greghaynes> I have no idea..
18:25:39 <clarkb> deboostrap for example
18:25:40 <greghaynes> I just know about it because I was helping her review some things
18:26:05 <Tango> clarkb: Are they documented somewhere?
18:26:06 <daneyon_> OK
18:26:15 <daneyon_> I'll ping her directly then.
18:26:17 <clarkb> Tango: should be in the element
18:26:23 <greghaynes> Tango: http://docs.openstack.org/developer/diskimage-builder/elements/ubuntu-minimal/README.html
18:27:11 <daneyon_> #topic Review Action Items
18:27:19 * daneyon_ Tango to implement option 2 in the flannel host-gw patch https://review.openstack.org/#/c/241866/
18:27:26 <daneyon_> Tango already provided this update
18:27:27 <daneyon_> Thanks.
18:27:33 * daneyon_ danehans to create a network-driver support matrix
18:27:41 <daneyon_> One sec and I'll provide the links
18:28:05 <daneyon_> #link https://wiki.openstack.org/wiki/Magnum
18:28:14 <daneyon_> From ^, go to Resources
18:28:26 <daneyon_> oops References
18:28:43 <daneyon_> and you will see Network Driver Support Matrix and Labels Support Matrix
18:29:08 <daneyon_> Pretty basic so far, but should be a good starting point as other network drivers get added
18:29:22 <daneyon_> Feel free to contribute to improve.
18:30:08 <daneyon_> I have also been working on a doc that goes through magnum networking details
18:30:10 <daneyon_> #link https://wiki.openstack.org/wiki/Magnum/Networking
18:30:29 <daneyon_> When you have time, pls review and feel free to contribute to improve.
18:31:09 <daneyon_> ^ doc will also get updated as we add network drivers.
18:31:20 <daneyon_> I will add a link to the doc from the main M wiki page
18:32:34 * daneyon_ danehans check with wanghua on implementation status of https://blueprints.launchpad.net/magnum/+spec/run-kube-as-container
18:32:59 <daneyon_> I did sync-up with wanghua ~ 2 weeks ago
18:33:24 <hongbin> I will give a brief update on behalf of him
18:33:28 <daneyon_> We discussed a few details re implementation options. We're on the same page re the details.
18:33:38 <daneyon_> great, thx hongbin
18:34:01 <hongbin> He put a review for the bootstrap docker daemon
18:34:34 <hongbin> A bootstrap docker daemon is a separated docker daemon for running etcd flannel
18:34:52 <hongbin> Once that patch landed, we are ready to containerize flannel
18:35:07 <daneyon_> great
18:35:08 <hongbin> #link https://review.openstack.org/#/c/250999/
18:35:30 <daneyon_> I believe Egor had some concerns regarding the bootstrap docker daemon.
18:36:06 <hongbin> Yes, he worried the extra complexity of introducing the additional docker daemon
18:36:15 <daneyon_> i would like to take 5 min to review the patch
18:36:16 <hongbin> eghobo_: yt?
18:36:26 <eghobo_> hongbin: yep
18:36:49 <hongbin> eghobo_: we are talking about the docker bootstrap daemon
18:37:34 <eghobo_> yes, I saw it. honestly I don't see the value
18:38:15 <eghobo_> flannel is just 4 files you can just copy them ;)
18:38:50 <eghobo_> also even coreos folks don't run flannel in contaner
18:39:18 <daneyon_> From #link https://github.com/kubernetes/kubernetes/blob/release-1.0/docs/getting-started-guides/scratch.md#selecting-images
18:39:31 <daneyon_> "For etcd, kube-apiserver, kube-controller-manager, and kube-scheduler, we recommend that you run these as containers, so you need an image to be built."
18:41:01 <daneyon_> hongbin do you have a kube doc reference that states why the docker bootstrap is needed?
18:41:16 <hongbin> let me find it
18:41:33 <eghobo_> ;) don't believe everything what you read, very often it's outdated
18:41:49 <daneyon_> It seems like the bootstrap is not a flannel-specific requirement.
18:41:58 <eghobo_> we can easy run 'etcd, kube-apiserver, kube-controller-manager, and kube-scheduler'
18:42:20 <eghobo_> i have PR for everything except etcd
18:42:35 <daneyon_> eghobo_ agreed, but i think it's important that we understand the recommendations from the kube community and use that as data for our decision
18:43:03 <hongbin> #link http://kubernetes.io/v1.1/docs/getting-started-guides/docker-multinode.html
18:43:19 <eghobo_> bootstrap is very flannel specific, you need to create bridge before real docker start
18:43:24 <daneyon_> It's my understanding that the M community is trying to ensure that our COE's are following upstream best practices. I want to make sure we're aligned from a networking standpoint.
18:43:54 <daneyon_> If CoreOS, Kube, etc.. are saying  flannel should not be containerized, then I will heavily lean on those recommendations
18:44:42 <daneyon_> taking a few minutes to review #link http://kubernetes.io/v1.1/docs/getting-started-guides/docker-multinode.html
18:45:15 <eghobo_> daneyon_: actually they are not saying it officially ):, but they just don't use
18:46:07 <daneyon_> all #link http://kubernetes.io/v1.1/docs/getting-started-guides/docker-multinode.html does not provide much background on the benefits to this 2 daemon approach.
18:46:42 <daneyon_> Is someone willing to address this further on the kube irc channel?
18:47:00 <daneyon_> I do have concerns about over complication
18:47:46 <hongbin> daneyon_: I think you are the best candidate :) or I can ask wanghua to do that
18:47:57 <daneyon_> ATM I don't see the benefits outweighing the concerns of over complication.
18:48:38 <Tango> we can ask if others have tried this and have any suggestion
18:48:39 <daneyon_> #action danehans to address the 2-daemon approach on the kube irc and provide add'l info through Magnum ML
18:48:51 <daneyon_> I'll take it on
18:49:03 <hongbin> daneyon_: thx
18:49:06 <daneyon_> Until then, I would like to see the other kube service containerized.
18:50:19 <Tango> So how do we select a version with the kube service containerized?
18:50:32 <Tango> like 1.1.1 ...
18:50:41 <daneyon_> I would think it would be through an image tag
18:51:53 <Tango> ok
18:52:15 <daneyon_> i'm trying to find the doc
18:52:22 <daneyon_> i'll shoot it to you if i can find it
18:52:29 <daneyon_> #topic Open Discussion
18:53:11 <Tango> On the last meeting, we had a brief discussion on networking for mesos cluster
18:53:23 <daneyon_> Their is a meetup re: Docker Trusted Registry tomorrow. Check it out if you can #link http://www.meetup.com/Docker-Online-Meetup/events/227287361/
18:53:50 <daneyon_> Yes Tango. I believe we agreed not to iomplement Flannel for Mesos
18:54:00 <daneyon_> at least until etcd is implemented in mesos
18:54:14 <Tango> Does anyone know if tenant isolation is supported within mesos?
18:54:25 <daneyon_> the general concensous was that not every network driver will be applicable for every coe
18:54:38 <Tango> In other words, can we allow different tenants on the same mesos cluster?
18:54:52 <daneyon_> i don't believe it is.
18:55:13 <Tango> OK, so we will still have to isolate by private neutron network
18:55:13 <hongbin> I don't think so either
18:55:58 <daneyon_> i believe multi-tenancy is addressed in some of the mesos frameworks
18:56:07 <daneyon_> #link https://engineering.twitter.com/university/videos/spark-on-mesos
18:56:25 <daneyon_> or by running mesos in magnum :-)
18:56:29 <daneyon_> lol!!!
18:56:56 <Tango> One of the concern about containers is different tenants running containers on the same host
18:57:17 <Tango> we solve that by VM's and networking
18:57:51 <Tango> Mesos claims to run on the whole data center, so I wonder how they address this
18:58:13 <daneyon_> Tango take a look at the Selecting Images section #link https://github.com/kubernetes/kubernetes/blob/release-1.0/docs/getting-started-guides/scratch.md#software-binaries
18:58:24 <daneyon_> All, we are down to our final 2 minutes.
18:59:25 <daneyon_> In general, mesos is focused on pooling all compute resources
18:59:46 <daneyon_> the different frameworks that run on mesos uses the pooled resources for different purposes
19:00:08 <Tango> Sounds like Magnum has the right approach for multi tenancy then
19:00:11 <daneyon_> i.e. the kube-mesos framework uses the pooled resources for container clustering engine purposes.
19:00:19 <daneyon_> we can talk more on the M irc channel.
19:00:26 <daneyon_> Thanks all for attending!!!
19:00:34 <daneyon_> #endmeeting