21:00:43 <strigazi> #startmeeting containers
21:00:44 <openstack> Meeting started Tue Sep 18 21:00:43 2018 UTC and is due to finish in 60 minutes.  The chair is strigazi. Information about MeetBot at http://wiki.debian.org/MeetBot.
21:00:45 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
21:00:47 <openstack> The meeting name has been set to 'containers'
21:01:05 <strigazi> #topic Roll Call
21:01:09 <strigazi> o/
21:01:10 <colin-> hello
21:01:13 <cbrumm> o/
21:01:27 <imdigitaljim> o/
21:02:38 <strigazi> Let's go  diretly to stories/ideas, no more bps :)
21:02:49 <strigazi> #topic Stories/Ideas
21:03:33 <strigazi> What I would like more review is on : Fix cluster update command https://storyboard.openstack.org/#!/story/1722573 Patch in review: https://review.openstack.org/#/c/600806/
21:03:58 <strigazi> To unblock cluster-updates
21:04:13 <cbrumm> we'll get some eyes on it
21:04:33 <strigazi> plus we need a patch to be able to fix clusters in UPDATE_FAILED
21:04:47 <strigazi> I can do it, I'll ping you
21:05:11 <strigazi> In heat at least you can do stack update and go back in a good state
21:05:50 <strigazi> #action strigazi to push a patch for allowing cluster-update on UPDATE_FAILED clusters
21:06:30 <strigazi> I would also like to ask for review on making flannel self hosted:
21:06:46 <strigazi> https://review.openstack.org/#/c/597150/
21:06:56 <imdigitaljim> oh awesome!
21:07:07 <imdigitaljim> that will make managing CNI a little cleaner :D
21:07:10 <cbrumm> we'll check that out too
21:07:18 <strigazi> Laste week we rebooted hypervisors for l1tf and many nodes didn't have network
21:07:38 <imdigitaljim> strigazi: i had been working on some changes that would be good to add to https://review.openstack.org/#/c/585420/
21:08:30 <strigazi> imdigitaljim: I'll strip the patch from the controller manager container, and just have CI code + git mv the container agent
21:08:41 <strigazi> other changes can be added on top
21:08:53 <strigazi> You can push a patch with this one as dependency
21:09:19 <strigazi> imdigitaljim: The changes are for containers or the ci?
21:09:40 <imdigitaljim> some additional changes for the containers
21:09:56 <imdigitaljim> sorry some additional containers*
21:10:00 <imdigitaljim> and only a couple changes
21:10:27 <imdigitaljim> I have a latest (3.3.9) minimal etcd container
21:10:32 <strigazi> if they are not for the CI let's do them ina follow-up patch
21:10:50 <imdigitaljim> and a heat-container-agent using alpine as its base
21:10:53 <imdigitaljim> sounds good
21:10:54 <strigazi> imdigitaljim: sounds good, based on quai.io?
21:11:08 <imdigitaljim> the etcd is
21:11:08 <strigazi> for etcd ^^
21:11:23 <imdigitaljim> FROM quay.io/coreos/etcd:$ETCD_VERSION
21:11:29 <strigazi> cool
21:11:47 <strigazi> 3.3.x cleans the log of the api server
21:11:56 <imdigitaljim> :)
21:11:58 <imdigitaljim> and its much smalelr
21:12:04 <strigazi> from blah blah compacted blah blah
21:12:57 <imdigitaljim> I've also finalized the calico update to latest
21:13:16 <imdigitaljim> on our side, i'll get it all updated soon
21:13:25 <imdigitaljim> had a slow ramp-up since our trip :]
21:13:44 <strigazi> :)
21:14:27 <strigazi> oh, right, can you also update the no-keypair option?
21:14:48 <strigazi> I'll also update the patch to pass the actual public key as a string
21:15:03 <flwang> sorry for the late
21:15:13 <strigazi> not a keypair object, to allow cluster updates
21:15:17 <strigazi> flwang: welcome
21:16:01 <strigazi> Since flwang is here,
21:16:07 <imdigitaljim> https://review.openstack.org/#/c/590443/
21:16:08 <imdigitaljim> this one/
21:16:08 <imdigitaljim> ?
21:16:16 <imdigitaljim> flwang: hi! o/
21:16:45 <strigazi> imdigitaljim: oh, I thought it was in conflict, looks good
21:16:55 <strigazi> yes this one
21:16:59 <flwang> imdigitaljim: strigazi: i'm keen  to know the output of Blizzard visiting to CERN ;)
21:17:58 <strigazi> flwang: we missed you :)
21:18:02 <imdigitaljim> strigazi: i think going into the next couple weeks ill be looking into the in-place upgrade mechanism as well
21:18:09 <imdigitaljim> flwang: ^
21:18:42 <strigazi> cool
21:19:08 <imdigitaljim> did you have any new updates to that?
21:19:17 <strigazi> flwang: eandersson might have a fix for the UTs for the cluster healing cmd
21:19:49 <colin-> strigazi, flwang are you guys using Designate? is anyone else who might be lurking using it? just curious
21:19:53 <strigazi> flwang: we just need to test if it works in an actual py36 env
21:20:33 <strigazi> imdigitaljim: not much, we can sync tmr if you can
21:20:39 <imdigitaljim> yeah sounds good!
21:20:44 <flwang> strigazi: yep, i saw that
21:21:03 <strigazi> colin-: we have our own DNS, many years old :)
21:21:06 <flwang> i will test it in with both UT and run it in a py36 env
21:21:18 <imdigitaljim> also ill rebase/update the previous PRs with some new changes, such as starting to use config file instead of flags to move forward from deprecation
21:21:24 <flwang> colin-: no, we haven't enabled Designate, but it's on our roadmap
21:21:36 <eandersson> I wish we knew someone on the Designate team
21:21:36 <colin-> ok
21:21:38 <imdigitaljim> and kube-proxy daemonset most likely
21:22:47 <strigazi> imdigitaljim: sounds good, I would do proxy first
21:24:09 <strigazi> imdigitaljim: flwang in calico, can you tell easily which pods dont' have network?
21:24:56 <strigazi> imdigitaljim: flwang with the l1tf reboots, the nodes were up, they had ips but not network acccess
21:25:03 <flwang> strigazi: what do you mean don't have network? there is a calico command can help you understand the network status of each pod
21:25:12 <imdigitaljim> yeah
21:25:19 <flwang> i had a blog but i didn't have time to finish it :( shame on me
21:25:20 <strigazi> flwang:  ping each other
21:25:25 <imdigitaljim> and you can generally see that the daemonset on the nodes isnt coming online
21:25:28 <imdigitaljim> you can see which nodes are having issues
21:26:07 <strigazi> well, in our case, flannel was up, the pods too but they couldn't connect to each other
21:26:17 <flwang> strigazi: yep, sure, the debug process with calico is fairly normal like general network debug
21:26:44 <strigazi> ok, I was looking for a heartbeat or smth
21:26:58 <flwang> 1. make sure pod can talk to local node, 2. make sure the node can talk each other
21:27:32 <flwang> need some iptables knowledge
21:27:39 <strigazi> yes, but it is a bit manual. I was doing this exacltly
21:27:48 <flwang> due to involving the network policy
21:28:32 <flwang> strigazi: yep, not sure if there is a fully automated way to debug it, but given it's normal process, we could probably write a script
21:29:03 <flwang> we're deploying magnum into prod in catalyst cloud this week, so pls forgive my latency for upstream work
21:29:16 <strigazi> no prob
21:29:17 <colin-> gl :)
21:29:21 <imdigitaljim> yeah gl!
21:29:38 <cbrumm> go flwang!
21:29:44 <eandersson> \o/
21:30:00 <flwang> cross finger for me :D
21:30:06 <strigazi> keep us posted :)
21:30:14 <flwang> sure
21:30:21 <flwang> ah, i do have a question
21:30:35 <strigazi> if heat is going well, magnum will go well too
21:30:41 <flwang> based on the code, magnum doesn't support rotate certs for k8s
21:30:53 <strigazi> no, it doesn't
21:30:57 <flwang> but we're showing the menu on magnum-ui
21:31:15 <flwang> shouldn't we drop it until we can really support it
21:31:25 <strigazi> I think if you change the policy, it will hide
21:31:41 <strigazi> not 100% sure
21:31:53 <flwang> strigazi: good point, will try
21:32:14 <strigazi> I can ask tmr in our team
21:33:02 <flwang> https://ibb.co/cFxT4z
21:33:25 <flwang> that's the dashboard we're using in catalyst cloud
21:34:06 <strigazi> it looks cool
21:34:38 <strigazi> are you in the openstack passport program, not sure how it is called
21:34:47 <flwang> strigazi: we're
21:35:04 <strigazi> uses will see magnum there?
21:35:18 <flwang> strigazi: yes, for sure, but not now
21:35:26 <strigazi> nice
21:35:29 <flwang> probably next couple of weeks
21:35:38 <strigazi> very cool
21:35:48 <flwang> https://catalystcloud.nz/services/paas/catalyst-kubernetes-service/
21:36:05 <flwang> we're using the vanilla upstream Magnum now
21:36:15 <flwang> and we'd like to stick on that as much as we can
21:36:42 <flwang> we'd like to upstream whatever can benefit others
21:36:56 <strigazi> :)
21:38:02 <strigazi> It seems we are covered, anything else to discuss?
21:39:15 <flwang> no
21:39:17 <flwang> im good
21:39:24 <imdigitaljim> no all set here :)
21:39:27 <flwang> ah, one small thing
21:39:38 <flwang> i can manage to get the ds work for keystone auth
21:39:51 <flwang> i will propose new patch set early next week
21:40:07 <strigazi> +1
21:41:14 <strigazi> Thanks for joining imdigitaljim flwang cbrumm colin- eandersson
21:41:22 <eandersson> Anytime!
21:41:32 <flwang> strigazi: thank you
21:41:32 <strigazi> #endmeeting