10:00:32 <strigazi> #startmeeting containers
10:00:33 <openstack> Meeting started Tue Mar 27 10:00:32 2018 UTC and is due to finish in 60 minutes.  The chair is strigazi. Information about MeetBot at http://wiki.debian.org/MeetBot.
10:00:34 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
10:00:36 <openstack> The meeting name has been set to 'containers'
10:00:51 <strigazi> #topic Roll Call
10:00:55 <flwang> o/
10:00:57 <ricolin> o/
10:01:03 <strigazi> o/
10:01:22 <slunkad> hi
10:02:36 <strigazi> agenda: https://wiki.openstack.org/wiki/Meetings/Containers#Agenda_for_2018-03-27_1000_UTC
10:02:37 <rochapor1o> o/
10:02:56 <strigazi> #topic Announcements
10:03:46 <strigazi> Two bugs have been found concerning the K8s_fedora_atomic driver
10:04:20 <strigazi> 1. etcd since queens runs with certs over https but accepts requests without auth
10:05:04 <strigazi> The I have built a new container to address this and I'll push to docker.io/openstackmagnum
10:05:35 <strigazi> I'll push a patch to have the right config inplace
10:06:03 <flwang> strigazi: sorry, i didn't get, why is it related to the etcd container image?
10:06:20 <strigazi> 2. for all releases, kubelet is unprotected. The fix is in this patch
10:06:26 <strigazi> flwang: give me a sec
10:06:38 <flwang> strigazi: sure
10:06:44 <strigazi> fix for 2 https://review.openstack.org/#/c/556213/
10:06:50 <strigazi> for 1.
10:07:14 <strigazi> flwang: this default is the problem https://github.com/projectatomic/atomic-system-containers/blob/master/etcd/manifest.json#L33
10:07:45 <strigazi> We just need to set it to true in our etcd config
10:07:51 <strigazi> flwang: makes sense?
10:08:17 <flwang> strigazi: ok, we should blame atomic system container team anyway
10:08:23 <flwang> ;)
10:08:30 <strigazi> :)
10:08:55 <strigazi> #topic Blueprints/Bugs/Ideas
10:10:03 <strigazi> I have found three bp that I would need some help and are pretty useful for the next release
10:10:06 <rochapor1o> i'm happy to take : https://blueprints.launchpad.net/magnum/+spec/add-hidden-flag-to-baymodel
10:10:19 <strigazi> the easiest one is Add functional tests for kubernetes service with Load Balancer https://blueprints.launchpad.net/magnum/+spec/lb-functional-tests
10:11:05 <strigazi> add-hidden-flag-to-baymodel is, when having public CTs
10:11:30 <strigazi> to be able to hide them from CT list
10:11:43 <strigazi> and not allow users to create new cluster with them
10:12:08 <flwang> strigazi: hmmm
10:12:19 <flwang> that's is the typical issue in Glance
10:12:37 <flwang> i'm happy to review the spec
10:13:13 <strigazi> Now, if you have a public CT and a single cluster created with it, you can't touch the CT with the API
10:13:19 <strigazi> cool
10:13:34 <rochapor1o> i'll have to leave a bit early today, will check the meeting log later
10:13:45 <strigazi> rochapor1o: thanks
10:13:47 <flwang> rochapor1o: waves from NZ
10:14:11 <strigazi> flwang: since you have octavia you might be interested in lb-functional-tests
10:14:27 <flwang> strigazi: yep, we can do it i think
10:14:55 <strigazi> flwang: it is just to add a new test case so when reviewing we can easily test it
10:15:02 <flwang> when we say 'functional test', is it in tempest or just in magnum repo?
10:15:07 <ricolin> strigazi, I wounder where you plan to run those k8s load balancer tests
10:15:09 <strigazi> magnum repo
10:15:20 <strigazi> ricolin: locally of course
10:15:32 <strigazi> ricolin: in infra we could only dream about it
10:15:46 <flwang> strigazi: got it
10:16:00 <ricolin> strigazi, okay
10:16:05 <slunkad> so the functional tests in the magnum repo can't be run via tempest?
10:16:14 <flwang> strigazi: have we fixed the k8s gate?
10:16:33 <ricolin> flwang, not for now
10:16:33 <flwang> will it be a part of the k8s gate job?
10:16:34 <strigazi> there is a repo just for the magnum tempest tests
10:16:35 <ricolin> infra issue
10:16:54 <strigazi> ricolin: flwang our job is fine, it just heavy
10:17:00 <strigazi> ricolin: flwang our job is fine, it is just heavy
10:17:17 <flwang> strigazi: i see
10:17:23 <strigazi> when it ends up in a fast machine it runs in under 50mins
10:18:37 <strigazi> flwang: Can you take it then?
10:18:44 <openstackgerrit> yanghuichan proposed openstack/magnum master: Signed-off-by: yanghuichan <yanghc@fiberhome.com>  https://review.openstack.org/556812
10:19:23 <flwang> strigazi: ok
10:19:50 <flwang> i think it's nice to have
10:20:08 <strigazi> it shouldn't be a big change
10:20:58 <flwang> strigazi: fair enough
10:21:46 <strigazi> ok next is strigazi to report back on cluster upgrades
10:23:28 <strigazi> While trying to implement it with passing parameters via a new CT, I found that since we use labels for many parameters we might need to sets of labels
10:24:10 <flwang> what do you mean 'sets of labels'?
10:24:31 <strigazi> Instead of going there dirrectly, I'll add in the driver upgrade logic which labels can be upgraded and which not.
10:24:39 <strigazi> flwang: for example
10:24:54 <strigazi> the network-driver options are not be changed
10:25:20 <strigazi> the network_cidr must not change but the network container tag might change
10:25:35 <flwang> i see
10:25:44 <strigazi> Some labels is good to have the option to change them, others not.
10:26:40 <flwang> strigazi: totally agree, and given we're heavily using labels, i think we do need some policies/rules how to use/manage them
10:26:58 <strigazi> Also, on client side I'll add a --dry-run or verify parameter to inform the user which values are going to change
10:28:44 <strigazi> At the moment, I'm moving most software configs to software deployments
10:29:18 <strigazi> So, we don't have to have separate scripts for upgrade
10:29:57 <strigazi> and when changing a software config the node it replaced completely VS rebuild in the same hypervisor
10:30:17 <strigazi> user_data are immutable in nova
10:30:23 <strigazi> that's why
10:30:51 <strigazi> any questions?
10:31:27 <ricolin> +1 on that:)
10:31:29 <flwang> so you mean with heat-container-agent, we can rerun those script on same node?
10:31:44 <strigazi> yes
10:32:08 <flwang> nice
10:32:40 <strigazi> next: slunkad to report on "trust invalid when user is disabled" https://bugs.launchpad.net/magnum/+bug/1752433
10:32:42 <openstack> Launchpad bug 1752433 in Magnum rocky "trust invalid when user is disabled" [High,New] - Assigned to Sayali Lunkad (sayalilunkad)
10:32:46 <flwang> is it possible create another kubelet container on same node and switch them when upgrade?
10:32:58 <flwang> never mind, we can discuss this later
10:33:02 <openstackgerrit> yanghuichan proposed openstack/magnum master: Fix wrong links in magnum  https://review.openstack.org/556812
10:33:25 <strigazi> slunkad: I'll discuss this also with the heat team and ricolin tmr at 1400UTC
10:33:35 <strigazi> slunkad: did have time to look at it?
10:33:39 <slunkad> ya, I've been a bit stuck with what goes where but I think i will push a patch soon which will need quite some work
10:34:08 <slunkad> strigazi: ok in the heat weekly meeting?
10:34:12 <strigazi> slunkad: ok
10:34:14 <strigazi> slunkad: yes
10:34:23 <slunkad> ok will try to drop in too
10:34:34 <strigazi> slunkad: ricolin added it in the agenda
10:34:42 <ricolin> strigazi, already add a topic in heat Agenda
10:35:05 <strigazi> cool, thanks
10:35:42 <strigazi> next, strigazi to push a fix for flannel + dashboard over kubectl proxy I'll push it this week, last week it was all about the two security issue
10:36:02 <strigazi> *issues
10:36:57 <strigazi> there is this related bug: https://bugs.launchpad.net/magnum/+bug/1757936
10:36:58 <openstack> Launchpad bug 1757936 in Magnum rocky "apiserver can't access heapster" [Undecided,In progress]
10:37:26 <flwang> strigazi: can i highlight my patch related this?
10:37:34 <strigazi> yes
10:37:36 <flwang> calico network driver has same issue
10:37:47 <strigazi> #action strigazi to push a fix for flannel + dashboard over kubectl proxy
10:37:52 <flwang> we need run calico-node on master as well
10:38:17 <flwang> patch is here https://review.openstack.org/548139
10:38:36 <strigazi> #action strigazi to review/merge https://review.openstack.org/548139
10:38:38 <flwang> strigazi: ykarel: pls help review it
10:39:45 <strigazi> noted, sorry about it flwang
10:40:25 <strigazi> two more items:
10:40:28 <flwang> strigazi: no worries
10:40:29 <strigazi> slunkad Factor out the terminology guide
10:40:57 <slunkad> strigazi: oh yes i will push that patch today. Is there a bp already?
10:40:58 <strigazi> ^^ is this done slunkad ? Is it ready for review?
10:42:54 <slunkad> yes will push it soon
10:43:03 <strigazi> slunkad: there is now slunkad Factor out the terminology guide
10:43:11 <strigazi> slunkad: there is now https://blueprints.launchpad.net/magnum/+spec/docs-refactor
10:43:21 <slunkad> thanks
10:43:30 <strigazi> what is your launchpad id?
10:43:57 <slunkad> sayalilunkad
10:44:08 <strigazi> thanks
10:44:19 <strigazi> and last item,
10:44:26 <strigazi> flwang "Investigate cotyledon as a replacement of oslo.service"
10:44:37 <strigazi> do you have some input from kong as well?
10:45:13 <flwang> strigazi: yes, it's doable and i have started to code
10:45:27 <flwang> will push a patch in next 1-2 weeks
10:46:00 * kong sees his name hilighted
10:46:25 <strigazi> flwang: this means we need to remove oslo.service completely right?
10:46:43 <strigazi> kong: told me you had some experience with cotyledon
10:46:48 <strigazi> kong: flwang told me you had some experience with cotyledon
10:47:16 <kong> strigazi: i just adtoped cotyledon in Qinling project
10:47:26 <kong> we met with the same issue as you
10:47:35 <kong> k8s client doesn't support eventlet
10:47:58 <strigazi> kong: is cotyledon still maintained?
10:48:05 <kong> strigazi: i think so
10:48:47 <strigazi> this is my only concern
10:49:09 <kong> strigazi: https://github.com/sileht/cotyledon/commits/master
10:49:32 <strigazi> i contacted the maintainer to ask about the status but he didn't reply
10:50:15 <kong> actually, you just need to check the commits in that repo
10:51:03 <strigazi> let's see how it goes. Yes but if he doesn't have stake at the project it is a bit different
10:51:26 <strigazi> the commits are a hint
10:51:35 <kong> strigazi: do you have another option to work around the eventlet issue?
10:52:31 <strigazi> not use the python kubernetes client and use python-requests  or use the kubectl binary or fix eventlet :)
10:53:37 <strigazi> I don't know if we have another option
10:53:40 <kong> or use an older version of k8s client
10:54:19 <strigazi> that's an option as well
10:54:39 <kong> i remember the problem was introduced in a recent version
10:54:40 <flwang> strigazi: or just write a simeple requests call
10:54:55 <flwang> or a wrapper
10:55:10 <flwang> so that we can easily replace the underhood pieces
10:55:11 <strigazi> yes, I mentioned that too, to use python-requests
10:55:44 * kong needs go to bed, leaves the decision to you guys
10:55:53 <strigazi> thanks kong
10:56:05 <kong> strigazi: you are welcome
10:56:21 <strigazi> @all anything else?
10:56:32 <strigazi> we have 4 mins left
10:56:50 <ricolin> not from me
10:56:55 <slunkad> strigazi: I wanted to ask about running the functional tests via tempest, but we can do that after too if you have time
10:56:59 <strigazi> flwang: let's discuss it with rochapor1o as well
10:57:13 <flwang> strigazi: ok
10:57:35 <strigazi> slunkad: ok https://github.com/openstack/magnum/blob/master/magnum/tests/contrib/post_test_hook.sh#L204
10:58:03 <strigazi> let's end the meeting then
10:58:12 <strigazi> thanks folks
10:58:15 <strigazi> #endmeeting