20:59:04 <strigazi> #startmeeting containers
20:59:06 <openstack> Meeting started Tue Aug 14 20:59:04 2018 UTC and is due to finish in 60 minutes.  The chair is strigazi. Information about MeetBot at http://wiki.debian.org/MeetBot.
20:59:07 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
20:59:09 <openstack> The meeting name has been set to 'containers'
20:59:15 <strigazi> #topic Roll Call
20:59:16 <imdigitaljim> o/
20:59:20 <strigazi> o/
20:59:33 <colin-> \o
21:00:47 <strigazi> I guess flwang prefers working at midnight not midday :)
21:00:51 <strigazi> #topic Announcements
21:00:52 <imdigitaljim> haha
21:01:32 <strigazi> we have stable/rocky and magnum 7.0.0
21:01:49 <imdigitaljim> \o/
21:01:55 <strigazi> we have still patches to add so 7.0.1 would be the release
21:02:17 <strigazi> \o/ indeed :)
21:02:33 <strigazi> #topic Blueprints/Bugs/Ideas
21:02:49 <strigazi> I promised canori01 to tests coreos and I did
21:03:05 <strigazi> and apparently it is me that kind of broke it
21:03:14 <strigazi> I mean the coreos driver
21:03:32 <strigazi> details here:
21:03:37 <strigazi> #link https://review.openstack.org/#/c/579026
21:03:57 <imdigitaljim> didnt you go to the fedora convention?
21:04:08 <imdigitaljim> what are the atomic/coreos plans that you migth know of
21:04:15 <strigazi> The issue is passing certs with heat and using string replacement
21:04:44 <imdigitaljim> #link https://review.openstack.org/#/c/590443/
21:04:50 <imdigitaljim> #link https://review.openstack.org/#/c/590346/
21:04:56 <imdigitaljim> #link https://review.openstack.org/#/c/589214/
21:05:00 <strigazi> cloud-config in coreos != cloud-init
21:05:02 <imdigitaljim> #link https://review.openstack.org/#/c/577570/
21:05:17 <imdigitaljim> are all ready for review
21:05:59 <strigazi> thanks
21:05:59 <colin-> this reminds me, i was wondering strigazi or others if anyone has used hashicorp's Vault as a sidecar certificate authority in the cluster?
21:06:25 <colin-> to ease some of these certificate management/distribution tasks?
21:06:43 <strigazi> colin-: no, at least at CERN we have barbican
21:07:03 <strigazi> there is work to integrate barbican and k8s
21:07:26 <colin-> ok
21:07:27 <strigazi> eg barbican sdk was added to gophercloud
21:07:52 <colin-> that's great, the interest is less in Vault in more in a more graceful way to manage all the TLS files
21:07:59 <imdigitaljim> were working on org approval to work on kubernetes org repo to contribute several PRs for the standalone openstack controller
21:08:13 <imdigitaljim> which will also translate to some magnum PRs
21:08:23 <strigazi> cool :)
21:08:26 <canori02> o/
21:09:05 <strigazi> canori02: o/
21:09:48 <strigazi> imdigitaljim: have you pushed a patch to add kube-proxy already?
21:10:05 <imdigitaljim> id need the two files merged to apply the rest
21:10:12 <imdigitaljim> make-cert and configure-master
21:10:28 <imdigitaljim> because it uses elements of those to complete it
21:10:42 <strigazi> imdigitaljim: ok
21:11:03 <imdigitaljim> however the work has been already done
21:11:07 <imdigitaljim> just a matter of sequence
21:11:41 <strigazi> you can push with depencies in gerrit, I guess you know this
21:11:56 <strigazi> *dependencies
21:12:14 <imdigitaljim> i did not actually know, im not super knowledgeable with gerrit
21:12:16 <imdigitaljim> ill check it out
21:12:23 <imdigitaljim> and push it
21:12:53 <strigazi> https://docs.openstack.org/infra/manual/developers.html#adding-a-dependency
21:12:58 <imdigitaljim> oh awesome
21:12:58 <imdigitaljim> thanks
21:13:03 <imdigitaljim> that will make it easy
21:13:18 <strigazi> three more items from me:
21:13:56 <strigazi> 1. i'm working on the unit tests for the upgrade API, of course I love writing unit tests
21:14:16 <strigazi> 2. with flwang we made some progress in :
21:14:43 <strigazi> #link https://review.openstack.org/#/c/572897/
21:14:57 <strigazi> pulling the k8s cluster for health status
21:15:25 <strigazi> and return something like {api_health: {}, nodes: []}
21:16:18 <strigazi> 3. k8s 1.11.x works without any issue, all patches for the certificates are merged and the contianer images are updated
21:16:41 <imdigitaljim> yep!
21:16:57 <imdigitaljim> we're also on 1.11.2
21:17:17 <strigazi> really easy now :)
21:17:56 <strigazi> the patch is 6 character, it would be 1 if I used a variable :)
21:18:08 <imdigitaljim> we recently updated calico to 3.1.3
21:18:18 <imdigitaljim> we're like 17 releases behind in magnum
21:18:34 <strigazi> args in dockerfiles require a newer docker version
21:18:36 <imdigitaljim> probably more relevant to flwang:
21:18:42 <strigazi> imdigitaljim: what we have in magnum atm
21:18:48 <strigazi> ?
21:18:48 <imdigitaljim> 2.6.7
21:19:14 <strigazi> I wonder why flwang pushed for 2.6.x
21:19:25 <imdigitaljim> there were some 3.x issues at the time
21:19:28 <strigazi> I think 3.x.y was out at that time
21:19:29 <imdigitaljim> but to us they appear to be resolved
21:19:34 <strigazi> oh, ok
21:19:37 <strigazi> makes sense
21:19:41 <imdigitaljim> yeah he was probably doing what was necessary
21:19:58 <strigazi> Finally, a comment for Fedora CoreOS
21:20:20 <strigazi> we need at least 6 months fedora having something solid to use
21:20:40 <strigazi> All the builds they have now are experimental and none of them are public
21:20:59 <strigazi> Fedora Atomic 30 will be the last one
21:21:17 <strigazi> Fedora CoreOS will be based on rpms
21:21:24 <strigazi> and will rpm-ostree
21:21:28 <strigazi> and will use rpm-ostree
21:21:29 <imdigitaljim> will we still be able to use the same files that you know of? "atomic install etc"
21:21:50 <strigazi> atomic cli no, but something similar
21:22:03 <imdigitaljim> ok so we'll need to make some changes
21:22:28 <imdigitaljim> maybe we can get a good way to prebuild the new images as well quickly
21:22:29 <strigazi> they said since there are users like us they will take it into account
21:22:41 <imdigitaljim> so we can inject extra stuff on the base fedora coreos image
21:23:00 <strigazi> regaring that, we will need to work with ignition
21:23:28 <strigazi> we can start investigating this with coreos and be ready
21:23:41 <strigazi> we need a way to compile the ignition json
21:24:11 <strigazi> we can join the fedora coreos meeting on the 21st
21:24:19 <imdigitaljim> https://coreos.com/ignition/docs/latest/
21:24:22 <imdigitaljim> got it
21:24:24 <strigazi> imdigitaljim: are you interested?
21:24:35 <imdigitaljim> that would be good actually
21:25:19 <strigazi> #link https://apps.fedoraproject.org/calendar/workstation/2018/8/20/#m9315
21:25:51 <strigazi> 7:30 for me, great...
21:25:56 <strigazi> imdigitaljim: for you?
21:26:50 <strigazi> I can ask for the other one, if it is better, I think they will alternate
21:26:54 <imdigitaljim> 1030PM
21:27:00 <imdigitaljim> i should be able to make it if i remember :p
21:27:12 <strigazi> imdigitaljim: I can ping you :)
21:27:58 <strigazi> speaking of atomic, I tested F28AH, works fine with magnum 7.0.0
21:28:18 <strigazi> and queens actually, I'll push a patch
21:29:28 <strigazi> I think that was all from me, imdigitaljim colin- canori02  do you want to add anything?
21:30:12 <canori02> How far away is fedora coreos?
21:30:43 <strigazi> at least 6 months
21:31:09 <canori02> I ask since the work I've been doing makes the coreos driver work through cloud-init and not ignition
21:31:25 <canori02> Should we just deprecate that?
21:31:57 <strigazi> I would help
21:32:00 <strigazi> It would help
21:32:31 <strigazi> I think we can add your patch in rocky and then work on ignition
21:32:40 <strigazi> canori02: makes sense?
21:33:00 <canori02> Yeah, makes sense
21:33:27 <imdigitaljim> nothing else here
21:33:33 <canori02> Also, had another question. Sorry if you covered this already since I was late
21:34:09 <canori02> What were your guy's thougts on the discovery.etcd.io issues they were having?
21:34:50 <strigazi> discovery.etcd.io is back, but they plan to deprecate it, but without a clear timeline
21:35:23 <imdigitaljim> canori02: if you want to put the effort with it, this can be run internally/locally
21:35:32 <strigazi> magnum has the option to use a local discovery so we can use that
21:35:57 <flwang> strigazi: re calico
21:36:09 <strigazi> I posted a link in the channel last week on how to do it
21:36:14 <flwang> when I worked on calico, GKE is also using 2.6.7
21:36:18 <canori02> Yeah, I did deploy one internally. Wasn't too bad
21:36:35 <flwang> so I trust google so I assume 2.6.7 is stable at least
21:36:53 <strigazi> sounds good ^^
21:37:13 <flwang> and given we have calico node tag, so user can easily upgrade if they want
21:37:34 <strigazi> flwang: we could change the default?
21:37:37 <flwang> new version is cool, but i'm always a old man style, so....
21:37:42 <flwang> strigazi: we can
21:37:56 <flwang> i can test with 3.x
21:38:12 <flwang> and propose the version upgrade if it can pass the sonobuoy testing
21:39:01 <colin-> do you remember what they were version locking on 2.6.x to support flwang? i seem to recall it being related to IPVS use for kube-proxy but could be wrong
21:39:08 <strigazi> canori02: we can start an issue in etcd repo to ask what they recommend
21:39:39 <strigazi> I imagine they will say, run your own discovery
21:40:20 <strigazi> bootstraping etcd without knowing the ips beforehand is tedious
21:42:54 <canori02> I saw they were asking for feedback on how we use the service.  So we can give them that
21:44:05 <flwang> colin-: i can't remember, sorry
21:44:18 <strigazi> yeap, you can reply, I can follow it up too
21:44:38 <flwang> for etcd discovery issue, at least, we can add a retry for the function
21:45:21 <imdigitaljim> also the calico_tag wont work entirely
21:45:24 <imdigitaljim> the yaml format changed
21:45:30 <imdigitaljim> specifically plugins
21:45:48 <imdigitaljim> minor changes though
21:45:57 <imdigitaljim> flwang:^
21:46:22 <imdigitaljim> flwang: we've also gotten sonobuoy set up as well
21:48:01 <strigazi> imdigitaljim: flwang it always takes one hour to run?
21:48:39 <imdigitaljim> yeah
21:48:47 <imdigitaljim> its like 67 minutes for us
21:49:03 <imdigitaljim> although sonobuoy make some more assumptions on a fwe things but overall its pretty good
21:49:51 <strigazi> what assumptions? example?
21:51:21 <imdigitaljim> 1 sec
21:55:01 <flwang> sorry, i was in a meeting and going into another one
21:55:17 <flwang> strigazi: yes, 1 hour
21:55:22 <strigazi> flwang: enjoy :)
21:55:30 <flwang> but i'm going to dig to see if we can have a smoke test set
21:56:43 <imdigitaljim> i was gonna try to link the codeline but i cant find it right this second
21:56:44 <strigazi> imdigitaljim: are you still there?
21:56:44 <flwang> imdigitaljim: yep, that's a good point, i will check if we should upgrade to 3.x
21:57:00 <imdigitaljim> but basically its assuming a master node is labeled in a specific way
21:57:03 <imdigitaljim> and its not even a good way
21:57:22 <imdigitaljim> i think sonobuoy pulls from the kk e2e anyways so it might actually just be a bad kk test
21:57:44 <imdigitaljim> so it skips like 100s of tests
21:57:49 <imdigitaljim> based solely on that
21:58:29 <strigazi> imdigitaljim: do you have the name of the test? I didn't see anything in the logs about the label
21:58:41 <imdigitaljim> yeah ill dig it up again
21:58:47 <strigazi> maybe I missed ti
21:58:51 <flwang> imdigitaljim: i'm interested in too
21:59:04 <strigazi> thanks, when I run again I'll look closer
21:59:07 <imdigitaljim> i could have totally missed something too so it would be good to find out
22:00:01 <strigazi> let's end the meeting then
22:00:18 <strigazi> see you next week everyone
22:00:47 <strigazi> #endmeeting