15:01:24 <mgoddard> #startmeeting kolla
15:01:25 <openstack> Meeting started Wed Sep 18 15:01:24 2019 UTC and is due to finish in 60 minutes.  The chair is mgoddard. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:01:26 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
15:01:30 <openstack> The meeting name has been set to 'kolla'
15:01:31 <mgoddard> #topic rollcall
15:01:37 <mgoddard> \o
15:01:39 <hrw> /o\
15:01:40 <yoctozepto> o/
15:01:40 <scottsol> o/
15:01:42 <chason> o/
15:01:56 <mnasiadka> o/
15:01:58 <generalfuzz> o/
15:02:00 <priteau> o
15:02:03 <priteau> o/
15:02:31 <yankcrime> \o
15:02:49 <mgoddard> welcome back yankcrime
15:02:52 <yankcrime> hi mgoddard
15:03:08 <mgoddard> #topic agenda
15:03:16 <mgoddard> * Roll-call
15:03:19 <mgoddard> * Announcements
15:03:21 <mgoddard> ** Vote for Kayobe Train planning meeting time: https://doodle.com/poll/y7fakbhbkfx5hqyk
15:03:23 <mgoddard> * Review action items from last meeting
15:03:26 <mgoddard> * CI status
15:03:27 <mgoddard> * Train release planning
15:03:29 <mgoddard> * Review priorities
15:03:31 <mgoddard> * Kayobe Train planning
15:03:45 <mgoddard> #topic announcements
15:03:51 <mgoddard> #info Vote for Kayobe Train planning meeting time
15:03:57 <mgoddard> #link https://doodle.com/poll/y7fakbhbkfx5hqyk
15:04:16 <mgoddard> Will discuss this a little more later
15:04:32 <mgoddard> #info CentOS 7.7 released yesterday
15:04:42 <mgoddard> only minimal CI breakage this time
15:05:19 <mgoddard> #info CentOS 8 released next week, 24th
15:06:03 <hrw> In october I may take a look at centos:8 builds
15:06:21 <mgoddard> #info OpenStack projects releasing RC1 next week, final release 14-18 Oct
15:06:32 <mgoddard> that will be good hrw
15:07:07 <mgoddard> we should try to get ahead as soon as possible in case there are problems we need to report back
15:07:12 <mgoddard> Any others?
15:07:25 <yoctozepto> ipv6 is not a goal of deployment projects in train
15:07:33 <yoctozepto> probably in ussuri
15:07:36 <mgoddard> Right
15:07:40 <yoctozepto> so we are step ahead 8-)
15:07:41 <mgoddard> still nice to have
15:07:45 <mgoddard> always
15:07:50 <yoctozepto> indeed nice
15:07:52 <mgoddard> (when complete)
15:08:01 <mgoddard> #topic Review action items from last meeting
15:08:16 <mgoddard> There were none
15:08:22 <mgoddard> #topic CI status
15:08:54 <openstackgerrit> Radosław Piliszek proposed openstack/kolla stable/rocky: base: fix OracleLinux build  https://review.opendev.org/682927
15:09:01 <openstackgerrit> Radosław Piliszek proposed openstack/kolla stable/queens: base: fix OracleLinux build  https://review.opendev.org/682928
15:09:11 <mgoddard> We have issues with oraclelinux on stable branches. Fixed in stein, looks like yoctozepto is backporting as we speak
15:09:12 <yoctozepto> getting unbroken as we speak 8-)
15:09:41 <hrw> +2 on both
15:09:42 <yoctozepto> ol is not an issue of train
15:09:54 <yoctozepto> and we are dropping it nicely
15:10:05 <mgoddard> we had some flakiness in kayobe CI bare metal deploys, 'fixed' with iptables
15:10:27 <yoctozepto> why 'fixed'?
15:10:36 <mgoddard> https://review.opendev.org/#/c/681700/
15:11:10 <yoctozepto> weird indeed
15:11:19 <yoctozepto> at least it should always fail
15:11:27 <mgoddard> cause was docker CE adding a default DROP policy to the FORWARD chain, in combination with bridge-nf-call-iptables
15:11:55 <mgoddard> I never really understood why it only sometimes failed, and exactly what the problem was
15:12:28 <mgoddard> but a blanket ACCEPT on the FORWARD chain did the trick
15:12:29 <yoctozepto> worth noting if it does not break real deploys somewhere
15:12:54 <mgoddard> +1, docker's policy has caused a few different issues for us
15:12:55 <priteau> I've had issues on a Queens deployment using DVR
15:13:26 <priteau> I ended up running docker with --iptables=false
15:13:52 <yoctozepto> maybe it does something funny on the lower layers as well (raw, mangle, nat stuff)
15:14:24 <yoctozepto> I seem to forget about raw and it can be tricky left unchecked :D
15:14:33 <yoctozepto> (vide that ipv6 problem with firewalld on)
15:14:43 <mnasiadka> maybe it's time to add iptables: false to /etc/docker/daemon.json ;-)
15:14:51 <mgoddard> priteau: that is tempting. It wouldn't hurt kolla-ansible, I was just worried in case people run other containers without net=host
15:15:06 <priteau> There are security implications
15:15:16 <yoctozepto> mnasiadka, mgoddard, priteau: zun impact
15:15:47 <mnasiadka> yoctozepto: I don't care about Zun, who uses it anyway :)
15:16:10 <mgoddard> ouch
15:16:38 <yoctozepto> behave, mnasiadka, behave
15:16:40 <mnasiadka> Frankly speaking I never understood why we use the same container runtime for control plane and Zun
15:17:05 <yoctozepto> mnasiadka: because we cannot otherwise
15:17:21 <mnasiadka> yoctozepto: I would challenge that, but this is not the meeting to do it :)
15:17:34 <yoctozepto> yeah, I would like to hear that
15:17:42 <mgoddard> I think we have digressed
15:17:48 <yoctozepto> indeed
15:17:54 <mgoddard> #topic Train release planning
15:18:33 <mgoddard> According to our release status section, we have 2 weeks until our feature freeze
15:18:39 <mgoddard> Kolla feature freeze: Sep 30 - Oct 04
15:19:29 <mgoddard> I think we'll need to exclude centos8 & centos py3 from that
15:19:48 <mgoddard> Let's assess what other priorities look like
15:19:48 <hrw> yep
15:20:00 <yoctozepto> mgoddard: or do the standard 2.5 month delay
15:20:05 <mgoddard> support matrix won't be affected by freeze
15:20:33 <mgoddard> yoctozepto: we still froze a few weeks after everyone else IIRC
15:20:55 <yoctozepto> mgoddard: seems it got hit by hot winds then
15:21:01 <mgoddard> we'd just need to backport anything after RC1
15:21:14 <openstackgerrit> Merged openstack/kayobe stable/stein: Fix kayobe-overcloud-centos CI job flakiness  https://review.opendev.org/682829
15:21:24 <mgoddard> I think that's still a good approach, agree?
15:21:44 <mgoddard> ^ oh nice I can mark kayobe CI GREEN again
15:22:48 <mgoddard> Nova cells could be at risk - dougsz is away next week
15:23:26 <mgoddard> I'd like to get the nova part of that work merged this cycle
15:23:35 <mgoddard> (at least)
15:23:44 <mgoddard> https://review.opendev.org/675659
15:24:37 <mgoddard> We finished the ansible upgrade \o/
15:24:54 <mgoddard> Internal TLS is coming along nicely, first patch merged
15:25:25 <mgoddard> I don't know if we'll get the backend patch in this cycle
15:25:30 <mgoddard> but we'll see
15:26:05 <mgoddard> this one is now a dependency of TLS: https://review.opendev.org/#/c/587590
15:26:34 <mgoddard> and then it's this one: https://review.opendev.org/#/c/664517
15:26:45 <mnasiadka> I love those multi file small fixes :)
15:27:19 <hrw> maybe we should start use BP+1/-1 to mark what we need in Train and what goes for Ussuri?
15:27:20 <mgoddard> oh yeah, get all those CI jobs going
15:27:48 <mgoddard> I've been adding RP+1 for train priorities
15:28:06 <mgoddard> RP-1 seems a bit mean :)
15:28:17 <mgoddard> but maybe nearer the freeze it will help
15:28:38 <mgoddard> what's the status of health checks mnasiadka?
15:29:17 <mnasiadka> I doubt I'll get them in this cycle, looks like some debugging needed on docker-py level to get this working - and then there's a lot of work ahead
15:29:30 <yoctozepto> :-(
15:29:47 <mgoddard> k, good to know so we can focus elsewhere
15:30:16 <mgoddard> yoctozepto: reckon IPv6 can land in time?
15:30:35 <yoctozepto> mgoddard: I confirm, it works, CI is just grumpy :-)
15:30:51 <yoctozepto> (well, ubuntu worked well)
15:31:13 <mgoddard> ok
15:32:11 <mgoddard> I added patches to generate PDF docs
15:32:12 <mgoddard> https://review.opendev.org/682610
15:32:15 <mgoddard> https://review.opendev.org/682611
15:32:17 <mgoddard> https://review.opendev.org/682616
15:32:23 <mgoddard> shamelessly adapted from ironic's hard work
15:32:35 <mgoddard> it's a community goal, so I'll mark RP+1
15:33:02 <yoctozepto> seems we need more irrelevant files around
15:33:22 <mgoddard> maybe
15:33:26 <yoctozepto> no need to test docs with ansible
15:33:53 <mgoddard> had to fix requirements.txt for those: https://review.opendev.org/#/c/682646
15:34:22 <openstackgerrit> Michal Nasiadka proposed openstack/kolla-ansible master: WIP: Ceph-Ansible CI  https://review.opendev.org/676376
15:34:29 <hrw> +2 on pdf
15:34:50 <mgoddard> I think that's all the big stuff. I marked some things as being pushed out to Ussuri
15:35:21 <mgoddard> Did I miss anything?
15:36:01 <mgoddard> #topic Review priorities
15:36:12 <mgoddard> Anything we should focus on this week?
15:36:27 <mgoddard> already marked PDFs
15:37:01 <mgoddard> Cells
15:37:06 <yoctozepto> vxlan
15:37:11 <yoctozepto> ipv6
15:37:16 <yoctozepto> I do, you review
15:37:20 <hrw> -OL
15:37:43 <mgoddard> I think -OL went in already?
15:37:48 <yoctozepto> same here
15:37:59 <mgoddard> ah, not kolla
15:38:09 <mgoddard> added
15:38:13 <hrw> OL stuff: https://review.opendev.org/#/c/682927/ https://review.opendev.org/#/c/682928/ https://review.opendev.org/#/c/682865/
15:39:27 <mgoddard> yoctozepto: could you add bp/ topic to your IPv6 patches?
15:39:30 <mgoddard> https://review.opendev.org/#/q/topic:bp/ipv6-control-plane
15:40:09 <mgoddard> yoctozepto: feel free to add RP+1 to your IPv6 patches
15:40:17 <mnasiadka> mgoddard: what's the plan with ceph-ansible spec? do we want to merge it now, in Ussuri or never? ;-)
15:40:27 <mgoddard> mnasiadka: hmm
15:40:51 <mnasiadka> Ussuri might make sense, we'll know more after the experimental CI job with ceph-ansible will be merged ;-)
15:41:27 <mgoddard> mnasiadka: +2
15:41:43 <yoctozepto> mgoddard: ok, right when done
15:42:06 <mgoddard> mnasiadka: I think the direction is sound, even if we don't know when we'll do it
15:42:30 <mgoddard> Kayobe priorities this week?
15:42:42 <mgoddard> priteau, jovial[m]?
15:43:28 <priteau> I am going over the storyboard and identifying things we can fix in Train
15:44:50 <mgoddard> great. we can discuss the train planning in the next section, just looking for review priorities for this week atm
15:45:22 <mgoddard> we have a few bug fixes waiting for review, I'll pick some and mark RP+1
15:45:33 <priteau> Oh, sorry
15:45:39 <mgoddard> priteau: np
15:45:43 <mgoddard> Anyone know about kolla-cli
15:45:48 <priteau> Review priority: stable/queens backport https://review.opendev.org/#/c/679269/
15:45:59 <mgoddard> priteau: +1. Feel free to add RP+1
15:46:21 <mgoddard> Is kolla-cli going to be ready to release?
15:46:50 <yoctozepto> mgoddard: it looks it was a temporary activity burst
15:46:52 <jovial[m]> I'll try and review 679269 now
15:47:15 <mgoddard> yoctozepto: I have seen a few patches recently
15:48:05 <mnasiadka> mgoddard: do you have any contact with somebody that is merging that work? ;-)
15:48:23 <yoctozepto> they don't seem to visit irc channel
15:48:38 <mnasiadka> probably great firewall of China is to blame
15:48:46 <yoctozepto> probably
15:48:53 <mgoddard> yeah
15:49:01 <yoctozepto> let's mail them then
15:49:17 <mgoddard> mnasiadka: not really. Jeffrey4l is the only one from 99cloud who is on IRC AFAIK
15:49:49 <mgoddard> ok, last 10 minutes
15:49:52 <mnasiadka> mgoddard: I guess this is not very easy for him, I worked with some Chinese people in IBM, and they had to do a lot of tunnelling :-)
15:50:09 <mgoddard> I expect so
15:50:20 <yoctozepto> mgoddard: maybe he has higher clearance ;-)
15:50:35 <mgoddard> #action mgoddard to email Jeffrey4l & co about kolla-cli
15:50:38 <mgoddard> #topic Kayobe Train planning
15:50:46 <mgoddard> Ok kayobians
15:50:57 <mgoddard> priteau & jovial[m]
15:51:12 * yoctozepto just eavesdropping
15:51:18 <mgoddard> Currently no slot next week suits everyone :(
15:51:52 <mgoddard> If I allow for even hours does that help?
15:51:59 <mgoddard> couldn't see any way to overlap in doodle
15:53:25 <mgoddard> I think I'll just need to pick one and apologise
15:53:51 <mgoddard> We should have a think about what to talk about in advance. priteau started an etherpad
15:54:04 <jovial[m]> +1
15:54:13 <mgoddard> #link https://etherpad.openstack.org/p/kayobe-train-release
15:55:21 <mgoddard> I think we could follow a similar pattern to kolla PTG
15:55:21 <mgoddard> dump a list of things to discuss, then vote on priorities
15:55:21 <mgoddard> bearing in mind we don't have much time
15:55:32 <mgoddard> but priorities can always roll over
15:56:20 <mgoddard> so, bring your ideas for discussion, and comb storyboard for anything that we should be picking up
15:56:34 <openstackgerrit> Michal Nasiadka proposed openstack/kolla-ansible master: WIP: Ceph-Ansible CI  https://review.opendev.org/676376
15:56:47 <mgoddard> anything else we should be doing in those 2 hours?
15:57:05 <priteau> Discuss path forward for Kolla and Kayobe?
15:57:07 <jovial[m]> is remaining patches for stuff that already has a change up for review?
15:57:16 <priteau> Are there things we need to change in how the two integrate together
15:57:40 <mgoddard> priteau: good question
15:57:49 <mgoddard> it's a good time for open questions like that
15:58:01 <mgoddard> chuck it on the pad
15:58:34 <mgoddard> the format of the pad probably needs cleaning up
15:58:58 <mgoddard> it's more of a status tracking pad
15:59:05 <mgoddard> should we have a fresh one for ths discussion?
15:59:11 <priteau> +1
15:59:14 <mgoddard> ok
15:59:28 <mgoddard> https://etherpad.openstack.org/p/kayobe-train-planning
15:59:35 <priteau> Since we just released Stein, this is a kind of PTG meeting
15:59:41 <mgoddard> #link Welcome to Etherpad Lite!
15:59:44 <mgoddard> This pad text is synchronized as you type, so that everyone viewing this page sees the same text. This allows you to collaborate seamlessly on documents!
15:59:46 <priteau> without the snow
15:59:46 <mgoddard> Etherpad Lite on Github: http://j.mp/ep-lite
15:59:48 <mgoddard> n
15:59:50 <mgoddard> oops
15:59:54 <mgoddard> #link https://etherpad.openstack.org/p/kayobe-train-planning
16:00:17 <mgoddard> could be snowing somewhere
16:01:20 <mgoddard> it's 1600UTC, regrettably we must end
16:01:38 <mgoddard> thanks everyone
16:01:43 <mgoddard> the train is fast approaching
16:01:51 <mgoddard> #endmeeting