15:01:21 <mgoddard> #startmeeting kolla
15:01:22 <openstack> Meeting started Wed Dec 11 15:01:21 2019 UTC and is due to finish in 60 minutes.  The chair is mgoddard. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:01:23 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
15:01:26 <openstack> The meeting name has been set to 'kolla'
15:01:33 <mgoddard> #topic rollcall
15:01:35 <mgoddard> \o
15:01:39 <jovial[m]> o/
15:01:45 <generalfuzz> o/
15:01:52 <osmanlicilegi> o/
15:01:57 <yoctozepto> o/
15:02:03 <chason> o/
15:02:50 <mnasiadka> o/
15:03:32 <mgoddard> #topic agenda
15:03:34 <mgoddard> * Roll-call
15:03:36 <mgoddard> * Announcements
15:03:38 <mgoddard> * Review action items from last meeting
15:03:40 <mgoddard> * CI status
15:03:42 <mgoddard> * Train release planning
15:03:44 <mgoddard> * Ussuri release planning
15:03:46 <mgoddard> ** Support for Train redeployment on C8, we are already diverging with support for Train
15:03:48 <mgoddard> * Review priorities
15:03:50 <mgoddard> * Should we check for supported OS distro & release in prechecks?
15:03:52 <mgoddard> * CI scenarios
15:03:54 <mgoddard> #topic announcements
15:03:56 <mgoddard> #info Kayobe RC1 planned this week
15:04:15 <mgoddard> #info kolla RC2 releases proposed
15:04:40 <mgoddard> I do wonder if the release team have become a bottleneck. Those patches have been up for a couple days
15:04:56 <mgoddard> *patch
15:05:24 <mgoddard> Any others?
15:05:27 <mnasiadka> they announced some time ago they are happy to teach new members... so maybe that's the case
15:05:28 <yoctozepto> yea, it's happening again
15:05:34 <yoctozepto> they were delayed previously too
15:05:51 <yoctozepto> mnasiadka: we should go get taught :-)
15:06:40 <mgoddard> seems to me like teams should have control over their own releases
15:06:49 <mgoddard> similar to the recent stable branch discussion
15:07:22 <mgoddard> anyway
15:07:32 <mgoddard> #topic Review action items from last meeting
15:07:36 <mgoddard> We had two
15:07:43 <mgoddard> mgoddard to update kayobe train release etherpad
15:07:44 <mgoddard> osmanlicilegi to mark senlin images as unbuildable for ubuntu & debian binary
15:07:48 <mgoddard> I did mine
15:07:53 <mgoddard> osmanlicilegi did his
15:07:55 <mgoddard> boom
15:07:59 <osmanlicilegi> yep
15:07:59 <mgoddard> #topic CI status
15:08:05 <mgoddard> thanks osmanlicilegi :)
15:08:13 <osmanlicilegi> yw
15:08:34 <mgoddard> CI looking good AFAIK, although we are running train tarballs on master
15:08:49 <yoctozepto> only for c7
15:09:02 <mgoddard> true
15:09:24 <openstackgerrit> Michal Nasiadka proposed openstack/kolla-ansible master: WIP: Ceph-Ansible CI  https://review.opendev.org/676376
15:09:29 <mgoddard> #topic Train release planning
15:09:51 <mgoddard> Kolla is looking good at this point, just waiting for release handle cranking
15:10:05 <mgoddard> jovial[m]: how about kayobe?
15:10:08 <openstackgerrit> Will Szumski proposed openstack/kayobe master: Add kayobe as openstack project for release notes  https://review.opendev.org/697701
15:11:15 <jovial[m]> Still need the nova cells work and anible version bump patches to be completed.
15:11:41 <mgoddard> jovial[m]: and this one: https://review.opendev.org/#/c/694616/
15:11:41 <jovial[m]> Other than that it's just reviews we need.
15:11:41 <patchbot> patch 694616 - kayobe - Use OpenStack Train release - 2 patch sets
15:12:34 <mgoddard> Release tracker here: https://etherpad.openstack.org/p/kayobe-train-release
15:12:35 <jovial[m]> Oh yeah, I will update that one shortly.
15:12:36 <mgoddard> #link https://etherpad.openstack.org/p/kayobe-train-release
15:13:24 <mgoddard> 7 patches left to review
15:13:28 <openstackgerrit> Michal Nasiadka proposed openstack/kolla-ansible master: WIP: OVN Support  https://review.opendev.org/696841
15:13:45 <mgoddard> let's keep pushing on it
15:14:26 <mgoddard> #topic Ussuri release planning
15:14:35 <mgoddard> Onto U
15:14:46 <yoctozepto> onto U!
15:14:55 <mgoddard> I've been spending some time on centos 8 recently
15:15:22 <mgoddard> current status is we have an aio centos8/source job passing in CI
15:15:37 <yoctozepto> hurrary
15:15:40 <yoctozepto> hurray*
15:15:50 <mgoddard> it required a few hacks & fudges, trying to extract the correct pieces into other patches to get them merged
15:16:07 <mnasiadka> so it's in good progress
15:16:14 <mnasiadka> although we're missing ceph client libraries, right?
15:16:16 <mgoddard> yatin pointed out that the main delorean RDO repo is now available, so I've just pushed a patch to test centos8/binary aio
15:16:28 <mgoddard> mnasiadka: yes
15:16:32 <mgoddard> and a few other things
15:16:59 <mnasiadka> ok, around ceph - we should get it built in a couple of weeks, there's a group that is pushing all the deps to EPEL el8
15:17:10 <mnasiadka> and there's only a couple left
15:17:34 <mnasiadka> mainly pecan and libs depending on pecan
15:18:15 <mgoddard> nice
15:18:16 <yoctozepto> very good
15:18:54 <mgoddard> we know that we need to support c7 & c8 on train, so bear that in mind when writing and reviewing these patches
15:18:59 <mgoddard> (backport)
15:19:28 <mnasiadka> we already know that backporting will be a bit painful, due to some changes in kolla (like repo enablement on a different level)
15:19:35 <mgoddard> I think the most sane way to support that is to support both in master, then remove c7
15:19:57 <mgoddard> i.e. parallel build & deploy jobs
15:20:11 <mnasiadka> yeah, that's probably a good idea to follow that path
15:20:13 <yoctozepto> yea, we already broke qinling now
15:20:19 <yoctozepto> sorry, senlin*
15:20:36 <mgoddard> I'm not sure what we do about publishing
15:20:51 <mgoddard> should we publish images with a base_distro of centos8?
15:21:14 <mnasiadka> mgoddard: I think we have no other option
15:21:35 <mgoddard> true for train, but we could avoid it on master if we wish
15:21:57 <mgoddard> maybe it doesn't matter
15:22:00 <mnasiadka> I think a sane approach would be to have base_distro = centos8, and deprecate/remove centos (as in centos7)
15:22:17 <mnasiadka> less people complaining maybe
15:22:29 <mnasiadka> but I guess we'll work it out on the way
15:22:31 <mgoddard> hmm, rather than switching back to centos on master?
15:23:04 <mnasiadka> maybe it doesn't make any sense, and we should switch back to centos on master
15:23:23 <openstackgerrit> Merged openstack/kayobe master: Add example to Bridges and VLANs section  https://review.opendev.org/693368
15:23:38 <yoctozepto> centos8 sounds nice
15:23:44 <yoctozepto> we might soon require that for ubuntu20
15:23:56 <mgoddard> would require a lot of conditional logic changes
15:23:56 <yoctozepto> encoding distro version sounds nice
15:23:59 <mnasiadka> my worry is that we'll get centos8 master tagged images we would need to remove from docker hub
15:24:13 <yoctozepto> mgoddard: hmm, could be
15:24:28 <yoctozepto> mnasiadka: yea, going back and forth sounds bad
15:24:33 <yoctozepto> my idea is to stay
15:24:39 <yoctozepto> rethink logic to accommodate it
15:24:48 <mgoddard> or we encode the OS version in the image name
15:25:24 <mnasiadka> like centos-8-source-xxx ?
15:25:41 <mgoddard> or kolla/centos-8-source-base:train
15:25:42 <mgoddard> yeah
15:25:45 <mnasiadka> well, we have time to rethink the pros and cons of both solutions I think
15:25:55 <mgoddard> or kolla/centos-source-base:train-centos8
15:25:59 <mgoddard> as a one-off
15:26:11 <mnasiadka> yeah, the tag thing as a one off should probably work
15:26:16 <mnasiadka> we could shorten it to train-c8
15:26:17 <mnasiadka> :)
15:26:20 <mgoddard> yeah
15:26:24 <mgoddard> that's probably best
15:26:39 <mnasiadka> yup
15:27:02 <mgoddard> we need something we can decide on at runtime in ansible, based on ansible_distro_major_version
15:27:22 <mgoddard> for mixed c7/8
15:27:40 <mgoddard> ok, we have some things to think about
15:28:11 <mgoddard> while we are on this topic, we have a subtopic proposed
15:28:13 <mgoddard> Support for Train redeployment on C8, we are already diverging with support for Train
15:28:18 <mgoddard> yoctozepto: is that you?
15:28:34 <yoctozepto> mgoddard: yea
15:28:48 <yoctozepto> so we need to think about it when reviewing
15:28:57 <yoctozepto> and also need some mechanism to support both ways
15:29:10 <yoctozepto> as we are going to lose train re-deployability
15:29:14 <yoctozepto> on c8
15:30:03 <mgoddard> you mean migrating an existing train c7 cloud to c8?
15:30:10 <yoctozepto> mhm
15:30:29 <yoctozepto> now senlin would fail to redeploy because of the logic change
15:30:51 <yoctozepto> it could be really hard to allow both releases if more projects start diverging
15:30:53 <mgoddard> but we'd be using stable/train k-a to do this
15:31:00 <yoctozepto> ah
15:31:08 <yoctozepto> did not get that
15:31:10 <yoctozepto> :D
15:31:17 <mgoddard> at least that was the generally preferred option I think
15:31:19 <yoctozepto> then all is fine
15:31:40 <mnasiadka> but maybe it would make sense to have a change with zuul job that does the migration?
15:31:57 <mgoddard> yes, we need that
15:32:01 <mgoddard> but on stable/train
15:32:40 <mnasiadka> of course
15:33:55 <mgoddard> I think we would have a multinode with mixed c7/8
15:34:15 <mgoddard> deploy to c7, then add the c8 node to the inventory and deploy again
15:34:26 <yoctozepto> so we need to support c8 as host on train
15:34:28 <mnasiadka> well, host can be c7, containers can be c8 :)
15:34:31 <yoctozepto> not sure we do already
15:34:32 <mgoddard> yes
15:34:40 <mgoddard> no, we do not
15:34:43 <yoctozepto> :D
15:34:44 <mgoddard> there will be much backporting
15:34:53 <yoctozepto> mhm
15:35:22 <mnasiadka> I guess we need to support all three mixes for the migration/upgrade process? c7 on c7, c8 on c7, c8 on c8?
15:35:46 <mnasiadka> or c7 on c8? :)
15:35:49 <mgoddard> mnasiadka: no - tripleo isn't going to mix host and container versions
15:36:02 <mgoddard> so we shoulnd't either
15:36:12 <mgoddard> c7 host runs c7 containers
15:36:18 <mnasiadka> mgoddard: well, they control the OS with Heat stack, we need to state what we support in the docs
15:36:28 <mgoddard> new c8 hosts come up with c8 containers
15:36:29 <mnasiadka> that's why I'm asking
15:36:56 <mgoddard> that's what I meant earlier with ansible_distro_major_version - we will select the container tag based on the host
15:37:09 <yoctozepto> c8 on c7 unsupported
15:37:09 <mnasiadka> makes sense
15:37:35 <mnasiadka> so we don't expose that to the user in any other way, we deploy c8 containers on c8 host
15:37:35 <mgoddard> we should probably write this stuff down somewhere :)
15:38:08 <mnasiadka> yeah, most probably - we need a draft of a proposed migration process and a zuul job that tests it :)
15:38:14 <mgoddard> it'll be a variable. they can shoot themselves in the foot if they want
15:38:42 <yoctozepto> :D
15:38:52 <mnasiadka> good, I love users shooting in the foot
15:39:15 <mgoddard> openstack_release & openstack_release_centos8?
15:39:30 <mgoddard> or maybe it needs to be configurable for each container?
15:39:44 <mgoddard> hmm, perhaps base_distro would be easier from ansible perspective
15:40:21 <mgoddard> would be a lot easier if they just supported upgrades from 7 to 8...
15:40:50 <mgoddard> Any other thoughts on this one?
15:41:08 <mnasiadka> by the way, the centos 8 containers will be based on Stream, right?
15:41:30 <mgoddard> mnasiadka: I think so
15:41:46 <mnasiadka> yeah centos:8 is stream
15:42:06 <mnasiadka> no more stupid questions, I guess as homework we all need to rethink the base_distro vs tag for next meeting
15:43:32 <mgoddard> yeah
15:43:41 <mgoddard> #topic Should we check for supported OS distro & release in prechecks?
15:44:08 <mgoddard> I noticed that OSA has some checks on the OS distro & release against a supported set
15:44:36 <mnasiadka> might make sense, do we plan to check the docker engine version as well?
15:44:41 <mgoddard> I also notice in bug reports people using untested ubuntu & kolla release combos
15:45:13 <yoctozepto> +1
15:45:16 <mnasiadka> especially when we merge healthchecks, anything prior to docker 1.12 will not work
15:45:21 <mgoddard> we could check docker engine version. I'm not sure we have much of a requirement on it
15:45:34 <osmanlicilegi> +1
15:45:41 <mnasiadka> we have some requirements on docker API versions rather
15:45:48 <mgoddard> hopefully no one is running anything earlier than 1.12 with ussuri :)
15:45:50 <mnasiadka> we could check for that
15:46:26 <mnasiadka> mgoddard: who knows... who knows :)
15:46:44 <mnasiadka> especially people might not be aware about the maintenance cycles for docker-ce
15:46:53 <mgoddard> how strict should we be with centos versions?
15:47:11 <mgoddard> should we just say we support c7, or 7.7?
15:47:45 <mgoddard> same with debian/ubuntu
15:47:49 <mnasiadka> I think it should be c7 - c7 GA is supported normally until Q4 2020
15:47:59 <mnasiadka> well, with Ubuntu it's easy - we say Bionic, right?
15:48:12 <mgoddard> I think so
15:48:13 <mnasiadka> and Debian Buster I think?
15:48:19 <mgoddard> yes
15:48:41 <yoctozepto> image must match host
15:48:54 <mgoddard> yeah
15:48:56 <yoctozepto> otherwise very limited support
15:49:02 <mgoddard> so does anyone want to pick this up?
15:49:06 <yoctozepto> i.e. not necessarily rejecting patches
15:49:09 <yoctozepto> but not caring much
15:50:40 <osmanlicilegi> I can take it if nobody wants it :) I'm not sure if I can catch ussuri timelines but I'll try
15:51:08 <mnasiadka> mgoddard: around picking up, maybe we could think about having a bucket list (or a more scary version - kanban board) - and each one of us with spare cycles could take something - at least I have a tendency of forgetting about things :)
15:51:37 <mgoddard> I've added it to the whiteboard
15:51:47 <mgoddard> https://blueprints.launchpad.net/kolla-ansible/+spec/os-prechecks
15:52:19 <mnasiadka> goodie
15:53:14 <mgoddard> always a good place to pick things up from
15:54:07 <mgoddard> I'll put your name on the blueprint osmanlicilegi, remove it if you think you won't have time
15:54:11 <mnasiadka> there are some... old blueprints with high priority - can we sort of think about cleaning it up?
15:54:28 <osmanlicilegi> mgoddard: ok
15:55:02 <mgoddard> mnasiadka: which ones?
15:55:20 <mnasiadka> mgoddard: example - https://blueprints.launchpad.net/kolla-ansible/+spec/sanity-check-container
15:55:29 <mnasiadka> it's there since Newton or longer :)
15:55:59 <mgoddard> oh right, I thought you meant on the whiteboard
15:56:05 <mgoddard> yeah, LP is a mess
15:56:12 <mnasiadka> naah, whiteboard is maintained :)
15:56:26 <mgoddard> I occasionally tidy it up until I get bored
15:56:52 <mnasiadka> mgoddard: maybe let's just remove priority tag from those things we don't do in Ussuri?
15:56:58 <mgoddard> +1
15:57:16 <mnasiadka> ok, I'll do that, because this mess hurts :)
15:57:45 <yoctozepto> meh, just filter for ussuri :D
15:58:13 <mgoddard> #action mnasiadka to remove priority from inactive blueprints
15:58:55 <mgoddard> #topic Open discussion
15:59:02 <mgoddard> Anything else to cover?
15:59:15 <yoctozepto> nah, /me overloaded
15:59:31 <osmanlicilegi> mgoddard: I have a question for https://blueprints.launchpad.net/kolla-ansible/+spec/docker-image-pruning. should I go with ansible docker module or kolla_docker.py ?
15:59:50 <yoctozepto> good question
16:00:18 <osmanlicilegi> maybe for the v cycle kolla_docker.py will retire :]
16:00:38 <mnasiadka> so, if kolla_docker.py lacks this functionality - I would go for upstream module
16:00:43 <mgoddard> osmanlicilegi: it still has some things docker_container does not
16:00:51 <yoctozepto> make it so that it's easy to go either way :D
16:00:58 <mnasiadka> unless upstream module lacks some functionality, so go for extending kolla_docker.py :)
16:01:05 <yoctozepto> looking at current shape of docker
16:01:17 <yoctozepto> i would not be surprised it's "deprecated" in a year
16:01:24 <mgoddard> https://docs.ansible.com/ansible/latest/modules/docker_image_module.html
16:01:29 <mgoddard> I can't see prune on there
16:01:39 <yoctozepto> probably getting a fork and more momentum toward podman and alike
16:01:47 <mnasiadka> yoctozepto: and what all those Docker Desktop users will do? :)
16:01:58 <yoctozepto> mnasiadka: cry just a little
16:02:11 <osmanlicilegi> mgoddard: https://docs.ansible.com/ansible/latest/modules/docker_prune_module.html
16:02:16 <mnasiadka> yoctozepto: Windows users cry louder than others :)
16:02:28 <yoctozepto> https://www.youtube.com/watch?v=ZBpWdwXzpMk
16:02:44 <mgoddard> osmanlicilegi: oh, nice. Use that. We'll need to bump ansible minimum to 2.8, but I think we planned to do that in U anyway
16:02:57 <osmanlicilegi> ok sir :]
16:03:08 <mnasiadka> yes, bump up Ansible :)
16:03:35 <yoctozepto> yes, bump as far as eyes can see
16:03:45 <yoctozepto> we are past 5pm
16:03:48 <mgoddard> ok, we're at time
16:03:50 <yoctozepto> let's wrap it up
16:04:08 <mgoddard> Thank you all
16:04:12 <mgoddard> #endmeeting