15:02:16 <mgoddard> #startmeeting kolla
15:02:17 <openstack> Meeting started Wed Dec  9 15:02:16 2020 UTC and is due to finish in 60 minutes.  The chair is mgoddard. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:02:18 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
15:02:20 <openstack> The meeting name has been set to 'kolla'
15:02:54 <mgoddard> #topic rollcall
15:03:01 <mgoddard> \o
15:03:05 <mnasiadka> o/
15:03:09 <yoctozepto> o/
15:03:10 <osmanlicilegi> o/
15:03:29 <rafaelweingartne> \o
15:05:07 <wuchunyang> o/
15:05:09 <mgoddard> #topic agenda
15:05:18 <mgoddard> * Roll-call
15:05:20 <mgoddard> * Announcements
15:05:22 <mgoddard> * Review action items from the last meeting
15:05:24 <mgoddard> * CI status
15:05:26 <mgoddard> * Victoria release planning
15:05:28 <mgoddard> * CentOS 8.3 & stream https://lists.centos.org/pipermail/centos-devel/2020-December/075451.html
15:05:30 <mgoddard> * Dockerhub pull rate limits https://etherpad.opendev.org/p/docker-pull-limits
15:05:32 <mgoddard> * Cinder active/active https://bugs.launchpad.net/kolla-ansible/+bug/1904062
15:05:34 <openstack> Launchpad bug 1904062 in kolla-ansible wallaby "external ceph cinder volume config breaks volumes on ussuri upgrade" [High,In progress] - Assigned to Michal Nasiadka (mnasiadka)
15:05:34 <mgoddard> * Wallaby PTG actions
15:05:51 <mgoddard> #topic announcements
15:06:15 <mgoddard> #info CentOS 8.3 released, triggering various breakages
15:06:37 <mgoddard> #info CentOS Linux deprecated in favour of CentOS stream
15:07:55 <yoctozepto> lovely week, my dears
15:08:25 <mgoddard> #topic Review action items from the last meeting
15:08:38 <mgoddard> There were none
15:08:43 <mgoddard> #topic CI status
15:08:46 <mgoddard> broken
15:08:51 <yoctozepto> very b0rken
15:09:48 <rafaelweingartne> Mark, about the review action, we have a bit of a problem in the patch we are working. People are requesting to add SAML or to split the patch in some other parts, but we are not understanding what is being requested. The patch is already complex, and long, but it is working and self-contained.
15:10:14 <rafaelweingartne> I would like to move on with it, and address extra extensions and changes in new further patches
15:10:21 <yoctozepto> rafaelweingartne: later please
15:10:47 <rafaelweingartne> Sorry, I saw the review action
15:11:21 <mgoddard> starting with kolla
15:11:28 <mgoddard> [major] lower-constraints broken
15:11:28 <yoctozepto> rafaelweingartne: ah; this is for reviewing *actions* from the previous meeting
15:12:05 <mgoddard> fixed by dropping the lower-constraints job
15:12:10 <yoctozepto> yes, need to backport
15:12:19 <mgoddard> +1
15:12:19 <mnasiadka> rafaelweingartne: you need to wait until the end of the meeting, then will be time for similar queries.
15:12:36 <mgoddard> backporting seems a lot slower with new gerrit
15:12:42 <yoctozepto> it does
15:12:51 <mgoddard> everything is slower :)
15:12:53 <yoctozepto> I will backport this drop
15:12:54 <mgoddard> anyway
15:12:56 <yoctozepto> mgoddard: +2
15:12:57 <mgoddard> cool
15:13:03 <yoctozepto> and also
15:13:05 <yoctozepto> the slimming
15:13:09 <yoctozepto> if you don't mind
15:13:14 <waverider> the Kolla meeting is in #openstack-meeting-3 ?
15:13:15 <mgoddard> yeah
15:13:16 <yoctozepto> it makes it much snappier to recheck
15:13:19 <yoctozepto> thanks
15:13:23 <yoctozepto> waverider: here
15:13:26 <yoctozepto> waverider: now
15:13:29 <maharg101> what would be the most efficient invocatiom of kolla-ansible to simply upgrade new TLS certificates which have been prepared and placed in ./config/certificates ?
15:13:53 <yoctozepto> folks, *this* is the meeting, please stay focused on the topic :-)
15:14:03 <maharg101> apols
15:14:15 <mgoddard> [major] [user-facing] [centos8.3] INFO:kolla.common.utils.base:[91mError: No matching repo to modify: PowerTools.
15:14:38 <mgoddard> fix looks good, "just" needs docker 20.10 patch to land
15:14:51 <mgoddard> I think that's all for kolla
15:14:55 <yoctozepto> mgoddard: did you try dropping the .* instead?
15:15:00 <yoctozepto> (just curious)
15:15:05 <mgoddard> yoctozepto: no, it worked
15:15:09 <mgoddard> saw no need to iterate
15:15:17 <yoctozepto> ok
15:15:39 <mgoddard> onto kolla-ansible
15:15:40 <mgoddard> [major] [user-facing] Docker 20.10 fails prechecks
15:15:51 <mgoddard> we have a patch, and reverse backport chain to stein
15:15:55 <mgoddard> hopefully it will pass
15:16:15 <yoctozepto> failing already
15:16:21 <yoctozepto> k-a needs the l-c drop
15:16:21 <mnasiadka> so all CI breakages are being handled?
15:16:26 <yoctozepto> yeas
15:16:37 <yoctozepto> ok, I will handle the l-c stuff all around
15:16:58 * yoctozepto patting himself on the back for making zuul not trigger our jobs if only zuul files are touched
15:17:13 <yoctozepto> so that we can now do something about it
15:18:23 <yoctozepto> Collecting bandit>=1.1.0
15:18:23 <yoctozepto> Downloading https://mirror.gra1.ovh.opendev.org/pypifiles/packages/25/2b/fe57beb362cb19145ab5eb809f65feec16febe056d97280e9a8d4b235824/bandit-1.6.3-py2.py3-none-any.whl (115 kB)
15:18:23 <yoctozepto> ERROR: Package 'bandit' requires a different Python: 2.7.17 not in '>=3.5'
15:18:27 <yoctozepto> oooh
15:18:32 <yoctozepto> this is k-a stein
15:18:36 <yoctozepto> trying tox py27
15:19:16 <mgoddard> ah yeah, py27 jobs are failing on stein & train
15:19:32 <mgoddard> everywhere from what I can tell
15:19:53 <mgoddard> quite the mess we have here
15:20:53 <yoctozepto> yes
15:21:00 <yoctozepto> add this to whiteboard please
15:21:14 <openstackgerrit> Radosław Piliszek proposed openstack/kolla stable/victoria: [CI] Stop testing lower-constraints  https://review.opendev.org/c/openstack/kolla/+/766240
15:22:50 <mgoddard> done
15:23:15 <mgoddard> onto kayobe
15:23:36 <mgoddard> obviously we are hit by the kolla-ansible docker precheck issue
15:23:59 <yoctozepto> (gerrit really is slow)
15:24:00 <openstackgerrit> Radosław Piliszek proposed openstack/kolla stable/ussuri: [CI] Stop testing lower-constraints  https://review.opendev.org/c/openstack/kolla/+/766241
15:24:55 <mgoddard> still trying to work out how we are hit by CentOS 8.3. I think we have patches to fix CI now
15:25:09 <mnasiadka> wonder if it's the Gerrit UI, or Gerrit API is also slow
15:25:19 <mgoddard> I expect there would be issues with a full universe-from-nothing run though, and we had a couple of bugs raised
15:25:20 <yoctozepto> mnasiadka: API seems slow too
15:25:29 <yoctozepto> mgoddard: how much is kayobe affected?
15:25:33 <yoctozepto> due to dib?
15:26:00 <mgoddard> still unsure
15:26:01 <mgoddard> https://review.opendev.org/c/openstack/kayobe/+/766203
15:26:09 <mgoddard> that fixes a few issues
15:26:29 <openstackgerrit> Radosław Piliszek proposed openstack/kolla stable/train: [CI] Stop testing lower-constraints  https://review.opendev.org/c/openstack/kolla/+/766242
15:26:57 <mnasiadka> we'll probably be fixing stuff related to 8.3 in the next days at least
15:27:05 <yoctozepto> mgoddard: gosh, that's a quite a bit
15:27:36 <yoctozepto> mnasiadka, mgoddard: well, 8.3 is said to be the taste of stream :-)
15:27:58 <mnasiadka> yoctozepto: I would call that different, but I think it violates some rules to be polite :)
15:27:59 <yoctozepto> I wonder if it makes me want to continue consuming it though
15:28:38 <yoctozepto> mnasiadka: well ;-)
15:29:41 <mgoddard> #topic Victoria release planning
15:30:20 <mgoddard> I was hoping we might push forward with V, but it seems firefighting will be the activity for a while
15:30:47 <yoctozepto> ++
15:30:58 <yoctozepto> very saddening
15:31:09 <mgoddard> release blockers are on L166 of the whiteboard
15:31:23 <mgoddard> not really any changes
15:31:34 <openstackgerrit> Radosław Piliszek proposed openstack/kolla stable/stein: [CI] Stop testing lower-constraints  https://review.opendev.org/c/openstack/kolla/+/766243
15:32:06 <yoctozepto> ok, dears, please approve all those l-c drops ^
15:32:18 <yoctozepto> now the same on k-a
15:32:47 <mgoddard> yoctozepto: did you look at py27?
15:33:05 <yoctozepto> mgoddard: not yet, I wonder if it breaks on kolla too
15:33:12 <yoctozepto> will do that as the next step, yes
15:33:16 <mnasiadka> Ok, seems I can put some cycles to cinder in the next two days, so we might have some progress on that one.
15:33:17 <yoctozepto> leave the basic CI to me :-)
15:33:30 <yoctozepto> mnasiadka: that would be wonderful
15:33:42 <mgoddard> yoctozepto: it does
15:33:47 <yoctozepto> start with that volume-preserving upgrade
15:34:09 <yoctozepto> mgoddard: that was fast :-)
15:34:17 <mgoddard> yoctozepto: checked emails..
15:34:27 <yoctozepto> I see
15:34:27 <mnasiadka> well, the volume will be preserved either way, just you can't do anything with it since the agent that owns it is dead :)
15:34:42 <yoctozepto> mnasiadka: yeah, I mean we are now cleaning them
15:34:44 <yoctozepto> so hard to check
15:34:53 <mgoddard> #topic Cinder active/active https://bugs.launchpad.net/kolla-ansible/+bug/1904062
15:34:55 <openstack> Launchpad bug 1904062 in kolla-ansible wallaby "external ceph cinder volume config breaks volumes on ussuri upgrade" [High,In progress] - Assigned to Michal Nasiadka (mnasiadka)
15:35:01 <mgoddard> since we are on it already
15:35:17 <mnasiadka> mgoddard: sorry to get in front of the agenda ;)
15:35:20 <mgoddard> I saw in last week's meeting a plan that did not align with the one in my head
15:35:41 <mgoddard> first, there was some suggestion that cluster is not supported until victoria
15:35:59 <mgoddard> that is incorrect, it is about a cluster_name field returned by the API
15:36:10 <mgoddard> cluster goes back to rocky IIRC
15:36:22 <mnasiadka> yeah, OSA changed way back to use cluster
15:36:46 <mgoddard> second, there was talk of using backend_host until victoria. I think we should use cluster and drop backend_host, all the way back to stein
15:36:54 <mgoddard> s/stein/train/
15:37:45 <mnasiadka> well, we never had backend_host in external ceph code I think
15:37:47 <mnasiadka> Only in docs
15:38:00 <mgoddard> correct, but we have to assume our users followed the docs :)
15:38:45 <mnasiadka> Ok, so after merging a fix and upgrade path - we should fix the docs as well.
15:38:49 <yoctozepto> mgoddard: I agree
15:39:24 <yoctozepto> hmm, bandit 1.6.3 was released Dec 6th
15:39:28 <mgoddard> mnasiadka: fix docs in stable?
15:39:34 <yoctozepto> it likely broke all CIs
15:39:58 <mgoddard> yoctozepto: requirements may have a patch in progress in that case
15:40:06 <mnasiadka> mgoddard: yeah, shouldn't we?
15:40:21 <mgoddard> mnasiadka: I agree, we should
15:40:41 <mnasiadka> ok, added to my list
15:40:42 <mgoddard> I suppose we need to consider what happens if a user keeps backend_host
15:40:52 <yoctozepto> https://github.com/PyCQA/bandit/issues/663
15:40:56 <mgoddard> should we fail?
15:41:01 <yoctozepto> mgoddard: it's not controlled by reqs
15:41:28 <mnasiadka> mgoddard: we could write a precheck for that, would make sense
15:41:42 <mgoddard> +1
15:41:52 <mnasiadka> and backport it all the way?
15:41:56 <mgoddard> yes
15:42:06 <mnasiadka> makes sense
15:43:33 <mgoddard> ready to move on?
15:43:37 <mnasiadka> yup
15:43:38 * yoctozepto fixing kolla py27 by blacklisting "universal" bandit
15:43:59 <mgoddard> #topic CentOS 8.3 & stream https://lists.centos.org/pipermail/centos-devel/2020-December/075451.html
15:44:24 <mgoddard> avoiding CentOS bashing for the moment
15:44:38 <dcapone2004> :-(
15:44:43 <mgoddard> what do we need to do?
15:45:04 <mgoddard> medium term, it seems the path forward it stream
15:45:07 <mgoddard> *is
15:45:15 <mgoddard> the question for us is when
15:45:28 <mnasiadka> I guess first we would need to understand if there will be container image tagged centos:8-stream
15:46:13 <yoctozepto> ++
15:46:21 <mnasiadka> I can ask on centos-devel
15:46:23 <yoctozepto> I would say go stream all the way
15:46:26 <yoctozepto> it's fine for our cadence
15:46:40 <yoctozepto> just let there be a proper container image of it
15:46:44 <mnasiadka> Second thing is - if Infra is planning to build a stream image we could use in CI
15:46:53 <yoctozepto> mnasiadka: we have it already
15:47:07 <yoctozepto> passes devstack CI ;-)
15:47:18 <mnasiadka> So then we could start running our simple CI jobs on it
15:47:25 <yoctozepto> you are in good hands with me :D
15:47:33 <yoctozepto> yes, we can switch this easily
15:47:46 <mnasiadka> do we want to switch, or have both?
15:48:08 <mnasiadka> It's rather a question do we deprecate normal CentOS 8 support in W, and support only Stream?
15:48:27 <openstackgerrit> Radosław Piliszek proposed openstack/kolla stable/train: [CI] Stop testing lower-constraints and fix py27 job  https://review.opendev.org/c/openstack/kolla/+/766242
15:48:34 <yoctozepto> mnasiadka: I wonder what docker will do
15:48:41 <yoctozepto> are its repos stream-friendly now?
15:48:53 <mgoddard> were they not?
15:49:01 <mnasiadka> I think they should be.
15:49:17 <yoctozepto> I think so too, just asking for clarity
15:49:19 <mnasiadka> I can waste some minutes and spawn a Stream image and try to build/deploy Kolla
15:49:31 <yoctozepto> that would not be a waste!
15:49:56 <mnasiadka> Rather thinking do we just support Stream from Wallaby, or support both 8.3 and Stream?
15:50:19 <mnasiadka> Anybody has any thoughts on that?
15:50:38 <mgoddard> I've added these questions to the whiteboard
15:51:05 <mgoddard> in order to drop support for releases we need a working migration path to stream
15:51:55 <mnasiadka> well, migration path shouldn't be complex - just dnf install centos-release-stream and dnf distro-sync
15:52:19 <yoctozepto> yes, that is simple
15:52:19 <mgoddard> mnasiadka: you are right - it shouldn't :)
15:52:31 <yoctozepto> mgoddard: it really is simple
15:52:33 <mnasiadka> but what breaks then, don't ask me yet
15:52:41 <yoctozepto> yeas, kolla-wise unknown :-)
15:52:48 <mnasiadka> well, then medium term we could test that migration path in upgrade jobs
15:53:07 <yoctozepto> too costly
15:53:26 <yoctozepto> we never tested 7.x -> 7.(x+1) anyhow
15:53:31 <mgoddard> we should probably switch the containers over
15:53:39 <ozzzo> we (and probably lots of other operators) are not planning to run OpenStack on stream. I'd like to see Centos 8.3 supported as long as possible while we work on switching to another distro
15:54:04 <mgoddard> ozzzo: good datapoint
15:54:43 <yoctozepto> hmm
15:54:45 <mgoddard> ozzzo: can you provide your reasoning?
15:54:51 <yoctozepto> ^
15:55:07 <ozzzo> stream isn't suitable for production. It is basically a beta, while still trailing RHEL for security fixes
15:55:30 <yoctozepto> they are changing this, it's an unfair claim
15:55:38 <yoctozepto> I would say wait till the end of Q1
15:56:15 <mnasiadka> well, anyway that's main concern of a lot of people
15:56:21 <mgoddard> I would at least say that you are going to want a stable local mirror of it
15:56:30 <mgoddard> that you can move forward at your own pace
15:56:41 <yoctozepto> well, with the current model you could have 1-2 months of no security fixes anyhow
15:57:02 <mnasiadka> mgoddard, yoctozepto: centos-devel folks are thinking of building centos-stream container images, but nothing concrete yet
15:57:09 <ozzzo> we can live with trailing security fixes; the concern is stability
15:57:16 <yoctozepto> mnasiadka: thanks
15:57:21 <mgoddard> ozzzo: see my last point
15:57:23 <waverider> Is there another rpm-based open source distro suitable for production? Operators prefer rpm-based instead of deb-distros, why? Or makes sense to consider Debian support (as alternative to Ubuntu, which is probably leaned towards commercial).
15:57:39 <yoctozepto> waverider: deb is just as fine ;-)
15:57:47 <mgoddard> waverider: we support debian
15:57:52 <mgoddard> and ubuntu
15:57:54 <dcapone2004> ozzzo:  I'm with you, I commented before meeting on that, but said I was waiting till after meeting to discuss that further
15:58:01 <yoctozepto> waverider: for openstack with rpms your best bet is centos still
15:58:16 <yoctozepto> otherwise we welcome you in ubuntu and debian flavours
15:58:28 <dcapone2004> fyi, original Centos founder
15:58:31 <yoctozepto> as for the current situation w/ centos
15:58:37 <yoctozepto> rocky linux is happening
15:58:39 <dcapone2004> just announced Rocky Linux....he is supposedly recreating another fork of RHEL
15:58:43 <waverider> nice, I didn't notice Debian, I'm used with Ubuntu and we used everywhere we are not forced to rpm-based (like cPanel servers)
15:58:50 <yoctozepto> https://forums.rockylinux.org/
15:59:09 <yoctozepto> and #rockylinux* on Freenode
15:59:13 <waverider> that was on my mind, who's gonna take over centos open source :)
15:59:39 <yoctozepto> anyway, another announcement is
15:59:47 <yoctozepto> we will be discussing centos with tc tomorrow
16:00:04 <yoctozepto> the infra is ready for stream
16:00:12 <yoctozepto> just need for decisions
16:00:27 <mnasiadka> mgoddard, yoctozepto: so for now I'll just add conversion to centos stream during base build to understand if something will fail very badly on stream builds.
16:00:41 <mnasiadka> and we can get better prepared
16:00:42 <yoctozepto> mnasiadka: ++
16:00:44 <mgoddard> I expect we need to hold tight and see which way this goes before doing anything drastic
16:00:46 <yoctozepto> lovely idea
16:00:52 <yoctozepto> mgoddard: ++ as well
16:01:28 <mnasiadka> yeah, no dynamic moves before Stream gets established properly.
16:01:47 <mgoddard> thanks all
16:01:50 <mgoddard> #endmeeting