15:01:03 <mnasiadka> #startmeeting kolla
15:01:03 <opendevmeet> Meeting started Wed Dec  8 15:01:03 2021 UTC and is due to finish in 60 minutes.  The chair is mnasiadka. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:01:03 <opendevmeet> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
15:01:03 <opendevmeet> The meeting name has been set to 'kolla'
15:01:07 <yoctozepto> priteau: nice marketing
15:01:36 <mnasiadka> #topic rollcall
15:01:44 <yoctozepto> (priteau: is it only related? does not it fix the bug?)
15:01:46 <yoctozepto> o/
15:01:50 <mgoddard> \o
15:01:56 <frickler> o/
15:02:07 <mnasiadka> o/
15:02:46 <parallax> .
15:03:23 <osmanlicilegi> o/
15:04:07 <mnasiadka> #topic agenda
15:04:25 <mnasiadka> * Review action items from the last meeting
15:04:25 <mnasiadka> * CI status
15:04:25 <mnasiadka> * Release tasks
15:04:25 <mnasiadka> * Current cycle planning
15:04:25 <mnasiadka> * Additional agenda (from whiteboard)
15:04:27 <mnasiadka> * Open discussion
15:04:56 <mnasiadka> * (yoctozepto) CI - CentOS 8 images are to be gone by January 2022 http://lists.opendev.org/pipermail/service-announce/2021-December/000029.html
15:04:56 <mnasiadka> * (mnasiadka) Kolla images single distro - blockers
15:05:10 <mnasiadka> (two last before Open discussion)
15:05:26 <mnasiadka> #topic Review action items from the last meeting
15:05:54 <mnasiadka> mnasiadka to triage security bugs and update them with resolution plan (if needed)
15:05:54 <mnasiadka> yoctozepto hide properly init-runonce
15:05:54 <mnasiadka> anybody not forget to go through backports for stable branches (L248 on Whiteboard) and do stable releases afterwards.
15:05:54 <mnasiadka> mnasiadka post a patch for docs - standard topics that should be discussed over PTG and then revisited in mid-cycle
15:05:58 <mnasiadka> I haven't done mine
15:06:06 <mnasiadka> yoctozepto: ?
15:06:21 <mnasiadka> #action mnasiadka to triage security bugs and update them with resolution plan (if needed)
15:06:28 <mnasiadka> #action mnasiadka post a patch for docs - standard topics that should be discussed over PTG and then revisited in mid-cycle
15:06:41 <mnasiadka> #action anybody not forget to go through backports for stable branches (L248 on Whiteboard) and do stable releases afterwards.
15:07:37 <yoctozepto> mnasiadka
15:07:42 <yoctozepto> I posted a patch
15:07:51 <yoctozepto> let me find it
15:08:04 <yoctozepto> #link https://review.opendev.org/c/openstack/kolla-ansible/+/820940
15:08:24 <yoctozepto> I think we could go more aggressive but it's a good first step to increase awareness
15:08:47 <mnasiadka> ok then, at least one off the table
15:08:52 <mnasiadka> #topic CI status
15:08:56 <yoctozepto> as this action point was made because people were following init-runonce as a mandatory step
15:09:02 <opendevreview> Dr. Jens Harbott proposed openstack/kolla-ansible master: Bump timeout for grafana startup  https://review.opendev.org/c/openstack/kolla-ansible/+/820400
15:09:03 <yoctozepto> ok :-)
15:09:10 <mnasiadka> So, how's the CI?
15:09:35 <mnasiadka> Kolla Ussuri is red on centos
15:09:38 <yoctozepto> k&k-a on regularly supported branches seemed ok
15:09:45 <yoctozepto> but yeah, em ussuri kolla is red
15:09:56 <yoctozepto> grafana does not like something
15:10:12 <mnasiadka> is there a volunteer to look into it?
15:10:26 <yoctozepto> not me
15:11:05 <parallax> I might
15:11:22 <mnasiadka> #action parallax look into Grafana Kolla build failures on Ussuri/CentOS
15:11:29 <mnasiadka> let's move on
15:11:34 <mnasiadka> thanks parallax
15:11:43 <mnasiadka> #topic Release tasks
15:12:05 <mnasiadka> How are we with those R-17 jobs?
15:12:15 <mnasiadka> all three projects switched?
15:12:32 <yoctozepto> no idea
15:12:35 <yoctozepto> let's take a look
15:13:25 <yoctozepto> #link https://review.opendev.org/c/openstack/kolla/+/820043
15:13:31 <yoctozepto> ^ kolla needs merging this
15:13:55 <mnasiadka> ok then, +w
15:13:58 <yoctozepto> I wonder if we should just kill vmtp
15:14:02 <yoctozepto> btw
15:14:04 <mnasiadka> watcher and vmtp are not probably very popular
15:14:07 <yoctozepto> https://review.opendev.org/q/project:x%252Fvmtp
15:14:10 <yoctozepto> seems unmaintained
15:14:23 <yoctozepto> is k-a doing something with vmtp?
15:14:44 <mnasiadka> I would ask what is vmtp useful for?
15:15:12 <mnasiadka> ah, data path performance tool - but it seems it's at least abandoned for a year
15:15:13 <yoctozepto> mnasiadka: well, that's a paraphrase
15:15:19 <yoctozepto> mnasiadka: yeah
15:15:56 <yoctozepto> I think it's broken anyway
15:16:01 <yoctozepto> looking at the contents of the vmtp role
15:16:11 <yoctozepto> I would refrain from touching it
15:16:27 <yoctozepto> maybe just drop? project unsupported, in x/, and we bid farewell?
15:17:03 <mnasiadka> Probably that's the case - but shouldn't we at least send a mail to openstack ML and wait a week for any replies?
15:17:07 <yoctozepto> watcher is at least in openstack/ and has some movement
15:17:31 <yoctozepto> mnasiadka: sure thing, but it's likely we are forced to drop it anyway
15:17:55 <mnasiadka> sure, just want to have it communicated so we don't have more angry supporters ;-)
15:18:01 <mnasiadka> yoctozepto: will you send the mail?
15:18:05 <yoctozepto> mnasiadka: ok
15:18:17 <mnasiadka> #action yoctozepto to send mail to openstack ML about dropping vmtp
15:18:30 <mnasiadka> so, kolla-ansible and kayobe patches for R-17 are merged?
15:18:51 <yoctozepto> about kayobe I don't know
15:18:59 <yoctozepto> k-a seems so, but you could verify
15:20:28 <mnasiadka> seems it is
15:20:37 <yoctozepto> ok then
15:20:38 <yoctozepto> :-)
15:21:03 <mnasiadka> let's go with the additional topics now, since they impact some planning.
15:21:17 <mnasiadka> #topic (yoctozepto) CI - CentOS 8 images are to be gone by January 2022 http://lists.opendev.org/pipermail/service-announce/2021-December/000029.html
15:22:23 <yoctozepto> yeah, we need to get rid of jobs using them
15:22:25 <mnasiadka> That means we should cease centos8 (non-stream) based jobs in January
15:22:33 <yoctozepto> indeed
15:22:41 <yoctozepto> and possibly notify the world we are doing it
15:23:04 <mnasiadka> Ok, mail to openstack ML + a reno?
15:23:47 <yoctozepto> I guess?
15:23:55 <mnasiadka> Ok, makes sense.
15:24:33 <mnasiadka> Should we also think about EOL for stable/train in the process?
15:24:58 <yoctozepto> hmm, maybe? though it seems yet another topic
15:25:19 <mnasiadka> Ok, so let's first carry on dropping of CentOS jobs
15:25:24 <mnasiadka> Is there a volunteer?
15:25:57 <mgoddard> why EOL train?
15:26:08 <mgoddard> CentOS isn't the only distro
15:27:10 <yoctozepto> mnasiadka: I can do it
15:27:45 <mnasiadka> #action yoctozepto to remove CentOS 8 based CI jobs and manage communication (ML and renos)
15:29:03 <mnasiadka> Actually, what about CentOS7?
15:29:25 <yoctozepto> still supported till 2024
15:29:26 <yoctozepto> (-:
15:29:37 <mnasiadka> so we only remove the python3 CentOS8 jobs? fantastic
15:29:53 <yoctozepto> yeah, the joke is big
15:30:03 <mnasiadka> Anyway
15:30:41 <mnasiadka> mgoddard: right, I know there are other distros - just asking
15:30:47 <mnasiadka> next topic
15:31:02 <mnasiadka> #topic (mnasiadka) Kolla images single distro - blockers
15:31:21 <mnasiadka> We've had this discussion initially last week - but mgoddard was absent - and we agreed to continue today
15:32:10 <mnasiadka> We've also had an internal StackHPC meeting, and given that on our roadmap we have FIPS certification and SELinux/AppArmor support - we can't unfortunately agree with the goal of going to single distro...
15:33:53 <yoctozepto> will debian images fail to support these reqs?
15:34:40 <yoctozepto> selinux/apparmor is supported well
15:34:50 <yoctozepto> certification for our images is tricky anyway
15:34:58 <yoctozepto> fips* certification ;-)
15:35:53 <mnasiadka> We don't want to close doors for that ;-)
15:36:30 <yoctozepto> will using debian close these doors? is using ubuntu better? we can decide on ubuntu instead
15:36:41 <yoctozepto> in fact that's the most tested distro anyway...
15:36:55 <mnasiadka> Around SELinux and AppArmor - I'm sure it's going to be more complicated to implement it when having different host os and in-container os
15:37:31 <yoctozepto> mnasiadka: re selinux and apparmor - you need custom policies anyway - better write them for one setup
15:38:41 <mnasiadka> We need support for both CentOS/RHEL/etc and Ubuntu on the Host OS, and there are customers that want to have the same ,,flavor'' on the Host OS and in the containers - it's not that we dictate what you get - we don't want to loose customers this way.
15:39:13 <mnasiadka> And I - as a PTL - am worried with users/operators wandering off somewhere else, because we do this - or because of issues that we'll bring on them ;-)
15:39:49 <mnasiadka> I think for most users dropping binary is a change, let's not push it to the limits.
15:40:13 <frickler> so that means you want to build CS9 support?
15:40:47 <mnasiadka> That's up for discussion, for now we'd like to stick to CS8 as long as it's possible.
15:41:04 <yoctozepto> not much longer beyond the Yoga cycle
15:41:06 <yoctozepto> ;-)
15:41:14 <mnasiadka> once RHEL9 is out, there might be other options viable than CentOS Stream 9
15:41:48 <mgoddard> we have a few related research items
15:41:53 <yoctozepto> I *think* long term you, as a company, would benefit more from streamlining clients to the same set of images
15:42:02 <mgoddard> 1. libvirt on the host
15:42:13 <yoctozepto> as you support them, they don't have to worry about discrepancies as you solve them
15:42:19 <mgoddard> 2. py3.9 in cs8 images
15:42:33 <yoctozepto> but surely short-term this sounds like some extra work
15:43:05 <mgoddard> there is certainly an argument for single distro - it hasn't been an easy call to make
15:43:28 <yoctozepto> ad 1) seemingly libvirt in container is quite fine with different kernels, though it may need some extra verification
15:43:37 <yoctozepto> ad 2) that we can try to extedn the life of it
15:44:01 <mgoddard> we have seen issues with mixing libvirt versions
15:44:02 <yoctozepto> since we don't have to have support for python-selinux/python-dnf inside
15:44:39 <yoctozepto> mgoddard: yeah, mixing is problematic; but still it contributes to making it better by controlling the version in the container
15:45:19 <mnasiadka> Well, we already allowed libvirt 6.9.0 in stream images, which has a peculiar bug spamming the logs with one message constantly ;-)
15:45:24 <mgoddard> or just run it on the host and don't worry :)
15:46:13 <mnasiadka> We can have both options I guess.
15:46:20 <bbezak> libvirt 7.9 ;)
15:46:27 <mnasiadka> ah, 7.9
15:46:34 <yoctozepto> mnasiadka: what does it say?
15:46:50 <mnasiadka> yoctozepto: https://www.mail-archive.com/libvir-list@redhat.com/msg222970.html this - but not only on power9
15:46:58 <mnasiadka> it's patched in 7.10 and doesn't exist in 7.6
15:47:13 <bbezak> I got this on some dell x86 as well
15:47:30 <yoctozepto> well, the issue with having both supported means more to support :D
15:47:39 <yoctozepto> mnasiadka: eh :D
15:47:51 <mnasiadka> yoctozepto: that's what we get with stream ;-)
15:48:07 <mnasiadka> And we probably get those messages every week that nobody wants to use stream.
15:48:34 <yoctozepto> well, it's odd they let this kind of issue slip in, seems quite widespread
15:48:50 <mnasiadka> well, RHEL sticks to 7.6
15:48:59 <mnasiadka> I think
15:49:02 <yoctozepto> eh
15:49:17 <mnasiadka> Anyway, we're diverting - is there anything more to add to this topic?
15:49:38 <yoctozepto> rocky linux in images instead?
15:49:43 <yoctozepto> but I don't want to do it
15:50:35 <mnasiadka> I think if our company wants to continue current path, we need to take this - but let's see how the situation evolves with the research activities mgoddard mentioned
15:50:50 <mnasiadka> and if Rocky Linux doesn't die before RHEL 9 gets out ;-)
15:51:13 <yoctozepto> hah, true that
15:51:38 <mnasiadka> Ok, I crossed out the single distro from the priorities for now.
15:51:56 <mnasiadka> Around it, let's try to grasp what is missing for deprecating source.
15:52:12 <mnasiadka> #topic Current cycle planning
15:52:18 <bbezak> binary?
15:52:25 <mnasiadka> binary, yes
15:52:29 <mnasiadka> my mind is not best today.
15:52:42 <mnasiadka> Kolla whiteboard L316
15:52:44 <yoctozepto> don't worry, I'm struggling these days too
15:52:49 <mnasiadka> #link https://etherpad.opendev.org/p/KollaWhiteBoard
15:53:08 <mnasiadka> TODO(): better document the differences between them, so that users can make an educated decision transition more easily
15:53:25 <mnasiadka> Is there a volunteer to do that? Do we still think we need to do that?
15:54:10 <yoctozepto> hmm
15:54:36 <yoctozepto> sounds nice to have, but not volunteering
15:55:15 <mgoddard> I think we could use some more info about what source images are - there seems to be a general lack of understanding
15:55:38 <yoctozepto> indeed
15:55:43 <mnasiadka> Yes, agreed
15:55:51 <mnasiadka> But there's no volunteer - I will keep asking on next meetings.
15:56:13 <mnasiadka> frickler: did you have any progress with your task?
15:56:19 <mnasiadka> Do you need any help?
15:56:36 <frickler> mnasiadka: well I found that osism has some framework around it https://github.com/osism/container-images-kolla
15:56:52 <frickler> with options to add patches and things into image builds
15:57:22 <frickler> but I haven't used it myself yet, so only basic understanding so far
15:57:22 <mnasiadka> Yes, but I think kevko was a special case here - especially around overriding pip packages versions and uppper-constraints
15:57:46 <mnasiadka> So if that's possible - we'd need some improved docs on overriding these, if not - we need to think how to implement it.
15:58:30 <mnasiadka> And I think that's most important task from all those that we have in this ,,priority item''
15:59:24 <mnasiadka> Anyway, time's up - let's try to at least iron out what needs to be done in this ;-)
15:59:36 <mnasiadka> Thanks for attending.
15:59:39 <mnasiadka> #endmeeting