15:01:03 #startmeeting kolla 15:01:03 Meeting started Wed Dec 8 15:01:03 2021 UTC and is due to finish in 60 minutes. The chair is mnasiadka. Information about MeetBot at http://wiki.debian.org/MeetBot. 15:01:03 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 15:01:03 The meeting name has been set to 'kolla' 15:01:07 priteau: nice marketing 15:01:36 #topic rollcall 15:01:44 (priteau: is it only related? does not it fix the bug?) 15:01:46 o/ 15:01:50 \o 15:01:56 o/ 15:02:07 o/ 15:02:46 . 15:03:23 o/ 15:04:07 #topic agenda 15:04:25 * Review action items from the last meeting 15:04:25 * CI status 15:04:25 * Release tasks 15:04:25 * Current cycle planning 15:04:25 * Additional agenda (from whiteboard) 15:04:27 * Open discussion 15:04:56 * (yoctozepto) CI - CentOS 8 images are to be gone by January 2022 http://lists.opendev.org/pipermail/service-announce/2021-December/000029.html 15:04:56 * (mnasiadka) Kolla images single distro - blockers 15:05:10 (two last before Open discussion) 15:05:26 #topic Review action items from the last meeting 15:05:54 mnasiadka to triage security bugs and update them with resolution plan (if needed) 15:05:54 yoctozepto hide properly init-runonce 15:05:54 anybody not forget to go through backports for stable branches (L248 on Whiteboard) and do stable releases afterwards. 15:05:54 mnasiadka post a patch for docs - standard topics that should be discussed over PTG and then revisited in mid-cycle 15:05:58 I haven't done mine 15:06:06 yoctozepto: ? 15:06:21 #action mnasiadka to triage security bugs and update them with resolution plan (if needed) 15:06:28 #action mnasiadka post a patch for docs - standard topics that should be discussed over PTG and then revisited in mid-cycle 15:06:41 #action anybody not forget to go through backports for stable branches (L248 on Whiteboard) and do stable releases afterwards. 15:07:37 mnasiadka 15:07:42 I posted a patch 15:07:51 let me find it 15:08:04 #link https://review.opendev.org/c/openstack/kolla-ansible/+/820940 15:08:24 I think we could go more aggressive but it's a good first step to increase awareness 15:08:47 ok then, at least one off the table 15:08:52 #topic CI status 15:08:56 as this action point was made because people were following init-runonce as a mandatory step 15:09:02 Dr. Jens Harbott proposed openstack/kolla-ansible master: Bump timeout for grafana startup https://review.opendev.org/c/openstack/kolla-ansible/+/820400 15:09:03 ok :-) 15:09:10 So, how's the CI? 15:09:35 Kolla Ussuri is red on centos 15:09:38 k&k-a on regularly supported branches seemed ok 15:09:45 but yeah, em ussuri kolla is red 15:09:56 grafana does not like something 15:10:12 is there a volunteer to look into it? 15:10:26 not me 15:11:05 I might 15:11:22 #action parallax look into Grafana Kolla build failures on Ussuri/CentOS 15:11:29 let's move on 15:11:34 thanks parallax 15:11:43 #topic Release tasks 15:12:05 How are we with those R-17 jobs? 15:12:15 all three projects switched? 15:12:32 no idea 15:12:35 let's take a look 15:13:25 #link https://review.opendev.org/c/openstack/kolla/+/820043 15:13:31 ^ kolla needs merging this 15:13:55 ok then, +w 15:13:58 I wonder if we should just kill vmtp 15:14:02 btw 15:14:04 watcher and vmtp are not probably very popular 15:14:07 https://review.opendev.org/q/project:x%252Fvmtp 15:14:10 seems unmaintained 15:14:23 is k-a doing something with vmtp? 15:14:44 I would ask what is vmtp useful for? 15:15:12 ah, data path performance tool - but it seems it's at least abandoned for a year 15:15:13 mnasiadka: well, that's a paraphrase 15:15:19 mnasiadka: yeah 15:15:56 I think it's broken anyway 15:16:01 looking at the contents of the vmtp role 15:16:11 I would refrain from touching it 15:16:27 maybe just drop? project unsupported, in x/, and we bid farewell? 15:17:03 Probably that's the case - but shouldn't we at least send a mail to openstack ML and wait a week for any replies? 15:17:07 watcher is at least in openstack/ and has some movement 15:17:31 mnasiadka: sure thing, but it's likely we are forced to drop it anyway 15:17:55 sure, just want to have it communicated so we don't have more angry supporters ;-) 15:18:01 yoctozepto: will you send the mail? 15:18:05 mnasiadka: ok 15:18:17 #action yoctozepto to send mail to openstack ML about dropping vmtp 15:18:30 so, kolla-ansible and kayobe patches for R-17 are merged? 15:18:51 about kayobe I don't know 15:18:59 k-a seems so, but you could verify 15:20:28 seems it is 15:20:37 ok then 15:20:38 :-) 15:21:03 let's go with the additional topics now, since they impact some planning. 15:21:17 #topic (yoctozepto) CI - CentOS 8 images are to be gone by January 2022 http://lists.opendev.org/pipermail/service-announce/2021-December/000029.html 15:22:23 yeah, we need to get rid of jobs using them 15:22:25 That means we should cease centos8 (non-stream) based jobs in January 15:22:33 indeed 15:22:41 and possibly notify the world we are doing it 15:23:04 Ok, mail to openstack ML + a reno? 15:23:47 I guess? 15:23:55 Ok, makes sense. 15:24:33 Should we also think about EOL for stable/train in the process? 15:24:58 hmm, maybe? though it seems yet another topic 15:25:19 Ok, so let's first carry on dropping of CentOS jobs 15:25:24 Is there a volunteer? 15:25:57 why EOL train? 15:26:08 CentOS isn't the only distro 15:27:10 mnasiadka: I can do it 15:27:45 #action yoctozepto to remove CentOS 8 based CI jobs and manage communication (ML and renos) 15:29:03 Actually, what about CentOS7? 15:29:25 still supported till 2024 15:29:26 (-: 15:29:37 so we only remove the python3 CentOS8 jobs? fantastic 15:29:53 yeah, the joke is big 15:30:03 Anyway 15:30:41 mgoddard: right, I know there are other distros - just asking 15:30:47 next topic 15:31:02 #topic (mnasiadka) Kolla images single distro - blockers 15:31:21 We've had this discussion initially last week - but mgoddard was absent - and we agreed to continue today 15:32:10 We've also had an internal StackHPC meeting, and given that on our roadmap we have FIPS certification and SELinux/AppArmor support - we can't unfortunately agree with the goal of going to single distro... 15:33:53 will debian images fail to support these reqs? 15:34:40 selinux/apparmor is supported well 15:34:50 certification for our images is tricky anyway 15:34:58 fips* certification ;-) 15:35:53 We don't want to close doors for that ;-) 15:36:30 will using debian close these doors? is using ubuntu better? we can decide on ubuntu instead 15:36:41 in fact that's the most tested distro anyway... 15:36:55 Around SELinux and AppArmor - I'm sure it's going to be more complicated to implement it when having different host os and in-container os 15:37:31 mnasiadka: re selinux and apparmor - you need custom policies anyway - better write them for one setup 15:38:41 We need support for both CentOS/RHEL/etc and Ubuntu on the Host OS, and there are customers that want to have the same ,,flavor'' on the Host OS and in the containers - it's not that we dictate what you get - we don't want to loose customers this way. 15:39:13 And I - as a PTL - am worried with users/operators wandering off somewhere else, because we do this - or because of issues that we'll bring on them ;-) 15:39:49 I think for most users dropping binary is a change, let's not push it to the limits. 15:40:13 so that means you want to build CS9 support? 15:40:47 That's up for discussion, for now we'd like to stick to CS8 as long as it's possible. 15:41:04 not much longer beyond the Yoga cycle 15:41:06 ;-) 15:41:14 once RHEL9 is out, there might be other options viable than CentOS Stream 9 15:41:48 we have a few related research items 15:41:53 I *think* long term you, as a company, would benefit more from streamlining clients to the same set of images 15:42:02 1. libvirt on the host 15:42:13 as you support them, they don't have to worry about discrepancies as you solve them 15:42:19 2. py3.9 in cs8 images 15:42:33 but surely short-term this sounds like some extra work 15:43:05 there is certainly an argument for single distro - it hasn't been an easy call to make 15:43:28 ad 1) seemingly libvirt in container is quite fine with different kernels, though it may need some extra verification 15:43:37 ad 2) that we can try to extedn the life of it 15:44:01 we have seen issues with mixing libvirt versions 15:44:02 since we don't have to have support for python-selinux/python-dnf inside 15:44:39 mgoddard: yeah, mixing is problematic; but still it contributes to making it better by controlling the version in the container 15:45:19 Well, we already allowed libvirt 6.9.0 in stream images, which has a peculiar bug spamming the logs with one message constantly ;-) 15:45:24 or just run it on the host and don't worry :) 15:46:13 We can have both options I guess. 15:46:20 libvirt 7.9 ;) 15:46:27 ah, 7.9 15:46:34 mnasiadka: what does it say? 15:46:50 yoctozepto: https://www.mail-archive.com/libvir-list@redhat.com/msg222970.html this - but not only on power9 15:46:58 it's patched in 7.10 and doesn't exist in 7.6 15:47:13 I got this on some dell x86 as well 15:47:30 well, the issue with having both supported means more to support :D 15:47:39 mnasiadka: eh :D 15:47:51 yoctozepto: that's what we get with stream ;-) 15:48:07 And we probably get those messages every week that nobody wants to use stream. 15:48:34 well, it's odd they let this kind of issue slip in, seems quite widespread 15:48:50 well, RHEL sticks to 7.6 15:48:59 I think 15:49:02 eh 15:49:17 Anyway, we're diverting - is there anything more to add to this topic? 15:49:38 rocky linux in images instead? 15:49:43 but I don't want to do it 15:50:35 I think if our company wants to continue current path, we need to take this - but let's see how the situation evolves with the research activities mgoddard mentioned 15:50:50 and if Rocky Linux doesn't die before RHEL 9 gets out ;-) 15:51:13 hah, true that 15:51:38 Ok, I crossed out the single distro from the priorities for now. 15:51:56 Around it, let's try to grasp what is missing for deprecating source. 15:52:12 #topic Current cycle planning 15:52:18 binary? 15:52:25 binary, yes 15:52:29 my mind is not best today. 15:52:42 Kolla whiteboard L316 15:52:44 don't worry, I'm struggling these days too 15:52:49 #link https://etherpad.opendev.org/p/KollaWhiteBoard 15:53:08 TODO(): better document the differences between them, so that users can make an educated decision transition more easily 15:53:25 Is there a volunteer to do that? Do we still think we need to do that? 15:54:10 hmm 15:54:36 sounds nice to have, but not volunteering 15:55:15 I think we could use some more info about what source images are - there seems to be a general lack of understanding 15:55:38 indeed 15:55:43 Yes, agreed 15:55:51 But there's no volunteer - I will keep asking on next meetings. 15:56:13 frickler: did you have any progress with your task? 15:56:19 Do you need any help? 15:56:36 mnasiadka: well I found that osism has some framework around it https://github.com/osism/container-images-kolla 15:56:52 with options to add patches and things into image builds 15:57:22 but I haven't used it myself yet, so only basic understanding so far 15:57:22 Yes, but I think kevko was a special case here - especially around overriding pip packages versions and uppper-constraints 15:57:46 So if that's possible - we'd need some improved docs on overriding these, if not - we need to think how to implement it. 15:58:30 And I think that's most important task from all those that we have in this ,,priority item'' 15:59:24 Anyway, time's up - let's try to at least iron out what needs to be done in this ;-) 15:59:36 Thanks for attending. 15:59:39 #endmeeting