14:00:20 <mnasiadka> #startmeeting kolla 14:00:20 <opendevmeet> Meeting started Wed May 25 14:00:20 2022 UTC and is due to finish in 60 minutes. The chair is mnasiadka. Information about MeetBot at http://wiki.debian.org/MeetBot. 14:00:20 <opendevmeet> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 14:00:20 <opendevmeet> The meeting name has been set to 'kolla' 14:00:31 <mnasiadka> #topic rollcall 14:00:41 <frickler> \o 14:00:45 <mnasiadka> o/ 14:00:51 <matfechner> o/ 14:01:15 <yoctozepto> o/ 14:01:27 <hrw> (_)(_) 14:02:02 <mmalchuk> hi 14:02:48 <mnasiadka> #topic agenda 14:02:48 <mnasiadka> * Announcements 14:02:48 <mnasiadka> * Review action items from the last meeting 14:02:48 <mnasiadka> * CI status 14:02:48 <mnasiadka> * Release tasks 14:02:49 <mnasiadka> * Current cycle planning 14:02:49 <mnasiadka> * Additional agenda (from whiteboard) 14:02:51 <mnasiadka> * Open discussion 14:02:52 <mnasiadka> #topic Announcements 14:03:20 <yoctozepto> mgoddard is back 14:03:29 <mnasiadka> Kolla has a Forum session on the upcoming Summit (I think it's on Wed) 14:03:59 <mnasiadka> I'll create an etherpad for planning - and I invite all people to join ;-) 14:04:12 <mnasiadka> yoctozepto: mgoddard is off today IIRC :) 14:04:18 <frickler> will there be remote access? 14:04:19 <mnasiadka> #topic Review action items from the last meeting 14:04:28 <hrw> is forum only for attendees or remote as well? 14:04:42 <yoctozepto> ++ 14:04:46 <mnasiadka> I can create a google meet meeting and send out on the list 14:05:01 <mnasiadka> with a link to the etherpad 14:05:06 <mnasiadka> but there will be no A/V in the room 14:05:06 <yoctozepto> though I will be not available most likely 14:05:19 <mnasiadka> so it's only my laptop as A/V equipment :) 14:05:29 <mnasiadka> I'll do it anyway. 14:05:39 <mnasiadka> last meeting items 14:05:41 <mnasiadka> let's see 14:05:49 <mnasiadka> frickler update grafana dashboard 14:05:55 <mnasiadka> frickler: how are you with this? 14:05:58 <frickler> still not there, but I pushed it to the top of my list now 14:06:16 <mnasiadka> ok then 14:06:30 <mnasiadka> #action frickler update grafana dashboard 14:06:38 <mnasiadka> #topic CI status 14:06:49 <mnasiadka> how's the CI? bleeding red? green as grass? 14:06:52 <frickler> that was a fun weekend 14:06:52 <yoctozepto> green now 14:06:53 <hrw> aarch64 broken in master 14:07:03 <hrw> iirc 14:07:11 <yoctozepto> broken everywhere 14:07:28 <mnasiadka> ok, x86 green, aarch broken 14:07:31 <frickler> kayobe was stiil broken when I last looked and blocking collection things 14:07:35 <yoctozepto> and we are green because of the weekend effort 14:07:39 <hrw> https://zuul.openstack.org/builds?job_name=kolla-ansible-debian-source-aarch64 lists xena and wallaby as fine 14:07:46 <yoctozepto> ah yes, kayobe sad 14:07:48 <mnasiadka> I think priteau is trying to fix kayobe master CI 14:07:58 <mnasiadka> to pin requirements to use yoga instead of master 14:08:00 <hrw> v/w/x fine, y/m broken 14:08:01 <mnasiadka> for now 14:08:14 <mnasiadka> until we'll get to RL9 14:08:15 <mmalchuk> https://review.opendev.org/c/openstack/kayobe/+/843117 can be merged 14:08:32 <hrw> #action hrw to look for aarch64 failures in yoga/master 14:08:40 <hrw> mnasiadka: ^^ 14:08:59 <mnasiadka> ok then, fantastic 14:09:07 <mnasiadka> #topic Release tasks 14:09:42 <hrw> mnasiadka: only chair can #commands so you have to repeat to make it saved 14:09:46 <mnasiadka> It's R-19 week 14:09:59 <mnasiadka> #action hrw to look for aarch64 failures in yoga/master 14:10:09 <hrw> thx 14:10:18 <mnasiadka> Yoga final releases are out 14:10:24 <hrw> yay 14:11:32 <mnasiadka> There was this point on the PTG of automating point releases for stable branches 14:11:39 <mnasiadka> But there is no obvious automation out there to do it 14:11:58 <mnasiadka> So we're sort of stuck with doing that manually, like every month or something 14:12:05 <frickler> there is some release tooling 14:12:18 <frickler> I added some comments on the ptg pad I think 14:12:38 <frickler> but you have to run the script manually and then submit the patch 14:13:08 <mnasiadka> yeah 14:13:33 <mnasiadka> It would be nice to add it to the weekly meeting page in the docs, so we don't forget - and maybe a Kolla Whiteboard line when was it done last time 14:13:42 <frickler> https://etherpad.opendev.org/p/kolla-zed-ptg#L261 14:13:55 <mnasiadka> Yeah, found that oneliner ;) 14:13:57 <frickler> ack, I can prepare that 14:14:04 <mnasiadka> great 14:14:34 <frickler> beginning of each month would match with next meeting 14:14:42 <mnasiadka> #action frickler add a mention of doing monthly point releases for stable branches in kolla meetings docs page 14:15:07 <mnasiadka> Anybody can propose the releases, just me or mgoddard need to approve it 14:15:34 <mnasiadka> Probably we could use more release liasons with this process so SHPC unavailability doesn't block it ;-) 14:16:00 <mnasiadka> #action mnasiadka add more release liasons 14:16:15 <mnasiadka> #topic Current cycle planning 14:16:15 <yoctozepto> ++ 14:16:27 <mnasiadka> yoctozepto mentioned Let's encrypt is on top of his review list - how is it going? 14:16:47 <yoctozepto> still on top 14:16:56 <mnasiadka> ok then 14:17:08 <yoctozepto> let's discuss in a week 14:17:28 <hrw> https://review.opendev.org/c/openstack/kolla/+/834211 (certbot -> lego) would be nice to have 14:17:39 <hrw> it builds on cs9 contrary to certbot 14:18:10 <mnasiadka> I've spent some time on trying to built Ubuntu 22.04 images - TL;DR - fluentd is broken, collectd-core has been removed from repo (so telegraf and collectd are unbuildable), mariadb is currently from Ubuntu repos, because MariaDB folks failed to build packages for latest 10.6 point release 14:18:26 <mnasiadka> so basically we need to wait for fluentd to shape up 14:18:49 <hrw> similar for cs9 - no fluentd, no elk, no grafana 14:18:53 <mnasiadka> I opened an issue in fluent builder around packages being built, let's see how many years do we need to wait 14:19:19 <hrw> mnasiadka: the good part is that they need to do same work for both cs9 and 22.04 14:19:37 <hrw> move to newer ruby, openssl3 etc 14:20:02 <mnasiadka> yeah 14:20:11 <mnasiadka> I think we have the priorities covered 14:20:20 <mnasiadka> yoctozepto: I think after LE - the next is podman? 14:20:34 <mhiner> Yep 14:20:37 <yoctozepto> yeah 14:20:52 <frickler> I plan to look at podman too 14:21:09 <frickler> and hopefully a colleague of mine, matfechner, can also get involved 14:21:56 <mnasiadka> lego is merging 14:22:05 <mhiner> So for starters, we decided to use podman-py after all 14:22:21 <hrw> mnasiadka: cool 14:22:27 <mnasiadka> I'll add my haproxy fqdn based frontend patches to the list on the whiteboard 14:22:34 <mnasiadka> if anybody wants to review ;-) 14:22:41 <mhiner> the previous solution using ansible modules didn't work 14:23:00 <mhiner> also can you take a look at systed patchset? 14:23:36 <mhiner> from our poit of view it's finished and we would like some feedback on it 14:24:02 <frickler> can we have a common topic for those patches? 14:24:03 <mnasiadka> yoctozepto: I guess that's for you and mgoddard 14:24:21 <yoctozepto> mnasiadka: and frickler I hope 14:24:23 <mnasiadka> common topic and a link on the whiteboard (L392) 14:24:33 <yoctozepto> yes, let's collect them 14:24:52 <yoctozepto> as the old adage says 14:24:57 <yoctozepto> gotta catch 'em all 14:25:07 <mnasiadka> pokemons? 14:25:13 <mnasiadka> anyway, let's move on :) 14:25:20 <mnasiadka> #topic Additional agenda (from whiteboard) 14:25:26 <yoctozepto> mnasiadka: czy już wszystkie masz? czy już masz? musimy ocalić świat 14:25:30 <mnasiadka> PTG - virtual, in person or mixed? (frickler) 14:25:45 <hrw> mixed 14:25:49 <mnasiadka> cf. https://lists.openstack.org/pipermail/openstack-discuss/2022-May/028601.html 14:25:49 <mnasiadka> I would very much like to see a commitment to do our PTGs virtual, forever. Arguments: 14:25:49 <mnasiadka> Climate crisis 14:25:49 <mnasiadka> Inclusiveness for people not able to travel for various reasons (kids, money, health, visa, ...) 14:25:49 <mnasiadka> These also would not be mitigated by a mixed (in person with remote participation) 14:25:49 <mnasiadka> yoctozepto agrees with the above, though he is open to having a meeting in-person with no strict agenda 14:26:29 <hrw> would love to attend PTG in person but without half of cores it would be sitting next to laptop anyway 14:26:30 <mnasiadka> Well, I think mixed makes sense - some people are going to be physically on the PTG for other projects, and some people for obvious reasons are going to stay home 14:26:36 <frickler> yeah, another kind of virtualization topic 14:26:51 <frickler> mixed is difficult 14:26:54 <hrw> also budget can be an issue - where next ptg takes place? 14:27:04 <mnasiadka> Columbus, Ohio 14:27:09 <frickler> because it prioritized in-person attendance still 14:27:19 <hrw> USA. mixed or virtual for most then 14:27:51 <hrw> I highly doubt that my manager will say "here is 3k USD for you" 14:28:35 <frickler> my suggestion would be to do virtual session(s) outside the in-person PTG timeframe 14:28:40 <frickler> to avoid conflict 14:29:14 <frickler> IMO the last PTGs worked very well 14:29:56 <mnasiadka> True, let's see how this topic evolves. 14:30:01 <frickler> and I'd also like to show the foundation that the way they are handling this isn't supported by the community 14:30:16 <mnasiadka> Kubecon in Detroit is week before PTG, so some people might be there as well. 14:30:55 <frickler> but we don't need to decide right now 14:31:16 <frickler> I just wanted to discuss early so not everyone says "oh we already booked everything" 14:31:47 <hrw> once they try to force in-person ptg the projects will start considering leaving openinfra 14:31:48 <mnasiadka> sure 14:32:07 <mnasiadka> I think they never forced that, but I was on one ,,mixed'' Kolla PTG in Denver 14:32:08 <yoctozepto> hrw: they can't "force" it tbh 14:32:51 <hrw> I was in Dublin. nice experience but not quite something I need to repeat 14:33:56 <mnasiadka> anyway, let's move on - we know what are our feelings about PTG, remote works out quite well 14:34:09 <mnasiadka> But we can't force somebody to not be physically there and connect with us ;-) 14:34:25 <yoctozepto> the truth has been spoken 14:34:46 <mnasiadka> (yoctozepto) CentOS Stream 8 fate, cf. https://review.opendev.org/c/openstack/kolla-ansible/+/842842 14:35:09 <yoctozepto> are we still testing it? 14:35:15 <yoctozepto> can we drop it from master CI? 14:35:19 <yoctozepto> when? 14:35:34 <hrw> once we move to master 14:35:46 <hrw> as we use yoga still 14:36:40 <yoctozepto> is that the common understanding? 14:36:51 <yoctozepto> i'm ok with it, though ready to drop now 14:36:59 <frickler> +2 14:37:07 <opendevreview> Maksim Malchuk proposed openstack/kolla master: Replace Certbot with Lego for Let's Encrypt container https://review.opendev.org/c/openstack/kolla/+/834211 14:37:08 <opendevreview> Maksim Malchuk proposed openstack/kolla master: Add ability to override repos https://review.opendev.org/c/openstack/kolla/+/842472 14:37:27 <mnasiadka> Dropping now - I think it's useful for merging bugs and backports - and making sure we don't break CentOS? 14:37:44 <yoctozepto> yes, probably easier this way now 14:37:59 <mmalchuk> ops... sorry 14:38:18 <mmalchuk> trying to solve conflicts 14:38:31 <mnasiadka> ok then, so once CS9 images are buildable - we'll get rid of CS8 14:38:50 <frickler> mmalchuk: np, its our fault using this channel for meetings, not yours ;) 14:38:58 <hrw> mnasiadka: once we move to master, we get rid of CS8 from CI. nevermind of CS9 state. 14:39:02 <mnasiadka> (yoctozepto) Use kolla-toolbox for clients in CI 14:39:02 <mnasiadka> this is related to CentOS Stream 8 fate and future-proofing as it avoids us doing clients installations (fewer things to fail) 14:39:02 <mnasiadka> this is also related to avoiding the use of u-c for kolla projects - clients would be the last bastion where this would have been used 14:39:17 <yoctozepto> frickler: or keeping the bot in it ;-) 14:39:34 <yoctozepto> yes, what's your opinion about that? 14:39:43 <yoctozepto> we install clients ad-hoc each time 14:39:54 <yoctozepto> maybe reusing kolla-toolbox for CI testing would be nicer 14:39:57 <mnasiadka> I think I'm all in, less installations in CI, less places where it breaks. 14:40:05 <yoctozepto> fewer* places 14:40:11 <mnasiadka> yeah, fewer 14:40:13 <hrw> +1 for it as we have them in k-t anyway 14:40:15 <yoctozepto> sorry, bad habits 14:40:26 <yoctozepto> we might be missing some clients 14:40:34 <yoctozepto> we can install them though and have better coverage 14:40:48 <yoctozepto> maybe we come up with some interesting 2nd-day actions 14:40:50 <mnasiadka> no worries, my brain feels like it's Friday evening (too much apocalypse-like issues on customers for these two days) 14:41:02 <yoctozepto> oh my, ack 14:41:59 <mnasiadka> ok then, direction approved ;) 14:42:02 <mnasiadka> (m.hiner): Podman & Systemd status review 14:42:02 <mnasiadka> https://review.opendev.org/c/openstack/kolla-ansible/+/816724 14:42:09 <mnasiadka> I think we discussed that already 14:42:44 <mhiner> mostly 14:42:49 <mnasiadka> Anything to add? 14:43:06 <mhiner> I just want to add that the whole podman patchset is now outdated as we are reworking it 14:43:27 <hrw> mhiner: add comment to patch? 14:43:46 <mnasiadka> change it's status to work in progress 14:44:08 <frickler> mhiner: so are you reworking 816724, too? 14:44:39 <mhiner> no, that is final for now 14:45:20 <mnasiadka> mhiner: just for patches that you're still working on - mark their status as in progress in Gerrit 14:45:32 <frickler> o.k., anything we can help you with for the reworking? 14:46:06 <opendevreview> Martin Hiner proposed openstack/kolla-ansible master: Adding support of podman deployment (WIP) https://review.opendev.org/c/openstack/kolla-ansible/+/799229 14:47:04 <halomiva> there is functionality in podman-py missing 14:47:36 <yoctozepto> let's document what is missing 14:47:39 <halomiva> for podman exec API call and we are kinda depend on it if we don't want to do it with some workaround 14:48:04 <halomiva> it should be in late stages of development https://github.com/containers/podman-py/pull/126 14:49:33 <frickler> did you test with that patch already? 14:49:45 <halomiva> not yet 14:50:11 <halomiva> and there is also other issue with volumes but it can be done with mounts instead 14:50:39 <frickler> is there an upstream issue for that, too? 14:52:24 <halomiva> no 14:52:38 <opendevreview> Michal Nasiadka proposed openstack/kolla master: veryWIP: Move to Ubuntu Jammy (22.04) https://review.opendev.org/c/openstack/kolla/+/839585 14:53:02 <mnasiadka> Ok, let's move on - time is ticking ;-) 14:53:05 <mnasiadka> #topic Open discussion 14:53:22 <mnasiadka> Anyone wants to discuss anything? 14:53:55 <hrw> who goes to Summit? 14:54:08 <yoctozepto> not me 14:54:10 <mmalchuk> masakari? 14:54:17 <mmalchuk> not me 14:54:31 <hrw> asked who goes, not who does not :D 14:54:32 <mnasiadka> I'm going 14:54:38 <mmalchuk> should we enable hacluster together with masakari? 14:54:38 <mnasiadka> frickler: ? 14:54:42 <rockey> I am, so would be fun meeting you guys which is attending :) 14:54:42 <frickler> nope 14:55:05 <yoctozepto> mnasiadka: others from shpc? 14:55:11 <hrw> I can drive to berlin for one afternoon. return train on ~21:30 14:55:36 <hrw> on tuesday or wednesday 14:55:43 <mnasiadka> yoctozepto: yeah, around 8-10 people 14:55:50 <yoctozepto> whoa 14:57:13 <mnasiadka> ok then, I guess that's it - thanks for coming! 14:57:15 <mnasiadka> #endmeeting