14:00:20 <mnasiadka> #startmeeting kolla
14:00:20 <opendevmeet> Meeting started Wed May 25 14:00:20 2022 UTC and is due to finish in 60 minutes.  The chair is mnasiadka. Information about MeetBot at http://wiki.debian.org/MeetBot.
14:00:20 <opendevmeet> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
14:00:20 <opendevmeet> The meeting name has been set to 'kolla'
14:00:31 <mnasiadka> #topic rollcall
14:00:41 <frickler> \o
14:00:45 <mnasiadka> o/
14:00:51 <matfechner> o/
14:01:15 <yoctozepto> o/
14:01:27 <hrw> (_)(_)
14:02:02 <mmalchuk> hi
14:02:48 <mnasiadka> #topic agenda
14:02:48 <mnasiadka> * Announcements
14:02:48 <mnasiadka> * Review action items from the last meeting
14:02:48 <mnasiadka> * CI status
14:02:48 <mnasiadka> * Release tasks
14:02:49 <mnasiadka> * Current cycle planning
14:02:49 <mnasiadka> * Additional agenda (from whiteboard)
14:02:51 <mnasiadka> * Open discussion
14:02:52 <mnasiadka> #topic Announcements
14:03:20 <yoctozepto> mgoddard is back
14:03:29 <mnasiadka> Kolla has a Forum session on the upcoming Summit (I think it's on Wed)
14:03:59 <mnasiadka> I'll create an etherpad for planning - and I invite all people to join ;-)
14:04:12 <mnasiadka> yoctozepto: mgoddard is off today IIRC :)
14:04:18 <frickler> will there be remote access?
14:04:19 <mnasiadka> #topic Review action items from the last meeting
14:04:28 <hrw> is forum only for attendees or remote as well?
14:04:42 <yoctozepto> ++
14:04:46 <mnasiadka> I can create a google meet meeting and send out on the list
14:05:01 <mnasiadka> with a link to the etherpad
14:05:06 <mnasiadka> but there will be no A/V in the room
14:05:06 <yoctozepto> though I will be not available most likely
14:05:19 <mnasiadka> so it's only my laptop as A/V equipment :)
14:05:29 <mnasiadka> I'll do it anyway.
14:05:39 <mnasiadka> last meeting items
14:05:41 <mnasiadka> let's see
14:05:49 <mnasiadka> frickler update grafana dashboard
14:05:55 <mnasiadka> frickler: how are you with this?
14:05:58 <frickler> still not there, but I pushed it to the top of my list now
14:06:16 <mnasiadka> ok then
14:06:30 <mnasiadka> #action frickler update grafana dashboard
14:06:38 <mnasiadka> #topic CI status
14:06:49 <mnasiadka> how's the CI? bleeding red? green as grass?
14:06:52 <frickler> that was a fun weekend
14:06:52 <yoctozepto> green now
14:06:53 <hrw> aarch64 broken in master
14:07:03 <hrw> iirc
14:07:11 <yoctozepto> broken everywhere
14:07:28 <mnasiadka> ok, x86 green, aarch broken
14:07:31 <frickler> kayobe was stiil broken when I last looked and blocking collection things
14:07:35 <yoctozepto> and we are green because of the weekend effort
14:07:39 <hrw> https://zuul.openstack.org/builds?job_name=kolla-ansible-debian-source-aarch64 lists xena and wallaby as fine
14:07:46 <yoctozepto> ah yes, kayobe sad
14:07:48 <mnasiadka> I think priteau is trying to fix kayobe master CI
14:07:58 <mnasiadka> to pin requirements to use yoga instead of master
14:08:00 <hrw> v/w/x fine, y/m broken
14:08:01 <mnasiadka> for now
14:08:14 <mnasiadka> until we'll get to RL9
14:08:15 <mmalchuk> https://review.opendev.org/c/openstack/kayobe/+/843117 can be merged
14:08:32 <hrw> #action hrw to look for aarch64 failures in yoga/master
14:08:40 <hrw> mnasiadka: ^^
14:08:59 <mnasiadka> ok then, fantastic
14:09:07 <mnasiadka> #topic Release tasks
14:09:42 <hrw> mnasiadka: only chair can #commands so you have to repeat to make it saved
14:09:46 <mnasiadka> It's R-19 week
14:09:59 <mnasiadka> #action hrw to look for aarch64 failures in yoga/master
14:10:09 <hrw> thx
14:10:18 <mnasiadka> Yoga final releases are out
14:10:24 <hrw> yay
14:11:32 <mnasiadka> There was this point on the PTG of automating point releases for stable branches
14:11:39 <mnasiadka> But there is no obvious automation out there to do it
14:11:58 <mnasiadka> So we're sort of stuck with doing that manually, like every month or something
14:12:05 <frickler> there is some release tooling
14:12:18 <frickler> I added some comments on the ptg pad I think
14:12:38 <frickler> but you have to run the script manually and then submit the patch
14:13:08 <mnasiadka> yeah
14:13:33 <mnasiadka> It would be nice to add it to the weekly meeting page in the docs, so we don't forget - and maybe a Kolla Whiteboard line when was it done last time
14:13:42 <frickler> https://etherpad.opendev.org/p/kolla-zed-ptg#L261
14:13:55 <mnasiadka> Yeah, found that oneliner ;)
14:13:57 <frickler> ack, I can prepare that
14:14:04 <mnasiadka> great
14:14:34 <frickler> beginning of each month would match with next meeting
14:14:42 <mnasiadka> #action frickler add a mention of doing monthly point releases for stable branches in kolla meetings docs page
14:15:07 <mnasiadka> Anybody can propose the releases, just me or mgoddard need to approve it
14:15:34 <mnasiadka> Probably we could use more release liasons with this process so SHPC unavailability doesn't block it ;-)
14:16:00 <mnasiadka> #action mnasiadka add more release liasons
14:16:15 <mnasiadka> #topic Current cycle planning
14:16:15 <yoctozepto> ++
14:16:27 <mnasiadka> yoctozepto mentioned Let's encrypt is on top of his review list - how is it going?
14:16:47 <yoctozepto> still on top
14:16:56 <mnasiadka> ok then
14:17:08 <yoctozepto> let's discuss in a week
14:17:28 <hrw> https://review.opendev.org/c/openstack/kolla/+/834211 (certbot -> lego) would be nice to have
14:17:39 <hrw> it builds on cs9 contrary to certbot
14:18:10 <mnasiadka> I've spent some time on trying to built Ubuntu 22.04 images - TL;DR - fluentd is broken, collectd-core has been removed from repo (so telegraf and collectd are unbuildable), mariadb is currently from Ubuntu repos, because MariaDB folks failed to build packages for latest 10.6 point release
14:18:26 <mnasiadka> so basically we need to wait for fluentd to shape up
14:18:49 <hrw> similar for cs9 - no fluentd, no elk, no grafana
14:18:53 <mnasiadka> I opened an issue in fluent builder around packages being built, let's see how many years do we need to wait
14:19:19 <hrw> mnasiadka: the good part is that they need to do same work for both cs9 and 22.04
14:19:37 <hrw> move to newer ruby, openssl3 etc
14:20:02 <mnasiadka> yeah
14:20:11 <mnasiadka> I think we have the priorities covered
14:20:20 <mnasiadka> yoctozepto: I think after LE - the next is podman?
14:20:34 <mhiner> Yep
14:20:37 <yoctozepto> yeah
14:20:52 <frickler> I plan to look at podman too
14:21:09 <frickler> and hopefully a colleague of mine, matfechner, can also get involved
14:21:56 <mnasiadka> lego is merging
14:22:05 <mhiner> So for starters, we decided to use podman-py after all
14:22:21 <hrw> mnasiadka: cool
14:22:27 <mnasiadka> I'll add my haproxy fqdn based frontend patches to the list on the whiteboard
14:22:34 <mnasiadka> if anybody wants to review ;-)
14:22:41 <mhiner> the previous solution using ansible modules didn't work
14:23:00 <mhiner> also can you take a look at systed patchset?
14:23:36 <mhiner> from our poit of view it's finished and we would like some feedback on it
14:24:02 <frickler> can we have a common topic for those patches?
14:24:03 <mnasiadka> yoctozepto: I guess that's for you and mgoddard
14:24:21 <yoctozepto> mnasiadka: and frickler I hope
14:24:23 <mnasiadka> common topic and a link on the whiteboard (L392)
14:24:33 <yoctozepto> yes, let's collect them
14:24:52 <yoctozepto> as the old adage says
14:24:57 <yoctozepto> gotta catch 'em all
14:25:07 <mnasiadka> pokemons?
14:25:13 <mnasiadka> anyway, let's move on :)
14:25:20 <mnasiadka> #topic Additional agenda (from whiteboard)
14:25:26 <yoctozepto> mnasiadka: czy już wszystkie masz? czy już masz? musimy ocalić świat
14:25:30 <mnasiadka> PTG - virtual, in person or mixed? (frickler)
14:25:45 <hrw> mixed
14:25:49 <mnasiadka> cf. https://lists.openstack.org/pipermail/openstack-discuss/2022-May/028601.html
14:25:49 <mnasiadka> I would very much like to see a commitment to do our PTGs virtual, forever. Arguments:
14:25:49 <mnasiadka> Climate crisis
14:25:49 <mnasiadka> Inclusiveness for people not able to travel for various reasons (kids, money, health, visa, ...)
14:25:49 <mnasiadka> These also would not be mitigated by a mixed (in person with remote participation)
14:25:49 <mnasiadka> yoctozepto agrees with the above, though he is open to having a meeting in-person with no strict agenda
14:26:29 <hrw> would love to attend PTG in person but without half of cores it would be sitting next to laptop anyway
14:26:30 <mnasiadka> Well, I think mixed makes sense - some people are going to be physically on the PTG for other projects, and some people for obvious reasons are going to stay home
14:26:36 <frickler> yeah, another kind of virtualization topic
14:26:51 <frickler> mixed is difficult
14:26:54 <hrw> also budget can be an issue - where next ptg takes place?
14:27:04 <mnasiadka> Columbus, Ohio
14:27:09 <frickler> because it prioritized in-person attendance still
14:27:19 <hrw> USA. mixed or virtual for most then
14:27:51 <hrw> I highly doubt that my manager will say "here is 3k USD for you"
14:28:35 <frickler> my suggestion would be to do virtual session(s) outside the in-person PTG timeframe
14:28:40 <frickler> to avoid conflict
14:29:14 <frickler> IMO the last PTGs worked very well
14:29:56 <mnasiadka> True, let's see how this topic evolves.
14:30:01 <frickler> and I'd also like to show the foundation that the way they are handling this isn't supported by the community
14:30:16 <mnasiadka> Kubecon in Detroit is week before PTG, so some people might be there as well.
14:30:55 <frickler> but we don't need to decide right now
14:31:16 <frickler> I just wanted to discuss early so not everyone says "oh we already booked everything"
14:31:47 <hrw> once they try to force in-person ptg the projects will start considering leaving openinfra
14:31:48 <mnasiadka> sure
14:32:07 <mnasiadka> I think they never forced that, but I was on one ,,mixed'' Kolla PTG in Denver
14:32:08 <yoctozepto> hrw: they can't "force" it tbh
14:32:51 <hrw> I was in Dublin. nice experience but not quite something I need to repeat
14:33:56 <mnasiadka> anyway, let's move on - we know what are our feelings about PTG, remote works out quite well
14:34:09 <mnasiadka> But we can't force somebody to not be physically there and connect with us ;-)
14:34:25 <yoctozepto> the truth has been spoken
14:34:46 <mnasiadka> (yoctozepto) CentOS Stream 8 fate, cf. https://review.opendev.org/c/openstack/kolla-ansible/+/842842
14:35:09 <yoctozepto> are we still testing it?
14:35:15 <yoctozepto> can we drop it from master CI?
14:35:19 <yoctozepto> when?
14:35:34 <hrw> once we move to master
14:35:46 <hrw> as we use yoga still
14:36:40 <yoctozepto> is that the common understanding?
14:36:51 <yoctozepto> i'm ok with it, though ready to drop now
14:36:59 <frickler> +2
14:37:07 <opendevreview> Maksim Malchuk proposed openstack/kolla master: Replace Certbot with Lego for Let's Encrypt container  https://review.opendev.org/c/openstack/kolla/+/834211
14:37:08 <opendevreview> Maksim Malchuk proposed openstack/kolla master: Add ability to override repos  https://review.opendev.org/c/openstack/kolla/+/842472
14:37:27 <mnasiadka> Dropping now - I think it's useful for merging bugs and backports - and making sure we don't break CentOS?
14:37:44 <yoctozepto> yes, probably easier this way now
14:37:59 <mmalchuk> ops... sorry
14:38:18 <mmalchuk> trying to solve conflicts
14:38:31 <mnasiadka> ok then, so once CS9 images are buildable - we'll get rid of CS8
14:38:50 <frickler> mmalchuk: np, its our fault using this channel for meetings, not yours ;)
14:38:58 <hrw> mnasiadka: once we move to master, we get rid of CS8 from CI. nevermind of CS9 state.
14:39:02 <mnasiadka> (yoctozepto) Use kolla-toolbox for clients in CI
14:39:02 <mnasiadka> this is related to CentOS Stream 8 fate and future-proofing as it avoids us doing clients installations (fewer things to fail)
14:39:02 <mnasiadka> this is also related to avoiding the use of u-c for kolla projects - clients would be the last bastion where this would have been used
14:39:17 <yoctozepto> frickler: or keeping the bot in it ;-)
14:39:34 <yoctozepto> yes, what's your opinion about that?
14:39:43 <yoctozepto> we install clients ad-hoc each time
14:39:54 <yoctozepto> maybe reusing kolla-toolbox for CI testing would be nicer
14:39:57 <mnasiadka> I think I'm all in, less installations in CI, less places where it breaks.
14:40:05 <yoctozepto> fewer* places
14:40:11 <mnasiadka> yeah, fewer
14:40:13 <hrw> +1 for it as we have them in k-t anyway
14:40:15 <yoctozepto> sorry, bad habits
14:40:26 <yoctozepto> we might be missing some clients
14:40:34 <yoctozepto> we can install them though and have better coverage
14:40:48 <yoctozepto> maybe we come up with some interesting 2nd-day actions
14:40:50 <mnasiadka> no worries, my brain feels like it's Friday evening (too much apocalypse-like issues on customers for these two days)
14:41:02 <yoctozepto> oh my, ack
14:41:59 <mnasiadka> ok then, direction approved ;)
14:42:02 <mnasiadka> (m.hiner): Podman & Systemd status review
14:42:02 <mnasiadka> https://review.opendev.org/c/openstack/kolla-ansible/+/816724
14:42:09 <mnasiadka> I think we discussed that already
14:42:44 <mhiner> mostly
14:42:49 <mnasiadka> Anything to add?
14:43:06 <mhiner> I just want to add that the whole podman patchset is now outdated as we are reworking it
14:43:27 <hrw> mhiner: add comment to patch?
14:43:46 <mnasiadka> change it's status to work in progress
14:44:08 <frickler> mhiner: so are you reworking 816724, too?
14:44:39 <mhiner> no, that is final for now
14:45:20 <mnasiadka> mhiner: just for patches that you're still working on - mark their status as in progress in Gerrit
14:45:32 <frickler> o.k., anything we can help you with for the reworking?
14:46:06 <opendevreview> Martin Hiner proposed openstack/kolla-ansible master: Adding support of podman deployment (WIP)  https://review.opendev.org/c/openstack/kolla-ansible/+/799229
14:47:04 <halomiva> there is functionality in podman-py missing
14:47:36 <yoctozepto> let's document what is missing
14:47:39 <halomiva> for podman exec API call and we are kinda depend on it if we don't want to do it with some workaround
14:48:04 <halomiva> it should be in late stages of development https://github.com/containers/podman-py/pull/126
14:49:33 <frickler> did you test with that patch already?
14:49:45 <halomiva> not yet
14:50:11 <halomiva> and there is also other issue with volumes but it can be done with mounts instead
14:50:39 <frickler> is there an upstream issue for that, too?
14:52:24 <halomiva> no
14:52:38 <opendevreview> Michal Nasiadka proposed openstack/kolla master: veryWIP: Move to Ubuntu Jammy (22.04)  https://review.opendev.org/c/openstack/kolla/+/839585
14:53:02 <mnasiadka> Ok, let's move on - time is ticking ;-)
14:53:05 <mnasiadka> #topic Open discussion
14:53:22 <mnasiadka> Anyone wants to discuss anything?
14:53:55 <hrw> who goes to Summit?
14:54:08 <yoctozepto> not me
14:54:10 <mmalchuk> masakari?
14:54:17 <mmalchuk> not me
14:54:31 <hrw> asked who goes, not who does not :D
14:54:32 <mnasiadka> I'm going
14:54:38 <mmalchuk> should we enable hacluster together with masakari?
14:54:38 <mnasiadka> frickler: ?
14:54:42 <rockey> I am, so would be fun meeting you guys which is attending :)
14:54:42 <frickler> nope
14:55:05 <yoctozepto> mnasiadka: others from shpc?
14:55:11 <hrw> I can drive to berlin for one afternoon. return train on ~21:30
14:55:36 <hrw> on tuesday or wednesday
14:55:43 <mnasiadka> yoctozepto: yeah, around 8-10 people
14:55:50 <yoctozepto> whoa
14:57:13 <mnasiadka> ok then, I guess that's it - thanks for coming!
14:57:15 <mnasiadka> #endmeeting