13:00:52 <mnasiadka> #startmeeting kolla
13:00:52 <opendevmeet> Meeting started Wed Jun 21 13:00:52 2023 UTC and is due to finish in 60 minutes.  The chair is mnasiadka. Information about MeetBot at http://wiki.debian.org/MeetBot.
13:00:52 <opendevmeet> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
13:00:52 <opendevmeet> The meeting name has been set to 'kolla'
13:00:55 <mnasiadka> #topic rollcall
13:00:56 <mnasiadka> o/
13:01:03 <ramona-beermann[m]> o/
13:01:04 <frickler> \o
13:01:11 <kevko> \o/
13:01:18 <bbezak> o/
13:01:39 <SvenKieske> o/
13:01:54 <ihalomi> \o
13:03:33 <mmalchuk> \o
13:03:35 <mnasiadka> #topic agenda
13:03:35 <mnasiadka> * CI status
13:03:35 <mnasiadka> * Release tasks
13:03:35 <mnasiadka> * Regular stable releases (first meeting in a month)
13:03:35 <mnasiadka> * Current cycle planning
13:03:37 <mnasiadka> * Additional agenda (from whiteboard)
13:03:37 <mnasiadka> * Open discussion
13:03:39 <mnasiadka> #topic CI status
13:03:54 <mnasiadka> It was red, but we're getting to green with RMQ and gnocchi unbuildable fixes
13:03:55 <opendevreview> Verification of a change to openstack/kayobe master failed: Revert "Use OpenStack 2023.1 release"  https://review.opendev.org/c/openstack/kayobe/+/886313
13:03:58 <opendevreview> Verification of a change to openstack/kayobe master failed: set previous_release to 2023.1  https://review.opendev.org/c/openstack/kayobe/+/886396
13:04:14 <kevko> btw, sorry for stupid question ..but where i can check ci status page for kolla, kolla-ansible ?
13:04:14 <frickler> well yellow at best I'd say
13:04:34 <frickler> lots of failures like https://review.opendev.org/c/openstack/kolla/+/886467/comments/ed6de50a_0c6c9a3b needing recheck
13:04:36 <mnasiadka> yeah, looking now it's yellow ;)
13:04:48 <mnasiadka> kevko: we basically update the whiteboard with status, of what we see every day
13:04:55 <SvenKieske> kevko: some of it here: https://grafana.opendev.org/d/c0d59dad13/kolla-failure-rate?orgId=1
13:04:59 <mnasiadka> there is a dashboard on grafana.opendev - but it needs some love
13:05:01 <spatel> \o/
13:05:54 <SvenKieske> it got some love from you, but it could certainly need more :)
13:05:55 <kevko> thanks, bookmarked
13:06:28 <SvenKieske> sources are here: https://opendev.org/openstack/project-config/src/branch/master/grafana/kolla.yaml
13:08:03 <mnasiadka> #topic Release tasks
13:08:14 <mnasiadka> So, we're on the path to use master again on master
13:08:17 <mnasiadka> so that's covered
13:08:25 <mnasiadka> #topic Current cycle planning
13:08:33 <mnasiadka> ihalomi: how's podman?
13:08:53 <ihalomi> not good, im stuck on problem with rabbitmq
13:09:34 <ihalomi> solution about mounting subdirectories of run and using tmpfiles.d config seems working but i cant make deployment finish successfully
13:10:10 <kevko> some log ?
13:10:40 <ihalomi> it fails on podman exec rabbitmq rabbitmqctl wait /var/lib/rabbitmq/mnesia/rabbitmq.pid
13:11:15 <ihalomi> where it timeouts with Waiting for pid file '/var/lib/rabbitmq/mnesia/rabbitmq.pid' to appear but pid file is there and when I check that process is running
13:11:36 <SvenKieske> that seems like not a correct directory to write a pid file? sure it is there? oO
13:11:45 <SvenKieske> ah ok
13:12:15 <SvenKieske> did you check permissions? can rabbitmq user access the pid file?` have seen such errors many times
13:12:25 <ihalomi> i didnt change anything in there so it looks like some hidden podman problem
13:12:41 <SvenKieske> if you could share all this stuff on some etherpad or somewhere maybe ppl could help :)
13:13:11 <ihalomi> i can write some summary and post the newest patchset so you can look
13:13:58 <SvenKieske> that would be nice
13:14:30 <mnasiadka> good progress anyway ;)
13:14:48 <mnasiadka> #topic Additional agenda (from whiteboard)
13:15:09 <mnasiadka> nothing on the whiteboard
13:15:12 <mnasiadka> but I have one
13:15:17 <mnasiadka> Wallaby and Xena EOL
13:15:23 <ihalomi> is this correct permissions for that pid? -rw-r--r--. 1 rabbitmq rabbitmq 2 Jun 21 08:56 /var/lib/rabbitmq/mnesia/rabbitmq.pid
13:15:43 <mnasiadka> Anybody wants to maintain those branches or are we marking them as EOL?
13:16:07 <mmalchuk> lets fixes on review landed and ok for EOL
13:16:22 <mnasiadka> ihalomi: correct
13:16:48 <mnasiadka> ok, so we should have a look on all reviews in open state for those branches and post EOL patch
13:16:51 <frickler> as already mentioned +1 for EOL from me
13:16:52 <mmalchuk> mnasiadka I can only Xena
13:17:07 <SvenKieske> ihalomi: looks like it; I hope nobody needs execute rights on this stuff
13:17:08 <ihalomi> also when i was rebasing on the newest patch there was few newly added docker commands so could you try keep in mind that now kolla_container_engine is used? fix is here https://review.opendev.org/c/openstack/kolla-ansible/+/886391
13:18:15 <mnasiadka> ihalomi: we'll try our best ;)
13:18:24 <frickler> ihalomi: oh, that's going to be tough I fear. can we lint that somehow?
13:19:30 <mnasiadka> we could probably define a custom check to ansible lint?
13:19:43 <frickler> we could make a CI job where we rename the command maybe. mv docker foobar
13:20:21 <frickler> or reviewers could just try to keep that in mind, yes
13:21:46 <mnasiadka> custom lint rule would probably be better, but can take some more time ;)
13:22:09 <mnasiadka> ok, so I think we have an agreement on EOL
13:22:16 <mnasiadka> #agreed to mark Wallaby and Xena EOL
13:22:31 <mnasiadka> #topic Open discussion
13:22:40 <mnasiadka> anyone anything?
13:22:53 <mmalchuk> https://review.opendev.org/q/topic:fix-app-cred-access-rules
13:22:58 <mmalchuk> discussed before
13:23:09 <mmalchuk> not breaking changes
13:23:14 <mmalchuk> lets merge
13:23:59 <mmalchuk> 2. lack of reviewers in Kayobe
13:24:09 <mmalchuk> https://review.opendev.org/c/openstack/kayobe/+/879554
13:24:16 <mmalchuk> https://review.opendev.org/c/openstack/kayobe/+/861397
13:24:29 <mmalchuk> ask from time to time and nothing changes(
13:24:52 <mnasiadka> and vacation season is approaching, so it's not going to get better
13:25:09 * mmalchuk sad
13:25:17 <mmalchuk> +RP ?
13:25:35 <mnasiadka> on the Kolla User Forum in Vancouver - there was a proposal on supporting multiple RabbitMQ clusters (or at least one additional for notifications)
13:25:44 <mnasiadka> anybody here has similar thoughts?
13:26:28 <mmalchuk> +RP ?
13:27:20 <kevko> mnasiadka this would be helfull for multiple cells
13:27:35 <mnasiadka> yeah, that's another thing - multiple cells
13:27:41 <SvenKieske> this is also a recommendation from the large scale sig afaik
13:28:01 <SvenKieske> so it's a good feature to have, I'd say
13:28:11 <mnasiadka> If somebody is giving - sure :)
13:28:14 <kevko> mnasiadka multiple rabbitmq feature can be copied from my patch for mariadb
13:29:15 <mnasiadka> labedz: if you guys are willing to work on this (as spoken on the Summit) and plan to submit a patch this cycle - I can add that to the cycle priorities
13:29:23 <frickler> and then we need a 6-node CI job to validate it?
13:29:32 <mnasiadka> we had 6 node CI job for Monasca
13:29:55 <kevko> well, you can spin 2/3 one node rabbitmq cluster i think
13:30:04 <kevko> two
13:30:06 <mnasiadka> I don't know how big the cells job is currently
13:30:21 <kevko> one node and two nodes if i remember correctly
13:30:42 <kevko> with two mariadb proxied through proxysql ..but one rabbit
13:31:04 <mnasiadka> so it should be easy to improve that, once somebody wants to work on it :)
13:31:23 <mnasiadka> ok, let's try to stabilise things before next week
13:31:35 <mnasiadka> btw - I'm going on vacation again for 3 weeks of July
13:31:47 <mnasiadka> Is anybody going to be around to run the meetings - or should we cancel them?
13:32:02 <mnasiadka> (that's mainly vacation season everywhere)
13:32:34 <mmalchuk> me too on vacation in July)
13:33:03 <frickler> fine to cancel I'd say, we can discuss things ad hoc as needed
13:33:14 <mnasiadka> ok then, I'll send a mail next week to the ML
13:33:32 <mnasiadka> That's all for me for today - thanks for coming :)
13:33:36 <mnasiadka> #endmeeting