14:00:34 <bbezak> #startmeeting kolla
14:00:34 <opendevmeet> Meeting started Wed Nov 13 14:00:34 2024 UTC and is due to finish in 60 minutes.  The chair is bbezak. Information about MeetBot at http://wiki.debian.org/MeetBot.
14:00:34 <opendevmeet> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
14:00:34 <opendevmeet> The meeting name has been set to 'kolla'
14:00:47 <bbezak> #topic rollcall
14:01:23 <SvenKieske> o/
14:01:27 <frickler> \o
14:02:14 <Guest9306> I can see that core team is around, good :)
14:03:32 <Guest9306> #topic agenda
14:03:33 <Guest9306> * Roll-call... (full message at <https://matrix.org/oftc/media/v1/media/download/Aaudh7XK0DL4kQY_zkWsYCLPncW_1ApJNdaQnSNYQH0FUwwGMopjXbBNAYF1ZXJN3K2wNqjWndpaEAzyoxh2jx5CeTbe8zCAAG1hdHJpeC5vcmcvbmtpdEZJUURZdVlObk13eExmaVFsU1Jj>)
14:03:41 <Guest9306> (i hope rendering is not very bad)
14:03:45 <frickler> it is
14:03:53 <SvenKieske> rendering is very bad (just a link) :D
14:04:06 <frickler> you need to paste each line on it's own
14:04:15 <frickler> but for now I guess we can live with it
14:04:17 <Guest9306> yeah. I'll try to remember that
14:04:26 <Guest9306> #topic CI status
14:04:55 <Guest9306> whiteboard says green
14:05:16 <Guest9306> I didn't see massive issues recently
14:05:25 <frickler> there were some infra issues yesterday, but should be fine now again
14:05:36 <Guest9306> ok
14:06:05 <Guest9306> #topic Release tasks
14:06:15 <Guest9306> how is kayobe doing
14:06:26 <Guest9306> jovial
14:06:39 <Guest9306> I'll check with him
14:07:47 <Guest9306> #topic Current cycle planning
14:08:26 <Guest9306> I guess we're just after PTG and in the beginning of the cycle
14:08:49 <Guest9306> rabbitmq is a good topic, but I saw discussion in the change SvenKieske
14:09:43 <SvenKieske> yes, so on one hand CI is "green" but looking at actual logs it's not that green and I wonder why that many errors are reported as green, but I didn't have time to dive deep into the issues and if they are even related.
14:10:14 <Guest9306> any particular example?
14:10:33 <SvenKieske> and if I'm not missing something it's really weird that CI is green, as afaik upstream removed some of our used queue types, so I wonder how that should work.
14:10:53 <SvenKieske> #link https://zuul.opendev.org/t/openstack/build/0037175825154a0bb76e0c6c65eab144
14:11:33 <SvenKieske> specifically https://zuul.opendev.org/t/openstack/build/0037175825154a0bb76e0c6c65eab144/log/secondary1/logs/kolla/all-ERROR.txt but it's a very very long error log so as I said I didn't deep dive into it
14:12:22 <SvenKieske> there's errors all over the place, no sql connection, keystone domain not found..various stuff, might be related to rmq, might be something else
14:12:54 <Guest9306> might be related to bootstrap. I'd compare other changes flows
14:13:07 <SvenKieske> not sure why that's reported as "success" though. afaik in k-a that should be a hard error. maybe zuul ci in kolla is different than in kolla-ansible? not sure.
14:13:28 <frickler> lot's of those errors are during the startup phase, so they are ignored
14:13:58 <Guest9306> yes, we're ignoring a lots of errors. recently we added some as well
14:14:03 <frickler> and keystone domain/user/whatever not found is normal ops
14:14:15 <Guest9306> in check-logs.sh
14:14:24 <Guest9306> tests/check-logs.sh that is
14:14:40 <Guest9306> sorry for autoparsing link
14:14:51 <SvenKieske> it's at least hard to distinguish multinode jobs from single node jobs in kolla, these seem to be differently labeled from the kolla-ansible jobs somehow
14:15:13 <SvenKieske> because I don't think we would find any issues in single node jobs.
14:15:55 <Guest9306> yes, multinodes are very helpful in that regard
14:16:20 <SvenKieske> or this: 2024-11-12 14:28:53.801 31 ERROR oslo_service.periodic_task oslo_messaging.exceptions.MessageDeliveryFailure: Unable to connect to AMQP server on 192.0.2.1:5672 after inf tries: Basic.publish: (404) NOT_FOUND - no exchange 'cinder-scheduler_fanout' in vhost '/'
14:16:30 <SvenKieske> basically last line in: https://zuul.opendev.org/t/openstack/build/19efb602a7734f06b495a0b2346bf345/log/secondary2/logs/kolla/all-ERROR.txt
14:16:43 <SvenKieske> so that doesn't look like a healthy rmq to me
14:17:34 <SvenKieske> still reported as success. I'm not sure what's filtering out "harmless" errors in our CI, but I think it's a little bit greedy
14:18:50 <SvenKieske> and in general I would be very careful about filtering out errors at all, because the required patterns often spin out of control and tend to shadow real issues.
14:19:25 <Guest9306> indeed. however some errors in bootstrap will occur anyway
14:19:28 <SvenKieske> but whatever, someone would have to dedicate some time, maybe I can make some on sunday, to look into it, no promises though
14:19:48 <Guest9306> ok. let's continue discussion in that change
14:19:54 <Guest9306> it make sense
14:19:55 <SvenKieske> sure, I understand the very good reasons to hide some errors, we can't fix all of them. we just need to be very careful :)
14:19:58 <Guest9306> to make it clearer
14:20:12 <Guest9306> #topic Additional agenda (from whiteboard)
14:20:16 <SvenKieske> I'll update the link to the latter link, that seems like a more real issue, thx
14:20:36 <Guest9306> I can see some changes to review from SvenKieske
14:21:12 <Guest9306> and backports
14:21:20 <Guest9306> that is on my list
14:21:27 <Guest9306> I'll take a look into that
14:22:00 <SvenKieske> nice, yeah most of those are backports I think :)
14:22:24 <SvenKieske> no reason to go over them individually imho. would just be happy to receive some reviews, thx.
14:23:18 <Guest9306> yeap, will take a look
14:23:22 <Guest9306> #topic Open discussion
14:24:04 <Guest9306> I guess we did some open discussion already :)
14:25:09 <Guest9306> I think we won't have meeting next week, as there are at least couple of conferences
14:25:21 <SvenKieske> okay
14:25:22 <Guest9306> I'll be in Prague on OVS/OVN con
14:25:34 <Guest9306> and some folks will be on Supercomputing in the US
14:25:53 <frickler> let's skip it then, I'd think
14:25:53 <Guest9306> I'm wondering why today's attendance is so low ;)
14:26:00 <SvenKieske> can you motivate someone at ovn con to revive the ovn-exporter project, please? ;)
14:26:09 <Guest9306> :)
14:26:16 <Guest9306> I'll try
14:26:37 <SvenKieske> upstream is dead and I don't feel particular inclined to maintain my one-off-patch fork, which isn't even finished :D
14:27:45 <Guest9306> it is a problem of one-people project. I think ovs/ovn should have and maintain that kind of stuff
14:27:46 <SvenKieske> btw, your IRC name somehow changed to "Guest9306" just if you're wondering bbezak :D
14:27:53 <Guest9306> lovely
14:28:03 <Guest9306> again :)
14:28:12 <SvenKieske> right, would be nice if ovn/ovs project could step up. no idea how many ressources they got though.
14:29:17 <SvenKieske> that looks better, but I guess we are finished? :)
14:29:36 <bbezak> I don't have anything else
14:29:57 <bbezak> let's wrap up then, thank you all!
14:29:57 <bbezak> #endmeeting