Wednesday, 2025-11-05

hamidlotfi__Hi everyone,06:15
hamidlotfi__I’m hitting a weird issue after a kolla-ansible upgrade and would appreciate any pointers.06:15
hamidlotfi__Environment06:15
hamidlotfi__kolla-ansible 18.8.106:15
hamidlotfi__rabbitmq 4.1.3 (deployed by Kolla)06:15
hamidlotfi__upgrade path: 2024.1 → 2025.106:15
hamidlotfi__What happened06:15
hamidlotfi__Ran kolla-ansible -i … upgrade from 2024.1 to 2025.1.06:15
hamidlotfi__The upgrade finished without obvious errors in kolla logs.06:15
hamidlotfi__Right after the upgrade every neutron-l3-agent container on all control/compute nodes is marked unhealthy (no ERROR/FATAL lines in the container logs).06:15
hamidlotfi__All tenant instances lose external connectivity.06:15
hamidlotfi__Simply docker restart neutron_l3_agent on each node is enough: containers become healthy and external access is restored instantly.06:15
hamidlotfi__No other OpenStack service shows the same behaviour.06:15
hamidlotfi__Any idea why the agents start in “unhealthy” state until manually restarted?06:16
hamidlotfi__Happy to collect more data if you tell me what to look for.06:16
opendevreviewPierre Riteau proposed openstack/kayobe master: [DNM] Test removal of python3.12-specific code  https://review.opendev.org/c/openstack/kayobe/+/96595307:56
mnasiadkahamidlotfi__: the health checks are checking IIRC if neutron agent is connected to RMQ - maybe there was some ,,glitch’’ due to change from HA queues to quorum queues08:13
opendevreviewPierre Riteau proposed openstack/kayobe master: [DNM] Test kayobe-overcloud-host-configure-centos10s  https://review.opendev.org/c/openstack/kayobe/+/96614508:24
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible master: keystone: Add support for deploying httpd for federation  https://review.opendev.org/c/openstack/kolla-ansible/+/96208308:45
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible master: keystone: Add support for deploying httpd for federation  https://review.opendev.org/c/openstack/kolla-ansible/+/96208308:59
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible master: keystone: Add support for deploying httpd for federation  https://review.opendev.org/c/openstack/kolla-ansible/+/96208308:59
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible master: keystone: Add support for deploying httpd for federation  https://review.opendev.org/c/openstack/kolla-ansible/+/96208309:25
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible master: CI: Drop test-core-openstack.sh  https://review.opendev.org/c/openstack/kolla-ansible/+/96476309:34
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible master: CI: Remove Rocky9 Python handling  https://review.opendev.org/c/openstack/kolla-ansible/+/96476409:35
hamidlotfi__mnasiadka: Thanks for the hint!09:51
hamidlotfi__Just to clarify: before I kicked off the 2024.1 → 2025.1 upgrade I had already:09:51
hamidlotfi__switched every queue to quorum09:51
hamidlotfi__upgraded RabbitMQ itself to 4.1.309:51
hamidlotfi__by setting in globals.yml:09:51
hamidlotfi__https://www.irccloud.com/pastebin/49QcA4sT/09:51
hamidlotfi__So at upgrade-time the deployment was already 100 % quorum queues; no HA → quorum migration happened during the upgrade.09:52
hamidlotfi__Anything else I should check that could make the l3-agent health-check fail right after the container starts?09:52
fricklerdebug logs might be helpful, but it doesn't sound like you could reproduce the issue once more now, or do you?10:14
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible master: keystone: Add support for deploying httpd for federation  https://review.opendev.org/c/openstack/kolla-ansible/+/96208310:30
opendevreviewPierre Riteau proposed openstack/kayobe master: [DNM] Test kayobe-overcloud-host-configure-centos10s  https://review.opendev.org/c/openstack/kayobe/+/96614510:35
opendevreviewBartosz Bezak proposed openstack/kolla-ansible master: Replace Redis with Valkey  https://review.opendev.org/c/openstack/kolla-ansible/+/95953910:38
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible master: keystone: Add support for deploying httpd for federation  https://review.opendev.org/c/openstack/kolla-ansible/+/96208310:53
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible master: keystone: Add support for deploying httpd for federation  https://review.opendev.org/c/openstack/kolla-ansible/+/96208310:57
opendevreviewPierre Riteau proposed openstack/kayobe master: CI: Fix CentOS Stream 10 host configure job  https://review.opendev.org/c/openstack/kayobe/+/96614511:08
opendevreviewMerged openstack/kolla-ansible master: Revert "ovn: Mark as non-voting due to db related failures"  https://review.opendev.org/c/openstack/kolla-ansible/+/96517511:33
opendevreviewVerification of a change to openstack/kolla-ansible master failed: Remove distro_python_version variable  https://review.opendev.org/c/openstack/kolla-ansible/+/94603911:33
opendevreviewMerged openstack/kolla-ansible master: CI: Drop test-core-openstack.sh  https://review.opendev.org/c/openstack/kolla-ansible/+/96476311:34
opendevreviewBartosz Bezak proposed openstack/kolla-ansible master: Replace Redis with Valkey  https://review.opendev.org/c/openstack/kolla-ansible/+/95953911:43
opendevreviewAmir Hossein Ahmadi proposed openstack/kolla-ansible master: swift: Reintroduce the Swift deployment role  https://review.opendev.org/c/openstack/kolla-ansible/+/95357612:34
opendevreviewBartosz Bezak proposed openstack/kolla-ansible master: Replace Redis with Valkey  https://review.opendev.org/c/openstack/kolla-ansible/+/95953912:41
opendevreviewBartosz Bezak proposed openstack/kolla master: debian: use valkey from bookworm-backports  https://review.opendev.org/c/openstack/kolla/+/96590712:43
opendevreviewBartosz Bezak proposed openstack/kolla master: debian: use valkey from bookworm-backports  https://review.opendev.org/c/openstack/kolla/+/96590712:46
opendevreviewBartosz Bezak proposed openstack/kolla master: debian: use valkey from bookworm-backports  https://review.opendev.org/c/openstack/kolla/+/96590712:47
opendevreviewBartosz Bezak proposed openstack/kolla master: debian: use valkey from bookworm-backports  https://review.opendev.org/c/openstack/kolla/+/96590712:48
opendevreviewVerification of a change to openstack/kolla-ansible master failed: Remove distro_python_version variable  https://review.opendev.org/c/openstack/kolla-ansible/+/94603913:04
mnasiadkaWe forgot to move the meeting due to DST13:09
mmalchukDST again)13:09
mnasiadkaBut since no one is complaining - we’ll meet in 51 minutes13:09
fricklerah, good catch, I briefly thought about this last week, but then things happened ;) let me do a revert of the revert of the revert to formalize this13:11
mnasiadkaThanks frickler13:12
mmalchukhttps://www.timeanddate.com/time/change/timezone/utc13:12
mmalchukDaylight Saving Time (DST) Not Observed in Year 2025 ;)13:12
mmalchukor we move to 14:00 UTC ?13:13
frickleroh, yes, let's all move to Utopia, problem solved ;)13:13
fricklerhttps://review.opendev.org/c/opendev/irc-meetings/+/96617213:14
mmalchukback to 14:00 UTC :)13:14
mmalchukdon't forget to change TOPIC here13:14
opendevreviewBartosz Bezak proposed openstack/kolla master: debian: use valkey from bookworm-backports  https://review.opendev.org/c/openstack/kolla/+/96590713:22
*** ChanServ changes topic to "IRC meetings on Wednesdays @ 14:00 UTC - https://docs.openstack.org/kolla/latest/contributor/meeting.html | Whiteboard: https://bit.ly/2MM7mWF | IRC channel is *LOGGED* @ http://goo.gl/3mzZ7b"13:24
fricklerone day we should change those tracking links to something nicer, too ... is what I think every 6 months ;-D13:25
mnasiadkaWell, at least we should decide on the url shortener service :)13:26
mmalchukwe already use bit.ly - why don't use it again?)13:29
mmalchukor goo.gl again)13:30
mmalchukor there is need a new/another url shortender? ;)13:31
mmalchukTinyURL ?13:32
fricklerhaving one that is GDPR compliant would be nice. likely need to build one ourselves13:32
mmalchukfrickler and deploy it in the opendev.org same as paste.opendev.org ?13:33
frickleryes, something like l.opendev.org running a small httpd just with static redirects served from a gerrit maintained .htaccess file is my rough idea13:36
jaydghttps://github.com/738/awesome-url-shortener has a list of self-hosted open source services13:36
mmalchukjaydg thanks for the list) did you know/tried the best small/simple self hosted solution from the list?13:38
jaydgmmalchuk: no sorry never tried any of those. I think I've read about pygmy, though.13:39
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible master: keystone: Add support for deploying httpd for federation  https://review.opendev.org/c/openstack/kolla-ansible/+/96208313:43
mnasiadkafrickler: https://urlr.me/en/ ;-)13:44
mnasiadkamnasiadka bbezak frickler kevko mmalchuk gkoper jovial mattcrees dougszu darmach pabloclsn ravlew amir58118 r-krcek - meeting in 11 minutes13:49
fricklermnasiadka: that still does tracking afaict. also needs javascript (ok, the latter is true for gerrit, too, although not when using gertty). maybe I need to define my requirements more exactly first :-/13:50
opendevreviewBartosz Bezak proposed openstack/kolla-ansible master: Replace Redis with Valkey  https://review.opendev.org/c/openstack/kolla-ansible/+/95953913:56
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible master: keystone: Add support for deploying httpd for federation  https://review.opendev.org/c/openstack/kolla-ansible/+/96208313:57
mnasiadka#startmeeting kolla14:00
opendevmeetMeeting started Wed Nov  5 14:00:43 2025 UTC and is due to finish in 60 minutes.  The chair is mnasiadka. Information about MeetBot at http://wiki.debian.org/MeetBot.14:00
opendevmeetUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.14:00
opendevmeetThe meeting name has been set to 'kolla'14:00
mnasiadka#topic rollcall14:00
mnasiadkao/14:00
amir58118-2o/14:00
bbezako/14:01
mmalchuk\o14:02
dougszu{o14:02
mnasiadka#topic agenda14:02
mnasiadka* CI status14:02
mnasiadka* Release tasks14:02
mnasiadka* Regular stable releases (first meeting in a month)14:02
mnasiadka* Current cycle planning14:02
mnasiadka* Additional agenda (from whiteboard)14:02
mnasiadka* Open discussion14:02
mnasiadka#topic CI status14:02
mnasiadkaSeems fine, unless anybody noticed anything - this is the time to post your observations14:03
bbezak(Lgtm)14:04
amir58118-2some warnings in test-init-tempest-discover14:04
amir58118-2#link https://zuul.opendev.org/t/openstack/build/b0c0cbf2db7c485da57c40fdb925b98a/log/primary/logs/ansible/test-init-tempest-discover14:04
mnasiadkaI’d say these are fine14:05
mnasiadka#topic Release tasks14:05
mnasiadkaSo, we’ve been officially warned - that it’s time to branch14:06
bbezakFinishing valkey as we speak14:06
mnasiadkaWe need to merge to things - Valkey and Keystone HTTPD server for federation (OIDC)14:06
mnasiadka#link https://review.opendev.org/c/openstack/kolla-ansible/+/95953914:06
mnasiadkaThe keystone one is still some WIP, but should be done before end of the week14:07
mnasiadka#link https://review.opendev.org/c/openstack/kolla-ansible/+/96208314:07
dougszuack - will try and help with reviews 14:08
mmalchukme too14:08
mnasiadkaOk then, next one14:09
mnasiadka#topic Regular stable releases (first meeting in a month)14:09
bbezakWill raise patch14:09
bbezakI think we missed last month14:09
mnasiadkaYeah, that could be last 2024.1 release14:10
mnasiadkaBefore we do EOM14:10
mnasiadkaI’ll skip current cycle planning, giving back cycles to review the Valkey patch :-)14:11
mnasiadkaNothing on the additional agenda14:11
mnasiadka#topic Open discussion14:11
mnasiadkaAnybody anything?14:11
amir58118-2I've developed Kolla and Kolla-Ansible patches to restore Swift support (topic: dev-swift-modernization). All Zuul tests passed successfully on Ubuntu, Debian, and Rocky. (Patchset #59)14:11
amir58118-2The patch is quite large, and review may take time. Given that HA for Swift should remain independent from OpenStack-wide HA (as discussed with core team, including mnasiadka), I propose:14:11
amir58118-2Merge the current patch as-is to bring back Swift support.14:12
amir58118-2Continue HA-related work in a separate follow-up patch.14:12
mnasiadkaSurely not this cycle.14:12
amir58118-2If the team prefers to keep everything in one patch, I'm happy to proceed that way too.14:12
amir58118-2mnasiadka: Of course14:13
mnasiadkaIt’s just too late14:13
mnasiadkaAnd since you have another full cycle - it’s a perfect opportunity to make things better.14:13
amir58118-2mnasiadka: I know we won’t hit this cycle. 14:14
mnasiadkaWell, for something to be merged - you need to core reviewers to be interested in it14:15
mnasiadkaI can be the second one - but you need to find another volunteer.14:15
amir58118-2after 6 months of developing? ...14:16
mnasiadkaEspecially - as you’ve mentioned, the patch is huge.14:16
mnasiadkaWe never promised it’s going to be merged, I remember kevko volunteered to review your patch - but he’s not around recently.14:16
mnasiadkaAnd I doubt you’re working only on this for the last 6 months14:17
amir58118-2I'll do my best. Many have shown interest in this patch.14:17
yuvalI can say in my env the updating of kolla and progress to ubuntu noble is not easy. the way everything is need to be install in virtual env raised up some issues14:18
mnasiadkayuval: well, that’s not us to be blamed for ;-)14:18
yuvalofcourse - but I saw issues in dev_mode where we try to do stuff but failed due to that issue14:19
yuvalcurrently fixed it locally for my branch14:19
bbezakhttps://review.opendev.org/c/openstack/releases/+/96618414:20
mnasiadkaRight, I didn’t have time to have a look in dev_mode - I know it’s not really working.14:20
mnasiadkaOk then - I think we’re out of topics.14:21
yuvalalso14:21
yuvalenable_fluentd: no - fails14:21
yuvalyou must enable it\14:21
mnasiadkaWell, I’d say majority of people don’t disable that - probably that’s a bug14:22
mnasiadkaBut we’re most probably not going to test that corner case14:22
mnasiadkaSo it might be broken14:22
yuvalno problem14:22
yuvalalso14:22
yuvalI see issue with enbale_proxysql stealing mariadb port - so I must disable it14:23
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible master: kolla-logs: break out from common role  https://review.opendev.org/c/openstack/kolla-ansible/+/96330314:23
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible master: common: Rename role to kolla-toolbox  https://review.opendev.org/c/openstack/kolla-ansible/+/96318814:23
BertrandLanson[m]we disabled fluentd ? it works without it ? 14:23
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible master: common: Rename role to kolla-toolbox  https://review.opendev.org/c/openstack/kolla-ansible/+/96318814:24
yuvalprevious version worked without fluentd14:24
mnasiadkaWell, raise bugs for these problems in launchpad and feel free to raise patches to fix them :-)14:25
yuvalyes yes I know14:25
yuvalnot complaining just raising awarness 14:25
mnasiadkasure14:25
mnasiadkaOk then - let’s finish for today14:26
mnasiadkaThank you all for coming14:26
mnasiadkaSee you next week14:26
yuvalalso14:26
mnasiadka#endmeeting14:26
opendevmeetMeeting ended Wed Nov  5 14:26:12 2025 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)14:26
opendevmeetMinutes:        https://meetings.opendev.org/meetings/kolla/2025/kolla.2025-11-05-14.00.html14:26
opendevmeetMinutes (text): https://meetings.opendev.org/meetings/kolla/2025/kolla.2025-11-05-14.00.txt14:26
opendevmeetLog:            https://meetings.opendev.org/meetings/kolla/2025/kolla.2025-11-05-14.00.log.html14:26
amir58118-2mnasiadka: Thanks anyway.14:26
yuvaljust kidding14:26
mmalchukmnasiadka thanks14:28
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible master: keystone: Add support for deploying httpd for federation  https://review.opendev.org/c/openstack/kolla-ansible/+/96208314:35
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible master: CI: Move playbooks from test/ to test/playbooks/  https://review.opendev.org/c/openstack/kolla-ansible/+/96476514:36
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible master: CI: Move playbooks from test/ to test/playbooks/  https://review.opendev.org/c/openstack/kolla-ansible/+/96476514:37
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible master: CI: Rework check-config.sh to Ansible role  https://review.opendev.org/c/openstack/kolla-ansible/+/96476714:38
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible master: kolla-logs: break out from common role  https://review.opendev.org/c/openstack/kolla-ansible/+/96330315:00
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible master: common: Rename role to kolla-toolbox  https://review.opendev.org/c/openstack/kolla-ansible/+/96318815:05
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible master: common: Rename role to kolla-toolbox  https://review.opendev.org/c/openstack/kolla-ansible/+/96318815:05
opendevreviewMichal Nasiadka proposed openstack/kolla master: WIP: Switch to Debian Trixie  https://review.opendev.org/c/openstack/kolla/+/96479315:07
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible master: Replace Redis with Valkey  https://review.opendev.org/c/openstack/kolla-ansible/+/95953915:34
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible master: Replace Redis with Valkey  https://review.opendev.org/c/openstack/kolla-ansible/+/95953915:35
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible master: Replace Redis with Valkey  https://review.opendev.org/c/openstack/kolla-ansible/+/95953915:36
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible master: Replace Redis with Valkey  https://review.opendev.org/c/openstack/kolla-ansible/+/95953915:36
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible master: Replace Redis with Valkey  https://review.opendev.org/c/openstack/kolla-ansible/+/95953915:36
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible master: Replace Redis with Valkey  https://review.opendev.org/c/openstack/kolla-ansible/+/95953915:38
opendevreviewBartosz Bezak proposed openstack/kolla-ansible master: Replace Redis with Valkey  https://review.opendev.org/c/openstack/kolla-ansible/+/95953915:46
opendevreviewMerged openstack/kolla master: debian: use valkey from bookworm-backports  https://review.opendev.org/c/openstack/kolla/+/96590715:48
opendevreviewBartosz Bezak proposed openstack/kolla-ansible master: Replace Redis with Valkey  https://review.opendev.org/c/openstack/kolla-ansible/+/95953915:52
opendevreviewBartosz Bezak proposed openstack/kolla-ansible master: Replace Redis with Valkey  https://review.opendev.org/c/openstack/kolla-ansible/+/95953915:57
opendevreviewMichal Nasiadka proposed openstack/kolla master: WIP: Switch to Debian Trixie  https://review.opendev.org/c/openstack/kolla/+/96479316:04
opendevreviewBartosz Bezak proposed openstack/kolla-ansible master: Replace Redis with Valkey  https://review.opendev.org/c/openstack/kolla-ansible/+/95953916:06
opendevreviewBartosz Bezak proposed openstack/kolla-ansible master: Replace Redis with Valkey  https://review.opendev.org/c/openstack/kolla-ansible/+/95953916:12
opendevreviewBartosz Bezak proposed openstack/kolla-ansible master: Replace Redis with Valkey  https://review.opendev.org/c/openstack/kolla-ansible/+/95953916:27
opendevreviewBartosz Bezak proposed openstack/kolla-ansible master: Replace Redis with Valkey  https://review.opendev.org/c/openstack/kolla-ansible/+/95953916:32
opendevreviewBartosz Bezak proposed openstack/kolla-ansible master: Replace Redis with Valkey  https://review.opendev.org/c/openstack/kolla-ansible/+/95953916:39
opendevreviewBartosz Bezak proposed openstack/kolla-ansible master: Replace Redis with Valkey  https://review.opendev.org/c/openstack/kolla-ansible/+/95953916:51
opendevreviewBartosz Bezak proposed openstack/kolla-ansible master: Replace Redis with Valkey  https://review.opendev.org/c/openstack/kolla-ansible/+/95953917:01
opendevreviewMerged openstack/kolla-ansible master: CI: Remove Rocky9 Python handling  https://review.opendev.org/c/openstack/kolla-ansible/+/96476417:45
opendevreviewBartosz Bezak proposed openstack/kolla-ansible master: Replace Redis with Valkey  https://review.opendev.org/c/openstack/kolla-ansible/+/95953919:04
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible master: keystone: Add support for deploying httpd for federation  https://review.opendev.org/c/openstack/kolla-ansible/+/96208319:23
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible master: keystone: Add support for deploying httpd for federation  https://review.opendev.org/c/openstack/kolla-ansible/+/96208319:33

Generated by irclog2html.py 4.0.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!