| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible stable/2023.2: Bump SHAs for EOL-ing 2023.2 https://review.opendev.org/c/openstack/openstack-ansible/+/950893 | 06:58 |
|---|---|---|
| kleini | Good morning. I feel a bit unconfident about upgrading A to C. After studying all guides, the only thing I found, was staying with RabbitMQ first on HA queues. Is there anything else to consider for a faster and smoother upgrade? I am going to switch to new defaults like quorum queues later. | 09:29 |
| noonedeadpunk | kleini: this was the only major thing afaik | 09:36 |
| noonedeadpunk | upgrade went really fine for us, to have that said | 09:36 |
| kleini | thank you very much. then I will try to upgrade staging and see, how smooth that works. | 09:39 |
| opendevreview | Ivan Anfimov proposed openstack/openstack-ansible master: docs: little update for compatibility-matrix https://review.opendev.org/c/openstack/openstack-ansible/+/950787 | 10:58 |
| opendevreview | Ivan Anfimov proposed openstack/openstack-ansible master: docs: little update for compatibility-matrix https://review.opendev.org/c/openstack/openstack-ansible/+/950787 | 11:23 |
| mgariepy | i did the 2023.1 > 2024.1 upgrade a few weeks ago. | 12:06 |
| mgariepy | I do have quite a load on my apis on this particular cloud the rabbitmq queue convertion took a bit more work for me. | 12:07 |
| mgariepy | i had trouble deleting the HA queues to re-create them in quorum mode. | 12:08 |
| noonedeadpunk | have you seen any improvements for load/stability after migration? | 12:08 |
| mgariepy | i did had much issues tho. | 12:09 |
| noonedeadpunk | yeah, you can only delete the vhost as a whole more or less | 12:09 |
| noonedeadpunk | pretty much the reason why there's re-naming in place | 12:09 |
| noonedeadpunk | huh, ok | 12:09 |
| mgariepy | i only have base services plus manila | 12:09 |
| noonedeadpunk | as for us it was a bit turbulent 30 mins when conversion was done, but it worked out like in the guide for us | 12:10 |
| noonedeadpunk | jsut scheduled rabbitmq migration separately out of business hours | 12:10 |
| mgariepy | i dont 'have buisness hours lol. it';s on 24/7 lol | 12:11 |
| mgariepy | the queue not deleting i fixed it by rebooting rabbit containers one by one. then it finally the throught. | 12:12 |
| mgariepy | i dont have rabbitmq issue because i'm with ovn on this one. i think rabbit pressure comes moslty fron neutron when there are issues. | 12:13 |
| opendevreview | Ivan Anfimov proposed openstack/openstack-ansible master: docs: little update for compatibility-matrix https://review.opendev.org/c/openstack/openstack-ansible/+/950787 | 12:37 |
| noonedeadpunk | I wonder how you had issues with queues not deleting, when process was dropping/replacing the vhost... | 12:41 |
| noonedeadpunk | yeah, we're on OVS, so rabbit is not happy there | 12:41 |
| mgariepy | not sure if it was stalled process or what | 12:41 |
| noonedeadpunk | but also on OVN we're having VPNaaS which still does use rabbitmq. Not as much as OVs though | 12:42 |
| mgariepy | i don't have vpnaas :) haha | 12:42 |
| noonedeadpunk | and business hours I meant, that usually load on API is significantly lower during off-hours, as even if we're talking about CI/CD systems, they are usually triggered by some action (unless these are periodic jobs, but then they also running in somehow wexpected timeframes) | 12:43 |
| noonedeadpunk | so customers usually can accept some API instability during announced maintenances. | 12:44 |
| mgariepy | i can also do announced maintenance, their workload is triggered when data is available so can be any time pretty much. | 12:47 |
| noonedeadpunk | ah, I see | 12:48 |
| kleini | I wanted to put whole API in maintenance mode, when switching from HA to quorum queues. Hopefully I avoid then issues. | 12:50 |
| mgariepy | i'm just not sure if the issue was caused by the load on the cluster or only gremlins hanging around because of some other reason. i didn't try to reproduce the issue also, | 12:57 |
| opendevreview | Ivan Anfimov proposed openstack/openstack-ansible master: docs: little update for compatibility-matrix https://review.opendev.org/c/openstack/openstack-ansible/+/950787 | 12:58 |
| jrosser | what did we have to fix apparmor things on recently? | 13:23 |
| jrosser | (looking to copy the approach) | 13:23 |
| noonedeadpunk | neutron? | 13:32 |
| noonedeadpunk | mean this? https://opendev.org/openstack/openstack-ansible-os_neutron/commit/600174f2168510942da25560b08580a9de7817d3 | 13:32 |
| jrosser | yeah, thats it | 13:37 |
| jrosser | th | 13:37 |
| jrosser | i can reuse some of that in ironic | 13:37 |
| opendevreview | Ivan Anfimov proposed openstack/openstack-ansible master: docs: little update for compatibility-matrix https://review.opendev.org/c/openstack/openstack-ansible/+/950787 | 13:45 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-os_ironic master: Add apparmor rules for ironic inspector https://review.opendev.org/c/openstack/openstack-ansible-os_ironic/+/951003 | 14:25 |
| jrosser | i do wonder why this apparmor stuff only broke very recently for dnsmasq | 14:27 |
| noonedeadpunk | #startmeeting openstack_ansible_meeting | 15:03 |
| opendevmeet | Meeting started Tue May 27 15:03:49 2025 UTC and is due to finish in 60 minutes. The chair is noonedeadpunk. Information about MeetBot at http://wiki.debian.org/MeetBot. | 15:03 |
| opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 15:03 |
| opendevmeet | The meeting name has been set to 'openstack_ansible_meeting' | 15:03 |
| noonedeadpunk | #topic rollcall | 15:03 |
| jrosser | o/ hello | 15:03 |
| noonedeadpunk | o/ | 15:04 |
| noonedeadpunk | sorry for being late | 15:04 |
| NeilHanlon | o/ | 15:06 |
| noonedeadpunk | #topic office hours | 15:07 |
| DavidGomez | o/ | 15:08 |
| noonedeadpunk | I think the main topic - what do we want/need to merge to Epoxy berfore doing RC2 and a 31.0.0 tag? | 15:08 |
| noonedeadpunk | As I'd love to propose a release by end of this week, not to face any unexpected delays | 15:09 |
| jrosser | it feels tlike things are reasonably stable? | 15:09 |
| noonedeadpunk | yes, I think they are | 15:10 |
| jrosser | though this cycle it is very unlikley that I will be able to get any lab testing of Epoxy done in the short term | 15:10 |
| noonedeadpunk | I think we will be doing this during July/August at very least | 15:11 |
| noonedeadpunk | Caracal -> Epoxy one | 15:11 |
| jrosser | this is probably our worst upgrade ever as it will be opesntack + ubuntu + lb->ovn | 15:11 |
| noonedeadpunk | Upgrading OVN with Ubuntu might be problematic | 15:12 |
| jrosser | oh yes, that not necessarily all in one shot | 15:12 |
| noonedeadpunk | there is a bug report I never looked into much | 15:12 |
| noonedeadpunk | https://bugs.launchpad.net/openstack-ansible/+bug/2059721 | 15:12 |
| noonedeadpunk | it's from 2023.1, but we have not changed much in logic around OVN clustering setup | 15:13 |
| noonedeadpunk | and it totally looks like a race condition/compatability issue between major OVN versions | 15:13 |
| noonedeadpunk | so that can be really a problem | 15:14 |
| opendevreview | Merged openstack/openstack-ansible master: docs: fix small mistake with MariaDB and RabbitMQ https://review.opendev.org/c/openstack/openstack-ansible/+/949733 | 15:14 |
| noonedeadpunk | I had one patch which was a bug fix, but can break the behaviour: | 15:14 |
| noonedeadpunk | #link https://review.opendev.org/c/openstack/openstack-ansible-os_trove/+/950491 | 15:14 |
| noonedeadpunk | which potentially might be a good fit for the Epoxy | 15:15 |
| noonedeadpunk | Other important topic EL10 | 15:21 |
| noonedeadpunk | I had not time last week to start working on it :( | 15:21 |
| noonedeadpunk | But apparently, I need to start it sooner then later | 15:22 |
| noonedeadpunk | not sure if there was any related progress on infra side of things though | 15:22 |
| opendevreview | Merged openstack/openstack-ansible master: 2025.1 (Epoxy) Release Candidate - fix for year https://review.opendev.org/c/openstack/openstack-ansible/+/950739 | 15:27 |
| noonedeadpunk | Actually, about docs | 15:28 |
| noonedeadpunk | I am thinking, if we should give up maintaining whole history of releases for our `latest` release | 15:28 |
| noonedeadpunk | #link https://docs.openstack.org/openstack-ansible/latest/ | 15:29 |
| noonedeadpunk | and just leave "current" one or none at all, just doc tree | 15:30 |
| noonedeadpunk | as current one not only need maintenance, but also needs translators to update each cycle | 15:32 |
| jrosser | do we end up having to go and fix that page for all the branches as the status of things changes? | 15:32 |
| noonedeadpunk | yup | 15:32 |
| noonedeadpunk | or well | 15:32 |
| noonedeadpunk | for all branches we have a different version of it | 15:33 |
| noonedeadpunk | ie https://docs.openstack.org/openstack-ansible/2024.2/ | 15:33 |
| jrosser | yes i was just looking and its a little random | 15:33 |
| noonedeadpunk | it really is | 15:33 |
| noonedeadpunk | and 2024.2 is an outstanding example | 15:34 |
| noonedeadpunk | and 2025.1 needs to be patched.... | 15:34 |
| noonedeadpunk | But I'd guess I'd align latest to the state of https://docs.openstack.org/openstack-ansible/2024.1/ | 15:36 |
| noonedeadpunk | or smth like that | 15:36 |
| noonedeadpunk | Also, 2023.2 was EOL-ed for projects | 15:42 |
| noonedeadpunk | so EOL-ing of roles were proposed | 15:42 |
| noonedeadpunk | #link https://review.opendev.org/c/openstack/releases/+/948217 | 15:42 |
| noonedeadpunk | With that I've also proposed to drop upgrade jobs for 2024.1 | 15:43 |
| noonedeadpunk | as eventually unmaintained branch would not exist | 15:43 |
| jrosser | yes | 15:43 |
| jrosser | i hope that soon some of the fixes we have done in recent branches will be trickling down to the new unmaintained ones | 15:43 |
| jrosser | it might now finally be possible to have some CI work on those without terrible difficulty | 15:44 |
| noonedeadpunk | Does this include upgrade jobs? | 15:45 |
| noonedeadpunk | As while there were improvements for respecting unmaintained branch, non-SLURP go straight to EOL | 15:46 |
| jrosser | i did do some work on making it better at picking the N-1 branch name | 15:46 |
| noonedeadpunk | and I can't recall we did anything to cover that | 15:46 |
| jrosser | and i think if the branch is deleted it should pick a previous one | 15:46 |
| noonedeadpunk | hu | 15:47 |
| noonedeadpunk | but then we still would need to be dropping upgrade from previous SLURP | 15:47 |
| noonedeadpunk | as we'd be testing same thing twice? | 15:47 |
| jrosser | https://github.com/openstack/openstack-ansible/blob/master/scripts/gate-check-commit.sh#L69 | 15:47 |
| noonedeadpunk | yeah, but this selects `stable` vs `unmaintained`? | 15:48 |
| jrosser | right yes so that automatically sorts out "stable/" vs "unmaintained", we'd still need to fix line 68 | 15:48 |
| noonedeadpunk | but now we kinda want to have $UPGRADE_SOURCE_RELEASE-eol ? | 15:48 |
| noonedeadpunk | if neither nor exist? | 15:48 |
| jrosser | hmm yes you are right | 15:49 |
| noonedeadpunk | which is a bit /o\ but yeah... | 15:49 |
| jrosser | if we can feed in UPGRADE_SOURCE_BRANCH from outside then all that logic is bypassed | 15:50 |
| jrosser | which could turn out to be the simplest | 15:50 |
| noonedeadpunk | True, but then we'd still need to do update of zuul jobs | 15:53 |
| noonedeadpunk | which we probably have to do anyway | 15:54 |
| jrosser | hrrm it is messy | 15:56 |
| jrosser | as we still want to run these locally | 15:56 |
| noonedeadpunk | true... | 15:56 |
| jrosser | so if the branch prefix is empty, the branch does not exist | 15:57 |
| jrosser | then we can assume it is <release>-eol ? | 15:58 |
| noonedeadpunk | Yeah, I think we can... | 15:59 |
| noonedeadpunk | that sounds like reasonable assumption... | 16:00 |
| noonedeadpunk | #endmeeting | 16:00 |
| opendevmeet | Meeting ended Tue May 27 16:00:58 2025 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 16:00 |
| opendevmeet | Minutes: https://meetings.opendev.org/meetings/openstack_ansible_meeting/2025/openstack_ansible_meeting.2025-05-27-15.03.html | 16:00 |
| opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/openstack_ansible_meeting/2025/openstack_ansible_meeting.2025-05-27-15.03.txt | 16:00 |
| opendevmeet | Log: https://meetings.opendev.org/meetings/openstack_ansible_meeting/2025/openstack_ansible_meeting.2025-05-27-15.03.log.html | 16:00 |
| opendevreview | Merged openstack/openstack-ansible-os_zun master: Remove centos-9 and jammy CI jobs https://review.opendev.org/c/openstack/openstack-ansible-os_zun/+/950696 | 16:39 |
| opendevreview | Merged openstack/openstack-ansible master: Update ansible to 2.18.6 https://review.opendev.org/c/openstack/openstack-ansible/+/938091 | 17:14 |
| opendevreview | Merged openstack/openstack-ansible master: Bump sha for barbican service https://review.opendev.org/c/openstack/openstack-ansible/+/950706 | 17:21 |
| opendevreview | Merged openstack/openstack-ansible master: Bump SHA for magnum service to pick up wsgi module changes https://review.opendev.org/c/openstack/openstack-ansible/+/950705 | 17:21 |
| opendevreview | Ivan Anfimov proposed openstack/openstack-ansible-rabbitmq_server master: wip https://review.opendev.org/c/openstack/openstack-ansible-rabbitmq_server/+/951039 | 21:13 |
| opendevreview | Ivan Anfimov proposed openstack/openstack-ansible-rabbitmq_server master: wip https://review.opendev.org/c/openstack/openstack-ansible-rabbitmq_server/+/951039 | 21:13 |
Generated by irclog2html.py 4.0.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!