Wednesday, 2024-09-25

opendevreviewMerged openstack/ansible-role-systemd_networkd master: Don't make VLAN/VXLAN/MACVLAN mutually exclusive  https://review.opendev.org/c/openstack/ansible-role-systemd_networkd/+/92934600:58
opendevreviewDmitriy Rabotyagov proposed openstack/ansible-role-systemd_networkd stable/2024.1: Don't make VLAN/VXLAN/MACVLAN mutually exclusive  https://review.opendev.org/c/openstack/ansible-role-systemd_networkd/+/93041007:33
jrossero/ good morning07:40
noonedeadpunko/07:47
opendevreviewMerged openstack/openstack-ansible master: Add global release note for Apache MPM alignment  https://review.opendev.org/c/openstack/openstack-ansible/+/93018507:56
noonedeadpunkwe also need that for skyline to land on master : https://review.opendev.org/c/openstack/openstack-ansible/+/92989208:00
noonedeadpunk*skyline mpm fix08:00
jrosserso for zuul stable branches08:08
jrosseri think we need to do an experiment and find out what actually happens today with the depends on stuff (vs. what might have happened in the past)08:08
opendevreviewJonathan Rosser proposed openstack/openstack-ansible master: DNM - test git checkout of zuul repos  https://review.opendev.org/c/openstack/openstack-ansible/+/93041308:25
noonedeadpunkI think we'd need to have a hold node for such experiment08:25
jrosser^ that should show the expected sha of master and 2024.108:26
noonedeadpunkand also I'm not sure what specifically you're checking with ^08:26
jrosserthen we can try adding a depends on, and see what happens08:26
jrosseri think that we are trying to understand the state of the prepared git repos when depends-on is in use08:26
noonedeadpunkAh, I need to somehow explain better where I see the current "issue"...08:26
noonedeadpunkwell, we kinda need to have a way to return "back" to the prepared state08:27
jrossermy understanding is that the branch positions are are modified to be the prepared state08:28
noonedeadpunkand even not for integrated repo, but all rest repos08:28
jrosserto allow that to happen08:28
noonedeadpunkyep08:28
jrosserok08:28
noonedeadpunkbut you can checkout from them anywhere easily08:28
noonedeadpunkthe question I have - how to checkout back to prepared state which originally was08:28
noonedeadpunkfor integrated repo specifically we do record it's state and can return back with that: https://opendev.org/openstack/openstack-ansible/src/branch/master/scripts/gate-check-commit.sh#L7308:30
jrossermaybe i misunderstand, but i thought that checking out the branch would do that by magic08:30
noonedeadpunkand then we return back to it quite easily08:30
noonedeadpunkoh well, you mean create a new brach out of current HEAD...08:31
noonedeadpunkyeah, that will work as well08:31
jrossermore that the branch positions are adjusted by zuul as part of preparing the repos08:32
jrosserto account for depends-on08:32
noonedeadpunkyep, they are indeed08:32
jrosserok so definatly i am missing what is wrong :)08:32
noonedeadpunkso we'd need to make automation to create new branches for all existing repos, then to checkout them all to UPGRADE_SOURCE_BRANCH, and return back for upgrade08:33
noonedeadpunk(all prepared by zuul repos)08:33
noonedeadpunkHighly likely that it's just me who thinks that's very complicated to do08:35
jrosserhmmm08:37
* jrosser confused08:37
jrosseris this all to do with how we symlink the role dirs?08:40
opendevreviewMerged openstack/openstack-ansible-repo_server master: Ensure that selected Apache MPM is enforced  https://review.opendev.org/c/openstack/openstack-ansible-repo_server/+/92969008:51
jrosserimho we should reduce the complexity, almost certainly at the expense of some supposed optimisiation08:55
opendevreviewMerged openstack/openstack-ansible-os_neutron stable/2024.1: Disable uWSGI usage by default  https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/92954908:55
jrosserhaving two different situations to deal with (upgrade source and target branch) only makes things much more difficult to understand08:56
jrosserif i now understand properly it is that for !master branch, a-r-r and a-c-r is not matching the prepared branches in the zuul repos?08:57
noonedeadpunkso IIRC for N-1 we jsut fetch everything from gitea08:58
noonedeadpunkand then on N we do use prepared repos08:58
jrosserperhaps a different way to look at this is we should strip all of the "in zuul" handling out of get-ansible-role-requirements.yml08:58
noonedeadpunkoh, that's a good one08:59
noonedeadpunkas in fact we should08:59
jrosserand there is a separate thing, which determines if the zuul repo branch != the a-r-r / a-c-r one, and drops the required user-role-requriements to make that work properly08:59
noonedeadpunkpotentially, by placing user-role-requirements08:59
noonedeadpunkyeah08:59
jrosserthat would be pretty simple, and would work for all branches 08:59
jrosserand this business of symlink / delete dirs / but only sometimes is just added complexity and confusion09:00
noonedeadpunkand code there is not readable at all either09:00
noonedeadpunkthis in any way doesn't solve our u-c thing on upgrade though :D09:01
jrosserso i would be in favour of now leveraging the user-role-requirements / user-collection-requriements (primarily in N-1 branches) to ensure we use the correct zuul branches and not the specific sha09:01
jrosserwell thats another issue09:02
jrosserand i think seperate?09:02
noonedeadpunktotally separate09:02
noonedeadpunkalso - I'm not 100% sure about anu stable branch - as collections at least were improved couple of times09:03
noonedeadpunkie - to allow avoid some role clones at all09:03
noonedeadpunkbut I think we should be leveraging them only on master as well?09:03
noonedeadpunkas actually all the complexity in get-ansible-role-requirements is related to master?09:04
jrosserit is09:04
jrosserwell we can handle that too - dropping a symlink and setting up user-role-requirements to not clone at all? I think it can do that09:05
jrosserand you would do that if the checked out sha/branch in the zuul repo was the one you wanted09:06
noonedeadpunkoh, yeah. true09:12
jrossermaybe i miss something but is it as simple as rewriting the a-r-r file to user-collection-requirements, fixing the url to be something like file:// and updating the version to be $branchname09:12
jrosserthat would be brute force but would work09:12
noonedeadpunkand jsut be part of gate-check-commit09:13
noonedeadpunkyeah09:13
jrosserhuh09:13
jrossermaybe that makes a ton of stuff easier to understand09:13
noonedeadpunkor? I thought that we never want to have trhe Zuul logic outside of CI09:13
noonedeadpunkso basically this whole block can go away https://opendev.org/openstack/openstack-ansible/src/branch/master/scripts/get-ansible-role-requirements.yml#L54-L10209:14
jrosserthat would be ideal, but perhaps difficult as both phases of the upgrade are run in the one script09:14
opendevreviewMerged openstack/openstack-ansible-os_neutron stable/2023.2: Do not kill ipsec on L3 cleanup  https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/93018609:14
jrosserso you need to be able to prepare at least the target branch user-role-requirements after the source branch run is done09:15
* noonedeadpunk also have a feeling now that we're talking about different issues right now09:15
jrosseror at minimum, copy them into place09:15
jrosserperhaps indeed :)09:15
jrosseri thought you were suggesting not having "in zuul" handling in gate-check-commit either09:15
jrosserso perhaps doing all this stuff in the pre- playbooks?09:16
noonedeadpunkI think it' fine to have that in gate-check-commit09:17
noonedeadpunkor well, pre-playbook is also a very good place09:17
noonedeadpunkthat is good idea09:17
opendevreviewMerged openstack/openstack-ansible-ceph_client master: Manage apt repositores and keys using deb822_repository module  https://review.opendev.org/c/openstack/openstack-ansible-ceph_client/+/92962209:17
noonedeadpunkbut then it indeed can be hard to mange pre-upgrade post-upgrade versions09:17
jrossermaybe some config files are prepared in the -pre playbooks and then moved into place if they exist in gate-check-commit09:18
noonedeadpunkif going baby steps, first patch would be just to clean-up `get-ansible-role-requirements` from zuul-related mess into a separate thing09:18
noonedeadpunkin favor of generation of u-r-r09:19
jrosseryes09:19
noonedeadpunkand then this new thing can evolve to better handle upgrade cases?09:19
jrossersounds good09:19
noonedeadpunkwell, the thing is that it's gate-check-commit then who should manage checkouts of repos09:20
noonedeadpunkor trigger smth which will do it09:21
opendevreviewMerged openstack/openstack-ansible-openstack_hosts master: Vendor osbpo gpg key into the role  https://review.opendev.org/c/openstack/openstack-ansible-openstack_hosts/+/93028009:21
opendevreviewMerged openstack/openstack-ansible-openstack_hosts master: Ensure python libselinux bindings for containers  https://review.opendev.org/c/openstack/openstack-ansible-openstack_hosts/+/92954009:21
noonedeadpunkas checkout back should be done after install of N-1 is done but before N is bootstrapped. And this all happens in gate-check-commit09:22
noonedeadpunkor well, could be that u-c is a very unique case we have09:22
noonedeadpunkso cases we have, which could act differently and have slightly different behaviour: roles, collections, u-c and services09:26
opendevreviewDmitriy Rabotyagov proposed openstack/openstack-ansible-openstack_hosts stable/2024.1: Ensure python libselinux bindings for containers  https://review.opendev.org/c/openstack/openstack-ansible-openstack_hosts/+/93042109:29
jrosserwe should note this all on the etherpad09:29
noonedeadpunkhttps://etherpad.opendev.org/p/osa-zuul-checkout-refactoring ?09:29
jrosseras we have (supposedly) already got handling for u-c in zuul 09:30
jrosserhere https://github.com/openstack/openstack-ansible/blob/master/scripts/bootstrap-ansible.sh#L133-L14209:30
noonedeadpunkwe have for all these bits09:30
jrosserand here https://github.com/openstack/openstack-ansible-repo_server/blob/master/tasks/repo_install_constraints.yml#L23-L3909:30
jrosserbut it's not working for $reasons09:30
noonedeadpunknot working on N-1 as we undefine ZUUL_SRC_PATH09:31
jrosserargh right i see09:31
jrosserso this is yet another reason to have a unified approach for all branches09:31
noonedeadpunkhttps://opendev.org/openstack/openstack-ansible/src/branch/master/scripts/gate-check-commit.sh#L261-L26709:32
noonedeadpunkbut we have something else somewhere else as well I believe09:32
noonedeadpunkyeah, so we don't export it for upgrade at all: https://opendev.org/openstack/openstack-ansible/src/branch/master/zuul.d/playbooks/pre-osa-aio.yml#L4409:33
noonedeadpunkso we explicitly avoid setting or running bootstrap-ansible for upgrade jobs, fully relying on gate-check-commit09:34
noonedeadpunkto checkout, bootstrap, deploy, set ZUUL_SRC_PATH, upgrade09:34
jrosserwell - you've done more work on all this than me - but i do find the complexity totally overwhelming09:41
noonedeadpunkI will try to describe in etherpad 09:41
jrosserthere is in a way a lot of complexity when handling upgrade jobs09:41
jrosserbut at the same time, a job which is !upgrade is just a subset of an upgrade job09:42
noonedeadpunkit's not _that_ complex, but complex enough if you want to completely refactor it09:42
noonedeadpunkwell, I always was thinking about that in reverse, but looking at code I understand how it's reversed now09:43
noonedeadpunkfeels like we'd need to move the etherpad to contributor docs once we're done09:47
jrosseri think we may have a rabbitmq breakage on noble09:54
jrosserseen this on two jobs now https://zuul.opendev.org/t/openstack/build/f2ebae81e6cb4f129855b166a07b7c36/log/job-output.txt#558109:55
noonedeadpunkwhat rabbitmq version get's in there?09:55
noonedeadpunkit feels to be coming not from repos we expect09:56
jrosser`rabbitmq-server=4.0.2-1`09:56
noonedeadpunkyeah09:56
noonedeadpunkexplains it09:57
noonedeadpunkso quite a decision to take09:57
noonedeadpunkalso nice release notes regarding support: https://github.com/rabbitmq/rabbitmq-server/releases/tag/v4.0.210:00
noonedeadpunkso the issue is following. 2024.2 is actually a good place for such upgrades, so we techincally can drop support of HA queues here10:03
noonedeadpunkand just leave CQv2 or Quorum queues10:03
noonedeadpunkbut we have Quorum for just 2024.1 and I'm not confident enough in them10:03
noonedeadpunk(but look good so far)10:03
noonedeadpunkor, we would have to stay on 3.13 until 2025.1 kinda10:04
noonedeadpunk(or maybe not?)10:04
noonedeadpunkas it actually feels we need to somehow convert old haqueues to just CQv210:05
noonedeadpunkif user opted out from quorum10:05
noonedeadpunkand there's also interesting lhepri replacement for mnesia as well...10:07
noonedeadpunkjrosser: oh, I probably found what went wrong - https://opendev.org/openstack/openstack-ansible/blame/commit/e16d741a78ab75586f70034bc749a6a03a62689e/zuul.d/playbooks/run.yml#L3110:14
noonedeadpunkas for upgrade jobs pre_run does not happen10:14
noonedeadpunkI'm actually not sure how it works at all....10:14
noonedeadpunkah, whatever, these are ignored, disregard10:15
opendevreviewMerged openstack/openstack-ansible-os_neutron stable/2023.1: Do not kill ipsec on L3 cleanup  https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/93018711:20
opendevreviewMerged openstack/openstack-ansible-haproxy_server stable/2023.2: Respect defined interface for external VIP with LE  https://review.opendev.org/c/openstack/openstack-ansible-haproxy_server/+/93038511:22
opendevreviewDmitriy Rabotyagov proposed openstack/openstack-ansible-rabbitmq_server master: Bump rabbitmq version to 3.13.7  https://review.opendev.org/c/openstack/openstack-ansible-rabbitmq_server/+/93044612:06
noonedeadpunkthis should fix noble rabbitmq ^12:06
jrosserthat is an ugly fallback to take completely the wrong version rather than failing12:10
jrosserwell, "wrong" is our opinion i guess, apt has its own ideas :)12:11
noonedeadpunkyeah...12:11
noonedeadpunkbut well, I think it has also smth to do with our pinning12:11
noonedeadpunkthat's the policy I saw: https://paste.openstack.org/show/bFC2rhWoawUYeTOOdvnY/12:12
noonedeadpunkso apt basically see there's nothing to increase priority on and proceeds with what it has12:12
noonedeadpunkso probably, we should drop the rest to some non-installable priority12:12
noonedeadpunklike set prio 1 or smth....12:13
noonedeadpunkbut then it will just take from the default repo instead12:13
jrosseroh that reminds me, we still could update the url of the ppa there12:14
noonedeadpunkoh, yes12:14
noonedeadpunkI have no idea though what's best to do12:14
noonedeadpunkas current repo seems quite reliable finally...12:14
noonedeadpunkwe have seen 1 outage during 4.0 release time or so I think12:15
noonedeadpunk(when jobs were just timeouting)12:15
opendevreviewJonathan Rosser proposed openstack/ansible-hardening master: Apply architecture specific audit rules  https://review.opendev.org/c/openstack/ansible-hardening/+/93045112:34
opendevreviewDmitriy Rabotyagov proposed openstack/openstack-ansible-os_octavia master: Align coordination_client_ssl value with other roles  https://review.opendev.org/c/openstack/openstack-ansible-os_octavia/+/93046413:32
opendevreviewDmitriy Rabotyagov proposed openstack/openstack-ansible-os_octavia master: Move rpc_conn_pool_size to oslo_messaging_rabbit  https://review.opendev.org/c/openstack/openstack-ansible-os_octavia/+/93046913:40
opendevreviewMerged openstack/openstack-ansible-galera_server master: Map all relevant architectures for deb822 repository setup  https://review.opendev.org/c/openstack/openstack-ansible-galera_server/+/93028214:36
opendevreviewMerged openstack/openstack-ansible-ceph_client master: Ensure apparmor folder exists for ceph caching  https://review.opendev.org/c/openstack/openstack-ansible-ceph_client/+/92960614:52
mnaserjrosser, noonedeadpunk: https://review.opendev.org/c/openstack/openstack-ansible-os_nova/+/817222 seems like osa is one of the only tools which are using this feature, we're having issues with it because we rotate certs often (3 months automatically) and noticing that when the cert expires, qemu doesn't "read" the new cert, and then we start to get errors that the cert is expired for vnc.. have you seen something similar?  unless 14:58
mnaseryou issue long lived cerst14:58
noonedeadpunkoh yes, by default cert is like 10y14:59
noonedeadpunkotherwise console is revived after live migration14:59
noonedeadpunkso 3m is not gonna really fly I guess14:59
jrosseris this a thing? https://patchwork.kernel.org/project/qemu-devel/patch/20210104071128.754-1-changzihao1@huawei.com/#2391001315:00
mnaserurgh, that's annoying, we do 3 months so after a while we've seen now tls stops working of course until a live migration or any operation that gets reloaded15:01
mnaserjrosser: oh that's nice, i have been googling but couldnt find much15:02
jrosserits unclear if that actually went anywhere from that15:02
mnaseri think it has not :(15:02
jrosserdoh :(15:02
opendevreviewDamian DÄ…browski proposed openstack/openstack-ansible-os_octavia master: Disable octavia_management_net_dhcp by default  https://review.opendev.org/c/openstack/openstack-ansible-os_octavia/+/93048615:02
mnaser3m auto-renewal feels really nice, switching to a 10y thing feels like a big hack boo15:03
mnaserthe whole system we built was a sidecar that automatically renewed certs every 3 months for libvirt api and vnc15:04
mnaseror whatever period of time..15:04
jrosserhmmm https://qemu-project.gitlab.io/qemu/interop/qemu-qmp-ref.html#qapidoc-174915:04
mnaseroh interesting let me git blame that15:05
mnaserhttps://github.com/qemu/qemu/commit/9cc07651655ee86eca41059f5ead8c4e5607c73415:05
mnasernow i think with libvirt we're not supposed to technically talk to qmp of vms15:06
mnaserso i wonder if it can call display-reload15:06
mnaserhttps://lists.libvirt.org/archives/list/devel@lists.libvirt.org/thread/LDTM3NIOBWCS2USQSHXEKASHJHDVPNDG/15:06
jrosser`virsh qemu-monitor-command ....` ?15:07
mnaserhttps://libvirt.org/html/libvirt-libvirt-domain.html#virDomainGraphicsReload15:08
mnaserhttps://github.com/libvirt/libvirt/commit/b25b071c7551495882cb8970bccd3e31d16b366715:09
mnaserneeds libvirt 10.2.0 and qemu 6.0.015:09
jrosserthats an api right? how would you invoke that from tooling that dropped a new cert?15:10
opendevreviewMerged openstack/openstack-ansible-openstack_hosts master: Rename libselinux python package bindings  https://review.opendev.org/c/openstack/openstack-ansible-openstack_hosts/+/92954215:10
mnaserjrosser: i was thinking once we drop the new cert we would just list all domains and run that against all domains, since the tooling that rotates certs is a small daemon15:11
mnaserim thinking i might even right a small tool that would just watch the cert file and issue the command if it changes, that can be used by any deployemnt tool for example15:12
mnasercause we manage/issue sidecars with this - https://github.com/vexxhost/pod-tls-sidecar15:13
jrossermnaser: btw did you get anywhere with mcapi CI on noble?15:15
* jrosser still needs virtualenv support for that15:15
mnaserjrosser: we are slowly making our way with noble here, not as fast as i'd like to15:15
mnaseri've had a few family stuff last week that put me out for a while so slowly ramping up15:15
mnaseri do have noble in the zuul so that kinda helps at least a bit15:16
jrosserdanger is we release OSA pointing to my personal fork of mcapi which would be baaaad15:16
mnaserwhat pieces are essential to it?  sorry15:16
jrosseroh sorry, the k8s collection i mean15:16
mnaserah that15:16
mnaseroh its the whole15:17
mnaserworks in docker ci container thing right15:17
jrosseri think so yes, as somehow that doesnt count as a "system python" probably15:17
noonedeadpunkbtw, very noob question I was not able to quickly google for - how in the world to select a driver for Magnum? As seems that's only config setting, so not smth you can do on tempalte level?15:28
noonedeadpunkie, if to have both capi and heat drivers at same time for more granular migration?15:29
jrossertheres some tuple that identifies the driver i think (k8s, vm, ubuntu) or something -> capi driver15:30
noonedeadpunkthat used to be like... based on image tags?15:30
jrosseri think it still is15:30
noonedeadpunkhuh, ok15:31
noonedeadpunkso kinda if I just get capi driver in, in theory, existing cluster templates should remain working15:31
jrosseri think so..... but never actually had heat stuff working at all15:32
noonedeadpunkyeah, we still do have it working and ppl using it. So starting thinging about migration plan15:33
noonedeadpunkor well, possiiblity15:33
mnaseryes noonedeadpunk, it relies on the actual combination of coe when creating cluster template, and the os_distro from the image attrs15:45
mnaserits not a great solution but indeed it wouldn't affect any of the existing users that use fedora15:45
noonedeadpunkokay, awesome, thanks for confirmation!15:45
opendevreviewJonathan Rosser proposed openstack/openstack-ansible-os_horizon master: Ensure that selected Apache MPM is enforced  https://review.opendev.org/c/openstack/openstack-ansible-os_horizon/+/92969515:46
opendevreviewJonathan Rosser proposed openstack/openstack-ansible master: Convert Skyline HAProxy httpcheck  https://review.opendev.org/c/openstack/openstack-ansible/+/92989215:46
opendevreviewJonathan Rosser proposed openstack/openstack-ansible master: Add Ceph test for Rocky Linux  https://review.opendev.org/c/openstack/openstack-ansible/+/92963615:47
opendevreviewJonathan Rosser proposed openstack/ansible-role-uwsgi master: Add libpython definition for ubuntu noble distro install  https://review.opendev.org/c/openstack/ansible-role-uwsgi/+/92960915:48
opendevreviewJonathan Rosser proposed openstack/openstack-ansible-os_keystone master: Ensure that selected Apache MPM is enforced  https://review.opendev.org/c/openstack/openstack-ansible-os_keystone/+/92969115:48
opendevreviewJonathan Rosser proposed openstack/openstack-ansible-os_keystone master: Change example to contain domain name instead of UUID  https://review.opendev.org/c/openstack/openstack-ansible-os_keystone/+/91956315:49
opendevreviewDmitriy Rabotyagov proposed openstack/openstack-ansible-os_nova master: Allow to apply custom configuration to Nova SSH config  https://review.opendev.org/c/openstack/openstack-ansible-os_nova/+/93050519:03
noonedeadpunkI wonder how we lived without that all this time ^19:03
noonedeadpunkI would also limit hosts to be frank....19:05
noonedeadpunkto the mgmt network or smth19:05
jrosserI guess limiting hosts is really an sshd/iptables problem which is not really in scope for OSA to set up?20:45

Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!