Wednesday, 2025-07-09

opendevreviewOpenStack Proposal Bot proposed openstack/openstack-ansible master: Imported Translations from Zanata  https://review.opendev.org/c/openstack/openstack-ansible/+/95440003:04
opendevreviewJonathan Rosser proposed openstack/ansible-role-pki master: Allow certificates to be installed by specifying them by name  https://review.opendev.org/c/openstack/ansible-role-pki/+/95423906:42
opendevreviewMerged openstack/openstack-ansible-ops master: Bump prometheus.prometheus to 0.27.0  https://review.opendev.org/c/openstack/openstack-ansible-ops/+/95416506:47
opendevreviewMerged openstack/openstack-ansible-os_barbican master: docs: add link to OSA deployment guide for adding a role  https://review.opendev.org/c/openstack/openstack-ansible-os_barbican/+/95230506:59
opendevreviewMerged openstack/openstack-ansible-os_cloudkitty master: docs: add link to OSA deployment guide for adding a role  https://review.opendev.org/c/openstack/openstack-ansible-os_cloudkitty/+/95236306:59
opendevreviewMerged openstack/openstack-ansible master: Imported Translations from Zanata  https://review.opendev.org/c/openstack/openstack-ansible/+/95440007:01
mossblaser(sorry about latency) noonedeadpunk: I mean the "?" syntax for defining repeated values with the same name in config_template08:22
noonedeadpunkyeah, right, you can do that. 08:23
noonedeadpunkbut it's probably not how you can do with https://opendev.org/openstack/ansible-role-systemd_networkd/src/branch/master/templates/systemd-netdev.j2 (except overrides ofc)08:24
mossblaseryeah; the current implementation appears to use a "bespoke" template for cases it supports08:25
jrosserlooks like the ceph people are starting to get 24.04 builds in place https://tracker.ceph.com/issues/7202008:50
jrossernoonedeadpunk: we were looking to use the systemd_networkd role here some more, but the non-nativeness particularly in templates/systemd-network.j2 is a bit of a problem08:52
jrosserso really the root of the question would be if we should look at an overhaul of the systemd_networkd role, or if we create something similar but different internally08:52
noonedeadpunkif you could be a bit more specific about blockers - that would be nice...08:57
noonedeadpunkas tbh I'm passing a bunch of things to it as overrides indeed08:58
noonedeadpunkbut I'm not against of some refactoring either08:58
noonedeadpunkalso - we're using just regular `ansible.builtin.template` for netdev defenition, so it's not having overrides...08:59
noonedeadpunkhttps://opendev.org/openstack/ansible-role-systemd_networkd/src/branch/master/tasks/main.yml#L149-L16008:59
jrossermossblaser: ^ you've been thinking about this mostly - thoughts?09:34
mossblaserthe background to some of my thinking here is that our network config effectively gets deployed "twice": once by the installer (for bare metal) or cloud-init (for containers) and then a second time via ansible playbooks during provisioning09:53
mossblaserthis means we have all sorts of bodges to translate at least some subset of our network config into curtin/cloud-init config; but this is an approximate game and so during initial provisioning things never quite match up09:54
mossblaserbecause we've historically used ifupdown for network config this leads to cludgey situations during intial provisioning where a reboot or re-run of the playbook is needed09:54
mossblaserbut we're wanting to migrate to networkd which would obviously help somewhat09:55
mossblaserbut ideally we'd like it if our networkd config were also deployed by the ubuntu installer (for baremetal) and cloud-init (for containers) -- after all, its "just a bunch of config files"09:55
mossblaserwith the way the systemd_networkd role is currently setup, there isn't a neat way to turn Ansible data into "a bunch of config files" to feed into our PXE boot or container configs09:57
mossblaserit feels like there's a couple of ways to approach this: if the mapping from Ansible data to networkd config files was 1:1 and simple, then obviously our PXE boot/container configs could use the Ansible data directly to populate their config.09:59
mossblaserAlternatively, if the systemd_networkd data-to-config-files process was implemented as an Ansible filter, this filter (data in - dict of files to generate out, for example) that filter could be neatly re-used in other settings (like our PXE/container configs)10:01
mossblaser(sorry, that's all a bit rambly, does that make some sense?)10:01
noonedeadpunkum, well10:04
noonedeadpunkso we're using combination of systemd_networkd and netplan right now10:04
noonedeadpunkand netplan config is provisioned by cloud-init10:05
noonedeadpunkand that is quite neat, as netplan does use systemd-networkd as backend anywa10:05
noonedeadpunkwith systemd-networkd role we can easily extend or re-configure what's done with netplan and cloud-init10:06
noonedeadpunkfirst by applying systemd overrides (via overrides_only) but then you can make provisioned config by role having higher precedence then what netplan does by naming files10:07
noonedeadpunkit kinda sounds that ifupdown is jsut not a team player here to me10:07
noonedeadpunk> there isn't a neat way to turn Ansible data into "a bunch of config files" - I guess I'm not sure what this really means10:09
noonedeadpunkI'm thinking in osa context now ofc, but I have setups where systemd_networkd does control all networking10:10
noonedeadpunksmth like that: https://paste.openstack.org/show/bRUdirKtBDc9ZrpNvTID/10:15
noonedeadpunkhttps://paste.openstack.org/show/bdRo8tC8iSKVmQi91HKh/ - with proper highlighting10:16
noonedeadpunk(and ofc you can do smarter in terms of group_vars and vars lookup instead of these conditions)10:17
noonedeadpunkbut I guess the question was about - what is problematic in this input to the role?10:17
noonedeadpunkLooking at the template, I guess right now it requires only 1 parameter outside of config_overrides, which is `interface`10:24
mossblaserin your setup how do you keep your initial netplan config in sync with the networkd config?10:24
noonedeadpunkI don't?10:24
noonedeadpunkor well10:24
noonedeadpunkI extend in with extra VLANs I need10:25
noonedeadpunkthrough `network_overrides_only`10:25
noonedeadpunkthen only extra parameters defined in role are applied, while netplan does configure interface10:26
noonedeadpunkin other cases I completely override netplan config with the role, when naming interfaces with higher precedence over netplan10:26
noonedeadpunknetplan generated systemd-networkd config to `/run/systemd/network/`10:28
noonedeadpunkso if you create /etc/systemd/network/10-netplan-bond0.network.d/override.conf - you can alter resulting interface10:28
noonedeadpunkor if you create /etc/systemd/network/05-netplan-bond0.network - it would re-define what's in netplan10:29
noonedeadpunkas netplan apply ultimatelly calling for systemd-networkd reload10:29
mossblaseris the disconnect between the initial netplan and networkd config out of a need to have different network config during setup or just because it isn't trivial to have netplan (or some other cloud-init-based step) deploy the same full networkd config?10:30
noonedeadpunkit's that we need to have ability to change networking during runtime in some cases10:30
noonedeadpunkwhere cloud-init would require reboot10:30
noonedeadpunkgood example of that are adding new VRFs to net nodes10:30
mossblaser(I had initially been considering going 'all in' on netplan but that plan hit the wall when I encountered things it didn't expose options for)10:31
noonedeadpunkas for VRF we are adding another vlan, so we need to extend bond010:31
noonedeadpunkand rebooting all netnodes is eh....10:31
mossblaserthe reconfig thing; yes we have a similar need -- but what I mean is it would be lovely if both cloud-init and the playbooks deployed the exact same config sourced from the same Ansible data10:32
noonedeadpunkbut again, in other deployment netplan and cloud-init only define ssh network and that's it10:32
noonedeadpunkall rest goes from playbook10:32
noonedeadpunkyeah, so you kinda want to run the role on PXE server to produce configs for the boot?10:33
noonedeadpunkand then be able to run them also against the target?10:34
mossblaseryeah -- that's kind of the idea10:34
noonedeadpunkI guess what I mostly miss is what is problematic in input that it can't fit this need?10:35
noonedeadpunkas if ansible has same context/vars - it should not be a big problem?10:36
mossblaseryeah; there's nothing inherently problematic (obviously the role would need some slight adjustments to work that way, of course)10:37
noonedeadpunkI guess we can make `/etc/systemd/network` configurable so you could place configs in arbitrary place/chroot10:37
mossblaserbut this all got mixed up with my wondering to Jon about how some parts of the config amount to "networkd config expressed as YAML" whilst others are "custom abstractions turned into networkd config by a template"10:38
mossblaserbut I imagine that is largely just history?10:38
mossblaseryeah -- and of course to be able to turn off everything that wasn't generating config files (e.g. installing networkd and restarting things)10:40
noonedeadpunkI'm not 100% sure, but I think in *.netdev you mostly don't have any repeating options. So you can do just regular structure10:40
noonedeadpunkwhich is totally not the case for *.network10:40
mossblaseryeah10:40
noonedeadpunkwe probably could supply all content as overrides there10:41
noonedeadpunkbut then it's question of compatability10:41
jrosserso i think really that is the ultimate question10:42
mossblaser(its only because I'd seen the repeating option support in config template I'd wondered if that might have been how it would be done)10:42
jrosserif we would like to do any changes to the networkd role to make it more "networkd-config-as-yaml" at the input for everything10:42
noonedeadpunkso if we make `interface` as optional argument here - https://opendev.org/openstack/ansible-role-systemd_networkd/src/branch/master/templates/systemd-network.j2#L410:42
noonedeadpunkfor rest you can just simply use overrides as a content10:43
mossblaseryeah; it looks like it10:43
mossblaserthat arguably would be useful for folks matching interfaces based on something other than names (e.g. bus addresses or MACs)10:44
noonedeadpunkright10:44
noonedeadpunkas I did that in paste above as well10:44
mossblaserah yes! I'd missed that10:45
noonedeadpunkI don't see why we can't make it optional as well10:46
noonedeadpunkand then you can pass native yaml as overrides, which should be fine I'd assume10:47
jrossermossblaser: does this give a route forward?10:48
jrosserlike, add some bools for disabling the setup steps/restarts10:48
jrosseradd some default var for the path to write to10:48
mossblaseryeah10:48
jrosserand make the contents of the network template have a conditional on the presence of `interface` in the inut10:49
jrosser*input10:49
jrossernoonedeadpunk: thanks for the discussion - its really helpful10:50
mossblaserthe only downside is running the performance cost of running the role for every host to generate PXE/container config is might be somewhat noticable10:50
mossblaserindeed; thanks very much for the help, noonedeadpunk10:50
noonedeadpunkare you going to run it in advance or when boot triggered?10:51
mossblaserin advance10:51
mossblaser(at least this is effectively how it is done at the moment)10:51
noonedeadpunkmaybe you can use some tags?10:51
noonedeadpunkthen really there should be almost no overhead if it's just place configs10:51
noonedeadpunks/use/introduce/10:55
mossblaserindeed!10:56
jrossercan you define tags in a playbook that calls a role?10:56
noonedeadpunkpassing `systemd_networkd_distro_packages: []` would skip all package-related stuff10:57
noonedeadpunkbut also, I think on playbook level you can run multiple copies of role in parallel?10:57
noonedeadpunksec10:58
mossblaserI'll continue on some prototyping/thinking for the rest of what I'm doing and if all goes well I'll put together some patches for the minor changes discussed here and see how things end up!10:58
noonedeadpunklike with batch filter https://opendev.org/openstack/openstack-ansible-plugins/src/branch/master/roles/openstack_resources/tasks/image.yml#L2811:00
noonedeadpunkI'm not 100% sure about that though...11:00
jrosserthis is a wierd case as you end up with "delegate_to: pxe_server"11:01
noonedeadpunkhttps://dmsimard.com/2017/05/10/limiting-the-amount-of-concurrent-asynchronous-tasks-in-ansible/11:01
jrosserand then a with_items on some tasks/role that write the pxe config with a loop on the results of some map() over all of hostvars11:02
jrosserand this can be very inefficient11:02
jrosserbut thats another matter11:02
mossblaserin another world where the mapping from Ansible data to config file contents were done a filter (rather than template [config] calls) then just calling that filter for each host would be both relatively quick and also easy to do from 'outside' the rest of the role11:04
mossblaserbut given that config template doesn't work that way either this likely isn't achievable11:04
jrosserdamiandabrowski: did you have any more thoughts on what we should do for `type` in the PKI role?12:06
darkhackerncHi, I am working on masakari deployment in (virtual lab), the deployment is completed, systemd services are running, created sergment nad added the nodes into the segment, while testing the migration/evacuation the instance is only migrating on the same node, not across the node, any hint please https://paste.centos.org/view/raw/b2872f5012:54
darkhackerncany one working on masakari 12:56
darkhackernccan please guide and help12:56
noonedeadpunkmasakari does not do migration on itself - it asks nova and process goes through it's scheduling process iirc13:07
damiandabrowskijrosser: I think we have 3 possible options. Didn't we agree on option a) already? But I'm open for a discussion13:15
damiandabrowskia) align all backends. Accept following string values: certificate, certificate_chain, key13:15
damiandabrowskib) keep it as is for standalone backend, while hashi_vault will accept a list format as well as string format.13:16
damiandabrowskiList can be used to combine multiple types in a same file, like ['certificate', 'ca_chain']. It's closely tied to the hashi_vault internals. This option was proposed in my patches.13:16
damiandabrowskic) modify standalone backend, add a support for a list type, like: ['certificate', 'ca_chain'] to align with hashi_vault backend. That's probably the most complex option.13:16
jrosserdamiandabrowski: well, we need to write on the etherpad :)13:22
damiandabrowskiack, moved them to etherpad13:25
jrosserare you happy to choose `a` then?13:29
darkhackerncnoonedeadpunk, yup yup, nova-scheduler not seeing any calls13:33
noonedeadpunkthen masakari does not do them13:38
noonedeadpunkfrankly - I'd need to revisit the whole pacemaker thing....13:39
noonedeadpunkbut as I have no masakari currently deployed - it doesn't have very high prio atm13:39
damiandabrowskijrosser: ouh, I just realized one problematic thing in option a)13:56
damiandabrowskihttps://opendev.org/openstack/ansible-role-zookeeper/src/branch/master/defaults/main.yml#L11413:56
damiandabrowskiwe refer directly to the intermediate certificate there, not the "service cert"13:57
damiandabrowskiin current hashi_vault implementation, I just used type: ca_chain and it worked14:00
damiandabrowskibut it may not be that easy for standalone, because as you mentioned in etherpad:14:00
damiandabrowski"When installing certificates, from just the name of the cert the standalone backend cannot tell which CA was used to sign"14:00
damiandabrowskinow I realized it's probably not a big issue, because we can add one more item(ca_chain) to the _source_files list in your patch14:20
damiandabrowskihttps://review.opendev.org/c/openstack/ansible-role-pki/+/95423914:20
damiandabrowskior not...we can't for the same reason(when installing cert with standalone backend, we don't have direct access to the information about the issuer)14:22
noonedeadpunkum, I'm not sure what do you mean14:30
noonedeadpunkas when we issue certificate, we totally know about issuer14:31
noonedeadpunkso why can't we place result in multiple formats right away by name?14:31
noonedeadpunkincluding just standalone CA, even as a symlink?14:31
noonedeadpunk(or hard link)14:32
damiandabrowskiwhen installing cert with standalone backend, you don't really know the issuer. you just define the source file and destination file14:32
damiandabrowskidon't confuse certificate creation with the installation14:33
damiandabrowskibut you're right, when we issue certificate, we can store the ca_chain in another file, besides the other files14:34
damiandabrowskibut naming this file properly would be tricky, because we currently use "*-chain.crt" for something that is rather a fullchain(cert + ca_bundle in a single file)14:35
noonedeadpunk> you don't really know the issuer14:36
noonedeadpunkwhy you need to know it, if it will be file with the name of certificate?>14:36
noonedeadpunkor a symlink to it14:36
noonedeadpunkor smth like that14:36
damiandabrowskiin that case, it won't be needed14:37
damiandabrowskibut we don't have that file as of now14:37
noonedeadpunkoh, yes, sure14:37
noonedeadpunkother way could be to get certificate details and register it as a fact14:37
noonedeadpunkbut I think this a very different topic now14:38
noonedeadpunkrealted to not storing certificates on deploy host14:38
noonedeadpunkand making them more short-lived14:38
damiandabrowskiI left some thoughts in the etherpad14:49
lowercaseRegarding RabbitMQ Classic mirror queues. Are we safe to take quorum queues at any point prior to Dalmation/RabbitMQ version <3.13?17:01
noonedeadpunklowercase: I'd even say until 4.017:23
noonedeadpunkas at 4.0 they're completely removed17:23
noonedeadpunkso probably inclkuding dalmatian17:23
lowercaseMy boss asked if you all were providing a path forward to convert from classic queues to quorum queues. Sorry, I know all of these answers already but he wants me to ask17:31
lowercaseI proposed using federated queues to point the old queue to a new, quorum queue.17:32
noonedeadpunklowercase: yeah, we had a code andf docs in place for that17:34
noonedeadpunkhttps://docs.openstack.org/openstack-ansible/2024.1/admin/upgrades/major-upgrades.html#implement-changes-to-osa-configuration17:35
noonedeadpunkour approach though is to drop old vhost and create a new one with durable queues17:35
noonedeadpunkthis also provides a way back. and yes - there's a downtime in the meanwhile for operations with this approach17:36
noonedeadpunkbut using that is really helpfule: https://docs.openstack.org/openstack-ansible/latest/admin/maintenance-tasks/rabbitmq-maintain.html#migrate-between-ha-and-quorum-queues17:36
lowercaseLooks like the process will delete the old vhost, create a new vhost as a quorum queue. Requires a restart to take the changes.17:40
lowercaseI'll go down the road looking at using the federated process and get back to you. See if its helpful17:40
lowercasehttps://www.rabbitmq.com/blog/2023/03/02/quorum-queues-migration#in-place-migration17:41
noonedeadpunkwell, you kinda still need to restart service, as in this case it's up to oslo.messaging and configuration of service if it will attempt to use classic or durable queue17:43
noonedeadpunk(or stream)17:43
noonedeadpunkas I'm not sure if you can actually use streams easily then anyway17:43
noonedeadpunk(if you would want them for some usecases)17:49
lowercasehmm... 17:52

Generated by irclog2html.py 4.0.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!