| opendevreview | OpenStack Proposal Bot proposed openstack/openstack-ansible master: Imported Translations from Zanata https://review.opendev.org/c/openstack/openstack-ansible/+/954400 | 03:04 |
|---|---|---|
| opendevreview | Jonathan Rosser proposed openstack/ansible-role-pki master: Allow certificates to be installed by specifying them by name https://review.opendev.org/c/openstack/ansible-role-pki/+/954239 | 06:42 |
| opendevreview | Merged openstack/openstack-ansible-ops master: Bump prometheus.prometheus to 0.27.0 https://review.opendev.org/c/openstack/openstack-ansible-ops/+/954165 | 06:47 |
| opendevreview | Merged openstack/openstack-ansible-os_barbican master: docs: add link to OSA deployment guide for adding a role https://review.opendev.org/c/openstack/openstack-ansible-os_barbican/+/952305 | 06:59 |
| opendevreview | Merged openstack/openstack-ansible-os_cloudkitty master: docs: add link to OSA deployment guide for adding a role https://review.opendev.org/c/openstack/openstack-ansible-os_cloudkitty/+/952363 | 06:59 |
| opendevreview | Merged openstack/openstack-ansible master: Imported Translations from Zanata https://review.opendev.org/c/openstack/openstack-ansible/+/954400 | 07:01 |
| mossblaser | (sorry about latency) noonedeadpunk: I mean the "?" syntax for defining repeated values with the same name in config_template | 08:22 |
| noonedeadpunk | yeah, right, you can do that. | 08:23 |
| noonedeadpunk | but it's probably not how you can do with https://opendev.org/openstack/ansible-role-systemd_networkd/src/branch/master/templates/systemd-netdev.j2 (except overrides ofc) | 08:24 |
| mossblaser | yeah; the current implementation appears to use a "bespoke" template for cases it supports | 08:25 |
| jrosser | looks like the ceph people are starting to get 24.04 builds in place https://tracker.ceph.com/issues/72020 | 08:50 |
| jrosser | noonedeadpunk: we were looking to use the systemd_networkd role here some more, but the non-nativeness particularly in templates/systemd-network.j2 is a bit of a problem | 08:52 |
| jrosser | so really the root of the question would be if we should look at an overhaul of the systemd_networkd role, or if we create something similar but different internally | 08:52 |
| noonedeadpunk | if you could be a bit more specific about blockers - that would be nice... | 08:57 |
| noonedeadpunk | as tbh I'm passing a bunch of things to it as overrides indeed | 08:58 |
| noonedeadpunk | but I'm not against of some refactoring either | 08:58 |
| noonedeadpunk | also - we're using just regular `ansible.builtin.template` for netdev defenition, so it's not having overrides... | 08:59 |
| noonedeadpunk | https://opendev.org/openstack/ansible-role-systemd_networkd/src/branch/master/tasks/main.yml#L149-L160 | 08:59 |
| jrosser | mossblaser: ^ you've been thinking about this mostly - thoughts? | 09:34 |
| mossblaser | the background to some of my thinking here is that our network config effectively gets deployed "twice": once by the installer (for bare metal) or cloud-init (for containers) and then a second time via ansible playbooks during provisioning | 09:53 |
| mossblaser | this means we have all sorts of bodges to translate at least some subset of our network config into curtin/cloud-init config; but this is an approximate game and so during initial provisioning things never quite match up | 09:54 |
| mossblaser | because we've historically used ifupdown for network config this leads to cludgey situations during intial provisioning where a reboot or re-run of the playbook is needed | 09:54 |
| mossblaser | but we're wanting to migrate to networkd which would obviously help somewhat | 09:55 |
| mossblaser | but ideally we'd like it if our networkd config were also deployed by the ubuntu installer (for baremetal) and cloud-init (for containers) -- after all, its "just a bunch of config files" | 09:55 |
| mossblaser | with the way the systemd_networkd role is currently setup, there isn't a neat way to turn Ansible data into "a bunch of config files" to feed into our PXE boot or container configs | 09:57 |
| mossblaser | it feels like there's a couple of ways to approach this: if the mapping from Ansible data to networkd config files was 1:1 and simple, then obviously our PXE boot/container configs could use the Ansible data directly to populate their config. | 09:59 |
| mossblaser | Alternatively, if the systemd_networkd data-to-config-files process was implemented as an Ansible filter, this filter (data in - dict of files to generate out, for example) that filter could be neatly re-used in other settings (like our PXE/container configs) | 10:01 |
| mossblaser | (sorry, that's all a bit rambly, does that make some sense?) | 10:01 |
| noonedeadpunk | um, well | 10:04 |
| noonedeadpunk | so we're using combination of systemd_networkd and netplan right now | 10:04 |
| noonedeadpunk | and netplan config is provisioned by cloud-init | 10:05 |
| noonedeadpunk | and that is quite neat, as netplan does use systemd-networkd as backend anywa | 10:05 |
| noonedeadpunk | with systemd-networkd role we can easily extend or re-configure what's done with netplan and cloud-init | 10:06 |
| noonedeadpunk | first by applying systemd overrides (via overrides_only) but then you can make provisioned config by role having higher precedence then what netplan does by naming files | 10:07 |
| noonedeadpunk | it kinda sounds that ifupdown is jsut not a team player here to me | 10:07 |
| noonedeadpunk | > there isn't a neat way to turn Ansible data into "a bunch of config files" - I guess I'm not sure what this really means | 10:09 |
| noonedeadpunk | I'm thinking in osa context now ofc, but I have setups where systemd_networkd does control all networking | 10:10 |
| noonedeadpunk | smth like that: https://paste.openstack.org/show/bRUdirKtBDc9ZrpNvTID/ | 10:15 |
| noonedeadpunk | https://paste.openstack.org/show/bdRo8tC8iSKVmQi91HKh/ - with proper highlighting | 10:16 |
| noonedeadpunk | (and ofc you can do smarter in terms of group_vars and vars lookup instead of these conditions) | 10:17 |
| noonedeadpunk | but I guess the question was about - what is problematic in this input to the role? | 10:17 |
| noonedeadpunk | Looking at the template, I guess right now it requires only 1 parameter outside of config_overrides, which is `interface` | 10:24 |
| mossblaser | in your setup how do you keep your initial netplan config in sync with the networkd config? | 10:24 |
| noonedeadpunk | I don't? | 10:24 |
| noonedeadpunk | or well | 10:24 |
| noonedeadpunk | I extend in with extra VLANs I need | 10:25 |
| noonedeadpunk | through `network_overrides_only` | 10:25 |
| noonedeadpunk | then only extra parameters defined in role are applied, while netplan does configure interface | 10:26 |
| noonedeadpunk | in other cases I completely override netplan config with the role, when naming interfaces with higher precedence over netplan | 10:26 |
| noonedeadpunk | netplan generated systemd-networkd config to `/run/systemd/network/` | 10:28 |
| noonedeadpunk | so if you create /etc/systemd/network/10-netplan-bond0.network.d/override.conf - you can alter resulting interface | 10:28 |
| noonedeadpunk | or if you create /etc/systemd/network/05-netplan-bond0.network - it would re-define what's in netplan | 10:29 |
| noonedeadpunk | as netplan apply ultimatelly calling for systemd-networkd reload | 10:29 |
| mossblaser | is the disconnect between the initial netplan and networkd config out of a need to have different network config during setup or just because it isn't trivial to have netplan (or some other cloud-init-based step) deploy the same full networkd config? | 10:30 |
| noonedeadpunk | it's that we need to have ability to change networking during runtime in some cases | 10:30 |
| noonedeadpunk | where cloud-init would require reboot | 10:30 |
| noonedeadpunk | good example of that are adding new VRFs to net nodes | 10:30 |
| mossblaser | (I had initially been considering going 'all in' on netplan but that plan hit the wall when I encountered things it didn't expose options for) | 10:31 |
| noonedeadpunk | as for VRF we are adding another vlan, so we need to extend bond0 | 10:31 |
| noonedeadpunk | and rebooting all netnodes is eh.... | 10:31 |
| mossblaser | the reconfig thing; yes we have a similar need -- but what I mean is it would be lovely if both cloud-init and the playbooks deployed the exact same config sourced from the same Ansible data | 10:32 |
| noonedeadpunk | but again, in other deployment netplan and cloud-init only define ssh network and that's it | 10:32 |
| noonedeadpunk | all rest goes from playbook | 10:32 |
| noonedeadpunk | yeah, so you kinda want to run the role on PXE server to produce configs for the boot? | 10:33 |
| noonedeadpunk | and then be able to run them also against the target? | 10:34 |
| mossblaser | yeah -- that's kind of the idea | 10:34 |
| noonedeadpunk | I guess what I mostly miss is what is problematic in input that it can't fit this need? | 10:35 |
| noonedeadpunk | as if ansible has same context/vars - it should not be a big problem? | 10:36 |
| mossblaser | yeah; there's nothing inherently problematic (obviously the role would need some slight adjustments to work that way, of course) | 10:37 |
| noonedeadpunk | I guess we can make `/etc/systemd/network` configurable so you could place configs in arbitrary place/chroot | 10:37 |
| mossblaser | but this all got mixed up with my wondering to Jon about how some parts of the config amount to "networkd config expressed as YAML" whilst others are "custom abstractions turned into networkd config by a template" | 10:38 |
| mossblaser | but I imagine that is largely just history? | 10:38 |
| mossblaser | yeah -- and of course to be able to turn off everything that wasn't generating config files (e.g. installing networkd and restarting things) | 10:40 |
| noonedeadpunk | I'm not 100% sure, but I think in *.netdev you mostly don't have any repeating options. So you can do just regular structure | 10:40 |
| noonedeadpunk | which is totally not the case for *.network | 10:40 |
| mossblaser | yeah | 10:40 |
| noonedeadpunk | we probably could supply all content as overrides there | 10:41 |
| noonedeadpunk | but then it's question of compatability | 10:41 |
| jrosser | so i think really that is the ultimate question | 10:42 |
| mossblaser | (its only because I'd seen the repeating option support in config template I'd wondered if that might have been how it would be done) | 10:42 |
| jrosser | if we would like to do any changes to the networkd role to make it more "networkd-config-as-yaml" at the input for everything | 10:42 |
| noonedeadpunk | so if we make `interface` as optional argument here - https://opendev.org/openstack/ansible-role-systemd_networkd/src/branch/master/templates/systemd-network.j2#L4 | 10:42 |
| noonedeadpunk | for rest you can just simply use overrides as a content | 10:43 |
| mossblaser | yeah; it looks like it | 10:43 |
| mossblaser | that arguably would be useful for folks matching interfaces based on something other than names (e.g. bus addresses or MACs) | 10:44 |
| noonedeadpunk | right | 10:44 |
| noonedeadpunk | as I did that in paste above as well | 10:44 |
| mossblaser | ah yes! I'd missed that | 10:45 |
| noonedeadpunk | I don't see why we can't make it optional as well | 10:46 |
| noonedeadpunk | and then you can pass native yaml as overrides, which should be fine I'd assume | 10:47 |
| jrosser | mossblaser: does this give a route forward? | 10:48 |
| jrosser | like, add some bools for disabling the setup steps/restarts | 10:48 |
| jrosser | add some default var for the path to write to | 10:48 |
| mossblaser | yeah | 10:48 |
| jrosser | and make the contents of the network template have a conditional on the presence of `interface` in the inut | 10:49 |
| jrosser | *input | 10:49 |
| jrosser | noonedeadpunk: thanks for the discussion - its really helpful | 10:50 |
| mossblaser | the only downside is running the performance cost of running the role for every host to generate PXE/container config is might be somewhat noticable | 10:50 |
| mossblaser | indeed; thanks very much for the help, noonedeadpunk | 10:50 |
| noonedeadpunk | are you going to run it in advance or when boot triggered? | 10:51 |
| mossblaser | in advance | 10:51 |
| mossblaser | (at least this is effectively how it is done at the moment) | 10:51 |
| noonedeadpunk | maybe you can use some tags? | 10:51 |
| noonedeadpunk | then really there should be almost no overhead if it's just place configs | 10:51 |
| noonedeadpunk | s/use/introduce/ | 10:55 |
| mossblaser | indeed! | 10:56 |
| jrosser | can you define tags in a playbook that calls a role? | 10:56 |
| noonedeadpunk | passing `systemd_networkd_distro_packages: []` would skip all package-related stuff | 10:57 |
| noonedeadpunk | but also, I think on playbook level you can run multiple copies of role in parallel? | 10:57 |
| noonedeadpunk | sec | 10:58 |
| mossblaser | I'll continue on some prototyping/thinking for the rest of what I'm doing and if all goes well I'll put together some patches for the minor changes discussed here and see how things end up! | 10:58 |
| noonedeadpunk | like with batch filter https://opendev.org/openstack/openstack-ansible-plugins/src/branch/master/roles/openstack_resources/tasks/image.yml#L28 | 11:00 |
| noonedeadpunk | I'm not 100% sure about that though... | 11:00 |
| jrosser | this is a wierd case as you end up with "delegate_to: pxe_server" | 11:01 |
| noonedeadpunk | https://dmsimard.com/2017/05/10/limiting-the-amount-of-concurrent-asynchronous-tasks-in-ansible/ | 11:01 |
| jrosser | and then a with_items on some tasks/role that write the pxe config with a loop on the results of some map() over all of hostvars | 11:02 |
| jrosser | and this can be very inefficient | 11:02 |
| jrosser | but thats another matter | 11:02 |
| mossblaser | in another world where the mapping from Ansible data to config file contents were done a filter (rather than template [config] calls) then just calling that filter for each host would be both relatively quick and also easy to do from 'outside' the rest of the role | 11:04 |
| mossblaser | but given that config template doesn't work that way either this likely isn't achievable | 11:04 |
| jrosser | damiandabrowski: did you have any more thoughts on what we should do for `type` in the PKI role? | 12:06 |
| darkhackernc | Hi, I am working on masakari deployment in (virtual lab), the deployment is completed, systemd services are running, created sergment nad added the nodes into the segment, while testing the migration/evacuation the instance is only migrating on the same node, not across the node, any hint please https://paste.centos.org/view/raw/b2872f50 | 12:54 |
| darkhackernc | any one working on masakari | 12:56 |
| darkhackernc | can please guide and help | 12:56 |
| noonedeadpunk | masakari does not do migration on itself - it asks nova and process goes through it's scheduling process iirc | 13:07 |
| damiandabrowski | jrosser: I think we have 3 possible options. Didn't we agree on option a) already? But I'm open for a discussion | 13:15 |
| damiandabrowski | a) align all backends. Accept following string values: certificate, certificate_chain, key | 13:15 |
| damiandabrowski | b) keep it as is for standalone backend, while hashi_vault will accept a list format as well as string format. | 13:16 |
| damiandabrowski | List can be used to combine multiple types in a same file, like ['certificate', 'ca_chain']. It's closely tied to the hashi_vault internals. This option was proposed in my patches. | 13:16 |
| damiandabrowski | c) modify standalone backend, add a support for a list type, like: ['certificate', 'ca_chain'] to align with hashi_vault backend. That's probably the most complex option. | 13:16 |
| jrosser | damiandabrowski: well, we need to write on the etherpad :) | 13:22 |
| damiandabrowski | ack, moved them to etherpad | 13:25 |
| jrosser | are you happy to choose `a` then? | 13:29 |
| darkhackernc | noonedeadpunk, yup yup, nova-scheduler not seeing any calls | 13:33 |
| noonedeadpunk | then masakari does not do them | 13:38 |
| noonedeadpunk | frankly - I'd need to revisit the whole pacemaker thing.... | 13:39 |
| noonedeadpunk | but as I have no masakari currently deployed - it doesn't have very high prio atm | 13:39 |
| damiandabrowski | jrosser: ouh, I just realized one problematic thing in option a) | 13:56 |
| damiandabrowski | https://opendev.org/openstack/ansible-role-zookeeper/src/branch/master/defaults/main.yml#L114 | 13:56 |
| damiandabrowski | we refer directly to the intermediate certificate there, not the "service cert" | 13:57 |
| damiandabrowski | in current hashi_vault implementation, I just used type: ca_chain and it worked | 14:00 |
| damiandabrowski | but it may not be that easy for standalone, because as you mentioned in etherpad: | 14:00 |
| damiandabrowski | "When installing certificates, from just the name of the cert the standalone backend cannot tell which CA was used to sign" | 14:00 |
| damiandabrowski | now I realized it's probably not a big issue, because we can add one more item(ca_chain) to the _source_files list in your patch | 14:20 |
| damiandabrowski | https://review.opendev.org/c/openstack/ansible-role-pki/+/954239 | 14:20 |
| damiandabrowski | or not...we can't for the same reason(when installing cert with standalone backend, we don't have direct access to the information about the issuer) | 14:22 |
| noonedeadpunk | um, I'm not sure what do you mean | 14:30 |
| noonedeadpunk | as when we issue certificate, we totally know about issuer | 14:31 |
| noonedeadpunk | so why can't we place result in multiple formats right away by name? | 14:31 |
| noonedeadpunk | including just standalone CA, even as a symlink? | 14:31 |
| noonedeadpunk | (or hard link) | 14:32 |
| damiandabrowski | when installing cert with standalone backend, you don't really know the issuer. you just define the source file and destination file | 14:32 |
| damiandabrowski | don't confuse certificate creation with the installation | 14:33 |
| damiandabrowski | but you're right, when we issue certificate, we can store the ca_chain in another file, besides the other files | 14:34 |
| damiandabrowski | but naming this file properly would be tricky, because we currently use "*-chain.crt" for something that is rather a fullchain(cert + ca_bundle in a single file) | 14:35 |
| noonedeadpunk | > you don't really know the issuer | 14:36 |
| noonedeadpunk | why you need to know it, if it will be file with the name of certificate?> | 14:36 |
| noonedeadpunk | or a symlink to it | 14:36 |
| noonedeadpunk | or smth like that | 14:36 |
| damiandabrowski | in that case, it won't be needed | 14:37 |
| damiandabrowski | but we don't have that file as of now | 14:37 |
| noonedeadpunk | oh, yes, sure | 14:37 |
| noonedeadpunk | other way could be to get certificate details and register it as a fact | 14:37 |
| noonedeadpunk | but I think this a very different topic now | 14:38 |
| noonedeadpunk | realted to not storing certificates on deploy host | 14:38 |
| noonedeadpunk | and making them more short-lived | 14:38 |
| damiandabrowski | I left some thoughts in the etherpad | 14:49 |
| lowercase | Regarding RabbitMQ Classic mirror queues. Are we safe to take quorum queues at any point prior to Dalmation/RabbitMQ version <3.13? | 17:01 |
| noonedeadpunk | lowercase: I'd even say until 4.0 | 17:23 |
| noonedeadpunk | as at 4.0 they're completely removed | 17:23 |
| noonedeadpunk | so probably inclkuding dalmatian | 17:23 |
| lowercase | My boss asked if you all were providing a path forward to convert from classic queues to quorum queues. Sorry, I know all of these answers already but he wants me to ask | 17:31 |
| lowercase | I proposed using federated queues to point the old queue to a new, quorum queue. | 17:32 |
| noonedeadpunk | lowercase: yeah, we had a code andf docs in place for that | 17:34 |
| noonedeadpunk | https://docs.openstack.org/openstack-ansible/2024.1/admin/upgrades/major-upgrades.html#implement-changes-to-osa-configuration | 17:35 |
| noonedeadpunk | our approach though is to drop old vhost and create a new one with durable queues | 17:35 |
| noonedeadpunk | this also provides a way back. and yes - there's a downtime in the meanwhile for operations with this approach | 17:36 |
| noonedeadpunk | but using that is really helpfule: https://docs.openstack.org/openstack-ansible/latest/admin/maintenance-tasks/rabbitmq-maintain.html#migrate-between-ha-and-quorum-queues | 17:36 |
| lowercase | Looks like the process will delete the old vhost, create a new vhost as a quorum queue. Requires a restart to take the changes. | 17:40 |
| lowercase | I'll go down the road looking at using the federated process and get back to you. See if its helpful | 17:40 |
| lowercase | https://www.rabbitmq.com/blog/2023/03/02/quorum-queues-migration#in-place-migration | 17:41 |
| noonedeadpunk | well, you kinda still need to restart service, as in this case it's up to oslo.messaging and configuration of service if it will attempt to use classic or durable queue | 17:43 |
| noonedeadpunk | (or stream) | 17:43 |
| noonedeadpunk | as I'm not sure if you can actually use streams easily then anyway | 17:43 |
| noonedeadpunk | (if you would want them for some usecases) | 17:49 |
| lowercase | hmm... | 17:52 |
Generated by irclog2html.py 4.0.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!