| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: Use haproxy_install playbook from openstack-ansible-plugins repo https://review.opendev.org/c/openstack/openstack-ansible/+/924168 | 09:10 |
|---|---|---|
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_octavia stable/2024.1: Define ovn provider agent when OVN is used https://review.opendev.org/c/openstack/openstack-ansible-os_octavia/+/927718 | 09:12 |
| noonedeadpunk | good morning | 09:12 |
| jrosser | o/ morning | 09:12 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible stable/2024.1: Remove the get_md5 parameter from ansible stat tasks https://review.opendev.org/c/openstack/openstack-ansible/+/927719 | 09:14 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible stable/2023.2: Remove the get_md5 parameter from ansible stat tasks https://review.opendev.org/c/openstack/openstack-ansible/+/927720 | 09:14 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible stable/2023.1: Remove the get_md5 parameter from ansible stat tasks https://review.opendev.org/c/openstack/openstack-ansible/+/927721 | 09:15 |
| opendevreview | Andrew Bonney proposed openstack/openstack-ansible-os_ceilometer master: Add support for Magnum notifications https://review.opendev.org/c/openstack/openstack-ansible-os_ceilometer/+/927724 | 09:43 |
| noonedeadpunk | andrewbonney: can you write a reno for ^ ? | 09:44 |
| andrewbonney | Ah yes, sure | 09:45 |
| opendevreview | Andrew Bonney proposed openstack/openstack-ansible-os_ceilometer master: Add support for Magnum notifications https://review.opendev.org/c/openstack/openstack-ansible-os_ceilometer/+/927724 | 09:48 |
| noonedeadpunk | It would be also kinda nice to have that backported to 2024.1 imo | 09:49 |
| opendevreview | Merged openstack/openstack-ansible master: [doc] Add documentation on spawning HAProxy inside LXC https://review.opendev.org/c/openstack/openstack-ansible/+/924353 | 10:13 |
| noonedeadpunk | andrewbonney: sorry, I have forgot again about octavia keypair issue you've reported. But now I can't find the bug report either.... | 10:22 |
| noonedeadpunk | (it's third time I'm about to look into that, I know) | 10:22 |
| noonedeadpunk | ah, it's fixed with https://opendev.org/openstack/openstack-ansible-plugins/commit/be620d3b3546d3a7a27c7d8d29803ac49c864142 ? | 10:23 |
| noonedeadpunk | yeah, it must be | 10:24 |
| andrewbonney | To be honest I've forgotten about it too. It certainly looks like we're using that patch locally | 10:26 |
| andrewbonney | It looks like we may have changed octavia_ssh_key_dir locally. Not sure if perhaps the default wouldn't work in the case of an upgrade | 10:28 |
| andrewbonney | Perhaps with that patch in place it's fine actually. It was perhaps just an issue before the deploy host fix | 10:30 |
| noonedeadpunk | yeah, probably. and we had just a different issue here with the same thing kinda | 10:44 |
| noonedeadpunk | but you patch worked nicely :) | 10:44 |
| opendevreview | Merged openstack/openstack-ansible-os_octavia stable/2023.2: Ensure Octavia communicates with Neutron through internal URL https://review.opendev.org/c/openstack/openstack-ansible-os_octavia/+/927233 | 10:56 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_octavia master: Provide better flexability for SSH keypair options https://review.opendev.org/c/openstack/openstack-ansible-os_octavia/+/927728 | 11:00 |
| opendevreview | Merged openstack/openstack-ansible-ops master: mcapi_proxy: allow overriding of systemd service environment https://review.opendev.org/c/openstack/openstack-ansible-ops/+/926889 | 11:01 |
| opendevreview | Merged openstack/openstack-ansible-ops master: Allow mcapi proxy git sources and python package versions to be overridden https://review.opendev.org/c/openstack/openstack-ansible-ops/+/926409 | 11:02 |
| gokhan | hello folks after the antelope upgrade we can not create images with glance, Error logs are: "glance_store.exceptions.StoreAddDisabled: Configuration for store failed. Adding images to this store is disabled." these are detailed logs https://paste.openstack.org/show/bjNCxjh6Km9Jkula3FKA/. I am using nfs for glance images in this environment. Have you seen this issue ? | 11:04 |
| opendevreview | Merged openstack/openstack-ansible-ops master: Add variables and hook for high-availability k8s control plane test https://review.opendev.org/c/openstack/openstack-ansible-ops/+/923173 | 11:12 |
| grauzikas | Hello, im trying to make work magnum and have issues there. The issue is that heat doesnt provide to fedora-coreos certificate keys and inside fedora coreos i can see errors: | 11:13 |
| grauzikas | https://paste.openstack.org/show/bBg7FijxjMK9EdjeZNNz/ | 11:15 |
| noonedeadpunk | gokhan: and how does you glance.conf look like? | 11:15 |
| noonedeadpunk | grauzikas: frankly - I have no idea if heat is capable of that even, given coreos design... | 11:16 |
| noonedeadpunk | but if you're to use FQDN for public VIP and enable Let's Encrypt usage - I think you should not have issues with certificates there | 11:17 |
| grauzikas | *** is ip address of external vip. So i tryed to configure letsencrypt, but seems it configures only haproxy external side. Tryed to set some of tls opinions but also not worked. By searching on google i found that inside coreos openstack-ca.pem is empty: https://paste.openstack.org/show/bxUxvRz2nQ62PhoKEosh/ | 11:17 |
| jrosser | you need to define openstack_ca_file in magnum.conf i think | 11:18 |
| noonedeadpunk | grauzikas: oh, well, there're options to control that | 11:18 |
| noonedeadpunk | yeah, that | 11:18 |
| noonedeadpunk | or set `verify_ca: false` | 11:19 |
| jrosser | letsecrypt is only for the external side, the internal side is not relevant for this | 11:19 |
| grauzikas | https://storyboard.openstack.org/#!/story/2010124 | 11:19 |
| jrosser | it is the CA used by the magnum workload cluster when it calls back to the external API endpoint | 11:19 |
| grauzikas | but this is for kolla ansible i mean this variable: openstack_ca_file | 11:20 |
| jrosser | that is a config option for magnum | 11:20 |
| gokhan | noonedeadpunk, https://paste.openstack.org/show/bCerKFrXEFqCDEDkw05A/ | 11:20 |
| jrosser | grauzikas: like the bug report there says, you need to configure openstack_ca_file - see https://docs.openstack.org/magnum/latest/configuration/sample-config.html | 11:21 |
| noonedeadpunk | grauzikas: you can do like that I gueess https://paste.openstack.org/show/bbQHEpNCwOBo75YSbnys/ | 11:22 |
| noonedeadpunk | in your user_variables.yml | 11:23 |
| jrosser | maybe we should add this as a default in the template | 11:23 |
| jrosser | but with the heat driver there are just so many ways for this to go wrong (like using wrong endpoint blah blah) | 11:23 |
| noonedeadpunk | yeah. might make sense to add it indeed, though it's kinda just for dev setups rather then prods. As I hardly see any prod env with untrusted certificate anyway | 11:24 |
| jrosser | perhaps - an internal cloud with company CA would need it | 11:25 |
| jrosser | but maybe thats a legitimate situation to use a manual override | 11:25 |
| noonedeadpunk | gokhan: hm, Im a bit confused why you're missing a [cinder] and [http] sections | 11:26 |
| noonedeadpunk | jrosser: and then you also don't want it to be ca-certificates I assume | 11:26 |
| jrosser | https://github.com/openstack/openstack-ansible-ops/blob/master/mcapi_vexxhost/playbooks/files/openstack_deploy/user_variables_z_magnum.yml#L27 | 11:27 |
| noonedeadpunk | but something way more specific | 11:27 |
| noonedeadpunk | well. this proves the point that it makes sense to do this by default | 11:27 |
| gokhan | noonedeadpunk, there is no override for glance in user_variables. these are default settings. I only defined glance nfs client | 11:32 |
| noonedeadpunk | ok, yeah, on 2023.1 I have kinda same config, was comparing with 2024.1 accidnetally | 11:35 |
| noonedeadpunk | gokhan: I assume that you've already ensured that `glance` user can write to the NFS still? I guess I would try also to disable all extra backends (like cinder/file) if they're not used jsut to ensure it's file that's causing troubles | 11:39 |
| noonedeadpunk | ie by setting `glance_additional_stores: []` | 11:40 |
| noonedeadpunk | as if you've re-setup OS during it's upgrade, it could be that now UID/GID of glance has changed, unless you've explicitly defined them | 11:40 |
| gokhan | drwxr-xr-x 2 nobody nogroup 65536 Jul 26 17:43 images | 11:43 |
| gokhan | noonedeadpunk, all of the nodes uuid/gid are same | 11:45 |
| noonedeadpunk | but um. I'\m not sure glance user can write with 0755? | 11:46 |
| gokhan | noonedeadpunk, now I am trying wth glance user | 11:48 |
| noonedeadpunk | as I don\t think that privilege escalation is gonna be used by glance | 11:48 |
| gokhan | noonedeadpunk, yes after changing its user to glance, it worked thanks. | 11:49 |
| gokhan | noonedeadpunk, I have another question. how can we install multi region with osa? | 11:49 |
| noonedeadpunk | you'd need to have some overrides. I think that from OSA prespective these will be just 2 deployments (or better say, 2 openstack_deploy folders). But it kinda depends on the architecture more. | 11:51 |
| noonedeadpunk | as there's totally a possibility to have different regions as part of extra groups as well | 11:51 |
| noonedeadpunk | I kinda have some "example" here locally where I did multiregion with shared keystone | 11:52 |
| gokhan | this is the first time I am trying to deploy multi region. | 11:52 |
| noonedeadpunk | where each region was it's own openstack_deploy folder and keystone was a separate "region" from osa prespective | 11:52 |
| gokhan | I am also thinking with a shared keystone | 11:52 |
| noonedeadpunk | as eventually, you'd need to have a different FQDNs (API endpoints) for each region | 11:53 |
| noonedeadpunk | then both of these should talk to the same keystone | 11:53 |
| gokhan | if it is possible can you share your overrides? | 11:53 |
| noonedeadpunk | but it was quite complex (though interesting setup) I was going to document when have some time... | 11:54 |
| noonedeadpunk | well, I did have tons of overrides but all due to different reasons, so I'd need some time to extract ones for multi-region setup | 11:56 |
| noonedeadpunk | as there were multiple AZs in each region as well | 11:56 |
| noonedeadpunk | and endpoints by path (ie cloud.com/identity) | 11:57 |
| noonedeadpunk | so this is atually how I wanted to do the keystone sharing: https://i.imgur.com/gngXklM.png | 11:57 |
| noonedeadpunk | where to maintain a separate mariadb cluster and keystone hosts from any of the existing regions | 11:59 |
| gokhan | thanks noonedeadpunk, keystone dbs will be seperate from other services dbs. instead of haproxy you prefer proxySQL | 12:01 |
| noonedeadpunk | Actually, I'm looking at overrides now, and kinda main point there - is that neither of regions did have keystone defined in them | 12:01 |
| noonedeadpunk | so I had like /etc/openstack_deploy_region1, /etc/openstack_deploy_region2 and /etc/openstack_deploy_keystone | 12:01 |
| * noonedeadpunk never finished proxysql part | 12:01 | |
| noonedeadpunk | and then as part of /etc/openstack_deploy_keystone you include jsut keystone, memcahced and galera while other 2 conatian jsut everything except keystone | 12:02 |
| noonedeadpunk | since each had different public_vip (and thus endpoints) for regions I defined manually keystone_service_adminuri, keystone_service_internaluri and keystone_service_publicuri: https://opendev.org/openstack/openstack-ansible/src/branch/master/inventory/group_vars/all/keystone.yml | 12:04 |
| noonedeadpunk | though I had to do a lot of hackery there in order to spawn keystone containers locally to these regions. and that part is kinda tricky | 12:05 |
| noonedeadpunk | so I wanted to re-use existing haproxy and controllers, so that each region was going to local to itself keystone only | 12:05 |
| noonedeadpunk | and only galera VMs were completely independent from regions | 12:05 |
| noonedeadpunk | so for `openstack_deploy_keystone` I defined just same controllers like that: | 12:06 |
| grauzikas | sorry need to go away for a short time. so if i will set self provided cert it should work? and letsencrypt didnt helped because it is only for haproxy? | 12:09 |
| opendevreview | Merged openstack/openstack-ansible-repo_server master: Remove references to lsync and rsync https://review.opendev.org/c/openstack/openstack-ansible-repo_server/+/927259 | 12:11 |
| noonedeadpunk | https://paste.openstack.org/show/brzpECoRijzjFryY3F7f/ | 12:13 |
| noonedeadpunk | grauzikas: let's encrypt should have helped. But it could be that the image just uses too old ca-certificates | 12:13 |
| noonedeadpunk | and let's encrypt changed it's root couple of times relatively recently | 12:14 |
| noonedeadpunk | but also you could jsut disable CA verification in overrides as well | 12:14 |
| noonedeadpunk | gokhan: but in fact there're kinda more overrides to make... It's really tough to layout all nuances in IRC without showing drawings and without having any architecture in mind | 12:15 |
| noonedeadpunk | But what I can totally tell is that osa is really flexible enough to support such crazy userstories | 12:15 |
| grauzikas | ok will try again letsencrypt. and reinstall everything again. was trying various ways :) : https://paste.openstack.org/show/bAXJrgGZbmXGxzhKdWzA/ | 12:17 |
| gokhan | thanks noonedeadpunk, thanks it seems ı got it. it is very benefical for me. I will try it. If I have issues, I will share it with you :) | 12:21 |
| jrosser | grauzikas: i think you created the overrides incorrectly for magnum openstack_ca_file there | 12:28 |
| grauzikas | yes yes i notices that, just spent a lot of time by figuring this out :) | 12:29 |
| jrosser | openstack_ca_file is a setting in magnum.conf, not a variable for openstack-ansible | 12:29 |
| noonedeadpunk | I shared some sample paste above for this | 12:30 |
| grauzikas | btw i have seen this somewhere, but cant find again. How i can append list of default packages for lxc containers? i want to add tools like net-tools, nano, wget, tcpdump - now all the time i need to install them after reinstall of system - and doing reinstall quite often while testing how OSA works | 12:30 |
| grauzikas | jrosser, nooonedeadpunk yes i already got this will try soon | 12:30 |
| jrosser | here is how to add extra packages to the base container image https://github.com/openstack/openstack-ansible-lxc_hosts/blob/master/defaults/main.yml#L165 | 12:31 |
| jrosser | though you would need to update the base image, then delete/recreate and existing containers for that to take effect | 12:31 |
| opendevreview | Merged openstack/openstack-ansible-os_neutron master: Ensure proper permissions for OVN Metadata service https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/927183 | 12:32 |
| jrosser | you can also set some packages here https://github.com/openstack/openstack-ansible-openstack_hosts/blob/master/defaults/main.yml#L142 | 12:32 |
| jrosser | those would be installed whenever the openstack_hosts role is run, which you could do for existing containers | 12:32 |
| grauzikas | and syntax “package1, packag2” or - package1 \n - package2 ? \n new line cant use in irc new lines :) | 12:44 |
| noonedeadpunk | it's a list, so new lines | 12:44 |
| noonedeadpunk | second option :) | 12:44 |
| grauzikas | yes yes understud :) thanks | 12:44 |
| opendevreview | Merged openstack/openstack-ansible-ops master: Add support for deploying mcapi control plane k8s on rocky linux https://review.opendev.org/c/openstack/openstack-ansible-ops/+/923447 | 13:01 |
| opendevreview | Merged openstack/openstack-ansible master: Properly apply `always` tag to install_defaults https://review.opendev.org/c/openstack/openstack-ansible/+/924938 | 13:06 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-ops master: Revert "Ensure that python3-cryptography is present in k8s control plane hosts" https://review.opendev.org/c/openstack/openstack-ansible-ops/+/921400 | 13:47 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_octavia master: Provide better flexability for SSH keypair options https://review.opendev.org/c/openstack/openstack-ansible-os_octavia/+/927728 | 13:51 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_magnum master: Define lock directory for oslo_concurrency https://review.opendev.org/c/openstack/openstack-ansible-os_magnum/+/921690 | 14:07 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: Use infrastructure playbooks from openstack-ansible-plugins repo https://review.opendev.org/c/openstack/openstack-ansible/+/924253 | 14:09 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: Use hosts setup playbooks from openstack-ansible-plugins repo https://review.opendev.org/c/openstack/openstack-ansible/+/924259 | 14:11 |
| opendevreview | Merged openstack/openstack-ansible-ops master: Add support for deploying mcapi control plane k8s on debian-12 https://review.opendev.org/c/openstack/openstack-ansible-ops/+/923586 | 15:00 |
| jrosser | noonedeadpunk: do we need to do anything about these? https://review.opendev.org/q/topic:%22remove-wsgi_scripts%22 | 15:01 |
| noonedeadpunk | oh yes, we do | 15:02 |
| noonedeadpunk | crap | 15:02 |
| jrosser | SHA bump time and see what breaks | 15:02 |
| noonedeadpunk | I totally missed that and thought it didn't make into the release | 15:03 |
| noonedeadpunk | All breaks then.... | 15:03 |
| noonedeadpunk | basically this what nicely covers what we need to https://review.opendev.org/c/openstack/nova/+/902687/4/releasenotes/notes/add-nova-wsgi-module-3cc250a78fef7365.yaml#19 | 15:04 |
| noonedeadpunk | and we should be able to do that before bump for some services | 15:04 |
| opendevreview | Merged openstack/openstack-ansible master: Skip importing haproxy_service_config with no haproxy hosts https://review.opendev.org/c/openstack/openstack-ansible/+/924308 | 16:56 |
| noonedeadpunk | it seems these 2 things are completely borked now: https://opendev.org/openstack/openstack-ansible-os_neutron/src/branch/master/handlers/main.yml#L33-L75 | 17:16 |
| noonedeadpunk | I don't really see any processes for `neutron-ns-meta` (as these are named differetly now) | 17:17 |
| noonedeadpunk | 1. I'm really not sure what `pgrep neutron-ns-meta` should be catching - I don't see anything anywhere | 17:46 |
| noonedeadpunk | 2. cat /sys/fs/cgroup/pids/neutron.slice/neutron-l3-agent.service/cgroup.procs does not exist anymore | 17:46 |
| noonedeadpunk | it should be /sys/fs/cgroup/neutron.slice/neutron-l3-agent.service/cgroup.procs I assume | 17:46 |
| noonedeadpunk | or at least depend on OS | 17:46 |
| jrosser | are those things breaking with a sha bump? | 19:09 |
| noonedeadpunk | no, not really. I just spotted plenty of atnelope processes on net nodes after upgrade to 2024.1 | 19:11 |
| *** priteau_ is now known as priteau | 19:12 | |
| noonedeadpunk | and issues with routers update... | 19:12 |
| jrosser | hrmm | 19:18 |
| jrosser | i wonder if that is something for the healthcheck playbook | 19:19 |
| jrosser | to test if there are any processes running from old version venvs | 19:19 |
| noonedeadpunk | well it would mandate running such playbook then | 19:19 |
| noonedeadpunk | And I'd guess this would be quite specific to neutron with ovs/lxb due to our KillMode: process | 19:20 |
| jrosser | i was thinking that the upgrade jobs would catch it | 19:20 |
| noonedeadpunk | So I guess I'd rather fixed handlers with commands that can result in success | 19:21 |
| jrosser | but yes you're right, old type networking | 19:21 |
| noonedeadpunk | > upgrade jobs would catch it - ah, that makes sense indeed | 19:21 |
| noonedeadpunk | I thought you was talking about moving handlers to a healthcheck playbook from neutron overall :D | 19:22 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_neutron stable/2024.1: Ensure proper permissions for OVN Metadata service https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/927761 | 21:15 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!
