| noonedeadpunk | mornings | 07:51 |
|---|---|---|
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-galera_server master: Bump galera version to 10.11.5 https://review.opendev.org/c/openstack/openstack-ansible-galera_server/+/894740 | 07:53 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_neutron master: Fix user for neutron-ovn-metadata-agent https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/870857 | 07:55 |
| jrosser | good morning | 07:58 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_neutron master: Workaround ovs bug that resets hostname with add command https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/897656 | 07:59 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: Add CI jobs for debian bookworm https://review.opendev.org/c/openstack/openstack-ansible/+/894561 | 07:59 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-rabbitmq_server master: Update RabbitMQ to 3.12 series https://review.opendev.org/c/openstack/openstack-ansible-rabbitmq_server/+/895804 | 08:02 |
| noonedeadpunk | really quite weird amount of timeouts. At the same time, I see some upgrade jobs taking 2h and being far from timeouting | 08:06 |
| noonedeadpunk | Looks a bit like heavily loaded provider, but not sure | 08:06 |
| noonedeadpunk | would be sweet to have eyes on https://review.opendev.org/c/openstack/openstack-ansible-ceph_client/+/897046 - this is not really hitting default setup, only if you wanna re-use the role and have keyrings from files. As then you might not have cinder keyring in the folder which will cause role failure | 08:16 |
| noonedeadpunk | (when you run it against nova_compute). Ie, we have that to setup and provision cephfs keyrings | 08:17 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_nova master: Fix example playbook linters https://review.opendev.org/c/openstack/openstack-ansible-os_nova/+/897659 | 08:25 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_nova master: Add barbican_service_user section https://review.opendev.org/c/openstack/openstack-ansible-os_nova/+/897562 | 08:25 |
| noonedeadpunk | and that one is nasty as well ^ | 08:29 |
| opendevreview | Merged openstack/openstack-ansible-os_rally stable/xena: Include proper commit in rally_upper_constraints_url https://review.opendev.org/c/openstack/openstack-ansible-os_rally/+/897447 | 08:53 |
| opendevreview | Andrew Bonney proposed openstack/openstack-ansible-ceph_client master: Add AppArmor configuration for ceph read/write caching https://review.opendev.org/c/openstack/openstack-ansible-ceph_client/+/897661 | 09:03 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-ops master: Update distros for CI https://review.opendev.org/c/openstack/openstack-ansible-ops/+/884637 | 09:39 |
| jrosser | we have q bunch of DISK_FULL for rocky upgrade jobs on 2023.1 | 09:48 |
| noonedeadpunk | somehow that's still old ovs getting there I believe. at least according to logs. Though I couldn't reproduce that behaviour somehow.... | 09:56 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: Remove common nova playbook https://review.opendev.org/c/openstack/openstack-ansible/+/897568 | 10:23 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: Run nova db post setup from nova playbook https://review.opendev.org/c/openstack/openstack-ansible/+/897570 | 10:24 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_nova master: Run nova_db_post_setup from playbook directly https://review.opendev.org/c/openstack/openstack-ansible-os_nova/+/897571 | 10:24 |
| opendevreview | Andrew Bonney proposed openstack/openstack-ansible-ceph_client master: Add AppArmor configuration for ceph read/write caching https://review.opendev.org/c/openstack/openstack-ansible-ceph_client/+/897661 | 10:31 |
| kleini | I am currently on 26.1.2 and I see a lot of timeouts happening on MariaDB connections. I see them in the MariaDB logs from various OpenStack service and especially Nova often loses the SSL connection during query execution. Do you have any ideas, what could causing this? | 10:47 |
| noonedeadpunk | Well loosing connection and loosing during query are 2 different things... While just loosing connection might be okeyish (as nova might have longer or same timeout for keeping pool), this should never happen during queries | 11:05 |
| noonedeadpunk | have you checked that VIP(keepalived) is not flapping? | 11:05 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: Remove common nova playbook https://review.opendev.org/c/openstack/openstack-ansible/+/897568 | 11:15 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: Run nova db post setup from nova playbook https://review.opendev.org/c/openstack/openstack-ansible/+/897570 | 11:15 |
| opendevreview | Merged openstack/openstack-ansible-os_nova stable/zed: Config has changed for pci passthrough. https://review.opendev.org/c/openstack/openstack-ansible-os_nova/+/893774 | 11:17 |
| kleini | noonedeadpunk, journalctl does not have any logs for keepalived, but has some after restarting on secondary node. So, I assume keepalived and VIP are stable. | 11:32 |
| opendevreview | Merged openstack/openstack-ansible-haproxy_server stable/2023.1: Add ability to have different backend port. https://review.opendev.org/c/openstack/openstack-ansible-haproxy_server/+/896768 | 11:37 |
| opendevreview | Andrew Bonney proposed openstack/openstack-ansible-ceph_client master: Add AppArmor configuration for ceph read/write caching https://review.opendev.org/c/openstack/openstack-ansible-ceph_client/+/897661 | 11:54 |
| noonedeadpunk | kleini: and haproxy also doesn't switch master node back and forth? | 12:14 |
| noonedeadpunk | as haproxy can mark a backend down for example during backups | 12:14 |
| kleini | nothing in haproxy logs regarding nodes going away | 12:17 |
| opendevreview | Merged openstack/openstack-ansible-lxc_hosts master: Sync additional apt config from the host to the container base image https://review.opendev.org/c/openstack/openstack-ansible-lxc_hosts/+/894556 | 12:19 |
| noonedeadpunk | then I'm not really sure what can be going on there.... | 12:19 |
| opendevreview | Christian Rohmann proposed openstack/openstack-ansible-galera_server stable/yoga: Added vars to override systemd for mariabackup https://review.opendev.org/c/openstack/openstack-ansible-galera_server/+/897665 | 12:38 |
| opendevreview | Merged openstack/openstack-ansible-os_neutron stable/2023.1: Fix conditional for non-OVN deployments https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/897049 | 13:15 |
| opendevreview | Merged openstack/openstack-ansible-haproxy_server master: Apply haproxy-service-config tag on include https://review.opendev.org/c/openstack/openstack-ansible-haproxy_server/+/896569 | 13:18 |
| opendevreview | Merged openstack/openstack-ansible-haproxy_server master: Add tags to PKI include https://review.opendev.org/c/openstack/openstack-ansible-haproxy_server/+/896612 | 13:24 |
| opendevreview | Merged openstack/openstack-ansible-galera_server stable/zed: Added vars to override systemd for mariabackup https://review.opendev.org/c/openstack/openstack-ansible-galera_server/+/896407 | 13:30 |
| opendevreview | Merged openstack/openstack-ansible-rabbitmq_server master: Update RabbitMQ to 3.12 series https://review.opendev.org/c/openstack/openstack-ansible-rabbitmq_server/+/895804 | 13:41 |
| NeilHanlon | is anyone looking at https://bugs.launchpad.net/openstack-ansible/+bug/2038826 yet? i can take a swing at updating that URL.. | 13:58 |
| noonedeadpunk | it looks like they've just dropped older versions from there? | 14:02 |
| noonedeadpunk | I don't think anyone is on that right now | 14:02 |
| noonedeadpunk | ah, it's in archive | 14:04 |
| noonedeadpunk | NeilHanlon: feel free to :) | 14:04 |
| opendevreview | Neil Hanlon proposed openstack/ansible-role-zookeeper master: update zookeeper to 3.7.2 https://review.opendev.org/c/openstack/ansible-role-zookeeper/+/897672 | 14:05 |
| opendevreview | Neil Hanlon proposed openstack/ansible-role-zookeeper master: update zookeeper to 3.7.2 https://review.opendev.org/c/openstack/ansible-role-zookeeper/+/897672 | 14:06 |
| noonedeadpunk | NeilHanlon: I though you will update URL to archive.apache.org/dist/zookeeper :) | 14:11 |
| noonedeadpunk | which contains everything | 14:11 |
| jrosser | we need a change thats backportable ideally | 14:12 |
| jrosser | then can separately update the version on master | 14:12 |
| NeilHanlon | yea i was going to look and see if the archive had all the versions, but didn't click it yet lol | 14:13 |
| NeilHanlon | was already half through the change :P | 14:13 |
| opendevreview | Neil Hanlon proposed openstack/ansible-role-zookeeper master: change to using archive.apache.org for source downloads https://review.opendev.org/c/openstack/ansible-role-zookeeper/+/897673 | 14:15 |
| opendevreview | Neil Hanlon proposed openstack/ansible-role-zookeeper master: update zookeeper to 3.7.2 https://review.opendev.org/c/openstack/ansible-role-zookeeper/+/897672 | 14:19 |
| NeilHanlon | there we g | 14:20 |
| NeilHanlon | o | 14:20 |
| NeilHanlon | oh i should have stacked them, eh? | 14:20 |
| jrosser | it'll be merge confict for one of them if you dont | 14:21 |
| noonedeadpunk | NeilHanlon: eventually, I think on master we can update to 3.8.3? | 14:26 |
| opendevreview | Merged openstack/openstack-ansible-os_nova master: Fix example playbook linters https://review.opendev.org/c/openstack/openstack-ansible-os_nova/+/897659 | 14:26 |
| NeilHanlon | noonedeadpunk: i think that makes sense, yeah -- but not sure if we want that for 2023.2 or not | 14:26 |
| opendevreview | Merged openstack/openstack-ansible-os_nova master: Add barbican_service_user section https://review.opendev.org/c/openstack/openstack-ansible-os_nova/+/897562 | 14:27 |
| noonedeadpunk | we have couple of weeks to check on that, so I think it's not too late | 14:28 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_nova stable/2023.1: Add barbican_service_user section https://review.opendev.org/c/openstack/openstack-ansible-os_nova/+/897457 | 14:28 |
| NeilHanlon | sounds good! | 14:28 |
| opendevreview | Neil Hanlon proposed openstack/ansible-role-zookeeper master: update zookeeper to 3.7.2 https://review.opendev.org/c/openstack/ansible-role-zookeeper/+/897672 | 14:34 |
| opendevreview | Neil Hanlon proposed openstack/ansible-role-zookeeper master: update zookeeper to 3.8.3 https://review.opendev.org/c/openstack/ansible-role-zookeeper/+/897672 | 14:37 |
| noonedeadpunk | I wonder how flawless upgrade of zookeeper is :D | 14:52 |
| jrosser | noonedeadpunk: it's a bit of a nit but this isnt from UCA? https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/897656 | 15:25 |
| noonedeadpunk | ah, I forgot to edit comment | 15:26 |
| noonedeadpunk | true | 15:26 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_neutron master: Workaround ovs bug that resets hostname with add command https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/897656 | 15:26 |
| -opendevstatus- NOTICE: The Gerrit service on review.opendev.org will be offline momentarily while we restart it for a combined runtime and platform upgrade | 16:32 | |
| opendevreview | Dmitriy Rabotyagov proposed openstack/ansible-role-python_venv_build master: Use distribution_major_version for Debian and CentOS https://review.opendev.org/c/openstack/ansible-role-python_venv_build/+/897701 | 16:47 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: Add CI jobs for debian bookworm https://review.opendev.org/c/openstack/openstack-ansible/+/894561 | 16:47 |
| NeilHanlon | noonedeadpunk, jrosser: so.. ran into an ansible-lint problem on an unrelated file for https://review.opendev.org/c/openstack/ansible-role-zookeeper/+/897673 -- should I fix in its own commit? | 16:47 |
| opendevreview | Merged openstack/openstack-ansible stable/xena: Use github to fetch ansible collections https://review.opendev.org/c/openstack/openstack-ansible/+/897356 | 16:48 |
| NeilHanlon | feels like yes, but also feels like more backporting work :\ | 16:48 |
| noonedeadpunk | yeah.... | 16:48 |
| jrosser | ah yes there are whole bunch of patches like that | 16:48 |
| NeilHanlon | it's fine.. i need more practice lol | 16:48 |
| noonedeadpunk | but this example playbook doesn't need backporting | 16:48 |
| noonedeadpunk | it's smth we break on master by using latest ansible-lint | 16:48 |
| noonedeadpunk | such rules simply didn't exist before | 16:49 |
| NeilHanlon | ah.. i wasn't sure if we'd end up seeing it on other branches too | 16:49 |
| noonedeadpunk | nah, only on master | 16:49 |
| NeilHanlon | in that case, i'll just submit it and stack the other changes on top | 16:49 |
| noonedeadpunk | ++ | 16:49 |
| opendevreview | Neil Hanlon proposed openstack/ansible-role-zookeeper master: change to using archive.apache.org for source downloads https://review.opendev.org/c/openstack/ansible-role-zookeeper/+/897673 | 16:49 |
| opendevreview | Neil Hanlon proposed openstack/ansible-role-zookeeper master: update zookeeper to 3.8.3 https://review.opendev.org/c/openstack/ansible-role-zookeeper/+/897672 | 16:49 |
| opendevreview | Neil Hanlon proposed openstack/ansible-role-zookeeper master: chore: fix linter for example playbook https://review.opendev.org/c/openstack/ansible-role-zookeeper/+/897702 | 16:49 |
| jrosser | noonedeadpunk: is there a way we can improve on this? https://github.com/openstack/openstack-ansible/blob/master/scripts/get-ansible-role-requirements.yml#L108 | 16:50 |
| jrosser | there a quite a few jobs failed downloading of u-c | 16:50 |
| jrosser | and they are almost always upgrade jobs | 16:51 |
| noonedeadpunk | NeilHanlon: btw, it seems that checksum is wrong here https://review.opendev.org/c/openstack/ansible-role-zookeeper/+/897672 | 16:51 |
| NeilHanlon | ah, thanks. i must've made a mistake when rebasing | 16:51 |
| noonedeadpunk | jrosser: well.... then we need to store state of all zuul repos somewhere, and check them back on upgrade before second bootstrap | 16:52 |
| jrosser | yes i was just looking at https://github.com/openstack/openstack-ansible/commit/c7ab46f620243d6fb0bf86e656007b512dd82bce | 16:53 |
| jrosser | and not really remembering what a mess there is | 16:53 |
| noonedeadpunk | so, after upgrade we wanna use repos prepared by zuul, while before upgrade take basically what;s in a-r-r | 16:54 |
| noonedeadpunk | as iirc you can't really checkout easily back if there's a depends-on or smth like that | 16:55 |
| noonedeadpunk | but not sure about the last one | 16:55 |
| NeilHanlon | hrm - that's the checksum i am getting.. | 16:55 |
| noonedeadpunk | was that sha512? | 16:57 |
| NeilHanlon | ye | 16:57 |
| NeilHanlon | 4f68434a55a93b88e422029a8583ebdc060e1cc2aed38cdc0170c5612be6f6209947d10b488eae671ff7279201c166d34095b0be446b930a918a24f509b159e5 apache-zookeeper-3.8.3.tar.gz | 16:57 |
| noonedeadpunk | as I'm getting eb1a5e79fe55fa50e36021ca8b752e40584d5e24f23985c43e9f56abfbf23c0239dce37d18f27213c9a38fcb6f68daf1a9468fc352e6ac2364faaf9b56145bdf | 16:57 |
| noonedeadpunk | https://paste.openstack.org/show/boAaxKE9VRU8EmNiam9U/ | 16:57 |
| * NeilHanlon slaps self in face | 16:58 | |
| NeilHanlon | -bin :) | 16:58 |
| noonedeadpunk | yeah) | 16:58 |
| noonedeadpunk | was just writting that ) | 16:58 |
| opendevreview | Neil Hanlon proposed openstack/ansible-role-zookeeper master: update zookeeper to 3.8.3 https://review.opendev.org/c/openstack/ansible-role-zookeeper/+/897672 | 16:59 |
| NeilHanlon | reading is hard | 16:59 |
| NeilHanlon | stay in school, kids | 16:59 |
| noonedeadpunk | I'm really not getting why https://review.opendev.org/c/openstack/openstack-ansible-galera_server/+/894740 fails on upgrade for jammy... And what 10.6 is doing there.... It's some nonsense... | 17:01 |
| noonedeadpunk | and what has happened to gluster for rocky? | 17:01 |
| noonedeadpunk | fatal: [aio1_repo_container-cd5e9530]: FAILED! => {"changed": false, "msg": "Failed to download packages: No URLs in mirrorlist", "results": []} | 17:01 |
| noonedeadpunk | Ah, I guess that's old good infra issue? | 17:02 |
| jrosser | i guess my upgrade question is really about if we really need to change this between source and target branches https://codesearch.opendev.org/?q=openstack_opendev_base_url&i=nope&literal=nope&files=&excludeFiles=&repos= | 17:04 |
| jrosser | like why do we need specifically the opendev.org version on the source branch and on-disk version of all that for the target branch | 17:05 |
| noonedeadpunk | well, because otherwise on-disk repos will be checkouted to N-1 version I guess? | 17:11 |
| noonedeadpunk | and then you don't know how to check-out them back | 17:12 |
| noonedeadpunk | unless do some rsync to the directory nearby as alternative path | 17:12 |
| noonedeadpunk | and defining it as place on disk | 17:12 |
| jrosser | but those are the openstack services | 17:12 |
| jrosser | which we install with pip and a git sha | 17:12 |
| noonedeadpunk | do we? | 17:13 |
| noonedeadpunk | I thought we override them as well for some reason | 17:13 |
| noonedeadpunk | https://opendev.org/openstack/openstack-ansible/src/branch/master/scripts/get-ansible-role-requirements.yml#L103-L104 | 17:13 |
| noonedeadpunk | ah, or you mean they're not "broken" by installation | 17:14 |
| noonedeadpunk | and we don't respect depnds-on either there | 17:14 |
| jrosser | yeah | 17:15 |
| jrosser | it's why is the task you linked controlled by this when: https://opendev.org/openstack/openstack-ansible/src/branch/master/scripts/get-ansible-role-requirements.yml#L108 | 17:15 |
| jrosser | i don't see what that does | 17:15 |
| noonedeadpunk | But I somehow was under impression that we do.... I can be really wrong here | 17:15 |
| jrosser | all the complexity in that block: is dealing with a-r-r | 17:16 |
| noonedeadpunk | yeah, then openstack_opendev_base_url should be right out of the block but still having `- "lookup('env', 'ZUUL_SRC_PATH') != ''"` condition? | 17:17 |
| noonedeadpunk | ok, yes, I think you're right here | 17:18 |
| jrosser | so so i have another question :( | 17:19 |
| noonedeadpunk | these are indeed not very related 2 things | 17:19 |
| jrosser | here we symlink https://opendev.org/openstack/openstack-ansible/src/branch/master/scripts/get-ansible-role-requirements.yml#L73 | 17:19 |
| jrosser | and here we clone https://opendev.org/openstack/openstack-ansible/src/branch/master/scripts/get-ansible-role-requirements.yml#L132 | 17:19 |
| jrosser | and i am wtf | 17:19 |
| jrosser | clearly i am missing something here | 17:20 |
| noonedeadpunk | like why not to symlink directly to dest? | 17:20 |
| jrosser | oh wait | 17:21 |
| noonedeadpunk | Ah, but we have clone_roles | 17:21 |
| jrosser | clone_roles excludes things that are linked | 17:21 |
| noonedeadpunk | So what's in zuul is not cloned | 17:21 |
| noonedeadpunk | yes | 17:21 |
| noonedeadpunk | :) | 17:21 |
| jrosser | more comments needed i think | 17:21 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible master: Always use on-disk openstack service git repos in CI jobs https://review.opendev.org/c/openstack/openstack-ansible/+/897707 | 17:29 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible master: Always use on-disk openstack service git repos in CI jobs https://review.opendev.org/c/openstack/openstack-ansible/+/897707 | 17:31 |
| NeilHanlon | noonedeadpunk: gluster for rocky comes from centos, right? | 17:42 |
| NeilHanlon | i do know fedora has been having some issues recently.. i wonder if related | 17:43 |
| noonedeadpunk | ah, might be from centos indeed | 18:12 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/ansible-role-python_venv_build master: Use distribution_major_version for Debian and CentOS https://review.opendev.org/c/openstack/ansible-role-python_venv_build/+/897701 | 18:18 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_nova stable/zed: Add barbican_service_user section https://review.opendev.org/c/openstack/openstack-ansible-os_nova/+/897721 | 18:25 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_nova stable/yoga: Add barbican_service_user section https://review.opendev.org/c/openstack/openstack-ansible-os_nova/+/897722 | 18:25 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_nova stable/xena: Add barbican_service_user section https://review.opendev.org/c/openstack/openstack-ansible-os_nova/+/897723 | 18:25 |
| noonedeadpunk | I clean forgot if we've backported journalctl trimm back to 2023.1.... | 18:27 |
| noonedeadpunk | Seems, that massive curl update is pending... https://www.phoronix.com/news/Curl-8.4-Coming | 18:48 |
| noonedeadpunk | Massive in terms of amount of things that needs to be updated ofc | 18:48 |
| * NeilHanlon cries | 18:50 | |
| NeilHanlon | who's buying us the beers | 18:51 |
| NeilHanlon | oof. we have a cycle between 897702 and and 897673 now. | 18:56 |
| NeilHanlon | i.e., 897702 can't pass CI because of 897673, and the reverse is also true | 18:57 |
| jrosser | it's probably reasonable to squash those two together as they would both backport cleanly | 19:01 |
| jrosser | and the example lint fix isnt going to have any risk on a stable branch | 19:01 |
| jrosser | alternatively you can disable the broken CI test in one of them and propose a revert directly afterwards, but thats more work | 19:02 |
| opendevreview | Neil Hanlon proposed openstack/ansible-role-zookeeper master: update zookeeper to 3.8.3 https://review.opendev.org/c/openstack/ansible-role-zookeeper/+/897672 | 19:09 |
| opendevreview | Neil Hanlon proposed openstack/ansible-role-zookeeper master: change to using archive.apache.org for source downloads https://review.opendev.org/c/openstack/ansible-role-zookeeper/+/897673 | 19:12 |
| opendevreview | Neil Hanlon proposed openstack/ansible-role-zookeeper master: update zookeeper to 3.8.3 https://review.opendev.org/c/openstack/ansible-role-zookeeper/+/897672 | 19:13 |
| opendevreview | Merged openstack/openstack-ansible-ceph_client master: Add AppArmor configuration for ceph read/write caching https://review.opendev.org/c/openstack/openstack-ansible-ceph_client/+/897661 | 20:55 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!