| * noonedeadpunk looking at status of https://review.opendev.org/q/topic:%22bug/2071734%22 | 08:02 | |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: [doc] Add support for Ubuntu 24.04 to docs https://review.opendev.org/c/openstack/openstack-ansible/+/924829 | 09:13 |
|---|---|---|
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-lxc_hosts stable/2024.1: Fix incorrect copying of sources.list.d to container image https://review.opendev.org/c/openstack/openstack-ansible-lxc_hosts/+/924830 | 09:15 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_cinder stable/2024.1: Disable heartbeat_in_pthread by default https://review.opendev.org/c/openstack/openstack-ansible-os_cinder/+/924831 | 09:17 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_cinder stable/2023.2: Disable heartbeat_in_pthread by default https://review.opendev.org/c/openstack/openstack-ansible-os_cinder/+/924832 | 09:18 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_cinder stable/2023.1: Disable heartbeat_in_pthread by default https://review.opendev.org/c/openstack/openstack-ansible-os_cinder/+/924833 | 09:19 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_nova stable/2024.1: Disable heartbeat_in_pthread by default https://review.opendev.org/c/openstack/openstack-ansible-os_nova/+/924834 | 09:19 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_nova stable/2023.2: Disable heartbeat_in_pthread by default https://review.opendev.org/c/openstack/openstack-ansible-os_nova/+/924835 | 09:19 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_nova stable/2023.1: Disable heartbeat_in_pthread by default https://review.opendev.org/c/openstack/openstack-ansible-os_nova/+/924836 | 09:20 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/ansible-role-python_venv_build master: Don't use local pip cache when re-building wheels https://review.opendev.org/c/openstack/ansible-role-python_venv_build/+/924761 | 09:23 |
| opendevreview | Merged openstack/openstack-ansible unmaintained/zed: Update .gitreview for unmaintained/zed https://review.opendev.org/c/openstack/openstack-ansible/+/924139 | 09:46 |
| opendevreview | Merged openstack/openstack-ansible-galera_server master: Remove installation of libaio1 https://review.opendev.org/c/openstack/openstack-ansible-galera_server/+/924355 | 09:49 |
| opendevreview | Merged openstack/openstack-ansible-lxc_hosts master: Add libpython entry for Ubuntu Noble https://review.opendev.org/c/openstack/openstack-ansible-lxc_hosts/+/924310 | 10:05 |
| opendevreview | Merged openstack/openstack-ansible-lxc_hosts master: Install systemd-resolved into container base image for Ubuntu Noble https://review.opendev.org/c/openstack/openstack-ansible-lxc_hosts/+/924339 | 10:15 |
| frickler | jrosser: noonedeadpunk: can we force-merge https://review.opendev.org/c/openstack/openstack-ansible-tests/+/922408 or do you plan to repair the jammy job soon? this is blocking further opendev cleanup | 10:17 |
| opendevreview | Merged openstack/openstack-ansible-os_keystone unmaintained/zed: Remove X-Forwarded-Proto header in apache https://review.opendev.org/c/openstack/openstack-ansible-os_keystone/+/922642 | 10:59 |
| noonedeadpunk | frickler: I was plannning to fix that on master... | 11:25 |
| noonedeadpunk | let me check this out now actually | 11:26 |
| noonedeadpunk | I was working on this some time ago, but obviously need to revise now https://review.opendev.org/c/openstack/openstack-ansible-tests/+/921434 | 11:27 |
| frickler | noonedeadpunk: thx, as long as it is moving forward, I'm fine for now. this is where you can see the config error appearing btw. https://review.opendev.org/c/opendev/base-jobs/+/922653 | 11:29 |
| jrosser | noonedeadpunk: we do not affect the fixing of the jammy job by force merging the centos-8-stream removal? | 11:33 |
| noonedeadpunk | well, no, but then there's no motivation to fix it lol | 11:33 |
| noonedeadpunk | and always better to fix then force-merge... | 11:33 |
| noonedeadpunk | imo | 11:33 |
| jrosser | once there is no centos-8-stream nodes we will run no CI on the affected branches | 11:33 |
| jrosser | which is kind of even worse | 11:33 |
| noonedeadpunk | yeah. true | 11:36 |
| noonedeadpunk | but we can force merge these already I guess? https://review.opendev.org/c/openstack/openstack-ansible-os_zun/+/923974 / https://review.opendev.org/c/openstack/openstack-ansible-lxc_hosts/+/922409 ? | 11:38 |
| opendevreview | Merged openstack/openstack-ansible-galera_server master: Update to version 10.11.8 https://review.opendev.org/c/openstack/openstack-ansible-galera_server/+/924357 | 11:57 |
| noonedeadpunk | jrosser: so, why https://review.opendev.org/c/openstack/openstack-ansible-galera_server/+/922377 is abandoned? I guess we still wanna have 11.4 by the release date? | 12:10 |
| noonedeadpunk | given that 11.4.3 will fix the issue? | 12:10 |
| jrosser | argh my mistage | 12:11 |
| jrosser | mistake | 12:11 |
| jrosser | i decided the same thing and didnt cancel the abandon dialog properly | 12:11 |
| noonedeadpunk | we can leave it this way for now ofc | 12:11 |
| noonedeadpunk | and restore when new release is out | 12:11 |
| jrosser | so i think we have a circular dependancy between https://review.opendev.org/c/openstack/openstack-ansible/+/924661 | 12:20 |
| jrosser | oh | 12:21 |
| jrosser | i was going to say https://review.opendev.org/c/openstack/openstack-ansible-lxc_hosts/+/924340 | 12:21 |
| jrosser | but that doesnt seem to be the case | 12:21 |
| noonedeadpunk | yeah, seems these 2 work together? | 12:22 |
| jrosser | once 924340 merges probably 924661 will pass | 12:22 |
| jrosser | yes it's almost two different sides to the same change | 12:22 |
| noonedeadpunk | huh, so https://review.opendev.org/c/openstack/openstack-ansible-tests/+/921434 just passed in a sandbox | 12:27 |
| noonedeadpunk | now I start recalling why I stopped previously with that patch :D | 12:37 |
| jrosser | well there is one possible thing | 12:40 |
| jrosser | see how these tasks run straight one into the next https://opendev.org/openstack/openstack-ansible-lxc_container_create/src/branch/master/tasks/lxc_container_config.yml#L55-L63 | 12:41 |
| jrosser | but we make a very specific step here to make sure the container is ready https://opendev.org/openstack/openstack-ansible-lxc_container_create/src/branch/master/handlers/main.yml#L40-L59 | 12:41 |
| noonedeadpunk | well, it's very confusing error then, as it shouldn't fail with `ssh: root@localhost: Permission denied (publickey)` then | 12:43 |
| noonedeadpunk | and ssh keys are created/distributed: https://zuul.opendev.org/t/openstack/build/cd9da1010d33419b8c1395283b963089/log/job-output.txt#4199-4218 | 12:43 |
| noonedeadpunk | with https://opendev.org/openstack/openstack-ansible-tests/src/branch/master/test-prepare-keys.yml#L20-L63 | 12:44 |
| noonedeadpunk | there can be smth connection related ofc.... | 12:44 |
| jrosser | hmm yes the error message is odd | 12:44 |
| jrosser | but we do definatley miss the step to wait for the container there | 12:44 |
| jrosser | which does a couple of things, does some sanity check, but also has retries to allow it to take some time / iterations | 12:45 |
| noonedeadpunk | I never catched any issues at this step... and haven't seen anything in aios either? | 12:45 |
| jrosser | no me neither | 12:45 |
| noonedeadpunk | while it would be good to have some wait indeed | 12:46 |
| jrosser | unless something in the CI jobs is super slow to boot them | 12:46 |
| jrosser | storage or whatever | 12:46 |
| jrosser | though wierdly, those "wait for ready" tasks had to be inserted because the container came up very quickly for ssh | 12:47 |
| noonedeadpunk | I'd kinda more suspicious about that: https://zuul.opendev.org/t/openstack/build/cd9da1010d33419b8c1395283b963089/log/job-output.txt#4153-4155 | 12:47 |
| jrosser | but systemd had not completed starting all the things, so there were some super wierd errors | 12:47 |
| jrosser | ewwww i wonder what that is | 12:48 |
| noonedeadpunk | I have same locally though | 12:50 |
| noonedeadpunk | this all somehow smells like sessions persistance or smth like that. where connection doesn't know about new keypair it needs to use | 12:51 |
| noonedeadpunk | but how to reproduce that.... | 12:52 |
| noonedeadpunk | and then it seems our plugin disregards local connection to the host as well | 12:56 |
| noonedeadpunk | which is probably doesn't matter much... | 12:58 |
| noonedeadpunk | I guess I'm gonna ask for hold.... | 13:09 |
| noonedeadpunk | I wonder if that has smth to do with permit root login in sshd or smth like that | 13:11 |
| noonedeadpunk | or key length.... | 13:12 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-tests master: Bump Ansible version for tests repo https://review.opendev.org/c/openstack/openstack-ansible-tests/+/921434 | 13:17 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-tests master: Bump Ansible version for tests repo https://review.opendev.org/c/openstack/openstack-ansible-tests/+/921434 | 13:40 |
| opendevreview | Merged openstack/openstack-ansible-lxc_hosts master: Use generated apparmor profile by default in lxc base config https://review.opendev.org/c/openstack/openstack-ansible-lxc_hosts/+/924340 | 13:56 |
| noonedeadpunk | seems like without unconfined debian is not happy? https://review.opendev.org/c/openstack/openstack-ansible/+/924661 | 14:14 |
| jrosser | i think that was never tested with a depends-on of https://review.opendev.org/c/openstack/openstack-ansible-lxc_hosts/+/924340 | 14:17 |
| jrosser | which i missed | 14:18 |
| jrosser | but with the whole stack of patches in place on this it is passing https://review.opendev.org/c/openstack/openstack-ansible/+/924342 | 14:18 |
| jrosser | this is where my concern about circular dependencies was | 14:19 |
| opendevreview | Merged openstack/openstack-ansible-plugins master: Use distribution specific name for ssh service in ssh_keypairs role https://review.opendev.org/c/openstack/openstack-ansible-plugins/+/924755 | 14:36 |
| opendevreview | Merged openstack/ansible-role-uwsgi master: Ensure uWSGI is built with pcre support https://review.opendev.org/c/openstack/ansible-role-uwsgi/+/924754 | 15:23 |
| opendevreview | Merged openstack/openstack-ansible-os_nova stable/2024.1: Disable heartbeat_in_pthread by default https://review.opendev.org/c/openstack/openstack-ansible-os_nova/+/924834 | 15:33 |
| opendevreview | Merged openstack/openstack-ansible-os_neutron master: Allow to nicely control list of L3 agent extensions https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/924769 | 15:36 |
| opendevreview | Merged openstack/openstack-ansible-os_nova stable/2023.2: Disable heartbeat_in_pthread by default https://review.opendev.org/c/openstack/openstack-ansible-os_nova/+/924835 | 15:45 |
| opendevreview | Merged openstack/openstack-ansible-os_nova stable/2023.1: Disable heartbeat_in_pthread by default https://review.opendev.org/c/openstack/openstack-ansible-os_nova/+/924836 | 15:45 |
| noonedeadpunk | btw, seems it might be time to update a logic a bit regarding python2: https://opendev.org/openstack/openstack-ansible/src/branch/master/playbooks/openstack-hosts-setup.yml#L27-L40 | 15:55 |
| jrosser | we can maybe remove that entirely | 15:59 |
| jrosser | is there ever a case python is not installed these days? | 15:59 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-tests master: Bump Ansible version for tests repo https://review.opendev.org/c/openstack/openstack-ansible-tests/+/921434 | 16:04 |
| noonedeadpunk | frankly - not sure | 16:06 |
| noonedeadpunk | but worth at least to fail early if there's none... | 16:06 |
| opendevreview | Merged openstack/ansible-role-python_venv_build master: Don't use local pip cache when re-building wheels https://review.opendev.org/c/openstack/ansible-role-python_venv_build/+/924761 | 16:09 |
| jrosser | oh dear ML post someone deleted/recreated their keystone containers due to upgrade trouble | 16:19 |
| jrosser | there is something pretty unhelpful with keystone-manage failing that it just fails but there is no useful information printed | 16:21 |
| noonedeadpunk | ooops | 16:24 |
| jrosser | i wonder if that is irc user ygk_12345, as there is basically the same question asked in the keystone channel earlier | 16:25 |
| noonedeadpunk | yeah, no idea wtf is that... | 16:25 |
| jrosser | for https://review.opendev.org/c/openstack/openstack-ansible-tests/+/921434 | 16:27 |
| jrosser | could make it fail very very early and run the whole tox command manually | 16:27 |
| opendevreview | Merged openstack/openstack-ansible-os_nova stable/2024.1: Allow to supply multiline overrides to vendor_data https://review.opendev.org/c/openstack/openstack-ansible-os_nova/+/924280 | 16:27 |
| jrosser | so see if it is "first time" issue, or perhaps due to the way the zuul exector runs the test compared to a user ssh in (as root) and run the test | 16:27 |
| jrosser | like root vs. zuul user | 16:28 |
| noonedeadpunk | well, then I likely can to run as ubuntu user on a vm? | 16:37 |
| noonedeadpunk | to mimic behavior more closely | 16:37 |
| jrosser | i would think so | 16:41 |
| jrosser | i'm seeing a few ceph jobs fail related to python 'packaging' module which was discussed elsewhere yesterday w.r.t setuptools update | 16:55 |
| jrosser | but thats surprising if it only affects the ceph job | 16:55 |
| noonedeadpunk | well, we're pinning setuptools for our venv: https://opendev.org/openstack/openstack-ansible/src/branch/master/global-requirement-pins.txt#L11 | 17:03 |
| noonedeadpunk | so, what's interesting, is that `ansible_user_id` is `root`, but $USER playbook is launched is `ubuntu`. | 17:14 |
| noonedeadpunk | so logic of copying ssh key for ubuntu just does not get triggered | 17:17 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-tests master: Bump Ansible version for tests repo https://review.opendev.org/c/openstack/openstack-ansible-tests/+/921434 | 17:20 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-tests master: Bump Ansible version for tests repo https://review.opendev.org/c/openstack/openstack-ansible-tests/+/921434 | 17:39 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/ansible-role-uwsgi stable/2024.1: Ensure uWSGI is built with pcre support https://review.opendev.org/c/openstack/ansible-role-uwsgi/+/924876 | 18:12 |
| noonedeadpunk | crap, seems this is not enough: https://review.opendev.org/c/openstack/ansible-role-uwsgi/+/924754 | 18:13 |
| noonedeadpunk | https://zuul.opendev.org/t/openstack/build/ee0846526232446788f1b602a2f12ab4/log/logs/host/glance-api.service.journal-15-10-21.log.txt#584 | 18:13 |
| noonedeadpunk | damn | 18:14 |
| noonedeadpunk | I got it working in sandbox /o\ | 18:14 |
| noonedeadpunk | that's really quite dissapointing | 18:16 |
| opendevreview | Merged openstack/openstack-ansible stable/2023.1: Use UCA mirror in CI for ubuntu https://review.opendev.org/c/openstack/openstack-ansible/+/924604 | 18:27 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-tests master: Remove legacy centos-8-stream job definition https://review.opendev.org/c/openstack/openstack-ansible-tests/+/922408 | 18:33 |
| noonedeadpunk | https://review.opendev.org/c/openstack/openstack-ansible-tests/+/921434 finally passed | 18:33 |
| noonedeadpunk | fwiw, https://opendev.org/openstack/openstack-ansible/src/branch/stable/2024.1/inventory/group_vars/ceph-rgw.yml#L2-L20 seems to be ignored on ceph-ansible 7.0 | 18:57 |
| noonedeadpunk | as I don't see a variable there | 18:58 |
| mgariepy | noonedeadpunk, seems like uwsgi wasn't configured correctly : see line 72912 of python_venv_build.log.txt | 18:58 |
| mgariepy | via the browser it doesn't quite work for me since the file is kinda .. big. | 19:00 |
| noonedeadpunk | or maybe I'm wrong... | 19:01 |
| noonedeadpunk | yeah | 19:01 |
| noonedeadpunk | I've already spotted that... | 19:01 |
| mgariepy | hehe ok. | 19:01 |
| noonedeadpunk | or well | 19:01 |
| noonedeadpunk | not that specifically... | 19:01 |
| mgariepy | uWSGI configuration seems kinda off from what i see. | 19:02 |
| noonedeadpunk | but kinda venv build is on desitanation? | 19:02 |
| noonedeadpunk | and what I saw is that wheels were built wrongly somehow before | 19:03 |
| noonedeadpunk | crap, file is already 50M lol | 19:04 |
| noonedeadpunk | how big is it? | 19:04 |
| mgariepy | 150ish | 19:04 |
| mgariepy | you need a better internet ! :P haha | 19:05 |
| noonedeadpunk | `pcre = False` | 19:07 |
| noonedeadpunk | https://paste.openstack.org/show/bNlobEoBzukVPib1TWRA/ | 19:08 |
| noonedeadpunk | but frankly - this looks like wheels build is disabled? or well - it does build wheels on package installation? | 19:09 |
| noonedeadpunk | nah, python setup.py bdist_wheel | 19:09 |
| noonedeadpunk | but inside keystone venv? | 19:09 |
| mgariepy | master log same config on uwsgi. | 19:11 |
| noonedeadpunk | yeah, wheels are not built there: https://zuul.opendev.org/t/openstack/build/ee0846526232446788f1b602a2f12ab4/log/job-output.txt#13227-13237 | 19:12 |
| noonedeadpunk | so it's kinda just package installation.... | 19:12 |
| noonedeadpunk | and then - there should be another variable I think to install pcre... | 19:13 |
| mgariepy | hmm yeah | 19:13 |
| noonedeadpunk | but that's good find, thanks! | 19:15 |
| noonedeadpunk | This one I haven't figured out :D | 19:15 |
| noonedeadpunk | (also disregard message about https://opendev.org/openstack/openstack-ansible/src/branch/stable/2024.1/inventory/group_vars/ceph-rgw.yml#L2-L20 - I had weird override) | 19:15 |
| mgariepy | hehe | 19:16 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!