| *** dviroel|brb is now known as dviroel | 00:03 | |
| *** dviroel is now known as dviroel_ | 00:19 | |
| *** dviroel_ is now known as dviroel | 00:19 | |
| *** dviroel is now known as dviroel|out | 01:47 | |
| rohit02 | hi team,as we all know centos 8 repos are EOL due to which centos8 victoria deployment failed | 04:07 |
|---|---|---|
| rohit02 | in deployment doc only centos 8 is supported os and not stream.so do we still support victoria deployment on centos 8 ? | 04:09 |
| rohit02 | noonedeadpunk: ^ | 04:11 |
| jrosser | rohit02: here is what the OS support looks like https://docs.openstack.org/openstack-ansible/latest/admin/upgrades/compatibility-matrix.html | 06:26 |
| jrosser | now that centos-8 is EOL, that column no longer works | 06:26 |
| jrosser | so for centos8, you need to be already on centos8-stream and Wallaby release of openstack-ansible | 06:27 |
| rohit02 | jrosser: so how we can move to wallaby with stream in production environment? Is there any latest doc? | 06:37 |
| jrosser | rohit02: you would follow the major upgrade guide, but re-install each node as you go with centos-8 stream | 06:39 |
| jrosser | this is the most difficult upgrade you could attempt, doing the OS and openstack at the same time | 06:39 |
| noonedeadpunk | spatel for example jsut jumped from centos to ubuntu once realized centos 8 got eoled ;) | 07:05 |
| jrosser | right, and centos-8-stream having EOL python3.6 you know that once you do the centos8 -> centos8-stream it will not be long before you need to move to centos-9 | 07:14 |
| jrosser | argh more mess to fix Status code: 404 for http://mirror.centos.org/centos/8/virt/x86_64/kata-containers/repodata/repomd.xml | 07:17 |
| noonedeadpunk | seems like we need to sytart building kata anyway | 07:29 |
| jrosser | oh what a total mess https://review.opendev.org/c/openstack/openstack-ansible-os_zun/+/827046 | 07:29 |
| noonedeadpunk | as now both centos and debina missing their support | 07:29 |
| jrosser | that "works" but of course the upgrade job breaks, as N-1 branch is also now broken | 07:29 |
| noonedeadpunk | pfffff | 07:29 |
| jrosser | and also circular dep on the previous patch | 07:30 |
| noonedeadpunk | and can we just backport to X it first? | 07:30 |
| noonedeadpunk | or it won't work ther because of cross dependency as well... | 07:30 |
| jrosser | install from snap - thats not cool | 07:31 |
| noonedeadpunk | I actually thought that centos would be last who drop repo for kata | 07:32 |
| noonedeadpunk | yeah either snap or source install | 07:32 |
| jrosser | i still have no idea why anyone thinks it is a good idea to install application/container runtimes as a snap | 07:32 |
| noonedeadpunk | Well, have you seen Juju charms ?:) | 07:32 |
| jrosser | perhaps thankfully - no :) | 07:32 |
| noonedeadpunk | Well, snap is not that bad in theory especially for software developers, when they don't need to care about cross-distro compatability... I think same applies to docker though | 07:34 |
| noonedeadpunk | But in fact that's a pita... | 07:34 |
| noonedeadpunk | Hm, I wonder if for centos it's just matter of kata v2 https://github.com/kata-containers/kata-containers/blob/main/docs/install/centos-installation-guide.md ? | 07:37 |
| jrosser | nothing starting with k http://mirror.centos.org/centos-8/8/virt/x86_64/advanced-virtualization/Packages/ | 07:39 |
| jrosser | right - so maybe we backport this to X like you say https://review.opendev.org/c/openstack/openstack-ansible-os_zun/+/827046 | 07:41 |
| jrosser | there is no centos-8 upgrade job there so that would be as far back as it needs to go | 07:41 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_zun stable/xena: kata: fix link to now removed mirror location https://review.opendev.org/c/openstack/openstack-ansible-os_zun/+/827230 | 07:42 |
| noonedeadpunk | wait, we had centos-8 as nv everywhere? | 07:50 |
| noonedeadpunk | for upgrade jobs? | 07:50 |
| noonedeadpunk | https://opendev.org/openstack/openstack-ansible-os_zun/src/branch/stable/xena/zuul.d/project.yaml#L28 | 07:51 |
| noonedeadpunk | So might be we just needed to make them nv in https://review.opendev.org/c/openstack/openstack-ansible-os_zun/+/824457/3/zuul.d/project.yaml as well | 07:51 |
| noonedeadpunk | and not depend on this patch for https://review.opendev.org/c/openstack/openstack-ansible-os_zun/+/827046 | 07:52 |
| noonedeadpunk | yes, there won't be ci tests for it, but no circular dependency as well then :) | 07:52 |
| jrosser | i think they were nv | 07:52 |
| jrosser | and then for the briefest moment they were all working | 07:52 |
| jrosser | \o/ followed by /o\ | 07:53 |
| noonedeadpunk | (usual centos stuff) | 07:53 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_zun master: kata: fix link to now removed mirror location https://review.opendev.org/c/openstack/openstack-ansible-os_zun/+/827046 | 07:58 |
| *** kleini_ is now known as kleini | 08:13 | |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-galera_server master: Convert xinetd clustercheck to systemd socket service https://review.opendev.org/c/openstack/openstack-ansible-galera_server/+/824042 | 08:27 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-galera_server master: Remove legacy cleanup tasks https://review.opendev.org/c/openstack/openstack-ansible-galera_server/+/827216 | 08:30 |
| agemuend | Hi all, we have an osa deployment on CentOS 8, can someone maybe point me to a link on how to handle the migration from centos 8 to stream? | 08:31 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-lxc_hosts master: Ansible systemd module can reload units without specifying a service https://review.opendev.org/c/openstack/openstack-ansible-lxc_hosts/+/827217 | 08:32 |
| agemuend | Should we just dist-upgrade the hosts and all good, or do we need to migrate the images as well? | 08:32 |
| jrosser | agemuend: this is the compatibility matrix https://docs.openstack.org/openstack-ansible/latest/admin/upgrades/compatibility-matrix.html | 08:32 |
| jrosser | we do not have support for centos-8 stream until the Wallaby release of OSA | 08:33 |
| agemuend | Oh I see, we're using victoria. So what would be the recommended approach? First upgrade to Wallaby and then upgrade the hosts? | 08:34 |
| jrosser | there is a set of documentation here about how someone did ubuntu bionic->focal | 08:34 |
| jrosser | so the same basic process would hold | 08:34 |
| jrosser | unfortunatley you are in the most difficult position by not having moved to Wallaby before the centos-8 EOL | 08:34 |
| jrosser | bionic-> focal notes are here https://docs.openstack.org/openstack-ansible/latest/admin/upgrades/distribution-upgrades.html | 08:35 |
| jrosser | really the only route is to do a combined operating system and openstack major release upgrade at the same time | 08:36 |
| jrosser | and i also feel kind of obligated to point out that centos-8-stream ships a python interpreter which is already EOL | 08:37 |
| jrosser | agemuend: you are going to need to reconfigure your repos as a minumum step https://www.centos.org/centos-linux-eol/ | 08:41 |
| jrosser | and you could try to upgrade from V -> W on centos-8, but there is absolutely no testing of that at all from the openstack-ansible side due to the EOL | 08:42 |
| agemuend | damn | 08:43 |
| jrosser | the loss of the repos and the EOL means that CI is no longer viable for the whole openstack project on centos-8 | 08:43 |
| agemuend | So maybe update the hosts to stream first, then attempt a full deploy? | 08:43 |
| jrosser | well i don't know | 08:43 |
| jrosser | you can't deploy victoria on centos-8-stream as we never supported that combination in openstack-ansible | 08:44 |
| jrosser | but you could maybe use an AIO build to test out modifying all the repo locations to point to the centos vault repos | 08:45 |
| jrosser | and then do an openstack-ansible upgrade of that to W on the old OS | 08:45 |
| jrosser | but really i can't say what might go wrong there, lots of the content has just disappeared | 08:45 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-os_keystone master: Remove legacy nginx cleanup tasks https://review.opendev.org/c/openstack/openstack-ansible-os_keystone/+/827220 | 08:53 |
| agemuend | Are the repo locations somewhere central or scattered? | 08:56 |
| agemuend | Maybe we can use a repo mirror, we do have CentOS repos mirrored anyway | 08:56 |
| jrosser | agemuend: there is info on here https://www.centos.org/centos-linux-eol/ about how content has been moved to vault.centos.org | 09:01 |
| agemuend | Yeah I did that already | 09:02 |
| agemuend | At least on the hosts | 09:02 |
| agemuend | All repos are available on vault | 09:03 |
| jrosser | i guess there are a bunch to check, EPEL and RDO may be in play as well | 09:03 |
| agemuend | rdo is available on vault as well | 09:06 |
| agemuend | EPEL is still there | 09:07 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-rabbitmq_server master: Remove old repos for Debian https://review.opendev.org/c/openstack/openstack-ansible-rabbitmq_server/+/827221 | 09:08 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-os_keystone master: Remove legacy db pooling variables https://review.opendev.org/c/openstack/openstack-ansible-os_keystone/+/827228 | 09:13 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-os_neutron master: Remove legacy db pooling variables https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/827250 | 09:18 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-os_neutron master: Remove legacy vpnaas agent service https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/827251 | 09:18 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-os_octavia master: Remove legacy db pooling variables https://review.opendev.org/c/openstack/openstack-ansible-os_octavia/+/827252 | 09:20 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-os_neutron master: Remove legacy db pooling variables https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/827250 | 09:21 |
| opendevreview | Jonathan Rosser proposed openstack/ansible-role-python_venv_build master: Remove legacy comment https://review.opendev.org/c/openstack/ansible-role-python_venv_build/+/827253 | 09:26 |
| jrosser | agemuend: if you manage to make some notes for centos8 -> centos8-stream we can add them to the documentation | 09:49 |
| agemuend | Well, lets see how it goes, I'm a bit pessimistic right now | 10:05 |
| opendevreview | Merged openstack/openstack-ansible-os_zun master: kata: fix link to now removed mirror location https://review.opendev.org/c/openstack/openstack-ansible-os_zun/+/827046 | 10:14 |
| opendevreview | Merged openstack/openstack-ansible stable/wallaby: Remove left-over centos-8 job from project template https://review.opendev.org/c/openstack/openstack-ansible/+/826937 | 11:15 |
| jrosser | agemuend: i don't really know what we could have done differently, 8-stream support was a quite big set of changes in openstack-ansible so major surgery for a new OS support is not something we really can backport | 11:18 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-openstack_hosts stable/ussuri: Remove legacy centos-8 jobs https://review.opendev.org/c/openstack/openstack-ansible-openstack_hosts/+/827266 | 11:26 |
| *** dviroel|out is now known as dviroel | 11:27 | |
| agemuend | jrosser: Yeah I completely understand | 11:28 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-tests stable/xena: Remove legacy centos-8 CI jobs https://review.opendev.org/c/openstack/openstack-ansible-tests/+/827269 | 11:42 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-tests stable/wallaby: Remove legacy centos-8 CI jobs https://review.opendev.org/c/openstack/openstack-ansible-tests/+/827239 | 11:43 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-tests stable/victoria: Remove legacy centos-8 CI jobs https://review.opendev.org/c/openstack/openstack-ansible-tests/+/827270 | 11:45 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-tests stable/ussuri: Remove legacy centos-8 CI jobs https://review.opendev.org/c/openstack/openstack-ansible-tests/+/827271 | 11:46 |
| agemuend | jrosser: I have now updated the hosts and tried to rebuild the containers after lxc-destroy, but they're being rebuilt with CentOS 8 and the old repos, can you point me where I'm going wrong? | 12:10 |
| jrosser | there is a base image built on each host, which is probably caching all of that from before | 12:11 |
| jrosser | you can use this to make it re-create the base image https://github.com/openstack/openstack-ansible-lxc_hosts/blob/stable/victoria/defaults/main.yml#L216-L217 | 12:12 |
| jrosser | run your playbook with `-e lxc_image_cache_refresh=true` | 12:13 |
| agemuend | Okay will try, thank you | 12:13 |
| agemuend | I'm just wondering, we should probably run that regularly in operations as well, right? Otherwise we'll have possibly insecure service containers? | 12:16 |
| jrosser | the quickest way to do that will be to use `playbooks/containers-lxc-host.yml` playbook directly | 12:16 |
| jrosser | that will just run the lxc_hosts role and refresh the base image | 12:16 |
| jrosser | then delete / re-create whatever you need to use the new base image | 12:16 |
| jrosser | the base image build grabs always the same starting point on victoria branch https://github.com/openstack/openstack-ansible-lxc_hosts/blob/stable/victoria/vars/redhat.yml#L16 | 12:18 |
| jrosser | but i think it should be brought up to date each time the base image is refreshed https://github.com/openstack/openstack-ansible-lxc_hosts/blob/stable/victoria/templates/prep-scripts/centos_8_prep.sh.j2#L16 | 12:19 |
| jrosser | it's up to you how to manage updates in the containers on a running deployment - you can treat them like hosts and update them with yum/dnf | 12:19 |
| agemuend | Mhm that fails actually | 12:23 |
| agemuend | > | 12:23 |
| agemuend | FAILED - RETRYING: Ensure that the LXC cache has been prepared (120 retries left). | 12:23 |
| agemuend | fatal: [ultronmgmt03]: FAILED! => {"ansible_job_id": "894964107053.443202", "attempts": 2, "changed": true, "cmd": "chroot /var/lib/machines/centos-8-amd64 /opt/cache-prep-commands.sh > /var/log/lxc-cache-prep-commands.log 2>&1", "delta": "0:00:01.837147", "end": "2022-02-01 13:21:33.599723", "finished": 1, "msg": "non-zero return code", "rc": 1, "start": "2022-02-01 13:21:31.762576", "stderr": "", "stderr_lines": [], "stdout": "", | 12:23 |
| agemuend | "stdout_lines": []} | 12:23 |
| agemuend | Oh sorry, that was probably our fault again | 12:24 |
| agemuend | No it wasnt, it still fails. | 12:25 |
| jrosser | `/var/log/lxc-cache-prep-commands.log` is your next stop | 12:27 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-tests stable/wallaby: Fix rich version for ansible-lint https://review.opendev.org/c/openstack/openstack-ansible-tests/+/827241 | 12:37 |
| jrosser | noonedeadpunk: i am wondering if [keystone_authtoken]:service_type has any bearing on your application credential thing with octavia | 12:57 |
| jrosser | did you limit the scope of the token? | 12:57 |
| jrosser | i am sure we had someone submit a patch about that but i can't find it right now | 12:57 |
| noonedeadpunk | Um, I'm not sure it's related... I mean - jsut check this out https://opendev.org/openstack/keystone/src/branch/master/keystone/api/_shared/authentication.py#L210-L212 | 12:59 |
| agemuend | Mhm, no mattter what I do, it always rebuilds the containers with the old centos | 13:00 |
| noonedeadpunk | And keystone fails with `KeyError: 'application_credential'` | 13:02 |
| noonedeadpunk | jrosser: regarding patch you meant this? https://review.opendev.org/c/openstack/openstack-ansible-os_glance/+/823009 | 13:03 |
| jrosser | agemuend: how do you mean with the old centos? | 13:03 |
| noonedeadpunk | I can recall there was one more coming somewhere actually... | 13:03 |
| noonedeadpunk | or it was jsut bug report | 13:03 |
| agemuend | jrosser: I mean it still builds base images with CentOS 8.4 although my hosts are upgraded for Stream | 13:04 |
| jrosser | agemuend: oh well, you're still deploying victoria? | 13:04 |
| agemuend | Well, I checked out the wallaby files | 13:05 |
| agemuend | Sorry if I'm being dense | 13:05 |
| jrosser | and did bootstrap-ansible.sh? | 13:05 |
| noonedeadpunk | there could be facts that are cached as well | 13:05 |
| jrosser | ^ good point | 13:06 |
| *** sshnaidm is now known as sshnaidm|afk | 13:06 | |
| * noonedeadpunk really need to sit and calmly read about tokens scope and coming changes | 13:06 | |
| jrosser | agemuend: clear out /etc/openstack_deploy/ansible_facts | 13:07 |
| jrosser | noonedeadpunk: yes, i was kind of concerned about the ML messages about the token scope changes | 13:07 |
| jrosser | there does not seem to be a clearly described "from a deployment tool point of view, you need to 1).... 2).... 3)....." | 13:07 |
| jrosser | and that seemed to be kind of being designed / discussed on the ML which was worrying | 13:08 |
| noonedeadpunk | exactly... But main point I get that services should be assigned service role.. | 13:09 |
| noonedeadpunk | What is worrying indeed is that it all sounds like a change to openstack collections would be required to enable some api calls I guess | 13:10 |
| jrosser | it also sounds like there is $gigantic changes landed in openstacksdk too | 13:10 |
| jrosser | perfect storm :) | 13:10 |
| noonedeadpunk | so to sum up - doesn't sound like early release for us again | 13:11 |
| jrosser | haha | 13:11 |
| jrosser | true | 13:11 |
| agemuend | That didn't help unfortunately | 13:13 |
| agemuend | destroyed the containers and then ran containers-lxc-host.yml, still trying to use victoria | 13:14 |
| noonedeadpunk | agemuend: I think in V you should override `lxc_hosts_container_image_url` | 13:14 |
| noonedeadpunk | as what we have by default is https://opendev.org/openstack/openstack-ansible-lxc_hosts/src/branch/stable/victoria/vars/redhat.yml#L16 | 13:15 |
| agemuend | If osa is still on victoria I'm doing something wrong, I want osa to be wallaby | 13:15 |
| agemuend | I'm rerunning bootstrap to make sure | 13:15 |
| noonedeadpunk | Likely what you'll need is a backport of https://opendev.org/openstack/openstack-ansible-lxc_hosts/commit/791b9e813ed29c1187803c53ad46e369b24eaad4 | 13:16 |
| noonedeadpunk | as centos doesn't have any valid lxc image for stream | 13:16 |
| noonedeadpunk | or well, you can prepare it yourself with "dnf --installroot" command, pack it to tar.gz and put to repo container | 13:17 |
| agemuend | So even in Wallaby it doesn't work? | 13:17 |
| jrosser | hold on | 13:17 |
| agemuend | If there is no stream base image? | 13:17 |
| noonedeadpunk | it does, why not? | 13:17 |
| noonedeadpunk | on W we changed way and don't need base image anymore | 13:17 |
| agemuend | Ok. I'm trying to use osa wallaby. | 13:18 |
| noonedeadpunk | as we build it with command locally | 13:18 |
| * noonedeadpunk holding now | 13:18 | |
| jrosser | on wallaby it should be doing this https://github.com/openstack/openstack-ansible-lxc_hosts/blob/stable/wallaby/vars/redhat.yml#L16 | 13:19 |
| jrosser | but having said that, W is the one release that we have support for both centos-8 and centos-8-stream | 13:20 |
| agemuend | Can I just delete /var/lib/machines/centos-8-amd64 ? | 13:20 |
| noonedeadpunk | looking at command, centos-8 could be only host - containers are stream anyway :D | 13:20 |
| agemuend | In case the lxc_image_cache_refresh doesn't really work? | 13:21 |
| jrosser | i think you can do that | 13:21 |
| jrosser | it would be really interesting if you are able to paste the output of the 'playbooks/containers-lxc-host.yml' to paste.opendev.org | 13:21 |
| agemuend | Okay even if I delete /var/lib/machines/centos-8-amd64 it will rebuild with the CentOS release | 13:27 |
| agemuend | I'm missing something, but I don't know what | 13:27 |
| jrosser | can you paste the output for us to see? | 13:30 |
| agemuend | Sure one sec | 13:34 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-tests stable/wallaby: Remove legacy centos-8 jobs https://review.opendev.org/c/openstack/openstack-ansible-tests/+/827239 | 13:35 |
| agemuend | This is just the last run which looks okay, I guess you need something else: https://paste.opendev.org/show/812462/ | 13:37 |
| jrosser | i think without lxc_image_cache_refresh=true it's not going to run the interesting part | 13:39 |
| agemuend | Here is a new one: https://paste.opendev.org/show/812464/ | 13:46 |
| agemuend | And in the prep commands log you see that it tried victoria again: Error: Failed to download metadata for repo 'openstack-victoria': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried | 13:46 |
| agemuend | Oh damnit, I'm so stupid. It runs the dnf installroot on the mgmt host right? | 13:48 |
| agemuend | So that broken repo must still be on the ansible machine *facepalm* | 13:49 |
| jrosser | the prep commands are run against the chroot created on the target | 13:51 |
| jrosser | the repo definitions are copied from the container host into the container | 13:52 |
| jrosser | because of this https://github.com/openstack/openstack-ansible-lxc_hosts/blob/stable/wallaby/vars/redhat.yml#L30 | 13:54 |
| *** sshnaidm|afk is now known as sshnaidm | 13:54 | |
| jrosser | the only place a repo called openstack-victora can have come from inside the container at this very early stage in it's creation if by being copied from the host | 13:54 |
| jrosser | noonedeadpunk: andrewbonney could you take a look at these? https://review.opendev.org/q/topic:osa%252Fremove-centos8 | 13:56 |
| jrosser | need to merge them bit by bit to get the jobs removed in the right order | 13:57 |
| agemuend | Short other question. Here its described that you should install the rdo-release package, is that required? They seem to be in a weird state due to stream. https://docs.openstack.org/project-deploy-guide/openstack-ansible/wallaby/deploymenthost.html#configure-centos | 14:12 |
| spatel | i believe centos-8 does support victoria but not plan is to drop centos-8 (why do you want to use centos-8 which is dead by Redhat also) | 14:20 |
| agemuend | I dont | 14:20 |
| spatel | i repled to rohit02 :) | 14:21 |
| agemuend | Ah sorry | 14:21 |
| spatel | i would highly recommend stay away from centos (any kind of deployment) | 14:23 |
| spatel | may be in future centos stream get stable and come back but otherwise it would be pain and nothing else | 14:24 |
| jrosser | agemuend: there could easily be an error in the documentation there | 14:24 |
| spatel | we are migrating all our cloud deployment from centos to ubuntu | 14:24 |
| jrosser | and those instructions could totally be for non-stream, as W was the first release to support both | 14:25 |
| jrosser | agemuend: you can look in the collected logs for our CI jobs to see where the installed packages came from https://zuul.opendev.org/t/openstack/build/7f950a4ea1fa43e5b2686c30d7841ea0/log/logs/redhat-rpm-list-installed-host-13-50-26.txt | 14:30 |
| jrosser | thats a recent patch to stable/wallaby running on a centos-8-stream node | 14:31 |
| jrosser | we can see there that the thing that matters is rdo-deps | 14:32 |
| jrosser | these should be automatically setup for you as part of openstack_hosts ansible role https://github.com/openstack/openstack-ansible-openstack_hosts/blob/stable/wallaby/vars/redhat-8.yml#L95-L107 | 14:36 |
| opendevreview | Merged openstack/openstack-ansible-tests stable/ussuri: Remove legacy centos-8 CI jobs https://review.opendev.org/c/openstack/openstack-ansible-tests/+/827271 | 14:40 |
| opendevreview | Merged openstack/openstack-ansible-tests stable/victoria: Remove legacy centos-8 CI jobs https://review.opendev.org/c/openstack/openstack-ansible-tests/+/827270 | 14:52 |
| opendevreview | Merged openstack/openstack-ansible-tests stable/xena: Remove legacy centos-8 CI jobs https://review.opendev.org/c/openstack/openstack-ansible-tests/+/827269 | 14:52 |
| noonedeadpunk | #startmeeting openstack_ansible_meeting | 15:01 |
| opendevmeet | Meeting started Tue Feb 1 15:01:16 2022 UTC and is due to finish in 60 minutes. The chair is noonedeadpunk. Information about MeetBot at http://wiki.debian.org/MeetBot. | 15:01 |
| opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 15:01 |
| opendevmeet | The meeting name has been set to 'openstack_ansible_meeting' | 15:01 |
| noonedeadpunk | #topic office hours | 15:03 |
| mgariepy | hey o/ | 15:04 |
| damiandabrowski[m] | hey! | 15:04 |
| noonedeadpunk | I saw mariadb has merged and seems even backported fixes to mysql_upgrade | 15:05 |
| noonedeadpunk | mgariepy: have you had a chance to look closer where their ended up?:) | 15:05 |
| mgariepy | nop i haven't | 15:05 |
| mgariepy | but the idea was only not to run 2 mysql_upgrade at the same time to prevent lock. | 15:08 |
| NeilHanlon | heyo folks :) | 15:09 |
| noonedeadpunk | yeah. But likely we can change some code and simplify things based on that | 15:09 |
| jrosser | hello | 15:09 |
| mgariepy | debian packaging needs to be upgraded also. | 15:10 |
| mgariepy | i can try to take a look a bit later this week or next week. | 15:11 |
| noonedeadpunk | I don't think there's any rush. I will try to check it as well if get some time before that :) | 15:12 |
| noonedeadpunk | NeilHanlon: how things going on with rocky dib btw? | 15:12 |
| NeilHanlon | fighting more CI issues than anything else, I think, but I believe the patch itself is ready to go once the underlying CI is all fixed.. but the image itself is working afaik | 15:13 |
| noonedeadpunk | ok, awesome. Btw, jrosser found that latest pip jsut dropped py36 support overall. | 15:14 |
| NeilHanlon | oh goody :( | 15:14 |
| noonedeadpunk | Which leads us to thought for how long we will be able to have 8-stream or rocky-8 | 15:14 |
| NeilHanlon | does anyone know where the powerkvm jobs live? There's a centos 8 job in there that needs removing. | 15:14 |
| noonedeadpunk | Maybe it's third-party test? | 15:15 |
| jrosser | NeilHanlon: this is useful https://zuul.openstack.org/jobs | 15:15 |
| noonedeadpunk | as I can't find any in opendev zuul | 15:16 |
| noonedeadpunk | but third-party tests are non-voting usually | 15:17 |
| noonedeadpunk | so they shouldn't result in overall failure | 15:17 |
| NeilHanlon | thanks jrosser. yeah i think it is a 3rd party test https://review.opendev.org/c/openstack/diskimage-builder/+/825957 | 15:17 |
| NeilHanlon | i'll sync up with the dib folks and see how to proceed | 15:18 |
| noonedeadpunk | Ah, so it's all green :) | 15:18 |
| noonedeadpunk | nice | 15:18 |
| NeilHanlon | and green is good :) | 15:18 |
| noonedeadpunk | it really is :) | 15:19 |
| jrosser | i imagine the rocky job should be voting in order to merge that | 15:19 |
| jrosser | noonedeadpunk does make a good point about py36 though | 15:20 |
| jrosser | in an ideal world we would get centos-9-stream support in for our Y release | 15:20 |
| jrosser | and then be dropping centos-8-stream in Z | 15:20 |
| jrosser | for 9 we are a little stuck on mariadb so i made a ticket in their JIRA https://jira.mariadb.org/browse/MDEV-27693 | 15:22 |
| NeilHanlon | jrosser: there was a comment saying it should be nonvoting for a bit, so I made it as such, happy to go back and talk with them, though | 15:23 |
| noonedeadpunk | doh and mariadb just released I believe... | 15:24 |
| jrosser | oh cool no worries - follow what the dib people suggest there | 15:24 |
| jrosser | somehow everything is kind of different in the centos-9 mariadb package, more than it just being a 10.5.x | 15:24 |
| jrosser | so i was not particularly motivated to spend a lot of time hacking in a version we don't want | 15:25 |
| jrosser | there are a few thing we need to fix | 15:27 |
| jrosser | some brokenness here https://review.opendev.org/q/topic:osa%252Finclude_vars | 15:27 |
| noonedeadpunk | lxc_hosts should be fine overall I believe | 15:28 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_zun master: Restore CI jobs https://review.opendev.org/c/openstack/openstack-ansible-os_zun/+/824457 | 15:29 |
| noonedeadpunk | For zun it seems we hacked way around? | 15:29 |
| noonedeadpunk | for now at least | 15:29 |
| noonedeadpunk | Haven't looked into cloudkitty and mistral though | 15:29 |
| jrosser | also a few here https://review.opendev.org/q/topic:osa%252Fcommon_tasks | 15:29 |
| jrosser | i did look at aodh a bit and it fails consistently on centos-8-stream (no comment) and i wondered if it was memory exhaustion | 15:30 |
| jrosser | fails to boot cirros iirc | 15:30 |
| jrosser | next would be the ssh_keypairs role https://review.opendev.org/q/topic:osa%252Fkeypairs | 15:32 |
| jrosser | this is pretty much working except for some issue in CI that needs fixing | 15:32 |
| noonedeadpunk | oh yes, thats interesting | 15:32 |
| jrosser | we have that running in the lab here for os_nova | 15:32 |
| jrosser | it makes a really nice cleanup of the existing complexity in os_nova and os_keystone | 15:33 |
| noonedeadpunk | and, that is indeed awesome! | 15:36 |
| noonedeadpunk | I spent some time on rabbitmq, and converted their erlang config format to smth more conventional https://review.opendev.org/c/openstack/openstack-ansible-rabbitmq_server/+/826338 | 15:37 |
| noonedeadpunk | As my eyes were bleeding each time I was to read it | 15:37 |
| jrosser | oh yes i remember adding the TLS support | 15:38 |
| noonedeadpunk | There're one issue though is that config_template can't handle ini config files without sections | 15:38 |
| jrosser | horrible syntax | 15:38 |
| noonedeadpunk | well, it's not _much_ better when it comes to listeners, but at least readable... | 15:39 |
| noonedeadpunk | somehow readable | 15:39 |
| jrosser | does it have sections at all in the config file? | 15:40 |
| noonedeadpunk | nope | 15:40 |
| *** dviroel is now known as dviroel|lunch | 15:40 | |
| noonedeadpunk | so when config_template is used, it just places blank file | 15:40 |
| noonedeadpunk | as default_overrides are {} so it's overriden with nothing | 15:41 |
| jrosser | this format is actually more like a dict that has been flattened out into K/V pairs | 15:41 |
| noonedeadpunk | yep, exactly. And they still split keys with dots for structure | 15:42 |
| noonedeadpunk | btw I'm a bit frustrated that there's still no erlang for debian 11 | 15:42 |
| noonedeadpunk | while rabbit is already built for it for a while | 15:42 |
| noonedeadpunk | oh, and also I was happy when realized that journald is now supported logging | 15:44 |
| jrosser | i wonder if config_template could grow another format 'flatjson' | 15:45 |
| jrosser | read in that flattened format, process it just like it were a regular json file, and write it out flattened | 15:45 |
| jrosser | maybe json is the wrong term, but 'flatdict' of some sort | 15:46 |
| noonedeadpunk | Rabbitmq claim it to be sysctl | 15:46 |
| noonedeadpunk | and eventually sysctl has really similar fromat | 15:47 |
| jrosser | hmm interesting | 15:47 |
| noonedeadpunk | so yes, I'd say it makes sense to patch config_template | 15:48 |
| noonedeadpunk | I wish sshd_config was also in this format :D | 15:49 |
| jrosser | oh well remember we now have sshd_config.d in all places except centos-8 | 15:51 |
| jrosser | and the ssh_keypairs role now sets it up on centos-8 to work like that too | 15:51 |
| jrosser | maybe sysctl format is just a special case of a generic key=value plain text file | 15:52 |
| jrosser | we would need to choose if we support passing in an actual dict or if its just a bunch of a.b.c=value strings | 15:53 |
| noonedeadpunk | I'd say it should be dict anyway? | 15:55 |
| noonedeadpunk | well, we can do split('=') but that would be weird | 15:55 |
| noonedeadpunk | but dict in terms of simple key/value imo | 15:56 |
| jrosser | can we do a.b.c: value ? | 15:56 |
| noonedeadpunk | so might be just enough to teach config_template how to work without sections if that's feasable | 15:56 |
| noonedeadpunk | yes, I was thinking about that exactly | 15:57 |
| noonedeadpunk | I haven't look in module code yet though | 15:57 |
| jrosser | should we talk about the os_tempest patches? there are lots | 15:57 |
| jrosser | (also we're short of time) | 15:57 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible master: Bump ansible to 2.12.2 https://review.opendev.org/c/openstack/openstack-ansible/+/827313 | 16:00 |
| jrosser | this is ready https://review.opendev.org/c/openstack/openstack-ansible-tests/+/827239 | 16:05 |
| jrosser | #endmeeting | 16:14 |
| opendevmeet | Meeting ended Tue Feb 1 16:14:18 2022 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 16:14 |
| opendevmeet | Minutes: https://meetings.opendev.org/meetings/openstack_ansible_meeting/2022/openstack_ansible_meeting.2022-02-01-15.01.html | 16:14 |
| opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/openstack_ansible_meeting/2022/openstack_ansible_meeting.2022-02-01-15.01.txt | 16:14 |
| opendevmeet | Log: https://meetings.opendev.org/meetings/openstack_ansible_meeting/2022/openstack_ansible_meeting.2022-02-01-15.01.log.html | 16:14 |
| agemuend | The installation of mod-auth-openidc doesn't seem to work on CentOS Stream | 16:16 |
| agemuend | Because it needs dnf module enable | 16:17 |
| agemuend | also the package name seems to be wrong | 16:17 |
| agemuend | Imho https://github.com/openstack/openstack-ansible-os_keystone/blob/master/vars/redhat.yml#L51 should be mod_auth_openidc | 16:21 |
| agemuend | And somewhere before it should do dnf module enable mod_auth_openidc | 16:23 |
| noonedeadpunk | jrosser: thanks for ending, as I got distracted :( | 16:23 |
| jrosser | agemuend: can you test that out for us? i can make a patch no problem | 16:24 |
| agemuend | To be honest I just hacked an additional step in for us where i run the dnf module enable, I don't know how you want to integrate it conditionally because its an extra step for redhat | 16:27 |
| agemuend | But sure I can test it | 16:27 |
| agemuend | If you have a nicer solution | 16:27 |
| *** dviroel|lunch is now known as dviroel | 16:35 | |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-nspawn_hosts stable/victoria: Remove legacy centos-8 jobs https://review.opendev.org/c/openstack/openstack-ansible-nspawn_hosts/+/827319 | 16:50 |
| noonedeadpunk | it's always one thing left.... | 16:51 |
| jrosser | each time we fix one it reveals another | 16:52 |
| jrosser | so this is taking forever | 16:53 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-os_keystone master: Fix apache oidc package name for redhat based distros https://review.opendev.org/c/openstack/openstack-ansible-os_keystone/+/827321 | 16:55 |
| jrosser | agemuend: ^ like this? | 16:55 |
| jrosser | agemuend: related to this my team did loads of work on OSA OIDC support in recent releases | 16:56 |
| jrosser | there are probably a bunch of new things you can do more easily or configure better now | 16:56 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible master: [DOC] A centos deployment host does not need the RDO repo installing https://review.opendev.org/c/openstack/openstack-ansible/+/827326 | 17:01 |
| jrosser | agemuend: ^ also this - i see no reason that the deployment host needs the RDO repo installing | 17:02 |
| agemuend | Yeah looks good. The installation still won't work without executing "dnf module enable mod_auth_openidc" once though | 17:06 |
| agemuend | And indeed, you don't need the RDO repos there, we just ran it without and are nearly through | 17:07 |
| agemuend | well, its working on horizon currently, so it passed a couple of services already | 17:07 |
| agemuend | Now this error appears as well in our env: https://bugs.launchpad.net/openstack-ansible/+bug/1956026 | 17:14 |
| jrosser | would the equivalent module enabling be like this? https://github.com/openstack/openstack-ansible-os_keystone/blob/bc053f483f72793c5451a452d10df5b1741c9946/tasks/keystone_apache.yml#L74-L85 | 17:18 |
| opendevreview | Merged openstack/openstack-ansible-os_zun stable/xena: kata: fix link to now removed mirror location https://review.opendev.org/c/openstack/openstack-ansible-os_zun/+/827230 | 17:18 |
| jrosser | agemuend: can you paste the error you get to paste.opendev.org? | 17:19 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-os_zun master: Use common service setup tasks from a collection rather than in-role https://review.opendev.org/c/openstack/openstack-ansible-os_zun/+/824372 | 17:20 |
| agemuend | jrosser: Ah no, thats the apache httpd module, I'm talking about a DNF module that you need to enable to be able to install the package in the first place. Independent from activating the module in httpd. | 17:20 |
| jrosser | ah right so there are two more things to deal with then | 17:20 |
| * jrosser doesnt use dnf much | 17:21 | |
| agemuend | jrosser: in fact the package brings /etc/httpd/conf.modules.d/10-auth_openidc.conf which contains the LoadModule already, so you don't need an extra step for activation in httpd | 17:22 |
| agemuend | in case of mod_auth_openidc at least, I just checked that. For other modules it could be different. | 17:23 |
| jrosser | seems ansible is kind of clumsy for enabling dnf modules https://github.com/ansible/ansible/issues/64852 | 17:24 |
| agemuend | Yes, thats true. We either set state=enabled in the module file (in that case /etc/dnf/modules.d/mod_auth_openidc.module) or call the dnf module enable as a command | 17:26 |
| jrosser | can you give me a copy of what it puts in /etc/dnf/modules.d ? | 17:27 |
| agemuend | https://paste.opendev.org/show/812471/ | 17:29 |
| agemuend | jrosser: And this is the error we now receive: https://paste.opendev.org/show/bBOeTw5go2b90Sc5EnkP/ | 17:34 |
| agemuend | I included the repo containers because I thought its related | 17:34 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-os_keystone master: Fix oidc apache module installation for centos https://review.opendev.org/c/openstack/openstack-ansible-os_keystone/+/827321 | 17:35 |
| jrosser | ^ ok so i updated that to handle enabling / disabling the module and fixing the name | 17:36 |
| jrosser | if you are able to test that it would be great | 17:36 |
| jrosser | noonedeadpunk: do we need to backport this? https://review.opendev.org/c/openstack/ansible-role-python_venv_build/+/823199 | 17:39 |
| noonedeadpunk | um I think for others we merged already fixed version... | 17:41 |
| jrosser | yeah it looks that way | 17:41 |
| jrosser | agemuend: have you redeployed the repo server containers onto centos-8-stream yet? | 17:41 |
| noonedeadpunk | It wasn't too late when bug was reported | 17:41 |
| agemuend | Shouldnt it have done that automatically in the above tasks? | 17:43 |
| agemuend | But yes, the repo containers that are running are Stream based | 17:43 |
| agemuend | I'm also wondering why it didn't complain before. The other roles like keystone, glance, cinder, nova, neutron all ran through without this error. | 17:44 |
| agemuend | And I thought they also include the venv playbook? | 17:44 |
| jrosser | yes it all uses the same stuff | 17:46 |
| jrosser | it uses the same venv build role, called from different playbooks | 17:47 |
| jrosser | i guess that the useful info is in venv_build_targets | 17:47 |
| opendevreview | Merged openstack/openstack-ansible-os_keystone master: Remove bugfix tasks for the Train release https://review.opendev.org/c/openstack/openstack-ansible-os_keystone/+/827140 | 17:48 |
| jrosser | and that takes into account `[ansible_facts['distribution_version']][ansible_facts['architecture']]`, and i only see architecture in the paste unfortunatley | 17:48 |
| jrosser | the architecture and distro have to match for the repo server to be considered valid for building wheels | 17:49 |
| jrosser | so one possiblity is a centos-8 / centos-8-stream mismatch between the placement containers and the repo containers | 17:49 |
| jrosser | oh wait, it's the '8' isnt it | 17:50 |
| opendevreview | Merged openstack/openstack-ansible-rabbitmq_server master: Use cloudsmith repo for rabbit and erlang https://review.opendev.org/c/openstack/openstack-ansible-rabbitmq_server/+/826444 | 17:50 |
| agemuend | Oh | 17:55 |
| agemuend | Damn, why is the venv_build_targets wrong | 17:55 |
| *** dviroel is now known as dviroel|biab | 17:55 | |
| jrosser | remeber that ansible distribution_version is also a bit of a car crash for centos-8-stream | 17:58 |
| jrosser | it will be just '8' | 17:58 |
| jrosser | but for old EOL centos you get 8.3 / 8.4 / 8.5 | 17:59 |
| agemuend | Oh man, its such a mess, why did CentOS have to do this to us | 18:01 |
| jrosser | all the conditional code we had for (if version > foo) just didnt work right any more | 18:02 |
| jrosser | as the newer version is now *less* than the older one | 18:02 |
| NeilHanlon | ugh | 18:03 |
| opendevreview | Merged openstack/openstack-ansible-os_keystone stable/xena: Fix ordering error enabling/disabling Apache modules https://review.opendev.org/c/openstack/openstack-ansible-os_keystone/+/826550 | 18:05 |
| *** dviroel|biab is now known as dviroel | 18:08 | |
| jrosser | agemuend: end of my day here now but if you are still stuck i would add some debug tasks to see what is happening with the setting of these https://github.com/openstack/ansible-role-python_venv_build/blob/master/defaults/main.yml#L110-L122 | 18:14 |
| agemuend | jrosser: We used the skip variable now and it ran through, I hope that was okay. Thanks a lot for your help | 18:14 |
| jrosser | its ok for just one or two targets | 18:14 |
| jrosser | if it does the same for nova on your compute hosts then please don't | 18:15 |
| jrosser | as thats what has DDOS the opendev git servers before | 18:15 |
| jrosser | nova is a massive repo + lots of hosts == bad | 18:15 |
| agemuend | Okay, I see | 18:17 |
| agemuend | We'll try to debug that tomorrow | 18:17 |
| opendevreview | Merged openstack/openstack-ansible-lxc_hosts master: Refactor use of include_vars https://review.opendev.org/c/openstack/openstack-ansible-lxc_hosts/+/824345 | 19:12 |
| noonedeadpunk | oh, so mariadb hasn;t released mysql_upgrade fix anyway as for now. Scheduled to 10.6.6. I hope that centos 9 will be built for 10.6.6 as well tbh | 19:18 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_zun master: Use common service setup tasks from a collection rather than in-role https://review.opendev.org/c/openstack/openstack-ansible-os_zun/+/824372 | 19:19 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-rabbitmq_server master: Use systemd_service role for overrides https://review.opendev.org/c/openstack/openstack-ansible-rabbitmq_server/+/826463 | 19:40 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-rabbitmq_server master: Use systemd_service role for overrides https://review.opendev.org/c/openstack/openstack-ansible-rabbitmq_server/+/826463 | 19:43 |
| jrosser | more centos-8 removals ready https://review.opendev.org/c/openstack/openstack-ansible-openstack_hosts/+/827266 | 19:43 |
| noonedeadpunk | doh, come on... https://review.opendev.org/c/openstack/openstack-ansible-nspawn_hosts/+/827319 | 19:44 |
| noonedeadpunk | let's drop all tests there?:) | 19:44 |
| noonedeadpunk | except docs and linters? | 19:44 |
| jrosser | yes please :) | 19:45 |
| NeilHanlon | and make sure you don't remove anything I need ! (jk) | 19:47 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-nspawn_hosts stable/victoria: Remove legacy centos-8 jobs https://review.opendev.org/c/openstack/openstack-ansible-nspawn_hosts/+/827319 | 19:48 |
| *** dviroel is now known as dviroel|afk | 20:35 | |
| noonedeadpunk | I bet we already did tbh :( | 21:01 |
| noonedeadpunk | we'll try to help to sort these out anyway) | 21:02 |
| NeilHanlon | all good :) you're just making my life harder but that's not your fault ;) | 21:11 |
| noonedeadpunk | Let's all blame rh :p | 21:12 |
| NeilHanlon | sounds like a plan ;) | 21:13 |
| *** dviroel|afk is now known as dviroel | 23:44 | |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!