| opendevreview | Dmitriy Rabotyagov proposed openstack/ansible-role-zookeeper master: Add upgrade jobs for zookeeper https://review.opendev.org/c/openstack/ansible-role-zookeeper/+/897754 | 06:55 |
|---|---|---|
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible master: Always use on-disk openstack service git repos in CI jobs https://review.opendev.org/c/openstack/openstack-ansible/+/897707 | 06:59 |
| opendevreview | Andrew Bonney proposed openstack/openstack-ansible-ceph_client stable/2023.1: Add AppArmor configuration for ceph read/write caching https://review.opendev.org/c/openstack/openstack-ansible-ceph_client/+/897730 | 07:00 |
| jrosser | wow https://zuul.opendev.org/t/openstack/build/b9556a690bc341e0a75cebb40b3ca9b9/log/job-output.txt#21425 | 07:54 |
| jrosser | noonedeadpunk: typo in the commit message here which i can't decode https://review.opendev.org/c/openstack/ansible-role-python_venv_build/+/897701 | 07:56 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/ansible-role-python_venv_build master: Use distribution_major_version for Debian and CentOS https://review.opendev.org/c/openstack/ansible-role-python_venv_build/+/897701 | 07:58 |
| noonedeadpunk | done | 07:58 |
| jrosser | aaaha | 07:59 |
| noonedeadpunk | sounds like I need to look into the tests repo and functional tests.... | 07:59 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/ansible-role-python_venv_build master: Use distribution_major_version for Debian and CentOS https://review.opendev.org/c/openstack/ansible-role-python_venv_build/+/897701 | 08:00 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible master: WIP - test Vexxhost CAPI driver for magnum https://review.opendev.org/c/openstack/openstack-ansible/+/893240 | 08:02 |
| opendevreview | Merged openstack/openstack-ansible-os_neutron master: [doc] Update example on how to define neutron_vpnaas_custom_config https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/897431 | 09:04 |
| opendevreview | Jonathan Rosser proposed openstack/ansible-role-zookeeper master: Add upgrade jobs for zookeeper https://review.opendev.org/c/openstack/ansible-role-zookeeper/+/897754 | 09:28 |
| damiandabrowski | i noticed that it's nearly impossible to pass gating(at least for magnum repo) due to the rocky timeouts | 09:38 |
| damiandabrowski | examples: | 09:38 |
| damiandabrowski | https://review.opendev.org/c/openstack/openstack-ansible-os_magnum/+/897526 | 09:38 |
| damiandabrowski | https://review.opendev.org/c/openstack/openstack-ansible-os_magnum | 09:38 |
| damiandabrowski | is it something that already has a fix proposed? | 09:39 |
| damiandabrowski | https://review.opendev.org/c/openstack/openstack-ansible-os_magnum/+/893362 * | 09:39 |
| noonedeadpunk | damiandabrowski: it looks like loaded infra... but not sure | 09:41 |
| noonedeadpunk | as ubuntu takes also 3h | 09:41 |
| noonedeadpunk | and same tls job 2h19m | 09:41 |
| damiandabrowski | ahh, that would make sense | 09:42 |
| jrosser | theres nothing very obvious on the grafana dashboards either | 09:45 |
| jrosser | except that the runtimes are generally creeping upward | 09:45 |
| jrosser | sad we don't have "old" ARA where you could quickly sort by task runtime | 09:46 |
| noonedeadpunk | And I haven't managed to work on new ara :( | 09:51 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: Define install_method default when hosts resolution depend on it https://review.opendev.org/c/openstack/openstack-ansible/+/891697 | 09:57 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible stable/yoga: Remove unreadable unicode symbols https://review.opendev.org/c/openstack/openstack-ansible/+/884219 | 09:59 |
| jrosser | i might have a quick hack on seeing what we can get out of the sqlite file | 10:03 |
| jrosser | like what the heaviest roles are | 10:03 |
| opendevreview | Merged openstack/ansible-role-zookeeper master: change to using archive.apache.org for source downloads https://review.opendev.org/c/openstack/ansible-role-zookeeper/+/897673 | 10:26 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/ansible-role-zookeeper stable/2023.1: change to using archive.apache.org for source downloads https://review.opendev.org/c/openstack/ansible-role-zookeeper/+/897801 | 10:30 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/ansible-role-zookeeper stable/2023.1: change to using archive.apache.org for source downloads https://review.opendev.org/c/openstack/ansible-role-zookeeper/+/897801 | 10:31 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/ansible-role-zookeeper stable/zed: change to using archive.apache.org for source downloads https://review.opendev.org/c/openstack/ansible-role-zookeeper/+/897802 | 10:31 |
| opendevreview | Merged openstack/openstack-ansible-os_nova stable/2023.1: Add barbican_service_user section https://review.opendev.org/c/openstack/openstack-ansible-os_nova/+/897457 | 10:40 |
| opendevreview | Merged openstack/openstack-ansible-os_keystone master: Fix example playbook linters https://review.opendev.org/c/openstack/openstack-ansible-os_keystone/+/896908 | 10:43 |
| opendevreview | Merged openstack/openstack-ansible-galera_server stable/yoga: Added vars to override systemd for mariabackup https://review.opendev.org/c/openstack/openstack-ansible-galera_server/+/897665 | 11:11 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-tests master: Bump ansible-core to 2.12.8 https://review.opendev.org/c/openstack/openstack-ansible-tests/+/897403 | 11:16 |
| opendevreview | Merged openstack/openstack-ansible-lxc_hosts master: Add ca-certificates into debian base image during debootstrap. https://review.opendev.org/c/openstack/openstack-ansible-lxc_hosts/+/894557 | 11:16 |
| opendevreview | Merged openstack/ansible-role-zookeeper master: update zookeeper to 3.8.3 https://review.opendev.org/c/openstack/ansible-role-zookeeper/+/897672 | 11:22 |
| opendevreview | Merged openstack/openstack-ansible-ceph_client master: Allow to distribute custom key with the role https://review.opendev.org/c/openstack/openstack-ansible-ceph_client/+/897046 | 11:34 |
| jrosser | noonedeadpunk: first go at stats from ara sqlite - not sure if it's helpful? https://paste.opendev.org/show/bJTmzPJb8xrlSR6Au0r0/ | 11:45 |
| opendevreview | Merged openstack/ansible-role-zookeeper master: Use TOX_CONSTRAINTS_FILE https://review.opendev.org/c/openstack/ansible-role-zookeeper/+/890877 | 12:04 |
| ThiagoCMC | Hey folks, is OSA 2023.2 Bobcat still under development? | 12:46 |
| opendevreview | Merged openstack/openstack-ansible-os_keystone master: oidc: fix recognition of x forwarded headers from v2.4.11 https://review.opendev.org/c/openstack/openstack-ansible-os_keystone/+/894616 | 13:00 |
| noonedeadpunk | ThiagoCMC: yes, pretty much is | 13:00 |
| opendevreview | Andrew Bonney proposed openstack/openstack-ansible-os_keystone stable/2023.1: oidc: fix recognition of x forwarded headers from v2.4.11 https://review.opendev.org/c/openstack/openstack-ansible-os_keystone/+/897806 | 13:01 |
| opendevreview | Andrew Bonney proposed openstack/openstack-ansible-os_keystone stable/zed: oidc: fix recognition of x forwarded headers from v2.4.11 https://review.opendev.org/c/openstack/openstack-ansible-os_keystone/+/897807 | 13:03 |
| noonedeadpunk | We;re still finalizing couple of work areas | 13:06 |
| opendevreview | Andrew Bonney proposed openstack/openstack-ansible-os_keystone stable/zed: oidc: fix recognition of x forwarded headers from v2.4.11 https://review.opendev.org/c/openstack/openstack-ansible-os_keystone/+/897807 | 13:07 |
| ThiagoCMC | noonedeadpunk, okdok! No problem, just curious... Thanks! ^_^ | 13:17 |
| opendevreview | Jonathan Rosser proposed openstack/ansible-role-python_venv_build master: Remove OS specific variable inclusion https://review.opendev.org/c/openstack/ansible-role-python_venv_build/+/897830 | 13:31 |
| noonedeadpunk | so, bookworm seems to be passing for metal, huh? | 13:33 |
| NeilHanlon | 🥳 | 13:33 |
| noonedeadpunk | https://zuul.opendev.org/t/openstack/build/215c90152a2247588fc213b4a2bced2a | 13:33 |
| NeilHanlon | rocky/centos jobs continue to be unstable due to centos mirror issues (at least they're not rocky mirror issues this time) | 13:33 |
| noonedeadpunk | but yeah, fails for horizon in lxc :( | 13:33 |
| NeilHanlon | i had to recheck the zookeeper backports for two different reasons :\ | 13:34 |
| NeilHanlon | rdo-deps, and then a package _in_ rdo deps | 13:34 |
| opendevreview | Jonathan Rosser proposed openstack/ansible-role-systemd_service master: Remove conditions on systemd version https://review.opendev.org/c/openstack/ansible-role-systemd_service/+/897832 | 13:42 |
| noonedeadpunk | and now that needs to land on stable branches as well.... | 13:44 |
| NeilHanlon | for https://review.opendev.org/c/openstack/ansible-role-zookeeper/+/897754/2 - does this require we land the 2023.1 backport for the archive.apache.org? I see zuul upgrade job failed because it was trying to download the old url for zk | 13:44 |
| noonedeadpunk | yes, backport and version bump | 13:45 |
| NeilHanlon | 👍 | 13:45 |
| NeilHanlon | btw please give me a ping if anyone needs reviews or anything. i'll try to keep my eye on the gerrit dashboard, too :) | 13:46 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-ceph_client stable/2023.1: Allow to distribute custom key with the role https://review.opendev.org/c/openstack/openstack-ansible-ceph_client/+/897808 | 13:57 |
| noonedeadpunk | sure, thanks a lot! | 13:57 |
| NeilHanlon | also if you want me to take a stab at any docs stuff, i can make time :) | 14:01 |
| noonedeadpunk | also zookeeper on stable branches block things like this: https://review.opendev.org/c/openstack/openstack-ansible/+/897284 | 14:07 |
| NeilHanlon | bleh | 14:09 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-tests master: Bump ansible-core to 2.12.8 https://review.opendev.org/c/openstack/openstack-ansible-tests/+/897403 | 14:14 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-haproxy_server master: Convert haproxy_service_config_external.yml to a symlink https://review.opendev.org/c/openstack/openstack-ansible-haproxy_server/+/897841 | 14:35 |
| opendevreview | Merged openstack/openstack-ansible stable/yoga: Gather extra networking facts for keepalived https://review.opendev.org/c/openstack/openstack-ansible/+/897285 | 14:40 |
| noonedeadpunk | #startmeeting openstack_ansible_meeting | 15:04 |
| opendevmeet | Meeting started Tue Oct 10 15:04:13 2023 UTC and is due to finish in 60 minutes. The chair is noonedeadpunk. Information about MeetBot at http://wiki.debian.org/MeetBot. | 15:04 |
| opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 15:04 |
| opendevmeet | The meeting name has been set to 'openstack_ansible_meeting' | 15:04 |
| noonedeadpunk | #topic rollcall | 15:04 |
| noonedeadpunk | o/ | 15:04 |
| damiandabrowski | hi! | 15:04 |
| NeilHanlon | o/ | 15:05 |
| NeilHanlon | sorry, last meeting running long 🙃 | 15:05 |
| noonedeadpunk | #topic office hours | 15:06 |
| noonedeadpunk | PTG. | 15:06 |
| noonedeadpunk | I've booked a room for us on Tuesday, Oct 24, 14:00 - 17:00 UTC | 15:07 |
| NeilHanlon | awesome | 15:07 |
| noonedeadpunk | Is that fine for everyone or you have some input on how better to re-schedule that? | 15:08 |
| NeilHanlon | that works fine for me | 15:08 |
| NeilHanlon | #link https://ptg.opendev.org/ptg.html | 15:08 |
| NeilHanlon | #link https://etherpad.opendev.org/p/oct2023-ptg-os-ansible | 15:08 |
| noonedeadpunk | Another thing, is that I didn't book operator hours this time, but then TC wrote a ML asking for projects to do so. | 15:08 |
| noonedeadpunk | I might re-name etherpad fwiw :) | 15:09 |
| noonedeadpunk | I haven't populated it yet | 15:09 |
| NeilHanlon | fair :) | 15:09 |
| damiandabrowski | I'll perform openstack upgrade during that time so not sure if i'll be able to join, but i'll try to | 15:09 |
| noonedeadpunk | What we think about operator hours? Do we see having any benefit from running these? | 15:09 |
| noonedeadpunk | They don't have a lot of attendance, though I can appear for really an hour on Wednesday just to make an opprotunity for ppl to show up | 15:10 |
| damiandabrowski | IIRC last time it was only you, me and amy :| | 15:11 |
| noonedeadpunk | Though I don't think anybody will | 15:11 |
| NeilHanlon | do we have a list of operators we can email directly and let them know about them? | 15:11 |
| NeilHanlon | besides just posting on the -discuss list | 15:11 |
| NeilHanlon | i feel as though they could be useful, if we got people to show up | 15:12 |
| noonedeadpunk | I'm not sure really... | 15:12 |
| noonedeadpunk | And besides openstack marketing... | 15:12 |
| noonedeadpunk | But ok, let's try the last time. | 15:12 |
| NeilHanlon | i can probably try and drum up some interest with Rocky, but. yeah. let's give it a shot :) | 15:13 |
| noonedeadpunk | And maybe do that on Monday as it's pretty much free | 15:13 |
| jamesdenton | TBH i end up with a conflict or simply forget. Apologies | 15:13 |
| noonedeadpunk | and like 17UTC doesn't have a conflict with anything else yet | 15:13 |
| NeilHanlon | sounds good to me | 15:14 |
| noonedeadpunk | ok, good. I will book and send a ML | 15:15 |
| noonedeadpunk | #action noonedeadpunk to book operator hour and send ML | 15:15 |
| noonedeadpunk | Other then that, today we got debian 12 passing for metal jobs. It's failing on horizon though and I'm spawning a sandbox to check what's there | 15:15 |
| noonedeadpunk | CI stability is not great - we're having TIMEOUTs and broken CentOS mirrors now | 15:16 |
| noonedeadpunk | So quite hard to land anything | 15:16 |
| jrosser | i think there was some discussion in horizon irc about debian12 broken with django 4 | 15:16 |
| noonedeadpunk | #link https://review.opendev.org/c/openstack/horizon/+/897310 | 15:17 |
| noonedeadpunk | I wonder why only Deb 12 is affected. Just py3.11? | 15:18 |
| noonedeadpunk | As u-c are quite explicit about Django===3.2.18 | 15:19 |
| noonedeadpunk | SO it's really interesting what's going on | 15:20 |
| jrosser | well, debian12 might have other ideas about that | 15:20 |
| noonedeadpunk | Then we have landed quite some bugfixes and some were already backported. | 15:21 |
| noonedeadpunk | So I'm thinking of pushing bumps for stable branches soon | 15:21 |
| noonedeadpunk | However, bump for master seems to be failing with weird nova issue | 15:22 |
| noonedeadpunk | during upgrade check | 15:22 |
| noonedeadpunk | #link https://review.opendev.org/c/openstack/openstack-ansible/+/897434 | 15:22 |
| noonedeadpunk | I haven't checked what;'s up yet, but that looks like missing uuid for computes in /var/lib/nova/ | 15:23 |
| noonedeadpunk | `Compute node objects without service_id linkage were found in the database. Ensure all non-deleted compute services have started with upgraded code.` | 15:24 |
| noonedeadpunk | So that is a blocker for landing 2023.2 for sure and needs sorting out | 15:25 |
| noonedeadpunk | Also very weird issue with mariadb upgrade, which I wasn't able to reproduce | 15:26 |
| jrosser | i did start today looking for bogus/old tasks in roles we use a lot | 15:26 |
| jrosser | but feels like really not going to be the solution to making CI faster | 15:26 |
| noonedeadpunk | #link https://review.opendev.org/c/openstack/openstack-ansible-galera_server/+/894740 | 15:26 |
| noonedeadpunk | yeah, those are nice clean-ups | 15:26 |
| noonedeadpunk | I wonder if we should revive parallel execution of things at least for CIs | 15:27 |
| jrosser | i wonder if theres some 12 vs 12.1 stuff going on in the galera role | 15:27 |
| noonedeadpunk | Like make some python script that would parse setup-infrastructure and setup-openstack and execute in threads openstack-ansible binary... | 15:27 |
| noonedeadpunk | But it fails for jammy? | 15:28 |
| noonedeadpunk | It somehow tries to isntall 10.6 instead of 10.11.5 | 15:28 |
| noonedeadpunk | like use built-in repos ignoring pinned priority | 15:29 |
| jrosser | 894740 fails in repo server for jammy | 15:29 |
| jrosser | oh well actually | 15:30 |
| noonedeadpunk | is it? https://zuul.opendev.org/t/openstack/build/204c318c9e204e01a6f48064ab9060d7/log/job-output.txt#25724 | 15:30 |
| jrosser | its 894561 where we need to look | 15:30 |
| noonedeadpunk | 'mariadb-server=1:10.6.12-0ubuntu0.22.04.1'' failed: E: Unmet dependencies. Try 'apt --fix-broken install' with no packages (or specify a solution). | 15:30 |
| noonedeadpunk | well... that fails differently... | 15:31 |
| jrosser | oh no /o\ | 15:31 |
| jrosser | its that systemd_mount rescue task that keeps catching me out | 15:32 |
| noonedeadpunk | but I wonder what mariadb it has installed | 15:32 |
| noonedeadpunk | We should do smth to it I guess.... | 15:32 |
| noonedeadpunk | but that's different topic | 15:32 |
| noonedeadpunk | So. 894561 has 10.6.12 at the end | 15:33 |
| noonedeadpunk | when it fails to re-bootstrap the cluster | 15:33 |
| noonedeadpunk | https://zuul.opendev.org/t/openstack/build/5086e874a1dc4ef0b13c072e3e3d4886/log/logs/host/dpkg.log.txt#3032 | 15:33 |
| jrosser | wtf https://zuul.opendev.org/t/openstack/build/5086e874a1dc4ef0b13c072e3e3d4886/log/logs/host/dpkg.log.txt#2948 | 15:35 |
| noonedeadpunk | it somehow looks like the infra cache mirror just doesn't have required version | 15:35 |
| noonedeadpunk | it's before upgrade | 15:35 |
| noonedeadpunk | on 2023.1 | 15:35 |
| noonedeadpunk | and then it gets removed in favor of 10.6 | 15:35 |
| jrosser | that line is like only 2 seconds before all the 10.6 stuff | 15:36 |
| noonedeadpunk | huh | 15:36 |
| jrosser | `install mariadb-common:all 1:10.11.2+maria~ubu2204 1:10.6.12-0ubuntu0.22.04.1` | 15:37 |
| jrosser | ^ what is this i wonder | 15:37 |
| noonedeadpunk | aha, and previous run is L1098 | 15:37 |
| noonedeadpunk | maybe we don't clean up enough? | 15:37 |
| noonedeadpunk | and some more packages needs to be wiped for upgrade now | 15:38 |
| noonedeadpunk | And L2931 it's being removed... | 15:39 |
| noonedeadpunk | status half-installed mariadb-common | 15:39 |
| noonedeadpunk | so no, it really installs 10.6 from default repos ignoring mariadb one | 15:40 |
| jrosser | maybe we need a patch that inserts a `fail:` at the point it should be cleaned up | 15:40 |
| jrosser | and get a held node to see what actually is there | 15:40 |
| noonedeadpunk | So I really wonder if smth is off with repo proxy | 15:41 |
| noonedeadpunk | https://zuul.opendev.org/t/openstack/build/5086e874a1dc4ef0b13c072e3e3d4886/log/logs/etc/host/apt/sources.list.d/MariaDB.list.txt | 15:41 |
| jrosser | can always try to use `http://mirror.iad.rax.opendev.org:8080/MariaDB/mariadb-10.11.5/repo/ubuntu/` in a local build | 15:43 |
| noonedeadpunk | is it available from outside? | 15:44 |
| noonedeadpunk | but yeah, will check that | 15:45 |
| noonedeadpunk | And hopefully I will be able to continue pshing stuff for quorum queues and identify more linter failures.... | 15:45 |
| jrosser | yes take the `-int` out of it | 15:46 |
| noonedeadpunk | fwiw, horizon didn't fail in my sandbox | 15:46 |
| noonedeadpunk | aha | 15:46 |
| noonedeadpunk | so maybe horizon will feel better on 2023.2 branch... | 15:48 |
| noonedeadpunk | or well... Maybe we're using specific SHA atm... | 15:49 |
| noonedeadpunk | (in gates) | 15:49 |
| noonedeadpunk | anyway... | 15:50 |
| noonedeadpunk | anything else to raise/talk about? | 15:50 |
| jrosser | we say it every year but would be good not to have a huge rush to release :) | 15:52 |
| jrosser | so anything that can fix up the CI reliability a bit would be a bonus | 15:52 |
| noonedeadpunk | Yeah, and each year I'm pretty much in agreement with that but it somehow doesn't work out at the end :( | 15:56 |
| noonedeadpunk | I think we really should not attempt to land smth extra other then what was already promised/agreed | 15:56 |
| noonedeadpunk | And if CI gods will be nice - it should give us to not be in rush | 15:57 |
| NeilHanlon | 🤞 | 16:00 |
| noonedeadpunk | #endmeeting | 16:00 |
| opendevmeet | Meeting ended Tue Oct 10 16:00:12 2023 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 16:00 |
| opendevmeet | Minutes: https://meetings.opendev.org/meetings/openstack_ansible_meeting/2023/openstack_ansible_meeting.2023-10-10-15.04.html | 16:00 |
| opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/openstack_ansible_meeting/2023/openstack_ansible_meeting.2023-10-10-15.04.txt | 16:00 |
| opendevmeet | Log: https://meetings.opendev.org/meetings/openstack_ansible_meeting/2023/openstack_ansible_meeting.2023-10-10-15.04.log.html | 16:00 |
| NeilHanlon | thank you noonedeadpunk for running, as always :) | 16:00 |
| noonedeadpunk | thanks for taking time to join! | 16:00 |
| jrosser | why do we not run ARA in upgrade jobs | 16:01 |
| noonedeadpunk | I actually don't remember | 16:01 |
| jrosser | i was going to run my script over the db to see if theres anything obvious for slowness | 16:02 |
| jrosser | but no db :( | 16:02 |
| *** louis is now known as Guest2868 | 16:04 | |
| noonedeadpunk | it was smth related to it's going crazy about verison upgrade or smth like that... | 16:07 |
| noonedeadpunk | but probably just didn't care anough as upgrade jobs were pretty much supplementary at first | 16:09 |
| opendevreview | Merged openstack/openstack-ansible-lxc_hosts master: Switch to native systemd-resolved from resolv.conf https://review.opendev.org/c/openstack/openstack-ansible-lxc_hosts/+/894558 | 16:09 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-lxc_hosts master: Stop installing openssh and rsync to containers https://review.opendev.org/c/openstack/openstack-ansible-lxc_hosts/+/889945 | 16:11 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-lxc_hosts master: Remove old cleaup task https://review.opendev.org/c/openstack/openstack-ansible-lxc_hosts/+/897855 | 16:25 |
| Guest2868 | Hi, im trying to install osa on my infrastructure im running setup-openstack.yml playbook and I got this error: https://paste.openstack.org/show/bxIf6SQ8nDFTw5clQhrE/ when I try to curl htpp://192.168.100.10:5000/ this is the output: https://paste.openstack.org/show/bYlSLdCuuIebzKdU5O4l/ I don't understand why sometimes I got a SSL handshake failure. | 16:26 |
| noonedeadpunk | Guest2868: How does your endpoint list look like? | 16:34 |
| opendevreview | Merged openstack/ansible-role-python_venv_build master: Use distribution_major_version for Debian and CentOS https://review.opendev.org/c/openstack/ansible-role-python_venv_build/+/897701 | 16:34 |
| noonedeadpunk | as it might be that public endpoint is using TLS and keystone log just shows them with handshake error | 16:34 |
| noonedeadpunk | like if someone tries to access TLS with plain HTTP | 16:35 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/ansible-role-python_venv_build stable/2023.1: Use distribution_major_version for Debian and CentOS https://review.opendev.org/c/openstack/ansible-role-python_venv_build/+/897810 | 16:35 |
| Guest2868 | noonedeadpunk: like that https://paste.openstack.org/show/bsiRDH16GsxF8l034nq6/ | 16:36 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/ansible-role-python_venv_build stable/2023.1: Use distribution_major_version for Debian and CentOS https://review.opendev.org/c/openstack/ansible-role-python_venv_build/+/897810 | 16:36 |
| noonedeadpunk | Guest2868: and you can do curl not only from the deploy host, but also from infra1 server? | 16:38 |
| noonedeadpunk | Also - you're running metal install, right? | 16:38 |
| Guest2868 | yes its a metal install and no I cant curl from infra1 i got a SSL handshake error in haproxy logs | 16:39 |
| noonedeadpunk | ah, so you do curl multiple times but once in a while you get empty reply | 16:40 |
| noonedeadpunk | And 192.168.100.10 is on infra1? | 16:40 |
| Guest2868 | Oh, I just retry curl and now I got result from haproxy | 16:40 |
| noonedeadpunk | and basially - you have more then 1 controller?:) | 16:41 |
| noonedeadpunk | or that is some kind of AIO setup | 16:42 |
| Guest2868 | my LB is alone on 192.168.100.10 and infra1 is 192.168.100.11 , and no I have only one controller on infra1, I tried to create a cluster with infra1, infra2, infra3 but I got an error with glusterfs, so I decided to have a working install on only one infra server | 16:42 |
| noonedeadpunk | Glusterfs can be disabled, but then it need to be replaced with some other shared storage, like nfs or smth like that. Or, you can jsut define to have `repo-infra_hosts` be on infra1 | 16:44 |
| Guest2868 | noonedeadpunk: this is my terminal output: https://paste.openstack.org/show/bKSwzKMI2yk1VSuLO3B2/ | 16:44 |
| noonedeadpunk | so, LB is a standalone host? | 16:44 |
| Guest2868 | yes with only haproxy on it | 16:44 |
| noonedeadpunk | I wonder if you can somehow accidentally have same IP both for internal and external endpoints? | 16:45 |
| noonedeadpunk | or well, internal vip and external VIP are different? | 16:45 |
| Guest2868 | I will give you my openstack_user_config | 16:45 |
| Guest2868 | here: https://paste.openstack.org/show/b2SJX9RBx5z7YISLEm30/ | 16:47 |
| noonedeadpunk | checking | 16:49 |
| noonedeadpunk | well, the only comment there I have to networking definition | 16:51 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-lxc_hosts master: Remove old tasks and vars from image download process https://review.opendev.org/c/openstack/openstack-ansible-lxc_hosts/+/897860 | 16:51 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-lxc_hosts master: Remove lxc_cache_map variable https://review.opendev.org/c/openstack/openstack-ansible-lxc_hosts/+/897861 | 16:51 |
| noonedeadpunk | I'm not sure that you can define `network-gateway_hosts` twice | 16:51 |
| noonedeadpunk | and then I guess it's worth to be `network-infra_hosts` rahter then `network_hosts` | 16:52 |
| noonedeadpunk | I have some weird feeling that your internal_lb_vip_address might be migrating between hosts somehow | 16:54 |
| noonedeadpunk | Though I would expect that you should not have keepalived installed on haproxy host. | 16:54 |
| noonedeadpunk | Can you check if you do have keepalived there or not to be sure? | 16:54 |
| noonedeadpunk | neither on any other hosts... | 16:54 |
| Guest2868 | on user_variable.yml i have a key: haproxy_use_keepalived: False | 16:55 |
| noonedeadpunk | if you do smth like `arp -n 192.168.100.10` - would it be the same MAC? | 16:56 |
| Guest2868 | interesting | 16:56 |
| noonedeadpunk | (like between getting empty reply and not) | 16:57 |
| opendevreview | Merged openstack/ansible-role-zookeeper stable/2023.1: change to using archive.apache.org for source downloads https://review.opendev.org/c/openstack/ansible-role-zookeeper/+/897801 | 17:05 |
| Guest2868 | I change the MAC address and I have some SSL handshake failed | 17:05 |
| Guest2868 | *changed | 17:06 |
| jrosser | noonedeadpunk: oh hmm https://github.com/openstack/openstack-ansible-tests/blob/master/zuul.d/jobs.yaml#L78 | 17:06 |
| opendevreview | Merged openstack/openstack-ansible-os_nova stable/yoga: Add barbican_service_user section https://review.opendev.org/c/openstack/openstack-ansible-os_nova/+/897722 | 17:07 |
| opendevreview | Merged openstack/openstack-ansible-os_nova stable/zed: Add barbican_service_user section https://review.opendev.org/c/openstack/openstack-ansible-os_nova/+/897721 | 17:07 |
| noonedeadpunk | jrosser: that would explain hussle i'm into right now I guess | 17:08 |
| jrosser | i was half way to change your patch to be master there | 17:08 |
| jrosser | and then i dont knwo what the deal will be with older ansible | 17:08 |
| noonedeadpunk | I think that also adding test-vars.yml shouldn't be done through tox... | 17:09 |
| noonedeadpunk | as they're included inside playbooks | 17:09 |
| jrosser | we have a big big mess here | 17:09 |
| noonedeadpunk | https://opendev.org/openstack/openstack-ansible-tests/src/branch/master/test-install-tempest.yml#L29 | 17:09 |
| noonedeadpunk | also that override-checkouts is only for requirements.... | 17:10 |
| noonedeadpunk | but yes, I think worth dropping that and see | 17:10 |
| jrosser | yeah i saw that broken on another job | 17:10 |
| noonedeadpunk | though for me locally things crash in a completely different way on pki stuff | 17:11 |
| jrosser | actually here https://review.opendev.org/c/openstack/openstack-ansible-tests/+/897403 | 17:12 |
| opendevreview | Merged openstack/ansible-role-zookeeper stable/zed: change to using archive.apache.org for source downloads https://review.opendev.org/c/openstack/ansible-role-zookeeper/+/897802 | 17:12 |
| jrosser | probably needs one massive patch on master to sort all that out and bring it to current ansible / current u-c all at once | 17:12 |
| jrosser | otherwise it will never pass | 17:13 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-tests master: Bump ansible-core to 2.12.8 https://review.opendev.org/c/openstack/openstack-ansible-tests/+/897403 | 17:14 |
| noonedeadpunk | I've actually tried to recall how that all worked back then but couldn't make myself to keep focused on all that crap... | 17:14 |
| jrosser | there is still services_branch set to yoga in there | 17:15 |
| noonedeadpunk | ugh | 17:15 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-tests master: Bump ansible-core to 2.12.8 https://review.opendev.org/c/openstack/openstack-ansible-tests/+/897403 | 17:15 |
| noonedeadpunk | and that will obviously fail | 17:16 |
| noonedeadpunk | or not... | 17:16 |
| noonedeadpunk | really very fishy part is why test-vars were not respected.... | 17:17 |
| noonedeadpunk | feels like some include/import thingy | 17:17 |
| noonedeadpunk | as it passed nicely until tempest... | 17:17 |
| Guest2868 | noonedeadpunk: i think you are pretty busy right now | 17:20 |
| noonedeadpunk | Guest2868: sorry :( so, you changed mac.... on haproxy host? and you got the issue with ssl handshake? | 17:21 |
| noonedeadpunk | and arp on the host you was using curl from shows new mac? | 17:21 |
| noonedeadpunk | or I misunderstood what you did? | 17:22 |
| Guest2868 | i had a doubt on duplicated MAC address, so I changed it on my LB server. When I run the apn i got the same MAC from the apn cmd and a `ip a` on lb server | 17:24 |
| noonedeadpunk | so it didn't change after all? | 17:25 |
| Guest2868 | the issue? yes same shit, sometimes i got ssl handshake error | 17:26 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-haproxy_server stable/2023.1: Add possibility to override haproxy_ssl_path https://review.opendev.org/c/openstack/openstack-ansible-haproxy_server/+/897167 | 17:26 |
| Guest2868 | the ssl error is from keystone_service-front-1/1 and when the response is ok is from keystone_service-front-2. Maybe its not relevant | 17:27 |
| noonedeadpunk | and if you check haproxy.cfg - you have different frontend for internal and external vips, right? | 17:27 |
| noonedeadpunk | I assume, that on lb1 you don't have any host record that would make to resolve lb00.airtop.io into 192.168.100.10 ? | 17:29 |
| Guest2868 | this i my /etc/hosts on lb1 https://paste.openstack.org/show/b3q9Yl3RBPBrJsaL10bB/ | 17:30 |
| noonedeadpunk | Guest2868: last line | 17:35 |
| noonedeadpunk | you can never have extrnal_lb_vip_address be same as internal | 17:35 |
| noonedeadpunk | You can explicitly define IP or interface for haproxy to listen on for extenral vip by defininig haproxy_bind_external_lb_vip_address | 17:36 |
| noonedeadpunk | or you can bind it to interface as well: https://opendev.org/openstack/openstack-ansible-haproxy_server/src/branch/master/defaults/main.yml#L277-L285 | 17:37 |
| noonedeadpunk | So I guess depending on how your fqdn will be resolved - you go either to internal endpoint which is plain HTTP or to external which is TLS | 17:37 |
| Guest2868 | should i add it on user_variables? | 17:39 |
| noonedeadpunk | yup | 17:40 |
| noonedeadpunk | and re-run os-keystone-install.yml if you're on 2023.1 (antelope) or haproxy-server.yml if earlier | 17:41 |
| Guest2868 | i dont really understad what should i put on theses keys | 17:41 |
| Guest2868 | i should rerun setup-infra? | 17:42 |
| noonedeadpunk | So somehow, in your /etc/hosts, 192.168.100.10 record is mapped to the external fqdn lb00.airtop.io | 17:43 |
| Guest2868 | i removed it but now, haproxy cant start | 17:44 |
| noonedeadpunk | and you have this in openstack_user_config: external_lb_vip_address: lb00.airtop.io | 17:44 |
| noonedeadpunk | and internal_lb_vip_address: 192.168.100.10 | 17:44 |
| noonedeadpunk | which under certain circumstances can make internal_lb_vip_address == external_lb_vip_address | 17:45 |
| noonedeadpunk | which should never happen | 17:45 |
| Guest2868 | i think i dont understand this key what is "external_lb_vip_address" | 17:45 |
| noonedeadpunk | So you need to have at least 2 IPs to handle internal_lb_vip_address and external_lb_vip_address independently | 17:45 |
| noonedeadpunk | It's many things | 17:46 |
| Guest2868 | this ip could be on the same range like 192.168.100.10 and 192.168.100.20 ? | 17:46 |
| noonedeadpunk | Among them it is a default external frontend for haproxy it binds to | 17:46 |
| noonedeadpunk | Yes, it can be from the same range, but they jsut must be different | 17:46 |
| Guest2868 | my domain should be mapped on which one? | 17:46 |
| Guest2868 | which service will use the external ip? | 17:47 |
| noonedeadpunk | Another usecase for *_lb_vip_address is provide default for your endpoints | 17:47 |
| noonedeadpunk | So, openstack service APIs get discovered from the catalog, which is part of the keystone. When you define a new service it get's created in the catalog along with it's endpoints | 17:48 |
| noonedeadpunk | Ther're multiple type of endpoints, like public and internal/admin | 17:48 |
| Guest2868 | ok and now, i should rerun setup-infra? or start with new setup-hosts? | 17:49 |
| noonedeadpunk | So external_lb_vip_address will participate in creation of public endpoint | 17:49 |
| noonedeadpunk | And internal_lb_vip_address will make an internal/admin endpoints | 17:49 |
| noonedeadpunk | And both of them can be domains, just in case | 17:49 |
| noonedeadpunk | re-run os-keystone-install.yml if you're on 2023.1 (antelope) or haproxy-server.yml if earlier | 17:50 |
| noonedeadpunk | And then haproxy also has 2 type of frontends - to handle public and internal traffic separately | 17:51 |
| noonedeadpunk | By default public one is covered with TLS while private one is not, though it's simple enough to enable TLS on both | 17:51 |
| Guest2868 | but the public endpoint should have my domain name url no? | 17:54 |
| Guest2868 | I have now un error on regenerate haproxy configuration 'bind' : invalid address: 'lb00.airtop.io' in 'lb00.airtop.io:80' | 17:56 |
| noonedeadpunk | Um but you set `haproxy_bind_external_lb_vip_address: 192.168.100.20`, or? | 17:59 |
| NeilHanlon | have to bind to an IP, not a FQDN | 17:59 |
| Guest2868 | i decide to have internal_lb_vip_address: 192.168.100.100 and external_lb_vip_address: 192.168.100.10 | 18:00 |
| noonedeadpunk | > but the public endpoint should have my domain name url no: haproxy_bind_external_lb_vip_address is affecting only haproxy. Endpoint will be still created with domain as long as external_lb_vip_address defined to domain | 18:00 |
| noonedeadpunk | Um. Well. For endpoints to be FQDN you need to keep external_lb_vip_address same, but define haproxy_bind_external_lb_vip_address on top of that | 18:02 |
| noonedeadpunk | let me find a doc for that... | 18:02 |
| noonedeadpunk | https://docs.openstack.org/openstack-ansible-haproxy_server/latest/configure-haproxy.html#overriding-the-address-haproxy-will-bind-to | 18:04 |
| Guest2868 | i do this https://paste.openstack.org/show/bSW7jMmx0Ql6tM2jSl6f/ | 18:05 |
| noonedeadpunk | So, both haproxy_bind_external_lb_vip_address and haproxy_bind_internal_lb_vip_address are already defaulting to these values | 18:05 |
| noonedeadpunk | on top of that, you won't have your domain to be in public endpoint this wasy | 18:05 |
| noonedeadpunk | let me edit it for you:) | 18:06 |
| Guest2868 | sorry but im just a web dev with no school skills with lan/network/linux/etc... Just learning by myself | 18:06 |
| noonedeadpunk | https://paste.openstack.org/show/bsFAwP8MenSzEsBEKB7y/ | 18:07 |
| noonedeadpunk | no worries :) | 18:07 |
| Guest2868 | its pretty hard to understand and find ressources on openstack, they is a lot of knowledges | 18:08 |
| noonedeadpunk | but community is rather friendly on other hand | 18:09 |
| noonedeadpunk | and nobody loves to write docs :D | 18:09 |
| Guest2868 | ok now im trying to run os-keystone-install ? | 18:11 |
| noonedeadpunk | yeah... but maybe it's wor | 18:12 |
| Guest2868 | its what? | 18:12 |
| noonedeadpunk | *worth to run setup-infrastructure as well just in case... | 18:12 |
| Guest2868 | ok | 18:12 |
| noonedeadpunk | there should be a tag to make it fast | 18:13 |
| noonedeadpunk | like openstack-ansible playbooks/setup-infrastructure.yml --tags haproxy-service-config,haproxy-config or smth like that | 18:14 |
| Guest2868 | the setup is retrying on Install HAProxy Pacakges | 18:14 |
| noonedeadpunk | ugh | 18:15 |
| Guest2868 | I know why my server have not internet anymore | 18:16 |
| Guest2868 | failed to start haproxy | 18:20 |
| Guest2868 | Starting frontend placement-front-1: cannot bind socket (Cannot assign requested address) [176.162.233.210:8780] | 18:20 |
| noonedeadpunk | wait. how you ended up with 176.162.233.210 there... | 18:21 |
| Guest2868 | this is the ip of lb00.airtop.io | 18:21 |
| noonedeadpunk | though I do see a potential issue with our new design of haproxy role damiandabrowski | 18:21 |
| noonedeadpunk | it's not easy to just reconfigure * when you have configuration that is broken | 18:22 |
| noonedeadpunk | we should add a flag to like - start from scratch | 18:23 |
| noonedeadpunk | Guest2868: can you just manually wipe content of `/etc/haproxy/conf.d/` on your lb01? | 18:24 |
| noonedeadpunk | we've changed the way we do configure haproxy to address issues in logic there and reduce potential issues during major upgrades, but new things arise :( | 18:25 |
| noonedeadpunk | and then run again the role | 18:25 |
| Guest2868 | ok running setup-infra | 18:26 |
| noonedeadpunk | (or other way around would be to return hosts records as it was until haproxy is re-configured properly) | 18:28 |
| noonedeadpunk | sorry I need to sign out for today | 18:29 |
| Guest2868 | should i wipe /etc/hosts? | 18:29 |
| noonedeadpunk | nah | 18:30 |
| noonedeadpunk | it's controlled by the role | 18:30 |
| noonedeadpunk | you might update them using openstack-hosts-setup.yml playbook if that's needed | 18:31 |
| noonedeadpunk | and use --tags openstack_hosts-file to make it fast | 18:32 |
| Guest2868 | everything was good with no error | 18:34 |
| Guest2868 | now should i run setup-openstack or only os-keystone? | 18:34 |
| noonedeadpunk | setup-openstack will execute all opensatck services setup | 18:35 |
| noonedeadpunk | os-keystone-install only keysrtone | 18:35 |
| noonedeadpunk | sry, really need to run now | 18:35 |
| Guest2868 | yes no pb | 18:35 |
| Guest2868 | really appreciate your help | 18:36 |
| Guest2868 | thank you | 18:36 |
| opendevreview | Merged openstack/openstack-ansible-os_rally stable/xena: Restore rally tests https://review.opendev.org/c/openstack/openstack-ansible-os_rally/+/897521 | 18:52 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!