| opendevreview | Merged openstack/openstack-ansible-os_neutron stable/zed: Fix typo for vpnaas_custom_config distribution https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/896711 | 00:22 |
|---|---|---|
| opendevreview | Merged openstack/openstack-ansible-lxc_hosts master: Remove lxc_cache_map variable https://review.opendev.org/c/openstack/openstack-ansible-lxc_hosts/+/897861 | 05:03 |
| opendevreview | Merged openstack/ansible-role-python_venv_build master: Drop unneeded become overrides https://review.opendev.org/c/openstack/ansible-role-python_venv_build/+/897948 | 05:35 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/ansible-role-python_venv_build stable/2023.1: Drop unneeded become overrides https://review.opendev.org/c/openstack/ansible-role-python_venv_build/+/898062 | 07:03 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/ansible-role-python_venv_build stable/zed: Drop unneeded become overrides https://review.opendev.org/c/openstack/ansible-role-python_venv_build/+/898063 | 07:03 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/ansible-role-python_venv_build stable/yoga: Drop unneeded become overrides https://review.opendev.org/c/openstack/ansible-role-python_venv_build/+/898064 | 07:03 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_neutron stable/zed: Check length of network_mappings https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/893952 | 07:04 |
| noonedeadpunk | mornings | 07:05 |
| Guest2868 | hi thank you for your help, now I have a ceph cluster OK | 07:25 |
| noonedeadpunk | hurray \o/ | 07:25 |
| Guest2868 | now I have a pb with my compute2 when I ran setup-openstack error: internal error: a secret with UUID 7e707ace-9279-4c6c-a12f-7e9dafc63a4d already defined for use with client.cinder secret | 07:26 |
| jrosser | good morning | 07:28 |
| jrosser | Guest2868: could you describe where that error was from? | 07:28 |
| Guest2868 | maybe a previous installation atempt create this problem | 07:28 |
| Guest2868 | TASK [ceph_client : Define libvirt nova secret] | 07:28 |
| noonedeadpunk | So does the output of `virsh secret-list` on compute contain `nova_ceph_client_uuid` value from user_secrets? | 07:35 |
| noonedeadpunk | Or well... what's the output of `virsh secret-list|grep 7e707ace-9279-4c6c-a12f-7e9dafc63a4d ; echo $?`? | 07:36 |
| noonedeadpunk | I kind of wonder if grep can exist with a code rather then 0 when accurance was matched | 07:40 |
| noonedeadpunk | (or command overall) | 07:41 |
| noonedeadpunk | maybe we should do smth like safe pipelining there, to ensure that it's grep exit code we're looking at | 07:41 |
| Guest2868 | im in meeting right now | 07:41 |
| noonedeadpunk | me too ヽ(。_°)ノ | 07:42 |
| Guest2868 | 7e707ace-9279-4c6c-a12f-7e9dafc63a4d ceph client.cinder secret | 07:48 |
| noonedeadpunk | what;s the exit code? | 07:48 |
| noonedeadpunk | echo $? | 07:48 |
| noonedeadpunk | should be executed right after the grep | 07:48 |
| Guest2868 | 0 | 07:48 |
| Guest2868 | sorry | 07:48 |
| noonedeadpunk | I'd suggest to re-run os-nova-install.yml.. Or well, it failed during nova installation, right? | 07:49 |
| Guest2868 | with --limit on compute2? | 07:50 |
| Guest2868 | should i remove the file in /tmp ? | 07:51 |
| noonedeadpunk | yeah. you can do that with limit, sure | 08:00 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-lxc_hosts master: Stop installing openssh and rsync to containers https://review.opendev.org/c/openstack/openstack-ansible-lxc_hosts/+/889945 | 08:01 |
| Guest2868 | same error | 08:02 |
| Guest2868 | https://paste.openstack.org/show/bXRxImrgetMq3zWdhoyF/ | 08:03 |
| admin1 | Guest2868, you should remove it with virsh secret-undefine and retry | 08:03 |
| admin1 | when a secret is already defined, it has to be manually undefined | 08:03 |
| Guest2868 | Hi, do you have the correct cmd? | 08:04 |
| admin1 | just gave it to you :) | 08:04 |
| Guest2868 | error: command 'secret-undefine' requires <secret> option | 08:04 |
| admin1 | yes | 08:04 |
| admin1 | virsh secret-list to list the secret | 08:04 |
| Guest2868 | ok thank you I deleted it | 08:05 |
| noonedeadpunk | But I guess most question is - why at all it tries to execute | 08:05 |
| noonedeadpunk | Can you provide couple of previous tasks as well? | 08:05 |
| admin1 | Guest2868, now if you re-run the playbook, it will not get stuck here | 08:05 |
| Guest2868 | Im re-running right now | 08:06 |
| noonedeadpunk | Like starting from `Check if nova secret is defined in libvirt` | 08:06 |
| admin1 | it will re-add the necessary secret and continue | 08:06 |
| Guest2868 | Do you think it's because of an old installation? | 08:07 |
| Guest2868 | Thank you admin1 setup is now OK | 08:07 |
| noonedeadpunk | Well, eventually, this task has condition when to run. And it should not run if the same secret is already present in virsh | 08:07 |
| noonedeadpunk | According to output you've provided it should be there and this task should not run simply | 08:08 |
| Guest2868 | Now, im in the dasboard | 08:09 |
| Guest2868 | dashboard | 08:09 |
| Guest2868 | I got this error: Policy doesn't allow os_compute_api:os-simple-tenant-usage:show to be performed. (HTTP 403) (Request-ID: req-1729b576-791b-4646-8762-1de5c39d3ec7) | 08:10 |
| admin1 | as admin ? | 08:11 |
| Guest2868 | yes | 08:12 |
| admin1 | i guess you are in service and if its newer 27 tag .. admin is not present in service tenant as admin | 08:12 |
| admin1 | so you have to switch to admin, edit admin role for service to add admin and then switch back | 08:12 |
| noonedeadpunk | Or use admin project I assume | 08:12 |
| admin1 | yeah .. by default it logs you in to service project | 08:12 |
| noonedeadpunk | oh? I wonder why though.... | 08:13 |
| admin1 | and if you are first timer or trying out, you get hit with this error .. | 08:13 |
| Guest2868 | yes I just view this, now on admin project everything are good | 08:13 |
| admin1 | in new ones, i have found that admin is not as admin role in service | 08:13 |
| noonedeadpunk | It should be pretty much easy to fix that actually... | 08:13 |
| Guest2868 | yes that was my conclusion | 08:13 |
| Guest2868 | how? noonedeadpunk | 08:13 |
| Guest2868 | to be admin in the service project? | 08:13 |
| admin1 | Guest2868, he meant via the code /playbooks | 08:13 |
| admin1 | you edit the project members and for admin user, select the admin role | 08:14 |
| noonedeadpunk | Well, I would not expect it to be admin in service project... Though I wouldn't expect service to be the default either | 08:14 |
| noonedeadpunk | Guest2868: somewhere in upper dropdowns you should have selection of projects | 08:14 |
| noonedeadpunk | admin1: actually, looking at nova policies, you should have a project reader role | 08:15 |
| Guest2868 | ok good, what the goal of the service project? | 08:15 |
| noonedeadpunk | and reader is implied by member... | 08:15 |
| noonedeadpunk | Guest2868: all "services" are assigned to the "service" project | 08:15 |
| admin1 | Guest2868, for those, you need to google and read a bit about it | 08:15 |
| noonedeadpunk | like nova/glance/cinder/etc to interact with each other | 08:16 |
| Guest2868 | ok, thank you for your time guys. Have a nice day | 08:16 |
| noonedeadpunk | so... if admin has access to service project at all (which I guess it should not?) - it should be able to read as well... | 08:16 |
| admin1 | it used to be admin role in service as well | 08:16 |
| noonedeadpunk | I'd really need to reproduce the env and see what's happening there... | 08:17 |
| admin1 | now its something else .. depending on what you install | 08:17 |
| * noonedeadpunk haven't checked on horizon for last 2 years | 08:17 | |
| admin1 | :) | 08:17 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_nova stable/yoga: Install libvirt-deamon for RHEL systems https://review.opendev.org/c/openstack/openstack-ansible-os_nova/+/898068 | 08:48 |
| noonedeadpunk | Huh | 10:03 |
| noonedeadpunk | Failed to download packages: Status code: 503 for https://mirrors.rockylinux.org/mirrorlist?arch=x86_64&repo=BaseOS-9 (IP: 199.232.198.132) | 10:03 |
| noonedeadpunk | NeilHanlon: ?:) | 10:03 |
| noonedeadpunk | At same time some pass | 10:04 |
| noonedeadpunk | https://zuul.opendev.org/t/openstack/build/566d23466d56482b819b0e88d9e46c0a/log/job-output.txt#13877-13894 | 10:04 |
| noonedeadpunk | I guess these are some individual repos though... | 10:05 |
| noonedeadpunk | The only way on how to solve that I guess to use infra mirrors actually | 10:05 |
| jrosser | do we yet mirror rocky? | 10:06 |
| noonedeadpunk | I think not | 10:11 |
| opendevreview | Merged openstack/openstack-ansible master: Always use on-disk openstack service git repos in CI jobs https://review.opendev.org/c/openstack/openstack-ansible/+/897707 | 11:03 |
| noonedeadpunk | jrosser: I assume that should be backported to 2023.1 to see any benefit on master for upgrade jobs? | 11:05 |
| jrosser | I think so yes | 11:06 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible stable/2023.1: Always use on-disk openstack service git repos in CI jobs https://review.opendev.org/c/openstack/openstack-ansible/+/898071 | 11:09 |
| opendevreview | Merged openstack/openstack-ansible master: Define install_method default when hosts resolution depend on it https://review.opendev.org/c/openstack/openstack-ansible/+/891697 | 11:18 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible stable/2023.1: Define install_method default when hosts resolution depend on it https://review.opendev.org/c/openstack/openstack-ansible/+/898072 | 11:23 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible stable/zed: Define install_method default when hosts resolution depend on it https://review.opendev.org/c/openstack/openstack-ansible/+/898073 | 11:23 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible stable/yoga: Define install_method default when hosts resolution depend on it https://review.opendev.org/c/openstack/openstack-ansible/+/898074 | 11:23 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible stable/xena: Define install_method default when hosts resolution depend on it https://review.opendev.org/c/openstack/openstack-ansible/+/898075 | 11:23 |
| NeilHanlon | noonedeadpunk: hm. looking, but I didn't get any pages... i wonder if we're having some micro outages | 12:15 |
| opendevreview | Marc Gariépy proposed openstack/openstack-ansible-openstack_hosts stable/2023.1: Remove rsyslog since we should use journald instead https://review.opendev.org/c/openstack/openstack-ansible-openstack_hosts/+/898076 | 13:08 |
| noonedeadpunk | NeilHanlon: well... 503 is that no backend available? So... they go down all at same time or smth in connection btw LB to backend that flaps? | 13:29 |
| noonedeadpunk | (or you ignore DOWN backends until only 1 is left?:) | 13:30 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible stable/xena: Define install_method default when hosts resolution depend on it https://review.opendev.org/c/openstack/openstack-ansible/+/898075 | 13:31 |
| noonedeadpunk | I wanna ask to do some reviews of stable branches: https://review.opendev.org/q/parentproject:openstack/openstack-ansible+branch:%255Estable/.*+status:open+ | 13:32 |
| NeilHanlon | it is _probably_ no backend available; i had one endpoint out of the balancer because I was testing CDN code, but I think it could lead to a situation where the other one restarts and leads to a 503. however, CDN is configured to serve a static backup response, which should mean no one ever "sees" a 503 when one of the services is restarting. I may | 13:32 |
| NeilHanlon | have broke that fallback code, though ... lol | 13:32 |
| noonedeadpunk | Was hoping to push version bumps this week, as I guess we've pretty much settled with existing bugs | 13:32 |
| NeilHanlon | i can take a look at reviews today | 13:32 |
| noonedeadpunk | Ah, damn. We also have that for 2023.1 kinda | 13:34 |
| noonedeadpunk | https://review.opendev.org/q/topic:bugfix-2034583 | 13:35 |
| noonedeadpunk | jrosser: I'm not sure if reply to your comment in https://review.opendev.org/c/openstack/openstack-ansible/+/897568 is fine or not? | 13:36 |
| jrosser | yes it's fine, sorry forgot to update that | 13:38 |
| jamesdenton | anyone here using a centralized keystone deployment for multiple regions? | 13:40 |
| noonedeadpunk | I actually dropped that section at once, until didn't come to compute part and realized it was needed for api as well... | 13:40 |
| noonedeadpunk | not anymore for me | 13:41 |
| jamesdenton | noonedeadpunk did you run into issue with that sort of deployment? what are you doing now? simply federated? | 13:41 |
| noonedeadpunk | right now simply standalone.... | 13:41 |
| noonedeadpunk | But yeah, looking towards switching to federation | 13:41 |
| jamesdenton | gotcha | 13:42 |
| noonedeadpunk | it's just a /o\ when you loose network connection to the region where keystone is | 13:42 |
| jamesdenton | yes, i could see that being an issue :D | 13:42 |
| noonedeadpunk | like due to upstream provider failures or anything like that | 13:42 |
| noonedeadpunk | Or if smth happened to galera... Or memcached in that region... | 13:43 |
| jamesdenton | yeah, it's a risk for sure | 13:43 |
| NeilHanlon | something something oauth2? | 13:43 |
| noonedeadpunk | Like all your HA cross-region deployment becomes a rubish | 13:43 |
| noonedeadpunk | Yeah, and KeyCloak. But our biggest issue is is that we have tons of domains | 13:44 |
| noonedeadpunk | And federation as of today is not designed a lot for that scenario | 13:44 |
| jamesdenton | 1 per "customer"? | 13:44 |
| noonedeadpunk | yeah | 13:44 |
| NeilHanlon | saw this last night, which is kinda neat https://github.com/anderspitman/obligator | 13:44 |
| noonedeadpunk | We were looking at this blueprint actually as long as yesterday https://review.opendev.org/c/openstack/keystone-specs/+/748042/12/specs/keystone/2023.1/versioning-for-attribute-mapping-schema.rst | 13:45 |
| noonedeadpunk | Which _supposedly_ could solve that multi-domain hussle | 13:45 |
| * NeilHanlon adds that to his 'to read' list | 13:45 | |
| noonedeadpunk | Or well, there's also a keystone provider from vexxhost that can jsut proxy keystone requests directly to keycloak without need of federation, but then you kinda loose pretty much of oauth features | 13:46 |
| mgariepy | hmm anyway to override uwsgi of nova-api-os-compute but not of nova-api-metadata ? | 13:47 |
| noonedeadpunk | nova_api_os_compute_uwsgi_ini_overrides ? | 13:51 |
| noonedeadpunk | or what do you mean by uwsgi override? | 13:51 |
| mgariepy | exactly that lol. | 13:52 |
| mgariepy | default config needs some tweaks for the load i get from some users .. | 13:54 |
| noonedeadpunk | would be interesting to hear what tweks :) | 13:54 |
| noonedeadpunk | (one day) | 13:55 |
| mgariepy | nova is tracebacking and overloaded.. | 13:56 |
| mgariepy | so i want to set the nova_api_db_max_overflow, nova_api_db_max_pool_size and uwsgi_processes a bit higher. | 13:57 |
| noonedeadpunk | ah, and nova_wsgi_processes is indeed same for api/metadata | 13:59 |
| mgariepy | yeah default works ok for most of my case.. but not that one. | 13:59 |
| opendevreview | Merged openstack/openstack-ansible-os_zun stable/zed: Install kata containers from source https://review.opendev.org/c/openstack/openstack-ansible-os_zun/+/883711 | 14:25 |
| -opendevstatus- NOTICE: The lists.openstack.org site will be offline over the next few hours for migration to a new server | 15:30 | |
| opendevreview | Merged openstack/openstack-ansible-os_keystone stable/zed: oidc: fix recognition of x forwarded headers from v2.4.11 https://review.opendev.org/c/openstack/openstack-ansible-os_keystone/+/897807 | 20:53 |
| opendevreview | Merged openstack/openstack-ansible-openstack_hosts stable/2023.1: Remove rsyslog since we should use journald instead https://review.opendev.org/c/openstack/openstack-ansible-openstack_hosts/+/898076 | 21:03 |
| opendevreview | Merged openstack/openstack-ansible-os_nova stable/yoga: Install libvirt-deamon for RHEL systems https://review.opendev.org/c/openstack/openstack-ansible-os_nova/+/898068 | 21:27 |
| opendevreview | Merged openstack/openstack-ansible-os_neutron stable/zed: Check length of network_mappings https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/893952 | 21:33 |
| opendevreview | Merged openstack/ansible-role-python_venv_build stable/2023.1: Drop unneeded become overrides https://review.opendev.org/c/openstack/ansible-role-python_venv_build/+/898062 | 21:38 |
| opendevreview | Merged openstack/openstack-ansible stable/2023.1: Always use on-disk openstack service git repos in CI jobs https://review.opendev.org/c/openstack/openstack-ansible/+/898071 | 21:42 |
| opendevreview | Merged openstack/openstack-ansible-os_neutron stable/zed: Stop haproxy on ovn-controller nodes https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/893451 | 21:43 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!