| noonedeadpunk | for base lxc image - we have a variable at very least to make it re-build. as otherwise it's expiring only by a timeout | 07:10 |
|---|---|---|
| noonedeadpunk | good morning :) | 07:10 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: Start 2024.2 (Dalmatian) development https://review.opendev.org/c/openstack/openstack-ansible/+/921367 | 07:15 |
| noonedeadpunk | but yeah, apparently we don't verify base image for alignment with underlying OS | 07:15 |
| jrosser | I would expect it to make a different named dir in /var/lib/machines then trigger all the rest of the image build because of that | 07:23 |
| jrosser | just like happens in the initial deployment | 07:23 |
| noonedeadpunk | yeah, me to and I did saw that happening at least in 2023.1 | 07:25 |
| noonedeadpunk | as deflateder said actually | 07:26 |
| noonedeadpunk | probably I messed up with some condition fixing cache timeouts.... | 07:26 |
| noonedeadpunk | I think it has smth to do with this logic https://opendev.org/openstack/openstack-ansible-lxc_hosts/src/branch/master/tasks/lxc_cache.yml#L26-L42 | 07:28 |
| noonedeadpunk | let me try to reproduce I guess. As that's really interesting | 07:29 |
| jrosser | yeah so the entire code path is now conditional only on the cache expiry | 08:26 |
| jrosser | this would be the thing you could register to know that an upgraded host wants to build a new image https://opendev.org/openstack/openstack-ansible-lxc_hosts/src/branch/master/tasks/lxc_cache_rootfs.yml#L51-L56 | 08:27 |
| noonedeadpunk | well, I think idea was, is that SLURP will fail | 08:31 |
| noonedeadpunk | this one: https://opendev.org/openstack/openstack-ansible-lxc_hosts/src/branch/master/tasks/lxc_cache.yml#L26-L34 | 08:31 |
| noonedeadpunk | and if it does - `expiry.content | default('MQo=') | b64decode | int` should result in evaluating next task as setting `lxc_image_cache_refresh: true` | 08:32 |
| noonedeadpunk | (as no content will exist) | 08:32 |
| noonedeadpunk | as cache_path_fact depends on distribution, distribution_major_version and architecture | 08:33 |
| noonedeadpunk | jrosser: so... I spawned Debian 11 infra_lxc aio. Then replaced repos and did upgraded to Debian 12. Did utility container removal. After that - ran lxc-hosts-setup.yml (to create new image), and lxc-containers-create.yml --limit aio1,utility_all created new utility container with Debian 12 in it. | 08:59 |
| noonedeadpunk | new image was created nicely: https://paste.openstack.org/show/bTzCM7HbNuzgAYUrS9VG/ | 09:00 |
| noonedeadpunk | so - can't really confirm there's an issue with logic we have right now | 09:01 |
| jrosser | interesting | 09:01 |
| noonedeadpunk | but also - I did removed facts for both aio and container | 09:01 |
| jrosser | well would be interesting to see what deflateder has in /var/lib/machines | 09:01 |
| noonedeadpunk | do you recall folks claoming on messaging timeout for nova during live migrations even with quorum queues? | 09:15 |
| noonedeadpunk | Apparently, the reason is this: https://opendev.org/openstack/openstack-ansible-os_nova/src/branch/master/defaults/main.yml#L136 | 09:16 |
| noonedeadpunk | as we seen the same, but seems that just setting it to False - did solve issue for us | 09:16 |
| noonedeadpunk | we're still monitoring/testing that, but I guess as soon as oslo has reverted default, enabling it now breaks uwsgi as well. | 09:17 |
| jrosser | i think andrew might have raised a nova bug for something like this | 09:21 |
| noonedeadpunk | yeah | 09:23 |
| jrosser | https://bugs.launchpad.net/nova/+bug/2060931 | 09:28 |
| noonedeadpunk | yes, so exactly that | 09:29 |
| noonedeadpunk | what we also saw | 09:29 |
| noonedeadpunk | and disabling pthread for nova-api seemed to help | 09:29 |
| noonedeadpunk | well. our stack trace is slightly different... | 09:37 |
| noonedeadpunk | https://paste.openstack.org/show/bsoMxqEjI6iQ7yXIzc7t/ | 09:37 |
| jrosser | i think we may not be on quorum queues yet | 09:40 |
| jrosser | if that matters | 09:40 |
| noonedeadpunk | we're also not | 09:43 |
| noonedeadpunk | and it's even antelope... | 09:43 |
| noonedeadpunk | though stack trace is slightly different, though taken from compute side | 09:43 |
| jrosser | looking at the bug it seems we have some difficulty reproducing it | 09:44 |
| noonedeadpunk | for us it occurs randomly during live migrations | 09:44 |
| jrosser | so if yours is more reproducible then its worth to try to see if the stack track is "real" or actually there is something else hidden under it | 09:44 |
| noonedeadpunk | so out of 50 live migrations we were catching it on around 10 | 09:44 |
| jrosser | hmm sounds familiar | 09:44 |
| noonedeadpunk | and as we're upgrading OS right now - we have plenty of them | 09:45 |
| jrosser | yes exactly the same as when we came on this, OS upgrades and shuffling VM all around to do that | 09:45 |
| noonedeadpunk | though both source and destination compute have this eventlent reference at the very beginning of stack trace | 09:46 |
| noonedeadpunk | which is absent on API side (while only api is using pthreads by our default) | 09:46 |
| noonedeadpunk | So, folks are monitoring state after fully disabling heartbeat_in_pthread (which was True for API and False for computes to jsut False) | 09:47 |
| jamesden- | good morning jrosser. Have an interesting one for ya. Did an A->C upgrade yesterday and enabled Skyline along the way. Found that the /horizon mapping was working properly to send to horizon-back, but the / mapping was not working and root domain (https://dashboard.com/ was matching on .* in the base_regex.map and sending to horizon-back, but actually returning the index.html page from /var/www/html/index.html instead of | 10:00 |
| jamesden- | horizon! Really wacky stuff. Still trying to figure out what's up. | 10:00 |
| jrosser | huh - you have some overrides for this i guess? | 10:04 |
| jamesden- | haproxy overides? | 10:05 |
| jamesden- | nothing related to mapping | 10:06 |
| jrosser | i think this might be a question for noonedeadpunk tbh | 10:08 |
| jamesden- | no worries - i will try and replicate in an AIO and see if there's anything funky happening there | 10:08 |
| noonedeadpunk | there was some `/` at the end difference iirc | 10:09 |
| noonedeadpunk | though I think AIO was working nicely for me | 10:09 |
| noonedeadpunk | unless, I screwed smth with moving skyline to apache from nginx | 10:09 |
| noonedeadpunk | jamesden-: might be you have some `horizon_webroot` or `skyline_webroot` override somewhere? | 10:11 |
| jamesden- | nothing like that, no | 10:11 |
| noonedeadpunk | then... my next guess would be some ordering issue in mapping for haproxy... | 10:12 |
| noonedeadpunk | as prio was important there iirc | 10:12 |
| noonedeadpunk | and I think that `base_regex` I've reduced in prio | 10:12 |
| noonedeadpunk | though. I never tried an upgrade.... | 10:13 |
| noonedeadpunk | as that was 99 : https://opendev.org/openstack/openstack-ansible/src/branch/master/inventory/group_vars/horizon_all/haproxy_service.yml#L43 | 10:13 |
| noonedeadpunk | jamesden-: that's how it looks in aio: https://paste.openstack.org/show/bSba8w8Jt0uabeFPGQ8O/ | 10:15 |
| noonedeadpunk | so I assume, if you have, say, /etc/haproxy/map.conf.d/base_regex/99-horizon.map.... | 10:15 |
| jamesden- | https://paste.opendev.org/show/bspNhwUtnTz0k1EPgsn6/ | 10:15 |
| jamesden- | interesting. I also have a 99-horizon | 10:15 |
| noonedeadpunk | that can be the issue | 10:15 |
| noonedeadpunk | ok, so upgrade path should actually include removal of 99-horizon.map then :( | 10:16 |
| jrosser | afaik there *should* be a state: absent possible for map entries | 10:17 |
| jamesden- | i have a 99-horizon and 99-skyline. i moved 99-skyline to 97-skyline, which my reorder in base_regex effectively did | 10:17 |
| noonedeadpunk | yeah | 10:17 |
| noonedeadpunk | nah, you'd ned to drop 99-horizon | 10:18 |
| jamesden- | ok | 10:18 |
| noonedeadpunk | it should be 98-horizon and 99-skyline left | 10:18 |
| noonedeadpunk | but eventually, assembled /etc/haproxy/base_regex.map is what makes difference | 10:19 |
| noonedeadpunk | yeah, I will push a patch to fix that | 10:19 |
| jamesden- | sure, thank you | 10:19 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-tests master: Bump Ansible version for tests repo https://review.opendev.org/c/openstack/openstack-ansible-tests/+/921434 | 10:28 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-tests stable/2024.1: Bump Ansible version for tests repo https://review.opendev.org/c/openstack/openstack-ansible-tests/+/921435 | 10:30 |
| opendevreview | OpenStack Release Bot proposed openstack/openstack-ansible-tests stable/2024.1: Update TOX_CONSTRAINTS_FILE for stable/2024.1 https://review.opendev.org/c/openstack/openstack-ansible-tests/+/921253 | 10:31 |
| opendevreview | Merged openstack/openstack-ansible-plugins master: reno: Update master for unmaintained/zed https://review.opendev.org/c/openstack/openstack-ansible-plugins/+/919204 | 10:37 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible stable/2024.1: Remove old horizon base mapping https://review.opendev.org/c/openstack/openstack-ansible/+/921436 | 10:38 |
| noonedeadpunk | jamesden-: can you check this out? ^ | 10:39 |
| jamesden- | looking | 10:39 |
| jamesden- | what happens if they don't enable skyline? will that also be absent? | 10:40 |
| jamesden- | and does it mattert? | 10:40 |
| opendevreview | Merged openstack/openstack-ansible-os_gnocchi master: reno: Update master for unmaintained/zed https://review.opendev.org/c/openstack/openstack-ansible-os_gnocchi/+/919161 | 10:45 |
| opendevreview | Merged openstack/openstack-ansible-os_tempest master: reno: Update master for unmaintained/zed https://review.opendev.org/c/openstack/openstack-ansible-os_tempest/+/919198 | 10:45 |
| opendevreview | Merged openstack/openstack-ansible-os_trove master: reno: Update master for unmaintained/zed https://review.opendev.org/c/openstack/openstack-ansible-os_trove/+/919200 | 10:45 |
| opendevreview | Merged openstack/openstack-ansible-rabbitmq_server master: reno: Update master for unmaintained/zed https://review.opendev.org/c/openstack/openstack-ansible-rabbitmq_server/+/919206 | 10:45 |
| opendevreview | Merged openstack/openstack-ansible-openstack_hosts master: reno: Update master for unmaintained/zed https://review.opendev.org/c/openstack/openstack-ansible-openstack_hosts/+/919138 | 10:45 |
| opendevreview | Merged openstack/openstack-ansible-os_nova master: reno: Update master for unmaintained/zed https://review.opendev.org/c/openstack/openstack-ansible-os_nova/+/919182 | 10:46 |
| opendevreview | Merged openstack/openstack-ansible-repo_server master: reno: Update master for unmaintained/zed https://review.opendev.org/c/openstack/openstack-ansible-repo_server/+/919208 | 10:46 |
| opendevreview | Merged openstack/ansible-hardening master: reno: Update master for unmaintained/zed https://review.opendev.org/c/openstack/ansible-hardening/+/919107 | 10:47 |
| noonedeadpunk | there's a record jsut with another priority | 10:47 |
| opendevreview | Merged openstack/openstack-ansible-os_placement master: reno: Update master for unmaintained/zed https://review.opendev.org/c/openstack/openstack-ansible-os_placement/+/919186 | 10:47 |
| opendevreview | Merged openstack/ansible-role-qdrouterd master: reno: Update master for unmaintained/zed https://review.opendev.org/c/openstack/ansible-role-qdrouterd/+/919112 | 10:47 |
| noonedeadpunk | we're placing it with prio 98 and removing with prio 99 | 10:47 |
| opendevreview | Merged openstack/openstack-ansible-os_tacker master: reno: Update master for unmaintained/zed https://review.opendev.org/c/openstack/openstack-ansible-os_tacker/+/919196 | 10:48 |
| noonedeadpunk | so that skyline could chime in with prio 99 | 10:48 |
| opendevreview | Merged openstack/openstack-ansible-haproxy_server master: reno: Update master for unmaintained/zed https://review.opendev.org/c/openstack/openstack-ansible-haproxy_server/+/919130 | 10:48 |
| opendevreview | Merged openstack/openstack-ansible-tests master: reno: Update master for unmaintained/zed https://review.opendev.org/c/openstack/openstack-ansible-tests/+/919210 | 10:48 |
| noonedeadpunk | but potentially... we need to drop prio even more to let users do smth on top of these 2 | 10:48 |
| opendevreview | Merged openstack/openstack-ansible-os_mistral master: reno: Update master for unmaintained/zed https://review.opendev.org/c/openstack/openstack-ansible-os_mistral/+/919176 | 10:48 |
| opendevreview | Merged openstack/openstack-ansible-os_swift master: reno: Update master for unmaintained/zed https://review.opendev.org/c/openstack/openstack-ansible-os_swift/+/919194 | 10:49 |
| jamesden- | i am also seeing the skyline user missing from keystone/ Hm | 10:49 |
| opendevreview | Merged openstack/openstack-ansible-os_octavia master: reno: Update master for unmaintained/zed https://review.opendev.org/c/openstack/openstack-ansible-os_octavia/+/919184 | 10:49 |
| opendevreview | Merged openstack/openstack-ansible-os_senlin master: reno: Update master for unmaintained/zed https://review.opendev.org/c/openstack/openstack-ansible-os_senlin/+/919192 | 10:49 |
| opendevreview | Merged openstack/openstack-ansible-os_horizon master: reno: Update master for unmaintained/zed https://review.opendev.org/c/openstack/openstack-ansible-os_horizon/+/919165 | 10:49 |
| opendevreview | Merged openstack/openstack-ansible-os_barbican master: reno: Update master for unmaintained/zed https://review.opendev.org/c/openstack/openstack-ansible-os_barbican/+/919147 | 10:50 |
| noonedeadpunk | jamesden-: huh, and did installation finished with success? | 10:50 |
| noonedeadpunk | it's where it should be included https://opendev.org/openstack/openstack-ansible-os_skyline/src/branch/master/tasks/main.yml#L76-L99 | 10:50 |
| jamesden- | yes. i have re-run the playbook multiple times since | 10:50 |
| jamesden- | Looks like it's skipped | 10:51 |
| noonedeadpunk | and you have `skyline_all` group in inventory? | 10:52 |
| noonedeadpunk | and don't have any custom env.d file for skyline? | 10:52 |
| opendevreview | Merged openstack/ansible-role-zookeeper master: reno: Update master for unmaintained/zed https://review.opendev.org/c/openstack/ansible-role-zookeeper/+/919122 | 10:52 |
| jamesden- | let me double check that | 10:52 |
| opendevreview | Merged openstack/openstack-ansible-os_glance master: reno: Update master for unmaintained/zed https://review.opendev.org/c/openstack/openstack-ansible-os_glance/+/919159 | 10:52 |
| noonedeadpunk | as that could change a bit from initial variant: https://opendev.org/openstack/openstack-ansible/src/branch/master/inventory/env.d/skyline.yml | 10:52 |
| noonedeadpunk | that you tested | 10:52 |
| opendevreview | Merged openstack/openstack-ansible-os_ironic master: reno: Update master for unmaintained/zed https://review.opendev.org/c/openstack/openstack-ansible-os_ironic/+/919167 | 10:53 |
| opendevreview | Merged openstack/openstack-ansible-os_heat master: reno: Update master for unmaintained/zed https://review.opendev.org/c/openstack/openstack-ansible-os_heat/+/919163 | 10:53 |
| opendevreview | Merged openstack/openstack-ansible-os_rally master: reno: Update master for unmaintained/zed https://review.opendev.org/c/openstack/openstack-ansible-os_rally/+/919188 | 10:53 |
| jamesden- | that looks OK and the inventory itself has the right hosts and children | 10:54 |
| jamesden- | there was no skyline previously here. I just updated openstack_user_config to include -- skyline_dashboard_hosts: *controller_hosts | 10:54 |
| jamesden- | so i might be missing something | 10:55 |
| jamesden- | https://paste.opendev.org/show/byrxixeMC0vzoYeFwTHs/ | 10:55 |
| noonedeadpunk | that should be fine | 10:56 |
| opendevreview | OpenStack Proposal Bot proposed openstack/openstack-ansible-plugins master: Updated from OpenStack Ansible Tests https://review.opendev.org/c/openstack/openstack-ansible-plugins/+/921439 | 10:58 |
| noonedeadpunk | jamesden-: so in aio (lxc one though) in openstack_inventory.json I have smth like that https://paste.opendev.org/show/b9wP0q8q4tVmY2RMhPcq/ | 10:58 |
| noonedeadpunk | and you should have all 3 controllers there pretty much | 10:59 |
| noonedeadpunk | the condition is actually here: https://opendev.org/openstack/openstack-ansible-os_skyline/src/branch/master/vars/main.yml#L16-L20 | 10:59 |
| noonedeadpunk | `skyline_services['skyline-api']['group']` - should result in skyline_all: https://opendev.org/openstack/openstack-ansible-os_skyline/src/branch/master/defaults/main.yml#L139 | 11:00 |
| jamesden- | https://paste.opendev.org/show/bQAQ5qdNbVK2bAbLee4v/ | 11:00 |
| noonedeadpunk | also, jsut in case, as playbook runs in serial, this should be only included for the first host | 11:00 |
| jamesden- | yep, that all looks OK. I will dig around | 11:00 |
| jamesden- | makes sense | 11:01 |
| noonedeadpunk | which would be 910400-ord-controller01 | 11:01 |
| jamesden- | i am running from a dedicated deploy host, but i don't think that would make any difference? | 11:01 |
| noonedeadpunk | oh | 11:02 |
| noonedeadpunk | that;s interesting | 11:02 |
| noonedeadpunk | https://paste.openstack.org/show/bwTq1TCA9Udci1D6Qgx1/ | 11:02 |
| jamesden- | yeah | 11:02 |
| noonedeadpunk | so it did included role | 11:02 |
| noonedeadpunk | (on the contrary to other controllers) | 11:03 |
| jamesden- | right, yes | 11:03 |
| noonedeadpunk | so that;s pretty much the condition then https://opendev.org/openstack/openstack-ansible-plugins/blame/branch/master/roles/service_setup/tasks/main.yml#L83-L88 | 11:05 |
| noonedeadpunk | I would try to chime-in above with some debug then | 11:06 |
| jamesden- | i will poke around and see if there is some var set here causing that to skip | 11:07 |
| noonedeadpunk | don't you happen to have `service_ldap_backend_enabled: True` or smth? | 11:07 |
| jamesden- | thabk you for the pointers | 11:07 |
| noonedeadpunk | that is the default for it: https://opendev.org/openstack/openstack-ansible/src/branch/master/inventory/group_vars/all/all.yml#L96 | 11:08 |
| jamesden- | def no ldap here | 11:08 |
| noonedeadpunk | ok | 11:08 |
| noonedeadpunk | I don't see why it would be skipped otherwise frankly speaking | 11:08 |
| jamesden- | user_secrets did get updated automatically the the changes task from docs | 11:09 |
| opendevreview | Merged openstack/openstack-ansible-lxc_container_create master: reno: Update master for unmaintained/zed https://review.opendev.org/c/openstack/openstack-ansible-lxc_container_create/+/919132 | 11:09 |
| jamesden- | so the service password is there | 11:09 |
| opendevreview | Merged openstack/ansible-role-uwsgi master: reno: Update master for unmaintained/zed https://review.opendev.org/c/openstack/ansible-role-uwsgi/+/919120 | 11:09 |
| opendevreview | Merged openstack/openstack-ansible-galera_server master: reno: Update master for unmaintained/zed https://review.opendev.org/c/openstack/openstack-ansible-galera_server/+/919128 | 11:10 |
| opendevreview | Merged openstack/openstack-ansible-lxc_hosts master: reno: Update master for unmaintained/zed https://review.opendev.org/c/openstack/openstack-ansible-lxc_hosts/+/919134 | 11:10 |
| jamesden- | running w/ debug and will see. Might add something to service_setup to see if i'm missing anything | 11:10 |
| opendevreview | Merged openstack/openstack-ansible-os_aodh master: reno: Update master for unmaintained/zed https://review.opendev.org/c/openstack/openstack-ansible-os_aodh/+/919145 | 11:10 |
| noonedeadpunk | yeah, that might give some more view | 11:10 |
| opendevreview | Merged openstack/openstack-ansible-os_ceilometer master: reno: Update master for unmaintained/zed https://review.opendev.org/c/openstack/openstack-ansible-os_ceilometer/+/919151 | 11:11 |
| opendevreview | Merged openstack/openstack-ansible-memcached_server master: reno: Update master for unmaintained/zed https://review.opendev.org/c/openstack/openstack-ansible-memcached_server/+/919136 | 11:11 |
| noonedeadpunk | ok, good news - upgrade jobs are fixed now for 2024.1 | 11:13 |
| noonedeadpunk | and https://review.opendev.org/c/openstack/openstack-ansible-os_cloudkitty/+/921285 is kinda ready | 11:13 |
| jamesden- | brilliant | 11:14 |
| noonedeadpunk | gnocchi catched timeouts: https://review.opendev.org/c/openstack/openstack-ansible-os_gnocchi/+/921284 | 11:14 |
| opendevreview | Merged openstack/openstack-ansible-openstack_openrc master: reno: Update master for unmaintained/zed https://review.opendev.org/c/openstack/openstack-ansible-openstack_openrc/+/919140 | 11:15 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: Switch services SHAs to track master https://review.opendev.org/c/openstack/openstack-ansible/+/921368 | 11:15 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: Drop support for Debian 11 (bullseye) https://review.opendev.org/c/openstack/openstack-ansible/+/921370 | 11:15 |
| opendevreview | Merged openstack/ansible-role-systemd_mount master: reno: Update master for unmaintained/zed https://review.opendev.org/c/openstack/ansible-role-systemd_mount/+/919114 | 11:16 |
| opendevreview | Merged openstack/ansible-role-systemd_service master: reno: Update master for unmaintained/zed https://review.opendev.org/c/openstack/ansible-role-systemd_service/+/919118 | 11:17 |
| jamesden- | oh man | 11:17 |
| jamesden- | roles | 11:17 |
| jamesden- | https://paste.openstack.org/show/baENUVTMjzoHPo2goCBD/ | 11:17 |
| jamesden- | skyline_role_name vs skyline_role_names | 11:17 |
| jamesden- | vaguely familiar | 11:18 |
| jamesden- | https://paste.openstack.org/show/baENUVTMjzoHPo2goCBD/ | 11:18 |
| noonedeadpunk | doh | 11:19 |
| noonedeadpunk | I missed to update it /o\ https://opendev.org/openstack/openstack-ansible-os_skyline/commit/677575b61c43306d50766279c3b462d5da1d0454 | 11:20 |
| opendevreview | Merged openstack/ansible-role-python_venv_build master: reno: Update master for unmaintained/zed https://review.opendev.org/c/openstack/ansible-role-python_venv_build/+/919110 | 11:20 |
| jamesden- | will service_setup iterate on the list of roles? | 11:21 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_skyline master: Fix skyline user creation https://review.opendev.org/c/openstack/openstack-ansible-os_skyline/+/921442 | 11:22 |
| noonedeadpunk | it pretty much should, yes | 11:22 |
| noonedeadpunk | I've pushed a hot fix ^ | 11:23 |
| jamesden- | testing | 11:23 |
| jamesden- | ahhh there we go, much better | 11:25 |
| noonedeadpunk | you're super nice in timing for your tests :) | 11:25 |
| noonedeadpunk | as it's the deadline for the final release today, lol | 11:26 |
| jamesden- | The login region menu is populated and logins are working. Not so great once i'm in but there might be something else wrong on my side. | 11:26 |
| jamesden- | haha yes, good timing indeed. thank you | 11:26 |
| noonedeadpunk | what;s inside? | 11:27 |
| noonedeadpunk | crap, I just need to spawn an env I guess | 11:27 |
| jamesden- | oh, my admin account can't pull up service project resources for some reason | 11:27 |
| jamesden- | so getting errors | 11:28 |
| noonedeadpunk | ah | 11:28 |
| noonedeadpunk | check the project | 11:28 |
| noonedeadpunk | or domain | 11:28 |
| noonedeadpunk | as admin does not have access to service project | 11:28 |
| jamesden- | it was better when i switched to admin project, but it's defaulting to service project | 11:28 |
| noonedeadpunk | there was a bug in... heat or magnum I've addressed quite recently | 11:28 |
| jamesden- | good to know | 11:29 |
| noonedeadpunk | https://opendev.org/openstack/openstack-ansible-os_heat/commit/408f794e2a1d4162cc5211d0c94953ef08b5c0b7 | 11:29 |
| jamesden- | should one retroactively address this? | 11:30 |
| noonedeadpunk | well... | 11:30 |
| noonedeadpunk | we can revoke role explicitly for upgrade path | 11:31 |
| noonedeadpunk | but I kinda afraid to do role revoking things... | 11:31 |
| jamesden- | of course | 11:31 |
| jamesden- | just wondering about a manual change | 11:32 |
| noonedeadpunk | but yeah, unlikely one needed a service role for their admin user... | 11:32 |
| noonedeadpunk | openstack role remove --user admin --project service service iirc | 11:33 |
| noonedeadpunk | --domain Default | 11:33 |
| jamesden- | let me check my role assignments | 11:33 |
| jamesden- | i have the heat_stack_owner role against service and admin projects | 11:35 |
| noonedeadpunk | ok, yeah, might be that... | 11:36 |
| jamesden- | is it enough to just unassign that particular role, then? | 11:36 |
| noonedeadpunk | yup | 11:36 |
| noonedeadpunk | at least from service project | 11:36 |
| noonedeadpunk | as the issue is that the only role assigned to the service project doens't give much privileges to it | 11:37 |
| jamesden- | ok, looks better in both horizon and skyline | 11:37 |
| jamesden- | thanks again | 11:37 |
| jamesden- | and good catch | 11:37 |
| jamesden- | what prompted the move from nginx to apache? | 11:38 |
| noonedeadpunk | well, ideally we wanna have one web server to manage | 11:40 |
| noonedeadpunk | (and maintain) | 11:40 |
| noonedeadpunk | and as for keystone nginx can't provide all reauired functionality (ie mod_oidc) | 11:40 |
| noonedeadpunk | and horizon also running apache... | 11:40 |
| jamesden- | gotcha | 11:40 |
| noonedeadpunk | the only thing we left is repo container | 11:40 |
| noonedeadpunk | which should be possible/trivial to move to apache as well... | 11:41 |
| noonedeadpunk | and then on metal you have -1 service to keep track of | 11:41 |
| jamesden- | side note - if you have skyline enabled, can you try hitting the Identity menu from the administrator panel? I get 'Error - Unable to get Data'. Not sure if this is me or skyline | 11:42 |
| jamesden- | lol n/m. this thing can be janky | 11:43 |
| jamesden- | now it's working | 11:43 |
| jamesden- | a hard refresh did the needful | 11:44 |
| noonedeadpunk | as it's nodejs it tends to cache plenty of stuff on client side | 11:45 |
| jamesden- | thanks for the quick turnaround on these issues :) | 11:46 |
| jamesden- | Re: https://review.opendev.org/c/openstack/openstack-ansible/+/921436 - won't this also delete 99-horizon even is skyline is not enabled? | 11:47 |
| noonedeadpunk | it will | 11:50 |
| noonedeadpunk | but it's fine as 98 will still be created? | 11:50 |
| noonedeadpunk | as otherwise record will be duplicated anyway? | 11:50 |
| jamesden- | ok sure, wasn't sure how important that second match was | 11:52 |
| noonedeadpunk | jamesden-: soI just did renamed 99 to 98 here without thinking of upgrades: https://opendev.org/openstack/openstack-ansible/commit/2c06a65a75f501c30376f7369f4b7eccd3f0523a#diff-4cabb73f913b9590085df9cbe0b7908b98855b84 | 11:54 |
| noonedeadpunk | so it should be just replaced | 11:54 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-plugins master: Allow to remove role assignment from users https://review.opendev.org/c/openstack/openstack-ansible-plugins/+/921443 | 12:00 |
| opendevreview | Merged openstack/openstack-ansible stable/2023.1: Enable RabbitMQ feature flags pre-upgrade https://review.opendev.org/c/openstack/openstack-ansible/+/921000 | 12:00 |
| jamesden- | ahh ok, i see now. thanks | 12:06 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_heat master: Handle removal of wrong role assignment https://review.opendev.org/c/openstack/openstack-ansible-os_heat/+/921444 | 12:09 |
| noonedeadpunk | jamesden-: well... I _guess_ this might be covering role assignment thing ^ | 12:10 |
| noonedeadpunk | though I kinda dunno if it makes much sense given timeline, and that it only affects admin user.... | 12:11 |
| noonedeadpunk | and how backportable is that even... | 12:11 |
| noonedeadpunk | probably would be good fit jsut for 2024.1.... | 12:12 |
| noonedeadpunk | jrosser: wdyt? | 12:12 |
| noonedeadpunk | just a bit scary to bring the dependency to 2024.1 right now | 12:12 |
| * jrosser gathers context..... | 12:13 | |
| noonedeadpunk | context basically - clean-up after fixing https://opendev.org/openstack/openstack-ansible-os_heat/commit/408f794e2a1d4162cc5211d0c94953ef08b5c0b7 | 12:13 |
| noonedeadpunk | where we were assigning heat_stack_owner role to admin user on service project | 12:14 |
| noonedeadpunk | which kinda makes admin to see service project, but not having any vaulable permissions there | 12:14 |
| noonedeadpunk | (which is annoying for admins) | 12:14 |
| jamesden- | i don't mind the removal, especially if the assignment was done automatically and provides no value | 12:16 |
| noonedeadpunk | I'm kinda scared to backport https://review.opendev.org/c/openstack/openstack-ansible-plugins/+/921443 as not sure in it a all | 12:18 |
| noonedeadpunk | but maybe for 29.0.1 or smth it would be fine... | 12:18 |
| noonedeadpunk | jamesden-: as you;'re around - would be very helpful if you could check on https://review.opendev.org/c/openstack/openstack-ansible/+/918979 | 12:21 |
| noonedeadpunk | smth we're quite overdue with at this point... | 12:21 |
| jamesden- | looking | 12:28 |
| jamesden- | comment left but otherwise looks OK and i can +2 that if it does or doesnt need to be addressed | 12:31 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible stable/zed: Transition Zed to Unmaintained https://review.opendev.org/c/openstack/openstack-ansible/+/918979 | 12:32 |
| noonedeadpunk | good catch | 12:32 |
| jamesden- | done, thank you | 12:34 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_horizon stable/2024.1: Fix horizon distro installation for EL https://review.opendev.org/c/openstack/openstack-ansible-os_horizon/+/921445 | 12:37 |
| opendevreview | Merged openstack/openstack-ansible master: [doc] Properly promote deploy guide https://review.opendev.org/c/openstack/openstack-ansible/+/921237 | 12:48 |
| opendevreview | Merged openstack/openstack-ansible master: [doc] Remove empty pages around ToC in PDF https://review.opendev.org/c/openstack/openstack-ansible/+/921273 | 13:14 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible stable/2024.1: [doc] Properly promote deploy guide https://review.opendev.org/c/openstack/openstack-ansible/+/921447 | 13:32 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-tests stable/2024.1: Bump Ansible version for tests repo https://review.opendev.org/c/openstack/openstack-ansible-tests/+/921435 | 13:36 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-tests master: Bump Ansible version for tests repo https://review.opendev.org/c/openstack/openstack-ansible-tests/+/921434 | 13:37 |
| noonedeadpunk | gnocchi backport just passed: https://review.opendev.org/c/openstack/openstack-ansible-os_gnocchi/+/921284 | 14:23 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-tests master: Bump Ansible version for tests repo https://review.opendev.org/c/openstack/openstack-ansible-tests/+/921434 | 14:48 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-tests master: Bump Ansible version for tests repo https://review.opendev.org/c/openstack/openstack-ansible-tests/+/921434 | 14:53 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-tests master: Bump Ansible version for tests repo https://review.opendev.org/c/openstack/openstack-ansible-tests/+/921434 | 14:55 |
| jamesden- | @noonedeadpunk did some rabbit vhosts get renamed as part of the upgrade? | 15:06 |
| jamesden- | ok, yeah i see the slash and no slash | 15:11 |
| jamesden- | lemme rerun octavia playbook, transport url didn't get updated | 15:12 |
| noonedeadpunk | yup | 15:16 |
| noonedeadpunk | so it's in release notes, but in short - depending on the quorum queues value | 15:16 |
| noonedeadpunk | if quorum queues are enabled - vhost will be changed | 15:17 |
| noonedeadpunk | and they are enabled by default now | 15:17 |
| jamesden- | got it - i think i had a failure in octavia playbook and it didnt make the change | 15:17 |
| noonedeadpunk | well, frankly - octavia is a scary case here. | 15:18 |
| noonedeadpunk | oh well. sorry. mixed it up with trove lol | 15:18 |
| jamesden- | ahh :D | 15:18 |
| noonedeadpunk | as there it might mess things up badly | 15:19 |
| noonedeadpunk | if someone around - Zed unmaintained was updated and ready now: https://review.opendev.org/c/openstack/openstack-ansible/+/918979 | 15:24 |
| jrosser | done | 15:28 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-tests master: Bump Ansible version for tests repo https://review.opendev.org/c/openstack/openstack-ansible-tests/+/921434 | 15:32 |
| noonedeadpunk | sorry for torturing you on the deadline day :( | 15:33 |
| noonedeadpunk | can't get myself organized enough to complete all these in advance :( | 15:33 |
| jamesden- | you're fine - thank you for your efforts | 15:33 |
| noonedeadpunk | I think kinda everything needed is in gates. | 15:34 |
| noonedeadpunk | I'll just need to update SHAs again and create 29.0.0 from that one. | 15:34 |
| jamesden- | I'm seeing neutron-rpc-server wanting /neutron even though it's a quorum queue now as 'neutron'. Digging into that | 15:35 |
| jamesden- | Jun 06 15:34:14 910400-ord-controller01 rabbitmq-server[2901]: Error on AMQP connection <0.1717969.0> (172.29.236.112:49174 -> 172.29.236.101:5671 - neutron-rpc-server:100973:baeb2828-8027-4b77-9323-5a48d6fbe478, user: 'neutron', state: opening): | 15:35 |
| jamesden- | vhost /neutron not found | 15:35 |
| noonedeadpunk | you're having ovs? | 15:36 |
| jamesden- | OVN | 15:36 |
| noonedeadpunk | how neutron-rpc-server is there | 15:36 |
| noonedeadpunk | OVN doesn't work in uwsgi mode | 15:36 |
| noonedeadpunk | what;s the runtime of the service? | 15:37 |
| jamesden- | lemme double check | 15:37 |
| noonedeadpunk | so neutron-rpc-server simply should not be there | 15:37 |
| jamesden- | sorry this is OVS not OVN | 15:37 |
| noonedeadpunk | aha | 15:37 |
| noonedeadpunk | and service was restarted on playbook run? | 15:37 |
| jamesden- | should've been, but i've since stopped/started manually | 15:38 |
| noonedeadpunk | gotcha | 15:39 |
| noonedeadpunk | I kinda really no idea of where it can take vhost except of transport_url in neutron.con | 15:41 |
| opendevreview | Merged openstack/openstack-ansible-os_cloudkitty stable/2024.1: Define region and interface for collector/fetcher https://review.opendev.org/c/openstack/openstack-ansible-os_cloudkitty/+/921285 | 15:42 |
| opendevreview | Merged openstack/openstack-ansible stable/2024.1: Remove old horizon base mapping https://review.opendev.org/c/openstack/openstack-ansible/+/921436 | 15:47 |
| noonedeadpunk | ah, this would need to be backported :( https://review.opendev.org/c/openstack/openstack-ansible-os_skyline/+/921442 | 15:48 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-tests master: Bump Ansible version for tests repo https://review.opendev.org/c/openstack/openstack-ansible-tests/+/921434 | 16:02 |
| jamesden- | i assume Glance and Keystone should also have been converted to quorum? | 16:17 |
| noonedeadpunk | so glance and keystone do not use rpc messaging at all | 16:18 |
| noonedeadpunk | my expectation would be that vhost will be just dropped | 16:19 |
| opendevreview | Merged openstack/openstack-ansible-os_skyline master: Fix skyline user creation https://review.opendev.org/c/openstack/openstack-ansible-os_skyline/+/921442 | 16:19 |
| noonedeadpunk | and no transport_url in config either | 16:19 |
| jamesden- | ahh ok. I did still see /keystone and /glance as vhosts, but the transport_url in [oslo_messaging_notifications] was updated | 16:21 |
| jamesden- | and one of my controllers had this: https://paste.openstack.org/show/b45LR9QOMBVOm5xHN6sy/ | 16:21 |
| noonedeadpunk | ok, yeah, maybe they're not dropped as role just not run now..... | 16:22 |
| noonedeadpunk | wait, but you have transport_url for glance? | 16:22 |
| jamesden- | i do, in glance-api.conf | 16:22 |
| noonedeadpunk | oh | 16:23 |
| jamesden- | for notifcations | 16:23 |
| noonedeadpunk | in oslo_messaging_notifications | 16:23 |
| noonedeadpunk | yeah | 16:23 |
| jamesden- | yeah | 16:23 |
| noonedeadpunk | though driver is noop? | 16:23 |
| jamesden- | but, that still does mean we need to convert that vhost? | 16:23 |
| jamesden- | no, my driver is set to messagingv2 - i have ceilometer enabled here | 16:23 |
| noonedeadpunk | aha | 16:24 |
| noonedeadpunk | so you have one of these, right? glance_ceilometer_enabled or oslomsg_notify_configure or glance_oslomsg_notify_configure ? | 16:25 |
| noonedeadpunk | so basically it will if https://opendev.org/openstack/openstack-ansible-os_glance/src/branch/master/defaults/main.yml#L130 resolves to True | 16:27 |
| jamesden- | ok - let me check | 16:27 |
| noonedeadpunk | and also - what's result of `openstack-ansible playbooks/os-glance-install.yml --tags common-mq --limit control01` | 16:29 |
| jamesden- | i had metering inventory configured but not glance_ceilometer_enabled | 16:30 |
| noonedeadpunk | but... how you ended up with non-noop driver | 16:31 |
| noonedeadpunk | as https://opendev.org/openstack/openstack-ansible-os_glance/src/branch/stable/2023.1/templates/glance-api.conf.j2#L75 | 16:31 |
| noonedeadpunk | (that's atelope you're upgrading from) | 16:31 |
| opendevreview | Merged openstack/openstack-ansible-os_gnocchi stable/2024.1: Drop default policy file location https://review.opendev.org/c/openstack/openstack-ansible-os_gnocchi/+/921284 | 16:32 |
| jamesden- | https://github.com/openstack/openstack-ansible-os_ceilometer/blob/master/defaults/main.yml#L293 | 16:33 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_skyline stable/2024.1: Fix skyline user creation https://review.opendev.org/c/openstack/openstack-ansible-os_skyline/+/921462 | 16:33 |
| jamesden- | so, it's auto-enabled | 16:34 |
| noonedeadpunk | yeah, though this wasn't in context of glance.... | 16:34 |
| noonedeadpunk | it's just for ceilometer kinda | 16:34 |
| jamesden- | yeah, but i am guessing ceilometer was the one trying to hit /glance vhost? | 16:34 |
| noonedeadpunk | oh well | 16:34 |
| noonedeadpunk | https://opendev.org/openstack/openstack-ansible-os_glance/src/branch/stable/2023.1/defaults/main.yml#L52 | 16:34 |
| jamesden- | oh, no glance-wsgi-api.. hmm | 16:35 |
| jamesden- | oh well there you go. | 16:35 |
| noonedeadpunk | ok, so https://opendev.org/openstack/openstack-ansible-os_glance/src/branch/stable/2024.1/defaults/main.yml#L130 should be true then | 16:35 |
| noonedeadpunk | so, can you paste results of `openstack-ansible playbooks/os-glance-install.yml --tags common-mq --limit control01`? | 16:36 |
| jamesden- | sure one sec | 16:36 |
| jamesden- | https://paste.openstack.org/show/b9Dq9Bo6ktoRMvydJGTN/ | 16:37 |
| noonedeadpunk | huh, skipped | 16:38 |
| jamesden- | https://paste.openstack.org/show/brgl9CbQAZbFx90vJWuU/ | 16:38 |
| jamesden- | let me check the collection and add some debug | 16:38 |
| noonedeadpunk | that;s the condition just in case: https://opendev.org/openstack/openstack-ansible-plugins/src/branch/master/roles/mq_setup/tasks/main.yml#L44-L46 | 16:39 |
| noonedeadpunk | oh | 16:40 |
| noonedeadpunk | crap | 16:40 |
| noonedeadpunk | crap | 16:41 |
| noonedeadpunk | I found the issue | 16:41 |
| jamesden- | ok | 16:41 |
| noonedeadpunk | give me a sec | 16:41 |
| jamesden- | sure | 16:41 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-plugins master: Run Notify part when RPC is disabled https://review.opendev.org/c/openstack/openstack-ansible-plugins/+/921464 | 16:45 |
| noonedeadpunk | jamesden-: can you try this out? ^ | 16:45 |
| jamesden- | sure, one sec | 16:45 |
| noonedeadpunk | thanks for RC testing that :D | 16:46 |
| jamesden- | https://paste.openstack.org/show/bWovLeJkRpRaZvwHUpQK/ | 16:47 |
| jamesden- | ship it! | 16:47 |
| jamesden- | looks good now, thank you | 16:48 |
| noonedeadpunk | yeah, looks like it does what;'s needed according to the log | 16:48 |
| jamesden- | indeed, thanks for the quick turnaround | 16:49 |
| noonedeadpunk | also, https://review.opendev.org/c/openstack/openstack-ansible-os_heat/+/921444 looks kinda healthy (regarding heat role removal) | 16:51 |
| noonedeadpunk | though it's not "really" tested, as we've backported fix back to Zed... | 16:51 |
| * noonedeadpunk needs to go for groceries now | 16:51 | |
| jamesden- | of course! | 16:52 |
| noonedeadpunk | if https://review.opendev.org/c/openstack/openstack-ansible-plugins/+/921443 works (and looks like it is), it also might be helpful on 2024.1.... | 16:54 |
| noonedeadpunk | if you're playing around - would be nice to add back heat_stack_owner assignment and see if these 2 patches do the trick :) | 16:55 |
| jamesden- | i will try, sure | 16:56 |
| opendevreview | Merged openstack/openstack-ansible-os_masakari stable/2024.1: Updated from OpenStack Ansible Tests https://review.opendev.org/c/openstack/openstack-ansible-os_masakari/+/921258 | 16:59 |
| opendevreview | Merged openstack/openstack-ansible-os_zun stable/2024.1: Updated from OpenStack Ansible Tests https://review.opendev.org/c/openstack/openstack-ansible-os_zun/+/921260 | 17:00 |
| opendevreview | Merged openstack/openstack-ansible-os_horizon stable/2024.1: Fix horizon distro installation for EL https://review.opendev.org/c/openstack/openstack-ansible-os_horizon/+/921445 | 17:19 |
| opendevreview | Merged openstack/openstack-ansible stable/zed: Transition Zed to Unmaintained https://review.opendev.org/c/openstack/openstack-ansible/+/918979 | 17:44 |
| opendevreview | Merged openstack/openstack-ansible-os_ironic stable/2024.1: Updated from OpenStack Ansible Tests https://review.opendev.org/c/openstack/openstack-ansible-os_ironic/+/921256 | 17:46 |
| opendevreview | Merged openstack/openstack-ansible-openstack_hosts unmaintained/zed: Allow openstack_hosts role to configure git safe.directory entries https://review.opendev.org/c/openstack/openstack-ansible-openstack_hosts/+/921028 | 18:38 |
| jrosser | oh whoops no proper CI there :( | 18:42 |
| deflated | Hi all, having issues with a cinder error on 29rc1 (i also have issues with adjutant magnum and skyline as well but one at a time), the paste is here: https://pastebin.com/RfPAbJVU | 19:02 |
| jrosser | deflated: perhaps something to do is introduce some debug: tasks just before the one that fails displaying the variables in use here | 19:08 |
| deflated | And how would i do that? The only variables i have set for cinder are here: https://pastebin.com/ArWpRGdB | 19:15 |
| jrosser | ah well i don;t think container_vars is something you can use in user_variables.yml | 19:19 |
| deflated | I got it from here: https://docs.openstack.org/openstack-ansible-os_cinder/latest/configure-cinder.html#configuring-cinder-to-use-ceph | 19:23 |
| deflated | removing it results in the same error, i actually added container_vars after it failed the first time | 19:24 |
| deflated | so the same values minus the words container_vars still fails | 19:25 |
| jrosser | note a bit further on it shows you "In openstack_user_config.yml" | 19:25 |
| deflated | it never made a difference as i have always had it there and it worked for 2023.1 but i can move it and try again no problem | 19:27 |
| jrosser | nothing has changed regarding this and i believe that container_vars is used when constructing the sdynamic inventory from openstack_user_config | 19:28 |
| jrosser | deflated: to add some debug: tasks you will find the cinder role in /etc/ansible/roles/os_cinder | 19:41 |
| jrosser | and you could try adding something like https://paste.opendev.org/show/bGwy9woSWcsMu90nrXxf/ just here https://opendev.org/openstack/openstack-ansible-os_cinder/src/branch/master/tasks/cinder_post_install.yml#L25 | 19:43 |
| deflated | Moved it over, same error, just realised the time so i've bookmarked the above and i'll come back to it in the morning, thanks for the guidance | 20:04 |
| opendevreview | Merged openstack/openstack-ansible-os_cloudkitty stable/2023.1: Define region and interface for collector/fetcher https://review.opendev.org/c/openstack/openstack-ansible-os_cloudkitty/+/921055 | 20:38 |
| opendevreview | Merged openstack/openstack-ansible-plugins master: Run Notify part when RPC is disabled https://review.opendev.org/c/openstack/openstack-ansible-plugins/+/921464 | 23:38 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!