| jrosser | morning | 08:13 |
|---|---|---|
| opendevreview | Katarina Strenkova proposed openstack/openstack-ansible-os_manila master: Replacedeprecated terms https://review.opendev.org/c/openstack/openstack-ansible-os_manila/+/890202 | 09:13 |
| noonedeadpunk | o/ | 10:10 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: Stop reffering _member_ role https://review.opendev.org/c/openstack/openstack-ansible/+/891400 | 10:59 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_ironic master: Stop reffering _member_ role https://review.opendev.org/c/openstack/openstack-ansible-os_ironic/+/891461 | 11:03 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_adjutant master: Stop reffering _member_ role https://review.opendev.org/c/openstack/openstack-ansible-os_adjutant/+/891462 | 11:04 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_sahara master: Stop reffering _member_ role https://review.opendev.org/c/openstack/openstack-ansible-os_sahara/+/891463 | 11:07 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_horizon master: Stop reffering _member_ role https://review.opendev.org/c/openstack/openstack-ansible-os_horizon/+/891464 | 11:08 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_swift master: Stop reffering _member_ role https://review.opendev.org/c/openstack/openstack-ansible-os_swift/+/891465 | 11:10 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_keystone master: Stop reffering _member_ role https://review.opendev.org/c/openstack/openstack-ansible-os_keystone/+/891466 | 11:18 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_keystone master: Install distro_packages in pre-main https://review.opendev.org/c/openstack/openstack-ansible-os_keystone/+/889934 | 11:49 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible stable/2023.1: Make `_member_` role to imply `member` https://review.opendev.org/c/openstack/openstack-ansible/+/891473 | 13:24 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_adjutant stable/zed: Install mysqlclient devel package https://review.opendev.org/c/openstack/openstack-ansible-os_adjutant/+/891447 | 13:27 |
| jrosser | wierd - someone else had this i think | 13:34 |
| jrosser | `fatal: [aio1_utility_container-efdc639c -> aio1_repo_container-effb8d8f(172.29.237.157)]: FAILED! => {"attempts": 5, "changed": false, "msg": "No package matching '{'name': 'ubuntu-cloud-keyring', 'state': 'latest'}' is available"}` | 13:34 |
| jrosser | like somewhere a dict gets parsed as a string | 13:34 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: Bump upstream SHAs https://review.opendev.org/c/openstack/openstack-ansible/+/891474 | 13:34 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_gnocchi master: Use proper galera port in configuration https://review.opendev.org/c/openstack/openstack-ansible-os_gnocchi/+/890100 | 13:35 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: Bump upstream SHAs https://review.opendev.org/c/openstack/openstack-ansible/+/891474 | 13:37 |
| noonedeadpunk | jrosser: ah, yes, I saw someone has reported that, but I didn't catch that it's indeed a string... | 13:41 |
| noonedeadpunk | I wonder if we should land this https://review.opendev.org/c/openstack/openstack-ansible/+/888517, as patches already voted depend on it, or abandon in favor of 891474? | 13:41 |
| jrosser | rather annoyingly i re-ran just utility-install.yml with -vvvv to see what was passed to the module, but of course it just-worked(tm) | 13:42 |
| noonedeadpunk | And in report it was same, actually | 13:42 |
| jrosser | my failure was in fresh AIO trying to run playbooks/setup-everything.yml | 13:43 |
| noonedeadpunk | master? | 13:43 |
| jrosser | yeah | 13:43 |
| jrosser | kind of ansible bug feeling somehow | 13:43 |
| noonedeadpunk | or import/inlcude | 13:45 |
| jrosser | how would that be? | 13:45 |
| noonedeadpunk | as I see ubuntu-cloud-keyring only in openstack_hosts | 13:46 |
| noonedeadpunk | and your output - utility delegates to repo | 13:46 |
| jrosser | yes | 13:46 |
| noonedeadpunk | Which I hardly imagine to be part of openstack_hosts? | 13:46 |
| jrosser | it's part of python_venv_build i think | 13:47 |
| noonedeadpunk | So it's like `_package_list` is used elsewhere? | 13:47 |
| jrosser | "install distro packages for...." | 13:47 |
| jrosser | i think i might have lost it off my scrollback, /me looks | 13:47 |
| noonedeadpunk | openstack_hosts doesn't run that? does it? | 13:47 |
| jrosser | oh i see what you mean | 13:47 |
| jrosser | like some var name aliasing with _package_list? | 13:48 |
| noonedeadpunk | yeah | 13:49 |
| jrosser | i'll check the ansible log when this finishes and i can paste what it said | 13:49 |
| noonedeadpunk | where it assumes to be dict rather then list | 13:49 |
| jrosser | lost it off my scrollback now | 13:49 |
| noonedeadpunk | *string | 13:49 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-openstack_hosts master: Drop Ubuntu 20.04 support https://review.opendev.org/c/openstack/openstack-ansible-openstack_hosts/+/891475 | 13:51 |
| noonedeadpunk | jrosser: and yes, same _package_list variable is smth we try to define in python_venv_build | 13:52 |
| noonedeadpunk | https://opendev.org/openstack/ansible-role-python_venv_build/src/branch/master/tasks/python_venv_wheel_build.yml#L20-L33 | 13:52 |
| noonedeadpunk | and these are the only places which intersect:) | 13:53 |
| noonedeadpunk | So there're proper and easy way :D Easy - just rename vars in both places to include role names, which is kinda fair to do. Proper would likely be to ensure we do import/include properly. | 13:54 |
| jrosser | https://paste.opendev.org/show/buvg5E7IpqJdVH3ma1hA/ | 13:54 |
| noonedeadpunk | Also I wonder if we already might have patch that's not merged... | 13:55 |
| jrosser | not sure - i remember seeing the report but having no time to take a look | 13:55 |
| noonedeadpunk | nah, I was jsut patching smth simmilar lately | 13:56 |
| noonedeadpunk | I guess I was thinking about this one: https://review.opendev.org/c/openstack/ansible-role-uwsgi/+/880344 | 13:58 |
| noonedeadpunk | apparently same happens with python_venv_build | 13:59 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-openstack_hosts master: Rename internal variable _package_list https://review.opendev.org/c/openstack/openstack-ansible-openstack_hosts/+/891476 | 14:03 |
| opendevreview | Jonathan Rosser proposed openstack/ansible-role-python_venv_build master: Rename _package_list variable https://review.opendev.org/c/openstack/ansible-role-python_venv_build/+/891478 | 14:08 |
| jrosser | humm well yes we do need to merge either 888517 or 891474 | 14:38 |
| jrosser | heat is broken either way | 14:38 |
| noonedeadpunk | and telemetry stack | 14:46 |
| noonedeadpunk | 888517 sounds easier and then 891474 on top | 14:46 |
| noonedeadpunk | let me rebase accordingly | 14:47 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-plugins master: Define default value for _service_adminuri_insecure https://review.opendev.org/c/openstack/openstack-ansible-plugins/+/889707 | 14:47 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-plugins master: Add openstack_resources role skeleton https://review.opendev.org/c/openstack/openstack-ansible-plugins/+/878794 | 14:47 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_octavia master: Adopt for usage openstack_resources role https://review.opendev.org/c/openstack/openstack-ansible-os_octavia/+/889879 | 14:49 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_tempest master: Adopt for usage openstack_resources role https://review.opendev.org/c/openstack/openstack-ansible-os_tempest/+/889741 | 14:53 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: Update SHAs for Gnoochi and plugins https://review.opendev.org/c/openstack/openstack-ansible/+/888517 | 14:55 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: Bump upstream SHAs https://review.opendev.org/c/openstack/openstack-ansible/+/891474 | 14:55 |
| noonedeadpunk | #startmeeting openstack_ansible_meeting | 15:00 |
| opendevmeet | Meeting started Tue Aug 15 15:00:34 2023 UTC and is due to finish in 60 minutes. The chair is noonedeadpunk. Information about MeetBot at http://wiki.debian.org/MeetBot. | 15:00 |
| opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 15:00 |
| opendevmeet | The meeting name has been set to 'openstack_ansible_meeting' | 15:00 |
| noonedeadpunk | #topic rollcall | 15:00 |
| noonedeadpunk | o/ | 15:00 |
| jrosser | o/ hello | 15:01 |
| noonedeadpunk | #topic office hours | 15:03 |
| noonedeadpunk | We have couple of things going on | 15:03 |
| noonedeadpunk | 1. Heat and telemetry are blocked, we need to land https://review.opendev.org/c/openstack/openstack-ansible/+/888517 to fix them | 15:03 |
| noonedeadpunk | 2. Fix for keystone regarding passwords that are longer 54 symbols has landed to master. Backports are proposed but not merged yet | 15:04 |
| noonedeadpunk | This blocks our stable bracnhes upgrade jobs | 15:04 |
| noonedeadpunk | I'm inlcined not to merge anything to our repos and just wait for keystone fixes | 15:05 |
| NeilHanlon | o/ | 15:05 |
| noonedeadpunk | Regarding _member_ role that's not invalid - I've proposed patch to mark the role as implied. And series of patches to stop reffering to _member_ | 15:06 |
| noonedeadpunk | #link https://review.opendev.org/q/topic:osa%252Fmember_role | 15:06 |
| jrosser | this is a big surprise at upgrade | 15:07 |
| noonedeadpunk | Interestingly, rocky failed a lot there. Either on tempest or with timeouts | 15:07 |
| noonedeadpunk | Yeah... Keystone has brought plenty of surprises this time I would say | 15:07 |
| jrosser | do we need a releasenote about the member changes, or should the patches take care of it automatically? | 15:07 |
| noonedeadpunk | Not saying about invalid tokens | 15:07 |
| noonedeadpunk | That is very good question | 15:08 |
| noonedeadpunk | I was thinking about release note, but I failed to find when Keystone has marked _memeber_ as deprecated | 15:08 |
| noonedeadpunk | So I got slightly confused on what to say in a release note | 15:08 |
| jrosser | wheres the implied role patch? | 15:08 |
| noonedeadpunk | um.... don't tell me I haven't pushed it and just did `git reset origin/master --hard` | 15:09 |
| * jrosser not seeing it under the topic link | 15:10 | |
| noonedeadpunk | ah | 15:10 |
| noonedeadpunk | https://review.opendev.org/c/openstack/openstack-ansible/+/891473 | 15:10 |
| jrosser | aaaah ok so this is addressed at upgrade | 15:11 |
| jrosser | we never skip 2023.1? :) | 15:11 |
| noonedeadpunk | According to officially supported upgrade process - we should not | 15:12 |
| jrosser | maybe *that* is actually where the releasenote needs to go | 15:12 |
| noonedeadpunk | ok, what we wanna mention there? | 15:15 |
| noonedeadpunk | refference the keystone bug that application credentials are still borked? | 15:16 |
| noonedeadpunk | (with _member_ role)? | 15:16 |
| jrosser | maybe saying that this is the release which OSA transitions from _member_ to member | 15:16 |
| jrosser | handled automatically in upgrade scripts blah blah | 15:17 |
| jrosser | or rather, existing deployments get an implied role added for backward compatibility | 15:17 |
| noonedeadpunk | also, I'd say that ones who want skip slurp releases during upgrades - doing that on their own risk and will have plenty of other hacks. And should execute all upgrade steps regardless | 15:17 |
| noonedeadpunk | yeah, last one sounds good | 15:18 |
| noonedeadpunk | Will add reno then | 15:18 |
| noonedeadpunk | And I guess we'd need to backport https://review.opendev.org/q/topic:osa%252Fmember_role to 2023.1 as well | 15:19 |
| jrosser | like you say about waiting for keytone to merge password length fixes.... | 15:19 |
| jrosser | we have also not taken 2023.1 upgrade past lab tests because of all the keystone things | 15:19 |
| jrosser | also waiting for proper upstream fixes to merge | 15:20 |
| noonedeadpunk | yeah, I was going to look into flushing memcached after keystone upgrade as well | 15:20 |
| noonedeadpunk | not sure when exactly to run this though | 15:20 |
| noonedeadpunk | and where to add | 15:20 |
| noonedeadpunk | sounds like adding a variable and running either in keystone playbook or in the role itself is by far only ways | 15:21 |
| jrosser | though that might be not needed once the password length thing is addressed? | 15:21 |
| noonedeadpunk | I think these are 2 independent regressions | 15:21 |
| noonedeadpunk | caused by different patches | 15:21 |
| noonedeadpunk | And I'm not sure how to fix this one on keystone side | 15:22 |
| noonedeadpunk | #link https://bugs.launchpad.net/keystone/+bug/2029134 | 15:24 |
| noonedeadpunk | I asked patch submitter to have a look into that, but not sure if they did... | 15:24 |
| noonedeadpunk | But if that's trivial to workaround - probably we should do that then... | 15:25 |
| noonedeadpunk | or well - possible to workaround at very least. | 15:25 |
| jrosser | i did look at that with andrewbonney and it looks like something that needs patching in keystone | 15:26 |
| noonedeadpunk | but issue goes after cache timeout, from what I've read? | 15:27 |
| jrosser | yes thats right | 15:27 |
| noonedeadpunk | so, if flush cache...? | 15:27 |
| jrosser | ah so this is where memcached flush | 15:27 |
| jrosser | uyes | 15:27 |
| noonedeadpunk | I haven't tested though, as that sounds not super trivial to reproduce/witness and ensure that flushing cache is not a co-incidence | 15:28 |
| noonedeadpunk | But was going to try it out later today | 15:29 |
| jrosser | you need a deployed cloud with services running | 15:29 |
| jrosser | i suspect with no monitoring you wont see anything | 15:29 |
| noonedeadpunk | Also openstack_resources role looks very close to get it's initial state. Tempest is passing now while using the role. Octavia was almost passing - keypair was owned by a wrong user. | 15:30 |
| noonedeadpunk | Yeah, and monitoring should be also "proper" one | 15:30 |
| * noonedeadpunk not even sure that their production monitoring will catch it either | 15:30 | |
| noonedeadpunk | But I guess, that if I run tempest test, then upgrade keystone, running tempest again should fail? | 15:31 |
| noonedeadpunk | as it's interaction between services that is affected? | 15:31 |
| noonedeadpunk | like nova can't query neutron or placement | 15:32 |
| jrosser | from our notes, as soon as you upgrade keystone then everything else fails to auth | 15:33 |
| jrosser | because it expects oauth2_thumbprint in the tokens, and it's missing | 15:33 |
| noonedeadpunk | but not CLI? As cli does not cache tokens? | 15:34 |
| noonedeadpunk | (I assume) | 15:34 |
| jrosser | i assume not | 15:35 |
| noonedeadpunk | I guess anything that uses keystone_authtoken | 15:35 |
| jrosser | but iirc our alerting (haproxy?) all went bananas at that point | 15:35 |
| jrosser | andrew is back tomorrow and might be able to say exactly what it did | 15:35 |
| noonedeadpunk | hm... maybe patch to keystone is more trivial then I thought | 15:36 |
| jrosser | right - it just needs to not try to parse that field if it's absent | 15:36 |
| jrosser | or have some non-failing accessor method to get() it | 15:36 |
| noonedeadpunk | Yeah, I will try to patch that actually as well | 15:38 |
| noonedeadpunk | As it's failing here https://opendev.org/openstack/keystone/src/commit/f6a0cce4409232d8ade69b7773dbabcf4c53ec0f/keystone/common/render_token.py#L145-L148 | 15:38 |
| jrosser | thats it | 15:39 |
| noonedeadpunk | Not sure if that's the only place that needs adjustment | 15:39 |
| noonedeadpunk | as such assumptions are everywhere in code kind of | 15:39 |
| noonedeadpunk | So it could be just first place | 15:39 |
| noonedeadpunk | anyway, will see :) | 15:40 |
| * noonedeadpunk hopes to get keystone fixes to land before September | 15:43 | |
| jrosser | i think there is at least a regular keystone meeting now | 15:44 |
| noonedeadpunk | yup, was on previous one | 15:44 |
| opendevreview | James Denton proposed openstack/openstack-ansible-os_nova master: Allow Glance region to be set via variable https://review.opendev.org/c/openstack/openstack-ansible-os_nova/+/891515 | 15:47 |
| noonedeadpunk | #endmeeting | 16:01 |
| opendevmeet | Meeting ended Tue Aug 15 16:01:53 2023 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 16:01 |
| opendevmeet | Minutes: https://meetings.opendev.org/meetings/openstack_ansible_meeting/2023/openstack_ansible_meeting.2023-08-15-15.00.html | 16:01 |
| opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/openstack_ansible_meeting/2023/openstack_ansible_meeting.2023-08-15-15.00.txt | 16:01 |
| opendevmeet | Log: https://meetings.opendev.org/meetings/openstack_ansible_meeting/2023/openstack_ansible_meeting.2023-08-15-15.00.log.html | 16:01 |
| jrosser | running a complicated aio scenario ends up with pretty random tempest test configuration] | 16:07 |
| jrosser | includelist from role X and excludelist from role Y | 16:08 |
| jrosser | oh even better SCENARIO=aio_lxc_magnum_barbican_octavia blows up straight away | 16:11 |
| jrosser | tempest_run.log `The specified regex doesn't match with anything` | 16:12 |
| noonedeadpunk | ugh, yeah, I can imagine this happening... | 16:14 |
| jrosser | ok i confirmed it's because of getting junk includelist / excludelist contents | 16:16 |
| jrosser | i think i should be able to make this aggregate all the tests from the enabled roles | 16:16 |
| jrosser | use one of those specially prefixed var names | 16:17 |
| noonedeadpunk | what makes it non-trivial, as we're defining these lists in different files | 16:17 |
| noonedeadpunk | (unless it wasn't fixed and I guess it wasn't) | 16:18 |
| noonedeadpunk | and then we should have some "default" tests when nothing else is defined | 16:18 |
| noonedeadpunk | I do recall trying to improve things, but it was non-trivial | 16:18 |
| jrosser | i think i can rename these vars https://github.com/openstack/openstack-ansible/blob/master/tests/roles/bootstrap-host/templates/user_variables_magnum.yml.j2#L56 | 16:19 |
| jrosser | then use a regex search through the vars names to gather them all, like we do in PKI role for CA definitions | 16:19 |
| jrosser | like this https://github.com/openstack/ansible-role-pki/blob/master/vars/main.yml#L17 | 16:20 |
| noonedeadpunk | ah, yes, true | 16:25 |
| noonedeadpunk | you're right | 16:25 |
| noonedeadpunk | that should work really nicely | 16:25 |
| -opendevstatus- NOTICE: Zuul job execution is temporarily paused while we rearrange local storage on the servers | 16:53 | |
| noonedeadpunk | jrosser: https://review.opendev.org/c/openstack/keystone/+/891521 "just works" | 17:39 |
| jamesdenton | nice | 17:40 |
| -opendevstatus- NOTICE: Zuul job execution has resumed with additional disk space on the servers | 17:43 | |
| opendevreview | Merged openstack/openstack-ansible-lxc_hosts master: Refactor LXC image expiration https://review.opendev.org/c/openstack/openstack-ansible-lxc_hosts/+/888278 | 19:26 |
| opendevreview | Merged openstack/openstack-ansible master: Remove dynamic-address-fact call which is no longer required https://review.opendev.org/c/openstack/openstack-ansible/+/890306 | 19:49 |
| opendevreview | Merged openstack/openstack-ansible master: haproxy: fix health checks for serialconsole in http mode https://review.opendev.org/c/openstack/openstack-ansible/+/890520 | 19:49 |
| opendevreview | Merged openstack/openstack-ansible-os_rally master: Fix linters and metadata https://review.opendev.org/c/openstack/openstack-ansible-os_rally/+/888605 | 21:46 |
| opendevreview | Merged openstack/openstack-ansible-os_keystone master: Install libldap-common for keystone https://review.opendev.org/c/openstack/openstack-ansible-os_keystone/+/890533 | 21:52 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!