| opendevreview | Andrew Bonney proposed openstack/openstack-ansible-haproxy_server stable/2023.1: Correct default Content-Type for security.txt https://review.opendev.org/c/openstack/openstack-ansible-haproxy_server/+/890007 | 07:04 |
|---|---|---|
| jrosser | good morning | 08:23 |
| noonedeadpunk | o/ | 08:40 |
| noonedeadpunk | fwiw I'm mostly away during this week | 08:40 |
| noonedeadpunk | I think most nasty things right now that we have - this keystone thing | 08:41 |
| noonedeadpunk | I tried to reach keystone folks but had no reply | 08:41 |
| jrosser | noonedeadpunk: yeah - we are just looking at Z->2023.1 upgrade here and wondering what to do about this | 09:08 |
| noonedeadpunk | I'm really not sure other then bug keystone folks or keep old keystone (or have a fork with revert) | 09:09 |
| * noonedeadpunk facepalms that we don't use <service>_galera_port in any <service>.conf | 09:27 | |
| noonedeadpunk | well, we do _only_ in nova | 09:32 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_octavia master: Use proper galera port in configuration https://review.opendev.org/c/openstack/openstack-ansible-os_octavia/+/890088 | 12:15 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_neutron master: Use proper galera port in configuration https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/890089 | 12:25 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_cinder master: Use proper galera port in configuration https://review.opendev.org/c/openstack/openstack-ansible-os_cinder/+/890091 | 12:28 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_adjutant master: Use proper galera port in configuration https://review.opendev.org/c/openstack/openstack-ansible-os_adjutant/+/890092 | 12:30 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_aodh master: Use proper galera port in configuration https://review.opendev.org/c/openstack/openstack-ansible-os_aodh/+/890093 | 12:32 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_barbican master: Use proper galera port in configuration https://review.opendev.org/c/openstack/openstack-ansible-os_barbican/+/890095 | 13:00 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_blazar master: Use proper galera port in configuration https://review.opendev.org/c/openstack/openstack-ansible-os_blazar/+/890096 | 13:01 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_cloudkitty master: Use proper galera port in configuration https://review.opendev.org/c/openstack/openstack-ansible-os_cloudkitty/+/890097 | 13:03 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_octavia master: Use proper galera port in configuration https://review.opendev.org/c/openstack/openstack-ansible-os_octavia/+/890088 | 13:03 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_designate master: Use proper galera port in configuration https://review.opendev.org/c/openstack/openstack-ansible-os_designate/+/890098 | 13:04 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_glance master: Use proper galera port in configuration https://review.opendev.org/c/openstack/openstack-ansible-os_glance/+/890099 | 13:07 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_gnocchi master: Use proper galera port in configuration https://review.opendev.org/c/openstack/openstack-ansible-os_gnocchi/+/890100 | 13:08 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_heat master: Use proper galera port in configuration https://review.opendev.org/c/openstack/openstack-ansible-os_heat/+/890122 | 13:09 |
| jrosser | Has anyone attempted a Z -> 2023.1 upgrade? | 13:12 |
| jrosser | we see total failure as soon as keystone is upgraded https://paste.opendev.org/show/bGfPvKRVJbHXS6I08QMT/ | 13:13 |
| jrosser | which is a result of this https://opendev.org/openstack/keystone/commit/f6a0cce4409232d8ade69b7773dbabcf4c53ec0f | 13:13 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_magnum master: Use proper galera port in configuration https://review.opendev.org/c/openstack/openstack-ansible-os_magnum/+/890123 | 13:13 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_masakari master: Use proper galera port in configuration https://review.opendev.org/c/openstack/openstack-ansible-os_masakari/+/890124 | 13:15 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_mistral master: Use proper galera port in configuration https://review.opendev.org/c/openstack/openstack-ansible-os_mistral/+/890125 | 13:17 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_manila master: Use proper galera port in configuration https://review.opendev.org/c/openstack/openstack-ansible-os_manila/+/890126 | 13:18 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_murano master: Use proper galera port in configuration https://review.opendev.org/c/openstack/openstack-ansible-os_murano/+/890127 | 13:20 |
| * noonedeadpunk don't use oauth as of today | 13:20 | |
| jrosser | it doesnt matter | 13:20 |
| jrosser | this blew up our test lab which doesnt either | 13:20 |
| noonedeadpunk | huh | 13:21 |
| jrosser | it's all over our upgrade CI jobs too | 13:21 |
| andrewbonney | https://zuul.opendev.org/t/openstack/build/0c561f4e655241e09e4517f2f9407da4/log/logs/host/keystone-wsgi-public.service.journal-09-24-38.log.txt#13607 | 13:21 |
| jrosser | like all the existing issued tokens suddenly don't work any more | 13:21 |
| jrosser | the upgrade jobs only "work" because they update/restart all the services which then get new tokens | 13:22 |
| jrosser | but from the point keystone is upgraded to the point that all the services are restarted, things are in a pretty bad state | 13:23 |
| noonedeadpunk | But upgrade job worked only until we had resetted passwords? | 13:23 |
| jrosser | this is different | 13:23 |
| jrosser | in our lab we have reverted the password length keystone change | 13:23 |
| noonedeadpunk | But I'm not sure that we have configured all services to have caching? | 13:23 |
| noonedeadpunk | it's likely limited to nova/neutron I believe | 13:24 |
| noonedeadpunk | I wonder if flushing memcacached should work? | 13:24 |
| jrosser | we lost placement / designate / radosgw / glance / etc etc | 13:24 |
| jrosser | basically as soon as we updated keystone the whole monitoring went red | 13:25 |
| noonedeadpunk | ugh... | 13:25 |
| noonedeadpunk | halali: should be working on testing upgrades right now | 13:26 |
| jrosser | that whole log that andrewbonney posted is full of WTF exceptions, like missing users / roles / project | 13:27 |
| noonedeadpunk | yeah, saw that.... | 13:28 |
| jrosser | after ~1hour it came good after the tokens expired and were reissued | 13:29 |
| noonedeadpunk | I think that doing smth like `ansible -m shell -a "echo 'flush_all' | nc localhost 11211" memcached_all` should also help? | 13:30 |
| jrosser | tried to understand the keystone code a bit but it's pretty difficult | 13:32 |
| jrosser | theres the token as a string, seems to be then decoded to a dict (?) and then theres also the TokenModel class | 13:32 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_placement master: Use proper galera port in configuration https://review.opendev.org/c/openstack/openstack-ansible-os_placement/+/890130 | 13:37 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_senlin master: Use proper galera port in configuration https://review.opendev.org/c/openstack/openstack-ansible-os_senlin/+/890131 | 13:39 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_sahara master: Use proper galera port in configuration https://review.opendev.org/c/openstack/openstack-ansible-os_sahara/+/890132 | 13:40 |
| noonedeadpunk | um, decoding string to dict sounds... weird... | 13:40 |
| andrewbonney | It may have been a string ID which is used to look up a token, but it's a little tricky to follow | 13:42 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_tacker master: Use proper galera port in configuration https://review.opendev.org/c/openstack/openstack-ansible-os_tacker/+/890133 | 13:43 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_zun master: Use proper galera port in configuration https://review.opendev.org/c/openstack/openstack-ansible-os_zun/+/890134 | 13:46 |
| halali | noonedeadpunk in progress... | 16:45 |
| jamesdenton_ | any known issues with 27.0.1 and first_found? | 17:07 |
| jrosser | jamesdenton_: do you have something breaking? | 18:19 |
| halali | upgrade from Yoga 25.4.0 -> stable/2023.1 I lost neutron API https://paste.openstack.org/show/bsx1QDpjKmPmUY0otWK1/ | 19:14 |
| jamesdenton_ | Yeah, I've done 2 deploys now using a dedicated deploy node, 27.0.0/27.0.1 on jammy, and the unbound plays are failing when trying to gather vars: https://paste.opendev.org/show/bH6D8p8BZzS35dDkzc8U/ | 19:51 |
| jamesdenton_ | doesn't happen with 26.1.2 | 19:51 |
| jamesdenton_ | not sure what the trigger is, so i'd like to narrow that down before anyone wastes time | 19:52 |
| mgariepy | isn't it a stalled/expired fact issue ? | 19:55 |
| jamesdenton_ | not sure - i've had this happen in two different envs | 20:04 |
| jamesdenton_ | i did t-shoot some last week i've just forgotten the details, i'll try it again | 20:05 |
| jamesdenton_ | halali have you tried restarting haproxy? | 20:11 |
| jrosser | ^ would going from yoga -> 2023.1 switch the networking backend to OVN unless $something is done to prevent that? | 20:12 |
| jamesdenton_ | there should be logic there to prevent that | 20:13 |
| halali | jamesdenton_ restarted haproxy did not help, | 20:20 |
| halali | jrosser does this suppose to keep OVS as is if I did not change it to OVN instead | 20:20 |
| jamesdenton_ | if you had defined ml2.ovs previous that should have stayed | 20:21 |
| jamesdenton_ | "journalctl -xe -f -u neutron-server" might show whats going on | 20:21 |
| halali | jamesdenton_ it's AIO using default user_variables.yml, I moved from Xena -> Yoga (was good and smooth with no issue) then 2023.1 that got failed | 20:22 |
| halali | and if the upgrade process suppose to transfer the neutron plugin from OVS -> OVN then the neutron container and inventory it suppose to change which is not currently | 20:24 |
| jamesdenton_ | well, even then using linuxbridge there was supposed to be something there, but if you went Y->A then maybe something in Z was missed | 20:24 |
| jamesdenton_ | there is no mechanism to migrate anyone to OVN automatically (on purpose, anyway) | 20:25 |
| jamesdenton_ | what does the journal show? | 20:25 |
| halali | you are right, it complaining on ml2 plugin https://paste.openstack.org/show/bkann3WINk1diMHLq0Yz/ | 20:27 |
| jrosser | oh, isnt that something we fixed? | 20:28 |
| jamesdenton_ | neutron-24.6.1? isnt that still xena? | 20:28 |
| halali | and the content of ml2_conf.ini is https://paste.openstack.org/show/b30brqUCvD3xjE3Ghelz/ | 20:28 |
| jrosser | `Failed to find some config files: /etc/neutron/neutron.conf,/etc/neutron/plugins/ml2/ml2_conf.ini` | 20:29 |
| jamesdenton_ | hum, well that's not what we have for ml2 | 20:29 |
| jamesdenton_ | if you were upgrading | 20:29 |
| halali | O_o 24.6.1 how it come | 20:29 |
| halali | neutron-api venv/neutron-27.0.1 and 25.4.0 and 24.6.1 is exist | 20:32 |
| jrosser | they don't get deleted during an upgrade | 20:33 |
| jamesdenton_ | https://github.com/openstack/openstack-ansible/blob/stable/zed/scripts/upgrade-utilities/define-neutron-plugin.yml | 20:34 |
| jamesdenton_ | https://docs.openstack.org/openstack-ansible/zed/admin/upgrades/major-upgrades.html | 20:34 |
| jamesdenton_ | that's a play that needs to run from Y->Z to ensure that the plugin doesn't change | 20:34 |
| jamesdenton_ | it's also listed here: | 20:35 |
| jamesdenton_ | https://docs.openstack.org/openstack-ansible/2023.1/admin/upgrades/major-upgrades.html | 20:35 |
| jamesdenton_ | but is missing from here: | 20:35 |
| jamesdenton_ | https://docs.openstack.org/openstack-ansible/latest/admin/upgrades/major-upgrades.html | 20:35 |
| jamesdenton_ | jrosser i was just observing that the service/binary still seemed to be running from that older venv | 20:36 |
| jrosser | yes, that could be a failed previous upgrade | 20:36 |
| jrosser | from X | 20:36 |
| jamesdenton_ | i haven't seen this particular issue, though.. the file not found | 20:36 |
| halali | interesting, but I wonder how is Y upgrade is pass the test and success, tbh did not check directly on neutron-api but the api was response and no haproxy complain on backend either, | 20:39 |
| jamesdenton_ | Must just be enough compatibility there, hard to say | 20:40 |
| halali | I see, so the process as follow | 20:41 |
| halali | from X -> Y then the define-neutron-plugin.yml script to make sure neutron plugin NOT change then I able to upgrade to 2023.1 | 20:42 |
| jamesdenton_ | Well, from Y->Z or Y->2023.1 you need(ed) to run define-neutron-plugin.yml | 20:43 |
| jamesdenton_ | which uses magic to determine that linuxbridge was the default and hardsets it moving forward | 20:43 |
| jamesdenton_ | if you were already using ml2.ovs or had hardcoded ml2.lxb then it shouldn't have been an issue | 20:43 |
| halali | I see, and on the way from X -> 2023.1 we must go through Y ? | 20:44 |
| halali | or at least X -> Z -> 2023.1 | 20:45 |
| jamesdenton_ | X->Y->Z->2023.1 for sure. Not sure if a skip is supported anywhere in there, yet | 20:46 |
| halali | OK thanks | 20:53 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!