| opendevreview | Merged openstack/openstack-ansible-ops master: Clarifications to mcapi_vexxhost README https://review.opendev.org/c/openstack/openstack-ansible-ops/+/916817 | 07:51 |
|---|---|---|
| opendevreview | Takashi Kajinami proposed openstack/ansible-config_template master: Remove compatibility with Python 2 https://review.opendev.org/c/openstack/ansible-config_template/+/917123 | 08:47 |
| noonedeadpunk | NeilHanlon: hey! seems rocky missing some ovn?:) "nothing provides ovn24.03 needed by rdo-ovn-2:24.03-1.el9s.noarch from rdo-deps" | 09:22 |
| noonedeadpunk | I think we've switched to RDO for OpenStack 2024.1 (Caracal) which requires more recent OVN version | 09:23 |
| noonedeadpunk | or well. we didn't even.... | 09:23 |
| opendevreview | Merged openstack/openstack-ansible-ceph_client stable/2023.1: Add backwards compatibility of ceph_components format https://review.opendev.org/c/openstack/openstack-ansible-ceph_client/+/917000 | 09:52 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_trove master: Add independent set of variables for guestagent configuration https://review.opendev.org/c/openstack/openstack-ansible-os_trove/+/917047 | 10:04 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-galera_server master: Ensure find is installed on MariaDB containers https://review.opendev.org/c/openstack/openstack-ansible-galera_server/+/917132 | 11:50 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-galera_server master: Add distro infra jobs https://review.opendev.org/c/openstack/openstack-ansible-galera_server/+/914691 | 11:50 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: Test distro installation method for MariaDB/RabbitMQ https://review.opendev.org/c/openstack/openstack-ansible/+/914649 | 14:11 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: Enable rabbitmq distro installation for distro scenario https://review.opendev.org/c/openstack/openstack-ansible/+/917148 | 14:14 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-rabbitmq_server master: Add distro infra jobs https://review.opendev.org/c/openstack/openstack-ansible-rabbitmq_server/+/914692 | 14:15 |
| nixbuilder | Getting this error... https://paste.openstack.org/show/bzRtvOuN6L2OIl1BIzCM/. This was working a few days ago just fine. This is with 28.2.0. | 15:02 |
| noonedeadpunk | nixbuilder: is that on rocky? | 15:05 |
| noonedeadpunk | we started catching this error just today | 15:05 |
| noonedeadpunk | apparently new ovn version was released/used by rdo, but it's not built yet for rocky I guess | 15:05 |
| nixbuilder | noonedeadpunk: Yes... Rocky | 15:06 |
| noonedeadpunk | NeilHanlon: if you around... | 15:08 |
| noonedeadpunk | I believe some binding to old version of OVN should be possible | 15:09 |
| noonedeadpunk | but I didn't checked on that as ususally Neil just quickly builds the version needed :D | 15:09 |
| nixbuilder | Hopefully building openvswitch3.3 will fix this. I will try that. | 15:15 |
| noonedeadpunk | though eventually, I think we shoul actually lock down prior version of ovs.... | 15:30 |
| noonedeadpunk | it was 3.2 yesterday I believe | 15:32 |
| noonedeadpunk | and also ovn got bumped from 23.09 to 24.03 | 15:36 |
| nixbuilder | noonedeadpunk: Can't catch a break :-( | 15:40 |
| noonedeadpunk | well, I can in my rocky | 15:41 |
| noonedeadpunk | https://paste.openstack.org/show/bGswkprTQAbYMrkom8cx/ | 15:42 |
| noonedeadpunk | the problem is, that there's no openvswitch3.2 either that was there yesterday... | 15:43 |
| noonedeadpunk | https://zuul.opendev.org/t/openstack/build/af621fafd083498e9cf3fe0f47a1b87d/log/logs/host/dnf.rpm.log.txt#438-442 | 15:44 |
| noonedeadpunk | ah. damn | 15:45 |
| noonedeadpunk | I was missing centos-release-nfv-openvswitch | 15:46 |
| noonedeadpunk | dnf install openvswitch3.2*.x86_64 or dnf install ovn23.09*.x86_64 does work | 15:49 |
| nixbuilder | noonedeadpunk: Thanks again... retrying AIO install | 15:56 |
| noonedeadpunk | nixbuilder: huh, what if you try `openstack_hosts_rdo_repo_type: cloudsig`? | 15:59 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-openstack_hosts master: Ensure codename is lowercased for cloudsig install https://review.opendev.org/c/openstack/openstack-ansible-openstack_hosts/+/917184 | 16:04 |
| noonedeadpunk | nah, it's same :( | 16:04 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_neutron master: Fix OVS/OVN versions for EL https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/917192 | 16:18 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_neutron master: Update OVN/OVS versions for EL https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/917193 | 16:18 |
| noonedeadpunk | nixbuilder: I assume https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/917192 should help you | 16:18 |
| noonedeadpunk | but I really don't like that approach as it's fishy | 16:18 |
| noonedeadpunk | would be really nice to catch up with Neil about his thoughts for packaging... | 16:18 |
| noonedeadpunk | spotz[m]: will also ping you, as that aggressing bumping of ovs/ovn versions for stable branches is not great, imo | 16:22 |
| noonedeadpunk | I can totally see how many troubles it might bring deployments... | 16:22 |
| noonedeadpunk | though I tried to discuss that previously already... | 16:23 |
| spotz[m] | noonedeadpunk: 917192? | 16:36 |
| nixbuilder | noonedeadpunk: 917192 seems like it might work for me... I'll try that. | 16:38 |
| noonedeadpunk | spotz[m]: well, yes, but that's I guess more about overall RDO policy to backport major ovn/ovs versions to stable releases | 16:43 |
| noonedeadpunk | As it feels that deployments might get weird mix of versions during their lifetime | 16:43 |
| noonedeadpunk | as well as get unexpected downtimes/race coditions due to that. Ie, we've catched that performing ovs minorly (ie that was for 2.6_>2.7) does result in brdige re-build and basically a downtime, which you might not expect to happen | 16:45 |
| noonedeadpunk | while bugfix releases are updated normally, so I assume there's also smth to schema update or need to rebuild flows that come into picture... | 16:46 |
| noonedeadpunk | So I'm really not sure how osa in this case should act, where without any release note or anything ppl just start getting upgrades or installed new major ovn versions to new hosts while old are kept as-is... | 16:49 |
| spotz[m] | Let me look | 16:51 |
| spotz[m] | In our case if you're going from say Bobcat to Caracal if there's an upgrade you should be upgraded. What you're proposing is hard coding in a specific version to prevent the upgrade from taking place. I think some downtime should be expected if you're not in a SLURP release and want to do the net one? Though tobe honest I can't remember if this is an upgrade yes or an upgrade no release | 16:54 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_trove master: Add service policies defenition https://review.opendev.org/c/openstack/openstack-ansible-os_trove/+/917225 | 16:56 |
| spotz[m] | RDO has been trying to more slowly (aka an extra cycle) retire things provided they pass our CI | 16:56 |
| noonedeadpunk | So I guess what I meant, is that like 2 days ago, bobcat was being installed rdo-ovn-2:23.09 and today it's trying to get rdo-ovn-2:24.03 | 16:57 |
| noonedeadpunk | which has really nothing to do with openstack upgrades, but backport policy | 16:58 |
| spotz[m] | jcapitao: ^ tried to ping Alfredo too but he's not here | 16:58 |
| spotz[m] | Getting an engineer who might know why:) | 16:58 |
| spotz[m] | It is unfortunately late for the team to respond | 16:59 |
| noonedeadpunk | and that's really not a single case. last time there was agressive backport from 2.17 to 3.1 for OVS on stable branches, which also brought some issues... | 16:59 |
| spotz[m] | First I'm hearing so let's see if we can figure it out something that works for everyone | 17:00 |
| noonedeadpunk | then motivation was that neutron team recommend to have latest / more modern ovs... but again there must be really solid reason imo | 17:00 |
| noonedeadpunk | if to check UCA - they do stick with same versions for stable releases and do make just bugfixes from time to time | 17:00 |
| spotz[m] | Unfortunately we canceled next week's meeting due to PTO/Holidays or I'd say come join. The following week will be our monthly video call if you want to join that | 17:01 |
| noonedeadpunk | I'm not saying to do the same or trying to compare, but I guess that's kind of what you would expect as a user... dunno | 17:01 |
| noonedeadpunk | I will be travelling during following week to openinfra days | 17:02 |
| noonedeadpunk | (which is unfortunate) | 17:02 |
| noonedeadpunk | but at least thanks for listening:) | 17:02 |
| jrosser_ | the aggressive backporting is something i also find pretty surprising | 17:04 |
| spotz[m] | I'll be at RH SUmmit:) I'll have someone from the team ping you to chat next week if someone is about | 17:04 |
| jrosser_ | we spent the last couple of weeks debugging some tcp stack wierdness on a RH '9' style kernel, happens to be rocky but the backported patches are the same | 17:05 |
| jrosser_ | and it turns out literally everyone, everywhere running the current RH-9-ish kernel on arm has a fatal memory accounting error | 17:06 |
| noonedeadpunk | huh | 17:07 |
| jrosser_ | and it's all due to a backport of some pretty speculative patch from 6.x kernels that turns out to have a hideous bug | 17:07 |
| spotz[m] | Ok I've pinged the team with the patch and some background and asked them to ping you next week | 17:07 |
| noonedeadpunk | thanks a ton:) | 17:07 |
| spotz[m] | You're welcome, I might try to help steer the boat but they'll know the why's and how's:) | 17:10 |
| spotz[m] | My superpower is knowing who to ask and getting folks together;) | 17:10 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_trove master: Add variable to globally control notifications enablement https://review.opendev.org/c/openstack/openstack-ansible-os_trove/+/917226 | 17:17 |
| nixbuilder | noonedeadpunk: Well your 917192 patch *almost* worked... for some reason I am still getting an error... fatal: [aio1]: FAILED! => {"attempts": 5, "changed": false, "failures": [], "msg": "Depsolve Error occurred: \n Problem: cannot install the best candidate for the job\n - nothing provides ovn24.03 needed by rdo-ovn-2:24.03-1.el9s.noarch from rdo-deps", "rc": 1, "results": []} | 17:21 |
| nixbuilder | noonedeadpunk: but it is just one error | 17:21 |
| noonedeadpunk | ugh | 17:22 |
| noonedeadpunk | seems there's smth I've missed | 17:23 |
| noonedeadpunk | ah | 17:23 |
| noonedeadpunk | I see now | 17:23 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_neutron master: Fix OVS/OVN versions for EL https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/917192 | 17:26 |
| noonedeadpunk | nixbuilder: can you check it now ^ ? | 17:26 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_neutron master: Update OVN/OVS versions for EL https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/917193 | 17:26 |
| nixbuilder | noonedeadpunk: Trying now :-D | 17:34 |
| nixbuilder | nonedeadpunk: 917192 now works. Thank you. But what is the patch 917193 for since ovn24.03 is not available yet? Or is this for other OS such as Ubuntu or RedHat? | 17:43 |
| noonedeadpunk | nah, it's for future when it become available | 18:18 |
| noonedeadpunk | as eventually we need to have 24.03 for caracal | 18:18 |
| noonedeadpunk | and then potentially backport 917192 to stable branches... | 18:28 |
| nixbuilder | noonedeadpunk: OK... thanks for the clarification and for your expert assistance. | 18:38 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!