| noonedeadpunk | o/ | 06:43 |
|---|---|---|
| jrosser | good morning | 07:40 |
| jrosser | ARA installs look broken https://zuul.opendev.org/t/openstack/build/246e0d7020034f84a940908c807702a0/log/job-output.txt#6459 | 07:48 |
| noonedeadpunk | Curiouser and curiouser..... | 07:52 |
| noonedeadpunk | I think we might also have issue in bootstrap for stable branches. At least ansible broke on us during bootstrap today with wrong version of ansible.netcommon | 07:55 |
| jrosser | maybe we got tzdata in the ansible-runtime venv by some other dependancy before | 07:56 |
| jrosser | we have some very long looking runtimes at the moment, i wonder if those are related to broken ARA | 08:08 |
| jrosser | like a failed callback per task might add up | 08:08 |
| admin1 | in a recent lab install, 27.0.1, i noticed controller unresponsive and console saying java -- oom .. .. checking further .. its used by zookeeper .. checking what is dependent on it and if we can tone down its memory limits .. | 08:57 |
| admin1 | don't recall seeing zookeeper till zed .. | 08:57 |
| noonedeadpunk | admin1: it's used for coordination and required for proper setup of cinder-volume in active/active mode | 09:03 |
| noonedeadpunk | Also used for things like Octavia jobboards and designate | 09:04 |
| noonedeadpunk | and yes, we didn't have it and it is not obligatory either | 09:04 |
| noonedeadpunk | (ah, and gnocchi) | 09:04 |
| admin1 | is there a way to bypass this , or at least limit memory ( i did not find anything in the vars ) to not have it oom and crash .. there were just 2 cirros instances in the cluster and nothing else | 09:21 |
| admin1 | specs of the controller in lab .. 12 vcpu, 32gb ram | 09:23 |
| noonedeadpunk | You may feel free not to install zookeeper - it's pretty much optional | 09:23 |
| admin1 | yes, but i did not installed it .. i do not have zookeer hosts or anything .. how do i ask it to not install | 09:24 |
| admin1 | oh .. | 09:25 |
| admin1 | coordination_hosts: *infrastructure_hosts | 09:25 |
| admin1 | its in lab, so i will drop and rebuild | 09:25 |
| noonedeadpunk | yup, that's it | 09:38 |
| jrosser | there are quite a few failed jobs due to `Retrieve upper constraints using https` | 10:43 |
| opendevreview | Marc Gariépy proposed openstack/openstack-ansible-haproxy_server stable/2023.1: Add possibility to override haproxy_ssl_path https://review.opendev.org/c/openstack/openstack-ansible-haproxy_server/+/897167 | 11:57 |
| opendevreview | Merged openstack/openstack-ansible-rabbitmq_server master: Fix linter errors in example playbook https://review.opendev.org/c/openstack/openstack-ansible-rabbitmq_server/+/896801 | 13:29 |
| openstack-ansib | Hi | 14:57 |
| noonedeadpunk | o/ | 14:59 |
| NeilHanlon | o/ | 14:59 |
| noonedeadpunk | #startmeeting openstack_ansible_meeting | 14:59 |
| opendevmeet | Meeting started Tue Oct 3 14:59:39 2023 UTC and is due to finish in 60 minutes. The chair is noonedeadpunk. Information about MeetBot at http://wiki.debian.org/MeetBot. | 14:59 |
| opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 14:59 |
| opendevmeet | The meeting name has been set to 'openstack_ansible_meeting' | 14:59 |
| * NeilHanlon needs more coffee | 14:59 | |
| noonedeadpunk | #topic rollcall | 14:59 |
| noonedeadpunk | Yeah, apparently me too :D | 14:59 |
| NeilHanlon | hehe | 14:59 |
| NeilHanlon | we're just a bit early today | 15:00 |
| mgariepy | hey ! | 15:00 |
| noonedeadpunk | \o/ | 15:01 |
| jrosser | o/ hello | 15:01 |
| damiandabrowski | hi! | 15:01 |
| noonedeadpunk | #topic bug triage | 15:01 |
| noonedeadpunk | To be frank - I in fact don't have much updates/to say from the last meeting. Was mostly occupied with internal things | 15:01 |
| noonedeadpunk | But, I've submitted couple of new bugs we've found during the upgrade | 15:01 |
| noonedeadpunk | But lets start from this one | 15:02 |
| noonedeadpunk | #link https://bugs.launchpad.net/openstack-ansible/+bug/2036421 | 15:02 |
| noonedeadpunk | I;ve replied to it, but I'm not sure if we wanna do smth there or just mark as invalid | 15:03 |
| openstack-ansib | I want to see if Horizon has been translated into Farsi or not?! | 15:03 |
| noonedeadpunk | As I see how this release note can be confusing, but I'm not sure what can be done | 15:03 |
| noonedeadpunk | openstack-ansib: you can see horizon translations here: https://translate.openstack.org/iteration/view/horizon/master?dswid=7334 | 15:04 |
| *** openstack-ansib is now known as roksana | 15:04 | |
| noonedeadpunk | regarding bug - I'm not sure what we can do at this point... We could add another release note back in the days, but not I guess it's useless | 15:05 |
| noonedeadpunk | *now | 15:05 |
| noonedeadpunk | So I guess I will mark it as invalid if no objections | 15:06 |
| NeilHanlon | that seems reasonable | 15:07 |
| noonedeadpunk | #link https://bugs.launchpad.net/openstack-ansible/+bug/2037554 | 15:08 |
| noonedeadpunk | this one is actually really bad | 15:08 |
| noonedeadpunk | and it's not limited to what is described there... | 15:08 |
| noonedeadpunk | I've catched another phase of this bug yesterday | 15:09 |
| jrosser | should we make that a separate play? | 15:10 |
| noonedeadpunk | So it failed on us when we were deploying new compute node | 15:10 |
| jrosser | just to make it simple | 15:10 |
| noonedeadpunk | As it attempted to do db migration against compute | 15:10 |
| noonedeadpunk | jrosser: yeah, I was thinking exactly about that tbh | 15:10 |
| noonedeadpunk | except we have quite some complexity in the playbook for nova... I guess that stopped me from doing this originally | 15:11 |
| noonedeadpunk | as I was thinking to add some post_tasks or do tasks_from, but that doesn't look too trivial with how we do things here https://opendev.org/openstack/openstack-ansible/src/branch/master/playbooks/os-nova-install.yml#L54-L58 | 15:12 |
| ayush | There is another bug here | 15:12 |
| ayush | https://bugs.launchpad.net/osops/+bug/2038315 | 15:12 |
| noonedeadpunk | Ok, moved it to the correct project | 15:13 |
| jrosser | common-playbooks/nova.yml is kind of nasty too | 15:14 |
| noonedeadpunk | sorry for not describing bugs in a good way - was pretty much in a rush and needed not to forget about them | 15:14 |
| jrosser | as it attempts to be common, then a whole bunch of the tasks are conditional for computes only | 15:14 |
| noonedeadpunk | yup, it pretty much is | 15:14 |
| jrosser | so that could be simplified too | 15:14 |
| jrosser | like too much optimisation there makes a mess | 15:14 |
| noonedeadpunk | I wonder if we need to have that common playbook | 15:15 |
| jrosser | true | 15:15 |
| jrosser | perhaps some tidying here would make a neat place to deal with that bug | 15:15 |
| noonedeadpunk | as it may be way more plain when moved to os-nova-install | 15:15 |
| noonedeadpunk | ok, sounds like a plan then | 15:15 |
| noonedeadpunk | hopefully, it will be backportable enough... | 15:16 |
| noonedeadpunk | #link https://bugs.launchpad.net/openstack-ansible/+bug/2038315 | 15:16 |
| noonedeadpunk | So, we have some stable branches borked due to ansible.netcommon breaking galaxy | 15:16 |
| noonedeadpunk | I didn't have time to look there today, but we're catching failures internally as well | 15:16 |
| noonedeadpunk | Except 2023.1 - it's working fine | 15:17 |
| noonedeadpunk | (as we moved to github from galaxy there) | 15:17 |
| noonedeadpunk | ayush: did workaround I've posted you helped? | 15:17 |
| noonedeadpunk | ie - create user-collection-requirements.yml with https://paste.openstack.org/show/bC5oFVEOBoOdB9yMb9Dg/ ? | 15:17 |
| NeilHanlon | i can probably find some time to look at the galaxy problem today/tomorrow | 15:18 |
| noonedeadpunk | damiandabrowski: actually, you've catched that yesterday as well | 15:18 |
| jrosser | what actually is the bug? | 15:18 |
| noonedeadpunk | that bootstrap fails of some stable branches | 15:18 |
| jrosser | oh sure but is there a new netcommon that breaks our old stuff? | 15:19 |
| noonedeadpunk | because it can't install ansible.netcommon for $reasons | 15:19 |
| jrosser | ahha ok | 15:19 |
| noonedeadpunk | yeah, I believe it just breaks galaxy overall | 15:19 |
| noonedeadpunk | I believe that's related to them having `v` in tags, while not having in versions or smth like that | 15:20 |
| noonedeadpunk | (as ansible wasn't happy either in some scenarios) | 15:20 |
| NeilHanlon | "unexpected error when getting available versions of collection" | 15:20 |
| NeilHanlon | such a helpful message 🙃 | 15:20 |
| noonedeadpunk | Skipping Galaxy server https://galaxy.ansible.com/api/. Got an unexpected error when getting available versions of collection ansible.netcommon | 15:20 |
| noonedeadpunk | yeah | 15:20 |
| NeilHanlon | #link https://github.com/ansible-collections/ansible.netcommon/issues/562 | 15:21 |
| ayush | @noonedeadpunk : i have not tried, i will try and let you know the results by tmrw | 15:21 |
| ayush | mrng | 15:21 |
| noonedeadpunk | But, I see no new commits in collection | 15:22 |
| noonedeadpunk | And web ui of galaxy doesn't seem broken | 15:22 |
| noonedeadpunk | https://galaxy.ansible.com/ui/repo/published/ansible/netcommon/ | 15:22 |
| noonedeadpunk | So... it's either already fixed... Or API != UI | 15:22 |
| NeilHanlon | ope | 15:22 |
| NeilHanlon | # https://github.com/ansible/awx/issues/14496 | 15:22 |
| jrosser | this is "galaxy ng" isnt it? | 15:22 |
| NeilHanlon | #link https://github.com/ansible/awx/issues/14496 | 15:22 |
| noonedeadpunk | "We've had similiar issue today but with 'ansible.netcommon' collection" | 15:23 |
| noonedeadpunk | " Removal of default Galaxy credential from organisation settings helped us mitigate the problem." | 15:23 |
| NeilHanlon | > However the old-galaxy.ansible.com settings seems to have solved the issue for us ;-) | 15:23 |
| noonedeadpunk | Lol | 15:23 |
| * NeilHanlon adds liquor to his coffee | 15:23 | |
| noonedeadpunk | This reminds me of some company.... that maintains CentOS Stream... | 15:23 |
| noonedeadpunk | What the name was that... | 15:24 |
| noonedeadpunk | Anyway | 15:24 |
| noonedeadpunk | Might be we can replace galaxy with github or just wait for the fix... | 15:24 |
| noonedeadpunk | OR, we need to update ansible version... | 15:25 |
| NeilHanlon | i'm gonna ping greg sutcliffe.. get his thoughts :) | 15:25 |
| noonedeadpunk | Yeah, would be nice to hear | 15:26 |
| NeilHanlon | seems like new ansible won't help | 15:26 |
| noonedeadpunk | #link https://bugs.launchpad.net/openstack-ansible/+bug/2037649 | 15:27 |
| NeilHanlon | #link https://forum.ansible.com/t/awx-cannot-download-collections-roles-from-the-new-galaxy/1251/4?u=neil | 15:27 |
| noonedeadpunk | that looks really messy... | 15:27 |
| noonedeadpunk | What makes it worse, that in branches like Xena we also had issues with rabbitmq versions... | 15:28 |
| NeilHanlon | :( | 15:28 |
| NeilHanlon | yeah | 15:28 |
| noonedeadpunk | so 2037649 is another race condition bug, but thankfully it affects only vpnaas | 15:29 |
| mgariepy | rabbitmq is a mess.. | 15:29 |
| noonedeadpunk | jamesdenton: you might be interested in https://bugs.launchpad.net/openstack-ansible/+bug/2037649 as well | 15:29 |
| noonedeadpunk | I will check what's the best approach, but I guess it would be change of the destination directory to the one inside the venv | 15:29 |
| noonedeadpunk | so that symlinking would be not a problem | 15:30 |
| noonedeadpunk | #topic office hours | 15:30 |
| noonedeadpunk | I failed to book a PTG slots until now :( | 15:30 |
| noonedeadpunk | Will try to do that during this week | 15:30 |
| noonedeadpunk | Another thing is that we likely should switch from master to tracking stable/2023.2 since all projects have branched already | 15:31 |
| noonedeadpunk | Will propose patches soonish | 15:31 |
| noonedeadpunk | #action noonedeadpunk book PTG slots | 15:31 |
| jrosser | do we have a TODO list for bobcat | 15:32 |
| noonedeadpunk | #action noonedeadpunk propose patch to track 2023.2 | 15:32 |
| jrosser | like actual patches to chase and deal with | 15:32 |
| jrosser | seems to be many many open | 15:32 |
| jrosser | i think we also need to give the CI some tlc | 15:32 |
| jrosser | jobs seem to be running slowly, upgrades are TIMEOUT regularly atm | 15:33 |
| noonedeadpunk | I think it's Debian 12, quorum queues at minimum | 15:33 |
| jrosser | then there is a fair number of failing to retrieve u-c with https | 15:33 |
| noonedeadpunk | we can discuss rest, like Skyline | 15:33 |
| jrosser | and I see a number of Rocky repo errors every day still | 15:34 |
| noonedeadpunk | But what we tried to scope at the beginning of cycle is here | 15:34 |
| noonedeadpunk | #link https://etherpad.opendev.org/p/osa-bobcat-ptg | 15:34 |
| noonedeadpunk | Ah, we have openstack-resources role | 15:35 |
| noonedeadpunk | #link https://review.opendev.org/q/topic:osa%252Fopenstack_resources | 15:35 |
| noonedeadpunk | Ugh, it's failing with same galaxy issue | 15:35 |
| NeilHanlon | bleh | 15:36 |
| jrosser | oh no curse of the tests repo | 15:36 |
| noonedeadpunk | yeah-yeah... | 15:36 |
| NeilHanlon | I will look into rocky issues, jrosser. i know of a few lingering problems I'm trying to snuff out.. but also wondering based on what you're saying and I'm seeing if I just need to give the CDN a good ol' flush. | 15:37 |
| noonedeadpunk | and modular libvirt would be also a thing nice to have | 15:37 |
| NeilHanlon | yea, i have an update on that | 15:37 |
| NeilHanlon | It's basically confirmed to be the only thing available in EL10 | 15:37 |
| NeilHanlon | and ELN composes have begun for what will become CentOS Stream 10 | 15:37 |
| NeilHanlon | https://composes.stream.centos.org/stream-10/ | 15:37 |
| noonedeadpunk | so we need to get to bottom of that sooner or later anyway | 15:38 |
| noonedeadpunk | And it seems that Debian is not going the same path, at least not with Debian 12? | 15:38 |
| NeilHanlon | yeah. i am going to schedule myself some time to come up with a plan, especially around how to migrate/upgrade to modular libvirt | 15:40 |
| noonedeadpunk | well. if it's only EL10... | 15:41 |
| noonedeadpunk | Maybe no migration needed - you still can't upgrade 9->10 | 15:41 |
| noonedeadpunk | and then we can wait for 10 for that | 15:42 |
| NeilHanlon | true, but i wonder about folks running mixed environments? | 15:42 |
| noonedeadpunk | make up some conditions? | 15:42 |
| NeilHanlon | i.e., 9 and 10 | 15:42 |
| NeilHanlon | maybe you're right--we can just not worry about it until we support EL10 | 15:42 |
| noonedeadpunk | like define services to restart in vars per distro | 15:42 |
| NeilHanlon | yeah | 15:42 |
| noonedeadpunk | like we have already here: https://opendev.org/openstack/openstack-ansible-os_nova/src/branch/master/vars/redhat.yml#L19-L21 | 15:43 |
| NeilHanlon | right right | 15:43 |
| noonedeadpunk | But I bet it would be way-way more complicated.... | 15:43 |
| noonedeadpunk | As I'm not sure I understand logic behind this modular thing and how to make tls/non-tls things to work... | 15:43 |
| noonedeadpunk | And the order to start/restart | 15:44 |
| NeilHanlon | yeah | 15:45 |
| NeilHanlon | i'm not looking forward to it, honestly. lol | 15:45 |
| noonedeadpunk | so, does the list of Debian 12, quorum queues support, and openstack-resource role (possibly) sounds good enough? | 15:48 |
| noonedeadpunk | I was planning also to work on PKI improvement to add vault support, but I'm not sure if I will be able to fit | 15:48 |
| noonedeadpunk | And also would better release earlier... | 15:48 |
| NeilHanlon | that sounds good to me | 15:50 |
| NeilHanlon | would rather less things we're more confident with | 15:50 |
| noonedeadpunk | I will also try to work on documentation review. I've already posted some changes, inlcuding some OVN drawing here https://review.opendev.org/c/openstack/openstack-ansible/+/894384 | 15:50 |
| noonedeadpunk | which would be great to get some feedbacks on... | 15:51 |
| noonedeadpunk | And was going to make openstack_user_config examples OVN-friendly | 15:51 |
| noonedeadpunk | As I guess most users copying them and running into issues afterwards | 15:51 |
| noonedeadpunk | ok, so if that's it | 15:53 |
| noonedeadpunk | #endmeeting | 15:53 |
| opendevmeet | Meeting ended Tue Oct 3 15:53:17 2023 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 15:53 |
| opendevmeet | Minutes: https://meetings.opendev.org/meetings/openstack_ansible_meeting/2023/openstack_ansible_meeting.2023-10-03-14.59.html | 15:53 |
| opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/openstack_ansible_meeting/2023/openstack_ansible_meeting.2023-10-03-14.59.txt | 15:53 |
| opendevmeet | Log: https://meetings.opendev.org/meetings/openstack_ansible_meeting/2023/openstack_ansible_meeting.2023-10-03-14.59.log.html | 15:53 |
| opendevreview | Merged openstack/openstack-ansible master: Add barbican api microversion limits for tempest tests https://review.opendev.org/c/openstack/openstack-ansible/+/897002 | 15:58 |
| NeilHanlon | thanks noonedeadpunk! | 15:59 |
| opendevreview | Merged openstack/openstack-ansible-os_tempest master: Allow tempest_tempest_conf_overrides to be defined in many variables https://review.opendev.org/c/openstack/openstack-ansible-os_tempest/+/894757 | 16:19 |
| opendevreview | Merged openstack/openstack-ansible-os_barbican master: Add quorum support for service https://review.opendev.org/c/openstack/openstack-ansible-os_barbican/+/895693 | 16:53 |
| opendevreview | Merged openstack/openstack-ansible-rabbitmq_server master: Add ability to define extra arguments for erlang https://review.opendev.org/c/openstack/openstack-ansible-rabbitmq_server/+/895686 | 17:18 |
| Moha | /etc/neutron/plugins/ml2/openvswitch_agent.ini: `bridge_mappings = provider:PROVIDER_INTERFACE_NAME` (src: | 21:41 |
| Moha | /etc/neutron/plugins/ml2/ml2_conf.ini: `flat_networks = provider` | 21:42 |
| Moha | `ovs-vsctl add-br provider` | 21:43 |
| Moha | `ovs-vsctl add-port br-ex <PROVIDER_INTERFACE_NAME, i.e enp0s9>` | 21:44 |
| Moha | `ovs-vsctl add-port provider* <PROVIDER_INTERFACE_NAME, i.e enp0s9>` | 21:44 |
| Moha | But OVS expects enp0s9 to be a bridge! | 21:46 |
| Moha | (in manual installation) | 21:46 |
| Moha | src: https://docs.openstack.org/neutron/latest/install/compute-install-option2-ubuntu.html | 21:53 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!