| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_placement master: Add online_data_migrations for placement https://review.opendev.org/c/openstack/openstack-ansible-os_placement/+/892159 | 06:19 |
|---|---|---|
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_keystone master: Fix docs jobs after sphinx update https://review.opendev.org/c/openstack/openstack-ansible-os_keystone/+/893701 | 06:26 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-lxc_hosts master: Stop installing openssh and rsync to containers https://review.opendev.org/c/openstack/openstack-ansible-lxc_hosts/+/889945 | 06:29 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_keystone master: Install distro_packages in pre-main https://review.opendev.org/c/openstack/openstack-ansible-os_keystone/+/889934 | 06:30 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_adjutant stable/2023.1: Use proper galera port in configuration https://review.opendev.org/c/openstack/openstack-ansible-os_adjutant/+/892474 | 06:44 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_adjutant stable/2023.1: Stop reffering _member_ role https://review.opendev.org/c/openstack/openstack-ansible-os_adjutant/+/892099 | 06:44 |
| noonedeadpunk | seems we have issues with rocky and systemd now. Or at least had yesterday. | 06:46 |
| noonedeadpunk | also lxb is broken for centos on Yoga | 06:47 |
| noonedeadpunk | fails on running iptables-restore | 06:47 |
| noonedeadpunk | kinda wonder if that's a missing rootwrap... | 06:56 |
| noonedeadpunk | like https://review.opendev.org/c/openstack/neutron/+/778444 | 06:56 |
| noonedeadpunk | but unlikely that we've spotted that just now... | 06:57 |
| noonedeadpunk | and then it's only centos.... | 06:58 |
| jrosser | i did see some unrelated centos+iptables update stuff in other openstack irc | 08:35 |
| jrosser | like updated package -> changed things -> brokenness | 08:35 |
| opendevreview | Slawek Kaplonski proposed openstack/ansible-config_template master: DNM Just testing some EDPM ansible issue in ovn role https://review.opendev.org/c/openstack/ansible-config_template/+/893711 | 08:48 |
| noonedeadpunk | hm, ok, this might be related in fact | 09:05 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible master: WIP - test Vexxhost CAPI driver for magnum https://review.opendev.org/c/openstack/openstack-ansible/+/893240 | 09:28 |
| jrosser | hmm also a notification in our internal slack that rocky repos are a bit broken | 09:50 |
| noonedeadpunk | wrt systemd | 10:04 |
| noonedeadpunk | ? | 10:04 |
| jrosser | i am not sure, it was more a generic "expect your CI to be broken" | 10:05 |
| jrosser | i think i have got k8s to properly initialise inside an LXC now | 10:05 |
| jrosser | thats relevant/required for any cluster-api implementation in magnum i think | 10:06 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible master: WIP - test Vexxhost CAPI driver for magnum https://review.opendev.org/c/openstack/openstack-ansible/+/893240 | 13:42 |
| NeilHanlon | jrosser: sorry about that :/ -- we should be stable again | 14:05 |
| NeilHanlon | for some value of stable | 14:05 |
| jrosser | NeilHanlon: is this related to iptables package at all | 14:06 |
| NeilHanlon | don't think so -- we were just having infra issues earlier | 14:06 |
| noonedeadpunk | iptables I assume only centos-related... | 14:09 |
| noonedeadpunk | at least from what I saw in stable branches | 14:09 |
| jrosser | hmm ok | 14:11 |
| jrosser | we do seem to have a lot of rocky9 fail but i did not manage to take a look | 14:11 |
| noonedeadpunk | in gates it was that packages are desynced in mirrors | 14:12 |
| noonedeadpunk | specifically failed in systemd, but it could be jsut first occurance | 14:12 |
| jrosser | maybe we should recheck a bunch and see what the state is now | 14:17 |
| NeilHanlon | probably not a bad idea. we had our whole haproxy tier go out to lunch | 14:18 |
| NeilHanlon | so mirrorlist wouldn't work, e.g. | 14:18 |
| noonedeadpunk | and smth was also off on the infra side, as ubuntu was failing with fetching hatop... | 14:31 |
| * NeilHanlon puts on his tinfoil hat | 14:33 | |
| NeilHanlon | > it's the aliens! | 14:33 |
| noonedeadpunk | #startmeeting openstack_ansible_meeting | 15:00 |
| opendevmeet | Meeting started Tue Sep 5 15:00:13 2023 UTC and is due to finish in 60 minutes. The chair is noonedeadpunk. Information about MeetBot at http://wiki.debian.org/MeetBot. | 15:00 |
| opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 15:00 |
| opendevmeet | The meeting name has been set to 'openstack_ansible_meeting' | 15:00 |
| noonedeadpunk | #topic rollcall | 15:00 |
| mgariepy | hello ! | 15:00 |
| noonedeadpunk | o/ hey | 15:02 |
| NeilHanlon | o/ hiya | 15:02 |
| noonedeadpunk | #topic gate health | 15:03 |
| noonedeadpunk | so lately we hit bunch of issues | 15:03 |
| noonedeadpunk | starting with infra failing to fetch hatop, then l3 agent breaks on lxb for centos 8 | 15:04 |
| noonedeadpunk | and something weird for centos 9 on Yoga - it can't find libvirt.d service again, despite we supposedly aready have a patch | 15:05 |
| noonedeadpunk | #link https://review.opendev.org/c/openstack/openstack-ansible-os_nova/+/884381 | 15:06 |
| noonedeadpunk | I wasn't able to dig deep enough in any area, except I assume that infra and rocky issues are intermittent and should be fixed soonish | 15:06 |
| noonedeadpunk | but please, limit your rechecks, as we're in quite bad shape right now | 15:07 |
| noonedeadpunk | #topic office hours | 15:07 |
| noonedeadpunk | I've holded 27.1.0 release due to https://review.opendev.org/c/openstack/openstack-ansible/+/893547 not being merged. It looks like very important patch, as I've seen multiple inquires in different places regarding need to not have gateways on computes | 15:08 |
| noonedeadpunk | We also should finally prioritize docs update, as that is quite major blocker for new users | 15:09 |
| NeilHanlon | Yes. I want to spend some time on that myself | 15:10 |
| noonedeadpunk | osa/quorum_queues has landed for core roles. Right now AIO is broken , because new SHA is needed for plugins repo | 15:11 |
| noonedeadpunk | It's already proposed here https://review.opendev.org/c/openstack/openstack-ansible/+/892373 but quite some requirements need to be merged first... | 15:12 |
| noonedeadpunk | So maybe it's worth to propose that as a separate one | 15:12 |
| noonedeadpunk | With that I will push patches for quorum_queues for the rest of repos during this week. | 15:13 |
| noonedeadpunk | (once gates will be unblocked) | 15:13 |
| noonedeadpunk | Also, adjutant is feeling much better now on 2023.1 (if not rocky :D) | 15:14 |
| noonedeadpunk | #link https://review.opendev.org/c/openstack/openstack-ansible-os_adjutant/+/891279/6 | 15:14 |
| noonedeadpunk | once this gets merged, and SHA updated in integrated repo - master should be quite happy as well. | 15:15 |
| mgariepy | should merge soon ;) | 15:15 |
| noonedeadpunk | osa/core-2.15 is also pretty much merged. | 15:16 |
| noonedeadpunk | #link https://review.opendev.org/q/topic:osa/core-2.15+status:open | 15:16 |
| NeilHanlon | nice | 15:16 |
| noonedeadpunk | for senlin to pass this needs to merge: https://review.opendev.org/c/openstack/openstack-ansible/+/892910 | 15:16 |
| noonedeadpunk | So really good progress | 15:17 |
| NeilHanlon | indeed. great work folks | 15:17 |
| NeilHanlon | +2 from me on 892910 | 15:17 |
| noonedeadpunk | awesome :) | 15:18 |
| noonedeadpunk | Right now we're upgrading our first region to 2023.1 and found 2 things. Both are small, but one is quite nasty and not sure how to solve it. | 15:18 |
| noonedeadpunk | Will fill a bug report shortly to discuss on the next bug triage | 15:19 |
| noonedeadpunk | in short - we delegate 1 task to nova_conductor, while running against last compute. And task execute handlers, but handlers run not against delegated host, but against one that it "was executiong against" | 15:20 |
| noonedeadpunk | so basically - handlers are not delegated, which sounds like... intended? | 15:20 |
| noonedeadpunk | but not sure how to overcome that | 15:20 |
| damiandabrowski | sorry for being late | 15:21 |
| * damiandabrowski promised to report a bug regarding handlers delegation, will do that tomorrow | 15:21 | |
| noonedeadpunk | I don't have any other updates yet. | 15:24 |
| noonedeadpunk | Didn't have chance to work on debian 12 support | 15:24 |
| noonedeadpunk | Ah, I also will be on vacation next week. Hopefully, will be able to join the meeting but no guarantees. | 15:25 |
| noonedeadpunk | Any volunteer to chair next meeting? | 15:25 |
| NeilHanlon | I am also out the next two weeks -- might be able to join next week but same as you, no guarantees | 15:26 |
| damiandabrowski | I'm also absent next week | 15:28 |
| noonedeadpunk | ok, then maybe we should cancel next week meeting? | 15:28 |
| noonedeadpunk | will cancel then :) | 15:30 |
| damiandabrowski | sounds good | 15:30 |
| NeilHanlon | sounds good to me | 15:30 |
| noonedeadpunk | awesome. since there's nothing more to discuss - let's make it short | 15:31 |
| noonedeadpunk | #endmeeting | 15:31 |
| opendevmeet | Meeting ended Tue Sep 5 15:31:51 2023 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 15:31 |
| opendevmeet | Minutes: https://meetings.opendev.org/meetings/openstack_ansible_meeting/2023/openstack_ansible_meeting.2023-09-05-15.00.html | 15:31 |
| opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/openstack_ansible_meeting/2023/openstack_ansible_meeting.2023-09-05-15.00.txt | 15:31 |
| opendevmeet | Log: https://meetings.opendev.org/meetings/openstack_ansible_meeting/2023/openstack_ansible_meeting.2023-09-05-15.00.log.html | 15:31 |
| Karni | Hi | 16:39 |
| Karni | noonedeadpunk: May you please this bandwith tests: https://i.imgur.com/SmMXan9.png | 16:41 |
| Karni | Design: VM1 <----> ASAv Cisco Firewall <----> VM2 | 16:42 |
| Karni | each row is a test where those 3 virtual machines are on a different compute | 16:43 |
| Karni | Column4: The compute hosting ASAv as the monitoring tool to cature bit rates. | 16:44 |
| Karni | Column5: where VM1 is located | 16:44 |
| Karni | Column6: the other VM | 16:44 |
| Karni | In column5&6 there are to number per cell. The upper number is the bit rate in Mb/s for the traffic that goes from VM1 to VM2 buth through the ASAv that is the gateway for VMs. | 16:45 |
| Karni | The second number is related to the traffic vice versa, that goes from VM2 forwM1 using the siwth `-r` in ethr command. | 16:46 |
| Karni | I used ethr for testing bandwith instead of the iperf3 as iperf3 is only single threaded. | 16:47 |
| Karni | The two last column are the result in MQ-enabled and MQ-disabled servers. | 16:49 |
| Karni | The results are weird! | 16:49 |
| Karni | I couldn't find a pattern! | 16:49 |
| Karni | Note: in our setup, controlers and network nodes are in virtual machines hosted on a ESXi cluster. | 16:50 |
| Karni | Do you have any suggestion for our bandwith fluctuation over different servers? | 16:51 |
| noonedeadpunk | Karni: I have close to zero idea, but I'm not sure if that's related to openstack overall comapring to the underlying infrastructure | 17:02 |
| noonedeadpunk | Also, if this is nested virtualization (ie you run nova-compute inside other VM) -that would affect VM performance | 17:03 |
| noonedeadpunk | but I have no experience neither with esxi nor with cisco firewalls, so I don't have any advices :( | 17:04 |
| opendevreview | Merged openstack/openstack-ansible-os_adjutant stable/2023.1: Use version mysqlclient<2.2.0 https://review.opendev.org/c/openstack/openstack-ansible-os_adjutant/+/891279 | 17:07 |
| opendevreview | Merged openstack/openstack-ansible master: Update Senlin SHA https://review.opendev.org/c/openstack/openstack-ansible/+/892910 | 17:36 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible stable/yoga: Use include_role in task to avoid lack of access to vars https://review.opendev.org/c/openstack/openstack-ansible/+/889021 | 17:47 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: Fix container bridge name for octavia https://review.opendev.org/c/openstack/openstack-ansible/+/893315 | 17:48 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: Define install_method default when hosts resolution depend on it https://review.opendev.org/c/openstack/openstack-ansible/+/891697 | 17:49 |
| opendevreview | Marc Gariépy proposed openstack/openstack-ansible-os_nova master: Config has changed for pci passthrough. https://review.opendev.org/c/openstack/openstack-ansible-os_nova/+/893797 | 17:53 |
| jamesdenton | was the ASAv model/flavor information ever provided? That makes a difference | 17:58 |
| noonedeadpunk | um not sure | 18:00 |
| mgariepy | noonedeadpunk, the config is the exact same only the key needs changed, if we do rename the var i need to patchup the upgrade script and so on. | 18:09 |
| noonedeadpunk | we can keep old one as default | 18:09 |
| noonedeadpunk | like we did with tempest | 18:09 |
| noonedeadpunk | like https://opendev.org/openstack/openstack-ansible-os_tempest/src/branch/stable/2023.1/defaults/main.yml#L170 | 18:10 |
| mgariepy | ok will update the patch | 18:12 |
| opendevreview | Marc Gariépy proposed openstack/openstack-ansible-os_nova master: Config has changed for pci passthrough. https://review.opendev.org/c/openstack/openstack-ansible-os_nova/+/893797 | 18:13 |
| noonedeadpunk | mgariepy: some more comments, sorry :( | 18:18 |
| noonedeadpunk | also we should backport that to.... antelope? zed? | 18:18 |
| mgariepy | zed | 18:18 |
| opendevreview | Marc Gariépy proposed openstack/openstack-ansible-os_nova master: Config has changed for pci passthrough. https://review.opendev.org/c/openstack/openstack-ansible-os_nova/+/893797 | 18:19 |
| mgariepy | no worry ;) should be better now. | 18:20 |
| opendevreview | Marc Gariépy proposed openstack/openstack-ansible-os_nova master: Config has changed for pci passthrough. https://review.opendev.org/c/openstack/openstack-ansible-os_nova/+/893797 | 18:21 |
| mgariepy | ok .. finishing reading the whole comment. now it should be better hahaha | 18:21 |
| noonedeadpunk | mgariepy: if you also could add a release note that variable is renamed and old one is deprecated.... | 18:24 |
| opendevreview | Marc Gariépy proposed openstack/openstack-ansible-os_nova master: Config has changed for pci passthrough. https://review.opendev.org/c/openstack/openstack-ansible-os_nova/+/893797 | 18:27 |
| opendevreview | Marc Gariépy proposed openstack/openstack-ansible-os_nova master: Config has changed for pci passthrough. https://review.opendev.org/c/openstack/openstack-ansible-os_nova/+/893797 | 18:29 |
| mgariepy | hopefully now it's ok :) haha | 18:29 |
| opendevreview | Merged openstack/openstack-ansible-os_neutron master: Revert "Workaround ovs bug that resets hostname with add command" https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/891453 | 18:38 |
| opendevreview | Merged openstack/openstack-ansible-os_senlin master: Fix linters and metadata https://review.opendev.org/c/openstack/openstack-ansible-os_senlin/+/888607 | 21:56 |
| opendevreview | Merged openstack/openstack-ansible-os_tempest master: Ensure test exclusion file is removed when there are no exclusions https://review.opendev.org/c/openstack/openstack-ansible-os_tempest/+/891586 | 22:01 |
| opendevreview | Merged openstack/openstack-ansible-os_senlin master: Use proper galera port in configuration https://review.opendev.org/c/openstack/openstack-ansible-os_senlin/+/890131 | 22:13 |
| opendevreview | Merged openstack/openstack-ansible-os_keystone master: Fix docs jobs after sphinx update https://review.opendev.org/c/openstack/openstack-ansible-os_keystone/+/893701 | 22:31 |
| opendevreview | Merged openstack/openstack-ansible master: Fix container bridge name for octavia https://review.opendev.org/c/openstack/openstack-ansible/+/893315 | 22:55 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!
