| opendevreview | OpenStack Proposal Bot proposed openstack/openstack-ansible master: Imported Translations from Zanata https://review.opendev.org/c/openstack/openstack-ansible/+/965438 | 04:07 |
|---|---|---|
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: Update rally SHA https://review.opendev.org/c/openstack/openstack-ansible/+/965308 | 08:52 |
| kleini | During upgrade to E, I ran into a broken glusterfs, too. I tried fixing through backup, container delete and recreate and wondered, why glusterfs always get back into same broken state until I found /var/lib/gluster being bind mount from underlying host. | 12:52 |
| noonedeadpunk | I was just looking into this | 12:52 |
| noonedeadpunk | somehow gluster seems quite unique in terms of bindings | 12:53 |
| noonedeadpunk | https://codesearch.openstack.org/?q=_container_bind_mounts&i=nope&literal=nope&files=&excludeFiles=&repos= | 12:53 |
| noonedeadpunk | like ceph, mysql, zookeeper, glance - all just bind mount to "/openstack/{{ inventory_hostname }}" | 12:53 |
| kleini | Just wanted to add that "plus 1" to the above gluster playbook discussion. | 12:53 |
| noonedeadpunk | and only gluster to "/openstack/glusterd/{{ inventory_hostname }}" | 12:54 |
| noonedeadpunk | so if we delete the container, this directory does not get cleaned up | 12:54 |
| noonedeadpunk | yeah... I wouldn't expect though it breaking during upgrade... | 13:00 |
| kleini | In my case the step to manully remove one brick might not have finished successfully (as mentioned in distro upgrade) and add_brick then kept failing. I was not able to fix it manually. | 13:14 |
| opendevreview | Merged openstack/openstack-ansible-rabbitmq_server master: Change repository for deb to deb1.rabbitmq.com https://review.opendev.org/c/openstack/openstack-ansible-rabbitmq_server/+/965196 | 13:16 |
| noonedeadpunk | so I just tried to drop one repo container and then spawn a new one, and it worked pretty much nicely for me. but I did not drop anything yet. | 13:20 |
| jrosser | isnt this something to do with it preserving the brick on the deleted node then re-adding that to a re-provisioned container? | 13:55 |
| jrosser | iirc its much more complicated to remove the old brick and re-add a new one, than to just re-add the one from before | 13:55 |
| kleini | https://docs.openstack.org/openstack-ansible/2025.1/admin/upgrades/distribution-upgrades.html#deploying-infrastructure-hosts <- chapter 3c mentions to remove the repo node aka brick | 13:56 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-plugins master: Allow to skip pacemaker/corosync setup https://review.opendev.org/c/openstack/openstack-ansible-plugins/+/966065 | 14:06 |
| noonedeadpunk | yeah, I think that's exactly the case... | 14:08 |
| noonedeadpunk | ok, so that's a distro upgrade, not openstack upgrade | 14:08 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-plugins master: Allow to skip pacemaker/corosync setup https://review.opendev.org/c/openstack/openstack-ansible-plugins/+/966065 | 14:16 |
| kleini | sorry, it is a distro upgrade not the upgrade to E. That was wrong in my initial statement. | 14:43 |
| noonedeadpunk | ++ thanks for clarification, makes much more sense now. And I was able to reproduce the behavior | 14:45 |
| noonedeadpunk | not sure how to fix it though :D | 14:45 |
| noonedeadpunk | gluster output format is not really helping either | 14:46 |
| noonedeadpunk | so we can collect current uuids, and we can collect peer uuids potentially. with some intersection, we can get ones that should be removed from peering | 14:47 |
| noonedeadpunk | but we need to map uuids to hostnames as well... | 14:47 |
| noonedeadpunk | and also remove bricks | 14:48 |
| noonedeadpunk | and also handle glusterfs_bootstrap_node logic, to ensure that we don't use currently missing gluster member for creating peers. | 14:53 |
| noonedeadpunk | oops | 15:06 |
| noonedeadpunk | timezones | 15:06 |
| noonedeadpunk | #startmeeting openstack_ansible_meeting | 15:06 |
| opendevmeet | Meeting started Tue Nov 4 15:06:43 2025 UTC and is due to finish in 60 minutes. The chair is noonedeadpunk. Information about MeetBot at http://wiki.debian.org/MeetBot. | 15:06 |
| opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 15:06 |
| opendevmeet | The meeting name has been set to 'openstack_ansible_meeting' | 15:06 |
| noonedeadpunk | #topic rollcall | 15:06 |
| noonedeadpunk | o/ | 15:06 |
| noonedeadpunk | #topic office hours | 15:09 |
| noonedeadpunk | We had a quite productive PTG session last week, I am still lagging behind a little bit in terms of sending out the summary | 15:10 |
| noonedeadpunk | I am expecting to send it out today by EOD | 15:10 |
| noonedeadpunk | We had also plenty of progress regarding Debian 13 support | 15:11 |
| noonedeadpunk | all votes are there, CI is passing | 15:11 |
| noonedeadpunk | If everything goes up to the plan - we have all chances to land patches today or tomorrow | 15:12 |
| noonedeadpunk | PKI progress - not much so far | 15:14 |
| noonedeadpunk | I am still struggling to review the patch to the pki role adding the vault driver | 15:15 |
| noonedeadpunk | it somehow always end up somewhere at the end of the list | 15:15 |
| noonedeadpunk | And we're coming to the end of the development cycle for 2025.2. Ideally, we need to branch next week, according to the ML sent by releases team yesterday | 15:17 |
| noonedeadpunk | so we're running on time wrt vault driver :* | 15:18 |
| noonedeadpunk | *out of time | 15:18 |
| noonedeadpunk | we also need to make new minor releases this week for all supported branches | 15:19 |
| noonedeadpunk | #link https://security.openstack.org/ossa/OSSA-2025-002.html | 15:19 |
| noonedeadpunk | so once patches are merged to keystone, I will propose a bump and a release | 15:31 |
| noonedeadpunk | Would be nice if we could backport and merge eligible to backport things | 15:32 |
| noonedeadpunk | and we probably should start with https://review.opendev.org/c/openstack/openstack-ansible-plugins/+/965941 | 15:33 |
| noonedeadpunk | as it's likely blocking this one: https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/965142 | 15:34 |
| noonedeadpunk | as well as https://review.opendev.org/c/openstack/ansible-role-uwsgi/+/965342 | 15:35 |
| noonedeadpunk | that's pretty much it from my side. | 15:36 |
| jrosser | o/ hello | 15:46 |
| noonedeadpunk | o/ | 15:50 |
| noonedeadpunk | Should we briefly discuss gluster thing? | 15:50 |
| NeilHanlon | ah crap, timezone change got me :( | 15:51 |
| opendevreview | Merged openstack/openstack-ansible-rabbitmq_server master: Bump RabbitMQ version to 4.1 series https://review.opendev.org/c/openstack/openstack-ansible-rabbitmq_server/+/965197 | 15:51 |
| NeilHanlon | gluster sure | 15:51 |
| noonedeadpunk | yup... | 15:51 |
| noonedeadpunk | so kind the thing is that we're not handling host being re-installed inside of the gluster role | 15:52 |
| noonedeadpunk | but also during container re-creation we leave stale data on the host | 15:52 |
| noonedeadpunk | thus I'm thinking about 2 possible actions here | 15:53 |
| noonedeadpunk | first - try to handle clean-up or better way to re-add containers/hosts to the cluster after OS re-install for instance | 15:53 |
| noonedeadpunk | and second - actually move /openstack/gluster/<hostname> to /openstack/<hostname>/gluster | 15:54 |
| noonedeadpunk | so that when you drop the container, gluster data was also cleaned up | 15:54 |
| mgariepy | it was working ok on 20.04 > 22.04 upgrade tho. | 15:54 |
| mgariepy | ho i did a dist-upgrade on the hosts. | 15:55 |
| noonedeadpunk | I think it works for us, but it's somehow unobvious behavior\ | 15:55 |
| noonedeadpunk | at least we get already couple of confused users in IRC withing couple of weeks | 15:55 |
| noonedeadpunk | meaning - we have smth to improve :) | 15:56 |
| mgariepy | yep. | 15:56 |
| noonedeadpunk | I have a sandbox to play with, and will try to come up with smth | 15:59 |
| noonedeadpunk | I wish gluster output could be in some parsable format, but I failed to find anything | 16:00 |
| jrosser | i think that i'd intended the gluster state to persist on the host if the container was deleted | 16:00 |
| jrosser | there is a uuid and other state in the data on each host which is shared across the cluster | 16:01 |
| jrosser | if you destroy a node and reprovision it, quite easy to end up with gluster being N+1 nodes but one missing | 16:02 |
| jrosser | this is quite easy to play with on a `infra` AIO because i think it makes 3 repo containers | 16:02 |
| noonedeadpunk | yeah, so I'm thinking if based of uuids we can drop the stale version of it | 16:02 |
| noonedeadpunk | as when container re-spawned it get's new UUID | 16:02 |
| jrosser | yes, then you have to somehow determine the one that should not be there and get rid of it | 16:03 |
| noonedeadpunk | so get all current uuids, get peers, intersect, drop missing | 16:03 |
| jrosser | indeed that could work | 16:03 |
| jrosser | but needs to be super careful if host down or something | 16:03 |
| noonedeadpunk | one problem I don't understand how to figure out, is mapping of old uuid to hostname | 16:03 |
| noonedeadpunk | as you need to drop peer by hostname specifically, you somehow can't do that by uuid | 16:03 |
| noonedeadpunk | I was thinking to do action only if uuid diff exists. if host is down, it should not be a problem I guess | 16:04 |
| noonedeadpunk | as it won't produce weird thing... | 16:05 |
| jrosser | gluster get-state | 16:08 |
| jrosser | ewwwwwww | 16:08 |
| jrosser | `gluster peer status` | 16:09 |
| jrosser | but i don't an example in the state with a deleted/recreated node just now | 16:09 |
| noonedeadpunk | yeah, it's not really parsable | 16:09 |
| noonedeadpunk | it looks smthj like this: https://paste.openstack.org/show/b6exkK3d7xbTlfwApghn/ | 16:10 |
| noonedeadpunk | `State: Peer Rejected (Connected)` | 16:10 |
| noonedeadpunk | but I was thinking of smth like this: https://paste.openstack.org/show/bD4Xhlsxp6Vfhwnwf8WG/ | 16:11 |
| jrosser | yes we can slurp these | 16:12 |
| noonedeadpunk | #endmeeting | 16:13 |
| opendevmeet | Meeting ended Tue Nov 4 16:13:11 2025 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 16:13 |
| opendevmeet | Minutes: https://meetings.opendev.org/meetings/openstack_ansible_meeting/2025/openstack_ansible_meeting.2025-11-04-15.06.html | 16:13 |
| opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/openstack_ansible_meeting/2025/openstack_ansible_meeting.2025-11-04-15.06.txt | 16:13 |
| opendevmeet | Log: https://meetings.opendev.org/meetings/openstack_ansible_meeting/2025/openstack_ansible_meeting.2025-11-04-15.06.log.html | 16:13 |
| noonedeadpunk | but like mapping of uuid to the hostname is biggest pita right now... | 16:13 |
| jrosser | maybe we should set a local fact as part of the deployment | 16:16 |
| noonedeadpunk | that would help, sure | 16:17 |
| jrosser | as we know the uuid for each node from glusterd.info when it gets created | 16:17 |
| noonedeadpunk | but if we're redeploying the thing... | 16:17 |
| noonedeadpunk | we are loosing the fact | 16:17 |
| jrosser | `this_uuid` `all_uuids` perhaps | 16:18 |
| jrosser | but not sure | 16:18 |
| noonedeadpunk | ah | 16:19 |
| noonedeadpunk | lol | 16:19 |
| noonedeadpunk | I can slurp file from peers | 16:19 |
| noonedeadpunk | https://paste.openstack.org/show/b08e52WF71QCKaK5oMDI/ | 16:20 |
| noonedeadpunk | and pretty much same with bricks https://paste.openstack.org/show/bdKxUJXsfSTiZBOpXuH1/ | 16:21 |
| noonedeadpunk | it's annoying though | 16:21 |
| noonedeadpunk | sounds more like a module :D | 16:21 |
| jrosser | yes i was thinking that | 16:22 |
| jrosser | and was just (disappointedly) looking at the gluster collection | 16:23 |
| jrosser | `gluster_info` or something | 16:23 |
| spatel | Hello noonedeadpunk | 16:41 |
| noonedeadpunk[e] | o/ | 16:42 |
| spatel | :) | 16:42 |
| spatel | after long time | 16:42 |
| spatel | I was super busy in new k8s deployment | 16:43 |
| spatel | how are you guys doing? | 16:43 |
| noonedeadpunk | yeah. been a while indeed :) | 16:48 |
| noonedeadpunk | so how you like helm charts these days ? :p | 16:48 |
| noonedeadpunk | was really a nice summit and ptg | 16:48 |
| noonedeadpunk | but now too many things on plate as well | 16:48 |
| spatel | noonedeadpunk hey! I am learning helm chart | 17:21 |
| spatel | our plan is to move out from openstack and build all containertized app | 17:22 |
| spatel | I am building k8s on bare metal and playing with to see how it goes | 17:22 |
| noonedeadpunk | omfg | 17:23 |
| noonedeadpunk | well. I guess good luck then :) | 17:23 |
| opendevreview | Merged openstack/openstack-ansible master: Update rally SHA https://review.opendev.org/c/openstack/openstack-ansible/+/965308 | 17:29 |
| spatel | lol | 17:44 |
| spatel | noonedeadpunk how is everything your side.. anything new outside openstack? | 17:45 |
| noonedeadpunk | no, not at all... just waaaay more openstack with VMware migrations now... | 17:46 |
| spatel | Hmm nice!! | 17:55 |
| spatel | How is your bgp deployment is in production with ovn? | 17:56 |
| noonedeadpunk | I think will know in a month when first upgrade time will come :D | 17:56 |
| noonedeadpunk | otherwise it works | 17:56 |
| noonedeadpunk | but I'm not sure if load in the region is enough to evaluate how good does it work | 17:57 |
| noonedeadpunk | I still don't like a design tbh | 17:57 |
| noonedeadpunk | but hopefully native bgp in ovn will make things better | 17:57 |
| noonedeadpunk | and it was really nice summit this time in Paris. It was almost twice from Vancouver. | 18:06 |
| noonedeadpunk | So felt really nice, but also super busy over there | 18:06 |
| spatel | my company not approving budget otherwise I would love to see you guys | 18:16 |
| spatel | noonedeadpunk I agreed about ovn-bgp stuff.. its kind of hacky when python script doing stuff | 18:17 |
| spatel | totally agreed with ovn bgp native support | 18:17 |
| spatel | In my company I am trying to deploy ceph with BGP (its over engineering at this point but plan is to do L3 last mile in DC) | 18:18 |
| spatel | going to run FRR in each host to advertise IPs in fabric | 18:19 |
| noonedeadpunk | well, I actually have ceph in L3 in one of deployment | 18:33 |
| noonedeadpunk | though it's OSPF for some reason (don't ask me why) | 18:34 |
| noonedeadpunk | and it was birdc | 18:34 |
| noonedeadpunk | and I got storage completely down for a while, as somehow ospf master was on controllers, and I messed up with prio on controllers during OS upgrade.... | 18:35 |
| mgariepy | that's a good day | 18:39 |
| noonedeadpunk | yeah.... | 18:41 |
| noonedeadpunk | so I'm shutting down a single controller node, and whole region goes black the same minute... | 18:41 |
| mgariepy | yep, from that single time you wont do that again :) | 18:42 |
| opendevreview | Merged openstack/openstack-ansible-os_neutron stable/2024.1: Rename OVN BGP Agent SHA pinning var https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/965144 | 19:16 |
| opendevreview | Merged openstack/openstack-ansible-os_neutron stable/2024.2: Rename OVN BGP Agent SHA pinning var https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/965143 | 19:16 |
| opendevreview | Merged openstack/openstack-ansible master: Add debian trixie job definitions https://review.opendev.org/c/openstack/openstack-ansible/+/954616 | 19:52 |
| opendevreview | Merged openstack/openstack-ansible master: Imported Translations from Zanata https://review.opendev.org/c/openstack/openstack-ansible/+/965438 | 19:52 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-plugins master: Split glusterd tasks to task files https://review.opendev.org/c/openstack/openstack-ansible-plugins/+/966116 | 20:22 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-plugins master: [DNM] Clean-up stale gluster peers https://review.opendev.org/c/openstack/openstack-ansible-plugins/+/966117 | 20:22 |
| opendevreview | Merged openstack/openstack-ansible stable/2024.2: Ensure that gluster mount is started after service https://review.opendev.org/c/openstack/openstack-ansible/+/965211 | 20:23 |
| noonedeadpunk | So I started pulling in together gluster thing: https://review.opendev.org/c/openstack/openstack-ansible-plugins/+/966117 | 20:26 |
| noonedeadpunk | I _think_ this might be enough to do some cleaning-up.... | 20:27 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-plugins master: [DNM] Clean-up stale gluster peers https://review.opendev.org/c/openstack/openstack-ansible-plugins/+/966117 | 20:29 |
| jrosser | blimey | 20:34 |
| *** mnaser[m] is now known as mnaser | 20:46 | |
Generated by irclog2html.py 4.0.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!