| opendevreview | OpenStack Proposal Bot proposed openstack/openstack-ansible master: Imported Translations from Zanata https://review.opendev.org/c/openstack/openstack-ansible/+/967794 | 03:12 |
|---|---|---|
| opendevreview | Merged openstack/openstack-ansible master: Imported Translations from Zanata https://review.opendev.org/c/openstack/openstack-ansible/+/967794 | 09:43 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-rabbitmq_server master: Install erlang for trixie from distro packages https://review.opendev.org/c/openstack/openstack-ansible-rabbitmq_server/+/967689 | 09:43 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-rabbitmq_server master: Use correct variable for erlang repo defenition https://review.opendev.org/c/openstack/openstack-ansible-rabbitmq_server/+/968172 | 09:49 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-rabbitmq_server master: Make upgrade jobs voting https://review.opendev.org/c/openstack/openstack-ansible-rabbitmq_server/+/968173 | 09:51 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible stable/2024.1: Bump rabbitmq SHA and stop using infra mirror for mariadb https://review.opendev.org/c/openstack/openstack-ansible/+/967864 | 11:03 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: Use uv for python packages install https://review.opendev.org/c/openstack/openstack-ansible/+/966409 | 11:08 |
| opendevreview | Andrew Bonney proposed openstack/openstack-ansible master: Fix indentation for aio networking-post-up service https://review.opendev.org/c/openstack/openstack-ansible/+/967986 | 12:16 |
| jrosser | are we in a circular dependancy with this https://review.opendev.org/c/openstack/ansible-role-python_venv_build/+/966404 | 13:01 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-galera_server stable/2024.1: Switch MariaDB to archive.mariadb.org https://review.opendev.org/c/openstack/openstack-ansible-galera_server/+/968190 | 13:13 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-galera_server stable/2024.1: Switch MariaDB to archive.mariadb.org https://review.opendev.org/c/openstack/openstack-ansible-galera_server/+/968190 | 13:50 |
| noonedeadpunk | we are | 13:50 |
| noonedeadpunk | or well. we are with upgrade jobs at least | 13:50 |
| noonedeadpunk | and then 2024.1 seems to have issue not only with rabbitmq but also mariadb repos | 13:51 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-rabbitmq_server stable/2024.2: Ensure no CQ mirroring policies applied https://review.opendev.org/c/openstack/openstack-ansible-rabbitmq_server/+/968194 | 13:55 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-galera_server stable/2024.1: Switch MariaDB to archive.mariadb.org https://review.opendev.org/c/openstack/openstack-ansible-galera_server/+/968190 | 14:41 |
| jrosser | by circular i think i meant that this tries to fix trixe for rabbitmq https://review.opendev.org/c/openstack/openstack-ansible-rabbitmq_server/+/967689 but depends on https://review.opendev.org/c/openstack/ansible-role-python_venv_build/+/966404, which itself fails on rabbitmq for trixie | 14:54 |
| jrosser | and fails not in an upgrade job | 14:55 |
| noonedeadpunk | doh, right..... | 14:55 |
| noonedeadpunk | ok, so I guess I will leave jobs as NV for rabbitMQ? | 14:56 |
| jrosser | i think we have to do that somewhere yes | 14:56 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-rabbitmq_server master: Install erlang for trixie from distro packages https://review.opendev.org/c/openstack/openstack-ansible-rabbitmq_server/+/967689 | 14:57 |
| noonedeadpunk | it's just sooo many conflicts now, that I got lost a little bit | 14:57 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-rabbitmq_server master: Make upgrade jobs voting https://review.opendev.org/c/openstack/openstack-ansible-rabbitmq_server/+/968173 | 14:57 |
| jrosser | in fact yes this one is blocking master completely | 14:57 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-rabbitmq_server master: Install erlang for trixie from distro packages https://review.opendev.org/c/openstack/openstack-ansible-rabbitmq_server/+/967689 | 14:58 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-rabbitmq_server master: Make upgrade jobs voting https://review.opendev.org/c/openstack/openstack-ansible-rabbitmq_server/+/968173 | 14:58 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-rabbitmq_server master: Make jobs voting again https://review.opendev.org/c/openstack/openstack-ansible-rabbitmq_server/+/968173 | 14:58 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-rabbitmq_server master: Make jobs voting again https://review.opendev.org/c/openstack/openstack-ansible-rabbitmq_server/+/968173 | 14:59 |
| jrosser | it would be so nice to be able to put voting: false on a project template | 14:59 |
| noonedeadpunk | I have no idea if you can do that | 15:00 |
| noonedeadpunk | maybe you can? | 15:00 |
| jrosser | do i miss where you make the jobs nv? | 15:02 |
| noonedeadpunk | it was here https://review.opendev.org/c/openstack/openstack-ansible-rabbitmq_server/+/967523/5/zuul.d/project.yaml | 15:03 |
| noonedeadpunk | it's already merged | 15:03 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-rabbitmq_server master: Use correct variable for erlang repo defenition https://review.opendev.org/c/openstack/openstack-ansible-rabbitmq_server/+/968172 | 15:04 |
| jrosser | ah ok | 15:06 |
| jrosser | you are right so many moving parts / broken things just now | 15:07 |
| jrosser | noonedeadpunk: btw do you think that this ever worked? https://github.com/openstack/openstack-ansible/blob/master/scripts/log-collect.sh#L171 | 15:12 |
| jrosser | not really finding when /var/www/repo was bind mounted to /openstack/repo in the history | 15:13 |
| noonedeadpunk | oh, it was before glusterfs I guess? | 15:13 |
| jrosser | i wonder if there is a risk it doesnt work with a bind mount | 15:14 |
| jrosser | becasue you'd have that made early in the container setup and then the glusterfs mount would be on top of that | 15:14 |
| noonedeadpunk | yeah, gluster being on top would make it not working for sure | 15:15 |
| noonedeadpunk | but with lsyncd I'd expect be working | 15:15 |
| noonedeadpunk | it would be useful to get this info though... | 15:15 |
| noonedeadpunk | but it's totally better to do that from the container | 15:16 |
| jrosser | yes its probably simpler to adjust that to instead use store_lxc_artifacts instead of try to make the bind mount work | 15:19 |
| noonedeadpunk | but if bind mount is working - then gluster is not | 15:20 |
| noonedeadpunk | so we can kinda just disable gluster setup for lxc and leave it only for lxc_infra... | 15:20 |
| jrosser | maybe - lets not make more complexity :) | 15:21 |
| noonedeadpunk | yeah | 15:26 |
| noonedeadpunk | just saying that changing order of mounts make another issue :) | 15:26 |
| noonedeadpunk | I am waiting for https://review.opendev.org/c/openstack/openstack-ansible-galera_server/+/968190 to pass, to remove dependency and CI tests for it... | 15:27 |
| noonedeadpunk | as it's another circular thing.... | 15:27 |
| noonedeadpunk | ok, so it's passing | 16:09 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-galera_server stable/2024.1: Switch MariaDB to archive.mariadb.org https://review.opendev.org/c/openstack/openstack-ansible-galera_server/+/968190 | 16:09 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-galera_server stable/2024.1: Re-enable CI jobs https://review.opendev.org/c/openstack/openstack-ansible-galera_server/+/968212 | 16:11 |
| noonedeadpunk | Would be nice to quick land this one | 16:11 |
| opendevreview | Merged openstack/openstack-ansible-galera_server stable/2024.1: Switch MariaDB to archive.mariadb.org https://review.opendev.org/c/openstack/openstack-ansible-galera_server/+/968190 | 17:23 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible stable/2024.1: Bump rabbitmq SHA and stop using infra mirror for mariadb https://review.opendev.org/c/openstack/openstack-ansible/+/967864 | 17:24 |
| jrosser | we are in "no meaningful testing" situation again here https://review.opendev.org/c/openstack/openstack-ansible-galera_server/+/968190 | 17:30 |
| jrosser | ah ok i see /o\ | 17:32 |
| noonedeadpunk | yeah, it was intended... | 17:34 |
| noonedeadpunk | but previous patchset was passing | 17:34 |
| noonedeadpunk | so I believe it should be fine | 17:34 |
| jrosser | which to keep an eye on next? | 17:44 |
| noonedeadpunk | so for fixing upgrade jobs - https://review.opendev.org/c/openstack/openstack-ansible/+/967864 (and https://review.opendev.org/c/openstack/openstack-ansible-galera_server/+/968212 after it) | 17:46 |
| noonedeadpunk | for master in general - https://review.opendev.org/c/openstack/openstack-ansible-rabbitmq_server/+/967689 and then recheck https://review.opendev.org/c/openstack/ansible-role-python_venv_build/+/966404 | 17:46 |
| spatel | noonedeadpunk hey! | 18:43 |
| spatel | I have deployed ceph on my BGP L3 datacenter fabric | 18:44 |
| noonedeadpunk[e] | o/ | 18:44 |
| mnaser | make sure no one restarts frr on an osd node :) | 18:46 |
| noonedeadpunk[e] | Hehe, indeed | 18:46 |
| noonedeadpunk[e] | Well, I guess you can add ceph osd set noout as pre-stop command to frr unit | 18:48 |
| spatel | mnaser ofc not :) | 18:48 |
| noonedeadpunk[e] | But kinda nasty given that flag is global and likely you're not using systemd anyway | 18:48 |
| spatel | mnaser Now we don't need to deal with LACP and L2 debugging :) | 18:49 |
| noonedeadpunk[e] | You have bigger things to worry about, but yeah | 18:49 |
| spatel | We had massive issue when we upgrade switches and found ARP not doing what it should do and entire infra went down | 18:49 |
| noonedeadpunk[e] | Yeah, this is indeed is solved nicely with l3 | 18:50 |
| spatel | We decided to say goodby to ARP | 18:50 |
| noonedeadpunk[e] | Not sure what you mean in case of ceph though, as you still have ARP for peering | 18:51 |
| spatel | I can give frr daemon to init to keep it alive or tell systemctl to lock down frr and not allow to restart | 18:51 |
| spatel | noonedeadpunk[e] I have peer IPv6 and ofc ARP will be there but it will stay on that link not impact entire DC | 18:52 |
| spatel | anyway there is no such thing call solved all problem :) but it will make life easier and traffic engineering... | 18:53 |
| noonedeadpunk[e] | Well, ok, NDP | 18:54 |
| noonedeadpunk[e] | Yeah, sure, I think it's nice, but for us ceph is not biggest contributor to the issue, as it was anyway having l2 only inside of the rack | 18:55 |
| spatel | We have 100 nodes in ceph across multiple racks | 18:55 |
| noonedeadpunk[e] | So it wasn't going further then a leaf | 18:55 |
| spatel | our workload is different :) | 18:55 |
| noonedeadpunk[e] | Except one setup with ospf I said about one day😅 | 18:56 |
| spatel | soon I am planning to play with RDMA network to handle more ceph IOPS and latency issue | 18:57 |
| jrosser | network is never normally the limiting factor for ceph | 18:58 |
| jrosser | unless you have stellar CPU and NVMe already | 18:58 |
| spatel | our workload is crazy which required engineering on network to get juice out of it... | 18:59 |
| spatel | Will share my experience when I go to RDMA base network for ceph and see how it goes | 19:00 |
| jrosser | general opinion seems to be there is no tangible benefit to rdma/roce for ceph until crimson OSD backend becomes default | 19:04 |
| jrosser | and even then you probably will be in a very small club trying it out | 19:04 |
| jrosser | current OSD is basically single threaded which makes it hard to take advantage of any aysnchronous behaviour in the network | 19:05 |
| jrosser | spatel: btw your leaf / spine diagram - you can put all the spines in the same AS | 19:09 |
| spatel | I put AS on different to avoid any loop + if spine on different AS then I can do traffic engineering to route my packet to specific spine. I will have more control in different AS | 19:10 |
| spatel | again spine are just 1 or 2 so not worry to have them on different AS | 19:11 |
| spatel | Please give "clap" if you like the article :) | 19:12 |
Generated by irclog2html.py 4.0.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!