Wednesday, 2025-09-10

opendevreviewOpenStack Proposal Bot proposed openstack/project-config master: Normalize projects.yaml  https://review.opendev.org/c/openstack/project-config/+/95799502:32
opendevreviewMichael Still proposed openstack/diskimage-builder master: Automatically accept "dnf mark" commands.  https://review.opendev.org/c/openstack/diskimage-builder/+/96032903:33
opendevreviewMichal Nasiadka proposed openstack/diskimage-builder master: almalinux-container: Add support for building 10  https://review.opendev.org/c/openstack/diskimage-builder/+/96033607:01
opendevreviewMerged openstack/diskimage-builder master: Automatically accept "dnf mark" commands.  https://review.opendev.org/c/openstack/diskimage-builder/+/96032908:03
opendevreviewMichal Nasiadka proposed openstack/diskimage-builder master: almalinux-container: Add support for building 10  https://review.opendev.org/c/openstack/diskimage-builder/+/96033609:30
opendevreviewMerged zuul/zuul-jobs master: Remove deprecated javascript jobs  https://review.opendev.org/c/zuul/zuul-jobs/+/92106713:12
opendevreviewMerged zuul/zuul-jobs master: Remove version defaults for nodejs jobs  https://review.opendev.org/c/zuul/zuul-jobs/+/95721913:18
opendevreviewMichal Nasiadka proposed openstack/diskimage-builder master: almalinux-container: Add support for building 10  https://review.opendev.org/c/openstack/diskimage-builder/+/96033613:24
fungiclarkb: i think what happened is that the upgrade from focal to jammy disabled the ppa but didn't remove the packages we had installed from it, then the upgrade from jammy to noble had some conflicting dependencies and uninstalled most of them (all but the dkms package)13:40
fungiso on the other servers we're running the focal builds of our afs packages on jammy right now, but since it seems to be working (and they're all the same afs source version anyway) i'm not inclined to disrupt things further by upgrading them again before the upgrade to noble13:41
fungias for lists.o.o, sounds like we're getting somewhere and database profiling/tuning will be next on the agenda13:42
fungiback on afs, as expected the docs volume move from ord to dfw is still in progress13:45
opendevreviewMichal Nasiadka proposed openstack/diskimage-builder master: Remove nodepool based testing  https://review.opendev.org/c/openstack/diskimage-builder/+/95295314:08
clarkbfungi: weird re the package selection choices15:29
fungii guess it doesn't seem all that weird to me, living most of my life on debian/unstable and wrangling those sorts of inconsistencies15:30
clarkbit must've rerun the dkms build against the new kernel though?15:30
clarkbbut ya since the afs versions doesn't change I don't think there is any real reason to upgrade the package at that point.15:30
fungiyes, that part was still working15:30
fungii mean, in some ways it's a testament to shared library symbol management that you can upgrade 2 years worth of packages and others continue to work with them just fine15:31
clarkbI have a meeting in 1.5 hours but afterwards we might want ot sync up on the mariadb for lists istuation if you have time15:35
clarkbI think my first step there is still to benchmark the disk io using something like fio just to see if there is something horribly wrong or potentailly solvable via an ssd volume15:36
fungiyeah, i think i'm going to disappear briefly for lunch, but will be around15:36
opendevreviewClark Boylan proposed opendev/system-config master: Expand old chrome UA rules  https://review.opendev.org/c/opendev/system-config/+/96039915:49
fungidocs afs volume move is still going16:48
clarkbthis is across the internet and not within the same region which explains some of the slowness16:48
fungiright, also one of our larger and most active wrt updates16:49
*** dhill is now known as Guest2627616:57
clarkbfungi: I'm around now if we want to look at lists17:39
clarkb(I'm also happy to just dive in myself, not sure if you are interested in following along17:39
clarkbI probably need to read the fio manpage first17:40
clarkboracle cloud (https://docs.oracle.com/en-us/iaas/Content/Block/References/samplefiocommandslinux.htm) suggests something like: `fio --direct=1 --ioengine=libaio --bs=4k --iodepth=256 --runtime=120 --eta-newline=1 --numjobs=4 --time_based --group_reporting --rw=read --size=2G --name=clarkb-test` to test sequential read speeds for database use cases. I have this runnign on my local18:00
clarkbmachine first. Fio has the potential to impact the running server so want to be careful18:00
clarkbthat reports `read: IOPS=532k, BW=2080MiB/s (2181MB/s)(244GiB/120001msec)` for my local nvme device18:02
clarkbmaybe we want to start with something smaller on lists though. Like 5 seconds and 250MB or so?18:05
clarkbnote that writes numjobs files to the local directory of size size to then start performing the reads against18:05
clarkbit does not appear to delete them when done18:05
clarkbthoughts on the safety of such a thing while lists is running (and possibly busy / sad) ?18:06
fungii mean, it can't get much more sad that it already is on a regular basis, short of being entirely offline18:07
fungii think the e-mail receipt/processing/forwarding isn't dependent on the database at all, other than to possibly do subscriber lookups for filtering18:08
clarkbfungi: ya but the whole system shares one device so potentially if we load it up more things can get sad?18:09
fungifio looks like it's just focused on block device performance though18:09
funginot anything database-specific18:10
clarkbits a generic io tesing tool. The idea is you configure it to mimic a use case. In this case databases should be sequential read heavy18:10
clarkbso we configure it to test sequential reads18:10
fungii guess you're wanting to see if that device is sad before we dig into database performance problems?18:10
clarkbyes18:11
clarkbthere isn't much point in trying to tune mariadb if the disk is always going to be too slow18:11
clarkbmy hunch here is that switchign to an ssd based volume for mariadb will help a lot18:11
fungiseems reasonable, sure18:11
clarkbbut I'm hoping to actually measure some of that before hand. But I also don't want to break the system sending a bunch of extra load at it18:11
fungiit looks less heavily-loaded than usual at the moment18:12
fungiseems like as good a time as any18:12
clarkbI have installed fio on lists18:15
clarkbI'm setting runtime to 3 seconds and file sizes to 200M and I'm doing it in ~clarkb/fio_test18:15
clarkb`read: IOPS=28.8k, BW=113MiB/s (118MB/s)(344MiB/3054msec)` is the result of that test18:17
clarkbdmesg -T doesn't show any complaints. I guess I'll do a longer test now 30 seconds?18:17
fungisounds great18:18
fungidocs afs volume move finished in 19h40m18:19
clarkb`read: IOPS=30.5k, BW=119MiB/s (125MB/s)(3584MiB/30033msec)` that seems faily consistent over short periods of time18:19
clarkb(still much less zoomy than my nvme drive, but not sure if that is bad numbers of mariadb)18:19
fungidid a vos release on docs now just for safety, but vos listvldb shows no rw volumes on afs01.ord any longer, and the expected ones on afs01.dfw18:22
fungii'm going to get the larger batch of afs02.dfw to afs01.dfw moves going now, and will then look at further upgrades to noble while that's underway18:23
clarkbfwiw I don't think those fio numbers point at anything drastically wrong. I would expect that to be plenty of bw for the amount of total transfer that iotop reported for mariadb18:23
fungithough obviously won't upgrade afs02.dfw until the rw moves off of it are done18:23
clarkbmariadb did a total read amount of like 10G over several minutes yesterday. Which seems like well below 120M/s18:24
fungiclarkb: so we probably shouldn't expect a huge performance boost from moving the db to a dedicated ssd-backed cinder volume, is what i'm hearing?18:24
clarkbfungi: thats kind of what I'm thinking based on ^18:24
clarkbunless there is noisy neighbor problems or something like that hitting us at times that are not now18:24
fungiright, i'm starting to wonder if right now things are fine, and our i/o bandwidth ends up horribly oversubscribed at other times18:24
clarkbya maybe we need to rerun this fio command when we notice slowness?18:25
fungithough trying to load the moderation queue for openstack-discuss just took my browser about 80 seconds wall clock time18:26
fungiafter that, deleting a message from the queue only took roughly 5 seconds though18:27
opendevreviewMichal Nasiadka proposed openstack/diskimage-builder master: almalinux-container: Add support for building 10  https://review.opendev.org/c/openstack/diskimage-builder/+/96033618:27
fungifeels like disk access vs resident memory speed differences18:27
clarkbor maybe we're filling up some queues (in apache, uwsgi, or mariadb) and you have to wait to get your spot in the line?18:28
clarkbI just reran my test and got `read: IOPS=24.9k, BW=97.2MiB/s (102MB/s)(2924MiB/30095msec)` so worse but not terribly so18:28
fungialso system load has climbed to 5, where it was 1.something just a few minutes ago18:28
fungiand now it's started dropping again18:28
fungiooh, we probably haven't looked at apache stats18:29
fungigood call, the scorecard might have some insights18:29
clarkbfungi: I think our uwsgi config is configured with 2 processes and 2 threads. That seems maybe low18:30
clarkbwe have 4vcpu there and if we're hitting a system load of 5 that seems liek we're probably loading things up pretty well already. We could try say 4 processes and double the total number of threads?18:32
clarkbinternet suggests 2 * cpu for process count18:34
clarkbso ya maybe bump that to 4 and see if it helps. If nothing bad happens bump all the way to 8?18:34
fungisounds great to me, worth a try for sure18:37
opendevreviewClark Boylan proposed opendev/system-config master: Double the total mailman3 uwsgi processes to 4  https://review.opendev.org/c/opendev/system-config/+/96041218:40
clarkbfungi: so I notice that ^ we're setting the value as a hardcoded value in the container itself. We could bind mount that file in and make the rtt on these updates a bit more efficient. Do you think we should do that now too?18:41
fungimaybe worth checking how much our copy of that file deviates from the one in maxking's repo18:43
fungii can't remember if we've edited it at all up to now18:43
fungibut yeah, it's configuration and if we're changing it then we're probably better off mounting our copy over it18:44
clarkbthere is no difference before my change18:44
funginot bothering with container rebuilds just to alter a setting18:44
clarkbok let me start on a followup change to bind mount that over instead18:45
fungithen that would explain why we didn't mount in our own copy previously. sounds like a great idea18:45
clarkber not a follow change. I'll do a second patchset18:46
fungiremember to include a breadcrumb comment at the top indicating the upstream repo commit it's copied from and what we altered18:46
fungii think there are a few others where i did that18:46
fungimakes it easier for when we update later18:46
fungiwe're probably do for another mailman update soon, but i know they've got a release pending that will add python 3.13 support so worth waiting for that i think18:47
fungis/ do / due /18:47
fungiafs moves from afs02.dfw to afs01.dfw are in progress. it skipped debian because the volume was already locked by another writer but i'll come back around to get that and any others that don't move in the first attempt18:50
fungiworking on upgrading afs01.ord to noble next... i'll adjust /etc/network/interfaces and /etc/default/grub similar to afs01.dfw, then post-upgrade i'll add our openafs ppa and reinstall the missing packages from it. i *think* that's all the gotchas?18:52
opendevreviewClark Boylan proposed opendev/system-config master: Double the total mailman3 uwsgi processes to 4  https://review.opendev.org/c/opendev/system-config/+/96041218:52
clarkbfungi: those three items are all that I recall from afs01.dfw.18:53
clarkbwas the docs volume the only RW volume on afs01.ord?18:53
fungino, it was merely the last one i moved18:53
fungithere were 5 others but they were much smaller18:53
fungiproject.airship was the next largest volume, for reference18:54
fungii saved docs for last because i knew it would take a while and i wanted it to happen in parallel with my meatspace low-power standby recharge18:54
opendevreviewClark Boylan proposed opendev/system-config master: Double the total mailman3 uwsgi processes to 4  https://review.opendev.org/c/opendev/system-config/+/96041218:55
clarkbgot it18:55
fungii stuck the list i've been working from in a paste, lemme see if i can find it in scrollback real quick18:56
fungiclarkb: https://paste.opendev.org/show/bWxZSt3nyaG8bxa3yW54/18:57
fungithat larger list at the top is the one i'm working from, though i hand-reordered it a bit to insert some smaller breather between the larger volumes in hopes that they'll vos release normally and avoid us running out of space again18:58
fungibut now that i'm fairly sure a vos release is what frees up the additional usage, it's not a big deal to address19:00
clarkback19:06
fungiit's working on mirror.centos-stream at the moment, which will likely continue until well after i've checked out for the evening, but i'll check in on it again when i wake up and see how far it's gotten19:15
fungii'm expecting it'll be the weekend at best before they're all finished though19:16
fungibut i should be able to get the rest of the servers upgraded in the meantime and then do afs02.dfw last19:20
clarkbfungi: we didn't delete the openeuler content yet did we?19:39
funginot yet, no19:39
clarkbshould we make sure that happens to speed things up?19:39
clarkbhttps://review.opendev.org/c/opendev/system-config/+/960412 just passed and you already +2'd it. Should we go ahead and land that? I doubt anyone will review it until tomorrow19:39
fungiat this point it probably doesn't matter, we can worry about additional cleanup once the upgrades are in the rear-view mirror19:39
clarkbwfm19:40
fungiit would speed up the moves by maybe 6-ish hours, but over the whole it's not that important19:41
fungithere's other stuff we could clean up too, i'm sure19:41
opendevreviewMerged opendev/system-config master: Double the total mailman3 uwsgi processes to 4  https://review.opendev.org/c/opendev/system-config/+/96041220:17
clarkbok looks like ^ should be deployed now20:23
clarkbI see processes restarted20:23
clarkbstill waiting for things to finish restarting and be available though20:23
clarkbfungi: any idea how long it typically takes ?20:25
clarkbok I get a site now but says something went wrong please start mailman-core20:27
clarkbits doing a chown -R /opt/mailman/var in the mailman-core container I think20:28
clarkbok thats the directory with all the caches and stuff. So it may be slow getting to all the files?20:29
clarkbI see high iowait now20:30
clarkbthe chown has been running for about 6 minutes now. This has me wondering if its back to general disk io problems given this behavior20:31
clarkbin theory with things not fully started up we're not handling any requests from elsewhere unless one of hyperkitty vs postorius is up and is getting hit and that is slow and slowing down other io?20:32
clarkbstrace shows the chown is chowning things so I don't think we've hit some bug or problem. Its just taking a while to run the command against all the files. I will attempt to practice patience20:33
clarkbthe expected uwsgi worker count of 4 is present so that is good20:33
clarkbfwiw I think that this behavior is pushing me back to "the disk is really slow"20:35
clarkband maybe the problem is writes not reads? or random not sequential?20:35
clarkbso I think I may be back to trying to quantify that again once things settle20:36
clarkbok things did eventually come up after teh chown completed20:39
clarkbdurign the chown there was high iowait. The chowns aren't going to be super heavy on the throughput but are potentially iops limited and random not sequential20:39
clarkbthese might be good clues for what we should be trying to test with fio20:40
clarkbthings aren't super speed right now but aren't super slow either20:42
fungiclarkb: oh sorry, was afk for a few but yeah, 10 minutes isn't unusual20:43
clarkbfungi: I think it was closer to 15 but that is good to know20:43
clarkblooks like the entrypoint script for mailman-core does a chown -R of the entire mailman var dir20:43
clarkbwhich includes a bunch of files from caches to messages20:44
clarkband while that was happening iowait was very high so maybe my fio test to emulate a database is looking at the wrong profile of io20:44
fungii guess they assume that will normally be very fast20:44
fungii mean, i normally expect a recursive chown or chmod to be very fast even when processing lots of files20:45
clarkbyes I would too. Its like getdirents in a loop and then a chown call20:46
clarkbnot significant amounts of data but lots of potentially random access?20:46
clarkblet me see about doing a random read test using fio going20:47
clarkbs/rw=read/rw=randread/ on my local machine produces better results than sequential reads. Yay nvme I guess20:48
clarkbactually thats more about solid state than the io itnerface, but still zoom zoom20:48
clarkbfungi: `read: IOPS=984, BW=3937KiB/s (4031kB/s)(116MiB/30099msec)` on lists running the same command as before but using the randread method20:50
clarkbI think this is the smoking gun20:50
clarkbless than 1k ops and less than 4MiB/s 20:50
clarkbthat is sloooooowwwww20:50
fungiah yeah20:51
clarkbI need to do a school run shortly. But maybe the next step is running that same randread test on other disks in the same provider to get more of a baseline. eg is using the root disk always this bad or is this host sad20:51
clarkb`fio --direct=1 --ioengine=libaio --bs=4k --iodepth=256 --runtime=30 --eta-newline=1 --numjobs=4 --time_based --group_reporting --rw=randread --size=200M --name=clarkb-test` this is the command if anyone else wants to try it20:51
clarkbfio will write files to the directory you are currently in of size 200MB and it won't delete them so just keep that in mind20:52
clarkbyou don't need root/sudo20:52
fungimight also make sense to do it on the lists server with mailman/mariadb processes stopped20:52
clarkbya and maybe take several samples over time to see if it is persistent20:52
clarkbnot persistent could point at a noisy neighbor etc20:53
fungiit does seem likely, and would explain why we see slow responses sometimes but not others20:53
clarkbI can take another sample when I get back from the school run20:54
clarkband then look at measuring some other nodes. If you have ideas for other locatiosn to test let me know.20:54
clarkb`read: IOPS=687k, BW=2682MiB/s (2812MB/s)(26.2GiB/10002msec)` is what I got on my local disk over 10 seoncds (not 3020:55
clarkbjust to put it in comparison20:55
fungimirror.dfw.rax maybe? its rootfs isn't used for much anything, all the i/o is on a separate block device20:56
clarkb++20:56
clarkbI think apache logs do go to the root disk but yes the majority of io is on dedicated devices20:56
fungiah, yeah /var/log is on the rootfs but that's about it20:57
clarkbok I'm off back soon21:02
fungihave fun storming the castle!21:03
fungipreparatory ifupdown and grub configs rebooted fine on afs01.ord in jammy, do-release-upgrade is running there now21:04
fungihopefully this one will come back up on its own afterward, and also have the correct cpu count21:05
fungithen i'll get our openafs ppa readded and those packages reinstalled21:06
clarkbjust got back new run data for randread: `read: IOPS=869, BW=3477KiB/s (3561kB/s)(102MiB/30102msec)` and read (sequential): `read: IOPS=22.0k, BW=85.8MiB/s (90.0MB/s)(2579MiB/30052msec)`21:42
clarkbnow to install the tool on the mirror node and test there21:42
fungiafs01.ord did come back up on noble with no additional work, and lists 8 processors21:43
clarkbexcellent21:46
clarkbon mirror.dfw randread: `read: IOPS=20.4k, BW=79.8MiB/s (83.7MB/s)(2395MiB/30004msec)` read (sequential): `read: IOPS=140k, BW=548MiB/s (575MB/s)(16.1GiB/30004msec)`21:47
clarkbit is possible that other system activity accounts for the difference however I suspect something else is going on21:47
clarkbthe difference in what is otherwise expected to be very similar hardware is quite large21:48
clarkbso maybe we need to start thinking about picking a time to shut down services and measure without as much noise as we can manage and if things are still sad then consider a volume or replacing the host or askign the cloud if there is something wrong?21:49
fungiwe could also consider moving it to a larger flavor and upgrading to noble21:53
clarkbyes, but if that doesn't change the root device location then it probably won't help21:54
fungion a new server i mean, not in-place21:54
clarkbya we could go through the fun of a new ip again. It wasn't that painful toher than windriver subscribing to that one list causing problems21:54
clarkbI do wonder if this is something the cloud would be interested in simply because it might indicate a problem they should address21:55
fungithough a resize in rackspace would reschedule it to a new host anyway, the downtime is unpredictably long21:55
clarkband with the very large root disk on this node probably longer than usual21:55
clarkb(also thats another difference with this node we picked one iwth a large root device to match the old server iirc)21:56
fungiyeah21:56
clarkbbut I think I'm reasonably convinced now that something is up with general disk io particularly with random access21:56
clarkband then that seems to impact our ability to serve content at a reliable response time21:56
fungiokay, the missing openafs packages have been reinstalled from our ppa and afs01.ord rebooted again. tomorrow i'll try to get the afsdb and kdc servers upgraded similarly22:02
fungithen afs02.dfw will happen once all the rw volume moves are done22:02
fungiwhich will probably be early next week22:02
clarkbsounds great22:03
clarkbfor lists any thoughts on when might be a good time to take some no services running samples22:04
fungiany time really, but friday might be lower-demand22:04
fungihowever, predicting the noisy neighbor bursts may prove challenging, if that's what's going on22:05
clarkbok lets plan for friday then and based on what we learn maybe we file a ticket to see what rax has to say about it but also maybe start planning a replacement or volume attachment22:05
clarkbya though so far my small sample set over a couple of hours seems to be pretty consistent22:05
clarkb`read: IOPS=1022, BW=4091KiB/s (4189kB/s)(120MiB/30120msec)` from just now22:06
fungimoving it to a cinder volume would be the easiest workaround, and shouldn't require much actual downtime22:06
clarkbright we can in theory rsync most of the data to reduce the delta then shut things down and rsync again then start stuff up minimizing the downtime22:06
clarkbthat doesn't help things that would stay on the root disk though but maybe that is good enough22:07
fungiarchive files and the database mainly22:13
clarkbvery hand wavy so don't take too seriously: 15k spinning disks have a random read iops limit of ~200 iops. So either we're on an array of disks and our data is distributed amongst them or we're on ssds22:18
clarkband if we are on ssds I believe that read performance can degrade over time as the drive consumes its extra blocks and has to distribute things less efficiently?22:19
clarkbso maybe this is a signal that we're on ssds that need some care and feedign?22:19
fungineeds a trim ;)22:21
fungipun intended22:21
clarkbit wouldn't surprise me22:21
clarkbif that is legitimately the solution here22:21

Generated by irclog2html.py 4.0.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!