| opendevreview | Dmitriy Rabotyagov proposed openstack/project-config master: Add access to Freezer Unmaintained for freezer-stable-maint https://review.opendev.org/c/openstack/project-config/+/974994 | 11:50 |
|---|---|---|
| kozhukalov | Hi team. I have a question regarding ARM nodes. I see that the only provider of ARM64 nodes is the University of Oregon. I am trying to build ARM64 docker images and use those nodes to run the build natively (not using qemu). | 20:49 |
| kozhukalov | I see that when I clone git repos from opendev, for example https://opendev.org/openstack/cinder the download speed is about 20KiB/s which is very slow. When I clone the same repo from the github the speed is about 20Mib/s. What is the best way to mitigate this? Should I use github? | 20:49 |
| kozhukalov | Also some apt mirrors are slow, and I use mirrors.mit.edu. | 20:49 |
| clarkb | kozhukalov: small clarification the arm nodes come from OSUOSL at oregon state univeristy. | 20:57 |
| clarkb | kozhukalov: all of our test nodes have the git repos cached in them so you shouldn't need to cloen from anywhere externally, you just need to update the cached repo to the latest code | 20:58 |
| clarkb | kozhukalov: or do you mean you are running our disk image builder builds locally and see that slowness? If so then again dib maintains a cache and you should only need to incur the cost once. Or you can prepopulate the caches | 20:59 |
| clarkb | The downside to using github is that if there are issues with replication you'll not see the latest updates (which happened recently after the key rotations since the github credentials took a few days to update) | 20:59 |
| kozhukalov | Oh, that's nice. Will try to use cached repos. Thanks for the hint. I appreciate this. | 21:00 |
| kozhukalov | No, this is not dib. I am trying to build docker images for later use with Openstack-Helm charts. | 21:00 |
| clarkb | then yes I would use the cached repos within the test node images as the starting point then you only have to update the minimal delta from there. This is what zuul does for every job automatically for required-projects when populating /home/zuul/src too (but all of the repos are cached just possible not up to date) | 21:01 |
| clarkb | as for apt mirrors we have a mirror in that cloud region as well that the nodes are configured to use so you can port that into your builds as well | 21:01 |
| clarkb | from my local machine I typically get about 2-3MiB/s as reported by git. I just cloned nova and it reports: 3.17 MiB/s. Repeating the same test against github I get: 4.14 MiB/s which is better but not on the same order as you've discovered. I suspect that this points to networking differences more than a functional difference in the opendev.org gitea service | 21:08 |
| clarkb | (both of those are well below my ISP limits so I don't think my local network is throttling things) | 21:08 |
| kozhukalov | Ok, I'll try to use cached repo with git fetch. Perhaps this would be enough to make it work. | 21:15 |
| kozhukalov | Regarding apt mirrors. I tried to use opendev mirror but encountered missing packages for the arm ubuntu noble. Then I decided to use upstream mirrors and focus on other things and make everything work. Later I'll try to switch to opendev again and will let you know where exactly I have the issue. BTW the default ubuntu mirror https://ports.ubuntu.com/ubuntu-ports/ is also slow for OSUOSL nodes, and that is why I am | 21:15 |
| kozhukalov | currently using MIT mirror. | 21:15 |
| fungi | http://mirror.regionone.osuosl.opendev.org/ubuntu-ports/ shouldn't be incomplete as far as arm64 packages are concerned | 21:19 |
| clarkb | it is a diffrent mirror than the x96 content though | 21:19 |
| clarkb | which may be why you saw that if trying to use the x96 mirror | 21:19 |
| clarkb | *x86 | 21:19 |
| fungi | though grafana and the timestamp indicate it's a couple months behind... checking in on that now | 21:22 |
| fungi | "The lock file '/afs/.openstack.org/mirror/ubuntu-ports/db/lockfile' already exists..." i'll clear it and manually update in a root screen session on the mirror-update server | 21:26 |
| fungi | timestamp on that is indeed 2025-11-21 too | 21:27 |
| clarkb | thanks | 21:28 |
| fungi | in progress now | 21:29 |
| fungi | guess this got missed and nobody noticed over the holidays | 21:29 |
| fungi | BDB0689 /afs/.openstack.org/mirror/ubuntu-ports/db/checksums.db page 24553 is on free list with type 5 | 21:31 |
| fungi | BDB0061 PANIC: Invalid argument | 21:31 |
| fungi | Internal error of the underlying BerkeleyDB database: | 21:31 |
| fungi | Within checksums.db subtable pool at put(uniq): BDB0087 DB_RUNRECOVERY: Fatal error, run database recovery | 21:31 |
| fungi | BDB0060 PANIC: fatal region error detected; run recovery | 21:31 |
| fungi | well that's no good | 21:31 |
| clarkb | looks like there is room in the quota for ubuntu-ports so not an out of disk corrupting the db issue | 21:33 |
| fungi | yeah, i'm thinking i'll need to have it completely rebuild the index | 21:39 |
| fungi | fingers crossed it's just checksums.db that needs to be rebuilt, i'll give that a shot | 21:43 |
| kozhukalov | If it helps somehow, here is the example of the broken dependencies (see timestamp 2026-01-28 21:49:29.532789) | 22:17 |
| kozhukalov | https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_c3c/openstack/c3c2904d3a5941689085c367447bd644/job-output.txt | 22:17 |
| clarkb | kozhukalov: note if you use the zuul.opendev.org url you can link to the specific line(s) with the issue | 22:19 |
| kozhukalov | sure, thanks https://zuul.opendev.org/t/openstack/build/c3c2904d3a5941689085c367447bd644/log/job-output.txt#4553 | 22:20 |
| clarkb | kozhukalov: and that log shows node-0 is using x86 mirrors and node-1 is using ubuntu-ports. Is it a mixed arch nodeset so I think that is correct | 22:21 |
| kozhukalov | yes, I deploy buildkit instance with the arm worker | 22:22 |
| clarkb | given that I suspect that fungi's mirror updates should fix things | 22:22 |
| fungi | yeah, this checksum database rebuild will take through the night probably, but i should know something more tomorrow | 22:34 |
Generated by irclog2html.py 4.0.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!