| *** ricolin_ is now known as ricolin | 00:16 | |
| opendevreview | Merged openstack/openstack-zuul-jobs master: Update 2024.1 Caracal related filters https://review.opendev.org/c/openstack/openstack-zuul-jobs/+/969949 | 11:46 |
|---|---|---|
| opendevreview | Henrik Wahlqvist proposed openstack/project-config master: Add back volvocars repos in volvocars tenant https://review.opendev.org/c/openstack/project-config/+/975154 | 13:39 |
| opendevreview | Dr. Jens Harbott proposed openstack/openstack-zuul-jobs master: Add jobs for Python 3.14 https://review.opendev.org/c/openstack/openstack-zuul-jobs/+/968510 | 13:41 |
| *** vhari_ is now known as vhari | 15:00 | |
| stephenfin | We're seeing "No space left on device" errors on the py313 job for oslo.utils, e.g. https://121a82104cc98f0bf47f-8e50817bcbd575f2341914da48a13cbe.ssl.cf5.rackcdn.com/openstack/ec5bc84328074010b9581b6ac670638b/job-output.txt | 18:06 |
| clarkb | stephenfin: any idea where genisoimage is writing? | 18:07 |
| fungi | stephenfin: have the zuul build link for that one handy?\ | 18:07 |
| stephenfin | It seems to be the tests that create fake images that are causing the issue, but idk why we see it on py313 only. I know we use a different distro for those (Debian?) but I'd have imagined the images were otherwise very similar. | 18:07 |
| * stephenfin looks | 18:07 | |
| clarkb | yes the images should be similar. There is also a pair of df outputs at the end of your log report (one for bytes and the other for inodes) | 18:07 |
| clarkb | fungi: https://zuul.opendev.org/t/openstack/build/ec5bc84328074010b9581b6ac670638b | 18:07 |
| stephenfin | clarkb: /tmp | 18:08 |
| * stephenfin gets the permalink | 18:08 | |
| clarkb | `tmpfs 4065544 10940 4054604 1% /tmp` is what the df output says (those are 1k blocks I think() | 18:09 |
| fungi | we log the disk layout and initial free space because it differs between providers | 18:09 |
| stephenfin | https://github.com/openstack/oslo.utils/blob/ea187a68e0b3d04668d0e1bda41365bb662438c8/oslo_utils/tests/imageutils/test_format_inspector.py#L89-L100 | 18:09 |
| clarkb | so thats 4GB on /tmp | 18:10 |
| clarkb | but it is tmpfs not disk | 18:10 |
| clarkb | so you're probably running out of memory actually | 18:10 |
| clarkb | and yes I'm guessing ubuntu /tmp is not tmpfs but instead real disk | 18:10 |
| clarkb | one solution may be to write to real disk in that case | 18:10 |
| stephenfin | Ah, that's a good assumption | 18:10 |
| fungi | yeah, just under 4gb space in /tmp initially on the trixie nodes | 18:11 |
| clarkb | fungi: is that tmps or a proper disk backed fs? | 18:11 |
| clarkb | *tmpfs | 18:11 |
| fungi | tmpfs | 18:11 |
| fungi | so you could umount -remount it | 18:11 |
| fungi | with a different size | 18:11 |
| stephenfin | I don't want to do that in a test though, assuming that's directed to me | 18:12 |
| clarkb | oh right you said trixie for some reason I parsed it as noble | 18:12 |
| fungi | just as long as you have enough swap to back it | 18:12 |
| clarkb | stephenfin: any idea how big the image youre dealing with is (just to rule out something less obvious. Like if it is a 2MB iso image for config drive this would be weird) | 18:12 |
| fungi | right, compare the equivalent of https://zuul.opendev.org/t/openstack/build/ec5bc84328074010b9581b6ac670638b/log/zuul-info/zuul-info.debian-trixie.txt on a working build | 18:13 |
| stephenfin | fwict these are all images generated by the test. The biggest I could find was 1057MB | 18:13 |
| stephenfin | https://review.opendev.org/c/openstack/oslo.utils/+/975179/1/oslo_utils/tests/imageutils/test_format_inspector.py#316 | 18:13 |
| clarkb | my tumbleweed install also has a tmpfs /tmp that is 1/2 the size of my memory | 18:14 |
| fungi | so possible something else is eating up space in /tmp | 18:14 |
| clarkb | 1GB is still pretty large | 18:14 |
| clarkb | that is ~1/8 the total memory on the system | 18:14 |
| stephenfin | yeah, and there's a 512 and 513 image being created in parallel | 18:14 |
| clarkb | stephenfin: fungi: I would use /var/tmp | 18:15 |
| clarkb | it is supposed to survive reboots meaning it isn't a tmpfs but it is still a temporary disk location | 18:15 |
| clarkb | (assuming debian has /var/tmp but I think it is standard) | 18:15 |
| fungi | /var/tmp is just part of / on these instances, looks like | 18:16 |
| fungi | but yes the directory should exist | 18:16 |
| clarkb | and at least on my local system permissiosn are set up that anything should be able to write to it just like /tmp so ya I think this a good consistent way to address the problem | 18:19 |
| stephenfin | okay I've respun https://review.opendev.org/c/openstack/oslo.utils/+/975179 to reduce image size further to see if that helps first | 18:22 |
| stephenfin | and added https://review.opendev.org/c/openstack/oslo.utils/+/975194 as a follow-up in case it doesn't | 18:23 |
| stephenfin | will check those in the morning. thanks for quick input 🙏 | 18:23 |
| fungi | any time! | 18:23 |
| clarkb | you're welcome. Hopefully one or both of those addres the issue | 18:23 |
| opendevreview | Jeremy Stanley proposed openstack/project-config master: Reset Zanata API key https://review.opendev.org/c/openstack/project-config/+/975228 | 21:38 |
| *** efoley_ is now known as efoley | 21:47 | |
| opendevreview | Merged openstack/project-config master: Reset Zanata API key https://review.opendev.org/c/openstack/project-config/+/975228 | 21:47 |
Generated by irclog2html.py 4.0.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!