| JayF | Hmm. I think I've found a bug wherein svc-map's package-install.yaml, it's using dib_python_version, and the logs imply it's not functional | 00:03 |
|---|---|---|
| JayF | https://www.irccloud.com/pastebin/GoFH2XRs/ | 00:03 |
| JayF | and then, the build later fails with pyyaml not being installed | 00:03 |
| clarkb | JayF: I probably need more words to understand ^ | 15:56 |
| clarkb | I don't know what dib_python_version has to do with it in particular. The pyyaml install is a in the image using hte distro package manager | 15:56 |
| JayF | https://opendev.org/openstack/diskimage-builder/src/branch/master/diskimage_builder/elements/svc-map/package-installs.yaml#L6 | 15:56 |
| JayF | those "dib_python_version" stanzas basically act as "if" statements | 15:57 |
| clarkb | dib_python_version is internal to dib to determien if it is running under python2 or python3 (somethign that can probably be removed at this point) iirc | 15:57 |
| JayF | and, at least in my gentoo example, are never getting installed | 15:57 |
| JayF | if I remove the dib_python_version (and the python 2 implementation) it works as expected | 15:57 |
| clarkb | oh I see hte pkg listings in svc map are python2 only | 15:57 |
| clarkb | oh no it has both? | 15:58 |
| JayF | Yeah, and neither get installed | 15:58 |
| JayF | looking at the code, I'm 90% sure there's a logic error in https://opendev.org/openstack/diskimage-builder/src/branch/master/diskimage_builder/elements/package-installs/bin/package-installs-squash#L170 | 15:58 |
| clarkb | the easy solution would be to flatten to python3 only since dib doesn't do python2 anymore | 15:58 |
| JayF | we don't support *building images that contain* python 2 anymore? | 15:59 |
| clarkb | but also you can run that code to generate the package list from the map files without running all of dib | 15:59 |
| clarkb | JayF: we do, but dib_python_version is the version dib is running under | 15:59 |
| JayF | so it's actually double-useless | 15:59 |
| JayF | because the code that uses pyyaml runs in-system | 15:59 |
| JayF | that's why we have to do package-installs | 15:59 |
| JayF | SGTM, I'm happy to remove the py2 support, although I do think there's a general bug in there somewhere ... I have a stack of things to fix so I'm happy to punt on the more complex solution :) | 16:00 |
| clarkb | I don't think its double useless | 16:00 |
| clarkb | its just that dib is python3 capable and all of the things dib builds in haev python3 | 16:00 |
| clarkb | so we don't need to distinguish between python2 vs pytho3n anymore for dib itself | 16:00 |
| JayF | well what I'm saying is, we're saying "if we're running DIB on the *buildhost* under python 2.x, clearly we are building an image that uses python2.x" | 16:01 |
| JayF | that's how the logic is written right now | 16:01 |
| JayF | if dib_python_version is being used to decide what version of a library to install /inside the image/ | 16:01 |
| JayF | (which is what's happening) | 16:01 |
| clarkb | JayF: dib_python_version determiens the python version to use within the image build for dib things alone. It gets removed in the cleanup phase | 16:01 |
| JayF | aha, okay | 16:02 |
| JayF | that makes it more align | 16:02 |
| clarkb | it still happens in the image but the python executable (a symlink really) is just for dib scripts | 16:02 |
| JayF | okay cool | 16:02 |
| clarkb | its not something that the final image care about and as long as all images support python3 we don't need python2 anymore | 16:02 |
| clarkb | and I think centos 7 was the last holdout? | 16:02 |
| clarkb | that said selecting packages based on variable definitions is a more general feature and fixing that in pkg map processing is probably still worth while | 16:04 |
| clarkb | is that log saying it thinks the python version is "collect-data" | 16:05 |
| JayF | no | 16:05 |
| JayF | it's saying it's happening /in the collect_data method/ | 16:05 |
| JayF | I'm fairly sure the bad logic is here https://opendev.org/openstack/diskimage-builder/src/branch/master/diskimage_builder/elements/package-installs/bin/package-installs-squash#L173 | 16:05 |
| JayF | I couldn't ever figure out a case where that would resolve to true | 16:06 |
| JayF | and in my tests, dib_py_version was 3, dib_py_version_env was '' | 16:06 |
| JayF | so it would [conjecturing] fit if we removed *some* of the py2 scaffolding in DIB but not all | 16:06 |
| JayF | leaving this broken | 16:06 |
| JayF | I think the "fix" is to remove this feature from package-installs and pull any similarly py2-compat code from package-installs.yaml throughout the repo | 16:07 |
| JayF | I can write that change right after breakfast | 16:07 |
| clarkb | DIB_PYTHON_VERSION is set by the dib-python element in an environment.d script | 16:07 |
| clarkb | JayF: I think we need to undersatnd why it fails first before we can prescribe a fix | 16:07 |
| clarkb | did dib-python run? | 16:07 |
| JayF | I need to hit pause on this conversation for like, 15 minutes | 16:08 |
| JayF | I'll check the output and get back to you | 16:08 |
| clarkb | svc-map doesn't seem to have any element deps | 16:08 |
| JayF | I suspect the answer is "no, it didn't" | 16:08 |
| clarkb | I suspect the fix is to add dib-python as an element dep to svc-map | 16:08 |
| JayF | ++ I will look into that | 16:08 |
| * JayF testing that fix | 16:24 | |
| JayF | clarkb: good call, PR incoming | 16:31 |
| opendevreview | Jay Faulkner proposed openstack/diskimage-builder master: Fix pyyaml install for svc-map https://review.opendev.org/c/openstack/diskimage-builder/+/937192 | 16:33 |
| clarkb | then separately if we want to start trimming out the use of dib-python entirely I think that is worthy of consideration too. But is a separate thing | 16:34 |
| clarkb | probably the main thing to check first is if any of the supported build targets are python2 only. Off the top of my head centos 7 was the last one and we very recently removed support for it | 16:34 |
| clarkb | (and it is EOL upstream too) | 16:34 |
| JayF | I'll note Gentoo has a 2.7 slot for python available (mainly because of things with build scripts that still depend on python 2.7, like some chromium versions :X) | 16:35 |
| JayF | I am very +1 to just pretending that doesn't exist | 16:35 |
| JayF | lol | 16:35 |
| JayF | plus it's not supported as a PYTHON_TARGET which means while we'll install the interpreter, we will not install any package-manager packages that depend on it directly (e.g. you can't install tox into python 2.7, only supported pythons, currently 3.10-3.12) | 16:36 |
| clarkb | JayF: many dib targets still support python2. That isn't in qustion. The thing that matters is can dib run all of its scripts under python3 only | 16:56 |
| clarkb | fungi dib functests for 937192 are failing trying to build openeuler I suspect due to the mirror changes | 18:33 |
| clarkb | fungi: should we stop worrying about dib testing openeuler for now or maybe figure out disabling the use of our mirrors? Wonder if you have any thoughts on that | 18:33 |
| clarkb | but thinking the scv map fix is probably a good thing to include in a release | 18:34 |
| fungi | lookin' (not that i have the first clue what i'm lookin' for nor really the first thing about that distro) | 18:34 |
| fungi | kinda amused that the job was previously passing, since the mirroring was totally broken and the volume was over quota | 18:34 |
| clarkb | I wonder if it used simple/basic/lowlevel enough packages to function in that capacit? | 18:35 |
| clarkb | https://zuul.opendev.org/t/openstack/build/d40f2ec691ed45ceb9db3d425b9b5076/log/logs/openeuler-minimal_22.03-LTS-build-succeeds.FAIL.log is the logfile | 18:35 |
| clarkb | and its dfienitely getting 404s from the mirror and is sad about it | 18:36 |
| clarkb | let me see if Ican sort out turning off the use of the mirror | 18:36 |
| clarkb | (because honestly turning off the mirror entirely might be a reasonable thing to try too) | 18:36 |
| fungi | might just be as simple as a hard-coded version string | 18:36 |
| fungi | i guess https://zuul.opendev.org/t/openstack/build/d40f2ec691ed45ceb9db3d425b9b5076/log/logs/openeuler-minimal_22.03-LTS-build-succeeds.FAIL.log#319 is the first error? | 18:37 |
| fungi | yeah, per the error it's looking for openEuler-22.03-LTS and what we've mirrored now is openEuler-24.03-LTS | 18:39 |
| opendevreview | Clark Boylan proposed openstack/diskimage-builder master: Disable openeuler mirror use in dib-functests https://review.opendev.org/c/openstack/diskimage-builder/+/937200 | 18:39 |
| clarkb | I think ^ that will drop the use of the mirror and we should find normal package repos I hope | 18:40 |
| clarkb | then the next step is for people who want openeulre images to figure out dib updates (which I think there is a change for actually) | 18:40 |
| clarkb | "Error: No package basesystem available." IGuess we have to provide a location explicitly | 19:09 |
| clarkb | maybe we should just disable the functest instead? | 19:09 |
| clarkb | I'm open to ideas or volunteers to fix this properly but I'm not sure I will be doing that (happy to workaround though) | 19:10 |
| JayF | TBH I hadn't even heard of openeuler until the noise about mirroring it :) | 19:51 |
| JayF | I keep getting out of disk space errors in the package install step | 21:24 |
| JayF | do we, at some point, create a limited sized image or something? | 21:24 |
| JayF | because I have lots of disk space, and I realized this is after a loopback mount step, which implies we've got an image file created | 21:24 |
| * JayF hrms | 21:24 | |
| clarkb | JayF: by default dib runs in a tmpfs which yo ucan override if that isn't big enough | 21:32 |
| JayF | Yeah, i've done that | 21:32 |
| JayF | https://www.irccloud.com/pastebin/mbyCYljc/ | 21:32 |
| JayF | and there is no tmpfs mount during the build | 21:32 |
| JayF | I ensured it both via code changes locally and checking my mounts | 21:33 |
| clarkb | I would expect that means you're filling whatever DIB_TMP is set to | 21:33 |
| JayF | that's 100% not the case, confirmed | 21:33 |
| clarkb | does portage try t obuild in some special location maybe? | 21:33 |
| JayF | I have 56GB free in / | 21:33 |
| clarkb | could be that is filling | 21:33 |
| JayF | I suspect that is what's happening, perhaps | 21:34 |
| JayF | but AFAICT nothing *inside the chroot* is tmpfs mounted, either | 21:34 |
| JayF | which is why I went around to "am I somehow in like, a loopback mounted qcow file that's too small" | 21:34 |
| JayF | I'm either missing something obvious or there's a misdirection I'm not catching | 21:34 |
| JayF | this is also a ramdisk build, which might be special | 21:35 |
| clarkb | dib doesn't do anything with mounting the actual target image until near the very end | 21:35 |
| clarkb | it should build everything in a chroot on the normal filesystem then it does some math to calculate image size and create san image that it mounts and copies content into | 21:36 |
| JayF | this /is/ happening in finalize.d | 21:36 |
| clarkb | why are you emerging the kenrel in finalize.d? | 21:36 |
| JayF | it's doing a general update and I changed the use | 21:36 |
| JayF | I can fix that ordering issue | 21:36 |
| JayF | you might have quacked me to the answer | 21:36 |
| clarkb | your package installs should generally occur in intsall.d | 21:36 |
| JayF | gentoo element finalize.d does an emerge update | 21:37 |
| clarkb | that seems like a bug to me | 21:37 |
| clarkb | but you can contorl the final disk size and make it bigger | 21:38 |
| JayF | I'm going to try just moving that 99-cleanup into install.d | 21:38 |
| JayF | and see if it goes kaboom | 21:38 |
| JayF | at this point I'm still in "trying to get a thing to work" phase, not even trying to get a submittable change | 21:38 |
| JayF | that worked, thank you for the pointer | 22:36 |
| JayF | I'll take all these fixes and split them up properly into PRs once I get to the end | 22:36 |
| * JayF down to ~300M compressed IPA image | 22:37 | |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!