| opendevreview | Ian Wienand proposed zuul/zuul-jobs master: container-build : add container_promote_method flag https://review.opendev.org/c/zuul/zuul-jobs/+/879009 | 00:05 |
|---|---|---|
| opendevreview | Ian Wienand proposed zuul/zuul-jobs master: remove-registry-tag: role to delete tags from registry https://review.opendev.org/c/zuul/zuul-jobs/+/878614 | 00:05 |
| opendevreview | Ian Wienand proposed zuul/zuul-jobs master: promote-container-image: use generic tag removal role https://review.opendev.org/c/zuul/zuul-jobs/+/878740 | 00:05 |
| opendevreview | Ian Wienand proposed zuul/zuul-jobs master: remove-registry-tag: update docker age match https://review.opendev.org/c/zuul/zuul-jobs/+/878810 | 00:05 |
| ianw | 2023-03-29 02:21:38.331815 | TASK [build-wheel-cache : Prevent using existing wheel mirror] | 02:05 |
| ianw | 2023-03-29 02:21:38.346571 | wheel-cache-ubuntu-jammy-python3 | changed: 1 line(s) removed | 02:05 |
| ianw | 2023-03-29 02:21:38.388891 | | 02:05 |
| ianw | 2023-03-29 04:40:45.052347 | RUN END RESULT_TIMED_OUT: [untrusted : opendev.org/openstack/openstack-zuul-jobs/playbooks/wheel-cache/build.yaml@master] | 02:05 |
| ianw | that's annoying | 02:05 |
| ianw | it really must be the next step | 02:06 |
| ianw | https://opendev.org/openstack/openstack-zuul-jobs/src/commit/699e811cb8fd3f0a72d97d6866cd0a4a07a191fd/roles/build-wheel-cache/tasks/main.yaml#L29 | 02:06 |
| ianw | https://zuul.opendev.org/t/openstack/builds?job_name=publish-wheel-cache-centos-9-stream-arm64&project=openstack/requirements was failing, but it's gone green. and i'd say that relates to the mirror resyncing itself | 02:10 |
| ianw | https://zuul.opendev.org/t/openstack/builds?job_name=publish-wheel-cache-ubuntu-jammy&project=openstack/requirements | 02:11 |
| ianw | looks like it's probably right on the edge | 02:11 |
| ianw | there's successful builds at 2 hrs 29 mins 20 secs | 02:11 |
| ianw | i think we can re-evaluate our branch list | 02:13 |
| ianw | the short story is i can't see anything systematically wrong with the wheel builds right now. i think need to monitor for next few days | 02:13 |
| fungi | with openstack's constraints freezing for stable branches, i wonder if building new wheels for them is necessary | 02:19 |
| ianw | i guess maybe zed branching has added another step | 02:27 |
| opendevreview | sean mooney proposed openstack/diskimage-builder master: [WIP] add apk element https://review.opendev.org/c/openstack/diskimage-builder/+/755411 | 02:35 |
| opendevreview | sean mooney proposed openstack/diskimage-builder master: [WIP] bootloader-alpine-support https://review.opendev.org/c/openstack/diskimage-builder/+/755412 | 02:35 |
| opendevreview | sean mooney proposed openstack/diskimage-builder master: [WIP] openssh-server-alpine-support https://review.opendev.org/c/openstack/diskimage-builder/+/755413 | 02:35 |
| opendevreview | sean mooney proposed openstack/diskimage-builder master: [WIP] simple-init-alpine-support https://review.opendev.org/c/openstack/diskimage-builder/+/755414 | 02:35 |
| opendevreview | sean mooney proposed openstack/diskimage-builder master: alpine support Change-Id: I7db02c1bdc8c6e466eee30a72ee8c70cf8ee0bf5 https://review.opendev.org/c/openstack/diskimage-builder/+/755410 | 02:35 |
| ianw | [iwienand@fedora19 requirements (master)]$ cat /tmp/u-c.txt | wc -l | 02:37 |
| ianw | 6578 | 02:37 |
| ianw | [iwienand@fedora19 requirements (master)]$ cat /tmp/u-c.txt | sort | uniq | wc -l | 02:37 |
| ianw | 598 | 02:37 |
| ianw | i'm really not sure why we don't de-dup the u-c across all branches | 02:37 |
| ianw | i have dejavu. i think we've discussed it all before. i think because each wheel builds with constraints of its own branch | 02:55 |
| *** dasm|off is now known as Guest9353 | 04:02 | |
| frickler | mnasiadka: mirror for ubuntu-ports should be in sync again | 07:18 |
| *** jpena|off is now known as jpena | 07:18 | |
| *** amoralej is now known as amoralej|off | 07:26 | |
| opendevreview | Sylvain Bauza proposed openstack/project-config master: Stop using Storyboard for Placement projects https://review.opendev.org/c/openstack/project-config/+/878932 | 07:39 |
| *** amoralej|off is now known as amoralej | 07:44 | |
| frickler | this is really weird, why does wheel building take 16 mins on bullseye but 2h on jammy? | 08:00 |
| frickler | maybe we should add some more detailed logging to the build step in order to find out | 08:01 |
| opendevreview | sean mooney proposed openstack/diskimage-builder master: add apk element https://review.opendev.org/c/openstack/diskimage-builder/+/755411 | 10:17 |
| opendevreview | sean mooney proposed openstack/diskimage-builder master: bootloader-alpine-support https://review.opendev.org/c/openstack/diskimage-builder/+/755412 | 10:17 |
| opendevreview | sean mooney proposed openstack/diskimage-builder master: alpine support https://review.opendev.org/c/openstack/diskimage-builder/+/755410 | 10:17 |
| opendevreview | sean mooney proposed openstack/diskimage-builder master: simple-init-alpine-support https://review.opendev.org/c/openstack/diskimage-builder/+/755414 | 10:17 |
| opendevreview | sean mooney proposed openstack/diskimage-builder master: [WIP] openssh-server-alpine-support https://review.opendev.org/c/openstack/diskimage-builder/+/755413 | 10:17 |
| mnasiadka | frickler: fantastic | 10:20 |
| *** amoralej is now known as amoralej|lunch | 11:13 | |
| fungi | frickler: the time to build will depend on how many things have prebuilt wheels on pypi for that platform, so if we're building old versions of things that already have wheels for the python version on focal never got wheels on pypi for the version of python on jammy, that would explain it | 11:44 |
| *** amoralej|lunch is now known as amoralej | 12:47 | |
| frickler | there is a config error in the openstack tenant for github.com/ansible/ansible: https://paste.opendev.org/show/bDAq06x3KmZfZmajB68F/ any idea how to fix that? | 13:42 |
| frickler | same for pytest-dev/pytest-testinfra | 13:43 |
| frickler | corvus: maybe zuul should ignore merge-mode for projects that don't import any config? | 13:44 |
| Clark[m] | frickler: I suspect zuul is detecting those projects do something like squash or cherry pick and complaining we do normal merges locally which is out of sync with upstream. We may be able to set that on a per project basis | 13:44 |
| Clark[m] | Merge mode matters just for testing even if you don't do configs because you can build invalid speculative state | 13:45 |
| frickler | what do you mean by invalid state? even if merging is enabled in github, the person applying the PR can still decide to squash or rebase, zuul can neither predict nor avoid that | 13:50 |
| Clark[m] | frickler: merge mode determines how zuul locally merges changes. This should match the merge mode for the project in the code review system to ensure the speculative state matches the actual state as much as possible | 14:02 |
| Clark[m] | Is there not a GitHub merge mode default for projects? | 14:02 |
| Clark[m] | Yes looks like you can enforce a specific merge strategy in GitHub. I suspect zuul is detecting it is out of sync with projects that have done this | 14:04 |
| frickler | there is a default, but you can also allow multiple strategies at the same time, so it doesn't go 100%. the question now is how to find out what is the correct mode for those projects | 14:09 |
| *** Guest9353 is now known as dasm | 14:48 | |
| corvus | the error is stating that the merge mode in zuul is not one of the possible merge modes selected by the project owners in github | 14:59 |
| corvus | so yeah, we can set that on the project in zuul to make the error go away | 15:00 |
| fungi | is it necessary to check that at all if we don't load any configuration from the project? | 15:03 |
| fungi | curious if it would make sense to have a short-circuit around the check in such cases | 15:03 |
| opendevreview | Bartosz Bezak proposed openstack/project-config master: Stop using Storyboard for Kayobe https://review.opendev.org/c/openstack/project-config/+/879055 | 15:04 |
| frickler | fungi: clarkb mentioned earlier it will still affect speculative merging | 15:05 |
| frickler | so zuul should at least try to do the most likely variant | 15:05 |
| clarkb | right we want the actual commits under test to look similar to what they will end up looking like upstream when we test with them. | 15:06 |
| fungi | oh, got it. i missed that | 15:07 |
| clarkb | sounds like we may not be able to 100% solve that with github's allowance for variance but it knows the current method will enver be correct so complains | 15:07 |
| corvus | yeah, user intervention may be required due to the ambiguity | 15:08 |
| frickler | but then, whether a PR is merged, rebased or squashed, the resulting code should be the same. so unless a check actually looks at git commits, it shouldn't matter | 15:15 |
| *** jpena is now known as jpena|off | 16:35 | |
| *** amoralej is now known as amoralej|off | 16:36 | |
| opendevreview | Merged opendev/system-config master: Redirect openstack-specs to git repo on opendev https://review.opendev.org/c/opendev/system-config/+/856834 | 17:08 |
| opendevreview | Thales Elero Cervi proposed openstack/project-config master: Add virtualization (virt) repo to StarlingX https://review.opendev.org/c/openstack/project-config/+/879076 | 18:40 |
| fungi | that ^ has made me wonder if we've updated the acl examples in infra-manual yet... | 18:52 |
| fungi | never mind, our examples in there are simple enough they don't require updating. they just inherit from stuff we already updated | 19:00 |
| Clark[m] | The error is due to pushSignedTag. Did we break that in the checker script? | 19:02 |
| fungi | should be createSignedTag instead | 19:08 |
| fungi | we updated them all in https://review.opendev.org/826334 | 19:09 |
| fungi | also we corrected it in the infra-manual examples | 19:10 |
| fungi | so probably was a copy from an old change | 19:10 |
| fungi | from more than a year ago | 19:10 |
| Clark[m] | Aha | 19:12 |
| fungi | i left a comment on the change giving them a heads up | 19:12 |
| opendevreview | Thales Elero Cervi proposed openstack/project-config master: Add virtualization (virt) repo to StarlingX https://review.opendev.org/c/openstack/project-config/+/879076 | 19:17 |
| ianw | fungi: do you have some time to discuss the wheel jobs? | 19:31 |
| ianw | i started about 4 things yesterday and just confused myself | 19:31 |
| fungi | ianw: sure | 20:07 |
| opendevreview | Thales Elero Cervi proposed openstack/project-config master: Add virtualization (virt) repo to StarlingX https://review.opendev.org/c/openstack/project-config/+/879076 | 20:10 |
| ianw | fungi: my first thought is, what are we actually expecting to work | 20:40 |
| ianw | specifically at https://opendev.org/openstack/openstack-zuul-jobs/src/commit/699e811cb8fd3f0a72d97d6866cd0a4a07a191fd/roles/build-wheel-cache/files/wheel-build.sh#L39 | 20:40 |
| ianw | there seems little point running stein -> ??? on jammy with python3.10, because a bunch of stuff doesn't build | 20:41 |
| fungi | i think the intent was that we would install the constraints lists for each maintained stable branch on the platforms where we run jobs for those branches | 20:42 |
| fungi | though given how we freeze constraints for stable branches, i'm not sure even that is necessary unless we're worried about the wheel cache getting wiped and having to replace all the content from scratch | 20:43 |
| fungi | obviously installing constraints for stein on jammy is absurd | 20:43 |
| fungi | i'm just increasingly uncertain we should bother to build any besides master | 20:44 |
| fungi | if we do intend to continue building wheels for maintained stable branches, we should probably do each branch's constraints file in a separate job anyway, so that we can select what platform(s) to do that on | 20:45 |
| ianw | yeah, things like centos-8 seem only to be used by swift | 20:46 |
| fungi | it's probably also worth bringing this topic up with prometheanfire and tonyb to get their perspectives | 20:47 |
| ianw | ++ | 20:47 |
| tonyb | how far back do I need to read? | 20:47 |
| ianw | about 10 lines :) | 20:48 |
| fungi | tonyb: the topic is periodically building and caching wheels for stable branches | 20:48 |
| ianw | i think we should be considering the case of "afs exploded" and we want to get back | 20:48 |
| ianw | i.e. probably should say "these platforms have previously built the wheels, which are now pushed so forget about it" | 20:49 |
| tonyb | that's something frickler has been looking at off and on. | 20:49 |
| fungi | if we want to cover that case reliably, we should probably have jobs for each branch+platform+arch we expect to continue running jobs on | 20:49 |
| ianw | right, i guess i got to making that list, then just left more confused :) | 20:50 |
| tonyb | yeah I'd like to see that. | 20:51 |
| fungi | but also it seems slightly wasteful to keep rerunning these stable branch jobs which are going to at best just keep redundantly rebuilding the same frozen set of packages every day | 20:51 |
| fungi | on the off-chance that the content they're building might disappear | 20:51 |
| tonyb | weekly? or monthly would probably suffice for stable branches | 20:51 |
| ianw | like for example, we've got swift running master jobs on centos-8-stream | 20:51 |
| ianw | but building master global requirements on 8-stream with python3.6 doesn't seem like it can work | 20:52 |
| fungi | another aspect of this is, like with our decisions about what to mirror, these are a convenience to help speed up jobs. if there are very few jobs using wheels for master branch constraints on centos-8-stream nodes, then maybe we just don't bother prebuilding and caching those | 20:53 |
| ianw | and i wonder the opposite, perhaps we have stable branch jobs running on jammy. again most of the stable branches g-r have stuff that isn't py3.10 happy | 20:53 |
| tonyb | I really don't think it can unless we add more annotations to the constrains files and are prepared to cap setuptools etc for the purposes of building | 20:53 |
| ianw | i think the problem with branch / platform jobs is the multiplication | 20:55 |
| tonyb | there is another aspect, iirc some system libraries in focal/jammy have moved forward to the point that you can't build the wheel and the only reason we have a working CI is because of that cache | 20:55 |
| ianw | we have in the requirements repo rocky -> 2023.1 branches == 10 | 20:56 |
| fungi | i would be fine only building master branch constraints for the platforms listed in the openstack pti, or perhaps even simply a subset that most of the jobs run on, and have some notes on how to similarly regenerate wheels for stable branches if the cache suffers a catastrophic loss | 20:56 |
| tonyb | I'd like to see us rebuild the cache for stable branches on some cadence | 20:57 |
| fungi | stable branches run far fewer jobs, and if there are no prebuilt wheels for those jobs to use then yes they'll take longer or maybe highlight that a project has failed to list a binary dependency required for some of their deps which are only available as sdists | 20:57 |
| tonyb | that way if there is a catastrophic loss we have some confidence we can rebuild | 20:57 |
| ianw | ... and 16 platforms with x86+arm64 == 160 potential jobs | 20:57 |
| tonyb | fungi at the moment if there wasn't a cache most of the stable branches wouldn't even run | 20:58 |
| fungi | they'd hit timeouts? or just that they're missing headers for some libs that extensions need to build against? | 20:58 |
| tonyb | I can publish a patch to show that tomorrow | 20:58 |
| tonyb | it's that the system libraries haveoved forward and the python bindings haven't | 20:59 |
| tonyb | for example if you have time build python-nss on jammy | 20:59 |
| fungi | why would stable branch jobs be running on jammy? | 21:00 |
| tonyb | the wheel fails and the only way to fix it is use code from hg | 21:00 |
| fungi | oh, stable/2023.1 runs on jammy i guess | 21:00 |
| tonyb | fungi I'd need to look | 21:00 |
| jrosser | looks like both kolla and osa deploy Z on jammy | 21:01 |
| tonyb | fungi you can just grab a jammy system/container and try pip install --user python-nss | 21:01 |
| tonyb | jrosser: do they currently benefit from a wheel cache? | 21:02 |
| jrosser | osa certainly uses the built wheel cache | 21:03 |
| tonyb | WRT to python-nss I'm working with $people to get it fixed and published but it's going | 21:03 |
| jrosser | and arm jobs are completely impossible without | 21:03 |
| fungi | but also the version of python-nss isn't changing in stable branch constraints | 21:04 |
| tonyb | jrosser: interesting. I'll have to poke at the logs. | 21:04 |
| jrosser | and by impossible i mean the wheel build time lands in the job instead and it's an inevitable timeout | 21:04 |
| ianw | hrm, i actually think the arm64 build isn't handling 2023.1 branch right | 21:05 |
| tonyb | fungi: true. but 1.0.1 isn't buildable today. so if we didn't have that in a wheel cache somewhere I suspect anything that installs it would fail | 21:05 |
| ianw | https://opendev.org/openstack/openstack-zuul-jobs/src/branch/master/roles/build-wheel-cache/files/wheel-build.sh#L33 | 21:05 |
| ianw | that lists the branches and does tail -2. but 2023.1 is going to be at the top | 21:05 |
| fungi | so the question is do we want automation rebuilding python-nss for stable/zed periodically (daily, weekly) or just a procedure for rebuilding it if something catastrophic happens to our cache? if the latter, is some days or weeks without a wheel cache for stable/zed disasterous? are there alternatives, like backing up the cache content? | 21:06 |
| fungi | do we really need to be able to rebuild the cache from sdists, or is restoring from a file backup sufficient? | 21:07 |
| ianw | ... yes it's only iterating yoga/zed https://d23c12b0aad63e0703e8-47c67d6f96b63324a4b586b546351bb6.ssl.cf5.rackcdn.com/periodic/opendev.org/openstack/requirements/master/publish-wheel-cache-ubuntu-jammy-arm64/ae56478/python3/ | 21:07 |
| tonyb | pragmatically the latter would be fine. | 21:07 |
| tonyb | in an ideal world I'd like the former | 21:08 |
| ianw | fungi: yeah, the gitops-y in me thinks that it's best if we have it codified somewhere, at least | 21:08 |
| fungi | if backups are a possibility, then we could really just run master branch wheel build jobs with the expectation that stable branch constraints don't change in ways that pull in new versions of things not available already as wheels | 21:09 |
| ianw | i guess arm64 has built everything in master, which eventually ended up in 2023.1 anyway, which is why this all hangs together, and what fungi is saying | 21:09 |
| ianw | as long as master is building every day, and we keep appending to the cache, then we don't have to worry about branches | 21:10 |
| fungi | and one daily backup is probably less costly resource-wise than maintaining 160 different wheel building jobs | 21:10 |
| ianw | i guess we would not maintain 160, but we'd have a matrix and have to at least decide which of the 160 to maintain | 21:11 |
| fungi | and that matrix would, itself, become a maintenance burden too | 21:11 |
| tonyb | ianw: does the current builder code start with an empty cache | 21:11 |
| tonyb | sorry I'm on my phone and can't get to it quickly | 21:11 |
| fungi | yeah, the builder runs on a fresh vm and doesn't use our central cache | 21:12 |
| tonyb | okay. | 21:13 |
| fungi | it's just a "normal" zuul job these days | 21:13 |
| ianw | sorry have to school run, back in 20 | 21:15 |
| ianw | https://review.opendev.org/c/opendev/infra-specs/+/703916/1/specs/wheel-modernisation.rst has some discussion to | 21:16 |
| fungi | the main optimization we added is that after it installs the constraints list, the job scans the pip install log to build a list of anything which was downloaded from pypi and cleans it out of the cache before we copy the remainder for publishing | 21:16 |
| fungi | so that we don't redundantly publish wheels which are already available on pypi | 21:16 |
| fungi | we had a few incidents in the past where a wheel was yanked from pypi and our jobs didn't see that because we had a copy of it in our central cache | 21:17 |
| tonyb | fungi: thanks. tomorrow I'll try to make time to check the last logs | 21:25 |
| fungi | i think one of the goals of rebuilding for all branches was that we could as a next step start removing anything from the cache that we hadn't built in the latest run, but that was before we had 10 open branches for openstack, and a second processor architecture, and as many distros in the pti | 21:27 |
| fungi | so i no longer think that goal is especially reasonable | 21:28 |
| fungi | i suppose one thing we *can* probably do safely is a cleanup pass of the current cache to 1. remove any wheels which are also available on pypi that we cached before the recent optimizations, and 2. remove any wheels which are for cpython versions or platforms we no longer run jobs on | 21:45 |
| fungi | that might reduce what needs backing up, if nothing else | 21:45 |
| fungi | and *maybe* we can check the versions in the cache against version numbers in constraints files, removing any that ended up cached because they once appeared as a constraint but then the constraint was changed while still in master | 21:51 |
| fungi | all of that seems probably scriptable | 21:51 |
| fungi | and could still get us close to the goal of having a cache that contains only packages we actively need for jobs | 21:52 |
| fungi | and those sorts of cleanup operations can happen asynchronously and are comparatively cheap compared to repeatedly rebuilding every last thing | 21:57 |
| ianw | as step 1 http://mirror.iad.rax.opendev.org/wheel/centos-7-x86_64/ can go | 22:03 |
| ianw | root@mirror01:/afs/openstack.org/mirror/wheel/centos-7-x86_64# du -hs . | 22:04 |
| ianw | 2.6G | 22:04 |
| ianw | tonyb: i must be missing something about python-nss, it doesn't seem to be in upper-constraints? | 22:12 |
| tonyb | hmm maybe it's only on older branches. | 22:15 |
| tonyb | https://opendev.org/openstack/requirements/src/branch/stable/zed/upper-constraints.txt#L388 | 22:15 |
| tonyb | that'd explain some of my confusion | 22:15 |
| tonyb | yup it looks like it isn't needed in > zed | 22:16 |
| ianw | what's weird is i don't see it in http://mirror.iad.rax.opendev.org/wheel/ubuntu-22.04-x86_64/ | 22:16 |
| ianw | it is in the log @ https://438434f768f3cb6b0f33-325b4a96905c9937625b58a50d186e45.ssl.cf2.rackcdn.com/periodic/opendev.org/openstack/requirements/master/publish-wheel-cache-ubuntu-jammy/e22233b/python3/yoga-job.log | 22:18 |
| ianw | [iwienand@fedora19 python-nss===1.0.1]$ cat stderr | 22:19 |
| ianw | error: subprocess-exited-with-error | 22:19 |
| ianw | 22:19 | |
| ianw | it suggests to me it's *never* build in these jobs :/ | 22:19 |
| ianw | https://paste.opendev.org/show/boC4QqG0WTgf5ShZvgiy/ is the full output | 22:20 |
| ianw | (that just came from the complete logs kept at https://438434f768f3cb6b0f33-325b4a96905c9937625b58a50d186e45.ssl.cf2.rackcdn.com/periodic/opendev.org/openstack/requirements/master/publish-wheel-cache-ubuntu-jammy/e22233b/python3/build-logs.tar.bz2) | 22:20 |
| ianw | https://zuul.opendev.org/t/openstack/build/e22233bb42554e769756091bc44e3cba/logs is the job | 22:21 |
| clarkb | fungi: ya I think having 10 branches intsead of 3 makes a huge difference here | 22:21 |
| ianw | it *has* built on focal -> http://mirror.iad.rax.opendev.org/wheel/ubuntu-20.04-x86_64/python-nss/ | 22:23 |
| ianw | so this comes down to the matrix thing again. yoga upper-constraints.txt can't install on jammy essentially | 22:24 |
| ianw | fungi: is it perhaps fair to say that we don't care about any .whl in our mirror that has -none-any.whl? | 22:26 |
| clarkb | ianw: I think those may still slightly speed up the install process but probably not to the extent that we'd need them | 22:27 |
| clarkb | (since pip will make a wheel of those packages first then install them when looking at an sdist) | 22:27 |
| ianw | is anything on pypi only an sdist though? | 22:28 |
| fungi | some things | 22:28 |
| fungi | yeah, i don't see significant harm is removing all pure python wheels, but would probably instead just check to see if what we have is already on pypi and remove it if so | 22:29 |
| clarkb | I ran into something the other day redoing my zuul env | 22:29 |
| clarkb | they definitely exist | 22:29 |
| ianw | yeah i guess pypi does no building at all for you | 22:29 |
| ianw | it looks like https://pypi.org/project/pypi-simple/#description might answer that for us | 22:29 |
| fungi | basically things which seem pretty safe to clean up are the platforms we no longer use, the wheels which are identical to ones on pypi, and wheels for versions of things which appear in no current constraints lists | 22:30 |
| ianw | if we fed it the package, then looked at the version from upper-constraints.txt, and checked if there is a .whl available | 22:30 |
| ianw | fungi: and then, essentially only build master and only on the platforms we expect to be able to run master? | 22:31 |
| fungi | right | 22:31 |
| fungi | and start backing up the cache | 22:31 |
| fungi | so that we can restore it rather than having to find a way to rebuild it | 22:32 |
| fungi | and figure out some periodic (semi-automated or automated) cleanup process to prune unneeded things based on the above rules | 22:32 |
| ianw | ok, rough theory is to cat all the upper-requirments.txt together and sort | uniq it, then run it through something using pypi-simple to see if there's a wheel available upstream | 22:33 |
| ianw | if yes; then candidate for removal | 22:33 |
| ianw | (cat all together from all branches in requirements repo) | 22:33 |
| clarkb | what does removing wheels we already have get us? Smaller archive that is easier to backup? | 22:34 |
| fungi | we need to compare to what we've got cached already though, right? how do we determine if there's a wheel upstream which satisfies the python version on focal but not jammy? | 22:34 |
| fungi | clarkb: yeah, reducing the footprint for backup purposes. i'm guessing (without actually having checked) that we likely have more cruft than actually used files | 22:35 |
| ianw | well if it's a py3-none presumably it's fine everywhere? | 22:35 |
| fungi | for those specifically, yes | 22:35 |
| fungi | if we're trying to figure out manylinux1 and abi3 stuff it gets more complicated | 22:36 |
| fungi | i was assuming we'd just make a list of all the files in our cacheş and then check pypi to see whether each one exists there or not | 22:37 |
| ianw | hrm, a manylinux1 would essentially work on all our platforms | 22:37 |
| fungi | a manylinux1 for the right python version would | 22:38 |
| fungi | also manylinux1 for abi3 would work on most of our platforms, but those with old pip may not support checking abi3 | 22:38 |
| ianw | fungi: how do we query for a specific .whl though? it seems like it's done by hash | 22:38 |
| ianw | e.g. https://files.pythonhosted.org/packages/59/af/dffede5a125dc50c23b81656a8e129fa3f3fad4d94f97ebe90749224c001/taskflow-5.1.0-py3-none-any.whl | 22:38 |
| fungi | those hashes are checksums, if we want to go about it that way. though i was thinking query pypi's (simple or json) api | 22:39 |
| fungi | the pypi simple api gives you a list of filenames | 22:39 |
| ianw | right, ours don't match | 22:39 |
| ianw | hashes | 22:39 |
| fungi | it's long enough to be a sha512sum, but doesn't actually match the sha512sum of what i download | 22:42 |
| ianw | ok, i see | 22:43 |
| ianw | curl https://mirror.iad.rax.opendev.org/pypi/simple/taskflow/ | 22:43 |
| fungi | apparently BLAKE2b-256 | 22:43 |
| ianw | compare to, say https://mirror.iad.rax.opendev.org/wheel/ubuntu-20.04-x86_64/taskflow/ | 22:44 |
| fungi | also i means sha256sum (which it also wasn't) | 22:44 |
| ianw | so i think all of those .whls for our taskflow are upstream; that whole thing could go | 22:45 |
| ianw | one interesting thing might be how much this hides problems because we don't specify the data-requires-python | 22:47 |
| fungi | for some reason i thought we were excluding those from publication by scanning the pip log and identifying what was downloaded from pypi, but maybe that check has quietly broken | 22:47 |
| ianw | fungi: yeah, we *should* be | 22:47 |
| ianw | some of them have date 2023-01-07 which is definitely after that check | 22:48 |
| fungi | but anyway, we should be able to either check the pypi simple api for a file list and compare filenames to decide what to remove, or generate blake2b-256 checksums of what we have and look for those on pypi though that sounds like more work when we ought to be able to assume same filename means same file | 22:48 |
| ianw | ++ | 22:50 |
| ianw | i think this is probably worth doing, as what's left over will be an interesting list of what we *actually* need to build for speed | 22:50 |
| fungi | i do wonder why warehouse settled on blake2b instead of sha2 for its primary checksumming | 22:50 |
| fungi | clarkb: a related reason to clean as much cruft as possible out of the cache is not just because it will take up space for backups but also because the backup process would be reading this from afs which means it could be pretty slow for a large amount of data | 22:52 |
| fungi | the smaller we can keep that cache, the faster backups will complete, the less load we'll put on afs servers, et cetera | 22:52 |
| Clark[m] | Hem when they broke the pip cache it was sha256 that must be a new change? | 23:36 |
| Clark[m] | Silly phone auto complete... 'hrm' | 23:39 |
| fungi | maybe pip uses sha2-256 and warehouse uses blake2b-256? or yeah, maybe that was the breakage | 23:52 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!