| mikal | I suspect I am doing something wrong, but I think I definitely need a hint... I have private CI on the patches I am proposing to OpenStack because I want to do CI things which are hard to express upstream. Whatever. That said, the most common reason for that CI to fail these days is because tarballs.opendev.org has replied with an error instead of | 03:16 |
|---|---|---|
| mikal | the source tarball the CI run needs. | 03:16 |
| mikal | So, I want to cache the things I need so as to reduce load on tarballs.opendev.org, and improve the reliability of my CI. A squid cache didn't work because it will just pass through the HTTP error from upstream. I tried syncing with AFS the bits I need, but the IO performance is terrible -- at least from my side of the world. | 03:17 |
| mikal | Attempt three is to write a python script which downloads the bits I need, and then serves it locally. I can use the releases git repo to determine the right version number of the various OpenStack projects for a given release, but I am struggling to come up with a way to turn that into filenames on tarballs.opendev.org. For example, oslo.db used | 03:18 |
| mikal | to use "oslo.db" as the releasable name on the web server, but now uses "oslo_db" instead. | 03:18 |
| mikal | Am I missing something? Is there a way to turn a project name and a version number into a list of all of the artifcats that project released for that version? | 03:18 |
| mikal | I can't find it if there is. | 03:19 |
| *** ykarel_ is now known as ykarel | 04:11 | |
| opendevreview | Michal Nasiadka proposed openstack/project-config master: Follow up for I170bdb3ffc89bc3307a08da7cd4c7f2793a4e491 https://review.opendev.org/c/openstack/project-config/+/984800 | 06:07 |
| opendevreview | Michal Nasiadka proposed openstack/project-config master: Follow up for I170bdb3ffc89bc3307a08da7cd4c7f2793a4e491 https://review.opendev.org/c/openstack/project-config/+/984800 | 06:08 |
| opendevreview | Michal Nasiadka proposed openstack/project-config master: Install pip-check-updates only in jobs requiring that https://review.opendev.org/c/openstack/project-config/+/984804 | 06:08 |
| opendevreview | Michal Nasiadka proposed openstack/project-config master: Follow up for I170bdb3ffc89bc3307a08da7cd4c7f2793a4e491 https://review.opendev.org/c/openstack/project-config/+/984800 | 06:09 |
| opendevreview | Michal Nasiadka proposed openstack/project-config master: Install pip-check-updates only in jobs requiring that https://review.opendev.org/c/openstack/project-config/+/984804 | 06:09 |
| opendevreview | Michal Nasiadka proposed openstack/project-config master: Install pip-check-updates only in jobs requiring that https://review.opendev.org/c/openstack/project-config/+/984804 | 06:49 |
| frickler | mikal: you can check the release pages, e.g. for your example https://releases.openstack.org/gazpacho/index.html#gazpacho-oslo-db lists the available tarball | 07:21 |
| frickler | for AFS you might want to configure a local cache, not sure how much that helps though with very high latency | 07:21 |
| mikal | Yeah, there is a cache by default so I do have one. The issue is that many small reads incur a huge amount of latency for me, whereas HTTP is pretty good a pipelining. | 07:26 |
| mikal | That webpage is basically exactly what I want. Is that driven under the hood by a yaml file or something that might be a bit easier for my pyhton to parse? | 07:27 |
| opendevreview | Merged openstack/project-config master: Follow up for I170bdb3ffc89bc3307a08da7cd4c7f2793a4e491 https://review.opendev.org/c/openstack/project-config/+/984800 | 07:32 |
| frickler | mikal: iirc it is generated by some tooling from the git release repo. adding a parseable version might not be a bad idea, we can discuss that with the release team | 08:13 |
| opendevreview | Michal Nasiadka proposed openstack/project-config master: DNM: Testing https://review.opendev.org/c/openstack/project-config/+/984903 | 08:35 |
| opendevreview | Michal Nasiadka proposed openstack/project-config master: DNM: Testing https://review.opendev.org/c/openstack/project-config/+/984903 | 08:36 |
| opendevreview | Michal Nasiadka proposed openstack/project-config master: DNM: Testing https://review.opendev.org/c/openstack/project-config/+/984903 | 08:37 |
| opendevreview | Michal Nasiadka proposed openstack/project-config master: DNM: Testing https://review.opendev.org/c/openstack/project-config/+/984903 | 08:39 |
| opendevreview | Michal Nasiadka proposed openstack/project-config master: DNM: Testing https://review.opendev.org/c/openstack/project-config/+/984903 | 08:41 |
| mikal | frickler: even if I could calculate it from the release repo that would meet my needs, but its not at all obvious to me how to do that. | 09:10 |
| opendevreview | Michal Nasiadka proposed openstack/project-config master: Fix pip-check-updates target https://review.opendev.org/c/openstack/project-config/+/984908 | 09:17 |
| opendevreview | Merged openstack/project-config master: Fix pip-check-updates target https://review.opendev.org/c/openstack/project-config/+/984908 | 09:47 |
| fungi | mikal: the tarball filenames and paths are constructed from the repo and version strings in per-deliverable yaml files organized by release name like https://opendev.org/openstack/releases/src/branch/master/deliverables/gazpacho for 2026.1/gazpacho | 12:00 |
| opendevreview | Sylvain Bauza proposed openstack/project-config master: Add openstack/agentic-workflows project https://review.opendev.org/c/openstack/project-config/+/984922 | 12:37 |
| frickler | fungi: I think the interesting question is how do you find out from that that there is oslo.db-1.0.0.tar.gz but oslo_db-18.0.0.tar.gz | 12:39 |
| frickler | (. vs. _ in case that isn't obvious) | 12:39 |
| fungi | python sdist name normalization rules, they're standardized (i forget the pep number off the top of my head) | 12:40 |
| fungi | it was part of pep 625... https://packaging.python.org/en/latest/specifications/source-distribution-format/#source-distribution-file-name | 12:45 |
| frickler | yes, but the rules changed sometime between 2024 and 2025 it seems. and we also still have oslo.db-master.tar.gz , so these aren't applied consistently everywhere | 12:45 |
| frickler | so mapping from deliverable entry to tarball name isn't a completely trivial function | 12:46 |
| opendevreview | Michal Nasiadka proposed openstack/project-config master: kolla: fix pcu packages and files https://review.opendev.org/c/openstack/project-config/+/984930 | 13:00 |
| opendevreview | Sylvain Bauza proposed openstack/project-config master: Add openstack/agentic-workflows project https://review.opendev.org/c/openstack/project-config/+/984922 | 13:08 |
| fungi | in our case, the mapping changed when we switched our release jobs from running `setup.py sdist` to `pyproject-build`, but regardless the easy solution is to try to grab the normalized name and then if it's a 404 not found try the older form | 13:25 |
| fungi | and if you're not going back very far you won't need the fallback | 13:26 |
| frickler | mikal: ^^ would that work for you? actually the links on the releases pages are broken in the same way, like 17.2.1 here works, but 17.0.0 doesn't https://releases.openstack.org/epoxy/index.html#epoxy-oslo-db | 13:41 |
| fungi | yeah, if memory serves it was around mid-late january last year that the tooling changed and we started getting normalized project names in sdist filenames | 13:43 |
| opendevreview | Michal Nasiadka proposed openstack/project-config master: pcu: run in project directory https://review.opendev.org/c/openstack/project-config/+/984930 | 15:11 |
| opendevreview | Merged openstack/project-config master: pcu: run in project directory https://review.opendev.org/c/openstack/project-config/+/984930 | 16:09 |
| opendevreview | Merged openstack/project-config master: Add openstack/agentic-workflows project https://review.opendev.org/c/openstack/project-config/+/984922 | 17:18 |
| fungi | bauzas: the repository is created now and i've added you as the first core reviewer (feel free to add whoever else you want). zuul is also aware of the project now so should react when you push the first change for review | 17:31 |
| bauzas | fungi: thanks a lot ! I'll fire up the first git init change and then I'll work on a series | 17:32 |
| fungi | well, you don't need to git init | 17:32 |
| fungi | clone https://opendev.org/openstack/agentic-workflows and then am your patches or whatever | 17:32 |
| fungi | at the moment it has a basic .gitreview file and a .zuul.yaml that just runs the built-in noop job | 17:33 |
| fungi | you probably want to add the official-openstack-repo-jobs template for it in project-config as a next step | 17:35 |
| bauzas | ah right | 17:35 |
| fungi | that way it will get github mirroring and similar functionality | 17:35 |
| bauzas | thanks for the help, will do it | 17:35 |
| fungi | yw | 17:35 |
| fungi | if i can't assist tact sig projects with setup, i'm pretty useless as a sig chair ;) | 17:36 |
| bauzas | lol | 17:36 |
| bauzas | does the sig has any meeting I need to join ? | 17:36 |
| bauzas | or how can I help for the sig ? | 17:36 |
| fungi | nope, it's very low-key | 17:37 |
| fungi | basically we're the human bridge between the openstack project and the opendev collaboratory, so try to offload openstack-specific requests in order to not pester the other opendev sysadmins with them as much | 17:38 |
| fungi | if you want to help with reviewing openstack-specific changes to openstack/project-config that's welcome, though not necessary. it's usually the entrypoint for people who have limited available time but want to pitch in | 17:40 |
| bauzas | ++ | 17:43 |
| bauzas | cool ping me then anytime you want me to look at those | 17:44 |
| fungi | it's mostly just random job configs, new project requests, acl changes, et cetera for things in the openstack/ git namespace | 17:45 |
| opendevreview | Sylvain Bauza proposed openstack/project-config master: Add Zuul jobs for openstack/agentic-workflows https://review.opendev.org/c/openstack/project-config/+/985017 | 17:52 |
| opendevreview | Merged openstack/project-config master: Add Zuul jobs for openstack/agentic-workflows https://review.opendev.org/c/openstack/project-config/+/985017 | 18:08 |
| opendevreview | Merged openstack/project-config master: Allow OpenStack-Ansible Cores to change WIP state https://review.opendev.org/c/openstack/project-config/+/981925 | 19:49 |
| opendevreview | Merged openstack/project-config master: Add GitHub mirroring for devstack-plugin-prometheus https://review.opendev.org/c/openstack/project-config/+/983728 | 19:49 |
| mikal | frickler: I think I am mildly disappointed there isn't a stronger linkage between the release deliverables yaml and the actual artifact names, but I'll give the underscore-then-fallback strategy a go today and see how I go. | 20:56 |
| clarkb | I think we're all annoyed by it for various raesons but python made their decisions and we don't have much choice but to follow. Otherwise things don't install properly | 20:58 |
| mikal | Oh, I'm not opposed to the change really. I guess I was expecting the release yaml for a given release to list the artifacts for each deliverable as well as just the deliverables? Otherwise what's to stop a bad person from inserting an extra artifact for a deliverable which didn't actually release? | 21:04 |
| clarkb | doesn't releases.yaml control the source to version mapping? Setuptools controls the sdists and wheel names. I guess the issue is that its operating at a level before your expectation? | 21:06 |
| clarkb | I don't work on the release team so may have that bit wrong | 21:06 |
| mikal | I think both of those are true. My point is that if I try to rely only on configuration in git from openstack, there is no way for me to tell that a given openstack release produced a specific list of artifacts. I need to infer than by guessing filenames from a web server. What the git config _does_ tell me is what deliverables should be present | 21:08 |
| mikal | for a release, and what version numbers they should have, but not the actual artifact names. | 21:08 |
| clarkb | got it. There is a bit of a chicken and egg there, but one that is deterministic as long as python doesn't change the naming rules again later. So the file probably could list things, but they may not exist yet until later | 21:09 |
| fungi | to reiterate, what's in the structured data in the releases repository is an input to tools provided by the python packaging community, and what comes out the other side is their translation of the inputs we supplied | 21:21 |
| fungi | we could add a post-run layer to try to capture those outputs and then feed them back into the releases repository, though it feels like a layer violation | 21:22 |
| clarkb | ya or predict them and record that that is the eventually expected state (which is what I was less explicit about above) | 21:23 |
| fungi | predictability and the python packaging ecosystem don't exactly go hand-in-hand | 21:28 |
| mikal | One thing I looked at was that I can ask the pypi JSON api for the source artifacts for a given release version, but I wasn't confident that's a complete mapping. | 21:28 |
| mikal | I guess I was assuming that the push to pypi would log what artifacts it created and then record those back somewhere. Sort of like how attestation works with moden github python release processes (to my limited understanding of such things). | 21:29 |
| mikal | Honestly, if tarballs.opendev.org was reliable for me, I would never have gone down this pathway at all. Its all a side quest while trying to make my CI more reliable. | 21:30 |
| fungi | it mostly started because pypi decided to deprecate and then remove publication of cryptographic signatures, so we do try to make sure that the detached openpgp signatures on tarballs.openstack.org match the packages on pypi | 21:31 |
| fungi | or in tarballs.opendev.org/openstack/ really, as that's where it redirects to these days | 21:32 |
Generated by irclog2html.py 4.1.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!