| *** benj_9 is now known as benj_ | 14:19 | |
| *** benj_4 is now known as benj_ | 15:21 | |
| *** benj_2 is now known as benj_ | 16:20 | |
| *** benj_3 is now known as benj_ | 16:29 | |
| *** __ministry is now known as Guest11263 | 18:06 | |
| sean-k-mooney | fungi: clarkb: what is the current stance on creating new repos in the x namespace? | 19:14 |
|---|---|---|
| clarkb | we try to get people to not do that and use a more appropriate namespace (and create a new one is necessary) | 19:15 |
| sean-k-mooney | im currently looking at creating a poc of a promethous plugin for horizon | 19:15 |
| clarkb | the x namespace is there because we couldn't get a number of projects to do anything better when we did the big migration | 19:16 |
| sean-k-mooney | we are goign to chat to the horizon and telemetyr teams about it at the ptg | 19:16 |
| sean-k-mooney | but im not sure that either team will want to incldue it in there deliverable so im not sure it will fit under openstack | 19:17 |
| sean-k-mooney | clarkb: i cloud create teh poc on my github and defer this conversation until after we chat to them in the ptg | 19:18 |
| fungi | uppenstock/horizon-prometheus-plugin ;) | 19:18 |
| sean-k-mooney | is ^ a thing | 19:18 |
| fungi | hopenslack | 19:19 |
| fungi | precariouslystacked | 19:20 |
| clarkb | sean-k-mooney: no I think fungi is merely suggesting that you can create an arbitrary but hopefully somewhat meaningful namespace | 19:21 |
| sean-k-mooney | ya i was debating about that | 19:21 |
| sean-k-mooney | openstack-contrib | 19:21 |
| sean-k-mooney | or something with out either software projects names in it | 19:21 |
| clarkb | oscloudtools/foo | 19:22 |
| fungi | yeah, i'm basically wondering if there's a need for an openstack-adjacent place to group efforts that teams aren't likely to adopt officially | 19:22 |
| fungi | s/place/namespace | 19:22 |
| sean-k-mooney | well it might be adopted offically we jsut have not had that converstation yet | 19:22 |
| sean-k-mooney | basiclly the horizon team is small and ususlaly ask the project teams to own the plugin form the governance point of view | 19:23 |
| sean-k-mooney | for prometheus there isnt a project team because its not part of openstack | 19:24 |
| sean-k-mooney | and im not sure if the telemetry team are interested or not | 19:24 |
| sean-k-mooney | so ya a generic namespace might be the approch we end up taking | 19:24 |
| clarkb | one thing I will say is that I think its a bug that we have this tendency to wait for the ptg to discuss things. | 19:25 |
| clarkb | If it were me I would send an email to the list and get an answer quicker | 19:25 |
| sean-k-mooney | well i may also do that | 19:25 |
| clarkb | the ptg should be for things where we've already done the async thing and can't reach a resoution. Not the default appraoch | 19:25 |
| sean-k-mooney | fair | 19:26 |
| fungi | though i can certainly understand that it's sometimes hard to get inout from the horizon team other than jumping into a ptg session of theirs | 19:26 |
| fungi | s/inout/input/ | 19:27 |
| sean-k-mooney | well i asked out internal folks that work on it and they are the one that said normally they would ask the project teams to own the plugins | 19:27 |
| sean-k-mooney | so they already gave the indicatoin that they would prefer if it was out side fo the upstream teams offical delvibele but suggeested i reach out ot folks publicly too | 19:28 |
| sean-k-mooney | fungi: clarkb the reaon i asked ye as 2 fold | 19:29 |
| sean-k-mooney | 1 to gague if i should jsut wait before creating work for ye if the repo jmight changed in a few weeks | 19:29 |
| sean-k-mooney | and too just to confirm that x should be avoided | 19:29 |
| sean-k-mooney | i can do my poc on github i just strongly prefer gerrit | 19:30 |
| clarkb | ya x should be avoided and ideally we also avoid renaming things shortly after creating them so having clarification upfront is a good thing | 19:30 |
| sean-k-mooney | ya ok ill start a mailing list topic | 19:31 |
| sean-k-mooney | if no one respond ill bring it up in a ptg session | 19:31 |
| sean-k-mooney | and for now ill just use my personal github to poc it | 19:31 |
| fungi | it's been a while since we've done a project rename, might be good for clearing out some cobwebs | 19:33 |
| sean-k-mooney | the more recent ones i assuem were more retirement then renames | 19:34 |
| sean-k-mooney | by the way i might need to create a couple of xstatic repos for this if i proceed with it too | 19:36 |
| clarkb | we have them all recorded in opendev/project-config iirc but I can't remember off the top of my head | 19:36 |
| clarkb | wow we're still using xstatic? | 19:37 |
| clarkb | I think the way zuul does js deps actually works ok | 19:37 |
| sean-k-mooney | .... yes | 19:37 |
| sean-k-mooney | i hate that it exists and that one of the thing i want to talk to them about | 19:37 |
| fungi | i'd love to see us get away from redistributing (eventually old and known-insecure) copies of random javascript libraries | 19:38 |
| sean-k-mooney | we are usinga 9 year old version of d3.js with a 7 year old verion of rickshaw | 19:38 |
| sean-k-mooney | so i want to use a modern version of chart.js and htmx instead fo the angular techdebt | 19:38 |
| sean-k-mooney | clarkb: how does zuul do it | 19:38 |
| fungi | a yarn lockfile | 19:39 |
| sean-k-mooney | in my poc i was just gong to use cdn links for now to avoid the packaging. ok that defintlly a topic i can bring up | 19:39 |
| fungi | run yarn to fetch the desired version of various js libs during build | 19:39 |
| sean-k-mooney | when building hte python wheel for the plugin? | 19:40 |
| sean-k-mooney | or a sperate build procees | 19:40 |
| fungi | separate, generally when installing or building container images | 19:40 |
| sean-k-mooney | ack | 19:40 |
| clarkb | fungi: sean-k-mooney: its part of the python packge build process too | 19:42 |
| fungi | oh, do we add them to wheels? | 19:42 |
| clarkb | if you run yarn first then run pip/wheel/build then we include all the js in the wheel iirc | 19:42 |
| clarkb | that may not be automatic and instead opportunistic I would have to go reread things | 19:42 |
| sean-k-mooney | dont worry about it | 19:44 |
| clarkb | its automatic. Look in setup.cfg for the _setup_hook.py and then look at that file | 19:44 |
| sean-k-mooney | its somethign i can look into when i get to that point | 19:44 |
| sean-k-mooney | ah yes https://opendev.org/zuul/zuul/src/branch/master/setup.cfg#L25 | 19:44 |
| sean-k-mooney | so here https://opendev.org/zuul/zuul/src/branch/master/zuul/_setup_hook.py | 19:45 |
| clarkb | yup those two things work together to include the js in python artifacts | 19:45 |
| fungi | though looking in the wheel it doesn't immediately appear that we're vendoring any js deps | 19:45 |
| fungi | unless they're compiled opaquely into larger nameless js blobs | 19:46 |
| sean-k-mooney | im not sure but this might only include it in the source tar | 19:47 |
| clarkb | fungi: its all in zuul/web/static | 19:47 |
| clarkb | no the js is there | 19:47 |
| sean-k-mooney | ye also have some level of webpack support | 19:47 |
| fungi | clarkb: i mean, yes there's a ton of opaquely-named js files in there | 19:47 |
| clarkb | yes taht the built javascript output | 19:47 |
| sean-k-mooney | i came across that term over the weekend but have not looked it up yet | 19:48 |
| fungi | but, like, is zuul/web/static/static/js/54.a96ce836.chunk.js jquery? who knows? | 19:48 |
| sean-k-mooney | the last time i did web deve was in php in 2012 | 19:48 |
| clarkb | fungi: the manifest file should in theory figure that out | 19:48 |
| sean-k-mooney | so its been a bit | 19:48 |
| fungi | for some reason i thought we still had a script that ran during installation or container image builds to download the js dependencies. feeling really uncomfortable with the idea that these packages are redistributing copies of random js libs we didn't write | 19:49 |
| fungi | for the same reason i've been uncomfortable with the xstatic packages | 19:50 |
| sean-k-mooney | for right now im being lazy and just using cdn links. | 19:50 |
| sean-k-mooney | fungi: ye do https://opendev.org/zuul/zuul/src/branch/master/Dockerfile#L27 | 19:50 |
| clarkb | I forget what the manifest system is called but its supposed to provide a mapping so that you can debug in your browser but also could be used to map things here I think | 19:50 |
| fungi | if there's a vulnerability in jquery, how does someone installing these packages know they need to patch it? are we even tracking that ourselves upstream? | 19:50 |
| fungi | or do we just wait for someone to report to us that we're distributing a vulnerable copy of jquery? | 19:52 |
| clarkb | well all of the deps are tracked in plaintext human readable form via the yarnlockfile you pointed out earlier | 19:52 |
| clarkb | thats a complete exhaustive list aiui and everything is pinned | 19:52 |
| sean-k-mooney | i assume there is a yarn update or similar | 19:52 |
| sean-k-mooney | to bump the deps | 19:52 |
| clarkb | what you have in the binary packeg is a binary build of those dependencies | 19:52 |
| fungi | is that included in the packages to indicate what versions were embedded in them though? | 19:53 |
| fungi | i don't see the lockfile included in the zuul wheel | 19:53 |
| clarkb | I believe the manifest system is what you have to use on the binary side | 19:53 |
| fungi | i guess in some near future state we'd include an sbom indicating what's in these packages that didn't come from the zuul repo itself | 19:54 |
| sean-k-mooney | now that i have nerd sniped you ill go o/ automtatic sboms would make lots fo folks happy | 19:55 |
| clarkb | if you open https://zuul.opendev.org in firefox with web developer tools running then go to debugger on the left side you get a pane with all the files mapped out in human readable form | 19:55 |
| fungi | but yeah, sort of dismayed that we're redistributing copies of arbitrary third-party js libs inside the packages, since we're not well-placed to evaluate and track vulnerabilities for all of them | 19:56 |
| clarkb | each of those files is also viewable in human readable form. I believe the various manifest files are responsible for making that happen to map from the optimized compressed minified code to stuff you can work with | 19:56 |
| clarkb | I think all of the info to do that is available. It may just be clunky to figure out how to work with the format they are in depending on the context | 19:56 |
| clarkb | liek I don't know how to do the mapping firefox has done for me with my editor | 19:57 |
| sean-k-mooney | it sound like just including the lock file would be less work but ya it does appare to be reable | 19:57 |
| sean-k-mooney | clarkb: chrome does it too for what its worth | 19:58 |
| fungi | yeah, i mean for the python side it's tractable because the packages aren't embedding copies of their dependencies, a package manager installs them. the same could be done for installing javascript dependencies (using e.g. npm) so that we wouldn't have to redistribute them ourselves and then consumers are on the hook to keep track of what versions of things they're installing | 19:58 |
| fungi | with npm similarly to what they're installing with pip | 19:58 |
| clarkb | but we use a lock file so its all the same versions | 19:59 |
| clarkb | you're not going to trivially edit that lockfile as an end user and change versions of random libs | 19:59 |
| clarkb | its often difficult to do that in python land but also often possible. AIUI with js its basically impossible | 19:59 |
| clarkb | things get updated in concert because bumping one thing means bumping 20 things | 20:00 |
| fungi | maybe i should just crawl back into my cave and continue pretending javascript doesn't exist, that whole ecosystem seems designed to silently propagate security vulnerabilities | 20:00 |
| fungi | same goes for docker containers really | 20:00 |
| clarkb | maybe a good middle groupnd is having regular yarn lock updates performed by a zuul job for zuul or something | 20:00 |
| clarkb | it should also be worth noting that I think you can configure zuul to not serve the js | 20:01 |
| clarkb | if you have a local build you can set up your webserver to serve that content itnead | 20:01 |
| clarkb | yes you can set the static_path configuration on the web server to change the path | 20:03 |
| fungi | i think i'm just going to start keeping a closer eye on pep 770 and then once that's settled work on a way for us to generate sbom additions that list all the javascript components | 20:04 |
| fungi | it's just taking an age for people to agree on the in-wheel implementation details | 20:05 |
| clarkb | that seems like the sort of thing that can be hsipped with a version identifier and iterative improvements can be easily made | 20:05 |
| clarkb | hell ship all the versions 10 years from now for maximum compatibility it shouldn't have a major impact on wheel size | 20:06 |
| fungi | well, it's less about file format and more about figuring out how to signal where in the wheel your sbom file(s) reside | 20:06 |
| fungi | consensus seems to be that they're going to get installed somewhere under packagename.dist-info/ | 20:08 |
| fungi | but it's not entirrely settled yet | 20:08 |
| clarkb | that seems like an odd thing to not already be settled. Wheels contain a ton of metadata already. Why can't you just put a new file alongside what exists? | 20:11 |
| clarkb | (or new directory or files) | 20:11 |
| fungi | tools need to know how to identify which file(s) is/are the sbom(s) | 20:12 |
| clarkb | right but you add the file(s) and then the tools look for them where they go? | 20:12 |
| fungi | that bikeshed hasn't been painted yet and people are still arguing over color schemes | 20:12 |
| clarkb | I shouldn't be surprised oncsidering what happend with setup.cfg and firends | 20:12 |
| fungi | it's (probably) going to be files somewhere under packagename.dist-info/ maybe in a sbom/ subdirectory maybe with a specific name pattern, but that's what's not settled yet | 20:13 |
| clarkb | going back to the concern about cross checking zuul wheel content I think you basically look at the lockfile in source form and get version/file content. Then you do the manifest mapping the chrome and firefox do to get the file content and compare. Clunky for sure but doable | 20:14 |
| clarkb | and I'm sure there are tools that you can do this on the command line with I'm just detachedfrom that world enough to not know what they are | 20:15 |
| fungi | yeah, i was thinking less about client-side evaluation and more about how would a scanner on the server where the software was deployed (or a package installer evaluating the package) identify what versions of which js libs were vendored inside it | 20:16 |
| fungi | how does the server admin find out that they've got a vulnerable version of some specific js lib that was installed embedded inside the zuul package | 20:17 |
| clarkb | you'd either check hashes of known bad content against what gets decompiled out of the manifest system or you'd say I haev zuul vfoo.bar, grab that source and check the lockfile confirm that version is vulnerable then compare (maybe with hashes again) to what the manifest map produced sourced content is | 20:18 |
| fungi | yeah, i guess unpacking and parsing the zuul/web/static/asset-manifest.json and/or zuul/web/static/manifest.json files from the wheel, for now | 20:19 |
| clarkb | essentially the same process we had to use with java during log4shell. Super clunky and not confidence inspiring but doable | 20:19 |
| fungi | though those json files are for the post-compilation state, looks like, so not as useful as a frozen snapshot of the yarn lockfile would be | 20:22 |
| fungi | yeah, log4shell is what basically spurred on the modern sbom standards | 20:22 |
| clarkb | ya adding the lockfile to the builds is probably another reasonable easish thing to do | 20:22 |
| clarkb | I'm not seeing anything obvious that is excluding it already but I didn't see it in the wheel | 20:23 |
| clarkb | oh its because zuul/web/yarn.lock is where the file is but we're building the zuul package which is everything in zuul/zuul/ including zuul/zuul/web which is the target of the zuul/web build? | 20:24 |
| fungi | right, i think it would need to be injected into the manifest | 20:24 |
| clarkb | but we might be able to put yarn.lock in MANIFEST.in and have it get included | 20:25 |
| clarkb | fungi: ^ that is an easy change to throw up if you want to write the commit message that explains it | 20:25 |
| fungi | yeah, working on it. thanks! | 20:28 |
| fungi | just need to do a bit of local testing to make sure it works as advertised | 20:28 |
| fungi | pyproject-build doesn't seem to run the setup_hook | 20:34 |
| clarkb | the hook is part of pbr | 20:35 |
| clarkb | fungi: are you missing the tools? | 20:36 |
| clarkb | that may be it. Its trying to run but skipping because the tools aren't present | 20:36 |
| clarkb | if subprocess.call(['which', 'yarn']) != 0: return | 20:37 |
| fungi | yeah, i've got it installing but unlike what's in bindep.txt the package name on debian seems to be yarnpkg | 20:38 |
| fungi | mmm, no, that didn't manage to install a yarn executable either | 20:39 |
| fungi | editing it to look for and invoke yarnpkg also isn't getting it | 20:42 |
| fungi | doesn't seem to support the expected command-line options, so must not be the same tool zuul expects | 20:44 |
| fungi | "This package provides Yarn Modern (berry). If you are looking for Yarn Classic (1.x), install node-corepack." | 20:45 |
| fungi | so maybe that's it | 20:45 |
| fungi | nope, the /usr/share/nodejs/corepack/shims/yarn from that doesn't seem to work either | 20:49 |
| clarkb | the zuul jobs seem to instll yarn with npm and don't have a version limiter | 20:55 |
| clarkb | I wonder if yarn from debian is just too old? | 20:55 |
| fungi | ah okay, so i need to get yarn installed from not the distro | 20:56 |
| clarkb | https://www.npmjs.com/package/yarn says version is 1.22.22 | 20:57 |
| clarkb | https://packages.debian.org/search?keywords=yarn says bookworm isn't all that far behind that | 20:57 |
| clarkb | then I'm not sure where trixie gets 4.0 | 20:57 |
| fungi | i think that's the "classic" vs "berry" | 20:58 |
| fungi | the 4.0 package says it installs modern yarn | 20:58 |
| clarkb | https://www.npmjs.com/package/yarn?activeTab=versions this says berry is 2.4.3 and that is 4 years old | 20:58 |
| fungi | i guess yarn 1.x is considered legacy/obsolete | 20:58 |
| fungi | https://github.com/yarnpkg/berry ys 4.x tags | 20:59 |
| clarkb | ya it kinda looks like 1.x is old/deprecated but still maintained | 20:59 |
| clarkb | then 2.x was replaced by 4.x? Not sure why npm doesn't have 4.x on it | 20:59 |
| clarkb | https://www.reddit.com/r/node/comments/19cu9bz/moving_later_versions_of_yarn_outside_of_npm_was/ | 21:00 |
| clarkb | anyway I would expect yarnpkg in bookworm to work for you | 21:01 |
| clarkb | after you symlink yarn to yarnpkg | 21:01 |
| fungi | yeah | 21:03 |
| clarkb | https://yarnpkg.com/getting-started/install so I guess nodejs ships corepack which includes some version of yarn then use yarn to update itself? | 21:06 |
| clarkb | that reddit post is interesting bceause its basically there is yarn 1, yarn 4, npm, and pnpm now | 21:07 |
| fungi | yes, i tried that one but was getting some sort of missing module error | 21:07 |
| clarkb | you could use nodeenv | 21:08 |
| fungi | the yarn shim executable script from node-corepack i mean | 21:08 |
| clarkb | (that is typically what I do because distro and nodejs often don't get along) | 21:08 |
| fungi | aha, right, https://pypi.org/project/nodeenv/ | 21:09 |
| fungi | now i remember using that | 21:09 |
| fungi | which is becoming an exercise in figuring out how to get a functioning yarn executable in my homedir | 21:12 |
| clarkb | Ithink you do python3 -m venv && venv/bin/pip install nodeenv && venv/bin/npm install yarn && ln -s $HOME/bin/yarn venv/bin/yarn | 21:15 |
| clarkb | I may have gotten the ln args backwards I always make that mistake | 21:15 |
| clarkb | https://dev.to/andreychernykh/yarn-npm-to-pnpm-migration-guide-2n04 | 21:16 |
| clarkb | I wonder if that makes sense for ap roject like zuul | 21:16 |
| clarkb | not that that solves your problem but if yarn classic is more or less a dead end then using something that isn't seems worthwhile | 21:16 |
| fungi | yeah, i've done an npm install yarn with the nodeenv i created, but yarn still doesn't seem to be installed anywhere in the env | 21:20 |
| clarkb | the zuul scripts do npm install -g yarn for a global install but I worry that will try to install to your /usr/local/bin | 21:21 |
| clarkb | but maybe it does the right thing in a nodeenv? | 21:21 |
| fungi | i was trying with debian's nodeenv package, but i'll revisit with the version from pypi after dinner | 21:21 |
| clarkb | https://docs.npmjs.com/resolving-eacces-permissions-errors-when-installing-packages-globally | 21:22 |
| clarkb | this implies global isn't truly global but rather global in a local sense :) | 21:22 |
| fungi | okay, with my nodeenv activated before invoking pyproject-build, zuul's setup_hook does find the yarn executable and seems to be using it successfully | 23:24 |
| clarkb | cool | 23:24 |
| fungi | and then i ran out of disk | 23:25 |
| clarkb | oops | 23:25 |
| fungi | Error: error:0308010C:digital envelope routines::unsupported | 23:33 |
| fungi | thrown when it's running `react-app-rewired --max_old_space_size=4096 build` | 23:34 |
| fungi | at leasst i didn't run out of disk that time | 23:34 |
| fungi | apparently that's the friendly way nodejs tells you that it disapproves of your openssl version | 23:38 |
| fungi | can be a sign that a too-new version of node is in use with an outdated js lib trying to use older unsupported crypto algorithms | 23:41 |
| fungi | with yarn>=3 there's an available plugin to generate sbom files automatically: https://github.com/CycloneDX/cyclonedx-node-yarn | 23:54 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!