| clarkb | chardet is listed in global requirements (though I'm not sure how important chardet is to openstack though) and ran across https://github.com/chardet/chardet/issues/327 just though this should be on our radar | 00:39 |
|---|---|---|
| mnasiadka | clarkb: whoa, thanks for raising attention - https://github.com/chardet/chardet/commit/7e25bf40bb4ae6884892c094080e011290494947 is interesting… complete AI rewrite of chardet | 06:24 |
| frickler | according to codesearch the only direct reference is in horizon | 09:24 |
| gouthamr | wow | 15:50 |
| * dansmith cringes | 15:53 | |
| fungi | not going to wade into that quagmire, but precedent in the usa so far is that llm-generated code is not copyrightable, so in theory the idea of relicensing (or licensing at all) wouldn't make sense if they are able to actually claim it made a "clean" rewrite | 15:57 |
| fungi | either it's derivative of the original and needs to remain compatible with the original license, or it's rewritten by an llm and doesn't get the benefit of copyright to begin with | 15:58 |
| dansmith | we're definitely in for some LLM copyright-washing of things.. "rewrite this github repo so it doesn't share a single line of identical code" -> new license | 16:00 |
| fungi | though legally speaking, it's probably effectively more like -> no license (public domain) | 16:07 |
| dansmith | that's the legal thing, but if you can erase the link, you can claim you wrote it all and thus own the copyright (and the license determination) | 16:08 |
| fungi | oh, sure. people can lie about anything they like, of course | 16:08 |
| dansmith | right.. I guess my point is, it used to be harder to lie about cleanrooming something :) | 16:08 |
| fungi | it used to be harder to lie about quite a number of things, thanks ai! | 16:09 |
| dansmith | indeed, except.. not thanks :) | 16:13 |
| fungi | right, sarcastic thanks, i meant | 16:14 |
| fungi | the latest newsletter from debian's project leader talks a bit about managing ai-assisted contribution in their community: https://lists.debian.org/debian-devel-announce/2026/03/msg00001.html | 16:19 |
| clarkb | in this particular case I suspect the main thing to be aware of on our side is that a dependency has had a 500k line rewrite and a license change and updating to the new version may have impacts to our software. Rather than necessarily needing to get into the argument over whether the license change was appropriate or not | 16:23 |
| mnasiadka | Well I think it makes sense to raise it to Horizon teams attention - since they are the only ones using it and be careful about bumping it’s version in requirements beyond the tag with 500k line rewrite | 16:49 |
| frickler | the history in https://github.com/chardet/chardet/issues/36 looks interesting. it also implies that it is a dependency of python-requests and therefore like ... everything? | 18:24 |
| fungi | doesn't appear so, unless it's vendored inside something and not installed from pypi | 18:27 |
| fungi | if i create a clean venv and pip install requests, the only other installed packages according to `pip list` are certifi, charset-normalizer, idna, and urllib3 | 18:29 |
| fungi | doesn't seem to be vendored inside any other packages either, at least a find across the entire venv's filetree for case-insensitive "chardet" turns up no matches | 18:30 |
| fungi | oh, i think charset-normalizer may be used by requests as a modern replacement for chardet | 18:31 |
| fungi | yeah, requests 2.26 (almost 5 years ago) included https://github.com/psf/requests/pull/5797 "Switch LGPL'd chardet for MIT licensed charset_normalizer" | 18:36 |
| fungi | so chardet *was* a dependency of requests a (relatively speaking) very long time ago now | 18:37 |
| fungi | so looks like this whole situation at least dates back to https://github.com/chardet/chardet/issues/36 circa 2014 | 18:40 |
| fungi | oh, that's the same one frickler linked | 18:40 |
| frickler | ah, yes, that happened after the above issue was closed. might be a possible path for horizon then, too | 18:40 |
| fungi | so basically requests solved the problem by switching to an alternative many years ago | 18:41 |
| fungi | if memory serves, ian was even working in openstack back when that issue was opened, so we might have been some of the catalyst for this entire fiasco | 18:42 |
| fungi | yeah, his resume confirms that was while he was at rackspace | 18:43 |
| fungi | so anyway, roundaboutly, once again dealing with a problem of our own creation | 18:48 |
| clarkb | I don't understand the lgpl concern in this context. Its a self contained library that performs a function. Is it really the end of the world to push your patches up if you modify this particular library? | 20:02 |
| clarkb | it isn't viral like the gpl so the scope is limtied to the library itself | 20:02 |
| fungi | some people/businesses are allergic to anything that sounds like "gpl" | 20:07 |
| gouthamr | i don't know, but, i don't see usage of chardet directly in horizon.. this is the only use: https://opendev.org/openstack/horizon/src/commit/65cb9ca7f305c5e8a7fb3761f96b3fc06be515e8/openstack_dashboard/local/local_settings.py.example#L286-L288 | 20:20 |
| fungi | agreed, it's in global requirements upper-constraints.txt and therefore duplicated into horizon's (hopefully temporary?) custom upper-constraints.txt, but the only match on master is in that example file | 20:22 |
| fungi | placement also has it installed as a workaround in the gate/perfload-runner.sh script | 20:23 |
| fungi | (but likely no longer necessary) | 20:24 |
| fungi | worth noting, puppet-horizon does use that example local_settings.py directly so relies on chardet | 20:24 |
| fungi | otherwise the only master branch appearances are in constraints files, not actually called/imported | 20:25 |
| fungi | and even the puppet-horizon settings is just supplying a logging override, so does nothing if chardet is missing (which it presumably will be if nothing is actually depending on it) | 20:26 |
| gouthamr | the placement usage is a different library: https://opendev.org/openstack/placement/src/branch/master/gate/perfload-runner.sh#L109 | 20:27 |
| fungi | oh, actually placement's script is installing cchardet, not chardet (note the extra "c") | 20:27 |
| fungi | yeah, just spotted that | 20:27 |
| gouthamr | so this might be an opportunity to cleanly excise the requirement if this was just dead code.. or a plugin of sorts that's not really necessary for horizon to run | 20:28 |
| fungi | cchardet looks like it has chardet as a dev dependency but not a direct dependency | 20:29 |
| gouthamr | ack, and placement's dependency is also a dev dependency | 20:30 |
| fungi | yep, so all of this sounds like we could just delete all references to chardet in master branches and probably nobody would even notice | 20:31 |
| gouthamr | horizon uses requests, and so maybe this settings.py example was crafted before requests itself switched to charset_normalizer.. | 20:32 |
| fungi | though the fact that it's ending up in https://opendev.org/openstack/requirements/src/branch/master/upper-constraints.txt#L9 indicates something in our transitive set of dependencies somewhere is actually depending on it | 20:32 |
| fungi | we'd need to pip install global-requirements.txt and work out from the pip logs what's pulling it in, if anyone cares | 20:33 |
| gouthamr | i don't know who could know the motivation, perhaps rdopiera or tmazur | 20:33 |
| gouthamr | oh | 20:33 |
| gouthamr | might know* | 20:33 |
| gouthamr | this was when it showed up in horizon: https://review.opendev.org/c/openstack/horizon/+/501635 | 20:34 |
| gouthamr | i think this was because of requests | 20:34 |
| gouthamr | maybe at some point, we cared about constraining this dependency, but no longer need to? | 20:35 |
| fungi | yeah, that was 4 years before requests switched off chardet | 20:35 |
| fungi | upper-constraints.txt is refreshed by doing a pip install of global-requirements.txt and then finding out what versions of which packages that pulled down. but maybe it doesn't remove old entries when they disappear, i can't recall | 20:36 |
| gouthamr | i'll poke folks on #openstack-horizon to see if they can drop this example config.. | 20:38 |
| fungi | the other maybe not obvious thing to point out is that requirements upper-constraints.txt is pinning to a 3-year-old version of chardet still, which doesn't have the new concerns | 20:40 |
| fungi | (5.2.0 uploaded to pypy 2023-08-01, while the problem rewrite seems to enter with 7.0.0 uploaded yesterday) | 20:41 |
| gouthamr | yes | 20:41 |
| gouthamr | https://github.com/chardet/chardet/issues/331 | 20:42 |
| gouthamr | "please do not foist this unnecessary controversy and risk upon your users." | 20:42 |
| fungi | granted, 5.2.0 was the most recent release until ~2 weeks ago when 6.0.0 was uploaded (2026-02-22) | 20:42 |
| clarkb | the job that installs global-requirements should say which dependency installs chardet if we remove it from the top level list (its a pip behavior to emit that info when it fetches things iirc) | 21:00 |
| fungi | i used to believe the same, but the last time i went looking it didn't actually log any of that | 21:05 |
| clarkb | I'm not finding the job in zuul's builds list probably isn't part of the periodic pipeline I guess | 21:07 |
| clarkb | oh it only runs weeklyu | 21:08 |
| fungi | yeah, https://zuul.opendev.org/t/openstack/build/65b8aed30f834ddc9c8f1073607d4582 was the most recent run | 21:09 |
| clarkb | fungi: looks like the script is called generate-constraints and it seems to eat all the output? | 21:11 |
| clarkb | the tool logs to a file but the job doesn't seem to capture it | 21:12 |
| clarkb | so ya would need to improve the tooling to better record the relationships between these packages | 21:12 |
| fungi | exactly | 21:17 |
Generated by irclog2html.py 4.1.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!