Monday, 2024-02-05

opendevreviewJay Faulkner proposed openstack/diskimage-builder master: Stop using deprecated pkg_resources API
tkajinamJayF, That deprecation is a huge killing for us :-(03:35
tkajinamI found it's being replaced in a few libraries which I was looking into during the weekend but I wasn't aware that was because of actual deprecation...03:36
JayFI mean, it is what it is :/ I only did it because Sam from gentoo community pointed it out03:36
JayFand apparently that is the stdlib solution (the pypi library is the stdlib solution before it goes into stdlib)03:36
JayFI suspect there are more elegant ways, but I'm interested to see if that works ... if it does we're probably fine (from dib standpoint)03:37
opendevreviewJay Faulkner proposed openstack/diskimage-builder master: Stop using deprecated pkg_resources API
ykarelfrickler, so now not seeing Disk quota error but it still fails05:24
ykarelrsync error: error in socket IO (code 10) at io.c(801) [receiver=3.1.3]05:24
ykarelrsync: connection unexpectedly closed (633901 bytes received so far) [generator]05:24
ykarelerrors around altimages, seems these are new content and we need to skip these ?05:36
ykarel vs
opendevreviewyatin proposed opendev/system-config master: [centos-stream] Exclude altimages from SIGs
ykarelpushed ^05:58
*** ralonsoh_ is now known as ralonsoh08:56
opendevreviewElod Illes proposed openstack/project-config master: [relmgt] Update reno when cutting unmaintained branch
opendevreviewMerged openstack/project-config master: [relmgt] Update reno when cutting unmaintained branch
opendevreviewMerged opendev/system-config master: [centos-stream] Exclude altimages from SIGs
fungisummarizing discussion from #openstack-infra, there was a launchpad and ubuntuone sso outage for a while (not sure how long) that cleared up a few minutes ago. if anyone was having trouble getting to or logging into things with, please try again and hopefully things are back to normal14:58
opendevreviewJay Faulkner proposed openstack/diskimage-builder master: DNM: CI science around gentoo fixes
fungifor SCIENCE!15:21
JayFSam is trying to get it working15:22
JayFmy task is to answer random questions and push his patches to gerrit lol15:22
JayFGentoo *needs* a tool like DIB something fierce, and I suspect once we get it working, it'll be off to the races to add it around the docs there, so lots of interest in keeping it going15:22
* JayF spent the weekend manually doing a dib-style stage4 for a cheap ARM chromebook he got last week15:23
fungivery cool!15:26
elibrokeittkajinam: for what it's worth, pkg_resources is going away no time soon at all, and quite possibly even "never". It has a few decades worth of people using it in pypi libraries that will never be updated and whose authors may be retired from the internet or dead15:28
elibrokeitbut pkg_resources is slow and non-performant anyways, and adds a very odd install_requires = setuptools, and will of course produce deprecation warnings15:29
clarkbfwiw the thing that replaces pkg_resources is slower on python2 (less of a concern for many users) and it isn't much faster on python3 (but at least not slower)16:12
clarkbunfortunately the underlying issues that made pkg_resources are somewhat inherent to the problem space16:12
clarkb(it has to scan the disk)16:12
elibrokeitclarkb: it's interesting that it would be slower *anywhere* since the big difference is supposed to be that pkg_resources scans every single package no matter what, and the replacement only scans the ones matching the filesystem glob you asked for17:30
elibrokeithmm, I suppose plugin entrypoints do require reading most things17:31
clarkbelibrokeit: yes I guess specifically for using entrypoints17:31
elibrokeitgetting metadata for a single package became *much* faster as it wasn't using the original pkg_resources functionality to its full potential17:31
fungielibrokeit: but the primary painpoint we had with projects using pkg_resources was where we already needed to scan every installed package to find entrypoints for plugins (e.g. stevedore)17:33
fungiso most of the likely speedups are going to be some variety of caching, and then you get into the usual cache invalidation challenges17:33
clarkbya I don't think importlib added any caching or package tracking db17:34
elibrokeityeah I guess the argument for moving away from pkg_resources is, in this case, mostly seen by upstream wanting to deprecate it because many of their users are better served elsewhere17:34
elibrokeitand therefore -> deprecation, somewhat compelling reason to start examining alternatives17:34
fungicorrect, and us as developers/maintainers not wanting to have to explain the constant deprecation warnings to our end users17:35
elibrokeitalso somewhat relevant here:
elibrokeit(speaking of deprecations)17:35
fungiyeah, i remember when that first came up as well17:38
elibrokeitwell, that last comment I linked to has an offer to maintain the functional API, including a "FWIW I find it much friendlier"17:39
elibrokeitso... fingers crossed that it gets restored in the 3.13 alphas?17:40
clarkbone way to make it faster would be to put entrpoint info in their targets so that when you go to import those paths they are discoverable directly without scanning the whole world17:40
clarkbinstead entrypoints are tracked on the packages that install them not the target17:41
funginot sure how that would work, since wheels just get unpacked in-place and can't install anything outside their own trees17:41
elibrokeitdepends on your architecture for registering plugins17:41
elibrokeityou don't have to use metadata for this17:41
clarkbfungi: you would have to add metadata to the extracted wheel17:41
clarkbI'm sure there are good reasons they don't do this17:42
fungiright, and something would have to check the metadata of every "extracted wheel" (install package), right?17:42
clarkbfungi: ya when you install you would reconcile it17:42
fungii thought that's what the entrypoints already were17:42
clarkbfungi: no entrypoints are metadata on the source side not the target side17:42
clarkbwhcih means when you want to load entrypoint you have to scan the entire world to see if anyone defines an entrypoint for that path17:43
clarkbthis is why pkg_resources is slow17:43
fungii guess i'm misunderstanding what you mean by source and target here. not sdist vs installed wheel?17:43
clarkbfungi: no location of the entrypoint code vs its import path17:43
elibrokeityou just use namespace packages17:43
clarkbelibrokeit: those don't work for other reasons17:44
elibrokeitfoo_plugins/ and expect people to stick a name.py17:44
clarkboh I see not actual namepsaces packages in python land17:44
clarkbbut ya17:44
fungiclarkb: but that's different for each plugin, right? so you still have to scan all the packages' metadata to find them, like with the entrypoint metadata17:44
clarkbfungi: but only at install time17:44
clarkbevery time you do a pip install or remove to add or remove code from the system you wouldh ave to reconcile the metadata17:45
fungiclarkb: "at install time" is the problem. there is no arbitrary code run at install time, wheels are just unpacked in-place unceremoniously17:45
clarkbthen runtime can just import17:45
clarkbfungi: yes pip would have to do it17:45
fungiokay, i see what you mean, add a feature to pip17:45
fungi(and all other installers, so start with a packaging pep)17:45
fungipretty sure i can predict how that discussion will play out. pip maintainers are unlikely to see the benefit, pep delegates are unlikely to want to accept the proposal without a reference implementation in an installer already, so you might have luck convincing one of the other package installers to be a "first mover" for that feature, but if you want to keep using pip you're not going to get17:47
fungithe benefits for a very long time17:47
elibrokeitclarkb: no I literally mean namespace packages in python land17:49
elibrokeitevery package with a plugin would install to the foo_plugins/ namespace17:49
elibrokeitand you pick up plugins like this:17:49
clarkbya I mean I'm not advocating for any of this17:50
elibrokeitimport foo_plugins; all_plugins = [ for m in pkgutil.iter_modules(foo_plugins.__path__, foo_plugins.__name__+'.')]17:50
clarkbI just htink that reconciling the links at install time and taking the hit at install time is better than taking the hit at runtime17:50
elibrokeitall_plugins is a list like ['foo_plugins.tech1', 'foo_plugins.tech2']17:50
elibrokeitpkgutil just scans each foo_plugins/ directory on sys.path for importable namespace sub-modules17:51
fungiclarkb: the other area this touches on is that the python packaging community is slowing coming to the conclusion that python packages as we know them now are really only useful for installing libraries/dependencies, not for installing applications, and that to install an application (say openstackclient) you should use some other kind of packaging and installation17:52
elibrokeitI have been saying that for a long time, mostly in a dig at the python packaging community for not wanting to do anything else17:52
fungiwhich has been a lot of the recent pushback on proposals which would ease application deployment17:53
elibrokeit(because spoiler: install worked quite fine for this)17:53
fungibasically they're starting to consider these sorts of concerns "out of scope" for tools like pip17:53
elibrokeit> Maybe I'm missing something, and historically people somehow used Python tools like distutils to install Unix system files. I don't think that was ever intended/supported use (I've heard a lot of Linux distro maintainers complaining that people should use the Linux management tools, not Python, to alter system-managed files) so there's no "modern way" of doing this, and if it was17:55
elibrokeitpossible in the past then I guess sorry, it's no longer supported...17:55
elibrokeitI had some... comments on that fascinating take17:55
elibrokeitthey seem to have been under the impression that it was NEVER possible17:55
elibrokeitnot that they're starting to declare it out of scope with the new wheel based approach17:55
elibrokeitit took major effort and investment to make distutils / setuptools handle data files for unix directories correctly17:56
elibrokeitthey *repealed* that effort and now don't realize that it ever worked?17:56
fungithey're suffering from a semi-recent and growing phenomenon in open source communities... the old guard have burned out or gone, and there was insufficient continuity to impart their history to those who came after17:58
fungisee also: chesterton's fence17:58
clarkbmaybe someone will write a cargo plugin for python packaging17:58
fungiclarkb: the irony is, now, folks are convinced that python packaging and cpython distribution need to be redone to mimic cargo/rustup17:59
fungiand switch to vendoring everything like all good rust (and go) projects do17:59
fungibecause wcpgw17:59
fungialso distros are unnecessary now that we have containers. because, um. well just because, right?18:00
clarkbfwiw the cargo user expereicen is really good. I'm not sure that necessarily requires static linking though18:03
clarkbI think the things that make it good are having one way to package things rather than a million with a bunch of arbitrary code mixed in, an init method that gets you a working start point18:03
fungiit's also a very different language with a very different usage model18:03
fungioh, and yes dynamic linking is still apparently a challenge for rust, based on recent discussion on the debian-devel ml18:04
fungisupposedly the rust maintainers don't want to commit to a stable abi, is the root of the problem18:05
elibrokeitthere's a crABI proposal but it appears to have lost steam18:09
elibrokeitI don't think the cargo user experience is very good at all though18:09
elibrokeitpeople superficially think it is really good because thus far, cargo has been able to get away with a userbase that only wants to build executables and sometimes libraries, in a mostly-rust codebase with occasional bits of C/C++ wrapped up in a rust API as a crate dependency18:11
clarkbI think the key bit is consistency18:11
elibrokeitany project that wants to do something as trivial as create a manpage by converting a markdown/rST/asciidoc file to roff, learns instantaneously, on the spot, that it's hopeless18:11
clarkbyes it is pluggable but for the most part everyone is using the same tool and it does the basic things one way and everyone does it that way18:12
clarkbthis is where python really suffers18:12
clarkbthere is too much choice and tribal knowledge18:12
fungithat's because rust is still relatively young18:12
clarkbya that certainly helps18:12
elibrokeitpep 517 proposed to solve this problem, and only made it 50x worse...18:12
elibrokeitproliferation of build backends that are all just flit with slightly different config file names18:13
clarkbmeanwhile you can't build a wheel with a custom build dir18:13
fungiit's already beginning to leak into other ecosystems. e.g., pyca/cryptography doing lots of stuff in a rust-based extension now18:13
elibrokeitcryptography shows exactly how well that works in mixed rust + !rust projects...18:15
clarkbinfra-root: I have two gitea related changes queued up. The first is to update the db connection limit and the second is to upgrade gitea a minor release version up from where we are. I don't think the db update will auto restart the services (and I'm not sure we want it to). I'm happy to land these changes in either order and manually restart things gracefully for the db change.18:15
clarkbThat way we don't conflate things if there are issues. Any opinions on how we want to approach these two chagnes?18:15
fungiand the resulting madness that is
elibrokeituse setuptools, setuptools-rust, and a set of eclectic and custom toml settings in pyproject.toml18:15
fungifrom a distribution's point of view, any python-based project that uses cryptography now has just opted into a build dependency on a dozen rust libs18:16
fungiand upstream cpython is busily evicting the last remnants of cryptographic functions from the stdlib owing to the increase in pyca/cryptography's popularity18:17
fungiso soon (if not already) that's basically anything with cryptographic needs written in python18:18
clarkbI think those issues are orthogonal to the developer experience using cargo though. My point is I think the lessons aren't necessarily "static linking good" but instead "make getting started easy, don't make users/developers understand super esoteric things riddled across 20 years of docs and 3 deprecated libs just to install a simple python tool, and act consistently when making18:18
clarkbdecisions (dep solving is good))18:18
clarkbpython has improved some of this and some of it is just as bad as ever18:19
fungiand some of it has gotten worse18:19
clarkbthe new warning about not specifying the distribution location or whatever it complains about in all our repos now is a good example of this18:19
clarkbthe docs on that don't help me and I'm a person that has dealt with python packaging for 15 years18:19
elibrokeitI think the answer to that was honestly just either saying "use flit instead of setuptools" or campaigning for people to use setup.cfg instead of setup.py18:19
elibrokeitbut they decided the big takeaway was that people felt stifled and wanted more choice :D18:20
elibrokeitnot specifying the distribution location?18:20
clarkbbsically you're left with copying the example of the day for a new project, editing it until you think it works then in a year or two giving up on being up to date any longer18:20
fungipbr (distutils2 really) solved that long ago, it's unfortunate that it took setuptools 10 years to copy the setup.cfg idea, and by then pyproject.toml was basically already coming into existence in parallel18:21
clarkbelibrokeit: setuptools wants you to specify the location of the package explicitly or something like that18:21
clarkbelibrokeit: when you don't (which many packages that are older than a couple years don't) you get a warning with a poitner to some docs. If you read the docs it doesn't explain anything just tells you to do X or Y but without background for understanding18:21
JayFclarkb: ++++++++++ to the "the docs don't help me and I've been working with python packaging for years and years"18:21
JayFLots of problems I didn't know exist getting solved at the expense of re-causing problems we had worked around18:22
elibrokeitah, that would be this change:
clarkbcompared to cargo you do a cargo init and you're off to the races18:23
JayFfungi: In some ways, elibrokeit has acted as a bridge preventing some of that ... burnout/ageout-caused-brain-drain in some communities. New generation of developer making a point to understand why things are the way they are before going about willy-nilly changing things18:24
JayFelibrokeit: ^ that may be the nicest thing I'll ever say about you, go get it framed :D 18:24
*** mtreinish_ is now known as mtreinish18:24
clarkbthat said I do apprecite it is difficult to pull back from an arbitrary code execution packaging system into something more rigid and well defined18:25
clarkband they have been making improvements.18:25
elibrokeitI think the bulk of that work really came from the addition of setup.cfg based config18:26
JayFYeah, I see most of this as people trying to reduce the scope of the project they work on18:26
JayFMost of us know the pain of working on a project with huge scope with not enough people to run it :)18:26
elibrokeitI am actually a bit confused by the automatic discovery thing18:26
JayFand just lots of evidence there's not enough context to know what scope to keep18:27
elibrokeitit seems to be tied to using pyproject.toml, but also tied to... flit envy?18:27
elibrokeitI think they want to make setuptools as simple as it is when using flit18:27
clarkbelibrokeit: yes the discovery thing is what I18:27
elibrokeitso that you can have a blank project with no config and it automatically detects stuff for you18:27
clarkbthat is what I'm thinking of18:27
clarkband yes it is confusing18:27
clarkbI think the issue we have is that the dir structure is such that auto discovery has to make a guess and the current guess it makes is working for us18:28
clarkbwhat isn't clear to me is what it wants us to supply for explicit discovery inputs18:28
JayFelibrokeit: this is very akin to the GUI vs CLI chat in -chat today. Typically that kinda autodiscovery is going to make the simpler case better at the expense of the complex cases.18:28
elibrokeitit is *supposed* to not try that nonsense, if you already used packages=find: or packages=list,of,packages18:28
clarkblooking at the two gitea changes only the db conn limit bump has multiple +2s. I'll go ahead and approve that one now then and plan to do the restarts necessary to pick that up after it applies18:34
fungidid i miss reviewing one?18:34
clarkbno you're on both18:35
clarkbone thing I noticed is that the mm3 mysqld conf uses a 99- prefix18:35
clarkbany idea if I dno't think this is necessary, but maybe I should update the change I wrote for gitea to do the same thing?18:36
clarkbya includedir says it reads them in alphabetical order18:36
clarkbso not strictly necessary but might be a good idea?18:36
fungiclarkb: it's only relevant if there ends up being more than one config fragment in the same directory adjusting the same value18:37
clarkbya and there are no other files in that dir currently18:37
fungibut for i did copy that behavior just for added surety18:37
clarkbI'll respin it for consistency but maybe not bother with full fledged rereview?18:37
opendevreviewClark Boylan proposed opendev/system-config master: Increase gitea db connection limit
clarkbI'll approve that if it passes testing18:40
clarkbfungi: I left a small note on your question on the keycloak change. Otherwise that change lgtm18:42
fungiand yeah, i guess we proved that the configuration works, now it's just a matter of figuring out the appropriate envvar names for that18:45
clarkbfungi: system-config/playbooks/roles/gitea/tasks/main.yaml has an example18:47
clarkbat the end of the file18:47
fungiclarkb: oh, i also misunderstood where you meant and thought you were saying to keep it out of the command string in the docker-compose file, which may also be important?18:49
fungii suppose that might also get logged or appear in /proc18:50
fungier, well, envvars also appear in /proc but might show up in process listings i mean18:51
clarkbfungi: my main concern is cron sending email with that content in it. Not an issue here but that is why I chose that form for gitea and other places18:52
fungiyeah, makes sense. as for the test itself, would it be better with a targeted query like the example in my comment?18:52
fungiwith the envvar substitution of course18:53
clarkbI don't think it is functionaly different since both check for db contents18:53
clarkband maybe I misunderstood the initial question18:53
clarkbya I think I did. I thought you were asking if we should use mysql on the host vs docker. Now realize its mysqldump + grep vs a query18:53
clarkbeither are fine I think18:53
fungiinstead of doing a mysqldump and checking the massive returned string for a specific substring, using a specific sql query to just return the data we're interested in looking for18:54
fungii simply didn't want to respin the change to adjust that one test unless others thought it was a good idea. if i'm updating it anyway for the envvar then i can also switch to an sql query while i'm at it18:54
clarkbI think both are fine and not worth a respin.18:56
fungiclarkb: as for the docker-compose file, what's your opinion on the --db-password={{ keycloak_db_password }} templating for the container command parameters list?18:56
clarkbya I guess the issue there is ps will show it. But still not an issue for cron + email18:56
clarkbdid we determine if the env vars would work?18:56
fungii can see an argument for figuring out what the envvar equivalent is so that the raw password string for the database account isn't in the process list18:56
clarkbya I think that would be prefereable if we can figure that out18:56
fungii'll look and see if i can figure it out18:57
opendevreviewJeremy Stanley proposed opendev/system-config master: Upgrade to Keycloak 23.0
fungilooks like the launchpad outage is still affecting their ppa system:
fungi"failed to fetch PPA information, error was: HTTP Error 503: Service Unavailable"19:26
clarkbwhy does the mirror job run for 907141? Maybe due to the group and host var updates?19:29
clarkbwe might be able to edit the file list for the job(s) and make that better19:29
fungialso that's the arm64 version of the job specifically19:30
fungibut the amd64 version is probably running too, i haven't looked19:31
clarkbthey match - roles/19:32
clarkbI don't think that is necessary19:32
clarkbI'll push a change to remove that line19:32
opendevreviewClark Boylan proposed opendev/system-config master: Don't run mirror jobs when roles/ updates
clarkbetherpad just made a new release that modifies how plugins install19:51
clarkbwe'll have to investigate if that impacts our image building19:51
fungippa outage is impacting system-config-run-base and other jobs too
fungidue to the "ensure-pip: Install backport pip" task20:04
fungiwhich we'll be able to drop when we're free of xenial, at least20:04
clarkbya it hit my gitea change as well20:05
fungithat's where i noticed it, actually20:06
fungianyway, looks like the DB_PASSWORD envvar isn't plumbed through according to
fungii'll try KC_DB_PASSWORD to see if it's any better20:09
opendevreviewJeremy Stanley proposed opendev/system-config master: Upgrade to Keycloak 23.0
fungiyeah, i should have remembered i have a held node i could just try it on20:19
fungibut also i just realized there's still an ansible task in there unnecessarily creating /var/log/keycloak that isn't used for anything20:20
opendevreviewJeremy Stanley proposed opendev/system-config master: Upgrade to Keycloak 23.0
fungithe only remaining unknown with that change is whether all the nested quoting to run an sql query from bash in docker via python will actually translate. i have quoting nested four layers deep20:23
opendevreviewJay Faulkner proposed openstack/diskimage-builder master: Fix various minor issues with Gentoo; make CI pass
opendevreviewJay Faulkner proposed openstack/diskimage-builder master: Re-enable voting on Gentoo DIB CI job
*** dasm is now known as Guest173920:33
opendevreviewJay Faulkner proposed openstack/diskimage-builder master: Fix various minor issues with Gentoo; make CI pass
opendevreviewJay Faulkner proposed openstack/diskimage-builder master: Re-enable voting on Gentoo DIB CI job
*** Guest1739 is now known as dasm20:38
fungiwell fudge, my db query isn't working as-written, likely something to do with escaped quotes20:43
fungiaha, it's because you can't nest escaped single quotes inside unescaped single quotes20:51
fungiyou can nest escaped double quotes though20:51
fungihope nobody needs a toothpick, i'll be using them all20:53
opendevreviewJeremy Stanley proposed opendev/system-config master: Upgrade to Keycloak 23.0
fungialso, the ppa outage seems to be resolved, for now anyway21:06
clarkbtime to recheck all the things21:09
fungiargh, it looks like \\\" is getting turned into \\" not \"21:22
fungimaybe making it a raw string will help21:24
opendevreviewJeremy Stanley proposed opendev/system-config master: Upgrade to Keycloak 23.0
clarkbfungi: I think the gitea db change will pass testing this time (happy PPAs). I'll go ahead and approve it now21:46
clarkband then I'll restart services once applied21:47
fungii'm around to help, just whipping up dinner real quick21:50
clarkbthe main thing is the coordinatioj between the proxy and gitea services so that end users don't notice and gerrit replication doesn't fail21:51
clarkboh I need to get a meeting agenda out too22:03
clarkbI think I can trim the inmotion/openmetal item. I'll add an ote about gitea things. Anything else to add or remove or edit?22:04
opendevreviewJay Faulkner proposed openstack/diskimage-builder master: Fix various minor issues with Gentoo; make CI pass
opendevreviewJay Faulkner proposed openstack/diskimage-builder master: Fix various minor issues with Gentoo; make CI pass
opendevreviewJeremy Stanley proposed opendev/system-config master: Upgrade to Keycloak 23.0
clarkbok some initial meeting agenda edits are in22:37
clarkbI'll send it out at the end of my day giving others time to make edits if they want to22:37
opendevreviewMerged opendev/system-config master: Increase gitea db connection limit
clarkbthat appears to be in place on gitea09 now. i'll work on the restarts now23:30
clarkbgitea09 is done. I connected to mariadb and ran `select @@global.max_connections;` to verify the limit bumped (and it did)23:34
clarkband now all six cluster members should be running with the updated db configs23:41
clarkb#status log Gracefully restarted the gitea cluster to pick up new database connection limit configs23:42
opendevstatusclarkb: finished logging23:42
clarkblast call on meeting agenda items. I'll send that out in about 10 minutes probably23:48
fungigitea's still working for me23:53
fungisorry, stepped away for a few just as it deployed23:54
clarkbno worries it all went well23:56
opendevreviewJeremy Stanley proposed opendev/system-config master: Upgrade to Keycloak 23.0

Generated by 2.17.3 by Marius Gedminas - find it at!