| *** Unknown123 is now known as Mike-- | 05:29 | |
| fungi | clarkb: i see pbr-7.0.3-py2.py3-none-any.whl listed there? keep in mind there's a block of text between the sdist link and list of wheels for the release | 12:09 |
|---|---|---|
| fungi | https://pypi.org/project/pbr/#pbr-7.0.3-py2.py3-none-any.whl has the details for it | 12:11 |
| clarkb | fungi: oh so it is what a weird ui thing | 13:21 |
| fungi | i think it's designed so that projects with 100 different platform-specific and python-version-specific wheel files can be filtered, but yes when you only have one wheel file it sort of disappears visually in the noise | 13:24 |
| -opendevstatus- NOTICE: Anubis is now deployed on our Gitea backends, and things are back to working normally though you may notice an Anubis screen flash briefly when starting to browse opendev.org; any jobs which failed prior to 15:00 UTC today can be safely rechecked | 15:35 | |
| fungi | would it be appropriate for me to mass-propose patches to remove precommit from the tox configs in all projects, until teams have time to work out how to make it not try to fetch git repos on its own over the network for things where it should be using local checkouts from zuul or packages from pypi? | 16:34 |
| fungi | i don't use the tool and have limited interest in learning how to make it align with the way our ci system expects jobs to be optimized, but also am tired of teams complaining about their jobs failing any time we have a problem with the git frontends when that situation could have been avoided entirely | 16:35 |
| dansmith | I don't have it enabled locally so it's not _actually_ in my way, even though I sort of object to the entire premise of it | 17:11 |
| dansmith | if it's doing anything negative with CI by default, I'd say that's justification to rip it out on its own | 17:12 |
| spotz[m] | I think it's worth a shot? | 17:19 |
| JayF | I think proposing the patches won't get you what you want though, we need the discussion if you want them to land. | 17:19 |
| JayF | In fact, mass proposing those patches before someone was able to read an ML post about them, describing the pain, for instance, might anchor people into being less likely to merge them | 17:19 |
| spotz[m] | Would it be worth attempting the change in one area to see if it even fixes the issue for sure? | 17:21 |
| JayF | It's a design problem with pre-commit, really | 17:22 |
| dansmith | well, proposing them and letting projects fix or abandon them seems fine to me, as long as there's some outcome | 17:23 |
| JayF | The question isn't "will it help with git load" it's "are the project teams OK with the tradeoff" (and maybe a little of "opendev doesn't wanna let the teams make that tradeoff anymore for $reasons") | 17:23 |
| dansmith | it's really "how you get stuff done on the internet" which is ... "propose a patch that isn't quite right so someone gets mad and fixes it properly" | 17:23 |
| JayF | A bad PR is the fastest path to getting someone to write a good PR :D | 17:24 |
| dansmith | the github corrollary ^ | 17:24 |
| dansmith | right, it's the opendev infra impact that needs some action regardless | 17:24 |
| fungi | it's really mostly that we designed robust solutions to supply artifacts (packages, git repositories, vm images, et cetera) to jobs so that they wouldn't run into so many of the problems inherent with fetching them over the internet on every run, so it's frustrating for projects to ignore those capabilities and introduce fragility into their jobs, then complain about the impact | 17:27 |
| fungi | from that decision | 17:27 |
| sean-k-mooney | fungi: is the main issue current the git clone or is it beyond that | 17:27 |
| sean-k-mooney | fungi: mainly askign to understand the current probleme yoru trying to solve | 17:28 |
| fungi | it's that it's cloning hacking over the internet and then checking out the 8.0.0 tag. we have solutions optimized for installing a normal hacking 8.0.0 package or from a locally-supplied git checkout on the test node's filesystem | 17:30 |
| sean-k-mooney | ack | 17:31 |
| fungi | so when we have some incident impacting our git offload servers intended for normal users, nova's zuul jobs are breaking unnecessarily and i have very little interest in helping solve that problem | 17:31 |
| sean-k-mooney | well its not just nova but ya ok i get the issue | 17:32 |
| sean-k-mooney | i thikn we could have zuul pre prepar them but im not sure that folks will want the extra infra of that | 17:33 |
| fungi | yes, nova was an example. my even bigger frustration is that the same antipattern got pushed out to a lot of other repos without considering the impact | 17:34 |
| sean-k-mooney | the main regression woudl be the extra linting that we do only via pre-commit | 17:34 |
| sean-k-mooney | perhas a better option woudl be to only use local hook or tool script althought that makes sharign ahrder | 17:35 |
| sean-k-mooney | as long as we dont regresss the linting and folks cna still use it lcoally we coudl go back to keepign the lint job and pre-commit config in sync | 17:37 |
| sean-k-mooney | it wasnt really much addtional work | 17:37 |
| sean-k-mooney | although we woudl need to replace https://github.com/openstack/nova/blob/master/.pre-commit-config.yaml#L3-L36 with something | 17:37 |
| sean-k-mooney | perhaps hackign checks in some cases? | 17:38 |
| fungi | the basic design principle with these systems is that projects should be able to continue testing and merging changes normally even if the opendev.org site is offline (and also if test nodes are unable to reach the gerrit server, for that matter) | 17:40 |
| sean-k-mooney | right htat why we have the error on clone flag for devstack | 17:40 |
| sean-k-mooney | ideally all content is either coming form a mirror or if its git prepared by zuul | 17:41 |
| sean-k-mooney | the simiple 2 secodn chang ewoudl be to use the github mirror | 17:41 |
| sean-k-mooney | but that not realy ideal either | 17:41 |
| fungi | yes, and zuul can supply clones from other sources like github too, so usable even if test nodes can't reach that | 17:42 |
| sean-k-mooney | so what i need to check is pre-commit i belive suprpot a env var for its "home" location | 17:59 |
| sean-k-mooney | which is wher eit loosk for the relevent repos | 17:59 |
| sean-k-mooney | i want to check can we just poitn that at zuul cloned repos or perhaps copy them | 17:59 |
| sean-k-mooney | the answe ris proably no | 17:59 |
| sean-k-mooney | but if that works it would be trival to add hacking to the pep8 job as a required repo and move it to the right place before runign or somethign like that | 18:00 |
| fungi | yes, that sounds viable, if precommit caches git repositories in a known location | 18:03 |
| opendevreview | Merged openstack/openstack-manuals master: Add note for reno link update for release task https://review.opendev.org/c/openstack/openstack-manuals/+/982088 | 18:37 |
| sean-k-mooney | instead of doing someting liek .cache/pre-commit/repos/opendev.org/openstack/hacking | 18:46 |
| sean-k-mooney | tis generating a repo###### has dir name | 18:46 |
| sean-k-mooney | and it has a sqlite db to know where everythign is | 18:47 |
| sean-k-mooney | prek dos someitng imilar so we shoud have to jsut sed the .pre-comite-cofnig.yaml to update the repo paths to file://${home}/src/opendev.org/openstack/hacking | 18:48 |
| sean-k-mooney | that woudl nto be hard to do i jsut dont know if we want to do that in genreal | 18:49 |
| sean-k-mooney | it woudl be triival to do for just the hackign repo but im thinking of the other repos that we dont currenly have in zuul | 18:49 |
| sean-k-mooney | but hackign is the only one hitting opendev.org i think today | 18:49 |
| clarkb | what I don't understand is why they are so resistant to just using pypi | 18:50 |
| sean-k-mooney | for the hook distirbution? | 18:51 |
| clarkb | yes. I think some of the other language systems do pull from the upstream package repos too | 18:51 |
| clarkb | not only that but they pip install the git dir and that pulls everything else from pypi | 18:51 |
| sean-k-mooney | yes so for python hooks it will use pip to install the deps fi you list any | 18:51 |
| sean-k-mooney | i.e. that how hacking and flake8 are isntall via the hackign pre-commit hook | 18:51 |
| clarkb | this is why I mentioned the work around is to have a shim in git that is just a list of things you want installed then pip install that and it downloads what you really care about | 18:51 |
| sean-k-mooney | yep | 18:52 |
| sean-k-mooney | so puting the hackign hook into its own tiny repo | 18:52 |
| clarkb | but its weird if that is the behavior anyway to avoid just going straight ot pypi in the first place | 18:52 |
| sean-k-mooney | that is just the hook | 18:52 |
| clarkb | right I'm saying why can't you define the hook in your local yaml file that says this is where the code is | 18:52 |
| clarkb | rather than a git repo it could be pypi | 18:52 |
| sean-k-mooney | you can | 18:52 |
| clarkb | ok then why are we not just doing that? | 18:53 |
| sean-k-mooney | the reason we are pullign hackign was jut to not copy it | 18:53 |
| sean-k-mooney | taht woudl be tirival to fix actully | 18:54 |
| clarkb | it just seems like this system wasn't really designed to be use outside of your laptop where you would run it once and cache it. Reminds me a lot of the sharp edges of docker fwiw. Oh you want to mirror an image? You must host it on docker hub can't have mirrors for anything else | 18:54 |
| sean-k-mooney | we coudl baisly copy https://github.com/openstack/hacking/blob/master/.pre-commit-hooks.yaml | 18:54 |
| sean-k-mooney | and run it as a local hook | 18:54 |
| clarkb | similarly here: oh you want to use code hosted in the python package repo sorry can't do that | 18:54 |
| sean-k-mooney | clarkb: well it was desced for ci | 18:55 |
| clarkb | no it wasn't | 18:55 |
| sean-k-mooney | but with the asusmtion that the cache is on a nfs/afs share | 18:55 |
| clarkb | or if it was it was by someone who has never designed anything for ci | 18:55 |
| sean-k-mooney | that is used acrss runs | 18:55 |
| clarkb | I say this as someone who has maintained CI systems since 2010 | 18:55 |
| sean-k-mooney | ant htat what bites us | 18:55 |
| clarkb | anyway what I'm saying is you should be able to point pre-commit at hacking on pypi and have it load /hacking/blob/master/.pre-commit-hooks.yaml from there which is in the package | 18:55 |
| clarkb | and if it isn't in the package then that is a package bug we can fix | 18:56 |
| clarkb | the fact that precommit still doesn't have this ability is a bit mind boggling to me | 18:56 |
| sean-k-mooney | ya so since hackign is the souce of bleedign let me create a tiny patch ot make that not git cloen | 18:56 |
| clarkb | I seem to recall looking into this the last time it came up and that file isin the package but things may have changed since then | 18:57 |
| sean-k-mooney | right but the isssuue is not with the file its that git is the protical that the tool uses for the distibutoion of the hooks | 18:58 |
| sean-k-mooney | taht likely shoudl be changed | 18:58 |
| clarkb | right I've been saying this since the first time ssbarnea was all hyped about pre commit. That this is a fundamental flaw and it shouldn't be that difficult to fix | 18:58 |
| sean-k-mooney | but it the choice they roginaly made the same as the go echosystem did | 18:58 |
| clarkb | but as I see this as a fundamental flaw and existing tools work just fine for me I have zero motivation to fix it for them | 18:58 |
| clarkb | we rejected pre commit on this basis years ago when ssbarnea was trying to get us to sue it | 18:59 |
| clarkb | and we've been fine if I'm honest | 18:59 |
| clarkb | but openstack didn't think my protests were enoguh reason to find another tool or fix the tool and prceeded anyway | 18:59 |
| clarkb | and this is not the first time w've had this discussion because its not the first time this has cuased us issues | 19:00 |
| clarkb | the problem is that opendev manages to fix the backend issues which removes the motivation to fix the flaky job sources | 19:01 |
| clarkb | and then we rinse and repeat this some time in the future when things go sideways again | 19:02 |
| sean-k-mooney | i dont think i was involev in those converstaion but lets see if we can fix things goign forward | 19:04 |
| sean-k-mooney | for what its worth i was execting the git clone to be cached vai the http proxies we have at teh providers | 19:05 |
| sean-k-mooney | but i guess the s in https prbably is preventign that form beign a thing | 19:05 |
| sean-k-mooney | well that and most proxies doent cache git conetnt by default i guess | 19:06 |
| sean-k-mooney | clarkb: so i belive we can fix hacking like this https://review.opendev.org/c/openstack/watcher/+/983925/1/.pre-commit-config.yaml | 19:12 |
| sean-k-mooney | that will get rid fo the clone and pull a pined version of hackign form pypi which will use our pip caches | 19:15 |
| sean-k-mooney | although in need to test that quickly by localy violateing a hackign rule | 19:16 |
| sean-k-mooney | clarkb: ok ya that works if i add "vim: set tabstop=4 shiftwidth=4\n#\n#\n#\n#\n#" to a file the local hackign version flag H106 | 19:21 |
| sean-k-mooney | so we coudl apply that change to all relevent repos as a first step | 19:22 |
| clarkb | you're still heavily dependent on other external services look like but I guess I'm not in the queue to fix those when they break | 19:25 |
| sean-k-mooney | we can transform any fo the ones that are just invokeing a cli in the same way if they are on pypi ectra | 19:26 |
| sean-k-mooney | that will cover codespell bandit sphinxlint and ruff/autopep8 | 19:27 |
| sean-k-mooney | for the thing likd detec-private-keys well that can jsut be a local tool script | 19:27 |
| sean-k-mooney | of if we used prek i think all ormost of the https://github.com/pre-commit/pre-commit-hooks hooks | 19:28 |
| sean-k-mooney | are built into the tool so they are provie with just the pip intsll of perk | 19:28 |
| sean-k-mooney | so we coudl likely remove almost all the clones and just vendor the few scrips that are left | 19:28 |
| sean-k-mooney | we have tools directores in most fo the repos where we keep things like this already it would just take a littel mroe time to do | 19:29 |
| sean-k-mooney | clarkb: ill leave this patch as it is as a minimal refence and ill see if i can quickly transform the the other low hanign fruit in a follwo up | 19:30 |
| sean-k-mooney | we will loos the "pre-commit autoupdate" but honestly that fine we bump these thing like once or twice a release we can jsut check when we do | 19:31 |
| sean-k-mooney | clarkb: so https://pypi.org/project/pre-commit-hooks/ is a thing i think we can skip the vendoring of the srcipt entirly adn jsut call them form local hooks | 19:47 |
| sean-k-mooney | that what im tryign to make work now at least | 19:48 |
| sean-k-mooney | it provides them all as console scripts https://github.com/pre-commit/pre-commit-hooks/blob/main/setup.cfg#L30-L62 | 19:49 |
| sean-k-mooney | clarkb: lookign at the repo history i think that has been posibel at least for those hoosk for at least 7 years | 19:50 |
| sean-k-mooney | there is just no documentaiotn for that at least not obevious docs | 19:51 |
| clarkb | I wonder if this is a happy accident and we're using something that wasn't intended for this purpose | 19:52 |
| clarkb | or its the classic forgot to document it problem. | 19:52 |
| sean-k-mooney | well they are intentionally exported as console_scripts | 19:52 |
| sean-k-mooney | so that to mee looks liek jsut never docuemtned | 19:53 |
| fungi | if memory serves, it was roughly 8 years ago that we called out this problem, but i guess enough people have cycled out and in that they forgot it was every discussed and reimplemented it fresh | 20:09 |
| sean-k-mooney | it looks like it was posibel form the sart the older console_script were added 12 years go | 20:10 |
| sean-k-mooney | it was just never docuemented or it was remvoed | 20:10 |
| sean-k-mooney | this woudl not be the normal way to use hooks as you hae to duplcit the hook logic in yoru local repo | 20:11 |
| sean-k-mooney | but the hook logic is juca calling a cli 99% of the time and is trivial | 20:11 |
| fungi | though this conversation has affirmed my suspicion it's a tool designed for the golang ecosystem (where dependencies are managed in a very particular way) that people are trying to apply to python projects (where dependencies are typically handled much differently) | 20:13 |
| sean-k-mooney | fungi: clarkb im tryign to convert watcher now but i messed up by tryign to do too much in one go (moving some check to ruff that it can do natively) | 20:13 |
| JayF | fungi: a lot of new tools are golang/rust ecosystem shaped; that's why they are git-first and package-second :/ | 20:14 |
| JayF | fungi: it's a pain point gentoo devs hit frequently when packaging things | 20:14 |
| fungi | i guess it'll all work out once we finish rewriting all of openstack in rust. i hear that'll be ready any day now ;) | 20:14 |
| clarkb | JayF: the secret is that go has a package system. Its the google go repo cache | 20:14 |
| clarkb | it just happens to use git for its protocol but it acts like package provider preventing unecessary git operations to the actual git locations | 20:15 |
| sean-k-mooney | the comuity proxy ya | 20:15 |
| clarkb | sr.ht helped them impriove it after they got ddos by all the go mod users | 20:15 |
| JayF | It didn't always work that way though | 20:15 |
| sean-k-mooney | you technially can turn that off but no one does unless there is a coperate firewall in the way | 20:15 |
| JayF | It's always been git based, but I think the proxying stuff came later | 20:15 |
| JayF | Realistically I haven't done meaningful greenfield go development in long enough I just don't know | 20:16 |
| sean-k-mooney | for what its worht pre-commit to my understanding is primarlly a python thing | 20:17 |
| fungi | weird | 20:17 |
| sean-k-mooney | makefiles for all there evil and i guess good is a go thing | 20:17 |
| sean-k-mooney | or maybe jsut a redhat thing | 20:17 |
| sean-k-mooney | redhat go proejct have make for some reason | 20:18 |
| fungi | makefiles are a c thing | 20:18 |
| sean-k-mooney | i dont mind simle make fiels | 20:18 |
| fungi | or at least were 30 years ago when c was what we had | 20:18 |
| fungi | battle-tested and solid | 20:19 |
| sean-k-mooney | but when you right a devstack equvlent to install openstack on opensift in it that a little bit more make then im ok with in my life | 20:19 |
| fungi | at one point clarkb had a poc to replace tox with make | 20:19 |
| fungi | i was +2 on the idea | 20:19 |
| sean-k-mooney | i coudl get behind justfiles which is basiclly make but where tab vs spaes does not change the bevhior... | 20:20 |
| clarkb | its still in the zuul gerrit change list somewhere | 20:20 |
| sean-k-mooney | https://review.opendev.org/c/openstack/watcher/+/983933/1/.pre-commit-config.yaml | 20:24 |
| sean-k-mooney | so its kind of ugly because i have too repeat things | 20:24 |
| sean-k-mooney | but with that watcher shoudl nolonger use git cone | 20:24 |
| sean-k-mooney | i obviously need to test that more but that every hooks converted to pullign the deps form pypi | 20:25 |
| sean-k-mooney | i shoudl clean up a few of the comments but that the simpletst 1:1 traslation. | 20:26 |
| clarkb | the thing make (and its descendents) does that tools like tox fail at (in my opinion anyawy) is the whole depednecy resolution don't do work you don't need to d othing | 20:28 |
| clarkb | by making that an upfront first class thing you have to think about it. The downside is you have to think about it and it can get complicated. But if you want fast tools that is a good place to start | 20:28 |
| fungi | yeah, simplistically, each section makes a target. if that target already exists then make skips it | 20:29 |
| sean-k-mooney | ya the slightly ugly thing is it expct your targts to be compliation step so it expect files to eb cureated | 20:29 |
| sean-k-mooney | so if your not doing that you need the PHONE hack | 20:30 |
| sean-k-mooney | *PHONY | 20:30 |
| fungi | debian (and so ubuntu and other derivative) source packages use make to this day. the debian/rules file is just a makefile | 20:30 |
| sean-k-mooney | really i know it was more imperitive then specfils ( they are also not declaritve enve if it looks like it) | 20:31 |
| sean-k-mooney | but not that they were basicaly make | 20:31 |
| fungi | not just basically make, entirely make | 20:31 |
| sean-k-mooney | so in our operator repos we just have pre-commit call make for our local hooks | 20:32 |
| sean-k-mooney | https://github.com/openstack-k8s-operators/nova-operator/blob/main/.pre-commit-config.yaml#L1-L39 | 20:32 |
| sean-k-mooney | the operators ahve generated code for thign like copy | 20:33 |
| sean-k-mooney | because go | 20:33 |
| clarkb | then if you want to go a level deeper you end up using something like bazel. Which I hated because it was too magical. But after using it a bit with Gerrit and learning the magic I'm more tolerant of it. In particular it does things like remember test results and won't rerun tests unless you force it to. As if java never has flaky tests | 20:33 |
| sean-k-mooney | so there are a bunch of meta programing stpe you have to do before you compile and or commit | 20:33 |
| fungi | clearly java isn't bugproof, that's why god created rust | 20:39 |
| clarkb | meanwhile haskell is written by god | 20:40 |
| sean-k-mooney | no that way javascript was created. because if your going to have bugs anyway you might as well use javascript | 20:40 |
| fungi | she has multiple side-projects, sure | 20:40 |
| fungi | (see also: lisp) | 20:40 |
| sean-k-mooney | https://zuul.opendev.org/t/openstack/build/848e9474ea0147009725f5095b0e560e/log/job-output.txt#904-956 | 20:48 |
| sean-k-mooney | so as far as i can tell no more git clones | 20:48 |
| sean-k-mooney | lcally "breakign" the lint seasm to also work | 20:48 |
| fungi | nice! i know it's more verbose, but you can actually see what's being used there, as compared to a bunch of references to git remotes | 20:49 |
| sean-k-mooney | ya its also fast locally | 20:50 |
| sean-k-mooney | like 6 seconds | 20:50 |
| fungi | also it was cloning relatively large repos just to use a few bits from a single file, i think? | 20:52 |
| sean-k-mooney | for our usecase yes | 20:52 |
| sean-k-mooney | hooks are not neessary packages | 20:52 |
| sean-k-mooney | they can be in any language | 20:52 |
| sean-k-mooney | but the oens watcher ues are just wraper around clis | 20:52 |
| sean-k-mooney | and that is true of almost every repo | 20:53 |
| sean-k-mooney | well i might be wrong on that last point | 20:53 |
| sean-k-mooney | but i bleive that ture of nova cybrog placement ectra | 20:53 |
| sean-k-mooney | just takign random oslo repo https://opendev.org/openstack/oslo.middleware/src/branch/master/.pre-commit-config.yaml | 20:54 |
| sean-k-mooney | that a subset of what wathcher is using so it can be converted the same way | 20:55 |
Generated by irclog2html.py 4.1.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!