| *** ykarel_ is now known as ykarel | 11:43 | |
| fungi | just a reminder that i'm not really around today and will be afk a lot of the day | 12:32 |
|---|---|---|
| fungi | though the vos release of mirror.debian and mirror.debian-security both completed successfully, mirror.ubuntu is still in progress but judging from grafana it's probably finishing any minute now | 12:36 |
| opendevreview | Pierre Crégut proposed openstack/diskimage-builder master: root password for dynamic-login made simpler https://review.opendev.org/c/openstack/diskimage-builder/+/961449 | 12:53 |
| TheJulia | Greetings, by chance is there a PTG etherpad? Or forum sessions at the summit? | 14:50 |
| clarkb | for opendev specifically? | 14:52 |
| TheJulia | yes | 15:04 |
| TheJulia | If there is any overlap with Zuul, that could also be a good vehicle for the crazy idea I have | 15:06 |
| clarkb | neither opendev or zuul are participating in the ptg so no. OpenDev has started doing a "pre ptg" where we try to get together a bit in advance of the ptg to ensure meetpad is working and it gives us a chance to focus compared to the ptg which tends to pull most of us in 10 directions. | 15:08 |
| clarkb | we do have a list of topics on an etherpad for the pre ptg. Let me dig up a link for that | 15:09 |
| TheJulia | perfect! | 15:09 |
| clarkb | https://etherpad.opendev.org/p/opendev-preptg-october-2025 | 15:10 |
| TheJulia | Much appreciated! | 15:11 |
| clarkb | TheJulia: maybe you can mark down which of the times I have listed for the ptg that work for you to discuss those items? that way we can do our best to discuss them during those times | 15:19 |
| clarkb | *for the pre ptg | 15:19 |
| TheJulia | I've added an item, feel free to scream, but also included a link with some examples and tried to convey usefulness. | 15:20 |
| TheJulia | clarkb: will do | 15:20 |
| clarkb | TheJulia: also I'm not convinced that claude's license terms are open source compatible (they limit field of endeavor) | 15:20 |
| clarkb | but I'm not a lawyer and don't know how that impacts open source projects vs say just the claude user | 15:20 |
| JayF | A method that was model-agnostic for that kinda integration would be ideal imo | 15:21 |
| clarkb | JayF: if the models were free to use that would be easier. But no I think you have to configure whichever you choose to use ahead of time with appropriate access | 15:23 |
| TheJulia | clarkb: Yeah, that would be an aspect we would need to look at, I'm speaking generally in that it might be poweful. And if we're just treating at a CI result.. meh. | 15:23 |
| TheJulia | clarkb: I wouldn't treat the model or cost as an aspect, there may be options there if we're willing to open the door | 15:23 |
| JayF | clarkb: I think the way I'm thinking about this wouldn't be condusive to a CI-run option anyway (I was thinking more: on demand code review comments in gerrit, which are doable I believe 100% client side via API if interesting) | 15:24 |
| * JayF asks claude to JfDI | 15:26 | |
| clarkb | JayF: I put notes on the etherpad. Yes I think two primary appraoches would be to do this in Gerrit or to do this via Zuul jobs and report back to gerrit. Experimenting and bootstrapping with the zuul stuff is someting you can do right now without any interaction from the opendev team | 15:26 |
| clarkb | doing it directly in Gerrit is going to be a much bigger lift | 15:27 |
| JayF | well I'm not even thinking zuul right now, I'm thinking smaller | 15:27 |
| JayF | like ./claude-code-review https://review.opendev.org/whatever/123456 -> uses my user creds to post a summary in the style of Julia's examples | 15:27 |
| clarkb | I mean zuul is the engine/framework/tool to organize it | 15:27 |
| JayF | if that has value in one offs, then go the next step for automagic | 15:27 |
| clarkb | oh sure you as a gerrit end user can do code reviews however you like. If people complain about spamming we might ask you to stop | 15:28 |
| clarkb | I'm just saying it would be pretty easy to have the ci system drive this if you want to do that and it wouldn't require any input from opendev | 15:28 |
| TheJulia | I'd really love to see a model where a contributor could easily get something | 15:28 |
| TheJulia | I really don't like an idea of someone trying to use an AI to replace their own engagement | 15:28 |
| TheJulia | That seems like asking for a disaster, tbh | 15:29 |
| JayF | TheJulia: I'm talking more about a prototyping tool :) | 15:29 |
| clarkb | I guess what I'm saying is you can already do this if you want and you don't need to invovle us. You really only need to involve us if you want to change zuul or gerrit beyond its already user exposed settings | 15:29 |
| TheJulia | oh, okay | 15:29 |
| TheJulia | good | 15:29 |
| JayF | TheJulia: not an end state | 15:29 |
| TheJulia | whew | 15:29 |
| JayF | I've seen the copilot PR stuff in github. | 15:29 |
| JayF | And I am ... skeptical | 15:29 |
| clarkb | I would ask you not to do that to any of our code repos | 15:29 |
| clarkb | our == opendev's own repos | 15:29 |
| clarkb | but if you want to experiment in ironic that seems fine | 15:29 |
| TheJulia | clarkb: well, we would likely need to store secrets if we're calling an API | 15:29 |
| clarkb | TheJulia: yes zuul has support for that | 15:29 |
| JayF | clarkb: it's not my intention to go spamming the world, using a tool that can do them one at a time is quite literally the opposite direction | 15:30 |
| TheJulia | But, there may be other interest, since I keep hearing the idea of AI assisted code review, so made sense to raise a discussion up | 15:30 |
| TheJulia | clarkb: to be clear, I'm not wearing an ironic hat with this topic, would be super cool, and maybe I'd build that project level consensus based upon the discussion | 15:30 |
| JayF | clarkb: TBH my suggestion is more because I don't love the idea of it running automatically on everyone's stuff, because I do have concerns it could be misleading to new contributors | 15:31 |
| JayF | so either going to prove myself wrong or write with some science :D | 15:31 |
| JayF | **right (wow that was /ironic/) | 15:31 |
| clarkb | JayF: interesting my concerns come more form a maintainer standpoint. It already feels like Ai is being used to dump more work in laps without consideration for the acutal mental overhead in reviewing thousands of lines of code and deploying it successfully | 15:31 |
| clarkb | sure you can generate a bunch of stuff but then you hand it off to a maintainer probably without even testing it or reviewing it yourself and you just make this thing my problem | 15:32 |
| TheJulia | clarkb: that is *absolutely* happening, and I've started pushing back on the idea of "just use AI to do the work" and this seems like an easier way to also help shorten the review/cycle loop to help reviewers and contributors. | 15:32 |
| clarkb | but ya I think zuul provides enough supporting infrastructure for typical CI jobs to be able to get something working without really changing anything about opendev's zuul deployment. At least iof you want to start experimenting | 15:32 |
| TheJulia | ++ | 15:33 |
| clarkb | its possible that those experiments provide information on what could be improved to integrate better or make the experience better. But I think starting there is a good first step (or starting at step 0 like JayF suggests then do zuul jobs as step 1) | 15:34 |
| clarkb | I also hesitate to invest a lot of my personal time in this sort of thing because I was working on this before AI was cool and it was extremely difficult to get anyone to care | 15:34 |
| clarkb | and eventually we turned off the entire elasticsearch cluster and handed it off as a result | 15:34 |
| TheJulia | totally fair as well :) | 15:35 |
| clarkb | so would be good to have something we can point to as "this is useful" and then go from there | 15:35 |
| TheJulia | My other reason for bringing the topic up was a just a temperature test on the infra side | 15:35 |
| clarkb | TheJulia: the approach we were taking in the past (pre covid) was attempting to analyze ci job failures to call out where things were going sideways to aid people in identifying sources of bugs. The elasticsearch cluster and elastic recheck were the more visible components of this | 15:37 |
| TheJulia | I've noticed a shift to broader recognition of mental taxing overall due to AI, so I suspect others are going to have it in mind and may want to look for solutions as well. | 15:37 |
| clarkb | but then beyond that we were also trying to train models of one variety or another (the initial attept was using spam filters then tristanC[m] and dirk started building more dedicated models iirc) | 15:37 |
| TheJulia | Yeah, but thats just end result, I'm talking purely at "look at the code, provide feedback" | 15:37 |
| TheJulia | The go and look at the logs is way after the fact | 15:37 |
| Clark[m] | arg my isp continues to have occasional packet loss problems. Lets see if this client is better | 15:40 |
| TheJulia | heh | 15:41 |
| TheJulia | it happens | 15:41 |
| Clark[m] | maybe slightly. Yes, at the time LLMs didn't really exist (people were probably working on them in secret at openai) so we couldn't really do code analysis in that manner, but text process for outliers is something that has been done for a long time to combat spam so we figured we could have success in that space | 15:42 |
| Clark[m] | I think the bigger issue is that there has been significant apathy within openstack around caring for the commons and ensuring quality to stand by | 15:42 |
| JayF | spamassassin for bugs :) | 15:42 |
| TheJulia | clarkb: ahh, interesting! | 15:43 |
| Clark[m] | so when the ask was for basically making our CI more reliable and running a large dataabse and data processing system to help there wasn't enough momentum | 15:43 |
| Clark[m] | I think that has shifted somewhat since then the issue now is more time (fewer contributors means focusing on what has the most impact etc) | 15:44 |
| TheJulia | Clark[m]: I'm sort of secretly hoping, since I know orgs have goals around using AI, that maybe we could also help remind folks about the commons as well. Consider it my secret hope. ;) | 15:44 |
| Clark[m] | and it is possible LLMs can help with that if done well | 15:44 |
| Clark[m] | from my perspective I'm all for people experimenting and supporting that as I'm able. But I'm not sure I can personally drive it at this point. | 15:46 |
| TheJulia | Agree, but also, if we want to drive overall consistency, we at least need to have a discussion your level of community to allow the ideas to flow :) | 15:46 |
| clarkb | I think my connection to irc is happier now? Thinking about this more if yuo do start down the do this with zuul jobs path then there may be engagement within the zuul community and that could be something that is at least partially maintained in zuul/zuul-jobs | 15:53 |
| clarkb | it wouldn't surprise me if some of the groups other than opendev using zuul have an interest in this | 15:54 |
| clarkb | similar to the framework tooling for dealing with container images in jobs there may be interest in a common set of tooling to mcp between zuul and llm | 15:54 |
| TheJulia | (This is also why I wanted to see if there was a discussion to be had at this level) :) | 15:55 |
| TheJulia | Anyway, I revised the etherpad a little, just to make it clear I'm not looking at opendev to *do* something, but maybe provide feedback/guidance if there is interest. | 15:56 |
| clarkb | ack thanks | 15:56 |
| JayF | clarkb: TheJulia: https://review.opendev.org/c/opendev/sandbox/+/961716 from https://codeberg.org/JayF/claude-gerrit | 16:00 |
| JayF | obviously the prompt should be updated to have it be shorter, but I think I have an MVP PoC :) | 16:00 |
| TheJulia | oh my | 16:01 |
| JayF | it cost $.01 fwiw | 16:01 |
| TheJulia | heh | 16:01 |
| JayF | only manual coding in that, fwiw, was pretty hilarious: claude-code kept using invalid model names, so I had to pass --model with a working name to make it work | 16:02 |
| TheJulia | That might be a bit further than I was thinking, but it would be pretty cool as well if Ironic could have "talk like a pirate month" and have the output style changed to be like a pirate..." | 16:05 |
| clarkb | I like that it calls the commit message vague and unprofessional. We need more of that on like 99.999% of commits pushed to github :P | 16:05 |
| * TheJulia resists the urge to ask claude code to do the same, although as shakespear is sort of amusing :) | 16:06 | |
| clarkb | our communities tend to be really good about writing at least something useful in commit messags which I always appreciate when I'm out in other repos trying to work through histroy | 16:06 |
| JayF | My #1 beef: people who put links in git messages and consider that good enough. Nope. Links die. Git is as forever as the software is, generally. | 16:07 |
| TheJulia | I prefer people explain, provide additional links, but at least give me enough context to understand. I get links are also process driven, especially when stuff gets cherry-picked down though. | 16:08 |
| JayF | oh, links as *additive* are great | 16:09 |
| JayF | links in *lieu of* real info are not | 16:09 |
| TheJulia | ++++++ | 16:12 |
| clarkb | from that example I think there are a few minor things that could be cleaned up like teaching it to quote things in a way that gerrit understands but even as is its readable enough. I htink the main thing I would wnat to improve fomr there JayF is figuring out what the correct amount of infromation is. But even then iterative review is a balancing act itself | 16:13 |
| JayF | adamcarthur5: this is where the conversation happened mostly fwiw :D | 16:13 |
| clarkb | sometimes its better to just dump all the info at once and other times it is better to get people to focus on one thing at a time and sometimes that depends on who you are working with | 16:13 |
| clarkb | re commit messages I find the big thing we often fail to do is properly explain why | 16:14 |
| clarkb | and I'm sure I fall into this trap too. It is easier to explain the what but why is often more subtle | 16:14 |
| clarkb | or we have inherent biases where the why is explicit to ourslves and not realize it may not be to anyone else | 16:15 |
| JayF | well-written code tells you how and what. well-written commit messages/comments tell you what-was-intended and why | 16:16 |
| JayF | you gotta have both to really know what's going on | 16:16 |
| JayF | I didn't fully grok this until I worked at a big purple behemoth which had lots of code that did stuff that nobody knew why | 16:16 |
| opendevreview | Merged zuul/zuul-jobs master: Raise connection pool for boto3 in s3 upload role https://review.opendev.org/c/zuul/zuul-jobs/+/957218 | 17:19 |
| tristanC[m] | TheJulia: here is the project I'm working on: https://github.com/logjuicer/logjuicer, it's available in Zuul with https://zuul-ci.org/docs/zuul-jobs/latest/logjuicer-roles.html . You can give it a try right away by pasting a zuul build url in this service: https://gateway-cloud-softwarefactory.apps.ocp.cloud.ci.centos.org/logjuicer/ | 17:36 |
| tristanC[m] | LogJuicer predates LLM, it's a fast log processor that can compare build if it finds a baseline (e.g. the last successful build), otherwise it extracts errors/traceback from the logs. | 17:40 |
| tristanC[m] | clarkb: this is not using a "model", the baselines are indexed using a hashing tokenizer, but that's more like a diff on steroid than a machine learning model. | 17:42 |
| clarkb | tristanC[m]: oh for some reason I thought you were doing training of something | 17:42 |
| clarkb | the old spam system did train good and bad results (just like you would with a spam detection system) and thati s a model (not an llm) | 17:43 |
| tristanC[m] | well, when a baseline is processed, the vector are stored in a file so that it can be re-used quickly, but I wouldn't call that a model, the core algorithm is a pairwise cosine distance | 17:44 |
| clarkb | I see | 17:45 |
| tristanC[m] | in other words, the more data you add, the slower it gets | 17:45 |
| clarkb | but ya I think the interest has always been there to provide smarter feedback to developers. I think the reason tools like these ones haven't really caught on with say openstack is that people woudl rather just recheck into oblivion | 17:49 |
| clarkb | even something as simple as elastic-recheck was a direct response to ^ ebcause people were supposed to identify causes of failures at the time and rehceck with that info so taht we could collect it and prioritize the fixes by greatest impact. We found that people were either A) bad at identifying root causes of B) didn't care enough to apply the effort to do it properly so the data | 17:51 |
| clarkb | was highly polluted. That led to elastic-recheck doing its own identification instead. Then when people stopped caring about the elastic-recheck identified problems e-r itself died | 17:51 |
| clarkb | then we deleted elsticsearch entirely | 17:51 |
| tristanC[m] | perhaps one of the challenge with openstack is that the failures can happens in distributed services and their logs may not be readily available in the build result. That was the main motivation for LogJuicer, to speed up the investigation where we would need to go through multiple files before finding the root cause. | 17:54 |
| JayF | That's kinda what I've been thinking reading this, tristanC[m] -- often times the failures are second-order things (at least in Ironic that's often true) -- like a timeout caused by something that will never print anything "wrong" in logs (but you can maybe assemble a story of what went wrong reading multiple service logs to put together a story) | 17:55 |
| clarkb | tristanC[m]: yes that was why we tried to do the identification with the precursor to logjuicer too. OpenStack is big and complicated and I think moreso than just having logs distributed across systems the project itself has always bee nvery siloed | 17:55 |
| clarkb | so people don't even want to look in neutron's logs if they are wiorking on nova or whatever | 17:55 |
| clarkb | the perspective isn't "the software I care about broke and I want to find out why" its "well I don't see anything wrong in what I did so recheck" | 17:55 |
| clarkb | and i think that is a deeply seated cultural behavior within openstack | 17:56 |
| tristanC[m] | JayF: yeah, LogJuicer also have this convenient "timeline" feature where it can merge the errors in chronological order when the log files have timestamp | 17:56 |
| clarkb | that functionality is why I spent foever arguing with projects that they needed to have consistent log formats and always include a timestamp | 17:57 |
| JayF | clarkb: I've often heard, especially from newer contributors, they don't have the knowledge to understand what blew up. This is more common with folks who have more of a development than systems/devops backgrounds. There's gotta be a better way than just asking every contributor to be an expert at devop-y troubleshooting :( | 17:57 |
| tristanC[m] | clarkb: though I missed elastic-recheck, it's annoying to observe known errors and it would be great to assign multiple failure to a single bug and be able to recheck automatically when it is fixed | 17:57 |
| JayF | To be clear: I don't have a better suggestion, and I agree blind rechecks are bad, but our solution is some equivalent of just telling people to try harder, which isn't really fair | 17:58 |
| clarkb | JayF: I mean realistically that is what is required to work on openstack | 17:58 |
| JayF | If that's legitimately true, and I sure as hell hope it isn't, that's a major flaw in our workflow that needs fixing. | 17:58 |
| clarkb | it is a large complicated system that requires linux knowledge, networking knowledge, virtualization knowdlege, python knoweldge etc | 17:58 |
| tristanC[m] | clarkb: it can be complicated, I had to handle logfile that only have time without date, using a britle logic to figure out what to do when the span range between 23:50 and 00:30 for example :) | 17:58 |
| clarkb | tristanC[m]: ya I never managed to get everyone to do it, but the coverage got a lot better over time with iso timestamps | 17:59 |
| clarkb | JayF: I mean a classic issue is mtus. These have affected all the things consistently over time. They require people have a basic understanding of some networking stuff. | 18:00 |
| JayF | Yeah, but if someone is adding a field to an API endpoint they shouldn't need to have a PhD in linux networking. | 18:00 |
| clarkb | ya I guess it depends on how you want to frame the sope of that work | 18:01 |
| JayF | That's more what I'm getting at; not that people should go mucking in the system areas of the code and just hope it works; but the idea that many of these issues the people rechecking *are victims* not the root cause | 18:01 |
| clarkb | are we adding a field to a REST API or are we adding a field to a linux rsource orchestration system's rest api? | 18:01 |
| tristanC[m] | To be fair, it's a bit ridiculous that incorrect mtu is a thing, there is a service somewhere which is not helping! | 18:02 |
| clarkb | ultimately if that api no longer functions at what it is designed to do someone has to deal with it | 18:02 |
| clarkb | tristanC[m]: part of the problem is that you can have l2 devices that cannot fragment so you have to ensure everything along the l2 only path has mtus as least as small as the l3 interfaces | 18:02 |
| tristanC[m] | but couldn't that be checked by the runtime, or does it expect PhD level of expertise to be used properly? | 18:03 |
| clarkb | JayF: and yes I think that openstack's testing approach probably over exposes those details. More fakes or simpler subsets of services could be tested together to limit this. But if neutron's network path no longer sets mtus correctly then you need someone to be able to debug that | 18:04 |
| clarkb | tristanC[m]: the problem is that for a long time neutron didn't enforce any of that the expectation was that operators would | 18:04 |
| JayF | tristanC[m]: my PhD comment was sorta a joke, but people like Julia and I have been working on Ironic for like ... ten years? And some of the issues around MTUs or similarly low level things took us many days to figure out. I don't wanna throw new contributors under that bus. | 18:04 |
| clarkb | tristanC[m]: in the CI system the operator was the person writing or modifying the CI job | 18:04 |
| clarkb | tristanC[m]: eventually we convinced neutron that it needed to more directly manage that stuff to avoid these problems and things got better but occasionally you see a new change that breaks mtu stuff somewhere and then someone has to track it down | 18:04 |
| JayF | many of those MTU issues, for Ironic at least, are in the fact we have to use various tunnelling protocols as part of devstack | 18:05 |
| clarkb | my point isn't so much that new contributors need to deal with it, its more that ultimately someone does and that requires having the ability to do it somewhere (which probably in an ideal world means teaching new contributors about these things over time) | 18:05 |
| clarkb | and so hiding that entirely away is a disservice. Again its all about balance | 18:05 |
| clarkb | the good news is a lot of that knowledge is transferable. If you've ever debugged k8s networking its basically the same as debugging neutron networking | 18:06 |
| JayF | Yeah, there is a balance to have, I just want the bulk of that blame/burden to not be shouldered by the newest or least-experienced members of our community. That's sorta why I dislike the focus on "bare rechecks" | 18:06 |
| clarkb | JayF: oh the problem was never bare rechecks | 18:07 |
| clarkb | JayF: the problem was that people would lie on their rechecks | 18:07 |
| clarkb | and we realized you can't stop them from doing that so we made elastic-recheck to track the info more directly | 18:07 |
| clarkb | elastic-recheck was definitely a better solution to the problem. But it doesn't exist anymore bceause it had a giant list of bugs that it was tracking that no one was fixing | 18:07 |
| JayF | I'll admit new-to-openstack jay might have polluted an e-r bug or two without realizing | 18:07 |
| tristanC[m] | sorry, I didn't meant to blame anyone in particular, my point was that if the systems are brittle to small changes, and they don't yell when something is wrong, then it's going to be hard for new comers. I picked mtu as an example because it sounds like something that, for example docker, could check if it is set incorrectly, but it does not, and that often results in obscure network timeout. | 18:08 |
| clarkb | tristanC[m]: ya a lot of tools that play with network interfaces ignore mtus | 18:09 |
| clarkb | and it isn't always safe to do so | 18:09 |
| clarkb | related apparenly the nintendo switch's network stack uses a very small mtu because nintendo got sick of dealing with broken home network setups | 18:09 |
| clarkb | I always found that funny | 18:09 |
| JayF | honestly it's pretty smart | 18:09 |
| tristanC[m] | haha, that's horrible! | 18:10 |
| JayF | outside of software downloads I bet there's a lot of tcp_nodelay use in online gaming, so it's probably not even that big of a real-world loss | 18:10 |
| clarkb | there are lots of other common problems that having general knowledge of if you work with linux will simply make your life better. SELinux (or apparmor), namespaces, and IP networking seem like the big ones | 18:13 |
| clarkb | oh DNS too | 18:14 |
| clarkb | 1.1.1.1 had that outage semi recently and most of the world using it thought that the internet was broken. The rest of us chagned our resolvers and were fine. Even in tech spaces I saw a lot of people not realize what had happened | 18:15 |
| clarkb | I think a cultural shift that accepts "I don't know but I'm willing to learn" and more of an identity attachemnt to openstack rather than specific subprojects would go a long way towards improving the quality of the software. At the same time identifying things that make it difficult to debug why something has gone wrong and improving logs/tools/etc to make it less difficult is also | 18:20 |
| clarkb | a good thing | 18:20 |
| clarkb | one common issue is that often times these problems require updates to three or more distinct areas of code so if you aren't willing to step outside of comfort zones there is no way to improve things | 18:20 |
| clarkb | (once you get over the hurdle of identifying them in the first place) | 18:20 |
| clarkb | this reminds me I should catch back up on where we are in re PBR and setuptools deprecation cleanups | 18:26 |
| clarkb | PBR has a deadline of October 31 and at that point setuptools will break it. stephenfin in particular has been amking excellent progress in cleaning up things to avoid that problem | 18:26 |
| clarkb | the smaller of the two backup servers needs pruning. I'll look into that (its been a while) shortly (just finishing up some lunch) | 19:55 |
| clarkb | the prune script is running in screen on that host now | 20:02 |
| clarkb | the prior prune was just under a month ago so we may need to look at clearing out some old unneeded data again (I can't remember if we have any in the pipeline for that) | 20:04 |
| clarkb | as a sanity check we are already excluding the gerrit h2 cache db dir from backups (those can grow quite large so I wanted to double check) | 20:12 |
| clarkb | I think we can retire and/or purge eavesdrop01, refstack01, and review02 backups | 20:19 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Retire eavesdrop01 and refstack01 backups on the smaller backup server https://review.opendev.org/c/opendev/system-config/+/961752 | 20:27 |
| clarkb | fungi: ^ you may be interested in that change in particular since it affects two services that you eithe replaced or shutdown | 20:27 |
| clarkb | prune is done and 67% of the disk is free | 20:36 |
| clarkb | *67% of the disk is used not free | 20:37 |
| clarkb | we started at 92% used | 20:37 |
| clarkb | #status log Pruned backups on backup02.ca-ymq-1.vexxhost.opendev.org which reclaimed space from 92% used to 67% used | 20:38 |
| opendevstatus | clarkb: finished logging | 20:38 |
| clarkb | I closed the screen and the log is in the usual location. I think we followup with those retirements then start working on the purges that we think are safe | 20:38 |
| fungi | i'm not really here, but strongly object to the idea that reading and understanding fundamental ietf rfcs for understanding basic networking concepts requires a phd... i don't even have a university degree at all and was able to work as a network engineer diagnosing and solving e.g. pmtud blackhole problems | 20:44 |
| fungi | if people entering the field think you need 6-8 years of academic extortion with a lifetime of student loans just to hook computers together, that's a sad state of affairs | 20:46 |
| fungi | and i really worry for the future | 20:46 |
| clarkb | I suspect that it is less about assuming you need advanced degrees and more just that people don't want to have to dig into something tertiary to their current focus | 20:47 |
| clarkb | management says add feature foo. Well the gate can be completely broken preventing that from happening but its easy to same I'm blocked on bar rather than take some time to learn what is required to undersatnd and fix bar | 20:48 |
| clarkb | *easy to say I'm blocked on bar | 20:48 |
| clarkb | and in many organizations things are structured so that you explicitly cannot delve into other areas so that is another cultural expectation that we are often fighting against I suspect | 20:49 |
| fungi | got it, so claims of needing "phd-level expertise" were hyperbole | 20:50 |
| clarkb | yes I think the epxress that debugging openstack can be daunting. But if the developers of openstack cannot do it then who can? | 20:50 |
| clarkb | and that is sort of why I think we should shift the culture. Its not that I expect everyone to be able to do it on day one. but I think more people should be willing to do it and we can create a learning pipeline so that more people are capable of doing it | 20:51 |
| clarkb | but you should go back to enjoying a day off the backup stuff is happy for now and we can followup on that later | 20:52 |
| fungi | cool, i'll take a look tomorrow | 20:55 |
Generated by irclog2html.py 4.0.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!