Wednesday, 2023-03-15

ianwbackup02 prune is running in a screen00:01
Clark[m]ianw: Foundation feedback is that the update looks great 00:04
ianwok, i can send that on then00:04
opendevreviewIan Wienand proposed opendev/system-config master: infra-prod: run job against linaro
wxy-xiyuanJust checked the openEuler job timeout problem. it looks that the network speed is very very slow when do `yum install`, I'll check why and how to speed up it.01:53
wxy-xiyuanoh, wrong channel. Sorry.01:55
ianw#status log pruned backups on backup0203:10
opendevstatusianw: finished logging03:10
opendevreviewIan Wienand proposed openstack/diskimage-builder master: Update Fedora to 37
opendevreviewIan Wienand proposed openstack/diskimage-builder master: Removal focal pins for testing
opendevreviewMerged openstack/diskimage-builder master: chore: support building Fedora on arm64 AKA aarch64
opendevreviewIan Wienand proposed openstack/diskimage-builder master: Update Fedora to 37
mnasiadkaClark[m]: we have a free account, actually it's an organisation - as long as all of the repositories are public it's free (but when you push the image for the first time it sets the repository as private, but can be changed with an API call)06:34
*** elodilles is now known as elodilles_pto06:49
*** jpena|off is now known as jpena08:37
opendevreviewDr. Jens Harbott proposed openstack/project-config master: Fix approval check for post-review trigger
fricklerinfra-root: discovered ^^ while debugging with gtema. maybe someone wants to also look into making zuul recognize the issue earlier09:17
opendevreviewMerged openstack/project-config master: Fix approval check for post-review trigger
opendevreviewArtem Goncharov proposed openstack/project-config master: Add post-review pipeline reporting
opendevreviewMerged openstack/project-config master: Add post-review pipeline reporting
fungiokay, this is downright frightening... is it an emerging trend in software development education?
*** blarnath is now known as d34dh0r5312:23
NeilHanlonfungi: i'd like to cancel my subscription to the internet now, thank you.14:28
fungipeople are are so insistent on avoiding actually reading documentation and learning things that, instead of annoying people who did that by asking them to answer things already covered in documentation, they just ask ai now14:33
fungii guess too many "rtfm" replies have backfired and created a worse problem14:33
NeilHanloni've been playing around with using 'chatgpt' as a starting point for some stuff (particularly: kernel inner workings and the like), but the idea that it could just give me answers on how to do X is ... I want to say laughable, but that's not quite it. There's some reason people think it's "magic" and not just.. guessing the next word14:46
fungii'll admit i've not actually tried that, but based on my crude understanding of the limitations of ai/ml technologies today it seems more likely to confuse learners and lead to a sort of ignorance feedback loop14:48
NeilHanlonI agree with that. If  you think the magic box is truly doing magic, bad things will happen. If you know the magic box is just a complex series of if statements... i guess it can be a useful tool. Or rather, it has been moderately useful to me in certain situations14:50
fungithe accuracy of the information provided to a student contributes significantly to the quality of their education. i'm remembering advice from my first guitar teacher: practicing on a poorly-made instrument is the best way to learn to play badly14:55
fungibut also, i'm imagining an ai trained on "answers" in the ubuntu forums. it's all too easy to find authoritative-looking advice that is in actuality entirely wrong, and often times even dangerous when followed14:57
jrossertheres nothing can be done but as an information point my connectivity to via zayo has turned terrible again15:31
fungiit's possible that padding announcements and setting prefs at the provider's edge would be sufficient to route most traffic around zayo, but i'm guessing they already tune their ebgp to optimize billing from the backbones they peer with15:34
fungistill, it's good feedback for mnaser that communication to/from vexxhost's sjc1 region has been frequently impacted by transit issues inside zayo's network15:36
fungiit may prove important when his contract with them comes up for renewal15:36
jrosserit's not a fix but since this happened before i patched openstack-ansible to allow this `openstack_opendev_base_url:`15:37
jrosserso at least i can continue working with everything swung over to the github equivalent git repos with just one variable now15:38
NeilHanlonfucking Zayo15:41
NeilHanlonsorry. i'm still bitter, apparently15:42
jrosserlooks like we and zayo/above meet and LINX and we're straight into their network in london15:42
jrosser*meet at15:42
NeilHanlonjrosser: v4 or v6, if I may ask15:42
jrosserv4 in this case15:43
NeilHanlondoes your latency increase at/around LINX? 15:43
NeilHanlon(I have a 'friend' I can... nag ;) )15:43
jrosserit looks like this
jrosserbut tbh i think people more geograpically widespread than london have seen similar trying to get to opendev.org15:45
NeilHanloni had a time when Zayo was shipping my traffic from Boston to New York to Boston to get to me15:46
fungijrosser: yes, the traces we looked at previously showed significant degradation in london and atlanta15:46
fungii want to say that the people who reported connectivity issues from germany had traceroutes that ended up going through london as well, but now i don't recall for certain15:47
NeilHanlon"The packets stopped for tea" - my friends are so helpful15:48
fungiit is about that time15:49
* fungi puts a kettle on15:49
fungiunfortunately, my packets going through atlanta are stopping for a tall glass of sweet tea15:50
NeilHanlonannndd now they have diabetes15:51
fungiknown to the rest of the world as tea-colored syrup15:52
* clarkb is having a very slow start today. Probably a good thing after yesterday15:56
fungitake your time, nothing is on fire15:57
clarkbya I'll attend the openstack TC meeting then probably double check gitea09 backups are running properly. THen I don't know what is next15:57
fungithough i have to duck out of the tc meeting early to pick up my grocery order. scheduling error since i booked that time a week ago and forgot to take the usa time change into account15:58
clarkbshould be fine, I'll keep an eye on it and fill in if necessary15:58
* fungi shakes his fist at twice-yearly timezone changes15:58
clarkbinfra-root do we want to set a time to discuss our docker decisions? We could do an IRC meeting or jitsi meet call if we want something more formal16:00
clarkbotherwise its probably fine to just have a time to discuss it in here16:00
fungii'm fine doing it in here, or in #opendev-meeting if we want to record minutes16:01
clarkbmaybe we can do 1900 UTC tomorrow? thinking that probably has the best overlap for ianw and frickler16:02
fungiwfm, i'm free then16:04
NeilHanloni'll attend as a fly on the wall.. still haven't heard back from Docker via email, unfortunately. I'm seriously considering and weighing if Rocky could support a community-owned container registry.. which I think we could, it's just.. time :) 16:04
clarkbcorvus: ianw frickler does 19:00 UTC Thursday March 16 work for docker decision making discussion?16:05
fungiNeilHanlon: well, time, but possibly also storage and bandwidth depending on the number/size of images you're hosting and their popularity16:06
NeilHanlonyeah. i need to chat with our sponsors (mostly, fastly) to see how they feel about it16:09
fungithat's going to be my biggest concern if we decide to run an image repository, since it only makes sense to me to put time into maintaining that service if we can offer it to all the projects we're hosting16:12
fungiand some of our projects have a lot of large images16:13
fungii should say, some of the projects hosted in the collaboratory have anyway16:13
NeilHanlonsure, that makes sense. i will chat with Fastly. maybe something that can be a collaborative effort to maintain and 'own'16:15
clarkbfungi: large images and they insist on building 5 versions of each one :)16:16
NeilHanlonsurely the cloud is free :P 16:17
fungiat least for a write-only repository backed by by the infinite storage of /dev/null16:19
clarkbit will be sunny and 11C today. I should make sure the laptop is charged16:20
NeilHanlonyes, you should. vitamin D is good (or something)16:22
fungia bit cold here on the east coast. was almost cold enough to freeze around the dock pilings overnight16:24
fungibut mainly just unpleasantly windy, which makes the cold feel... colder16:24
clarkbwe had a couple of nice weeks in january and it has been pretty meh until the last couple of days or so16:27
clarkbdecember was really cold too16:27
clarkbHopefully spring is here now. Some of the flowers seem to think so too16:27
NeilHanlonyeah we just got dumped on here in Massachusetts... so.. spring next month, if all goes well :P 16:30
fungiby the time you're able to dig out16:31
fungiokay, headed out to run errands, back within the hour i hope16:47
corvusclarkb: that time wfm17:20
*** jpena is now known as jpena|off17:42
clarkbI've double checked the gitea09 backups by mounting them for each backup server and I see mysql dumps (the main things we are concerned with) as well as content in my homedir that acts as a good fs check17:45
clarkbI think that means backups are working as expected17:46
clarkbin the process I discovered that you don't always need to escape : in bash?17:46
clarkbthe timestamp based backup dirs when FUSE mounting have :'s in them and I was able to ls the names without escaping the noop char. Interesting17:48
clarkbif anyone else wants to dobule check that isprobably not a bad idea but I think other than confidence in handling user demands and cleaning up the remaining old servers this is all done17:49
clarkbhrm gitea has two 1.19.0 release RCs but no updated changelog info for them yet. Probably best to ait until we have that info before worrying about a change to test the update17:51
clarkboh except synchronizing the templates may be worthwhile. /me looks17:53
fungiokay, i'm back. took longer than hoped but not by too much17:54
opendevreviewJulia Kreger proposed openstack/diskimage-builder master: Add a FIPS element
opendevreviewClark Boylan proposed opendev/system-config master: WIP Begin gitea 1.19.x update
clarkbThe templates do need a fair bit of updates but it all seems pretty straightforward and mechanical. I also updated the golang and nodejs version to match the upstream dockerfile18:23
clarkbhtought now that I think of it I may have looked at the master dockerfile not 1.19 /me double checks18:23
clarkbI did, but main and 1.19.-rc1 use the same compiler versions18:24
clarkbfungi: NeilHanlon: the mailman maintainer responded to that email saying "this is not what you want to do" and described a very different path to adding the functionality18:33
clarkbmaybe ya'll saw that alraedy but I just caught up and thought that was interesting18:40
fungiyeah, it's definitely been an amusing ml thread18:43
clarkbcorvus: has more quay info. Though after reading this it feels more like you have to pay for at least a dev account then public repos are free. But they keep reiterating you can use it for free os maybe I'm just thinking too hard about it18:45
clarkbmnasiadka: ^ do you know if you can just sign up to quay creating a new account without purchasing anything?18:45
clarkbpoking around it kinda feels like we should be able to create a shared acount then use that account to create an organization. And maybe do that separately for both opendev and zuul. Then assuming this doesn't require any billing we'll be set?18:51
fungisounds worth a try18:52
fungiand at least our communities already have relationships with red hat, unlike for the owners of other popular public registries18:54
corvusi agree it's worth a try -- note that it looks like a phone # is required19:42
corvusclarkb: i agree though, it seems like at least one dev account is required and it has a cost19:44
corvusbut also that there are sentences like "The service is free for those who want to set up their own public repositories and available for a fee, if you want to create private repositories."19:46
corvusit seems like they're trying to say it's possible to do the thing we want, but in all the steps that describe how to set stuff up, they don't indicate how to do the thing we want :)19:46
Clark[m]Exactly 19:47
ianwclarkb: i can do 19:00 utc tomorrow, only time i'm out is 21:30-21:50 for school drop off19:58
ianwone interesting thing just cleaned from a HN comment is
clarkbianw: ya I suspect that those images we rely on will be ok due to their status20:04
ianwi think most (all) of our base images that we're pulling are "docker official"20:05
clarkbwhich really leaves the exposure for other images to jitsi, jaeger, and a few others20:05
fungithis is an interesting discrepancy:
ianwname: publish-tox-docs-releases doesn't find anything at all ... suggesting project-config isn't indexed?20:07
fungiwell, for me codesearch is turning it up in two projects but gitea is only finding it in one20:07
fungimaybe it depends on the gitea backend i hit20:08
ianwright, it doesn't find it in a direct search on project-config etiher (on gitea)20:09
ianwdoes give some results, so it has *something*20:09
fungigitea01 has both hits, gitea09-12 don't20:09
opendevreviewClark Boylan proposed opendev/system-config master: WIP Begin gitea 1.19.x update
fungiso maybe something's incomplete in the search indices on the new gitea servers?20:10
clarkbthat change removes a no_log setting because it seems some sort of api has changed around listing orgs. Needs more debugging20:10
clarkbfungi: ya maybe has to do with how we populate things with gerrit replication. I think it builds the indexes on startup20:10
clarkbfungi: its possible we haven't indexed some thingsbecause we haven't restarted since the content was pushed in?20:10
fungiyep, 01-04 all find it in project-config as well as releases, while 09-12 only find it in releases and not project-config20:11
fungimaybe we haven't merged anything in project-config? seems unlikely, but...20:11
clarkbpretty sure ianw's acl changes have occurred since at least 09 is in place20:11
clarkbmy hunch is it is more to do with the startup index processing that happens20:11
fungiyeah, we merged things in project-config as recently as 5 hours ago20:12
clarkbBut I'm not super familar with the mechanisms that update the indexes there20:12
fungijust odd that the index would have one repo but not the other in that case20:12
clarkb specifically seems to be the issue we've got20:14
fungistill somewhat baffling that it decided to index one of two repos20:17
clarkbya maybe there is some event that triggers things that has happened for one repo but not the other?20:18
clarkbreleases isn't branched and has no tags so that isn't it20:18
clarkbits the same code running on both gitea01 and gitea09 (same containers etc)20:19
fungimaybe it's per-file and when a file is updated it indexes it20:19
clarkboh that could very well be it20:19
fungithough no, that file in releases was last updated in january20:20
clarkbit says "last indexed 1 day ago"20:20
clarkbwhich is less recent the most recent push to releases20:21
clarkbfungi: if you search for "originate" it shows up in project-config/zuul.d/pipelines.aml which was update a few hours ago20:22
clarkband it indicates it last indexed that a few hours ago. I'm guessing your hunch here is very close20:22
fungithough project-config is the one it hasn't indexed20:23
fungiat least as far as search results are concerned20:23
clarkbno, it is indexed is my point20:23
clarkbat least some of it20:23
clarkbI bet it is something like it only indexes the file or directories that are modified when things are changed and not the entire repo20:23
clarkbmax-concurrency is another term that shows up and was recently added to a file20:27
opendevreviewJeremy Stanley proposed openstack/project-config master: Temporarily remove release docs semaphores
opendevreviewJeremy Stanley proposed openstack/project-config master: Revert "Temporarily remove release docs semaphores"
fungiclarkb: ^ your idea20:28
clarkbre gitea it is definitely indexing project-config but it seems like it is consistently indexing files that have been updated since the service is up but not others. However, I agree releases/.zuul.yaml doesn't appear to have updated recently either. Maybe someone pushed a change that updates releases/.zuul.yaml that is unmerged and that tripped the indexer?20:31
clarkbsince unmerged changes get replicated too20:31
fungithat would be interesting, since they don't get included in the search results20:32
clarkboh yup it even says indexed 5 days ago when you search max-concurrency and that is when it merged20:32
clarkbso definitely seems tied towhen files are being updated somehow20:32
clarkbbut the originate typo was more recent and gives a different indexed time stamp20:33
clarkbtrying other terms from that missing file also shows no results. Really makes me think it just hasn't indexed that file because we haven't crated and event that would cause that to happen20:41
fungimaybe the file is too large and gitea's search doesn't index files over a certain size?20:46
clarkbthat could be. Gerrit definitely has those file size short circuits for rendering the files20:46
clarkbwe just got an alert that the translate-MySQL_upgrade DB has less than 10% of its disk free20:51
clarkbI suspect this is probably fine since openstack is moving towards weblate now20:51
clarkband if this was the first time we've gotten the warning we should be just under that threshold, but I have a school run shortly so not really going to double check those assumptions right now :)20:51
fungimy guess is that new copies of strings were made or something for creation of the 2023.2 branches20:55
fungibut i agree, growth there is slow enough we can likely ignore it until weblate has fully reached production and zanata is retired20:56
clarkbya exactly if it took this long to reach 90% chances are we're ok for a while20:58
ianwclarkb: was the result of the discussion yesterday about partially managed hosts20:59
clarkbianw: I wonder if "!foo:!bar" means not foo and not bar or not foo or not bar21:00
clarkb doesn't make it clear. But I think understanding that is important otherwise we could still run the base playbook there (should be easy enough to test with a mock up inventory could even use a job for that?)21:03
opendevreviewJulia Kreger proposed openstack/diskimage-builder master: Correct boot path to cover FIPS usage cases
opendevreviewJulia Kreger proposed openstack/diskimage-builder master: Add a FIPS element
opendevreviewJay Faulkner proposed opendev/infra-manual master: Trivial: fix image sizing in creators guide
ianwi hope it's "or"21:20
NeilHanlonsomething something demorgan's law21:24
clarkbI always end up with ansible groups not doing what I expect when I set up my local ansible install22:00
clarkbthe whole inventory in fact and eventually I give up and move on and just use the default localhost thing to test whatever task I was actually interested in. Hence my caution22:00
ianwyeah i'll test22:05
clarkb I think that may be the document we want22:05
ianwi started writing up a .dot of the image build requirements.  we have a lot of image builds22:05
clarkbI think it does what we want. a:b:!e:!d:&c means Host in/is (a or b) AND host in/is all(c) AND host NOT in/is all(e, d).22:06
clarkbianw: we haev a lot of builds but most of them are basically take either an upstream official base image or our own python base image and then build an image on that. Which is nice because it means we aren't doing a ton of nesting its all only a few levels of image22:07
ianwyeah it's not too deep, just long22:07
ianwi just just trying to colorize between the "library" images upstream and things under other orgs22:08
ianwwhich you've already done with anyway22:08
opendevreviewClark Boylan proposed opendev/system-config master: WIP Begin gitea 1.19.x update
clarkbapparently ansible's uri module expects a 401 back to know it needs to send the basic auth headers by default. But if you specify expected http return codes (like we do to 200) then that 401 is an error and it doesn't proceed22:28
clarkbso you have to force it to send the auth header upfront22:28
clarkbthat was a fun one22:29
clarkbfungi: pip's dep solver doesn't take into account existing pacakge install's and their dependency requirements? if it does then I think requests may have a buggy set of dependencies (it seems my git-review venv at some point updated chardet to a version requests didn't like so it warned about it but I think that could only happen if the dep solver didn't consider requests deps or22:42
clarkbrequests simply doesn't express that dep properly)22:42
fungipip's dep solver does take installed packages into account now22:52
fungithat's why the situation with using pip in the system python context has gotten increasingly volatile, pip wants to know how to satisfy dependencies for installed packages, and can't upgrade or necessarily even figure those out when installed by something other than pip22:54
clarkbya so maybe requests has a buggy dep specification22:58
clarkbbecuase it would've prevented something else from intsalling newer chardet otherwise22:58
fungior would have upgraded requests23:21
fungii'm not sure which23:22
Ramerethfungi clarkb NeilHanlon: update from yesterday WRT OSS docker program. This morning I got an email that stated Docker Team - Annual was added to the osuosl account. I confirmed that was the case. I also finally got a confirmation email for the other submission I had done for another project (cinc)23:33
fungithat's fast, thanks for the details!23:33
clarkbRamereth: nice that sounds like it was quick and easy compared to our previous attempt. Good to see they have streamlined things23:34
Ramerethhopefully that's the same for others if they had re-submitted using the form23:34
fungithe github issue chronicling all this seemed to be full of people saying "we applied in [2020...2021...2022] and at most only got back an automated reply or two that they'll tell us something eventually but heard nothing else"23:35
Ramerethyeah I noticed that too23:35
fungii guess the sudden flood of new applicants and publicity has forced them to get more serious about processing those23:35
fungiinfra-root: i guess is likely to impact our gerrit upgrade plans? at least for the broken relation chain links (we don't use submitted together as far as i know)23:36
fungilikely there will be a 3.7.2 fixing it by the time we're ready to upgrade, i guess23:37
clarkbfungi: yes, but also I think that may just be informational?23:39
clarkbwe may decide it isn't broken enough to prevent the upgrade?23:39
fungiright, i was mainly thinking from a user inconvenience perspective, but maybe upgrading anyway outweighs that23:39
clarkbya I'm not sure yet myself. Definitely worth considering23:39
clarkbOn the one hand that info is nice on the other its never been rich enough to address me needing to look at shas anyway23:40
fungii mostly point people to the relation chain when they ask why a change isn't merging and its because they never approved its parents23:40
clarkbthis is something gertty does a million times better than the web ui23:40
fungiwell, what gertty does is what the old "new" change screen did23:41
fungiwhich, yes, is far more useful to me anyway23:41
clarkbgerrit has always been linear though gertty has branching which is one of the main things23:41
clarkbyou can't tell just by looking at what the web ui shows you if the change is a peer or a parent23:42
opendevreviewClark Boylan proposed opendev/system-config master: WIP Begin gitea 1.19.x update
clarkbhopefully thats 90% of the necessary changes to upgrade to 1.19.x once there is a changelog23:43
fungiOpenSSH 9.3 (Bugfixes): ssh-add(1), ssh-keygen(1): use RSA/SHA256 when testing usability of private keys as some systems are starting to disable RSA/SHA1 in libcrypto.23:44
fungicloser, but still they haven't changed the client connection default, doesn't sound like23:44
funginot that it matters for us any more since gerrit's finally fixed it at the server end23:48

Generated by 2.17.3 by Marius Gedminas - find it at!