Friday, 2022-03-18

opendevreviewIan Wienand proposed opendev/system-config master: gitea: set custom avatars for orgs
ianwacessing the gitea api reminded me of which switches our ansible things to use token auth 00:23
ianwnot sure if that's worth restoring?  it was to avoid problems with the hashing algorithm that i think upstream reverted.  anyone think of other advantages?00:24
Clark[m]Ya upstream reverted iirc and it was to make auth quicker and use less memory. I don't think it is necessary due to upstream's pivot00:53
opendevreviewMerged opendev/system-config master: Test gitea 1.16 partial clones
ianwour ci system really is something :)01:38
mnaserianw: aw man we want a logo too now :p02:34
fungivery nice!02:39
fungimnaser: nobody has a logo up yet, but 834085 should make that possible02:39
ianwmnaser: i did a vexxhost logo :)02:51
mnaseroh no way, that's awesome02:51
ianwif it should be something else, feel free to update02:52
mnaserthats more than ok :)02:52
*** arxcruz is now known as arxcruz|rover07:47
*** jpena|off is now known as jpena07:48
*** arxcruz|rover is now known as arxcruz|ruck09:00
*** rlandy|PTO is now known as rlandy10:26
*** dviroel|afk is now known as dviroel10:58
priteauGood morning. Sorry, I have another issue today :) The cloudkitty-dashboard release notes are not getting published.11:02
priteauSee, yoga is missing although we merged the patch a while ago11:03
priteauI've also just merged which should have changed the title11:03
priteaupromote-openstack-releasenotes : SUCCESS in 1m 01s11:04
priteauMaybe I should ask in #openstack-infra instead?11:05
fricklerpriteau: asking here is fine, I can take a look after my current meeting11:15
priteauThanks frickler11:18
fricklerpriteau: I do see but it is empty, do you expect actual content to be there?12:14
priteauI swear it wasn't there earlier12:15
priteauThe index page has updated too12:15
fricklermaybe it just took a bit to publish after that last promote job? or browser cache?12:15
priteauAnd yes, I am expecting it to be empty12:15
priteauI tried in an incognito window, maybe this still uses browser cache12:16
fungipriteau: frickler: note that there is a delay between when documentation files are written to the read-write volume in afs and when that update is synchronized to the read-only replicas like the one the docs site is served from. normally we run a `vos release` every 5 minutes on a timer out of cron, but that's a serialized set of calls for all the static afs volumes and some of them (e.g. the12:23
fungitarballs volume) can have large updates which take longer to copy across the network so can in some rare cases delay publication of updates for other volumes by an hour or more12:23
priteauok, I must have check right after the promote job finished12:23
priteauStill doesn't explain why the initial reno update didn't publish, its promote job was successful:
fungiyeah, generally expect uploads from a promote job to be reflected on the docs site within 5-10 minutes, but longer delays are possible. if it's more than an hour, definitely reach out because something might be stuck (it happens, because clouds... like an unexpected hypervisor host reboot impacting an afs server could leave some volumes in a dirty state)12:25
priteauAnyway, problem solved for now, thanks frickler and fungi12:25
fungipriteau: was the page a 404, or just didn't have any notes in it?12:35
fungipriteau: the promote job uploaded the release notes built in the gate job, which you can see a preview of here:
fungiand that's empty12:36
fungiso maybe your question is more why 832582 built an empty set of release notes?12:40
*** dhill is now known as Guest248112:50
*** TheJulia is now known as needsleep12:57
*** needsleep is now known as TheJulia12:57
priteaufungi: the empty yoga page is expected, there were no changes in the release13:01
priteauBut the notes built by 832582 were not published, yoga.html was a 404 until this morning when I merged another patch to publish again13:02
priteauYou can see it fail on this releases patch:,
priteauCould not fetch release notes page 40413:03
fungiyeah, that's weird. the promote job claims to have uploaded it to days before that 404 was logged13:17
fungier, ten days before13:17
priteauWhen I checked the release notes this morning, the timestamp at the bottom was from 202113:27
priteauThe upload must have failed silently somehow13:27
fungidoesn't look like it from the promote build log13:35
fungithat has the list of files copied to /afs/ and includes "<<CHANGED>>>f+++++++++ yoga.html"13:36
fungithe only think i can think of is that some other build came along after it and removed that file (maybe building from an earlier repository state?)13:37
fungilooking at there were two builds with identical start times that day, one for master and one for stable/yoga13:38
*** TheJulia is now known as needssleep13:38
fungii wonder if the promote job uploaded moments after and removed yoga.html because that wasn't included in its gate build13:39
fungi"<<CHANGED>>*deleting   yoga.html"13:41
fungipriteau: mystery solved13:41
fungiwe probably need to think about how to address races between these sorts of publication jobs13:42
*** sboyron_ is now known as sboyron13:43
*** sboyron is now known as Guest248513:43
*** Guest2485 is now known as sboyron_13:44
opendevreviewTakashi Kajinami proposed openstack/project-config master: Add puppet-manila-core
*** dviroel is now known as dviroel|lunch15:23
*** dviroel|lunch is now known as dviroel16:10
clarkbinfra-root is a fairly straightforward review that improves our testing of firewall rules16:11
* clarkb is going through system-config change sagain to try and prune that list furhter16:12
corvusthe rolling restart of executors completed yesterday; i plan on rolling the schedulers and web servers later today16:15
opendevreviewClark Boylan proposed opendev/system-config master: Finish removing references
gmannfungi: frickler clarkb fixed config error in legacy job file
gmannand l-c job moving to focal
*** jpena is now known as jpena|off16:48
*** marios is now known as marios|out16:49
opendevreviewMerged opendev/system-config master: Add firewall behavior assertions to testinfra testing
clarkbwe're down to like 165 open changes now17:12
fungithat's remarkable17:13
fungidown by an order of magnitude, i'm betting17:13
clarkbI think we were around 300 when this started17:13
fungiokay, so not an order of magnitude, but down by almost half anyway17:15
clarkbwe're definitely hitting the set that is more difficult to wade through. Having to do a bit more investigation for each one that I think is abandonable and half of them I've left open as they represent some value17:15
fungii had just assumed we had far more open system-config changes than we really did17:16
fungiit was more than my watership down counting threshold anyway17:16
clarkbfungi: any thought son whether should stay open in the hopes of it getting reimplemented in ansible? I don't know that we see connection pressure against gerrit over http today so this is probably fine to remove?17:17
fungimmm, yeah, i pushed that in 2016. i think it's safe to say if we needed it we would have done it in the intervening 6 years. i'll abandon that17:19
clarkbtopic:system-config-cleanup has a number of change sthat I evaluated as being worthwhile and I either reabsed them or rewrote them against the current state of things too17:21
clarkbI don't think any of them are particularly urgent though as they have been hanging out for a while17:21
fungiwe did merge my thread tuning change for gerrit around the same time, so apparently that was sufficient for whatever slowdowns we were experiencing at the time17:21
clarkbDoes anyone know what the motiviation for and was? That log file doesn't seem to exist in the current containers17:39
clarkbmordred: ^ do you recall17:39
opendevreviewJeremy Stanley proposed opendev/system-config master: Add (lack of) license information for org logos
*** dviroel_ is now known as dviroel17:46
clarkbfungi: fyi that change is one we're going to pick up when lands and we restart gerrit. Not sure if elections tooling and similar scripts may rely on that18:01
fungicodesearch says we don't call --no-limit18:03
clarkbfungi: what about just no-limit? it is useable in both ssh and http iirc and http would not hvae the --18:04
fungiyeah, that's actually what i queried:
clarkbcool we should be fine then18:04
fungipretty sure all the implementations i've written have relied on normal paging18:05
fungiand offsets18:05
fungiin the opendev/engagement implementation i actually constrain the queries as much as possible (so basically one batch per repo) and do overlapping offsets with client-side deduplication in order to try to eliminate undercount from nondeterministic paging18:06
fungiit roughly doubles the amount of data pulled, but increases accuracy to near perfect18:08
fungistateless pagination in query apis is such a mess18:08
clarkbooof ya18:08
fungii also use the largest page sizes allowed by the api in order to minimize the number of requests, but i think i may not have even been aware there was a no-limit option18:09
clarkbI think it must be newer18:11
fungii would believe that18:13
fungii may have written most of the engagement query script prior to our 3.x upgrade18:14
fungican't exactly recall now18:14
opendevreviewMerged opendev/system-config master: gitea: set custom avatars for orgs
clarkbthe logo change seems to have applied ok19:17
clarkb(yay for testing)19:17
clarkbthe gerrit image rebuild change will finally +1 after a couple of network related failures19:18
mnaseranyone ever thought into where the 'updated' time comes from gitea?19:37
clarkbmnaser: I think it updates when changes are replicated to it19:39
mnaserhmm, i see it the same time for all vexxhost projects, maybe it has to do when the system was redeployed19:40
clarkbone thing that makes it weird is that gerrit stores votes and similar in git no wtoo so it isn't just when you push changes but when reviews are made aiui. is listed as updating an hou rago but the last chnge merged more than that19:42
mnaseryeah i have stuff which havent had any reviews but still showing last updates all same time19:52
clarkbmnaser: I wonder if that update is from the logo update19:55
clarkbI wonder if the db update trips a timestamp update19:56
clarkbthe timing seems to line up for that19:57
clarkband the airship repos show the same thing which lends more evidence to that being the case19:58
*** dviroel is now known as dviroel|out21:37
fungi#status log Rebooted all 12 Zuul executors for Linux kernel updates to address CVE-2022-2563621:41
opendevstatusfungi: finished logging21:41
fungicorvus: ^ in doing that, the executors have restarted onto 5.0.1.dev188 2a50989821:42
mordredclarkb: I honestly don't remember. I believe it has to do primarily with clearing out that tc.log file, which exists for XA transaction support and will be happily re-created if needed if it doesn't exist.22:25
mordredthe file is created on start ... that location would be in the volume we bind-mount in22:27
clarkbmordred: ya the location is bind mounted but the file doesn't exist. I wonder if we just don't make use of whatever generates it22:59
clarkbfungi: I think if we can get landed an early week restart of gerrit next week is a good idea23:00
clarkbthough happy for tha tto get approved early next week if preferred23:00
clarkbyou're probably enjoying a beer and bbq23:00
clarkbso no rush :)23:01
funginah, nothing so exciting as all that. i can take a look23:02
clarkbthat will get us the release that they just made today23:02
clarkbnothing super important out of it for our uses that I can see but just general fixes which are good to keep up with23:03
* fungi accidentally nicks himself on the bleeding edge23:03
opendevreviewGuillaume Chauvel proposed opendev/system-config master: Move xenial distutils check in OS specific files.

Generated by 2.17.3 by Marius Gedminas - find it at!