Friday, 2023-02-17

clarkbthe osf/refstack -> openinfra/refstack redirect works. Logos are not all happy here: and that is beacuse those orgs don't exist in projects.yaml anymore.00:01
clarkbI was worried about that sort of thing with the transplanted database, but I suspect fixing that is simpler than doing a fresh db update to address the redirects00:01
clarkbI think I can manually copy the files in /var/gitea/data/gitea/avatars to address that. Trying now00:02
fungiyeah, maybe we should document a step of rsyncing those and/or include them in backups00:04
clarkbI'm going to finish cleaning those up and then push up changes for the next steps00:04
clarkband really we could have a step that creates the old orgs00:04
clarkbas step 0 in sorting out redirect creation directly rather than as a db transplant00:04
opendevreviewClark Boylan proposed opendev/system-config master: Add gitea09 as a Gerrit replication target
opendevreviewClark Boylan proposed opendev/system-config master: Add gitea09 to the gitea load balancer
opendevreviewClark Boylan proposed opendev/system-config master: Convert gitea99 test node to Jammy
clarkbI'm going to WIP those first two changes. The first one because I'd like to land it tomorrow when I can monitor the potentially very long sync from gerrit (also I need to update the host vars for it). And the second because we need to be replicating before it is part of the lb pool00:08
clarkbinfra-root ^ if you want to poke around gitea09 both on the host and via the web ( that would be appreciated just to look out for any oddities from the db transplant00:09
clarkbThen assuming you're happy with the results please review the above changes which will get it into production00:09
clarkbfungi: fwiw I believe gitea01 does have backups which should include the non git portions of gitea00:11
clarkbbut we can/should doulbe check that00:11
*** soniya29 is now known as soniya29|ruck03:36
*** soniya29|ruck is now known as soniya29|ruck|lunch07:57
*** marios is now known as marios|ruck08:23
*** jpena|off is now known as jpena08:36
*** soniya29|ruck|lunch is now known as soniya29|ruck08:54
mnasiadkafrickler: is there a guide how to encrypt secrets? seems we need to update the docker hub secret in Kolla09:44
fricklermnasiadka: this is what is linked from what I posted yesterday
mnasiadkaah, thanks, missed that10:34
mnasiadkawas mangling with zuul-client10:34
fungimnasiadka: also the zuu side documentation is at and
fungiwe need to update that section of the infra-manual to refer to zuul-client12:27
fungiit hasn't been updated since zuul-client existed12:28
fungibut usage is similar to the old script12:28
*** artom_ is now known as artom12:31
fricklerinteresting issue arising from the update:
fricklerI guess nobody else has needed to change secrets that are defined in branched repos yeet?12:38
frickleror similar12:39
fricklerdo we have to rename secrets every time we branch? or drop them from stable branches completely?12:40
fungifrickler: i thought zuul was supposed to ignore secrets if they're defined on other branches, but maybe that has changed or regressed12:42
opendevreviewJeremy Stanley proposed opendev/infra-manual master: Update subsection on Using Secrets
fungifrickler: ^ as discussed13:03
fricklerfungi: thx, I didn't realize that the old script reference was also obsolete. also I need to test the --infile option, haven't used that one before13:10
fungithe prior example used --infile too, i didn't change that part (though i did test to make sure it works as written there)13:10
opendevreviewJeremy Stanley proposed opendev/infra-manual master: Update subsection on Using Secrets
*** dasm|off is now known as dasm14:11
clarkbI've updated host vars for and I think that can be approved as soon as we are comfortable. Note the comment around reloading replication configs (won't be automatic but I think we can use the `gerrit plugin reload replication` command to avoid a gerrit restart16:27
clarkbI even managed to make sure I was talking to the correct ssh port16:27
opendevreviewMerged opendev/infra-manual master: Update subsection on Using Secrets
*** jpena is now known as jpena|off17:01
clarkbfungi: re should we continue to proceed with that today? I feel like its on the borderline between send it and be cautious during openstack feature freeze. In theory landing it won't make any impacts until we reload the replication plugin or restart gerrit17:50
clarkbbut I'm also happy to try and find time for that next week between meetings if wewant feature freeze to burn itself out first17:50
clarkbI've got some zuul reviews I should do now anyway17:50
fungiclarkb: today seems like a fine time to push forward there17:58
clarkbok want to approve the change?17:59
opendevreviewMerged opendev/system-config master: Add gitea09 as a Gerrit replication target
clarkbok that didn't actually enqueue infra-prod-review or whatever the job is18:10
* clarkb writes a followup change to fix that that should trigger he job18:10
opendevreviewClark Boylan proposed opendev/system-config master: Trigger infra-prod-service-review when review02 hostvars update
clarkbfungi: ^18:12
fungiunrelated, clarkb do you have an opinion on ?18:13
clarkbfungi: approved.18:15
fungithanks! i'll take care of the cleanup once it rolls out18:15
opendevreviewMerged opendev/system-config master: Redirect openstack-infra specs to opendev docs
opendevreviewMerged opendev/system-config master: Trigger infra-prod-service-review when review02 hostvars update
clarkbinfra-prod-bootstrap-bridge is running in opendev-prod-hourly at the same time as infra-prod-letsencrypt in deploy. I don't think this is an issue but it is unexpected19:03
clarkbinfra-prod-bootstrap-bridge doesn't share the semaphore I think because what it does is actually pretty minimal so this should be fine19:05
clarkbI can't tell if my fix job change is actually going to enqueue the job i wnt though :( might need another followup to force it19:14
clarkbah we don't see the jobs until they are done waiting for the semaphore?19:28
* clarkb attempts patience19:28
fungii think my specs index change is blocking it19:49
clarkbya no job ran20:08
clarkbthe file doesn't exist because we store everything in secret vars.20:09
clarkbSo I think some of these public things like host keys could be moved into the public file? Since we use review99 in CI?20:10
clarkbLets sort that out another time though. /me figures out how to trigger this job20:10
fungioh, yeah20:12
opendevreviewClark Boylan proposed opendev/system-config master: Update Gerrit role readme
clarkbthats a bit of an easy mode so I don't have to trigger the job by hand20:15
clarkbI need to go figure out lunch now too20:15
opendevreviewMerged openstack/project-config master: Step 3: Remove project from infra System - Retire xstatic-font-awesome
clarkbfungi: just a heads up that I'll probably be doing a school run when the change finally lands. But that should be fine as it is typically a few more minutes until the job runs21:55
clarkbfungi: once the job has run I think the next steps are to `gerrit plugin reload replication` and then trigger replication for a small repo to see if all is well. Then trigger it for everything against gitea0921:56
fungii'll be around, but probably eating dinner soon21:58
opendevreviewMerged opendev/system-config master: Update Gerrit role readme
fungideploy completed at 22:42:32 utc22:44
Clark[m]Cool I'm on my way home now22:45
clarkbfungi: ok I'll reload the replication plugin now and ask bindep to replicate22:57
clarkbhrm remote plugin administration is disabled os that didn't work22:58
clarkbok remoate plugin admin defaults to false and we don't override that. I think we need to do a gerrit restart.23:00
clarkbI didn't expect this. Maybe the thing to do is consider a config change that has the replication plugin automatically reload its config. Then restart to pick that up as well as the gitea09 addition23:01
clarkbthis way we don't need to do a bunch of restarts as we roll new giteas in and old out. /me works on that change23:02
opendevreviewClark Boylan proposed opendev/system-config master: Enable Gerrit replication autoreload
clarkbI think that may be our best option. But I'm open to ideas. Not sure I want to do a gerrit restart right now during openstack feature freeze23:07
clarkbre ideas: I think we can enable remote plugin admin including plugin installs in order to manually reload the replication plugin (I think this has the same dropping events problem that auto reloads of the config has though)23:09
clarkbI'd rather take autoreloading with that problem than create potential plugin management issues23:10
fungiclarkb: one of the reasons we have that disabled is that it used to wipe the replication queue on reload. no idea if it still does, but it resulted in incomplete mirrors missing objects/commits23:20
clarkbfungi: yup I made note of that in my change. I think if we make `gerrit plugin replication reload` work it would have the same problem23:21
clarkbso I prefer auto to that to avoid opening the whole can of worms for remote admin of all the plugin things23:21
clarkbIf we prefer gerrit restarts we can do that too. Mostly thinking we could do this temporarily and perform a full sync after each config reload or something23:22
clarkbthen when giteas are all rotated in/out undo the config23:22
clarkbfungi: if we feel more comfortable with restarts that fine too. Feel free to -1 the change.23:25
clarkbI just didn't want to do a restart until we had considered making it so future restrats aren't necessary23:25

Generated by 2.17.3 by Marius Gedminas - find it at!