Wednesday, 2022-03-02

clarkbI'm not sure we want a new super group to contain zuul and the registry. Double accounting is probably the best option :/00:00
jentoioregistry.yaml has registry_user: zuul00:02
clarkbjentoio: that appears to configure the user inside the docker registry service itself00:05
clarkb(I didn't realize these layers might make things confusing)00:05
clarkbjentoio: maybe registry_process_user_id: 10001 and registry_process_group_id: 10001 as well as registry_process_user and registry_process_gruop to help differentiate? 00:06
jentoioyeah, its a little confusing. You have a container user (inside) and the user the docker-compose is running as00:07
jentoiookay, so we want to separate the registry from zuul, that makes sence. I will work on this task assuming a new registry user per your examples00:08
clarkbya. Maybe add comments to registry.yaml as well to explain the difference even though they are similar.00:08
jentoiosounds good, start here system-config/inventory/service/group_vars/registry.yaml00:10
jentoioI'll start here..00:10
clarkband dont' be afraid of pushing something that seems half done. We can usually takl about things more conretely in review once we have something to look at00:12
*** ysandeep|out is now known as ysandeep00:15
fungii constantly push half-done work into gerrit, just so i don't have to worry about accidentally blowing it away on my workstation when i switch to some other task00:28
jentoiohow do I push my changes to gerrit again? I don't see it in gerrit after git commit01:35
opendevreviewJack Morgan proposed opendev/system-config master: Adds support for running zuul-registry as a non-root user
fungithe `git review` command will add a remote named "gerrit" and will also download the commit hook which adds the change-id footer to your commit message and will rerun it if needed01:37
fungiaha, you found the docs, i guess ;)01:37
jentoionevermind, figured it out01:37
fungijentoio: in case you found other less accurate docs01:39
jentoioI used this one, opendev/infra-manual/latest/developers.html01:40
jentoiobut didn't scrol down enough to see submitting a change for review. Got stuck at running unit tests ;)01:40
jentoioDo I need to add reviews in gerrit?01:47
*** ysandeep is now known as ysandeep|afk01:49
*** pojadhav|afk is now known as pojadhav01:50
fungijentoio: add reviewers? nah01:54
fungijust pester people in here if we don't get around to looking at it01:54
jentoiowell, 1st one (zuul-registry) was kinda fun. Once its done, I'll take the learnings and apply them to other containers.01:58
fungiyeah, hopefully we actually exercise that one with tests, more than just making sure it deploys... seeing if we do01:59
*** rlandy|ruck is now known as rlandy|out02:00
fungiwe do, not a lot, but we check that the service starts enough to bind a listening socket, so that's better than nothing:
Clark[m]I'll be sure to review it tomorrow morning02:11
ianwjentoio: thanks!  i'm pretty sure there's a typo in there, but what's more interesting than that is why that typo didn't cause a CI failure?02:20
ianwthis did actually fail :
ianwi think i've screwed up catching the error with the | tee to logs02:37
opendevreviewIan Wienand proposed opendev/system-config master: zuul run-base: make sure we catch failures when teeing to logs
*** ysandeep|afk is now known as ysandeep03:37
opendevreviewIan Wienand proposed zuul/zuul-jobs master: ensure-pip: test with install/upgrade of pip
ianwfrickler: thanks for investigating this, i've approved your other change.  if you think ^ might avoid even more problems, we can go with it, but i'm not that fussed04:09
opendevreviewMerged zuul/zuul-jobs master: Fix ensure-pip test on Debian Buster
opendevreviewJack Morgan proposed opendev/system-config master: Fixed minor spelling mistake
jentoioWell, I fixed my spelling mistake but I created a new commit instead of updating my previous commit. Need to read the docs more.04:49
ianwjentoio: in short, update whatever it is, "git add -i" and then "git commit --amend", then push.  gerrit will know it's the same because the change-id in the commit message doesn't update05:01
jentoioianw: thanks, for pointer. 05:04
ianwonce you've gone with amending/rebase you'll never go back :)  the fact there was a typo is of historical interest in the gerrit review, where we can discuss such things.  but it's not something we want to commit to the tree, which should just be the final revision :)05:07
*** ykarel_ is now known as ykarel06:39
*** ysandeep is now known as ysandeep|lunch08:13
fricklerianw: I like the idea, but if I do this on buster, pip is already at the latest version, which may not be as good a test since it doesn't change anything. I'm also wondering why the affected jobs aren't executed for either of our patches, I guess we should fix that, too08:15
frickleralso your comment change now failed on f35 with what looks like broken repo mirrors. does this never stop?08:17
fricklerfinally, whom to ping about the Software Factory CI node failures?08:19
*** pojadhav- is now known as pojadhav08:24
*** jpena|off is now known as jpena08:34
*** ysandeep|lunch is now known as ysandeep08:57
ianwfrickler: haha, it hasn't ended in the 9 years or so i've been here, so i guess not :)08:57
ianwyeah, the --reinstall should get it to try again?  but there might be better ideas too08:58
ianwi noticed the fedora thing first in zuul-repository, which is for some reason using fedora nodes.  i haven't looked, i don't have time right now but will in the morning if nobody else does! :)08:59
*** ykarel_ is now known as ykarel09:54
fricklerI checked that the rsync job is finishing fine, so likely we are tracking some broken upstream mirror. will need someone with more fedora knowledge to verify10:06
*** rlandy|out is now known as rlandy|ruck10:51
*** ysandeep is now known as ysandeep|afk10:51
*** ysandeep|afk is now known as ysandeep11:05
*** pojadhav is now known as pojadhav|brb11:58
*** pojadhav|brb is now known as pojadhav12:42
opendevreviewSzymon Datko proposed zuul/zuul-jobs master: [ensure-python] Improve check for CentOS/RHEL 9 packages
opendevreviewSzymon Datko proposed zuul/zuul-jobs master: [ensure-python] Improve check for CentOS/RHEL 9 packages
tristanCfrickler: i've disabled the Software Factory CI jobs that produce in node failure with 13:37
fricklertristanC: great, thx13:39
*** ykarel_ is now known as ykarel14:17
*** timburke__ is now known as timburke15:23
*** ykarel is now known as ykarel|away15:40
*** ysandeep is now known as ysandeep|out15:51
dpawlikprobably this mirror was unsynchronized  15:51
dpawlikthat's why we have such error15:52
dpawlikfrickler, fungi: should we switch fedora mirror to other?15:54
fungidpawlik: our record of picking mirrors is apparently not great...
fungican you get a good recommendation?15:59
fungior should we switch back to mit's mirror again?16:00
* dpawlik asking folks for recommendation 16:01
mnasiadkawhat's the latest on rocky linux 8 nodes availability?16:02
dpawlikwondering if Facebook mirror is not stable 16:02
dpawlikfungi: I will propose a patch to switch to (
fungimnasiadka: they're available, though they may not yet be booting in all of our providers, they're at least in place enough to be able to try out16:08
fungidpawlik: the rsync socket on that server says not to use rsync directly and instead mirror with
dpawlikfungi: nhicher propose tu take this one
dpawlikfungi: oh, same repo 16:12
dpawlikbut then it can be more difficult to find what is an issue16:13
clarkbfungi: mnasiadka: the dib update for our nodepool builders did land yesterday so I think any new problems are unknown at this point? Definitely in the try it out and lets see how it goes stage16:18
opendevreviewdaniel.pawlik proposed opendev/system-config master: Change Fedora mirror to
opendevreviewdaniel.pawlik proposed opendev/system-config master: Change Fedora mirror to
clarkbfungi: looks like an important one16:21
clarkbdpawlik: is the master mirror. I'm pretty sure they don't want us using that16:24
clarkbdpawlik: typically the master mirrors are meant to only be pulled from the tier 1 mirrors and then you talk to those or tier 216:24
clarkbiirc the rules for being tier 1 involve being publicly accessible which we aren't really (even though we are, we don't want to advertise them because they come and go, for example we just shutdown a mirror yesterday)16:25
clarkb describes this. Anyway I think you need to find an alternative tier 2 mirror is what I'm getting at16:26
dpawlikclarkb: yeah, I see on doing a test that it does not work as expected16:26
dpawlikclarkb: send me later your PS. I will check. Now I need to go16:30
clarkbwell I don't have any suggestions for a better mirror. Just saying we cannot use the one that is proposed16:36
*** marios is now known as marios|out16:44
opendevreviewMerged opendev/system-config master: zuul run-base: make sure we catch failures when teeing to logs
clarkbjentoio: left some notes on the change. Overall looks good, but we should use the variables consistently or remove them (as noted on the review). Thanks again. I'll be sure to rereview once we've got updates16:53
jentoioclarkb: okay, i'll make the change but I just copied the gerritbot example,
clarkbjentoio: ya I think we can just drop the variables and use the hardcoded value for the name and group name17:26
clarkbjentoio: that would be in line with gerritbot17:27
jentoioclarkb: nm, I'm understanding your gerrit comments now that the coffee is kicking in17:31
jentoioi will abondon my 2nd commit,, revert my git tree back to 1st commit and make the changes there17:32
clarkbsounds good17:32
fungijentoio: i meant to link it last night, but we have documentation on the change revision workflow too if you ever need a reminder:
*** jpena is now known as jpena|off17:44
yoctozeptoI report there is something wrong with gitea18:06
yoctozeptowhen I git clone/fetch from gitea18:07
yoctozeptoI only get 83fa9079611b359e390d3a7a0f2c7aa935a4494618:07
clarkbyoctozepto: can you be more specific?18:07
yoctozeptoas the latest commit18:07
yoctozeptoclarkb: yes, sorry, writing multiple lines18:07
yoctozeptobad habit18:07
yoctozeptonewer commits are not propagated18:07
clarkbok so the clone is functional but the data is old18:07
yoctozeptothe web ui shows new commits well18:08
yoctozeptothe gerrit endpoint also serves fresh data18:08
yoctozeptodiscovered doing rebases and figured out I'm not current with gerrit heh18:08
clarkbyoctozepto: can you go to and check the names in the cert to see which backend you map to18:11
clarkbI see some errors in the replication log that may be specific to gitea0118:11
clarkbwant to corss check with where you are pulling from18:11
clarkbI'm checking gitea01 and gieta02 now to compare as well18:13
clarkbya gitea02 seems up to date and gitea01 is not. I'm going to try forcing full replication to gitea01 of kolla-ansible18:13
clarkbthis may be related to the gitea01 outage we had18:14
clarkband if that corrects it I'll ask for a full replication of gitea01 for everything I Guess18:14
clarkbok that didn't seem to help still getting errors. Says "reason: missing necessary objects"18:17
clarkbBut this is a force push so you'd think it would push the missing objects?18:17
clarkbinfra-root I've pulled gitea01 out of the haproxy rotation18:20
clarkbI suspect that we may need to move the repo aside and recreate it. But I'm not currently sure how to do that so I've pulled the server out of rotation so the other 7 can serve the up to date data18:21
clarkbyoctozepto: ^ can you check that you get up to date content now that I've pulled thsi out of rotation?18:21
clarkband git fsck says error: object file ./objects/0a/a678cd8d48015b9db75cc5bd926660bd7aeb89 is empty and then complains it is corrupt18:23
clarkbalmost certainly related to the server dying a painful death recently.18:24
clarkbinfra-root ^ any ideas on the best way to approach this? We should be able to move the repo aside and have gerrit re replicate it but I'm unsure how to make gitea aware of that cleanly18:24
clarkbI think we can shutdown all of gitea, move the unhappy repo aside, git init a new repo, start gitea, tell gerrit to replicate18:28
clarkber shutdown all gitea services on gitea01. not the entire gitea service18:33
clarkbinfra-root ^ if that plan seems reasonable I'll start doing that18:34
clarkbmy only real concern with it I guess is I'm not sure what flags gitea is using to init the repo18:35
clarkbI should probably go look and see if I can determine how it does that18:35
clarkblooks like git init and if bare is set then --bare and I think these are bare repos18:36
clarkbfunc InitRepository in gitea/modules/git/repo.go18:37
yoctozeptoclarkb: sorry, I went out for a moment18:38
yoctozeptoclarkb: yes, it's up to date now18:38
yoctozeptothank you!18:39
clarkbyoctozepto: thanks, I think I've identified the issue then. I'ev stopped the gitea services on gitea01 while I prepare to make a bare repo then tell gerrit to rereplicate it18:39
fungiclarkb: any guess as to whether this could also be impacting other repos on the server?18:39
clarkbfungi: I think the answer is yes it is possible, but no I do not know of any that are impacted other than this one18:41
clarkbsince we can take it back out of the rotation again later easily I think fixing this one and proving out the system is fine18:41
clarkbthen we can audit and do additional fixes if necessary18:41
fungijust wondering if it might make more sense to blow away all the repos on the server and repopulate them for good measure18:42
clarkbmaybe, but we don't even know if this surgery will work18:44
fungiyeah, i agree it's a good first experiment. if that seems to fix it, i guess we can do the rest easily in a shell loop18:46
clarkbif anyone wants to check I've put the sad repo in /root/corrupt_repos and the kolla-ansible.git in /var/gitea/..... is a new git init --bare repo that I chowned to gitea18:47
clarkbI think I'm ready to start gitea up again and ask gerrit to replicate.18:47
clarkbok proceeding to start services back up again18:50
opendevreviewJack Morgan proposed opendev/system-config master: Adds support for running zuul-registry as a non-root user
clarkbvisiting gitea01 kolla-ansible via the web ui is a 500 error currently.18:51
jentoiopatchset 2 pushed. thanks for help / feedback18:51
clarkbNo branches in non-empty repository <- that is the error reported by gitea. So maybe replication will fix it18:52
clarkb has content now18:54
clarkbyoctozepto: ^ can you check it please? (note the backend specific url)18:54
clarkbfungi: `grep 'Failed replicate of' /home/gerrit2/review_site/logs/replication_log | grep -v kolla-ansible` is empty implying kolla-ansible is the only active repo with the problem from today's log18:55
fungiclarkb: that seems reasonably safe that it's the only one affected then, unless nothing's tried to fetch commits from some repos on 01 today18:56
clarkbfungi: note that is gerrit's replication log not gitea logs18:57
clarkbfungi: basically no one has pushed changes to repos to gerrit that exhibit the issue18:57
clarkbother than kolla18:57
fungioh, errors on push18:57
clarkbalso no other gitea exhibits the issue18:57
clarkbI changed grep -v kolla-ansible git grep -v gitea01 to check that18:57
fungiand i agree the kolla-ansible repo seems to match gerrit (same commit is my origin/master and gerrit/master)18:57
fungiafter recloning from 0118:58
clarkbif yoctozepto is still around and confirm it looks good I will put gitea01 back into the haproxy rotation. Then maybe I should write up a doc on how to do this18:59
clarkbI suspect that we could've taken another approach of deleting broken things in the old git repo until we got to a "consistent" state then have gerrit push. But starting clean felt well cleaner :)19:00
clarkbI wonder if there are more safeguards we could put into place to avoid these issues in the first place?19:01
clarkbI believe gitea01 is boot from volume and gerrit's git repos are hosted on a volume in a different region of the same cloud19:01
clarkbdoes ceph or whatever backs this maybe have some sort of stronger write back garuntees? cc mnaser 19:01
clarkbif we could opt into that via /etc/fstab options or /sys/ settings that might be a good idea?19:02
clarkbinfra-root I've just realized that we document an admin gitea function to create missing repos. If you like I can move this new repo I created aside and have gitea create it for us that way. This would ensure that gitea is creating it exactly the way it wnts it to be. But I think I found the code and it just does a `git init --bare` so we should be fine19:05
clarkbI suppose if we run into additional problems that would be the next step to take.19:05
opendevreviewClark Boylan proposed opendev/system-config master: Add docs on restoring a gitea repository
clarkbinfra-root ^ that should capture what I did pretty well. Please look it over and if the process looks good we can land the docs update. If you see problems we should address them on gitea01 and then update the docs too19:17
mnaserim trying to capture context, what happened19:20
clarkbmnaser: when the gitea01 server's hypervisor crashed we wrote an empty git object file to the kolla-ansible repo19:23
mnaserfwiw there is no more different region stuff, everything is all teh same place19:23
clarkbmnaser: this corrupted the repo and prevented further pushing to it to keep it replicated. I'm wondering if the volumes provided by vexxhost support certain writeback options that we might be able to set to tryand avoid those problems? Like I know the nvme in my local machines has different write back options. The default is fast and maybe less safe but I can opt into slower and more19:24
clarkbmnaser: well gitea01 is in sjc1 and review is in ca-ymq-1 iirc. But both use ceph backed volumes for the git data iirc19:24
clarkbMostly me talking out loud wondering if there are options we can set to say "this mount is important be more careful"19:24
mnaserok so you're saying the hard power off resulted in data loss19:25
fungimore like data corruption19:25
fungia truncated write, essentially19:25
clarkbya it created the file but with no content so ^19:25
clarkbwe've since recovered because the source of truth is gerrit not gitea01. But my concern is if gerrit were to experience similar we'd be pulling from backups most likely19:25
clarkband wondering if we can mitigate that via mount or device options19:25
fungiprobably the filesystem layer allocated the inode but never actually got a chance to flush buffered data to it before the disruption19:26
mnasiadkaclarkb: seems to launch ok, thanks :)19:26
clarkb/sys/block/nvme.*/queue/write_cache is how linux controls this for nvme devices for example19:27
clarkbdefaults to write back typically but you can set it to write through19:27
clarkbwhich will more than likely affect your write speed, but you get better integrity on power loss19:27
clarkbthese aren't nvme devices as they are ceph backed volumes mounted via iscsi or similar. But wondered if there was similar we could do maybe19:28
fungithough if filesystem allocation and writing out the blocks are performed as separate steps, i'm not sure the caching layer can necessarily solve it19:28
clarkbfungi: thats a good point. And I would've expected git to do atomic renames19:28
clarkbthis may also point to a git bug as it should be more resilient to this sort of thing too19:28
fungiis gitea using cgit behind the scenes?19:30
clarkbfungi: yes19:30
clarkbfungi: well for the writes anyway19:30
clarkbthe reads are done via a golang libarary iirc19:30
fungicool, i couldn't remember. i know gerrit doesn't19:30
clarkbfungi: the way it works is we run an sshd that can execute git commands and then those commands run to accept the pushes and those pushes trigger hooks which update the gitea application side of things19:31
clarkbthe application side then does some things with golang lib and other things with forking cgit tools iirc19:31
mnaseri think ceph handles fsync's correctly19:32
jrosserceph should be synchronous underneath iirc but it presents a block device for volumes so I expect the behaviour of whichever file system used under power loss will dominate what happens19:32
mnaseryep, what jrosser said as well -- now, there is a cache on the host level which combines writes and pushes them out to the cluster19:33
clarkbmnaser: ya so that might be the issue19:34
clarkbdo you know if there is a way to tell the ceph mount to act write through rather than write back?19:34
clarkbI bet that completely destroys performance though19:34
clarkbbut may be worth testing if it is an option19:35
clarkbI'm guessing we won't hear back from yoctozepto today due to timezones. I'll put gitea01 back in rotation after lunch if we don't find any reason to keep it out by then19:40
jrosserceph disk write cache settings are somewhat counterintuitive
jrosserfwiw my Intel nvme come set with the write cache off by default19:43
clarkbjrosser: is it an "enterprise" device? Apparently consumer devices are far less safe19:45
jrosseryes these are enterprise spec drives19:45
clarkbsupposedly the other thing enterprise drives do is include extra capacitors/batteries to ensure writes make it all the way down into non volatile storage and consumer drivers are far less likely to do that (even with write through set, people have started testing this after the apple m1 nvme controller issues with linux came to light)19:46
mnaseri think the issue here is that the writes never made it out to the ceph cluster19:47
mnaserwe didnt lose power to the ceph nodes19:47
mnaserthe problem is the qemu writes didnt make it to the cluster19:47
clarkbmnaser: right. And I think if we were operating in write through mode the software would know the writes hadn't completed yet? But if in write back as soon as it hit the caches the software running ont top thought it was good to go19:48
clarkbthe downside to write through is that it is almost definitely going to be slower. So not necessarily something you'd want to set globally I don't think19:48
mnaserwrite but the performance will be terrible19:49
clarkbya if it is possible to opt in on a per mount basis that would be interesting to benchmark at least19:49
clarkbbut I'm not sure if that is possible. Looking at jrosser's doc it seems this may be set pretty globally?19:49
mnaserid start by making sure the fs level is flushing writes19:49
clarkbglobally per local cache at least19:50
clarkbmnaser: I mean its git19:50
clarkbon ext4, I'd be super surprised if they were doing something naive19:50
clarkb(but it is possible)19:50
clarkbwe are mounted data=ordered which is "All data are forced directly out to the main file system prior to its metadata being committed to the journal."19:52
clarkbbarrier=1 would be another thing to set, we don't set it explicitly and I'm not sure what the default is19:53
clarkbnot sure if git supports that either19:53
clarkbext4 enables write barriers by default19:53
clarkb(ext3 didn't)19:54
clarkbThose seem to be the primary fs options for ensuring integrity (the other is commit=nsec which forces flushes every n seconds)19:56
clarkbthe default for that one is 5 seconds19:56
clarkbanyway we don't have to solve this now. Just wanted to throw ideas out there to see if there were any good options or things we might be missing. Seems like maybe not?19:57
clarkbor at least nothing obvious to me19:58
clarkbfungi: any idea why it doesn't find the Backend Maintenance heading at the top of the file?20:00
clarkboh I may need to enable an extension for that20:01
opendevreviewClark Boylan proposed opendev/system-config master: Add docs on restoring a gitea repository
fungiclarkb: yeah, i occasionally try to use implicit labels in docs and then remember that they're fragile. using explicit labels also helps you keep old bookmarks and external links working even if you rename the section at a later date20:07
fungiso preferable all around20:07
clarkbmakes sense20:08
clarkblunch now. Back in a bit and I can restore gitea01 in haproxy at that point20:11
clarkbgitea01 is back in the rotation now20:58
clarkbmy docs change passes CI now too21:13
clarkbready for review if yall can take a look21:13
fungii've apparently clogged gertty by resubscribing to all the ancient opendev namespace repos so i can review retirement changes on them21:16
fungicatching up on other things while i watch its sync counter slowly decrease21:17
fungiit'll be convenient though, because after approving the retirement changes in them, i can bulk abandon all other open changes21:17
fungicourtesy of gertty's awesome process mark feature21:18
clarkbooh yes21:18
ianwi saw some chat about the fedora mirror but it doesn't seem we merged anything?21:18
clarkbianw: correct the proposal from dpawlik was to use the master repos to sync from which if possible (I don't think it is) is not recommended21:19
clarkbwe need to find another tier 2 mirror to pull from aiaui21:19
fungiwe could just unrevert the mit mirror change if they're looking stable again21:20
fungi(see git history on that file)21:20
ianwok, i need to do school runs but will sort something out after21:20
opendevreviewMerged opendev/system-config master: Add docs on restoring a gitea repository
fungimy gitea sync seems to be about 75% done. hopefully after dinner i'll be able to go back to using it for reviewing stuff21:26
fungi(anyone need a toothpick?)21:26
*** dviroel is now known as dviroel|out21:31
clarkbjentoio: I think I diagnosed the job failure for the latest patchset. I think this is really close. I left a comment on how I think youcan work around it21:31
jentoioclarkb: yup, working on it. patchset 3 coming shortly. thanks21:33
clarkbno rush jsut wanted to make sure you saw that21:35
jentoionp, I just happended to be working on it.21:36
opendevreviewJack Morgan proposed opendev/system-config master: Adds support for running zuul-registry as a non-root user
clarkbjentoio: heh the last error shold've been a hint for me to look closer. Hopefully comment on most recent patchset explains things well enough22:12
jentoioclarkb: yeah, looking into it now. I'll implement your feedback. thanks22:18
opendevreviewJack Morgan proposed opendev/system-config master: Adds support for running zuul-registry as a non-root user
ianw just passed so i'll assume the mirror fixed itself22:31
fungioh good22:37
fungidon't tell the technicians from central services, but sometimes these things *do* fix themselves22:38
fungino need to involve harry tuttle22:38
fungi(though we are all in it together!)22:38
clarkbjentoio: woot that passes testing now. I noticed two more small things. Sorry for not catching them earlier. But I think if we make those updatse and fidn a second reviewers we may be about done here22:59
fungii can be a second reviewer now that my gertty has caught up23:02
clarkbfungi: well I think we need a couple more updates. but passes testing23:02
fungiexcellent work!23:02
clarkband then maybe land tomorrow when we can watch it? I'm feeling like today might be an early one for me23:02
fungiit has seemed like a long day for me too for some reason23:03
jentoioclarkb: cool, taking a look now23:10
corvusfungi: what happened to your gertty?   also, internally it has a priority queue, so if you know what changes you want to review, you can jump right to them and it'll sync them first.23:14
corvusfungi: oh, nm, i found the explanation in scrollback23:14
fungicorvus: i subscribed to all the ancient opendev namespace repos so that i can use it to review clarkb's retirement changes and mass abandon old reviews on those23:14
fungiand yeah, i could have jumped the sync queue, but it was pretty sluggish anyway23:15
fungiso preferred to do other things until it finished23:15
fungimostly it was busy cloning repos and such23:15
* corvus didn't know there were other things to do23:15
opendevreviewJack Morgan proposed opendev/system-config master: Adds support for running zuul-registry as a non-root user
*** rlandy|ruck is now known as rlandy|ruck|bbl23:23

Generated by 2.17.3 by Marius Gedminas - find it at!