20:27:58 <fungi> #startmeeting opendev-maint
20:27:59 <openstack> Meeting started Fri Jun 12 20:27:58 2020 UTC and is due to finish in 60 minutes.  The chair is fungi. Information about MeetBot at http://wiki.debian.org/MeetBot.
20:28:00 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
20:28:02 <openstack> The meeting name has been set to 'opendev_maint'
20:28:09 <fungi> #link https://etherpad.opendev.org/p/gerrit-2020-06-12
20:29:16 <fungi> clarkb: do you recall what file it is we create on bridge.o.o to pause deployments?
20:29:58 <clarkb> fungi: it should be in the etherpad
20:30:11 <clarkb> its also documented in bridges system config dics
20:30:14 <clarkb> *docs
20:30:29 <fungi> in which etherpad?
20:30:48 <fungi> i wasn't finding where we did it in the last one from 2020-03-20
20:31:42 <clarkb> the one I did for this time
20:31:46 * clarkb looks
20:31:50 <fungi> oh, there was another pad?
20:31:55 <fungi> anyway, yeah, docs say /home/zuul/DISABLE-ANSBILE
20:32:29 <clarkb> https://etherpad.opendev.org/p/gerrit-project-renames-20200612
20:32:34 <fungi> #info did `sudo touch /home/zuul/DISABLE-ANSBILE` on bridge.o.o
20:32:46 <fungi> aha, i'll switch to that pad. didn't realize you already had one
20:32:59 <clarkb> I put that together yesterday and libked it in irc. Your url earlier looked similar enough that I thought they were the same
20:33:08 <fungi> #link https://etherpad.opendev.org/p/gerrit-project-renames-20200612
20:33:15 <fungi> nah, i just missed that you had already created one
20:35:38 <fungi> clarkb: the pad says to put review.o.o in emergency disable *and* stop deployments, is the emergency disable list addition actually necessary?
20:36:29 <fungi> also i guess i should pull a copy of renames/20200612.yaml from https://review.opendev.org/735211 onto bridge.o.o
20:41:05 <clarkb> fungi: I don't think it is necessary
20:41:23 <clarkb> and the renames file should already be there at the path I list (double check though)
20:42:29 <fungi> yep, i see now in the pad you already staged it at /root/renames/20200612/20200612.yaml
20:43:31 <fungi> confirmed it's there and checksum matches the one from 735211
20:44:15 <fungi> i've started a screen session as root on bridge.o.o
20:44:33 <clarkb> I've joined it
20:45:09 <corvus> i'm standing by
20:45:54 <fungi> and yeah, since these are all repositories moving to different namespaces i wouldn't have expected any groups to get renamed (as we aren't namespacing groups... yet anyway)
20:46:18 <clarkb> ya I double checked just to be sure fwiw, but there weren't any I saw
20:46:36 <fungi> i looked through the changes as well and didn't see any
20:47:44 <fungi> the acls in them are only git moves, with the exception of cla enforcement added ni one
20:49:49 <fungi> 10 minutes to go. i suppose at the 5 minute mark i can send a status notice that gerrit is being restarted
20:50:31 <clarkb> sounds good
20:52:00 <fungi> something like...
20:52:02 <fungi> status notice The Gerrit service on review.opendev.org is going offline momentarily at 21:00 UTC for project rename maintenance, but should return within a few minutes: http://lists.opendev.org/pipermail/service-announce/2020-June/000004.html
20:52:10 <clarkb> lgtm
20:52:34 <fungi> hopefully if that takes anyone by surprise, it'll be incentive to subscribe to service-announce
20:55:28 <fungi> #status notice The Gerrit service on review.opendev.org is going offline momentarily at 21:00 UTC for project rename maintenance, but should return within a few minutes: http://lists.opendev.org/pipermail/service-announce/2020-June/000004.html
20:55:28 <openstackstatus> fungi: sending notice
20:57:39 <fungi> i've staged the ansible-playbook command from the pad in the screen session, ready to engage at 21:00
20:58:44 <openstackstatus> fungi: finished sending notice
20:58:58 <clarkb> doing last minute sanity checks and the flag to run gerrit init before gerrit start seems to default to false which is what we want
20:59:22 <fungi> thanks
20:59:48 <clarkb> oh we explicitly set it to false when we run it too
21:00:00 <fungi> go time. green light from mission control?
21:00:24 <clarkb> the command looks correct to me
21:00:29 <clarkb> corvus: ^ you ready and go?
21:00:29 <fungi> #info running `ansible-playbook /home/zuul/src/opendev.org/opendev/system-config/playbooks/rename_repos.yaml -e@/root/renames/20200612/20200612.yaml`
21:00:48 <corvus> clarkb: yep
21:01:20 <fungi> repolist is undefined
21:01:33 <fungi> edit repos to repolist?
21:01:41 * clarkb looks at historical lists
21:02:05 <clarkb> it has always been repos before /me looks at the code now
21:02:40 <corvus> -e repolist=ABSOLUTE_PATH_TO_VARS_FILE
21:02:46 <corvus> is the command wrong?
21:02:57 <clarkb> oh ya we didn't do repolist=
21:02:58 <corvus> that ^ is from the docs
21:03:19 <clarkb> is that -e repolist=@filename or -e@repolist=filename?
21:03:25 <fungi> -e@repolist=/root/renames/ or?
21:03:33 <fungi> checking docs
21:03:36 <corvus> i think it should not have @
21:03:45 <corvus> playbooks/rename_repos.yaml:    - include_vars: "{{ repolist }}"
21:03:48 <corvus> cause that's how it's used
21:04:18 <fungi> docs say -e repolist=
21:04:24 <clarkb> corvus: got it and ya I think that is correct. -e repolist=filepath
21:04:57 <fungi> #info running `ansible-playbook /home/zuul/src/opendev.org/opendev/system-config/playbooks/rename_repos.yaml -e repolist=/root/renames/20200612/20200612.yaml`
21:04:59 <clarkb> sorry I took that from the etherpad we used last time and didn't cross check the docs
21:05:14 <fungi> looking better, thanks ;)
21:05:19 <corvus> i updated the etherpad for today to avoid perpetuating the error in case that happens again
21:05:33 <clarkb> ++
21:06:36 <clarkb> hrm looking at zuul status I think our jobs are not pausing
21:06:58 <clarkb> we don't have any hourly gitea or gerrit jobs queued though so we are probably ok
21:07:36 <fungi> the jobs themselves start, but ansible blocks if that file is present, from what i gather
21:07:46 <fungi> okay, i just saw some errors scroll by
21:08:02 <fungi> user id changes biting us?
21:08:13 <corvus> seems likely
21:08:16 <fungi> the project key moves failed
21:08:51 <clarkb> ya that seems likely related to the regrouping and reuiding
21:08:53 <fungi> that seems to have been the only issue it encountered
21:09:09 <clarkb> re the jobs, that is correct but two jobs have succeeded since after you reported touching that file
21:09:19 <clarkb> fungi: it should stop completely once it hits the first error
21:09:32 <fungi> ahh, okay, so there was still more to run
21:09:55 <corvus> i'll make a new playbook
21:10:20 <clarkb> it should be zuuld and ya we should make a new playbook for things that happen from failure forward. thank you corvus
21:10:42 <fungi> yeah, so looks like we're using zuuld on the schdeduler now, not zuul
21:10:51 <clarkb> fungi: ya this was missed when we changed that stuff
21:10:52 <corvus> ~corvus/rename_repos.yaml look good?
21:10:52 <fungi> and the playbook tried to chown to zuul which dne
21:11:14 <clarkb> corvus: yes lgtm
21:11:31 <fungi> yep, that looks like the remaining tasks with the first one fixed to the new username
21:11:59 <corvus> should we move that playbook into place?
21:12:12 <fungi> i was simply going to run `ansible-playbook ~corvus/rename_repos.yaml -e repolist=/root/renames/20200612/20200612.yaml`
21:12:13 <corvus> i can't remember if it references any roles or includes
21:12:23 <corvus> if it does, those will be relative to the playbook path
21:12:24 <clarkb> corvus: it does for the gitea stuff
21:12:25 <fungi> but yeah, i can move it
21:12:29 <clarkb> and for gerrit start
21:12:38 <corvus> then best to cp it to pwd
21:12:39 <clarkb> hold on
21:12:50 <clarkb> if zuul is running jobs the jobs could undo that copy
21:12:51 <corvus> or rather, the system-config/playbooks dir
21:13:23 <clarkb> would it be ebtter to make a copy of the subtree and then copy your edit into that?
21:13:31 <clarkb> (I'm trying to understand what zuul is doing now fwiw)
21:13:54 <fungi> i could just put it at /home/zuul/src/opendev.org/opendev/system-config/playbooks/fixed_rename_repos.yaml and then reference that new playbook name, right?
21:14:06 <fungi> and then clean up the checkout when we're done
21:14:36 <clarkb> fungi: assuming that workspace setup doesn't reset --hard
21:15:21 <fungi> if it does, we'll just get an error at that point anyway, right? non-destructive if so
21:15:40 <clarkb> good point
21:15:43 <clarkb> then ya I think thats the way to do it
21:15:47 <clarkb> corvus: ^ that sound good?
21:16:06 <clarkb> fwiw I only see an older (possibly stale) remoet_puppet else run from zuul ansible
21:16:15 <clarkb> and we're seeing jobs rotate attempts now
21:16:27 <corvus> wfm
21:16:29 <clarkb> so I think we may be good either way
21:16:33 <fungi> #info ran `cp ~corvus/rename_repos.yaml /home/zuul/src/opendev.org/opendev/system-config/playbooks/fixed_rename_repos.yaml`
21:16:54 <fungi> #info running `ansible-playbook /home/zuul/src/opendev.org/opendev/system-config/playbooks/fixed_rename_repos.yaml -e repolist=/root/renames/20200612/20200612.yaml`
21:17:13 <fungi> it's gotten further, thanks
21:18:08 <clarkb> we are to the step wehere we check gerrit is up again and happy before hitting enter
21:18:23 <fungi> yeah, ssh: connect to host review.opendev.org port 29418: Connection refused
21:18:27 <clarkb> the process is running but not yet httpd'ing for me yet
21:19:36 <clarkb> I wnt to say this takes ~3-4 minutes now?
21:19:45 <fungi> yeah, i'm not worried (yet)
21:20:05 <fungi> responding!
21:20:06 <clarkb> [2020-06-12 21:20:00,219] [main] INFO  com.google.gerrit.pgm.Daemon : Gerrit Code Review 2.13.12-11-g1707fec ready
21:20:16 <clarkb> httpd still not there yet
21:20:42 <fungi> well, ssh api is responding so we can proceed
21:20:47 <fungi> any objections?
21:21:07 <clarkb> there are sshd exceptions in the error log
21:21:10 <fungi> (at least our reminder there says "Make sure that Gerrit ssh api is accepting requests" which it seems to be
21:21:13 <clarkb> did you login and run a command?
21:21:17 <fungi> i did
21:21:30 <clarkb> web is up for me now anyway
21:21:30 <fungi> i mean, i asked it for ssh api help, but i can do more
21:21:32 <clarkb> I think we can proceed
21:21:44 <clarkb> I was thinking something like ls-projects but I expect its fine as web responds now too
21:21:45 <fungi> i can get it to list projects
21:21:46 <corvus> i just pushed up a change
21:21:53 <fungi> i'd call that working
21:21:56 <clarkb> cool lets proceed
21:21:56 <fungi> okay, continuing
21:21:58 <corvus> 21:21 < openstackgerrit> James E. Blair proposed opendev/system-config master: Fix rename playbook after zuul user rename  https://review.opendev.org/735397
21:22:13 <fungi> #info gerrit api is responding, continued
21:22:22 <fungi> #info playbook run completed
21:22:31 <fungi> no new errors
21:22:51 <fungi> anybody want to double-check anything before we start approving the rename changes?
21:23:13 <clarkb> if I search for changes on the renamed proejcts I get no results
21:23:21 <clarkb> do we have to wait for reindexing to complete for that to work?
21:23:30 <fungi> yes
21:23:45 <clarkb> fungi: that was the only other thing I was going to check
21:23:46 <fungi> searches hit the index
21:23:55 <clarkb> but if we have to reindex I don't know if we want to wait
21:24:02 <fungi> if you know a change for one of those projects, you should still be able to access it
21:24:11 <clarkb> ah ok let me see
21:24:28 <fungi> queries which involve the project name will fail to find it though
21:25:02 <fungi> once the online reindex reaches the renamed projects, they'll be searchable again
21:25:29 <fungi> which thankfully is long, long before the reindex completes
21:25:58 <fungi> (unless we rename nova, openstack-manuals, neutron, or something similarly large)
21:26:22 <clarkb> https://review.opendev.org/#/c/726006/ errors which I think is a refstack or interop change
21:26:44 <clarkb> java.lang.IllegalArgumentException: passed project openstack/refstack when creating ChangeNotes for 726006, but actual project is osf/refstack
21:26:56 <clarkb> maybe we need reindexing to finish for that to work too?
21:27:05 <fungi> ahh, yeah okay so i guess they can't be reached until the index is done
21:27:20 <fungi> though we don't typically wait for reindex to complete to do the project-config changes
21:27:52 <fungi> but will likely need to wait to be able to update .gtireview files for those repos
21:28:50 <clarkb> I've double checked that refstack hasn't reindexed it (just to be sure the problem is likely related to idnexing) and it has not so thats good
21:29:39 <fungi> corvus: you still have a wip on 714478, mind if i delete it? or do you want to undo that (or approve the change)?
21:29:52 <corvus> i'll +w if we're ready
21:30:31 <clarkb> ya I think I'm about as ready as I'll be without waiting for reindexing to finish
21:30:39 <corvus> done
21:30:44 <fungi> cool, thanks
21:30:47 <clarkb> should we rm the DISABLE-ANSIBLE file to ensure those jobs run properly?
21:30:55 <clarkb> in particular I think we need them to update zuul config
21:31:02 <corvus> that's probably a stellar idea
21:31:22 <fungi> it wasn't clear to me whether we needed to wait until the changes merged to clear DISABLE-ANSIBLE
21:31:54 <fungi> in the past we didn't resume configuration management for the gerrit server until the rename changes merged, so that manage-projects wouldn't recreate old stuff
21:32:05 <mordred> yeah - that's probably a good idea ...
21:32:08 <clarkb> fungi: thats a good point
21:32:11 <mordred> we use tip of project-config
21:32:17 <clarkb> I think the correct thing now is to rm the file bceause of ^
21:32:19 <clarkb> oh wait
21:32:23 <clarkb> no I see
21:32:27 <clarkb> we want them to merge then rm I get it
21:32:29 <mordred> no - I think let's wait to rm until the patches merge
21:32:31 <mordred> yeah
21:32:42 <fungi> yeah, we've scripted changes to match the future state of project-config, not the present satte
21:32:59 <fungi> so if manage-projects runs with the present state, we'll wind up with a mess
21:33:24 <clarkb> yup
21:33:34 <clarkb> and I guess we can always reenqueue if we need to once things have merged
21:33:40 <clarkb> rather than try and sync with zuul perfectly
21:33:54 <clarkb> gerrit is slowly chewing the the largest repos now
21:34:22 <fungi> 734669 fails zuul configuration. are we going to need to bypass gating on that one?
21:35:40 <corvus> that probably could be merged by zuul if it were split into 2 changes
21:35:47 <corvus> change main.yaml first, then the rest.
21:36:05 <corvus> but as written, yes, that would need to be force-merged
21:36:08 <fungi> i can split it up
21:36:22 <corvus> depends on how long you want to stick around.  :)
21:36:35 <fungi> we can always bypass check
21:38:33 <corvus> i'd vote to force-merge
21:38:44 <clarkb> ya I'm ok with force merge too
21:38:57 <fungi> about halfway through the dance to split the patch, but i'll just do that instead
21:39:31 <clarkb> one thing I notice is we haev a lot more "big" repos to reindex now
21:39:39 <clarkb> we've only finished a handful so far
21:39:45 <clarkb> but it is progressing
21:40:07 <clarkb> also we should really do that gerrit disk cleanup I never do because its scary
21:40:14 <clarkb> we haev 23GB free where the git repos and indexes live
21:40:20 <clarkb> (enough for now but steadily getting smaller)
21:40:37 <mordred> yeah. we have a LOT of saved indexes in the homedir too
21:40:47 <clarkb> mordred: ya exactly
21:41:17 <fungi> #info bypassed gating on 734669 so as not to spend time splitting the change during maintenance
21:41:27 <clarkb> zuul's reindexing now
21:41:35 <clarkb> that means zuul is one of our bigger repos :)
21:42:57 <clarkb> fungi: re removing the disable file has the change replicated first?
21:43:07 <clarkb> I think we want to wait for that as well?
21:43:12 <fungi> i haven't removed it yet
21:43:21 <fungi> just documenting in the pad that it's a step
21:43:23 <clarkb> (I want to say reindexing and replication are different worker threads so that should work well)
21:44:25 <fungi> though yes, i guess the question remains whether we need to make sure something is replicated to gitea before we turn jobs back on... they use branch tip so do they get that directly from gerrit or gitea?
21:44:46 <mordred> gitea
21:44:47 <clarkb> I want to say it is from gitea
21:44:53 <mordred> definitely gitea
21:45:13 <fungi> okay, so we need to make sure the project-config changes have replicated to gitea i guess
21:45:25 <clarkb> ugh local network has suddenly gone flaky for me so whwen I run the show-queue commands ssh says review.o.o is unreachable. Then I run it again and it is reachable
21:45:42 <clarkb> its not review I notice it typing in this shell for irc too
21:46:11 <fungi> i've noted the additional wait condition in the pad
21:47:08 <fungi> #link https://opendev.org/openstack/project-config/commits/branch/master
21:47:18 <fungi> i see merge commits for both rename changes
21:47:28 <fungi> so we should be safe?
21:48:53 <clarkb> I like the check the individual backends and I just do that they all look good
21:49:12 <fungi> okay, thanks for confirming
21:50:02 <clarkb> progress is picking up with reindexing which is what we expect since it starts with the most expensive ones and proceeds to the cheaper ones
21:50:08 <fungi> so we should be all clear to remove /home/zuul/DISABLE-ANSIBLE now
21:50:22 <clarkb> fungi: yes I think so
21:50:26 <clarkb> mordred: corvus ^ do you agree?
21:50:43 <mordred> ++
21:50:57 <corvus> abstain (weak agree)
21:52:02 <fungi> so i was *going* to remove it
21:52:09 <fungi> but it's no longer there
21:52:24 <fungi> i wonder if that explains the behavior clarkb was seeing
21:52:31 <mordred> uh - what removed it?
21:52:41 <fungi> i'd love to know
21:52:47 <mordred> fungi: I see it
21:53:01 <mordred> root@bridge:/home/zuul# ls -ltra /home/zuul/DISABLE-ANSBILE
21:53:03 <mordred> -rw-r--r-- 1 root root 0 Jun 12 20:32 /home/zuul/DISABLE-ANSBILE
21:53:07 <mordred> oh
21:53:09 <mordred> nope
21:53:14 <mordred> I see a file named incorrectly
21:53:57 <clarkb> oh ya looking at scrollback thats the file that was there when I was trying to figure out why things were running
21:54:08 <mordred> this is maybe a stupid idea - but maybe we should make a helper utitlity called "disable-ansible" that touches the file
21:54:10 <clarkb> so uh
21:54:14 <mordred> so that we can dis<tab>
21:54:17 <clarkb> we may have recreated the refstack and interop repos?
21:54:17 <mordred> and not worry about misspelling
21:54:24 <mordred> yeah. we probably did
21:54:31 <corvus> ansibile is great
21:54:35 <fungi> wow
21:54:44 <mordred> corvus: so is ansbile
21:54:51 <fungi> i picked the ONE place in our documentation where it's mistyped, and didn't notice
21:55:08 <fungi> https://docs.opendev.org/opendev/system-config/latest/bridge.html#running-ansible-on-nodes "When done, don’t forget to remove /home/zuul/DISABLE-ANSBILE"
21:55:19 <corvus> okay, so how do we remove those repos?
21:55:32 <mordred> I think we have to shut gerrit down again
21:55:53 <mordred> I'm not 100% sure what the right choice is on the gitea replicas
21:56:02 <mordred> since I'm guessing the new repos will have removed the redirects
21:56:07 <clarkb> mordred: they did I just checked
21:56:22 <mordred> I mean - obvs we can do db surgery to fix the redirects
21:56:31 <mordred> but that'll take some investigation
21:56:41 <clarkb> can we even delete projects?
21:56:42 <corvus> i'll start working on the gitea stuff
21:56:47 <clarkb> in gerrit I mean
21:56:57 <clarkb> or do we toggle some flag saying this isn't visible instead?
21:57:07 <corvus> please shut gerrit down now
21:57:18 <clarkb> can we safely stop it if it is reindexing?
21:57:25 <clarkb> I assume so since its online
21:57:28 <corvus> because we can delete projects if they haven't changed
21:57:32 <clarkb> gotcha
21:57:44 <fungi> #status notice gerrit is being taken offline for emergency cleanup, will return to service again shortly
21:57:44 <openstackstatus> fungi: sending notice
21:57:46 <mordred> yeah - I think worst case it'll just be another reindex
21:58:24 <fungi> do we use docker-compose to take it down now?
21:58:38 <clarkb> fungi: yes, cd /etc/gerrit-compose && sudo docker-compose down
21:58:41 <mordred> yeah
21:58:44 <fungi> doing that now
21:59:10 <fungi> it's stopped
21:59:21 <mordred> corvus: removing from gerrit is removing the git repo - do we need to delete from db there too?
21:59:25 * mordred can work on that
21:59:36 <fungi> yes, needs to be removed from the mysql db
21:59:36 <corvus> mordred: dunno
21:59:40 <fungi> same tables we rename in
22:00:37 <fungi> account_project_watches where project_name = oldname
22:00:55 <openstackstatus> fungi: finished sending notice
22:01:01 <fungi> changes where dest_project_name = oldname
22:01:20 <fungi> though hopefully those match zero rows
22:01:22 <clarkb> I found https://stackoverflow.com/questions/8723716/delete-a-project-in-gerrit which seems to indicate that removing the git dirs may be sufficient on some versions of gerrit (maybe those with notedb though?)
22:01:31 <mordred> yeah - probably so
22:01:47 <mordred> now I'm wondering why /root/.my.cnf doesn't have the ssl=true line that's in our ansible config management
22:01:58 <mordred> but maybe that can be a task for older monty
22:01:59 <fungi> clarkb: yeah, i think that stuff winds up in special refs in those repos with notedb
22:02:26 <fungi> but also if nobody created new changes or watches on those recreated repos, it's likely we won't have anything to remove from the db
22:02:42 <mordred> fungi: yeah - I'm going to quick select to see if that's true
22:02:53 <corvus> for gitea -- how about we use the web ui to delete the old name, rename the new name to the old, then rename the old to the new?
22:03:13 <clarkb> corvus: we should be able to test that on a single backend too to ensure it works properly
22:03:17 <clarkb> but that plan sounds good to me
22:03:22 <mordred> me too
22:03:23 <clarkb> separately, should we touch that ansible file now?
22:03:26 <clarkb> the proper one?
22:03:28 <mordred> yes
22:03:32 <corvus> clarkb: yes; i expect it too because i did that when testing renames originally
22:03:37 <fungi> corvus: that sounds like it should create redirects again... though doesn't our redirect creation also have the same effect if it reruns after we delete the old repos?
22:03:56 <fungi> (part of the reason for recording those in yaml)
22:04:02 <clarkb> fungi: no
22:04:07 <clarkb> that hasn't been implemented yet
22:04:12 <clarkb> we've been reocrding things so that we can do that
22:04:13 <fungi> oh, okay
22:04:22 <fungi> sometimes i confuse the future with the present, sorry :/
22:04:35 <clarkb> I'm touching the ansible disable file now
22:04:37 <corvus> we could just manually delete the old names in gitea and then run that part of the playbook
22:05:02 <corvus> or rather, manually delete the old names, move the new to the old, then run the playbook
22:05:05 <corvus> that only saves one step though
22:05:21 <clarkb> corvus: ya I think your plan was great myself and gets us to the end state we want
22:05:27 <fungi> yep, agreed
22:05:31 <clarkb> does anyone else want to double check the file in /home/zuul on bridge/
22:06:12 <mordred> I do not believe there are any gerrit db changes
22:06:15 <clarkb> thinking out loud on how to address zuul running in the future. Maybe we should edit authorized keys?
22:06:18 <mordred> mysql> select * from account_project_watches where project_name in ('openstack/refstack', 'x/whitebox-tempest-plugin', 'openstack/python-tempestconf', 'openstack/refstack-client');
22:06:20 <mordred> Empty set (0.01 sec)
22:06:25 <fungi> clarkb: it looks correct this time, thanks
22:06:27 <mordred> mysql> select * from changes where dest_project_name in ('openstack/refstack', 'x/whitebox-tempest-plugin', 'openstack/python-tempestconf', 'openstack/refstack-client');
22:06:29 <mordred> Empty set (0.64 sec)
22:06:54 <clarkb> mordred: is there a general projects table too?
22:07:04 <clarkb> basically what does ls-projects read?
22:07:14 <mordred> no. no projects table
22:07:16 <fungi> clarkb: i expect the current method for disablement is just fine as long as we type it correctly
22:07:32 <clarkb> fungi: sure, but its fragile whereas breaking ssh is robust
22:07:33 <mordred> I believe it reads from a cache that's related to git repos on disk
22:07:39 <clarkb> mordred: rgr
22:07:53 <clarkb> mordred: and to confirm I t hink what I hear you saying is we only need to remove the git dirs?
22:07:58 <fungi> i should have looked more closely at the command i cut and pasted, and not just assumed our documentation is somehow magically immune to typos
22:08:02 <mordred> clarkb: yes - I believe that is correct
22:08:27 <mordred> clarkb, fungi: I think making a utility script to set the file would be a very easy one-liner and prevent future mistakes while keeping things simple
22:08:38 <fungi> that sounds reasonable
22:08:41 <clarkb> I like that
22:08:56 <mordred> clarkb: I will now delete the git repos
22:09:19 <mordred> clarkb: I think you already have done that
22:09:23 <mordred> as I do not see them
22:09:55 <fungi> or manage-projects never actually made it that far?
22:10:01 <clarkb> mordred: I have not
22:10:13 <mordred> perhaps we only made the projects in gitea
22:10:14 <clarkb> fungi: manage projects succeeded
22:10:18 <clarkb> the job I mean
22:10:48 <mordred> well - that's disturbing - because there are no projects named that in gerrit now
22:10:50 <fungi> does it only apply repos which changed, or do we do a full run when we trigger it?
22:11:02 <mordred> OH
22:11:04 <mordred> you know what?
22:11:04 <corvus> https://gitea01.opendev.org:3000/openstack/interop   should be complete
22:11:13 <mordred> those repos are going to be in the manage-projects cache on gerrit
22:11:25 <mordred> so manage-projects isn't going to try to re-created them gerrit side - purely accidentally
22:11:43 <clarkb> mordred: yup I'm reading the logs on bridge now and I concur that is what happened
22:11:48 <mordred> \o/
22:11:49 <clarkb> it basically said "my shas match so I'm good"
22:12:02 <mordred> cool. so gerrit is actually fine (and will remain fine)
22:12:03 <clarkb> someone else may want to double check that manage projects logs on brdige to concur too but that is my read of it
22:12:19 <clarkb> corvus: redirect wfm there
22:12:45 <mordred> yes - I concur
22:13:00 <mordred> corvus: did you do that by clicking things?
22:13:29 <corvus> mordred: yes
22:14:26 <fungi> okay, so we're ready to start gerrit again?
22:14:44 <corvus> do we want to wait until gitea is done?  replication
22:15:02 <clarkb> ya that way everything works properly when it starts functioning
22:15:12 <fungi> oh, right, that, sorry missed it wasn't done on all the backends
22:15:42 <mordred> corvus: yeah - I think that's safest
22:15:58 <fungi> i also have the documentation correction for "ansbile" ready to push once it's up (interesting fact, it's not the only place in our documentation where we did that)
22:16:12 <corvus> im confused -- 1 sec
22:16:48 <corvus> i don't see an x/whitebox-tempest-plugin; it still has a redirect
22:17:00 <corvus> maybe it just didn't get around to it?
22:18:24 <clarkb> corvus: that one is ok
22:18:35 <clarkb> corvus: because it was in the first chagne that merged which is what ran manage-projects
22:18:43 <corvus> ah ok
22:18:45 <clarkb> its was the gap between first and second changes merging that bit us
22:18:59 <clarkb> and so only ssecond change's projects are sad (I noted the list on the etherpad too)
22:18:59 <fungi> oh, right
22:19:14 <fungi> so if the three-rename change had merged first we'd only have to correct x/whitebox-tempest-plugin
22:20:14 <clarkb> yes
22:21:03 <corvus> okay i've done all the things for gitea01
22:21:21 <corvus> are other folks idle?  should we just divide up the clicking?
22:21:30 <corvus> it's silly, but it's probably ~= to the time to make a playbook
22:21:43 <clarkb> I can click buttons if we decide that is easiest
22:21:44 <mordred> I can click. should we divide up backends?
22:22:00 <corvus> k;  ya'll grab the root password from bridge
22:22:07 <corvus> i'll lay stuff out in the etherpad
22:22:09 <mordred> corvus: it's in hostvars right?
22:22:12 <corvus> yep
22:22:28 <corvus> git grep gitea_root_password
22:23:00 <clarkb> yup found it. is the username admin?
22:23:04 <corvus> root
22:23:14 <clarkb> heh that makes sense given the key name
22:24:10 <mordred> corvus has done 01 - actually, why don't I make an etherpad list and we can grab backends
22:24:50 <mordred> corvus has done it better than me
22:24:50 <corvus> oh heh i was about to say i like that better
22:25:11 <corvus> okay we'll do it my way :)
22:25:24 <corvus> clarkb, mordred: etherpad make sense? gtg?
22:25:31 <mordred> corvus: we go to /login ?
22:25:34 <corvus> oh right
22:25:36 <corvus> /user/login
22:26:06 <mordred> that did not work :(
22:26:27 <mordred> nevermind. I'm dumb
22:26:30 <clarkb> https://gitea0X.opendev.org:3000/user/login
22:26:32 <clarkb> that worked for me
22:26:56 <corvus> then click the settings in the top right for the repo
22:27:13 <fungi> grabbing backends might get me in trouble, but i'll try one anyway
22:27:18 <corvus> you could put that on the url too, but i liked to see the repo readme
22:27:22 <fungi> stole 8 from corvus
22:27:24 <corvus> fungi: take 8
22:27:25 <corvus> good
22:31:59 <mordred> k. 2 is done
22:32:26 <clarkb> 4 is done
22:34:58 <corvus> 6,7 done
22:35:15 <clarkb> 5 is done now
22:35:19 <mordred> 3 is done
22:35:48 <mordred> also - let me express how happy I am that clicking things is not our normal job
22:36:33 <clarkb> once 8 is done we start gerrit, trigger reindexing, check things are happy again then delete the zuul ansible disable file?
22:36:39 <fungi> i'm so unbelievably terrible at it
22:36:44 <fungi> almost there
22:37:03 <clarkb> fungi: you need to get yourself a gamer mouse
22:37:15 <fungi> just one double-rename left
22:39:30 <fungi> and fniished
22:39:38 <fungi> so i can start gerrit again now?
22:39:53 <corvus> go from me
22:39:58 <clarkb> I think so. commadn for that is docker-compose up -d in the the /etc/gerrit-compose dir
22:40:21 <fungi> it's on its way up again now
22:40:31 <fungi> #info started gerrit again after cleanup
22:41:47 <mordred> woot
22:41:51 <mordred> that was exciting
22:42:11 <fungi> #link https://review.opendev.org/735400 Correct "ansbile" typos
22:42:17 <clarkb> now we need to trigger reindexing I think
22:43:29 <fungi> status notice The Gerrit service on review.opendev.org is available again
22:43:34 <fungi> should we? ^
22:43:42 <clarkb> fungi: I think we should trigger reindexing first?
22:44:08 <mordred> yeah
22:44:21 <fungi> i can do that now
22:44:24 <fungi> gerrit index start accounts --force
22:44:30 <fungi> gerrit index start changes --force
22:44:35 <fungi> is what the playbook does
22:44:41 <clarkb> ya that should be what we want
22:45:56 <fungi> #info restarted reindexing for accounts and changes
22:47:02 <clarkb> I saw all the tasks queue up and now gerrit is nomming on them
22:47:13 <clarkb> now I Think we can maybe make the notice?
22:47:22 <corvus> ++
22:47:27 <fungi> #status notice The Gerrit service on review.opendev.org is available again
22:47:27 <openstackstatus> fungi: sending notice
22:48:27 <fungi> other than keeping an eye on things (particularly reindexing) i guess we're basically done?
22:48:44 <fungi> someone will also need to push .gitreview updates to the 5 repos i suppose
22:48:51 <fungi> but doesn't need to be us necessarily
22:48:54 <clarkb> and removing the ansible zuul disable when happy with things (is that now?)
22:49:01 <fungi> aha, right
22:49:03 <fungi> that
22:49:16 <clarkb> I think we should remove that then watch manage projects run (which is queued up and waiting)
22:49:28 <fungi> i expect we are? and yes to watching
22:49:28 <clarkb> the one queued up should be for the most recent merge so all of that should be happy now
22:50:13 <clarkb> ya I think we are
22:50:19 <clarkb> mordred: corvus ^ any other thoughts before we do that?
22:50:43 <openstackstatus> fungi: finished sending notice
22:51:04 <fungi> once that's cleared, i may switch to a more evening-appropriate part of my residence
22:51:29 <fungi> but will still be around to check/test stuff, and fix anythnig else we don't know is broken
22:54:26 <clarkb> I'm around, but also enjoying tea becaues it is cold and rainy
22:55:49 <mordred> clarkb: yeah - I thnk so
22:55:55 <mordred> (like, it sounds good)
22:55:58 <clarkb> fungi: ^ do you want to remove it or should I/
22:57:25 <fungi> i'm happy to
22:57:40 <fungi> and done
22:57:48 <fungi> i removed the stray mistyped one as well
22:58:21 <fungi> #info ran `rm /home/zuul/DISABLE-ANSIBLE` to reenable deployments
22:59:06 <fungi> once again, sorry about all that, i should have looked more closely at commands i was cutting and pasting
22:59:47 <clarkb> we're back to having zuul get indexed so things are moving along there
23:00:51 <fungi> i guess once we see the manage-projects run complete as expected, we can endmeeting. probably no need to wait for reindexing as that will be hours still
23:04:59 <clarkb> if anyone is wondering run cloud launcher is running on bridge
23:05:08 <clarkb> its just slow because of the previous disable and now it has to do its things
23:09:29 <clarkb> cloud launcher is finsihing up now I think manage projects should be next?
23:09:36 <clarkb> I want to say priority is such that that happens
23:09:52 <clarkb> openstack/gnocchi is being reindexed
23:10:09 <clarkb> I wonder if there are ways to optimize this ti ignore ancient history until the end
23:11:17 <fungi> which may not be what you want if you wind up renaming ancient history for some reason
23:12:01 <fungi> right now it optimizes to fill all the parallel workers with the largest workloads first so that everything is finished sooner
23:13:15 <clarkb> it is moving pretty quickly now, enough of the big repos have cleared out and it can work on the smaller ones
23:16:22 <clarkb> mange projects ran
23:16:42 <clarkb> redirects for openstack/interop still work
23:18:30 <fungi> awesome
23:18:58 <fungi> shall we call it a wrap?
23:19:16 <clarkb> I think so? It would be nice to confirm that that refstack change I linked earlier works once reindexing hits refstack
23:19:23 <clarkb> but thats the only thing outstanding that I can see
23:19:45 <clarkb> and looks like reindexing may be done for it /me tries it
23:19:49 <fungi> ls-projects lists a osf/refstack and no openstack/refstack
23:20:07 <clarkb> https://review.opendev.org/#/c/726006/ loads now with no http 500
23:20:49 <mnaser> before i mucharound too much
23:20:57 <mnaser> https://review.opendev.org/#/c/714480/ was a change i wanted to land relating to the rename
23:21:14 <mnaser> it lists "Cannot Merge" but i just rebased it to the tip of the master (Add njohnston liaison preference)
23:21:27 <mnaser> and the dependency is merged too?
23:21:37 <fungi> the "cannot merge" may have been from before reindexing completed
23:21:52 <mnaser> it looks like after my rebase, now zuul also thinks it cannot merge either
23:22:20 <fungi> zuul may not have reconfigured yet, if our configuration management hasn't gotten around to triggering a reload of the tenant config
23:22:24 <clarkb> I think zuul asks gerrit
23:22:29 <clarkb> fungi: that change is to governance
23:22:30 <fungi> ahh, or that
23:22:40 <fungi> oh, hrm
23:22:44 <mnaser> yeah and it depends on a project-config change
23:22:50 <mnaser> so really its not an 'affected' project
23:22:54 <clarkb> but we can check the zuul logs for it to see why zuul is unhappy
23:22:57 <mordred> clarkb, fungi: don't forget about: https://review.opendev.org/#/c/735401/ - I imagine we'll forget about it if we don't grab it today
23:23:10 <fungi> gerrit doesn't take depends-on into account for showing "cannot merge"
23:23:11 <mordred> clarkb: and https://review.opendev.org/#/c/735400/ from fungi
23:23:57 <mnaser> fungi: right -- so that's why when i rebased to the most recently merged change, i figured it.. should be ok?
23:24:08 <mnaser> i mean, i can pull it down and fully push back under another commit id but..
23:24:17 <fungi> also claims "Patch in Merge Conflict"
23:25:44 <mnaser> i mean if we think its just a weird thing inside openstack/governance, i can pull it down and push it back up
23:26:00 <fungi> mnaser: i see a merge conflict with that change and head of master over reference/projects.yaml
23:26:02 <mnaser> under another commit id and that might unblock whatever is stuck
23:26:20 <fungi> maybe gerrit/zuul aren't wrong?
23:26:27 <mnaser> fungi: i'm curious as to how/why gerrit let me rebase it through the UI
23:26:36 <mnaser> usually it would whine about a merge conflict in the UI
23:26:36 <clarkb> oh you used the ui
23:26:57 <mnaser> i should have clarified earlier, it sounds like the ui has some dark magic?
23:27:05 <clarkb> well the ui uses jgit
23:27:09 <clarkb> so weirdness to be expected :P
23:27:20 <clarkb> I'm still tracking down how zuul failed
23:27:21 <fungi> ahh, yeah i really don't know what the ui does for its magic/remote rebase
23:27:44 <clarkb> we don't seem to log the merger that merges things though :/
23:27:54 <mnaser> let me cherry-pick it locally and see if git on my machine complains
23:27:55 <fungi> clarkb: i expect zuul is actually unhappy because that change is really merge-conflicting with current master state
23:28:02 <mnaser> if it does, maybe that's why zuul is unhappy
23:28:03 <mnaser> ^
23:28:23 <fungi> yeah, i mean, i tried to rebase it on master locally just now which is how i know it's conflicting over reference/projects.yaml
23:28:47 <mnaser> yeah, it conflicts locally
23:29:04 <mnaser> fungi: always a step ahead of me :)
23:29:16 <clarkb> ERROR: content conflict in reference/projects.yaml
23:29:23 <clarkb> zuul tripped that too
23:29:30 <clarkb> so ya I think its actually in an unmergable state
23:29:41 <clarkb> I wonder if rebase in gerrit ui relies on the index to check
23:29:47 <clarkb> and if index is stale you just get a weird result
23:30:04 <clarkb> only nova and openstack-manuals are reindexing now
23:30:21 <fungi> i want to say that gerrit's rebase function is for when you have a parent change which gets a new patchset and you want your change rebased on the new state of the parent
23:30:21 <clarkb> so ya I'd push a new ps properly rebased using git and see if it complains from there
23:31:01 <clarkb> or maybe if it has a conflict it just rebases to the most recent thing that doesn't conflict
23:31:08 <clarkb> because it did change the HEAD^
23:31:19 <mnaser> it is weird, because sometimes it does also complain about merge conflict in the UI too
23:31:22 <clarkb> and its different than what the latest ps is at
23:31:23 <mnaser> (anyways, pushed it up and its good now)
23:31:43 <fungi> cool. maybe this story has a happier ending in newer gerrit
23:32:21 <clarkb> mordred: bug on https://review.opendev.org/#/c/735401/2
23:32:51 <fungi> mnaser: regardless, thanks for bringing that up, thankfully i don't see anything to suggest it indicates a problem with the maintenance
23:33:10 <mnaser> yeah, i figured it would be good to have an early signal _incase_ it was something
23:33:23 <mnaser> but no we can all have our rest of friday :
23:33:24 <mnaser> :)
23:33:38 <fungi> yay!
23:34:42 <mordred> clarkb: fixed - and thus why we code-review :)
23:34:47 <clarkb> I've detached from the root screen on brdige
23:34:51 <clarkb> I think we can probably shut it down now?
23:35:16 <clarkb> mordred: +2 now.
23:36:04 <mordred> fungi: got a sec for a +A on 735401?
23:36:57 <clarkb> I've rechecked https://review.opendev.org/#/c/735397/ as its gating was lost by gerrit being down I think
23:37:26 <clarkb> everything else looks merged or on its way except for 735491
23:37:31 <clarkb> *401
23:38:18 <fungi> lookin
23:39:04 <clarkb> I'm going to take a break for a bit now. Will check back in a bit to see if nova and openstack manuals are done reindexing
23:39:08 <clarkb> but I think this is looking happy now
23:39:44 <fungi> okay, shall i endmeeting and declare the maintenance done? reindexing should take care of itself
23:40:34 <clarkb> ++
23:45:41 <fungi> #endmeeting