| *** yamamoto has joined #openstack-infra | 00:02 | |
| *** mattw4 has quit IRC | 00:02 | |
| clarkb | corvus: we also need an update to the zuul tenant config right? | 00:03 |
|---|---|---|
| clarkb | your zuul key file moves made me think of that | 00:03 |
| *** igordc has quit IRC | 00:04 | |
| fungi | oh, great point | 00:04 |
| *** yamamoto has quit IRC | 00:08 | |
| *** yamamoto has joined #openstack-infra | 00:08 | |
| *** samueldmq has quit IRC | 00:08 | |
| fungi | i've now got to the point where it's producing `git diff` outputs like this per project+branch: http://paste.openstack.org/show/749517/ | 00:12 |
| *** eernst has quit IRC | 00:16 | |
| *** eernst has joined #openstack-infra | 00:16 | |
| fungi | so the basic pattern matching and tempfile handling seems to be working | 00:16 |
| corvus | clarkb: yes... fungi maybe we can just add "zuul/main.yaml" to the list of files to rewrite? :) | 00:16 |
| fungi | yep | 00:16 |
| fungi | added | 00:19 |
| *** diablo_rojo has quit IRC | 00:22 | |
| fungi | is there one true and canonical mailing list archive entry i should link in commit messages? | 00:25 |
| fungi | seems like http://lists.openstack.org/pipermail/openstack-infra/2019-April/006314.html may have been the first announcement to the infra ml? | 00:26 |
| corvus | hrm. or perhaps at least reasonably truthy? :) | 00:26 |
| corvus | http://lists.openstack.org/pipermail/openstack-discuss/2019-March/003603.html has more background but is less current | 00:26 |
| corvus | and http://lists.openstack.org/pipermail/openstack-discuss/2019-April/004920.html is relevant to some (most?) repositories | 00:28 |
| corvus | fungi: i think the answer to your question is "of course not" :) | 00:28 |
| fungi | i'll link both for good measure | 00:28 |
| corvus | yeah, i think multiple links is the way to go here | 00:29 |
| fungi | is https://opendev.org/ broken for anyone else? | 00:29 |
| corvus | fungi: no, but i'm on ipv4, are you using v6 now? | 00:29 |
| fungi | nope, i'm tethered to a wireless modem using ipv4 | 00:29 |
| fungi | i'll just blame t-mobile | 00:30 |
| fungi | our broadband is still offline | 00:30 |
| * fungi mutters | 00:30 | |
| fungi | can someone who has a real internet connection remind me whether we have contact info displayed at that homepage? | 00:30 |
| corvus | perhaps they object to the magenta logo? | 00:30 |
| fungi | that's likely it | 00:30 |
| corvus | fungi: yes, mailing list and irc | 00:31 |
| fungi | excellent, thanks! | 00:31 |
| corvus | it's even under a section headed "Contact info" | 00:31 |
| fungi | i'll just link to the main url for now but we can tweak in review | 00:31 |
| *** gyee has quit IRC | 00:31 | |
| corvus | https://opendev.org/#contact-info if you want to deep-link (we do have anchors!) | 00:32 |
| corvus | also, we should update that page, since it says we're going to move gerrit later. | 00:32 |
| *** markvoelker has quit IRC | 00:37 | |
| mordred | corvus: time is relative? | 00:41 |
| corvus | time is an illusion. lunchtime doubly so. | 00:41 |
| * corvus saves fungi from typing that over the wireless modem | 00:41 | |
| fungi | thank you h2g2 | 00:42 |
| fungi | nearly done now. i think all that's left is the gerrit acl file moves | 00:51 |
| *** eernst has quit IRC | 00:54 | |
| *** eernst has joined #openstack-infra | 00:55 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: web: add triggers information to pipeline list https://review.openstack.org/637670 | 00:55 |
| openstackgerrit | Tristan Cacqueray proposed openstack/diskimage-builder master: openssh-server: enforce sshd config https://review.openstack.org/653890 | 00:58 |
| *** mriedem has quit IRC | 00:59 | |
| *** michael-beaver has quit IRC | 01:00 | |
| *** nicolasbock has quit IRC | 01:09 | |
| *** ricolin has joined #openstack-infra | 01:09 | |
| *** zhangfei has joined #openstack-infra | 01:17 | |
| *** zhangfei has quit IRC | 01:23 | |
| *** zhangfei has joined #openstack-infra | 01:25 | |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/system-config master: Add a opendev migration repo rename scripts https://review.openstack.org/653138 | 01:33 |
| fungi | there it is now minus the git mv logic for the acl files | 01:34 |
| fungi | clarkb: corvus: mordred: ^ | 01:34 |
| fungi | i'm doing a benchmark run on it to see how long to expect it to take tomorrow | 01:34 |
| clarkb | fungi: for the run now should we go aheadandpush up the projects.yaml and zuul tenant config changes? | 01:36 |
| clarkb | that way they can go through ci and all that? | 01:36 |
| clarkb | if we need a new ps tomorrow we can do that | 01:36 |
| fungi | that's certainly an option | 01:37 |
| fungi | once i get the acl file moves added i can push a copy of the result up for review | 01:37 |
| fungi | though spot checks of the entire corpus are probably easier done locally on review.o.o | 01:37 |
| fungi | i'm rsyncing a fresh copy into /opt/fungi-testing/ now and will start the opendev-patching run for it once it concludes, though i need to generate a more updated repos.yaml with the opendev-migrate script first | 01:39 |
| fungi | okay, test run is underway | 01:43 |
| *** hwoarang has quit IRC | 01:46 | |
| *** hwoarang has joined #openstack-infra | 01:48 | |
| fungi | and i've got the acl mover routine written now too, just waiting for the current test run to complete so we can get some metrics | 01:56 |
| fungi | i also tweaked the regex not consider _ and - as possible separators for repository names, just in case we end up with any unwanted substring matches from that (though they're likely to be benign if we do, easy to include at least, unlike . which also tends to get used as a separator in filenames, like for extensions) | 01:58 |
| *** bhavikdbavishi1 has joined #openstack-infra | 01:59 | |
| *** bhavikdbavishi has quit IRC | 01:59 | |
| *** bhavikdbavishi1 is now known as bhavikdbavishi | 01:59 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: web: honor allowed-labels setting in the REST API https://review.openstack.org/653895 | 02:01 |
| clarkb | fungi: not sure I grok the thing about _/- | 02:04 |
| clarkb | we split on / right? | 02:04 |
| fungi | if we're moving (made-up example) openstack/nova and not openstack/nova-tempest-plugin then the regex seeing - as a possible boundary for the match could result in the latter getting the former's rules applied | 02:05 |
| clarkb | ah | 02:05 |
| fungi | ([^a-z0-9_-]|^)(%s)([^a-z0-9_-]|$) | 02:06 |
| fungi | is what we're searching for in lines of text | 02:07 |
| fungi | where %s is a list of possible repository name | 02:07 |
| fungi | s | 02:07 |
| fungi | previously i had ([^a-z0-9]|^)(%s)([^a-z0-9]|$) | 02:08 |
| fungi | and in an even earlier iteration is was something like ([ /]|^)(%s)([ /]|$) | 02:09 |
| fungi | before i realized i was missing lines where there was a = before the repository name (for example in .gitreview files) | 02:09 |
| fungi | okay, not too shabby... 32m34.109s (real) | 02:16 |
| fungi | so we should plan tomorrow for roughly half an hour to patch all the repos | 02:16 |
| clarkb | and is the plan to push all that into gerrit or do it in place or? | 02:16 |
| fungi | push it locally on the filesystem | 02:17 |
| clarkb | bypassing gerrit? | 02:17 |
| fungi | the script clones from the /home/gerrit2/review_site/git bare repos into temporary directories (context-managed so there's only one existing at a time) and then pushes directly back to the corresponding branches of the bare repos after committing edits for them | 02:18 |
| fungi | bypassing the gerrit software, right | 02:18 |
| *** roman_g has quit IRC | 02:18 | |
| fungi | all local file i/o on the review server | 02:19 |
| clarkb | ok so we need to plan to do that with gerrit off, to avoid any races | 02:20 |
| clarkb | though zuul is unlikely to merge anything before hand anyway :) | 02:20 |
| fungi | yes, that was the expectation | 02:20 |
| fungi | basically do this as part of the rename outage for the gerrit service | 02:21 |
| clarkb | got it | 02:21 |
| fungi | along with running the rename_repos playbook | 02:21 |
| fungi | i'm just testing the acl file moves now and then will push up the project-config change for preliminary review | 02:22 |
| clarkb | sounds good tha ks | 02:26 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/system-config master: Add a opendev migration repo rename scripts https://review.openstack.org/653138 | 02:36 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/project-config master: OpenDev Migration Patch https://review.openstack.org/653900 | 02:38 |
| fungi | clarkb: corvus: mordred: ^ that's the change the script would push locally for project-config on the gerrit server | 02:38 |
| fungi | zuul is, of course, instantly unhappy with it because of the zuul configuration changes | 02:39 |
| mordred | silly zuul | 02:41 |
| fungi | i mean, understandably unhappy as we're renaming a ton of projects in zuul/main.yaml to things which are not (yet) present on gerrit | 02:42 |
| fungi | my faculties are waning... anybody know why the environment variables i set in https://review.openstack.org/653138 tools/opendev-patching line 30 aren't used by git commit? | 02:48 |
| *** jamesmcarthur has joined #openstack-infra | 02:48 | |
| fungi | yay! looks like my real internet connection finally returned to its definition of operational | 02:49 |
| *** rosmaita has left #openstack-infra | 02:50 | |
| mordred | fungi: yay! | 02:51 |
| mordred | fungi: mine has gone away, along with the power | 02:51 |
| fungi | :/ | 02:51 |
| mordred | yay for tethering laptops to phones! | 02:51 |
| fungi | nideed | 02:52 |
| *** jamesmcarthur has quit IRC | 02:53 | |
| clarkb | fungi: not at first glance | 02:58 |
| *** jamesmcarthur has joined #openstack-infra | 02:58 | |
| clarkb | arethey the right env vars for git? | 02:59 |
| mnaser | clarkb: on my googling it seems like they are | 03:00 |
| fungi | yeah, i'm trying to figure out which manpage covers those | 03:00 |
| mnaser | apparently | 03:00 |
| mnaser | Popen starts a new process so it maybe doesn't super properly inherit the env | 03:01 |
| mnaser | maybe subprocess.call might work better, I dunno. | 03:01 |
| mordred | the use of env should make that work | 03:01 |
| mnaser | "If env is not None, it must be a mapping that defines the environment variables for the new process; these are used instead of inheriting the current process’ environment, which is the default behavior." | 03:02 |
| mnaser | yeah.. | 03:02 |
| mordred | mnaser, fungi: maybe you can't use update like that | 03:02 |
| mnaser | oh maybe os.environ is weird like that | 03:03 |
| fungi | oh! i think you may be right | 03:03 |
| mordred | update is in place modification, it doesn't return anything | 03:03 |
| fungi | i'll move the update out to a named dict | 03:03 |
| mordred | yeah | 03:03 |
| fungi | hurr | 03:03 |
| mnaser | there is a relationship between the type of mistakes you do and how much experience in software dev you have :p | 03:04 |
| mnaser | the more you do, the less likely you consider fundamental stuff as solutions :p | 03:04 |
| * mordred thinks update should return the dict it's updating | 03:04 | |
| fungi | for some reason i assumed it did, but yeah | 03:05 |
| mnaser | it's the logical thing | 03:05 |
| *** irclogbot_2 has quit IRC | 03:05 | |
| mnaser | but then again words.join(' ') is the logical thing too but python be python | 03:05 |
| *** edmondsw has quit IRC | 03:08 | |
| *** irclogbot_0 has joined #openstack-infra | 03:08 | |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/system-config master: Add a opendev migration repo rename scripts https://review.openstack.org/653138 | 03:14 |
| fungi | yep, updating a named dict and then using that seems to have solved it, as expected | 03:14 |
| fungi | also i put a few more comments in | 03:14 |
| clarkb | with that weshould be quite ready now? | 03:17 |
| fungi | i believe so | 03:18 |
| fungi | i'm leaving comments on that change indicating how i've been invoking it | 03:18 |
| clarkb | looking at the project config acl moves I wouldve expected more | 03:19 |
| clarkb | jjb/jenkins-job-builder for example | 03:19 |
| fungi | hrm, yeah | 03:21 |
| fungi | oh! | 03:21 |
| fungi | i need to ensure the parent directories for new namespaces are created | 03:22 |
| fungi | fix coming | 03:22 |
| clarkb | skimming the gerrit/projects.yaml and zuul.d/projects.yaml things look as I expect them | 03:25 |
| clarkb | we may have to disable the sort checkers on some of these files | 03:27 |
| clarkb | or plan to do followups that sort things | 03:28 |
| fungi | yeah, likely so | 03:28 |
| clarkb | but I'm not seeing any other errors | 03:28 |
| *** eernst has quit IRC | 03:28 | |
| *** jamesmcarthur has quit IRC | 03:32 | |
| *** markvoelker has joined #openstack-infra | 03:33 | |
| *** d34dh0r53 has joined #openstack-infra | 03:36 | |
| *** cloudnull has joined #openstack-infra | 03:37 | |
| *** eglute has joined #openstack-infra | 03:39 | |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/system-config master: Add a opendev migration repo rename scripts https://review.openstack.org/653138 | 03:42 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/project-config master: OpenDev Migration Patch https://review.openstack.org/653900 | 03:42 |
| fungi | now with more acl! | 03:42 |
| clarkb | woo | 03:43 |
| fungi | i added a bunch of notes in a review comment on https://review.openstack.org/653138 too | 03:50 |
| fungi | since i likely won't be around when these scripts get run, i want to make sure i have provided adequate guidance (as well as sufficient code comments to aid in debugging, should that become necessary) | 03:51 |
| fungi | also feel free to take over the change and make adjustments to commit message wording, author/committer info or whatever else | 03:53 |
| fungi | it's approaching midnight here, so i really need to disappear | 03:53 |
| *** bhavikdbavishi has quit IRC | 03:58 | |
| *** bhavikdbavishi has joined #openstack-infra | 04:00 | |
| *** jamesmcarthur has joined #openstack-infra | 04:01 | |
| clarkb | good night | 04:01 |
| fungi | thanks! you too | 04:04 |
| fungi | i'll likely already be out of the house doing anniversary things by 15:00z, but i'll try to check in before i leave and whenever possible when i'm back | 04:05 |
| fungi | and plan to be around all weekend/week to fix all the things | 04:06 |
| *** imacdonn has quit IRC | 04:08 | |
| *** jamesmcarthur has quit IRC | 04:08 | |
| *** imacdonn has joined #openstack-infra | 04:08 | |
| *** kjackal has joined #openstack-infra | 04:34 | |
| *** mujahidali has joined #openstack-infra | 04:36 | |
| *** bhavikdbavishi1 has joined #openstack-infra | 04:36 | |
| *** bhavikdbavishi has quit IRC | 04:37 | |
| *** bhavikdbavishi1 is now known as bhavikdbavishi | 04:37 | |
| *** jamesmcarthur has joined #openstack-infra | 04:44 | |
| *** kjackal has quit IRC | 04:45 | |
| *** jamesmcarthur has quit IRC | 04:48 | |
| *** lpetrut has joined #openstack-infra | 05:12 | |
| *** mujahidali has quit IRC | 05:19 | |
| *** jamesmcarthur has joined #openstack-infra | 05:21 | |
| *** jamesmcarthur has quit IRC | 05:27 | |
| *** mujahidali has joined #openstack-infra | 05:34 | |
| *** kjackal has joined #openstack-infra | 05:44 | |
| openstackgerrit | Masayuki Igawa proposed openstack/os-testr master: Deprecate ostestr command https://review.openstack.org/573636 | 05:56 |
| openstackgerrit | Masayuki Igawa proposed openstack/os-testr master: Deprecate ostestr command https://review.openstack.org/573636 | 05:58 |
| *** kjackal has quit IRC | 06:14 | |
| *** kjackal has joined #openstack-infra | 06:14 | |
| *** whoami-rajat has joined #openstack-infra | 06:15 | |
| *** ykarel|away has joined #openstack-infra | 06:15 | |
| *** pgaxatte has joined #openstack-infra | 06:16 | |
| *** jamesmcarthur has joined #openstack-infra | 06:23 | |
| *** e0ne has joined #openstack-infra | 06:26 | |
| *** markvoelker has quit IRC | 06:27 | |
| *** e0ne has quit IRC | 06:28 | |
| *** jamesmcarthur has quit IRC | 06:28 | |
| *** sajauddin has joined #openstack-infra | 06:29 | |
| sajauddin | I am from PowerKVM team, working on Openstack CI for Power hardware. | 06:30 |
| sajauddin | I am trying to setup ZuulV3 so that we can upgrade existing ZuulV2 node to ZuulV3 | 06:30 |
| sajauddin | Installation part is done. Trying to configure dependent nodes like nodepool, gerrit and setup few pipelines/jobs for testing | 06:32 |
| sajauddin | Can anyone help me with any documentation for reference? Anyway I am using zuul official documentation. | 06:32 |
| *** lpetrut has quit IRC | 06:34 | |
| *** e0ne has joined #openstack-infra | 06:36 | |
| *** e0ne has quit IRC | 06:38 | |
| *** e0ne has joined #openstack-infra | 06:38 | |
| *** chandankumar is now known as raukadah | 06:40 | |
| *** mujahidali has quit IRC | 06:42 | |
| *** apetrich has quit IRC | 06:44 | |
| *** jamesmcarthur has joined #openstack-infra | 06:45 | |
| *** sajauddin has quit IRC | 06:51 | |
| *** jamesmcarthur has quit IRC | 06:51 | |
| *** rpittau|afk is now known as rpittau | 06:52 | |
| *** pcaruana has joined #openstack-infra | 07:12 | |
| *** yolanda_ has joined #openstack-infra | 07:24 | |
| *** bhavikdbavishi has quit IRC | 07:25 | |
| *** bhavikdbavishi has joined #openstack-infra | 07:25 | |
| *** jamesmcarthur has joined #openstack-infra | 07:27 | |
| *** markvoelker has joined #openstack-infra | 07:27 | |
| *** dciabrin_ has joined #openstack-infra | 07:29 | |
| *** jamesmcarthur has quit IRC | 07:32 | |
| *** lpetrut has joined #openstack-infra | 07:34 | |
| *** e0ne has quit IRC | 07:37 | |
| *** liuyulong has quit IRC | 07:39 | |
| *** e0ne has joined #openstack-infra | 07:43 | |
| *** ykarel|away has quit IRC | 07:50 | |
| *** ykarel has joined #openstack-infra | 07:50 | |
| *** slaweq has quit IRC | 07:53 | |
| *** bhavikdbavishi has quit IRC | 07:54 | |
| *** e0ne has quit IRC | 07:55 | |
| *** helenafm has joined #openstack-infra | 07:59 | |
| *** bhavikdbavishi has joined #openstack-infra | 08:00 | |
| *** markvoelker has quit IRC | 08:01 | |
| *** e0ne has joined #openstack-infra | 08:04 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Sort gerrit/projects.yaml https://review.openstack.org/653918 | 08:05 |
| *** kjackal has quit IRC | 08:06 | |
| *** e0ne has quit IRC | 08:07 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Sort zuul/main.yaml again https://review.openstack.org/653919 | 08:09 |
| *** jamesmcarthur has joined #openstack-infra | 08:12 | |
| *** e0ne has joined #openstack-infra | 08:15 | |
| *** jamesmcarthur has quit IRC | 08:17 | |
| *** bhavikdbavishi has quit IRC | 08:18 | |
| *** sajauddin has joined #openstack-infra | 08:18 | |
| *** tkajinam has quit IRC | 08:19 | |
| *** whoami-rajat has quit IRC | 08:24 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Sort zuul.d/projects.yaml https://review.openstack.org/653922 | 08:25 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Fix starlingx queue https://review.openstack.org/653923 | 08:25 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Add system-required back for repos https://review.openstack.org/653924 | 08:25 |
| AJaeger | infra-root, above are a couple of cleanups on top of fungi's large change 653900 to allow linters to work again. I thought it easier to make them separate changes... | 08:26 |
| *** ykarel is now known as ykarel|away | 08:27 | |
| *** kjackal has joined #openstack-infra | 08:28 | |
| *** electrofelix has joined #openstack-infra | 08:32 | |
| *** e0ne has quit IRC | 08:35 | |
| *** yamamoto has quit IRC | 08:41 | |
| *** e0ne has joined #openstack-infra | 08:41 | |
| *** e0ne has quit IRC | 08:43 | |
| *** ykarel|away has quit IRC | 08:44 | |
| *** yamamoto has joined #openstack-infra | 08:44 | |
| sajauddin | Can anyone help me in setting up ZuulV3? Already installed Zuul, looking for documentation which can help me in configuring dependent modules like ansible playbooks, nodepool, pipeline/jobs etc | 08:44 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Sort gerritbot/channels.yaml https://review.openstack.org/653928 | 08:46 |
| sajauddin | Currently following Zuul official documentation... it is a bit confusing. If anyone has any documentation related to Zuul V3, please share. | 08:48 |
| AJaeger | sajauddin: I doubt anybody is around today - and we have a huge maintenance task today as well. For Zuul v3, I suggest to ask on #zuul channel but since this is a 3rd party question, #openstack-third-party-ci is the better choice. The folks on #zuul won't have time either today. | 08:48 |
| sajauddin | Thanks Ajaeger. I work for Openstack CI for Power (PowerKVM CI). Will connect to #openstack-third-party-ci. | 08:51 |
| *** dikonoor has quit IRC | 08:54 | |
| *** eumel8 has quit IRC | 08:55 | |
| *** whoami-rajat has joined #openstack-infra | 08:55 | |
| *** kjackal has quit IRC | 09:01 | |
| *** kjackal has joined #openstack-infra | 09:01 | |
| *** Basante has joined #openstack-infra | 09:08 | |
| *** Basante is now known as basante | 09:12 | |
| basante | hello | 09:15 |
| *** raukadah has quit IRC | 09:15 | |
| *** chandankumar has joined #openstack-infra | 09:16 | |
| *** markvoelker has joined #openstack-infra | 09:58 | |
| *** yamamoto has quit IRC | 09:59 | |
| *** larainema has quit IRC | 10:09 | |
| *** jamesmcarthur has joined #openstack-infra | 10:11 | |
| *** pcaruana has quit IRC | 10:17 | |
| *** kjackal has quit IRC | 10:18 | |
| *** jamesmcarthur has quit IRC | 10:18 | |
| *** e0ne has joined #openstack-infra | 10:18 | |
| *** jamesmcarthur has joined #openstack-infra | 10:23 | |
| *** jamesmcarthur has quit IRC | 10:27 | |
| *** bhavikdbavishi has joined #openstack-infra | 10:30 | |
| *** yamamoto has joined #openstack-infra | 10:30 | |
| *** markvoelker has quit IRC | 10:31 | |
| *** yamamoto has quit IRC | 10:41 | |
| *** nicolasbock has joined #openstack-infra | 10:45 | |
| tristanC | Good luck everybody with the opendev migration! | 10:46 |
| *** yamamoto has joined #openstack-infra | 10:49 | |
| *** kjackal has joined #openstack-infra | 10:50 | |
| *** yboaron_ has quit IRC | 10:50 | |
| *** yamamoto has quit IRC | 10:54 | |
| *** jamesmcarthur has joined #openstack-infra | 10:54 | |
| *** jamesmcarthur has quit IRC | 10:59 | |
| *** raissa has joined #openstack-infra | 11:01 | |
| *** e0ne has quit IRC | 11:04 | |
| *** yamamoto has joined #openstack-infra | 11:27 | |
| *** markvoelker has joined #openstack-infra | 11:29 | |
| *** lpetrut has quit IRC | 11:40 | |
| *** Lucas_Gray has joined #openstack-infra | 11:42 | |
| *** edmondsw has joined #openstack-infra | 11:50 | |
| *** yamamoto has quit IRC | 11:57 | |
| *** yamamoto has joined #openstack-infra | 12:00 | |
| *** markvoelker has quit IRC | 12:02 | |
| *** whoami-rajat has quit IRC | 12:05 | |
| *** yamamoto has quit IRC | 12:07 | |
| *** Lucas_Gray has quit IRC | 12:08 | |
| *** e0ne has joined #openstack-infra | 12:08 | |
| *** Lucas_Gray has joined #openstack-infra | 12:10 | |
| *** dikonoor has joined #openstack-infra | 12:22 | |
| *** yamamoto has joined #openstack-infra | 12:31 | |
| *** raissa has quit IRC | 12:33 | |
| *** bhavikdbavishi has quit IRC | 12:40 | |
| *** sajauddin has quit IRC | 12:40 | |
| *** bhavikdbavishi has joined #openstack-infra | 12:41 | |
| *** ricolin has quit IRC | 12:41 | |
| *** e0ne has quit IRC | 12:41 | |
| *** e0ne has joined #openstack-infra | 12:42 | |
| *** jamesmcarthur has joined #openstack-infra | 12:55 | |
| mordred | tristanC: thanks! | 12:56 |
| *** mriedem has joined #openstack-infra | 12:56 | |
| *** markvoelker has joined #openstack-infra | 12:58 | |
| fungi | okay, heading out for a while but will be back later to see how the migration's going and what i can do to help (but possibly not until 1900z or so) | 12:59 |
| *** aaronsheffield has joined #openstack-infra | 12:59 | |
| *** jamesmcarthur has quit IRC | 13:00 | |
| fungi | looks like things are pretty well prepared, so hopefully the unanticipated factors won't be overwhelming | 13:00 |
| fungi | to be honest, i mostly worried about the volume of questions we're going to get from folks who can't push changes to gerrit because they need to clear/update their git remotes on their systems | 13:00 |
| *** jamesmcarthur has joined #openstack-infra | 13:01 | |
| mordred | fungi: yeah | 13:02 |
| mnaser | wouldn’t pointing review.o.o by dns to OpenDev server solve that? | 13:07 |
| *** eernst has joined #openstack-infra | 13:07 | |
| *** Lucas_Gray has quit IRC | 13:07 | |
| *** eernst has quit IRC | 13:08 | |
| *** jamesmcarthur has quit IRC | 13:11 | |
| openstackgerrit | Matt Riedemann proposed openstack-infra/elastic-recheck master: Update query for bug 1825435 https://review.openstack.org/653943 | 13:16 |
| openstack | bug 1825435 in OpenStack Compute (nova) "TestRPC unit tests intermittently fail with "'>' not supported between instances of 'NoneType' and 'datetime.datetime'"" [Undecided,Confirmed] https://launchpad.net/bugs/1825435 | 13:16 |
| *** dims has quit IRC | 13:21 | |
| *** ykarel|away has joined #openstack-infra | 13:22 | |
| *** e0ne has quit IRC | 13:23 | |
| *** jamesmcarthur has joined #openstack-infra | 13:24 | |
| *** efried is now known as fried_rice | 13:25 | |
| *** basante has quit IRC | 13:26 | |
| *** ykarel|away is now known as ykarel | 13:27 | |
| ykarel | is the issue around too much RETRY_LIMIT already known? | 13:27 |
| ykarel | http://zuul.openstack.org/builds?result=RETRY_LIMIT | 13:27 |
| ykarel | fungi, AJaeger , clarkb ^^ | 13:29 |
| *** whoami-rajat has joined #openstack-infra | 13:31 | |
| *** markvoelker has quit IRC | 13:31 | |
| *** jamesmcarthur has quit IRC | 13:35 | |
| *** e0ne has joined #openstack-infra | 13:36 | |
| mnaser | ykarel: I disabled ovh-bhs1 yesterday as it was causing a lot of these issues | 13:43 |
| mnaser | maybe infra-root can check where those VMs were running | 13:44 |
| mnaser | but yeah I see a lot of our jobs being hit by RETRY_LIMIT too | 13:44 |
| ykarel | mnaser, okk, if it was disabled yesterday, then may be some other host/hosts have issue today | 13:44 |
| mnaser | I am indeed seeing a high # of retry_limit too | 13:45 |
| mnaser | and we can't get much logs out of them | 13:45 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Update opendev website to be more present tense https://review.openstack.org/653946 | 13:46 |
| mordred | infra-root: ^^ corvus mentioned yesterday that we should update the website text, so there's a stab at that | 13:47 |
| *** jamesmcarthur has joined #openstack-infra | 13:47 | |
| *** dims has joined #openstack-infra | 13:48 | |
| *** larainema has joined #openstack-infra | 13:48 | |
| mordred | infra-root: I'm going to disable ansible/puppet cron | 13:51 |
| mordred | oh - nevermind. looks like it's already disabled | 13:51 |
| *** jamesmcarthur has quit IRC | 13:52 | |
| corvus | mordred: good morning! | 13:54 |
| mordred | corvus: morning! | 13:54 |
| *** jamesmcarthur has joined #openstack-infra | 13:55 | |
| mnaser | I know everyone is busy preparing for opendev, but I'd really appreciate a quick look to see if there's a specific provider that is having issues with the high # of retry_limits in gate | 13:55 |
| mnaser | it's causing a lot of churn | 13:55 |
| corvus | oh, it's retry limits with no logs | 13:57 |
| mordred | yeah | 13:58 |
| corvus | i'll dig | 13:58 |
| mordred | I'm looking at one in rax-ord | 13:58 |
| mnaser | yeah I'd dig into those but the no logs make it hard | 13:59 |
| corvus | my goodness there are a lot | 13:59 |
| mordred | corvus: on ze01, grep 4d3c695030434c5b936df9f47218d728 executor-debug.log has stuff | 13:59 |
| *** jamesmcarthur has quit IRC | 13:59 | |
| mordred | I don't see any smoking gun - it looks like it was happily running devstack commands | 13:59 |
| mordred | then we get an unreachable | 14:00 |
| mordred | 2019-04-19 13:28:22,512 DEBUG zuul.AnsibleJob: [build: 4d3c695030434c5b936df9f47218d728] Ansible exit code: 4 | 14:00 |
| mnaser | is it possible that this whole time, the issue is the Zuul-executors and not the clouds? :X | 14:00 |
| corvus | mnaser: well, they are in the clouds | 14:01 |
| mnaser | yes there's that too I guess :) | 14:01 |
| corvus | they are all in rax-dfw, so if that has a network issue, then the whole system would appear unstable | 14:01 |
| mnaser | https://status.apps.rackspace.com shows nothing | 14:02 |
| mordred | corvus: in the one I'm looking at, the last run command is 2019-04-19 12:59:14,536 DEBUG zuul.AnsibleJob: [build: 4d3c695030434c5b936df9f47218d728] Ansible output: b'./safe-devstack-vm-gate-wrap.sh' | 14:02 |
| mordred | which then runs for 30 minutes before we get ansible unhappy about unreachable | 14:02 |
| corvus | none of the jobs in the build list are short | 14:03 |
| corvus | the failures appear to be grouped by time | 14:04 |
| mordred | corvus: another one I looked at, 3f199b2f59b44926932b7acfd98c2170 on ze06, is not running devstack-gate (so rule that part out) but is also in rax - in dfw | 14:05 |
| corvus | failures: http://paste.openstack.org/show/749534/ | 14:05 |
| mnaser | oh I think this is the right status page https://rackspace.service-now.com/system_status/ | 14:05 |
| mordred | corvus: and in this one, the last thing before unreachable is starting to run tox - which runs for 40 minutes | 14:05 |
| *** nicolasbock has quit IRC | 14:06 | |
| corvus | we should really log provider in the zuul log :( | 14:06 |
| mordred | corvus: yeah. I've been looking at the ansible hostname | 14:07 |
| mordred | corvus: oh - you can grep for provider: | 14:07 |
| mordred | grep 3f199b2f59b44926932b7acfd98c2170 executor-debug.log | grep Provider: | 14:07 |
| mordred | 2019-04-19 09:47:22,846 DEBUG zuul.AnsibleJob: [build: 3f199b2f59b44926932b7acfd98c2170] Ansible output: b' "msg": "# Node Information\\nHostname: ubuntu-bionic-rax-dfw-0005349268\\nProvider: rax-dfw\\nLabel: ubuntu-bionic\\nInterface IP: 104.239.150.58\\n"' | 14:07 |
| mordred | oh - but yeah - on the scheduler side | 14:08 |
| mnaser | speaking of potential neat features too, statsd logging per provider would be nice | 14:08 |
| mnaser | like job status => provider mapping can help identify those things easiler | 14:08 |
| corvus | yeah, i'll get awk to give us a list of shas and we can run that grep on bridge across all the executors | 14:08 |
| mnaser | easily* | 14:09 |
| mordred | corvus: awk-lover | 14:09 |
| corvus | bridge:~corvus/jobs | 14:10 |
| * dtroyer recalls that the path from awk + sed + grep leads to …… perl | 14:11 | |
| *** jamesmcarthur has joined #openstack-infra | 14:13 | |
| corvus | sjc1 ord ord ord iad iad dfw ord gra1 iad gra1 ord ord ord iad dfw iad ord gra1 | 14:16 |
| corvus | that's the last several | 14:16 |
| mordred | corvus: that unfortunately looks like all of them | 14:16 |
| corvus | ya | 14:16 |
| corvus | so there may be a network problem in dfw, or transiting out of dfw, or the whole internet | 14:17 |
| *** jamesmcarthur has quit IRC | 14:18 | |
| mordred | yeah. and we haven't landed any changes to the executors recently that seem likely to have screwed up ansible return code processing or timeouts or anything | 14:18 |
| *** nicolasbock has joined #openstack-infra | 14:18 | |
| corvus | we also haven't even restarted them since march 21 | 14:18 |
| mordred | yeah | 14:19 |
| *** ykarel is now known as ykarel|away | 14:20 | |
| *** dims has quit IRC | 14:20 | |
| corvus | we should set up monitoring from a server in dfw to all of our mirrors | 14:21 |
| corvus | so we can see if pings from dfw -> any provider fail | 14:21 |
| *** e0ne has quit IRC | 14:21 | |
| mordred | corvus: yeah - although also these do all seem to be long-running remote ansible ssh commands | 14:21 |
| mnaser | I saw this REALLY weird behavior btw | 14:23 |
| mordred | so maybe also something to run a really long ssh command? | 14:23 |
| mnaser | where a job would complete in ansible, but *post* would fail | 14:23 |
| mnaser | part of why I ended up disabling ovh-bhs1, the whole job would run successfully (and log everything) | 14:23 |
| mnaser | but when the jobs ends and post starts running, it fails to ssh to the server / times out | 14:24 |
| corvus | mordred: maybe, but the failures appear to be clustered suggesting that they may all happen at the same time (ie, the problem happens at 6:30UTC regardless of when the job started not after running for 30 minutes) | 14:24 |
| mordred | corvus: you know - maybe we should still try running post playbooks even if we get unrechable - since the log upload playbook still might work | 14:24 |
| openstackgerrit | Merged openstack-infra/elastic-recheck master: Update query for bug 1825435 https://review.openstack.org/653943 | 14:24 |
| openstack | bug 1825435 in OpenStack Compute (nova) "TestRPC unit tests intermittently fail with "'>' not supported between instances of 'NoneType' and 'datetime.datetime'" - maybe due to "Fatal Python error: Cannot recover from stack overflow."" [High,Confirmed] https://launchpad.net/bugs/1825435 | 14:24 |
| corvus | mordred: it won't upload many logs | 14:24 |
| corvus | (since it also fetches them) | 14:25 |
| mordred | no - but it would have the console log | 14:25 |
| mnaser | http://logs.openstack.org/92/649892/13/check/openstack-ansible-deploy-aio_distro_metal-centos-7/0e486a2/job-output.txt.gz | 14:25 |
| mnaser | here's a job that ran fine, streamed all log fine, but when it tried to re-ssh to the machine, it timed out | 14:25 |
| corvus | mordred: i think you could probably put an "ignore_errors" on the rsync task | 14:25 |
| mordred | yeah - the one to fetch things in teh log upload playbook | 14:26 |
| *** e0ne has joined #openstack-infra | 14:26 | |
| mordred | oh - wait- duh | 14:26 |
| mordred | these hit unreachable, so we return so that zuul will re-run them | 14:26 |
| *** Lucas_Gray has joined #openstack-infra | 14:27 | |
| corvus | mordred: these are exiting with exit code 4 | 14:27 |
| *** michael-beaver has joined #openstack-infra | 14:28 | |
| mordred | yeah. that's aborted | 14:28 |
| corvus | elif ret == 4: | 14:28 |
| corvus | # Ansible could not parse the yaml. | 14:28 |
| corvus | ? | 14:28 |
| mordred | well - that's not what the log output looks like - the log output says 218d728] Ansible output: b'primary : ok=6 changed=6 unreachable=1 failed=0' | 14:29 |
| corvus | oh, maybe this is the issue with the wrong return code and tobiash's unreachable file is in play | 14:30 |
| mordred | oh yeah | 14:30 |
| corvus | yeah, exit code 4 and RESULT_UNREACHABLE is that case | 14:34 |
| corvus | it's happening now | 14:34 |
| openstackgerrit | Clint 'SpamapS' Byrum proposed openstack-infra/nodepool master: Ensure config is always validated https://review.openstack.org/653953 | 14:35 |
| *** dims has joined #openstack-infra | 14:35 | |
| corvus | it is curious that dfw is in the list of hits | 14:38 |
| mordred | yeah | 14:40 |
| clarkb | hello | 14:40 |
| *** ykarel|away has quit IRC | 14:41 | |
| clarkb | one thing I realized in my sleep is that we may need to run fungi's script while zuul is off and before we run our migrate playbook | 14:41 |
| clarkb | because I think it operates on the old side repos? | 14:41 |
| clarkb | also I think we may want to merge the project-config change that fungi pushed up last night before we stop gerrit so that it can be replicated | 14:42 |
| corvus | clarkb: i think fungi's script generates the input for the migrate playbook? | 14:42 |
| clarkb | corvus: the other script | 14:42 |
| clarkb | corvus: there is the input to migrate playbook one and then there is the modify all the git repos in place script | 14:43 |
| *** e0ne has quit IRC | 14:43 | |
| * clarkb is double checking that | 14:43 | |
| corvus | i've started several pings on ze01, i see notable packet loss in ipv6 routes, but not v4. as we learned yesterday, we should be using v4 for all the rax regions | 14:44 |
| corvus | and the recent surge of failures seems to be over, last was at 14:41 | 14:44 |
| corvus | so there was a rash from 14:34 to 14:41 | 14:44 |
| corvus | (it's possible the event was shorter than that, it does take zuul a variable amount of time to clean up after a job) | 14:45 |
| clarkb | also the other thing I thought of was kick.sh would reinstall the cron in the past | 14:46 |
| clarkb | we should double check it won't do that on us today | 14:46 |
| * mordred goes to verify | 14:46 | |
| corvus | mnaser, mordred: the last 100 failures: http://paste.openstack.org/show/749536/ | 14:47 |
| *** helenafm has quit IRC | 14:47 | |
| mordred | clarkb: ansible-cron is in the bridge.yaml playbook | 14:47 |
| mordred | clarkb: that is run by run_all but not by kick | 14:48 |
| mnaser | be nice if we have region-local executors but im not the one who has the time :p | 14:48 |
| clarkb | mordred: thanks for checking | 14:49 |
| clarkb | reading fungi's script it doesn't actually care if the repos on disk on review have been moved yet or not | 14:49 |
| corvus | mnaser: not sure that would solve this problem | 14:49 |
| mordred | mnaser: yeah - although the scheduler would still need to talk to them - so if it's networking issues that might just move the ball | 14:49 |
| corvus | that | 14:49 |
| clarkb | it lists what the repos are on disk and then clones them and applies the ruleset against that | 14:49 |
| *** yamamoto has quit IRC | 14:49 | |
| clarkb | so I think we are fine there too | 14:49 |
| mordred | however, it might be helpful for certain classes of problems | 14:50 |
| clarkb | I do think we should get any changes we want to be replicated early merged prior to stopping gerrit though | 14:50 |
| corvus | clarkb: can you paste the etherpad link? | 14:50 |
| mordred | https://etherpad.openstack.org/p/the-great-opendev-git-migration | 14:50 |
| *** yamamoto has joined #openstack-infra | 14:50 | |
| mordred | corvus: ^^ | 14:50 |
| corvus | thx | 14:50 |
| clarkb | https://etherpad.openstack.org/p/the-great-opendev-git-migration | 14:50 |
| *** yamamoto has quit IRC | 14:51 | |
| *** e0ne has joined #openstack-infra | 14:52 | |
| corvus | clarkb: i'm not sure we need to worry about replication there | 14:52 |
| corvus | clarkb: i don't think we'll start up any ansible crons until replication after restart has completed | 14:53 |
| clarkb | and we can manually apply things otherwise? | 14:53 |
| *** dims has quit IRC | 14:53 | |
| clarkb | that works for me | 14:53 |
| *** e0ne has quit IRC | 14:53 | |
| corvus | and in fact, i doubt we'd even want to start zuul until startup replication has finished | 14:54 |
| clarkb | oh thats a good point | 14:54 |
| clarkb | it will just thrash otherwise | 14:54 |
| clarkb | (and may thrash after but less so) | 14:54 |
| *** jamesmcarthur has joined #openstack-infra | 14:55 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Update opendev website to be more present tense https://review.openstack.org/653946 | 14:55 |
| clarkb | mordred: corvus the bridge crontab seems to be enabled? | 14:55 |
| mordred | clarkb: it does? | 14:55 |
| clarkb | ya no # prefix on the crontab -l entries | 14:56 |
| *** dims has joined #openstack-infra | 14:56 | |
| mordred | the only thing I see in crontab is cloud launcher | 14:56 |
| mordred | oh - nope | 14:56 |
| mordred | I simply cannot see things | 14:56 |
| clarkb | and the run started at 14:45UTC so it won't be done until about 15:30 unless we kill it | 14:57 |
| mordred | disabling now | 14:57 |
| clarkb | k | 14:57 |
| clarkb | probably want to disable both | 14:57 |
| clarkb | just to keep ansible out of ps | 14:57 |
| mordred | done | 14:57 |
| clarkb | that looks better | 14:57 |
| corvus | status alert Gerrit is offline for several hours starting at 15:00 UTC to perform the opendev migration; see http://lists.openstack.org/pipermail/openstack-discuss/2019-April/005011.html | 14:58 |
| clarkb | lgtm | 14:59 |
| *** nicolasbock has quit IRC | 14:59 | |
| clarkb | mordred: can we safely stop the run_all.sh on bridge too? | 14:59 |
| corvus | think alert is appropriate? (sets topic, etc) | 14:59 |
| clarkb | corvus: probably especially if it will help cut down on questions in here | 14:59 |
| mordred | clarkb: I think so? I can't think of anything that would be broken by it | 14:59 |
| mnaser | maybe suggesting using openstack-infra-incident to help remove noise from y'all work :) | 14:59 |
| mnaser | surely there will be questions | 15:00 |
| clarkb | mordred: more just thinking if we stopped ansible halfway through but most of our stuff is idempotent so it should be fine | 15:00 |
| corvus | oom killer occasinoally does that anyway | 15:00 |
| clarkb | ha | 15:00 |
| clarkb | mnaser: ya I expect if it becomes a problem we'll move | 15:00 |
| mordred | yeah. ok - I'm going to kill run_all.sh first - then will kill ansible-playbook processes. yes? | 15:00 |
| corvus | #status alert Gerrit is offline for several hours starting at 15:00 UTC to perform the opendev migration; see http://lists.openstack.org/pipermail/openstack-discuss/2019-April/005011.html | 15:00 |
| openstackstatus | corvus: sending alert | 15:00 |
| clarkb | mordred: ++ | 15:01 |
| *** dims has quit IRC | 15:01 | |
| mordred | ok. no more ansible-playbook processes | 15:01 |
| *** Goneri has joined #openstack-infra | 15:01 | |
| mordred | clarkb: there's still a few thing in ps with ansible in them - you wanna look and see what you think? | 15:02 |
| mordred | it's mostly control persist stuff I thnik | 15:02 |
| clarkb | ya looks like ssh processes. Those should be fine (and claen up themselves after a timeout) | 15:03 |
| mordred | kk | 15:03 |
| -openstackstatus- NOTICE: Gerrit is offline for several hours starting at 15:00 UTC to perform the opendev migration; see http://lists.openstack.org/pipermail/openstack-discuss/2019-April/005011.html | 15:03 | |
| *** ChanServ changes topic to "Gerrit is offline for several hours starting at 15:00 UTC to perform the opendev migration; see http://lists.openstack.org/pipermail/openstack-discuss/2019-April/005011.html" | 15:03 | |
| *** e0ne has joined #openstack-infra | 15:03 | |
| *** lpetrut has joined #openstack-infra | 15:04 | |
| *** ykarel|away has joined #openstack-infra | 15:05 | |
| clarkb | so next steps. Stop gerrit, perform db backups, run fungi's first script to get the todo list, run corvus' rename playbook, then fungi's second script to rename everything in place ? | 15:05 |
| *** jamesmcarthur has quit IRC | 15:05 | |
| *** nicolasbock has joined #openstack-infra | 15:05 | |
| mordred | yes. I'll take db backups | 15:05 |
| *** jamesmcarthur has joined #openstack-infra | 15:06 | |
| clarkb | fungis notes are in gerrit so I'm copying those elsewhere now | 15:06 |
| clarkb | since well gerrit will be off | 15:06 |
| corvus | hrm | 15:06 |
| corvus | people were not consistent with the topics | 15:06 |
| *** gyee has joined #openstack-infra | 15:06 | |
| openstackstatus | corvus: finished sending alert | 15:07 |
| corvus | i should have everything in gertty if we need something | 15:07 |
| *** dims has joined #openstack-infra | 15:07 | |
| clarkb | k | 15:07 |
| corvus | also, should we rsync a backup copy of the repos before running the script? | 15:08 |
| zbr | 🙏 that everything will go smooth. | 15:08 |
| clarkb | corvus: ++ | 15:08 |
| mordred | corvus: ++ | 15:09 |
| mordred | zbr: ++ :) | 15:09 |
| corvus | i think i can do that as a normal user too (that's what fungi was doing) | 15:09 |
| corvus | i'll get that started now | 15:09 |
| corvus | (and obvs run again after stopping) | 15:09 |
| clarkb | I can stop gerrit whenever we are ready | 15:10 |
| corvus | ready | 15:10 |
| clarkb | ok stopping gerrit now | 15:10 |
| corvus | i'll stop zuul | 15:10 |
| corvus | should i bother saving the zuul queues? | 15:11 |
| corvus | (some project names will change) | 15:11 |
| clarkb | ya I don't think so | 15:11 |
| corvus | that's how i lean | 15:11 |
| clarkb | chances are good they won't have working jobs initially too | 15:11 |
| clarkb | gerrit is stopped | 15:12 |
| mordred | k. I'm running the db backups | 15:12 |
| corvus | second pass of rsync is done | 15:13 |
| mordred | storyboard db backup done - still waiting on gerrit | 15:13 |
| clarkb | gerrit takes ~4 minutes iirc | 15:13 |
| mordred | waiting for mysqldump is boring | 15:16 |
| corvus | brb | 15:17 |
| mordred | still running | 15:18 |
| corvus | 3 executors have stopped | 15:19 |
| clarkb | I've compared review.openstack.org:/home/fungi/opendev-migrate against a local up to date copy and they match | 15:20 |
| clarkb | should I go ahead and run that to get our repos.yaml file? | 15:20 |
| mordred | yeah. that can go in parallel to mysqldump | 15:21 |
| clarkb | ok doing that now | 15:21 |
| mordred | remind me I really need to make us a new db cluster for gerrit ... | 15:21 |
| mordred | this would have been so much quicker if we could have just copied the ibdata file :) | 15:22 |
| mordred | ok. mysqldump is finished | 15:22 |
| clarkb | ok that broke /me figuring out why | 15:22 |
| clarkb | it loads the projects list from the gerrit api | 15:23 |
| clarkb | and gerrit is off :) | 15:23 |
| mordred | yeah. we shoudl have run it before turning off gerrit | 15:23 |
| corvus | we all reviewed that | 15:23 |
| mordred | yup | 15:23 |
| clarkb | we did, but fungi's notes said do it before turning off gitea not gerrit and that replaced in my head | 15:23 |
| clarkb | we can generate a list from the on disk contents. Or start gerrit and do it again | 15:24 |
| mordred | probably starting gerrit and running it tehn stopping is the least likely to involve quick dev work we might get wrong | 15:24 |
| clarkb | ya | 15:24 |
| corvus | well, if we restart gerrit we will have to do db and disk backups again | 15:24 |
| clarkb | also yes | 15:25 |
| mordred | yeah | 15:25 |
| corvus | we might be able to firewall it off and restart it | 15:25 |
| corvus | then avoid the backup cost | 15:25 |
| mordred | gerrit db backup took 10 minutes | 15:26 |
| clarkb | stopping apache would cover the http blockage | 15:26 |
| clarkb | but would have to do sshd still | 15:26 |
| *** yamamoto has joined #openstack-infra | 15:26 | |
| mordred | but not git-review submissions | 15:26 |
| corvus | the firewall is easy to do; let's do that? | 15:26 |
| clarkb | corvus: ++ | 15:26 |
| *** pgaxatte has quit IRC | 15:27 | |
| corvus | done | 15:27 |
| corvus | clarkb: if iptables lgty, you want to restart gerrit? | 15:28 |
| clarkb | corvus: yup was just checking. iptables and ip6tables -L -n lgtm and I cannot connect to webserver over ipv4 | 15:28 |
| clarkb | corvus: mordred ^ ready to start gerrit? | 15:28 |
| mordred | clarkb: ++ | 15:28 |
| clarkb | ok starting it now | 15:29 |
| corvus | then the script will need an update to hit localhost:8081 | 15:29 |
| clarkb | yup was just realizing that | 15:30 |
| clarkb | gerrit says it is ready but usually it needs a minute before it actually is | 15:30 |
| *** jamesmcarthur has quit IRC | 15:30 | |
| clarkb | and to double check I still cant hit it from home via https | 15:30 |
| corvus | my gertty is still offline | 15:31 |
| *** jamesmcarthur has joined #openstack-infra | 15:31 | |
| mordred | I also cannot hit it over 29418 | 15:32 |
| clarkb | I have 820 entries in ~clarkb/repos.yaml | 15:32 |
| mordred | which host? | 15:32 |
| clarkb | review.o.o | 15:32 |
| clarkb | there were 820 in fungis last repos.yaml file too | 15:32 |
| mordred | yes. I agree with you | 15:33 |
| clarkb | if that looks good to you I'll stop gerrit again | 15:33 |
| mordred | it's passing the eyeball test | 15:33 |
| clarkb | zuul and jjb look good and I see a variety of opendev | 15:34 |
| mordred | yeah | 15:34 |
| clarkb | corvus: ^ you ready for me to stop gerrit again? | 15:34 |
| corvus | ++ | 15:34 |
| clarkb | ok stopping it now | 15:34 |
| mordred | I thnik it's as good as it's possible for me to manually verify | 15:34 |
| *** yamamoto has quit IRC | 15:35 | |
| *** dims has quit IRC | 15:35 | |
| clarkb | gerrit is stopped | 15:36 |
| clarkb | are we ready to run corvus' rename playbook? review.o.o:~clarkb/repos.yaml is the input to that | 15:37 |
| mordred | I can't think of any reason why not | 15:37 |
| corvus | wait | 15:37 |
| clarkb | waiting | 15:37 |
| corvus | fungi did not squash https://review.openstack.org/653881 | 15:38 |
| corvus | which means we don't have the zuul file moves | 15:38 |
| corvus | is that correct? | 15:38 |
| * clarkb looks at local copies | 15:38 | |
| clarkb | hmr that wasn't on the etherpad. | 15:39 |
| corvus | it was supposed to be squashed | 15:39 |
| *** tjgresha has joined #openstack-infra | 15:40 | |
| clarkb | I ran opendev-migrate out of 653138 | 15:40 |
| clarkb | I have not run the thing that modifies repo contents | 15:41 |
| *** zhangfei has quit IRC | 15:41 | |
| corvus | right. i wrote a change to opendev-migrate, which, since fungi was mostly working offline, i made as a followup and asked that it be squashed into the script | 15:41 |
| clarkb | gotcha | 15:41 |
| corvus | gimme a sec and i can run it | 15:42 |
| clarkb | corvus: is it in the repos.yaml generation script? | 15:42 |
| corvus | clarkb: it's opendev-migrate | 15:42 |
| corvus | so we need to start gerrit again | 15:42 |
| clarkb | k | 15:42 |
| corvus | so i can run my version and get the zuul key moves | 15:42 |
| clarkb | oh right the key moves | 15:42 |
| clarkb | (sorry zuul files kept parsing as zuul yaml files) | 15:42 |
| corvus | i will start gerrit | 15:43 |
| clarkb | I'm starting gerrit after double checking iptables | 15:43 |
| clarkb | k /me gets out of the way | 15:43 |
| *** dims has joined #openstack-infra | 15:44 | |
| corvus | nice | 15:44 |
| *** Lucas_Gray has quit IRC | 15:44 | |
| corvus | i think the output is not sorted | 15:44 |
| corvus | so my repos.yaml is completely different than yours | 15:45 |
| clarkb | oh ya because python3.5 on review not 3.6:/ | 15:45 |
| corvus | okay ~corvus/zuul-rename.sh | 15:45 |
| mordred | corvus: your repos.yaml also looks good on general perusing | 15:45 |
| corvus | gerrit is stopped | 15:46 |
| clarkb | corvus: there are 1638 zuul file moves and not 1640 (820 * 2 == 1640) | 15:46 |
| clarkb | is that expected? | 15:46 |
| *** fmount has joined #openstack-infra | 15:46 | |
| corvus | i have 819 repos in repos.yaml | 15:47 |
| corvus | so do you | 15:47 |
| corvus | first line isn't a move | 15:47 |
| clarkb | oh right the repos: line | 15:47 |
| clarkb | ++ things look good to me then | 15:47 |
| corvus | \o/ | 15:47 |
| mordred | corvus: also - I think we need to make sure we create target directories for new org | 15:47 |
| corvus | mordred: good point | 15:47 |
| corvus | mordred: i'll work on commands to do that | 15:48 |
| mordred | do we have a set of new orgnames? | 15:48 |
| corvus | i don't think so. i will soon though | 15:49 |
| mordred | kk | 15:49 |
| corvus | ~corvus/zuul-dirs.sh | 15:51 |
| mordred | corvus: lgtm | 15:51 |
| clarkb | I see nothing missing from the list lgtm | 15:52 |
| corvus | i think we're good to run the playbook now | 15:52 |
| clarkb | I agree. db and repo backups are done. gerrit doesn't seem to be running | 15:52 |
| mordred | ++ | 15:52 |
| clarkb | we have our repos.yaml and zuul secrets mv data/scripts | 15:52 |
| mordred | corvus: should we rsync a backup of the zuul keys as a belt/suspenders? | 15:54 |
| corvus | sure i'll do that | 15:54 |
| corvus | done | 15:55 |
| mordred | that's all I can think of to be specifically paranoid about atm | 15:55 |
| clarkb | before running the playbook I think we may need to update the hostline for storyborad | 15:55 |
| clarkb | I'm double checking against inventory now | 15:55 |
| clarkb | storyboard01.opendev.org is the hostname now | 15:56 |
| clarkb | other than that I think playbook is ready and good to go. (its interesting how the extra level of paranoia of being in the moment helps you catch stuff like that | 15:56 |
| mordred | clarkb: what do you mean update the hostline? | 15:57 |
| clarkb | mordred: - hosts: storyboard.openstack.org should be -hosts: storyboard01.opendev.org | 15:57 |
| *** e0ne has quit IRC | 15:57 | |
| corvus | it shouldn't be storyboard01 | 15:58 |
| corvus | it should be a group with storyboard01 in it | 15:58 |
| corvus | i mean, that's fine for today | 15:58 |
| clarkb | good point. storyboard group does exist | 15:58 |
| corvus | but assuming we don't want our rename playbook to keep breaking everytime we change the host | 15:58 |
| clarkb | ++ | 15:58 |
| mordred | ++ | 15:58 |
| corvus | when we fix the real one, that's what we should do :) | 15:58 |
| mordred | and I concur about the line needing to be opendev.org | 15:58 |
| clarkb | also its - hosts: not -hosts: I fail at yaml | 15:59 |
| clarkb | corvus: ^ did you want to run that (you made the playbook edits and generated the final repos.yaml) | 15:59 |
| corvus | not especially... i don't think i've ever run it | 16:00 |
| corvus | i just removed some things from it | 16:00 |
| clarkb | k | 16:00 |
| corvus | clarkb: want to use screen? | 16:00 |
| clarkb | ya lets starts a root screen on bridge | 16:01 |
| clarkb | someone with a small terminal want to do it? otherwise you'll get my large termainl | 16:02 |
| corvus | i'm fine with a large terminal :) | 16:02 |
| clarkb | ok I'll start it then | 16:02 |
| mordred | I'll get by with it ... :) - make it the size you need for driving | 16:02 |
| clarkb | started | 16:02 |
| mordred | on and watching | 16:03 |
| clarkb | corvus: can you copy your playbook and repos.yaml onto bridge? | 16:03 |
| corvus | yep | 16:03 |
| clarkb | https://docs.openstack.org/infra/system-config/gerrit.html#renaming-a-project are the docs on running the playbook | 16:05 |
| corvus | playbook is in place | 16:05 |
| clarkb | and looks like you fixed the storyboard hosts line | 16:05 |
| corvus | ya | 16:06 |
| corvus | how could you tell it was me? :) | 16:06 |
| clarkb | something about emacs | 16:06 |
| mordred | corvus: you could have done that edit with sed and awk | 16:07 |
| corvus | mordred: i'll edit you with sed and awk | 16:07 |
| clarkb | let us know when repos.yaml is in place. I think we can make a copy of that file and reduce it to a single entry and run with that first. Then run with the other partition after that looks good | 16:07 |
| mordred | probably a good idea. I need a good editing | 16:07 |
| corvus | clarkb: i sorted both of our repos.yaml files and the result is identical | 16:08 |
| corvus | i'll just copy my repos.yaml | 16:08 |
| clarkb | corvus: oh good check | 16:08 |
| corvus | so that it's the same order as the zuul script | 16:08 |
| corvus | done | 16:09 |
| *** e0ne has joined #openstack-infra | 16:09 | |
| clarkb | ok I'll make two new files. One with a single entry and one with everything else so that we can do this in two passes (first confirmation it is happy, second everything else) | 16:10 |
| corvus | good | 16:10 |
| corvus | lg | 16:11 |
| mordred | ++ | 16:12 |
| corvus | just one thing i'd like to double or triple check | 16:13 |
| clarkb | ok waiting | 16:13 |
| corvus | the patching script runs on the new locations? | 16:13 |
| clarkb | corvus: I think it can run on the old or new locations. It ls's the review_site dir and then operates within whatever repos are there | 16:13 |
| mordred | yeah | 16:13 |
| clarkb | corvus: but you should independently confirm that | 16:14 |
| mordred | and does any replacements in the repos based on the input mapping, not based on what repo it's in | 16:14 |
| corvus | yep, that's what it looks like to me. ok | 16:14 |
| corvus | 47 # the gerrit git directory | 16:15 |
| corvus | 48 top = sys.argv[1] | 16:15 |
| corvus | 84 # find all second-level directories on which we will operate | 16:15 |
| corvus | 85 repos = run(['find', top, '-maxdepth', '2', '-mindepth', '2', '-name', '*.git', '-type', 'd']) | 16:15 |
| *** mattmceuen has joined #openstack-infra | 16:15 | |
| corvus | all clear from me | 16:15 |
| clarkb | mordred: ready for me to run the pasted command? | 16:15 |
| mordred | clarkb: yes | 16:16 |
| clarkb | ok proceeding | 16:16 |
| clarkb | ok is that a werd ansible interaction with something? | 16:17 |
| corvus | maybe that was on puppetmaster but not bridge? | 16:17 |
| clarkb | oh ya | 16:17 |
| corvus | [ERROR] unknown variable 'defaults-file=/root/.gerrit_db.cfg' | 16:17 |
| corvus | there is a /root/.gerrit_db.cnf on gerrit | 16:18 |
| corvus | can probably copy that over | 16:18 |
| clarkb | corvus: if you've already got ssh things set up for that copy want to do that? | 16:18 |
| corvus | yeah, lemme double check the creds match the gerrit config file | 16:18 |
| mordred | I used .gerrit_db.cfg for the db backup earlier | 16:19 |
| mordred | cnfg that is | 16:19 |
| mordred | cnf | 16:19 |
| clarkb | actually that should run on review | 16:19 |
| corvus | oh you're right | 16:19 |
| corvus | so we just need to fix the filename? | 16:19 |
| mordred | then also update it to make the filename .gerrit_db.cnf instead of .cfg | 16:19 |
| corvus | i think the task does already run on reviwe | 16:20 |
| clarkb | ya I think we just have to update the filename | 16:20 |
| mordred | ++ | 16:20 |
| corvus | clarkb: you drive | 16:20 |
| clarkb | k | 16:20 |
| corvus | (has the rename playbook ever been run?) | 16:20 |
| clarkb | the name is /root/.gerrit_db.cnf ? | 16:21 |
| corvus | yes | 16:21 |
| clarkb | corvus: ya we used it last time we renamed projects iirc | 16:21 |
| corvus | this seems like a strange error to have in that case | 16:21 |
| clarkb | I agree | 16:21 |
| corvus | May 2 2018 /root/.gerrit_db.cnf | 16:21 |
| clarkb | I'm double checking the storyboard defaults file | 16:22 |
| mordred | I used /etc/mysql/debian.cnf for the db backup on storyboard | 16:23 |
| clarkb | hrm I wonder if this changed at some point? | 16:23 |
| clarkb | er chagned in puppet | 16:23 |
| mordred | the storyboard one likely changed when we moved to local db | 16:24 |
| clarkb | ah | 16:24 |
| corvus | i don't see any puppet or ansible to write the gerrit_db file out | 16:24 |
| clarkb | I thought the puppet mysql db backups module did that but maybe it doesn't | 16:25 |
| clarkb | that could explain how we've diverged | 16:25 |
| corvus | oh, then maybe i'm not searching right | 16:25 |
| mordred | yes | 16:25 |
| mordred | $defaults_file = "/root/.${name}_db.cnf" | 16:25 |
| corvus | clarkb: yes it does | 16:25 |
| corvus | ya | 16:26 |
| clarkb | but ya I'm fairly certain we've run this playbook before because I remember editing repos yaml files before | 16:26 |
| corvus | 27ecc02c (Clark Boylan 2013-09-05 18:22:21 -0700 30) $defaults_file = "/root/.${name}_db.cnf" | 16:26 |
| corvus | so *that* isn't what changed :) | 16:26 |
| clarkb | heh thats been there a long time | 16:26 |
| *** smarcet has joined #openstack-infra | 16:26 | |
| clarkb | Let me know if you think we are ready to run it on this first project again | 16:26 |
| corvus | clarkb: if you've double checked storyboard then ++ | 16:27 |
| mordred | ++ | 16:27 |
| corvus | i have started a list of things to clean up at the end of the etherpad | 16:27 |
| clarkb | corvus: thank you for that | 16:27 |
| clarkb | I'll rerun now | 16:27 |
| clarkb | heh is the arg not --defaults-file? | 16:28 |
| clarkb | mordred: ^ | 16:28 |
| mordred | should be ... one sec | 16:28 |
| corvus | mysql reviewdb --defaults-file=/root/.gerrit_db.cfg | 16:29 |
| corvus | running that on review01 fails | 16:29 |
| mordred | clarkb: --defaults-file=# Only read default options from the given file #. | 16:29 |
| mordred | database comes after arguments | 16:29 |
| mordred | mysql --defaults-file=/root/.gerrit_db.cnf reviewdb | 16:30 |
| clarkb | ok so change the order of the db and arguments | 16:30 |
| corvus | mysql --defaults-file=/root/.gerrit_db.cnf reviewdb | 16:30 |
| corvus | works | 16:30 |
| clarkb | editing | 16:30 |
| *** smarcet has quit IRC | 16:31 | |
| corvus | looks like the storyboard one is already correct | 16:31 |
| clarkb | yup | 16:31 |
| mordred | what the heck happend to that line? | 16:31 |
| clarkb | if you think that looks good I am ready to rerun | 16:31 |
| mordred | ++ | 16:31 |
| corvus | does this mean we don't have db backups on storyboard? | 16:32 |
| mordred | I did db backups on storyboard | 16:32 |
| corvus | sorry i mean *regular* backups | 16:32 |
| clarkb | no there is a crontab entry that uses the debian defaults file | 16:32 |
| clarkb | so I think storyboard backups were working | 16:32 |
| *** igordc has joined #openstack-infra | 16:32 | |
| clarkb | corvus: anything else we should check before trying again? | 16:33 |
| corvus | ok, we use /etc/mysql/debian.cnf if running local | 16:33 |
| corvus | i'm good :) | 16:33 |
| clarkb | alright proceeding | 16:33 |
| *** mgoddard has quit IRC | 16:34 | |
| clarkb | ok its the dest dir problem we anticipated for zuul keys on review_site now :) | 16:34 |
| corvus | agreed | 16:34 |
| corvus | i'll make a script | 16:34 |
| clarkb | I think we'll have that same issue on git0* | 16:34 |
| clarkb | and the local review mirror | 16:34 |
| corvus | oh maybe we should just incorporate this into the playbook then | 16:34 |
| clarkb | probably not a bad idea | 16:35 |
| *** mgoddard has joined #openstack-infra | 16:35 | |
| corvus | is there a jinja filter to get dirname? | 16:35 |
| mordred | mkdir $(dirname $(dirname /opt/lib/git/{new})) | 16:35 |
| mordred | ? | 16:35 |
| *** electrofelix has quit IRC | 16:36 | |
| mordred | looking for jinja | 16:36 |
| corvus | looks like there's a |dirname | 16:36 |
| corvus | maybe shell is easier | 16:36 |
| corvus | i'll drive | 16:36 |
| mordred | ++ | 16:36 |
| *** e0ne has quit IRC | 16:36 | |
| clarkb | corvus: use -p so that it doesn't error when the dir exists | 16:37 |
| clarkb | also do we need to chown to gerrit2? | 16:38 |
| corvus | hrm i guess so | 16:38 |
| mordred | corvus: wait - reove /opt/lib/git | 16:38 |
| mnaser | |dirname works in Ansible (but not jinja2 native fwiw) | 16:38 |
| mordred | that's what's in item[0] | 16:38 |
| mordred | ++ | 16:38 |
| corvus | this might be worth making a new task | 16:38 |
| mordred | yeah | 16:39 |
| mordred | corvus: was nI wrong - do we need the double dirname? | 16:40 |
| mordred | corvus: I think just one dirname | 16:40 |
| clarkb | mode is 755 on the other dirs | 16:40 |
| *** lpetrut has quit IRC | 16:41 | |
| mordred | corvus: ++ | 16:42 |
| corvus | mordred: i think you're right :) | 16:42 |
| mordred | corvus: :) | 16:42 |
| *** rpittau is now known as rpittau|afk | 16:42 | |
| mordred | \o/ | 16:43 |
| mordred | we made a directory! | 16:43 |
| mordred | on review, /opt/lib/git/x is there | 16:44 |
| corvus | derp thanks | 16:44 |
| *** mgoddard has quit IRC | 16:44 | |
| corvus | that play look good? | 16:44 |
| mordred | yes | 16:44 |
| clarkb | I thought the ls said it wasn't there? | 16:45 |
| clarkb | (sorry trying to follwo along and may have missed it when it was there) | 16:45 |
| corvus | clarkb: i was on the wrong host | 16:45 |
| mordred | clarkb: it's there on review.o.o | 16:45 |
| clarkb | oh right | 16:45 |
| corvus | git0* play look good? | 16:46 |
| mordred | yes. I think it's solid | 16:46 |
| *** igordc has quit IRC | 16:46 | |
| clarkb | yes I think so | 16:46 |
| corvus | i *think* that's covered for gitea...oh we need to pull in those tasks files | 16:46 |
| corvus | and... give it a root password don't we? | 16:47 |
| corvus | and it looks like that does not create the orgs | 16:47 |
| clarkb | I think the gitea apssword is in the ansible vars | 16:47 |
| corvus | oh, right, and we have access to that because we're on bridge | 16:48 |
| *** mattw4 has joined #openstack-infra | 16:48 | |
| corvus | and it's in the normal inventory hostvars stuff | 16:48 |
| mordred | yeah | 16:48 |
| clarkb | re creating the orgs that is another playbook right? | 16:48 |
| corvus | ok, so we do need to add in an org creation | 16:48 |
| corvus | yeah | 16:48 |
| corvus | let's come back to that | 16:48 |
| *** gyee has quit IRC | 16:48 | |
| corvus | ah, well, that looks like the last thing actually | 16:48 |
| corvus | i don't see any more places where we'd need to mkdir | 16:48 |
| clarkb | agreed I think we are good mkdiring on git0* and review | 16:49 |
| mordred | ++ | 16:49 |
| clarkb | leaving us with the gitea org creation | 16:49 |
| *** gyee has joined #openstack-infra | 16:49 | |
| corvus | for this, it may be easiest to write a one-time playbook to create the orgs | 16:49 |
| mordred | corvus: yeah - we have the list of them you made earlier | 16:50 |
| clarkb | I'm good with that | 16:50 |
| *** cmurphy is now known as cmurpheus | 16:50 | |
| corvus | we are missing one var -- gitea_url | 16:51 |
| mordred | I think it's in the playbook | 16:51 |
| clarkb | that might be in the public data? | 16:51 |
| corvus | yep it is | 16:51 |
| mordred | right there | 16:51 |
| mordred | yeah | 16:51 |
| corvus | does that look right? | 16:53 |
| clarkb | corvus: yes, though ansible loop syntax always confuses me. We can also simplify it by removing the orgs we already have | 16:53 |
| clarkb | openstack opendev zuul come to mind | 16:54 |
| corvus | yeah, i think i just removed those 3 | 16:54 |
| corvus | okay, i think we're gtg on that, should we run it now? | 16:55 |
| clarkb | corvus: ++ | 16:55 |
| mordred | ++ | 16:56 |
| clarkb | heh 201 not 200 | 16:56 |
| mordred | or, 200 not 201 | 16:57 |
| *** mriedem is now known as mriedem_lunch | 16:57 | |
| corvus | hrm, i don't understand the situation | 16:58 |
| clarkb | the playbook expects a 201 but it got a 200 | 16:59 |
| corvus | right, but the org was not created | 16:59 |
| clarkb | oh | 16:59 |
| corvus | so that's a correct error | 16:59 |
| corvus | ooooh | 17:01 |
| mnaser | I think inagust and Vexxhost are gitea orgs too | 17:01 |
| mnaser | inaugust* | 17:01 |
| corvus | they're pre-existing; we're looking for airship to be created | 17:01 |
| mordred | corvus: oh god - it's because it's redirecting us to opendev.org? | 17:02 |
| corvus | yeah i think it's the ssl redirect | 17:02 |
| clarkb | how did that work when we created vexxhost/ and inaugaust? | 17:03 |
| mordred | yeah | 17:03 |
| corvus | hrm | 17:03 |
| corvus | https | 17:05 |
| mordred | yes! | 17:05 |
| mordred | and it worked | 17:05 |
| corvus | i've updated this playbook and the migration playbook | 17:05 |
| corvus | i should have run -f10 sorry | 17:05 |
| mordred | no worries - I'm just thrilled it's working | 17:06 |
| clarkb | so the issue was we needed https://localhost? | 17:06 |
| corvus | yeah | 17:06 |
| clarkb | before we rerun repos-first.yaml the mysql udpates we've already done should be safe to reapply right? | 17:07 |
| corvus | i think we're ready to run the migration with first again; you have the conn :) | 17:07 |
| clarkb | we do an update where and that where won't match anymore so it should noop right? | 17:07 |
| corvus | i believe so because nothing will match the where clause | 17:07 |
| clarkb | mordred: ^ | 17:07 |
| mordred | yes | 17:08 |
| corvus | i highlighted the relevant lines | 17:08 |
| clarkb | ok I'm ready to rerun repos-first.yaml when you hand over the con again :) | 17:08 |
| mordred | the where clause should not match | 17:08 |
| corvus | clarkb: it's yours | 17:08 |
| clarkb | mordred: ready? | 17:08 |
| mordred | yup | 17:08 |
| clarkb | ok gitea failed | 17:10 |
| clarkb | but everything else looked fine | 17:10 |
| corvus | how is repos not defined? | 17:10 |
| corvus | oh | 17:11 |
| corvus | we need an include vars there | 17:11 |
| clarkb | ah its a separate play | 17:11 |
| corvus | ya | 17:11 |
| clarkb | like that? | 17:11 |
| corvus | i vote we fix it, then copy that play to its own playbook and run it | 17:11 |
| corvus | clarkb: lgtm | 17:12 |
| mordred | ++ | 17:12 |
| corvus | or maybe even that play and all the ones that follow? | 17:13 |
| clarkb | corvus: ya let me update to do the following ones too | 17:13 |
| clarkb | and actually we don't need the index backup until later so I'll remove that | 17:14 |
| corvus | ++ | 17:14 |
| clarkb | how does that look now | 17:14 |
| corvus | good | 17:14 |
| mordred | ++ | 17:15 |
| clarkb | running the new playbook now | 17:15 |
| clarkb | ok xenapi-os-testing should be done | 17:15 |
| corvus | https://opendev.org/x/xenapi-os-testing looks good | 17:15 |
| mordred | it does - but the redirect doesn't work | 17:15 |
| mordred | isn't that the thing you fixed with that upstream patch corvus ? | 17:17 |
| corvus | wow, it's not in a final release yet | 17:18 |
| corvus | it's in 1.8.0-rc1 | 17:18 |
| mordred | crap | 17:19 |
| clarkb | hrm I thought we upgraded gitea to pick that stuff up. Maybe they needed to backport and we didn't realize? | 17:19 |
| mordred | we could potentially cherry pick it out and apply it in our docker build - but that would obviously take some work with stuff back online | 17:19 |
| clarkb | mordred: we could trust an infra-root to push the image out of band | 17:19 |
| clarkb | then correct when stuff is online again | 17:19 |
| mordred | that is true | 17:19 |
| corvus | or we could manipulate the database | 17:19 |
| mordred | is it just a db fix? | 17:20 |
| corvus | i believe i understand what needs to be done | 17:20 |
| mordred | ok | 17:20 |
| corvus | we would insert records in the db based on our move mapping to cause the redirects to happen | 17:20 |
| mordred | I'm comfortable with that for today - it seems somehow like the least janky option | 17:20 |
| corvus | and yeah, the redirection is handled entirely by the db. | 17:20 |
| mordred | cool | 17:20 |
| corvus | so maybe we continue for now, and perform "add redirects to gitea db" after we're back online? | 17:21 |
| clarkb | corvus: is that safe to add after the fact or do we need to record project ids or similar? | 17:21 |
| clarkb | I'm trying to verify storyboard and utterly failing because i don't understand its search | 17:21 |
| corvus | should be fine to do after the fact | 17:22 |
| clarkb | xenapi-os-testing doesn't return a thing like it does for nodepool | 17:23 |
| mordred | clarkb: I don't get anything for https://storyboard.openstack.org/#!/project/openstack/xenapi-openstack-testing or https://storyboard.openstack.org/#!/project/x/xenapi-openstack-testing | 17:23 |
| mordred | clarkb: is it really a storyboard project? | 17:24 |
| corvus | was it in storyboard in the first place? | 17:24 |
| clarkb | oh good question | 17:24 |
| mordred | probably not I'd guess | 17:24 |
| clarkb | no it was not | 17:24 |
| mordred | good. well, it still isn't | 17:24 |
| clarkb | ok if we are doing gitea org redirects later should we rerun our playbook with another test repo and then move on to the rest? | 17:25 |
| corvus | since it stops on failure, i'm okay running the rest now | 17:26 |
| clarkb | it won't continue the next item in the loop? it completely exits? | 17:26 |
| mordred | yeah - I'm also ok running the rest now | 17:26 |
| clarkb | ok | 17:26 |
| clarkb | corvus: is the https://localhost fix in the opendev_migration.yaml too? | 17:28 |
| corvus | yes | 17:28 |
| *** igordc has joined #openstack-infra | 17:28 | |
| clarkb | alright ready to go then? | 17:28 |
| clarkb | command is up | 17:28 |
| corvus | ++ | 17:29 |
| clarkb | mordred: ? | 17:29 |
| mordred | ++ | 17:29 |
| clarkb | alright here goes | 17:29 |
| mordred | fingers crossed | 17:29 |
| mordred | this isn't going to be the FASTEST thing :) | 17:32 |
| mnaser | purposely done so | 17:32 |
| mnaser | for lunch (or breakfast) time | 17:32 |
| mnaser | :) | 17:32 |
| mordred | :) | 17:32 |
| clarkb | it is about halfway through the changes db update | 17:37 |
| *** jamesmcarthur has quit IRC | 17:40 | |
| clarkb | about 100 repos to update changes db for | 17:40 |
| clarkb | so should be done in a minute or two I expect | 17:41 |
| clarkb | well then it goes on to do the other things | 17:41 |
| corvus | mordred, clarkb: drat; there is a code change needed for the redirects | 17:42 |
| corvus | i think i should build a gitea image manually with the patch and push it up | 17:43 |
| clarkb | corvus: ++ | 17:43 |
| corvus | should we stop the migration script so that we don't have to do the database entries manually, or should we plan on doing that and restarting gitea afterwords? | 17:44 |
| clarkb | stopping it seems potentially painful to figure out what needs what updates but that is doable | 17:44 |
| corvus | we could take gitea down | 17:44 |
| clarkb | corvus: oh that would do it | 17:44 |
| corvus | that would probably stop it at exactly the right place :) | 17:44 |
| clarkb | indeed | 17:45 |
| clarkb | sounds good to me if you can do that relly quick | 17:45 |
| corvus | will do | 17:45 |
| corvus | gitea is down | 17:46 |
| corvus | i will go build the image now | 17:46 |
| clarkb | ok and we haven't gotten to that play yet | 17:46 |
| corvus | should i just rebuild our image based on upstraem 1.8.0-rc3? | 17:48 |
| *** eernst has joined #openstack-infra | 17:49 | |
| clarkb | maybe stick to 1.7.x for now? | 17:49 |
| corvus | ok. 1.7.4 which we were running or 1.7.6? | 17:49 |
| clarkb | the paranoid in me says 1.7.4 | 17:49 |
| clarkb | :) | 17:49 |
| clarkb | we can do A/B testing of gitea updates when not in the middle of the migration and be sure that the other versions work | 17:49 |
| corvus | k | 17:50 |
| clarkb | (though I guess we can always set the image label back to the older version too) | 17:50 |
| mordred | ++ to 1.7.4 | 17:51 |
| corvus | we need to be running one of these patches for the redirects to work at all | 17:51 |
| corvus | so this is worth doing so that if anything goes wrong in later versions, we have a working version to fall back on | 17:51 |
| mordred | ++ | 17:52 |
| clarkb | that is a good point | 17:52 |
| corvus | https://github.com/jeblair/gitea/tree/opendev | 17:53 |
| corvus | and i promise you, the irony is not lost on me. :) | 17:53 |
| corvus | (our docker file wants to clone from a web accessible location...) | 17:53 |
| clarkb | that apepars to be two jeblair commits on top of 1.7.4 | 17:53 |
| mordred | yes. I concur, and think that's a good location | 17:54 |
| mordred | and also ironic | 17:54 |
| *** jamesmcarthur has joined #openstack-infra | 17:55 | |
| mordred | corvus: maybe once gerrit is back up, we should also propose a system-config patch to update the dockerfile to point to that location until there isa 1.8.0 release we upgrade to | 17:55 |
| corvus | ++ | 17:55 |
| mordred | so we don't accidentally revert | 17:55 |
| corvus | i'll just push up what i'm doing right now | 17:55 |
| *** jamesmcarthur has quit IRC | 17:55 | |
| mordred | cool | 17:55 |
| corvus | building now | 17:56 |
| corvus | i have manually inserted the redirect for xenapi-os-testing (which is how i discovered the missing piece) | 17:56 |
| corvus | so we should be able to confirm that's working when we bring gitea up with the new images | 17:57 |
| corvus | looks like the process has stopped as expected | 17:57 |
| clarkb | I agree | 17:57 |
| clarkb | I am goign to take this as an opportunity for a short bio break | 17:58 |
| clarkb | back in a minute | 17:58 |
| corvus | shall we bring gitea back online now? (with the old images) | 17:58 |
| mordred | corvus: crappit | 17:58 |
| *** eernst has quit IRC | 17:58 | |
| mordred | corvus: https://github.com/go-gitea/gitea/pull/5987 is also in 1.8.0 not in 1.7.x | 17:58 |
| corvus | heh | 17:58 |
| corvus | ok i'll cherry pick that too | 17:58 |
| corvus | mordred: https://github.com/jeblair/gitea/commits/opendev lg? | 17:59 |
| *** roman_g has joined #openstack-infra | 17:59 | |
| mordred | ++ | 18:00 |
| corvus | rebuilding | 18:00 |
| clarkb | I think we've gotten too used to CD :P | 18:00 |
| clarkb | surely if the code is merged it goes straight to production | 18:00 |
| corvus | shall i restart gitea now? i stopped it by stopping docker, which i'll need to restarct anyway to pull the new images... | 18:00 |
| corvus | clarkb: yeah... :) | 18:00 |
| clarkb | corvus: ya the playbook is not running so should be safe to do that | 18:01 |
| corvus | i mean, there have been at least 2 point releases since then which these didn't end up in... :| | 18:01 |
| clarkb | I am going to make a new playbook that removes everything before the gitea stuff | 18:01 |
| corvus | clarkb: ++ | 18:01 |
| * mordred was scanning through the gitea 1.8 changelog | 18:02 | |
| mordred | doesnm't look like anything scary | 18:02 |
| smcginnis | So not just gerrit is offline? All services? | 18:03 |
| clarkb | smcginnis: gerrit, gitea and zuul | 18:03 |
| clarkb | smcginnis: everything else is up and running | 18:03 |
| smcginnis | kk, thanks | 18:03 |
| corvus | https://opendev.org/ is back up | 18:03 |
| clarkb | and we should have gitea back and happy momentarily | 18:03 |
| corvus | and https://opendev.org/openstack/xenapi-os-testing is doing the infinite redirect which will be fixed by the new version | 18:04 |
| mordred | corvus: neat! | 18:05 |
| corvus | mordred: are you still personally a member of opendevorg? | 18:05 |
| corvus | mordred: if so, can you add me? | 18:05 |
| mordred | let's find out | 18:05 |
| mordred | nope | 18:05 |
| corvus | ok, i'll dig out the passwords :) | 18:06 |
| clarkb | smcginnis: rought state of things is once gitea is back with the redirect fix we will run the playbook to update gitea configs, then the next step after confirming it and storyboard are happy is to run fungi's inplace update of repos for the renaming. Then when we are happy with that we can start gerrit and the real fun begins :) | 18:06 |
| corvus | the password we have for opendevzuul is not working | 18:08 |
| clarkb | corvus: is ours the base64 version? | 18:08 |
| clarkb | I ran into that debugging the registry stuff once | 18:08 |
| corvus | i'm looking in the password file | 18:08 |
| clarkb | (we store base64 in zuul and that gets converted on the output) | 18:09 |
| mnaser | corvus: I think the stuff stored in .docker/config is a base64 of username:password | 18:09 |
| clarkb | oh or maybe it wants it base64 as input | 18:09 |
| smcginnis | clarkb: Thanks for that status. ;) | 18:09 |
| corvus | no i'm just trying to log into the web ui with the password stored in our encrypted password store | 18:09 |
| corvus | there should be no base64 involved | 18:09 |
| corvus | the password for opendevzuul is the same as openstackzuul which seems suspicious to me | 18:10 |
| corvus | we may need to decrypt the zuul secret :| | 18:10 |
| mnaser | do we recall who created the opendevorg .. org? | 18:10 |
| mordred | yeah - that was me - but I have since divested myself of personal ownership | 18:11 |
| mordred | corvus: can you log in as openstackzuul with the password that's there - and does openstackzuul have access to opendevorg ? | 18:12 |
| mnaser | ah gotcha | 18:12 |
| corvus | mordred: no that's what i've been trying to type here | 18:12 |
| corvus | not sure why this is confusing :) | 18:12 |
| corvus | mordred: oh openstackzuul for opendevorg | 18:13 |
| corvus | mordred: no | 18:13 |
| mordred | yeah | 18:13 |
| mordred | k | 18:13 |
| mordred | just checking | 18:13 |
| corvus | mordred: login works, only access to zuul orgs | 18:13 |
| mordred | sad panda | 18:13 |
| mnaser | I guess someone is gonna have to work on decrypting https://opendev.org/openstack-infra/system-config/src/branch/master/.zuul.yaml#L25-L40 | 18:13 |
| corvus | i'm working on decrypting the zuul secret | 18:13 |
| clarkb | corvus: root history has some decryption stuff if your grep for base64 | 18:13 |
| clarkb | er on zuul01 | 18:14 |
| corvus | i have a script | 18:14 |
| corvus | https://review.openstack.org/639771 | 18:14 |
| *** mriedem_lunch is now known as mriedem | 18:15 | |
| *** cgoncalves has quit IRC | 18:15 | |
| corvus | logged in and password store updated | 18:17 |
| mordred | \o/ | 18:17 |
| corvus | added myself | 18:17 |
| corvus | pushing | 18:19 |
| *** e0ne has joined #openstack-infra | 18:19 | |
| corvus | i do not have a large upstream pipe. | 18:19 |
| corvus | clarkb: so the expanse was really good. | 18:20 |
| clarkb | corvus: yup | 18:20 |
| corvus | have you read the books? are they good? | 18:20 |
| clarkb | I've read the first 4 or 5 after that they kinda fizzled out for me | 18:21 |
| clarkb | but the first set are worth a read | 18:21 |
| corvus | ok, i'll give 'em a shot | 18:21 |
| clarkb | are you on a dsl pipe? I seem to recall your area of the world has a good independnet dsl provider | 18:22 |
| corvus | i know you said the show could seem slow due to worldbuilding, and i agree you should commit to at least a few eps. but one of the things i liked about it it never felt like a wasted episode. 3 seasons and they hit the ball every time at bat. even bsg had some episodes that were like "well what was the point of that?" | 18:23 |
| clarkb | ya if you pay attention the early episodes do lead up to later episodes well. But with minimal context it can be hard to pick up on all of that | 18:23 |
| fungi | okay, i'm back at a keyboard, but no idea what the current state is or if i should be helping break something... should i catch up on scrollback or is there anything urgent to jump into right away? | 18:24 |
| clarkb | fungi: let me tldr for you | 18:24 |
| fungi | much obliged! | 18:24 |
| clarkb | fungi: we have done everything in the migrate playbook up to the point where we do teh gitea renames. This is because we have discovered we need a newer version of gitea to include corvus and mordred's fixes to gitea for our redirecting stuff | 18:24 |
| clarkb | fungi: corvus has built that image and is uploading it to dockerhub now. We'll restart gitea on that image as soon as it is available | 18:25 |
| clarkb | fungi: the next step after that is to run the rest of the playbook, then run your in place git updates, then start gerrit assuming things look good | 18:25 |
| fungi | okay, cool... i saw nick highlights about needing to squash something into one of the scripts but i missed which one. something about zuul moves? | 18:26 |
| clarkb | fungi: problems we ran into include stopping gerrit before getting the repos.yaml (your comment said gitea not gerrit and I didn't double check so my bad). Also our migrate playbook had bugs around mysql config speicification and needing to create org dirs before moving repos | 18:26 |
| *** dims has quit IRC | 18:26 | |
| clarkb | fungi: the moves of zuul secret keys | 18:26 |
| clarkb | fungi: corvus had a change up that needed to be squashed into your script change. He took care of that manually | 18:26 |
| fungi | oh! right, opendev-migrate does also hit the gerrit api for a list of actual repos, my bad :( | 18:27 |
| clarkb | overall things are looking good right now. Its just been a slow one item at a time slog | 18:27 |
| fungi | and what was the modification which needed squashing? i included the requested addition of zuul/main.yaml | 18:27 |
| corvus | clarkb: yeah, i'm thinking of re-watching it, actually... i think it'll hold up... | 18:27 |
| corvus | fungi: the generation of the zuul key file moves | 18:27 |
| fungi | oh! i totally missed you asking about that. yes i can see where that would be needed, sorry! | 18:28 |
| corvus | clarkb: and yeah, i use sonic.net. they are rolling out fiber, but haven't gotten to me, so i'm still on bonded adsl (~40mbit down) | 18:28 |
| clarkb | fungi: as for where you can help we have a root screen running on bridge if you want to be a second set of eyeballs for when we run the next playbook. And if you can stick around long enough maybe you want to be the one to run your edit git in place | 18:28 |
| fungi | we probably should do zuul keyfile moves in the rename_repos playbook though, right? i assume that's going to come up any time we do a repository rename in the future? | 18:28 |
| corvus | https://hub.docker.com/u/opendevorg is updated | 18:29 |
| clarkb | fungi: as a future improvement yes, but for now we are just going to run that script I think | 18:29 |
| *** dims has joined #openstack-infra | 18:29 | |
| fungi | sure, that makes total sense. the scripts i pushed up are entirely utilitarian and (hopefully) one-shot | 18:29 |
| corvus | i'm ready to docker-compose pull and restart | 18:29 |
| mordred | corvus: ++ | 18:29 |
| clarkb | corvus: ++ | 18:29 |
| mordred | fungi: corvus started a list of improvements we need to make for the future on the bottom of the etherpad | 18:29 |
| fungi | awesome | 18:30 |
| fungi | also joined screen session on bridge | 18:30 |
| corvus | running docker-compose pull on all gitea servers | 18:30 |
| corvus | sorry i should have done that in the screen | 18:30 |
| corvus | "docker-compose up -d --timeout 60" is what our playbooks do next; i think that will restart? | 18:32 |
| mordred | I think so | 18:32 |
| corvus | i'll do that now then | 18:32 |
| clarkb | ya that should be fine | 18:33 |
| mordred | redirect worked! | 18:33 |
| corvus | https://opendev.org/openstack/xenapi-os-testing works | 18:33 |
| *** edmondsw has quit IRC | 18:33 | |
| clarkb | yay | 18:33 |
| mordred | ok. that was exciting | 18:33 |
| corvus | i think we're ready to pick up | 18:33 |
| mordred | corvus: ++ | 18:33 |
| clarkb | I've pulled up the edited playbook for just gitea and after tasks in the screen | 18:34 |
| clarkb | does that look good to everyone? | 18:34 |
| mordred | clarkb: yes. | 18:34 |
| corvus | ++ | 18:34 |
| clarkb | alright drafting up the command now | 18:34 |
| clarkb | how does that look? do we want to run it on a single repo first? | 18:35 |
| clarkb | or jsut go with it since it will fail fast? | 18:35 |
| corvus | i think go | 18:35 |
| clarkb | mordred: fungi ^ | 18:35 |
| clarkb | fwiw I'm just go too considering it fails fast | 18:35 |
| mordred | I thnk just go | 18:36 |
| * clarkb gives fungi a few more seconds to object before hitting enter | 18:36 | |
| clarkb | alright hitting enter | 18:37 |
| clarkb | seems to be running | 18:37 |
| mordred | yeah. and picking out random ones the rename/redirect seem to be working | 18:38 |
| fungi | yep, sorry, still catching up | 18:38 |
| mordred | https://opendev.org/openstack/fuel-ccp-searchlight -> https://opendev.org/x/fuel-ccp-searchlight | 18:38 |
| mordred | for instance | 18:38 |
| mordred | works | 18:38 |
| mordred | https://opendev.org/airship/drydock is there - yay new orgs | 18:39 |
| mordred | I am comfortable that that is working properly | 18:39 |
| clarkb | any objections if I go make a kettle of tea real quick while we wait for this? | 18:39 |
| * fungi is glad he wrote the opendev-patching script so that it will work regardless of whether it happens before or after the renames playbook | 18:39 | |
| mordred | corvus: how long does the crsf token last? | 18:39 |
| mordred | fungi: ++ | 18:39 |
| mordred | clarkb: do it | 18:39 |
| fungi | clarkb: i suspect you have time to get through several cups of tea before this playbook is done | 18:40 |
| fungi | looks like we're getting on the order of 1-2 renames per second | 18:40 |
| corvus | mordred: i'm not sure | 18:40 |
| mordred | corvus: here's hoping the time is "longer than we need it to" | 18:40 |
| fungi | it'll either be that, or the other thing | 18:41 |
| corvus | https://github.com/go-macaron/csrf/blob/master/csrf.go | 18:42 |
| corvus | maybe indefinite | 18:42 |
| mordred | yeah | 18:43 |
| corvus | i will start heating up lunch | 18:43 |
| mordred | fungi: so - it should be done in about 13 minutes | 18:43 |
| mordred | ++ | 18:43 |
| * mordred stands up and walks around for no reason | 18:43 | |
| *** rf0lc0 has joined #openstack-infra | 18:43 | |
| clarkb | I looked at my yard asthey just treated it with a larvacide | 18:44 |
| *** eglute has quit IRC | 18:45 | |
| clarkb | japanese beetles are bad and state is killing them | 18:45 |
| fungi | won't someone think of the larva?!? | 18:45 |
| fungi | ahh, yeah | 18:45 |
| fungi | i need to get out and address some shrubbery this weekend. semi-annual bulk pickup starts monday | 18:46 |
| *** rfolco has quit IRC | 18:46 | |
| fungi | i have a *lot* of neglected pruning | 18:46 |
| *** e0ne has quit IRC | 18:46 | |
| clarkb | this is the sort of thing wher ethey say "you can opt out but if you do you need a valid medical excuse from a doctor otherwise we'll get a court order overriding your opinion" | 18:47 |
| *** dims has quit IRC | 18:48 | |
| *** eglute has joined #openstack-infra | 18:48 | |
| clarkb | thinking out loud here about next steps. When we start gerrit do we want to keep the iptables rules preventing access in place? that way gerrit can replicate in a more controlled manner? | 18:48 |
| corvus | clarkb: i think that's a stellar idea | 18:49 |
| fungi | what amount of replication are we anticipating? | 18:49 |
| clarkb | fungi: all of it because gerrit replicates everything on start | 18:49 |
| corvus | we could probably be running the zuul move scripts now (i don't expect that to take long though) | 18:49 |
| fungi | i guess just the usual "i don't know what may have changed so i'll replicate everything to be safe" that gerrit does, right | 18:49 |
| clarkb | yup | 18:49 |
| fungi | yeah, file moves within the same filesystem will be near-instant | 18:50 |
| clarkb | I have ginger lemon tea now | 18:50 |
| fungi | at most a few seconds | 18:50 |
| fungi | to complete the full set | 18:50 |
| mordred | https://opendev.org/jjb/jenkins-job-builder \o/ - happy to see that with it's own nice little home | 18:51 |
| *** dims has joined #openstack-infra | 18:51 | |
| clarkb | I think we are about 1/3 of the way through | 18:51 |
| corvus | mordred: it'll have a friend there too with python-jenkins | 18:51 |
| mordred | corvus: it should have a friend | 18:51 |
| corvus | shall i run the zuul scripts now? | 18:51 |
| clarkb | corvus: I'm good with that if you are | 18:52 |
| clarkb | don't forget to mkdir first | 18:52 |
| clarkb | fungi: ^ this is probably the biggest lesson of the day | 18:52 |
| clarkb | none of our renaming stuff assumed new orgs :) | 18:52 |
| mordred | funny that :) | 18:52 |
| mordred | corvus: ++ | 18:52 |
| corvus | i've run the dir creation | 18:53 |
| corvus | now i'll run "bash -xe zuul-rename.sh" | 18:53 |
| corvus | that should fail fast if there's a prob | 18:53 |
| clarkb | ++ | 18:53 |
| mordred | and succeed fast if there's not | 18:53 |
| corvus | mv: cannot stat '/var/lib/zuul/keys/ssh/project/gerrit/openstack/deb-os-refresh-config': No such file or directory | 18:53 |
| clarkb | ah we are renaming some repos that never zuulv3'd | 18:53 |
| corvus | maybe i should just run without -e then | 18:54 |
| corvus | it moved a bunch of other stuff before that successfully | 18:54 |
| clarkb | ya maybe write to a log file then we can grep all the cannot stats | 18:54 |
| clarkb | and check if they look sane | 18:54 |
| mordred | ++ | 18:54 |
| corvus | bash -x zuul-rename2.sh >zuul-rename.log 2>&1 | 18:55 |
| corvus | that look right? | 18:55 |
| corvus | did i get the redirect order correct? | 18:56 |
| clarkb | 2>&1 is correct | 18:56 |
| clarkb | I don't think the order of that vs >zuul-rename.log matters much? | 18:56 |
| mordred | no, it doesn't | 18:56 |
| corvus | k. that's done | 18:57 |
| mordred | woot | 18:57 |
| *** Goneri has quit IRC | 18:57 | |
| corvus | http://paste.openstack.org/show/749545/ | 18:57 |
| clarkb | the deb-* and loci-* are fine aiui (loci transitioned to a single repo for all their images) | 18:58 |
| clarkb | nothing is really jumping out at me as a mistake there | 18:59 |
| mordred | me either | 18:59 |
| clarkb | we are past the halfway mark on the gitea renames | 18:59 |
| mordred | woot | 19:00 |
| clarkb | fungi: once this is done I think the next thing is your in place git repo edits | 19:00 |
| fungi | 139 nsfod errors some of which are for the same project, seems like a relatively small subset at least | 19:00 |
| clarkb | fungi: nsfod? | 19:00 |
| AJaeger | no such file or directory? | 19:01 |
| clarkb | oh no such file or directory ya | 19:01 |
| * AJaeger never heard that abbreviation before, I might be wrong... | 19:01 | |
| fungi | i was being hip, channeling my inner millennial | 19:02 |
| *** auristor has quit IRC | 19:02 | |
| AJaeger | ;) | 19:02 |
| AJaeger | clarkb: https://opendev.org/opendev/puppet-vinz is a RETIRED repo, should that really have gone to opendev? Something to double check aferwards? | 19:03 |
| fungi | where is the copy of repos.yaml file we're working from for the current playbook run? | 19:03 |
| clarkb | AJaeger: I think we moved all the puppetry | 19:03 |
| AJaeger | clarkb: ah | 19:03 |
| clarkb | AJaeger: I'm not too worried about it since over time we will grow retired repos anyway | 19:03 |
| AJaeger | ;) | 19:03 |
| corvus | fungi: bridge:~root | 19:04 |
| fungi | thanks | 19:04 |
| clarkb | so in the ansible output it says it is redirecting openstack/stx-manifest to starlingx/stx-manifest but https://opendev.org/openstack/stx-manifest redirects to https://opendev.org/starlingx/manifest properly | 19:04 |
| fungi | and yeah, things like puppet-vinz makes more sense as a retired opendev repo than a retired openstack repo | 19:05 |
| clarkb | a bug in our log entries but not in the actual redirecting? | 19:05 |
| fungi | clarkb: is it maybe handling org transfers and renames separately? | 19:05 |
| clarkb | fungi: oh maybe | 19:05 |
| corvus | i think we moved vinz to x but forgot to move its puppet with it | 19:05 |
| clarkb | corvus: ya I'm not going to lose any sleep over that | 19:05 |
| corvus | and yes, those are two separate operations in gitea | 19:05 |
| clarkb | the vinz thing | 19:06 |
| corvus | "transfer repo ownership" is the org move and "update repo name" is the rename within the org | 19:06 |
| clarkb | got it | 19:06 |
| clarkb | just over 200 to go now | 19:07 |
| fungi | was someone else planning to fire up the opendev-patching script or shall i get prepared to do so? | 19:07 |
| clarkb | fungi: I would be most comfortable if you did it :) | 19:07 |
| *** auristor has joined #openstack-infra | 19:08 | |
| clarkb | assuming that doesn't interfere with your holidaying today | 19:08 |
| fungi | happy to if folks want. i'm expecting it to require roughly half an hour but will start it in a screen session just in case my internet goes back to broken (it was offline again for several hours this morning but was back before i got home) | 19:09 |
| fungi | i've just rejoined a root screen session on review.o.o dated from march 18... looks like maybe renidex stuff | 19:10 |
| fungi | i'm going to close that one out and start a fresh one | 19:10 |
| fungi | and then su to gerrit2 within it | 19:10 |
| fungi | also going to copy the repos.yaml from bridge.o.o into ~gerrit2 | 19:11 |
| clarkb | fungi: ++ | 19:12 |
| clarkb | about 100 to go now | 19:13 |
| mnaser | seems like a rate of 100 per 5m | 19:13 |
| fungi | which would be ~1 every 3 seconds | 19:14 |
| clarkb | fungi: let us know when we should attach to the review screen | 19:14 |
| fungi | you can attach at any time | 19:14 |
| fungi | i've got the repos.yaml copied into place already and just confirming the checksum now | 19:14 |
| fungi | looks like it matches | 19:15 |
| *** auristor has quit IRC | 19:16 | |
| *** ijw has joined #openstack-infra | 19:17 | |
| fungi | gerrit's offline so i can't be sure, did anyone make any additional comments or updates to the opendev-patching script after i last touched it? | 19:18 |
| mordred | I did not - I don't think any of us touched the patching script | 19:18 |
| clarkb | I did not | 19:18 |
| clarkb | storyboard is updating now. gitea should be done | 19:19 |
| fungi | okay, i'll copy the last version of it i had onto review.o.o in that case | 19:20 |
| clarkb | the last update I know for it was your fix for the acls in projects.yaml | 19:20 |
| clarkb | doing the mkdir | 19:20 |
| clarkb | storyboard is done, it is backing up the old gerrit index now | 19:20 |
| fungi | is the rename-repos playbook going to try to start gerrit? | 19:21 |
| fungi | and if so, is that a problem? | 19:21 |
| corvus | no it's done and gerrit is offline | 19:21 |
| fungi | okay, cool | 19:22 |
| mordred | woot it is done! | 19:22 |
| clarkb | yup I'm just trying to confirm where the index backup is | 19:22 |
| fungi | ~44 minutes runtime | 19:22 |
| fungi | i'm ready with the opendev-patching script in gerrit2's homedir on review.o.o now whenever we're satisfied it's time to start that | 19:22 |
| mordred | repos in storyboard look corret | 19:22 |
| clarkb | drwxr-xr-x 6 gerrit2 gerrit2 4096 Apr 19 15:43 index.backup.1555701637 | 19:23 |
| clarkb | I think that was from our first pass through with the test repo | 19:23 |
| fungi | that looks like today | 19:23 |
| clarkb | but I don't see one from the most recent run | 19:23 |
| AJaeger | fungi, just one comment on your script: We need to merge a couple of cleanups for project-config before we can merge anything there, I hope I pushed everything up for that. Our linters are not happy with your change as is (files are not sorted anymore). | 19:23 |
| clarkb | I think I'm ok with that given that file was from after gerrit was stopped | 19:23 |
| fungi | AJaeger: yep, that was expected | 19:23 |
| AJaeger | fungi: but let's go ahead with your script... | 19:23 |
| corvus | i'm "go" | 19:24 |
| fungi | i wasn't sure what wuold need to be reordered, but it's just the one repo so should be a single patch | 19:24 |
| *** kranthikirang has joined #openstack-infra | 19:24 | |
| mordred | ++ | 19:24 |
| fungi | clarkb: ready? | 19:24 |
| * AJaeger pushed several ones (one per file) - we can squeeze them together... | 19:24 | |
| clarkb | I'm go with the note that we should have a second backup there and don't but it seems to be ok because we have the one from earlier today | 19:24 |
| clarkb | fungi: ^ | 19:24 |
| *** kaspars__ has joined #openstack-infra | 19:25 | |
| * AJaeger expects not to be around in time for helping with those, so, please take them and do what is needed... | 19:25 | |
| fungi | yep, wfm. starting it now, eta ~20:00z | 19:25 |
| fungi | started and hasn't crashed yet, so that's a plus | 19:25 |
| clarkb | AJaeger: thank you for that | 19:25 |
| clarkb | I wonder if the ansible warning about perms in /tmp is why we don't have a second | 19:26 |
| clarkb | in any case I think we have the backup we need so I won't worry too much | 19:26 |
| fungi | top says python3 and git are the most active processes and the got processes are churning pids so seems to be going | 19:26 |
| corvus | clarkb: we should have only one, and that's the one. it did work. we never ran the backup play previously | 19:27 |
| fungi | it did cinder, and `git --git-dir=/home/gerrit2/review_site/git/openstack/cinder.git show master` has our commit | 19:27 |
| clarkb | corvus: oh the timestamp seems off | 19:28 |
| fungi | same for `git --git-dir=/home/gerrit2/review_site/git/openstack/cinder.git show stable/pike` | 19:28 |
| corvus | clarkb: i think it's preserved from the original due to "cp -ax" | 19:28 |
| clarkb | ah | 19:28 |
| clarkb | yay for more than one set of eyeballs :) | 19:28 |
| corvus | between all of us, we add up to a complete sysadmin :) | 19:28 |
| clarkb | don't remind me of my ls/ln mixup. I'm still annoyed at myself over that one | 19:29 |
| clarkb | ls -s does not do what ln -s does :) | 19:29 |
| corvus | i forgot ansible does things on remote hosts | 19:29 |
| fungi | heck, last night i was trying to do a dict.update() in a variable assignment | 19:30 |
| clarkb | next step at ~2000zulu is start gerrit, check replication and online reindexing? | 19:31 |
| fungi | i believe so | 19:31 |
| fungi | i have a feeling the missing openstack repos on github will cause some replication failures | 19:31 |
| clarkb | should I create the CNAMEs for git.airshipit.org git.starlingx.io and git.openstack.org to point at files.openstack.org now? or wait on that? zuul-ci is in gerrit so it will have to happen when gerrit is running | 19:32 |
| fungi | we're not bouncing gitea again, right? so probably fine to start redirecting nowish? | 19:32 |
| corvus | ++ | 19:32 |
| clarkb | From where I stand gitea is happy and in its final state (other than needing replication to) | 19:33 |
| *** kaspars__ has left #openstack-infra | 19:33 | |
| clarkb | ok I'll start working on those cname updates | 19:33 |
| mordred | ++ | 19:33 |
| corvus | and since we updated the cgit farm, i think we've started 404ing on some things, so now is a good time | 19:34 |
| clarkb | and to double check files.o.o is my target right? | 19:34 |
| clarkb | I should say files.openstack.org | 19:34 |
| corvus | afaik :) | 19:34 |
| fungi | in theory the content on gitea is as intact as the content on the cgit farm | 19:35 |
| fungi | "replication" should be a (probably lengthy) no-op | 19:35 |
| corvus | agreed | 19:35 |
| clarkb | fungi: it will replicate all of your updates happening now | 19:35 |
| clarkb | other than that agreed | 19:35 |
| fungi | ahh, right, those | 19:36 |
| *** igordc has quit IRC | 19:36 | |
| mordred | what did we decide the story is on the renames for the github repos? | 19:37 |
| clarkb | mordred: dmsimard wrote a thing that will transfer ownership of a repo from our org to another org if the target org adds our user as an owner temporarily | 19:39 |
| clarkb | mordred: this was already tested with ara so should be in good shape. Its just a matter of going through those motions | 19:39 |
| clarkb | I have updated dns | 19:39 |
| mordred | yeah - so that's good for things in openstack that are going somewhere else on github and where people care | 19:39 |
| *** auristor has joined #openstack-infra | 19:39 | |
| fungi | for things we're moving into the openstack org today we can probably just run it with our shared admin account | 19:40 |
| mordred | but what about openstack repos that renamed within openstack orgs | 19:40 |
| clarkb | mordred: oh I see devstack etc | 19:40 |
| mordred | nod - so we just need, at some point, to make a filtered versio of repos.yaml and feed it to dmsimard's thing | 19:40 |
| mordred | clarkb: yeah | 19:40 |
| clarkb | mordred: ++ to that plan | 19:40 |
| mordred | clarkb: so I guess that would be anything in repos.yaml where to: starts with openstack/ | 19:40 |
| *** whoami-rajat has quit IRC | 19:40 | |
| fungi | should be able to filer that down really easily | 19:40 |
| corvus | mordred: feel like doing that? might be able to get it done before we bring gerrit up... | 19:40 |
| mordred | yeah. I'll go make the filtered version | 19:41 |
| fungi | s/to/new/ but that, yes | 19:41 |
| clarkb | http://git.airshipit.org/openstack/airship-berth | 19:41 |
| clarkb | be amazed ^ | 19:41 |
| clarkb | (sorry I'm just really excited that worked) | 19:41 |
| corvus | a *lot* went into that :) | 19:41 |
| *** kranthikirang has quit IRC | 19:41 | |
| * clarkb checks project-config | 19:42 | |
| *** kranthikirang has joined #openstack-infra | 19:42 | |
| clarkb | project-config redirects in my browser but git clone not working yet | 19:43 |
| clarkb | I don't think its cached dns so maybe something is not working? | 19:43 |
| corvus | clarkb: git.zuul-ci.org is a cname for git.openstack.org, so that's already in place | 19:43 |
| * fungi is just amused that the top contributor in stackalytics for a very long time to come will be "OpenDev Sysadmins" | 19:43 | |
| mordred | ok. /root/repos-github.yaml | 19:43 |
| clarkb | corvus: ah | 19:44 |
| corvus | clarkb: i agree, the cloning isn't working as expected | 19:44 |
| clarkb | fungi: ^ I know you tested this a bit any idea of why that may be? | 19:44 |
| clarkb | git seems to send a git for the old path | 19:44 |
| fungi | huh, checking | 19:45 |
| clarkb | so is it not respecting the path part of the redirect and only doing the host? | 19:45 |
| *** jamesmcarthur has joined #openstack-infra | 19:45 | |
| clarkb | https://opendev.org/openstack-infra/project-config doesn't clone either | 19:45 |
| clarkb | so I don't think it is files fault | 19:46 |
| fungi | cloning https://git.openstack.org/openstack-infra/zuul doesn't work for me but cloning https://git.openstack.org/zuul/zuul does | 19:46 |
| fungi | leading me to think this may be a gitea issue | 19:46 |
| mordred | yeah. it doesn't work for me just pointing to opendev urls | 19:46 |
| clarkb | ya I wonder if the redirects don't apply to cloning | 19:47 |
| mordred | this is ... very unfortunate | 19:47 |
| clarkb | one workaround would be to do a large lookup table in apache redirects and point to the target | 19:47 |
| fungi | and yeah, i tested cloning insofar as i could, but only tested the apache to gitea redirects, not the inner-gitea redirects | 19:48 |
| clarkb | ya | 19:48 |
| *** slaweq has joined #openstack-infra | 19:48 | |
| clarkb | but basically have apache do a redirect from openstack-infra/zuul to zuul/zuul to address that? | 19:48 |
| corvus | we don't have an apache in front of gitea | 19:49 |
| clarkb | no but we are updating opendev.org and paths at the same time where we do that so its less of a concern | 19:49 |
| mordred | https://github.com/niemeyer/gopkg/issues/50 <-- mention of latest git not following redirects by default | 19:50 |
| clarkb | is that git config somethign we can apply maybe? | 19:51 |
| corvus | i don't understand the relevance of that | 19:51 |
| mordred | I just tried - it doesn't help | 19:51 |
| mordred | so - potentially irrelevant | 19:51 |
| fungi | yeah, git *is* following redirects from apache on files.o.o to gitea on opendev.org | 19:51 |
| mordred | fungi: ah - yeah - good point | 19:52 |
| clarkb | ya its gitea taht isn't redirecting the git client | 19:52 |
| corvus | patches are done | 19:52 |
| *** eernst has joined #openstack-infra | 19:53 | |
| fungi | i also keep getting "Corrupted Content Error: The site at https://opendev.org/openstack-infra/zuul/ has experienced a network protocol violation that cannot be repaired. The page you are trying to view cannot be shown because an error in the data transmission was detected. Please contact the website owners to inform them of this problem." with firefox | 19:53 |
| clarkb | fungi: I don't get that I go straight to https://opendev.org/zuul/zuul/ | 19:54 |
| fungi | though that may be me... i also keep getting it for https://opendev.org/ | 19:54 |
| AJaeger | fungi: works for me with firefox | 19:54 |
| mordred | fungi: I do not get that with that URL in chrome | 19:54 |
| fungi | i'll restart firefox for good measure | 19:54 |
| clarkb | so with the git repo edits done do we need a plan for the redirects not working with git or do we think that is something we can fix in a followup? | 19:54 |
| clarkb | I think that doing a lookup table in apache for the old domains will get us ~90% of what we need | 19:55 |
| clarkb | people using new domain will liekly have updated to new path too | 19:55 |
| kranthikirang | Team, don't want to disturb you and work; Here is the test I have done from my setup; git clone https://git.openstack.org/openstack/airship-promenade/ | 19:55 |
| kranthikirang | Cloning into 'airship-promenade'... | 19:55 |
| kranthikirang | fatal: repository 'https://opendev.org/openstack/airship-promenade/' not found | 19:55 |
| kranthikirang | redirect works but the path is not correct? | 19:56 |
| clarkb | kranthikirang: that is the issue we are discussing now | 19:57 |
| fungi | kranthikirang: yes, that's what we're working on solving now | 19:57 |
| kranthikirang | cool; thanks | 19:57 |
| clarkb | we could do a quick hand edit on files.o.o for say project-config to prove that that redirect table will work | 19:57 |
| fungi | i suppose we could reset the dns entry in the short term while we continue troubleshooting the gitea redirect | 19:57 |
| corvus | i'm attempting to find the problem with gitea | 19:57 |
| clarkb | fungi: I'm not sure that will help | 19:58 |
| clarkb | fungi: we've moved the repos on cgit servers | 19:58 |
| fungi | ahh, right-o | 19:58 |
| fungi | great point | 19:58 |
| corvus | did someone get a trace of the http interaction? | 19:59 |
| clarkb | corvus: GIT_TRACE_CURL=1 git clone https://git.openstack.org/openstack-infra/project-config | 19:59 |
| clarkb | that will get you one | 20:00 |
| corvus | thx | 20:00 |
| *** eernst has quit IRC | 20:00 | |
| fungi | fwiw, a firefox restart got me working again (at least solved the weird protocol error i was getting) | 20:01 |
| fungi | and thanks for the git envvar magic. i was trying git clone -v -v -v | 20:01 |
| mordred | wow. that's fascinating | 20:01 |
| mordred | 20:01:21.153291 http.c:545 => Send header: GET /openstack/airship-berth/info/refs?service=git-upload-pack HTTP/1.1 | 20:01 |
| mordred | which gets a 404 | 20:02 |
| clarkb | ya gitea must be using different rules for git-upload-pack | 20:02 |
| mordred | just doublechecked that redirects work for other subpath entries | 20:03 |
| clarkb | fungi: looking at the htaccess rules can we do a rewriterule for every rename before the ignore all args comment ? but rewrite to git.openstack.org and fallthrough for the other rules to apply? | 20:03 |
| clarkb | so git.openstack.org/openstack-infra/project-config -> gets rewritten to git.openstack.org/openstack/project-config then we apply any rules below for hashes or whatever | 20:04 |
| clarkb | then that gets sent to opendev.org | 20:04 |
| mordred | in browser, https://opendev.org/airship/berth/info/refs works - but https://opendev.org/openstack/airship-berth/info/refs does not | 20:04 |
| fungi | clarkb: that's probably a viable workaround, if the bug in gitea turns out to be nontrivial to fix | 20:05 |
| fungi | easy enough to test by just editing the .htaccess file on disk. no apache restarts/reloads needed | 20:05 |
| clarkb | yup I'm drafting up a rule like that for project-config and will paste it here before I modify .htaccess | 20:06 |
| *** arunkant has joined #openstack-infra | 20:06 | |
| *** basante has joined #openstack-infra | 20:07 | |
| clarkb | fungi: RewriteRule RewriteRule "^(cgit/|)openstack-infra/project-config(.*)$" "$1openstack/project-config/$2" | 20:07 |
| clarkb | er only one RewriteRule prefix | 20:07 |
| *** ralonsoh has joined #openstack-infra | 20:08 | |
| clarkb | need to remove the / before $2 | 20:08 |
| AJaeger | clarkb: LGTM | 20:08 |
| clarkb | RewriteRule "^(cgit/|)openstack-infra/project-config(.*)$" "$1openstack/project-config$2" | 20:08 |
| clarkb | ok I'm going to add that and test it | 20:09 |
| fungi | i think so... may want to do something to better separate the .* at the end lest we end up with unwanted substring matches | 20:09 |
| *** jcoufal has joined #openstack-infra | 20:10 | |
| fungi | not that there's going to be a openstack-infra/project-config-something-else but for other renames there might be (and might be matched by something which was rewritten in an earlier entry from the mapping, and might even end up causing a circular redirect under the right circumstances) | 20:11 |
| mordred | ++ | 20:11 |
| clarkb | RewriteRule "^(cgit/|)openstack-infra/project-config(/(.*)|)$" "$1openstack/project-config$2" | 20:11 |
| clarkb | fungi: ^ is that better? | 20:11 |
| fungi | (((.git/?)?|/).*|)$ maybe"? | 20:13 |
| clarkb | I think taht requires you to have a / if you .git but let me improve it | 20:14 |
| *** jamesmcarthur has quit IRC | 20:14 | |
| clarkb | RewriteRule "^(cgit/|)openstack-infra/project-config(.git|)(/(.*)|)$" "$1openstack/project-config$2$3" | 20:15 |
| fungi | well, that's the reason i had the additional ? in it | 20:15 |
| clarkb | oh I see | 20:16 |
| fungi | but what you have there would probably be roughly the same matches | 20:16 |
| clarkb | ok giving that a go | 20:16 |
| fungi | hopefully we'll either not need it or at least not need it for very long, because this doesn't solve future renames for folks who are already using opendev.org remotes | 20:17 |
| fungi | so it's a short-term stop-gap at best | 20:18 |
| corvus | i've identified a potential change; testing now. | 20:18 |
| corvus | (to gitea) | 20:18 |
| fungi | also the opendev-patching script seems to have completed, 27m19s | 20:18 |
| *** basante has quit IRC | 20:18 | |
| mordred | corvus: nice! | 20:18 |
| fungi | slightly faster than when i was testing in /opt, but that could also be due to having the server entirely idle | 20:19 |
| clarkb | ok my fix doesn't work even though I redirect to the desired location | 20:19 |
| clarkb | this is beacuse git does a listing of all the repos and branches at that location and the original url isn't one of them | 20:19 |
| fungi | `git --git-dir=/home/gerrit2/review_site/git/openstack/project-config.git show master` shows acl moves too, so i think we're good | 20:20 |
| clarkb | http.followRedirects may be what I need for my fix to work | 20:20 |
| clarkb | testing | 20:20 |
| corvus | my first fix almost worked: http://paste.openstack.org/show/749550/ | 20:22 |
| corvus | it lost the query string in the redirect | 20:23 |
| clarkb | corvus: thats a good sign | 20:23 |
| corvus | clarkb: i don't understand what you're saying about http.followRedirects | 20:23 |
| mordred | oh - so close | 20:23 |
| fungi | i have serious doubts git's http.followRedirects is relevant at all here | 20:24 |
| clarkb | corvus: with my htaccess update git is listing the branches properly on opendev for openstack/project-config but none of them are openstack-infra/project-config so it rejects the data there | 20:24 |
| clarkb | supposedly you can tell git to follow those redirects ? | 20:25 |
| corvus | well, any solution that requires us to ask people to set a git option is a non-starter | 20:25 |
| clarkb | yes I agree | 20:25 |
| corvus | that's like, roll-back territory. | 20:25 |
| clarkb | (I'm just trying to understand what the options are here, mroe than happy for your proper fix to properly fix it :)) | 20:26 |
| *** ralonsoh has quit IRC | 20:26 | |
| corvus | well, i'm worried about whether mine will given that a simple apache redirect doesn't work? | 20:26 |
| clarkb | oh wait I see anotheri ssue int he apache redirect | 20:27 |
| clarkb | GET /openstack/project-config/info/refs/project-config/info/refs?service=git-upload-pack HTTP/1.1 | 20:27 |
| clarkb | its redirecting to a bad path so maybe I misinterpreted the error. I will work on understanding that rewrite | 20:27 |
| fungi | http.followRedirects: Whether git should follow HTTP redirects. If set to `true`, git will transparently follow any redirect issued by a server it encounters. If set to `false`, git will treat all redirects as errors. If set to `initial`, git will follow redirects only for the initial request to a remote, but not for subsequent follow-up HTTP requests. Since git uses the redirected URL as the base | 20:28 |
| fungi | for the follow-up requests, this is generally sufficient. The default is `initial`. | 20:28 |
| fungi | so i suppose it's not an all-or-nothing option, it's a trinary | 20:28 |
| fungi | looks like that was added in git 2.11.1 | 20:29 |
| fungi | so has been in place more than a couple years | 20:29 |
| clarkb | fungi: I don't understand how I end up with that GET above since I shouldn't hit any of the cgit/ rules | 20:30 |
| clarkb | fungi: any idea what might be going on there? | 20:30 |
| mordred | for contrast, I did the GIT_TRACE_CURL thing with github.com/openstack/ara - github does not return a redirect, it returns the actual content directly from the old path | 20:31 |
| fungi | clarkb: i'm missing what's wrong with the path you pasted | 20:31 |
| fungi | what should it have been/what did you expect there? | 20:32 |
| clarkb | GET /openstack-infra/project-config/info/refs?service=git-upload-pack is the initial request and gets rewritten to GET /openstack/project-config/info/refs/project-config/info/refs?service=git-upload-pack | 20:32 |
| clarkb | fungi: the rewrite has project-config twice | 20:32 |
| mordred | clarkb: I agree - that rewritten url is definitely wrong | 20:33 |
| fungi | ahh, /project-config/info/refs is repeated. so perhaps the positional vars aren't what we expected | 20:35 |
| clarkb | RewriteRule "^(cgit/|)openstack-infra/project-config(.git|)(/(.*)|)$" "$1openstack/project-config$2$3" is the rule | 20:36 |
| corvus | \o/ http://paste.openstack.org/show/749552/ | 20:37 |
| corvus | that appears to work for a simple in-org rename | 20:37 |
| corvus | i'll try an org tranfer and org tranfser + rename next | 20:37 |
| *** slaweq has quit IRC | 20:37 | |
| mordred | corvus: woot! | 20:37 |
| fungi | an encouragingly small patch | 20:38 |
| mordred | corvus: since you posted the first one, I've been reading that code trying to see if I could spot the second thing ... I'm glad you did :) | 20:38 |
| mordred | so - we STILL might need to get clarkb's thing to work if recent git only follows the first redirect | 20:39 |
| clarkb | I'm still at a loss | 20:40 |
| clarkb | do I need to root with / at the beginning of my regex? | 20:40 |
| fungi | shouldn't | 20:40 |
| fungi | the earlier matches work and don't start with a / | 20:40 |
| corvus | all 3 cases check out: http://paste.openstack.org/show/749553/ | 20:41 |
| *** dwalt has joined #openstack-infra | 20:41 | |
| clarkb | corvus: nice | 20:41 |
| mordred | corvus: excellent. I support restarting our giteas with that | 20:41 |
| fungi | awesome! we need gitea images built from a fork with that patch applied though, right? | 20:41 |
| clarkb | I'll keep banging my head on the rewrite rule in case that becomes useful | 20:41 |
| clarkb | fungi: yup corvus already has one of those handy | 20:41 |
| corvus | okay. i'll start the process for that | 20:42 |
| mordred | clarkb: do you really need the (cgit/|) ? | 20:42 |
| clarkb | mordred: no I don't think so | 20:42 |
| clarkb | so why don't I start by simplifying that | 20:42 |
| mordred | maybe remove that for ... yeah | 20:42 |
| clarkb | RewriteRule "^openstack-infra/project-config(.git|)(/(.*)|)$" "openstack/project-config$1$2" is the new rule | 20:42 |
| mordred | give it a shot | 20:43 |
| clarkb | still getting GET /openstack/project-config/info/refs/project-config/info/refs?service=git-upload-pack but one less thing to debug now | 20:43 |
| fungi | oh, right, the cgit urls aren't an issue anyway, just the git+http protocol urls | 20:44 |
| mordred | it's so weird that project-config gets repeated and not openstack/ | 20:44 |
| clarkb | mordred: right or openstack-infra | 20:44 |
| mordred | yeah | 20:45 |
| corvus | this look good to everyone? https://github.com/jeblair/gitea/commit/cae7c35bfc9c834cdd8ca14f5ec73bd120f51493 | 20:45 |
| clarkb | corvus: I think so | 20:46 |
| corvus | and we're up to 4 commits on that branch: https://github.com/jeblair/gitea/commits/opendev | 20:47 |
| mordred | corvus: ++ | 20:47 |
| corvus | i'm rebuilding the image now (using that, rather than the docker source directory) to make sure it's clean | 20:48 |
| mordred | awesome | 20:48 |
| *** ykarel|away has quit IRC | 20:48 | |
| corvus | pushing | 20:49 |
| *** jcoufal has quit IRC | 20:50 | |
| fungi | i have hopes the additional rewrites on files.o.o will be unnecessary. my reading of the http.followRedirects=initial default is that it follows however many redirects it encounters to arrive at content for its initial request, but then expects subsequent requests to be served from the same place the initial request finally was | 20:50 |
| mordred | ah - ok | 20:51 |
| mordred | good | 20:51 |
| mordred | that would be way more pleasing | 20:51 |
| clarkb | I think adding a [L] may fix my redirect | 20:51 |
| clarkb | so if it is necessary I'm close to a solution there I think | 20:52 |
| mordred | cool | 20:52 |
| mordred | I agree with fungi - I hope that we don't need that | 20:52 |
| fungi | clarkb: oh, yes you likely do need a [L] (the ones i added needed one as well) | 20:52 |
| mordred | and that reading seems to make more sense | 20:52 |
| fungi | yeah, only following the first in a chain of redirects would be an odd misfeature | 20:53 |
| clarkb | http://git.openstack.org/openstack-infra/project-config works now with the [L] | 20:53 |
| mordred | clarkb: awesome | 20:53 |
| clarkb | if corvus fix doesn't work for some reason we have an out with my redirect | 20:53 |
| corvus | images pushed | 20:54 |
| mordred | fungi: we live in an age of constant odd misfeatures | 20:54 |
| clarkb | RewriteRule "^openstack-infra/project-config(.git|)(/(.*)|)$" "openstack/project-config$1$2" [L] | 20:54 |
| clarkb | I'm going to comment that out now | 20:54 |
| clarkb | so that we don't have it interfere with corvus' things | 20:54 |
| corvus | i will dig out my bridge screen and use it to restart gitea | 20:54 |
| clarkb | I've commented out my files.o.o htaccess change | 20:55 |
| corvus | bridge command look good? | 20:56 |
| clarkb | corvus: yes | 20:57 |
| *** kjackal has quit IRC | 20:58 | |
| fungi | clarkb: puppet would likely eventually revert it anyway | 20:58 |
| clarkb | fungi: ya though for puppet to work it needs that redirect to work aiui | 20:58 |
| corvus | git clone https://git.openstack.org/openstack-infra/project-config | 20:59 |
| clarkb | because project-config | 20:59 |
| mordred | clarkb, fungi: I've got repos-github.yaml - what's the dmsimard playbook I should feed it to? | 20:59 |
| corvus | clarkb: that's what you were trying ^? | 20:59 |
| clarkb | corvus: yes | 20:59 |
| corvus | seems to work for me now | 20:59 |
| clarkb | corvus: I agree you are a magician | 20:59 |
| mordred | woot! | 20:59 |
| mordred | I cloned airship-berth and it worked | 20:59 |
| fungi | `git clone https://git.openstack.org/openstack-infra/zuul` is working for me now | 21:00 |
| clarkb | woo I think we can call that fixed now | 21:00 |
| clarkb | kranthikirang: can you check too? | 21:00 |
| clarkb | kranthikirang: try the clone/fetch you were doing before that failed | 21:00 |
| fungi | as is `git clone https://git.zuul-ci.org/zuul` | 21:00 |
| kranthikirang | yeah, git clone https://git.openstack.org/openstack/airship-promenade/ | 21:00 |
| kranthikirang | its working | 21:00 |
| * corvus quickly closes stackoverflow window with "what do they call null in go again?" before anyone notices | 21:00 | |
| kranthikirang | cool, team | 21:00 |
| clarkb | kranthikirang: thank you for checking and sorry for the trouble | 21:00 |
| smcginnis | corvus: ;) | 21:00 |
| kranthikirang | clarkb: No, I am watching all your conversation | 21:00 |
| mordred | kranthikirang: you can now also git clone https://opendev.org/airship/promenade | 21:01 |
| kranthikirang | thank you; I was struck because of this :) | 21:01 |
| clarkb | ok so where were we | 21:01 |
| fungi | kranthikirang: thanks for helping out! | 21:01 |
| clarkb | gitea redirects work | 21:01 |
| clarkb | gerrit in place commits have been made | 21:01 |
| mordred | I think we were at fungi changes | 21:01 |
| clarkb | are we ready to start gerrit? | 21:01 |
| mordred | ah - I think we are | 21:01 |
| mordred | unless we want to run the github renames | 21:01 |
| clarkb | or at least verify those changes then start gerrit? | 21:01 |
| fungi | which fungi changes? those finished like an hour ago | 21:01 |
| mordred | so that gerrit doesn't derp on replicating to thigns that have moved | 21:02 |
| clarkb | fungi: the in place git repo updates | 21:02 |
| clarkb | mordred: thats a good point | 21:02 |
| clarkb | mordred: are you able to drive that? | 21:02 |
| corvus | i dig doing the github stuff if it's ready, it will save weirdness later | 21:02 |
| corvus | i also dig mordred driving it :) | 21:02 |
| mordred | clarkb, corvus : happy to drive - what's the playbook? | 21:02 |
| mordred | I have the data file ready | 21:02 |
| fungi | me too. the less i touch github, the less murderous i feel | 21:02 |
| clarkb | uh it was dmsimard change of course with gerrit off its harder to find these things :) | 21:02 |
| mordred | I think we maybe didn't land it | 21:03 |
| clarkb | ya | 21:03 |
| corvus | i'm on it. :) | 21:03 |
| mordred | well piddle | 21:03 |
| mordred | yay! | 21:03 |
| clarkb | mordred: its ok corvus has everything cached | 21:03 |
| mordred | corvus is the magical pony of today | 21:03 |
| fungi | gertty ftw | 21:03 |
| * mordred dubs corvus Star Baker | 21:03 | |
| *** Emine has quit IRC | 21:04 | |
| corvus | oh, i only score half a point on this one | 21:04 |
| corvus | my gertty doesn't have the latest rev cached. however, it's 644937 in system-config | 21:04 |
| corvus | so we can pull it out of gerrit's repos with that | 21:04 |
| clarkb | latest rev did have a bugfix too | 21:04 |
| * mordred is certain if Paul Hollywood were here, he'd shake corvus' hand | 21:04 | |
| fungi | i have no idea what a paul hollywood is, but sounds glamorous anyway | 21:05 |
| clarkb | mordred: how many projects are there to move like that out of curiousity? | 21:05 |
| clarkb | beacuse we can probably click buttons too | 21:05 |
| corvus | i'll grab the script from gerrit's repos | 21:05 |
| clarkb | corvus: k | 21:05 |
| mordred | clarkb: 31 | 21:05 |
| fungi | fewer than i thought | 21:05 |
| clarkb | ya 31 is within the realm of reasonable sanity if we have to do it by hand | 21:05 |
| mordred | yeah | 21:05 |
| clarkb | I'll go get my keychain in case that becomes necessary | 21:05 |
| *** diablo_rojo has joined #openstack-infra | 21:06 | |
| mordred | fungi: paul hollywood is one of the judges on the Great British Baking Show ... and if he's really impressed with your cake, he shakes your hand and it's a big deal when he does | 21:06 |
| pabelanger | o/ I'm online for the next few hours, if there is anything I can help with. Otherwise, keep up the awesome work! | 21:06 |
| corvus | looks like refs/changes/37/644937/6 is the one | 21:07 |
| fungi | mordred: oh, i think i've seen that, but only notable because it was co-hosted by noel fielding (a.k.a. vince noir from the mighty boosh) | 21:08 |
| corvus | mordred: bridge:~root/github-org-transfer.py | 21:08 |
| mordred | fungi: yes. noel is a lovely co-host | 21:08 |
| clarkb | fungi: I think you mean it crowd | 21:08 |
| mordred | corvus: aweome. looking at that now | 21:08 |
| fungi | clarkb: that too | 21:09 |
| fungi | clarkb: he was richmond on the it crowd, yes | 21:09 |
| fungi | but i liked him on boosh better | 21:09 |
| kranthikirang | I just ran update_software to see how the git clone works on repos and looks like everything is good; just FYI | 21:09 |
| mordred | kranthikirang: woot! | 21:09 |
| mordred | corvus: so - looks like it would be maybe easiest to write a playbook that does the loop on the repos | 21:09 |
| fungi | kranthikirang: thanks! that's an excellent real-world test, i expect | 21:09 |
| mordred | clarkb: becuase the script wants username and password set - and I'm guessing we can find those in hiera somewhere | 21:10 |
| kranthikirang | fungi: sure thing | 21:10 |
| corvus | mordred: ++ | 21:10 |
| fungi | mordred: i believe the password ends up being the otp code | 21:10 |
| fungi | so may need to get a new otp passcode each call, i don't exactly know how that works | 21:10 |
| corvus | # TODO: Add support for two factor authentication by passing the "x-github-otp" | 21:11 |
| clarkb | ya so I think dmsimard got a token with the otp | 21:11 |
| mordred | oh good | 21:11 |
| corvus | # The password can be the account's password or a personal access token created | 21:11 |
| corvus | # at https://github.com/settings/tokens | 21:11 |
| clarkb | so you go into the web ui, get a token using otp then you don't ahve to otp when you run the script | 21:11 |
| fungi | aha, that's easier then | 21:12 |
| mordred | ... neat ... so, I don't think I have yet used the otp thing - anybody got a pointer to the instructions on that? | 21:12 |
| clarkb | mordred: its in the secrets file | 21:13 |
| fungi | they're in the password list | 21:13 |
| mordred | cool | 21:13 |
| fungi | easy cut-n-paste commands | 21:13 |
| clarkb | I wonder if we even need ansibel for that? | 21:15 |
| mordred | I'm sorry - I'm feeling really dumb here - I have no idea how this works or how I'm supposed to use it. | 21:15 |
| fungi | try to log into the github webui with that username | 21:16 |
| clarkb | mordred: so you login with the password listed as per normal. Then github will send you to a secondary screen saying enter your otp code | 21:16 |
| fungi | it will prompt for an otp code | 21:16 |
| mordred | ok | 21:16 |
| clarkb | mordred: then you run the command given in the secrets file to get the otp code and enter its output in that second login screen | 21:16 |
| fungi | run the command on bridge with the secret to generate an otp code and then paste that into github | 21:16 |
| fungi | after that you should be able to go to account settings or something and find where to generate an api token | 21:17 |
| mordred | ok. neat. that worked | 21:17 |
| mordred | is this a Personal Access Token perhaps? | 21:18 |
| fungi | sounds reasonable | 21:18 |
| clarkb | mordred: that is what the comment above that corvus pasted says | 21:18 |
| mordred | cool | 21:19 |
| *** cgoncalves has joined #openstack-infra | 21:19 | |
| mordred | ok. I think those are good. now we just need to run that script with the arguments | 21:20 |
| mordred | does that look good to folks? | 21:23 |
| clarkb | where do we supply the token? | 21:23 |
| corvus | looking | 21:23 |
| clarkb | also might want to run it on a single repo to start like we did with the other moves | 21:23 |
| mordred | clarkb: I put it in GITHUB_PASSWORD env var | 21:24 |
| mordred | clarkb: good point | 21:24 |
| clarkb | the env var won't get passed through ansible will it? | 21:24 |
| corvus | may as well put it in the script | 21:24 |
| corvus | openstack/git-review doesn't look right | 21:25 |
| clarkb | hrm https://opendev.org/opendev/git-review is a 404 :/ | 21:26 |
| mordred | that does seem to be what we rewrote it to | 21:26 |
| corvus | however, that's what's in repos.yaml | 21:26 |
| corvus | yes; i think that may be an error that goes back to the first script | 21:26 |
| corvus | we did spot-check some infra things | 21:26 |
| mordred | yup | 21:26 |
| mordred | git-restack is correctly in opendev | 21:26 |
| dtroyer | catching up a bit with some testing: should "git clone https://git.openstack.org/openstack/stx-manifest.git" work? It does without the trailing .git | 21:26 |
| clarkb | the ethercacl has it as opendev/git-review | 21:27 |
| mordred | that work for folks? | 21:28 |
| clarkb | dtroyer: I can confirm that doesn't seem to work in the general case but dropping .git does | 21:28 |
| clarkb | mordred: if that question is re the edits to the moving script yes I think so | 21:28 |
| clarkb | do we think we move git-review later? | 21:28 |
| mordred | cool | 21:28 |
| clarkb | (to opendev that is) | 21:28 |
| corvus | yeah, mordred dropped it from the github list | 21:29 |
| mordred | well - I put it in the single-list to use as a test | 21:29 |
| mordred | but I can pick up a different one | 21:29 |
| mordred | how's that for a test run | 21:29 |
| clarkb | mordred: wfm | 21:30 |
| corvus | wfm | 21:30 |
| mordred | cool. going to try it | 21:30 |
| mordred | seems to have worked | 21:30 |
| corvus | agree | 21:30 |
| fungi | it's not clear to me what happened with git-review... some problem earlier when i was afk? | 21:30 |
| clarkb | yup I agree too | 21:30 |
| corvus | fungi: it ended up in openstack rather than opendev | 21:31 |
| mordred | k. I'm going to run on the remainder if everyone is ok with that | 21:31 |
| *** Lucas_Gray has joined #openstack-infra | 21:31 | |
| clarkb | and seems to be due to original repos.yaml state | 21:31 |
| corvus | mordred: ++ | 21:31 |
| clarkb | mordred: ++ | 21:31 |
| fungi | weird, repos.yaml should have taken the value from the ethercalc | 21:31 |
| clarkb | mordred: hrm is the token one shot? | 21:31 |
| mordred | do I need to generate a new token for each thing? | 21:31 |
| corvus | fungi: i see the problem; it's a typo in the ethercalc | 21:32 |
| mordred | oh - openstack-dev | 21:32 |
| clarkb | oh wow ya I see it too | 21:32 |
| mordred | openstackadmin may not have openstack-dev access | 21:32 |
| corvus | mordred: i'll see if my account can fix that | 21:32 |
| clarkb | I can fix it if you can't | 21:33 |
| fungi | corvus: oh, good, not a systemic issue with the script then | 21:33 |
| *** yamamoto has joined #openstack-infra | 21:33 | |
| mordred | yeah - it looks still owned by us indivdually | 21:33 |
| clarkb | openstack/ openstack-infra/ openstack-dev/ are the three that that needs to be owner on right? are there any old entries from a different source? | 21:34 |
| fungi | mordred: i think dmsimard noted that he had to do something to add permissions for the openstackadmin account in the openstack org too | 21:34 |
| clarkb | ya that was the owner perms from team member I think | 21:34 |
| * fungi checks the saved notifications from that | 21:34 | |
| mordred | yeah - it's working on openstack and openstack-infra | 21:34 |
| *** ijw has quit IRC | 21:34 | |
| corvus | clarkb, mordred: i see openstackadmin as a maintainer in openstack-dev | 21:34 |
| mordred | can you make it an owner? | 21:34 |
| corvus | i may not understand the acl situation | 21:35 |
| *** diablo_rojo has quit IRC | 21:35 | |
| corvus | didn't they get rid of owners? | 21:35 |
| fungi | notification from that said "granted admin access" | 21:35 |
| fungi | so needs to be set as an admin of the org | 21:35 |
| clarkb | corvus: if you clikc the little gear next to it in people/ then select manage there is an otpion to make it owner | 21:35 |
| mordred | https://github.com/orgs/openstack-dev/people show 7 of us as owners, whatever that means | 21:35 |
| clarkb | I think that is what we need | 21:35 |
| fungi | seems like org members can modify and add/remove repositories but admin perms are needed to do transfers? | 21:36 |
| corvus | done | 21:36 |
| corvus | makes perfect sense <EYE ROLL> | 21:36 |
| *** ijw has joined #openstack-infra | 21:36 | |
| clarkb | looks much happier | 21:36 |
| corvus | so that's set on -dev and -infra | 21:36 |
| fungi | yeah, i can't make heads or tails of their new rbac | 21:36 |
| corvus | anything else for me to check? | 21:36 |
| clarkb | corvus: I think that is all of them that are going into openstack/ | 21:37 |
| mordred | ok. the devstack redirect works | 21:37 |
| ianw_pto | o/ ... i can't imagine i'm much help at this point but am aroundish | 21:37 |
| corvus | i guess we'll find out if i see red :) | 21:37 |
| *** ianw_pto is now known as ianw | 21:37 | |
| mordred | they all seem happy now | 21:37 |
| corvus | shall we move git-review? | 21:37 |
| clarkb | corvus: now is probably a reasonable time to do it | 21:37 |
| clarkb | everything is off etc | 21:37 |
| mordred | yeah | 21:37 |
| *** yamamoto has quit IRC | 21:38 | |
| corvus | run the one-off rename playbook with just that... we'll also have to manually fixup the zuul config for it | 21:38 |
| clarkb | yup | 21:38 |
| clarkb | I can drive that | 21:38 |
| corvus | clarkb: go for it | 21:38 |
| mordred | clarkb: con is yours | 21:38 |
| mordred | corvus: also - move zuul keys | 21:38 |
| mordred | since I don't think the one-off rename playbook has that yet | 21:38 |
| corvus | i'll move the zuul keys | 21:38 |
| clarkb | how does that repo yaml file look? | 21:39 |
| clarkb | ianw: we are in root screen on bridge if interested in watching | 21:39 |
| mordred | looks great | 21:40 |
| corvus | clarkb: lgtm. zuul keys moved. | 21:40 |
| *** eernst has joined #openstack-infra | 21:40 | |
| mordred | clarkb: I'm assuming repos in github that we're now going to be not replicating to and will become stale are something we'll chalk up for later, yeah? | 21:40 |
| mordred | something something archive flag something I think? | 21:40 |
| mordred | (not urgent we do before gerrit update of course) | 21:40 |
| corvus | mordred: (that's the assumption i'm working under) | 21:41 |
| clarkb | mordred: ya plan was to help people do org transfers like you just did when we can coordinate on both sides | 21:41 |
| *** ijw has quit IRC | 21:41 | |
| clarkb | how does my command look in the screen? ready to run that? | 21:41 |
| fungi | which may also involve unarchiving then first | 21:41 |
| corvus | clarkb: ++ | 21:41 |
| clarkb | ok running that now | 21:41 |
| mordred | nod. I think while I'm in there and looking at it, maybe I should go ahead and archive git-review, since it's moving to 'not mirrored to github' land | 21:42 |
| fungi | since at some point in the near future whoever takes over curating the openstack org on gh will probably want to archive all the stuff which got moved out of there | 21:42 |
| mordred | fungi: ++ | 21:42 |
| clarkb | ok that looks happy should double check the gerrit index backup again maybe | 21:42 |
| clarkb | and test that cloning works | 21:42 |
| corvus | https://git.openstack.org/openstack-infra/git-review goes to https://opendev.org/opendev/git-review | 21:43 |
| clarkb | I can clone it too | 21:43 |
| mordred | awesome | 21:43 |
| clarkb | ok before we continue a few things. dtroyer points out that the .git suffix redirects aren't working | 21:43 |
| fried_rice | In case nobody has mentioned this yet today: https://www.youtube.com/watch?v=KSQyW_l8OgE&feature=youtu.be&t=46 | 21:43 |
| clarkb | also in the etherpad we have it saying we should update the gerrit config and apache before starting gerrit | 21:44 |
| clarkb | do we actually want to do ^ before hand or later? | 21:44 |
| clarkb | fried_rice: ha | 21:44 |
| corvus | fried_rice: lol :) | 21:44 |
| mordred | so - I'm not going to archive git-review in github - it mentions it makes it read-only but it can still be cloned - we might want to push a 'this repo is no longer here, please see opendev' commit before archiving | 21:44 |
| mordred | so let's leave that til later | 21:44 |
| corvus | mordred: agree | 21:44 |
| clarkb | wfm | 21:44 |
| clarkb | I think we already expect people to update remotes so its not the end of the world to me that .git doesn't work (I don't knwo if gitea expects to support that? /me tests) | 21:45 |
| corvus | if not, we can have the redirect drop it | 21:45 |
| clarkb | ya even a direct clone with .git without any redirects fails | 21:46 |
| *** eernst has quit IRC | 21:46 | |
| mordred | kk. so we just need to have the redirects drop .git | 21:46 |
| clarkb | so this may be a slightly annoying corner case that we roll forward with | 21:46 |
| mordred | yeah | 21:46 |
| mordred | we can fix it after the fact | 21:46 |
| clarkb | ++ | 21:46 |
| clarkb | dtroyer: ^ fyi we plan to fix it but not a blocker for now | 21:46 |
| clarkb | dtroyer: thank you for catching that and letting us know | 21:46 |
| dtroyer | clarkb: I think if the redirects can drop it we can live with it | 21:47 |
| clarkb | so now we have to decide if we update gerrit before we start it or if we start it as review.openstack.org one last time | 21:47 |
| clarkb | also I need a quick bio break | 21:47 |
| clarkb | brb | 21:47 |
| corvus | here are the redirects if anyone feels like hacking on that: https://opendev.org/opendev/system-config/src/branch/master/modules/openstack_project/files/git-redirect.htaccess | 21:48 |
| fungi | i thought i tested with .git extensions in urls | 21:48 |
| clarkb | git clone https://opendev.org/opendev/git-review.git does not work for me fwiw | 21:49 |
| dtroyer | fungi: this is specifically with git clone | 21:49 |
| clarkb | with the browser it works | 21:49 |
| dtroyer | agreed ^^^ | 21:49 |
| smcginnis | Would it be it is getting a "/" appended to the URL when it should not? | 21:50 |
| corvus | yeah, i think we did not know that we needed the apache redirects to drop .git when going to gitea because gitea does not support .git | 21:50 |
| smcginnis | Hmm, that works in browser with "/" too. | 21:50 |
| fungi | yeah, looks like maybe we're redirecting .git suffixes (which i tested) but that perhaps gitea doesn't support them? | 21:50 |
| corvus | oh weird | 21:50 |
| corvus | gitea web ui does render with '.git' | 21:50 |
| clarkb | corvus: yup | 21:50 |
| * corvus starts to get worried we may want another gitea patch | 21:50 | |
| mordred | nah - I think just dropping them in the redirects is still fine | 21:51 |
| fungi | it does seem less like a missing feature and more like an actual bug, if they support browsing with .git extensions | 21:51 |
| mordred | I don't think it's important to support cloning with .git extension from the opendev urls | 21:51 |
| mordred | but yeah | 21:51 |
| mordred | what fungi said | 21:51 |
| corvus | (i'm really tired of searching for the word "git" in the gitea source) | 21:51 |
| *** fried_rice is now known as efried | 21:51 | |
| mordred | I think if we never get around to fixing that, the world will continue turning | 21:52 |
| clarkb | ya I still don't think it is a blocker. If we note that as a gotch and drop .git in our redirect htaccess file I think we'll be good | 21:52 |
| mordred | ++ | 21:52 |
| fungi | well, the rewrite commits very well may have updated clone urls in some jobs with .git extensions, granted jobs cloning from git rather than from on-disk copies is not recommended | 21:52 |
| mordred | clarkb: I'm torn on updating gerrit - I could go either way | 21:52 |
| mordred | fungi: yeah - if people are cloning from git.openstack.org in their jobs, they get to keep both parts | 21:53 |
| fungi | i agree "fixing" it on files.o.o seems like a fine solution | 21:53 |
| clarkb | I'm kinda leaning towards turning it on as review.openstack.org so that we can merge the chagne "normally" (even if that means a force merge) | 21:53 |
| mordred | clarkb: yeah. I think me too | 21:53 |
| *** mriedem has quit IRC | 21:53 | |
| mordred | we can always hup gerrit later to pick up the changes - it's not necessary to do while off | 21:53 |
| fungi | which change are we needing to merge? | 21:53 |
| *** jamesmcarthur has joined #openstack-infra | 21:53 | |
| mordred | the change to turn review.openstack.org into review.opendev.org | 21:54 |
| clarkb | fungi: the one that updates apache and gerrit config to serve gerrit as review.opendev.org | 21:54 |
| fungi | oh, hah. how very catch-22 | 21:54 |
| mordred | also - do we want to update zuul's main.yaml and restart it before we turn gerrit back on? | 21:54 |
| mordred | or are we ok with maybe missing events? | 21:54 |
| *** kranthikirang has quit IRC | 21:54 | |
| corvus | i think we should get gerrit up and running, replicate, then update zuul's config and start zuul | 21:55 |
| mordred | kk. cool | 21:55 |
| clarkb | corvus: before we turn it into review.opendev.org (to be clear)? | 21:55 |
| fungi | what else do we need to update in main.yaml which the opendev-patching script didn't get? | 21:55 |
| mordred | fungi: nothing - it's just that we need to roll that file out to zuul.o.o | 21:55 |
| fungi | ahh | 21:55 |
| fungi | righty-o | 21:56 |
| mordred | fungi: or, that is - nothing that we know of | 21:56 |
| clarkb | also we can possibly leave the firewall rules in place so that we can merge what we want without anyone else pushing stuff | 21:56 |
| clarkb | though that might get weird with openid login redirects | 21:57 |
| mordred | I'm going to revoke the personal access token I created for openstackadmin | 21:57 |
| corvus | i lean toward manually merging the review.opendev.org change, then turning it on | 21:57 |
| clarkb | mordred: ++ | 21:57 |
| clarkb | corvus: that also works for me | 21:57 |
| corvus | also fwiw, i think i found the one line to change in gitea to make .git clones work... i can start testing that in the background if we want | 21:57 |
| clarkb | corvus: that might be a good task while we wait for gerrit to replicate once it is up? | 21:58 |
| clarkb | so what I'm hearing is merge review.opendev.org switchover change, apply it to review.opendev.org ?, wait for replication and start zuul ? | 21:58 |
| *** jamesmcarthur has quit IRC | 21:58 | |
| mordred | we'll need to manually merge the change, then run ansible to apply the stuff | 21:58 |
| mordred | clarkb: that is what I'm hearing | 21:58 |
| clarkb | fwiw we could also just apply it to bridge.o.o | 21:59 |
| clarkb | then merge it when gerrit is up | 21:59 |
| mordred | yeah | 21:59 |
| *** jamesmcarthur has joined #openstack-infra | 22:00 | |
| clarkb | 653507 is that change | 22:00 |
| mordred | corvus: do you have it in your gertty cache? | 22:00 |
| corvus | yes. patchset 8 is current | 22:00 |
| mordred | cool | 22:00 |
| fungi | sounds good | 22:01 |
| corvus | want me to push it onto /opt/system-config on bridge? | 22:01 |
| mordred | yes | 22:01 |
| corvus | will do | 22:01 |
| mordred | and then we want to kick.sh review yeah? | 22:01 |
| clarkb | mordred: ya as long as we don' | 22:01 |
| clarkb | er | 22:01 |
| clarkb | as long as we don't expect it to update /opt/system-config on bridge or reenable the cron | 22:02 |
| mordred | kick doesn't run bridge.yaml | 22:02 |
| clarkb | k | 22:02 |
| mordred | which is what does that | 22:02 |
| clarkb | sorry I'm really paranoid about all that because it did that in the past | 22:02 |
| mordred | TOTALLY | 22:02 |
| mordred | I'm only saying that confidently because I checked earlier due to your paranoia | 22:02 |
| clarkb | roger | 22:02 |
| clarkb | note that that is going to do a git fetch against https://git.openstack.org/openstack-infra/project-config | 22:03 |
| clarkb | (just something to keep an eye out for) | 22:03 |
| clarkb | on review01 that is | 22:03 |
| clarkb | oh hold on | 22:03 |
| ianw | FYI on the .git extension; mitm proxying it the request is for 127.0.0.1:43346: GET https://opendev.org/opendev/git-review.git/info/refs?service=git-upload-pack | 22:03 |
| clarkb | that git fetch is not going to have the renamed repos in it | 22:04 |
| mordred | oh right | 22:04 |
| clarkb | do we have to worry about manage-projects undoing all our work? | 22:04 |
| clarkb | (sigh) | 22:04 |
| mordred | yeah. I think we need to start gerrit and get stuff replicated | 22:04 |
| mordred | otherwise rolling that change out to gerrit is gonna potentially cause havoc | 22:05 |
| corvus | clarkb, mordred: i have cherry-picked 653507 onto /opt/system-config on bridge | 22:05 |
| mordred | corvus: I think clarkb has noted a reason this is dangerous | 22:05 |
| corvus | i will read back then :) | 22:05 |
| clarkb | also we need to update opendev/git-review in gerrit/projects.yaml for that | 22:05 |
| mordred | yeah | 22:06 |
| corvus | what are we expecting kick.sh to do? | 22:06 |
| mordred | run manage-projects after cloning project-config | 22:06 |
| clarkb | corvus: it may run manage-project (jeepyb) and create all the repos we just moved | 22:06 |
| corvus | oh that would be very bad | 22:06 |
| mordred | yeah | 22:06 |
| corvus | i thought it only did that as part of a special playbook | 22:07 |
| clarkb | beacuse it fetches project-config from https://git.openstack.org/openstack-infra/project-config | 22:07 |
| clarkb | no it is in puppet | 22:07 |
| corvus | "neat" | 22:07 |
| clarkb | it may only run it if the file has updated | 22:07 |
| clarkb | in which case we can check the file on both sides and cross our fingers | 22:07 |
| fungi | yeah, and i guess that will successfully redirect to the current state in gitea which is still missing replication of the opendev-patching result, so agreed bad | 22:08 |
| corvus | i don't think there's anything we need to do to gerrit to bring it up to replicate | 22:08 |
| mordred | me either | 22:08 |
| fungi | i concur | 22:08 |
| clarkb | the one gotcha is git-review | 22:08 |
| corvus | so we can do that, maybe even manually merge the git-review fixup, | 22:08 |
| mordred | we could manually merge the git-review fixup real quick before we start | 22:09 |
| corvus | then after that it should be safe to merge 653507 then kick, then restart | 22:09 |
| mordred | yeah | 22:09 |
| corvus | mordred: that works too | 22:09 |
| clarkb | yup | 22:09 |
| fungi | yeah, should be able to bypass zuul and manually submit the change | 22:09 |
| corvus | so: 1) fix git-review in project-config; 2) start gerrit; 3) merge 653507 in gerrit; 4) wait for replication; 5) kick | 22:10 |
| *** kaspars__ has joined #openstack-infra | 22:10 | |
| clarkb | corvus: yes that sounds right to me | 22:10 |
| mordred | 6) restart gerrit | 22:10 |
| *** ijw has joined #openstack-infra | 22:10 | |
| corvus | ++ | 22:11 |
| corvus | who wants to do #1? :) | 22:11 |
| clarkb | fungi: ^ you probably have the most familiarity with it | 22:11 |
| corvus | http://paste.openstack.org/show/749554/ appears to cause "git clone foo.git" to work with gitea | 22:11 |
| mordred | corvus: I'm so glad you understand the code there | 22:12 |
| corvus | mordred: i absolutely positively 100% understand it and all the implications. | 22:12 |
| fungi | clarkb: hrm, i suppose there could be incorrect git-review references in other repos now too, but hopefully not | 22:13 |
| *** diablo_rojo has joined #openstack-infra | 22:13 | |
| corvus | fungi: yeas, it seems unlikely, and we can fix them later i think. | 22:13 |
| clarkb | ya though in theory they will just cause things to break not create new projects | 22:13 |
| corvus | mordred: (i understand that it looks a lot like the code 20 lines up) | 22:13 |
| fungi | so i guess we need to just update zuul/main.yaml, gerrit/projects.yaml, gerritbot/channels.yaml and maybe move an acl file | 22:13 |
| *** ijw has quit IRC | 22:13 | |
| clarkb | thinking out loud here we could also as a step 4.5 update the project-config on review.openstack.org and manually run manage-projects to confirm it works. Then we don't ahve to worry about puppet hiding an issue and by updating it ourselves puppet shouldn't run it too | 22:14 |
| *** ijw has joined #openstack-infra | 22:14 | |
| corvus | fungi: sounds about right | 22:14 |
| mordred | clarkb: ++ | 22:14 |
| corvus | clarkb: i would not object to that | 22:14 |
| *** rpioso is now known as rpioso|afk | 22:14 | |
| fungi | yeah, wfm | 22:14 |
| *** kaspars__ has left #openstack-infra | 22:15 | |
| fungi | i can push a fix for git-review directly into project-config on disk too, is that what we want? | 22:15 |
| mordred | fungi: yes | 22:15 |
| clarkb | yup then we can start gerrit and let it replicate | 22:15 |
| *** rh-jelabarre has quit IRC | 22:15 | |
| fungi | working on it already | 22:17 |
| fungi | should have it done here in a few minutes | 22:17 |
| fungi | `git grep --files-with-match openstack/git-review` only identifies 4 files | 22:18 |
| fungi | zuul.d/projects.yaml being the only one i didn't already consider | 22:18 |
| corvus | should i rebuild the gitea images with this? https://github.com/jeblair/gitea/commits/opendev | 22:24 |
| clarkb | corvus: +1 if you are happy with it | 22:24 |
| clarkb | the route addition makes sense to me | 22:24 |
| corvus | if you look up a bit in the file you'll see something similar for the web ui | 22:24 |
| mordred | corvus: ++ | 22:24 |
| corvus | 21 lines up. i guess wrong :) | 22:25 |
| corvus | k, i kicked that off, i'll let you know when it's pushed | 22:25 |
| mordred | corvus: we should also get those patches sent in as a PR and see if they'd be willing to squeeze them in to 1.8 - which they might not, given it's on rc4 already - but maybe they're simple enough that they'd accept them for an rc5 | 22:26 |
| corvus | yeah, i'll push up 2 new branches and open prs | 22:26 |
| fungi | does http://paste.openstack.org/show/749556/ look like the patch folks are expecting me to push into the canonical project-config repo? | 22:26 |
| fungi | (that is to say, /home/gerrit2/review_site/git/openstack/project-config.git on review.o.o) | 22:27 |
| clarkb | fungi: yes | 22:27 |
| mordred | fungi: yes | 22:27 |
| mordred | I'm looking forward to the followup fun of moving things from the openstack zuul tenant to the opendev zuul tenant | 22:28 |
| mnaser | how neat, I was going through launchpad, clicked https://git.openstack.org/cgit/openstack/neutron/commit/?id=37d33b2fcd4a1dde356df3b4b4a2f739afb6876c and went straight to the right url, awesome work so far everyone :) | 22:29 |
| fungi | `git --git-dir=/home/gerrit2/review_site/git/openstack/project-config.git show master` shows it applied now | 22:29 |
| mordred | mnaser: \o/ | 22:29 |
| mordred | fungi: awesome. | 22:29 |
| clarkb | ok any last items to check before we start gerrit? | 22:29 |
| clarkb | maybe someone can figure out how to list the queues with the firewall in place | 22:30 |
| *** kaspars__ has joined #openstack-infra | 22:30 | |
| corvus | clarkb: should be able to with agent forwarding | 22:31 |
| ianw | clarkb: ssh to 127.0.0.1 as gerrit2? | 22:31 |
| corvus | or that | 22:31 |
| clarkb | ok should I start it then? | 22:31 |
| clarkb | lets tail the error log in fungi's screen | 22:32 |
| mordred | clarkb: ready as I'll ever be | 22:32 |
| clarkb | I started a tail -f in that screen | 22:32 |
| clarkb | I can start it in another window | 22:32 |
| clarkb | just give me the go ahead | 22:32 |
| fungi | wfm | 22:33 |
| mordred | ++ | 22:33 |
| fungi | no need for another window | 22:33 |
| fungi | that one has nothing useful in it | 22:33 |
| clarkb | fungi: I mean the tail is useful :) | 22:33 |
| clarkb | corvus: ^ ready? | 22:33 |
| fungi | oh, right, yes another window besides the tail | 22:34 |
| * fungi makes universal "thumbs-up" sign | 22:34 | |
| clarkb | should I keep waiting for corvus or go for it? | 22:35 |
| corvus | back | 22:37 |
| clarkb | corvus: I'm ready to start gerrit if you are | 22:37 |
| corvus | 1 sec and i'll catch up | 22:37 |
| clarkb | and fungi's root screen has a tail -f of the gerrit error log | 22:37 |
| clarkb | k | 22:37 |
| corvus | ftr: https://github.com/go-gitea/gitea/pull/6688 | 22:37 |
| corvus | i'm ready | 22:37 |
| clarkb | alright I am starting gerrit now. here goes | 22:38 |
| *** slittle1 has quit IRC | 22:38 | |
| clarkb | it says it is ready | 22:38 |
| corvus | 18800 tasks | 22:39 |
| corvus | 37871 tasks | 22:39 |
| fungi | burning down quickly | 22:39 |
| corvus | 37023 tasks | 22:39 |
| fungi | oh, wait, climbing | 22:39 |
| fungi | but now falling | 22:39 |
| corvus | now heading downhill yeah | 22:40 |
| clarkb | should I start online reindexing? or wait on that for after the second restart? | 22:40 |
| mordred | https://opendev.org/airship/divingbell/commit/c45e4faa126e9a1cad7b5d917b71616b674f5ab4 | 22:40 |
| mordred | lookie there! it's a transition patch | 22:40 |
| corvus | i don't see errors in the replication log | 22:40 |
| clarkb | no errors in the error log either | 22:40 |
| corvus | clarkb: i vote index after second start | 22:40 |
| clarkb | corvus: roger | 22:40 |
| mordred | inaugust.com even got one: https://opendev.org/inaugust/inaugust.com/commit/6b49bac1ae320565ca3704d95cb6cd8262ba9217 | 22:41 |
| fungi | woo! | 22:41 |
| mordred | I find that unreasonably exciting :) | 22:41 |
| *** slittle1 has joined #openstack-infra | 22:41 | |
| corvus | mordred: of course, you somehow tricked fungi into maintaining your website | 22:41 |
| mordred | corvus: yes I did | 22:41 |
| mordred | corvus: I consider 2019 a success now | 22:41 |
| fungi | i'm telling you, "opendev sysadmins" will be the stackalytics stats envy of everyone | 22:41 |
| clarkb | ls-projects works and its output looks coorect on a short skim to me | 22:41 |
| fungi | well, of everyone who thinks stackalytics stats matter | 22:42 |
| dtroyer | fungi: especially now that it has been all fixed up… will anyone believe it? | 22:42 |
| *** Goneri has joined #openstack-infra | 22:42 | |
| fungi | tune in next week to find out | 22:42 |
| clarkb | now down to 36k events | 22:43 |
| corvus | we maybe could have gone ahead and dropped the git farm from the config... | 22:43 |
| clarkb | oh right | 22:43 |
| corvus | not sure if it's slowing us down or not | 22:43 |
| mordred | I stopped caring about stackalytics when I got beat out for the top committer by AJaeger and dhellmann | 22:43 |
| corvus | probably not; the system doesn't have anything else to do | 22:43 |
| corvus | [2019-04-19 22:44:16,319] [d77c9ae5] Cannot replicate to git@github.com:openstack/activity-board.git | 22:44 |
| mordred | corvus: weird. that repo exists | 22:45 |
| corvus | oh, did it git 'github archived'? | 22:45 |
| mordred | it doesn't appear to have been | 22:45 |
| fungi | i don't think we've archived anything | 22:45 |
| corvus | then hrm. | 22:46 |
| clarkb | is it possible they thottled us on that one? | 22:46 |
| mordred | gerrit isn't in its team | 22:46 |
| clarkb | or maybe the repo perms are wrong | 22:46 |
| mordred | teams | 22:46 |
| clarkb | ya thats the thing | 22:46 |
| clarkb | mordred: are you fixing that? | 22:46 |
| fungi | also possible that repo has had broken/incomplete gerrit permissions on gh since it was created | 22:46 |
| fungi | yeah, that | 22:46 |
| mordred | clarkb: I'm not - I logged out already | 22:46 |
| clarkb | ok I can fix it | 22:46 |
| corvus | i thought that was at org level? | 22:46 |
| clarkb | its both | 22:46 |
| mordred | corvus: yeah - manage-projects adds gerrit to a team in the repo | 22:46 |
| fungi | maddening gh rbac once again | 22:47 |
| mordred | or somethign | 22:47 |
| corvus | well, we've established nothing about the github rbac makes sense to me anyway. so ... i'll do something else. | 22:47 |
| mordred | the team exists in the org | 22:47 |
| mordred | the team is granted write permission on the repo | 22:47 |
| mordred | the gerrit user is in the team | 22:47 |
| fungi | i'm looking forward to being able to care much less about gh in the very near future | 22:47 |
| mordred | yup | 22:48 |
| corvus | thirty four thousand eight-hundred eighty three bottles of beer on the wall...thirty four thousand eight-hundred eighty three bottles of beer.... | 22:48 |
| clarkb | ok I added the infra-core team as admin and the gerrit team as writer | 22:48 |
| mordred | corvus: remind me never to play that game with you | 22:48 |
| clarkb | I expect that gerrit will be happy with pushing to that repo in github on its next pass | 22:49 |
| mordred | "yay" | 22:49 |
| corvus | gitea images are pushed, but i think restarting them with gerrit replicating would be counter-productive | 22:49 |
| mordred | I agree | 22:49 |
| mordred | we can restart it when we restart gerrit | 22:49 |
| *** jamesmcarthur has quit IRC | 22:49 | |
| fungi | sounds great | 22:49 |
| corvus | so afaict, we're at "wait for 34464 tasks to complete" | 22:49 |
| mordred | oh - we can merge the review.opendev.org patch now | 22:49 |
| mordred | if anybody has the brain powder to do that with the firewall in place | 22:50 |
| corvus | mordred: brain powder sounds explosive | 22:50 |
| * mordred is very low on powder | 22:50 | |
| clarkb | it might be easiset to do it via ssh | 22:50 |
| clarkb | since that doesn't need the openid login round trip[ | 22:50 |
| * fungi has jelly beans | 22:50 | |
| corvus | yeah, though we'll need to add us to bootstrappers first | 22:51 |
| corvus | also via ssh | 22:51 |
| corvus | i'm up for working that out unless someone else is eager to. | 22:51 |
| clarkb | go for it | 22:51 |
| mordred | I do not feel smart enough to do that without breaking something | 22:51 |
| mordred | right now | 22:51 |
| corvus | on it then | 22:51 |
| clarkb | I think I need to get blood flowing a bit better than it currently is and am gonna walk around a bit | 22:51 |
| *** diablo_rojo has quit IRC | 22:52 | |
| corvus | $ ssh corvus@localhost -p29418 gerrit set-members --add corvus "'Project Bootstrappers'" | 22:53 |
| corvus | Members added to group Project Bootstrappers: corvus@inaugust.com | 22:53 |
| corvus | one down | 22:53 |
| fungi | that's how i've done it in the past, yes | 22:53 |
| corvus | gerrit query doesn't work because of the out of date index i assume | 22:54 |
| corvus | maybe i can still blind review | 22:54 |
| corvus | but first, i have to remove mordred's wip | 22:54 |
| mordred | corvus: piddle. | 22:55 |
| corvus | $ ssh corvus@localhost -p29418 gerrit set-reviewers -p opendev/system-config -r mordred 653507 | 22:55 |
| corvus | fatal: passed project openstack-infra/system-config when creating ChangeNotes for 653507, but actual project is opendev/system-config | 22:55 |
| corvus | we may need to do that reindex now | 22:56 |
| mordred | blerg | 22:56 |
| mordred | yeah. although actually... | 22:56 |
| mordred | corvus: you already put the change in /opt/system-config | 22:56 |
| *** jamesmcarthur has joined #openstack-infra | 22:56 | |
| corvus | mordred: oh you just want me to force-push the commit? | 22:56 |
| clarkb | I think we can index the single change | 22:56 |
| mordred | we don't actually have to land that patch before kicking - it's just project-config that needs to be replicated | 22:56 |
| corvus | mordred: well, gerrit isn't going to replicate it until it merges | 22:57 |
| mordred | we don't need that patch replicated | 22:57 |
| clarkb | https://review.gerrithub.io/Documentation/cmd-index-changes.html | 22:57 |
| mordred | we only need the fungi project-config patch replicated | 22:57 |
| clarkb | mordred: corvus ya thats a good point its the manage-projects metadata we need for puppet to do the right thing | 22:58 |
| mordred | so that kicking with the manually applied patch won't pull old project-config and run manage-projects on it | 22:58 |
| clarkb | if we've got the other change lined up to go we can run it from bridge as is | 22:58 |
| corvus | ok. i'll go circulate my blood then. :) | 22:58 |
| mordred | https://opendev.org/opendev/project-config/commit/565357e66adc66867b6b4fbea2518622feea1084 | 22:58 |
| mordred | wrong project-config :) | 22:59 |
| mordred | openstack/project-config not yet replicaed | 22:59 |
| clarkb | confirmed I don't see it yet either | 22:59 |
| corvus | mind if i take a 15-20 minute break? | 22:59 |
| corvus | we're at 32290 tasks | 22:59 |
| clarkb | corvus: nope I think its gonna take at least that long to eat through the replication queue | 22:59 |
| mordred | go for it | 22:59 |
| fungi | corvus: please take a break | 22:59 |
| clarkb | I'm going to keep walking in circles (it just started raining otherwise I'd lap the block) | 23:00 |
| fungi | it's a holiday for a lot of folks | 23:00 |
| *** diablo_rojo has joined #openstack-infra | 23:03 | |
| clarkb | at 30k ish now | 23:08 |
| clarkb | so this might take another 1.5 hours? | 23:08 |
| mordred | yeah. | 23:09 |
| clarkb | ok since I've got nothing better to do apparently. After we do the steps listed above We should have an up and running gerrit at review.opendev.org. The step after that is to update zuul's config and restart it? | 23:10 |
| clarkb | I'm guessing once we have a zuul mostly working we might be in a decent spot to pause then check in on things in the morning | 23:10 |
| mordred | yeah. I thnik so | 23:11 |
| *** jamesmcarthur has quit IRC | 23:11 | |
| mordred | corvus, clarkb: since it's going to be another 1.5 (or possibly 2 hours) for replication - maybe we _should_ start the reindex too | 23:11 |
| mordred | otherwise we're going to need to reindex post 2nd start - and zuul really can't land changes until the reindex is done | 23:12 |
| *** mattw4 has quit IRC | 23:12 | |
| mordred | (given the error we saw above when corvus tried to land a change in a project that had been renamed) | 23:12 |
| clarkb | that may slow down the replication as they compete for io and cpu but the overall wall time might be less | 23:13 |
| clarkb | I'm not opposed to starting the reindexes now. Do we want to wait for corvus to get back before committing to that? | 23:15 |
| clarkb | one concern is if gerrit won't stop if it is reindexing | 23:15 |
| clarkb | (I don't know if that is the case) | 23:16 |
| clarkb | bashate and bugdaystats also failed against github. I am checking their permissions now | 23:16 |
| corvus | the only reason i can think of to avoid doing the reindex now, is if we wanted to see if we happen to replicate the project-config repo soonish; then we could restart gerrit before completing the initial repliation run | 23:17 |
| *** jamesmcarthur has joined #openstack-infra | 23:18 | |
| clarkb | corvus: oh that is an interesting idea since ya it will kick off replication again after | 23:18 |
| mordred | that is an interesting idea | 23:18 |
| clarkb | zuul might not be super happy about that when it comes up | 23:18 |
| clarkb | but that will correct itself in time | 23:18 |
| mordred | I think we can't start zuul until reindex is done | 23:18 |
| corvus | oh, hey, we can find out when project-config will be replicated; i'll look | 23:19 |
| mordred | would it help anything to do an explicit "replicate openstack/projecr-config" ? | 23:19 |
| fungi | yeah, normally post-rename reindexing isn't time-sensitive because it only has incorrect indices cached for one or a handful of repos | 23:19 |
| corvus | mordred: i think it would go at the end of the queue | 23:19 |
| corvus | mordred: but maybe worth a try :) | 23:19 |
| clarkb | fungi: well and we doubled our target count | 23:19 |
| corvus | i'm pretty sure the ssh error just logged was me :) | 23:20 |
| clarkb | corvus: mordred ya it goes to the end of the queue iirc | 23:20 |
| mordred | kk | 23:20 |
| corvus | itas 9540 tasks away from the present position | 23:20 |
| clarkb | I fixed bashate and bugday stats | 23:20 |
| corvus | actually, the first one is at 9540, the last is at 10063 | 23:21 |
| corvus | so what's that? something like an hour from now? | 23:22 |
| clarkb | ya about an hour I think | 23:22 |
| clarkb | in the 17k absolute queue size range | 23:22 |
| mordred | clarkb: didnt' we rename bashate from openstack-dev ? | 23:24 |
| corvus | we should also drop the k8s gitea replication target | 23:24 |
| mordred | corvus: ++ | 23:24 |
| corvus | (since we didn't do the renames there, we'll need to reset it) | 23:25 |
| clarkb | mordred: maybe? maybe we lose the member data on those moves? | 23:25 |
| mordred | clarkb: whatcha wanna bet that a repo org transfer .. yeah | 23:25 |
| mordred | clarkb: since teams are owned by orgs | 23:25 |
| mordred | you can't move a team membership | 23:25 |
| clarkb | thinking out loud here I think we may want to update system-config to use https://opendev.org/openstack/project-config as project-config before we reenable ansible + puppet | 23:25 |
| clarkb | that way we are always getting that straight from the source | 23:25 |
| mordred | so we should update dmsimard's script - or at least our implementation of it - to then do the "add gerrit team" thing | 23:25 |
| clarkb | ++ | 23:26 |
| clarkb | er though in the general case we won't want that | 23:26 |
| clarkb | because it will be up to the new non us affiliated org to set it up how they want | 23:26 |
| mordred | eyah - actually - we donm't need to fix that | 23:26 |
| mordred | yeah | 23:26 |
| mordred | so it's actually 'correct' - just annoying for 31 of our repos | 23:26 |
| corvus | we just need to go click a bunch of things now, huh? | 23:26 |
| mordred | yeah. I'm pretty sure we'll need to do this for everything in the repos-github.yaml file | 23:27 |
| clarkb | looking like it | 23:27 |
| corvus | we could each take a few | 23:27 |
| clarkb | mordred: want to put them in an etherpad and we can claim some? | 23:27 |
| mordred | yeah - one sec | 23:27 |
| corvus | someone will probably have to tell me what gear icons to click. sorry. | 23:27 |
| clarkb | corvus: et al you go to eg https://github.com/openstack/bugdaystats/settings/collaboration then click the add a team drop down then select infra core and gerrit | 23:28 |
| clarkb | make infra core admin and gerrit write perms in the drop downs that appear in their rows | 23:28 |
| mordred | k. they're in there | 23:28 |
| corvus | clarkb: thank you | 23:28 |
| mordred | I need to go log in as openstackadmin again, right? | 23:28 |
| *** jamesmcarthur has quit IRC | 23:28 | |
| clarkb | mordred: if you've removed your account perms to that org ya | 23:29 |
| *** diablo_rojo has quit IRC | 23:34 | |
| clarkb | ok I think the clicking is done | 23:35 |
| mordred | y'all's so much faster at clicking | 23:35 |
| clarkb | 23.7k so about 6k from project-config | 23:36 |
| corvus | i have a dent in my finger from the trackpoint. | 23:36 |
| mordred | and we're at a rate of about 4 minute / k I think - so probably 20/25 minutes until project-config | 23:37 |
| corvus | i'll go take another 20m break | 23:37 |
| mordred | yeah. I'm going to go look for pants | 23:37 |
| clarkb | reminder of the current plan/todolist 1) fix git-review in project-config; 2) start gerrit; 3) no longer worried about merging this change 4) wait for replication; 4.5) manually update project-config on review.o.o and run jeepy manage-projects 5) kick 6) restart gerrit | 23:38 |
| clarkb | we are at 4) | 23:38 |
| mordred | clarkb: when we kick - we probably also want to kick zuul | 23:38 |
| mordred | to get the updated main.yaml | 23:39 |
| clarkb | zuul isn't running | 23:39 |
| fungi | roadmap was helpful, thanks | 23:39 |
| mordred | kick the puppet | 23:39 |
| clarkb | but ya we need to update the main.yaml and the configparser config | 23:39 |
| mordred | to roll ut the main.yaml | 23:39 |
| mordred | yeah | 23:39 |
| clarkb | so there is a different zuul change to apply that we don't have on /opt/system-config yet. (I think we should do them one at a time?) | 23:39 |
| mordred | ah. kk. yeah | 23:39 |
| clarkb | mordred: its the stuff that says canonical name is opendev.org and connect to review.opendev.org | 23:40 |
| mordred | oh yeah. | 23:40 |
| clarkb | now at 22.7k | 23:40 |
| clarkb | so about 4 minutes per 5k | 23:40 |
| clarkb | or an hour and a half so our estimates were pretty good I think. Though I think we are also saying we can stop worring about replication once projects.yaml replicates | 23:41 |
| clarkb | for 5.5) we can edit the replication config before starting zuul again to remove the git0* and kubernetes gitea to speed that up | 23:41 |
| fungi | the surprised thread on openstack-discuss where people didn't realize gerrit was going to be offline? comedy gold | 23:43 |
| * fungi can't count the number of times/places/ways we announced that, even on that very ml | 23:44 | |
| clarkb | I was worried people were getting annoyed by it too | 23:44 |
| *** nicolasbock has quit IRC | 23:45 | |
| clarkb | alright I'm just counting beer bottles i mean repos that need replication again now | 23:46 |
| clarkb | I'm gonna do a few more laps of the living room | 23:46 |
| *** yamamoto has joined #openstack-infra | 23:48 | |
| paladox | Your running around the living room? | 23:49 |
| clarkb | it is pouring rain outside and I need yo stretch muscles | 23:50 |
| mordred | clarkb: ++ 5.5 | 23:50 |
| paladox | clarkb oh, didn't realise it was raining :) | 23:53 |
| clarkb | project-config has replicated to some mirrors | 23:59 |
| clarkb | gitea07 git03 github and k8s gitea still queued | 23:59 |
| *** jamesmcarthur has joined #openstack-infra | 23:59 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!