| opendevreview | Merged opendev/zuul-providers master: Remove build_diskimage_image_name variable https://review.opendev.org/c/opendev/zuul-providers/+/956373 | 02:15 |
|---|---|---|
| opendevreview | Ivan Anfimov proposed opendev/zuul-providers master: wip https://review.opendev.org/c/opendev/zuul-providers/+/970049 | 11:38 |
| opendevreview | Ivan Anfimov proposed opendev/zuul-providers master: wip https://review.opendev.org/c/opendev/zuul-providers/+/970049 | 11:41 |
| opendevreview | Ivan Anfimov proposed opendev/zuul-providers master: Remove build_diskimage_image_name variable https://review.opendev.org/c/opendev/zuul-providers/+/970049 | 11:44 |
| opendevreview | Ivan Anfimov proposed opendev/zuul-providers master: wip https://review.opendev.org/c/opendev/zuul-providers/+/970042 | 11:48 |
| opendevreview | Dmitriy Chubinidze proposed opendev/zuul-providers master: Use standard x86-64 for AlmaLinux 10 image build https://review.opendev.org/c/opendev/zuul-providers/+/970042 | 12:51 |
| opendevreview | Ivan Anfimov proposed opendev/zuul-providers master: Use standard x86-64 for AlmaLinux 10 image build https://review.opendev.org/c/opendev/zuul-providers/+/970042 | 13:14 |
| opendevreview | Ivan Anfimov proposed opendev/zuul-providers master: Use standard x86-64 for AlmaLinux 10 image build https://review.opendev.org/c/opendev/zuul-providers/+/970042 | 13:15 |
| opendevreview | Ivan Anfimov proposed opendev/zuul-providers master: Use standard x86-64 for AlmaLinux 10 image build https://review.opendev.org/c/opendev/zuul-providers/+/970042 | 13:17 |
| opendevreview | Ivan Anfimov proposed opendev/zuul-providers master: Limit arm64 image builds to producing raw images https://review.opendev.org/c/opendev/zuul-providers/+/968029 | 13:20 |
| opendevreview | Ivan Anfimov proposed opendev/zuul-providers master: Limit arm64 image builds to producing raw images https://review.opendev.org/c/opendev/zuul-providers/+/968029 | 13:20 |
| opendevreview | Ivan Anfimov proposed opendev/zuul-providers master: Limit arm64 image builds to producing raw images https://review.opendev.org/c/opendev/zuul-providers/+/968029 | 13:25 |
| opendevreview | Ivan Anfimov proposed opendev/zuul-providers master: Remove build_diskimage_image_name variable https://review.opendev.org/c/opendev/zuul-providers/+/970049 | 14:44 |
| opendevreview | Ivan Anfimov proposed opendev/zuul-providers master: Remove build_diskimage_image_name variable https://review.opendev.org/c/opendev/zuul-providers/+/970049 | 14:44 |
| opendevreview | Ivan Anfimov proposed opendev/zuul-providers master: Remove build_diskimage_image_name variable https://review.opendev.org/c/opendev/zuul-providers/+/970049 | 14:47 |
| fungi | 5.5 hours until the gerrit upgrade maintenance window starts (21:00 utc) | 15:30 |
| fungi | in 4.5 hours we can send the status notice | 15:30 |
| clarkb | I resemble some sort of operational human being. I've got a change staged locally that adds system-config-core to bindep's gerrit acls for code review +/-2. I figure I can push that up post upgrade which will help us check functionality of gerrit itself and replication. Then if we merge that it will check if project acl updates work | 16:35 |
| clarkb | then we can revert it whenever is convenient. I think it is functionally a noop for us but gerrit won't see it that way | 16:35 |
| clarkb | otherwise I plan to do normal weekend things for a bit but should be back around 2000 UTC | 16:36 |
| clarkb | #status notice Gerrit on review.opendev.org is being upgraded to version 3.11 and will be offline starting at 2100 UTC. We have allocated an hour for the outage window lasting until 2200 UTC | 19:59 |
| opendevstatus | clarkb: sending notice | 19:59 |
| -opendevstatus- NOTICE: Gerrit on review.opendev.org is being upgraded to version 3.11 and will be offline starting at 2100 UTC. We have allocated an hour for the outage window lasting until 2200 UTC | 19:59 | |
| clarkb | I've updated the emergency.yaml file on bridge as well if anyone wants to double check that | 20:01 |
| clarkb | https://etherpad.opendev.org/p/gerrit-upgrade-3.11 is the plan we'll be following | 20:01 |
| opendevstatus | clarkb: finished sending notice | 20:02 |
| fungi | i see the gerrit, gitea, storyboard and zuul scheduler servers in the emergency disable list | 20:07 |
| clarkb | fungi: and they all look typed correctly? | 20:07 |
| fungi | yes | 20:07 |
| clarkb | excellent | 20:07 |
| clarkb | any preferences on who drives? I think the main consideration is whoever drives should start the root screen so that they can size the terminal for their needs (I'm happy to drive as I've gone through the process on the test node a few times, but I'll make a larger terminal window than fungi will for example) | 20:08 |
| clarkb | going to eat a quick lunch now while we wait for the 2100 timestamp | 20:11 |
| fungi | i'm happy to cut and paste commands from the etherpad | 20:13 |
| fungi | i've started a root screen session on review03 in any case | 20:14 |
| clarkb | thanks I'll join soon. Has it been ^a H'd for logging purposes? | 20:18 |
| clarkb | I did look and zuul nodes seem to have upgraded as expected yesterday. So we should do the manual restart of web andschedulers and that won't interfere with the automated background process | 20:19 |
| fungi | i did not turn on hardcopy, but can | 20:20 |
| fungi | it's logging now | 20:20 |
| clarkb | thanks. It is one of the items noted in the task list | 20:21 |
| clarkb | and I have attached to the screen session now | 20:21 |
| clarkb | note item 7 isn't done yet. Thats another reminder at 2100. But not a big deal I won't forget to send that one | 20:22 |
| fungi | ah, good point, we could have moved #8 before #7 | 20:25 |
| clarkb | ya the order in the etherpad is probably backwards | 20:25 |
| tonyb | did I do the math wrong? | 20:25 |
| clarkb | tonyb: no we are 35 minutes away | 20:26 |
| tonyb | phew! | 20:26 |
| fungi | you're early | 20:26 |
| clarkb | we're just sending notices early (we got feedback once that letting people know ahead of time like this allows them to save comments and fetch changes locally if necessary) | 20:26 |
| clarkb | and then doing the other steps that can be done before we turn anything off | 20:26 |
| tonyb | ah okay | 20:27 |
| tonyb | I'll make another coffee and then I'll be ready | 20:27 |
| fungi | yeah, trying to do as much as we can ahead of the service outage in order to keep it as brief as possible | 20:27 |
| tonyb | sounds good | 20:30 |
| clarkb | I added a few more notes about restarting the zuul services to the etherpad. They weren't there before because my testing didn't have a zuul. But I did this once semi recently to pick up some changes and you basically stop web on one scheduler and scheduler on the other so they can spin their wheels on each server without any service interruption and minimizing impact to each server | 20:30 |
| tonyb | where is the etherpad? | 20:39 |
| clarkb | tonyb: https://etherpad.opendev.org/p/gerrit-upgrade-3.11 | 20:40 |
| tonyb | Thanks I looked at gerrit-3.11-upgrade ;P | 20:40 |
| tonyb | I don't see the zuul pause step corvus mentioned last time | 20:43 |
| tonyb | which I expect we should do before we down the containers | 20:44 |
| fungi | 15 minutes until go time | 20:45 |
| clarkb | oh ya I guess we could do that | 20:45 |
| clarkb | that said we announced this more than a week in advance so I'm not super concerned about it | 20:46 |
| tonyb | doesn't is stop in flight testing from failing if gerrit is down? | 20:46 |
| clarkb | tonyb: yes, it pauses reporting to gerrit so you avoid failed reports (and merges if things are gating) | 20:47 |
| clarkb | the feature didn't exist last time we upgraded gerrit which is why it isn't on the list already | 20:47 |
| clarkb | currently there is one check and one experimental change enqueued in all of zuul from what I can see | 20:48 |
| clarkb | so impact is non zero but also minimal | 20:48 |
| tonyb | Yeah. I added what I think is the correct command to the etherpad | 20:49 |
| tonyb | mostly for "correctness" today | 20:49 |
| clarkb | that looks correct | 20:49 |
| clarkb | tonyb: I added the unpause command on list item 24 | 20:50 |
| tonyb | Thanks. I was going to figure that out next :) | 20:51 |
| clarkb | when I send the status notice at 2100 tonyb do you want to pause zuul then let fungi know when it is safe to stop gerrit and start the process there? | 20:53 |
| tonyb | ack | 20:53 |
| fungi | 4 minutes | 20:56 |
| fungi | 60 seconds | 20:59 |
| clarkb | I'll send the status notice as soon as I see my clock tick over to 2100 | 20:59 |
| tonyb | As it has minimal impact today I've paused zuul | 20:59 |
| clarkb | #status notice Gerrit on review.opendev.org is being upgraded to version 3.11 and will be offline momentarily. We have allocated an hour for the outage window lasting until 2200 UTC | 21:00 |
| opendevstatus | clarkb: sending notice | 21:00 |
| fungi | thanks tonyb | 21:00 |
| -opendevstatus- NOTICE: Gerrit on review.opendev.org is being upgraded to version 3.11 and will be offline momentarily. We have allocated an hour for the outage window lasting until 2200 UTC | 21:00 | |
| clarkb | I see the banner on the zuul status page too | 21:00 |
| fungi | i can in theory down gerrit now, while the notifications are going out | 21:00 |
| clarkb | fungi: yes i think we canstart. We already did the one hour warning too | 21:00 |
| clarkb | we expect this step to take upwards of 5 minutes too | 21:01 |
| fungi | too bad we didn't get the matrix config in place yet, would have been a great functional test | 21:02 |
| fungi | but at least it doesn't seem like we broke it | 21:02 |
| clarkb | ya I'm sure we'll find something new to test the cross platform notices aginst | 21:03 |
| opendevstatus | clarkb: finished sending notice | 21:03 |
| fungi | maybe we'll have it installed by the time i do the project rename maintenance on friday | 21:03 |
| fungi | gerrit is finally down, bringing mariadb back up now and backing it up | 21:04 |
| clarkb | it stopped quicker than the timeout too which is cool | 21:04 |
| clarkb | fungi: give mariadb a few seconds before starting the backup | 21:04 |
| fungi | yeah, 205.7s | 21:04 |
| clarkb | just to be sure it is up and running before we back it up | 21:04 |
| tonyb | Looks like it stopped within the timeout value | 21:04 |
| clarkb | this is probably long enough. | 21:05 |
| fungi | it doesn't seem to have errored, so maybe the second or so i gave it was enough | 21:05 |
| clarkb | fungi: ya I think it backs up the fs first too before the db | 21:05 |
| clarkb | so there is a built in delay anyway as well | 21:05 |
| fungi | the backup exited 0, want to check anything else? | 21:05 |
| clarkb | I pulled up the log file in another terminal and it looks ok to me | 21:06 |
| clarkb | I think we can proceed | 21:06 |
| fungi | 3.9M /var/log/borg-backup-backup02.ca-ymq-1.vexxhost.opendev.org.log | 21:06 |
| tonyb | ++ | 21:06 |
| fungi | oh, i guess that doesn't tell us much without knowing what was in the log before | 21:06 |
| clarkb | the timestamps look correct | 21:06 |
| fungi | terminating with success status, rc 0 | 21:06 |
| clarkb | and db backup returned rc 0. fs backup returned rc 1 which means there were warnings | 21:07 |
| fungi | yeah, log looks like it recorded a successful backup | 21:07 |
| clarkb | usually some file updatse as it goes maybe our screenlog for example | 21:07 |
| fungi | mariadb is down again, backing up configs next | 21:07 |
| fungi | and indices | 21:07 |
| clarkb | this step isn't strictly necessary since indexes don't update but I left it in place ebcause it seems liek good belts and suspenders | 21:08 |
| fungi | takes a moment, doesn't it | 21:10 |
| clarkb | ya :/ I think it may keep older indexes aroudn which we mgiht be able to clean up or exclude from the copy potentially but I think keeping steps like this simple is probably worth the tradeoff | 21:10 |
| clarkb | it appears to be copying the last changes index should be done soon I hope | 21:11 |
| fungi | moving replication tasks aside now | 21:12 |
| fungi | and cleaning up h2 caches | 21:12 |
| fungi | mv: target '/home/gerrit2/review_site/cache/modified_files.h2.db': Not a directory | 21:12 |
| clarkb | fungi: I think you cut off the end of that command | 21:12 |
| fungi | forget a target directory? | 21:13 |
| clarkb | yes its in there on the etherpad | 21:13 |
| fungi | oh! | 21:13 |
| fungi | there we go | 21:13 |
| fungi | very long line, wrapped in the browser and i didn't spot it | 21:13 |
| clarkb | that error should've prevented any copies we don't want going into the proper cache dir right? | 21:13 |
| fungi | correct | 21:13 |
| clarkb | basically it nooped rather than doing anything | 21:13 |
| clarkb | cool just want to make sure we didn't accidentally load up the cache with bad data | 21:14 |
| fungi | it was a command parsing error, so nothing was executed | 21:14 |
| fungi | compose file updates now | 21:14 |
| clarkb | ya timetsamps in the existing cache dir contents look fine | 21:14 |
| fungi | looks correct to me | 21:14 |
| tonyb | and me | 21:15 |
| fungi | ready to pull images next | 21:15 |
| clarkb | ready | 21:15 |
| fungi | underway | 21:15 |
| clarkb | fungi: that inspect command requires some manual var updates | 21:15 |
| fungi | oh, yep | 21:16 |
| clarkb | 127c7 is the image id we want to inspect I think' | 21:16 |
| clarkb | both of those hashes lgtm | 21:17 |
| tonyb | ++ | 21:18 |
| clarkb | I think we can start mariadb give it 20 seconds then do the site init | 21:18 |
| fungi | https://quay.io/repository/opendevorg/gerrit/manifest/sha256:065f7b03859065a2ba2305cab7c18bac77778c3f452e5fd0cc9c92ac11d24fa5 has an unknown entry, what's that? | 21:18 |
| clarkb | fungi: its some artifact of how we build images I think because we're using the multi arch builds universally now | 21:18 |
| fungi | okay | 21:19 |
| clarkb | fungi: the client side ignores that because they don't have a matching arch | 21:19 |
| fungi | also worth noting, that tls deprecation banner still links to a "subscriber only" kb article, there's a jira ticket filed about that courtesy of TheJulia but i haven't seen any activity on it for the month it's been open yet | 21:20 |
| fungi | okay, gerrit init time | 21:20 |
| clarkb | yup lets start mariadb then wait ~20 seconds then do the gerrit init | 21:20 |
| fungi | waiting | 21:21 |
| fungi | it's been about 20 seconds | 21:21 |
| fungi | proceeding | 21:21 |
| clarkb | that looks great just as expected | 21:21 |
| TheJulia | fungi: It got assigned to someone, but also have not seen any activity otherwise. | 21:22 |
| clarkb | I'm ready to start gerrit if you are | 21:22 |
| fungi | lgtm too, time to bring gerrit up? | 21:22 |
| fungi | starting | 21:22 |
| fungi | thanks again TheJulia! | 21:22 |
| fungi | i started tailing the gerrit error_log in a second screen window just to avoid polluting the primary | 21:22 |
| clarkb | [2025-12-07T21:22:31.336Z] [main] INFO com.google.gerrit.pgm.Daemon : Gerrit Code Review 3.11.7-2-g46f2be98bb-dirty ready | 21:22 |
| clarkb | that traceback is a new one for me | 21:23 |
| clarkb | but it also says it is trying to delete a trash file that awsn't there so its probably ok? | 21:23 |
| fungi | i'm surprised that "i wanted to delete a nonexistent file" is an error, but whatevs | 21:24 |
| fungi | are we all set for tonyb to un-pause zuul then? | 21:24 |
| clarkb | the web ui is up for me and I appear to still be logged in | 21:24 |
| clarkb | diffs load too so yes I think we can unpause zuul | 21:24 |
| clarkb | I made a local note of that deletion traceback and can ask upstream about it later | 21:25 |
| clarkb | and no config diff is good means the testing was accurate | 21:25 |
| fungi | the config diff at step #26.2 returns an empty result, yes | 21:25 |
| tonyb | done | 21:26 |
| clarkb | I'm going to propose that bindep acl update now which should test several things for us | 21:26 |
| opendevreview | Clark Boylan proposed openstack/project-config master: Add system-config-core to bindep ACLs https://review.opendev.org/c/openstack/project-config/+/970091 | 21:26 |
| fungi | thanks | 21:26 |
| fungi | `gerrit show-queue -w -q` lists only 17 tasks | 21:27 |
| clarkb | ya that is expected since there is no index update | 21:27 |
| clarkb | it should largely be business as usual on startup here | 21:27 |
| fungi | k, makes sense | 21:27 |
| clarkb | that delete trash files exception occurred against content in All-Users | 21:27 |
| fungi | ready for me to exit the screen session and back up the log? | 21:27 |
| clarkb | does someone want ot log out and log back in just to sanity check that works? | 21:27 |
| fungi | i can | 21:28 |
| tonyb | 970091 looks good to me, enqueued in zuul, replicated to at least one gitea | 21:28 |
| clarkb | fungi: lets hold off on closing screen until we've gone through our short list of checks | 21:28 |
| tonyb | I have logged in since the update and that's fine | 21:28 |
| fungi | i was able to log out and into the webui | 21:28 |
| clarkb | cool thanks for checking | 21:29 |
| clarkb | recheck is the last item on the things to check list that I haven't seen cehcked yet | 21:29 |
| * fungi looks for a candidate | 21:29 | |
| clarkb | https://review.opendev.org/c/openstack/project-config/+/969846 | 21:30 |
| clarkb | I just rechecked thsi one since it only runs one job it is cheap | 21:30 |
| clarkb | and I see it in the zuul status page now | 21:30 |
| fungi | cool, thanks | 21:30 |
| fungi | yeah, i agree it enqueued | 21:31 |
| clarkb | I detached from screen. I think we can probably shut it down and move the log file into its more permanent home | 21:31 |
| clarkb | then we haev a few tasks listed in an order that don't necessarily need to be in that specific order | 21:31 |
| clarkb | specifically the zuul web and scheduler restarts should be able to happen while we do other things | 21:31 |
| fungi | okay, did `mv /root/screenlog.0 /home/gerrit2/tmp/upgrade-3.11` after shutting down the session | 21:32 |
| clarkb | tonyb: do you want to review and approve https://review.opendev.org/c/opendev/system-config/+/968349 ? I can clean up the emergency file list | 21:32 |
| clarkb | once ^ is in and we're happy then we can merge the bindep acl update if we think that is safe enough to test things | 21:33 |
| tonyb | Done. | 21:34 |
| clarkb | oh shoot that is going to take at least an hour to gate | 21:34 |
| clarkb | maybe we don't need to trigger the gitea jobs on these gerrit changes... | 21:34 |
| clarkb | thats fine we have other thigns to do while we wait like restarting zuul services. Does someone else want to do that step? I wrote down the directions on the etherpad basically we do web on one node and scheduler on the other. Wait for both to come back tino the cluster then flip the two around and do the services the other way around | 21:35 |
| clarkb | I can do it too if we prefer | 21:35 |
| clarkb | but I think things are looking happy other than that one unexpected exception | 21:37 |
| clarkb | the expected exception showed up too and since those two there haven't been any others | 21:37 |
| tonyb | I can do the zuul restarts | 21:37 |
| tonyb | The directions make sense, other than not rolling restarting the executors | 21:38 |
| clarkb | tonyb: we only need to restart scheduler and web beacuse they are the only ones that interact with code review systems | 21:38 |
| clarkb | tonyb: and the restart is because zuul asks gerrit for its version number on startup | 21:38 |
| clarkb | (executors don't do that so we can leave them be) | 21:38 |
| tonyb | Ok | 21:39 |
| clarkb | also we don't have to test acl updates today I dont' think. I suspect that if anything goes wrong there we won't be downgrading but will instead be updating acls which we can do with services running. So if we want to end the process at gerrit 3.11 is the image version in config management that is probably good enough? Thought seeing manage-projects run regardless is always a good | 21:39 |
| clarkb | thing | 21:39 |
| fungi | seems sufficient to m | 21:41 |
| fungi | w | 21:41 |
| fungi | me | 21:41 |
| clarkb | but also the downgrade here is relatively cheap as they go since there is no reindexing | 21:41 |
| fungi | i'll be around fairly early tomorrow in case something has gone terribly wrong, have a morning meeting to be up for anyway | 21:42 |
| clarkb | so if we do find need to downgrade later it is slightly less painful than usual | 21:42 |
| fungi | yeah | 21:42 |
| clarkb | ok 968349 will trigger manage-projects but as a noop | 21:44 |
| clarkb | I think if that comes back happy then we can worry about testing the positive case tomorrow | 21:44 |
| fungi | wfm | 21:44 |
| clarkb | (a noop because that change doesn't include acl updates and jeepyb should do very little as a result) | 21:44 |
| fungi | we don't make acl changes that often | 21:44 |
| clarkb | tonyb: looks like the first half of the zuul restarting has completed based on the components list | 21:46 |
| clarkb | oh line 300 has a post upgrade task of checking if Blocked Users were added to All-Projects acls | 21:48 |
| clarkb | again thats mostly so we can update our documentation if it did and not anything urgent so I'm happy to defer that but want to call it out as a task that could be done now | 21:49 |
| clarkb | there is a certain part of me that is trying to optimize the work done today for the fact that it is the weekend :) | 21:50 |
| tonyb | Yup, sorry it took longer than expected and I got distracted | 21:50 |
| clarkb | tonyb: it is a lot faster today than in the past. It took about 20 mintues before but takes about 5 now | 21:50 |
| tonyb | Noted | 21:52 |
| clarkb | I was leaving things in the emergecny file for now since the hourlies are about to start and the chagne we want to run jobs is some time away from merging | 21:54 |
| clarkb | but thinking about it further we don't run jobs in hourly that should affect any of these nodes really (zuul is the only ones but zuul should be ok) | 21:54 |
| clarkb | any concern with me removing the nodes from the file now? | 21:54 |
| fungi | yeah, starting 5 minutes out | 21:54 |
| fungi | should be safe | 21:54 |
| fungi | would be good to see them run before i call it done for me | 21:55 |
| clarkb | yup same here | 21:55 |
| clarkb | emergency file is cleaned up | 21:55 |
| fungi | thanks! | 21:55 |
| tonyb | zuul-web on zuul02 is that last thing I'm waiting on | 21:56 |
| clarkb | new exception but I think thsi is a common one: Connection reset by peer | 21:56 |
| tonyb | and done | 21:57 |
| clarkb | basically crawlers grab git repo archive files then time out before they can be put together and close the connection then gerrit complains | 21:57 |
| clarkb | I've seen it before on older gerrit so not concerned about ti being related to the upgrade | 21:57 |
| clarkb | cool I think we're just waiting on that chagne to merge and trigger manage-projects and infra-prod-service-review. Both should noop for us | 21:58 |
| clarkb | heh jetty has a routine called eat what you kill | 22:02 |
| opendevreview | Merged opendev/system-config master: Bump Gerrit container image to 3.11 https://review.opendev.org/c/opendev/system-config/+/968349 | 22:07 |
| clarkb | that was much quicker than expected jobs probably ran on raxflex | 22:07 |
| clarkb | and the two jobs we wnt to see have queued up | 22:07 |
| clarkb | theyare waiting for hourlies to finish | 22:08 |
| clarkb | Dec 7 21:14 docker-compose.yaml <- in theory that modtime won't change | 22:08 |
| clarkb | ok service-review finished successfull and that mod time did not change so that looks good | 22:12 |
| clarkb | manage-projects is running now | 22:12 |
| clarkb | of course as that is happening I realize we never updated gerritlib's integration testing to gerrit 3.11... | 22:12 |
| clarkb | I'll make a note of that | 22:12 |
| fungi | oh, yep | 22:12 |
| clarkb | I think we use the upstream images in that job so we could theoretically test 3.11 3.12 and 3.13 | 22:14 |
| clarkb | whcih is a nice way of getting ahead of it and not even needing to forget for the next couple of upgrades | 22:14 |
| fungi | sgtm | 22:14 |
| fungi | i guess we want to try to power through 3.12 to 3.13 early in 2026 | 22:15 |
| clarkb | yes, though we have to get on java 21 with 3.11 first | 22:15 |
| fungi | aha, yeah that's the next step | 22:15 |
| clarkb | 3.11 supports java 17 and 21 so is the transition release. We're running with 17 right now | 22:16 |
| clarkb | 3.12 only supports java 21 | 22:16 |
| clarkb | manage-projects job reports success and in the log file I see a whole lot of skipping | 22:18 |
| clarkb | if it looks good to everyone else I think we can call this done for now then pick up the post upgrade tasks tomorrow including testing an actual acl update | 22:18 |
| clarkb | but also feel free to disagree with me on that point and point out anything else you feel is necessary to check or test first | 22:19 |
| fungi | and looks like the gitea task completed a couple of minutes ago | 22:19 |
| clarkb | fungi: gitea task? | 22:19 |
| fungi | er, i guess it's gerrit and gitea, because it runs the plan on review03.opendev.org as well | 22:20 |
| fungi | s/plan/play./ | 22:20 |
| fungi | PLAY RECAP includes review03.opendev.org : ok=4 changed=2 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0 | 22:20 |
| clarkb | oh yes manage-projects runs against gitea first to update the repos there so that when we then update gerrit and potentially replicate from gerrit to gitea the gitea info is up to date and ready to receive updates | 22:20 |
| fungi | once upon a time, the manage-projects process wrote to a log on the gerrit server, but i'm not seeing where that is these days | 22:22 |
| clarkb | fungi: it writes it on bridge now in /var/log/ansible/manage-projects.yaml.log which is then also included in the infra-prod-manage-projects job logs (its the only job where we include the log because we're confident we don't leak things) | 22:23 |
| clarkb | fungi: basically we stopped having it write to local disk and instead emit to stdout so ansible captures it | 22:23 |
| clarkb | I don't remember why we decided that was desireable. Maybe it happened as part of switching it to an ansible job run on demand | 22:23 |
| fungi | ah okay, so everything that the manage-projects script used to log on the gerrit server is just captured by ansible now | 22:23 |
| clarkb | yes should be | 22:23 |
| fungi | anyway, i don't see anything amiss there | 22:24 |
| clarkb | great. tonyb any concerns with considering this done for now? | 22:25 |
| clarkb | I did update the post upgrade tasks list which yall might want to check to see if we feel any of them need to be done now rather than tomorrow/later | 22:25 |
| clarkb | (that list is on the etherpad) | 22:25 |
| fungi | aty line 297 | 22:26 |
| clarkb | I'm hearing no objections. I'm probably go to go and not look at a computer screen for a bit but can check in before dinner | 22:30 |
| fungi | should be fine, i'm still looking through it but i don't expect any concerns | 22:30 |
| clarkb | fungi: ack | 22:31 |
| clarkb | thank you for the help! | 22:31 |
| fungi | my pleasure! | 22:33 |
| fungi | yeah, i don't see anything urgent in the post upgrade tasks list, and i'm +2 on all the changes linked there so fart | 22:34 |
| fungi | so far | 22:34 |
Generated by irclog2html.py 4.0.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!