| @gtema:matrix.org | gerrit seems to be down, at least unreachable to me. 1h ago was there | 16:53 |
|---|---|---|
| @clarkb:matrix.org | I'm starting to look | 16:56 |
| @mnaser:matrix.org | I was just gonna check if the VM oom somehow.. | 16:56 |
| @clarkb:matrix.org | `review03.opendev.org | SHUTOFF` | 16:56 |
| @clarkb:matrix.org | mnaser: cool should I hold off on starting it while you do that? | 16:57 |
| @mnaser:matrix.org | let me check real quick | 16:57 |
| @mnaser:matrix.org | in case its not oom (i doubt it) and if it is ill start it back up and we'll try to figure out a longer term story | 16:58 |
| @clarkb:matrix.org | makes sense. Just let me know if/when you're ready for me to start the server then work on things on the service side | 17:00 |
| @mnaser:matrix.org | ok we need to probabaly lookdeeper into this because the vm died with 121gb rss on the host | 17:05 |
| @mnaser:matrix.org | Clark: i think you're good to start it up and it'll maybe have to be a monday thing to see why it ate up so much memory on the host side | 17:05 |
| @clarkb:matrix.org | mnaser: ok I'm starting it now | 17:06 |
| @clarkb:matrix.org | thank you for looking into it on the hypervisor side | 17:06 |
| @mnaser:matrix.org | np | 17:06 |
| @clarkb:matrix.org | the server is up, apache is running but gerrit isn't (this is expected). I'm going to take the opportunity to make clear out some of the ever growing gerrit cache h2 db files so will be a few more minutes before I actually start the gerrit service | 17:07 |
| @clarkb:matrix.org | `[2026-04-25T17:11:54.169Z] [main] INFO com.google.gerrit.pgm.Daemon : Gerrit Code Review 3.12.6-dirty ready` | 17:12 |
| @clarkb:matrix.org | I am able to reach the web ui now too | 17:12 |
| @clarkb:matrix.org | and diffs load | 17:13 |
| @gtema:matrix.org | git review worked as well, thanks a lot | 17:14 |
| @clarkb:matrix.org | loading diffs produced this in the log `org.h2.jdbc.JdbcBatchUpdateException: Timeout trying to lock table "DATA"; SQL statement:` whcih we saw last time too and it didn't reoccur after the initial startup. Last time I theorized it was lock contention with some cache cleanup routines | 17:14 |
| @clarkb:matrix.org | viewing more fiel diffs does not seem to produce those warnings. I think that gives more weight to the theory it is startup contention that goes away as startup routines complete | 17:15 |
| @clarkb:matrix.org | https://review.opendev.org/c/openstack/codegenerator/+/985819 this appears to be the change gtema pushed to and I see the new patchset on gitea https://opendev.org/openstack/codegenerator/commit/069eaec0099c13a965e7331bf3d3932771c7db43 so I think replication is happy | 17:16 |
| @clarkb:matrix.org | infra-root once the server was back up I moved the replication waiting queue and the large h2 v2 cache db files to /home/gerrit2/tmp/unexpected_stop_20260425/ I didn't simply delete the cache files as this is the first time we've done this with h2 v2 and I figured having old copies may be useful. It looks like moving them aside is just fine like it was with h2 v1 (there was talk on the gerrit mailing list about this a couple months ago so I was like 98% confident it was fine) | 17:21 |
| @clarkb:matrix.org | then the only other thing of note is the timeout trying to get lock warnings which seem to be due to contention during startup activity and user activitiy (like viewing file diffs) | 17:22 |
| @clarkb:matrix.org | Those seem to have stopped after a minute or so | 17:22 |
| @clarkb:matrix.org | #status log Started review03 after it was found in a SHUTOFF state. Then moved replication waiting queue and cleaned up h2 caches before starting Gerrit containers again. | 17:24 |
| @status:opendev.org | @clarkb:matrix.org: finished logging | 17:24 |
| @clarkb:matrix.org | For some additional testing I left myself a comment on https://review.opendev.org/c/opendev/system-config/+/985834 to indicates that gitea 1.26.1 exists and we should skip the .0 release | 17:27 |
Generated by irclog2html.py 4.1.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!