Friday, 2026-05-29

« Thursday, 2026-05-28 Index Saturday, 2026-05-30 »
-@gerrit:opendev.org- OpenStack Proposal Bot proposed: [openstack/project-config] 990586: Normalize projects.yaml https://review.opendev.org/c/openstack/project-config/+/99058602:25
@fungicide:matrix.orgcatching up from last night's discussion, happy to help with a gerrit restart but i'm planning to disappear for lunch after my morning meetings wrap up, expect i'll be gone between 15:00 and 16:50 utc12:50
@fungicide:matrix.organd yeah, i expect it'll all just magic itself12:50
-@gerrit:opendev.org- ayyappa proposed: [openstack/project-config] 990651: Add repo app-ejbca for starlingx https://review.opendev.org/c/openstack/project-config/+/99065113:43
@mordred:waterwanders.comfungi: it would be really neat to magically understand what happened :)13:46
@jim:acmegating.comi may not be around all day, but i can help get things started before Clark shows up, so i'll approve the gerrit build change13:51
@fungicide:matrix.orgthanks. we might be able to squeeze a restart in before i leave for lunch, even13:53
@fungicide:matrix.orgturns out my 14-15z meeting is cancelled this week so i won't be as distracted13:53
-@gerrit:opendev.org- ayyappa proposed: [openstack/project-config] 990651: Add repo app-ejbca for starlingx https://review.opendev.org/c/openstack/project-config/+/99065113:54
-@gerrit:opendev.org- ayyappa proposed: [openstack/project-config] 990651: Add repo app-ejbca for starlingx https://review.opendev.org/c/openstack/project-config/+/99065114:06
@clarkb:matrix.orgYa I've got a school run to do in a few minutes but then I should be back14:20
@fungicide:matrix.orgzuul's estimating another half hour to merge anyway14:21
@fungicide:matrix.organd deploy could very well end up losing the race against hourly jobs14:21
@fungicide:matrix.orgoh, i was looking at elapsed time, it hasn't even started running system-config-run-review-3.13 yet14:30
@fungicide:matrix.orgokay, zuul does think it still could merge before the top of the hour14:40
-@gerrit:opendev.org- Zuul merged on behalf of Clark Boylan: [opendev/system-config] 990570: Build gerrit replication plugin using stable branches https://review.opendev.org/c/opendev/system-config/+/99057015:00
@clarkb:matrix.orgNow to see if deploy or hourly wins the enqueue race15:01
@fungicide:matrix.orgmerged about 40 seconds too late i think, but they're all waiting behind a reconfigure event15:01
@jim:acmegating.comaww, missed the top of the hour by 42 seconds :)15:01
@clarkb:matrix.orgWe have some variance in the hourly pipeline so deploy can still win iirc15:02
@fungicide:matrix.orgyeah, unblocked now but they did indeed get scheduled in chronological order15:02
@clarkb:matrix.orgHourly for the semaphore I think15:02
@clarkb:matrix.org* Hourly got the semaphore I think15:02
@fungicide:matrix.orgright, but at least the promotes are able to run in parallel with that15:02
@fungicide:matrix.organd i'm overdue to step away for my lunch appointment, will check on things in ~1.5 hours when i get back, or help with the restart at that point if it hasn't happened yet15:04
@clarkb:matrix.orghttps://quay.io/repository/opendevorg/gerrit/manifest/sha256:736d3f4e5d6260fc334526fe1c3ca629d6db8d9bbd09345f005f47ae708235c2 is the new image15:06
@clarkb:matrix.orghow does this look for when we're ready to actually restart things: #status notice Gerrit will be restarted to pick up a bugfix in the replication plugin. You may notice a short outage of a few minutes.15:06
@clarkb:matrix.orgf4812a7aaf95 appears to be the image we are running with right now15:08
@clarkb:matrix.org`docker-compose -f /etc/gerrit-compose/docker-compose.yaml down && mv ~gerrit2/review_site/data/replication/ref-updates/waiting ~gerrit2/tmp/waiting_queue_2026-05-29 && rm ~gerrit2/review_site/cache/{gerrit_file_diff,git_file_diff,git_modified_files,modified_files,comment_context}-v2.* && docker-compose -f /etc/gerrit-compose/docker-compose.yaml up -d` This is the command I'll plan to run to do the actual restart15:10
@clarkb:matrix.orgI'll start a root screen on review03 and pull the image now15:10
@clarkb:matrix.orgI've pulled the image and checked it against the quay.io url above. That lgtm15:13
@clarkb:matrix.orginfra-root should I proceed with sending that status notice and then run the restart command or would we prefer to wait for fungi to return?15:13
@jim:acmegating.comClark: i'm around for the restart if you want to go for it15:30
@clarkb:matrix.orgOk I will get started momentarily 15:31
@clarkb:matrix.org#status notice Gerrit will be restarted to pick up a bugfix in the replication plugin. You may notice a short outage of a few minutes.15:31
@status:opendev.org@clarkb:matrix.org: sending notice15:31
@clarkb:matrix.orgonce that is complete I will run the docker compose down etc in the command above (it is also commented out in the screen for the moment)15:32
@jim:acmegating.comis that command documented somewhere?15:33
@clarkb:matrix.orgcorvus: probably not :( the normal docker compose down and up will work. This just optimizes a few things. But it is a good comment that this should be documented15:33
@jim:acmegating.comi'll work on that, mostly to see if i'm missing anything :)15:33
@clarkb:matrix.orgthe waiting queue cleanup avoids a few thousand tracebacks on startup that make things look bad (and hard to see real issues with all that spam) and the cache file cleanups helps contain their growth15:34
@clarkb:matrix.orgsince those files don't really shrink taking advantage of the downtime to remove them and restart from 0 is a nice thing to do15:34
-@status:opendev.org- NOTICE: Gerrit will be restarted to pick up a bugfix in the replication plugin. You may notice a short outage of a few minutes.15:34
@status:opendev.org@clarkb:matrix.org: finished sending notice15:34
@clarkb:matrix.orgok I will proceed with the restart now15:34
@clarkb:matrix.orgstopping gerrit takes a few minutes typically. like 3-5 I think15:35
@clarkb:matrix.org`[2026-05-29T15:37:23.245Z] [main] INFO  com.google.gerrit.pgm.Daemon : Gerrit Code Review 3.12.7-dirty ready`15:37
@clarkb:matrix.orgI can view diffs like https://review.opendev.org/c/opendev/system-config/+/990570/1/zuul.d/docker-images/gerrit.yaml15:37
@jim:acmegating.comyep lgtm15:38
@clarkb:matrix.orgI'm going to check show-queue and if that looks settled i'll issue a replicate start command for something like bindep15:39
@jim:acmegating.comyeah i was just looking at the queues15:39
@clarkb:matrix.orgthey look pretty empty (which is good)15:40
@jim:acmegating.comyeah, i think it's just doing current events maybe15:40
@jim:acmegating.comyep, mostly idle with the occasional replication for a new change15:40
@clarkb:matrix.orgyup and then it lists the diskcachepruner things which I believe are scheduled for the future15:41
@clarkb:matrix.orgthey just end up on the list when you start the service15:41
@clarkb:matrix.orgI issued `replication start opendev/bindep` via ssh and my ssh command exited 015:42
@clarkb:matrix.orgI believe that jeepyb was seeing a non zero exit with the error message which generated the traceback15:42
-@gerrit:opendev.org- James E. Blair https://matrix.to/#/@jim:acmegating.com proposed: [opendev/system-config] 990679: Document current gerrit restart process https://review.opendev.org/c/opendev/system-config/+/99067915:42
@jim:acmegating.comshould we see general replication tasks in the queue soon?15:43
@jim:acmegating.comoh you just did bindep sorry15:43
@jim:acmegating.comi had in my mind you were going to replicate all; nvm.  i don't think that's necessary, i think testing one repo is fine.15:43
@jim:acmegating.comis it the case that the current running tasks are not in the queue?15:44
@jim:acmegating.com(like if we see a task at the top like replicate to gitea09 does that mean that we're currently running replicate to giteo08, or that we're currently running replicate to gitea09)15:46
@clarkb:matrix.orgcorvus: I saw some the first time I ran show queue15:43
@clarkb:matrix.orgthere are some now `51840bab              15:41:34.354      [f16bbfc4] push ssh://git@gitea09.opendev.org:222/openstack/nova.git [refs/changes/41/981141/meta]`15:44
@clarkb:matrix.orgso I think automatic replication si working but I'll try to find positive confirmation of this via a gerrit change update that got replicated15:44
@clarkb:matrix.orghttps://opendev.org/opendev/system-config/commit/2d71e37b6684b3305740a6b4d4781e619117c762 your change appears to have replicated15:45
@clarkb:matrix.orgcorvus: I think if the state column of the task is empty then it is currently executing15:47
@clarkb:matrix.orgcorvus: otherwise things that are queued up/waiting for threads or just schedueld for another time have data within the state column15:47
@jim:acmegating.comyeah -- show-queue and ps show the same tasks15:48
@clarkb:matrix.orgone of the things we also like to do after a restart is trigger reindexing of changes15:48
@jim:acmegating.comso currently running tasks *are* in show-queue15:48
@clarkb:matrix.orgto avoid the duplicate change problem15:48
@clarkb:matrix.organd that will enqueue a lot of stuff and create queues as well as tasks in progress which may make this easier to see15:48
@clarkb:matrix.orgshould I go ahead and issue a reindex of changes now?15:48
@jim:acmegating.comClark: before doing that, is there anything we want to do with drizzle?15:49
@jim:acmegating.commy understanding is that we want to re-push into gerrit, so i think the answer to my question is no15:49
@jim:acmegating.combut i just wanted to check before we fill queues15:49
@clarkb:matrix.orgdrizzle shouldn't have any changes yet right? So I don't think we need to figure out drizzle before reindexing existing changes15:49
@mordred:waterwanders.comI *think* based on yesterday's investigation that a manual clone/push (probably doing it the jeepyb way) would be a thing to try. but yeah - agree, doesn't need to be sorted before reindexing15:50
@clarkb:matrix.organd ya I think the next step with drizzle is to repush it. But Let me double check that it didn't magically get refs created after the resetart15:50
@clarkb:matrix.orgya drizzle.git/refs/heads is still an empty dir and drizzle.git/packed-refs only has tags and refs/meta/config in it15:51
@jim:acmegating.comreindex sgtm15:51
@clarkb:matrix.orgso I think we're still ni the same state as before there with four missing branches and issuing a reindex of changes should be fine15:51
@clarkb:matrix.orgack I'll trigger that now15:51
@jim:acmegating.comClark: what's the command you're using for that?  i'll add it to the docs15:53
@clarkb:matrix.orgcorvus: `gerrit index start changes --force` via ssh on port 2941815:53
@mordred:waterwanders.comI swear I did a local reproduction of the jeepyb steps to check the fsck and the heads - and I cannot find the dir I did it in :)15:53
@clarkb:matrix.orgthe force is required beacuse the index version doesn't change. If gerrit sees the index version is the same it won't reindex by default15:53
@clarkb:matrix.organd show-queue -w has lots of entries now15:53
@clarkb:matrix.orgcorvus: and you can see state is `waiting ...` for things that are queued15:54
-@gerrit:opendev.org- James E. Blair https://matrix.to/#/@jim:acmegating.com proposed: [opendev/system-config] 990679: Document current gerrit restart process https://review.opendev.org/c/opendev/system-config/+/99067915:55
@clarkb:matrix.orgthis typically takes about half an hour or so to reindex15:56
@clarkb:matrix.orgnot terribly quick but much quicker than it used to be15:56
@clarkb:matrix.orgthat should finish around when fungi returns. Did we want to proceed with pushing drizzle content then?15:57
@clarkb:matrix.orgI'll leave the screen up for now too in case we want to use that for the push (though maybe someone will do it locally?)15:57
@jim:acmegating.comyeah, no need to wait on the drizzle stuff16:00
@clarkb:matrix.orgoh fungi when you get back can you rerun the command that failed for you when doing replication start? Just to be sure we're using a known reproducer? I did `replication start opendev/bindep` which I think is sufficient, but figured we should double check with a known reproducer which you had iirc16:01
@jim:acmegating.comhttps://paste.opendev.org/show/bEdKVOKKuw6L6BRNCJw9/16:02
@jim:acmegating.comwe're just going to manually run those commands right?16:03
@mordred:waterwanders.comyour way was easier than mine: https://paste.opendev.org/show/bRGwYC6v96VRWKkLnQbY/ 16:03
@mordred:waterwanders.com(but yes, I think those are the right reproducing sequence)16:03
@jim:acmegating.comthose look very similar except one swap in order16:03
@mordred:waterwanders.commight not be a bad idea to do a git for-each-ref refs/copy/ after the fsck 16:04
@clarkb:matrix.orgI think we can try to have manage-projects do it again by running manage-projects (I think there may even be a flag to do one project instead of all of them). But the downside to that is we won't be able to control each step and debug as we go so I do think manually running the commands manage-projects would run may be useful16:04
@clarkb:matrix.orgmordred: ++ exactly that we would miss that if we ran manage-projects rather than the individual commands16:04
@jim:acmegating.comi'm happy to run the commands; i can get started on that16:04
@jim:acmegating.comi have added myself to bootstrappers and am starting the process now16:06
@jim:acmegating.com```16:08
corvus@fuligin /t/d/drizzle (main)> git for-each-ref refs/copy/
269bf1ede95c17736f45d7fa7cdc2172a677892a commitrefs/copy/heads/drizzle-7.0
3873683eeb4cf650f9cdceece80eca10ca888f23 commitrefs/copy/heads/drizzle-7.1
b4de3d25e752315ef7fd020efb3f14ef62bc151d commitrefs/copy/heads/drizzle-7.2
51e535c15ba38fb5c09bd101a374bba14d7c3487 commitrefs/copy/heads/main
```
@jim:acmegating.comthat's everything before the push; ready to push now?16:08
@clarkb:matrix.orgok so fsck didn't delete the refs16:08
@clarkb:matrix.org(but it didn't for me or mordred either)16:08
@clarkb:matrix.orgmordred: ^ you probably want to double check those refs? But it looks good from my perspective without checking the specific refs16:08
@mordred:waterwanders.comdarn. I was hoping that would magically break and be the smoking cun.16:09
@mordred:waterwanders.comyeah - that looks right. main definitely is16:09
@clarkb:matrix.orgreindexing is nearing the halfway point16:09
@jim:acmegating.comremote: You need 'Create' rights to create new references.16:10
@mordred:waterwanders.com*wow* - and you're in project bootstrappers!16:10
@clarkb:matrix.orgcorvus: add yourself to admins too I think16:10
@jim:acmegating.comoh i uh16:10
@jim:acmegating.commay not have added corvus to bootstrappers16:10
@clarkb:matrix.orgbut maybe this was the issue if bootstrappers is not in admins?16:10
@jim:acmegating.comi think i added corvus.admin :)16:10
@clarkb:matrix.orgah that would do it :)16:10
@mordred:waterwanders.com:)16:11
@jim:acmegating.comyeah because that's what the instructions say :)16:11
@jim:acmegating.comhttps://docs.opendev.org/opendev/system-config/latest/sysadmin.html#force-merging-a-change16:11
@jim:acmegating.comanyway, i'll just push as corvus.admin :)16:11
@jim:acmegating.comokay... so... there were more warning messages than my very large scrollback can handle, so i lost the top of this.  but here's a sample of them, and then the bottom: https://paste.opendev.org/show/bkKN2EGM46kcLL7Ulce0/16:12
@mordred:waterwanders.comhuh. skip-validation16:13
@mordred:waterwanders.comso - like - it's trying to run commit validation16:13
@jim:acmegating.comi can try again and tee it to a logfile if we want to see stuff at the top16:13
@jim:acmegating.combut yeah, that validation message is completely new to me i don't know whats up16:13
@mordred:waterwanders.commaybe that's also a new gerrit behavior and it's just been a while since a new project imported preexisting state?16:14
@clarkb:matrix.orgif this is the issue jeepyb hit then A) drizzle's history was a stress test after all :) and B) its weird that we didn't log the error in the manage-project log. But also we're learning things16:14
@jim:acmegating.comor the pre-existing state was smaller?16:14
@clarkb:matrix.orgmordred: a project of this size in particular. It is complaining about the total number of commits I think16:14
@mordred:waterwanders.comapparently you can add "-o skip-validation" to the git push commabnd16:14
@mordred:waterwanders.comwhich we probably want to do in manage-projects for completeness16:14
@clarkb:matrix.orghttps://gerrit-review.googlesource.com/Documentation/user-upload.html#skip_validation ya16:14
@clarkb:matrix.orghttps://gerrit-review.googlesource.com/Documentation/error-too-many-commits.html16:15
@jim:acmegating.com(i ran it again and captured the whole log; nothing interesting at the top.  the log file was 27987 lines, most of those commit message warnings)16:16
@mordred:waterwanders.comI don't fully understand why that didn't cause m-p to error out16:16
@clarkb:matrix.orgso that 10000 limit is a default config value for the server. We could change it, but also having a sanity check seems fine and then have jeepyb's import process override it via the option16:16
@clarkb:matrix.orgcorvus: is it exiting 0?16:16
@jim:acmegating.comexit 116:16
@clarkb:matrix.orghuh then ya really weird this didn't generate an error in jeepyb. Maybe we don't properly check errors when pushing the code but do when we replicate?16:17
-@gerrit:opendev.org- Monty Taylor https://matrix.to/#/@mordred:inaugust.com proposed: [opendev/jeepyb] 990685: Add skip-validation when importing from upstream https://review.opendev.org/c/opendev/jeepyb/+/99068516:18
@jim:acmegating.comready for me to `git push -o skip-validation ssh://corvus.admin@review.opendev.org:29418/drizzle/drizzle.git '+refs/copy/heads/*:refs/heads/*'`16:18
@jim:acmegating.comoh mordred you put it on the end in jeepyb, i put it at the start16:19
@jim:acmegating.comwhich one do we want to do? :)16:19
@mordred:waterwanders.comwe should be running: "    log.debug("Return code: %s" % p.returncode)16:19
log.debug("Command said: %s" % out.strip())" in git_command
@mordred:waterwanders.comso I'd have expected a logged non-zero return code at least16:19
@clarkb:matrix.orgcorvus: the documented version is at the start16:19
@mordred:waterwanders.comcorvus: oh, I'll do it at the start16:19
@clarkb:matrix.orgmordred: I don't know that we log at a debug level16:19
-@gerrit:opendev.org- Monty Taylor https://matrix.to/#/@mordred:inaugust.com proposed: [opendev/jeepyb] 990685: Add skip-validation when importing from upstream https://review.opendev.org/c/opendev/jeepyb/+/99068516:20
@jim:acmegating.comall of that output, including the reject lines, went to stderr16:20
@jim:acmegating.comnothing to stdout16:20
@jim:acmegating.comnot sure if jeepyb merges them or not16:20
@clarkb:matrix.organd then as far as pushing it goes. I think we all knew this was a project with a big history and that was why I asked some questions before we landed the chagne to import it. I don't think the risks around that have changed. Gerrit is just checking more aggressively than we expected. So ya I think we can proceed with pushing16:20
@mordred:waterwanders.comClark: good point. and yeah, looks like we're ignoring exit codes16:20
@clarkb:matrix.orgmordred: aha ok mystery probably solved then16:21
@mordred:waterwanders.comwe invoke run_command in a form that returns the error code, but then at the callsite don't do anything about it, only trapping for exceptions16:22
@mordred:waterwanders.comso it looks like the callsite might have expected the git_command to throw on error, but that's not the model :)16:22
@clarkb:matrix.orgat the very least we should probably log at an error level non zero return codes16:22
@mordred:waterwanders.comyeah16:23
@jim:acmegating.com ! [remote rejected]     refs/copy/heads/main -> main (prohibited by Gerrit: not permitted: skip validation)16:23
@clarkb:matrix.orgI don't know that I want to do massive manage-projects refactoring just due to how difficult it is to test and be confident in. But some extra logging would've helped tremendously here I think16:23
@jim:acmegating.comi feel like i'm at the dmv16:24
@mordred:waterwanders.com"Forge Author16:24
Forge Committer
Forge Server
Push Merge Commits
"
@mordred:waterwanders.comgah16:24
@mordred:waterwanders.com* ```16:25
Using the skip-validation option requires the user to have a specific set of permissions, in addition to those permissions already required to bypass review:
Forge Author
Forge Committer
Forge Server
Push Merge Commits
```
@clarkb:matrix.orgI'm guessing forge server is the missing permission for bootstrappers16:25
@clarkb:matrix.orgthat list is documented at https://gerrit-review.googlesource.com/Documentation/user-upload.html#skip_validation16:25
@mordred:waterwanders.comcorvus: I also feel like you're at the dmv16:26
@jim:acmegating.comi'm going to add myself to administrators so that i can click the gui for access controls16:26
@clarkb:matrix.orgya looking at our documented all projects acls I see registered users has forge author, bootstrappers has forge commiter, bootstrappers has push merge.16:27
@clarkb:matrix.orgI don't see a forge server acl16:27
@jim:acmegating.comit's there in the gui16:28
@clarkb:matrix.orghttps://gerrit-review.googlesource.com/Documentation/access-control.html#category_forge_server documents what that means16:28
@clarkb:matrix.orgI'm not sure I fully understand it, but I guess the idea is that the check in there is one of the validations. So if you're skipping vlidations then you need the permission to forge the data and bypass validation?16:28
@clarkb:matrix.orgmordred: this repo has no gerrit notedb history in it right?16:28
@clarkb:matrix.org(we aren't replicating refs/changes/ so that shouldn't actually matter)16:29
@mordred:waterwanders.comit does not16:29
-@gerrit:opendev.org- Monty Taylor https://matrix.to/#/@mordred:inaugust.com proposed: [opendev/jeepyb] 990689: Log git errors if they happen https://review.opendev.org/c/opendev/jeepyb/+/99068916:29
@clarkb:matrix.orgso ya I think this is a non issue for this repo. It would potentially come up if importing somethign that was filter branched and hosted ing errit itself? But even then I suppose we would want imports to just work?16:30
@clarkb:matrix.orgcorvus: when you say it is there in the gui you mean boostrappers has the permission or that we have the ability to add it in the gui for eg bootstrappers?16:30
@jim:acmegating.comability to add16:30
@mordred:waterwanders.comyeah. I think when we say "please import this" we mean it16:31
@clarkb:matrix.orgack I think I'm good with adding it. But fungi should be back in a few minutes and it may be worth having his opinion on that too?16:31
@jim:acmegating.comis there a way we can tell gerrit to perform the validation on more commits?16:31
@clarkb:matrix.orgyes there is a server wide config setting: https://gerrit-review.googlesource.com/Documentation/config-gerrit.html#receive.maxBatchCommits16:32
@jim:acmegating.commaybe we could increase that; we carefully control who can bypass review (typically just mp)16:33
@clarkb:matrix.orgya that would be another option. We would have to restart gerrit again to pick that up16:33
@clarkb:matrix.organy idea how many commits are in each of those branches? It sounded like maybe on the order of 27k?16:33
@jim:acmegating.com22100 is the number of commits in main16:34
@clarkb:matrix.orgmaybe we bump that up to a round 32768?16:34
@clarkb:matrix.orgreindexing completed with the expected 3 errors16:34
@mordred:waterwanders.comI think it's kind of fascinating that there's a number at all.16:34
@jim:acmegating.commordred: yeah; i'm assuming it's something about how gerrit works with jgit and hooks into the git protocol16:35
@mordred:waterwanders.comyah16:35
@clarkb:matrix.orgit is probably expensive to do the checks16:35
@clarkb:matrix.organd so they want to limit how much damage you can do to system performance. But if we do this once in a while and not regularly I don't expect it to be a major issue16:36
@fungicide:matrix.orgokay, back now16:36
@fungicide:matrix.orgwow that's a lot of discussion16:36
@jim:acmegating.comyeah.  i just pushed again (with the 10k limit), and gerrit uses a couple extra cpus during the push.  doesn't affect the memory usage.  whole thing takes 12.5 seconds.16:37
@clarkb:matrix.orgfungi: tl;dr is we think replication start commands are working again (you should probably double check since you got it to fail yseterday). THen corvus manually ran the manage-project push commands and found the failure source. It is related to https://gerrit-review.googlesource.com/Documentation/user-upload.html#skip_validation there is a limit of 10000 commits to push at once by default and we hit that limit16:37
@jim:acmegating.comso maybe if we bump this to 32k, the result is that gerrit consumes 2 cpus for 30 seconds.  definitely notable, but not going to kill us for something that almost never happens.16:38
@clarkb:matrix.orgcorvus: agreed16:38
@mordred:waterwanders.comand ... project bootstrappers doesn't have the ACL to bypass the validation manually, is missing a setting16:38
-@gerrit:opendev.org- Monty Taylor https://matrix.to/#/@mordred:inaugust.com proposed: [opendev/jeepyb] 990689: Log git errors if they happen https://review.opendev.org/c/opendev/jeepyb/+/99068916:39
@clarkb:matrix.orgspecifically this one https://gerrit-review.googlesource.com/Documentation/access-control.html#category_forge_server We can change the limit via https://gerrit-review.googlesource.com/Documentation/config-gerrit.html#receive.maxBatchCommits16:39
@mordred:waterwanders.comspeaking of - I split that from the validation patch. they're not related ^^16:39
@jim:acmegating.comi'm okay with either approach, slight preference for increasing the limit instead of skipping validation, but like, very very slight.16:39
@jim:acmegating.comafk for 5 min while fungi catches up16:40
@clarkb:matrix.orgI'm good with that too. But also I think forge server is probably also safe16:40
@clarkb:matrix.orgso ya let's see what if any input fungi has16:40
@mordred:waterwanders.comI kind of think both. add forge-server so that project-bootstrappers has the oomph when an admin uses it to evelvate permissions, but then instead of updating manage-projects to skip validation, leave that as a human override and instead increase the limit in config 16:41
@mordred:waterwanders.combecause when an admin decides to override something, it's weird to be told no16:41
@clarkb:matrix.orgya that seems reasonable. Give ourselves the escape hatch but rely on sanity checking for typical operations16:42
@mordred:waterwanders.comlet's be honest - this specific scenario is not frequent. I'm guessing the initial import of nova was probably the last time we imported 10k commits in one go :)16:42
@clarkb:matrix.orgwhen everyone reconvenes are we good with my closing the root screen on review03?16:42
-@gerrit:opendev.org- Monty Taylor https://matrix.to/#/@mordred:inaugust.com proposed: [opendev/system-config] 990726: Increase receive.maxBatchCommits https://review.opendev.org/c/opendev/system-config/+/99072616:47
@fungicide:matrix.orgoh, that is indeed a very interesting discovery16:48
@fungicide:matrix.orgi never realized there was such a limit, but it makes some sense16:49
-@gerrit:opendev.org- Monty Taylor https://matrix.to/#/@mordred:inaugust.com proposed: [opendev/system-config] 990727: Start mirroring ubuntu resolute container images https://review.opendev.org/c/opendev/system-config/+/99072716:49
@clarkb:matrix.orgI am +2 on https://review.opendev.org/c/opendev/system-config/+/990726 but didn't approve it. Mainly because we need to restart Gerrit again to pick it up and I don't want the 3.13 upgrade in a week to be the time we do that. Do we want to approve it now and do another gerrit restart in an hour or two?16:51
@jim:acmegating.comi'm okay with that.16:55
@clarkb:matrix.orgYa I'm ok with it too as long as I'm not the only one around on a Friday to do it :)16:56
@jim:acmegating.comthat is a concern.  i may not be around by then.  but i would be happy to restart tomorrow if it comes to it.  or we could just restart on monday.16:56
@clarkb:matrix.orgAck Monday would work as well.16:57
@jim:acmegating.comi think we're still waiting for fungi's thoughts on the 2 approaches (one, the other, both, neither), yeah?16:57
@clarkb:matrix.orgWorst case we can remove the setting before the 3.13 upgrade too I guess16:57
@clarkb:matrix.orgYes I think fungi should weigh in and then we can decide what to do next16:57
@jim:acmegating.combecause that also sets us up for the question "do we also add the acl" and if the answer is yes, then i can push now without that change.  (but doing so means we don't test that change.)16:58
@fungicide:matrix.orgi'm still catching up on an eventful 1.5 hours of absence from several different spaces, but will prioritize feedback for this. i don't plan to disappear again until eod16:59
-@gerrit:opendev.org- Douglas Lopes Pereira proposed: [openstack/project-config] 990730: Add PTP Metrics app to StarlingX https://review.opendev.org/c/openstack/project-config/+/99073017:02
@jim:acmegating.comi have removed myself from project bootstrappers and administrators17:14
@harbott.osism.tech:regio.chatdo we have someone from airship around here? their config errors account for about half of the total errors in the openstack tenant by now17:43
@fungicide:matrix.orgildikov might be able to make some contacts17:44
@clarkb:matrix.orgDoug Goldstein may also know who to talk to17:45
@ildikov:matrix.orgI haven’t talked to folks in a while, but can try to reach them. We communicated in email, so happy to try make connections that way as well.17:49
@harbott.osism.tech:regio.chatjust pointing them to https://zuul.opendev.org/t/openstack/config-errors?skip=0 to check the airship projects listed there would be great. they can come here if they need help17:50
@fungicide:matrix.orgClark: corvus: so the "two approaches" are 990726 and... what alternative?17:55
@jim:acmegating.comfungi: adding the "forge server identity" acl to project bootstrappers so we have the option of telling it to skip validation17:56
@jim:acmegating.comwe think either in isolation is sufficient for this particular repo; we may have other larger repos in the future where we may need to skip validation17:57
@clarkb:matrix.organd as mordred mentioned we could do both things together. Update the ACL so that we have a manual out if we need it, but also bump the limit in 990726 so that cases like drizzle fall into the automatable category17:57
@jim:acmegating.comyep17:57
@fungicide:matrix.orgif both were applied then since the automated import still uses project bootstrappers so would never exercise the increased limit, unless skipping validation was a separate manual action18:00
@fungicide:matrix.orger, forgive my frankensentence there18:00
@clarkb:matrix.orgYes you need an explicit -o skip-validarion argument set on the git push to skip validation18:00
@clarkb:matrix.orgSo manage-projects would not add that option and rely on the limit alone. If we have problems in the future we can manually opt into skipping validation via git push -o skip-validation18:01
@fungicide:matrix.orgokay, so the suggestion does not include adding `-o skip-validation` to manage-projects18:01
@clarkb:matrix.orgNot if we do both. I think it may if we only update the acl18:01
@fungicide:matrix.orgi can definitely see arguments in favor of each. this is an unusual circumstance (first time we've hit it), so would we want to unknowingly allow it in future imports for other projects?18:02
@fungicide:matrix.orgthis gave us the opportunity to go, "wow that's a lot of commits you're importing, are you sure?"18:03
@clarkb:matrix.orgPossibly not. I think that is an argument in favor of the config update. And keeping the use of the push flag manually doesn't negate that value I don't think as any use of it would be intentional 18:03
@fungicide:matrix.orgbut we could also potentially do that when linting project creation requests in order to provide earlier feedback with less pain on our part18:04
@clarkb:matrix.orgYes though the upstream repo could change in the time between gating and deployment. But that seems like an unlikely situation 18:04
-@gerrit:opendev.org- Douglas Lopes Pereira proposed: [openstack/project-config] 990730: Add PTP Metrics app to StarlingX https://review.opendev.org/c/openstack/project-config/+/99073018:05
@fungicide:matrix.orgalso nothing (other than lack of efficiency) to keep a project from pushing that many changes through gerrit to merge with noop tests once the project is created18:05
-@gerrit:opendev.org- Douglas Lopes Pereira proposed: [openstack/project-config] 990730: Add PTP Metrics app to StarlingX https://review.opendev.org/c/openstack/project-config/+/99073018:06
@mordred:waterwanders.comdarn - I I should have thought of that - I totally could have claimed change 1000000 easily that way :)18:07
@fungicide:matrix.organyway, i think i'm convinced just upping the limit is the better option18:07
@fungicide:matrix.organd i'm around for another restart once that lands, to make up for taking an inconveniently-timed lunch break earlier ;)18:07
@clarkb:matrix.orgOk do you want to approve the change then? Assuming you haven't yet18:08
@fungicide:matrix.orgyou assume incorrectly18:08
@mordred:waterwanders.comwith that landed it's probably reasonable to just re-kick m-p and expect it to succeed18:08
@clarkb:matrix.orgAssuming manage-projects will attempt to push again. It may bot18:08
@fungicide:matrix.orgmodulo a gerrit restart in between18:09
@clarkb:matrix.org* Assuming manage-projects will attempt to push again. It may not18:09
@fungicide:matrix.orgi don't think gerrit will pick up that config change without restarting18:09
@clarkb:matrix.orgCorrect we need a Gerrit restart18:09
-@gerrit:opendev.org- Monty Taylor https://matrix.to/#/@mordred:inaugust.com proposed: [opendev/jeepyb] 990689: Log git errors if they happen https://review.opendev.org/c/opendev/jeepyb/+/99068918:10
@jim:acmegating.comi'm afk for a bit; i yield the responsibility of doing manual pushes back to the collective.  i'll try to check back later today and/or tomorrow if stuff needs doing18:10
@mordred:waterwanders.comalso that ^^ if we're going to run m-p again ... assuming it can pass its integration test18:11
@fungicide:matrix.orgalso https://review.opendev.org/c/openstack/project-config/+/989446 is another project creation request which is ready to approve as soon as we're done fiddling with gerrit18:12
@clarkb:matrix.orgI should be around today. Lunch is approaching but can work around that. The only other big thing in my list is to test etherpad 3.2.0 on the held node when I get a moment18:14
@clarkb:matrix.orgI am hoping my bike will be functioning again and I can pick it up. But haven't heard anything on that yet18:16
@fungicide:matrix.orgwalks can also be good18:16
@fungicide:matrix.org(i say as someone who only occasionally tries to ride a very ergonomically terrible beach cruiser bike and ends up with a very sore behind)18:17
@clarkb:matrix.orgThat goes away after a week or two. But yes walks are also good.18:19
@fungicide:matrix.orgi think this weekend we're planning for a ~6mi walk to the wright bros. memorial, up and over, then home again18:23
@fungicide:matrix.orgif we go early enough, we can be done at the national park before it opens to vehicular traffic for the morning18:24
@mordred:waterwanders.comalso, I imagine it'll be cooler that time of the day too18:30
@mordred:waterwanders.com(I assume it's already hot)18:30
@fungicide:matrix.orgit's variable this time of year, though supposed to be cooler for the next few days again at least18:46
@fungicide:matrix.orgthough how warm or cool it feels is more to do with wind speed, direction, cloud cover, and relative humidity18:47
@fungicide:matrix.orgour route follows the water mostly, so that helps18:47
-@gerrit:opendev.org- Zuul merged on behalf of Monty Taylor https://matrix.to/#/@mordred:inaugust.com: [opendev/system-config] 990726: Increase receive.maxBatchCommits https://review.opendev.org/c/opendev/system-config/+/99072618:54
@clarkb:matrix.orgthat is deploying now and `/home/gerrit2/review_site/etc/gerrit.config` appears to have updated as expected18:57
@clarkb:matrix.orgfungi: I'm about to start on lunch, but if you're able to drive the second restart I can put food down as necessary. Note since we already did a restart today the waiting queue move may need an additional suffix in the date/name18:57
@fungicide:matrix.orgyep, happy to18:59
@fungicide:matrix.orgthere was no remaining screen session on review03 so i've started a new one19:00
@fungicide:matrix.orgstatus notice The Gerrit service on review.opendev.org will be offline again monentarily while we restart for a configuration adjustment, but should return to service within a few minutes19:01
@fungicide:matrix.orgthat look good?19:01
@clarkb:matrix.orgyup I stopped the old screen session. Andthat notice lgtm19:01
@clarkb:matrix.orgI have attached to the screen you started19:02
@fungicide:matrix.orgcool, i'll send that now19:07
@fungicide:matrix.org#status notice The Gerrit service on review.opendev.org will be offline again monentarily while we restart for a configuration adjustment, but should return to service within a few minutes19:07
@status:opendev.org@fungicide:matrix.org: sending notice19:07
@fungicide:matrix.org`docker-compose -f /etc/gerrit-compose/docker-compose.yaml down && mv ~gerrit2/review_site/data/replication/ref-updates/waiting ~gerrit2/tmp/waiting_queue_2026-05-29.1 && rm ~gerrit2/review_site/cache/{gerrit_file_diff,git_file_diff,git_modified_files,modified_files,comment_context}-v2.* && docker-compose -f /etc/gerrit-compose/docker-compose.yaml up -d`19:08
@fungicide:matrix.orgthat's what i've queued up in screen, based on the last restart19:08
@clarkb:matrix.orgthat command lgtm19:08
@fungicide:matrix.orgi'll wait until statusbot is done, then execute it19:09
-@gerrit:opendev.org- Douglas Lopes Pereira proposed: [openstack/project-config] 990730: Add PTP Metrics Exporter app to StarlingX https://review.opendev.org/c/openstack/project-config/+/99073019:10
@status:opendev.org@fungicide:matrix.org: finished sending notice19:10
-@status:opendev.org- NOTICE: The Gerrit service on review.opendev.org will be offline again monentarily while we restart for a configuration adjustment, but should return to service within a few minutes19:10
@fungicide:matrix.orgengaging19:11
@fungicide:matrix.orggraceful stop took 23.6s19:11
@fungicide:matrix.orgon its way back up now19:11
@clarkb:matrix.orgso quick when the caches are only a few hours old19:12
@fungicide:matrix.orgwebui is loading for me, `Powered by Gerrit Code Review (3.12.7-dirty)`19:12
@fungicide:matrix.orgwatching the gerritbot log for a new patchset-created event19:14
@fungicide:matrix.orgbut if anybody has something to revise and push, that could speed replication checking along19:15
@fungicide:matrix.org(i don't personally)19:15
@fungicide:matrix.orgi'm also prepare to start reindexing with `gerrit index start changes --force` via the ssh api once that's confirmed19:16
@clarkb:matrix.orgI don't and I'm deep in a bowl of curry19:16
@clarkb:matrix.orgCan just push a useless DNM change if you need to19:16
@fungicide:matrix.orggo deeper, curry priorities!19:16
@fungicide:matrix.orgi'm in no hurry19:16
@fungicide:matrix.orgmadras? massaman? panang?19:17
@clarkb:matrix.orgThai green curry19:18
@fungicide:matrix.orgyessss! one of my faves. with lots of fresh basil i hope19:18
@clarkb:matrix.orgUnfortunately not. These are leftovers someone else got yesterday so I'm just cleaning out the fridge. Still tasty19:19
@fungicide:matrix.orgnext-day curry is often even better anyway19:20
@fungicide:matrix.orghttps://review.opendev.org/c/openstack/freezer-api/+/990735 got a new revision just a minute ago19:21
@fungicide:matrix.org74a02dc662bf446d8c99e03b78248c9c6f92cd5a19:21
@fungicide:matrix.orgthe gitea link takes me to https://opendev.org/openstack/freezer-api/commit/74a02dc662bf446d8c99e03b78248c9c6f92cd5a which loads19:21
@clarkb:matrix.orgexcellent19:21
@fungicide:matrix.orgi'll go ahead and start reindexing19:22
@clarkb:matrix.orgsounds good19:22
@fungicide:matrix.orgi'm filtering `~gerrit2/review_site/logs/error_log` for lines containing "Reindex" in the same screen session19:22
@fungicide:matrix.orgjust to give us a convenient place to monitor progress19:23
@clarkb:matrix.orgthen the last step would be for someone to try repushing the drizzle repo. We can potentially land 990730 and see if that does it for us. Or do it manually like corvus was doing earlier19:23
@fungicide:matrix.orgin theory 989446 also will rerun manage-projects19:24
@mordred:waterwanders.comI think you'd need to update the project-cache to set pushed-to-gerrit to false for drizzle/drizzle19:24
@fungicide:matrix.org(990730 is not ready to merge yet)19:24
@clarkb:matrix.orgAh yup that is a better option19:24
@mordred:waterwanders.combut - flipping that to false and then re-running m-p would likely be the best test of whether the config fix fixed it19:25
@clarkb:matrix.orgmordred: thats the json file right /me looks for it19:37
@clarkb:matrix.org`/opt/lib/jeepyb/project.cache` this is the file and yes the flag is set to true and would need to be toggled to false cc fungi 19:38
@clarkb:matrix.orgfungi: I +2'd https://review.opendev.org/c/openstack/project-config/+/989446 but didn't approve it so that we can edit that project.cache file if we want to use 989446 to try and resync drizzle/drizzle19:39
@clarkb:matrix.orgbut I think I'm comfortable with that if you are fungi 19:40
@mordred:waterwanders.com++ feels like the least manual re-try option19:40
@fungicide:matrix.orgyeah, i was going to wait for reindexing to complete first, it's nearly at 50% niow19:40
@clarkb:matrix.orgsounds good19:40
@fungicide:matrix.org80%19:51
@clarkb:matrix.orglooks like it completed with the expected 3 failuers again19:57
@fungicide:matrix.orgagreed. sorry my broadband went offline for about 20 minutes but seems to have returned finally, at least for the moment20:10
@fungicide:matrix.orgi switched `pushed-to-gerrit` to `false` for `drizzle/drizzle` in `/opt/lib/jeepyb/project.cache`20:12
@fungicide:matrix.orgapproving 989446 now20:12
@clarkb:matrix.orgthanks20:13
-@gerrit:opendev.org- Zuul merged on behalf of Sabbir Ahmed: [openstack/project-config] 989446: Add starlingx/app-machine-operator project https://review.opendev.org/c/openstack/project-config/+/98944620:21
@clarkb:matrix.orgmanage-projects should be running nowish20:23
@clarkb:matrix.orgthough I guess it does the giteas first? so maybe not quite yet. The job has staretd though20:23
@clarkb:matrix.orgthe job reports success20:28
@clarkb:matrix.orghttps://opendev.org/drizzle/drizzle is still not right. Maybe we didn't try to push I'll look at logs20:29
@clarkb:matrix.orghttps://paste.opendev.org/show/bE1gg3bhjQjAGwSBri1K/ ya I think it decided it didn't need to push for whatever reason20:29
@clarkb:matrix.orgso not the outcome we wanted but also not an emergency. Maybe we just concede and try the manual push again?20:30
@clarkb:matrix.orgI detached from the screen. Not sure that we need it any longer20:31
@fungicide:matrix.orglooks like the deploy reported a few minutes ago20:36
@fungicide:matrix.orgm-p did set that flag back to true when it ran, fwiw20:37
@fungicide:matrix.orgi've closed out the screen session now20:37
@fungicide:matrix.orgmordred: i added you as the initial member of drizzle-core20:38
@mordred:waterwanders.comfungi: \o/ thank you!!20:40
@mordred:waterwanders.combut also - yeah, opendev.org/drizzle/drizzle is still sad WTF20:41
@fungicide:matrix.orgnow we just need to get the heads pushed for the one repo, i guess the suggestion is manual `git push --force`?20:41
@mordred:waterwanders.comyeah - there's actually a set of commands we had in a paste - one sec20:41
@fungicide:matrix.orghappy to temporarily stick myself in bootstrappers and do that20:42
@mordred:waterwanders.comfungi: https://paste.opendev.org/show/bRGwYC6v96VRWKkLnQbY/ is a paste I put together based on https://paste.opendev.org/show/bEdKVOKKuw6L6BRNCJw9/20:43
@clarkb:matrix.orgI'm probably going to pop out for the school run in 15 minutes but happy for fungi to do git pushing 20:43
@mordred:waterwanders.comthat has the specific ones jeepyb specifically uses to do things.20:43
@fungicide:matrix.orgsure, i can do it now, just a sec20:43
@mordred:waterwanders.comwoot! I see branches!20:47
@fungicide:matrix.orgit's done and i see content at https://opendev.org/drizzle/drizzle including the expected branches now20:47
@mordred:waterwanders.com2.3% m4 :) 20:47
@fungicide:matrix.orgplease double-check it looks like you expect before you merge any reviews20:47
@mordred:waterwanders.comyup! That's the expected tip. now to push up a gitreview file :) 20:48
@mordred:waterwanders.comthat's probably the only repo in opendev that lists more perl than python code20:48
@fungicide:matrix.orgi've removed my fungi.admin account from the project bootstrappers group again now20:48
@mordred:waterwanders.comthanks a bunch - now I get to do the fun stuff - like setting up proper multi-repo testing stuffs20:49
@fungicide:matrix.orgperl... for great justice20:49
@clarkb:matrix.orgWhat an adventure too. But we learned things about Gerrit so yay?20:49
@mordred:waterwanders.comright? I wasn't expecting nearly that much fun20:49
@fungicide:matrix.orgfeels like an after-school special20:50
@clarkb:matrix.org158.69.66.89 is the held ehterpad 3.2.0 node. I have created a clarkb-test pad there and tested it with one firefox client and one chrome client. It seems to work22:03
@clarkb:matrix.orgunfortunately it doesn't really test the session cleanup routine and the oom problem due to the lack of a decade of db records. But we can always go back to disabling the cleanups if necessary.22:03
@fungicide:matrix.orgyeah, i think that's a fine approach22:05
@fungicide:matrix.orgthe held etherpad is working fine for me too22:10
@clarkb:matrix.orggreat. Maybe monday we can approve that change then? I think I'm done with upgrading servers for the day :)22:11
@fungicide:matrix.orgyep, sounds great22:12
@mordred:waterwanders.comfwiw: https://review.opendev.org/c/drizzle/drizzle/+/990762 22:17
@fungicide:matrix.orgwoohoo!22:20
@fungicide:matrix.orgit lives22:20
@fungicide:matrix.orgis archon spren your llm agent companion?22:28
@fungicide:matrix.orgi thought the name might be an anagram, but my algorithmic finder didn't come up with much. person ranch? sharpen corn?22:28
@mordred:waterwanders.comit's one of them :) Archon is the review agent - his job is to watch the gerrit event stream and review the changes (on a whitelist of opt-in projects of course)22:29
@mordred:waterwanders.comno - it's not an anagram. the archon name is just sort of made up. But in Sanderson's Stormlight Archive, spren are beings that are physical manifestations of human emotions, natural phenomena, and abstract concepts. So, sort of like souls of objects and concepts.22:30
@mordred:waterwanders.comso I've been personifying my various agents as-if they are spren. With the exception of my coding agent, sword-nimi,(aka nightblood) who is a direct reference to a sapient sword that was the result of someone from a different world having learned about spren from Roshar and infusing an object with life energy on their own.22:32
@mordred:waterwanders.comI may have a sanderson problem :) 22:32
@mordred:waterwanders.com(nimi is an honorific like -san and is how that particular sword is referred to respectfully by one of his wielders. )22:35
@mordred:waterwanders.comI particularly like this quote from one of the sanderson wikis ... "Personality: Despite its apocalyptic power, the sword possesses the naive personality of a child, cheerfully asking if it has "destroyed lots of evil today." ... which I think is very apt for an AI agent22:35
@fungicide:matrix.orgokay, as long as the answer is geeky, i approve22:39
@fungicide:matrix.organd now i have more novels to add to the reading list i'll already never get through, thanks!22:40
@mordred:waterwanders.comoh - yeah. Let me help you with that: https://emonty.github.io/reading-order/#/?view=linear&categories.other-pub=false&categories.potential-plan=false&categories.future-plan=false&categories.unpublished=false&categories.apocryphal=false&categories.forthcoming=false 22:43
@fungicide:matrix.orgmuahahahaha22:43
@mordred:waterwanders.comit turns out cosmere is MCU-like in that there are a bunch of interconnections. but he intends the various books and series to be able to read _without_ needing to read all of them22:43
@fungicide:matrix.orgseems prolific22:44
@mordred:waterwanders.comhowever - I can report that it is enjoyable to read all of them, and then do all of them a second time and notice the references to things throughout22:44
@mordred:waterwanders.comthe man is a machine22:44
@clarkb:matrix.orgI picked up Esoteric Ebb recently which is basically a DnD reskin of Disco Elysium. Both games are essentially graphic choose your own adventure novels in video game form with some light world navigation22:44
@clarkb:matrix.orgsorry the discussion about reading made me want to bring it up22:44
@mordred:waterwanders.com++ I'll add that to my list :)22:44
@mordred:waterwanders.comspeaking of books though - if you haven't all been preached to about the gospel of Dungeon Crawler Carl ... let me express my pleasure at how good book 8 (which just came out) was. HIGHLY recommend the audiobook versions, because the audiobook actor is really *really* good. (so good that the author and the audiobook actor frequently do appearances together, which is not the _most_ normal thing for a book series)22:46
@clarkb:matrix.orgI've avoided Dungeon Crawler Carl mostly out of an annoyance that they don't do smaller format prints22:47
@clarkb:matrix.orgI don't mind spending money to buy books but I hate the giant tomes that make it difficult to read22:47
@mordred:waterwanders.comI hear that. I've actually been enjoying hardbacks for things recently, but they are much harder to schlep around22:48
@mordred:waterwanders.comsandy reports that actually reading carl is also enjoyable, she's been reading them after we've listened to them. I'm going to try that next after I finish the Robin Hobb books I'm doing now22:48
@clarkb:matrix.orgI think Dungeon Crawler Carl is an audible exclusive too?22:49
@clarkb:matrix.orgfor the audio version I mean22:49
@mordred:waterwanders.comyeah, they are.22:53
« Thursday, 2026-05-28 Index Saturday, 2026-05-30 »

Generated by irclog2html.py 4.1.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!