| @priteau:matrix.org | Hello. Would there be a known change that would explain why the periodic kolla-publish-* jobs have stopped running daily? https://zuul.opendev.org/t/openstack/builds?job_name=kolla-publish-rocky-10-quay | 09:23 |
|---|---|---|
| @priteau:matrix.org | The last run on the master branch was on 2026-06-12 | 09:23 |
| @priteau:matrix.org | Specifically for the master branch, but stable/2026.1 also had a gap between June 12 and today | 09:41 |
| @harbott.osism.tech:regio.chat | iiuc this is because the arm jobs were blocked due to some provider issue. and IIUC when one part of a buildset is still pending, no new periodic buildsets are triggered. we might want to set up a dedicated periodic-arm pipeline similar to the check-arm pipeline to separate these | 11:08 |
| @priteau:matrix.org | Do you think it will get resolved automatically for the next periodic run? | 11:18 |
| @mnasiadka:matrix.org | Well, the backlog for arm jobs seems to be like… 5 days? | 12:15 |
| @mnasiadka:matrix.org | https://zuul.opendev.org/t/openstack/nodeset-requests | 12:15 |
| @priteau:matrix.org | So the new builds you manually queued today won't help… | 13:08 |
| @priteau:matrix.org | Should we remove the aarch64 jobs temporarily? | 13:08 |
| @jim:acmegating.com | i think there is a problem with the zuul launchers | 13:15 |
| @jim:acmegating.com | 2026-06-16 13:14:39,940 ERROR zuul.Launcher: openstack.exceptions.ConfigException: https://ovhcloud.com is a remote profile that could not be fetched: 404 Not Found | 13:15 |
| @jim:acmegating.com | i'm not sure if that's due to a new openstacksdk version or a change in ovh (like, did they move their profile?) | 13:19 |
| @jim:acmegating.com | there has been an openstacksdk release since the last time opendev upgraded zuul | 13:20 |
| @jim:acmegating.com | i'm going to do a quick test to figure out if the issue is sdk or ovh | 13:22 |
| @jim:acmegating.com | okay, the issue is present with 4.15.0 and 4.14.0, so i'm assuming something changed in ovh (maybe due to blocking ai scraping bots) | 13:24 |
| @jim:acmegating.com | curl -Lv https://ovhcloud.com/.well-known/openstack/api (i think this is the url sdk should be using) redirects me to https://us.ovhcloud.com/.well-known/openstack/api/ which does 404 | 13:29 |
| @jim:acmegating.com | it also 404s without the trailing / | 13:29 |
| @jim:acmegating.com | https://www.ovhcloud.com/.well-known/openstack/api exists | 13:31 |
| @jim:acmegating.com | i think for the moment we need to import that information into our clouds.yaml; then let the sdk and ovh teams know the ovh well-known file is no longer well-known. | 13:32 |
| @harbott.osism.tech:regio.chat | arnaudm: ^^ can you check this? iirc this has happened before | 13:32 |
| @arnaudm:matrix.org | hey, let me read the thread | 13:34 |
| @arnaudm:matrix.org | so the well-known works with www, but is not without, am I right?> | 13:36 |
| @jim:acmegating.com | arnaudm: that appears to be the case from what i'm seeing, yes | 13:37 |
| @harbott.osism.tech:regio.chat | that's how I read this, yes | 13:37 |
| @arnaudm:matrix.org | I will let the team know about this, I wasn't even aware of that stuff | 13:38 |
| @harbott.osism.tech:regio.chat | actually testing locally I'm only getting a big JS blob from www, too, so likely not the expected response there, either | 13:39 |
| @jim:acmegating.com | i mean, a js blob is expected | 13:39 |
| @jim:acmegating.com | not sure what big means in this case | 13:39 |
| @jim:acmegating.com | https://paste.opendev.org/show/b75hYA3XatYCVAXAmpak/ | 13:40 |
| @jim:acmegating.com | that's what the content should be | 13:40 |
| @jim:acmegating.com | (and what i get at the www url) | 13:40 |
| @arnaudm:matrix.org | doing curl here and it works with www, but not without | 13:40 |
| -@gerrit:opendev.org- James E. Blair https://matrix.to/#/@jim:acmegating.com proposed: [opendev/system-config] 993598: Use local instead of remote profile for OVH https://review.opendev.org/c/opendev/system-config/+/993598 | 13:41 | |
| @harbott.osism.tech:regio.chat | ah, yes, I'm getting redirected to https://www.ovhcloud.com/de/.well-known/openstack/api/ , missed the `/de/` part | 13:41 |
| @harbott.osism.tech:regio.chat | fancy cdn setup ;) | 13:42 |
| @jim:acmegating.com | infra-root: ^ i'm going to manually apply that to the zuul launchers | 13:42 |
| @mnasiadka:matrix.org | Oh fun every day | 13:43 |
| @mnasiadka:matrix.org | So, should we have periodic-arm pipeline now? | 13:43 |
| @harbott.osism.tech:regio.chat | corvus: I assume you tested this locally? lgtm, but right, even force merge won't help if we can get a node to deploy it | 13:43 |
| @mnasiadka:matrix.org | I’m happy to propose it, if we think this is the way to go until we get more arm64 providers | 13:43 |
| @jim:acmegating.com | Jens Harbott: yes, that works with my test script which goes as far as "get a cloud session" it does not do anything after that. | 13:44 |
| @harbott.osism.tech:regio.chat | that should be good enough to verify all the parameters, ack | 13:44 |
| @jim:acmegating.com | the launchers immediately loaded that and resumed work | 13:46 |
| @jim:acmegating.com | #status log manually applied https://review.opendev.org/993598 to launchers to fix error with ovh providers | 13:47 |
| @status:opendev.org | @jim:acmegating.com: finished logging | 13:47 |
| @harbott.osism.tech:regio.chat | arnaudm: if that helps the change seems to have happened not more than 3 hours ago, if needed we can likely find a pretty exact timeframe our logs | 13:49 |
| @arnaudm:matrix.org | yes if you have the exact timeframe | 13:50 |
| @jim:acmegating.com | Jens Harbott: arnaudm i think the change could have happened any time in the last week | 13:50 |
| @jim:acmegating.com | i think this only happens on startup, so the last time we would have initialized is around june 6 | 13:51 |
| @arnaudm:matrix.org | damn | 13:51 |
| @jim:acmegating.com | (we're currently finishing this past weekends restart (which was delayed due to the osuosl issue)) | 13:51 |
| @harbott.osism.tech:regio.chat | oh, that's ... unexpected, but well ... | 13:52 |
| @jim:acmegating.com | i'm not sure if there's any other playbook or cron job that uses the openstack api on bridge that could narrow it down though? | 13:52 |
| @harbott.osism.tech:regio.chat | so we did disable log uploads yesterday, that might also have been related | 13:53 |
| @harbott.osism.tech:regio.chat | * so we did disable log uploads yesterday, that might also have been related then | 13:53 |
| @harbott.osism.tech:regio.chat | https://review.opendev.org/c/opendev/base-jobs/+/993335 | 13:54 |
| @jim:acmegating.com | the log uploads do use the "profile:" keyword, so it's possible they would have seen the same error. | 13:55 |
| @priteau:matrix.org | Nice roller coaster 😄 https://grafana.opendev.org/d/21a6e53ea4/zuul-status?orgId=1&from=now-6h&to=now&timezone=utc&viewPanel=panel-15 | 13:56 |
| @harbott.osism.tech:regio.chat | arnaudm: so in the opendev-prod-hourly pipeline I see post failures only yesterday between 10 UTC and 12 UTC, the latter is when the fix was merged, so likely the error was introduced between 9 and 10, possibly a bit earlier if all the 9 UTC log uploads were lucky and went to rax. not as exact as I had hoped for, but maybe still good enough to help you | 14:04 |
| @arnaudm:matrix.org | perfect, thanks | 14:04 |
| -@gerrit:opendev.org- Zuul merged on behalf of James E. Blair https://matrix.to/#/@jim:acmegating.com: [opendev/system-config] 990679: Document current gerrit restart process https://review.opendev.org/c/opendev/system-config/+/990679 | 14:50 | |
| -@gerrit:opendev.org- Monty Taylor https://matrix.to/#/@mordred:inaugust.com proposed: [zuul/zuul-jobs] 993612: Support uv venvs in fetch-subunit output https://review.opendev.org/c/zuul/zuul-jobs/+/993612 | 14:53 | |
| -@gerrit:opendev.org- Monty Taylor https://matrix.to/#/@mordred:inaugust.com proposed: [zuul/zuul-jobs] 993612: Support uv venvs in fetch-subunit output https://review.opendev.org/c/zuul/zuul-jobs/+/993612 | 15:12 | |
| -@gerrit:opendev.org- Monty Taylor https://matrix.to/#/@mordred:inaugust.com proposed: [zuul/zuul-jobs] 993612: Support uv venvs in fetch-subunit output https://review.opendev.org/c/zuul/zuul-jobs/+/993612 | 15:36 | |
| @arnaudm:matrix.org | hey, the team identified the issue, they will fix it | 15:40 |
| @fungicide:matrix.org | thanks arnaudm!!! | 15:42 |
| @fungicide:matrix.org | corvus: should we abandon/cancel 993598 in that case, before it merges? | 15:55 |
| @fungicide:matrix.org | or i guess it's not fixed yet so we can revert it later if we prefer | 15:56 |
| @jim:acmegating.com | yeah let's wait | 15:57 |
| -@gerrit:opendev.org- Stephen Finucane proposed: | 16:03 | |
| - [openstack/project-config] 993628: Remove packaging configuration https://review.opendev.org/c/openstack/project-config/+/993628 | ||
| - [openstack/project-config] 993629: Apply ruff https://review.opendev.org/c/openstack/project-config/+/993629 | ||
| - [openstack/project-config] 993630: Bump Python versions when creating a new release https://review.opendev.org/c/openstack/project-config/+/993630 | ||
| - [openstack/project-config] 993631: trivial: Fix shellcheck issues https://review.opendev.org/c/openstack/project-config/+/993631 | ||
| - [openstack/project-config] 993632: trivial: Add PEP 723 headers for Python scripts https://review.opendev.org/c/openstack/project-config/+/993632 | ||
| -@gerrit:opendev.org- Zuul merged on behalf of James E. Blair https://matrix.to/#/@jim:acmegating.com: [opendev/system-config] 993598: Use local instead of remote profile for OVH https://review.opendev.org/c/opendev/system-config/+/993598 | 16:04 | |
| -@gerrit:opendev.org- Monty Taylor https://matrix.to/#/@mordred:inaugust.com proposed: [zuul/zuul-jobs] 993612: Support uv venvs in fetch-subunit output https://review.opendev.org/c/zuul/zuul-jobs/+/993612 | 16:06 | |
| -@gerrit:opendev.org- Monty Taylor https://matrix.to/#/@mordred:inaugust.com proposed: [opendev/base-jobs] 990828: Honor container_command in buildset-registry playbooks https://review.opendev.org/c/opendev/base-jobs/+/990828 | 16:11 | |
| -@gerrit:opendev.org- Michal Nasiadka proposed: [zuul/zuul-jobs] 993649: use-buildset-registry: Rename vars/Fedora.yaml to vars/RedHat.yaml https://review.opendev.org/c/zuul/zuul-jobs/+/993649 | 17:48 | |
| -@gerrit:opendev.org- Monty Taylor https://matrix.to/#/@mordred:inaugust.com proposed: [opendev/system-config] 993655: Add gerritbot to #drizzle:drizzle.org https://review.opendev.org/c/opendev/system-config/+/993655 | 18:47 | |
| -@gerrit:opendev.org- Monty Taylor https://matrix.to/#/@mordred:inaugust.com proposed: | 19:00 | |
| - [opendev/base-jobs] 990828: Honor container_command in buildset-registry playbooks https://review.opendev.org/c/opendev/base-jobs/+/990828 | ||
| - [opendev/base-jobs] 993657: Fix ansible-lint grumps from earlier force-land https://review.opendev.org/c/opendev/base-jobs/+/993657 | ||
| @mordred:waterwanders.com | fwiw - 990828 you can see working here: https://review.opendev.org/c/drizzle/drizzle/+/990830 ... and 993657 is fallout from this morning's ovh fun. | 19:01 |
| @priteau:matrix.org | build-openstack-releasenotes failures for the last hour: https://zuul.opendev.org/t/openstack/builds?job_name=build-openstack-releasenotes | 19:04 |
| @priteau:matrix.org | This should fix it: https://review.opendev.org/c/openstack/openstack-zuul-jobs/+/993658 | 19:05 |
| @arnaudm:matrix.org | fungicide I was told that the fix may take few days before landing on prod, I am trying to accelerate this but out of my scope unfortunately | 19:08 |
| @mordred:waterwanders.com | Pierre Riteau: that patch seems reasonable - but any idea how that could have ever worked? | 19:08 |
| @fungicide:matrix.org | arnaudm: no worries! we have the temporary fix in place | 19:08 |
| @priteau:matrix.org | The broken code was merged an hour ago: https://review.opendev.org/c/openstack/openstack-zuul-jobs/+/968398 | 19:09 |
| @priteau:matrix.org | I guess the CI jobs don't cover this | 19:09 |
| @mordred:waterwanders.com | ah, nod. makes sense. yeah. :( | 19:09 |
| @mnasiadka:matrix.org | Yeah, waiting for Verified+1 and will merge it | 19:11 |
| @priteau:matrix.org | Checks completed just now | 19:12 |
| @mordred:waterwanders.com | if folks have a sec, could I get eyes on https://review.opendev.org/c/openstack/project-config/+/993214 and https://review.opendev.org/c/opendev/system-config/+/993655 - they're both super easy :) ... Then also, https://review.opendev.org/c/opendev/base-jobs/+/993657 is unrelated but fixes the base-jobs gate. | 19:28 |
| -@gerrit:opendev.org- Michal Nasiadka proposed on behalf of Martin André: [ttygroup/gertty] 934535: Fix compatibility with SQLAlchemy v2 https://review.opendev.org/c/ttygroup/gertty/+/934535 | 19:28 | |
| -@gerrit:opendev.org- Michal Nasiadka proposed on behalf of Martin André: [ttygroup/gertty] 934535: Fix compatibility with SQLAlchemy v2 https://review.opendev.org/c/ttygroup/gertty/+/934535 | 20:08 | |
| -@gerrit:opendev.org- Monty Taylor https://matrix.to/#/@mordred:inaugust.com proposed: | 20:17 | |
| - [zuul/zuul-jobs] 993612: Support uv venvs in fetch-subunit output https://review.opendev.org/c/zuul/zuul-jobs/+/993612 | ||
| - [zuul/zuul-jobs] 993677: Override gce universe for gce uploads https://review.opendev.org/c/zuul/zuul-jobs/+/993677 | ||
| @jim:acmegating.com | mordred: fungi https://opendev.org/opendev/zuul-providers/src/branch/master/zuul.d/providers.yaml#L86 | 20:31 |
| i've been trying various experiments running zuul tests under python 3.14. the latest experiment is to run it with more ram. but our 16gb rax classic nodes have only 4 vcpus, not 8 (like the 8g nodes). i think at this point i would suggest that the answer to the question in that TODO comment is "no" | ||
| @jim:acmegating.com | (the cpus are not significantly better, so it's basically just 50% cpu resources) | 20:31 |
| @mordred:waterwanders.com | ugh. jeez | 20:32 |
| @mordred:waterwanders.com | like - I still can't get over why 3.14 is so much worse but still is a thing | 20:32 |
| @jim:acmegating.com | there's no 16gb version in the performance1 series. we can go to performance2-30 | 20:32 |
| @jim:acmegating.com | compute1-15 could be an option -- that's 8vcpu and 15g | 20:33 |
| @jim:acmegating.com | i don't know anything about it | 20:33 |
| @jim:acmegating.com | oh, no disk on those | 20:34 |
| @jim:acmegating.com | (i mean, we can work with that, but i think it suggests it may be substantially different than other flavors) | 20:34 |
| -@gerrit:opendev.org- James E. Blair https://matrix.to/#/@jim:acmegating.com proposed: [opendev/zuul-providers] 993689: Increase rax-classic 16GB flavor to next size up https://review.opendev.org/c/opendev/zuul-providers/+/993689 | 20:38 | |
| -@gerrit:opendev.org- Dr. Jens Harbott proposed: [openstack/project-config] 993691: Clean up log-classify jobs https://review.opendev.org/c/openstack/project-config/+/993691 | 20:44 | |
| @mnasiadka:matrix.org | corvus: but that’s 30G, not 16G - how is quota defined in rax classic? cpu only? | 20:49 |
| @jim:acmegating.com | mnasiadka: yes, it's not the only flavor that offers more than the minimum in one category in order to meet the minimum in another | 20:49 |
| @jim:acmegating.com | mnasiadka: it's weird | 20:49 |
| @jim:acmegating.com | https://paste.opendev.org/show/bzhgGnbX7m0luGsn55ky/ is the list | 20:50 |
| @jim:acmegating.com | this looks like the best way to get the cpu/ram needed for that while using the performance series (which, i think rax preferred us to use), but i'm open to suggestions :) | 20:51 |
| @mnasiadka:matrix.org | Well, nobody is going to complain, but then we could extend the 32G flavor to use this as well (to extend it to other providers than VexxHost) - but maybe not in the same change | 20:52 |
| @mnasiadka:matrix.org | That list is a bit weird, not mentioning that 15G and 30G sounds funny anyway | 20:52 |
| @jim:acmegating.com | yep we could add more 32gb flavors | 20:53 |
| @jim:acmegating.com | i think originally we wanted to avoid using those in rax because they used up so much quota, but zuul-launcher will favor vexxhost for those anyway now because of its low utilization, so it's not as much of a problem. | 20:54 |
| @mordred:waterwanders.com | corvus: 2026-06-16 20:54:00.198178 | TASK [upload-logs-swift : Upload logs to swift] | 20:57 |
| 2026-06-16 20:54:02.196811 | localhost -> localhost | Output suppressed because no_log was given | ||
| failure | ||
| 2026-06-16 20:54:02.199394 | | ||
| @mordred:waterwanders.com | corvus: Provider: opendev.org%2Fopendev%2Fzuul-providers/rax-dfw-main | 20:58 |
| @mordred:waterwanders.com | more jobs failed with post_failure that time | 20:58 |
| @jim:acmegating.com | oh but that may not be the log upload provider, right? | 20:59 |
| @mordred:waterwanders.com | (and a different subset are failing this time through) | 20:59 |
| @mordred:waterwanders.com | yeah - do we log which log upload provider we're goign to use? | 20:59 |
| @mordred:waterwanders.com | (I still have the console up) | 20:59 |
| @jim:acmegating.com | i thought so but i don't know the keyword | 21:00 |
| @mnasiadka:matrix.org | That no_log is not really helping in getting that, last time I had to check executor-debug logs to get that | 21:00 |
| @jim:acmegating.com | i thought we put it in a task before the upload? | 21:00 |
| @mnasiadka:matrix.org | Oh, maybe - I didn’t try scrolling up ;) | 21:00 |
| @mordred:waterwanders.com | I don't see it anywhere | 21:00 |
| @mordred:waterwanders.com | but - I _am_ scanning text with my eyeballs | 21:01 |
| @mordred:waterwanders.com | https://zuul.opendev.org/t/zuul/build/a5f6c4434198461ca98d019b4f140029 is one of the failed zuul jobs, fwiw | 21:02 |
| @jim:acmegating.com | okay these are coming from the base-minimal playbook | 21:04 |
| @jim:acmegating.com | maybe something about that code path is different -- that could explain why we're not seeing widespread problems, but also, why it's affecting a change that isn't touching logs | 21:04 |
| @jim:acmegating.com | um | 21:04 |
| @jim:acmegating.com | maybe base-minimal still have ovh configured? | 21:04 |
| @mordred:waterwanders.com | oh - maybe so? | 21:05 |
| @mordred:waterwanders.com | yup | 21:05 |
| -@gerrit:opendev.org- Monty Taylor https://matrix.to/#/@mordred:inaugust.com proposed: [opendev/base-jobs] 993694: Disable ovh from base-minimal https://review.opendev.org/c/opendev/base-jobs/+/993694 | 21:06 | |
| -@gerrit:opendev.org- James E. Blair https://matrix.to/#/@jim:acmegating.com proposed: [opendev/base-jobs] 993695: Update base-minimal post-logs https://review.opendev.org/c/opendev/base-jobs/+/993695 | 21:09 | |
| @jim:acmegating.com | mordred: lgtm but also ^ | 21:09 |
| @jim:acmegating.com | mnasiadka: mordred that solves the mystery of the missing task | 21:10 |
| @mordred:waterwanders.com | ah - neat | 21:10 |
| @jim:acmegating.com | the normal upload playbook *does* have a task that outputs the selected log provider | 21:10 |
| @jim:acmegating.com | now that change will add it to the base-minimal upload playbook | 21:10 |
| @mordred:waterwanders.com | also- you're going to need https://review.opendev.org/c/opendev/base-jobs/+/993657 | 21:10 |
| @jim:acmegating.com | okay i approved 657 and also 694 because those are pro-forma changes (add spaces, disable ovh which is already disabled). i think 695 deserves normal review. | 21:12 |
| @mordred:waterwanders.com | corvus: 695 lgtm conceptually - it's going to hit the same space error in ansible-lint :( | 21:15 |
| -@gerrit:opendev.org- Zuul merged on behalf of Monty Taylor https://matrix.to/#/@mordred:inaugust.com: [opendev/base-jobs] 993657: Fix ansible-lint grumps from earlier force-land https://review.opendev.org/c/opendev/base-jobs/+/993657 | 21:17 | |
| -@gerrit:opendev.org- James E. Blair https://matrix.to/#/@jim:acmegating.com proposed: [opendev/base-jobs] 993695: Update base-minimal post-logs https://review.opendev.org/c/opendev/base-jobs/+/993695 | 21:19 | |
| @mordred:waterwanders.com | the second ovh change hit it in the gate, but since the space fix landed I just popped a recheck on it. it should work this time | 21:20 |
| @mordred:waterwanders.com | I should have stacked them in the first place - duh | 21:21 |
| -@gerrit:opendev.org- Zuul merged on behalf of Monty Taylor https://matrix.to/#/@mordred:inaugust.com: [opendev/base-jobs] 993694: Disable ovh from base-minimal https://review.opendev.org/c/opendev/base-jobs/+/993694 | 21:29 | |
| -@gerrit:opendev.org- Monty Taylor https://matrix.to/#/@mordred:inaugust.com proposed: [opendev/base-jobs] 990828: Honor container_command in buildset-registry playbooks https://review.opendev.org/c/opendev/base-jobs/+/990828 | 21:32 | |
| @priteau:matrix.org | Release notes jobs are still broken: https://zuul.opendev.org/t/openstack/build/6c2e95bcfbe5414da5533bd2efe29a82 | 21:54 |
| @priteau:matrix.org | FileNotFoundError: [Errno 2] No such file or directory: 'tox' | 21:55 |
| @fungicide:matrix.org | Pierre Riteau: it may make more sense to revert and let the people who were working on that implementation try again later | 21:56 |
| @fungicide:matrix.org | i don't currently have bandwidth to dig into the errors enough to be able to confidently review a fix, but am happy to rubber-stamp a revert | 21:57 |
| -@gerrit:opendev.org- Stephen Finucane proposed: [openstack/project-config] 993628: Remove packaging configuration https://review.opendev.org/c/openstack/project-config/+/993628 | 21:59 | |
| -@gerrit:opendev.org- Stephen Finucane proposed: [openstack/project-config] 993629: Apply ruff https://review.opendev.org/c/openstack/project-config/+/993629 | 21:59 | |
| -@gerrit:opendev.org- Stephen Finucane proposed: [openstack/project-config] 993630: Bump Python versions when creating a new release https://review.opendev.org/c/openstack/project-config/+/993630 | 21:59 | |
| -@gerrit:opendev.org- Stephen Finucane proposed: | 21:59 | |
| - [openstack/project-config] 993631: trivial: Fix shellcheck issues https://review.opendev.org/c/openstack/project-config/+/993631 | ||
| - [openstack/project-config] 993632: trivial: Add PEP 723 headers for Python scripts https://review.opendev.org/c/openstack/project-config/+/993632 | ||
| @priteau:matrix.org | https://review.opendev.org/c/openstack/openstack-zuul-jobs/+/993700 | 22:06 |
| @priteau:matrix.org | Revert proposed | 22:06 |
| @priteau:matrix.org | But stephenfin looks like he's around, maybe he has a quick fix to the code? | 22:06 |
| @priteau:matrix.org | Maybe tox needs replacing by the value of tox_executable | 22:08 |
| -@gerrit:opendev.org- Pierre Riteau proposed: [zuul/zuul-jobs] 993701: Fix typo in task name https://review.opendev.org/c/zuul/zuul-jobs/+/993701 | 22:08 | |
Generated by irclog2html.py 4.1.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!