Tuesday, 2023-05-30

opendevreviewDavid Hill proposed openstack/nova master: Inject passwords in a rescue call  https://review.opendev.org/c/openstack/nova/+/88464700:41
opendevreviewDavid Hill proposed openstack/nova master: Inject passwords in a rescue call  https://review.opendev.org/c/openstack/nova/+/88464700:52
opendevreviewDavid Hill proposed openstack/nova master: Inject passwords in an instance rescue call  https://review.opendev.org/c/openstack/nova/+/88464700:55
opendevreviewKiran Pawar proposed openstack/nova-specs master: SR-IOV NIC device tracking in Placement  https://review.opendev.org/c/openstack/nova-specs/+/88456904:55
*** auniyal8 is now known as auniyal04:57
bauzasgood morning folks07:08
sahido/08:44
sahidsean-k-mooney[m]: ++ thanks a lot for your review and points regarding neutronclient replacement I appreciate :-)08:46
dvo-plvgibi, bauzas : Hello, are you around today?09:22
bauzasdvo-plv: not really, but shoot09:23
dvo-plvI would like to ask about review, but if you still have a vacation, I belive that it is unacceptable to ask you for this today09:24
bauzasdvo-plv: I'll try to do a few reviews today 09:29
bauzassahid: I'll also look at your chnage today09:29
dvo-plvGreat, thanks, this is a link https://review.opendev.org/c/openstack/nova/+/87607509:34
sahidbauzas: ack thank you !11:59
opendevreviewDanylo Vodopianov proposed openstack/nova master: Packed virtqueue support was added.  https://review.opendev.org/c/openstack/nova/+/87607512:22
opendevreviewDanylo Vodopianov proposed openstack/nova master: Packed virtqueue support was added.  https://review.opendev.org/c/openstack/nova/+/87607512:24
sahidsean-k-mooney[m], gibi, a quick question regarding https://review.opendev.org/c/openstack/nova-specs/+/86837712:38
sahidusually want a way for operators to fix the usage of a feature enabled from image properties that by using flavor extra_specs, no?12:39
sahidthe current state of the spec does not give a chance for operator to define a certain group of flavors allowed to use the feature12:40
sahid(or I may have missed some points ;D)12:40
sahid(or You already have discussed it ;D)12:40
LarsErikPHi! saw that this failed the tests ~2 weeks ago. any progress? https://review.opendev.org/c/openstack/nova/+/88292112:48
Uggla_Hello, just to let you know that last week I discovered that readonly is not supported by virtiofs. So at the moment we cannot mount the scaphandre fs in ro. I have discussed that with the virt team, they already have an open ticket about it.12:50
Uggla_So I guess, this is not a blocker. Also, a bind mount can be used to workaround that issue for the moment.12:52
fricklerLarsErikP: zuul voted verified+1, it's just some broken 3rd party CI that failed, so that patch is in the normal process for waiting for reviews13:04
bauzasUggla_: hmmm, ok, we should discuss this later then13:34
opendevreviewMichel Nederlof proposed openstack/nova master: Add ability to flatten RBD disks upon clone  https://review.opendev.org/c/openstack/nova/+/88459513:54
opendevreviewMichel Nederlof proposed openstack/nova master: Add ability to flatten RBD disks upon clone  https://review.opendev.org/c/openstack/nova/+/88459514:01
gibibauzas: I might need to skip today's meeting or disappeare in the middle15:22
bauzasack15:22
bauzasreminder : nova meeting in 5 mins here15:55
bauzas#startmeeting nova16:00
opendevmeetMeeting started Tue May 30 16:00:08 2023 UTC and is due to finish in 60 minutes.  The chair is bauzas. Information about MeetBot at http://wiki.debian.org/MeetBot.16:00
opendevmeetUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.16:00
opendevmeetThe meeting name has been set to 'nova'16:00
bauzas#link https://wiki.openstack.org/wiki/Meetings/Nova#Agenda_for_next_meeting16:00
bauzaswelcome folks16:00
elodilleso/16:00
dansmithOj16:02
bauzasmmm16:02
bauzasdoesn't sound a lot of people around16:02
bauzasbut we can make it a slow start and hopefully people will join16:03
bauzas#topic Bugs (stuck/critical) 16:04
bauzas#info No Critical bug16:04
bauzas#link https://bugs.launchpad.net/nova/+bugs?search=Search&field.status=New 15 new untriaged bugs (+0 since the last meeting)16:04
bauzasauniyal: any bug you wanted to discuss ?16:04
bauzaslooks he's not arounf16:05
bauzasmoving on16:05
bauzas#info Add yourself in the team bug roster if you want to help https://etherpad.opendev.org/p/nova-bug-triage-roster16:05
bauzas#info bug baton is being passed to bauzas16:05
bauzasok, next topic then16:05
bauzas#topic Gate status 16:05
bauzas#link https://bugs.launchpad.net/nova/+bugs?field.tag=gate-failure Nova gate bugs 16:05
bauzas#link https://etherpad.opendev.org/p/nova-ci-failures16:05
bauzasI'll be honest, gerrit was a bit off my radar since the last week16:06
bauzasI plead guilty but I have an OpenInfra prezo to prepare16:06
bauzasany CI failures people wanna discuss ?16:06
dansmithgate has been not too bad I think16:06
bauzasnice16:07
* gibi lurks16:07
bauzasthe very few gerrit emails I glanced were indeed positive16:07
bauzaslet's assume we're living on a quiet world and move on16:07
Uggla_o/16:07
bauzas#link https://zuul.openstack.org/builds?project=openstack%2Fnova&project=openstack%2Fplacement&pipeline=periodic-weekly Nova&Placement periodic jobs status16:07
bauzasthe recent failures we've seen on the periodics seem to be resolved ^16:08
bauzasall of them are green16:08
bauzasso I guess it's fixed16:08
bauzas#info Please look at the gate failures and file a bug report with the gate-failure tag.16:08
bauzas#info STOP DOING BLIND RECHECKS aka. 'recheck' https://docs.openstack.org/project-team-guide/testing.html#how-to-handle-test-failures16:08
bauzasvoila for this topic, anything else to address on that ?16:09
bauzas-16:09
bauzas#topic Release Planning 16:09
bauzas#link https://releases.openstack.org/bobcat/schedule.html16:09
bauzas#info Nova deadlines are set in the above schedule16:09
bauzas#info Nova spec review day next week !16:10
bauzasI should write in caps actually16:10
bauzasUPLOAD YOUR SPECS, AMEND THEM, TREAT THEM, MAKE THEM READY16:10
elodillesyepp, a reminder on that day might help ;)16:10
bauzaselodilles: indeed16:10
bauzas#action bauzas to notify -discuss@ about the spec review day16:11
bauzasI've seen a couple of new specs 16:12
bauzasI'll slowly make a round in advance if I have time16:12
bauzasnext topic if nothing else16:12
bauzas,16:12
bauzas#topic Vancouver PTG Planning 16:12
bauzas#info please add your topics and names to the etherpad https://etherpad.opendev.org/p/vancouver-june2023-nova16:12
bauzasso, I created an etherpad (actually I reused the one automatically created)16:13
bauzasgiven it will be an exercice to guess who's around and when, I'd love if people could chime into this etherpad their ideas of topics and ideally mention their presences16:13
bauzasalso, if people not able to join wanna bring some topics worth discussing at the PTG, that'd be nice16:14
bauzasnot sure we'll have a quorum, but I just hope we could somehow try to have some kind of synchronous discussion at the PTG16:15
bauzasso we would capture the outcome of such discussions for following them up after the PTG16:15
bauzasI won't lie, that'll be a challenge anyway.16:16
bauzasworth helping,16:16
bauzas#info The table #24 is booked for the whole two days. See the Nova community there !16:16
bauzasso, yeah, I should stick around this table for the two days, except when I need to present a breakout session or when I have a Forum session to moderate, or depending on my bath needs16:17
bauzas(the last one being a transitive result of the number of coffee shots I'll take)16:18
* gibi drops16:18
bauzasanyway, the word is passed.16:18
bauzas#topic Review priorities 16:18
bauzas#link https://review.opendev.org/q/status:open+(project:openstack/nova+OR+project:openstack/placement+OR+project:openstack/os-traits+OR+project:openstack/os-resource-classes+OR+project:openstack/os-vif+OR+project:openstack/python-novaclient+OR+project:openstack/osc-placement)+(label:Review-Priority%252B1+OR+label:Review-Priority%252B2)16:18
bauzasgibi: \o16:18
bauzas#info As a reminder, cores eager to review changes can +1 to indicate their interest, +2 for committing to the review16:18
bauzas#topic Stable Branches 16:18
bauzaselodilles: wanna bring some points ?16:19
elodilleso716:19
elodilles#info stable/train is unblocked, as openstacksdk-functional-devstack job is fixed for train16:19
bauzashuzzah16:19
elodilles\o/16:19
elodillesso:16:19
elodilles#info stable gates should be OK16:19
bauzasthat'll somehow change a bit of the next topics we'll discuss16:19
elodilles#info stable branch status / gate failures tracking etherpad: https://etherpad.opendev.org/p/nova-stable-branch-ci16:19
elodillesthat's all from me16:20
bauzaselodilles: thanks16:20
bauzasand yeah, I added a bullet point16:20
bauzasbut it seems to me we don't have quorum today, so I'll reformulate16:20
bauzasbased on the ML thread https://lists.openstack.org/pipermail/openstack-discuss/2023-May/033833.html,16:21
bauzasdo people think it's reasonable to EOL stable/train ?16:21
dansmithpersonally I do16:22
bauzasI saw the discussion that happened while I was away, and I wanted to reply this morning16:22
bauzasbut I preferred to defer any reply after this meeting16:22
dansmiththe point of keeping those around was so communities could form around them if needed,16:22
dansmithbut if even we (redhat) aren't keeping it up to date, I think that's a good sign that it should go away16:22
bauzasdansmith: well, some folks are continuing to backport a few things to train16:23
bauzasand we have some stable cores that do reviews on that branch16:23
bauzasbutn,16:23
dansmithbut not critical CVEs16:23
bauzasas I said in the email, the two critical CVEs don't have a path forward16:23
bauzasdansmith: this is unfortunately the problem16:23
dansmithokay, I see there's more traffic than I expected16:24
dansmiththen I guess I don't care16:24
bauzasthat's not exactly that I'm against backporting the VMDK fix16:24
dansmithbut I do think it's confusing for people to see a community-maintained repo that is missing such large fixes16:24
bauzasbut we'll break olso.utils semver if we do so16:24
dansmithyeah, idk, I lean towards EOLing16:25
bauzasfor the other CVE (the brick one), I'd say I don't know how much it would be difficult to propose a backport16:25
dansmiththe brick fix is not going to be backported AFAIK16:25
dansmithand I don't think the cinder one (which is the most important) is being backported past xena16:25
dansmithour part of the fix for that CVE is pretty minor and we could backport it, but without the FC part of the brick fix it's not complete16:26
bauzasthat's why I personnally too lean forward to EOLing stable/train16:26
dansmithonly debian spoke up about/agains EOLing right/16:26
bauzascorrect16:26
bauzasI can reply and explain the problem again16:27
dansmithshrug16:27
dansmithcinder does still have a train branch16:27
dansmithand the last thing there was the VMDK fix16:27
dansmithso does brick, but the last thing was last summer16:28
elodillesi also stated that we could keep open train as long as the gate is working, though indeed it is unfortunate that CVE fixes doesn't get backported :/16:28
dansmithidk, almost seems like there's not enough people to care to even push us one way or the other :)16:28
bauzasI would indeed be more inclined if some other project like cinder would make the same step than us quite at the same time16:28
bauzasdansmith: elodilles: honestly I guess I'll propose a patch for -eol and people can -1 if they care16:29
bauzasthat's probably where we'll capture most of the concerns16:29
dansmithack16:29
bauzasor we could round forever16:29
elodilles++16:29
bauzas#action bauzas to propose a gerrit change for tagging stein-eol so people could vote on it 16:30
elodillesyou mean train-eol :)16:30
bauzasdamn16:30
bauzas#undo16:30
opendevmeetRemoving item from minutes: #action bauzas to propose a gerrit change for tagging stein-eol so people could vote on it 16:30
elodillesstein-eol is long gone for nova ;)16:30
bauzas#action bauzas to propose a gerrit change for tagging train-eol so people could vote on it 16:30
bauzaselodilles: my brain fscked16:30
bauzasanyway, I guess we're done on this16:31
bauzas#topic Open discussion 16:31
elodilles++16:31
bauzasgeguileo: you had a point :)16:31
bauzas(geguileo) Change to os-brick's connect_volume idempotency 16:31
geguileobauzas: yes, thanks16:31
bauzas#link https://review.opendev.org/c/openstack/os-brick/+/88284116:31
bauzas#link https://bugs.launchpad.net/nova/+bug/202069916:31
geguileoSo with the latest CVE on SCSI volumes we decided in os-brick to make some changes16:32
geguileoSpecifically os-brick would remove any existing devices before starting the connect_volume code16:32
geguileoand then proceeding with the actual attachment16:32
geguileothis means that connect_volume will no longer be idempotent16:33
geguileo(which is never something we promised would be)16:33
geguileothat seems to break some Nova operations16:33
geguileoparticularly the rescue/unrescue operations16:33
dansmithis there some way we can check to see if we need to run connect_volume()?16:34
bauzasgeguileo: thanks for spotting this in advance16:34
dansmithbecause otherwise it's hard to know if we're restarting after a recovery or just a service restart16:34
dansmithwhether or not we should run that16:34
bauzasyeah that's one of the problems16:35
geguileodansmith: that could be solved if Nova didn't stash and then unstash the instance config16:35
geguileoand instead rebuilt the instance config16:35
dansmithmeaning the guest xml?16:35
geguileoI believe sean-k-mooney[m] mentioned that16:35
geguileodansmith: yes16:35
bauzasgeguileo: we also recreate the XML on stop/start fwiw16:36
dansmithokay, but if we're running on a readonly-root host or recovering from a disaster, none of that would be available after a restart for us16:36
dansmithyeah, the guest XML isn't something we can assume hangs around long-term IMHO16:36
geguileodansmith: could nova know if it's attaching the volume for the recovery instance on the same host that was running the instance?16:36
dansmithmost of nova is designed to assume that it's tempoary16:36
bauzasyeah and in general, we don't consider libvirt a source of truth for persisting some instance metadata16:36
dansmithbauzas: right16:37
dansmithgeguileo: for the rescue case specifically? if the instance was already stopped and you go straight to rescue, we wouldn't really know when the last time the connect was run.. could have been weeks ago and multiple host reboots since16:38
bauzaswould the spec on dangling volumes helping the problem ?16:38
dansmithbauzas: I don't think so16:38
bauzasI mean, our BDM/volume reconciliation 16:38
dansmiththat's not the problem, it's the underlying host devices16:38
bauzasyeah I remember the context of the CVE :)16:38
bauzasand I guess only brick knows whether there is a residue ?16:39
dansmithgeguileo: so let's say we reboot with one instance stopped that points to /dev/vdf,16:39
dansmiththen we reboot, and someone spawns a new instance, we call connect_volume() for that new instance, it gets /dev/vdf,16:39
dansmiththen the user starts the old was-stopped instance, we can't look at the instance xml config and know that the volume is wrong right?16:40
dansmithbecause /dev/vdf exists, but it's no longer relevant for this instance16:40
geguileodansmith: well, Nova could check the ID of the volume and see if it matches the information returned by os-brick16:40
dansmithpoint being, there could be multiple instances with disks that point to stale host devices at any given point..16:41
geguileobut I don't know if we want Nova to be on the business of doing those checks16:41
dansmithyeah, ideally not16:41
geguileodansmith: yes, that could happen16:41
bauzascould nova ask brick such thing ?16:42
dansmithgeguileo: maybe everywhere we currently do connect_volume() we do a disconnect first and ignore an error? that generates a lot of churn, but maybe that's safer?16:42
dansmithbauzas: that's what I was wondering.. if there was some validate_volume() call or something we could run to know if we should run connect16:42
bauzaslike, before doing connect_volume(), do a check_vol()16:42
bauzasahah16:42
geguileodansmith: os-brick will be doing that already with optimized code16:42
geguileodansmith: that's the issue that I'm bringing, that the new code that cleans things up break the idempotency, so there's no need for nova to do that call you mention16:43
geguileoos-brick cannot do a proper check_vol like we would want16:43
geguileobecause the volume can have changed in the background16:43
geguileofor example change the size, point a different volume in the backend (like happened in the CVE)16:44
dansmithokay, I'm just not sure how we can know what the right thing to do is16:44
geguileothere are a bunch of messes that could happen, and I'm not sure we would want this to depend on nova calling a method16:44
geguileodansmith: well, if you call connect_volume all instances in Nova should use the device it returns from that moment onwards16:45
bauzasgeguileo: I'm confused, who's responsible for knowing that all connectors are present on the host ?16:45
bauzasI thought it was brick16:45
dansmithgeguileo: okay I thought you didn't want us to do that16:45
geguileowhat do you mean by connectors?16:45
bauzaswrong wording, my bad.16:45
dansmithgeguileo: or are you just saying any time we call connect_volume() we need to examine the result and make sure the instance is (changed, if necessary) to use that/16:45
bauzasgeguileo: I'd say the device number16:46
geguileodansmith: what I don't want is nova to call connect_volume, get the value and stash it (/dev/sdc), then call connect_volume again (/dev/sda) to use it in the rescue instance, and then use the stashed value (/dev/sdc) that no longer exists16:46
geguileodansmith: yes, that's it16:47
dansmithgeguileo: okay16:47
geguileodansmith: if connect_volume is called, make sure that is used in all the instances that use the same cinder volume on that host16:47
dansmithsounds like the solution is that we need to call connect_volume every time we're going to start an instance for any reason, and update the XML (rescue or otherwise) to use that value before we start16:47
geguileodansmith: that would be slower but safe16:48
geguileothe optimised mechanism where we could tweak Nova to only do that when necessary could lead to unintended bugs16:49
bauzasyeah, I was hoping some mechanism that would prevent the unnecessary roundtrip but if that's hard, then...16:49
dansmithyeah I just don't know how we could do it safely otherwise, unless there's something that maintains an exclusive threadsafe set of host devices to make sure we never get them confused16:49
geguileodansmith: if the host doesn't reboot and connect_volume is not called again, then we could assume that the device is correct16:50
bauzasgeguileo: but we don't know thisq16:50
bauzasgeguileo: we only know about the service restart16:50
geguileobauzas: could nova query the uptime or something?16:50
dansmithno :)16:50
geguileook, then the whole connect_volume is probably the only way  :'-(16:51
dansmithlet's not do anything like that for host or service uptime.. I think we just need to do the safe/slow approach16:51
bauzasgeguileo: we have other host-dependent resources like mdevs that we treat this way without requiring to know if it's a reboot16:51
geguileoso I guess that's the solution then?16:53
dansmithI think so.. not ideal for sure16:53
bauzasin general, we follow some workflow which is "lookup my XML, find the specific resources attached, then query whatever needed to know whether those still exist, and if not, ask whatever else to recreate them"16:53
bauzas(at service restart I mean)16:54
geguileowith changes to cinder drivers we could have a proper check if it's valid, or have the connect-volume be faster (and idempotent when it can)16:54
geguileothey would need to provide extra information to validate the device16:54
dansmithlet's do the slow/safe thing now, and if connect_volume() can be idempotent and faster in the future, then that's cool16:55
geguileodansmith: sounds like a plan16:55
geguileosince that way the newer approach would probably not require additional nova changes16:55
geguileojust os-brick and cinder16:55
dansmithyep16:55
geguileook, then I have nothing else to say on the topic16:56
dansmithgeguileo: unrelated, but since you're here.. what's cinder's plan for stable/train?16:57
dansmithI assume no backports of this CVE back that far right?16:57
dansmithand are you all thinking about EOLing at some point?16:57
geguileodansmith: definitely no backports that far16:57
dansmithwe have concerns about keeping branches open that look maintained but don't have backports of high-profile CVEs (two for us now)16:57
geguileoI believe at some point it was discussed to stop supporting anything before Yoga16:58
dansmithack16:58
bauzasanyway, we're on time 16:58
bauzasthe nova meeting is ending in 1 min16:58
dansmiththanks geguileo 16:59
geguileothank you all16:59
bauzasgeguileo: dansmith: I guess we've arrived on a conclusion16:59
bauzasthanks both of you16:59
bauzasand thanks all16:59
bauzasif nothing else,16:59
bauzasbye16:59
bauzas#endmeeting16:59
opendevmeetMeeting ended Tue May 30 16:59:59 2023 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)16:59
opendevmeetMinutes:        https://meetings.opendev.org/meetings/nova/2023/nova.2023-05-30-16.00.html16:59
opendevmeetMinutes (text): https://meetings.opendev.org/meetings/nova/2023/nova.2023-05-30-16.00.txt16:59
opendevmeetLog:            https://meetings.opendev.org/meetings/nova/2023/nova.2023-05-30-16.00.log.html16:59
elodillesthanks o/17:00
stephenfinDidn't bring it up in the meeting cos it doesn't belong there, but another round of +2s on the outstanding sqlalchemy-20 stuff would make my week and get us towards unblocking that in Bobcat. sean-k-mooney[m] has already spun through them17:01
stephenfinAll can be found here https://review.opendev.org/q/project:openstack/nova+topic:sqlalchemy-20+is:open17:01
stephenfinHappy to answer any questions you may have on any/all of them17:01
bauzasstephenfin: ack, no promises but I'll try to review them soon17:02
bauzasstephenfin: bump me again next week if not17:03
stephenfinwill do, thanks bauzas17:03
opendevreviewDan Smith proposed openstack/nova master: Populate ComputeNode.service_id  https://review.opendev.org/c/openstack/nova/+/87990417:06
opendevreviewDan Smith proposed openstack/nova master: Add compute_id columns to instances, migrations  https://review.opendev.org/c/openstack/nova/+/87949917:06
opendevreviewDan Smith proposed openstack/nova master: Add dest_compute_id to Migration object  https://review.opendev.org/c/openstack/nova/+/87968217:06
opendevreviewDan Smith proposed openstack/nova master: Add compute_id to Instance object  https://review.opendev.org/c/openstack/nova/+/87950017:06
opendevreviewDan Smith proposed openstack/nova master: Online migrate missing Instance.compute_id fields  https://review.opendev.org/c/openstack/nova/+/87990517:06
opendevreviewDan Smith proposed openstack/nova master: Add online migration for Instance.compute_id  https://review.opendev.org/c/openstack/nova/+/88475217:06
dansmithSo, the last GPF kernel crash we had was a week ago17:24
dansmithit was on the new cirros but it doesn't look exactly like what we saw before17:25
dansmithbut it does look related to acpi hotplug, which I assume is related to the volume attach or detach, and _is_ in one of those volume-having tests17:25
dansmithso at this point I'd say the new cirros didn't fix it, but we'll have to wait for a few weeks of data to see if it's better or different17:26
fricklerdansmith: do you have a link to that crash handy? also we're right now publishing 0.6.2 with an updated kernel, that might help https://github.com/cirros-dev/cirros/actions/runs/512391818717:44
dansmithfrickler: here's the 0.6.1 one: https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_ad6/826755/16/check/nova-lvm/ad65569/job-output.txt17:45
dansmithsearch for "general protection"17:45
opendevreviewSylvain Bauza proposed openstack/nova stable/xena: Fix get_segments_id with subnets without segment_id  https://review.opendev.org/c/openstack/nova/+/88372517:48
opendevreviewSylvain Bauza proposed openstack/nova stable/xena: Fix get_segments_id with subnets without segment_id  https://review.opendev.org/c/openstack/nova/+/88372517:50
opendevreviewSylvain Bauza proposed openstack/nova stable/yoga: Fix get_segments_id with subnets without segment_id  https://review.opendev.org/c/openstack/nova/+/88372417:53
opendevreviewSylvain Bauza proposed openstack/nova stable/xena: Fix get_segments_id with subnets without segment_id  https://review.opendev.org/c/openstack/nova/+/88372517:55
opendevreviewSylvain Bauza proposed openstack/nova stable/wallaby: Fix get_segments_id with subnets without segment_id  https://review.opendev.org/c/openstack/nova/+/88372618:02
opendevreviewMerged openstack/nova master: db: Remove the legacy 'migration_version' table  https://review.opendev.org/c/openstack/nova/+/87242920:57
melwittjjm9ASDFGHJJKOL;;;;;''''21:50
melwittgeguileo: w2f21:51
melwittargh sorry21:51
dansmithnoice21:56
melwitt😆 21:59

Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!