*** tosky has quit IRC | 00:02 | |
TheJulia | https://bugs.chromium.org/p/gerrit/issues/detail?id=9740 | 00:02 |
---|---|---|
fungi | not sure, i suppose you've got a bunch you want to clean up? | 00:05 |
TheJulia | Yeah, tons orphaned from before the last upgrade | 00:08 |
TheJulia | but, that link has the command in it | 00:08 |
fungi | ahh, cool | 00:09 |
openstackgerrit | Merged opendev/system-config master: Increase some of Gerrit's in-memory cache pools https://review.opendev.org/c/opendev/system-config/+/764779 | 00:11 |
fungi | TheJulia: reading through it, supposedly there was at one time also a dashboard banner which had a button to bulk delete draft comments | 00:13 |
*** sboyron has quit IRC | 00:23 | |
TheJulia | There could have been, and maybe it went poof at some point... | 00:24 |
fungi | yeah, maybe after some number of releases they removed it because what are the odds someone's going to upgrade a site straight from <2.16 to whatever point that got removed in? | 00:31 |
fungi | so it eventually became noise for sites which had long since been running a version which showed you all your unsubmitted draft comments | 00:31 |
TheJulia | Yeah | 00:32 |
* TheJulia looks at the tons of line breaks in test results and may need to get her eyes chcked | 00:32 | |
fungi | it will take time to get everything back in shape, yes | 00:33 |
openstackgerrit | Merged opendev/system-config master: Disable Gerrit's automatic Git GC on push https://review.opendev.org/c/opendev/system-config/+/764807 | 01:05 |
openstackgerrit | likui proposed openstack/diskimage-builder master: update tox https://review.opendev.org/c/openstack/diskimage-builder/+/759384 | 01:41 |
fungi | changeid_project cache is now over 8192 since the restart | 03:09 |
*** sgw has quit IRC | 03:09 | |
*** larainema has joined #opendev | 03:45 | |
larainema | https://review.opendev.org/c/openstack/project-config/+/763742 can someone help merge this patch? | 03:50 |
*** sgw has joined #opendev | 04:33 | |
*** icey has quit IRC | 06:16 | |
*** hamalq has quit IRC | 06:21 | |
*** hamalq has joined #opendev | 06:22 | |
*** marios has joined #opendev | 06:32 | |
*** zbr has quit IRC | 06:52 | |
*** sboyron has joined #opendev | 06:52 | |
*** zbr has joined #opendev | 06:52 | |
*** ysandeep|away is now known as ysandeep | 06:53 | |
*** icey has joined #opendev | 07:01 | |
*** ralonsoh has joined #opendev | 07:37 | |
*** eolivare has joined #opendev | 07:42 | |
openstackgerrit | Merged openstack/project-config master: Remove ceilometer-zvm entry https://review.opendev.org/c/openstack/project-config/+/763742 | 07:45 |
*** slaweq has joined #opendev | 07:52 | |
*** fressi has joined #opendev | 07:57 | |
*** rpittau|afk is now known as rpittau | 07:59 | |
*** fressi has quit IRC | 08:02 | |
*** fressi has joined #opendev | 08:02 | |
*** andrewbonney has joined #opendev | 08:03 | |
*** hashar has joined #opendev | 08:10 | |
*** tosky has joined #opendev | 08:40 | |
*** mgoddard has joined #opendev | 09:36 | |
*** DSpider has joined #opendev | 09:50 | |
*** zbr has quit IRC | 09:51 | |
*** zbr has joined #opendev | 09:56 | |
*** chengsheng1 has joined #opendev | 10:18 | |
chengsheng1 | hello | 10:20 |
*** zbr has quit IRC | 10:58 | |
*** zbr has joined #opendev | 10:58 | |
*** zbr has quit IRC | 11:00 | |
*** zbr has joined #opendev | 11:00 | |
*** zbr has quit IRC | 11:03 | |
*** dtantsur|afk is now known as dtantsur | 11:04 | |
*** zbr has joined #opendev | 11:04 | |
*** zbr has quit IRC | 11:07 | |
*** zbr has joined #opendev | 11:17 | |
gibi | clarkb, fungi: I dropped the keybord at that time already. And so far I'm in review mode so I did not push any changes today. I will let you know if I see issues with push | 11:26 |
*** mgoddard has quit IRC | 11:55 | |
*** iurygregory|pto has joined #opendev | 12:08 | |
*** waverider has joined #opendev | 12:16 | |
*** eolivare_ has joined #opendev | 12:21 | |
*** eolivare has quit IRC | 12:24 | |
gibi | fungi: one addition to the yesterday gerrit load issue is that I was able to push a series of patches yesterday that resulted in an internal error but the commits ended up in gerrit. Today I realized that CI was not triggere for them so I assume that the internal error was in somewhere the event emitting part of gerrit | 12:26 |
gibi | fungi: this is the chain https://review.opendev.org/q/topic:%22bp%252Fsupport-interface-attach-with-qos-ports%22+(status:open%20OR%20status:merged) | 12:26 |
*** brinzhang_ has joined #opendev | 12:46 | |
*** eolivare_ has quit IRC | 12:48 | |
*** brinzhang0 has quit IRC | 12:50 | |
*** eolivare_ has joined #opendev | 13:23 | |
*** TeamDevOps has joined #opendev | 13:27 | |
*** TeamDevOps has quit IRC | 13:27 | |
*** slaweq has quit IRC | 13:34 | |
*** slaweq has joined #opendev | 13:36 | |
*** sshnaidm has quit IRC | 13:38 | |
*** sshnaidm has joined #opendev | 13:46 | |
*** sshnaidm has quit IRC | 13:48 | |
*** mlavalle has joined #opendev | 13:58 | |
*** chrome0 has quit IRC | 14:04 | |
*** chrome0 has joined #opendev | 14:06 | |
*** fressi has left #opendev | 14:43 | |
*** ysandeep is now known as ysandeep|away | 15:07 | |
*** waverider has quit IRC | 15:16 | |
*** hashar is now known as hasharAway | 15:20 | |
fungi | looking at the load graph for gerrit, i don't think expanding the caches or disabling autogc have necessarily helped | 15:20 |
fungi | we've nearly (but not quite yet) filled the new changeid_project cache, but hit ratio is still not particularly good (35%) | 15:22 |
fungi | no more "missing tree" errors today though | 15:22 |
*** TeamDevOps99 has joined #opendev | 15:23 | |
fungi | so maybe it's somewhat of an improvement | 15:23 |
*** TeamDevOps99 has quit IRC | 15:23 | |
lourot | Hi o/ we have been having intermittently today the following error: https://zuul.opendev.org/t/openstack/build/7a2b8ffdad83412abed573f250994b13 | 15:26 |
fungi | gibi: thanks, i tried looking into that one yesterday when you mentioned it and didn't find anything fatal in the log, but i think i saw at least one anomalous warning related to one of your pushes around that time | 15:27 |
lourot | this is not happening all the time, so we suspect it depends on what node the zuul job runs | 15:27 |
lourot | also this is not due to pip 20.3 published yesterday we think as we get other errors that we are solving already because of pip 20.3 | 15:27 |
lourot | does that ring a bell? | 15:28 |
gibi | fungi: no worries, I could trigger the CI with a recheck. | 15:29 |
fungi | lourot: pip tries to avoid retrieving wheels which are marked as not supporting the version of python you're running under, but if it gets a copy of the pypi index which lacks the necessary metadata for some reason then it will fall back to just fetching the newest version of the package which satisfies your requested requirement | 15:30 |
fungi | we've seen this in the past when the fastly cdn pypi uses can't reach the primary warehouse servers for some reason and winds up retrieving and caching an index from their backup bandersnatch-built mirror which doesn't include the python version metadata | 15:31 |
fungi | so my initial guess is that the fastly cdn endpoint near some of our cloud providers is having trouble getting to the primary copy of the pypi index | 15:32 |
fungi | this is usually transient, but we've sometimes seen it persist for days before they fix it | 15:32 |
lourot | ok thanks fungi | 15:33 |
fungi | lourot: if you're running openstack projects under python 3.5, you really should be using requirements and constraints contemporary with the last release which supported that | 15:34 |
lourot | yes I'll tweak the constraints, thanks! | 15:35 |
fungi | lourot: be aware that it's going to be fairly old openstack software... even stein was not tested with python 3.5: https://governance.openstack.org/tc/reference/runtimes/stein.html | 15:36 |
lourot | fungi, yes we still support Ubuntu 14.04 (up to Mitaka) and 16.04 (up to Queens) and they have python 3.5, but we have other CI jobs for this and I wonder if we really need the zuul jobs to be running python 3.5 | 15:39 |
lourot | gnuoy, maybe that's just a matter of adapting the .zuul.yaml files to not have the zuul jobs running on python 3.5? ^ | 15:40 |
lourot | yes indeed it looks like we have slightly outdated .zuul.yaml files pointing explicitly to py35 in the problematic repos | 15:41 |
lourot | thanks for the input fungi! | 15:42 |
fungi | you're welcome! | 15:42 |
*** whoami-rajat__ has joined #opendev | 16:00 | |
*** sshnaidm has joined #opendev | 16:05 | |
*** TeamDevOps43 has joined #opendev | 16:07 | |
*** TeamDevOps43 has quit IRC | 16:07 | |
fungi | changeid_project | 32768 | | 36% | 16:13 |
fungi | we just filled it in the last few minutes | 16:13 |
fungi | now to see if the cache hit rate improves or gets worse | 16:13 |
fungi | i'm noticing that show-caches doesn't list a couple of the caches we're overriding in our gerrit.config: accounts_byemail and accounts_byname | 16:18 |
clarkb | fungi: maybe those caches have gone away? most of the caches are documented in the gerrit docs but not all | 16:19 |
clarkb | could cross check against that I guess | 16:19 |
fungi | also the accounts cache is disk-backed, so i don't know that our memorylimit override for that is relevant any longer | 16:19 |
*** marios is now known as marios|out | 16:46 | |
*** rpittau is now known as rpittau|afk | 17:01 | |
*** marios|out has quit IRC | 17:13 | |
openstackgerrit | Ghanshyam proposed openstack/project-config master: Update retiring Qinling repo ACL to openstack/retired.config https://review.opendev.org/c/openstack/project-config/+/764532 | 17:28 |
openstackgerrit | Ghanshyam proposed openstack/project-config master: Remove Qinling projects from infra https://review.opendev.org/c/openstack/project-config/+/764536 | 17:32 |
openstackgerrit | Ghanshyam proposed openstack/project-config master: Update retiring Searchlight repo ACL to openstack/retired.config https://review.opendev.org/c/openstack/project-config/+/764534 | 17:35 |
openstackgerrit | Ghanshyam proposed openstack/project-config master: Remove Searchlight projects from infra https://review.opendev.org/c/openstack/project-config/+/764535 | 17:39 |
*** dtantsur is now known as dtantsur|afk | 17:55 | |
*** d34dh0r53 has quit IRC | 18:13 | |
*** andrewbonney has quit IRC | 18:40 | |
*** d34dh0r53 has joined #opendev | 18:54 | |
*** eolivare_ has quit IRC | 19:04 | |
*** sshnaidm has quit IRC | 19:15 | |
*** sshnaidm has joined #opendev | 19:18 | |
ianw | the cert on review test expired 11/16/2020 , can we refresh it? | 19:42 |
clarkb | ianw: it is an LE cert so yes, except I don't know that we can run ansible against it in its current state | 19:42 |
fungi | ianw: "we" can, what's the easiest way there? (noting that we don't want to blow away some of the specific gerrit configs there) | 19:43 |
clarkb | ianw: not sure if the best thing there is to get ansible in sync somehow, or to jsut do the LE dance manualyl (or maybe we can just run that playbook? | 19:43 |
ianw | could probably just run the le playbook quickly with review-test un-emergencied | 19:43 |
ianw | there's no real "manual" way to just run LE on the host because you need to deploy the DNS records | 19:43 |
clarkb | ianw: rather than unmergency it, it might be better to use a copy of the playbook without !disabled and then limit it to the relevant hosts | 19:43 |
clarkb | that way we don't have to worry about our gerrit config updates and image changes being switched on us | 19:44 |
ianw | yep, that could work too | 19:44 |
clarkb | also if we want to be lazy I can go spend $8 for a cert | 19:44 |
ianw | fwiw i've clicked around on review-test and everything works as expected for me | 20:01 |
ianw | fungi / clarkb: I forgot to mention in the zuul discussion a couple of fixes i found https://review.opendev.org/q/topic:%22gerrit-ssh%22+(status:open%20OR%20status:merged) | 20:02 |
ianw | maybe if we look at those, then i can hold a fresh node and fungi can maybe help me with the steps to add a admin, project, change, fake zuul user and a fake zuul comment. that would be a minimal framework we can expand on | 20:08 |
fungi | once i get the jgit config pushed i need to work on cooking dinner, but can look after that | 20:09 |
*** hasharAway is now known as hashar | 20:14 | |
openstackgerrit | Jeremy Stanley proposed opendev/system-config master: Move jgit autogc disablement into correct config https://review.opendev.org/c/opendev/system-config/+/765004 | 20:23 |
fungi | infra-root: ^ | 20:23 |
fungi | once that's deployed we can restart and then look at openjdk 11 as our next step | 20:24 |
mordred | fungi: +2 - +A at will | 20:36 |
fungi | thanks! | 20:36 |
*** savihou has joined #opendev | 20:56 | |
openstackgerrit | Jeremy Stanley proposed opendev/system-config master: Move jgit autogc disablement into correct config https://review.opendev.org/c/opendev/system-config/+/765004 | 20:56 |
fungi | mordred: clarkb: ^ updated based on feedback | 20:57 |
ianw | per clarkb's comment in 764396 i'm just going to restart review-test with a db/ dir mount, to make sure it doesn't do anything with it | 20:57 |
fungi | ianw: sounds good | 21:00 |
*** slaweq has quit IRC | 21:47 | |
*** slaweq has joined #opendev | 21:49 | |
clarkb | ianw: on my bike ride I also realized that we should check there isn't a dir created there already (not bind mounted) that we would mask off with a bind mount | 22:07 |
clarkb | if that makes sense | 22:07 |
clarkb | ianw: however if we don't bind mount it then that content would be ephemeral anyway? so I think we're like 99% fine | 22:09 |
ianw | clarkb: don't think there is, but let me jump in container | 22:09 |
ianw | drwxr-xr-x 2 root root 4096 Nov 17 20:47 db | 22:10 |
clarkb | ianw: ya I think I realized that it would have to be ephemeral content anyway after thinking about it on the bike a bit | 22:10 |
clarkb | so should be safe | 22:10 |
clarkb | fungi: tox linters failed on teh jgit change | 22:11 |
clarkb | fungi: https://zuul.opendev.org/t/openstack/build/ac85ac8eeb544704b8dee0360b99b1a9/log/job-output.txt#757 I think it may be a legit error | 22:12 |
clarkb | fungi: ya mode is listed twice | 22:12 |
clarkb | I made note of the issue with a -1 on the change | 22:13 |
clarkb | pabelanger: we don't know why that happens, we think possibly beacuse your browser is telling gitea to use not english as the locale | 22:14 |
pabelanger | is it just me, or is https://opendev.org/zuul/zuul and https://opendev.org/openstack/project-config displaying in non english | 22:14 |
pabelanger | ah | 22:14 |
clarkb | at the bottom right is a globe icon, click on that and you can select english again | 22:14 |
pabelanger | okay, thanks | 22:15 |
pabelanger | some reason pyccnn is selected | 22:15 |
*** hashar has quit IRC | 22:16 | |
pabelanger | I just upgraded to f33, the other day, so likely related | 22:16 |
clarkb | pabelanger: in firefox you go to preferences -> general -> Language and double chekc the language tehre | 22:16 |
clarkb | Mine says English (United States) | 22:16 |
pabelanger | on chrome, checking its default | 22:17 |
pabelanger | hmm, english (canada) is default | 22:18 |
pabelanger | weird | 22:18 |
openstackgerrit | Clark Boylan proposed opendev/system-config master: Build Gerrit 3.3 images https://review.opendev.org/c/opendev/system-config/+/765021 | 22:36 |
clarkb | pabelanger: ya we haven't tracked that behavior down yet | 22:36 |
clarkb | pabelanger: if you figure out why it happened let us know | 22:36 |
clarkb | fungi: ^ when not looking at release notes things can you check your jgit.config change? I think it needs edits | 22:38 |
fungi | clarkb: thanks, sorry got busy with dinner | 22:39 |
clarkb | fungi: no worries, I got busy with a two wheeled mobility device | 22:40 |
corvus | fungi: (from #zuul) we don't seem to say that forge committer is required here: https://docs.opendev.org/opendev/infra-manual/latest/drivers.html#merge-commits | 22:47 |
corvus | (i'm trying to push up a merge commit, but it's being rejected with remote: ERROR: commit 713b533: email address review@openstack.org is not registered in your account, and you lack 'forge committer' permission. | 22:48 |
corvus | that commit is a pretty standard zuul merge commit, which i would expect to be on any branch in gerrit, so this should be a well-tested process for any feature branch, etc | 22:49 |
corvus | is there any chance this is a gerrit version behavior change? | 22:49 |
corvus | Author: Zuul <zuul@review.opendev.org> / Commit: Gerrit Code Review <review@openstack.org> | 22:50 |
clarkb | corvus: the docs for that permission state that this is usually only necessary when mirroring commits from an upstream repository. That makes me think the assertion fungi made is how it is expected to work (basically if gerrit is already aware of the commits then it is fine) | 22:51 |
clarkb | corvus: is it possible that gerrit isn't already aware of those commits for some reason (a rebase gone wrong?) | 22:51 |
fungi | so at one point it was possible to push a history with merges into the "wrong" branch and as long as the commits were already known in gerrit they would get added. this resulted in mishaps like people with dirty histories pushing master into a stable branch and all of master coming along for a ride. gerrit adjusted some of that in later versions so make it so that commits not already on the branch would not | 22:51 |
fungi | be silently added in a merge | 22:51 |
clarkb | ah fungi has it paged in | 22:52 |
fungi | though i thought to get that we needed to add receive.rejectImplicitMerges=false, which we haven't yet | 22:52 |
corvus | i found this: https://repo-discuss.narkive.com/Ps1VbayI/forge-committer-required-when-pushing-a-change-on-top-of-an-unsubmitted-merge-commit | 22:53 |
corvus | i'm pretty sure the commit history is clean, git-review shows me exactly what i expect | 22:53 |
fungi | thinking harder, receive.rejectImplicitMerges is about non-merge changes with fast-forwardable parents already in another branch | 22:54 |
corvus | but i am pushing a locally-authored change on top of the pending merge commit, which is what that link describes | 22:54 |
corvus | let me try only pushing up to the merge commit | 22:54 |
corvus | no joy, but there's still a locally authored commit *before* the merge commit | 22:55 |
fungi | it does still seem likely this is a behavior change between 2.13 and 3.2 i'm just not sure which one | 22:56 |
*** slaweq has quit IRC | 22:56 | |
corvus | it's possible that's affecting it too, but at this point even if that did work it would be a good solution (i'd really like to see this whole thing work before we start merging temporary changes) | 22:56 |
corvus | i wonder if i should just ask the opendev admins to add forge committer for this case? | 22:57 |
corvus | (or invoke project bootstrappers or something) | 22:57 |
fungi | it seems like forge committer would be reasonable if that's really now a requirement for adding other committers' commits to a different branch | 23:01 |
fungi | we could just update our documentation to say that's expected in combination with allowing merge commits | 23:02 |
corvus | i'd probably want to test it without the 'local commit sandwich' first before doing that; this situation may still be unusual | 23:02 |
clarkb | review-test would be a good place for that sort of stuff | 23:03 |
clarkb | though I ugess we can do it on sandbox too | 23:03 |
corvus | yeah, was thinking sandbox would be sufficient | 23:04 |
corvus | i could also probably just push up a throwaway change to zuul with a different changeid and see | 23:04 |
*** jaicaa has quit IRC | 23:04 | |
fungi | i wouldn't be surprised if this is the first time since the upgrade anyone's tested pushing a legitimate merge commit, and it's not something i thought to test when we were evaluating pre-upgrade | 23:05 |
corvus | okay, i get the same result when pushing up only the merge commit | 23:06 |
corvus | i really don't like the idea of generally adding forge committer :( | 23:06 |
clarkb | the docs do seem to imply that shouldn't be necessary? I wonder if we should file a bug? | 23:08 |
fungi | if it turns out to just be that the zuul merge commit situation is a weird one-off situation, then i'm also not opposed to the opendev sysadmin whose username starts with c and rhymes with corvus temporarily adding themselves to project bootstrappers to push it on behalf of the zuul contributors, but i'll readily acknowledge that's not scalable solution if it's a commonplace merge commit | 23:08 |
corvus | fungi: yeah, i'm running out of ideas of how this could be extraordinary at this point | 23:09 |
fungi | does sound likely to be a bug in that case | 23:10 |
corvus | the 3.19.1 tag i merged locally has the same sha that gerrit does (and master matches too) | 23:11 |
clarkb | fungi: were you going to push a new ps up for 765004? and if so are we reaching a point in the day where we should consider manually making that change and restarted gerrit? | 23:12 |
clarkb | or leave it to tomorrow I guess? | 23:12 |
fungi | sorry, yeah i | 23:14 |
fungi | 'm working on it now | 23:14 |
openstackgerrit | Jeremy Stanley proposed opendev/system-config master: Move jgit autogc disablement into correct config https://review.opendev.org/c/opendev/system-config/+/765004 | 23:16 |
corvus | i've gone through the git log and every change in 3.19.0..3.19.1 is in gerrit; so i'm doubly confident that isn't the issue | 23:17 |
fungi | yeah, it's definitely sounding like a bug, so temporarily adding your account to project bootstrappers to push it seems reasonable and we can file something upstream to get clarity either in (docs or fixing the access controls) | 23:19 |
fungi | parenthetical location error #37 | 23:19 |
corvus | okay, i'll do the bootstrap process | 23:20 |
corvus | wow there's now a new merge conflict according to git-review | 23:22 |
corvus | i'll redo the stack | 23:22 |
corvus | remote: ERROR: commit 713b533: missing Change-Id in message footer | 23:24 |
corvus | sigh | 23:24 |
corvus | something is really weird here :/ | 23:25 |
openstackgerrit | Clark Boylan proposed opendev/system-config master: Build Gerrit 3.3 images https://review.opendev.org/c/opendev/system-config/+/765021 | 23:26 |
clarkb | hoping ^ fixes an error with its-base building against 3.3 | 23:26 |
ianw | infra-root: response about the results box per discussion i'm working on @ https://etherpad.opendev.org/p/XwWVVI9zBa8LMSZFeQoz | 23:26 |
clarkb | corvus: just so I'm clear on what you are attempting to do: you have created a new zuul branch and are now attempting to push a merge commit to it as well as a couple of other changes? | 23:27 |
clarkb | corvus: but all of those commits have previously merged on other zuul branches? | 23:28 |
corvus | clarkb: not quite; i'm running 'git checkout master; git merge 3.19.1; git review' (essentially) | 23:28 |
corvus | 3.19.1 is a tag made from a no-longer-extant stable/3.x branch | 23:28 |
clarkb | I see | 23:28 |
corvus | clarkb: i wonder if the branch is required | 23:29 |
clarkb | oh that is a good qusetion | 23:29 |
corvus | like, maybe we need to resuscitate stable/3.x @ 3.19.1 then do this, then delete | 23:29 |
corvus | ianw: good summary i think | 23:29 |
clarkb | ianw: I like the point about using the checks plugin as a reference | 23:31 |
clarkb | and ++ on the testing plan | 23:31 |
fungi | corvus: great point. re-adding the stable/3.x might be the ticket, now that you mention it i wonder if that's the difference. some of the commits parented to that tag don't belong to a branch gerrit has | 23:31 |
ianw | that's about the most fleshed out example i can find | 23:31 |
fungi | (even though they are represented by gerrit changes) | 23:31 |
corvus | fungi, clarkb: shall i recreate the branch then? | 23:32 |
fungi | corvus: it literally can't hurt | 23:32 |
clarkb | corvus: I guess so, that idea does seem likely | 23:32 |
fungi | and it's a quick test | 23:32 |
fungi | also it's a good source of information in case we see this crop up again | 23:32 |
*** cgoncalves has quit IRC | 23:33 | |
*** cgoncalves has joined #opendev | 23:34 | |
fungi | on a related note, i've got a to do list entry reminding me to push up a change sometime soon for rejecting "implicit merges" now that we can... the situation where a normal (non merge commit) change proposed by someone can be parented on changes from another branch and gerrit doesn't even indicate anything's out of the ordinary | 23:44 |
fungi | we reported that problem years ago, we ought to at least make use of the fix finally | 23:45 |
corvus | ftr in this channel: the branch creation worked | 23:52 |
*** jaicaa has joined #opendev | 23:52 | |
corvus | fungi: nice | 23:52 |
*** DSpider has quit IRC | 23:54 | |
*** tosky has quit IRC | 23:57 |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!