| *** pojadhav|out is now known as pojadhav | 01:42 | |
| frickler | clarkb: good point. maybe then we just need to remove the installation of the system pkg here https://opendev.org/openstack/devstack/src/branch/master/lib/nova_plugins/functions-libvirt#L61 or make it optional | 06:29 |
|---|---|---|
| opendevreview | Dr. Jens Harbott proposed openstack/devstack master: Drop the installation of system pkg python3-libvirt on Ubuntu https://review.opendev.org/c/openstack/devstack/+/804950 | 06:37 |
| opendevreview | Dr. Jens Harbott proposed openstack/devstack master: Install libvirt-python with pip instead from system pkg https://review.opendev.org/c/openstack/devstack/+/804951 | 06:37 |
| frickler | rpittau|afk: iurygregory_: ^^ testing those options now against ironic, too, with https://review.opendev.org/c/openstack/ironic-python-agent/+/804953 | 06:40 |
| *** iurygregory_ is now known as iurygregory | 06:42 | |
| yoctozepto | frickler: going back and forth, are we? :D we have to decide on one of the approaches duh; also, if we decide on using the one from u-c, we should do it on all the platforms ;/ | 06:45 |
| frickler | yoctozepto: well, I'm always open to reasonable arguments. and consider those patches wip for now, guess I should flag them as such | 06:47 |
| yoctozepto | frickler: so am I, but going back and forth ain't gonna help :D I know ceilometer has already reacted; masakari patch (by me) on hold | 06:50 |
| yoctozepto | I will check if masakari passes with the new arrangement | 06:53 |
| yoctozepto | frickler, iurygregory, rpittau|afk: so, on that issue with ironic and libvirt - where does ironic actually touch libvirt? basically, from my kolla-based understanding we touch libvirt only in nova-compute, ceilometer-compute and masakari-instancemonitor (both variants) - all these components live on a machine with running libvirtd that they | 07:20 |
| yoctozepto | interact with | 07:20 |
| rpittau|afk | yoctozepto: ironic itself does not interact with libvirt, it's another component that uses the bindings, virtualbmc | 07:22 |
| *** rpittau|afk is now known as rpittau | 07:22 | |
| * iurygregory just noticed he didn't get the ping notifications =( | 07:24 | |
| yoctozepto | rpittau: ack, I have just completed a codesearch and was about to ask if it's virtualbmc ;D | 07:24 |
| rpittau | ;) | 07:24 |
| yoctozepto | ok, so ceilometer has actually kinda followed nova before and they have already been *not* including libvirt in requirements, yet they had some code duplication with devstack that started failing and they simply removed it https://review.opendev.org/c/openstack/ceilometer/+/804587 | 07:32 |
| yoctozepto | makes sense | 07:33 |
| yoctozepto | they mock libvirt in their unit tests | 07:33 |
| *** jpena|off is now known as jpena | 07:34 | |
| yoctozepto | so, your situation is more alike masakari-monitors | 07:35 |
| rpittau | unfortuantely in our case it's not just testing | 07:35 |
| rpittau | virtualbms is literally built on top of libvirt | 07:36 |
| yoctozepto | rpittau: so are the others, at least interesting parts of them; so there must be a trick to deliver the best experience for all projects; I need to learn how virtualbmc gets included in your jobs | 07:37 |
| yoctozepto | ah, I see now | 07:38 |
| yoctozepto | you are just installing it in devstack plugin | 07:38 |
| yoctozepto | ok, that's how it gets pulled in and then breaks on pip | 07:39 |
| rpittau | we could remove the system pkg before installing virtualbmc, as in one of the tested options | 07:42 |
| rpittau | isnide the install_virtualbmc function | 07:42 |
| rpittau | https://opendev.org/openstack/ironic/src/branch/master/devstack/lib/ironic#L932 | 07:43 |
| yoctozepto | based on the way I read your tests, you could also mock the whole libvirt away for that and then simply drop libvirt from reqs; that would make it work like nova-compute and ceilometer-compute work now; that could possibly be the most optimal route; the lightweight option I have found is to do like for masakari-monitors | 07:44 |
| yoctozepto | https://review.opendev.org/c/openstack/masakari-monitors/+/804913 | 07:44 |
| yoctozepto | I did not have time to mock it for masakari-monitors so doing it less elegantly | 07:44 |
| yoctozepto | rpittau: yeah, I would not do that though, to be more compatible with the base devstack jobs | 07:45 |
| yoctozepto | meh, forgot to add to test-requirements there | 07:47 |
| yoctozepto | perhaps I will just mock it away because it gets on my nerves with the back-and-forth nature | 07:48 |
| rpittau | yoctozepto: our priority right now is to unlock the CI, not sure I have the time to do a radical change, I can propose both, removing the sys pkg as temp solution, and then work on the virtualbmc part itself as long term solution | 07:52 |
| yoctozepto | rpittau: ok, for a minimal change try moving libvirt-python to test-requirements | 07:53 |
| yoctozepto | that might help immediately | 07:53 |
| rpittau | yoctozepto: ack | 07:54 |
| yoctozepto | rpittau: I proposed in the meantime | 07:55 |
| yoctozepto | I don't know what jobs that will trigger | 07:55 |
| yoctozepto | so you might need to test depending on that | 07:55 |
| rpittau | oh thanks! | 07:55 |
| rpittau | we probably need to add the system package to bindep at least, but let's see how the tests go first | 08:00 |
| iurygregory | we probably need to push using Depends-On on ironic and ipa to see how it goes | 08:00 |
| rpittau | yep, I was exactly doing that :) | 08:00 |
| yoctozepto | rpittau: nah, based on working examples, you shouldn't :D | 08:02 |
| yoctozepto | it will likely create further issues down the road | 08:02 |
| yoctozepto | maybe not | 08:02 |
| yoctozepto | but who knows | 08:02 |
| iurygregory | well virtualbmc only runs 1 job :D | 08:02 |
| rpittau | mmm ok I just hope to not see libvirt-python removed from some RPM specs at some point :D | 08:03 |
| yoctozepto | iurygregory: yeah, I'm not negating your statement | 08:03 |
| iurygregory | hehehehe | 08:03 |
| yoctozepto | rpittau: yeah, it's in my nightmares already :-) | 08:03 |
| yoctozepto | though I can see it proactively installed by kolla, tripleo, osa and loci | 08:03 |
| opendevreview | Slawek Kaplonski proposed openstack/devstack master: Stop creating userrc_early https://review.opendev.org/c/openstack/devstack/+/780417 | 08:41 |
| opendevreview | Slawek Kaplonski proposed openstack/devstack master: Deploy Neutron with enforced new RBAC rules https://review.opendev.org/c/openstack/devstack/+/797450 | 08:41 |
| *** brinzhang0 is now known as brinzhang | 11:05 | |
| *** dviroel|ruck|out is now known as dviroel|ruck | 11:12 | |
| *** jpena is now known as jpena|lunch | 11:34 | |
| *** jpena|lunch is now known as jpena | 12:32 | |
| slaweq | gmann: hi | 12:45 |
| slaweq | gmann: may I have a question about system scopes and tempest testing? | 12:45 |
| slaweq | gmann: I started patch https://review.opendev.org/c/openstack/neutron-tempest-plugin/+/796612 to see how it will work with enforced new defaults | 12:46 |
| slaweq | but it seems that it is terrible now | 12:46 |
| slaweq | so my question is about is there any guidance about how we should do it? Should we try to modify existing tests so they will work with both old and new scopes? | 12:47 |
| slaweq | or maybe we should skip old tests if new scopes are enabled and add completly new API tests for system scope personas? | 12:47 |
| slaweq | gmann: I'm not sure in which direction I should go with all that | 12:48 |
| dansmith | slaweq: in glance we're testing both in parallel and will for some time, using a tempest plugin and job config | 13:36 |
| slaweq | dansmith: but do You have the same tests and run them for old and new personas? | 13:36 |
| slaweq | or do You have completly different tests for each? | 13:36 |
| dansmith | well actually, we don't really have any system-scope things yet, so maybe I'm not helping here, but we have kinda "old policy" and "new rbac-style policy" and we largely run the same tests in both modes | 13:38 |
| dansmith | when we do have system scope-y things I think we'll end up with tests specifically for system scope details, but also tests that run both ways until we're ready to drop non-scoped personas | 13:39 |
| dansmith | our tempest plugin tests a bunch of new-style rbac stuff, and will test any system scope stuff we have, and those tests are specific to the new arrangement and won't run on jobs in the old mode | 13:40 |
| rpittau | yoctozepto: good and bad news, the virtualbmc change should fix the issue there, although we'll have to release a new version to fix the jobs that do not have virtualbmc in the required-projects list (e.g. metalsmith) | 13:41 |
| slaweq | dansmith: ok, the reason I'm asking is mostly because when I want to use new personas, like e.g. system_admin, I need to modify a lot of things to pass project_id in the request body - otherwise it won't work at all | 13:41 |
| rpittau | we also found one more porject that has libvirt-python in requirements, virtualpdu | 13:41 |
| slaweq | and we have a lot of tests which are testing e.g. what "admin" can do | 13:41 |
| dansmith | slaweq: I think you should avoid doing that, personally.. have you seen the x-project-id proposal? | 13:42 |
| slaweq | dansmith: yes, I did | 13:42 |
| dansmith | basically a header where a system-scoped token holder can say "and this is the project I want to be acting on" | 13:42 |
| dansmith | okay | 13:42 |
| slaweq | but that is kind of workaround for now at least | 13:42 |
| dansmith | why do you say that? | 13:42 |
| slaweq | at least it works in devstack https://review.opendev.org/c/openstack/devstack/+/797450 and let's me deploy neutron with new scopes :) | 13:43 |
| dansmith | just to be clear, I'm talking about this: https://review.opendev.org/c/openstack/keystone-specs/+/787640 | 13:44 |
| opendevreview | Lukas Piwowarski proposed openstack/tempest master: Fix cleanup of default security group when preprov creds are used https://review.opendev.org/c/openstack/tempest/+/804088 | 13:44 |
| dansmith | nova and glance aren't modifying all of our api calls to take project_id in the body for system scope, because that would be really destabilizing for a project that has been using project-scoped tokens forever, and doesn't really make sense to change, IMHO | 13:45 |
| dansmith | if a system scoped token holder wants to do something on a project-scoped resource, they need to declare some project affiliation for that operation | 13:46 |
| slaweq | dansmith: I agree, I'm just saying about temporary solution for now as neutron already allows passing project_id in body and it works even with old personas | 13:46 |
| dansmith | okay | 13:47 |
| slaweq | so to make it properly I probably should add whole bunch of new api tests for new personas | 13:48 |
| slaweq | as their privileges are different than in old rules | 13:48 |
| yoctozepto | rpittau: meh! how are you going to proceed then? | 13:48 |
| dansmith | slaweq: or wait until the project header is available and wrap it in the client behavior so you can run both, where it makes sense | 13:49 |
| rpittau | yoctozepto: apparently this https://review.opendev.org/c/openstack/devstack/+/804951 works better for us at least on IPA, we're testing ironic now | 13:49 |
| slaweq | dansmith: ok | 13:49 |
| yoctozepto | rpittau: it seems fine for masakari and ceilometer too - I checked it; but it needs qa team discussion; I don't want to go back and forth with this; in kolla we know already that python-libvirt package compatibility is not best-in-breed, so to speak, so we risk hitting a wall the next time something moves with it, hence why I'm concerned :/ | 13:53 |
| yoctozepto | relying on distro seems more reliable | 13:53 |
| yoctozepto | but seemingly it breaks some assumptions in the ecosystem | 13:54 |
| yoctozepto | ironic is quite complex nowadays so I see you are at worse position to tackle this than others | 13:54 |
| yoctozepto | we could also merge this now, since it seems to be reliable for now, but discuss soon | 13:55 |
| yoctozepto | frickler, gmann, kopecmartin: wdyt? ^ my discussion with rpittau and ironic gate breakage due to libvirt-python | 13:55 |
| rpittau | yoctozepto: I totally understand, although I don't see a way to fix all the projects that has libvirt-python in the requirements, especially the ones that we don't control, like virtualpdu | 13:56 |
| rpittau | I still have the idea of removing the sys packaage when installing virtualbmc in ironic devstack plugin | 13:56 |
| yoctozepto | rpittau: if you could leave a comment on my test change that "it helps but..." and list the issues then that would help others to digest the flow of the possibilites | 13:56 |
| rpittau | yoctozepto: aboslutely, I think iurygregory already left a comment, I'll add one too :) | 13:57 |
| yoctozepto | rpittau: I have seen none :D | 13:57 |
| yoctozepto | I mean the virtualbmc one | 13:57 |
| yoctozepto | oh, I see, virtualpdu is in x/ | 13:57 |
| yoctozepto | if ironic removes the package to reinstall it with pip, then it leaves ironic vulnerable to breakages from elsewhere still | 13:58 |
| yoctozepto | we have already discussed that | 13:58 |
| yoctozepto | but I guess it could be accepted as well | 13:58 |
| yoctozepto | we are also planning to gate on ironic in devstack so it should be safer in the future anyhow | 13:59 |
| iurygregory | well we need the CI fixed asap, and we can try to figure out better approaches later I would say :D | 14:00 |
| rpittau | consider that we use virtualbmc in 2 ways (excluding CI): in virtualenv, or packaged; in both cases there should not be risk of conflicts. The issue happens when installing virtualbmc (and libvirt-python dependency) as root using pip | 14:00 |
| frickler | yoctozepto: I was just looking at https://docs.openstack.org/nova/latest/reference/libvirt-distro-support-matrix.html but I'm not sure what nova's plans are for y and beyond | 14:01 |
| frickler | maybe I should wrap my change with a flag, so that ironic can switch while others keep the current behavior | 14:02 |
| yoctozepto | iurygregory: agreed | 14:04 |
| frickler | lyarwood: ^^ can you share some thoughts as to whether nova would prefer to have libvirt-python from distro or keep using latest from pypi? | 14:04 |
| yoctozepto | ^ yeah, good thinking to actually ask the main consumer :D | 14:04 |
| yoctozepto | but keep in mind the issues were never nova failing on libvirt but rather libvirt failing to compile against another libvirtd | 14:05 |
| opendevreview | Lukas Piwowarski proposed openstack/tempest master: Fix cleanup of default security group when preprov creds are used https://review.opendev.org/c/openstack/tempest/+/804088 | 14:07 |
| dansmith | has anyone been seeing py36 jobs fail with timeout all the sudden? | 14:21 |
| dansmith | glance team is seeing our py36 functional tests time out with one worker stuck like 80% of the time, but the same job on py38 works fine 100% of the time | 14:21 |
| dansmith | it looks like maybe some other jobs are hitting timed_out, some of which may be py36, but we're kinda stuck on what is going on | 14:21 |
| dansmith | doesn't repro locally either | 14:21 |
| yoctozepto | not me | 14:23 |
| yoctozepto | but I remember there was a buggy patch proposed to masakari and it was failing on py38, yet timing out on py36 - it looked like a stack handling difference then | 14:24 |
| yoctozepto | but it's not the case here if py38 is entirely here | 14:24 |
| yoctozepto | entirely green* | 14:24 |
| dansmith | yeah.. if I could repro locally I'd start bisecting, but... I can't | 14:25 |
| dansmith | there are also some py36-specific deps that I figured could be related, but none that look super close to us, so I dunno | 14:26 |
| opendevreview | Merged openstack/devstack stable/queens: [stable-only] Use queens-eol for horizon https://review.opendev.org/c/openstack/devstack/+/804889 | 14:29 |
| lyarwood | frickler: sorry was on a call, I had thought it was safer from the distros | 14:32 |
| lyarwood | at least that was the impression I had after talking to the libvirt folks about this a few weeks ago when https://review.opendev.org/c/openstack/devstack/+/798514 came up | 14:33 |
| lyarwood | ah and I now see ianw 's comment in the bug | 14:34 |
| frickler | lyarwood: oh, interesting, I hadn't seen that patch. what I did with https://review.opendev.org/c/openstack/devstack/+/804951 for ironic would essentially revert it, I didn't intend to do that | 14:46 |
| frickler | so likely having a flag to choose either variant looks like the better option even more | 14:47 |
| yoctozepto | frickler: duh, I thought you did | 14:47 |
| yoctozepto | ok, I think I have just imagined a better approach, will propose in a bit | 14:47 |
| lyarwood | frickler: what's the reasoning for using pip over the distro? | 14:54 |
| lyarwood | frickler: the fact that some projects have it listed in requirements? | 14:54 |
| lyarwood | frickler: tbh I'd rather drop it from requirements given the way the bindings are generated at build time against a specific version of libvirt | 14:54 |
| gmann | slaweq: dansmith on system scope testing, plan is to move all the existing tempest plugins tests towards new policy defaults. it will take time. and we need to have flag which is CONF.<service>.enforce_scope to keep using old defaults for stable branch testing (for stable those does not have new policy). | 14:59 |
| yoctozepto | lyarwood: yeah, ceilometer dropped it, masakari followed as well, but ironic ecosystem is overly complicated to handle quickly | 14:59 |
| lyarwood | yoctozepto: kk, then could we do this just when ironic is used? | 15:00 |
| gmann | slaweq: dansmith and once all stable using master tempest has the new policy defaults then we can only test the new personas. basically at the end we need to migrate all the test tempest and plugins tests to new defaults | 15:01 |
| * dansmith nods | 15:01 | |
| opendevreview | Radosław Piliszek proposed openstack/devstack master: Remove libvirt-python from upper-constraints https://review.opendev.org/c/openstack/devstack/+/805040 | 15:02 |
| yoctozepto | rpittau, iurygregory, frickler, lyarwood: better approach ^ | 15:02 |
| yoctozepto | just help me testing it, I need to focus on something else now | 15:03 |
| iurygregory | yoctozepto, https://review.opendev.org/c/openstack/requirements/+/804874 ? :D | 15:03 |
| rpittau | yoctozepto: ok, thanks | 15:03 |
| rpittau | iurygregory: that's different though | 15:03 |
| iurygregory | oh ok | 15:03 |
| iurygregory | I saw u-c so I thought it was in requirements lol | 15:03 |
| rpittau | we remove that in devstack but not from general u-c | 15:03 |
| iurygregory | :D | 15:03 |
| yoctozepto | iurygregory: requirements team will not allow that; also, u-c is also affecting unit tests and there we have venv and much more flexibility in-project | 15:03 |
| rpittau | :) | 15:04 |
| rpittau | I'll add another DNM patch to test that | 15:04 |
| rpittau | yoctozepto: thanks a lot for the help | 15:04 |
| yoctozepto | rpittau: yw :-) | 15:04 |
| iurygregory | tks! | 15:04 |
| frickler | yoctozepto: I like your patch, if it works for ironic, we can go ahead with it | 15:38 |
| clarkb | frickler: re libvirt-python I think using the system pacakges is fine. I just wanted to be clear with what hte prior problem was. Previously we were mixing wheels built for one platform on another platform and that isn't a good idea. But if we had built a libvirt-python wheel for the actual platform it is expected to work fine and it not doing so is a bug according to upstream | 15:41 |
| yoctozepto | frickler: :D it came to my mind while I was brushing my teeth (always remember to brush your teeth at least twice a day, it helps with fixing openstack gates 8-) ) | 15:41 |
| yoctozepto | clarkb: the upstream does not have good back-and-forward-compatibility testing it seems | 15:42 |
| *** jpena is now known as jpena|off | 15:42 | |
| clarkb | yoctozepto: I don't think that is true the error we experienced was related to the wheel linking and not a fault of their forward and backward compatibility | 15:43 |
| clarkb | yoctozepto: we compiled and linked libvirt-python against one versions of libvirt then tried to use it against a different version of libvirt | 15:43 |
| clarkb | if you compiled libvirt-python against the actual libvirt in use it should work | 15:43 |
| opendevreview | Elod Illes proposed openstack/devstack stable/pike: [stable-only] Use pike-eol for horizon https://review.opendev.org/c/openstack/devstack/+/805047 | 15:43 |
| yoctozepto | clarkb: yeah, that one not, I agree; but libvirt-python was failing to compile on too new package in fedora iirc | 15:44 |
| clarkb | I am not aware of that one | 15:44 |
| yoctozepto | also, it was failing on stream with rdo repos bumping libvirt to fedora one | 15:44 |
| yoctozepto | 7.5.0 | 15:44 |
| clarkb | if you mean the more recent issue that came up around failing to uninstall the system packges that is why we avoided using system packages in the first place. It is a system packge issue nothing to do with the pip wheel build direclty | 15:45 |
| yoctozepto | and it tried to compile it | 15:45 |
| yoctozepto | clarkb: no, I am about compilation here | 15:45 |
| yoctozepto | the pip thing we are fixing | 15:45 |
| yoctozepto | we decided we want the latest pip | 15:45 |
| yoctozepto | but libvirt-python from distro | 15:45 |
| yoctozepto | and now juggling this | 15:45 |
| yoctozepto | ;d | 15:45 |
| clarkb | yes that is a problem with the system libvirt-python packaging using distutils | 15:46 |
| clarkb | when something else tries to install libvirt-python from pypi | 15:46 |
| clarkb | this is why we stopped installing these packages from the system in the first place | 15:46 |
| clarkb | do you have a link to logs for the compilation failure on centos-8-stream? | 15:47 |
| clarkb | was the libvirt-python an older version thanthe libvirt version? I think that is also not supposed to work | 15:48 |
| clarkb | libvirt python must be >= libvirt | 15:48 |
| yoctozepto | clarkb: yeah, it was older | 15:50 |
| yoctozepto | I think the links have expired | 15:51 |
| yoctozepto | but it was older | 15:51 |
| clarkb | ya so that isn't really a problem either | 15:51 |
| clarkb | just bump libvirt-python and move on | 15:52 |
| yoctozepto | yeah, it's supposed not to be a problem but u-c holds it to be a problem | 15:52 |
| yoctozepto | so we have to work around one way or another | 15:52 |
| clarkb | yes you bump u-c when that happens and move on | 15:53 |
| clarkb | that is why u-c exists to help us manage these problems and make controlled updates | 15:53 |
| clarkb | conversely rdo shouldn't bump libvirt if upstream says the newest working version is < | 15:54 |
| clarkb | its meant to communicate in both directions | 15:54 |
| yoctozepto | clarkb: well, fwiw, we don't have any bells and whistles from libvirt-python newer than libvirt | 15:57 |
| yoctozepto | so I guess I accept the current workaround as requiring the least work... for now at least :D | 15:58 |
| *** rpittau is now known as rpittau|afk | 16:06 | |
| opendevreview | James Parker proposed openstack/whitebox-tempest-plugin master: Limit returned compute hosts https://review.opendev.org/c/openstack/whitebox-tempest-plugin/+/805054 | 16:17 |
| gmann | yoctozepto: rpittau|afk we can test libvirt things in devstack gate itself in ironic-tempest-ipa-wholedisk-bios-agent_ipmitool-tinyipa job which is n-v for now | 16:34 |
| yoctozepto | gmann: yeah, we will be testing further impact on ironic there | 16:34 |
| yoctozepto | for now, we are trying to fix ironic's gates | 16:34 |
| yoctozepto | this seems to be doing it https://review.opendev.org/c/openstack/devstack/+/805040 | 16:34 |
| gmann | yeah ironic-tempest-ipa-wholedisk-bios-agent_ipmitool-tinyipa job currently fail on devstack gate too | 16:35 |
| yoctozepto | you can read more backlog for backstory | 16:36 |
| yoctozepto | but the commit should have all the necessary details to understand it in the future | 16:37 |
| gmann | yeah, i am saying we do not need another testing patch on ironic gate instead we can test it in devstack gate itself if it work or not | 16:38 |
| yoctozepto | gmann: ironic's already testing it; they have a multitude of jobs; another approach was fixing only a part of the problem so we are verifying it to the fullest with that | 16:39 |
| yoctozepto | but yeah, in the future that should be enough | 16:39 |
| yoctozepto | if not, we can gate on a better job | 16:39 |
| gmann | yeah | 16:41 |
| yoctozepto | for now, review that patch as it seems to be worky | 16:41 |
| yoctozepto | and we will want to merge it asap | 16:41 |
| gmann | yoctozepto: yeah lgtm, waiting for full set of ironic gate to pass. | 16:43 |
| yoctozepto | ack | 16:43 |
| yoctozepto | cc'ed you on https://review.opendev.org/c/openstack/ironic/+/805041/ | 16:44 |
| gmann | and that seems passing all job on this | 16:44 |
| yoctozepto | looks passing in zuul though | 16:44 |
| gmann | yeah | 16:44 |
| gmann | +2 | 16:45 |
| yoctozepto | thanks | 16:46 |
| *** dviroel|ruck is now known as dviroel|ruck|out | 19:07 | |
| opendevreview | Merged openstack/devstack stable/pike: [stable-only] Use pike-eol for horizon https://review.opendev.org/c/openstack/devstack/+/805047 | 22:29 |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!