| *** lifeless_ has joined #refstack | 00:05 | |
| *** lifeless has quit IRC | 00:05 | |
| *** lifeless_ has quit IRC | 01:43 | |
| *** lifeless has joined #refstack | 01:45 | |
| *** markvoelker has quit IRC | 05:01 | |
| openstackgerrit | yatin proposed openstack/python-tempestconf master: Handle role creation as they are not case-sensitive https://review.openstack.org/576356 | 05:24 |
|---|---|---|
| *** lifeless has quit IRC | 05:53 | |
| *** lifeless has joined #refstack | 05:53 | |
| *** markvoelker has joined #refstack | 07:01 | |
| *** kopecmartin has joined #refstack | 07:30 | |
| openstackgerrit | megan guiney proposed openstack/refstack master: Subunit Data upload API https://review.openstack.org/530681 | 07:30 |
| *** tosky has joined #refstack | 07:34 | |
| tosky | uhm uhm | 07:34 |
| *** markvoelker has quit IRC | 07:36 | |
| arxcruz | tosky: ping | 08:05 |
| tosky | arxcruz: pong (please ping with a reason, no need to wait for the pong to explain the issue) | 08:06 |
| arxcruz | you looks like filip now :) | 08:06 |
| tosky | no, it's just right - it wastes less time from all sides | 08:06 |
| arxcruz | tosky: so https://review.openstack.org/#/c/568869/ swift enabled, admin tests failing | 08:06 |
| tosky | if I'm away, I can just answer directly when I'm back, and not wait for the pong | 08:07 |
| tosky | arxcruz: admin test failing for demo only? Didn't I mention that it looks correct to me? | 08:07 |
| tosky | test.api.admin tests should always work for users with admin permissions | 08:07 |
| tosky | I think | 08:07 |
| tosky | we may recheck on #openstack-qa | 08:08 |
| arxcruz | tosky: yes, correct to fail, but enabling swift, the skip reason is missed | 08:08 |
| tosky | arxcruz: does it mean that in the normal set of tests we skip other admin tests for other services, but not for swift? | 08:08 |
| tosky | or that the admin tests for other services work? | 08:09 |
| tosky | I know that I can dig the log, but you probably checked already | 08:09 |
| arxcruz | tosky: on admin job, it pass, on demo job it fails, the admin tests are being skipped if swift is disabled, if enabled, it runs even with demo user | 08:10 |
| tosky | arxcruz: the question is: how is that, with a demo user, admin tests are run only for swift? Do we explicitly skip admin tests for other services? | 08:10 |
| tosky | are they not part of the default selector that we use? | 08:11 |
| arxcruz | tosky: they are part, but without swift = True, these tests are skipped | 08:11 |
| arxcruz | check other patches, and you'll see admin tests being skipped on demo jobs | 08:12 |
| tosky | arxcruz: well, let's check the condition that triggers tests like tempest.api.identity.admin etc then | 08:14 |
| arxcruz | tosky: already check, it checks for admin credentials | 08:15 |
| arxcruz | but i don't know why when swift is enabled, this tests run | 08:15 |
| tosky | then it may be something else | 08:15 |
| tosky | it must be there | 08:15 |
| tosky | let's compare tempest.conf for https://review.openstack.org/#/c/575233/ and https://review.openstack.org/#/c/568869/ | 08:17 |
| kopecmartin | already doing that, why accounts.yaml generated in ...869 patch is longer, with more credentials? | 08:18 |
| tosky | ehm, that's a starting point | 08:19 |
| kopecmartin | there is a role "admin" | 08:19 |
| kopecmartin | i don't know how this works, but it's about the changes to the roles | 08:20 |
| kopecmartin | maybe | 08:20 |
| tosky | and then why it did not affect https://review.openstack.org/#/c/568869/ ? The review was updated after the keystone change | 08:20 |
| kopecmartin | how did not affect, the gates for the patch are failing now | 08:21 |
| tosky | not the demo job | 08:21 |
| tosky | jobs* | 08:21 |
| tosky | ups, I meant the other review | 08:22 |
| tosky | the lat run of https://review.openstack.org/#/c/575233 | 08:22 |
| kopecmartin | oh, i see, maybe there is a change in that review, it doesn't have to be related to already merged code | 08:23 |
| tosky | do you have the environment to test 568869 locally? | 08:23 |
| kopecmartin | It just looks strange and suspicious that suddenly the accounts.yaml contains ResellerAdmin and admin roles | 08:23 |
| tosky | in an up-to-date devstack or packstack environment? | 08:23 |
| kopecmartin | i should have | 08:24 |
| tosky | maybe 575233 would fail anyway if swift is forced as enabled | 08:24 |
| tosky | that would be a good test | 08:24 |
| openstackgerrit | Martin Kopec proposed openstack/python-tempestconf master: DNM: Enable swift and test 575233 https://review.openstack.org/576424 | 08:27 |
| tosky | that's a stretch version of "test locally" :D | 08:28 |
| kopecmartin | well :D | 08:28 |
| *** lifeless has quit IRC | 08:28 | |
| kopecmartin | my guess is, that swift makes tempest account generator to use roles which affects the tempest tests | 08:29 |
| tosky | I agree with that, the only difference in the tempest.conf is the operator_role = admin line in the [object-storage] section when the tests fail (568869) | 08:30 |
| tosky | now, the question is why | 08:30 |
| tosky | kopecmartin: uhm, why do we rely still on tempest account-generator? Don't we have our own way to create accounts.yaml now? | 08:32 |
| arxcruz | tosky: if a accounts generator option in tempest is true, it will try to create the accounts | 08:32 |
| tosky | arxcruz: where "it" is? | 08:33 |
| *** markvoelker has joined #refstack | 08:33 | |
| arxcruz | tosky: let me check | 08:33 |
| tosky | arxcruz: right now we invoke tempest account-generator explicitly in roles/generate-accounts-file | 08:33 |
| kopecmartin | tosky, hm, it's because we have concurrency = 2 and in that case the simple minimal accounts.yaml is not enough | 08:34 |
| *** luzC_ has quit IRC | 08:34 | |
| tosky | kopecmartin: do you mean that the new code which creates accounts.yaml does not produce a real usable accounts.yaml? | 08:35 |
| tosky | (because we would need more accounts?) | 08:35 |
| tosky | (for other reasons?) | 08:35 |
| kopecmartin | it produces, but only for concurrency = 1, if i get it right | 08:35 |
| kopecmartin | yes, bigger concurrency , more accounts | 08:35 |
| arxcruz | kopecmartin: where are you seeing this code ? | 08:36 |
| *** rmart04 has joined #refstack | 08:36 | |
| tosky | kopecmartin: but then is refstack-client always running without concurrency? Because otherwise it's not useful | 08:36 |
| *** luzC has joined #refstack | 08:36 | |
| kopecmartin | arxcruz, https://github.com/openstack/python-tempestconf/blob/master/playbooks/python-tempestconf-tempest-packstack.yaml#L57 | 08:37 |
| kopecmartin | arxcruz, https://github.com/openstack/python-tempestconf/blob/master/roles/generate-accounts-file/tasks/main.yaml | 08:37 |
| kopecmartin | tosky, yes, i think the refstack is running with concurrency = 1 | 08:38 |
| tosky | oki | 08:38 |
| kopecmartin | tosky, https://github.com/openstack/refstack-client/blob/master/etc/accounts.yaml.sample#L2 | 08:38 |
| tosky | so then, regardless of how we use it (and I still think that we should use our own generator more in some test), tempest account-generator should be investigated | 08:38 |
| kopecmartin | and there is also a comment, that swift needs roles, that's why | 08:38 |
| tosky | but that's not relevant | 08:39 |
| kopecmartin | so the comment explains why we use tempest generator and also why swift's accounts yaml file is different | 08:39 |
| kopecmartin | is not? | 08:39 |
| tosky | it just says that the minimal file is not working with swift | 08:39 |
| tosky | it does not explain: | 08:40 |
| tosky | - why adding operator_role = admin in the swift section writes down also other roles in tempest.conf | 08:40 |
| tosky | and then why those roles affect, for example, identity tests, if the credentials specified are not admin | 08:41 |
| kopecmartin | admin role in operator role is there i guess because https://review.openstack.org/#/c/568869/32/config_tempest/services/object_storage.py@57 | 08:42 |
| tosky | yes, the problem it's not why it's there | 08:42 |
| kopecmartin | we need to find out, what triggers admin tests - admin role in operator section, or content of accounts.yaml | 08:43 |
| tosky | think about the usage of tempest account-generator for a "normal" user who would like who generates accounts.yaml and would like to run the tests using a non admin user | 08:43 |
| tosky | regardless of tempestconf | 08:43 |
| tosky | yes, that too (I suspect more the content of accounts.yaml, but let's see, the scientific method needs experimental proof) | 08:44 |
| arxcruz | tosky: https://github.com/openstack/tempest/blob/master/tempest/test.py#L282-L287 | 08:51 |
| arxcruz | this is where it check for admin | 08:52 |
| arxcruz | so admin is in cls.credentials | 08:52 |
| tosky | arxcruz: not only that, it's important the next if, credentails is_admin_available | 08:53 |
| arxcruz | oh boy | 08:54 |
| arxcruz | tosky: https://github.com/openstack/tempest/blob/master/tempest/common/credentials_factory.py#L165 | 08:54 |
| tosky | oh, makes sense | 08:55 |
| tosky | but if we use accounts.yaml, shouldn't we test dynamic credentials to False? | 08:56 |
| openstackgerrit | yatin proposed openstack/python-tempestconf master: Handle role creation as they are not case-sensitive https://review.openstack.org/576356 | 08:56 |
| openstackgerrit | Chandan Kumar proposed openstack/python-tempestconf master: Use healthcheck api to determine swift service https://review.openstack.org/568869 | 08:58 |
| arxcruz | tosky: dynamic_credentials is set to False | 08:58 |
| tosky | arxcruz: then it's the next if, maybe? elif CONF.auth.test_accounts_file: | 08:59 |
| tosky | or it could be get_configured_admin_credentials | 08:59 |
| tosky | needs manual debugging | 08:59 |
| arxcruz | tosky: it's set | 09:00 |
| arxcruz | the test_accounts_file is set to /etc/openstack/accounts.yaml | 09:01 |
| openstackgerrit | yatin proposed openstack/python-tempestconf master: Handle role creation as they are not case-sensitive https://review.openstack.org/576356 | 09:04 |
| *** markvoelker has quit IRC | 09:06 | |
| kopecmartin | so if admin is removed from roles in accounts.yaml , the admin tests are skipped as expected .. so why it's there? | 09:11 |
| openstackgerrit | Arx Cruz proposed openstack/python-tempestconf master: DNM - Testing collect logs https://review.openstack.org/576444 | 09:15 |
| tosky | yes, why is it assigned to all generated users? | 09:15 |
| arxcruz | kopecmartin: no idea | 09:15 |
| arxcruz | so, where's this accounts.yaml file is being generated ? | 09:16 |
| arxcruz | if someone already know, i can dig into the code | 09:16 |
| tosky | arxcruz: see the role mentioned before | 09:18 |
| kopecmartin | this tool generates that | 09:19 |
| kopecmartin | https://github.com/openstack/tempest/blob/master/tempest/cmd/account_generator.py | 09:19 |
| arxcruz | okay, turns out the accounts.yaml uses tempest | 09:20 |
| arxcruz | so let's see why is this creating the admin | 09:20 |
| kopecmartin | because of this probably | 09:20 |
| kopecmartin | https://github.com/openstack/tempest/blob/master/tempest/cmd/account_generator.py#L120 | 09:20 |
| kopecmartin | https://github.com/openstack/tempest/blob/master/tempest/cmd/account_generator.py#L162 | 09:20 |
| tosky | uhm, does it (spec) apply for each user created? | 09:21 |
| tosky | weird | 09:21 |
| tosky | resources.extend(generate_resources(cred_provider, opts.admin)) | 09:22 |
| tosky | and there is an if admin: | 09:22 |
| arxcruz | https://github.com/openstack/tempest/blob/master/tempest/cmd/account_generator.py#L157 | 09:22 |
| tosky | yes, I was wondering about that opts.admin, do we pass it? | 09:23 |
| kopecmartin | it's true we are creating the accounts.yaml with admin credentials, because there are needed admin credentials in the file .. | 09:24 |
| kopecmartin | we don't pass it explicitely | 09:24 |
| kopecmartin | *explicitly | 09:24 |
| arxcruz | nope | 09:24 |
| kopecmartin | maybe we can add --with-admin False? | 09:24 |
| tosky | as I said, it needs manual debugging | 09:25 |
| arxcruz | but the default is False | 09:25 |
| tosky | we are speculating, and while reading the code could lead to the solution, there are many variables around and it could be easier to see where it fails exactly in a live execution | 09:26 |
| * kopecmartin is checking | 09:29 | |
| arxcruz | tosky: agree | 09:30 |
| arxcruz | even beter if we can do a bj session with a sharing screen | 09:30 |
| arxcruz | from what I notice, the accounts.yaml are creating accounts with role: admin | 09:31 |
| tosky | that's what kopecmartin pointed out, yes | 09:31 |
| arxcruz | and the https://github.com/openstack/tempest/blob/master/tempest/common/credentials_factory.py#L172 | 09:31 |
| arxcruz | calls https://github.com/openstack/tempest/blob/master/tempest/lib/common/preprov_creds.py#L331 | 09:32 |
| arxcruz | that calls at some point https://github.com/openstack/tempest/blob/master/tempest/lib/common/preprov_creds.py#L326 | 09:32 |
| arxcruz | so, role admin is available, then return true to is_admin_available | 09:32 |
| openstackgerrit | Arx Cruz proposed openstack/python-tempestconf master: DNM - Testing collect logs https://review.openstack.org/576444 | 09:44 |
| arxcruz | kopecmartin: let me know when you have the env so we can bj and debug this | 09:47 |
| arxcruz | this is high priority (at least for our team, except for tosky) since it's blocking the master prod chain | 09:48 |
| kopecmartin | arxcruz, I already have it, first, i was able to reproduce it .. but when inserted pudb breakpoints there everything works as expected :/ I'm gonna try from the beginning | 09:49 |
| tosky | arxcruz: it's high priority for everyone if it's blocking the production chain, but I'd like to avoid a butchered solution, thanks | 09:51 |
| arxcruz | tosky: sure, i just say at least for us, because i don't know how your project works :) | 09:52 |
| tosky | I can start debugging this in a proper way too, but I think it's pointless to have different people do exactly the same task multiple time at this point | 09:52 |
| arxcruz | yup, let's debug in a bj session | 09:52 |
| tosky | my project is blocked thanks to the keystone change | 09:52 |
| arxcruz | but i need a few minutes | 09:52 |
| arxcruz | 15-20 | 09:53 |
| * tosky bbl | 09:53 | |
| *** markvoelker has joined #refstack | 10:03 | |
| kopecmartin | Found it | 10:04 |
| kopecmartin | arxcruz, tosky for generating accounts.yaml a tempest.conf is needed, .. due to recent changes in roles, when the tempestconf is run as admin, object-storage has a admin operator role | 10:12 |
| kopecmartin | and that role makes tempest account-generator to use admin role in accounts.yaml ... | 10:13 |
| kopecmartin | So i just override the operator role to Member, when tempest.conf for account-generator is generated and it should be ok | 10:13 |
| arxcruz | kopecmartin: wait a sec | 10:16 |
| arxcruz | we need operator_role as admin for tripleo jobs | 10:16 |
| arxcruz | tosky: so, it seems that chandankumar just overwrite my patchset on his patch :( | 10:18 |
| kopecmartin | arxcruz, it will not be affected, the change will affect only demo jobs and only the tempest.conf used for generation of accounts.yaml, not the tempest.conf used for running tempest tests | 10:18 |
| arxcruz | kopecmartin: do you have the patch already ? | 10:21 |
| arxcruz | let's merge it ASAP | 10:21 |
| kopecmartin | arxcruz, one second, writing a commit message :D | 10:21 |
| openstackgerrit | Arx Cruz proposed openstack/python-tempestconf master: Use healthcheck api to determine swift service https://review.openstack.org/568869 | 10:26 |
| openstackgerrit | Martin Kopec proposed openstack/python-tempestconf master: Fix operator_role value for account-generator https://review.openstack.org/576472 | 10:26 |
| arxcruz | come on! I want to +2 your patch :D | 10:26 |
| kopecmartin | arxcruz, https://review.openstack.org/576472 | 10:26 |
| openstackgerrit | Arx Cruz proposed openstack/python-tempestconf master: Use healthcheck api to determine swift service https://review.openstack.org/568869 | 10:27 |
| arxcruz | awesome, i jupdate chandankumar patch with a depends-on | 10:27 |
| kopecmartin | hopefully it will work, I'm going for a lunch, which will take a little longer, in 3 hours I'll be back | 10:36 |
| *** markvoelker has quit IRC | 10:37 | |
| openstackgerrit | Arx Cruz proposed openstack/python-tempestconf master: Use healthcheck api to determine swift service https://review.openstack.org/568869 | 10:40 |
| openstackgerrit | Nir Magnezi proposed openstack/python-tempestconf master: Configure tempest for Octavia tests https://review.openstack.org/571177 | 11:12 |
| openstackgerrit | Nir Magnezi proposed openstack/python-tempestconf master: DNM: Test Octavia https://review.openstack.org/575063 | 11:13 |
| *** lifeless has joined #refstack | 11:19 | |
| *** lifeless has quit IRC | 11:28 | |
| *** lifeless has joined #refstack | 11:28 | |
| *** markvoelker has joined #refstack | 11:34 | |
| chandankumar | arxcruz: kopecmartin tosky Anything needed from my side on this patch https://review.openstack.org/568869 | 11:53 |
| arxcruz | chandankumar: yes, don't do any changes | 11:54 |
| chandankumar | arxcruz: sure | 11:54 |
| *** markvoelker has quit IRC | 12:01 | |
| *** markvoelker has joined #refstack | 12:02 | |
| tosky | uhm, I'm still not sure | 12:16 |
| tosky | I mean, if we are forcing that role, isn't it exactly what we wanted to avoid with https://review.openstack.org/568869 ? | 12:17 |
| chandankumar | tosky: please comment on this revew what are the things I am missing on this one https://review.openstack.org/541273 | 12:22 |
| chandankumar | i will update later tomorrow morning | 12:22 |
| tosky | we have kind of other urgent stuff... | 12:23 |
| chandankumar | ack | 12:23 |
| tosky | arxcruz: so, about https://review.openstack.org/568869, why that if service == 'object-store': special case? | 12:27 |
| tosky | I think I already asked about it: why should that be a special case? | 12:27 |
| arxcruz | tosky: removed on latest patch | 12:29 |
| tosky | arxcruz: not that one | 12:30 |
| arxcruz | tosky: sorry, i'm not following | 12:31 |
| tosky | https://review.openstack.org/#/c/568869/36/config_tempest/services/services.py line 211 on the right | 12:31 |
| arxcruz | tosky: shit, removing | 12:32 |
| chandankumar | arxcruz: tosky then where are setting to false? swift service or check service availability will take care of that how? | 12:33 |
| tosky | I just raised the point that there should be no special exceptions | 12:33 |
| tosky | at least, no new special exceptions | 12:34 |
| openstackgerrit | Arx Cruz proposed openstack/python-tempestconf master: Use healthcheck api to determine swift service https://review.openstack.org/568869 | 12:34 |
| arxcruz | tosky: done | 12:34 |
| arxcruz | tosky: habemus tempest.conf and tempest log | 12:35 |
| arxcruz | http://logs.openstack.org/44/576444/2/check/python-tempestconf-tempest-devstack-demo/01a85c4/controller/logs/ | 12:35 |
| chandankumar | if no special exception, without heathcheck how we can se to false? | 12:35 |
| chandankumar | for swift case? | 12:35 |
| tosky | write the code so that it's general | 12:36 |
| tosky | see my comment to patch 20 | 12:36 |
| arxcruz | kopecmartin: shit huh, failed because of tripleo | 12:40 |
| tosky | arxcruz: I think we need the other patch by ykarel first | 12:42 |
| tosky | which depends on another patch on instack | 12:42 |
| arxcruz | tosky: instack patch still ongoing | 12:42 |
| tosky | argh | 12:43 |
| openstackgerrit | Arx Cruz proposed openstack/python-tempestconf master: DNM - Testing collect logs https://review.openstack.org/576444 | 12:45 |
| openstackgerrit | Nir Magnezi proposed openstack/python-tempestconf master: Configure tempest for Octavia tests https://review.openstack.org/571177 | 13:41 |
| openstackgerrit | Nir Magnezi proposed openstack/python-tempestconf master: DNM: Test Octavia https://review.openstack.org/575063 | 13:41 |
| kopecmartin | arxcruz, did it work? | 14:12 |
| arxcruz | kopecmartin: what ? | 14:12 |
| arxcruz | nothing is working lol | 14:12 |
| kopecmartin | :D , I meant that patch, did it fix the issue with credentials? | 14:13 |
| tosky | which of the 14135 open patches? | 14:14 |
| * kopecmartin looking for a link, but review.openstack is too slow | 14:15 | |
| kopecmartin | this one https://review.openstack.org/#/c/576472/ | 14:15 |
| kopecmartin | i was out for last three hours | 14:16 |
| tosky | kopecmartin: my question about that patch is: is it to fix the issues with demo jobs in python-tempestconf? But the tripleo jobs should work anyway because the role assigned to swift is overridden ? | 14:18 |
| kopecmartin | tosky, it should fix the issue with admin tests, it's not touching anything what is related to tripleo, I noticed the tripleo job failed multiple times since yesterday with other patches too | 14:20 |
| kopecmartin | so the tripleo failure is unrelated to that patch | 14:20 |
| tosky | I know that the tripleo failure is unrelated, we are waiting for two (or 3) fixes on tripleo | 14:20 |
| tosky | and one in tempestconf related to the keystone change | 14:21 |
| tosky | the question is: will the tripleo job work after their (non tempestconf) issues are fixed, or will they require another fix related to the issue fixed in 576472? | 14:21 |
| kopecmartin | great, one could think it's Monday today :D | 14:23 |
| kopecmartin | I don't know what's the problem with tripleo, if they will fix everything from their side, the job should work | 14:23 |
| tosky | pki | 14:23 |
| tosky | oki* | 14:23 |
| tosky | let's see, we can't merge anything anyway | 14:23 |
| tosky | uh, healthcheck passed (with the exception of tripleo), did you checke the logs already? | 14:28 |
| arxcruz | tosky: checking right now | 14:31 |
| arxcruz | tosky: 40 minutes run, doesn't even touch tempestconf code | 14:31 |
| tosky | I mean, the logs of the devstack and packstack jobs | 14:34 |
| tosky | and apparently object_storage tests were executed, and admin tests skipped | 14:34 |
| tosky | nice | 14:34 |
| tosky | thanks also to the underlying patch | 14:35 |
| tosky | but we can't do anyting until the other patch under recheck is merged (hoping that it will fix the gates) | 14:35 |
| kopecmartin | tosky, arxcruz I know gates are not working now, but any objections on this? https://review.openstack.org/#/c/575233/ | 15:24 |
| kopecmartin | so that I can fix them meanwhile | 15:24 |
| arxcruz | kopecmartin: gates are supposed to be working now | 15:25 |
| arxcruz | i would not like to see anything on the gates before fix the pipeline | 15:25 |
| kopecmartin | oh, great! | 15:25 |
| tosky | yeah, that was my idea as well | 15:43 |
| tosky | urgent fixes first | 15:43 |
| arxcruz | kopecmartin: http://logs.openstack.org/69/568869/37/check/python-tempestconf-tempest-devstack-demo/1581bd6/job-output.txt.gz#_2018-06-19_15_12_17_917607 | 16:04 |
| arxcruz | still failing one job on demo | 16:04 |
| kopecmartin | arxcruz, I've seen it before, it's flaky, I remember recheck helped last times | 16:06 |
| arxcruz | kopecmartin: ok | 16:18 |
| *** rmart04 has quit IRC | 16:22 | |
| tosky | arxcruz: should we recheck 576472? | 16:38 |
| arxcruz | tosky: yes | 16:39 |
| *** kopecmartin has quit IRC | 17:11 | |
| *** lifeless_ has joined #refstack | 17:12 | |
| *** lifeless has quit IRC | 17:12 | |
| *** lifeless has joined #refstack | 21:59 | |
| *** lifeless_ has quit IRC | 21:59 | |
| openstackgerrit | Luigi Toscano proposed openstack/python-tempestconf master: More compatibility with Python 3 https://review.openstack.org/574773 | 23:16 |
| openstackgerrit | Luigi Toscano proposed openstack/python-tempestconf master: More Python 3 porting: filter and map https://review.openstack.org/575874 | 23:16 |
| *** lifeless_ has joined #refstack | 23:41 | |
| *** lifeless has quit IRC | 23:42 | |
| *** tosky has quit IRC | 23:45 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!