| *** gyee has quit IRC | 00:28 | |
| *** schwicht has quit IRC | 00:41 | |
| *** joshualyle has quit IRC | 01:11 | |
| *** kplant has quit IRC | 01:41 | |
| *** dasp has quit IRC | 02:46 | |
| *** dasp has joined #openstack-ansible | 03:03 | |
| *** djhankb has quit IRC | 03:38 | |
| *** udesale has joined #openstack-ansible | 03:56 | |
| *** partlycloudy has quit IRC | 04:28 | |
| *** partlycloudy has joined #openstack-ansible | 04:29 | |
| *** pcaruana has joined #openstack-ansible | 05:06 | |
| janno | jrosser: so host="rbd:volumes" is just for display and has no other impact? | 05:34 |
|---|---|---|
| *** udesale has quit IRC | 05:38 | |
| logan- | janno: it has a big (positive) impact if you lose a cinder-volume host, because you no longer have to manually migrate volumes between cinder-volume instances. they'll all be sharing the same "hostname" so the volumes will not be mapped to a specific cinder-volume instance anymore. | 05:52 |
| logan- | with the old setup, if you had volumehost1 and volumehost2, both using ceph backends, they are basically just API interfaces to ceph. both have equal potential to manage volumes. but if volumehost1 goes down, you can no longer manage any of the volumes that were managed by volumehost1, even though volumehost2 could manage them just fine. | 05:55 |
| logan- | with the backend host setting, the volumes are owned by a fake/virtual host that both volumehost1 and volumehost2 serve requests for. no remapping is required if one of the hosts is lost. | 05:55 |
| jrosser | logan-: we still need to address this renaming in R-S upgrade | 05:57 |
| logan- | jrosser: yeah :/ | 05:57 |
| *** luksky has joined #openstack-ansible | 06:27 | |
| *** miouge36 has quit IRC | 06:27 | |
| *** miouge36 has joined #openstack-ansible | 06:27 | |
| *** cshen has joined #openstack-ansible | 06:30 | |
| *** luksky has quit IRC | 06:48 | |
| *** luksky has joined #openstack-ansible | 07:06 | |
| *** rpittau|afk is now known as rpittau | 07:06 | |
| janno | alright, thank you. I was asking because I created to backends with two different pools, both backends are displayed in horizon, but all volumes are always created in the same pool | 07:09 |
| *** tosky has joined #openstack-ansible | 07:20 | |
| *** luksky11 has joined #openstack-ansible | 07:43 | |
| *** luksky has quit IRC | 07:45 | |
| *** miloa has joined #openstack-ansible | 07:55 | |
| miloa | Morning :) | 07:55 |
| *** luksky11 has quit IRC | 07:55 | |
| *** rgogunskiy has quit IRC | 08:01 | |
| *** trident has quit IRC | 08:08 | |
| *** luksky11 has joined #openstack-ansible | 08:08 | |
| *** trident has joined #openstack-ansible | 08:09 | |
| *** luksky11 has quit IRC | 08:11 | |
| *** shyamb has joined #openstack-ansible | 08:21 | |
| *** jsquare has quit IRC | 08:25 | |
| *** jsquare has joined #openstack-ansible | 08:25 | |
| *** luksky11 has joined #openstack-ansible | 08:27 | |
| *** shyamb has quit IRC | 08:28 | |
| *** luksky11 has quit IRC | 08:33 | |
| *** ivve has joined #openstack-ansible | 08:34 | |
| *** shyamb has joined #openstack-ansible | 08:53 | |
| jrosser | morning | 09:02 |
| *** udesale has joined #openstack-ansible | 09:04 | |
| *** sum12 has joined #openstack-ansible | 09:10 | |
| *** luksky11 has joined #openstack-ansible | 09:12 | |
| openstackgerrit | Florian Haas proposed openstack/openstack-ansible stable/stein: Ensure systemd-journal-remote.socket service is started https://review.opendev.org/668843 | 09:30 |
| *** tosky has quit IRC | 09:39 | |
| *** tosky has joined #openstack-ansible | 09:40 | |
| *** dmsimard9 has joined #openstack-ansible | 10:08 | |
| *** dmsimard has quit IRC | 10:08 | |
| *** shyamb has quit IRC | 10:20 | |
| *** udesale has quit IRC | 10:28 | |
| *** udesale has joined #openstack-ansible | 10:29 | |
| *** shyamb has joined #openstack-ansible | 10:40 | |
| jamesdenton | mornin | 10:49 |
| *** udesale has quit IRC | 10:52 | |
| *** udesale has joined #openstack-ansible | 10:54 | |
| *** udesale has quit IRC | 10:58 | |
| *** udesale has joined #openstack-ansible | 10:58 | |
| *** cshen has quit IRC | 10:59 | |
| *** cshen has joined #openstack-ansible | 11:01 | |
| *** udesale has quit IRC | 11:19 | |
| jamesdenton | Anyone notice that Cinder is not configured to log to /var/log/cinder in recent Master? | 11:21 |
| *** kplant has joined #openstack-ansible | 11:22 | |
| jamesdenton | and possibly Keystone? | 11:23 |
| *** schwicht has joined #openstack-ansible | 11:41 | |
| *** ansmith has quit IRC | 11:52 | |
| janno | jamesdenton: I just was looking for a cinder log but I wasn't able to find any. | 11:52 |
| *** jsquare has quit IRC | 11:53 | |
| *** schwicht has quit IRC | 11:53 | |
| *** jsquare has joined #openstack-ansible | 11:53 | |
| jamesdenton | what version are you on? | 11:53 |
| kaiokmo | jamesdenton: janno, same here | 11:54 |
| janno | stein | 11:54 |
| kaiokmo | on rocky | 11:54 |
| jamesdenton | hmm. ok. Well, the journal has cinder-related logs if you need to check something | 11:55 |
| jamesdenton | but in the meantime, may wanna wait for one of the cores to pop in and we can ask them | 11:55 |
| kaiokmo | jamesdenton: take a look at rocky re-no https://docs.openstack.org/releasenotes/openstack-ansible/rocky.html#relnotes-18-0-0-stable-rocky | 11:56 |
| jamesdenton | Oh, well alrighty then. I guess this is expected | 11:57 |
| kaiokmo | logs from a couple services were moved to journal, such as cinder and keystone | 11:57 |
| kaiokmo | yeah | 11:57 |
| jamesdenton | thanks, kaiokmo | 11:57 |
| jamesdenton | janno kaiokmo - 'journalctl -u cinder-api -f' fits the bill | 11:58 |
| jamesdenton | or whatever service | 11:58 |
| janno | ah, thx | 11:59 |
| *** schwicht has joined #openstack-ansible | 12:15 | |
| *** shyamb has quit IRC | 12:16 | |
| janno | again a question regarding backend_host: is there something special about the name rbd:volumes? is there any association between the part after : and the ceph pool(s)? | 12:21 |
| jamesdenton | i don't know, sorry. | 12:26 |
| *** shyamb has joined #openstack-ansible | 12:27 | |
| kplant | it might just be a friendly naming scheme? <driver>:<vol_name> ? | 12:30 |
| kplant | i mindlessly configured mine the same way :-/ | 12:30 |
| janno | kplant: rather :<pool_name> than :<vol_name> | 12:33 |
| janno | kplant: at the moment rbd:volumes is hard-coded in cinder.conf.j2 | 12:34 |
| kplant | yeah, pool name | 12:36 |
| logan- | janno: no there's not an association between that and the pool name. it just needs to be consistent across all volume hosts serving that backend and all volumes on a particular backend | 12:36 |
| kplant | logan-: i didn't mean i direct mapping, i meant a choice to name it that way | 12:36 |
| kplant | a direct mapping* | 12:37 |
| janno | logan-: ok, but when I let osa create two ceph backends. Do they need different backend_host entries? | 12:37 |
| logan- | janno: you'll see that cinder also appends the backend name to the host of a volume, so no | 12:38 |
| logan- | if you look at the 'host' of a volume, it is <backend host>@<volume type>#<backend name> | 12:39 |
| *** ansmith has joined #openstack-ansible | 12:40 | |
| *** dmsimard9 is now known as dmsimard | 12:43 | |
| *** schwicht has quit IRC | 12:50 | |
| janno | I have created two backends. one is named 'slow' and the other is named 'RBD'. I see both backends when creating a volume in Horizon, but when I choose slow as backend the RBD one gets set and the volume is created in the pool belonging to backend RBD, which is named volumes. | 12:54 |
| janno | That's my underlying problem. | 12:55 |
| janno | full host value for each volume is rbd:volumes@RBD#rbddriver | 12:56 |
| *** udesale has joined #openstack-ansible | 13:03 | |
| janno | well, does volume_backend_name need to be unique per backend? | 13:07 |
| *** schwicht has joined #openstack-ansible | 13:08 | |
| *** nicolasbock has joined #openstack-ansible | 13:11 | |
| *** BjoernT has joined #openstack-ansible | 13:20 | |
| logan- | yes | 13:23 |
| *** shyamb has quit IRC | 13:23 | |
| janno | 'volume_backend_name: rbddriver' from https://docs.openstack.org/openstack-ansible/stein/de/user/ceph/full-deploy.html sounded too necessary. | 13:26 |
| janno | Thank you! | 13:27 |
| *** vnogin has joined #openstack-ansible | 13:31 | |
| *** spatel has joined #openstack-ansible | 13:37 | |
| spatel | morning | 13:57 |
| spatel | https://bugs.launchpad.net/openstack-ansible/+bug/1835225 | 13:57 |
| openstack | Launchpad bug 1835225 in openstack-ansible "rsyslog client not getting deployed with stein release " [Undecided,New] | 13:57 |
| spatel | why rsyslog_client not getting deploy | 13:57 |
| spatel | very strange issue | 13:57 |
| *** BjoernT_ has joined #openstack-ansible | 13:59 | |
| *** BjoernT has quit IRC | 14:02 | |
| BjoernT_ | wasnt that replaced in favor of the systemd journal client ? | 14:08 |
| *** BjoernT_ is now known as BjoernT | 14:08 | |
| spatel | BjoernT: really ? | 14:13 |
| spatel | i am seeing this error in my logs | 14:14 |
| *** ivve has quit IRC | 14:14 | |
| spatel | systemd-journal-upload: #033[1;31mUpload to http://172.28.8.5:19532/upload failed: Failed connect to 172.28.8.5:19532; Connection refused#033[0m | 14:15 |
| spatel | BjoernT: is this documented that we are replacing rsyslog to systemd journal? | 14:16 |
| jrosser | spatel: that is different, that is systemd log forwarding not rsyslog | 14:17 |
| spatel | So we are continue supporting rsyslog_client with stein right? (still trying to understand why ryslog client didn't get deploy) | 14:18 |
| spatel | everywhere /etc/rsyslog.d/ directory is empty | 14:19 |
| jrosser | well lets deal with that error first | 14:19 |
| BjoernT | the role is still there but only on or the other would make sense | 14:19 |
| BjoernT | see releasenotes/notes/rsyslog-disable-ffc4ebf27cc77330.yaml | 14:20 |
| BjoernT | its deprecated | 14:20 |
| jrosser | there is this https://review.opendev.org/#/c/668843/1 or https://review.opendev.org/#/c/659433/ as alternative ways to fix that systemd log forwarding | 14:20 |
| BjoernT | deprecations: | 14:21 |
| BjoernT | - With the implementation of `systemd-journal-remote` the | 14:21 |
| BjoernT | `rsyslog_client` role is no longer run by default. To enable the legacy | 14:21 |
| BjoernT | functionality, the variable `rsyslog_client_enabled` and | 14:21 |
| BjoernT | `rsyslog_server_enabled` can be set to **true**. | 14:21 |
| spatel | BjoernT: huh... that make sense.. | 14:22 |
| spatel | that is why i wonder why my syslog server and clients are doing nothing | 14:22 |
| spatel | jrosser: do you want me to put that patch by hand ? | 14:24 |
| jrosser | either should fix the journal log forwardingh | 14:24 |
| jrosser | logan-: version is likley more elegant than mine but i've not had time to really look very hard at it | 14:25 |
| spatel | why did we think of not continue with rsyslog_client and replace with journal ? | 14:26 |
| BjoernT | one service less to manage and journald is more efficient I would assume | 14:27 |
| spatel | BjoernT: we have centralized ELK, how do i tell journal to send all logs to ELK ? | 14:27 |
| BjoernT | only if logstash feed the logs into elasticsearch | 14:28 |
| BjoernT | right now I would assume no | 14:28 |
| BjoernT | the stein patch seems to be ok | 14:30 |
| BjoernT | ● systemd-journal-upload.service - systemd-journal-upload service | 14:30 |
| BjoernT | Loaded: loaded (/etc/systemd/system/systemd-journal-upload.service; enabled; vendor preset: disabled) | 14:30 |
| BjoernT | Active: active (running) since Wed 2019-07-03 10:26:15 EDT; 3min 32s ago | 14:30 |
| jrosser | afaik the issue is in the log host where the socket isnt listening, even though the process runs | 14:31 |
| jrosser | all magical systemd-ness | 14:31 |
| spatel | jrosser: i did restart all services on log host systemctl restart systemd-journa* | 14:32 |
| spatel | in container (not host itself) | 14:32 |
| jrosser | it's systemd-journal-remote.socket that matters here | 14:33 |
| spatel | but still 19532 nothing listening | 14:33 |
| spatel | let me restart that specific | 14:33 |
| jrosser | the patch does two things, enable the service, and starts it | 14:34 |
| jrosser | the default is not enabled so restarting on its own wont help | 14:34 |
| spatel | oh!! | 14:35 |
| spatel | i thought patch just restart service.. | 14:35 |
| spatel | let me hand edit patch and run rsyslog-install.yml on log host | 14:35 |
| spatel | thats all right? | 14:35 |
| BjoernT | this patch is not for rsyslog, just config_overrides: | 14:36 |
| BjoernT | infra-journal-remote.ym | 14:37 |
| * jrosser confused | 14:37 | |
| *** vnogin has quit IRC | 14:37 | |
| spatel | BjoernT: yes i got that part, that it will fix journal issue, my end goal is to fix syslog_client which i am going to enable because i want to send logs to ELK | 14:40 |
| spatel | if journal doesn't support that then i have no choice | 14:41 |
| spatel | BjoernT: so all i need to add in user_variable.yum is rsyslog_client_enabled: ture & rsyslog_server_enabled: true | 14:43 |
| spatel | jrosser: that patch worked and now i can see log_host listening on 19532 | 14:45 |
| BjoernT | I would look at https://github.com/mheese/journalbeat | 14:47 |
| BjoernT | not to ship stuff twice over the network | 14:47 |
| *** miloa has quit IRC | 14:48 | |
| spatel | interesting.. | 14:48 |
| spatel | i still like to go back to use syslog over journal because we have already running cloud with syslog and its known for years to troubleshoot etc.. | 14:49 |
| spatel | I am sure journal is best but running out of time to learn new stuff in short time. | 14:50 |
| spatel | can i disable journal completely so i don't duplicate stuff? | 14:50 |
| BjoernT | stopping the service systemd-journal-upload would do it, I could not find a override to disable largely infra-journal-remote.yml which gets automatically imported inside setup-infrastructure | 14:53 |
| BjoernT | which would enable it | 14:54 |
| spatel | hmm! i would put systemd-journal-upload stop in /etc/rc.local then to make it disable and stopped | 14:56 |
| spatel | let me enable rsyslog so i can troubleshoot some issue :( | 14:56 |
| BjoernT | you can just mask it | 14:57 |
| spatel | Yes that should work also.. | 14:57 |
| BjoernT | rc.local is slowly disappearing and/or not really working since systemd | 14:57 |
| spatel | BjoernT: rsyslog_client_enabled: ture is this enough to add in user_variables ? | 14:58 |
| spatel | same with server also | 14:58 |
| spatel | anything else? | 14:58 |
| BjoernT | that would hopefully emulate the old behavior to forward to the existing rsyslog server which should be untouched at this point | 15:00 |
| spatel | perfect! which playbook i should need to run deploy it everywhere? | 15:01 |
| spatel | or i can do "openstack-ansible setup-openstack.yml --limit <infra_host>" ? | 15:02 |
| spatel | or i can do "openstack-ansible setup-openstack.yml --limit <infra_host> --tags rsyslog_client" ? | 15:03 |
| spatel | i can tags to just touch rsyslog client right? | 15:03 |
| BjoernT | the latter yes | 15:03 |
| BjoernT | you need to run all openstack roles with a tag | 15:03 |
| BjoernT | across all hosts | 15:03 |
| spatel | got it | 15:03 |
| BjoernT | but it depends if /etc/rsyslog.conf.d | 15:03 |
| BjoernT | is empty | 15:04 |
| spatel | i will run on one container and will see | 15:04 |
| BjoernT | I have not checked if we wipe it on existing installs | 15:04 |
| spatel | I have /etc/rsyslog.conf.d dir but is has no files | 15:04 |
| spatel | openstack-ansible os-barbican-install.yml --tags rsyslog_client-config --limit ostack-infra-2-1_barbican_container-7684c7d5 | 15:06 |
| spatel | it did nothing... | 15:06 |
| spatel | let me try rsyslog_client-install tags | 15:07 |
| spatel | still nothing | 15:07 |
| -spatel- rsyslog_server_enabled: true | 15:08 | |
| -spatel- rsyslog_client_enabled: true | 15:08 | |
| spatel | in user_variables.yml | 15:08 |
| spatel | i run playbook without tags and it dropped one file /etc/rsyslog.d/50-default.conf | 15:12 |
| spatel | still missing remote file.. i think i need to run server playbook | 15:13 |
| *** cshen has quit IRC | 15:14 | |
| spatel | BjoernT: still something missing i am not seeing 99-remote.conf file which is going to push logs out | 15:18 |
| spatel | above changed only drop 50-default.conf file | 15:18 |
| spatel | BjoernT: if you interested - http://paste.openstack.org/show/753831/ | 15:30 |
| spatel | is this broken ? | 15:30 |
| *** gyee has joined #openstack-ansible | 15:31 | |
| *** udesale has quit IRC | 15:45 | |
| *** rpittau is now known as rpittau|afk | 16:14 | |
| *** markvoelker has joined #openstack-ansible | 17:02 | |
| *** openstackgerrit has quit IRC | 17:04 | |
| NobodyCam | Good Morning OSA folks, | 17:14 |
| NobodyCam | Off the wall question, will OSA handle adding a second cinder backend to an existing deployment? and if so what playbooks would I need to run after updating the `storage_hosts:` section? | 17:17 |
| BjoernT | @spatel you can not run tasks as playbooks. You need to run setup-openstack with the rsyslog client tags | 17:24 |
| spatel | let me give it a try again.. | 17:24 |
| spatel | i did run specific playbook limit with container to try but that only put 50-default.conf file | 17:25 |
| *** weifan has joined #openstack-ansible | 17:37 | |
| *** luksky11 has quit IRC | 17:46 | |
| *** jsquare has quit IRC | 17:49 | |
| *** jsquare has joined #openstack-ansible | 17:50 | |
| *** weifan has quit IRC | 17:57 | |
| *** weifan has joined #openstack-ansible | 17:57 | |
| *** weifan has quit IRC | 17:58 | |
| *** weifan has joined #openstack-ansible | 17:58 | |
| *** weifan has quit IRC | 17:58 | |
| *** weifan has joined #openstack-ansible | 17:58 | |
| *** weifan has quit IRC | 17:59 | |
| *** weifan has joined #openstack-ansible | 17:59 | |
| *** BjoernT has quit IRC | 17:59 | |
| *** weifan has quit IRC | 18:00 | |
| *** weifan has joined #openstack-ansible | 18:00 | |
| *** weifan has quit IRC | 18:01 | |
| *** weifan has joined #openstack-ansible | 18:01 | |
| *** weifan has quit IRC | 18:01 | |
| *** weifan has joined #openstack-ansible | 18:02 | |
| *** weifan has quit IRC | 18:02 | |
| *** weifan has joined #openstack-ansible | 18:02 | |
| *** weifan has quit IRC | 18:03 | |
| *** weifan has joined #openstack-ansible | 18:03 | |
| *** weifan has quit IRC | 18:04 | |
| *** weifan has joined #openstack-ansible | 18:04 | |
| *** weifan has quit IRC | 18:05 | |
| *** weifan has joined #openstack-ansible | 18:05 | |
| *** weifan has quit IRC | 18:05 | |
| *** weifan has joined #openstack-ansible | 18:09 | |
| *** BjoernT has joined #openstack-ansible | 18:22 | |
| *** kplant has quit IRC | 18:26 | |
| spatel | folks, I am talking to some folks over openstack-lbaas and look like i have strong feeling that os_octavia playbook is broken and has some bug | 18:27 |
| spatel | way it create lb-mgmt-net network and wire up with br-vlan it creating some strange loop or something which block my br-lbaas internet and i can't ping that interface | 18:28 |
| johnsom | I'm pretty sure the octavia role still has gates that test it, so I doubt the role has a networking bug. | 18:30 |
| *** hwoarang has quit IRC | 18:31 | |
| spatel | hmm should i stop deploying it in production then? | 18:31 |
| spatel | if its not fully tested then i don't wan to go ahead :( | 18:31 |
| johnsom | spatel I said it is still tested | 18:32 |
| *** weifan has quit IRC | 18:33 | |
| spatel | before deploying octavia my br-lbaas was working fine.. but as soon as deployed octavia it blocked my nic | 18:34 |
| *** hwoarang has joined #openstack-ansible | 18:34 | |
| spatel | not sure how do i debug it.. | 18:34 |
| spatel | i can try something destroy octavia and see if that fix my bridge loop | 18:37 |
| *** weifan has joined #openstack-ansible | 18:38 | |
| jrosser | spatel: Octavia works in the AIO and passes the Octavia tempest tests as well, so I’m fairly sure it is working | 18:41 |
| spatel | yes in AIO it works, i did test that | 18:42 |
| spatel | in 3 node controller its not | 18:42 |
| *** luksky11 has joined #openstack-ansible | 18:42 | |
| spatel | AIO has all internal dummy interface and possible it not seeing issue which i am seeing.. | 18:42 |
| jrosser | Maybe your config has created a loop? And STP is blocking somewhere? | 18:43 |
| spatel | even STP is block in all interface.. | 18:44 |
| spatel | Did anyone running OSA with 3 node controller here? | 18:44 |
| spatel | with octavia | 18:45 |
| jrosser | So this means you have an error in your networking somewhere | 18:45 |
| spatel | not sure where i should look, all i can see its very simple bridge network. everything was working and pinging before i start deploying OSA | 18:46 |
| spatel | I can ping br-vxlan, br-mgmt etc.. all good only br-lbaas is bricks (no ping no arp reply) | 18:47 |
| spatel | I am thinking i can destroy lb-mgmt-net network in neutron and see if that break my loop or something if any | 18:47 |
| *** weifan has quit IRC | 18:49 | |
| *** KeithMnemonic has joined #openstack-ansible | 18:51 | |
| *** devx has quit IRC | 19:12 | |
| *** devx has joined #openstack-ansible | 19:14 | |
| *** markvoelker has quit IRC | 19:19 | |
| *** markvoelker has joined #openstack-ansible | 19:19 | |
| *** weifan has joined #openstack-ansible | 19:22 | |
| *** ansmith has quit IRC | 19:23 | |
| *** weifan has quit IRC | 19:26 | |
| *** kplant has joined #openstack-ansible | 19:44 | |
| *** hamzaachi has joined #openstack-ansible | 19:48 | |
| *** goldyfruit has quit IRC | 20:24 | |
| *** goldyfruit has joined #openstack-ansible | 20:24 | |
| *** trident has quit IRC | 20:43 | |
| *** weifan has joined #openstack-ansible | 20:45 | |
| *** trident has joined #openstack-ansible | 20:45 | |
| *** goldyfruit has quit IRC | 20:47 | |
| *** weifan has quit IRC | 20:49 | |
| *** goldyfruit has joined #openstack-ansible | 21:02 | |
| *** weifan has joined #openstack-ansible | 21:02 | |
| *** pcaruana has quit IRC | 21:16 | |
| *** schwicht has quit IRC | 21:31 | |
| *** ansmith has joined #openstack-ansible | 21:33 | |
| *** ansmith has quit IRC | 21:40 | |
| *** schwicht has joined #openstack-ansible | 21:59 | |
| *** spatel has quit IRC | 22:05 | |
| *** schwicht has quit IRC | 22:12 | |
| *** schwicht has joined #openstack-ansible | 22:14 | |
| *** tosky has quit IRC | 22:38 | |
| *** hamzaachi has quit IRC | 22:48 | |
| *** luksky11 has quit IRC | 22:48 | |
| *** schwicht has quit IRC | 22:54 | |
| *** weifan has quit IRC | 23:30 | |
| *** gyee has quit IRC | 23:39 | |
| *** nicolasbock has quit IRC | 23:41 | |
| *** gyee has joined #openstack-ansible | 23:54 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!