| *** thorst_afk has quit IRC | 00:03 | |
| *** chhavi has joined #openstack-powervm | 00:06 | |
| *** chhavi has quit IRC | 00:10 | |
| *** jwcroppe has joined #openstack-powervm | 00:14 | |
| *** jwcroppe has quit IRC | 00:32 | |
| *** svenkat has joined #openstack-powervm | 00:56 | |
| *** jwcroppe has joined #openstack-powervm | 02:07 | |
| *** jwcroppe has quit IRC | 02:12 | |
| *** svenkat has quit IRC | 02:20 | |
| *** esberglu has joined #openstack-powervm | 02:35 | |
| *** esberglu has quit IRC | 02:40 | |
| *** jwcroppe has joined #openstack-powervm | 02:57 | |
| *** chhavi has joined #openstack-powervm | 04:09 | |
| *** esberglu has joined #openstack-powervm | 04:23 | |
| *** esberglu has quit IRC | 04:28 | |
| *** thorst_afk has joined #openstack-powervm | 04:38 | |
| *** thorst_afk has quit IRC | 04:43 | |
| *** jwcroppe has quit IRC | 05:30 | |
| *** esberglu has joined #openstack-powervm | 06:11 | |
| *** esberglu has quit IRC | 06:16 | |
| *** thorst_afk has joined #openstack-powervm | 06:39 | |
| *** thorst_afk has quit IRC | 06:44 | |
| *** thorst_afk has joined #openstack-powervm | 08:40 | |
| *** thorst_afk has quit IRC | 08:45 | |
| *** esberglu has joined #openstack-powervm | 08:53 | |
| *** esberglu has quit IRC | 08:57 | |
| *** openstackgerrit has quit IRC | 10:17 | |
| *** thorst_afk has joined #openstack-powervm | 10:24 | |
| *** smatzek has joined #openstack-powervm | 11:27 | |
| *** smatzek has quit IRC | 11:29 | |
| *** smatzek has joined #openstack-powervm | 11:29 | |
| *** svenkat has joined #openstack-powervm | 12:04 | |
| *** svenkat has quit IRC | 12:23 | |
| *** svenkat has joined #openstack-powervm | 12:23 | |
| *** jwcroppe has joined #openstack-powervm | 12:25 | |
| *** jwcroppe has quit IRC | 12:30 | |
| *** edmondsw has joined #openstack-powervm | 12:36 | |
| *** apearson has joined #openstack-powervm | 12:50 | |
| *** esberglu has joined #openstack-powervm | 12:55 | |
| *** jwcroppe has joined #openstack-powervm | 12:59 | |
| esberglu | #startmeeting powervm_driver_meeting | 13:00 |
|---|---|---|
| openstack | Meeting started Tue Jul 25 13:00:56 2017 UTC and is due to finish in 60 minutes. The chair is esberglu. Information about MeetBot at http://wiki.debian.org/MeetBot. | 13:00 |
| openstack | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 13:00 |
| openstack | The meeting name has been set to 'powervm_driver_meeting' | 13:01 |
| mdrabe | o/ | 13:01 |
| edmondsw | o/ | 13:01 |
| esberglu | #topic In Tree Driver | 13:01 |
| efried | \o | 13:01 |
| esberglu | #link https://etherpad.openstack.org/p/powervm-in-tree-todos | 13:01 |
| esberglu | Oh and here's the agenda | 13:02 |
| esberglu | #link https://etherpad.openstack.org/p/powervm-in-tree-todos | 13:02 |
| esberglu | #link https://etherpad.openstack.org/p/powervm_driver_meeting_agenda | 13:02 |
| esberglu | No topics for in tree. | 13:02 |
| edmondsw | yeah, I don't have anything for IT | 13:03 |
| esberglu | Reminder that this week (thursday) is the pike 3 milestone (feature freeze) | 13:03 |
| efried | https://review.openstack.org/#/c/476945/ | 13:03 |
| efried | Keeps failing CI. I imagine it's for whatever reason everything is failing CI, and we'll get to that later? | 13:04 |
| esberglu | efried: Yeah. I'll update on that status later | 13:04 |
| edmondsw | efried should we open a bug with which to associate that change? | 13:04 |
| efried | Wouldn't think so. It's not a bug. | 13:05 |
| edmondsw | k | 13:05 |
| efried | and we don't need to "backport" it. | 13:05 |
| edmondsw | I thought it fixed something, but must be wrong | 13:05 |
| efried | even if it doesn't make Pike, we don't need to backport it to Pike. | 13:05 |
| edmondsw | fine | 13:05 |
| esberglu | edmondsw: efried: I've been seeing errors about power_off_progressive in CI, assuming this is related somehow? | 13:06 |
| efried | oh? | 13:06 |
| efried | Yes, that would be related. | 13:07 |
| efried | I haven't seen those. | 13:07 |
| efried | Well, shit. | 13:07 |
| efried | Guess we spend some time on that soon. | 13:07 |
| efried | But maybe not too soo. | 13:07 |
| efried | Because feature freeze. | 13:07 |
| esberglu | efried: I don't have logs handy atm, I'll link them when I get to CI status | 13:08 |
| efried | Thought we ran that bad boy through when it was first proposed. But mebbe not. | 13:08 |
| efried | Twill suck if power_off_progressive is STILL broken. | 13:09 |
| efried | Hate that code. | 13:09 |
| esberglu | efried: The related error is not hitting every run so we may have ran it and just not hit the issue | 13:09 |
| efried | We would have needed to run with a develop pypowervm and in-flight nova[-powervm] at the time, which I guess we might have done. But anyway... | 13:10 |
| efried | I don't have anything else in-tree. Moving on? | 13:10 |
| esberglu | Anyone have anything OOT or ready to move straight to PCI passthrough? | 13:11 |
| mdrabe | NVRAM stuff | 13:11 |
| mdrabe | efried: Discuss that later perhaps? | 13:11 |
| efried | thorst_afk is home with sick kid, would like him involved in both of those discussions. | 13:12 |
| efried | So the NVRAM improvement was what, again? 3.4%? | 13:12 |
| mdrabe | Mhm | 13:12 |
| efried | That's... | 13:12 |
| mdrabe | Annoying | 13:12 |
| edmondsw | #topic Out Of Tree Driver | 13:12 |
| efried | ...not nothing, but pretty marginal. | 13:13 |
| thorst_afk | I'm here. | 13:13 |
| thorst_afk | slowly | 13:13 |
| *** thorst_afk is now known as thorst | 13:13 | |
| efried | mdrabe And this was run through a pvc suite that actually exercises the NVRAM/slotmgr code paths? | 13:13 |
| efried | Has it been run through pvc fvt kind of thing that hammers those code paths for functionality? | 13:14 |
| mdrabe | thorst: This is about https://review.openstack.org/#/c/471926/ | 13:14 |
| mdrabe | It can't run through fvt until it merges | 13:14 |
| thorst | that is ... quite the change set. | 13:14 |
| efried | Yeah, that's the problem. | 13:14 |
| thorst | mdrabe: it'd have to be cherry picked. | 13:14 |
| *** jay1_ has joined #openstack-powervm | 13:15 | |
| mdrabe | thorst: Not necessarily within nova-powervm | 13:15 |
| efried | It completely swizzles the way nvram and slot management is stored & retrieved in swift. | 13:15 |
| thorst | for a 3.4% improvement in deploy path? | 13:15 |
| efried | Backward compatible, right mdrabe? Upgrade-able? | 13:15 |
| thorst | overall deploy path | 13:15 |
| mdrabe | Yes backward compatible | 13:15 |
| mdrabe | It doesn't affect the slot manager path greatly | 13:15 |
| efried | By which I mean: if you have pvc running the old code, so objects are stored in swift the old way, then you upgrade pvc, it still works on the old objects? | 13:16 |
| thorst | and if you have some nova computes running at old level and new nova computes running at new level | 13:16 |
| thorst | does it all just work... | 13:16 |
| thorst | cause agents don't get upgraded all at once. | 13:17 |
| mdrabe | efried: The objects aren't stored any differently | 13:17 |
| thorst | I think I need to spend some time reviewing this (I'm just asking naive questions) | 13:17 |
| mdrabe | It's still keyed off the instance UUID regardless | 13:17 |
| efried | Oh, right, they were always keyed by UUID in swift - the difference is the way they were passed around nova-powervm. | 13:17 |
| efried | So yeah, thorst, that's the debate. This is a nontrivial change across code that we chucked together at the last minute during that one TCC to cover the RR emergency, so I'm kinda scared of touching it. | 13:18 |
| mdrabe | Actually efried this isn't that code | 13:19 |
| thorst | yeah, that nightmare is in pypowervm mostly | 13:19 |
| mdrabe | Most of the NVRAM stuff was already there, we slapped together the slot manager stuff | 13:19 |
| thorst | this nvram stuff was from kyleh long ago | 13:19 |
| thorst | so the basic change here is - move from list to set, and allow passing in the uuid or the instance, instead of just the instance. | 13:20 |
| thorst | thus reducing the number of instance lookups I assume | 13:20 |
| efried | That's the gist | 13:20 |
| mdrabe | thorst: Yes, the performance improvement comes from avoiding getting the instance object in the NVRAM event handler | 13:20 |
| thorst | let me review in more detail, but I think its a solid change. The net though is, yeah, it needs some good test pre-merge | 13:20 |
| thorst | so we've got to figure out a plan to do that....patch onto some PVC's I assume | 13:21 |
| thorst | since PVC always has this turned on | 13:21 |
| mdrabe | It might be worth noting there's also performance improvement in idle compute time | 13:21 |
| efried | ? | 13:21 |
| mdrabe | ? to me or thorst? | 13:21 |
| thorst | I think he's asking for an explanation | 13:22 |
| thorst | and if he isn't, I am | 13:22 |
| mdrabe | Any NVRAM events that come in won't grab the instance object | 13:22 |
| thorst | that's ... nice | 13:22 |
| mdrabe | NVRAM events aren't exclusive to deploys | 13:22 |
| thorst | right right... | 13:22 |
| mdrabe | thorst: I'm not the biggest fan of the caching that's done in the change at the moment | 13:23 |
| efried | Okay, so the plan is to move forward carefully, get some good pvc fvt (somehow) | 13:23 |
| thorst | +1 | 13:24 |
| mdrabe | I'd be curious on your review thoughts for that bit | 13:24 |
| thorst | I'll review today | 13:24 |
| esberglu | Move on to PCI? | 13:25 |
| edmondsw | I think so | 13:25 |
| esberglu | #topic PCI Passthrough | 13:26 |
| thorst | k | 13:26 |
| efried | No progress on my end since last week. | 13:26 |
| edmondsw | I don't have anything new here, but I wanted to make sure this is a standing item on the agenda going forward | 13:26 |
| edmondsw | so that's all for today then, probably | 13:26 |
| *** kylek3h has joined #openstack-powervm | 13:26 | |
| esberglu | Okay on to CI | 13:27 |
| esberglu | #link https://etherpad.openstack.org/p/powervm_ci_todos | 13:27 |
| esberglu | #topic Devstack generated tempest.conf | 13:27 |
| esberglu | I've got 3 changes out in devstack, need a couple more | 13:27 |
| esberglu | https://review.openstack.org/#/c/486629/ | 13:27 |
| esberglu | https://review.openstack.org/#/c/486671/ | 13:27 |
| esberglu | https://review.openstack.org/#/c/486701/ | 13:28 |
| esberglu | Then just need to backport those as needed and add the new options to the local.conf files | 13:28 |
| esberglu | At that point we should be ready to use the generated tempest conf | 13:29 |
| edmondsw | sweet | 13:29 |
| esberglu | #topic CI Status | 13:29 |
| *** smatzek has quit IRC | 13:29 | |
| esberglu | The last couple weeks the networking issues have been affecting the CI, taking down systems and leading to poor performance | 13:30 |
| esberglu | Now that all of that seems to have settled down (fingers crossed) I'm gonna take inventory of tempest failures | 13:30 |
| efried | zat what's been causing the timeouts? | 13:30 |
| esberglu | efried: Seems like they have been way up when the network has been poor. Anecdotal though | 13:31 |
| efried | k, well, we'll keep an eye on it. | 13:31 |
| efried | Progress on the mystery 500? | 13:31 |
| esberglu | Anyhow I'll let you guys know what I find when looking through the failures. | 13:32 |
| esberglu | efried: Haven't looked back into it | 13:32 |
| efried | And the power_off_progressive thing - you'll pop me links when you see it again? | 13:33 |
| esberglu | efried: Yep | 13:33 |
| efried | That's prolly all we can do here for CI, then. | 13:34 |
| esberglu | mdrabe: Any progress on the jenkins/nodepool upgrade stuff? Haven't touched base on that in a while | 13:34 |
| mdrabe | esberglu Nah been entirely pvc of late | 13:35 |
| mdrabe | but I wanted to ask | 13:35 |
| mdrabe | How tough would it be to go to all zuul? | 13:35 |
| mdrabe | It's zuulv3 right? | 13:35 |
| esberglu | mdrabe: Last time I checked zuul v3 wasn't ready for 3rd party CI's. But that may have changed | 13:36 |
| esberglu | Once zuul v3 is ready I want to move to it | 13:36 |
| esberglu | I shouldn't be too hard. Mostly just moving our jenkins jobs definitions over to zuul (where they would be defined with ansible instead) | 13:37 |
| esberglu | Alright let's move on | 13:38 |
| esberglu | #topic Driver Testing | 13:38 |
| esberglu | Developments here? | 13:38 |
| efried | jay1_ chhavi How's iSCSI going? | 13:40 |
| efried | thorst wanted to make sure we got an update here. | 13:40 |
| jay1_ | Yeah, Last week encountered the volume attach issue | 13:40 |
| chhavi | efried: ravi and jay is looking into it, i am not currently working on it. | 13:41 |
| jay1_ | and by the time Chhavi/Ravi debug it further system UI was not loading might be bcz of the NW issue we got last week | 13:41 |
| jay1_ | so, I reconfigured the system and again facing the Volume creation issue | 13:42 |
| jay1_ | Ravi is looking into it | 13:42 |
| edmondsw | from talking to ravi, he has very limited time to spend on this | 13:45 |
| edmondsw | so we're going to have to figure that out | 13:45 |
| edmondsw | next topic? | 13:49 |
| esberglu | #topic Open Discussion | 13:49 |
| esberglu | #subtopic Test subtopic | 13:49 |
| esberglu | Just curious if that command is a thing or not | 13:50 |
| esberglu | I don't have anything further | 13:50 |
| esberglu | Thanks for joining | 13:50 |
| esberglu | #endmeeting | 13:50 |
| openstack | Meeting ended Tue Jul 25 13:50:56 2017 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 13:50 |
| openstack | Minutes: http://eavesdrop.openstack.org/meetings/powervm_driver_meeting/2017/powervm_driver_meeting.2017-07-25-13.00.html | 13:51 |
| openstack | Minutes (text): http://eavesdrop.openstack.org/meetings/powervm_driver_meeting/2017/powervm_driver_meeting.2017-07-25-13.00.txt | 13:51 |
| openstack | Log: http://eavesdrop.openstack.org/meetings/powervm_driver_meeting/2017/powervm_driver_meeting.2017-07-25-13.00.log.html | 13:51 |
| jay1_ | edmondsw You have any plans on the RTC story creation? or any other thoughts ? | 13:51 |
| edmondsw | jay1_ I'm trying to get to it, but I honestly have no idea what I would create... | 13:52 |
| jay1_ | Okay.. but that would give more clarity on sprint by sprint target | 13:53 |
| jay1_ | also on the overall goal | 13:53 |
| *** smatzek has joined #openstack-powervm | 14:04 | |
| *** smatzek has quit IRC | 14:05 | |
| *** smatzek has joined #openstack-powervm | 14:05 | |
| *** tjakobs has joined #openstack-powervm | 14:56 | |
| thorst | efried: should I W+1 485571? | 15:20 |
| efried | thorst sho | 15:21 |
| thorst | I'm also thinking of W+1 on 468515 | 15:22 |
| efried | thorst Would be nice to have the cherry-pick source in there. | 15:24 |
| efried | though it's linked in gerrit, so no biggie. | 15:24 |
| efried | But sure, that was just a cherry pick with a mod for log translation, right? | 15:24 |
| *** openstackgerrit has joined #openstack-powervm | 15:29 | |
| openstackgerrit | Merged openstack/nova-powervm master: Removed older version of python added 3.5 https://review.openstack.org/485571 | 15:29 |
| *** smatzek has quit IRC | 15:29 | |
| thorst | efried: 468515? | 15:29 |
| *** smatzek has joined #openstack-powervm | 15:30 | |
| thorst | that was an OVS live migration one | 15:30 |
| thorst | I know pvc is using a cherry pick of it now, so its tested. | 15:30 |
| efried | thorst I'm fine with it. I'll promote to +2 | 15:30 |
| efried | done | 15:31 |
| thorst | kewl | 15:31 |
| *** jwcroppe has quit IRC | 15:34 | |
| *** smatzek has quit IRC | 15:58 | |
| *** smatzek has joined #openstack-powervm | 15:59 | |
| *** edmondsw has quit IRC | 15:59 | |
| *** edmondsw has joined #openstack-powervm | 16:00 | |
| *** edmondsw_ has joined #openstack-powervm | 16:03 | |
| *** jwcroppe has joined #openstack-powervm | 16:04 | |
| *** edmondsw has quit IRC | 16:05 | |
| *** edmondsw_ has quit IRC | 16:07 | |
| *** tjakobs has quit IRC | 16:07 | |
| *** edmondsw has joined #openstack-powervm | 16:18 | |
| *** edmondsw has quit IRC | 16:22 | |
| *** edmondsw has joined #openstack-powervm | 16:24 | |
| *** edmondsw has quit IRC | 16:28 | |
| *** edmondsw has joined #openstack-powervm | 16:52 | |
| *** edmondsw has quit IRC | 16:56 | |
| *** smatzek has quit IRC | 17:06 | |
| *** jwcroppe has quit IRC | 17:38 | |
| *** jwcroppe has joined #openstack-powervm | 17:48 | |
| efried | esberglu Seen this one before? | 18:03 |
| efried | 2017-07-25 17:48:37.755 | +++functions-common:oscwrap:2538 openstack project show admin -f value -c id | 18:03 |
| efried | 2017-07-25 17:48:41.183 | Failed to discover available identity versions when contacting http://9.47.79.13/identity. Attempting to parse version from URL. | 18:03 |
| efried | 2017-07-25 17:48:41.183 | Could not determine a suitable URL for the plugin | 18:03 |
| *** apearson has quit IRC | 18:03 | |
| esberglu | efried: Is that from CI or from a system you are stacking? | 18:03 |
| efried | the latter | 18:03 |
| *** tjakobs has joined #openstack-powervm | 18:04 | |
| esberglu | efried: Ping me creds and I can take a look | 18:04 |
| esberglu | efried: Not much to go off in the stack log. And none of the services are logging at this point. Looking at your local.conf right now but it looks standard | 18:09 |
| efried | esberglu Yuh, that's where I had gotten to. | 18:10 |
| esberglu | efried: Have you been stacking and unstacking this system? | 18:10 |
| efried | First time. | 18:10 |
| efried | Fresh install | 18:10 |
| esberglu | Welp there goes that theory. Thought maybe something got left around and a ./clean.sh would fix it | 18:11 |
| efried | Trying unstack, pull devstack, restack. | 18:12 |
| esberglu | efried: Stacks that fail this early are hard to debug. Sometimes I just wipe out all of the openstack directories and let them get recloned which occaisonally works | 18:14 |
| efried | vay. | 18:14 |
| efried | the openstack dirs were absent before this stack. | 18:15 |
| efried | So refreshing them seems unlikely to yield a different result. | 18:15 |
| efried | But will try, if this doesn't work. | 18:15 |
| esberglu | efried: Not for your system. I was just saying when they fail this early I end up having to resort to solutions like that | 18:16 |
| efried | yeah, I get it. | 18:17 |
| efried | same result | 18:18 |
| esberglu | efried: Try the syslog. I'm gonna look through recent devstack commits quick | 18:19 |
| efried | esberglu How do I look at the syslog? | 18:19 |
| esberglu | efried: /var/log/syslog | 18:19 |
| esberglu | With whatever you like looking at logs with | 18:20 |
| efried | nothing interesting in there, I don't think. | 18:22 |
| esberglu | efried: Nothing recent in devstack standing out either | 18:22 |
| efried | We just dropped a big ol' keystoneauth1 change that tries to do things like getting version numbers from URLs. | 18:22 |
| efried | But I can't imagine this wouldn't have shown up in some other dsvm gate if that was rotten. | 18:23 |
| efried | including our CI, for that matter. | 18:23 |
| esberglu | efried: Yeah I imagine it would have been caught. Just looked at some CI systems, that URL is correct | 18:27 |
| * efried has never, ever, ever stacked successfully the first time. | 18:27 | |
| esberglu | efried: It's not hitting the CI stacks (at least not yet) | 18:29 |
| esberglu | Maybe some sort of dependency issue? | 18:30 |
| esberglu | rabbitmq-server is running, sometimes that doesn't start up properly and causes weird failures | 18:30 |
| efried | rebooting. Cause whytf not. | 18:32 |
| *** apearson has joined #openstack-powervm | 18:45 | |
| *** chhavi has quit IRC | 19:02 | |
| efried | esberglu Is there a reason we're using mod_wsgi instead of uwsgi? | 19:04 |
| esberglu | efried: It didn't work for us right away and other stuff has been higher priority | 19:05 |
| esberglu | Its on the TODO list | 19:05 |
| efried | esberglu Well, maybe it's time I tried it. How would I do that? | 19:05 |
| esberglu | efried: Does this mean you got your base stack working? | 19:05 |
| efried | no | 19:05 |
| esberglu | In the local.conf there is a line WSGI_MODE=... | 19:06 |
| esberglu | Just remove/comment that out | 19:06 |
| esberglu | And it will then default to uwsgi | 19:06 |
| efried | k | 19:07 |
| esberglu | Can't remember what the issue with it was. IIRC switching to uwsgi didn't cause the stack itself to fail | 19:07 |
| esberglu | But caused issues with the services after stacking | 19:07 |
| efried | oh, goodie. | 19:08 |
| efried | The comment in the local.conf says something about placement not starting. | 19:08 |
| efried | Mebbe that's fixed. | 19:08 |
| efried | Anyway, trying it... | 19:08 |
| *** smatzek has joined #openstack-powervm | 19:13 | |
| *** jay1_ has quit IRC | 19:25 | |
| esberglu | efried: We have been enabling cinder in CI but not running any of the cinder tests. I enabled the cinder tests just to see what would happen and it didn't look good | 19:43 |
| esberglu | You okay with just disabling cinder until I have a chance to look into it? | 19:43 |
| efried | esberglu Sure. | 19:43 |
| efried | esberglu Once I converted to uwsgi, keystone logs showed up under journalctl. The logs showed tracebacks with pastedeploy | 19:44 |
| esberglu | efried: I'm disabling it with the other local.conf changes for the devstack generated tempest.conf. I'll leave a comment | 19:44 |
| efried | Stupid, stupid pastedeploy. | 19:44 |
| esberglu | efried: sudo pip install -U pastedeploy?? | 19:46 |
| esberglu | I thinks that's what I had to do last time I hit issues with pastedeploy | 19:47 |
| efried | I generally apt-get remove python-paste* and pip remove Paste* | 19:47 |
| efried | pip only showed Paste, which seems like less than usual. And I've never actually nailed down what the root of the problem is; I just know that typically removing everything paste-related will fix it. | 19:48 |
| esberglu | efried: Any luck? | 20:34 |
| efried | esberglu Got past that part, now it's failing on image create, weirdly. Cause when I run that image create command manually, it works fine. | 20:35 |
| efried | Looks like g-api may be choking on uwsgi, so gonna revert that to mod_wsgi and try again. | 20:36 |
| *** edmondsw has joined #openstack-powervm | 20:40 | |
| *** apearson has quit IRC | 20:56 | |
| *** smatzek has quit IRC | 20:57 | |
| *** esberglu has quit IRC | 21:31 | |
| *** svenkat has quit IRC | 21:39 | |
| *** esberglu has joined #openstack-powervm | 21:45 | |
| *** kylek3h has quit IRC | 22:12 | |
| *** svenkat has joined #openstack-powervm | 22:38 | |
| *** thorst has quit IRC | 23:06 | |
| *** tjakobs has quit IRC | 23:08 | |
| *** edmondsw has quit IRC | 23:10 | |
| *** edmondsw has joined #openstack-powervm | 23:11 | |
| *** edmondsw has quit IRC | 23:15 | |
| *** jwcroppe has quit IRC | 23:36 | |
| *** jwcroppe has joined #openstack-powervm | 23:37 | |
| *** jwcroppe has quit IRC | 23:37 | |
| *** svenkat has quit IRC | 23:46 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!