tristanC | anyway, i'll give this selenium simple-usage a try today using a simple tox target, we can look after how to properly integrate this | 00:00 |
---|---|---|
*** Shrews has quit IRC | 00:26 | |
*** austinsun[m] has joined #zuul | 00:46 | |
openstackgerrit | Merged openstack-infra/zuul master: web: fix status page flickering https://review.openstack.org/578226 | 00:46 |
*** pwhalen has joined #zuul | 01:10 | |
*** pwhalen has joined #zuul | 01:10 | |
openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: web: add selenium tests for the status page https://review.openstack.org/579039 | 01:55 |
tristanC | corvus: clarkb: initial selenium tests looks like this ^ | 01:55 |
tristanC | sadly i couldn't make the firefox driver works and had to install chrome to get the browser console logs | 01:57 |
corvus | tristanC: can you use chromium? | 01:58 |
tristanC | probably, but Chromium isn't officially supported | 01:59 |
corvus | we won't be able to use chrome as it's non-free | 02:00 |
corvus | how important is the logging interface? | 02:00 |
tristanC | it sounds important to assert no javascript errors appears in the logs | 02:01 |
tristanC | we could get away with firefox by using a custom console.log handler in tests that write them to a list | 02:01 |
corvus | that might be a workable compromise | 02:02 |
tristanC | then we get access using driver.execute_script("return console_logs_list") | 02:02 |
tristanC | well 579039 is a first draft, agree that chrome is a blocker, but unfortunately that was the path of least resistance | 02:03 |
corvus | it looks like a great start -- really easy to follow :) | 02:03 |
tristanC | next PS could try adding "click()" action and checks for correct element-id presence in the dom | 02:04 |
tristanC | also, Shrews fix for pool's labels works like a charm | 02:10 |
tristanC | hum, wait, now quota calculation seems off, requests are paused while there is plenty of room in the tenant | 03:01 |
clarkb | tristanC: each pool has its own max server though right? could it be running up against that? | 03:02 |
clarkb | the angular updates appear to be working fine this time around | 03:02 |
openstackgerrit | Tristan Cacqueray proposed openstack-infra/nodepool master: status: add pool column to nodepool list output https://review.openstack.org/579046 | 03:06 |
tristanC | clarkb: indeed, i got confused by some ready-node eating up available pool quota | 03:08 |
openstackgerrit | Tristan Cacqueray proposed openstack-infra/nodepool master: launcher: add pool quota debug and log information https://review.openstack.org/579048 | 03:12 |
tristanC | All hail to the angular6 overlord :-) | 03:14 |
openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: web: add selenium tests for the status page https://review.openstack.org/579039 | 03:45 |
openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: web: add selenium tests for the status page https://review.openstack.org/579039 | 04:42 |
openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: DNM: verify selenium test by reproducing the pathname whitelabel bug https://review.openstack.org/579052 | 04:46 |
tristanC | corvus: mordred: hopefully the selenium test should catch the bug from 579052 ^ | 04:46 |
tristanC | I switched them back to firefox and make the console.log check for js warning to a noop | 04:47 |
*** CrayZee has joined #zuul | 04:58 | |
*** pcaruana has joined #zuul | 05:20 | |
openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: executor: add executing job INFO log https://review.openstack.org/578704 | 06:29 |
openstackgerrit | Tristan Cacqueray proposed openstack-infra/nodepool master: status: add pool column to nodepool list output https://review.openstack.org/579046 | 06:32 |
*** yolanda__ has joined #zuul | 06:42 | |
*** yolanda__ is now known as yolanda | 06:43 | |
*** gtema has joined #zuul | 06:46 | |
gtema | tobiash: ping | 07:46 |
*** jpena|off is now known as jpena | 07:56 | |
*** jpena has left #zuul | 07:56 | |
*** jpena has joined #zuul | 07:57 | |
openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: gerrit: use baseurl for change uris lookup https://review.openstack.org/579086 | 08:02 |
openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: gerrit: use baseurl for change uris lookup https://review.openstack.org/579086 | 08:02 |
gtema | tristianC: do you know any way of consuming environment variables in zuul.conf? I am creating Zuul@Openshift as APB (based on templates shared by Tobias) and want to add/provision sql connection. But I do not want to store pass in configmap. There doesn't seem a way of substituting variables in ConfigMap with Secrets in K8 so far. There is also no way to offload credentials for sqlalchemy. | 08:04 |
tobiash | gtema: you probably need to create a boot.sh that runs envsubst on the config file | 08:06 |
tristanC | gtema: as tobiash said, environment variable override is not supported atm | 08:06 |
gtema | tobiash, tristianC: sad. Thanks. boot.sh is not a proper solution, since then configmap update will not be handled correctly | 08:07 |
gtema | s/handled/detected | 08:07 |
tristanC | tobiash: could we just add a .format(os.environ) on zuul.conf string access? | 08:08 |
tristanC | rather .format(**os.environ) | 08:09 |
tobiash | gtema, tristanC: afaik a zuul.conf change requires a zuul restart | 08:09 |
tristanC | like that, dburi could be "mysql://{ZUUL_DB_USER}:{ZUUL_DB_PASSWORD}..." | 08:09 |
tobiash | tristanC: that could be an option, but that still requires the possibility to reload that config during runtime | 08:10 |
tobiash | maybe I'm wrong but I thought reconfiguration only re-reads main.yaml | 08:10 |
tristanC | tobiash: reconfiguration doesn't reload zuul.conf indeed | 08:10 |
gtema | maybe, wasn't pointing attention to that | 08:10 |
tobiash | tristanC, gtema: nodepool does a reload regularly so there it might help | 08:11 |
gtema | I think anyway adding env substitution would be helpful also to hide github secrets | 08:11 |
gtema | tobiash, tristianC: since you both are working with Zuul and Openshift, do you have interest in seeing APB of installing zuul on openshift? | 08:12 |
gtema | with "full-blown" ;-) s2i | 08:13 |
tristanC | gtema: environment isn't very secure for handling secrets as it can leaks through /proc/ read access | 08:13 |
gtema | tristianC: storing them in configmap isn't better I suppose | 08:13 |
tristanC | gtema: sure, a ready to use openshift template for zuul would be handy | 08:14 |
gtema | tristianC: it's not template, it's AnsiblePlaybookBundle, which works like template for provisioning services | 08:14 |
gtema | tristianC: anyway it's not yet ready for public, but will try to publish it in couple of days | 08:15 |
gtema | s2i's are ready. But the diskimage-builder is a hige challenge (as also tobiash noticed) | 08:15 |
gtema | current diskimage-builder is not capable of running in container even with basic rpm distros, due to selinux | 08:16 |
tobiash | gtema: cool | 08:16 |
gtema | ubuntu is not working out-of-box as well | 08:16 |
tobiash | gtema: what's the problem with selinux? | 08:17 |
tobiash | doesn't privileged solve that? | 08:17 |
gtema | runcon -t setfiles_mac_t -- | 08:17 |
gtema | from rpm-distro/cleanup.d/99-selinux-fixfiles-restore | 08:17 |
gtema | in container selinux is disabled, therefore it fails | 08:17 |
tobiash | gtema: which version of openshift you are using? | 08:18 |
gtema | privileged is solving losetup troubles, but I needed to add additional image flush, since in some cases after empty image is being created and tried to `losetup` it does not exist on fs | 08:18 |
gtema | 3.9 | 08:18 |
tobiash | I'm running 3.9 and I thought selinux is supported inside the containers | 08:19 |
tobiash | I'm running with the overlay2 storage backend | 08:19 |
gtema | it is "suported". From what I understand it is checked, but the container itself reports it being disabled. | 08:20 |
tobiash | gtema: ah maybe I'm not facing that issue because I don't have an rpm based image atm | 08:20 |
openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: config: enable using environ template format for secrets https://review.openstack.org/579091 | 08:20 |
gtema | tobiash: you are using old diskimage-builder from what I see | 08:20 |
gtema | and this is present in the newer one | 08:20 |
gtema | exactly this step of `runcon` | 08:20 |
tobiash | oh right, but that was due to failing to install some java package I don't use anymore | 08:21 |
tobiash | upgrading diskimage builder is on my backlog since some time... | 08:21 |
gtema | tobiash: you might want to wait with that ;-) | 08:22 |
gtema | selinux: https://serverfault.com/questions/757606/how-to-enable-selinux-inside-of-a-centos-docker-container/757609 | 08:22 |
tobiash | gtema: thanks for that hint so I should probably try that in the test environment first... | 08:23 |
gtema | yupp | 08:24 |
*** Shrews has joined #zuul | 08:25 | |
*** Shrews has joined #zuul | 08:25 | |
*** Shrews has quit IRC | 08:25 | |
gtema | tobiash, tristianC: ok. I will further polish APB installation, publish it and present here. If there would be interest I would be happy to move development of it under Openstack from my copmany | 08:27 |
gtema | those privileged are however making troubles - you can't really create scc or add scc to sa from APB. So this remains as a manual admin step | 08:28 |
tristanC | gtema: i'd like to work on making zuul be able to run in un-privileged container, but that needs some more discussions first | 08:33 |
gtema | tristianC: great. Me either | 08:33 |
tristanC | my plan is to be able to replace the bubblewrap execution driver by a k8s interface to delegate ansible-playbook execution context to another pod | 08:40 |
gtema | tristianC: this is nice. However we need not to forget, that diskimage-builder is really requiring privileged. And I guess there is also much work to do | 08:41 |
tristanC | gtema: perhaps the nodepool-builder service could be managed by nodepool-launcher? | 08:42 |
gtema | how would this help? launcher itself does not need privileged | 08:43 |
gtema | it is itself "strage" to create image for VM from container running inside of VM :-| | 08:43 |
gtema | s/strage/strange | 08:44 |
tristanC | gtema: e.g. nodepool-launcher launch a builder vm inside each provider and to run disk-image-process inside openstack instead | 08:46 |
gtema | tristian: ah so, very elegant +1 | 08:47 |
*** electrofelix has joined #zuul | 08:53 | |
gtema | tobiash: hint - diskimage-builder==2.6.1 does not have problem with selinux (rpm distros). This is last release before the breaking change was introduced. I've opened a bug against diskimage-builder: https://bugs.launchpad.net/diskimage-builder/+bug/1779273 | 08:55 |
openstack | Launchpad bug 1779273 in diskimage-builder "building images in container fails due to selinux fixes" [Undecided,New] | 08:55 |
tobiash | gtema: thanks | 08:55 |
gtema | welcome | 08:55 |
*** spsurya has quit IRC | 09:14 | |
openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: executor: add executing job INFO log https://review.openstack.org/578704 | 09:14 |
*** jesusaur has quit IRC | 09:32 | |
*** jesusaur has joined #zuul | 09:52 | |
*** sshnaidm is now known as sshnaidm|off | 10:02 | |
*** electrofelix has quit IRC | 11:02 | |
*** jpena is now known as jpena|lunch | 11:06 | |
*** gregdek is now known as Guest62962 | 11:19 | |
*** sshnaidm|off has quit IRC | 11:29 | |
*** nchakrab has joined #zuul | 11:30 | |
*** bcoca has joined #zuul | 11:35 | |
*** bcoca has joined #zuul | 11:35 | |
*** electrofelix has joined #zuul | 11:40 | |
*** jpena|lunch is now known as jpena | 12:07 | |
*** hashar has joined #zuul | 12:41 | |
*** rlandy has joined #zuul | 12:42 | |
*** Shrews has joined #zuul | 12:42 | |
*** openstackgerrit has quit IRC | 12:49 | |
*** bcoca has quit IRC | 12:50 | |
*** zxiiro is now known as zxiiro-pto | 13:20 | |
*** hashar has quit IRC | 13:27 | |
*** snapiri- has joined #zuul | 13:32 | |
*** elyezer_ has joined #zuul | 13:32 | |
*** snapiri- has quit IRC | 13:33 | |
*** snapiri- has joined #zuul | 13:33 | |
*** CrayZee has quit IRC | 13:35 | |
*** elyezer has quit IRC | 13:36 | |
fungi | tristanC: i've always thought it would be cool if nodepool could handle elastic scaling of zuul executor and merger servers too | 13:42 |
fungi | rather, if zuul+nodepool could do that | 13:42 |
mnaser | mordred: the change is in (i might be late), zuul status page is so clean! | 14:22 |
mnaser | it's so much easier on my cpu/browser too | 14:22 |
mordred | mnaser: yay! | 14:30 |
*** nchakrab has quit IRC | 14:42 | |
*** nchakrab has joined #zuul | 14:43 | |
*** nchakrab has quit IRC | 14:47 | |
*** openstackgerrit has joined #zuul | 14:54 | |
openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Make websocket streaming more event-driven https://review.openstack.org/558646 | 14:54 |
Shrews | TIL that I have no idea how log streaming works today ^^^ | 14:56 |
Shrews | I also learned that a total VM rebuild at a non-sleeping 4am helps IRC disconnects | 14:57 |
mordred | Shrews: do any of us understand how log streaming works? | 14:57 |
Shrews | mordred: none of us will until we recode all of this with C++ | 14:59 |
corvus | i'm really looking forward to porting all the javascript to c++ | 15:00 |
Shrews | totes | 15:00 |
openstackgerrit | Merged openstack-infra/zuul master: Translate zuulStartStream into typescript https://review.openstack.org/558618 | 15:05 |
mordred | Shrews, corvus: we can compile rust down into webasm ... | 15:08 |
*** openstack has quit IRC | 15:22 | |
*** openstack has joined #zuul | 15:26 | |
*** ChanServ sets mode: +o openstack | 15:26 | |
Shrews | corvus: mordred: I think we're gtg with merging https://review.openstack.org/578831 now based on feedback from tristanC | 15:28 |
corvus | ++ | 15:29 |
*** acozine1 has joined #zuul | 15:35 | |
*** acozine1 has left #zuul | 15:36 | |
*** acozine1 has joined #zuul | 15:38 | |
tobiash | mordred: that log streaming reminds me that we should also add a keep alive mechanism to the websocket streaming | 15:55 |
mordred | tobiash: ++ | 15:55 |
*** sshnaidm|off has joined #zuul | 15:55 | |
*** nchakrab has joined #zuul | 15:56 | |
*** nchakrab has quit IRC | 16:02 | |
*** tobasco has quit IRC | 16:06 | |
*** tobasco has joined #zuul | 16:08 | |
*** weshay has quit IRC | 16:09 | |
*** Guest19794 has quit IRC | 16:09 | |
*** mugsie has quit IRC | 16:09 | |
*** zigo has quit IRC | 16:09 | |
*** lennyb has quit IRC | 16:09 | |
clarkb | then compile the C++ to webasm and run it on javscript? | 16:11 |
*** weshay has joined #zuul | 16:14 | |
*** Guest19794 has joined #zuul | 16:14 | |
*** mugsie has joined #zuul | 16:14 | |
*** zigo has joined #zuul | 16:14 | |
*** lennyb has joined #zuul | 16:14 | |
tobiash | mordred: added a question on 558646 | 16:16 |
*** aspiers[m] has quit IRC | 16:18 | |
*** austinsun[m] has quit IRC | 16:18 | |
openstackgerrit | Merged openstack-infra/nodepool master: Fix for pools with different labels https://review.openstack.org/578831 | 16:21 |
mordred | tobiash: good question. I have responded - and fixed | 16:22 |
openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Make websocket streaming more event-driven https://review.openstack.org/558646 | 16:22 |
tobiash | yay I could help with js stuff :) | 16:23 |
*** mattclay has quit IRC | 16:25 | |
*** mattclay has joined #zuul | 16:26 | |
*** electrofelix has quit IRC | 16:29 | |
*** nchakrab has joined #zuul | 16:36 | |
*** nchakrab has quit IRC | 16:37 | |
*** pcaruana has quit IRC | 16:40 | |
*** rlandy is now known as rlandy|brb | 16:43 | |
openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Enable heartbeat on websocket streaming https://review.openstack.org/579222 | 16:43 |
tobiash | mordred: websocket keepalive might be as easy as this ^ | 16:44 |
openstackgerrit | Merged openstack-infra/nodepool master: zk: retry initial zookeeper connection attempts https://review.openstack.org/576047 | 16:50 |
*** jpena is now known as jpena|off | 17:00 | |
*** gtema has quit IRC | 17:01 | |
*** austinsun[m] has joined #zuul | 17:01 | |
*** openstack has quit IRC | 17:11 | |
*** openstack has joined #zuul | 17:15 | |
*** ChanServ sets mode: +o openstack | 17:15 | |
*** rlandy|brb is now known as rlandy | 17:16 | |
openstackgerrit | Merged openstack-infra/zuul master: Shift log streaming code into StreamComponent https://review.openstack.org/558619 | 17:17 |
*** aspiers[m] has joined #zuul | 17:19 | |
corvus | tobiash: did you see https://review.openstack.org/579033 ? | 17:25 |
tobiash | corvus: yes, cool, but only had a quick look so far | 17:27 |
corvus | tobiash: no rush, it's failing all the jobs and i haven't written tests yet. but i wondered if you think it'll address the problem sufficiently. | 17:28 |
tobiash | TIL about git log -L | 17:36 |
tobiash | I think that could work :) | 17:36 |
mordred | corvus: ooh neat | 17:53 |
corvus | yeah, it looks like if you give it exactly one line, you'll end up with exactly a one line diff for each commit | 17:54 |
*** kmalloc is now known as vmalloc | 18:06 | |
*** vmalloc is now known as kmalloc | 18:06 | |
*** elyezer_ is now known as elyezer | 18:09 | |
openstackgerrit | Jeremy Stanley proposed openstack-infra/zuul master: Change "core developer" references https://review.openstack.org/579241 | 18:26 |
fungi | that's neat. more convenient than pickaxe or iterating over blame | 18:32 |
openstackgerrit | Merged openstack-infra/zuul master: Enable heartbeat on websocket streaming https://review.openstack.org/579222 | 18:33 |
fungi | corvus: is the use case for that series that a linter job could leave inline comments on each thing it's complaining about? | 18:35 |
mordred | fungi: s/linter/any/ really - but yeah | 18:35 |
fungi | well, i was picking that as a likely example | 18:35 |
mordred | fungi: I imagine linter jobs would be ... yeah - first likely example | 18:35 |
fungi | i should have said analyzer | 18:36 |
mordred | fungi: a something | 18:36 |
mordred | fungi: a something and do a something and leave something for somebody | 18:36 |
fungi | as a more general class of thing with that sort of line-by-line reporting behavior | 18:36 |
fungi | though i suppose if a job caused a python exception to be raised, the exception and traceback could be inserted as a comment at the line where it was raised | 18:37 |
fungi | or similar for other languages which support line numbers in their debugging | 18:37 |
fungi | and presumably zuul's configuration validator could be one of the first users of that feature as well i guess, reporting inline on proposed configuration changes where the problem is suspected? | 18:38 |
*** elyezer has quit IRC | 18:39 | |
mordred | fungi: yah - I think reporting on those will be cool | 18:40 |
fungi | after all, zuul should be a first-class user of its own features ;) | 18:40 |
mordred | ++ | 18:42 |
*** elyezer has joined #zuul | 18:49 | |
corvus | yes. the use case is definitely a smurfer that leaves smurfs for smurfs. | 19:04 |
fungi | best use case EVAR | 19:08 |
*** yolanda_ has joined #zuul | 19:19 | |
*** yolanda__ has joined #zuul | 19:22 | |
*** yolanda has quit IRC | 19:23 | |
*** yolanda_ has quit IRC | 19:24 | |
*** acozine1 has quit IRC | 19:59 | |
*** SpamapS has quit IRC | 20:22 | |
openstackgerrit | David Moreau Simard proposed openstack-infra/zuul-jobs master: Ensure iptables-services is installed when persisting firewall config https://review.openstack.org/579283 | 20:47 |
dmsimard | pabelanger: ^ for some reason that package isn't installed in our RDO image | 20:47 |
dmsimard | breaks multi node firewall setup | 20:48 |
pabelanger | dmsimard: -1, but ya | 20:49 |
pabelanger | we can fix it in DIB too | 20:49 |
openstackgerrit | David Moreau Simard proposed openstack-infra/zuul-jobs master: Ensure iptables-services is installed when persisting firewall config https://review.openstack.org/579283 | 20:49 |
dmsimard | fixed | 20:49 |
clarkb | from openstack's perspective you really want to have a firewall configured and ready when the host boots | 20:50 |
dmsimard | pabelanger: yeah, I was going to send a patch to the dib element but figured I'd fix it here too | 20:50 |
clarkb | to protect the jobs from themselves | 20:50 |
dmsimard | clarkb: iptables is up and running, there's just no systemd unit file so doing a "systemctl restart iptables" or "systemctl start iptables" doesn't work | 20:50 |
dmsimard | ¯\_(ツ)_/¯ | 20:50 |
clarkb | dmsimard: how are the rules loaded? the kernel module is probably there just no rules | 20:50 |
dmsimard | I'm not sure if this image has any rules by default | 20:52 |
clarkb | ya I'm suggesting that you want those :) | 20:53 |
clarkb | if only to protect jobs from affecting each other on shared network segments | 20:53 |
dmsimard | yeah good point | 20:53 |
pabelanger | what do we do for single node jobs today | 20:54 |
clarkb | openstack bakes a firewall ruleset into the image | 20:54 |
pabelanger | right, forgot that | 20:54 |
dmsimard | it's in one of the elements | 20:54 |
pabelanger | I'd have to check what default SF.io elements do | 20:55 |
pabelanger | because that is where that node (rdo-centos-7) is coming from | 20:55 |
clarkb | (Updating the playbook seems fine too, we shouldn't assume firewalls will allow the traffic we need for the overlay so we should explicitly allow it ourselves which involves installing a firewll management toolchain) | 21:01 |
clarkb | but more generally I would encourage people running CI systems to have test node firewalls to keep jobs from affecting each other as much as possible | 21:02 |
pabelanger | agree, I think we could create something for base to ensure a basic firewall for all jobs | 21:14 |
pabelanger | rather then baking in to images | 21:14 |
pabelanger | but that is for another day | 21:14 |
clarkb | oh I prefer baking into images because then it is there on boot | 21:15 |
clarkb | waiting for a job to run is too late | 21:15 |
fungi | completely agree | 21:29 |
corvus | i think we've got the all clear from clarkb, Shrews, and tristanC for a nodepool release, so i'll tag fe80343 (current master) | 22:11 |
corvus | as 3.0.2 | 22:11 |
corvus | that look right? (clarkb) | 22:11 |
* clarkb double checks the version launchers ar erunning | 22:11 | |
clarkb | fe80343 is what I restarted on lgtm | 22:12 |
clarkb | oh quick question | 22:12 |
corvus | the only wonky thing about this upgrade is that it requires a full restart. but no other action. i could probably go either way on 3.1.0 vs 3.0.2 | 22:12 |
clarkb | was the zk schema change on a release already? | 22:12 |
corvus | clarkb: no, that's ^ | 22:13 |
clarkb | ya I think because it comes with a zk schema change and requires a full restart 3.1.0 might be appropriate | 22:13 |
corvus | https://zuul-ci.org/docs/nodepool/releasenotes.html#upgrade-notes | 22:13 |
corvus | okay. i'm convinced. :) | 22:13 |
corvus | any objections to 3.1.0 ? | 22:13 |
corvus | http://paste.openstack.org/show/724620/ | 22:15 |
clarkb | lgtm | 22:15 |
corvus | nodepool 3.1.0 pushed | 22:17 |
openstackgerrit | James E. Blair proposed openstack-infra/nodepool master: Publish docs on release https://review.openstack.org/579300 | 22:42 |
openstackgerrit | James E. Blair proposed openstack-infra/zuul-jobs master: WIP: Add a role to return file comments https://review.openstack.org/579033 | 23:02 |
*** eikke has quit IRC | 23:11 | |
openstackgerrit | James E. Blair proposed openstack-infra/nodepool master: Handle node no longer in pool error https://review.openstack.org/579309 | 23:14 |
tristanC | corvus: that's great thanks. | 23:17 |
corvus | tristanC: thanks for your help :) | 23:19 |
*** eikke has joined #zuul | 23:36 | |
*** rlandy has quit IRC | 23:41 | |
*** SpamapS has joined #zuul | 23:54 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!