*** smyers has joined #zuul | 00:46 | |
*** bhavikdbavishi has joined #zuul | 02:14 | |
SpamapS | who is giving this talk? https://www.meetup.com/OpenStack-LA/events/255390642 | 02:31 |
---|---|---|
pabelanger | SpamapS: I believe John Studarus, who is running packethost for openstack-infra | 02:36 |
pabelanger | via packet.net | 02:36 |
SpamapS | ahhh | 02:52 |
dmsimard | He's an openstack community ambassador iirc :) | 02:56 |
dmsimard | Yup: https://groups.openstack.org/ambassador-program | 02:56 |
*** bhavikdbavishi has quit IRC | 02:59 | |
SpamapS | sweet | 03:04 |
*** jesusaur has quit IRC | 03:21 | |
*** jesusaur has joined #zuul | 03:27 | |
*** bhavikdbavishi has joined #zuul | 03:41 | |
*** bhavikdbavishi1 has joined #zuul | 03:44 | |
*** bhavikdbavishi has quit IRC | 03:46 | |
*** bhavikdbavishi1 is now known as bhavikdbavishi | 03:46 | |
*** quiquell|off is now known as quiquell | 05:37 | |
*** pcaruana has quit IRC | 06:37 | |
*** slaweq has joined #zuul | 06:43 | |
*** bhavikdbavishi1 has joined #zuul | 06:49 | |
*** bhavikdbavishi has quit IRC | 06:51 | |
*** bhavikdbavishi1 has quit IRC | 06:54 | |
*** quiquell is now known as quiquell|brb | 06:59 | |
*** pcaruana has joined #zuul | 07:02 | |
*** threestrands has quit IRC | 07:20 | |
*** evrardjp_ is now known as evrardjp | 07:21 | |
*** fbo has joined #zuul | 07:32 | |
*** themroc has joined #zuul | 07:33 | |
*** quiquell|brb is now known as quiquell | 07:42 | |
*** ssbarnea has joined #zuul | 07:43 | |
*** AJaeger has quit IRC | 07:45 | |
*** AJaeger has joined #zuul | 07:48 | |
*** hashar has joined #zuul | 07:49 | |
*** bhavikdbavishi has joined #zuul | 07:55 | |
*** bhavikdbavishi has quit IRC | 07:55 | |
*** bhavikdbavishi has joined #zuul | 07:55 | |
*** anticw has quit IRC | 08:07 | |
*** hashar is now known as hasharAway | 08:15 | |
*** goern has joined #zuul | 09:15 | |
*** rfolco has joined #zuul | 09:31 | |
*** electrofelix has joined #zuul | 10:25 | |
*** hasharAway is now known as hashar | 11:08 | |
quiquell | Hello | 11:19 |
quiquell | It's possible to override ansible_python_interpreter from jobs "vars: " | 11:20 |
*** jesusaur has quit IRC | 11:31 | |
*** jesusaur has joined #zuul | 11:37 | |
tobiash | I don't think so, that's currently hard coded in zuul | 11:49 |
*** quiquell is now known as quiuell|lunch | 12:00 | |
*** bhavikdbavishi1 has joined #zuul | 12:07 | |
*** bhavikdbavishi has quit IRC | 12:09 | |
*** bhavikdbavishi1 is now known as bhavikdbavishi | 12:09 | |
openstackgerrit | Tobias Henkel proposed openstack-infra/nodepool master: Use quota handling code for min-ready declines https://review.openstack.org/610495 | 12:17 |
openstackgerrit | Tobias Henkel proposed openstack-infra/nodepool master: Remove unneeded todo comment https://review.openstack.org/610496 | 12:22 |
pabelanger | tobiash: quiuell|lunch: still hardcoded to python2 right now | 12:25 |
*** bhavikdbavishi1 has joined #zuul | 12:28 | |
*** bhavikdbavishi has quit IRC | 12:29 | |
*** bhavikdbavishi1 is now known as bhavikdbavishi | 12:29 | |
*** rlandy has joined #zuul | 12:34 | |
*** quiuell|lunch is now known as quiquell | 12:41 | |
quiquell | pabelanger: ack | 12:41 |
openstackgerrit | Tobias Henkel proposed openstack-infra/nodepool master: Ensure that completed handlers are removed frequently https://review.openstack.org/610029 | 12:41 |
*** EmilienM has quit IRC | 12:46 | |
*** EmilienM has joined #zuul | 12:47 | |
*** bhavikdbavishi has quit IRC | 13:04 | |
openstackgerrit | Tobias Henkel proposed openstack-infra/nodepool master: Use quota handling code for min-ready declines https://review.openstack.org/610495 | 13:07 |
openstackgerrit | Tobias Henkel proposed openstack-infra/nodepool master: Remove unneeded todo comment https://review.openstack.org/610496 | 13:07 |
*** themroc has quit IRC | 13:29 | |
*** themroc has joined #zuul | 13:30 | |
*** sshnaidm_ has joined #zuul | 13:53 | |
*** sshnaidm has quit IRC | 13:54 | |
*** jesusaur has quit IRC | 14:00 | |
mordred | pabelanger: we should probably swing back around and revisit that at some point - modules should be py3 safe at this point | 14:03 |
pabelanger | +1 | 14:04 |
*** jesusaur has joined #zuul | 14:07 | |
tobiash | mordred, pabelanger: how do you shard the providers in nodepool to different launcher instances? | 14:18 |
tobiash | mordred, pabelanger: our nodepool (single instance atm) is currently struggling to keep the usage at quota with two cloud providers (with two pools each) and a few static providers | 14:20 |
tobiash | and it's accepting node requests very slowly | 14:20 |
*** sshnaidm_ has quit IRC | 14:25 | |
*** quiquell is now known as quiquell|off | 14:26 | |
*** jamielennox has quit IRC | 14:27 | |
*** jamielennox has joined #zuul | 14:29 | |
mordred | tobiash: right now it's just manual - we have different nodepool.yaml files for different launchers | 14:32 |
tobiash | mordred: how many providers do you have per launcher? | 14:32 |
mordred | tobiash: http://git.openstack.org/cgit/openstack-infra/project-config/tree/nodepool the nl*.yaml files are the config files | 14:33 |
tobiash | thx | 14:33 |
mordred | tobiash: it varies - we've got some with 3 - some with 6 | 14:33 |
mordred | I think we've been trying to keep the total max-servers count somewhat even across launchers | 14:34 |
pabelanger | tobiash: yup, what mordred said. I've been toying with the idea of just doing a launcher per provider / region, but haven't made that switch yet | 14:34 |
pabelanger | I'd love to have multiple launchers per provider, but will wait until that is supported | 14:35 |
tobiash | in the code I found that it does a 10s sleep in the provider loop even if there are quite some requests to fulfill (and there is quota left) | 14:37 |
tobiash | https://git.zuul-ci.org/cgit/nodepool/tree/nodepool/launcher.py#n314 | 14:37 |
tobiash | because we probably skip many/several requests per loop because they are locked by other providers | 14:38 |
*** sshnaidm_ has joined #zuul | 14:45 | |
openstackgerrit | Merged openstack-infra/nodepool master: Update pypi metadata https://review.openstack.org/557738 | 14:53 |
tobiash | maybe we should skip that sleep under some conditions | 14:54 |
openstackgerrit | Merged openstack-infra/zuul master: Include enqueue and report timestamps in MQTT msg https://review.openstack.org/609355 | 15:03 |
*** jimi|ansible has quit IRC | 15:10 | |
openstackgerrit | Merged openstack-infra/zuul master: Retry jobs failed with MERGER_FAILURE https://review.openstack.org/609945 | 15:15 |
openstackgerrit | Merged openstack-infra/zuul master: docs: add Project Testing Interface guide https://review.openstack.org/571420 | 15:15 |
*** ssbarnea has quit IRC | 15:16 | |
*** chandankumar is now known as chkumar|off | 15:35 | |
*** themroc has quit IRC | 15:41 | |
*** sdoran has joined #zuul | 15:44 | |
*** openstackgerrit has quit IRC | 15:47 | |
*** openstackgerrit has joined #zuul | 15:49 | |
corvus | tristanC, mordred: it looks like the "builds" button at the top of the screen still displays even if there is no sql connection... i can't remember if that's a regression or not, but it does seem like something that could be improved... do we expose that in the info endpoint? | 16:16 |
mordred | corvus: we either do - or there is a patch floating out there to expose it - it's not a regression, it's been on the "we should improve this" list | 16:17 |
mordred | corvus: job_history in the Capabilities object is the flag that is supposed to indicate this | 16:18 |
mordred | corvus: oh - but I don't think that's wired up | 16:19 |
corvus | mordred: ok, i guess it was one of those things where i imagined the future was here today :) | 16:19 |
mordred | corvus: I frequently imagine that | 16:19 |
mordred | so we have a thing in the api response that is the flag that should be checked, but we are not yet setting it | 16:19 |
mordred | or reading it | 16:19 |
mordred | corvus: we do validateWebConfig on all of the Connection plugins in zuul.web - but we don't do a similar thing with the info object ... perhaps we should add a method to Connection that a plugin can use to update an Info object? | 16:22 |
mordred | corvus: that way we can have the sqlconnection plugin set job_history to True | 16:23 |
*** sshnaidm_ is now known as sshnaidm|afk | 16:24 | |
mrhillsman | do we have a way to ignore a pr via commit message or another mechanism for github driver | 16:27 |
mrhillsman | like the depends-on syntax has special implications | 16:28 |
mrhillsman | use case is an update to like readme for example that there is no need to run jobs on | 16:29 |
mrhillsman | or is that just a matter of needing better config-fu | 16:29 |
corvus | mordred: that sounds like a good idea | 16:30 |
corvus | mrhillsman: jobs can have 'files' or 'irrelevant-files' matchers, so you can configure jobs to ignore changes that only touch certain files with, for example, "irrelevant-files: README" | 16:31 |
mrhillsman | ahhh...did not know that, awesome, ty sir | 16:31 |
corvus | mrhillsman: https://zuul-ci.org/docs/zuul/user/config.html#attr-job.irrelevant-files has more info | 16:32 |
mrhillsman | reading is fundamental hehe | 16:33 |
mrhillsman | asking is quicker | 16:33 |
mrhillsman | :) | 16:33 |
* corvus is a documentation oracle | 16:33 | |
mrhillsman | ++ | 16:34 |
mrhillsman | i was just looking in the wrong place :( | 16:34 |
corvus | yeah, that's the heart of the documentation, but it's a bit too buried. i need to figure out a way to make that easier to find and work with. | 16:37 |
*** ssbarnea has joined #zuul | 16:38 | |
clarkb | corvus: https://review.openstack.org/#/c/610159/ adds zuul quick start job to nodepool since that job builds nodepool from source it could break on the zuul side without gating them together | 16:39 |
corvus | clarkb: thanks! | 16:40 |
mordred | corvus: speaking of that - in the #openstack-infra scrollback, there was a discussion about the gerrit used in the git-review testing and I brought up the zuul quick-start job as a potential model | 16:41 |
pabelanger | q: For zuul / github, I'd like to write a pipeline trigger when somebody 'edits' a specific comment (eg: 1st in the PR) zuul reacts to it. Looking at https://developer.github.com/v3/activity/events/types/#issuecommentevent I should be able to collect all the information, but unsure how to best add this to zuul today: https://git.zuul-ci.org/cgit/zuul/tree/zuul/driver/github/githubconnection.py#n317 This is all | 16:43 |
pabelanger | around if / when somebody updated commit 1 to add depends-on header. Today, zuul doesn't see that an a user needs to then do a 'recheck'. | 16:43 |
*** hashar is now known as hasharAway | 16:46 | |
openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Add status page to the quick-start https://review.openstack.org/610660 | 16:48 |
clarkb | pabelanger: you could probably do a comment matcher like for recheck that matched on depends on? I don't think that will give you a delta though so it may run more than you like | 16:50 |
corvus | mordred, clarkb, Shrews: ^ that closes the circle on the status stuff | 16:51 |
*** ssbarnea has quit IRC | 16:51 | |
pabelanger | clarkb: right now, it seems we hardcode issue_comment event to only be created. Guess we'd have to allow edit | 16:52 |
*** jimi|ansible has joined #zuul | 16:52 | |
corvus | pabelanger: the "body" of the pull request is what is consulted for depends-on headers... is that the same thing as the first comment? | 16:55 |
clarkb | corvus: aiui yes | 16:55 |
clarkb | this tripped me up a bunch getting zuul with kata going | 16:55 |
corvus | pabelanger: in that case, i think you're on the right track. does github provide any special indication that a comment edit is for the first comment/pr body? is it a special kind of event, or are any special flags set in the event? | 16:56 |
clarkb | to update a depends on I have to go back and edit the first commit. A small concern is I'm not sure if that means other people can fix or set depends on or if it is PR owner only | 16:56 |
corvus | clarkb: did you meant 'comment' rather than 'commit'? | 16:57 |
clarkb | yes sorry | 16:57 |
corvus | ok good :) | 16:57 |
pabelanger | I don't believe there is a special event for the first commit. We'd need to somehow map that out using issue object I think, it has number field, which I assume is related to which comment in a PR | 16:58 |
pabelanger | I haven't tested that yet | 16:58 |
corvus | pabelanger: did you mean "first comment" rather than "first commit"? | 16:58 |
pabelanger | yes, sorry first comment | 16:58 |
corvus | ok good :) | 16:58 |
pabelanger | but, this is just a guess. I need to actually test | 16:59 |
corvus | pabelanger: so yeah, i think the question is whether there's some combination of existing data that we can just expose (like "triggers: github: comment-edited: sequence:1") or whether we need a new synthetic event ("triggers: github: pr-body-edited"). | 17:01 |
corvus | pabelanger: it sounds probable that we can work something out and it's worth continuing to explore. tobiash and jlk may be of help. | 17:02 |
pabelanger | corvus: okay good to know, not a major issue, but small annoyance in github UI when needing to add depends-on. When I have time, I'll dig more into it | 17:04 |
corvus | ++ | 17:04 |
*** ssbarnea has joined #zuul | 17:06 | |
corvus | clarkb: you found a fascinating edge case in depends-on: https://review.openstack.org/610159 | 17:07 |
corvus | clarkb: that worked as long as the target change had not merged, but once it had merged, it didn't work any more :) | 17:07 |
clarkb | corvus: do you know why? | 17:08 |
openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Add Zuul to zuul-quick-start https://review.openstack.org/610664 | 17:08 |
clarkb | is it because the job def changes? | 17:08 |
corvus | clarkb: it's because while the depends-on change was unmerged, zuul included zuul in the projects list (due to the dependency), but once it had merged, it wasn't necessary, so it wasn't there. | 17:09 |
corvus | clarkb: 610664 fixes the issue | 17:09 |
openstackgerrit | James E. Blair proposed openstack-infra/nodepool master: Run zuul-quick-start job https://review.openstack.org/610159 | 17:09 |
clarkb | oh you know what I chceked that inr everse. I made sure nodepool was in the required projects list for zuul | 17:09 |
corvus | clarkb: yep. me too. :) | 17:10 |
corvus | i don't have an easy solution to the zuul config puzzle. it's a fun one. :) | 17:10 |
*** ssbarnea_ has joined #zuul | 17:19 | |
*** ssbarnea has quit IRC | 17:23 | |
corvus | i was thinking it might be good to add mysql to the quickstart, so that the builds page works anyway... | 17:23 |
*** ssbarnea_ has quit IRC | 17:31 | |
*** panda is now known as panda|off | 17:37 | |
mordred | corvus: ++ | 17:57 |
*** ssbarnea has joined #zuul | 17:58 | |
corvus | mordred: http://paste.openstack.org/show/732166/ | 17:59 |
corvus | mordred: that appears to be a mysql deadlock, but i'm not sure how. it's entirely possible the second connection is another zuul component racing with the first. they may both be trying to create the tables. but one of them says "SLEEP" so it's hard to confirm. | 18:00 |
corvus | mordred: should i just assume that's the case and try to avoid the deadlock, or is there something else i can do to confirm/deny that and see if there's something else going on? | 18:00 |
corvus | mordred: i ran 'kill 8' (the sleeping one) and deadlock was broken | 18:01 |
corvus | so i guess that's confirmation of a sort :) | 18:01 |
*** dmellado has quit IRC | 18:05 | |
mordred | corvus: I agree with all of your words | 18:05 |
*** dmellado has joined #zuul | 18:05 | |
mordred | corvus: although this doesn't come up super-often - it does show a potential gotcha with our auto-sqlalchemy approach when starting up a pile of things | 18:07 |
corvus | mordred: yeah... i'm not sure how to deal with this... | 18:07 |
corvus | in zuul v4, i'd suggest we lock a znode and let one component create tables | 18:07 |
corvus | but i'm not sure how to avoid it in zuul v3. we could probably only have the scheduler do it, since it's a singleton; but there's some complexity to that. | 18:08 |
mordred | maybe in zuul v3 we do $something so that only the scheduler process tries to create tables? | 18:08 |
corvus | mordred: i don't suppose mysql has a nice way to avoid this? | 18:08 |
mordred | mysql has a named lock facility - one sec, I get you link | 18:09 |
mordred | corvus: https://dev.mysql.com/doc/refman/8.0/en/miscellaneous-functions.html#function_get-lock | 18:09 |
mordred | corvus: I'm also not 100% sure how to wire that in - or whether that would be more or less work than using the scheduler as a singleton | 18:11 |
corvus | cool that might work. i'll poke at that and the scheduler-only idea and see which is easiest. | 18:11 |
corvus | exactly :) | 18:11 |
pabelanger | jlk: do you happen to know what type of event an email reply for comment triggers? | 18:11 |
mordred | pabelanger: I assume jlk knows all things | 18:11 |
pabelanger | jlk: for exmaple: https://github.com/ansible-network/vyos/pull/31#issuecomment-429956180 | 18:11 |
pabelanger | randomly tried 'recheck' via email, didn't work | 18:12 |
jlk | pabelanger: an email reply? I thought emails were just a way to get a comment in there, and it would generate a comment event. | 18:12 |
jlk | issue comment | 18:12 |
pabelanger | okay, I'll ask to look at zuul logs in a bit | 18:13 |
*** electrofelix has quit IRC | 18:13 | |
jlk | 99.9% certain it'd be https://developer.github.com/v3/activity/events/types/#issuecommentevent | 18:13 |
pabelanger | ack | 18:13 |
*** ssbarnea|bkp2 has joined #zuul | 18:13 | |
jlk | If you have access to the admin page of the app in question, you would be able to see a listing of all the events that have come in | 18:14 |
pabelanger | Agree, sadly not an admin | 18:14 |
corvus | mordred: oh, right, we support postgres. | 18:16 |
mordred | corvus: yah | 18:16 |
pabelanger | mordred: mnaser: here is a weird traceback from upload-logs-swift with vexxhost: https://softwarefactory-project.io/paste/show/1260/ | 18:16 |
corvus | mordred: i guess i'll do the singleton. | 18:17 |
pabelanger | I don't think the issue is cloud side, but something weird with zuul-executor and DNS maybe? | 18:17 |
mordred | pabelanger: yeah - that looks like you're trying to hit vexxhost but instead hitting softwarefactory-project.io | 18:17 |
mordred | pabelanger: so I'd vote for environmental in sf.io - and I don't have much insight into what might cause that | 18:18 |
pabelanger | mordred: yes, same, so far only once. Was our first post_failure for swift in almost a month of using | 18:19 |
mordred | \o/ | 18:23 |
mordred | pabelanger: that's great new | 18:23 |
pabelanger | Yah, so far been happy how well it is working | 18:26 |
openstackgerrit | James E. Blair proposed openstack-infra/zuul master: SQL: only create tables in scheduler https://review.openstack.org/610696 | 18:30 |
openstackgerrit | James E. Blair proposed openstack-infra/zuul master: WIP: add mysql to quick-start https://review.openstack.org/610697 | 18:31 |
corvus | mordred: ^ that might work. i think i need to figure out how to build images locally so i can test locally | 18:32 |
corvus | (i mean, that shouldn't be hard, i have the commands in a playbook :) | 18:32 |
corvus | but maybe zuul will get around to running the quick-start test on the second patch first :) | 18:32 |
mordred | corvus: yes - being able to build images locally might wind up being a thing you find yourself wanting to do | 18:35 |
*** ssbarnea has quit IRC | 18:50 | |
corvus | that was pretty easy | 18:56 |
*** sshnaidm|afk has quit IRC | 19:06 | |
*** jimi|ansible has quit IRC | 19:30 | |
*** hasharAway has quit IRC | 19:31 | |
openstackgerrit | Merged openstack-infra/zuul master: Add status page to the quick-start https://review.openstack.org/610660 | 20:07 |
*** j^2 has quit IRC | 20:12 | |
*** j^2 has joined #zuul | 20:18 | |
*** j^2 has quit IRC | 20:20 | |
openstackgerrit | Merged openstack-infra/zuul master: Add Zuul to zuul-quick-start https://review.openstack.org/610664 | 20:35 |
openstackgerrit | Monty Taylor proposed openstack/pbrx master: Allow mounting replacement source in a volume https://review.openstack.org/610726 | 20:48 |
corvus | i filled up my disk with docker images | 20:57 |
jlk | thumbsup | 20:58 |
mordred | corvus: you're not cloud native enough | 21:03 |
mordred | corvus: if you were more cloud native, all your storage would be ephemeral | 21:03 |
clarkb | there is a docker rmi prune command | 21:04 |
clarkb | something like that at least, that should remove images that aren't needed for any defined containers | 21:04 |
mordred | clarkb: maybe it should have been called "vacuum" | 21:05 |
corvus | "docker image prune" | 21:09 |
corvus | also "docker volume prune" | 21:11 |
corvus | that was 6.784 GB | 21:11 |
clarkb | volume prune is scarier if you run real services on docker as you may want the volume to persist across time even when not associated with a container today | 21:12 |
clarkb | but ya in the context of the zuul quickstart probably fine | 21:12 |
mordred | corvus: Total reclaimed space: 8.868GB | 21:15 |
mordred | corvus: yeah. it adds up | 21:16 |
clarkb | I noticed something is creating "temporary" volumes when doing zuul quick start in addition to the specified volumes in the composefile | 21:16 |
corvus | clarkb: yeah. "docker rm -v" will apparently "remove container and its volumes" | 21:16 |
corvus | clarkb: well, some images specify they use volumes internally (gerrit for example) | 21:17 |
corvus | it has like 5 volumes | 21:17 |
clarkb | ah | 21:17 |
clarkb | that is probably it then | 21:17 |
clarkb | corvus: still plan to fix https://review.openstack.org/#/c/605856/2 ? I'm curious to start using that logging data | 21:35 |
corvus | mordred: RAN: /usr/bin/docker exec b64f8c1d2529d17b008cfa0ac1faf056fefc2ffd83a43c662aeb790ed858f025 sh -c mkdir /usr/src | 21:44 |
corvus | mordred: mkdir: cannot create directory ‘/usr/src’: File exists | 21:44 |
corvus | mordred: that seems like a minor design flaw | 21:44 |
corvus | clarkb: yeah... but i'm knee deep right now. i can get to it in a day or two, or if you want to fix it sooner that's ok with me | 21:45 |
clarkb | ok | 21:45 |
clarkb | I'll probably throw up a patch later today | 21:46 |
*** openstackgerrit has quit IRC | 21:56 | |
*** openstackgerrit has joined #zuul | 21:58 | |
*** openstackgerrit has quit IRC | 21:58 | |
*** openstackgerrit has joined #zuul | 22:01 | |
*** slaweq has quit IRC | 22:05 | |
mordred | corvus: oh - piddle. yeah. | 22:06 |
*** openstackgerrit has quit IRC | 22:12 | |
*** openstackgerrit has joined #zuul | 22:15 | |
*** openstackgerrit has quit IRC | 22:43 | |
*** jesusaur has quit IRC | 22:45 | |
*** jesusaur has joined #zuul | 22:53 | |
*** caphrim007 has joined #zuul | 22:53 | |
*** openstackgerrit has joined #zuul | 23:24 | |
openstackgerrit | Clark Boylan proposed openstack-infra/zuul master: Provide some accounting of node usage in logs https://review.openstack.org/605856 | 23:24 |
openstackgerrit | Monty Taylor proposed openstack/pbrx master: Allow mounting replacement source in a volume https://review.openstack.org/610726 | 23:26 |
*** caphrim007 has quit IRC | 23:30 | |
dmsimard | corvus: got a good genuine laugh out of "i filled up my disk with docker images", thanks for that | 23:32 |
dmsimard | It's the modern equivalent of what would've been on bash.org | 23:33 |
mordred | dmsimard: :) | 23:35 |
dmsimard | mordred: it's the "that was pretty easy" that came a few hours before that made it good | 23:37 |
mordred | dmsimard: ++ | 23:53 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!