*** sarob_ has quit IRC | 00:00 | |
*** otherwiseguy has quit IRC | 00:07 | |
mordred | SpamapS: I still stand by my statement that any of these services having ports at all is stupid | 00:11 |
---|---|---|
*** Ryan_Lane has joined #openstack-infra | 00:13 | |
SpamapS | mordred: keystone is the one... | 00:13 |
*** otherwiseguy has joined #openstack-infra | 00:13 | |
mordred | SpamapS: keystone's port should be "443" | 00:13 |
SpamapS | yeah or even 80 with starttls .. I would agree with that position. But .. Ipv4 strikes again. :-/ | 00:14 |
mordred | SpamapS: you know - I should hit "https://keystone.hpcloud.net" - which should give me other endpoints which could look like "https://glance.hpcloud.net" | 00:14 |
mordred | OR | 00:14 |
mordred | "https://hpcloud.net/keystone" and "https://hpcloud.net/glance" | 00:14 |
mordred | this has been a solved problem for 30 years | 00:14 |
mordred | why the HELL we're re-inventing it and making it harder is beyond me | 00:15 |
*** sarob has joined #openstack-infra | 00:15 | |
mordred | SpamapS: I do not see what IPv4 has to do with anything | 00:15 |
mordred | unless the problem is "can't figure out how to configure virtualhost and proxypass" | 00:16 |
mordred | in which case - stop trying to run cloud software and go back to writing iphone apps | 00:16 |
* mordred grumps around and strokes his beard | 00:16 | |
SpamapS | mordred: Well, SSL requires a 1 IP to 1 certificate relationship.... | 00:16 |
mordred | SpamapS: yup. but you already need an ip for the https://hpcloud.net entrypoint | 00:17 |
mordred | if you LB/proxypass through URLS to the sub-endpoints you're fine if you only have 1 IP | 00:17 |
mordred | of course, if you only have 1 ip and you're tring to run a cloud you're going to run in to other issues ... | 00:17 |
mordred | :) | 00:17 |
SpamapS | mordred: you're pushing an architecture on people.. we can't possibly ask people to do things the usual way. EVER. | 00:18 |
SpamapS | They'll... they'll cry. | 00:18 |
*** gokrokve has joined #openstack-infra | 00:18 | |
mordred | SpamapS: oh. right. I'm sorry. I forgot. we should make terrible decisions because someone somewhere might not want to learn antyhing about anything | 00:18 |
mordred | and would prefer to just hipster their apple to make bouncing icons and expect that to produce a working cloud | 00:19 |
SpamapS | mordred: or even worse, someone, somewhere, might have learned something else than the entire rest of the ops world learned. | 00:19 |
*** sarob has quit IRC | 00:19 | |
*** otherwiseguy has quit IRC | 00:19 | |
mordred | SpamapS: btw- matsunobu-san wrote a great blog-post on the new-improved semi-sync replication | 00:19 |
mordred | SpamapS: it has a new mode | 00:20 |
mordred | that's pretty good | 00:20 |
mordred | SpamapS: which of course made me think about all of the "MySQL can't possibly be used at scale for real things" people :) | 00:20 |
mordred | SpamapS: http://yoshinorimatsunobu.blogspot.com/2014/04/semi-synchronous-replication-at-facebook.html | 00:21 |
mordred | other people who want to learn about how massive scale mysql deployments really deal with HA ^^ | 00:21 |
SpamapS | semi-sync was pretty good already for the DR case IIRC | 00:22 |
mordred | SpamapS: yeah - they added a new thing called "loss-less semi-sync" | 00:22 |
mordred | it changes when the master commit is commited, so that you don't potentially show a commit to other clients before it's propagated to at least one slave | 00:23 |
mordred | SpamapS: which allows them to be able to combine it with slave promotion safely | 00:23 |
mordred | SpamapS: also, matsunobu-san is pretty ludicrously smart and thorough | 00:24 |
SpamapS | mordred: nice | 00:26 |
SpamapS | mordred: I still think Galera is easier to reason about and thus a more generically consumable thing. | 00:27 |
SpamapS | mordred: at the cost of requiring your app to be resilient to rollbacks. | 00:28 |
SpamapS | and that whole "have to write everything everywhere" bit :) | 00:29 |
mordred | yah | 00:29 |
mordred | I agree - galera seems to be a great small to mid-sized solution | 00:29 |
mordred | I think we may want to think about a loss-less semi-sync deployment and what that would look like just for completeness - but it may be more trouble than it's worth until we start hitting mega clouds | 00:30 |
SpamapS | mordred: thing is, I'd almost go for sharding galera before I'd go for the loss-less semi-sync because our challenge isn't page load time... it is simply scale-out | 00:32 |
SpamapS | but yeah.. at some point.. when we're topping 500,000 vms and 10k + tenants .. we'll probably want something better | 00:35 |
SpamapS | like MongoDB | 00:35 |
SpamapS | cause its webscale | 00:35 |
* SpamapS goes back to Saturday | 00:35 | |
harlowja_at_home | rewrite it all in node.js to, webscale -> infinity | 00:37 |
*** adrian_otto has quit IRC | 00:37 | |
*** harlowja_at_home has quit IRC | 00:57 | |
mordred | god. I look away for a second and I come back and see the two solutinos that have been suggested as "mongo" and "node.js" | 01:07 |
*** aconrad_ has joined #openstack-infra | 01:14 | |
*** sarob has joined #openstack-infra | 01:15 | |
*** alexpilotti has joined #openstack-infra | 01:15 | |
*** yamahata has joined #openstack-infra | 01:19 | |
*** sarob has quit IRC | 01:19 | |
*** alexpilotti has quit IRC | 01:24 | |
fungi | SpamapS: domain wildcard certs let you do http 1.1 host headers to have multiple https subdomains served as vhosts on one common ip address | 01:31 |
fungi | though srv rrs in dns could be another option for things like application clients too (unlike for browsers, where they still haven't caught on after more than a decade) | 01:33 |
*** Ryan_Lane has quit IRC | 01:37 | |
*** michchap has joined #openstack-infra | 01:42 | |
*** rustlebee has quit IRC | 01:54 | |
*** wenlock has quit IRC | 01:58 | |
*** homeless has quit IRC | 02:15 | |
*** sarob has joined #openstack-infra | 02:15 | |
*** wenlock has joined #openstack-infra | 02:16 | |
*** homeless has joined #openstack-infra | 02:16 | |
*** sarob has quit IRC | 02:19 | |
*** otherwiseguy has joined #openstack-infra | 02:23 | |
*** praneshp has quit IRC | 02:32 | |
*** greghaynes has quit IRC | 02:34 | |
*** otherwiseguy has quit IRC | 02:36 | |
*** olaph has quit IRC | 02:52 | |
*** Longgeek has joined #openstack-infra | 02:52 | |
*** harlowja_at_home has joined #openstack-infra | 02:53 | |
*** harlowja_at_home has quit IRC | 02:57 | |
*** mriedem has joined #openstack-infra | 03:01 | |
*** otherwiseguy has joined #openstack-infra | 03:03 | |
*** mriedem has quit IRC | 03:07 | |
*** penguinRaider has quit IRC | 03:08 | |
*** changbl has quit IRC | 03:08 | |
*** otherwiseguy has quit IRC | 03:11 | |
*** sarob has joined #openstack-infra | 03:15 | |
*** sarob has quit IRC | 03:19 | |
*** otherwiseguy has joined #openstack-infra | 03:23 | |
*** harlowja_at_home has joined #openstack-infra | 03:23 | |
*** fifieldt_ has quit IRC | 03:23 | |
*** fifieldt_ has joined #openstack-infra | 03:24 | |
*** Ryan_Lane has joined #openstack-infra | 03:39 | |
*** harlowja_at_home has quit IRC | 03:41 | |
*** penguinRaider has joined #openstack-infra | 03:48 | |
*** otherwiseguy has quit IRC | 03:48 | |
*** Longgeek has quit IRC | 03:52 | |
*** Longgeek has joined #openstack-infra | 04:00 | |
*** Ryan_Lane has quit IRC | 04:00 | |
*** homeless has quit IRC | 04:10 | |
*** homeless has joined #openstack-infra | 04:10 | |
*** godp1301 has joined #openstack-infra | 04:12 | |
*** sarob has joined #openstack-infra | 04:15 | |
*** godp1301 has quit IRC | 04:15 | |
*** sarob has quit IRC | 04:19 | |
*** Longgeek has quit IRC | 04:27 | |
*** harlowja_at_home has joined #openstack-infra | 04:41 | |
*** Longgeek has joined #openstack-infra | 04:43 | |
*** yfried has joined #openstack-infra | 05:00 | |
*** talluri has joined #openstack-infra | 05:03 | |
*** sarob has joined #openstack-infra | 05:15 | |
*** sarob has quit IRC | 05:20 | |
*** Longgeek has quit IRC | 05:25 | |
*** hunner has joined #openstack-infra | 05:31 | |
*** hunner has joined #openstack-infra | 05:31 | |
*** hunner is now known as Hunner | 05:32 | |
*** harlowja_at_home has quit IRC | 05:53 | |
*** basha has joined #openstack-infra | 06:14 | |
*** sarob has joined #openstack-infra | 06:15 | |
*** wenlock has quit IRC | 06:17 | |
*** sarob has quit IRC | 06:19 | |
*** Longgeek has joined #openstack-infra | 06:20 | |
*** praneshp has joined #openstack-infra | 06:26 | |
*** Longgeek has quit IRC | 06:29 | |
*** Longgeek has joined #openstack-infra | 06:29 | |
*** homeless has quit IRC | 06:38 | |
*** talluri has quit IRC | 07:04 | |
*** nati_ueno has joined #openstack-infra | 07:04 | |
*** talluri has joined #openstack-infra | 07:04 | |
*** pcrews has quit IRC | 07:05 | |
*** talluri has quit IRC | 07:09 | |
*** sarob has joined #openstack-infra | 07:10 | |
*** sarob has quit IRC | 07:14 | |
*** sarob has joined #openstack-infra | 07:15 | |
*** sarob has quit IRC | 07:19 | |
*** basha has quit IRC | 07:22 | |
*** rcarrill` has joined #openstack-infra | 07:31 | |
*** rcarrillocruz has quit IRC | 07:32 | |
*** talluri has joined #openstack-infra | 07:35 | |
*** talluri has quit IRC | 07:39 | |
*** basha has joined #openstack-infra | 07:40 | |
*** _nadya_ has joined #openstack-infra | 07:41 | |
*** _nadya_ has quit IRC | 07:42 | |
*** kmartin has quit IRC | 07:44 | |
*** kmartin has joined #openstack-infra | 07:44 | |
*** basha has quit IRC | 07:45 | |
*** kmartin has quit IRC | 07:47 | |
*** kmartin has joined #openstack-infra | 07:47 | |
*** basha has joined #openstack-infra | 07:55 | |
*** basha has quit IRC | 07:57 | |
*** _nadya_ has joined #openstack-infra | 08:03 | |
*** talluri has joined #openstack-infra | 08:04 | |
*** praneshp has quit IRC | 08:07 | |
*** talluri has quit IRC | 08:08 | |
*** sarob has joined #openstack-infra | 08:15 | |
*** basha has joined #openstack-infra | 08:20 | |
*** sarob has quit IRC | 08:20 | |
*** mestery has quit IRC | 08:29 | |
*** mestery_ has joined #openstack-infra | 08:29 | |
*** nati_ueno has quit IRC | 08:31 | |
*** andreykurilin_ has joined #openstack-infra | 08:34 | |
*** zzelle has joined #openstack-infra | 08:45 | |
*** _nadya_ has quit IRC | 08:45 | |
*** basha has quit IRC | 08:47 | |
*** achuprin has quit IRC | 08:59 | |
*** talluri has joined #openstack-infra | 09:04 | |
*** talluri has quit IRC | 09:09 | |
*** achuprin has joined #openstack-infra | 09:14 | |
*** sarob has joined #openstack-infra | 09:15 | |
*** toabctl has joined #openstack-infra | 09:16 | |
*** sarob has quit IRC | 09:20 | |
*** basha has joined #openstack-infra | 09:40 | |
*** andreykurilin_ has quit IRC | 10:09 | |
*** alexpilotti has joined #openstack-infra | 10:10 | |
*** sarob has joined #openstack-infra | 10:15 | |
*** sarob has quit IRC | 10:19 | |
*** basha has quit IRC | 10:20 | |
*** alexpilotti has quit IRC | 10:21 | |
*** basha has joined #openstack-infra | 10:26 | |
*** HenryG has quit IRC | 10:26 | |
*** basha has quit IRC | 10:55 | |
*** salv-orlando has quit IRC | 11:01 | |
*** rcarrill` has quit IRC | 11:02 | |
*** sarob has joined #openstack-infra | 11:15 | |
*** sarob has quit IRC | 11:19 | |
*** lcheng has joined #openstack-infra | 11:45 | |
*** paul-- has quit IRC | 12:00 | |
*** paul-- has joined #openstack-infra | 12:02 | |
*** tnurlygayanov has quit IRC | 12:03 | |
*** tnurlygayanov has joined #openstack-infra | 12:06 | |
*** sarob has joined #openstack-infra | 12:15 | |
*** sarob has quit IRC | 12:20 | |
*** jerryz has quit IRC | 12:20 | |
*** yolanda has quit IRC | 12:21 | |
*** mriedem has joined #openstack-infra | 12:23 | |
*** lcheng has quit IRC | 12:33 | |
*** trinaths has joined #openstack-infra | 12:43 | |
trinaths | Hello all | 12:43 |
openstackgerrit | Matt Riedemann proposed a change to openstack-infra/elastic-recheck: Make query for ssh timeout bug 1298472 more general https://review.openstack.org/98631 | 12:44 |
uvirtbot | Launchpad bug 1298472 in tempest "tempest.scenario.test_volume_boot_pattern.TestVolumeBootPattern.test_volume_boot_pattern fails with TimeoutException on SSH connection" [Undecided,In progress] https://launchpad.net/bugs/1298472 | 12:44 |
*** adalbas has joined #openstack-infra | 13:05 | |
*** sarob has joined #openstack-infra | 13:15 | |
*** sarob has quit IRC | 13:20 | |
*** plars is now known as plars-away | 13:43 | |
*** zul has quit IRC | 13:43 | |
*** zul has joined #openstack-infra | 13:47 | |
*** roeyc has joined #openstack-infra | 13:54 | |
*** changbl has joined #openstack-infra | 14:01 | |
*** adrian_otto has joined #openstack-infra | 14:04 | |
*** adrian_otto has quit IRC | 14:09 | |
*** adrian_otto has joined #openstack-infra | 14:09 | |
*** adrian_otto has quit IRC | 14:14 | |
*** sarob has joined #openstack-infra | 14:15 | |
*** sarob has quit IRC | 14:19 | |
*** HenryG has joined #openstack-infra | 14:26 | |
*** HenryG has quit IRC | 14:27 | |
*** jhesketh has quit IRC | 14:39 | |
*** ramashri has joined #openstack-infra | 14:43 | |
openstackgerrit | A change was merged to openstack-infra/elastic-recheck: Make query for ssh timeout bug 1298472 more general https://review.openstack.org/98631 | 14:46 |
uvirtbot | Launchpad bug 1298472 in tempest "SSHTimeout in tempest scenario tests using nova-network" [Undecided,In progress] https://launchpad.net/bugs/1298472 | 14:46 |
*** lbragstad has quit IRC | 14:48 | |
*** timrc-afk is now known as timrc | 14:48 | |
*** roeyc has quit IRC | 14:50 | |
*** jhesketh has joined #openstack-infra | 14:51 | |
*** homeless has joined #openstack-infra | 15:00 | |
*** gokrokve has quit IRC | 15:04 | |
*** gokrokve has joined #openstack-infra | 15:04 | |
*** jhesketh has quit IRC | 15:05 | |
*** sarob has joined #openstack-infra | 15:15 | |
*** wenlock has joined #openstack-infra | 15:15 | |
*** portante has quit IRC | 15:16 | |
*** gema has quit IRC | 15:16 | |
*** flashgordon has quit IRC | 15:16 | |
*** flashgordon has joined #openstack-infra | 15:16 | |
*** gema has joined #openstack-infra | 15:16 | |
*** portante has joined #openstack-infra | 15:16 | |
*** jhesketh has joined #openstack-infra | 15:18 | |
*** talluri has joined #openstack-infra | 15:20 | |
*** sarob has quit IRC | 15:20 | |
SpamapS | fungi: true the wildcard cert does do that, and most if not all API-driving libraries will support wildcards now. | 15:20 |
*** flaper87|afk has quit IRC | 15:23 | |
*** flaper87|afk has joined #openstack-infra | 15:23 | |
fungi | SpamapS: subject alt names in a cert are also a possibility there for supporting multiple domains off one cert | 15:24 |
fungi | though again, client-side support has been spotty in the past (but may be mostly a non-issue with modern libraries) | 15:25 |
SpamapS | fungi: Last I checked (2010 time frame) that RFC was contentious. | 15:34 |
SpamapS | perhaps it was put to bed | 15:34 |
*** HenryG has joined #openstack-infra | 15:37 | |
*** portante has quit IRC | 15:42 | |
*** portante has joined #openstack-infra | 15:42 | |
*** HenryG has quit IRC | 15:42 | |
*** HenryG has joined #openstack-infra | 15:42 | |
*** yfried has quit IRC | 15:42 | |
*** ramashri has quit IRC | 15:44 | |
mordred | SergeyLukjanov: you aroudn? | 15:51 |
*** homeless_ has joined #openstack-infra | 15:52 | |
* mordred still contends that even if you need to support one-ip-per-cert that the ports don't really solve anything | 15:54 | |
*** homeless has quit IRC | 15:54 | |
mordred | because you can _totally_ choose to install an apache or nginx with a rewrite/proxypass in front of a bunch of services running on non-public ports to map services to path locations | 15:55 |
mordred | so if you're short on ipv4, there is a way to deal with it that doesn't involve registering iana ports | 15:55 |
*** wenlock has quit IRC | 15:59 | |
*** wenlock has joined #openstack-infra | 16:06 | |
*** homeless_ has quit IRC | 16:11 | |
*** adalbas has quit IRC | 16:12 | |
*** gokrokve has quit IRC | 16:15 | |
*** sarob has joined #openstack-infra | 16:15 | |
*** gokrokve has joined #openstack-infra | 16:15 | |
*** aconrad_ has quit IRC | 16:16 | |
*** sarob has quit IRC | 16:19 | |
*** gokrokve has quit IRC | 16:20 | |
*** timrc is now known as timrc-afk | 16:20 | |
*** andreykurilin_ has joined #openstack-infra | 16:22 | |
*** penguinRaider has quit IRC | 16:23 | |
*** homeless has joined #openstack-infra | 16:24 | |
*** talluri has quit IRC | 16:27 | |
openstackgerrit | Alex Gaynor proposed a change to openstack-infra/config: Added a puppet module for livegrep. https://review.openstack.org/96877 | 16:35 |
*** yamahata has quit IRC | 16:51 | |
*** harlowja_at_home has joined #openstack-infra | 16:52 | |
*** adrian_otto has joined #openstack-infra | 17:04 | |
anteaya | saw a cow and a calf moose and stared with a wolf this weekend, missed seeing the bear | 17:06 |
anteaya | so all in all, pretty good | 17:06 |
*** rcarrillocruz has joined #openstack-infra | 17:06 | |
*** gokrokve has joined #openstack-infra | 17:06 | |
*** talluri has joined #openstack-infra | 17:07 | |
mordred | anteaya: excellent | 17:09 |
anteaya | mordred: any wildlife in your weekend, so far? | 17:10 |
anteaya | or just you? | 17:10 |
*** gokrokve_ has joined #openstack-infra | 17:11 | |
*** gokrokve has quit IRC | 17:11 | |
*** tkelsey has joined #openstack-infra | 17:13 | |
*** sarob has joined #openstack-infra | 17:15 | |
*** godp1301 has joined #openstack-infra | 17:16 | |
openstackgerrit | Christian Berendt proposed a change to openstack-infra/bugdaystats: Prettified all HTML files https://review.openstack.org/98653 | 17:16 |
trinaths | anteaya: you went to a zoo today.. ? | 17:16 |
anteaya | no | 17:17 |
anteaya | I walked around outside | 17:17 |
anteaya | there are moose and wolves and bear where I live | 17:18 |
openstackgerrit | Christian Berendt proposed a change to openstack-infra/publications: Prettified all HTML files https://review.openstack.org/98654 | 17:18 |
trinaths | anteaya: oh!. okay.. living very near to wild life. | 17:18 |
anteaya | yeah | 17:19 |
anteaya | and trees and water | 17:19 |
*** sarob has quit IRC | 17:19 | |
*** gokrokve_ has quit IRC | 17:20 | |
trinaths | anteaya: okay. surrounded by green | 17:20 |
anteaya | yeah | 17:20 |
*** gokrokve has joined #openstack-infra | 17:20 | |
trinaths | daily you go for a walk.. hearing music.. of birds and air.. | 17:21 |
*** tkelsey has quit IRC | 17:21 | |
openstackgerrit | Christian Berendt proposed a change to openstack-infra/reviewday: Prettified all HTML files https://review.openstack.org/98655 | 17:21 |
*** harlowja_at_home has quit IRC | 17:22 | |
openstackgerrit | Monty Taylor proposed a change to openstack-infra/config: Run stackalytics in infra https://review.openstack.org/98656 | 17:22 |
mordred | SergeyLukjanov: ^^ there is one thing about that that I'm unsure of - namely when and how stackalytics-processor should be run - but I think that's good enough to start | 17:23 |
mordred | SergeyLukjanov: I'm going to file a few bugs/feature requests after having dived in a little bit - what's the best way to chat with Ilya - is he aroudn on IRC? | 17:23 |
*** jpeeler has quit IRC | 17:23 | |
*** Shrews has quit IRC | 17:23 | |
*** arunkant has quit IRC | 17:23 | |
*** cyli has quit IRC | 17:23 | |
*** mdenny has quit IRC | 17:23 | |
*** isviridov|away has quit IRC | 17:23 | |
anteaya | yep | 17:23 |
*** cyli has joined #openstack-infra | 17:24 | |
mordred | awesome | 17:24 |
*** isviridov has joined #openstack-infra | 17:24 | |
*** mdenny has joined #openstack-infra | 17:24 | |
*** jpeeler has joined #openstack-infra | 17:24 | |
*** jpeeler has joined #openstack-infra | 17:24 | |
*** arunkant has joined #openstack-infra | 17:24 | |
anteaya | are we bring stackalytics under infra? | 17:24 |
mordred | oh. yeah. there he is | 17:24 |
mordred | anteaya: yeah - we chatted about it briefly at the summit | 17:24 |
anteaya | I ether missed that or I missed that | 17:24 |
anteaya | but yay | 17:24 |
mordred | well, it was more informal ish - there may have been beer | 17:25 |
anteaya | ah | 17:25 |
anteaya | I don't think I saw you at night anywhere | 17:25 |
*** Shrews has joined #openstack-infra | 17:25 | |
mordred | but in general, it's become a thing that people look at a lot, so it seemed that we should bring it in | 17:25 |
*** gokrokve has quit IRC | 17:25 | |
mordred | anteaya: I agree with that | 17:25 |
mordred | anteaya: I feel like I saw you for like 15 minutes | 17:25 |
anteaya | I saw you for about 2 minutes during daytime, I remember that | 17:25 |
anteaya | by the waterfall | 17:25 |
anteaya | ah okay | 17:25 |
anteaya | time moves more slowly for me apparently | 17:26 |
mordred | ilyashakhat: what sorts of ways would you prefer for me to make feature requests? | 17:26 |
fungi | i think it also bears mentioning in a reply to http://lists.openstack.org/pipermail/openstack-infra/2014-June/001292.html | 17:26 |
mordred | fungi: yah. funny story - I was in the middle of writing a response to that | 17:26 |
*** godp1301 has quit IRC | 17:26 | |
mordred | fungi: I was going to point them to that patch as an example of a patch they could cargo cult | 17:27 |
pleia2 | good, once we bring that in I'd like to add the reviewstats project to that server too | 17:27 |
fungi | perfect | 17:27 |
* fungi fades back into the weekend | 17:27 | |
* pleia2 weekend too | 17:27 | |
*** wenlock has quit IRC | 17:27 | |
mordred | yah. there's a thing the activity-board email mentions that may be worth thinking about | 17:28 |
fungi | i have an entire server room here i need to disassemble and junk. still need to move some of it into a virtual server provider first. should get back to it :/ | 17:28 |
mordred | which is the idea of having _one_ thing do the data collection, and then have various thigns be able to grab that data | 17:28 |
mordred | so if we have reviewstats, and stackalytics and activity-board, then perhaps at some point in the future a single "I know how to collect data" thing can emerge | 17:29 |
fungi | yes, having a raw data earehouse for the various stats-generating tools to mine would be doubleplusgood | 17:29 |
fungi | s/earhouse/warehouse/ | 17:29 |
mordred | right now, I'd actually imagine that's likely to be stackalytics since it does git, gerrit and launchpad and has an API | 17:29 |
fungi | not sure what an earhouse is | 17:29 |
mordred | but I could be wrote | 17:29 |
mordred | fungi: if you don't know what an earhouse is, then I can't explain it | 17:30 |
fungi | in fact, i cannot explain it, no | 17:30 |
fungi | an earhouse is where you keep your ear wigs | 17:30 |
*** trinaths has quit IRC | 17:31 | |
*** pcrews has joined #openstack-infra | 17:32 | |
*** ildikov_ has joined #openstack-infra | 17:34 | |
openstackgerrit | Monty Taylor proposed a change to openstack-infra/config: Run stackalytics in infra https://review.openstack.org/98656 | 17:35 |
*** Alec-A2 has joined #openstack-infra | 17:37 | |
*** ildikov has quit IRC | 17:38 | |
anteaya | I don't want an earhouse | 17:39 |
mordred | fungi: btw - I've spun up a server, added a key to hiera and am running the puppet above ^^ on it by hand out of the mordred environment | 17:40 |
*** yfried has joined #openstack-infra | 17:40 | |
*** yfried has joined #openstack-infra | 17:40 | |
openstackgerrit | Monty Taylor proposed a change to openstack-infra/config: Run stackalytics in infra https://review.openstack.org/98656 | 17:41 |
*** gema has quit IRC | 17:44 | |
*** gema has joined #openstack-infra | 17:46 | |
*** gokrokve has joined #openstack-infra | 17:54 | |
openstackgerrit | Monty Taylor proposed a change to openstack-infra/config: Run stackalytics in infra https://review.openstack.org/98656 | 17:59 |
*** ildikov_ has quit IRC | 18:01 | |
*** ildikov_ has joined #openstack-infra | 18:02 | |
mordred | ilyashakhat: _retrieve_project_list_from_github should grab list-projects from gerrit and not github :) | 18:02 |
YorikSar | fungi: On stackalytics-processor: I've heard it's run once in 4 hours by cron now. | 18:03 |
mordred | YorikSar: cool. I was sort of imagining something like that might be the case - it's not in the "HowToRun" doc though | 18:03 |
YorikSar | fungi, mordred: and bout bugs and features. It's on stackforge - I guess noone would object using https://bugs.launchpad.net/stackalytics | 18:04 |
mordred | YorikSar: awesome! I will file bugs | 18:05 |
*** andreykurilin_ has quit IRC | 18:05 | |
* YorikSar hopes ilyashakhat won't bite in the Monday morning | 18:06 | |
YorikSar | mordred: Oh... You're going to run it in Apache... | 18:08 |
mestery_ | fungi: FYI, I don't think the failure in the VM you saved is the same one. armax's code should have kept the guest around, and it didn't. | 18:08 |
mestery_ | fungi: So you can recycle that one please. If it triggers again, we can have anther look. I'll see about recreating it locally tomorrow. | 18:08 |
mestery_ | fungi: Have to head out to a family gathering for hte rest of the day. Thanks and enjoy your Sunday! | 18:09 |
*** mestery_ is now known as mestery | 18:09 | |
YorikSar | mordred: I know that stackalytics web process is very memory-hungry. It holds lots of data and indexes in memory. I'm not sure as of how Apache would react o that. | 18:09 |
mordred | YorikSar: well, we'll have to poke around at it - if we need to do an out-of-process and have apache do proxypass that's possible too | 18:10 |
mordred | YorikSar: we just try to keep things as consistent across the board and don't have nginx anywhere | 18:10 |
YorikSar | mordred: Currently they're using uwsgi there. | 18:11 |
YorikSar | mordred: I guess there's no dependency on Nginx - Apache can proxy to uwsgi as well. | 18:12 |
*** sarob has joined #openstack-infra | 18:15 | |
mordred | YorikSar: from apt-cache search it seems you have to install nginx to get uwsgi installed | 18:18 |
YorikSar | mordred: Weird Ubuntu... | 18:18 |
mordred | YorikSar: mordred@camelot:~/src/stackforge/stackalytics$ apt-cache search uwsgi | 18:19 |
mordred | nginx-extras - nginx web/proxy server (extended version) | 18:19 |
mordred | nginx-full - nginx web/proxy server (standard version) | 18:19 |
mordred | :( | 18:19 |
*** sarob has quit IRC | 18:20 | |
mordred | YorikSar: oh - pip install uwsgi | 18:20 |
mordred | YorikSar: k. nevermind. I can work with uwsgi | 18:20 |
mordred | I'll try that next | 18:20 |
YorikSar | mordred: On Gentoo uwsgi may depend on Apache :D | 18:21 |
YorikSar | mordred: I guess it's up to package maintainer's preference :) | 18:22 |
mordred | lovely | 18:23 |
mordred | YorikSar: stackalytics doesn't seem to be reading /etc/stackalytics/stackalytics.conf by default | 18:24 |
mordred | I had to run stackalytics-processor with --config-dir=/etc/stackalytics | 18:24 |
*** yfried has quit IRC | 18:24 | |
mordred | (I'm just complaining now) | 18:24 |
*** yfried has joined #openstack-infra | 18:24 | |
YorikSar | mordred: That's strange. They use oslo.config... | 18:26 |
mordred | yeah. I'm wondering if it's not getting initialized properly through a code path ... | 18:27 |
*** Longgeek has quit IRC | 18:30 | |
*** homeless has quit IRC | 18:30 | |
YorikSar | mordred: Aha. project name is not passed to CONF() call. | 18:31 |
mordred | well, that's poopy | 18:32 |
* mordred makes patch | 18:32 | |
*** pcrews has quit IRC | 18:32 | |
* mordred is currently making a patch to remove github urls | 18:32 | |
pleia2 | \o/ | 18:32 |
*** pcrews has joined #openstack-infra | 18:34 | |
* YorikSar looks how mordred is going to become active contributor to stackalytics | 18:34 | |
mordred | YorikSar: I don't even see the CONF() call | 18:35 |
mordred | ah. I see it now | 18:36 |
YorikSar | https://github.com/stackforge/stackalytics/blob/master/stackalytics/processor/main.py#L266 | 18:36 |
mordred | yah. got that | 18:36 |
YorikSar | mordred: I'm too slow this evening | 18:37 |
mordred | now, in the web app, it looks like it's expecting me to pass in an env var | 18:37 |
YorikSar | mordred: Yeah... | 18:38 |
mordred | YorikSar: https://review.openstack.org/98659 | 18:40 |
YorikSar | mordred: You shoulld pass project='stackalytics' to the call or default locations won't work | 18:41 |
YorikSar | mordred: Oh, wait... Is project the first argument? | 18:41 |
mordred | YorikSar: think so - that's how we do it in storyboard | 18:41 |
mordred | YorikSar: also, is this: logging.setup('stackalytics.dashboard') | 18:42 |
YorikSar | https://github.com/openstack/oslo.config/blob/master/oslo/config/cfg.py#L1596 | 18:42 |
mordred | YorikSar: valid usage? it is, right? | 18:42 |
mordred | oh. piddle | 18:42 |
YorikSar | mordred: It looks it's the second. Anyway, cryptic positional args should be avoided. | 18:42 |
*** praneshp has joined #openstack-infra | 18:43 | |
mordred | YorikSar: yah. just pushed an update | 18:44 |
YorikSar | mordred: Oh, here it is: https://pypi.python.org/pypi/kwonly | 18:45 |
YorikSar | mordred: I guess, some libs should enforce this more often. | 18:46 |
YorikSar | mordred: Although I'm barely involved in stackalytics here's a +1 from me :) | 18:51 |
mordred | :) | 18:51 |
mordred | lifeless: awake? | 18:53 |
mordred | lifeless: run.py: error: unrecognized arguments: discover -t ./ ./tests --list | 18:57 |
* anteaya readies to move locations | 19:02 | |
*** yfried has quit IRC | 19:06 | |
*** yfried has joined #openstack-infra | 19:06 | |
*** gokrokve has quit IRC | 19:07 | |
*** gokrokve has joined #openstack-infra | 19:07 | |
openstackgerrit | Monty Taylor proposed a change to openstack-infra/config: Run stackalytics in infra https://review.openstack.org/98656 | 19:08 |
Alex_Gaynor | mordred: ping | 19:10 |
*** gokrokve has quit IRC | 19:11 | |
mordred | Alex_Gaynor: heyho | 19:12 |
mordred | YorikSar: ok. I fixed a thing and also uploaded another patch to move dashboard into stackalytics :) | 19:15 |
*** sarob has joined #openstack-infra | 19:15 | |
Alex_Gaynor | mordred: So, my understanding is that the desire for config stuff is that the modules are standalone, and then the manifest is where the openstack specific stuff. Right now for livegrep it's just sort of all in one place, what's the right way to split that stuff up? | 19:16 |
*** talluri has quit IRC | 19:16 | |
mordred | Alex_Gaynor: so - there should be a module that's standalone - and then there shoudl be something in openstack_project that includes that module and also insludes openstack_project::server (which puts our base things on it) | 19:17 |
YorikSar | mordred: Nice. I've been looking for the web-related code there for a long time when I first took a look at it. | 19:17 |
*** wenlock has joined #openstack-infra | 19:18 | |
mordred | Alex_Gaynor: and then manifests/site.pp should instantiate the thing in openstack_project | 19:18 |
mordred | Alex_Gaynor: let me look at your patch real quick | 19:18 |
Alex_Gaynor | thanks | 19:19 |
mordred | Alex_Gaynor: that looks fine to me - I may want to go in and make a follow up patch to have it consume projects.yaml instead of keeping its own list - but this is probably great for now | 19:19 |
*** sarob has quit IRC | 19:20 | |
Alex_Gaynor | mordred: ok. We'd also discussed having it either be a mirror for cgit, or run on the cgit machine itself. Should that be a blocker for getting this landed? | 19:20 |
mordred | I dont' think so | 19:20 |
mordred | I think we'll learn something about it from landing this | 19:20 |
*** cody-somerville has quit IRC | 19:20 | |
Alex_Gaynor | Ok, let me remove the notes from teh commit message and put these into a comment in the code | 19:20 |
mordred | Alex_Gaynor: how does the indexer know to re-index things? | 19:21 |
Alex_Gaynor | mordred: the codesearch service (the indexer) is subscribed to the repos its indexing | 19:21 |
*** pcrews has quit IRC | 19:22 | |
mordred | Alex_Gaynor: ah. I see it | 19:22 |
openstackgerrit | Alex Gaynor proposed a change to openstack-infra/config: Added a puppet module for livegrep. https://review.openstack.org/96877 | 19:23 |
mordred | Alex_Gaynor: I'm going to eat some brunch - but I'll review that a little bit deeper in just a bit | 19:24 |
Alex_Gaynor | mordred: thanks! I'll be around all afternoon | 19:24 |
mordred | Alex_Gaynor: I, for one, look forward to our new livegrep overlords | 19:24 |
Alex_Gaynor | :-) | 19:24 |
*** wenlock has quit IRC | 19:29 | |
*** rcarrillocruz has quit IRC | 19:46 | |
*** rcarrillocruz has joined #openstack-infra | 19:50 | |
mordred | YorikSar: http://stackalytics.openstack.org/ | 20:00 |
mordred | YorikSar: at least works to some degree | 20:00 |
mordred | fungi: ^^ that's using the puppet module | 20:00 |
openstackgerrit | Monty Taylor proposed a change to openstack-infra/config: Run stackalytics in infra https://review.openstack.org/98656 | 20:01 |
mordred | with those changes | 20:01 |
mordred | outstanding questions: a) uwsgi b) cron job for processor | 20:01 |
fungi | mordred: darned awesome! | 20:01 |
YorikSar | Hm... It doesn't data in dropdowns... | 20:01 |
lifeless | mordred: hi | 20:02 |
mordred | YorikSar: it's still processing data | 20:02 |
mordred | lifeless: nm. I foudn the error | 20:02 |
mordred | lifeless: I was having trouble diagnosing a testr issue | 20:02 |
lifeless | mordred: thats being thrown by a unit test that is consulting the real sys.argv | 20:02 |
YorikSar | mordred: Ah... Ok. | 20:02 |
mordred | fungi: I also submitted like 10 patches to stackalytics that I think we should probably get landed before rolling that live | 20:02 |
mordred | YorikSar: I'm also seeing things like this: | 20:02 |
mordred | 2014-06-08 20:02:50.996 10166 WARNING stackalytics.processor.utils [-] Error while reading uri: HTTP Error 404: Not Found | 20:02 |
mordred | YorikSar: which I just submitted a patch to add the url to the message :) | 20:03 |
fungi | mordred: not at all unexpected | 20:03 |
mordred | fungi: mostly little things - the normal result of running one in the second location | 20:03 |
mordred | fungi: so things that were assumptions about their install that are not the same for ours and not properly picking up config | 20:04 |
mordred | fungi: but I've got to say in all fairness, it took me relatively low amount of effort to get that up and running and fully puppetted | 20:04 |
openstackgerrit | Monty Taylor proposed a change to openstack-infra/config: Run stackalytics in infra https://review.openstack.org/98656 | 20:08 |
mordred | YorikSar, fungi: ^^ ok. there it is running processor in cron instead | 20:08 |
*** sarob has joined #openstack-infra | 20:15 | |
*** sarob has quit IRC | 20:19 | |
*** michchap has quit IRC | 20:20 | |
*** _nadya_ has joined #openstack-infra | 20:20 | |
*** _nadya_ has joined #openstack-infra | 20:28 | |
*** _nadya_ has quit IRC | 20:30 | |
openstackgerrit | Monty Taylor proposed a change to openstack-infra/config: Split the database grants into their own class https://review.openstack.org/98586 | 20:35 |
openstackgerrit | Monty Taylor proposed a change to openstack-infra/config: Add support for disk-image-builder in nodepool https://review.openstack.org/88479 | 20:35 |
*** praneshp has quit IRC | 20:37 | |
*** ayoung has quit IRC | 20:45 | |
*** salv-orlando has joined #openstack-infra | 20:57 | |
*** zz_gondoi is now known as gondoi | 20:57 | |
*** adrian_otto has quit IRC | 21:00 | |
*** ayoung has joined #openstack-infra | 21:00 | |
*** chuckC has quit IRC | 21:01 | |
fungi | pleia2: this just in... DebConf14 - Call for talks, BoFs and events https://lists.debian.org/debian-devel-announce/2014/06/msg00000.html | 21:13 |
*** sarob has joined #openstack-infra | 21:15 | |
*** salv-orlando has quit IRC | 21:15 | |
lifeless | hmm | 21:19 |
*** sarob has quit IRC | 21:19 | |
*** salv-orlando has joined #openstack-infra | 21:20 | |
*** gondoi is now known as zz_gondoi | 21:22 | |
*** otter768 has joined #openstack-infra | 21:24 | |
*** mgagne has quit IRC | 21:24 | |
fungi | pleia2: given your posited interest in presenting on possible debian/openstack infra cross-pollination, it might be worth bouncing ideas off Mithrandir (since he's on the dsa team) | 21:30 |
*** otter768 has quit IRC | 21:37 | |
*** otter768 has joined #openstack-infra | 21:38 | |
*** mgagne has joined #openstack-infra | 21:40 | |
*** mgagne is now known as Guest89234 | 21:40 | |
*** pcrews has joined #openstack-infra | 21:40 | |
*** Guest89234 is now known as mgagne1 | 21:41 | |
*** mgagne1 has quit IRC | 21:42 | |
*** mgagne1 has joined #openstack-infra | 21:42 | |
mgagne1 | XenServer CI keeps failing on my change, who or which channel should I ask to help me debug? https://review.openstack.org/#/c/91727/ | 21:43 |
mgagne1 | All I managed to find is "'amqplib' is not in global-requirements.txt", I'm not sure if it's an error or not. And '/usr/sbin/apache2ctl: command not found' but fungi looks to say it's not a big deal in bug irc://grim.calavera.ca:6697/#1326209 | 21:43 |
mgagne1 | bug #1326209 * | 21:44 |
uvirtbot | Launchpad bug 1326209 in devstack "/usr/sbin/apache2ctl: command not found" [Undecided,New] https://launchpad.net/bugs/1326209 | 21:44 |
fungi | mgagne1: "XenServer CI" is an unofficial, third-party test system. BobBall may be able to look into it and provide suggestions (i think he may have some insight into it) | 21:48 |
fungi | mgagne1: er, not BobBall (he's citrix) | 21:49 |
fungi | mgagne1: oh, maybe so. the contact address for that account is at citrix.com | 21:50 |
* fungi can't keep all the third-party systems straight any longer | 21:51 | |
mgagne1 | fungi: right, as a change owner, where can I find information about 3rd party contact? Emails are ok but irc nicknames would be great so I can bug them | 21:51 |
*** salv-orlando has quit IRC | 21:51 | |
mgagne1 | fungi: and the UX of a change owner about how to debug such results isn't great. No README or explanation on how to analyse the result. :-/ | 21:52 |
fungi | mgagne1: there is work in progress to get wiki pages set up for all of them and actively maintained (anteaya is spear-heading that effort). in the meantime, hovering your mouse cursor over the name of the account should give you a valid contact e-mail address | 21:52 |
fungi | mgagne1: and yeah, they need to start including debugging instructions and links to additional details in their respective failure messages | 21:53 |
*** pcrews has quit IRC | 21:55 | |
*** salv-orlando has joined #openstack-infra | 21:57 | |
*** alkari has joined #openstack-infra | 22:02 | |
*** alkari1 has joined #openstack-infra | 22:06 | |
alkari1 | hello | 22:07 |
mordred | fungi: actually - you know - instaed of getting debugging instrauctions in the failure message | 22:10 |
*** alkari has quit IRC | 22:10 | |
mordred | fungi: if we expand on that "hide all the CI comments" UI patch - we could potentially store a single copy of debug instructions per-test system | 22:11 |
mordred | fungi: and have the UI thing provide a way to request the info in a discoverable way | 22:11 |
openstackgerrit | Monty Taylor proposed a change to openstack-infra/publications: Use reveal.js instead of slidy https://review.openstack.org/98670 | 22:14 |
mgagne1 | mordred: are the tests ran across all 3rd party the same with the same captured logs? | 22:14 |
*** salv-orlando has quit IRC | 22:15 | |
mordred | mgagne1: nope | 22:15 |
mordred | mgagne1: the tests for a given 3rd party are completely up to the 3rd party | 22:15 |
mgagne1 | mordred: How can debug instructions be the same for all? | 22:15 |
*** sarob has joined #openstack-infra | 22:15 | |
mgagne1 | mordred: oh, "per test system" right | 22:15 |
mordred | yah | 22:15 |
mgagne1 | mordred: what if the test system is updated, would having a single copy of the instruction be confusing for someone debugging results from a previous version? | 22:17 |
*** otter768 has quit IRC | 22:18 | |
mordred | mgagne1: possibly ... I suppose maybe just having the instructions in the results are maybe not terrible | 22:18 |
*** sarob has quit IRC | 22:20 | |
*** harlowja_at_home has joined #openstack-infra | 22:21 | |
*** lcheng has joined #openstack-infra | 22:23 | |
*** aconrad_ has joined #openstack-infra | 22:23 | |
alkari1 | folks, im looking for a way to consolidate openstack logs.. any advice here? | 22:24 |
*** gokrokve has joined #openstack-infra | 22:25 | |
*** harlowja_at_home has quit IRC | 22:32 | |
*** Ryan_Lane has joined #openstack-infra | 22:34 | |
*** salv-orlando has joined #openstack-infra | 22:34 | |
*** gokrokve has quit IRC | 22:36 | |
*** praneshp has joined #openstack-infra | 22:36 | |
*** gokrokve has joined #openstack-infra | 22:37 | |
*** adrian_otto has joined #openstack-infra | 22:38 | |
*** andreykurilin_ has joined #openstack-infra | 22:38 | |
*** lcheng has quit IRC | 22:40 | |
*** gokrokve has quit IRC | 22:41 | |
*** salv-orlando has quit IRC | 22:42 | |
*** praneshp_ has joined #openstack-infra | 22:44 | |
mordred | alkari1: hi! so, we're mainly the ones who run the openstack developer infrastructure - but we are fans of logstash/elasticsearch | 22:44 |
*** praneshp has quit IRC | 22:46 | |
*** praneshp_ is now known as praneshp | 22:46 | |
*** salv-orlando has joined #openstack-infra | 22:46 | |
*** andreykurilin_ has quit IRC | 22:56 | |
*** pcrews has joined #openstack-infra | 22:57 | |
*** salv-orlando has quit IRC | 23:04 | |
*** zzelle has quit IRC | 23:10 | |
*** sarob has joined #openstack-infra | 23:15 | |
anteaya | XenServer CI is citrix and BobBall is the contact | 23:15 |
anteaya | mgagne1: | 23:16 |
* anteaya is looking forward to the renaming | 23:17 | |
anteaya | 24 in the gate an only two left at over 40 hours | 23:18 |
anteaya | yay for the weekend | 23:18 |
*** sarob has quit IRC | 23:19 | |
Alex_Gaynor | anteaya: pacifica is starting to wake up and get to work I think, hopefully they don't destroy our shot at emptying the queue | 23:25 |
anteaya | pacifica is a gentle sort | 23:25 |
anteaya | hopefully they will be kind | 23:25 |
openstackgerrit | Monty Taylor proposed a change to openstack-infra/publications: Use reveal.js instead of slidy https://review.openstack.org/98670 | 23:28 |
mgagne1 | anteaya: I sent him an email =) | 23:30 |
anteaya | mgagne1: cool | 23:30 |
*** _nadya_ has joined #openstack-infra | 23:30 | |
mgagne1 | anteaya: yha, both of my changes failed for unclear reason or transient bugs :-/ | 23:31 |
anteaya | mgagne1: rather opaque | 23:32 |
anteaya | BobBall is rather responsive and he also is a contributor | 23:32 |
anteaya | so he should understand what you are talking about, which is a huge bonus | 23:32 |
ianw | anteaya: public holiday in Australia today, so that might slow things down | 23:32 |
mgagne1 | anteaya: I'll wait for his answer on Monday. | 23:32 |
anteaya | ianw: ack | 23:32 |
anteaya | and good, more weekend to clear the queue | 23:33 |
anteaya | mgagne1: very good | 23:33 |
mgagne1 | anteaya: looking forward for those improvements in the 3rd party area =) | 23:33 |
anteaya | me too | 23:34 |
anteaya | it is slow, like pushing mud with a rope, but there is an agreed upon direction | 23:34 |
*** _nadya_ has quit IRC | 23:34 | |
anteaya | at least agreed upon within infra, which is a start | 23:35 |
mordred | anteaya: wow. pushing mud with a rope seems like about the slowest thing possible | 23:35 |
anteaya | oh no | 23:36 |
anteaya | much slower movement | 23:36 |
anteaya | glaciers are slower | 23:36 |
anteaya | so are the crystalization of gems inside of cooling rock | 23:36 |
anteaya | the thing with using rope to push mud is the mud keeps being not in front of the rope | 23:37 |
anteaya | so you have to pull back and try again | 23:37 |
mordred | nod | 23:38 |
anteaya | that's my best analogy atm | 23:40 |
*** oomichi has quit IRC | 23:42 | |
*** yamahata has joined #openstack-infra | 23:42 | |
*** markmcclain has joined #openstack-infra | 23:44 | |
*** oomichi has joined #openstack-infra | 23:44 | |
*** markmcclain1 has joined #openstack-infra | 23:45 | |
*** markmcclain has quit IRC | 23:48 | |
anteaya | mordred: I am going of the tc meeting notes to do the thing I have an action item for | 23:48 |
anteaya | mordred: you too have an action item: ACTION: mordred still to prepare draft binary proposal for the "TC direction" column scores | 23:49 |
anteaya | s/going of/going over | 23:50 |
mordred | anteaya: thanks | 23:50 |
anteaya | np | 23:52 |
anteaya | fungi: you had responded to me thusly during the tc meeting: <fungi> anteaya: i can imagine a two-dimensional plot of voter turnout vs. commits | 23:52 |
anteaya | is it realistic to expect that any such plot might see the light of day this week? | 23:53 |
*** nati_ueno has joined #openstack-infra | 23:53 | |
anteaya | or how is your expected workload shaping up for the next few days | 23:53 |
*** godp1301 has joined #openstack-infra | 23:55 | |
Alex_Gaynor | Pretty sure the requirements gate isn't going to pass at all right now because of: http://bpaste.net/show/TGgG5c6kY9qTjzKiqqwy/ | 23:58 |
anteaya | Alex_Gaynor: is that an error in six? | 23:59 |
Alex_Gaynor | anteaya: it looks like it only happens during the installation process, I can't reproduce it except inside of pip. (six did a new release today, that's definitely the cause) | 23:59 |
anteaya | oh no | 23:59 |
anteaya | no no no | 23:59 |
*** godp1301 has quit IRC | 23:59 | |
mordred | Alex_Gaynor: sigh | 23:59 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!