Thursday, 2015-07-30

*** tlian has quit IRC00:07
*** darrenc_afk is now known as darrenc00:14
*** annashen has joined #openstack-ansible00:29
*** annashen has quit IRC00:34
*** daneyon has quit IRC00:48
*** daneyon has joined #openstack-ansible00:48
*** daneyon has quit IRC00:49
*** daneyon has joined #openstack-ansible00:49
*** daneyon has quit IRC00:51
*** alop has quit IRC00:51
*** daneyon has joined #openstack-ansible00:52
*** daneyon has quit IRC00:56
*** abitha has quit IRC01:24
*** annashen has joined #openstack-ansible01:30
*** metral has quit IRC01:33
openstackgerritChristopher H. Laco proposed stackforge/os-ansible-deployment: Remove reference to missing user_group_vars file  https://review.openstack.org/20726401:33
*** annashen has quit IRC01:35
*** abitha has joined #openstack-ansible02:16
*** annashen_ has joined #openstack-ansible02:31
*** annashen_ has quit IRC02:36
*** sdake has joined #openstack-ansible03:05
*** markvoelker_ has quit IRC03:20
*** markvoelker_ has joined #openstack-ansible03:22
*** metral_zzz has joined #openstack-ansible03:27
*** metral_zzz is now known as metral03:27
*** annashen_ has joined #openstack-ansible03:32
*** sdake has quit IRC03:36
*** annashen_ has quit IRC03:37
*** JRobinson__ is now known as JRobinson__afk04:25
*** fawadkhaliq has joined #openstack-ansible04:27
*** jlvillal has quit IRC04:32
*** annashen has joined #openstack-ansible04:33
*** annashen has quit IRC04:38
*** misc has quit IRC04:50
*** misc has joined #openstack-ansible04:52
*** fawadkhaliq has quit IRC04:52
*** yapeng has joined #openstack-ansible04:54
*** darrenc is now known as darrenc_afk04:56
*** shausy has joined #openstack-ansible04:57
*** JRobinson__afk is now known as JRobinson__04:58
*** yapeng has quit IRC04:59
*** fawadkhaliq has joined #openstack-ansible05:05
*** yapeng has joined #openstack-ansible05:10
*** abitha has quit IRC05:11
*** annashen has joined #openstack-ansible05:34
*** shausy has quit IRC05:37
*** shausy has joined #openstack-ansible05:38
*** annashen has quit IRC05:39
*** darrenc_afk is now known as darrenc05:40
*** javeriak has joined #openstack-ansible05:55
*** javeriak has quit IRC05:58
*** javeriak has joined #openstack-ansible06:05
*** javeriak_ has joined #openstack-ansible06:07
*** javeriak has quit IRC06:07
*** shausy has quit IRC06:15
*** shausy has joined #openstack-ansible06:15
*** annashen_ has joined #openstack-ansible06:35
*** annashen_ has quit IRC06:39
evrardjphello everyone07:15
evrardjpshould I vote for some talks to bring more OSAD visibility for the next summit?07:16
*** gparaskevas has joined #openstack-ansible07:35
*** annashen has joined #openstack-ansible07:35
stevelleevrardjp: there are probably a half dozen or more related talks07:37
*** gparaskevas has quit IRC07:38
*** gparaskevas has joined #openstack-ansible07:39
evrardjpThat's why I'd vote for a pre-defined list ;)07:39
-openstackstatus- NOTICE: Our CI system is broken again today, jobs are not getting processed at all.07:39
evrardjpI've voted for what I've found07:39
*** ChanServ changes topic to "Our CI system is broken again today, jobs are not getting processed at all."07:39
*** annashen has quit IRC07:40
*** JRobinson__ has quit IRC07:47
-openstackstatus- NOTICE: CI system is broken and very far behind. Please do not approve any changes for a while.07:48
*** ChanServ changes topic to "CI system is broken and very far behind. Please do not approve any changes for a while."07:48
*** abitha has joined #openstack-ansible08:11
*** abitha has quit IRC08:16
*** javeriak_ has quit IRC08:21
*** javeriak has joined #openstack-ansible08:33
*** annashen has joined #openstack-ansible08:36
*** annashen has quit IRC08:41
*** fawadkhaliq has quit IRC08:53
*** ChanServ changes topic to "Weekly Meetings: https://wiki.openstack.org/wiki/Meetings/openstack-ansible"08:57
-openstackstatus- NOTICE: CI is back online but has a huge backlog. Please be patient and if possible delay approving changes until it has caught up.08:57
openstackgerritMerged stackforge/os-ansible-deployment: set correct swift dispersion tenant  https://review.openstack.org/20691309:24
*** fawadkhaliq has joined #openstack-ansible09:25
*** fawadkhaliq has quit IRC09:26
*** fawadkhaliq has joined #openstack-ansible09:26
*** ig0r_ has joined #openstack-ansible09:32
*** annashen has joined #openstack-ansible09:37
openstackgerritMerged stackforge/os-ansible-deployment: Set default inventory file for inventory-manage.py  https://review.openstack.org/20715809:38
*** annashen has quit IRC09:42
*** javeriak has quit IRC10:07
matttsvg: would it be possible to push up your changes from yesterday so we can do further testing?10:09
svgSure, was waiting for you to ask that :)10:10
matttsvg: thanks :)10:11
*** shausy has quit IRC10:12
openstackgerritSerge van Ginderachter proposed stackforge/os-ansible-deployment: Ceph/RBD support  https://review.openstack.org/18195710:12
*** shausy has joined #openstack-ansible10:13
svgmattt: ^^10:13
matttthanks svg!10:18
git-harrysvg: are you going to have any time today to address the last few outstanding issues?10:22
git-harryWe'd like to try and get this patch merged in tomorrow10:23
git-harryIf not do you mind if mattt and I upload some changes to address those things?10:23
svgLet me see, IIRC, besides the obvious #TODO's, there was your last comment about 'services are configured before ceph_client installs'. Anything I am mising?10:25
*** javeriak has joined #openstack-ansible10:26
svgabout that latter, I'm not sure which of your proposed solutions, is the best. Perhaps having the ceph role restart those services?10:26
git-harrysvg: I think that's everything.10:28
git-harrysvg: I was inclined to have the role install libvirt and run it before the service role because virsh is required by the role10:29
svgOf the TODO's in the ceph role, so far I left them there b/c I was waiting for feedback from leseb, but given he seems offline, I suspect he might be on holidays10:29
svggit-harry: which service-role are you referring to exactly, os-cinder os-nova os glance I suppose?10:31
*** shausy has quit IRC10:32
openstackgerritJesse Pretorius proposed stackforge/os-ansible-deployment: Keystone Federation Identity Provider Configuration  https://review.openstack.org/20738610:32
openstackgerritJesse Pretorius proposed stackforge/os-ansible-deployment: Keystone Federation Service Provider Configuration  https://review.openstack.org/19439510:37
openstackgerritJesse Pretorius proposed stackforge/os-ansible-deployment: Enable Horizon to consume a Keystone v3 API endpoint  https://review.openstack.org/20657510:37
*** annashen has joined #openstack-ansible10:38
*** shausy has joined #openstack-ansible10:38
svghmz, git-harry, doing an implementation of ceph before service thingz: this won't help if ceph support is installed later, in a subsequent run, as then the os services won't be restarted either10:42
*** annashen has quit IRC10:42
svgso doing a handler restart from the ceph role seems better. thoughts?10:42
git-harrysvg: the conf files for the openstack services would be updated to include the ceph stuff so the services should get restarted11:01
*** jlvillal has joined #openstack-ansible11:04
svgyes, that was already clear11:05
svgI'm looking at implementing handlers, but that would mean I need to reimplement some data, lik ethe list of *_service_names11:05
odyssey4memancdaz there's some discussion about the ceilometer bit in https://review.openstack.org/181560 which could warrant a minor update, or a follow-on patch. Thoughts?11:06
odyssey4meIf you could do a follow-on patch to improve that today, then we can +w that patch now.11:07
openstackgerritJesse Pretorius proposed stackforge/os-ansible-deployment: Add sample Keystone Federation SP configuration for ADFS  https://review.openstack.org/20373611:09
matttsvg git-harry : any thoughts on this approach to removing ceph_conf from defaults/main.yml and either a) providing a path to ceph.conf or b) obtaining from a running mon?11:12
matttsvg git-harry : https://gist.github.com/mattt416/f00a30f9a7d9b3c9df5c11:12
mattt(still need to rebase against latest patchset)11:12
openstackgerritJesse Pretorius proposed stackforge/os-ansible-deployment: Keystone SSL cert/key distribution and configuration  https://review.openstack.org/19447411:12
svgmattt is it b/c of not rebased yet you are removing that # look for 1 ceph monitor host that is up task?11:15
svgO, dit you move that to  include: ceph_get_mon_host.yml ? that one's not in the diff11:17
openstackgerritJesse Pretorius proposed stackforge/os-ansible-deployment: Remove hardcoded config drive enforcement  https://review.openstack.org/19540311:18
svgmattt perhaps better to do a local_action: command instead of  delegate_to: localhost   for  name: Get ceph.conf and store contents as a var11:21
odyssey4memancdaz there are some notes in the reviews for https://review.openstack.org/199572 which I think are pertinent... it is probably better to configure the logging rather than just remove the package11:21
git-harrydoes local_action give all hosts access to the var?11:22
svgsure, it's actyally more of a syntactic difference, though locat_action will do a connection: local, whilst delegate might still go over ssh iirc11:23
matttsvg: cool, i can make that change11:23
svgit's a connection thing, doesnt change scope of host11:23
mancdazodyssey4me well, after discussion (in the bug) it was decided we don't want the package there at all11:23
matttsvg: if i remove ceph_conf from defaults/main.yml will that break your workflow significantly?11:24
svgmattt so the local ceph file, where would that one live? personally, I'd rather make that a variable in inventory, than a loose file somewhere11:24
mancdazodyssey4me but I see the note from kevin around moving the task out somewhere else11:24
matttsvg: it will read it from anywhere on the deployment host i guess, so location doesn't matter11:25
mancdaznot sure I agree. If you want to install the package again in the future you just remove it from the uninstall list and add it to the install list11:25
odyssey4memancdaz I see that it's merged into juno already, and also see the discussion in the bug - I think perhaps a subsequent patch can do the suggested reconfiguration? your thoughts?11:25
svgwhich could be anywhere, so also outside of a regular dir with data for osad11:25
svgI'd rather enforce it to be in some user var file11:25
svgbut gotta go or i don;t have lunch! brb11:26
mancdazodyssey4me I'm not sure I understand the value of moving the task into a separate file11:26
matttsvg: go go !11:26
odyssey4memancdaz I think the idea is similar to pinning - it has broader use beyond just neutron11:26
mancdazodyssey4me well the argument being made is that moving it out makes it easier to remove in future?11:27
odyssey4memeh, I don't agree with that11:27
odyssey4meI think we should merge this as-is. It's come specifically from a deployer and has been agreed to by the SME's.11:28
odyssey4meThe only thing that has been disagreed to was the method of how we do it in the playbooks/roles.11:29
odyssey4meAndy's -1 is related to rather doing the config change instead of removing the package.11:29
mancdazodyssey4me I responded to both comments11:31
*** annashen has joined #openstack-ansible11:39
*** yapeng has quit IRC11:39
odyssey4memancdaz thanks - I've approved it and am registering a wish list bug for the generic removal method11:40
*** annashen has quit IRC11:43
mancdazodyssey4me https://bugs.launchpad.net/openstack-ansible/+bug/140088111:56
openstackLaunchpad bug 1400881 in openstack-ansible "Cannot rebuild a VM created from a Cinder volume backed by NetApp" [Medium,Triaged]11:56
odyssey4memancdaz sure, we could do that - but it seems silly... do we do this for every other upstream patch?11:57
odyssey4mewe will automatically consume the fixes as they merge, if they merge11:57
mancdazodyssey4me yes but being able to show the users that we have bumped our sha to consume that fix, is useful11:57
odyssey4memancdaz I really doubt those backports will end up being merged though11:58
mancdazso wontfix when they get rejected?11:59
odyssey4memancdaz fair enough, we'll just keep moving the milestone until there's a conclusion of some sort11:59
mancdazeither way, I think if you're wontfixing something, you need to remove the target milestone11:59
mancdazodyssey4me yeah I bumped juno to 10.1.1212:00
*** fawadk has joined #openstack-ansible12:02
*** fawadkhaliq has quit IRC12:03
odyssey4memancdaz the same has been done for kilo and they're marked as in-progress12:05
odyssey4meI'll keep tabs on them (they're assigned to me)12:05
mancdazodyssey4me kewl12:05
odyssey4methis review is a bit pointless: https://review.openstack.org/19670212:06
odyssey4mewe need to add the reserved port, but in containers it can't be done due to apparmor12:06
odyssey4meso the role only does it on a physical keystone server, which we don't have (except in an edge case where someone chooses to do that)12:07
odyssey4meI think we need an alternative approach and that perhaps this bug should be moved along to the next release.12:07
*** tlian has joined #openstack-ansible12:10
*** fawadk has quit IRC12:10
*** javeriak has quit IRC12:30
matttsvg: updated diff https://gist.github.com/mattt416/84fd07c8dd70378ff0e4 (this one includes the file i missed before)12:36
matttsvg: so were you suggesting to create a variable w/ contents of your local ceph.conf file? how would that be done?12:37
*** annashen has joined #openstack-ansible12:40
openstackgerritMerged stackforge/os-ansible-deployment: remove conntrackd package  https://review.openstack.org/19957212:40
openstackgerritMerged stackforge/os-ansible-deployment: Remove reference to missing user_group_vars file  https://review.openstack.org/20726412:44
*** annashen has quit IRC12:44
openstackgerritJesse Pretorius proposed stackforge/os-ansible-deployment: remove conntrackd package  https://review.openstack.org/20744112:47
*** javeriak has joined #openstack-ansible12:55
*** javeriak has quit IRC13:07
*** ig0r_ has quit IRC13:10
*** fawadkhaliq has joined #openstack-ansible13:11
matttsvg: oops, updated that commit w/ your local_action suggestion13:15
*** fawadkhaliq has quit IRC13:16
*** ig0r_ has joined #openstack-ansible13:17
*** Mudpuppy has joined #openstack-ansible13:21
*** Mudpuppy has quit IRC13:23
*** Mudpuppy has joined #openstack-ansible13:24
*** ig0r_ has quit IRC13:31
*** TheIntern has joined #openstack-ansible13:34
*** KLevenstein has joined #openstack-ansible13:39
*** annashen has joined #openstack-ansible13:40
svgmattt:  sorry, had also a blitz meeting when I got back :)13:43
svgsee http://sprunge.us/LYNU on how to put ceph conf in a var13:43
*** annashen has quit IRC13:45
*** Mudpuppy has quit IRC13:56
matttsvg: maybe it's best to just create the var as you did before, and have it then build the ceph.conf via the template13:58
matttsvg: that way we can obtain the mon list from the variable without having to have to respecify ceph_mons13:59
svgWell, thought of that too, the difference is small13:59
svghm, but how do you then say "I have no cephconf, go pick it up"?13:59
matttassuming there is no variable specified then it'd go out and pick it up14:00
mattt(ceph_conf_file or whatever you want to call it)14:00
matttsvg: maybe i'll just implement what you suggested, we can iterate on this later as needs arise14:01
svga list of mon hosts is always necessary, either way14:01
*** spotz_zzz is now known as spotz14:01
*** Mudpuppy has joined #openstack-ansible14:02
matttsvg: not if you can do ceph_conf.global.mon_host as your patch currently does14:02
svgWell, then it still is needed :)14:03
matttsvg: well, not as a separate variable :)14:03
mattt(that's gleaned from the config you've dumped in)14:03
*** Mudpuppy has quit IRC14:04
svgbut I mean, if you dont provide the ceph conf, he has to fetch it, he needs the list, so a spearate variable in that case - better keep it that way in all cases, no?14:04
*** yaya has joined #openstack-ansible14:04
matttsvg: suppose it's more consistent yes14:04
svgyes14:04
*** Mudpuppy has joined #openstack-ansible14:04
svgunless we ight need other components from ceph-conf later on, but currently not the case14:05
*** prad has joined #openstack-ansible14:05
matttsvg: actually good point, we will need something to indicate if you're using cephx or not14:06
*** markvoelker_ has quit IRC14:06
*** alop has joined #openstack-ansible14:10
evrardjpAm I the only one to find interesting to have a line log_path= in ansible.cfg in my playbooks folder?14:12
*** javeriak has joined #openstack-ansible14:13
*** spotz is now known as spotz_zzz14:13
*** sigmavirus24_awa is now known as sigmavirus2414:13
openstackgerritJesse Pretorius proposed stackforge/os-ansible-deployment: Change ansible forks used  https://review.openstack.org/20747414:14
odyssey4meevrardjp that's primarily there for gating - I'd like to change it so that it doesn't do it when you test with an AIO14:16
odyssey4mein fact, let me do that now - it causes issues for development and testing14:16
*** javeriak has quit IRC14:17
*** Mudpuppy has quit IRC14:18
evrardjpwhat do you mean by "that"? the ansible.cfg file?14:18
*** spotz_zzz is now known as spotz14:19
evrardjpother question, I've been away for 2 weeks, should I run all the playbooks to upgrade my environment, or just the openstack ones?14:20
odyssey4meevrardjp I need to adjust the bootstrapping to not edit ansible.cfg unless it's part of a gate check process, and also need to not setup the link to the logs directory unless the execution is part of a gate check14:20
odyssey4meevrardjp well, what tag/branch were you on and where are you now?14:21
evrardjpkilo14:21
evrardjpI just did a git pull so hard to say... I guess .414:21
evrardjplet me check14:21
palendaeWithin the last 2 weeks kilo had some rabbitmq bugs that affect upgrades there; so I'd say run all14:22
odyssey4meevrardjp you can check /etc/*release on any of your hosts14:22
*** jwagner_away is now known as jwagner14:23
evrardjpyup14:24
evrardjp.414:24
evrardjp11.0.4 to be complete14:24
evrardjpso no big changes14:24
odyssey4meok, so that was the last point release14:24
*** javeriak has joined #openstack-ansible14:25
odyssey4mewe're planning to release 11.1.0 tomorrow night - so perhaps you should wait a little14:25
*** Mudpuppy has joined #openstack-ansible14:25
evrardjpok14:25
odyssey4mealso, how you do the upgrade is really up to you - the deployment is designed in such a way that things shouldn't change what you already have - but you may wish to inspect all the changes and run one play at a time so that you can inspect the changes and catch anything unexpected14:26
*** alop has quit IRC14:26
*** markvoelker has joined #openstack-ansible14:27
odyssey4methe list of patches in review and not yet merged for 11.1.0 is https://review.openstack.org/#/q/starredby:%22Jesse+Pretorius%22+project:stackforge/os-ansible-deployment,n,z if you're curious14:27
palendaeDoes anyone know off the top of their head if we have an example of running an ansible task/role only on the deployment host?14:27
evrardjpI'd like to test (in a near future) the upgrade process for minor versions and also test new features (SSL support on HAproxy + ceph + federation)14:28
evrardjpso I'll wait for 11.114:28
*** markvoelker_ has joined #openstack-ansible14:28
prometheanfireSam-I-Am: only if you ask in here14:29
svgmattt: right, the current ceph_conf var is also used in the conditional in playbooks/roles/ceph_client/tasks/ceph_all.yml14:29
svgso we will need some ceph_auth_cephx: True var14:30
Sam-I-Amprometheanfire: what?14:31
prometheanfireSam-I-Am: don't ask for reviews in that channel :P14:32
Sam-I-Amoh14:32
odyssey4mepalendae something like this: echo 'localhost ansible_connection=local' > /tmp/ansible_inventory; ansible localhost -i /tmp/ansible_inventory -m shell -a 'ls -al /tmp'14:32
Sam-I-AmApsu prometheanfire rackertom can y'all look at https://review.openstack.org/#/c/204597/14:32
Sam-I-Amthere :)14:32
prometheanfirethanks14:32
*** markvoelker has quit IRC14:32
* prometheanfire puts away the open source bat14:32
*** ig0r_ has joined #openstack-ansible14:32
ApsuSam-I-Am: hi14:32
git-harrypalendae: is the 'Wait for ssh to be available' task in playbooks/lxc-containers-create.yml what you're looking for?14:33
*** alop has joined #openstack-ansible14:34
palendaeodyssey4me: Well, I'm doing it with the OSAD inventory; I'd like to distribute an SSH key that's only on the deploy host. The containers are failing the role because they don't have that file present14:34
odyssey4mepalendae we have a precent for distributing stuff from the deployment host14:34
odyssey4mepalendae eg: https://review.openstack.org/#/c/194474/19/playbooks/roles/os_keystone/tasks/keystone_ssl_user_provided.yml,cm14:35
odyssey4me*precedent14:35
*** Mudpuppy has quit IRC14:36
odyssey4mepalendae we do it for horizon in the same way: https://github.com/stackforge/os-ansible-deployment/blob/master/playbooks/roles/os_horizon/tasks/horizon_ssl_user_provided.yml14:37
*** javeriak_ has joined #openstack-ansible14:37
palendaeodyssey4me: Ok, thanks, I'll dig into these. I figured we had precedent, wanted to follow those14:38
*** jmckind has joined #openstack-ansible14:38
matttsvg: https://gist.github.com/mattt416/84fd07c8dd70378ff0e4 updated, i added a 'cephx' variable cheekily since that's what ceph-ansible also uses14:39
ApsuSam-I-Am: In the l3_agent.ini.j2 template, line 25 of your changed version... you've got an ansible assignment: {{ neutron_max_l3_agents_per_router | default(groups['neutron_agent'] | length) }}14:40
ApsuSam-I-Am: Are you sure you can put a jinja pipe inside of a jinja function like default()?14:41
*** javeriak has quit IRC14:41
*** annashen has joined #openstack-ansible14:41
*** ccrouch has joined #openstack-ansible14:42
odyssey4meApsu after trying some fancy stuff in default() recently, I doubt it14:43
odyssey4methat said, you should probably review in the review so that the discussion and changes have history :)14:43
*** bapalm_ has joined #openstack-ansible14:44
Apsuodyssey4me: mmhmm14:44
prometheanfireSam-I-Am: done14:45
*** annashen has quit IRC14:46
svgmattt: ok; cephx might also need a reference in the example vars file in etc/...14:47
matttsvg: i'm going to leave that out only because i'm guessing most installs will not be using ceph14:52
svgDo you know if we need to restart all the os services after ceph setup?14:54
matttsvg: git-harry and i were discussing that this morning, i don't know 100% but i don't think so14:57
svgAlso: I'm inclined to leave the ceph_client role run after the services role, and restart os services from a handler, which allows to just use the defaults defined in those roles14:58
svgdoing it with a handler makes sure it restarts after a tiny change too14:58
svgI was looking at the vars where the services are listed, and those are quite extensive, especially for nova; I think it's better to avoid needing to reimplement all those?14:59
*** jmckind has quit IRC15:00
git-harrymattt: svg I think they all need restarting because the rdb/rados python libraries don't exist when the opnestack services are started15:01
git-harryWhen ceph_client is run for the first time15:01
svgeither way, restarting them all is the safest and simpliest15:01
*** klindgren_ is now known as klindgren15:01
git-harrysvg: I'm not too concerned as to how it is done. Each way has its pluses and minuses, but it does need to be done15:02
*** ig0r_ has quit IRC15:03
*** ig0r__ has joined #openstack-ansible15:04
*** jmckind has joined #openstack-ansible15:05
*** logan2 has quit IRC15:08
*** ig0r__ has quit IRC15:09
*** ig0r_ has joined #openstack-ansible15:09
*** fawadkhaliq has joined #openstack-ansible15:11
*** ig0r__ has joined #openstack-ansible15:13
*** ig0r__ has quit IRC15:15
gparaskevasHello there15:15
*** ig0r_ has quit IRC15:15
gparaskevasa small problem when running the swift installation playbook15:15
gparaskevasi get this erros "'dict object' has no attribute 'swift_vars'"15:16
gparaskevashttp://paste.openstack.org/show/406450/15:16
gparaskevasmore here15:16
*** Mudpuppy has joined #openstack-ansible15:17
svgmattt: I'm going to need your patch on ceph-conf to put min to update the service restarts on top of it15:18
matttsvg: you want me to update your review with https://gist.github.com/mattt416/84fd07c8dd70378ff0e4?15:20
openstackgerritJesse Pretorius proposed stackforge/os-ansible-deployment: Move ansible logging to gate-check script only  https://review.openstack.org/20750215:20
svgmattt: I gues that's the easies?15:21
svgt15:21
matttsvg: yeah i can do that, sec15:21
*** phalmos has joined #openstack-ansible15:24
openstackgerritMatt Thompson proposed stackforge/os-ansible-deployment: Ceph/RBD support  https://review.openstack.org/20750415:27
*** sdake has joined #openstack-ansible15:27
matttbleh, ignore15:27
svg:)15:28
svgor I could just apply your patch on what I'm doing here15:28
*** rackertom has quit IRC15:29
prometheanfireabandoned, nice15:29
openstackgerritMatt Thompson proposed stackforge/os-ansible-deployment: Ceph/RBD support  https://review.openstack.org/18195715:29
matttsvg: ^^15:30
*** gparaskevas has quit IRC15:32
sigmavirus24zuul is really struggling: http://status.openstack.org/zuul/15:35
sigmavirus24I know we want to tag 11.1 tomorrow but it might not be realistic15:35
palendaeHelp them help us15:36
palendaeWonder if we can ask OSIC to lend some capacity to Zuul15:37
palendaeI guess we can and they can say no :)15:37
*** jmckind has quit IRC15:38
*** jmckind has joined #openstack-ansible15:39
*** jmckind has quit IRC15:39
*** jmckind has joined #openstack-ansible15:40
prometheanfireI would be happy with that :D15:41
sigmavirus24prometheanfire: "that"?15:42
prometheanfirezuul going faster15:42
prometheanfireboth locally and remotely...15:42
palendaeMake zuul use web workers and farm jobs out to browsers with the page open :)15:42
*** annashen has joined #openstack-ansible15:42
prometheanfirepalendae: must have been what happened to me :(15:43
prometheanfirehttp://bellard.org/jslinux/15:43
openstackgerritSerge van Ginderachter proposed stackforge/os-ansible-deployment: Add Ceph/RBD support to playbooks  https://review.openstack.org/18195715:43
svgmattt: git-harry ^^15:44
prometheanfireohgod15:45
Apsuprometheanfire: Been a while since I've seen the jslinux15:46
prometheanfireit only came out a year or two ago15:46
matttim hoping the js means NODEJS15:46
matttoh.15:46
prometheanfiremattt: ffffffuuuuu15:46
*** annashen has quit IRC15:47
Sam-I-Amprometheanfire: ping15:48
prometheanfireSam-I-Am: wat15:48
Sam-I-Amprometheanfire: your comment on my patch15:50
Sam-I-Amyou +1d it, then -1d it?15:50
prometheanfireyes, didn't see that comment15:50
prometheanfirethen I did15:51
Sam-I-Amthe vxlan_udp_port business15:51
svgmattt: git-harry that leaves 2 tasks with todo's in playbooks/roles/ceph_client/tasks/ceph_auth.yml15:51
git-harrysvg: great15:52
prometheanfireSam-I-Am: ya, you remove it yet?15:52
git-harrysvg: would you like us to pick those task up or are you working on fixes for those15:52
*** ig0r_ has joined #openstack-ansible15:53
svgDo you guys think the first one (TODO: also be able to create users, keys and pools on ceph) ca be deferred to later?15:53
Sam-I-Amprometheanfire: i guess it is not needed with the lb agent, so i s'pose we could15:53
matttsvg: we're going to handle that separately15:53
svgThat's one that will prolly pull in roles from ansible-ceph15:53
git-harrysvg: yeah I don't think that's an issue for now15:53
prometheanfireSam-I-Am: it's really a nit, but I'd prefer to do it now rather then forget about it15:53
* prometheanfire is very anti cruft15:53
prometheanfireI wonder if there's a type of coverage tool for ansible15:54
svgok, so for now just needing to fix the group owner on the ceph keyring file15:54
Sam-I-Amprometheanfire: makes sense15:55
svggit-harry: mattt I'm going afk for the day, I might have a look at that tomorrow - if you think there is a timeconstraint let me know and feel free to go forward?15:55
*** rward has quit IRC15:55
svgthose latest patches will need some extensive testing though, and right now I can't easily do that15:56
git-harrysvg:we'd like to get it merged tomorrow or at least have it ready for review by others.15:57
git-harrysvg: mattt and I will test it tomorrow15:57
svgI'm not sure by what time I'll be able to implement that last todo15:57
git-harrysvg: don't worry about the group issue, if you haven't got time that is fine. mattt and I will get it sorted15:57
openstackgerritMatthew Kassawara proposed stackforge/os-ansible-deployment: Reduce neutron configuration  https://review.openstack.org/20459715:58
Sam-I-Amprometheanfire: fixeraterated15:58
svgI'll get back to you tomorrow then, we'll see from there then15:58
svgcya15:58
git-harrysvg:bye15:58
evrardjpcya svg15:58
prometheanfireSam-I-Am: reviewing now, was that the only change you made?15:59
Sam-I-Amprometheanfire: yaep15:59
prometheanfireok15:59
prometheanfiremaking it easy then15:59
odyssey4mecloudnull, mattt, andymccr, d34dh0r53, hughsaunders, b3rnard0, palendae, Sam-I-Am, odyssey4me, serverascode, rromans, mancdaz, dolphm, _shaps_, BjoernT, claco, echiu, dstanek, jwagner, ayoung meeting in #openstack-meeting-4 starting in a minute15:59
dstaneko/16:01
matttsvg: cool, have a good night, talk to you tomorrow16:01
odyssey4medstanek in #openstack-meeting-4 :)16:01
dstanekodyssey4me: ha, then why are you pinging me here :-)16:02
odyssey4memiguelgrinberg can you join us in #openstack-meeting-4 please :)16:02
odyssey4medstanek 'cos this is where you like to hang out :p16:02
palendaed34dh0r53: If you can get https://bugs.launchpad.net/openstack-ansible/+bug/1479870 fixed, that would be awesome16:03
openstackLaunchpad bug 1479870 in openstack-ansible "teardown.sh does not cleanup lvm.conf" [Undecided,New]16:03
b3rnard0hello16:03
palendaeI think rackertom was looking at that issue, too16:03
*** TheIntern has quit IRC16:05
*** TheIntern has joined #openstack-ansible16:07
*** jmckind has quit IRC16:11
openstackgerritDavid Wilde proposed stackforge/os-ansible-deployment: Enable Horizon to consume a Keystone v3 API endpoint  https://review.openstack.org/20657516:11
*** jmckind has joined #openstack-ansible16:11
*** jmckind has quit IRC16:11
*** jmckind has joined #openstack-ansible16:11
*** woodard has joined #openstack-ansible16:14
*** logan2 has joined #openstack-ansible16:19
*** shausy has quit IRC16:24
openstackgerritgit-harry proposed stackforge/os-ansible-deployment: Add Ceph/RBD support to playbooks  https://review.openstack.org/18195716:27
*** jmckind has quit IRC16:28
git-harrysvg: I've uploaded a new patchset that creates the group. We can discuss if it's the best approach tomorrow but I wanted d34dh0r53 to be able to test a patchset that should work.16:29
*** TheIntern has quit IRC16:31
*** jmckind has joined #openstack-ansible16:32
evrardjpabout https://review.openstack.org/194395, I'm willing to test, but I may need some help to grasp the concepts (Ok for federation, not ok for integration with openstack right now)16:32
evrardjpcan give a review afterwards16:32
*** yaya has quit IRC16:36
dstanekwhere's chapter 1 in the install-guide?16:43
*** annashen_ has joined #openstack-ansible16:43
*** rackertom has joined #openstack-ansible16:43
dstanekalso is there any reason my configuring-ceilometer.rst isn't linked in the configuration section?16:47
*** annashen_ has quit IRC16:47
palendaedstanek: Oversight probably16:52
dstanekpalendae: i'd like to get it in there. is there any particular ordering or is at the end ok?16:54
palendaedstanek: None that I'm aware of16:54
palendaeSam-I-Am may have something to say about it :)16:55
dstanekpalendae: Sam-I-Am: not being in there is wrecking my patch!17:00
sigmavirus24evrardjp: if you want to help get the ceph work into 11.2 we'll need thorough reviews for at least the functionality if not also correctness of approach :D17:00
evrardjpagreed17:00
sigmavirus24dstanek: patch ch-ch-ch-ch-chain17:00
*** rward has joined #openstack-ansible17:00
odyssey4meevrardjp so it seems likely that ceph client support will merge pretty soon into master, and will possibly be backported into kilo for a point release like 11.1.1 depending on the state of it17:00
sigmavirus24evrardjp: I personally have no knowledge of ceph or a way to test against it (yet) so for me reviewing that patch will be pointless because I can only offer helpful criticisms around the ansibles and style (which svg already is better at than me)17:01
sigmavirus24odyssey4me: ceph support should be 11.x not 11.1.y17:01
sigmavirus24odyssey4me: that's a significant feature and since we're following semver, that belongs in a 11.x release17:01
odyssey4meit looks like mattt git-harry and svg are close to a client patch merge for the major starting point - they'll then iterate in smaller patches afterwards which are easier to review17:01
sigmavirus24evrardjp: regardless, please also review the ceph work. more eyes will make it better sooner17:02
evrardjpI have to go for today, let's continue this conversation tomorrow. I'll stick with the team to test them17:02
odyssey4mesigmavirus24 well, there's ceph cluster deployment and simply enabling the use of a ceph back-end - the current patch is just enabling the back-end17:02
evrardjpto test the integration*17:02
evrardjpsigmavirus24: ofc, same for other stuff like federation17:02
odyssey4mebut yeah, when that merges into kilo is above my pay grade :p17:03
odyssey4meis there such a thing as pay grade in open source? ;)17:03
evrardjpit will never be if you're not interested by it ;p17:03
evrardjpanyway I'm off17:03
*** abitha has joined #openstack-ansible17:04
d34dh0r53yeah, I'm testing the ceph-client side stuff today, the server side deployment will live in rpc-openstack AFAIK17:04
d34dh0r53ceph is slated for 11.2 right now, but that is not firm17:05
d34dh0r53afk lunch17:05
sigmavirus24odyssey4me: right I understand the patch17:12
sigmavirus24hence "no way to test against it (yet)"17:12
palendaed34dh0r53: Should specify "Rackspace's server side deployment will live in rpc-openstack" :)17:15
palendaeOther implementations can go elsewhere17:15
*** woodard has quit IRC17:16
openstackgerritNolan Brubaker proposed stackforge/os-ansible-deployment: Upgrade Elasticsearch and Kibana  https://review.openstack.org/20755217:22
odyssey4mepalendae there were some other playbook/role changes in rpc-openstack for ELK which may be required for that build to work - did you test it all?17:23
palendaeNot yet17:23
palendaeWill mark WIP17:24
odyssey4meah ok, cool17:24
palendaeIdeally we'd bump versions in Kilo...but those all have big changes17:24
palendaeSo that's not really on deck for Juno in my mind17:24
*** bapalm_ has quit IRC17:25
palendaeWe being RAX, not OSAD17:25
*** abitha has quit IRC17:31
*** abitha has joined #openstack-ansible17:32
openstackgerritMerged stackforge/os-ansible-deployment: Add python-cinderclient to os_glance role  https://review.openstack.org/20713017:38
*** daneyon has joined #openstack-ansible17:43
*** annashen has joined #openstack-ansible17:44
*** daneyon_ has joined #openstack-ansible17:44
*** daneyon has quit IRC17:47
*** annashen has quit IRC17:49
*** javeriak_ has quit IRC17:52
*** javeriak has joined #openstack-ansible17:52
*** TheIntern has joined #openstack-ansible17:54
*** annashen has joined #openstack-ansible17:56
*** woodard has joined #openstack-ansible17:59
*** woodard has quit IRC18:04
*** jmckind has quit IRC18:14
*** woodard has joined #openstack-ansible18:14
*** jmckind has joined #openstack-ansible18:14
*** jmckind has quit IRC18:15
*** ig0r__ has joined #openstack-ansible18:25
*** woodard has quit IRC18:26
meteorfoxquick question, I'm drafting the spec for the Rally role. I think I understand the role of an administrator, but is there are difference between deployer and operator?18:28
meteorfoxin my mind the operator is the same as the deployer, in the case of RPC, that would be us (Rackspace)18:29
*** javeriak_ has joined #openstack-ansible18:32
*** misc_ has joined #openstack-ansible18:34
*** jlvillal_ has joined #openstack-ansible18:34
*** jmccrory has quit IRC18:34
*** andymccr has quit IRC18:34
*** jlvillal has quit IRC18:34
*** misc has quit IRC18:34
*** jmccrory has joined #openstack-ansible18:34
*** javeriak has quit IRC18:35
*** jlvillal_ is now known as jlvillal18:35
*** woodard has joined #openstack-ansible18:39
*** alop has quit IRC18:40
*** britthouser has quit IRC18:43
*** britthouser has joined #openstack-ansible18:44
*** ig0r_ has quit IRC18:50
openstackgerritDavid Stanek proposed stackforge/os-ansible-deployment: Adds ceilometer link in configure.rst  https://review.openstack.org/20757618:54
openstackgerritDavid Stanek proposed stackforge/os-ansible-deployment: Uses tox for automating documentation builds  https://review.openstack.org/20757718:54
*** bapalm_ has joined #openstack-ansible19:00
dstanekif we like those changes i'll propose a docs job to infra19:01
dstanek^19:01
dstaneklate lunch time!19:03
*** ig0r_ has joined #openstack-ansible19:04
*** bapalm_ has quit IRC19:05
palendaedstanek: Will try to check this afternoon19:06
alextricityHey..I have a big problem I was hoping someone might help me with.19:08
*** jmckind has joined #openstack-ansible19:08
*** annashen has quit IRC19:08
alextricityTurns out some servers that were provisioned for OSAD only have 4G on the / moun19:08
*** jmckind has quit IRC19:08
sigmavirus24So fwiw, I think we should be prioritizing what we review and while I want dstanek's  stuff, I'd like that to go in next week at the earliest19:08
alextricityAny way I can configure the containers to use the /var mount for storage? This mount was configured with 500G19:09
palendaesigmavirus24: True19:09
*** jmckind has joined #openstack-ansible19:09
*** annashen has joined #openstack-ansible19:09
*** jmckind has quit IRC19:10
*** javeriak_ has quit IRC19:10
*** annashen has quit IRC19:10
*** jmckind has joined #openstack-ansible19:10
*** annashen has joined #openstack-ansible19:10
*** jmckind has quit IRC19:11
*** jmckind has joined #openstack-ansible19:12
*** jmckind has quit IRC19:13
*** jmckind has joined #openstack-ansible19:14
*** jmckind has quit IRC19:14
dstaneksigmavirus24: i'm in no hurry. i saw my bug get tagged with a release number last night and addressed it this morning19:20
*** jmckind has joined #openstack-ansible19:20
*** jmckind has quit IRC19:21
sigmavirus24dstanek: cool19:21
sigmavirus24I appreciate the tox stuff myself19:22
sigmavirus24I was hoping to tackle that + flake8 + bash8 etc for our stuff soonish19:22
mgariepyalextricity, you could create a directory on /var/ and bind-mount it to /openstack ?19:23
*** jmckind has joined #openstack-ansible19:23
*** annashen has quit IRC19:26
*** annashen has joined #openstack-ansible19:26
*** jwagner is now known as jwagner_away19:27
alextricitymgarie..don't know. How would I tell lxc to use that mount for disks?19:27
*** andymccr has joined #openstack-ansible19:28
mgariepyalextricity, containers will use /openstack/<container_name> folder, so everything will finish on the /var disk19:35
mgariepyalextricity, for my part tho, i use lvm on the nodes and will create /openstack with enough room on the lvm.19:37
alextricityTurns out I won't really get very far with a 4gb root partition on each node anyway19:37
alextricityI'm just going to re-provision19:38
mgariepyalextricity, http://osad.readthedocs.org/en/latest/install-guide/targethosts-configlvm.html19:38
*** alop has joined #openstack-ansible19:38
mgariepyalextricity, what are you using to provision your server ?19:38
alextricityjust PXE19:38
mgariepyok19:38
alextricitymount19:39
*** annashen has quit IRC19:40
*** annashen has joined #openstack-ansible19:41
*** fawadkhaliq has quit IRC19:42
*** fawadkhaliq has joined #openstack-ansible19:43
*** openstackgerrit has quit IRC19:46
*** openstackgerrit has joined #openstack-ansible19:47
*** alop has quit IRC19:47
*** alop has joined #openstack-ansible19:50
*** TheIntern has quit IRC19:51
*** alop has quit IRC19:53
*** jwagner_away is now known as jwagner19:55
*** annashen has quit IRC19:56
*** ig0r_ has quit IRC19:56
*** annashen has joined #openstack-ansible19:57
*** woodard has quit IRC19:57
*** jmckind has quit IRC20:04
*** TheIntern has joined #openstack-ansible20:12
*** woodard has joined #openstack-ansible20:12
*** openstackgerrit has quit IRC20:16
*** openstackgerrit has joined #openstack-ansible20:17
*** woodard has quit IRC20:18
palendaesigmavirus24 posted this http://lists.openstack.org/pipermail/openstack-dev/2015-July/071034.html20:33
dstaneksigmavirus24: ha, i have bash8 and spell checker patches that i haven't pushed yet20:34
sigmavirus24dstanek: go go go go20:34
palendaedstanek: Run bash8 against run-upgrade.sh20:34
*** jmckind has joined #openstack-ansible20:36
dstaneksigmavirus24: i was just pulling over the keystone stuff i find useful20:39
dstanekpalendae: yeah, that's why i haven't pushed yet :-P there are a few hundred things that need to be cleaned up20:39
palendaedstanek: laffo20:39
sigmavirus24dstanek: naaaaw20:39
palendaeWell, fwiw, I think the plan for that script in Liberty is to have most, if not all that logic removed20:40
palendaeBecause we're only doing jumps of 120:40
palendaeJuno to Kilo, Kilo to Liberty, Liberty to M20:40
palendaeSo Kilo -> Liberty will not have the scrubbing of rax-isms20:40
palendaeBut yeah, the kilo version of that file needs to be linted with like, 4 programs20:40
dstanekso.... 588 bashate error(s) found20:42
palendaeRight20:42
dstanekthat's across all shell scripts20:42
palendaeIt's 50% bash, 50% yaml, 50% python20:42
palendaeAnd 90% of the bash errors are probably in the bash :)20:42
openstackgerritIan Cordasco proposed stackforge/os-ansible-deployment: Fix Keystone URI/URL defaults  https://review.openstack.org/20519220:43
dstanekso the question is....is it worth it to have that stuff cleaned up?20:44
sigmavirus24probably20:44
sigmavirus24most of the bash scripts (besides run-upgrade.sh) keep with us20:44
palendae^20:45
palendaeI would say exclude run-upgrade.sh20:45
sigmavirus24for now20:45
sigmavirus24dstanek: do we know anyone that can get us a new release of bash8?20:45
sigmavirus24the one that's out is about a year old and missing many bug fixes20:45
palendaeThe in-line ansible playbooks were to prevent people from re-running them. However, i think a better approach in the future is to put them in an upgrade directory and use that20:45
sigmavirus24== palendae20:46
palendaebecause we're leaving them behind in the current version20:46
sigmavirus24that way ansible lint check will run against them20:46
palendaeSimilarly with any python scripts20:46
palendaeRight20:46
palendaeEach thing can then be linted20:46
palendaeAs intended20:46
*** openstackgerrit has quit IRC20:46
palendaeMake an 'upgrade/' junk drawer20:46
sigmavirus24clean it out every spring20:46
*** openstackgerrit has joined #openstack-ansible20:47
palendaes/spring/6 months/20:47
sigmavirus24spring comes every 6 months20:47
palendaeMore like ~8 months20:47
dstaneksigmavirus24: i don't know who runs that20:48
sigmavirus24dstanek: me either20:49
sigmavirus24I think the person who started it disappeared20:49
sigmavirus24it's an openstack thing20:49
dstanekoh, it's joe gordon20:50
sigmavirus24is it?20:50
dstanekhe's at least one of the committers - since it's an openstack-dev thing i'm surprise that it hasn't seen a release20:51
*** TheIntern has quit IRC20:52
*** JRobinson__ has joined #openstack-ansible20:52
dstaneki'll submit a patch where i ignore all of the errors and another one for each check to start enabling the checks20:52
dstaneki have a vacation day tomorrow and a long drive so i might as well get something done20:53
palendaedstanek: Nah20:53
palendaeI forget, are you presenting at PyOhio?20:53
dstanekno, i didn't have to have to worry about it at mid-cycle20:54
dstaneki would have probably had to work on it there instead of making friends and influencing people20:54
palendaeHaha20:55
*** jwagner is now known as jwagner_away20:56
*** yapeng has joined #openstack-ansible20:58
*** jmckind has quit IRC20:59
*** jmckind has joined #openstack-ansible21:00
*** openstack has joined #openstack-ansible21:11
Sam-I-Amdstanek: ping21:11
*** prad has quit IRC21:12
*** tlian has quit IRC21:13
*** annashen has quit IRC21:15
sigmavirus24dstanek: fly you fool21:16
*** annashen has joined #openstack-ansible21:18
*** tomjoseka has joined #openstack-ansible21:20
Sam-I-Amsigmavirus24: haha21:23
*** yapeng has quit IRC21:24
*** yapeng has joined #openstack-ansible21:31
*** yapeng has quit IRC21:32
*** yapeng has joined #openstack-ansible21:38
*** yapeng has quit IRC21:38
*** markvoelker_ has quit IRC21:46
dstanekSam-I-Am: pong21:49
*** alop has joined #openstack-ansible21:49
Sam-I-Amdstanek: you were asking me about some ceilometer docs?21:50
Sam-I-Amor asking someone...21:50
*** ig0r__ has quit IRC21:51
dstanekSam-I-Am: i didn't know where to link it so i did this: https://review.openstack.org/20757621:52
Sam-I-Amdstanek: o hey cool, there's another patch underneath this one21:54
Sam-I-Amsomething we needed to do21:54
dstanekSam-I-Am: i added a tox target for docs and i wanted it to fail for warnings and the ceilometer thing was the only warning21:55
Sam-I-Amyep, thats slick21:55
*** jmckind has quit IRC21:55
Sam-I-Amnot sure why ceilometer wasn't fully added. i didnt see it come through.21:55
dstaneki have a few more in the queue to add bashate and some others things i find useful21:56
dstanekjust have to polish the code a little21:56
Sam-I-Amthose look pretty good21:57
Sam-I-Amis there any CI on that stuff?21:57
Sam-I-Ami think our docs ci job broke (or something)21:57
dstaneki plan on adding ci jobs for these21:57
dstanekdo we currently publish the docs anywhere?21:58
Sam-I-Amyeah... sort of21:59
Sam-I-Amthey're on readthedocs21:59
Sam-I-Am(which is sort of funny, because no one ever does)21:59
Sam-I-Ami dont see why the ci couldn't publish them somewhere closer to the repo22:00
Sam-I-Amnot sure what the plans are for docs with big tent stuff22:01
Sam-I-Amdevref is not the place, imho... unless its dev stuff22:01
dstanekno idea, but when i make the docs job i'll ask infra22:01
*** yapeng has joined #openstack-ansible22:01
dstaneki would think this would go on docs.openstack.org22:01
*** annashen has quit IRC22:02
*** yapeng has quit IRC22:03
Sam-I-Amdstanek: thats my thought too22:03
Sam-I-Ambut i dont know if its set up for that yet22:04
*** sdake has quit IRC22:05
*** sdake has joined #openstack-ansible22:05
*** phalmos has quit IRC22:08
*** JRobinson__ has quit IRC22:08
*** markvoelker has joined #openstack-ansible22:08
*** Mudpuppy has quit IRC22:10
*** sdake has quit IRC22:10
*** sdake has joined #openstack-ansible22:11
dstanekSam-I-Am: i'll let you know when i start poking at the CI jobs22:14
Sam-I-Amdstanek: yeh, keep me in the loop22:15
*** yapeng has joined #openstack-ansible22:15
*** yapeng has quit IRC22:18
*** markvoelker_ has joined #openstack-ansible22:32
*** markvoelker has quit IRC22:35
*** harlowja has quit IRC22:44
*** harlowja has joined #openstack-ansible22:44
*** KLevenstein has quit IRC22:52
*** markvoelker_ has quit IRC23:14
*** Mudpuppy has joined #openstack-ansible23:21
*** jmckind has joined #openstack-ansible23:26
openstackgerritDavid Stanek proposed stackforge/os-ansible-deployment: Adds a pep8 target to tox.ini  https://review.openstack.org/20766223:27
openstackgerritDavid Stanek proposed stackforge/os-ansible-deployment: Removes trailing whitespace for bashate  https://review.openstack.org/20766323:27
openstackgerritDavid Stanek proposed stackforge/os-ansible-deployment: Removes tab indents for bashate  https://review.openstack.org/20766423:27
openstackgerritDavid Stanek proposed stackforge/os-ansible-deployment: Fixes deprecated arithmetic expansion for bashate  https://review.openstack.org/20766523:27
openstackgerritDavid Stanek proposed stackforge/os-ansible-deployment: Fixes function declarations for bashate  https://review.openstack.org/20766623:27
openstackgerritDavid Stanek proposed stackforge/os-ansible-deployment: Fixes loops for bashate  https://review.openstack.org/20766723:27
*** spotz is now known as spotz_zzz23:28
*** daneyon_ has quit IRC23:41

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!