16:02:34 <b3rnard0> #startmeeting OpenStack Ansible Meeting
16:02:34 <openstack> Meeting started Thu Feb 26 16:02:34 2015 UTC and is due to finish in 60 minutes.  The chair is b3rnard0. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:02:36 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
16:02:38 <openstack> The meeting name has been set to 'openstack_ansible_meeting'
16:02:40 <sigmavirus24> o/
16:02:44 <b3rnard0> #topic Agenda & rollcall
16:02:49 <cloudnull> present
16:02:53 <mancdaz> hello
16:02:53 <b3rnard0> #link https://wiki.openstack.org/wiki/Meetings/openstack-ansible#Agenda_for_next_meeting
16:02:59 <d34dh0r53> pres ent
16:03:06 <b3rnard0> 'ello!
16:03:10 <stevelle> hello
16:03:15 <mattt> o/
16:03:16 <Apsu> preSent
16:03:33 <b3rnard0> #topic Review action items from last week
16:03:45 <odyssey4me> o/
16:03:51 <Sam-I-Am> hi
16:03:53 <palendae> Hi
16:04:43 <b3rnard0> I did not see any meeting conflicts, so I think we are good to go for this room/meeting time
16:05:11 <b3rnard0> d34dh0r53 update blueprint with template from cloudnull -- status?
16:05:20 <d34dh0r53> done
16:05:42 <b3rnard0> cloudnull ping jhesketh about creating a separate repo -- status?
16:06:21 <cloudnull> i pinged him, but have to ping him again about all the things needed.
16:06:26 <cloudnull> that should be carried.
16:06:56 <b3rnard0> #action cloudnull continue pinging jhesketh about creating a separate repo and other things
16:07:14 <b3rnard0> BjoernT to help out with reviewing open PRs https://review.openstack.org/#/q/status:open+project:stackforge/os-ansible-deployment,n,z -- status?
16:07:26 <sigmavirus24> Bjoern isn't here
16:07:45 <b3rnard0> okay, adding that as ongoing
16:07:48 <odyssey4me> I suppose that's more of an info item anyway.
16:08:07 <b3rnard0> #info BjoernT to help out with reviewing open PRs https://review.openstack.org/#/q/status:open+project:stackforge/os-ansible-deployment,n,z
16:08:24 <b3rnard0> #topic Blueprints
16:08:37 <cloudnull> odyssey4me: you have the mic
16:08:41 <b3rnard0> #link https://blueprints.launchpad.net/openstack-ansible/+spec/apt-package-pinning
16:08:46 <BjoernT> hey
16:08:52 <b3rnard0> #link https://blueprints.launchpad.net/openstack-ansible/+spec/tunable-openstack-configuration
16:09:03 <odyssey4me> alright - apt pinning
16:09:06 <b3rnard0> hello BjoernT : we missed you :-)
16:09:35 <BjoernT> yes my calendar  alarm popped up to early to I got carried away
16:09:51 <odyssey4me> essentially apt-pinning is one part of helping to make a consistant environment to reduce the possibility of unexpected changes in upstream apt repositories from affecting a production environment
16:10:30 <odyssey4me> this is not the same as freezing a repository source - I'm looking into options there as a separate item which I think will go into a separate blueprint
16:11:03 <palendae> odyssey4me: Do we know how apt behaves if you pin a version and it gets removed from the repos?
16:11:07 <odyssey4me> the idea here is simply to be able to pin a known set of important packages at a particular version, and to ensure that the list of packages is configurable by a user of os-ansible-deployment
16:11:15 <cloudnull> palendae it dies in a fire
16:11:34 <odyssey4me> palendae I would think that it'll just fail an apt-get install...
16:11:38 <Apsu> odyssey4me: I disagree it's not the same as freezing. You may not be copying packages but you still have to A) identify the set of packages to consider as a unit in order to pin them and B) continue this process as a form of continual maintenance to update over time.
16:11:42 <odyssey4me> but if the package is already there, it should be fine
16:11:51 <Apsu> odyssey4me: Freezing is the same thing, just copying/packaging vs pinning.
16:12:29 <cloudnull> odyssey4me: for kilo could this be an rpc-extras thing?
16:12:36 <Apsu> Both methods require package-set focused testing to try to identify what the right combination of versions is.
16:12:44 <cloudnull> arguing that deployers may have a frozen image / repo that they want to use
16:12:55 <odyssey4me> Apsu: I think that's largely an academic discussion. The purpose of the pinning framework is not to have an entire copy of a repo, or even a partial copy... it's rather to configure apt to respect that you want a specific set of packages kept at a certain level.
16:13:11 <cloudnull> Im defining image as an iso / net boot image.
16:13:43 <cloudnull> and their pinning may be different than what we've stated should be pinned, with regard to ap
16:13:46 <cloudnull> *apt
16:13:55 <odyssey4me> The maintenance of the pinning list we provide in the project is ours to live with, but a deployer is capable of putting their own process in to augment/override that.
16:14:32 <odyssey4me> anyway, the blueprint's focus is to provide the capability to do pinning...
16:15:18 <odyssey4me> we have already done so in icehouse/juno, but for master I'm looking at something a little more robust
16:15:30 <BjoernT> We do pin already , or is this not really implemented and we put a file in place
16:15:49 <odyssey4me> BjoernT see my last statement :)
16:16:17 <mancdaz> BjoernT we removed the pinning in master, kind of
16:16:30 <cloudnull> right, but if we pin enforcing pinning of packages outside of our control, IE we're not providing a repo for them to be downloaded from, I'd argue again that deployers will balk at the potential of pinning packages that do not exist.
16:17:02 <cloudnull> just playing devils advocate here.
16:17:06 <odyssey4me> cloudnull sure, this is why I started looking at a more holistic view
16:17:07 <mancdaz> but the versions would be configurable?
16:17:18 <mancdaz> so they can pin to a version they do have in their repo
16:17:30 <odyssey4me> this was highlighted specifically when a package was pulled from Ubuntu's LTS repo
16:17:30 <palendae> mancdaz: I think cloudnull's saying if they don't carry the package at all
16:17:31 <cloudnull> mancdaz they would be configurable, but what if i wanted to deploy using 14.10
16:17:53 <BjoernT> Hmm, we from the support team wrapping up currently all the pining stuff which goes towards frozen apt repos on rackspace mirrors so pining will be another requirement so double seal it
16:17:56 <mancdaz> well that's a whole different kettle of fish
16:18:09 <Apsu> I think as soon as you have to actually do the process of picking a set of "vital" packages to pin, is the exact instant you've begun a frozen repo. For one thing, regular (fast) upstream repos don't keep long archives of previous package versions for a given distro release. Which means you need to copy the packages somewhere else and maintain that set yourself.
16:18:12 <cloudnull> mancdaz: or debian 7
16:18:14 <odyssey4me> BjoernT exactly
16:18:17 <Apsu> So even if your argument is the list is small, it's still a requirement.
16:18:18 <git-harry> Why not say this is deployer responsibility and then if there is a community clamour for something it can be revisited.
16:18:23 <mancdaz> cloudnull well yeah, or centos etc
16:18:27 <odyssey4me> having a frozen repo is one part, pinning is an extra protection on top of it
16:18:35 <BjoernT> right
16:18:36 <cloudnull> there's nothing techincally stopping us from doing that now. we just dont support it
16:18:56 <odyssey4me> this allows you to update your repo without causing an update across infrastructure entirely, then you can release/update the pins seperately in stages (if you want)
16:19:03 <BjoernT> Because I know that some packages just install their own apt sources and then they might override the rackspace mirrors
16:19:16 <mancdaz> odyssey4me or you do what we do with the python packages, in that you only provide the right version in the frozen repo
16:19:20 <mancdaz> hence no pinnig needed
16:19:43 <BjoernT> Yes, I agree, no pinning needed to python
16:19:56 <BjoernT> since we don't install apt packages in the first place
16:20:00 <odyssey4me> mancdaz yes, except as BjoernT has just mentioned, what happens if there's another repo on the host (like the default sources.list)
16:20:24 <odyssey4me> so the way I see it it's a combination of the two to really seal it
16:20:30 <Apsu> odyssey4me: An alternate solution is to make your repo updates use different series names, based on the revision of the software that matches the repo update, so a given deployment that's pointed at a previous series won't see the new packages anyway
16:20:31 <BjoernT> +1
16:20:33 <odyssey4me> 1) provide the ability to pin
16:20:49 <odyssey4me> 2) provide a frozen repo solution, similar to the python frozen repo solution
16:20:51 <Apsu> That's actually what I originally proposed during the primary "to freeze or not to freeze" discussions.
16:21:04 <odyssey4me> 1 gets used to allow staged updates in an environment
16:21:10 <odyssey4me> 2 is the backing repository
16:21:40 <Apsu> I.e., instead of a series target like "trusty" in your deployment repo, you have a semver tag, matching the code revision that goes along with the package set that was tested with that code revision at deployment time.
16:22:08 <odyssey4me> I'd like to separate the two items as, generally, pinning is actually not a bad stop-gap for the short-term needs...
16:22:09 <Apsu> Then you get pinning automagically, and can update your repo as you desire -- under new series targets.
16:22:30 <mancdaz> Apsu yes, except for the case mentioned above where someone drops a different sources.list entry
16:22:36 <mancdaz> pointing at trusty, or w/e
16:22:53 <odyssey4me> To have a repo mirror and freeze it, especially within our gating infrastructure, is slightly complicated. Do-able, but it'll take longer to get to a working state.
16:23:57 <Apsu> odyssey4me: We actually had one going already. It's just that it was determined to not be a good investment of resources to have to actually do the frozen-repo-ownership-and-testing-and-maintenance part.
16:23:59 <odyssey4me> So, in order to try and keep the discussion clear and on-point - I ask that we keep the view that the two items, while linked, are separate.
16:24:04 <Apsu> Because it's an awful lot of work to test and identify the versions.
16:24:51 <Apsu> I'll stop derailing now. I just think actually accomplishing a pinning config is a major microcosm of freezing, and has identical challenges and resource costs.
16:25:17 <BjoernT> I would skin the cat from a different way, QE tests one version with latest packages for instance. If good, generate a list of packages/version and use that to create the repos
16:25:19 <odyssey4me> Apsu: yes, a full frozen repo is more complex... which is why I'd like to keep the topic on pinning first, we can get to z frozen repo method later
16:25:51 <odyssey4me> BjoernT that's similar to the idea I have in mind - something that is relatively low maintenance and easily generated
16:26:42 <odyssey4me> the aim would be that a deployer of os-ansible-deployment would be able to generate a working set for their environment... but anyone who does many deploys or who wants to generate a working set for a versioned release can also do that
16:27:08 <odyssey4me> but again, now we're talking frozen repo... the blueprint is for putting the pinning infrastructure down first
16:27:17 <odyssey4me> so, technically, this is entirely off-topic
16:27:50 <odyssey4me> I'm hoping to have a blueprint ready for a frozen repo discussion next week, or perhaps a week after
16:28:02 <cloudnull> BjoernT:  but again to argue both sides, why does Rackspace QE get to say what goes into that set of "good packages"? there has to be some other criteria? so any solution should be flexible enough to work outside of the Rackspace private cloud.
16:28:18 <mancdaz> odyssey4me so aside from just adding package versions to the package names that apt installs, what is the blueprint framework achieving?
16:28:18 <cloudnull> odyssey4me sounds good.
16:28:21 <b3rnard0> #info odyssey4me I'm hoping to have a blueprint ready for a frozen repo discussion next week, or perhaps a week after
16:28:53 <odyssey4me> mancdaz there are a few things to agree on
16:29:30 <odyssey4me> 1) using the debops.apt_preferences Ansible Galaxy role, instead of developing something specific to os-ansible-deployment
16:29:43 <odyssey4me> is everyone happy with that approach?
16:29:46 <cloudnull> +1
16:29:57 <BjoernT> cloudnull: Because they sign off on a tested package/RPC release set. They don't care what version we use in terms of apt packages
16:30:19 <BjoernT> cloudnull: So they don't have a say, we just give them the latest versions we think are appropriate
16:30:24 <cloudnull> right BjoernT  thats an RPC thing
16:30:50 <andymccr> is this an extras thing though or should it go in the os-ansible-deployment repo? It feels a bit like this is a "how we deploy openstack specifically" rather than a more general thing.
16:31:07 <cloudnull> ^ thats what im arguing
16:31:09 <Apsu> Technically it has nothing to do with os-ansible-deployment.
16:31:18 <Apsu> It's the environment o-a-d lives in.
16:31:19 <d34dh0r53> +10
16:31:26 <odyssey4me> any opinions around using debops.apt_preferences as the role to handle pinning from now in master? or do we need to give everyone time to review and form an opinion?
16:31:31 <cloudnull> i like the ability for a package pinning framework using the upstream role.
16:31:52 <cloudnull> but idk if its a os-ansible-deployment thing, or an rpc thing
16:32:00 <Apsu> Separating concerns, o-a-d isn't concerned with its environment as long as it 'works' in the way it expects. Which is primarily a question of python and minor tooling, which are already specified in the pip wheel system.
16:32:00 <andymccr> i'd say rpc - imho
16:32:01 <b3rnard0> odyssey4me: we could put it out for comments/feedback and vote on it next week
16:32:15 <Apsu> OS package pinning/freezing is moreso a question of avoiding bugs in those bits.
16:32:20 <mancdaz> surely if you're providing anyone who uses o-a-d the opportunity to pin packages cleanly, it is an o-a-d thing?
16:32:27 <odyssey4me> ok, so I understand the view - but I would argue that it benefits os-ansible-deployment in that it helps with our development process... specifically with gating
16:32:29 <palendae> No arguments against debops.apt_preferences, but the o-a-d vs rpc discussions is important
16:32:42 <git-harry> What about soliciting feedback from the mailing list as to whether os package management should be part of the project?
16:32:46 <b3rnard0> #info odyssey4me:	 1) using the debops.apt_preferences Ansible Galaxy role, instead of developing something specific to os-ansible-deployment -- is everyone happy with that approach?
16:33:07 <cloudnull> git-harry: that sounds like a good idea .
16:33:09 <b3rnard0> #info odyssey4me: any opinions around using debops.apt_preferences as the role to handle pinning from now in master? or do we need to give everyone time to review and form an opinion?
16:33:18 <odyssey4me> if we pin stuff we use in-repo, then we avoid unnecessary distractions while we're trying to get os-ansible-deployment right
16:33:39 <b3rnard0> git-harry: thanks for volunteering on that!
16:34:09 <git-harry> b3rnard0: I think you mean odyssey4me
16:34:14 <odyssey4me> git-harry agreed, and I will send a mail out regarding the blueprint and soliciting feedback
16:34:27 <odyssey4me> I just want to do some tweaks before I do so.
16:34:37 <palendae> git-harry: Which mailing-list were you referring to? openstack-dev?
16:34:43 <b3rnard0> #action odyssey4me Solicit feedback from the mailing list as to whether os package management should be part of the project?
16:35:03 <mattt> openstack-operators may be well suited also
16:35:08 <git-harry> palendae: and the other one
16:35:14 <cloudnull> i'd say both
16:35:29 <mancdaz> all the lists
16:35:47 <odyssey4me> I'll do both.
16:35:48 <palendae> So we'll have both public and private discussions of it?
16:35:51 <b3rnard0> #info for odyssey4me action, send out to the openstack-dev and openstack-operators mailing list
16:36:06 <Apsu> mancdaz: https://www.drupal.org/files/x-all-the-things-template.png
16:36:32 <cloudnull> ok, we've beaten this BP to death. lets get it on the ML once its ready and we'll go from there.
16:36:49 <Apsu> Sorry, and you're welcome. Carry on :)
16:36:54 <cloudnull> palendae: so long as its a bp on o-a-d its a public discussion .
16:37:16 <palendae> cloudnull: ok, I wasn't sure what 'the other one' git-harry was referring to was
16:37:23 <cloudnull> ah.
16:37:38 <cloudnull> no point on putting it on our internal MLs
16:37:43 <palendae> ^
16:37:44 <git-harry> palendae: sorry, yeah I meant openstack
16:37:45 <palendae> Agreed
16:37:56 <cloudnull> NEXT: BP
16:38:14 <cloudnull> Implement tunables for all OpenStack roles -- odyssey4me
16:38:27 * cloudnull hands mic to odyssey4me
16:38:56 <odyssey4me> right
16:39:12 <odyssey4me> so see the BP as an idea, not necessarily well formed at this stage
16:39:44 <odyssey4me> we're often getting requests for 'please add the ability to configure <x> in <y>.conf'
16:40:03 <odyssey4me> that goes through a cycle of adding some variables, and the motions of test and release
16:40:12 <BjoernT> yes, that will only increase the more we roll RPC out
16:40:20 <git-harry> Yeah, I don't see the point in hard coding templates
16:40:29 <odyssey4me> slowly, over time, we're going to end up with a ton of variables and may also end up with a situation where many of them don't apply any more
16:40:38 <odyssey4me> (as upstream has removed/changed them)
16:40:57 <odyssey4me> so I was thinking that perhaps we can introduce the concept of tunables
16:41:11 <palendae> odyssey4me, BjoernT: Would that relate to https://bugs.launchpad.net/openstack-ansible/+bug/1424525 then?
16:41:13 <openstack> Launchpad bug 1424525 in openstack-ansible trunk "Make policy.json configurable / replaceable" [Medium,Triaged] - Assigned to Nolan Brubaker (palendae)
16:41:13 <odyssey4me> simply put, have our base required stuff as the variables as we do now
16:41:13 <andymccr> we did used to have almost exactly what you describe in the bp for a few of the openstack projects
16:41:25 <odyssey4me> but allow other bits to be added into tunables
16:41:39 <odyssey4me> then, for any deployment, these tunables can be set as desired
16:41:47 <odyssey4me> very flexible, future proof
16:41:48 <BjoernT> the policy is probably different from normal key/value template changes
16:41:53 <palendae> Ok
16:42:18 <cloudnull> so we did have this before and it didn't work really well. IE https://github.com/stackforge/os-ansible-deployment/blob/icehouse/rpc_deployment/vars/config_vars/keystone_config.yml https://github.com/stackforge/os-ansible-deployment/blob/icehouse/rpc_deployment/roles/keystone_common/templates/template_gen
16:42:22 <odyssey4me> palendae that bug is a different kettle of fish
16:42:37 <palendae> odyssey4me: Ok, carry on then :)
16:42:53 <cloudnull> the issue was related to our support team having to manage all of that.
16:43:03 <odyssey4me> cloudnull that, I would say, is exactly how not to do it
16:43:08 <cloudnull> but if we're good with that, it does making it flexable.
16:43:18 <odyssey4me> that is far worse than what I'm proposing
16:43:38 <odyssey4me> what I'm saying is that we keep the essentials in normal variables, assigned in the normal ways
16:43:58 <andymccr> the problem is
16:44:03 <cloudnull> and add generators to specific things ?
16:44:10 <git-harry> half and half is a bad idea
16:44:15 <andymccr> if you want to override a certain var you have to override the whole nova_conf_settings: var for example?
16:44:18 <odyssey4me> it's the fringe settings that should be tunable in the manner I'm suggesting - settings which may or may not exist yet
16:44:33 <cloudnull> which will result in 1 request to add a "normal" item and another on how to add something complext to the generator.
16:45:12 <git-harry> andymccr: recursion
16:45:22 <cloudnull> but if we can think of a better way to make it work, i'm down.
16:45:30 <cloudnull> itll fix a never ending request.
16:45:40 <Apsu> ^
16:46:15 <odyssey4me> I think the idea needs development, and there are positive and negative aspects to each approach...
16:46:50 <odyssey4me> but yes, I really do think that it'll help us focus development effort in the right places and can make things super flexible (and reduce turnaround) for deployers
16:47:13 <b3rnard0> do we want to solicit feedback as well for this blueprint?
16:47:17 <cloudnull> agreed. its an awesome idea.
16:47:19 <odyssey4me> so I'd like to recruit volunteers to help develop the idea
16:47:45 <odyssey4me> perhaps what I could do is prepare an email with a variety of approaches and send it to the ML list
16:47:53 <cloudnull> im down to help, but it may be good to get some other opinions in there.
16:47:53 <b3rnard0> +1
16:47:56 <odyssey4me> but, I'd like someone to help come up with those approaches
16:48:13 <odyssey4me> someone(s)
16:48:16 <odyssey4me> :)
16:48:24 * cloudnull looks around for volunteers
16:48:29 <b3rnard0> #info odyssey4me: perhaps what I could do is prepare an email with a variety of approaches and send it to the ML list
16:48:30 <mancdaz> I like the 'do it all in one' rather than half and half
16:48:35 <mancdaz> I volunteer git-harry
16:48:37 * b3rnard0 prepares the voluntell machine
16:48:43 <cloudnull> hahahaha
16:48:58 <mancdaz> git-harry has quite a good clear idea of how it could work
16:49:11 <cloudnull> boom typie typie, make it go
16:49:18 <cloudnull> git-harry voluntold.
16:49:24 <git-harry> I'm allergic to being voluntold
16:49:29 <cloudnull> hahahaha
16:49:44 <b3rnard0> #action git-harry help odyssey4me on blueprint https://blueprints.launchpad.net/openstack-ansible/+spec/tunable-openstack-configuration
16:50:02 <b3rnard0> anyone else?
16:50:25 <b3rnard0> okay let's move on then
16:50:32 <cloudnull> last few min , BjoernT you want to hit on those bugs listed?
16:50:33 <odyssey4me> The help would largely be to play devil's advocate.
16:50:39 <b3rnard0> skipping revews let's do the bugs
16:50:50 <b3rnard0> #topic Bugs
16:50:56 <Apsu> cloudnull: The bugs are definitely lookers. They could use some BjoernT hitting-on
16:51:03 <b3rnard0> #link https://bugs.launchpad.net/openstack-ansible/+bug/1424808
16:51:04 <openstack> Launchpad bug 1424808 in openstack-ansible juno "Add nova configuration options image_cache_manager_interval" [Low,Triaged] - Assigned to Andy McCrae (andrew-mccrae)
16:51:09 <odyssey4me> #link https://github.com/stackforge/os-ansible-deployment/blob/icehouse/rpc_deployment/vars/config_vars/keystone_config.yml
16:51:12 <odyssey4me> #link https://github.com/stackforge/os-ansible-deployment/blob/icehouse/rpc_deployment/vars/config_vars/keystone_config.yml https://github.com/stackforge/os-ansible-deployment/blob/icehouse/rpc_deployment/roles/keystone_common/templates/template_gen
16:51:13 <b3rnard0> #link https://bugs.launchpad.net/openstack-ansible/+bug/1424971
16:51:13 <openstack> Launchpad bug 1424971 in openstack-ansible "ansible keystone module breaks with mixed case name" [Wishlist,New]
16:51:13 <cloudnull> if not ill wishlist them
16:51:22 <odyssey4me> sorry - just adding those for reference
16:51:36 <b3rnard0> odyssey4me: np
16:51:43 <b3rnard0> where did BjoernT go?
16:51:49 <BjoernT> yeah I know what the background is on this one
16:52:36 <BjoernT> I mean 1424971 charles requested that this one get's fixed but I think he didn't know it was caused by manual changes
16:52:46 <cloudnull> imo this is someone made bad life choices and now we want a way to deal with those choices.
16:52:49 <BjoernT> so I would wait with this bug until I talked with everyone
16:53:56 <cloudnull> BjoernT: on https://bugs.launchpad.net/openstack-ansible/+bug/1424808 is that something you'd like to see in 10.1.3 ?
16:53:57 <openstack> Launchpad bug 1424808 in openstack-ansible juno "Add nova configuration options image_cache_manager_interval" [Low,Triaged] - Assigned to Andy McCrae (andrew-mccrae)
16:53:57 <BjoernT> 1424808 looks ok to me
16:54:07 <BjoernT> yes 10.1.3 is ok
16:54:29 <cloudnull> ^ b3rnard0  action item
16:55:07 <cloudnull> any other open issue we want to talk about ?
16:55:18 <b3rnard0> #action andymccr to fix https://bugs.launchpad.net/openstack-ansible/+bug/1424808 in 10.1.3
16:55:19 <openstack> Launchpad bug 1424808 in openstack-ansible juno "Add nova configuration options image_cache_manager_interval" [Low,Triaged] - Assigned to Andy McCrae (andrew-mccrae)
16:55:51 <b3rnard0> #info BjoernT: I mean 1424971 charles requested that this one get's fixed but I think he didn't know it was caused by manual changes
16:56:06 <b3rnard0> #topic Open discussion
16:56:29 <cloudnull> ok lets call it. give everyone back a few mins.
16:57:09 <Apsu> Woot, meeting rollover minutes!
16:57:13 <cloudnull> thanks everyone .
16:57:17 <Apsu> Good talk.
16:57:18 <odyssey4me> :)
16:57:20 <cloudnull> Apsu hahaha.
16:57:21 <b3rnard0> #endmeeting