16:05:07 <DuncanT-> #startmeeting cinder
16:05:08 <openstack> Meeting started Wed Aug  6 16:05:07 2014 UTC and is due to finish in 60 minutes.  The chair is DuncanT-. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:05:09 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
16:05:11 <openstack> The meeting name has been set to 'cinder'
16:05:24 <thingee> o/
16:05:24 <glenng> Hi
16:05:26 <DuncanT-> Hopefully John will be along in a moment.... who else is about?
16:05:27 <tbarron> hi
16:05:28 <eharney> hi
16:05:32 <dustins> \o
16:05:32 <e0ne> hi
16:05:33 <xyang1> hi
16:05:34 <jkremer> hi
16:05:36 <jungleboyj> o/
16:05:37 <kmartin> o/
16:05:40 <Yogi1> hi
16:05:41 <ronenkat_> hi
16:05:56 <smcginnis> o/
16:05:59 <DuncanT-> #topic Mid cycle meetup
16:06:07 <DuncanT-> Do we have Scott yet?
16:06:22 <Swanson> hello
16:06:32 <thingee> scottda: ^
16:06:40 <jgriffith> o/
16:06:44 <thingee> jgriffith: yay!
16:06:49 <DuncanT-> Hey John
16:06:53 <jgriffith> sorry folks
16:06:59 <jgriffith> traffic
16:07:03 <DuncanT-> Right, we'll come back to him
16:07:15 <DuncanT-> #topic Replication
16:07:21 <jgriffith> sweet
16:07:23 <ronenkat> hi,
16:08:03 <jgriffith> has anybody else looked at the two approaches?
16:08:07 <ronenkat> Following discussions with John, I looked at how you would do replication via drivers, with limited support in Cinder iteself
16:08:17 <ronenkat> I put a design at https://etherpad.openstack.org/p/juno-cinder-volume-replication-apparochs
16:08:55 <scottda> Hi all. Sorry I'm late
16:08:55 <asselin_> hi
16:08:55 <thingee> jgriffith: I've have read both
16:09:01 <thingee> I have *
16:09:20 <ronenkat> it puts more burden on the driver, creating the volumes, but reduces the burden of the generic Cinder code
16:09:31 <ronenkat> I have very little db footprint
16:09:59 <ronenkat> I think it is less flexiable and harder on the driver, but can be a good start
16:10:15 <jgriffith> ronenkat: I'd rather put the burden there for a start at least :)
16:10:19 <DuncanT-> I like the simple approach first, though I wonder if people who know more about replication can see anything missing?
16:10:37 <DuncanT-> Being harder on the driver is fine I think, drivers might chose to share some code where that makes sense
16:10:40 <jgriffith> DuncanT-: full disclosure there are missing pieces
16:10:48 <jgriffith> I fully admit
16:10:59 <jgriffith> and ronenkat has pointed some of them out
16:11:14 <jgriffith> fail over, disabling etc
16:11:14 <kmartin> agree with the simply approach for us anyways
16:11:24 <eharney> i'm mostly trying to sort out whether this leaves the driver doing things it shouldn't really be responsible for... seems like it could be reasonable
16:11:36 <jgriffith> eharney: expand?
16:11:44 <thingee> jgriffith: I think that's what we wanted anyways. a simple first pass to build off of
16:11:49 <thingee> espescially this late
16:11:51 <jgriffith> eharney: ie "what should a driver NOT be doing"
16:11:51 <ronenkat> the major point I am concerned with is that a driver may need to talk with another backend
16:12:17 <eharney> jgriffith: i'm just trying to make sure this doesn't bring manager/service type behavior too far down into the driver itself... so far i think it doesn't but i'm still sorting that out
16:12:18 <thingee> not that this late is ronenkat's fault. there have been slow responses back and it's a difficult problem.
16:12:26 <jgriffith> ronenkat: I actually thought  who better to talk to a SolidFire backend than a SoidFire driver :)
16:12:35 <jgriffith> eharney: ahh... got ya
16:12:53 <hemna> morning
16:12:53 * jgriffith has been the blocker on this for sure
16:13:26 <ronenkat> but I think that the scheduler-aware pools is actually a good concept that address talk to multi backends
16:13:35 <jgriffith> eharney: fair point... my position would be as soon as you do dissimilar backends it needs to move
16:14:01 <jgriffith> ronenkat: hmm... ya know that's a really good point
16:14:07 <eharney> jgriffith: that sounds about right to me
16:14:14 <jgriffith> model it off of the pools design?
16:14:35 <jgriffith> ronenkat: that's interesting....
16:14:41 <ronenkat> jgriffit: I really liked how the driver based apparoch works with the pool-aware scheduler
16:15:03 <jgriffith> Ok.. so maybe we should tackle that
16:15:11 <jgriffith> ronenkat: are you available next week?
16:15:26 <ronenkat> availables yes, but remotly
16:15:34 <jgriffith> that's almost as good :)
16:15:48 <ronenkat> jgriffit: I plan to get the code up by Monday, so we can review it next week
16:15:49 <jgriffith> maybe we can work this out as a team next week?
16:15:58 <jgriffith> thingee: DuncanT- hemna kmartin ^^ ?
16:16:06 <thingee> sure
16:16:09 <hemna> the pool scheduler or replication ?
16:16:13 <kmartin> +1
16:16:27 <hemna> or both :P
16:16:29 <kmartin> both I hope
16:16:30 <jungleboyj> +1
16:17:27 <ronenkat> jgriffit: I think that the code you put up for driver support, would work nicely with what I put up on the etherpad
16:17:47 <DuncanT-> Makes sense to beat on it in person, given the timing
16:18:34 <jgriffith> hemna: combo
16:18:38 <jungleboyj> Yeah, definitely seems like something to talk through in person.
16:18:43 <thingee> DuncanT-: seems like no disagreements?! moving on?
16:18:44 <hemna> yah
16:18:49 <jgriffith> ronenkat: yeah... reading
16:19:02 <DuncanT-> #topic mid cycle meetup
16:19:03 <hemna> I had some feedback for winston-d_ on the pool scheduler that I think closes some holes with it.
16:19:07 <thingee> scottda: ^
16:19:07 <DuncanT-> scottda: You're up
16:19:20 <scottda> I've not much to say. The etherpad is up to date https://etherpad.openstack.org/p/CinderMidCycleMeetupAug2014
16:19:26 * thingee stops being a backseat driver to DuncanT-
16:19:29 <winston-d_> ronenkat: mind post the url for etherpad if it's not this one: https://etherpad.openstack.org/p/juno-cinder-volume-replication-apparochs
16:19:46 <scottda> I just want to make sure there's nothing else that needs to be done from my end re: the physical meetup
16:20:00 <hemna> What time is it starting Tuesday?
16:20:00 <scottda> I've done nothing for the Virtual meetup details.
16:20:03 <DuncanT-> ~17 of us, nice
16:20:05 <rushiagr> hi all!
16:20:09 <winston-d_> hemna: will go through your concerns about pool-aware scheduler after meeting
16:20:14 <ronenkat> winston-d_: that is the link
16:20:16 <hemna> winston-d_, sweet
16:20:30 <kmartin> scottda: thanks for helping put this together
16:20:37 <scottda> no problem
16:20:39 <jgriffith> yeah scottda thanks a ton
16:20:54 <scottda> Thanks go to our admin Pam. You'll be able to thank her in person.
16:20:54 <jungleboyj> hemna: I was wondering that too.
16:21:07 <hemna> 9am?
16:21:14 <jungleboyj> scottda: You guys have admins!?!
16:21:15 <scottda> start whenever. I've arranged coffee/snacks in the meeting room.
16:21:25 <kmartin> scottda: I think we can do a google hangout on the fly
16:21:29 <hemna> scottda, great thanks
16:22:01 <scottda> HP employees can maybe help escort non-hp from the front Security to the meeting room.
16:22:24 <hemna> yah will do.
16:22:29 <kmartin> scottda: sure
16:22:45 <scottda> I'll be onsite early and my phone # is on the etherpad
16:22:49 * jgriffith needs to call his old coworkers at HP FC
16:23:03 <thingee> winston-d_, DuncanT-: since we have time, could talk about the pool aware scheduler?
16:23:26 <DuncanT-> 'escort' http://www.hooverae.com/upload/files/050108/7825617.jpeg
16:23:28 <jungleboyj> hemna: Should we call it 9 am with no other suggestions?
16:23:50 <jungleboyj> DuncanT-: Nice.
16:23:52 <DuncanT-> Let's call it 9am...
16:23:53 <hemna> yah that's what I was thinking
16:23:56 <jgriffith> +1 for 9
16:23:57 <hemna> 9am sounds reasonable
16:24:03 <bswartz> scottda: you're providing the catering right?
16:24:03 <scottda> sure
16:24:09 <bswartz> lol
16:24:17 <scottda> We've breakfast every day and lunch Tues...
16:24:25 <DuncanT-> And coffee
16:24:32 <scottda> I could have lunch brought in the other days, but thought people might want to escape the room.
16:24:40 <hemna> :)
16:24:44 <scottda> And I'll work out some beers somewhere Tues night.
16:24:49 <winston-d_> oh, I wish I could be there, for food's sake. :)
16:24:55 * thingee should make some hipster coffee for folks
16:24:59 <hemna> for beer's sake!
16:25:05 <winston-d_> hemna: ha!
16:25:18 <DuncanT-> 'hipster coffee'.... I'm intreageud
16:25:23 <DuncanT-> Anyway....
16:25:25 <DuncanT-> #topic pool scheduler
16:25:28 <winston-d_> thingee: sure, I can talk about pool-aware scheduler
16:25:30 <hemna> hopefully it doesn't have a beard.
16:25:55 <DuncanT-> winston-d_: You're up
16:26:14 <jungleboyj> hemna: Ew.
16:26:43 <winston-d_> thingee: since you bring this up, what is in your mind you want to talk about pool-aware scheduler?
16:27:03 <bswartz> winston-d_: merge it!
16:27:10 <hemna> heh
16:27:16 <winston-d_> or anyone else have specific question/concern?
16:27:20 <winston-d_> bswartz: +2, done!
16:27:40 <hemna> so one of the issues I have with it now is that it doesn't seem to handle the case where folks put a pool in a volume type.
16:27:48 <thingee> winston-d_: I don't have anything specific. You said talk about it after the meeting. figured now would be fine
16:27:52 <winston-d_> bswartz: you can show your support by +1, though.
16:27:54 <kmartin> thingee: I think it's getting good reviews and markstur posted a WIP patch with the 3PAR driver
16:28:07 <hemna> our drivers will have a cinder.conf that lists the pools we want to support, but that doesn't prevent someone from putting another pool into a volume type.
16:28:15 <thingee> I honestly haven't reviewed the code since I've been focusing on data/control path in driver for jgriffith and the replication changes. Pool scheduler is next on my list
16:28:18 <hemna> we want to be able to report on all the pools for stats.
16:28:38 <DuncanT-> hemna: Your driver can (and should) just refuse the create
16:28:40 <winston-d_> hemna: actually, I didn't quite get the part you mentioned, rushiagr said the same.
16:28:43 <hemna> I think the drivers need to have another param passed in to get_volume_stats
16:29:00 <hemna> we use volume types extensively today for pools
16:29:12 <hemna> and we don't require admins to put all of those pools into cinder.conf
16:29:23 <hemna> so there could be pools that only exist in volume types
16:29:37 <winston-d_> hemna: yeah, like DuncanT- said, if someone accidentally put something wrong into vol types, you driver can just refuse to create
16:29:49 <hemna> so we need to have the volume types passed in to get_Volume_Stats, so we can report on any pools that are specified there
16:30:01 <hemna> they didn't put anything wrong into the type
16:30:18 <hemna> they put a pool into the type, because they want to ensure that when the volume is created it goes to that specific pool
16:30:20 <hemna> we do this today.
16:30:24 <jgriffith> hemna: you'll need to change your driver :)
16:30:28 <winston-d_> hemna: types are dynamic
16:30:33 <rushiagr> hemna: are there any other drivers which do it the way you do it?
16:30:34 <DuncanT-> Supporting legacy, HP only type based pools is not currently in the design
16:30:35 <hemna> because that pool has certain characteristics that they operators charge customers for.
16:31:12 <hemna> winston-d_, yes, they are dynamic.  that's the point of them.  :)  And it's also why we use types to specify pools
16:31:14 <winston-d_> hemna: how about you report all pools the driver (admin) would like to support and let scheduler decide which ones are good?
16:31:29 <rushiagr> hemna: I 'sort of' understood what your driver is doing, but couldn't understand how it will be affected if that param is not specified in the get_vol_stats
16:31:42 <hemna> the scheduler needs to be aware of what's in the type
16:32:08 <hemna> or we'll get a request to create a volume on what the scheduler thinks as well as a pool listed in the type, which would be different.
16:32:09 <markstur> scheduler shouldn't pick a "type pool" for the wrong type or a "non-type pool" for a type
16:32:16 <winston-d_> hemna: scheduler is, it has a capabilities filter to take care extra specs in vol types
16:32:23 <hemna> the host entry at that point would be what the scheduler chose, not what the user chose from the type.
16:32:46 <hemna> ok, so how do we report the pool if it only exists in the type?
16:32:48 <jgriffith> hemna: just use scoped keys to differentiate
16:32:54 <jgriffith> hemna: then you can support both
16:33:12 <hemna> admins can create types after cinder is up, which would mean that certain pools only live in the type
16:33:25 <winston-d_> hemna: what do you mean by pool that only exists in the type?
16:33:29 <hemna> the driver won't report on pools in types, because it doesn't have access to the types at get_volume_stats time
16:33:42 <DuncanT-> So you can have a working pool that your driver doesn't know anything at all about? That's nuts!
16:34:08 <hemna> winston-d_, 1) admin configures cinder.conf for Pools A-C   2) at a later date creates a type with Pool D
16:34:18 <jgriffith> hemna: I'm having a hard time understanding what you're getting at....
16:34:20 <hemna> DuncanT-, it does and it doesn't.
16:34:23 <markstur> driver knows about it, scheduler doesn't know how to associate pools with types (or not)
16:34:48 <markstur> scheduler says create volume in pool1 w/ type2, type2 says use "premier" pool only.
16:34:48 <hemna> the driver only knows about it by the pool only existing in the type.
16:34:53 <hemna> it's not in cinder.conf
16:35:09 <jgriffith> bottom line is I can't imagine you can't *fix* it, and holding up everything for the 3par driver doesn't seem *good*
16:35:25 <hemna> This just seems like a hole in the scheduler for us.
16:35:29 <DuncanT-> hemna: Pools don't have to all be listed in cinder.conf.... the driver can just report them
16:35:45 <hemna> as 1) we won't be able to report stats correctly unless we get the list of volume types at get_volume_stats time.
16:35:57 <jgriffith> hemna: that's a bug
16:36:16 <jgriffith> hemna: I don't understand why you'd rely on an arbitrary thing like types for stats reporting?
16:36:20 <DuncanT-> hemna: That's an insane design
16:36:32 <hemna> *sigh*
16:36:34 <hemna> it's not.
16:36:47 <winston-d_> hemna: this approach is broken if there were other types using similar extra specs for their backend
16:36:50 <hemna> we aren't going to report every single pool on the 3PAR
16:37:03 <hemna> because we don't want volumes going to every pool on the array.
16:37:07 <hemna> only certain pools
16:37:10 <jgriffith> hemna: so let me see if I understand this...
16:37:13 <hemna> which are defined in cinder.conf
16:37:16 <hemna> but
16:37:23 <jgriffith> hemna: currently.. you inspect the volume-type and look for a pool spec in it
16:37:25 <hemna> admins can add pools to types that aren't in cinder.conf
16:37:28 <winston-d_> hemna: say 3par has pool a,b,c, and the driver is able to report pool d if there was a type has extra specs says pool d, right? is this your desgin?
16:37:31 <hemna> jgriffith, yes
16:37:32 <jgriffith> hemna: if it's there you report that back in stats????
16:37:39 <jgriffith> that makes no sense
16:37:40 <hemna> jgriffith, currently no,
16:37:46 <jgriffith> ok... good :)
16:37:49 <hemna> because we don't have that information at get_volume_stats time
16:37:49 <jgriffith> so what do you do
16:38:01 <hemna> but we want to
16:38:06 <jgriffith> no you don't :)
16:38:09 <hemna> lol
16:38:11 <jgriffith> seriously.. you don't
16:38:17 <jgriffith> that's bad juju
16:38:20 <hemna> ?
16:38:27 <hemna> I don't understand that
16:39:07 <winston-d_> hemna: if there is a storwize and netapp in the same cinder deployment, admin create a type trying to associate that type to netapp, with pool d. Would your driver try to find a pool d simply because it looks at all types?
16:39:07 <jgriffith> You sould report stats based on what you have and leave it to scheduler
16:39:14 <DuncanT-> hemna: Just report stats on all pools.... admin needs *somewhere* authoritative to know what he's allowed to put in a type
16:39:25 <jgriffith> DuncanT-: +1
16:39:36 <hemna> winston-d_, no, we should only get the volume type that has the host associated with our instance.
16:39:40 <hemna> types
16:40:11 <hemna> DuncanT-, that authoritative knowledge on what pools he can add were added by himself on the 3PAR.
16:40:28 <DuncanT-> hemna: That's an erm, weak, design
16:40:30 <hemna> the admin is the person who created the pools to begin with.
16:40:42 <hemna> pools don't magically get created on our array.
16:40:42 <DuncanT-> Just report them all
16:40:47 <hemna> no
16:40:52 <jgriffith> I don't understand
16:40:55 <hemna> we don't want volumes getting created on all pools on the array.
16:41:00 <winston-d_> hemna: if admin accidentally put the wrong pool name into 3par related type, in your design, your driver will keep complaining 'couldn't find the target pool to report'?
16:41:15 <hemna> winston-d_, it's not the wrong type.
16:41:16 <DuncanT-> The admin created the pools, he knows which ones he does or doesn't want types to be able to access, so what is the problem?
16:41:22 <hemna> he created a type with a correct pool name
16:41:33 <hemna> DuncanT-, correct
16:41:38 <hemna> like I explained earlier
16:41:48 <hemna> he set Pools A-C in cinder.conf
16:41:51 <winston-d_> hemna: i am thinking about corner cases, type with wrong pool name
16:41:56 <hemna> then later created a type that contains Pool D.
16:42:03 <DuncanT-> hemna: Report them all and use the type definition to select the one(s) that are desired... that's the point
16:42:11 <DuncanT-> hemna: No need to selectively report
16:42:14 <markstur> if conf says pools a,b,c and type says pool d -- even if we report stats for all 4 pools, we still need the scheduler to pick pool d for the type and a|b|c if not the type
16:42:34 <hemna> markstur, correct, so the host entry is acurate
16:42:41 <winston-d_> markstur: scheduler is already able to do that
16:42:59 <winston-d_> markstur: as long as your driver reports pool d
16:43:01 <markstur> scheduler picks pool a when type says it needs pool d
16:43:06 <jgriffith> hemna: not sure what the problem is... stats are stats and should be reported
16:43:12 <hemna> winston-d_, but we don't report pool d, because it only lives in the type.
16:43:16 <hemna> it's not in cinder.conf
16:43:21 <jgriffith> types are unknown at init and periodic reporting intervals
16:43:38 <jgriffith> hemna: maybe you should write up and diagram in an etherpad
16:43:41 <jgriffith> and share with everyone
16:43:44 <hemna> ok
16:43:49 <hemna> there is a big disconnect here
16:43:50 <jgriffith> since we can't seem to figure out exactly what you're getting at
16:43:56 <hemna> and it may very well be my own fault.
16:44:00 <winston-d_> hemna: that's the issue, i think you should report pool d, regardless what is in cinder.conf.
16:44:08 <jgriffith> hemna: yeah... communication is hard on IRC sometimes
16:44:13 <DuncanT-> hemna: It exists on the array, report it. Stop hiding it until there's a type pointing at it.
16:44:19 * jgriffith likes white boards
16:44:29 <hemna> winston-d_, so say we report every pool on the array
16:44:32 <winston-d_> jgriffith: +1
16:44:48 <jgriffith> DuncanT-: I'd have to agree with you there
16:44:56 <hemna> but our cinder.conf only lists a subset of those.  how does the scheduler know not to pick pools outside of our desired list ?
16:45:06 <jgriffith> DuncanT-: or at least have specific "OpenStack" related things you always report
16:45:09 <jgriffith> no games
16:45:14 <hemna> we aren't trying to hide anything
16:45:30 <hemna> *sigh*
16:45:37 <hemna> it's not about hiding things or games
16:45:49 <jgriffith> hemna: then only report those spec'd in cinder.conf
16:45:53 <jgriffith> and if you don't work, you don't work
16:45:54 <DuncanT-> hemna: Can this wait until Tuesday and a whiteboard
16:45:58 <hemna> it about having a selected list of pools we want to use for the driver.   we dont' want to use every pool on the array.
16:46:01 <jgriffith> DuncanT-: good idea
16:46:02 <winston-d_> hemna: if you want to use cinder.conf to enforce a list of supported pools, fine, don't report pool d, and scheduler won't put any volume to pool d
16:46:11 <jgriffith> winston-d_: +1
16:46:18 <hemna> winston-d_, ok and when pool D is in a volume type ?
16:46:19 <jgriffith> winston-d_: I'm so confused... I don't see what the issue is here
16:46:39 <jgriffith> hemna: the scheduler won't send it to your backend if you don't report it in stats
16:46:47 <DuncanT-> hemna: having pools magically appear once a type references them is not a sensible design
16:46:53 <hemna> ok, that's definitely a problem then.
16:46:58 <winston-d_> hemna: any volume created with that type will be in 'error' state 'cos 'No Valid Host can be found'.
16:47:05 <jgriffith> hemna: then fix it
16:47:23 <hemna> ok lets talk about this next week
16:47:26 <hemna> I'm not getting through.
16:47:28 <bswartz> perhaps we will get some good examples of how to do pools after the funtionality is implemented in several drivers
16:47:37 <winston-d_> hemna: if you want to use pool d, put it into cinder.conf and restart c-vol, done
16:47:40 <bswartz> and the confusion will go away
16:47:46 <bswartz> if any real issues remain, we can fix them
16:47:58 <winston-d_> bswartz: i'm looking at you
16:48:03 <hemna> then that completely negates the idea of volume types for us.
16:48:14 <winston-d_> bswartz: when will netapp driver be ready for pool scheduler? :)
16:48:14 <hemna> we've always used volume types to specify the pool.
16:48:15 <markstur> winston-d: scheduler would still pick 'd' for other types and sometimes pick a|b|c for the type that requires 'd'
16:48:27 <bswartz> winston-d_: very soon
16:48:33 <winston-d_> hemna: you still do, don't you?
16:48:42 <hemna> yah
16:49:12 <markstur> Now if our pool-aware driver uses the type's pool, then it won't match the one the scheduler said to use
16:49:22 <winston-d_> markstur: not, scheduler will not, it doesn't know about pool d, simply 'cos your driver deson't report pool d
16:49:24 <jgriffith> note
16:49:30 <jgriffith> drivers in Cinder are not a static thing
16:49:41 <jgriffith> they need maintenance and they change over time
16:49:55 <hemna> sure.
16:50:00 <jgriffith> we've done this in the past for MBE, QoS etc
16:50:01 <markstur> so scheduler picks 'b' which doesn't apply to the type.  how do we get it into the type's pool?
16:50:32 <DuncanT-> markstur: A type is not the right place to announce the existance of a pool
16:50:44 <DuncanT-> markstu: Taht is the tail wagging the dog
16:50:45 <jgriffith> let's wrap this up for today
16:51:01 <winston-d_> markstur: if the type specifically says, 'i need pool d', scheduler won't put any volume with this type to pool a|b|c.
16:51:02 <jgriffith> maybe hemna can explain next week and we can help fix it :)
16:51:23 <markstur> DuncanT-, OK the pool should be in the stats, but the scheduler needs to know which volume types are allowed to use which pools
16:51:26 <hemna> DuncanT-, that's doesn't work for 3PAR.   we use types to specify a pool, because that pool has certain characteristics the admins want to charge for, or the user wants because of their application.
16:51:32 <winston-d_> DuncanT-: +1 for type is not the right place to anounce pools.
16:51:40 <hemna> ok guys.  sorry to suck up the meeting time.
16:51:46 <hemna> moving on.
16:51:53 <markstur> someone had to fill in the meeting time
16:51:58 <jungleboyj> :-)
16:52:05 <DuncanT-> Nothing else on the agenda, or I'd have called time a while ago, don't worry
16:52:08 <winston-d_> hemna, markstur: let's take it to cinder channel
16:52:15 <hemna> coolio
16:52:22 <markstur> yep
16:52:34 <DuncanT-> Any other business? Speak now or forever....
16:52:44 <jgriffith> peace
16:52:57 <DuncanT-> #endmeeting