16:00:09 <jungleboyj> #startmeeting Cinder 16:00:10 <openstack> Meeting started Wed Apr 5 16:00:09 2017 UTC and is due to finish in 60 minutes. The chair is jungleboyj. Information about MeetBot at http://wiki.debian.org/MeetBot. 16:00:12 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 16:00:14 <openstack> The meeting name has been set to 'cinder' 16:00:29 <xyang> hi 16:00:29 <Swanson> Hello 16:00:32 <jungleboyj> Courtesy ping: dulek duncant eharney geguileo winston-d e0ne jungleboyj jgriffith thingee smcginnis hemna xyang1 tbarron scottda erlon rhedlind jbernard _alastor_ bluex karthikp_ patrickeast dongwenjuan JaniceLee cFouts Thelo vivekd adrianofr mtanino yuriy_n17 karlamrhein diablo_rojo jay.xu jgregor baumann rajinir wilson-l reduxio wanghao thrawn01 16:00:32 <jungleboyj> chris_morrell watanabe.isao,tommylikehu mdovgal ildikov wxy viks ketonne abishop sivn breitz 16:00:39 <mgagne> hi 16:00:40 <rajinir> o/ 16:00:40 <geguileo> hi! 16:00:41 <wxy|> hi 16:00:43 <jgriffith> o/ 16:00:44 <karthikp> hi 16:00:45 <hemna> Chewbacca 16:00:49 <patrickeast> Hi 16:01:01 <jungleboyj> Will give everyone a minute to join. Sean is is Tokyo so I am manning the ship today. 16:01:14 <Apoorva> hi 16:01:14 <hemna> lord help us 16:01:25 <hemna> :P 16:01:25 <jungleboyj> hemna: Thanks for the vote of confidence. :-p 16:01:27 <bluex> hi 16:01:35 <tbarron> hi 16:01:48 <jungleboyj> tbarron: Wow! Thanks for joining. 16:02:07 <tbarron> jungleboyj: I usually show up; don't feel too special. 16:02:13 <DuncanT> Hi 16:02:18 <jungleboyj> tbarron: Awww. Ok. 16:02:36 <jungleboyj> Ok, we have a good showing. Lets get started. 16:02:47 <jungleboyj> #topic Announcements: 16:03:11 <jungleboyj> #link https://etherpad.openstack.org/p/cinder-spec-review-tracking 16:03:23 <viks> o/ 16:03:34 <jungleboyj> Weekly reminder of review tracking priorities for Pike. Please take a look at those and keep up with reviews. 16:03:47 <jungleboyj> Also, tomorrow is the cut-off for stable/newton patches. 16:03:57 <jungleboyj> I went through a bunch of things that were out there yesterday. 16:03:59 <abishop> o/ 16:04:02 <jungleboyj> I will do another round today. 16:04:15 <eharney> i just posted https://review.openstack.org/#/c/453697/ for Newton, seems like a good thing to get in 16:04:18 <jungleboyj> If there is anything I am missing please let me know and I will get to it todya or tomorrow. 16:04:35 <jungleboyj> eharney: Tab opened. 16:04:54 <jungleboyj> Any other ones that need immediate attention? 16:05:16 <eharney> i've been eyeballing the list, it's mostly driver fixes 16:05:17 <jungleboyj> Ok, if others come up ping me in IRC please. 16:05:28 <xyang> jungleboyj: we have a few newton backports that need a 2nd +2 16:05:42 <xyang> https://review.openstack.org/#/c/452234/ 16:05:44 <jungleboyj> eharney: I have been pretty relaxed about letting those in. Figure if the maintainers want them...Ok. 16:05:55 <xyang> https://review.openstack.org/#/c/453163/ 16:06:03 <jungleboyj> eharney: ^^ 16:06:08 <xyang> https://review.openstack.org/#/c/451813/ 16:06:25 <xyang> stable cores, please help! 16:06:48 <jungleboyj> jgriffith: If you and/or eharney could look at those it would be appreciated. 16:06:56 <jgriffith> jungleboyj sure 16:07:02 <jungleboyj> jgriffith: Thank you! 16:07:13 <xyang> thanks! 16:07:17 <eharney> we can figure this out in #openstack-cinder later, but i dunno why "add replication" is a backport... 16:07:34 <jungleboyj> eharney: I will fill you in on that one. Look closer. ;-) 16:07:40 <xyang> eharney: I can explain to you later, just a small portion of that 16:07:43 <jungleboyj> A bug fix that went in with that change. 16:07:55 <jungleboyj> I had them same heartburn when I saw it the other day. 16:08:05 <jungleboyj> Ok ... Anything else on this topic? 16:08:27 <jungleboyj> Ok, moving on. 16:08:38 <jungleboyj> #topic Updates from Oslo Meeting: 16:08:57 <jungleboyj> #link https://review.openstack.org/328692 16:09:07 <jungleboyj> The review above may break the gate for Cinder. 16:09:43 <jungleboyj> The change set the method method set_override's parameter enforce_type=True by default, this enforces developer to write valid test cases about config options. 16:10:08 <jungleboyj> When I get back next week I want to try to pull the change down and see if it breaks us. 16:10:22 <jungleboyj> Wanted to make everyone aware of the change if there were any immediate concerns. 16:10:34 <eharney> it won't be allowed by upper-constraints requirements bounds until it passes w/ our unit tests, right? 16:10:50 <jungleboyj> eharney: I believe that is true. 16:11:21 <jungleboyj> I am not sure how much work will be involved to fix things up if it does break us. I think it is mostly busy work that I can tackle. 16:11:43 <jungleboyj> None-the-less, wanted to share the fact that that change is coming. 16:12:18 <jungleboyj> Also anohter note from the Oslo team. 16:12:33 <jungleboyj> With all the discussion and proposals around messaging ... 16:13:17 <jungleboyj> I wanted to make note that Ken Giusti is going to be doing a presentation on the different messaging backends and how to set up systems to run with multiple messaging queues. 16:13:34 <jungleboyj> May want to look for that in Boston. Sounds interesting. 16:13:58 <jungleboyj> So, that was what I had there. 16:14:21 <jungleboyj> #topic Extend Volume Discussion: 16:14:29 <jungleboyj> Now things get interesting. :-) 16:14:43 <jungleboyj> tommylikehu_: Has brought this functionality back up. 16:14:44 <mgagne> =) 16:14:53 <tommylikehu> :) 16:15:03 <jungleboyj> I am glad as it kind-of fell off the radar with other things going on in the community. 16:15:18 <jungleboyj> So, tommylikehu brought it up and mgagne has been helping out. 16:15:38 <hemna> mgagne, have you tested out the os-brick extend_volume stuff yet ? 16:15:40 <jungleboyj> mgagne: Shared with following solution with mriedem : 16:15:47 <jungleboyj> #link https://gist.github.com/mgagne/9402089c11f8c80f6d6cd49f3db76512 16:16:05 <jungleboyj> mriedem: Was willing to consider the solution. 16:16:10 <hemna> I'd like to verify that it still works :) 16:16:11 <mgagne> hemna: please take a look at the proposed implementation and see if it's considered "testing" os-brick. 16:16:37 <jgriffith> BTW. https://blueprints.launchpad.net/nova/+spec/online-volume-extend-extension 16:16:43 <jungleboyj> Updated spec for cinder: https://review.openstack.org/#/c/453286/ 16:16:52 <hemna> mgagne, can you just put up a cinder WIP ? 16:16:59 <jungleboyj> Updated spec for Nova: https://review.openstack.org/#/c/453272/ 16:17:00 <hemna> then I can pull that and play with it 16:17:01 <jgriffith> Some Trove folks had a patch up for this at one point as well (somewhere) 16:17:10 <eharney> does the proposal from mgagne mostly line up with our discussions on this, or is it going a different direction? 16:17:30 <mgagne> hemna: sure, will do that by tomorrow 16:17:35 <hemna> ok coolio 16:17:45 <jungleboyj> I think the people with the strongest feelings here are jgriffith and hemna 16:18:06 <jungleboyj> mgagne: That would be great. I would like to play with it as well. 16:18:29 <jgriffith> the problem from my perspective has never been the Cinder side 16:18:35 <hemna> I had hacked the cinder API to allow this while I worked on the os-brick piece a year or so ago when I wrote the brick code. 16:18:47 <jgriffith> Nova has rejected the proposals to do the rescan in the past 16:19:08 <jungleboyj> jgriffith: Agreed. 16:19:18 <hemna> as of yesterday, I think mriedem was happy with cinder doing a call into Nova to initiate the workflow on the nova side after cinder had resized the volume. 16:19:19 <jgriffith> and there was some debate around how it should work in terms of notifications to Nova 16:19:22 <mgagne> jgriffith: the current proposal suggests reusing existing external-events API endpoint instead of introducing a new API call 16:19:25 <jungleboyj> mriedem: Seems to be ok with making changes if they are triggered from Cinder. 16:19:35 <jgriffith> hemna and that was/is the problem :) 16:19:43 <hemna> yah, as was mine too. 16:19:54 <jgriffith> hemna how do you "call nova" from Cinder? 16:20:12 <mriedem> cinder already calls nova today 16:20:15 <mriedem> for migration and retype 16:20:15 <hemna> Cinder does that now with swap volume 16:20:16 <eharney> the same way we call it for other things now? 16:20:18 <jungleboyj> jgriffith: Isn't there a call back service. 16:20:25 <jungleboyj> There is the guy I was looking for. 16:20:36 <mriedem> nova doesn't want more apis like that, we have the external server events api that nova<>neutron use 16:20:50 <mriedem> if we're doing new things like that between nova<>cinder we use the same api with new event types 16:20:50 <jgriffith> mriedem +1 16:21:38 <mgagne> ok, will post code to cinder so people can play with it and gather comments/feedback 16:21:43 <jgriffith> eharney IIRC the only thing doing any interaction today is the instance-assisted snapshot stuff you worked on no? 16:22:04 <jgriffith> suppose I could look for myself :) 16:22:04 <mriedem> jgriffith: and swap volume 16:22:11 <mriedem> but yes those are the two 16:22:15 <jungleboyj> #action mgagne To post WIP code for Cinder team to download and try. 16:22:19 <mriedem> retype/migration and guest-assisted snapshot 16:22:28 <jgriffith> so maybe step 1: convert those to the new event system 16:22:36 <jgriffith> step2: propose the extend call? 16:22:58 <jgriffith> I don't want to have another case of *2 ways to do it in Cinder* that never gets cleaned up 16:22:59 <mriedem> i wouldn't waste time on converting snapshot/swap volume to the event api 16:23:13 <mriedem> those were never even tested in the gate until probably ocata 16:23:23 <mriedem> guest-assisted snapshot still isn't b/c it's fs-style backends only 16:23:25 <jgriffith> mriedem alrighty then 16:23:29 <hemna> mgagne, added a comment on your gist....FYI 16:23:41 <mgagne> thanks, will take a look 16:23:42 <eharney> we have an NFS job testing assisted snaps now 16:23:48 <jungleboyj> mriedem: Yeah, lets work on getting this to go forward first. 16:23:59 <mriedem> eharney: yeah, but that was ocata 16:24:07 <jgriffith> mriedem but I am going to voice my concern about adding yet another divergent path in Cinder on how to do things 16:24:11 <mriedem> but yeah we have that coverage now somewhere at least 16:24:19 <jgriffith> mriedem voiced.. and done. Moving on 16:24:46 <jungleboyj> jgriffith: Sounds like it would be good to go back and fix that in the future. 16:24:51 <jungleboyj> Maybe get a bug open for that. 16:25:03 <jgriffith> jungleboyj if I had a dollar for every time we said that and *didn't* do it :) 16:25:09 <jungleboyj> Agreed that we don't want to continue to diverge. 16:25:53 <jungleboyj> #action jungleboyj To open a bug for moving the snapshot/swap API to using events 16:25:57 <jungleboyj> jgriffith: :-) 16:26:07 <jungleboyj> It is an action. Has to happen now. 16:26:09 <mriedem> note, 16:26:17 <mriedem> swap is two callback apis :) 16:26:21 <mriedem> cinder -> nova -> cinder 16:26:23 <mriedem> it's the best 16:26:38 <mriedem> and if your policy doesn't line up, well, sucks to be you 16:27:17 <jungleboyj> mriedem: Noted. 16:28:08 <jungleboyj> So, I think we have visibility to the extend volume issue. We need to get everyone to look at the code that mgagne shares and we need reviews of the specs I linked above. 16:28:11 <jgriffith> mriedem not sure I follow what you mean? 16:28:50 <jgriffith> mriedem are you being sarcastic, or pointing out that without changing it on Nova the we're shit out of luck? 16:28:53 <jgriffith> mriedem or both? 16:28:54 <jgriffith> :) 16:29:33 <eharney> i think policy.json 16:29:49 <jungleboyj> #action Try mgagne 's patch, review extend volume specs . 16:30:23 <jungleboyj> mriedem: Has disappeared behind the great firewall of China. 16:30:26 <mriedem> jgriffith: retype 16:30:27 <mriedem> in cinder, 16:30:33 <mriedem> calls swap_volume in nova, 16:30:38 <mriedem> and nova calls back to cinder to tell cinder that the swap is done 16:30:41 <mriedem> and complete the migration 16:30:46 <mriedem> i'm being serious 16:30:52 <mriedem> there was a bug in nova at one point, 16:31:03 <jgriffith> mriedem yeah, I know about that sadly :( 16:31:18 <mriedem> where the nova and cinder policy files didn't line up such that a non-admin user initiating a swap volume in the comptue api (w/o cinder) would fail b/c of the callback to cinder, which is admin-only by default 16:31:29 <mriedem> swap volume used to be admin or owner in nova 16:31:31 <mriedem> it's admin-only now 16:31:32 <mriedem> by default 16:31:47 <mriedem> i had to dig through the history to figure out how swap volume even came about 16:31:51 <jgriffith> mriedem I just wasn't sure as you stated you didn't think it was worth changing if that meant you didn't want it changed in Nova so if our *policy* was consistency and didn't align you were saying "suck to be you" 16:32:01 <jgriffith> mriedem vish and avishay IIRC 16:32:17 <jungleboyj> mriedem: So that was a bug that has been fixed unless the end user changes the policy? 16:32:18 <mriedem> the good old days of wild west vendor specific apis 16:32:21 <mriedem> yeeehaWWW!!! 16:32:29 <mriedem> jungleboyj: the end user doesn't change the policy 16:32:34 <mriedem> the operator does 16:32:38 <jgriffith> mriedem it was a *thing* with IBM 16:32:53 <mriedem> vish wasn't ibm 16:32:56 <mriedem> anyway 16:33:00 <jungleboyj> mriedem: Sorry, operator. or packager. 16:33:05 <jgriffith> mriedem no, but avishay who owned/drove it was :) 16:33:13 <mriedem> and guest assisted snapshots was gluster 16:33:13 <mriedem> and red hat 16:33:16 <jgriffith> mriedem but as you say... "anyway" 16:33:16 <mriedem> but we digress 16:33:30 <jgriffith> mriedem that was actually eharney and russellb 16:33:33 <mriedem> i know 16:33:34 <mriedem> red hat 16:33:36 <mriedem> glusterf 16:33:57 <jungleboyj> So, does anyone have more on this topic or are we good with the action items going forward? 16:34:38 <Swanson> Was floating into the light so I think I'm good. 16:34:57 <jungleboyj> Ok, so we shall continue. 16:35:22 <jungleboyj> Thank you mgagne for helping with that code and to everyone else who is contributing to that effort. 16:35:45 <mgagne> =) 16:35:46 <jungleboyj> #topic Encrypted volumes and Glance: 16:35:52 <jungleboyj> eharney: You are up. 16:36:11 <jungleboyj> #link https://review.openstack.org/#/c/453342/ 16:36:18 <eharney> sure, just wanted to see if anyone was interested in this work, i'll give some background... 16:36:21 <jungleboyj> #link https://review.openstack.org/#/c/453343/ 16:36:32 <jungleboyj> Go for it man! 16:36:34 <eharney> upload-to-image is pretty good, volume encryption is pretty good, the combination of the two is pretty bad 16:37:18 <eharney> and fortunately or unfortunately this used to "work", long ago, so people upgrade expecting this to be a functional workflow, and are disappointed 16:37:34 <jgriffith> eharney oh, that's a bit horrid 16:38:10 <jungleboyj> eharney: Yuck. 16:38:28 <eharney> it would have needed rework anyway to work with multiple encryption keys and barbican, as opposed to our conf key manager that has one key 16:38:55 <eharney> so these patches attach a key id to images in glance when we upload them, and use that to find the encryption key when we download the image to a volume 16:39:09 <eharney> there are a handful of other bugs and issues in there (see the dependent patches), but that's the main gist of it 16:39:39 <eharney> without this, attempting that flow will get you either a) a doubly-encrypted volume that you can't read, or b) failure to create the volume from image 16:40:06 <eharney> the patches are at the "works in devstack" phase now, i'll be polishing this up a lot and moving forward, unless anyone has big objections 16:40:16 <jungleboyj> eharney: Small patches, that is good. 16:40:24 <jungleboyj> Sounds like we need to get that fixed in pike though. 16:40:38 <jgriffith> eharney have to say nice work disecting that 16:40:48 <jungleboyj> jgriffith: ++ 16:41:01 <eharney> i have to test a lot of combinations of things, but the code isn't bad so far 16:41:10 <jungleboyj> eharney: Sounds like we should get those patches on the review priorities list? 16:41:16 <eharney> also, would like to give thanks to LisaLi who started down this road a while ago 16:41:20 <eharney> jungleboyj: yeah, probably so 16:41:44 <eharney> they are loosely attached to the blueprint: https://blueprints.launchpad.net/cinder/+spec/improve-encrypted-volume 16:41:56 <jungleboyj> #action Get patches to fix Encrypted volumes and Glance on the review priority list. 16:42:02 <eharney> this blueprint is basically "fix everything related to encryption" so i'd like to finish it out and not keep adding things to it, but this is there for now 16:42:27 <eharney> that's about it from me 16:43:41 <jungleboyj> eharney: Thank you for the efforts there and for bringing it up. I will get the patches on the review list and make sure we follow up on that. 16:43:45 <Swanson> nice work 16:44:04 <jungleboyj> Wow Swanson is impressed. *gasping* ;-) 16:44:09 <eharney> :) 16:44:21 <Swanson> jungleboyj, it happens! 16:45:03 <jungleboyj> :-) 16:45:15 <jungleboyj> Ok. So we have the action items there. 16:45:27 <jungleboyj> #topic Open Discussion 16:45:46 <jungleboyj> Anything else we need to discuss today or should we wrap this baby up? 16:46:01 <karthikp> Guys...I needed some feedback on the generate driver matrix 16:46:06 <karthikp> https://review.openstack.org/#/c/371169/ 16:47:06 <jungleboyj> karthikp: Ok, have a tab open for that and will take a look. 16:47:57 <karthikp> jungleboyj: Thanks Appreciate that! 16:48:07 <bluex> It would be nice to see some feedback on blueprint proposing HA races fixes - https://review.openstack.org/#/c/429947/ 16:49:03 <_alastor_> Does anyone know the simplest way to get a project name in a driver if you have the project id? 16:49:29 <bluex> Also my life would be much easier if we got https://review.openstack.org/#/c/417257/ (quota module split) merged or abandoned 16:49:41 <jungleboyj> bluex: Ok, Spec cutoff is coming up. I am hoping to get some time to get through Spec reviews. Will coordinate with smcginnis_vacati When he is back. I know he is pretty busy over the next few weeks but I am going to try to meet up with him to work through things. 16:49:47 <jungleboyj> End of next week. 16:50:01 <bluex> awesome, thanks 16:50:35 <jungleboyj> bluex: Welcome. 16:50:46 <jungleboyj> _alastor_: Not sure. Anyone have feedback there? 16:50:55 <xyang> jungleboyj: I thought we have until Pike-2 for spec merge? 16:51:26 <_alastor_> jungleboyj: It's not a huge deal if it's not possible. I'd just like to store name on the backend for easy human-readable mapping 16:51:32 <xyang> jungleboyj: https://releases.openstack.org/pike/schedule.html#cinder-spec-freeze 16:51:37 <xyang> june 7th 16:51:37 <jungleboyj> xyang: That will be here before we know it. ;-) 16:51:46 <xyang> that's true:) 16:52:11 <jungleboyj> xyang: With the Forum in there and everything. That was more beating on myself to start reading specs which I haven't been doing so well at lately. ;-) 16:52:39 <xyang> jungleboy: I need to review specs too:) 16:54:06 <jungleboyj> Also, a reminder that we don't yet have a docs liaison. 16:54:26 <jungleboyj> I could take that on if no one else is interested ... but, if someone would be interested ... 16:55:06 <jungleboyj> Anyone have anything else for today? 16:55:14 <xyang> Also Pike-1 is around the corner. Please submit a patch to add CG capability to generic volume groups if you have not done so 16:55:24 <jungleboyj> xyang: ++ 16:55:38 <jungleboyj> Do we have to have those merge before Pike-1? 16:55:49 <xyang> no, just submit a patch by Pike-1 16:56:03 <hemna> so what happens if folks don't ? 16:56:08 <jungleboyj> xyang: Ok, good. Need to work on those reviews too. 16:56:09 <xyang> should be merged by Pike-3 16:56:09 <Swanson> https://review.openstack.org/#/c/443364/ 16:56:18 <Swanson> Oh, never mind that link, then. 16:56:25 <xyang> hemna: we'll open a bug 16:56:27 <hemna> and if drivers don't update their code for the generic groups in PIke ? 16:56:46 <jungleboyj> They get a generic warning. ;-) 16:56:54 <xyang> hemna: At PTG, we decided to submit a bug for that driver 16:57:01 <hemna> ok that's it ? 16:57:10 <hemna> just curious 16:57:16 <xyang> hemna: yes, that was the decision at PTG 16:57:20 <hemna> ok coolio. 16:57:34 <hemna> does the driver stop functioning ? 16:57:40 <hemna> wrt CG's ? 16:57:49 <xyang> hemna: In Queens though, I want to remove the conversion part, so if they didn't convert, they lose the CG function 16:58:10 <jungleboyj> xyang: So the deadline is just encouraging everyone to get the work done. 16:58:14 <xyang> hemna: actually in Pike, it will still work, because there's a conversion from CG to groups 16:58:15 <hemna> ok, so CG will still work in Pike 16:58:25 <xyang> jungleboyj, hemna: yes 16:58:32 <hemna> do we throw a warning in the logs at startup 16:58:36 <hemna> or during CG actions ? 16:58:57 <xyang> hemna: I think we should probably do something like that 16:58:57 <hemna> 'CG has been deprecated, please migrate to generic groups." or something along those lines 16:59:06 <hemna> for every CG call 16:59:09 <xyang> hemna: deprecation of CG API is in Queens 16:59:10 <hemna> $0.02 16:59:16 <hemna> wait 16:59:23 <hemna> you said you wanted to remove CG in queens 16:59:31 <hemna> which means we need to deprecate it in Pike no ? 16:59:42 <xyang> hemna: CG API will still work in Pike. In Queens we deprecate it 16:59:55 <xyang> hemna: and we remove it when we can 16:59:59 <hemna> and remove it later 17:00:00 <hemna> ok 17:00:18 <xyang> hemna: in Queens, I want to remove the piece of code in manager that convert a group object to a cg object 17:00:35 <xyang> once that's removed, driver will lose CG function if they don't add it to groups 17:00:36 <hemna> which would break CG ? 17:00:39 <Swanson> xyang, isn't that effectively removing it? 17:00:47 <hemna> yah, so we need to deprecate it now then 17:00:49 <xyang> Swanson: no 17:01:11 <jungleboyj> It is removal of the conversion. 17:01:11 <Swanson> xyang, That doesn't break CGs? 17:01:18 <xyang> For drivers already added CG capability in Groups, that will not break CG 17:01:24 <Swanson> Oh, wait, forgot about the conversion thingy. 17:01:30 <hemna> but it doesn't create groups 17:01:40 <hemna> which then leaves us with what? 17:01:42 <hemna> I'm confused 17:01:43 <xyang> only if drivers don't care and don't implement groups, they won't have CG function in groups for free any more 17:01:56 <jungleboyj> Yikes. We are out of time. 17:01:59 <hemna> seems like that could lead to upgrade problems 17:02:06 <Swanson> Deprecate now. 17:02:12 <jungleboyj> Lets take this over to openstack-cinder 17:02:13 <xyang> hemna: upgrade is taken care of in Pike already 17:02:14 <hemna> ok 17:02:21 <hemna> *sigh* 17:02:28 <Swanson> jungleboyj, Unusually smoothly run meeting this week. Wonder what has changed? 17:02:28 <xyang> hemna: there will be no entries in CG tables in pike 17:02:31 <jungleboyj> Thanks everyone for a productive meeting! 17:02:37 <xyang> thanks 17:02:49 <jungleboyj> Swanson: Ouch, I won't tell Sean you said that. 17:02:55 <jungleboyj> It was going so well until the end. 17:03:04 <jungleboyj> Anyway, thanks everyone for your work on Cinder. 17:03:13 <jungleboyj> See you all next week. 17:03:18 <jungleboyj> #endmeeting