21:00:22 #startmeeting swift 21:00:25 Meeting started Wed Jun 29 21:00:22 2016 UTC and is due to finish in 60 minutes. The chair is notmyname. Information about MeetBot at http://wiki.debian.org/MeetBot. 21:00:26 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 21:00:28 The meeting name has been set to 'swift' 21:00:34 who's here for the swift team meeting? 21:00:36 o/ 21:00:37 o/ 21:00:38 o/ 21:00:40 o/ 21:00:41 Hi 21:00:42 hi 21:00:43 o/ 21:00:44 o/ 21:00:45 o/ 21:00:45 o/ 21:00:47 o/ 21:00:48 o/ 21:00:54 o/ 21:00:58 o/ 21:01:02 helo 21:01:04 hi 21:01:43 acoles_: torgomatic: ping 21:02:15 I'm guessing acoles_ might me getting back online shortly. might need to give him a few minutes :-) 21:02:32 but before we get to the crypto stuff... 21:02:38 welcome, everyone 21:02:41 #topic hackathon 21:02:54 ohai 21:02:58 the hackathon in san antonio is coming quickly. just a week and a half from now 21:03:14 \o/ 21:03:16 today I set up an etherpad to start collecting topics 21:03:18 #link https://etherpad.openstack.org/p/swift_SA_hackathon_2016 21:03:26 so feel free to add your ideas to that 21:03:49 also, if you haven't booked your hotel yet, DO IT NOW! the room block is about to expire 21:04:07 hurricanerix: (are you here?) anything to add about the hackathon? 21:04:14 any questions from anyone about the hackathon? 21:04:53 Do we have to get a car to get to RackSpace? 21:04:54 doesn't look like it 21:05:00 Or shuttle is provided? 21:05:17 I'm told there is a bus from that hotel to the RAX office 21:05:24 nice :) 21:05:29 dmorita: there is a shuttle if you stay at the aloft 21:05:42 it's not quite like acoles_ had for us in bristol, I think 21:05:45 mattoliverau: what's it like? 21:05:52 I mean, will it even seat all of us? 21:06:09 Do they have Uber out there? 21:06:23 bkeller`: yes 21:06:24 does it run multiple times? 21:06:55 yeah, it runs on the hour, you just have to ask at the front desk, or call to get picked up 21:07:34 its a minibus, so probably seats 20 or so. 21:07:44 ok 21:07:45 maybe less, I've never counted 21:08:11 if there are more they might run a little more frequent 21:08:19 I'm goign to plan on using the shuttle and uber while I'm there and not get a rental car, FWIW 21:08:30 +1 21:08:38 ok :) 21:08:48 if all else fails you can get a shuttle to the airport and hire a car 21:08:53 I'll have a car - cause i am drivign from Austin. so i can take a few passengers 21:09:03 the hotel is *very* close to the airport 21:10:07 I'm looking forward to it. coming right off of finishing crypto, it will be a great launching point, of sorts, for the next things :-) 21:10:20 well, that, and I just really like seeing everyone at hackathons :-) 21:10:35 the folks working on crypto deserve an uberchopper ride 21:10:40 any other questions (that I'll punt to mattoliverau and pdardeau to answer)? 21:10:58 briancline: :-) 21:11:00 briancline: lol 21:11:29 ok, let's move on to crypto 21:11:33 #topic crypto 21:11:51 acoles_: sent me a message saying he's having some network problems and will be on ASAP 21:12:03 #link http://not.mn/swift/swift_community_dashboard.html 21:12:07 that page has 2 links on it 21:12:17 the 2nd is a nice dashboard that timburke made 21:12:27 IMO, crypto seems *really* close to landing 21:12:30 acoles: hi! 21:12:51 acoles: just started the crypto topic. you didn't miss any of that 21:12:52 hi sorry, got myself tethered 21:13:22 acoles: did those people in bussels cut your internet connection already? 21:13:29 lol 21:13:31 seems that way ;) 21:13:48 acoles: can you give a summary of where we are with crypto-review? 21:14:17 I think we are in good shape (although I have not checked for -ve reviews in last hour) 21:14:34 Since last week we have addressed the concerns over etag encryption 21:14:47 and the concerns over constraints checking 21:15:23 and the final "last minute" change to have just 2 middleware filters (keymaster + encryption) looks pretty simple imho 21:15:33 and that will make config life much easier ^^ 21:16:01 So first four patches have +2s , then we have patch 328208 21:16:12 hello patchbot? 21:16:27 patch 328208 21:16:27 notmyname: https://review.openstack.org/#/c/328208/ - swift (feature/crypto-review) - Enable object body and metadata encryption 21:16:31 yay 21:16:41 that is the meat of the encryption feature ^^ 21:16:48 nice! 21:16:48 for the etag, we're now storing the hmac of the etag and using that for comparisons in conditional requests. it's a clever update that means we simplify some of the crypto code (no more derived IVs) and still support the full API 21:17:31 hmac was a great idea acoles. glad we have a better solution there now 21:17:36 for the constraints checking, we're now storing user metadata keys encrypted in the [transient] sysmeta space. this means we don't have to worry as much about constaints being violated or overwriting the user namespace 21:18:07 and the unification of decrypter and encrypter should make deployments easier for operators (less pipeline management) 21:18:20 so all eyes on that patch 328208, and I am primarily interested in whether we are making any mistakes in what we write to disk or how we do config i.e. things that are hard to go back and change 21:18:21 acoles: https://review.openstack.org/#/c/328208/ - swift (feature/crypto-review) - Enable object body and metadata encryption 21:18:30 although there's a good question from mathias and response from timburke about it 21:19:36 acoles: right. other stuff can be changed later on master without worrying nearly as much about migrations and upgrades 21:20:00 oic, I will reply to that too, but IMHO the decrypter has to drive when the keymaster returns keys because only the decrypter understand when it is decrypting metadata from a POST vs data from a PUT 21:20:33 acoles: jrichli: although it's not in the docs (which is ok IMO), I'd like to add something to the CHANGELOG about "crypto should probably be enabled for new clusters instead of added to existing clusters". how do you feel about that? 21:20:56 notmyname: +1 21:21:04 (in addition to the lots of other words I need to write int he changelog about crypto) 21:21:11 but really as long as it gets in before the next *release* 21:21:13 it's just words 21:21:50 right. that's what i was looking forward to. for whenever the next release happens. probably won't be right after the crypto branch lands, but I don't expect it to be too long 21:22:01 notmyname: it begs the question "why, does it break existing cluster?". We've worked very hard to not break existing clusters. 21:22:43 yes, i was trying to somehow say that we'd want to communicate that you *could* do it. I am not sure why we want to discourage it, TBH 21:22:58 acoles: true. my reasoning is that adding it to a new cluster is the only way to be sure that all the data is actually encrypted. not that I expect it to break on existing clusters. just trying to give guidance about when to use it for best results 21:23:34 notmyname: so those words sound appropriate "adding it to a new cluster is the only way to be sure that all the data is actually encrypted" 21:23:42 jrichli: ok. so something like "while you can enable this for existing clusters, best practice would be to create a crypto cluster and migrate your data" 21:23:56 notmyname: i like that 21:24:20 fwiw, i think there are two pieces missing that should make operators hesitate to turn it on in existing clusters: first, a process to go around encrypting unencrypted data, and second, some way of monitoring that process's progress 21:24:21 yeah, it's similar to my thinking on "hey, you shoudl encrypt client side instead of trusting swift to do it for you; but if you don't here's this crypto feature we wrote for you" 21:24:26 a copy would encrypt right 21:24:27 ? 21:24:34 notmyname: I'd prefer the recommendation around "whild you *can*" to be a little stronger 21:24:59 I don't know the right words yet. I was just asking about the general idea to find everyone's comfort with that 21:25:08 anyone have an idea how much extra overhead there is for encryption? 21:25:11 I expect it to be a few weeks before I actually have to write the words 21:25:16 pdardeau: non-zero 21:25:16 and if we ever get torgomatic's audit watchers patch landed, it would be useful for that second half, as well as monitoring progress of key-rotation (once *that's* implemented) 21:25:17 tdasilva: yes 21:25:22 I have *no* understanding of a use-case where you only want to turn on encyrpiton for the whole cluster - but only want to encrypt some of your data 21:25:37 heh, I should resurrect that 21:25:44 timburke: it'll be great in the future! 21:26:07 clayg: wouldn't the rma drives count? 21:26:08 last time I submitted it, though, it clogged up zuul pretty well, so I've been really hesitant to touch it 21:26:09 everything is gonna be just grand in the future! 21:26:14 pdardeau: in seriousness, crypto is expected to have some degree of cost, but that hasn't been exactly quantified 21:26:15 clayg: I don't understand what you said 21:26:15 (and yes, it worked on my machine (tm)) 21:26:17 pdardeau: i'm looking at this - it's like *a bunch* - although I don't think it's the encryption as much as all the other stuff to support it (md5 in the proxy, footers, chunked transfer, mime-doc, etc) 21:26:48 pdardeau: can't rma drives that have unencrypted data on them? so "upgrading" is non-sensical for that use-case 21:27:06 tdasilva: ^ 21:27:14 clayg: sure you can, but i thought that was the primary motivation behind adding it 21:27:23 tdasilva: about "upgrading" a cluster to encrypt "everything" is a mis-feature? 21:27:23 pdardeau: clayg: my measurements also showed the extra md5's to cost more than the encryption 21:27:33 or non-feature; depending how you look at it? 21:27:38 acoles: yes :'( 21:28:26 pdardeau: I think encrypting everything as an operator feature is totally a thing - that's why I suggest you steal some capacity out of your existing cluster and plan a migration 21:29:17 right, because if only *some* of your data in encrypted, then you can't RMA any of the drives in the cluster because there might be plaintext data on it 21:29:47 but you might not care about the data that is unencrypted 21:29:56 then RMA anyway ;-) 21:30:22 notmyname: all: so back to the near term (this week) - I need feedback on the change in patch https://review.openstack.org/335641, ideally in next 12 hours so I can squash it in (or not), then barring anything that is broken, I hope we are close to done. 21:31:01 tdasilva: maybe if you have some use-cases that are *not* on your swift because of lack of encyrption, and once you turn it on they can start using it - encyrption of the new data from the old use-cases is just ... for fun? 21:31:02 acoles: right before the meeting I had said in -swift that aside from the few small comments in docs and that patch, I'm pretty happy with it all 21:31:15 notmyname: oic. thanks 21:31:18 clayg: yeah, that's what i was thinking 21:31:41 acoles: I still need to grok the conversation on the unification patch ( timburke's reply). but yeah 21:32:05 ok, so here's a question for everyone, based on what acoles jsut said 21:32:08 tdasilva: ok, but I think that lends better to either standing up an *encrypted* cluster for those use-cases; or moving to encryption on/off per-policy 21:32:29 who here will expect to be able to leave a review on teh crypto branches within the next 12 hours? 21:32:36 * notmyname will 21:32:42 o/ 21:32:56 * mattoliverau will 21:33:08 * cschwede too 21:33:09 o/ 21:33:24 it is only 730am here, so I have all day :) 21:33:39 I just left feedback before the meeting, so probably nothing new from me 21:33:45 mattoliverau: 6:30 for me ;) 21:34:00 * jrichli will review the most recent patch from acoles for sure 21:34:10 ok, great :-) 21:34:14 * mattoliverau is more east then the land of the rising sun.. i still find that crazy 21:34:19 kota_ and mattoliverau are cheating! ;) 21:34:26 acoles: so 12 hours or when all these people leave a review. whichever is first :-) 21:34:29 cschwede: :P 21:34:36 lol 21:35:26 acoles: and I'm expecting that there will likely be just one more patch chain push. then the final one should be able to get all the approvals and we'll do a merge commit. ideally on my friday 21:35:49 ok thanks everyone, so that govesme tomorrow to process comments and then...yeah, that ^^ 21:35:58 gives me* 21:36:06 early next week, those of us in the US will be celebrating the anniversary of the first brexit, so might not be online (I'll be unavailable until my tuesday pm) 21:36:29 lol 21:36:31 brexit v1 21:36:34 so let's get it landed by friday :-) 21:36:34 notmyname: have you been saving that one? 21:36:40 notmyname: "culturally insensitive holiday" ;) 21:36:41 clayg: yes ;-) 21:36:47 ymmd! 21:37:35 anything else from anyone on crypto for this meeting today? 21:38:13 i'm freakishly dissappointed at my inability to find *BUGS* 21:38:26 clayg: PM me and I'll share a few 21:38:27 clayg: have you met acoles? ;-) 21:38:28 I keep beating on it with all kinds of crazy chaos crap and and can't get any tracebacks 21:38:29 lol, first brexit. before it was cool 21:38:29 its acoles's fault! 21:39:01 for expectations-setting, after crypto lands, I want to get some of the other bugfixes landed and some things waiting like the container sync updates. then after that do a release 21:39:07 I'm just the editor, I just type what timburke dictates ;) 21:39:19 the closest I got was filling up my drives and getting some currpt metadata that seems to makes the objects unreadable; haven't quite sussed that one out - but christ - the drives were out of bytes 21:39:31 lol. timburke rulz 21:39:33 all hail timburke ! 21:39:44 clayg clearly should have gotten involved earlier :P 21:39:51 notmyname: did the api docs land yet? 21:39:54 no - this is perfect 21:40:13 acoles: no. that's another thing. I'll look at those later this afternoon, in fact 21:40:27 ok, that's all that's on the agenda for this week 21:40:39 thank you, everyone, for reviewing the crypto work 21:40:55 and thanks expecially to acoles for managing it and timburke and jrichli for pouring so much work into it 21:41:05 #endmeeting