21:00:15 #startmeeting swift 21:00:15 Meeting started Wed Jan 25 21:00:15 2017 UTC and is due to finish in 60 minutes. The chair is notmyname. Information about MeetBot at http://wiki.debian.org/MeetBot. 21:00:16 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 21:00:19 The meeting name has been set to 'swift' 21:00:23 who's here for the swift team meeting? 21:00:28 o/ 21:00:32 morning and happy Australia day. 21:00:41 o/ 21:00:44 o/ 21:00:44 hi 21:00:44 hi 21:00:47 hi 21:00:52 o/ 21:00:53 mattoliverau: is today a holiday in Oz? 21:00:57 yup 21:01:01 o/ 21:01:01 o/ 21:01:02 everyday is a holiday in Oz 21:01:09 mattoliverau: and you are still at the swift meeting. what devotion :-) 21:01:22 hi 21:01:29 planning on firing up the spit and slow cooking a leg of lamb ;) 21:01:37 nice 21:01:47 welcome, everyone 21:01:49 mattoliverau: just another ordinary day then 21:01:57 acoles: thats right, lol 21:02:07 acoles: he forgot to mention: "overlooking the beach" 21:02:12 agenda for this week is at... 21:02:15 hello 21:02:15 #link https://wiki.openstack.org/wiki/Meetings/Swift 21:02:25 I'll go slightly out of order from what's there 21:02:33 #topic FYI things 21:02:49 tdasilva: I'm not cruel :P 21:02:55 heh 21:03:13 just a few things to be aware of here 21:03:19 * notmyname gets organized with links 21:03:20 * kota_ is almost read only 21:03:34 kota_: no worries. get well soon 21:03:39 #link https://governance.openstack.org/tc/goals/pike/index.html 21:03:50 * clayg feels bad about giving kota my bug :'( 21:03:50 that's the TC-approved goals for Pike 21:04:16 what were our goals for octocat? did we do it!? 21:04:35 the only one for ocata was to stop using deprecated oslo libs 21:04:40 so...done! 21:04:45 good job, team 21:04:53 \o/ 21:04:53 we are so awesome 21:05:10 the first one (make projects use WSGI) I *think* might not have anything for us to do in swift. I think the implementation for us are just to make sure that the proxy server can work under apache/mod_wsgi 21:05:36 notmyname: it states: "Control Plane API endpoints" 21:05:44 swift is not control plane, is it? 21:06:08 however, when I looked at the devstack code, it seemed to indicate that if you give it the "use mod_wsgi" flag, it startes *everything* that way. which won't work for storage services 21:06:26 tdasilva: true, so there's some wiggle room there 21:06:56 so there might be a small change to devstack needed that we may be consulted on (or one of us might need to do) 21:07:14 the second goal listed is "support py35" 21:07:22 this one, of course, is rather bigger 21:07:25 hmm, a change in devstack or in swift? like making swift storage services wsgi compatible? 21:07:33 cschwede: in devstack 21:08:18 this might need some clarification at the PTG 21:08:20 for py35, obviously there's been a lot of discussion before on this, and there's a lot of work to do in swift (including some tricky questions that don't yet have answers) 21:08:32 notmyname: timburke basically already has swift working under py3 - he's just keeping it himself 21:08:46 ssssh! it's a surprise! 21:08:52 rofl! 21:08:54 cschwede: if you look at yesterday's TC meeting log, you can see the end of the discussion on it, including some of my comments 21:09:07 ok no need to worry about py3, timburke's got it covered ;P 21:09:19 mattoliverau: i *never* worry about py3 21:09:21 I do not expect to have py3 done during the Pike openstack time period 21:09:24 lol 21:09:25 really, sounds like dims is making solid progress on it 21:09:34 notmyname: do we get kicked out of openstack!? 21:09:36 notmyname: ok, will read TC log after the mtg 21:10:01 but we'll likely need to show some progress, including identifying tricky or blocking or just not-done issues 21:10:23 cschwede: ok. yesterday i found some part of the devstack code that I can point you to 21:10:48 also, I believe there will be some discussion at the PTG about these two pike goals 21:11:16 ...which brings us to the next topic 21:11:18 #topic PTG 21:11:25 #link https://wiki.openstack.org/wiki/PTG/Pike/Etherpads 21:11:32 that's the list of all known etherpads for the event 21:12:01 the way the event will work is that each of these teams listed on there will have a room, and the team in the room will be in complete control of their own schedule 21:12:43 which means (1) you gotta find where you want to be and hope it doesn't conflict with something else and (2) for the event will looks very very similar to our last summit 21:13:03 our planning etherpad is at 21:13:05 #link https://etherpad.openstack.org/p/swift-ptg-pike 21:13:15 so please add topics and/or details to that 21:13:44 over the next few weeks we'll be able to get a much better idea of what the wednesday-friday times will look like 21:13:55 for us 21:14:34 any questions about the PTG right now? 21:14:44 who's going!? 21:14:46 o/ 21:15:02 I'll be there all week 21:15:04 o/ 21:15:06 o/ 21:15:06 o/ 21:15:07 ... forever alone (with notmyname) 21:15:08 o/ 21:15:09 o/ 21:15:14 oh... or not - good :D 21:15:15 o/ 21:15:21 o/ 21:15:24 o/ 21:15:30 acoles: ? 21:15:38 not sure yet 21:15:40 torgomatic and timur 21:15:41 ok 21:15:55 mattoliverau? 21:16:17 hurricanerix: ? 21:16:19 I'm still figuring out attendance, so far I might only be allowed to come for 3 nights, so I'd be very jetlagged. 56 hours of transit for 74 hours on ground. Also it's so late it'll all be economy. Tho I hope to have a plan in the next day or 2. 21:16:37 looks like the usual suspects! will be great to see y'all there :) 21:16:46 but at this stage.. yeah I hope to be there. 21:17:19 #topic upcoming releases 21:17:42 I need to tag a swiftclient release this week to match the openstack ocata release schedule 21:18:10 I've talked to both timburke and joel about it (joel is sick, too) and they don't have anything major outstanding (ie blocking stuff) 21:18:29 so I will write up the authors/changelog patch and get that in, either today or tomorrow 21:18:43 would not call it blocking, but it would be nice to see https://review.openstack.org/#/c/423377/ land, i believe the server side already landed 21:19:04 tdasilva: ah, nice. yeah if the sever-side landed, that would be great to have in 21:19:13 tdasilva: but you've got a -1 on it :-) 21:19:20 :/ 21:19:38 cbartz has been very responsive... 21:19:54 for swift, we need to provide a tag for the ocata release during the week of Feb 13 21:20:10 so we've got a few weeks on that (and we'll talk about critical bugs in just a moment) 21:20:34 aside from the swift bugs that could block a release, any questions about these upcoming releases? 21:21:30 ok, then I'll come back to a couple of bugs in just a bit 21:21:34 but first... 21:21:56 jcaron: wanted to bring up a TXT lookup middleware 21:22:01 sure 21:22:02 :) 21:22:02 #link https://wiki.openstack.org/wiki/Swift/ideas/txt_lookup_middleware 21:22:08 let me give you the context 21:22:08 jrichli: what's up? 21:22:22 Few years ago, some customers complained about the cname lookup feature; after few exchanged with them, we understood 21:22:22 that the authoritated name servers were hosted by Cloudflare. they resolve the CNAME(s) on their side without providing the CNAME string (CNAME Flattening). Surely their is other dns providers doing the same. 21:22:22 I believe that using the TXT record looks like a good option to handle this use case. 21:22:22 First of all, what do you guys think about it ? 21:22:47 cname flattening :  https://support.cloudflare.com/hc/en-us/articles/200169056-CNAME-Flattening-RFC-compliant-support-for-CNAME-at-the-root 21:23:13 so we'd look for a well-defined dns TXT record and parse the contents? and use that for account/container resolution? 21:23:19 so the story is "if you use some dns provider you maybe can't use cname middleware as is"? 21:23:21 yep 21:23:40 exactly 21:24:13 I think it sounds like an interesting idea. and I'm assuming you've got customers at OVH who need this? 21:25:01 Actually, we have an implementation, but maybe we can make it more general 21:25:13 jcaron: do you have any idea how broad the support for adding custom TXT entries is on popular dns providers (godaddy, ghandi)? 21:25:40 I don't yet about it 21:25:42 know* 21:25:43 FWIW (anec-data), I've seen that in every DNS provider I've used 21:25:55 But it's the purpose of it, isn't it ? 21:26:03 jcaron: well perhaps as a start we could file the bug against cname middleware in lp - and add the "less general" solution as associated project (reference from bug) 21:26:25 Well you need to use TXT for things in email like SPF, so they all need to support TXT 21:26:34 I gathered that a question jcaron has is if it should be done in the existing cname middleware or in a new one 21:26:34 mattoliverau: gtk! 21:26:53 Yes 21:26:58 notmyname: IMHO that might have something to do with if we need to get cname? 21:27:11 Looks like it's better to have a shared middleware for it 21:27:25 but it breaks the previous versions 21:27:25 notmyname: it sounds like mattoliverau thinks using TXT might just be a better and cover more use-cases 21:27:43 jcaron: what breaks with previous versions? 21:27:52 clayg: I tend to agree with that statement :-) 21:28:15 Not really sue sorry, but like in the conf file for example ? 21:28:23 jcaron: yeah we can't drop the cname support out of the gate - but we can deprecate it - I don't think it's super widely deployed - and it could continue to live as an out of tree project if we decide we want to deprecate and only support TXT 21:28:46 Ok sounds perfect to me 21:28:51 so if we *like* cname - and it solves something you can't solve with TXT - then we will want to keep it forever - and it should all just be one big bal of mud we maintain forever 21:29:04 if we hate cname now and txt is the way to go forever - gah if only we had known!!!! 21:29:19 ... then we can make a new things to replace the old thing and someday my grandchildren can get rid of cname 21:29:45 long-term, my gut reaction is to have one "dns lookup" middleware that can do both 21:29:47 what if we use cname and TXT in different middlewares ? 21:30:14 notmyname : I agree 21:30:18 notmyname: ok, i'm just really not sure if cname covers some use-case that won't be better served by txt 21:30:33 and to get there, we can either start with updating the existing cname middleware or start a new dns_lookup middleware and deprecate the cname one later after the functionality is rolled into dns_lookup 21:30:46 if it's like "look two ways to do the same thing and one way is better we're not sure which" - then... that's less good reason to maintain the old way 21:30:47 we could have an entrypoint for cname_lookup indefinitely, as long as it accepts the old options. we could still rename the thing and/or provide some shim into a more-general solution 21:31:32 I had assumed one of the main use cases of cname lookup was S3 compat 21:31:32 i just really want to know if we have two different use-cases 21:32:03 if it's only one use-case and our implementation sucks because you can't deploy it with some dns providers - there's a lot more questions about what to do 21:32:03 and it makes me wonder if the TXT record idea might be an interesting way to solve some of the "we don't know the swift account" problems with s3 21:32:16 if we have two use-cases and need two solutions it's super stright forward 21:32:24 even if there's a little overlap 21:33:04 clayg: so how would you feel about having a new dns_lookup one that does TXT records, and over time we can merge int eh cname stuff if it looks like it will overlap? 21:33:09 jcaron: can you write up more about your txt implementation - your use case and how your using it? 21:33:17 jcaron: or share your existing code? 21:33:33 jcaron: do you feel like you could get exactly what you wanted from cname except the for the referenced cloudflare issue? 21:33:58 notmyname: yes code always helps - just publish it somewhere on the internet doesn't matter where 21:34:07 The implementation is directly done in the cname_lookup 21:34:22 So it does not make much sense 21:34:28 ah, interesting. so you're running your own patch on it 21:34:29 I think for now a bug report against cname middleware and reference to cloudflare related issue is the best thing 21:34:43 clayg : yes 21:34:44 notmyname: fwiw, i don't think it's an s3 necessity? not exactly, anyway. swift3 already rolls in something like domain_remap 21:34:55 "I want to cname and can't because THIS" - I need to do some learning in order to understand next steps 21:35:10 https://community.runabove.com/kb/en/object-storage/how-to-put-object-storage-behind-your-domain-name.html 21:35:14 a completely alternative implemenation for the use-case is cirtainly one way 21:35:22 Timburk: that's what i thought 21:35:39 jcaron: that's awesome! 21:35:46 jcaron: ok, cool. thanks 21:36:11 jcaron: still worth getting the diff on the inetnert - maybe not for people to use - but it'll give us a better frame of reference for the dicussion 21:36:26 Ok, I'll ask rdisez about it ;) 21:36:35 yeah, lets define the problem, and proposed TXT solution in a bug report or something. I feel like there cname and TXT might behave a little different, but I think I need to dwell on it a bit more. 21:36:49 jcaron: I think that gives you a good starting point: bug against cname in launchpad and show the middleware you have 21:36:50 jcaron: you *can* just push it to gerrit and mark it WIP so you can refenrece form the ideas page 21:37:07 one of the big differences is that typically you will resolve a CNAME chain all the way down and many resolvers can do that somewhat automagically for you 21:37:17 I think that the main diff will be the recursive step of cname 21:37:19 jcaron: also it sounds like *no one* is "against" the idea of TXT dns entry support in upstream 21:37:25 that will not be necessary with TXT 21:37:31 but that isn't the case with a TXT record, so if the domain in the TXT content is not a terminal you have to manually recurse things 21:37:37 jcaron: so you're solving an issue that has broad support/intrest - good work! 21:37:40 (potentially) 21:38:07 clarkb: great context - thank you! 21:38:17 and along with that comes loop detection 21:38:44 jcaron: if you're at the PTG maybe we can go find a designate guy - they know things about dns (more than me, probably everyone at the PTG will know more about dns than me) 21:38:56 Sorrrrry, what is PTG ? 21:39:14 (newbie spotted) 21:39:14 clarkb: oh yeah nice points, thanks man 21:39:15 https://www.openstack.org/ptg/ 21:39:17 jcaron: the in-person event that's next month in atlanda 21:39:22 *atlanta 21:39:46 notmyname: can we keep moving :\ 21:39:58 Ok thanks for the info 21:40:03 clayg: yeah, was just typing that :-) 21:40:18 jcaron: looking forward to seeing what you've got 21:40:19 mmmmm kauphy! 21:40:26 now moving on to some bugs... 21:40:29 #topic bugs 21:40:35 https://bugs.launchpad.net/swift/+bug/1651530 21:40:35 Launchpad bug 1651530 in OpenStack Object Storage (swift) "suffix hash invalidation may be lost" [Critical,Fix released] 21:40:38 fix released? we're done? 21:40:40 clayg: ? 21:40:41 notmyname : I'll reach to you soon 21:40:57 never before has there been a better patch than https://review.openstack.org/#/c/419787 21:41:15 storage polcies - scoff - ec - scoff - crypto - scoff 21:41:18 it's all about the hashes! 21:41:28 srly tho https://review.openstack.org/#/c/419787 fixes the last remaining issues 21:41:33 oh, but the bug already says fix released, and that patch doesn't reference a bug 21:41:37 but we need that patch to land too? 21:41:44 acoles and PavelK have been amazing 21:41:49 the final diff is pretty manageable 21:41:53 I mean, I'm told it's the best patch ever 21:42:02 we should really get that in before we cut another request 21:42:04 *release 21:42:16 ok 21:42:29 who else should look at it? 21:42:30 Only question I have is for backports (only I don't really *care* about backports - so we can *not* talk about backports too - don't care) 21:42:40 tdasilva: ? cschwede: ? 21:43:03 acoles: obviously if he *can* 21:43:05 i'll have a look at the patch tomorrow morning - ie in a few hours 21:43:15 I intend to 21:43:16 yeah and cschwede - I think that would be good 21:43:24 we're good! 21:43:27 thanks guys! 21:43:28 acoles: cschwede: thanks! 21:43:47 ok, let's talk about https://bugs.launchpad.net/swift/+bug/1639691 then 21:43:47 Launchpad bug 1639691 in PyECLib "EC: Swift can return corrupted Data and be able to go data lost at isa_l_rs_vand policy with >=5 parities" [Undecided,In progress] 21:43:56 clayg: you set it back to critical 21:43:58 I think mattoliverau also has some context - jrichli and mahatic might have looked at it 21:44:05 notmyname: ok yeah - thanks for bringing that up 21:44:07 ok... 21:44:21 so this only effects ida-l 21:44:24 *isa-l 21:44:27 that's intels' thing 21:44:34 and it only effects parity > 4 21:44:38 ok, so know that 21:44:46 Ah? 21:44:54 It opens? 21:45:00 but basically all major distros ship a liberasurecode that is silently corrupting your data 21:45:15 if you upgraded you are maybe getting errors - but it's not so scary - but it will never get better 21:45:42 they don't have libec >=1.3.1 right? 21:45:48 now that isa-l-rs-cauchy is out I think you should never let have someone have a isa-l-rs-vand policy with > 4 parity - it just sucks too much 21:46:16 so then the question is what to do if we detect a policy with that config 21:46:36 how is "they" major distros? yeah I mean we just released this shit - unless your swiftstack and you use isa-l with a policy parity > 4 - you're probably in such bad shape - it's really bad 21:46:46 and again - upgrading liberasurecode only makes it no EAT DATA 21:46:58 it's still bad - it throws errors on bad erase lists (if you have parity > 4) 21:47:14 so I want swift to not let someone set themselves up for that footgun - don't care *what* version your running 21:47:17 not sure how to do that 21:47:22 or you know... if we *need* to 21:47:33 it's not my fault liberasurecode sucks and you don't know what ec policies are good 21:47:53 I'm square - so I'd be cool to make that bug like - w/e just close it 21:47:55 well, it's easy to disallow that config. it's hard to disallow it and still allow access to data that may have been stored under a policy like that 21:48:11 ok, yeah so that would be the resolution to the bug IMHO 21:48:39 how about a disallow this config unless there's a "YES_I_KNOW_I_NEED_TO_CHANGE=true" in the policy config? 21:48:41 swift would not start if you have that policy unless you ... deprecate it? patch would be simple and close the bug IMHO 21:48:43 notmyname: isn't there a deprecated flag on the swift.conf options? 21:48:56 tdasilva: I think that would make sense 21:49:07 yeah, but if it's deprecated, can you still read the data? 21:49:08 you can still read data from it - but not create new containers with that policy 21:49:13 ah ok 21:49:13 yeah 21:49:15 then that's it 21:49:36 but that's a pretty big warning to ops 21:49:39 and once you upgrade your liberasure you can make a new isa_l_rs_cauchy with whatever parity you want 21:49:55 *needs to be a pretty big warning 21:50:02 clayg: is that true, i thought ceph guys still limited to less than 20 or something... 21:50:06 "warning" - you mean the swift not starting when it finds a isa_l_rs_vand with > 4 parity that's not deprecated? 21:50:07 you can also still create new data in any existing containers with that policy -- should we still allow that in this case? 21:50:19 I think it's sufficent to not start swift in this case 21:50:19 clayg: warning in release notes, etc 21:50:44 tdasilva: AIUI, pyeclib/libec will work as long as m+k<=32 21:50:49 timburke: i don't have an obvious way to prevent it - also I don't care - I don't ahve any of those policies 21:50:57 notmyname: gotcha! yes! 21:51:25 clayg: potentially even a major version bump (not sure on that) 21:51:25 notmyname: the change that makes existing configs cause swift to not start is one you might want make sure is g2g before rolling it out to all the nodes ;) 21:51:45 no idea 21:51:59 not only will it not start after upgrade, but the only way to make it start is to set it so you can't create new contaienrs with that policy 21:52:18 so everyone likes the suggested fix? what is the sevarity of the bug and if critical who's working on it? 21:52:32 notmyname: yeah! that's what I'm talking about! 21:52:50 wfm 21:52:52 I'm just saying that's going to be a Big Deal (tm) for deployers 21:53:07 only if they use isa-l-rs-vand and have parity > 4 21:53:14 and in that case - yeah sorry - it's a big deal for you 21:53:19 also you have to migrate everything 21:53:21 you're so screwed 21:53:23 i'm so sorry 21:53:36 ok, here's a first stab at a way to handle this... 21:53:53 next release, mention it in release notes, but don't make this a release blocking bug 21:54:09 later, implement the check so it doesn't start unless that policy is deprecated 21:54:28 notmyname: ok for now let's just add a warning then 21:54:41 yeah, that would be good too 21:54:42 notmyname: is the next release going to be the ocata release? 21:54:43 notmyname: move and then drop it down to HIGH - good plan 21:54:48 tdasilva: yes 21:54:51 I think we need to at least log a warning to mention it in release notes 21:55:03 ok, so who's going to work on implementing the warning? 21:55:16 tdasilva: and so you get a warning "your config is going to be deprecated - you need to upgrade liberasurecode - you've got a huge data migration on your hands" 21:55:19 I really like this plan 21:55:22 someone should make notmyname PTL 21:55:58 acoles: +100 log a deprecation warning and put in the release notes that we're going to put a huge wall around this footgun in a future release 21:56:12 does anyone else use ISA-L aside from swiftstack? if not, I'd suggest someone from swiftstack do the warning 21:56:19 ie clayg, timburke, or me 21:56:22 if you're effected you have a release to prepare for our killing of this dangerous config 21:56:35 Using 21:56:44 not using 21:56:47 kota_: oh? interesting. i thought you just had ntss 21:56:49 we're going to run out of time :'( 21:56:51 ntts? 21:56:56 nhss? 21:56:57 sshs? 21:56:59 nssh? 21:57:00 Shss 21:57:02 whatever 21:57:05 rofl 21:57:06 rofl 21:57:08 lol 21:57:12 bah. i was closest! 21:57:14 rfol 21:57:29 ok, so kota_'s sick. timburke or clayg? can one of you take it? 21:57:41 sure 21:57:45 thanks timburke 21:57:53 Thx 21:58:03 #topic open discussion 21:58:03 if it's not done by the PTG I'll do it - it'd be like... sometime after 2/9 21:58:11 I thinkw e made it through the meeting 21:58:23 clayg: nah, gotta be before the release (whcih is before the PTG) 21:58:33 anythign else to bring up? anythign I missed? we have 1.5 minutes 21:58:40 https://review.openstack.org/#/c/415473 21:58:58 i accidently said that "someone" should break up proxy.test_server (so should!) - but then someone did 21:59:04 and now I don't know how to review it 21:59:06 any tips? 21:59:29 Can I just +A like "whoops" - it's only tests - if they pass what's the worst that can happen? 21:59:30 that's...a really big patch 21:59:43 lots of coffee or scotch (depending on the time of day) and no interruptions :P 22:00:01 my only suggestion to make it more managable would be to do the same i did for the func tests 22:00:09 is it really going to take me a day of not working on swift to land a *test refactoring* 22:00:14 smaller patches, one for each controller 22:00:20 it should be a quick easy review? how else are we going to split up that file? 22:00:22 oh get tdasilva to do it.. I like that plan :P 22:00:26 lol 22:00:39 ok, we're at time. we can take this to #openstack-swift 22:00:45 sure 22:00:46 thanks everyone for working on swift 22:00:48 #endmeeting