21:00:47 #startmeeting swift 21:00:48 Meeting started Wed Jun 24 21:00:47 2015 UTC and is due to finish in 60 minutes. The chair is notmyname. Information about MeetBot at http://wiki.debian.org/MeetBot. 21:00:49 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 21:00:51 The meeting name has been set to 'swift' 21:00:57 hello everyone 21:01:01 hola 21:01:03 o/ 21:01:04 who's here for the swift meeting? 21:01:05 hello 21:01:07 o/ 21:01:07 yo 21:01:07 o/ 21:01:07 o/ 21:01:12 o/ 21:01:13 o/ 21:01:17 Hi 21:01:29 hello 21:01:39 👾 21:01:52 * MooingLemur squints at glyph 21:02:10 ok, another full agendat this week. let's get started and see if we can make good progress 21:02:14 #link https://wiki.openstack.org/wiki/Meetings/Swift 21:02:19 hi 21:02:25 first up: hackathon info 21:02:44 #link https://www.eventbrite.com/e/swift-hackathon-tickets-17308818141 21:02:53 week of august 10 in austin 21:03:07 hello 21:03:26 yay summer in TX! 21:03:32 yay :-) 21:03:34 as always, this is designed to be productive and review code, design stuff, catch up, etc 21:03:53 the only "rules" are no powerpoint and no intro to swift topics ;-) 21:04:00 thanks to IBM and jrichli for hosting 21:04:05 yw! 21:04:37 if you have any questions about the hackathon, feel free to ask me or jrichli 21:04:42 mattoliverau: summer in TX isn't that bad, everyone has AC. =) 21:04:43 can we get the city air conditioned by any chance? 21:04:44 hi 21:04:56 peluse: lol 21:05:01 peluse: wear tank tops and shorts and sandals, then you'd be fine. 21:05:02 :) 21:05:16 under armor tank tops 21:05:16 next up... 21:05:17 wel, its like 112F here today so I guess I can handle it :) 21:05:17 peluse: you're an Arizonian, aren't you? 21:05:24 #topic priority reviews 21:05:42 tdasilva, lol! know me too well 21:05:46 good news is that the 1+ object server per disk landed, along with the ring port patch 21:05:49 peluse is AZ? *waves from Central/Osborn* 21:06:09 no way, cool! chandler 21:06:51 please take a look at the review dashboard for starred patches and the "needs final approval" patches 21:06:59 there are a _lot_ in the latter category 21:07:12 * peluse slides down in chair... been sucked up in non-sense for the last week or so... 21:07:16 notmyname: do you have the ec bug list? 21:07:17 #link http://goo.gl/8IUcKl 21:07:30 notmyname: I think peluse landed the keyerror fix 21:07:30 clayg: ya, we;ll get there next topic 21:07:33 oh :\ 21:07:34 peluse: +1 (me too) 21:07:35 great! 21:07:45 peluse: you and me too 21:07:46 before we get to bugs... 21:08:26 looking at the landed specs, any updates from that ongoing work? encryption, fast-post, part power, symlinks, notifications? 21:08:49 qq: what hapened to the increase ring part power spec? I thought it was a spec that acoles had submitted, no? 21:09:09 peluse: http://specs.openstack.org/openstack/swift-specs/specs/in_progress/increasing_partition_power.html ? 21:09:11 notmyname: with acoles mia - I'd say fast-post is at risk, jrichli probably also needs more help with encryption (acoles was key there as well) 21:09:22 It's landed cause it's something we want to do.. at some point.. if i remember correctly 21:09:30 peluse: I think we were sliding that one before we even left the summit 21:09:34 notmyname: i’m working on notifications. takes some time though 21:09:39 Oh, OK. confused me to not see it on the dashboard 21:09:41 cschwede: cool 21:09:55 peluse: check http://specs.openstack.org/openstack/swift-specs/ for landed ones 21:10:02 or the top of the priority review wiki page 21:10:08 clayg: yeah sorry i'm down a rabbit hole on internal stuff but wanting to get back on it asap 21:10:10 gracias 21:10:14 jrichli: you've split up some of your encryption work. can you talk about it now? 21:10:22 clayg: spoke to acoles earlier this week - he said he's super busy atm 21:10:24 sure, There are new reviews that are more modular and easier to review 21:10:34 trivial keymaster: https://review.openstack.org/#/c/193749/ 21:10:42 cryptography module: https://review.openstack.org/#/c/193826/ 21:10:48 acoles: that's great news - i wasn't expecting to hear that - it's definately no hard feelings - but acoles not being 100% is a thing you have to plan around - because acoles @ 100% is a HUGE impact 21:10:49 encrypter and decrypter for obj body: https://review.openstack.org/#/c/194191/ 21:10:59 clayg: +1 :-) 21:11:10 it would really help me to get some feedback to work on 21:11:13 acoles: didn;'t spot you in here 21:11:15 jrichli: looks like those a part of a chain 21:11:18 acoles, jrichli : mahati starts Jul 6 and wil be able to jump in help on crypto if that works for you guys 21:11:20 joeljwright1: acoles: I shouldn't have said MIA - acoles was so gracious to give us all the heads up 21:11:23 you dont have to cover everything: just some things for me to improve 21:11:50 clayg: mia but making tracks back to the front line as soon as i can ;) 21:12:10 looks like the first blocker on encryption is https://review.openstack.org/#/c/158401/5 21:12:12 notmyname: peluse: cschwede: we talked at the summit spreading too thing - we have some more help now - but there's still a lot of ground to cover - we need to pick some winners and stay on them (i.e. jrichli needs our support) 21:12:12 peluse: good to hear, thanks! 21:12:16 joeljwright1: just lurking 21:13:13 yes. and encryption is a thing that should be one of the ones emphasized in the short term 21:13:15 notmyname: ok - i have patch 158401 open - i'll get eyes on it today 21:13:15 clayg: https://review.openstack.org/#/c/158401/ 21:13:21 clayg: thanks 21:13:25 notmyname: it's only 322 lines ;) 21:13:37 clayg : thanks!!! 21:13:44 clayg: also, it's on the feature branch, not master :-) 21:13:47 jrichli: if it's all nits - can I just push over? 21:13:58 sure 21:14:47 cschwede: looks like you had eyes on it too - anything you want to point out that's maybe not in the review? 21:15:25 mattoliverau: you're working with hurricanerix on large container DBs for cotainer sharding? 21:15:45 clayg: hmm, i think i added my concern to the review. nothing else afair, but i’ll have another look tomorrow 21:15:48 yeah, hurricanerix is awesome and helping me out there :) 21:15:53 good 21:16:05 anything else from anyone on ongoing work? 21:16:19 while we're talking about bugs 21:16:26 could really do with some help here 21:16:28 https://review.openstack.org/#/c/189815/ 21:16:37 clayg: cschwede jrichli re patch 158401 I'll look at any fixes needed tomorrow unless clayg pushes them over first 21:16:37 acoles: https://review.openstack.org/#/c/158401/ 21:16:44 the test fail seems genuine, but no idea what's causing it 21:16:45 FYI I have some partial PUT error handling code (on the GET side) for Ec about 30% done that I should b able to get back on by Fri or Mon at he latest 21:16:49 cschwede: sorry i didnt get back to your comments on that one sooner 21:17:10 peluse: good 21:17:21 acoles: yeah i'll probably push to address cschwede's concerns and leave a +2 for one of you eu guys to +A 21:17:24 *progress* 21:17:30 great! 21:17:35 thanks acoles clayg cschwede 21:17:39 clayg: cool 21:17:50 time to move on to bugs? 21:17:52 regarding the patch joeljwright1 mentioned - it’s super strange, is if there is some other client using a named arg 21:17:53 and 3x plus EC perf numbers are roling in as we speak... will get it all published once its done. On track for end of this week (well, then a few days to sort/post somewhere) 21:18:03 #topic bugs 21:18:09 peluse: where would you post the EC perfor numbers? 21:18:31 beats me :) notmyname where do you want this stuff? will ahve pictures as well 21:18:48 mine is about 95% done - need to trim some extraneous fat around the results. 21:18:49 yay pcitures! 21:18:53 peluse: let's figure that out over email this week 21:19:01 with circles and arrows and everything :) 21:19:03 OK 21:19:07 notmyname: peluse: can you include me on that regard plz? 21:19:11 you bet 21:19:20 wbhuber: not to hide it. just to figure out the best place to put it :-) 21:19:28 everyone will be included in seeign the results :-) 21:19:29 notmyname: lol 21:19:53 thought "email" was outside the domain 21:20:18 joeljwright1: does patch 189815 *really* help the docs - or it's like work in preperation to help the docs? 21:20:18 clayg: https://review.openstack.org/#/c/189815/ 21:20:21 kota_: https://review.openstack.org/#/c/194518/ is an item you added to the meeting agenda. a test for ssync 21:20:45 ah, yes but it's not so big one I think 21:21:00 the related patch is priority and 21:21:00 clayg: 189815 just reduces the number of names we use for 'object_name' by 1 21:21:08 kota_: and it already landed 21:21:09 joeljwright1: i know louis i'll ping him to find out what its all about 21:21:14 acoles: thansk 21:21:18 yes 21:21:23 acoles: the patch is fair 21:21:33 too late, I was playing the patch 21:21:36 acoles: the failure of gate-tempest-dsvm-neutron-src-python-swiftclient is bloody confusing 21:21:50 How often do the "updates from global requirements" jobs happen? 21:22:00 mikal: lots 21:22:05 * mikal can't see an update from nova for the change yesterday 21:22:06 I was playing that I deleted some lines from patch but all tests are passed succusessfully 21:22:12 around here https://review.openstack.org/#/c/189815/ 21:22:14 So I assume I am waiting for cron somewhere 21:22:14 acoles: all the patch does is rename 'obj' to 'name' in non-kwarg use in the client.py code3 21:22:15 acoles, while we on bugs can you take another look at this low hanging fruit patch 185459 21:22:16 peluse: https://review.openstack.org/#/c/185459/ 21:22:16 oh no 21:22:24 here https://github.com/openstack/swift/blob/master/swift/obj/ssync_receiver.py#L354-L356 21:22:51 kota_, hey, thanks for the +2 on that one :) 21:22:52 ok. we've got about 3 things going on at once here 21:23:00 * notmyname is getting confused ;-) 21:23:00 peluse: sure will do 21:23:06 only three jobs man?? 21:23:25 acoles: will help look at the patch that is failing that joeljwright1 brought up 21:23:28 * peluse in his best jamiacan accent 21:23:34 right? 21:23:40 or was that cschwede? 21:23:49 notmyname: will try to 21:23:53 thansk 21:24:00 notmyname: cschwede looked at it 21:24:08 joeljwright1: ah, ok 21:24:09 notmyname: we're both confused 21:24:18 I saw I got a ping in here 21:24:19 mikal: your not on infra :P 21:24:19 * cschwede cschwede is confused about the patch failing 21:24:21 oh if cschwede didn't figure it i have no chance :) 21:24:28 joeljwright1: yeah, I have no idea why it's failing 21:24:37 mattoliverau: yeah, I just figured that out, I was hoping to slink away unnoticed... 21:24:48 kragniz: hey! 21:24:51 mikal: too bad!.. tho you can fire me :P 21:24:52 the patch doesn't really do anything 21:25:05 kragniz: thanks for taking a look, we're all at a total loss 21:25:07 acoles: hey there! 21:25:46 i still think some lib is using swiftclient with a keyword arg. that’s the only reason i can think of 21:26:11 cschwede: would love to track it down 21:26:38 cschwede: but I'm super busy for a software release atm and no time to dig for hours 21:26:39 it’s not cinder and glance, if i didn’t miss anything there - checked the code 21:27:27 kragniz: joeljwright1: cschwede: sounds like a real mystery! which patch now? 21:27:37 clayg: https://review.openstack.org/#/c/189815/ 21:27:59 clayg: the patch is so simple it should be a no-brainer 21:28:31 lol - yeah that's awesome - i totally imagined someone might be using named arguments and get bit by that 21:29:07 ok, so what's the next step for this one? 21:29:24 do we need to talk to enutron people? 21:29:30 *neutron 21:29:35 I guess so 21:29:52 what does neutron do with swift?! 21:29:59 it's one more step on making the swiftclient api easier to understand 21:30:11 clayg: no idea, but it's their gate job that's failing 21:30:16 can i download the whole logs? or do I have to wget? 21:30:25 hey, try changing 'name' to 'obj' and see what happens :) 21:31:43 joeljwright1: thanks for bringing it up. a nice mystery :-) 21:31:49 :) 21:31:53 looking at the failed test logs there are compute/nova client errors 21:31:55 it'll be nice if we solve it! 21:32:05 ok, other bugs 21:32:13 here's the EC bug list 21:32:14 #link https://bugs.launchpad.net/swift/+bugs?field.tag=ec 21:32:22 clayg: you wanted to comment on something here, ithink 21:32:34 joeljwright1: not sure how errors like "Details: {u'message': u'No valid host was found. There are not enough hosts available.', u'code': 500, u'created': u'2015-06-21T21:32:06Z'}" have anything to do with swiftclient? 21:33:16 swifterdarrell: me either, but honestly it seems to be repeatable! this thing is blowing my mind! 21:33:48 awwwwzzz yeah i'm downloading now - -e robots=off was the key 21:33:55 comment on what? 21:34:01 I think minwoob_ had a patch on ec 21:34:02 clayg: an EC bug 21:34:16 it was somethign about not rebuilding something 21:34:21 there was a bug open for it 21:34:28 man, that list is too long 21:34:30 caused segfaults in liberasure 21:34:37 clayg: It's for https://bugs.launchpad.net/swift/+bug/1452553 21:34:39 Launchpad bug 1452553 in OpenStack Object Storage (swift) "don't rebuild existing fragments" [Undecided,In progress] - Assigned to Minwoo Bae (minwoob) 21:34:49 peluse: maybe there's some duplicates :-) 21:35:15 clayg: The mixed_old_etag and mixed_new_etag tests aren't working with the bad response injected in. 21:35:21 yeah, I dunno. Once I get the PUT failure/GET changes posted I'll start on whatever is up there 21:35:23 clayg: I've been trying to figure out why. 21:35:31 minwoob_: do you have a patch in gerrit or just in the bug report? 21:35:44 The initial patch is up there, but that one had 5 failing tests. 21:35:53 I've brought it down to 2. 21:35:55 swifterdarrell: joeljwright1: that error is because an image (or was it volume?) is missing, thus the node is not ready 21:36:03 minwoob_: oh RIGHT i had that gist to pull the frag index off the metadata in the ec fragments 21:37:02 minwoob_: but i forget why that was important - your code would... noop or error or something if it got duplicates? 21:37:05 Apparently that doesn't help for the bad response. 21:37:22 for all the EC bugs listed on https://bugs.launchpad.net/swift/+bugs?field.tag=ec, please look at the ones labeled "New" 21:37:36 we need to either confirm them, get more info, or get a patch for them 21:37:53 "new" status == needs attention from us (swift devs) 21:37:56 clayg: Yeah. It would only find either 0 or 1 of 10 fragments available for reconstruction. 21:38:07 notmyname: yayayayayay!!! i love that link - can it go somewhere i already have book marked or do I need a new book mark for that? 21:38:08 clayg, minwoob_ not sure what the dups comment was but FYI the patch I'm working on (not posted yet) will handle getting dup frag indexes on GET 21:38:29 same with https://goo.gl/uO4b7l (python-swiftclient new bugs) 21:38:32 peluse: nice - minwoob_'s patch is in the reconstructor tho 21:38:41 peluse: do you still plan to provide design thoughts on how to generally handle insufficient fragments: https://bugs.launchpad.net/swift/+bug/1457691 21:38:42 Launchpad bug 1457691 in OpenStack Object Storage (swift) "Insufficient Fragments occurred on EC GET object" [Undecided,New] - Assigned to Bill Huber (wbhuber) 21:38:45 clayg: that's the LP search by tag link. new bookmark 21:39:08 wbhuber, yeah I've just been scuked away for the last 10 days or so 21:39:12 notmyname: maybe here -> https://wiki.openstack.org/wiki/Swift/PriorityReviews 21:39:25 clayg: yeah, I'll them there 21:39:27 clayg, ahh, OK. wasn't sure 21:39:49 minwoob: did you already see issue list on erasure code side? 21:39:56 minwoob_: 194518 21:40:00 minwoob_: no 21:40:07 ok, let's spend no more than 5 more minute on bugs, then move on to the specs that are open 21:40:10 minwoob_: https://bitbucket.org/tsg-/liberasurecode/commits/5e6592d557e6d2006fa0dc176049c196784b336f 21:40:47 clayg: I might just go ahead and post the next patch set, even though 2 of the original 5 tests are still failing. 21:40:50 any other bugs to discuss? any resolution on what the next step is on these bugs we're talking about now? 21:41:06 minwoob_: sure that'd be great - i'm sure we can figure out the tests 21:41:19 minwoob_: yeah, generally keeping the code in gerrit is better. it's the normal place people go look for it and it's easy to pull it down to test 21:41:20 clayg: Okay. 21:41:25 minwoob_: thanks 21:41:31 notmyname: i'll follow up and make sure minwoob_ has what he needs for the ec bug he's working on and go over the list of new ec bugs 21:41:33 Sure. 21:41:37 notmyname: someone else needs to help joeljwright1 21:41:46 clayg: thanks 21:42:18 joeljwright1: will be helped by acoles (as he's available) and cschwede has looked into it. I'll ping someone from neutron this afternoon 21:42:44 oh - is that all he needs - finding who's calling the swift client with named params? 21:42:54 clayg: AFAICT 21:42:59 i'll probably know that as soon as my wget finishes 21:43:28 any other bugs to raise in here this week? 21:44:20 ok, moving on to specs 21:44:21 notmyname: what about the ring assigning duplicate partitions to same device? 21:44:26 . 21:44:37 link? 21:44:40 idk 21:44:52 :-) 21:44:58 https://bugs.launchpad.net/swift/+bug/1452431 21:44:58 Launchpad bug 1452431 in OpenStack Object Storage (swift) "some parts replicas assigned to duplicate devices in the ring" [Undecided,New] 21:45:03 thanks 21:45:13 notmyname: oh! it's tagged ec - nm so we're on it ;) 21:45:19 perfecto! go work bug triage man! 21:45:39 still in new status though. needs further eyes :-) 21:45:42 notmyname: you should tell us all about how that went and reivew your ideas on how it's going to be awesome in the future? 21:46:00 notmyname: you gave me a link of ec bugs - i'm all over it 21:46:01 s/went/is going/ (FTFY). and slowly 21:46:28 ok, let's move on to specs 21:46:34 #topic open specs 21:46:38 #link https://review.openstack.org/#/q/project:openstack/swift-specs+status:open,n,z 21:46:47 these are the specs patches that haven't landed 21:46:57 most are net-new (not updates to existing ones) 21:47:09 for those that are updates, they /should/ be able to land quickly 21:47:44 mattoliverau: will you get to the spec lifecycle rules patch? that one will be easy to land 21:48:15 yeah, I started last night, i'll post it after breakfast :) 21:48:18 ok 21:49:03 mattoliverau: which spec is that? 21:49:09 https://review.openstack.org/#/c/183014/ <-- keymaster updates for encryption spec 21:49:16 notmyname: did you red the ML thread on specs are broken (I know nova doens't use them like we do) 21:49:19 clayg: just a patch to the specs repo 21:49:29 clayg: that's why we don't use them like nova ;-) 21:49:36 clayg: wasn't that your idea? 21:49:51 notmyname: yeah well - it was just an idea - i'm not sure if we've really refined the process yet 21:50:00 I'm pretty sure we haven't 21:50:03 notmyname: I thought mattoliverau had something going about "this is our process for specs" 21:50:05 case in point, the current topic 21:50:14 clayg: https://review.openstack.org/#/c/190066/ 21:50:19 #link https://review.openstack.org/#/c/190066/ 21:50:37 notmyname: so that I was interested in reviewing - i'm not interested in yet another pass over fast-post spec, or +2ing symlinks *again*, or... really much with specs :\ 21:50:38 ^ just some rules when writing or thinking about specs. 21:50:51 mattoliverau: thanks 21:50:53 notmyname: htanks 21:51:39 ok, so other specs that are out there that we talked about in vancouver are 21:51:51 changing policies https://review.openstack.org/#/q/project:openstack/swift-specs+status:open,n,z 21:51:57 https://review.openstack.org/#/c/168761/ 21:52:24 mattoliverau: why is there -1's all over the make specs better spec! :'( 21:52:34 and https://review.openstack.org/#/c/151335/ but that one is being worked on 21:52:44 clayg: cause my english is.. well australian :P 21:52:48 lol 21:52:53 and crap :P 21:52:59 sigh 21:53:24 cschwede: you've got 3 open specs. metadata search, undelete, and container alias 21:53:28 clayg: I just quickly put something up to start discussions, so that worked :) 21:54:01 cschwede: yeah! what gives! :P 21:54:14 cschwede: you should be sharing 21:54:24 metadata is currently -1, i need to get back to it - was too busy with other things. the other two (containeralias and undelete) could take a review 21:54:37 tdasilva: if you have a moment, you can change your +1s to a +2 21:54:58 notmyname: on which spec? 21:55:11 torgomatic: and I volunteer you to review containeralias and undelete? 21:55:14 tdasilva: https://review.openstack.org/#/c/155524/ 21:55:16 For 21:55:19 or... maybe just undelete? 21:55:37 I can look at undelete once I'm done with the #$%^ ring builder 21:55:48 torgomatic: fair point :'( 21:56:12 torgomatic: sorry - you've been buys while i was off dicking around - i assume and apologise 21:56:42 ok, any other specs to specifically bring up? 21:56:42 clayg: it's not like I've got anything to show for it :| 21:56:59 dmorita: looks like yours needs some eyes too https://review.openstack.org/#/c/168761/ 21:56:59 Changing policies, i dont know what should i add to get approaval, actually, but i start to write codes, 21:57:01 I did manage to type in some code that accidentally squared the overload (so (1 + overload)^2) 21:57:09 dmorita: oh, good! 21:57:10 but that's off-topic, so I'll stop yammering now 21:57:51 can we land the sharding spec.. so it's in the right place (it is a POC) and so I can wrinte new patches without adding to the current patchsets? 21:57:56 Yes, i woul like a lot of guys to take a look 21:58:30 dmorita: were you able to make any progress adapting https://review.openstack.org/#/c/173580/ 21:58:34 mattoliverau: if it's good we can ;-) 21:58:38 lol 21:59:07 dmorita: could you take over that review - assuming a general policy change from replicated to ec is worth while even if the replicated to replciated case could be optomized later? 21:59:20 dmorita: there's all the prxoy work with a container that's in flight that needs work 21:59:30 I figure the stuff we discussed at summit, if we decided there worth following up on and haven't been updated since should land. but thats just me :) 21:59:35 dmorita: you could learn alot by trying out something that works and picking at the gaps 21:59:43 1 minute left 22:00:12 any last minute things? 22:00:25 ok, it's time 22:00:32 thanks for coming. thanks for working on swift 22:00:37 #endmeeting