21:00:15 <notmyname> #startmeeting swift
21:00:16 <openstack> Meeting started Wed Jan 11 21:00:15 2017 UTC and is due to finish in 60 minutes.  The chair is notmyname. Information about MeetBot at http://wiki.debian.org/MeetBot.
21:00:17 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
21:00:20 <openstack> The meeting name has been set to 'swift'
21:00:24 <notmyname> who's here for the swift team meeting?
21:00:26 <mattoliverau> o/
21:00:28 <cschwede> o/
21:00:28 <jrichli> o/
21:00:30 <bkeller`> o/
21:00:30 <mathiasb> o/
21:00:30 <joeljwright> hey
21:00:32 <dmorita> o/
21:00:34 <milkman__> o/
21:00:37 <pdardeau> hi
21:00:41 <kota_> o/
21:00:42 <m_kazuhiro> o/
21:01:01 <sgundur_> hi
21:01:24 <acoles> hi
21:01:30 <tdasilva> hi
21:01:36 <notmyname> welcome everyone
21:01:43 <kmARC> o/
21:01:46 <notmyname> agenda for this week is at
21:01:47 <notmyname> #link https://wiki.openstack.org/wiki/Meetings/Swift
21:02:12 <notmyname> and because of timezones, I'd like to start at the bottom with cschwede
21:02:26 <notmyname> cschwede: what's up with part power increase? I'm looking forward to that!
21:02:33 <notmyname> https://review.openstack.org/#/c/337297/
21:02:55 <cschwede> notmyname: thx! well, i just wanted to check if there are any questions regarding the patch, if i could help reviewers getting this forward
21:03:37 <notmyname> looks like in the past both mattoliverau and pdardeau have commented on it
21:03:49 <notmyname> cschwede: but you're currently blocked needing reviews on it, right?
21:03:51 <timburke> hey, that was another one i wanted to look at from the "Needs Final Approval" category on http://not.mn/reviews.html
21:04:06 <pdardeau> i'll review again
21:04:10 <mattoliverau> I'm mostly back in the swing after my vacation, so I'll have another play with it :)
21:04:16 <cschwede> yes, and I hopefully adressed all comments from Matt and Paul
21:04:18 <notmyname> pdardeau: mattoliverau: thanks!
21:04:27 <cschwede> Pete already added his +2 to that patch
21:04:39 <notmyname> great
21:05:02 <cschwede> so ping me if you have time for review and questions on that patch - would be great!
21:05:19 <mattoliverau> cschwede: well pdardeau and I will find some time :)
21:05:28 <pdardeau> yep
21:05:37 <cschwede> mattoliverau, pdardeau: thx a lot!
21:05:42 <notmyname> ok, back up to the top of the list
21:06:01 <notmyname> https://bugs.launchpad.net/swift/+bug/1651530 and https://review.openstack.org/#/c/402043/
21:06:01 <openstack> Launchpad bug 1651530 in OpenStack Object Storage (swift) "suffix hash invalidation may be lost" [Critical,New]
21:06:09 <notmyname> acoles: clayg: what's the status here?
21:07:41 <notmyname> I see clayg had pushed some review comments, but also a bunch of other related patches?
21:07:47 <acoles> notmyname: I saw clayg did a great review
21:08:03 <acoles> notmyname: yes, I saw those but not had time today to follow up
21:08:11 <clayg> hi!
21:08:17 <notmyname> oh hi!
21:08:20 <cschwede> so that's a race condition?
21:08:20 <acoles> I think maybe he broke it up into several patches
21:08:20 * clayg busts over some boxes for acoles
21:08:33 <clayg> acoles: oh yeah... i was feeling really bad about that last night
21:08:40 <acoles> hehe
21:08:56 <notmyname> clayg: so that's the relation of https://review.openstack.org/#/c/402043/ and the patches you pushed up?
21:09:05 <clayg> does anyone know Pavel?  Can I tell him ... idk something ... that i'm not a jerk on purpose?
21:09:18 <acoles> I already told him :P
21:09:22 <acoles> JK
21:09:24 <tdasilva> lol
21:09:36 <joeljwright> :D
21:09:37 <clayg> so... three days in I *think* - I *think* I understand patch https://review.openstack.org/#/c/402043
21:09:40 <clayg> *almost*
21:09:50 <acoles> clayg: you were faster than me
21:09:52 <clayg> I spit it up into smaller patches that I do understant
21:09:55 <clayg> *understand
21:10:11 <clayg> so... what to do?  acoles has a +2 on patch 402043
21:10:16 <notmyname> ah ok
21:10:35 <clayg> I'm like... I can *almost* think - i'm scared... but I keep with it and believe in myself
21:10:46 <clayg> I might be able to +A it without ruining everyone's clusters
21:10:55 <acoles> apart from now needing to re-review them, I see the merit in splitting up, there were definitely several issues being addressed, and as someone pointed out the commit message "optimize.." hides fact that there is a bug fix
21:11:16 <kota_> true
21:11:17 <notmyname> clayg: you cplit it into 4 patches right? https://review.openstack.org/418692 https://review.openstack.org/418691 https://review.openstack.org/418689 https://review.openstack.org/405134
21:11:19 <clayg> or rather I'm not sure if doing that should constitute negligence - since we *obviousl* should not fix three unrelated things in a critcal bug fix
21:11:48 <cschwede> so Pavel is not around? onovy maybe?
21:12:00 <onovy> o/
21:12:34 <acoles> I feel bad that I did not push back more on the multiple-fixes-in-a-patch, I guess that's the beauty of fresh eyes
21:12:34 <cschwede> onovy: hello :) so clayg and acoles are just discussing Pavel's patch ^^
21:12:39 <onovy> hi
21:12:54 * onovy is fine with spliting
21:13:10 <clayg> acoles: you see Closes-Bug and it says CRITICAL it's really hard to say "no"
21:13:19 <cschwede> splitting sounds good to me too
21:13:29 <acoles> clayg: are your patches in a chain, if I pull one will I see the same end result?
21:13:34 <onovy> and pavel is on vacation (family reason, new baby)
21:13:46 <clayg> honestly tho - i'm more pissed about the double rehash optimization than the teeny little race on new parts
21:13:56 <clayg> i mean... both should be fixed obviously :P
21:14:23 <clayg> but I have this deep burning hate for with REPLICATE requests
21:14:47 <clayg> did we netsplit?
21:14:53 <onovy> no, i'm here
21:14:54 <clayg> acoles: no my patches are *not* in a chain
21:14:54 <notmyname> no, just quiet :-)
21:14:59 <acoles> clayg: k
21:15:10 <clayg> the changes were all orthogonal and I didn't want review of one to hold up any other
21:15:29 <clayg> if you're super into eliminating the write iop on noop replicate reqeusts - go review that one
21:15:48 <clayg> if you don't like the race with invalidate of a part while doing the initial rehash to create the hashes.pkl - go review that one
21:15:51 <onovy> should Pavel review that three patches too, right?
21:16:04 <clayg> if you think doing rehash of every suffix twice every replicate call is a bad idea - go review that one
21:16:31 <clayg> onovy: that would be super helpful
21:16:33 <notmyname> seems that on one hand we've got a complex patch that closes an important issue and does a few other nice things too, and already has one +2. on the other, we've got a few unlrelated patches that each do only one thing, but they're all unreviewed
21:16:34 <mattoliverau> clayg: I've been playing with that patch. So I'll continue to review it (noop one)
21:16:42 <onovy> clayg: i will ask him tomorrow
21:16:56 <acoles> onovy: let him enjoy his new baby! I'll take a look over them, but he may want to before they merge
21:17:00 <clayg> notmyname: that's a great description of the situation!
21:17:21 <clayg> notmyname: but please understand - i wasn't trying to make extra work - or even get on a high horse about splitting them up
21:17:32 <onovy> acoles: he is working from home. sometimes :)
21:18:02 <clayg> notmyname: if you look at my review I'm all like "why the ^%&* is this gross code here?" and it's not till you understand all the things going on that you can sort to understand the motivations and reasons for all the changes
21:18:06 <clayg> and I think...
21:18:07 <notmyname> ok, so the risk of sticking with the first is that we might land broken code. the benefit is that it might land sooner. the risk of the second method is one or more just languish for a long time, and the important one(s) will take longer anyway for rereviews
21:18:17 <clayg> well acctually I still don't unerstand the fix for the new part race
21:18:26 <clayg> I ended up with a totally different solution to make the tests pass
21:19:51 <notmyname> clayg: what do you want to do? stick with the new patches or just use your work there to understand the first one?
21:20:11 <clayg> notmyname: "might land sooner" - if Pavel is not intested in collaborating on the split up changes to fix these issues independently I think I would prefer not to side step his amazing efforts in the combined patch
21:20:14 <notmyname> acoles: what do you want to do? rereview or land the one you've already +2;;d
21:21:04 <clayg> notmyname: that would leave me trying to +A the existing combined patch - which so far I don't have a sufficient behavioral deficiency to -1
21:21:27 <clayg> ... but I don't understand it yet either
21:21:28 <acoles> notmyname: I think my +2 came with caveat that I had some hand in the patch, so ideally we'd have two more +2's on something critical
21:21:30 <clayg> ... trying
21:22:16 <clayg> acoles: can you maybe explain to me after the meeting how it managed to fix invalidate_hashes dropping suffix invalidation on the floor when the hashes.pkl doesn't exist without changing invalidate_hashes?
21:22:23 <onovy> just my two cents: if we can wait a bit, Pavel can take a look into 3 splitted/clayg's patches
21:22:24 <acoles> clayg: are your patches a decomposition of the original or wildly different?
21:22:26 <clayg> I think that's he only piece I really still don't understand
21:22:32 <notmyname> ok. I'm not leaning too strongly one way or the other, but if we need a decision made, I'd vote for the splits
21:22:36 <clayg> the rest is style nits that I think I can fix in a follow up
21:22:46 <onovy> clayg: Pavel should explain it to you :)
21:22:54 <clayg> acoles: the tests transfer - the solutions are ment to be "more obviously correct"
21:22:58 <clayg> no wildly different
21:23:24 <kota_> notmyname: +1
21:23:26 <acoles> clayg: I can try explain but I'll need to reload it first.
21:23:26 <notmyname> onovy: aside from the cosmic injustice of knowing there's a bug and having a patch unmerged that seems to fix it, no, there is no particular rush
21:23:30 <clayg> except for the race - i still don't understand pavel's fix for the race - I think my fix is obviously correct and I stole your tests
21:23:57 <kota_> I'm moving to review the original combined to the splitted 4.
21:23:58 <clayg> *obviously correct* modulo whatever could be claned up pending review (e.g. the weird mkdirs call under directory lock?  wtf?  cc kota)
21:24:15 <onovy> notmyname: yep. critical bug (data loss) was already fixed. this should be well tested
21:24:25 <onovy> don't added another regression here (again :)
21:24:37 <notmyname> kota_: I don't follow what you mean by "review the original combined to the split 4"
21:24:41 <clayg> onovy: if Pavel could do that it would be *very* helpful!  what's his handle?  does he lurk?
21:25:10 <mattoliverau> notmyname: I think kota_ means he's faviouring the combined.
21:25:17 <mattoliverau> I'm not sure myself.
21:25:21 <kota_> notmyname: ah, i like the second method to review clayg's patches, i mean and I has started to review them.
21:25:22 <mattoliverau> personaly I like them split up, only because they're more targeted and easier for my head to get around (eventually). But that might just be my brain being slow.
21:25:44 <clayg> acoles: well could you read the commit message and glance at the diff in https://review.openstack.org/#/c/418690/
21:25:47 <onovy> clayg: you mean just now? it's 10pm here, so i think he is sleeping :)
21:26:09 <notmyname> ok, let's do that. it's decided. we'll get Pavel's view on the split patches, acoles will (may?) rereview, clayg will continue to own the split patches, kota_ et al will review
21:26:20 <onovy> notmyname: +1
21:26:22 <notmyname> everyone agree?
21:26:30 <clayg> acoles: you filed the bug - so you should at least be able to reload the failure quickly - and I think my diff is small/obvious enough that you might either agree "yeah that's the fix" or "oh, now I remember what the other change did!"
21:26:31 <onovy> i will send note to Pavel
21:26:34 <mattoliverau> +1
21:26:52 <notmyname> onovy: thanks. but remind him that babies come before code ;-)
21:26:58 <onovy> :]
21:27:13 <clayg> onovy: if at any point Pavel wants to restore his authorship of the four spit patches - I really only consider that I have split his work on his behalf
21:27:28 <clayg> i am but a humble servent and beg forgiveness for my hubris
21:27:54 <notmyname> clayg: you can `git commit --ammend --author="foo bar <foo@bar>"` to change the commit
21:27:55 <clayg> oh goodness the poor bloke is replicating!?
21:28:16 <notmyname> ok, moving on!
21:28:22 <notmyname> kota_: any progress on global ec this week?
21:28:24 <onovy> clayg: Co-Author: Pavel Kvasnička <pavel.kvasnicka@firma.seznam.cz> // so i'm fine with authorship
21:28:50 <kota_> notmyname: unfortunately nothing except i rebased it onto the newest master
21:29:02 <onovy> clayg: but yep. if it's just split, you can --amend author and push it yourself :)
21:29:05 <kota_> notmyname: i really want volunteer for reviews.
21:29:06 <onovy> move on, sry
21:29:21 <kota_> volunteers
21:29:44 <notmyname> specifically for https://review.openstack.org/#/c/219165/
21:29:57 <kota_> yes, that one.
21:30:12 <notmyname> EC fragment duplication. can anyone volunteer to review this patch, please?
21:30:23 <clayg> I obviously should step up - i have it loaded in my brain form... Austin?
21:30:35 <clayg> no... San Antonio
21:30:38 <notmyname> timburke: will you have a chance?
21:30:41 <clayg> I remember looking at it in the hotel lobby
21:30:49 <timburke> i was just thinking about that. sure
21:30:57 <notmyname> timburke: awesome, thanks
21:31:01 <kota_> clayg: yeah, that review was helpful to progress, thx.
21:31:05 <clayg> I have bandwidth this week (and into early next) for reviews - i'd like to spend it there for sure.
21:31:24 <notmyname> clayg: cool, thanks
21:31:42 <notmyname> jrichli: symlink progress from this week. what's up?
21:32:03 <jrichli> thanks for all the great feedback on the api patch.
21:32:19 <jrichli> i am squashing it into the implementation patch now.
21:32:24 <notmyname> ah, cool
21:32:32 <notmyname> and you've mapped that into a trello board right? https://trello.com/b/oQHkQuV9/swift-symlinks
21:32:37 <notmyname> either you or tdasilva did
21:33:07 <jrichli> yes, tdasilva created cards for outstanding issues
21:33:18 <clayg> wtg tdasilva!
21:33:31 <notmyname> thankd tdasilva
21:33:42 <tdasilva> just copy and paste from clayg's awesome comments ;)
21:34:10 <notmyname> if anyone wants access to that board, please ping me on irc with your trello username and i'll add you
21:34:22 <notmyname> access == write access. IIRC it's public read
21:34:51 <notmyname> jrichli: acoles: how was the keymaster v2 call last week? anything to report from that?
21:35:11 <kota_> notmyname: thanks for adding me ;-)
21:35:25 <kota_> notmyname: and i think m_kazuhiro may want to join?
21:35:29 <kota_> m_kazuhiro: ?
21:35:33 <clayg> tdasilva: I told you that was a roast right?  I just trying to keep notes from torgomatic timburke and notmyname !
21:35:52 <m_kazuhiro> kota_: yes. I want to join.
21:36:32 <acoles> notmyname: call was good, thanks tdasilva for hosting
21:36:34 <jrichli> notmyname: it went well.  acoles created an etherpad and recorded the results.
21:36:38 <acoles> notmyname: https://etherpad.openstack.org/p/swift_keymaster
21:36:42 <notmyname> thanks
21:36:48 <notmyname> any summary?
21:37:02 <acoles> and for the keen there is a link at bottom for call replay courtesy of tdasilva
21:37:07 <notmyname> or what's next?
21:37:39 <acoles> summary each keymaster impl loads key from one source. keep it simple.
21:37:39 <jrichli> is mathiasb here?
21:38:05 <mathiasb> notmyname: I tried to address the comments from the call and on gerrit, new version is available for review
21:38:18 <notmyname> mathiasb: what's the link to the patch?
21:38:32 <mathiasb> https://review.openstack.org/#/c/364878/
21:39:04 <notmyname> mathiasb: great, thanks. so the current status is needing reviews? or are you expecting more dev work first?
21:39:37 <mathiasb> it is ready for reviews
21:39:40 <notmyname> ok, great
21:39:50 <notmyname> bkeller`: any progress on notifications this week?
21:39:59 <notmyname> any updates from last week?
21:40:14 <bkeller`> not really
21:40:17 <notmyname> ok
21:40:37 <notmyname> tdasilva: what's the status of pyeclib/libec merge? any progress for merging and keeing history?
21:42:09 <notmyname> tdasilva: ? (you're talking in -swift) ;-)
21:42:09 <tdasilva> notmyname: no progress on that this week
21:42:15 <notmyname> ok :-)
21:42:17 <clayg> notmyname: IIRC onovy told us to keep them seperate and pyeclib should support old versions of liberasurecode and I sorta had a conniption
21:42:36 <notmyname> tdasilva: anything we can do to help?
21:42:47 <tdasilva> notmyname: i just need to ping infra again
21:42:50 <notmyname> clayg: yes, I remember that :-)
21:42:56 <clayg> onovy: sorry about that :\
21:43:37 <onovy> i don't have problem with merging, if there will be correctly working build scripts for both
21:43:46 <onovy> and i can than build two package, so same as now
21:44:05 <tdasilva> onovy: i created this repo: https://github.com/thiagodasilva/libec
21:44:07 <onovy> i will "lock" version between this two packages, so same version will be installed
21:44:27 <notmyname> onovy: that makes sense. one code repo, two deliverables
21:44:28 <onovy> tdasilva: i will take a look
21:44:37 <tdasilva> based on the talks i had with infra folks, that's how i understood it should look like at the end
21:45:22 <onovy> tdasilva: why not just put two dirs inside one repo and have that projects "directory separated"?
21:45:44 <tdasilva> onovy: it has to do with the gate jobs and where they look for certain files
21:45:53 <onovy> ah, right
21:45:59 <tdasilva> my initial plan was sort of what you described
21:46:09 <onovy> that's strange
21:46:24 <onovy> you can use one build system for one language only
21:47:15 <notmyname> joeljwright: how's the pre/post -amble patch going?
21:47:33 <joeljwright> I'm still working on the comments from timburke/jrichlie
21:47:40 <notmyname> joeljwright: ok
21:47:43 <joeljwright> I have one qn for general consumption
21:47:47 <notmyname> ok
21:47:54 <joeljwright> would anyone mind reducing min seg size to 0 bytes?
21:48:02 <onovy> tdasilva: if i understand correctly: make will build liberasurecode only, and python setup.py python-pyeclib only, right?
21:48:14 <clayg> tdasilva: but that's just merging the repo's right?  it's not really making the changes in pyeclib that turn it into a true python-liberasurecode - the c code still does a dynmic reference to whatever .so you have in the LDPATH or whatever?  you can publish a python-liberasurecode wheel to pypi and have it installed with pip?
21:48:49 <tdasilva> onovy: right
21:48:55 <notmyname> joeljwright: yeah, timburke asked me about that
21:49:03 <onovy> tdasilva: that's really confusing to have it in one directory :/
21:49:20 <notmyname> I think that if we actually write down the zero-byte chunk and jsut optimize out the read, I'm ok with that (IIRC our discussion)
21:49:21 <onovy> just because "gates want it this way"
21:49:30 <joeljwright> notmyname, excellent
21:49:35 <joeljwright> that's what I've been working on
21:49:48 <jrichli> joeljwright: I would expect torgomatic might an opinion.  it'd be good to get his 2 cents
21:49:48 <tdasilva> clayg: right, have not attempted to change any behavior yet, thought it could be done on a follow-up change?
21:49:50 <onovy> clayg: i still need dynamic reference :]
21:49:54 <timburke> notmyname: joeljwright: that still gets a little weird as you can't necessarily COPY an slo you can GET
21:50:04 <clayg> onovy: lol - "why do it wrong just cause gates want it that way" - you are one of us
21:50:05 <notmyname> joeljwright: I'm also pretty sure that others are likely to have a different opinion. ;-)
21:50:12 <joeljwright> :D
21:50:21 <joeljwright> well, I can only propose it
21:50:24 <onovy> clayg: :] maybe we can create some kind of workaround?
21:50:35 <onovy> put it into subdirectory, but add makefile + setup.py into root
21:50:40 <joeljwright> I'm aiming to have something for comments by Friday
21:50:44 <notmyname> joeljwright: great
21:50:46 <onovy> make just width 'cd ... ; make'
21:50:56 <onovy> and something similar for setup.py
21:51:01 <onovy> *with
21:51:10 <clayg> onovy: why do you want it dynamic?  I thougth you *just* said you have to lock the packages together?
21:51:34 <onovy> clayg: yep. because i will lock packages together i don't want to have same ELF inside two packages
21:51:47 <onovy> statical linked inside python_c and inside liberasurecode for "C users"
21:52:16 <clayg> which is my whole beef - I *want* all of the ec code tightly coupled - I can only barely stand swift "supporting" multiple versions of pyeclib because we build libec from source and bump pyeclib in our requirements like we own the whole damn thing
21:52:20 <notmyname> I want to mention one last thing before we're completely out of time. the TC approved https://governance.openstack.org/tc/reference/new-language-requirements.html this week. it's a process for getting not-python into openstack code repos
21:52:31 <clayg> drives me nuts that pyeclib and liberasure code don't even *try* to say they're tightly coupled
21:52:33 <joeljwright> \o/
21:52:39 <timburke> oh! joeljwright! you might have some insight into how best to record the fact that an object is a symlink in the container db :-) i was suggesting stuff it in the etag (or at least, the etag that gets sent to the container server)
21:52:48 <onovy> clayg: right. package lock will tightly couple it
21:53:09 <onovy> *package version lock
21:53:17 <clayg> onovy: what is the benifit of the project "supporting" loose coupling - and then requiring packagers to implement tight coupling?
21:53:32 <joeljwright> timburke: I'd need to think about that one!
21:53:43 <mattoliverau> timburke: shouldn't you prepend an 'l' to it or something :P
21:53:43 <onovy> clayg: benefit is simple. we don't want to have same binary inside two packages
21:53:47 <notmyname> (all agenda items have been brought up, but I don't want to cut off the packaging discussion before our time is up)
21:53:49 <onovy> deduplication :]
21:54:01 <onovy> notmyname: clayg: let's continue in -swift?
21:54:08 <timburke> joeljwright: for some reason i had it in my head that you'd done similar things...
21:55:10 <joeljwright> I have abused the content type in middlewares, but nothing I'd suggest in public :)
21:55:15 <notmyname> lol
21:55:44 <notmyname> joeljwright: to echo timburke, yeah you're name came up when we were talking about how to (if to) expose symlinks in listings. we'd love your input
21:56:06 <notmyname> ok, I think that's it
21:56:19 <notmyname> thanks, everyone, for coming and participating. thank you for working on swift
21:56:23 <notmyname> #endmeeting