21:01:03 <notmyname> #startmeeting swift
21:01:04 <openstack> Meeting started Wed Sep 23 21:01:03 2015 UTC and is due to finish in 60 minutes.  The chair is notmyname. Information about MeetBot at http://wiki.debian.org/MeetBot.
21:01:06 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
21:01:09 <openstack> The meeting name has been set to 'swift'
21:01:13 <notmyname> swift meeting time! who's here?
21:01:15 <wbhuber> o/
21:01:17 <jrichli> here
21:01:17 <cschwede> o/
21:01:20 <mattoliverau> o/
21:01:20 <peluse> hola
21:01:21 <minwoob> o/
21:01:21 <hurricanerix> o/
21:01:22 <cutforth> o/
21:01:23 <tdasilva> hello
21:01:24 <ho> hi
21:01:30 <kota_> Hi
21:01:46 <notmyname> torgomatic: clayg: ?
21:02:04 <torgomatic> ‽
21:02:06 <notmyname> acoles_: said he'd be a few minutes late
21:02:09 <MooingLemur> 🐄
21:03:21 <notmyname> torgomatic: do you see clayg in the office? (assuming you're there)
21:03:33 <torgomatic> notmyname: yeah, he's over there
21:03:34 <acoles> here
21:03:44 <notmyname> ok, let's get started then
21:03:49 <notmyname> so, Liberty is upon us
21:03:53 <notmyname> all together now:
21:03:59 <notmyname> AAAAAUGHHHHH!
21:04:10 <notmyname> /thatsoutoftheway
21:04:21 <mattoliverau> lol
21:04:32 <torgomatic> hooray, another time-based release
21:04:37 <notmyname> so yeah, we've got to cut our liberty release at the end of next week
21:04:38 <torgomatic> deadline-driven development
21:04:51 <peluse> 3 cheers for technical debt
21:05:13 <notmyname> yeah, I wonder if the concept of shorter cycles would go over with the rest of the world
21:05:21 <notmyname> oh well...
21:05:25 <torgomatic> hip hip... segmentation fault
21:05:32 <jrichli> lol
21:05:58 <mattoliverau> lol, is there something in the water over in the US today, you all seem extra sarcastic :P
21:05:59 <notmyname> looking at what we've got left and what we want our collective name on for the release, I've put together a list of priorities for this week and next
21:06:09 <notmyname> #link https://wiki.openstack.org/wiki/Swift/PriorityReviews
21:06:19 <notmyname> I want to go through that list today
21:06:33 <notmyname> #topic patch 226707
21:06:34 <patchbot> notmyname: https://review.openstack.org/#/c/226707/
21:06:44 <notmyname> this patch has a little history
21:06:52 <notmyname> so it was filed as a security bug
21:07:10 <notmyname> but a few hours ago donagh published the patch publicly to gerrit
21:07:27 <notmyname> so we need to make sure we resolve it quickly and get it in the release
21:07:32 <notmyname> cschwede has looked at it
21:07:51 <notmyname> and it looks like it needs another patch set
21:07:53 <cschwede> … and will publish a new patchset in the morning, if there is no change inbetween
21:08:02 <notmyname> cschwede: thanks
21:08:21 <notmyname> I'd like to have another core on it too, so we can land it quickly
21:08:30 <notmyname> preferably not someone who is neck-deep in EC work
21:08:39 <acoles> i was about to offer
21:08:43 <mattoliverau> I'll take a look
21:08:43 <peluse> no!
21:08:50 <notmyname> thanks mattoliverau
21:08:54 <notmyname> acoles: heh
21:08:59 <peluse> hey mattoliverau you still have to look at patch 211338 also :)
21:09:00 <patchbot> peluse: https://review.openstack.org/#/c/211338/
21:09:11 <notmyname> yeah, mattoliverau has several things :-)
21:09:15 <acoles> i am a little familar with it
21:09:23 <notmyname> so between mattoliverau and cschwede, get it reviewed and landed asap
21:09:25 <cschwede> acoles: mattoliverau: that would be great, because we’ll be online in a few hours again ;)
21:09:27 <mattoliverau> yeah, I'll look at all the things!
21:09:37 <peluse> go mattoliverau!
21:09:49 <acoles> cschwede: is it really just a few hours ;)
21:09:59 <notmyname> #topic patch 222799
21:09:59 <patchbot> notmyname: https://review.openstack.org/#/c/222799/
21:10:10 <clayg> ohia
21:10:11 <notmyname> this patch has to do with some ring stuff. which is hairy
21:10:35 <notmyname> clayg: is looking at this. with (I think) torgomatic and maybe timburke
21:10:57 <notmyname> clayg: what do you need on this? where do you need help? what's the high-level summary?
21:11:19 <notmyname> and is it going to require another patch for the duplicate parts to land next week too?
21:11:53 <clayg> notmyname: so i'm not really looking at that patch anymore
21:12:02 <notmyname> heh, ok
21:12:06 <notmyname> new plan!
21:12:09 <notmyname> what is it?
21:12:10 <clayg> notmyname: I *thought* that if we just validated against it we'd be fine since it probably only comes up if your ring topology is stupid
21:12:33 <clayg> but it turns out the stupidity of your topology is fairly wide and I'd rather just not do that anymore
21:13:28 <clayg> so I'm pretty sure I'm going to add some specific fixes for that particular bad and then add the validate
21:13:56 <clayg> I'm trying *not* to get caught in the ever elusive general solution which has tempted so many of our brave soldiers
21:14:11 <cschwede> clayg: so that means you’re still working on it? or is it ready for review?
21:14:27 <notmyname> cschwede: I think it might mean we don't care about that particular patch
21:14:27 <clayg> but it's really hard!  when you see the parts are just not *quite* right it's very tempting to add a sort or check a tier and look and a weight and tweak tweak tweak
21:14:48 <cschwede> notmyname: k
21:14:50 <peluse> cschwede: I speak clayg, let me help.. "no, I'm not"
21:15:00 <clayg> cschwede: I'll *care* about it - in that it'll be the first in a series of a chain of 2 or 3 patches
21:15:08 <clayg> 1) blow if if we do something stupid
21:15:12 <clayg> 2) don't do something stupid
21:15:14 <clayg> 3) do something smart
21:15:22 <MooingLemur> I don't think it's as simple as "bad" topology.  At this very moment I can't seem to get a 4-device (two host) three replica ring of equal weights not to end up with duplicate replica assingments.
21:15:22 <clayg> 3 is optional
21:15:33 <clayg> MooingLemur: quite right sir!
21:15:44 <clayg> hince 1) has been deamed not sufficient!
21:15:48 <clayg> but maybe nessecary
21:16:00 <notmyname> got it
21:16:17 <notmyname> so patch 222799 should still be reviewed, but there will be at least one more after it
21:16:17 <patchbot> notmyname: https://review.openstack.org/#/c/222799/
21:16:37 <notmyname> right?
21:16:43 <clayg> notmyname: yeah - but i'm going to WIP it because i'm finding some more tests with stupid rings that I want to fix in the first batch
21:16:51 <notmyname> ok
21:17:12 <clayg> basically once we add this check I want every test to be solvalbe - i've found a few that don't call validate and hence weren't currently blowing up even after adding the check
21:17:21 <notmyname> so regardless of it being WIP, we need some other cores (and non-cores) to keep up with it to be ready to review asap
21:17:29 <notmyname> any volunteers?
21:17:36 <cschwede> o/
21:17:38 <notmyname> normally for ring stuff I'd say torgomatic and cschwede
21:17:46 <clayg> cschwede: thanks!  torgomatic is probably on the hook too :P
21:17:49 <notmyname> cschwede: thanks
21:17:56 <clayg> torgomatic: don't try to back away now
21:18:06 <notmyname> :-)
21:18:08 <clayg> notmyname: he said "I'm really curious to see what you come up with"
21:18:17 <clayg> notmyname: but it sure sounded like he said "good fucking luck smartass"
21:18:26 <notmyname> clayg: that pretty much means he has to co author it, right? ;-)
21:18:40 <clayg> notmyname: only if he wants to keep me from doing something horribly stupid
21:18:46 <notmyname> heh
21:18:53 <notmyname> ok, we've got that patch triaged
21:18:57 <notmyname> now the next hairy one
21:19:03 <notmyname> #topic path 215276
21:19:06 <clayg> notmyname: it's not so bad - I think we can get in a world of better way shortly
21:19:09 <notmyname> acoles: this is yours
21:19:21 <notmyname> it's WIP and you and clayg have been beating on it for a while
21:19:29 <acoles> so this about being able to get an object that has missing durables
21:19:36 <notmyname> optimistic gets?
21:19:41 <acoles> yes its WIP but its getting close
21:19:46 <clayg> notmyname: acoles is a real champ - he's making tons of progress - peluse and I owe him much appreciation!
21:19:48 <acoles> yes optimistic gets
21:19:53 <peluse> and beer
21:19:59 <acoles> the optimistic part is simple
21:20:08 * peluse is optimistic
21:20:09 <clayg> peluse: i'll take some beer!  or saki or w/e we can find
21:20:15 <acoles> but being smart about what to do when it doesn't work out is harder
21:20:25 * acoles consumes anything in a glass
21:20:27 <peluse> oh yeah... so acoles are you ready for more eyes on it or do you need a bit more time?
21:20:31 <clayg> acoles: i threw in one more random idea that seems like it might be good in the last comment
21:20:38 <clayg> acoles: it's something torgomatic and I have been kicking around
21:20:39 <mattoliverau> there is beer in vending machines in Japan, so finding beer should be easy ;)
21:21:04 <acoles> peluse: yes eyes would be good, as long as you're ok with things changing after you thought they were great already!
21:21:11 <acoles> clayg: ok. thanks
21:21:16 <peluse> story of my life!
21:21:21 <acoles> peluse: heh
21:21:37 <notmyname> ok, so peluse will be looking at this with acoles
21:21:42 <peluse> for sure
21:21:46 <notmyname> clayg: you're doing ring stuff instead of this, right?
21:22:05 <peluse> I heard "both" like bo Jackson type shit
21:22:30 <notmyname> kota_: could you take some time to look at this patch and help out where possible?
21:22:34 <clayg> notmyname: indeed I am - but I feel very guilty about it!  (and the 100 other things that I would love to do)
21:22:37 * clayg is such a poser
21:22:47 <clayg> kota_: !!!
21:22:48 <kota_> ok
21:22:50 <acoles> turns out that i got the alt-frags thing that peluse started working on for free with 21526
21:22:57 <notmyname> clayg: it's good you're looking at the ring stuff
21:22:59 <clayg> notmyname: kota_ has his *own* awesome patches that need review - minwoo too
21:23:00 <notmyname> kota_: thanks
21:23:05 <peluse> FREE!  Wheeeeee!
21:23:08 <kota_> patch 186735
21:23:08 <patchbot> kota_: https://review.openstack.org/#/c/186735/
21:23:15 <clayg> mattoliverau: acctually it may be more helpful to t... oh shit - you already +2'd those :\
21:23:21 <notmyname> hang on that's next :-)
21:23:30 <peluse> what was the middle thing?
21:23:47 <clayg> which middle?
21:23:50 <notmyname> #topic patch 186735
21:23:50 <patchbot> notmyname: https://review.openstack.org/#/c/186735/
21:23:55 <peluse> never saw fish called wanda huh?
21:24:05 <notmyname> this is kota's patch to fix the possible missing container updates
21:24:14 <notmyname> mattoliverau has already been looking at this
21:24:31 <notmyname> I had just started before the meeting, so I'll take it too
21:24:31 <kota_> yeah, that's one I am working.
21:24:58 <acoles> ohyeah thats important
21:25:01 <clayg> oh yeah that was it - mattoliverau saw some better to grab
21:25:06 <notmyname> kota_: is there anything you need on this one? I see that mattoliverau has some suggestions for the next patch set
21:25:35 <kota_> I think, what I have to do is fixisng according matt's comment.
21:25:44 <kota_> that's all for this one, maybe?
21:25:48 <notmyname> ok, great
21:26:01 <notmyname> we'll get it reviewed as soon as you get the new patch set up
21:26:13 <notmyname> #topic patch 211338
21:26:13 <patchbot> notmyname: https://review.openstack.org/#/c/211338/
21:26:15 <kota_> ok, I'll push it after this meeting..
21:26:19 <peluse> so kota looked at this as did mattoliverau and clayg added some rockin updates.  I think its ready for review although there's a small grammar thingy in the commit message
21:26:22 <notmyname> peluse: this one is yours. same questions
21:26:28 <notmyname> ok
21:26:33 <peluse> answers before you asked.... ^
21:26:34 <notmyname> commit message grammar is a nit
21:26:42 <peluse> yeah that's why I left it
21:26:42 <notmyname> who cares unless there is another patch set
21:27:01 <notmyname> so kota_ and mattoliverau are on this patch as reviewers?
21:27:09 <peluse> if mattoliverauand kota can give it a possible final once over its done
21:27:09 <clayg> notmyname: idk, i like commit message grammer to be correct :\
21:27:09 <kota_> yup
21:27:20 <notmyname> clayg: you of all people?! ;-)
21:27:20 <clayg> is there a nit on that one - i'm sure me or paul could fix it
21:27:21 <mattoliverau> yup
21:27:26 <peluse> clayg: I think you messed up the commit message actually :)
21:27:38 <clayg> notmyname: no I mean - i just get embarassed when I say something stupid in a commit message :\
21:27:45 <peluse> I can fix for sure and need to add clayg as author as well
21:28:04 <clayg> so when someone spots it for me I *prefer* to fix it on my patches - but I almost *never* spot it in others - i read to fast - go stright to intent
21:28:08 <peluse> I'l do that real quick here, no cod echanges though
21:28:15 <notmyname> ok, fine. there's not a +2 on it already so no real harm
21:28:27 <notmyname> kota_: are you ok to review this one too?
21:28:52 <kota_> ok
21:28:57 <notmyname> thanks
21:29:13 <notmyname> good news! that's the end of the "Critical" list
21:29:20 <notmyname> bad news! there's quite a few of them
21:29:30 <notmyname> #topic patch 220059
21:29:31 <patchbot> notmyname: https://review.openstack.org/#/c/220059/
21:29:42 <kota_> quickly my turn!
21:29:56 <notmyname> high priority, this is kota_'s patch so that bad status codes don't have a quorum of good
21:30:16 <kota_> oh, yeah simlar one with previous container update
21:30:27 <notmyname> kota_: any updates here or is it ready for review?
21:30:28 <kota_> but this is a part of successful commit.
21:30:36 <clayg> kota_: should this be critical?
21:30:48 <kota_> might be
21:30:52 <clayg> kota_: I don't really understand the implications - it'd be easier on me if you just told me
21:31:02 <kota_> and just waiting to be reviewed.
21:31:27 <kota_> clayg: nice
21:32:10 <kota_> the patch cares of the handling of quorum before commits.
21:32:59 <kota_> currently the quorum check before commit put might think quorum 4xx as success and will put commit.
21:33:11 <notmyname> yeah, probably should be moved up to critical
21:33:14 <notmyname> acoles: torgomatic: can you take a look at this one for review?
21:33:23 <clayg> not acoles
21:33:24 <acoles> yup
21:33:29 <acoles> nope
21:33:32 <notmyname> heh
21:33:33 <clayg> lol
21:33:39 <torgomatic> what?
21:33:41 <acoles> yes i can
21:34:01 <kota_> thanks.
21:34:03 <notmyname> torgomatic: can you review this patch?
21:34:14 <torgomatic> sure
21:34:25 <notmyname> acoles: torgomatic: thanks
21:34:39 <notmyname> #topic patch 211726
21:34:40 <patchbot> notmyname: https://review.openstack.org/#/c/211726/
21:34:47 <notmyname> this is a lower priority, difinitely
21:34:52 <notmyname> cool new feature
21:35:03 <notmyname> already has 1 +2, and I think mattoliverau is looking at it today
21:35:13 <mattoliverau> yeah I am
21:35:23 <notmyname> however, if it doesn't land today or tomorrow, I think it probably drops the list
21:35:32 <mattoliverau> I see a new patchset is up, so I'll probably do that first thing this morning while it's till in my brain
21:35:48 <notmyname> #topic patch 177195
21:35:48 <patchbot> notmyname: https://review.openstack.org/#/c/177195/
21:35:49 <acoles> mattoliverau: what are you *not* doing today? :)
21:36:01 <notmyname> another kota_ patch
21:36:10 <notmyname> this one has a merge conflict on it, and it's set as low priority
21:36:21 <kota_> oh, yeah
21:36:27 <mattoliverau> acoles: lol
21:36:28 <kota_> I'm not working on this for now.
21:36:41 <notmyname> ok
21:36:51 <wbhuber> kota_: i looked at the patch - it needs a lot of rebasing and some change in the code to prevent decode when there's an invalid EC scheme
21:37:06 <kota_> exactly
21:37:13 <wbhuber> kota_: on top of the changes introduced by etag buckets, etc. etc.
21:37:23 <notmyname> then at this point, I think we should drop it from the list of liberty patches
21:37:27 <notmyname> any objections?
21:38:18 <kota_> It's ok to drop this, I have to reconsider the way to validate that.
21:38:21 <notmyname> ok
21:38:33 <notmyname> #topic patch 196848
21:38:33 <patchbot> notmyname: https://review.openstack.org/#/c/196848/
21:38:39 <notmyname> minwoob: this one is yours
21:38:48 <notmyname> it's also set at low priority. is that correct?
21:39:02 <notmyname> minwoob: what is it doing, what do you need for it?
21:39:21 <acoles> kota_: we might be able to add that feature into the new get/etag bucket class in patch 21526
21:39:21 <patchbot> acoles: https://review.openstack.org/#/c/21526/
21:39:30 <acoles> patch 215276
21:39:30 <patchbot> acoles: https://review.openstack.org/#/c/215276/
21:39:59 <notmyname> minwoob: ?
21:40:16 <kota_> acoles: thanks, I'll take a look at that!
21:40:16 <notmyname> perhaps he stepped away
21:40:22 <wbhuber> notmyname: checking on minwoob....
21:40:25 <notmyname> cschwede: can you look at minwoob's patch for review?
21:40:33 <wbhuber> notmyname: IRC stopped working for minwoob
21:40:38 <notmyname> ah
21:40:52 <notmyname> wbhuber: do you know the status of the patch?
21:40:58 <wbhuber> notmyname: a second...
21:41:01 <notmyname> ok
21:41:10 <cschwede> notmyname: i’ll have a look
21:41:11 <wbhuber> notmyname: minwoob said it's ready for rewview.
21:41:21 <clayg> notmyname: last I looked it was in pretty good shape
21:41:22 <notmyname> wbhuber: thanks for the relay :-)
21:41:25 <notmyname> cschwede: thanks
21:41:41 <wbhuber> notmyname: "it's an optimization moreso than a make-everything-ok fix
21:41:54 <notmyname> ok, makes sense as low priority then
21:42:01 <notmyname> #topic etc
21:42:12 <notmyname> ok, those are that patches that I'm tracking for liberty
21:42:23 <notmyname> we've got cores signed up for review on them. thank you
21:42:32 <notmyname> so what else?
21:42:34 <peluse> EC perf
21:42:53 <wbhuber> peluse: well done on compiling the results (a living one tho)
21:42:55 <notmyname> wbhuber: minwoob: ho: your help would be greatly appreciated on these patches too
21:42:58 <peluse> so I need to update the report out and post and have some discussion.  FYI the 50/50 numbers were all half of reality (my error making the graphs)
21:43:25 <peluse> not sure if everyone saw them but watch IRC later and I'll post the next ver on google doc
21:43:33 <notmyname> peluse: cool
21:43:46 <peluse> the one concerning thing is the amount of mem used at a storage node - need to dig into that.  it was was higher than I epxecte
21:44:28 <jrichli> It would be great to get some feedback on Conditional GETs latest proposal: https://etherpad.openstack.org/p/swift_encryption_issues
21:44:43 <peluse> other than that I think it suports "yes, Ec is usable and yeah it makes more sense for larger objects but still does pretty well with smaller (4MBish) as well"
21:44:58 <acoles> jrichli: ack
21:44:58 <notmyname> peluse: that's good news :-)
21:45:09 <peluse> plus I'll run once more test to prove the suggstion on segment size (proxy chunk * n_data is best)
21:45:45 <peluse> torgomatic: if you have any thoughts on how the mime additions to obj setver PUT might dramatically increase mem usage let me know :)
21:45:51 <jrichli> acole: thx
21:45:53 <notmyname> as a reminder, any critical bugs on https://bugs.launchpad.net/swift will block a release
21:46:06 <torgomatic> peluse: I can't see it being that much, but what do I know?
21:46:09 <peluse> torgomatic: I have no evidence of that BTW, just thinking out loud.  Tomorrow I devise some tests to try and see where the hog is
21:46:22 <minwoob_> peluse: With the new recommendation on segment size, are we still going with object size < segment size as the threshold for using replication within EC policies?
21:46:35 <peluse> whoa, huh?
21:46:42 <clayg> wait - what's the new recommendation on segment size?
21:46:43 <peluse> replication within Ec policies?
21:46:49 <notmyname> minwoob_: none of that has been implemented yet
21:47:06 <tdasilva> peluse: I don't remember if it stated on the ppt, but is it safe to assume it is using the intel ec library? are there any comparisons between other alg?
21:47:07 <peluse> clayg: thinking that its proxy chunk size * n_data
21:47:16 <clayg> i think if you sub within/instead that sentence parses better for me
21:47:33 <peluse> tdasilva: not in this report but there are lots of micro reports on the librariies, they're all very similar
21:47:36 <notmyname> tdasilva! I didn't see you there :-)
21:48:01 <minwoob> The plan was to store objects less than a certain size, with replication, rather than EC, even if a client had specified a particular EC scheme.
21:48:04 <peluse> tdasilva: but yes my testing was using isa-l
21:48:05 <notmyname> tdasilva: isa-l is better. unless you're using amd chips. then it's the same
21:48:13 <clayg> peluse: wow that's a really specific suggestion - and makes it hard to tweak your proxy_chunk_size - but i guess most folks just eat the default neway
21:48:21 <peluse> minwoob: we're a ways from doing something like that automatically
21:48:21 <tdasilva> peluse, notmyname: got it
21:48:24 <minwoob> So that's what I mean by 'threshold'
21:48:27 <peluse> minwoob: right now its one or the other
21:48:59 <peluse> clayg: I think thinking you'd set your Ec segment size based on what proxy chunk is set to and what ratio you choose
21:49:18 <notmyname> tdasilva: I didn't sign you up for any of the patch reviews ;-)
21:49:29 <peluse> clayg: so all defaults considered if using 10:14 you're better off using 640K segment isntead of 1MB segment (per data)
21:49:42 <tdasilva> notmyname: i was planning to look at Tim's SLO one
21:49:58 <acoles> tdasilva gets everything else ;)
21:49:58 <tdasilva> unless mattoliverau merges that first
21:50:02 <notmyname> tdasilva: I'm hoping that lands very very soon (as soon as mattoliverau looks at it again)
21:50:18 <peluse> clayg: wrt the comment on it beign very specific, keep in mind that 1MB was pulled out of our asses 18 months ago :)
21:50:51 <notmyname> tdasilva: patch 186735 and patch 211338 are where I'd start. or helping acoles with patch 215276
21:50:51 <patchbot> notmyname: https://review.openstack.org/#/c/186735/
21:50:51 <torgomatic> pseudorectal number generation
21:50:53 <patchbot> notmyname: https://review.openstack.org/#/c/211338/
21:50:54 <patchbot> notmyname: https://review.openstack.org/#/c/215276/
21:51:12 <acoles> torgomatic: LOL
21:51:20 <mattoliverau> maybe tdasilva can also look at the cross-reseller COPY thing, he is copy middlewaer man. :P
21:51:27 <notmyname> oh yeah!
21:51:32 <tdasilva> lol
21:51:34 <notmyname> anything else from anyone this week? any questions on what to work on?
21:51:37 <clayg> well and I think we got pretty close to 640 guessing at 1000 - i'm just wondering how much better 640 is than 512 and if an 12:4 would really benifit from approaching backup twoards 1MB
21:51:58 <peluse> clayg: see the slides... I tested both.  not a HUGE gain but better
21:52:11 <clayg> peluse: great!
21:53:06 <peluse> clayg if 12:4 one would think 768 would be only slightly better than 1MB
21:53:10 * tdasilva is sorry for not keeping up with copy middleware...been consumed by swift-on-everything recently
21:53:29 <notmyname> please don't hesitate to reach out to me (or anyone) this week if you have questions. ask earlier so we make best use of the time we have left
21:53:37 <mattoliverau> tdasilva: your punishment is reviews! :P
21:53:38 <notmyname> thanks for coming, and thanks for working on swift
21:53:45 <peluse> rock and rolla!!!
21:53:47 <notmyname> #endmeeting