21:00:27 <notmyname> #startmeeting swift
21:00:28 <openstack> Meeting started Wed Feb  8 21:00:27 2017 UTC and is due to finish in 60 minutes.  The chair is notmyname. Information about MeetBot at http://wiki.debian.org/MeetBot.
21:00:29 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
21:00:31 <openstack> The meeting name has been set to 'swift'
21:00:36 <notmyname> who's here for the swift team meeting?
21:00:38 <mattoliverau> o/
21:00:42 <ntata> hello
21:00:42 <m_kazuhiro> o/
21:00:45 <kota_> hi
21:00:45 <timburke> o/
21:00:56 <mmotiani1> Hi
21:00:57 <cschwede_> o/
21:00:58 <mathiasb> o/
21:01:04 <jrichli> o/
21:01:13 <clayg> PARTY TIME
21:01:35 <notmyname> wooo
21:01:40 <notmyname> welcome, everyone
21:01:40 <dmorita_> hi
21:01:45 <notmyname> agenda this week is at...
21:01:50 <notmyname> #link https://wiki.openstack.org/wiki/Meetings/Swift
21:02:00 <tdasilva> hi
21:02:06 <notmyname> sure would be nice to be able to do #links inline
21:02:22 <acoles> o/
21:02:23 <notmyname> #topic PTG
21:02:27 <notmyname> let's talk PTG to start with
21:02:43 <notmyname> we've got a week and a half before the PTG in atlanta
21:02:56 <notmyname> the etherpad is coming along nicely
21:02:57 <notmyname> #link https://etherpad.openstack.org/p/swift-ptg-pike
21:03:13 <notmyname> so please continue to add stuff there (topics and details on existing topics)
21:03:31 <clayg> yeah that's looking good folks!
21:03:53 <joeljwright1> looks like I'm gonna miss out on some great discussions
21:03:54 <notmyname> as I've mentioned before, the plan at the PTG is to be very similar to previous in-person events: we'll move tables and have several conversations going on at once
21:04:18 <notmyname> so next week I will start grouping some of the topics together to make sure some of the big stuff avoids overlaps
21:04:37 <notmyname> joeljwright1: you will be missed
21:04:50 <notmyname> and joeljwright1's comment reminds me of one more thing...
21:05:14 <notmyname> I'm told that those of us attending the PTG will be getting a ticket to the boston summit (ie instead of a free ticket based on code contributions)
21:05:44 <notmyname> so, is there anyone here who is thinking they might be attending the boston summit but who will *not* be attending the PTG in atlanta?
21:05:55 <notmyname> if so, I have one ticket that I can give out
21:06:00 <notmyname> so please speak up
21:07:00 <notmyname> ok :-)
21:07:13 <mattoliverau> I guess people may not know if they can go yet
21:07:20 <notmyname> any other question about the PTG before we move on to the next topic?
21:07:32 <notmyname> 3...
21:07:39 <notmyname> 2...
21:07:45 <notmyname> 1...
21:07:51 <clayg> notmyname: will the PTG be awesome?
21:08:06 <mattoliverau> it will be what we make it :)
21:08:12 <notmyname> all in-person meetings of swift contributors are awesome
21:08:17 <notmyname> #topic release
21:08:27 <notmyname> we're getting close to the ocata release date
21:08:40 <notmyname> I have started taking a look at the changelog updates
21:09:00 <notmyname> and I expect to be able to propose that early next week
21:09:15 <notmyname> so... if it's gonna be in the release, it needs to land soon
21:09:27 <notmyname> there are no critical priority open bugs in LP
21:09:33 <clayg> woooo!1
21:09:41 <notmyname> so let's talk about the stuff on the priority reviews page
21:09:49 <notmyname> #link https://wiki.openstack.org/wiki/Swift/PriorityReviews
21:10:00 <notmyname> clayg: what about https://review.openstack.org/#/c/425493/
21:10:15 <notmyname> you've moved it to WIP
21:10:19 <notmyname> is there a differnet patch?
21:10:40 <clayg> so i cherry picked that change for my release - so i'm not under the gun anymore
21:10:55 <clayg> kota_: and others had some good thoughts for some direction we might evolve that patch
21:11:26 <clayg> i wanted to get some feedback from ops to test my experience - happy to be a unique snowflake - i think there's room for compromise
21:11:39 <clayg> any ops in the room?  ahale onovy briancline - who else?
21:11:46 <clayg> maybe cschwede_
21:11:53 <clayg> basically "how do you use handoffs_first for replication"
21:12:09 <clayg> IME, you turn it on when when you rebalance is going to slow and monitoring handoff parts closely
21:12:22 <clayg> we do all that work to warn you that it's on because its "a bad thing" to leave it running
21:12:25 <cschwede_> turn it on if you're running out of space and add new disks, turn it off if things settled down...
21:12:55 <clayg> so I haven't really had so much of a *problem* with replicated policies dropping back into sync mode after they finish their handoffs
21:13:06 <clayg> but I do feel like it's not efficient - esp when I'm adding new nodes
21:13:13 <cschwede_> true
21:13:33 <clayg> some nodes will start making nodes do suffix hashing before i'm really done with my "pushing out handoffs" from the cluster wide perspective
21:13:56 <clayg> when we first started talking about making handoffs_first work better for rebalance (because ust sorting the list didn't really work in the face of failures)
21:14:16 <clayg> everyone agreed we wnated handoffs_only - but then I wrote handoffs_first++ and it's more or less worked out
21:14:44 <clayg> but I'm backing off that idea for EC - because IME with these patches in a EC reblance that was way off the rails - I really really really didn't want to go back to syncing
21:14:58 <clayg> a sync is much more expensive in EC land than in replicated - so that's the main driver for the difference
21:15:19 <clayg> ... but there's a consistency issue ... what's easier for ops?  do other people have replicated and ec policies?
21:15:36 <clayg> I really want handoffs_only for EC and that's what's implemented
21:16:02 <clayg> i'm probably going to "deprecate" the broken EC handoffs_first mode and just call it handoffs_only - maybe i'll honor the old config setting for bckwards compat!?
21:16:10 <clayg> anyone have any opions
21:16:34 <notmyname> yeah, honoring the old setting is good. redefine it to handoffs_only
21:16:48 <notmyname> IMO you make a good case for handoffs_only over handoffs_first
21:16:49 <cschwede_> +1. sounds reasonable to me
21:16:58 <notmyname> kota_: you had comments in gerrit about this
21:17:02 <notmyname> kota_: what do you think?
21:17:13 <kota_> i support that idea to add handoff_only thinking of the efficiency of reconstruct
21:17:25 <kota_> notmyname: just was typing ;-)
21:17:29 <jrichli> I am with you on this idea.  change to handoffs_only for EC and honor old cfg
21:17:30 <notmyname> kota_: thanks :-)
21:17:34 <clayg> and for the replicator?  is good for the goose good for the gander?  or do we leave well enough alone?
21:18:37 <jrichli> that is a harder question, only because it changes an existing behavior.  but hopefully just for the better.  like default is now fast-post.
21:18:38 <mattoliverau> there was a note on the last ops feedback session about always trying to prioritise reverts over sync.. which make sense.. I know this has nothing to do with handoffs_only but do we still need to visit the idea of handoff_first in normal operation, or at least a better implementation of it?
21:18:45 <clayg> it's super trival to make handoffs_first handoffs_only in the replicator - and my experience and consumption of the option leads me to believe that's what i want
21:19:13 <kota_> for the replicator, i think it's nice to have but I'm still thinking of how it works with global cluster use case.
21:19:29 <cschwede_> i need to think more about that and test things with the replicator. sounds like a topic for the PTG?
21:19:33 <clayg> mattoliverau: yes - I think the whole engine (replicated & EC) needs to self manage prioritization of rebalance vs sync automatically
21:19:48 <cschwede_> or is there any urgent reason to include it in the next release? i mean the replicator change
21:19:49 <clayg> having ops flip into rebalance mode - then push rings - then after reblance - switch back to sync mode - is annoying/stupid
21:20:17 <notmyname> I kinda like the conceptual simplicity of handoffs_only vs handoffs_first. _first is a reordering and still tries to do a lot. _only is what I want! I have a problem and need to get handoffs off the drive, so do the handoffs_only. seems easier to document, therefore easier to grok, therefore *maybe* better
21:20:29 <clayg> like I said - EC rebalance is screwed - I needed handoffs_only for EC - i wrote it and included in my release - this only effects people not doing SwiftStack
21:20:52 <notmyname> ok, so let's tackle the replicator question later
21:21:05 <cschwede_> notmyname: that's true (the easier part)
21:21:22 <notmyname> for now, the important question is does patch https://review.openstack.org/#/c/425493/ need to land for this release or should it wait?
21:21:23 <clayg> notmyname: ok - so i'll respin the EC patch to name the option _only - and honor _first for backwards compat with a warning (you said _first - you get _only)
21:21:38 <clayg> then we'll maybe support handoffs_only in the replicator
21:21:43 <clayg> is that really less confusing?
21:21:48 <notmyname> hey, briancline just joined
21:22:00 <briancline> first things first: i didn't do it
21:22:07 <notmyname> lol
21:22:17 <clayg> notmyname: ah... i think if you're doing EC rebalance at scale you want something that lets you prioritize rebalance (revert) in a meaningful way
21:23:00 <clayg> briancline: I just have the strong opinion "no one" runs with handoffs_first mode turned on after the rebalance
21:23:11 <clayg> if you have it on - you know it - and want to turn it off as soon as the fire goes out
21:23:42 <clayg> for EC I need it to be a more agressive handoffs_only mode for "reasons" - there's some support that the same reasoning would apply to repicated
21:24:19 <clayg> basically replicated handoffs_first tends to start introducing update/suffix-sync jobs into the cluster maybe before all of the updated_deleted/handoffs-revert/rebalance jobs have finished on all the nodes
21:24:31 <clayg> and it might be better if those nodes that finish all their handoffs just cool their heels
21:24:47 <clayg> rather than spinning up a bunch more jobs and slowing everyone down
21:24:55 <clayg> but the risk is "what if you forget to turn it off and don't sync"
21:25:29 <clayg> so... if you have anything to say on that - that's the topic
21:25:45 <clayg> notmyname: but I think we already agreed waht happens with patch 425493
21:25:46 <briancline> after a crisis-induced rebalance? yeah, i definitely would not want it on after a fire is out
21:26:03 <notmyname> on the one hand, I want swift to "just work" over time and do the right thing. on the other hand, supporting "I wasn't paying attention and things broke" doesn't seem like a good idea?
21:26:09 <clayg> and if i can respin and get reviews it'll land - if not - EC rebalance at scale is probably a timebomb for another release
21:26:23 <mattoliverau> maybe this will be like part-power increase. step 1 lets have the mechanics. step 2 add more automation.
21:26:32 <clayg> mattoliverau: ++^
21:27:04 <notmyname> +1
21:27:05 <clayg> mattoliverau: in my situation - i'm very interested in making ec reconstruction magic - but having a leaver to throw when shit hits the fan was step 0
21:27:20 <clayg> so along that reasoning maybe replication is fine
21:27:36 <clayg> it has other advantages that make handoffs_only less *needed*
21:27:45 <notmyname> so clayg will attempt to respin https://review.openstack.org/#/c/425493/ to be handoffs_only and then we'll review and hope for it in the next release
21:28:05 <clayg> only last question is confusion and differences in the options and behavior between the two deamons
21:28:16 <clayg> notmyname: that sounds like the right plan
21:28:21 <clayg> sorry to take up so much time
21:28:26 <kota_> I'll be able to circleback if handoffs_only get into that
21:28:37 <clayg> sounds like everyone understand's why the patch implemented handoffs_only and why that's a good thing for EC
21:28:41 <notmyname> no worries. good conversation
21:28:43 <notmyname> yeah
21:28:45 <clayg> only questions left are about future work
21:28:46 <briancline> re-reading, i'm not sure i know what handoffs-revert above is in reference to, but i agree on the sentiments about using that option
21:29:10 <notmyname> briancline: copying the fragments to the right place instead of recreating them via the EC math
21:29:38 <clayg> notmyname: don't forget all the extra read IO to gather the k frags too!
21:29:56 <notmyname> yes. EC math + lots of other terrible
21:30:00 <clayg> rofl
21:30:06 <mattoliverau> lol
21:30:16 <notmyname> ok, I think we're good on this patch for this meeting
21:30:20 <notmyname> other stuff for the release
21:30:22 <clayg> g2g!
21:30:26 <briancline> oh, wow. yeah
21:30:30 <notmyname> we've been tracking and reviewing the part power increase from cschwede_
21:30:51 <notmyname> cschwede_: where are we? how's it going? what's next? should it land in the release?
21:31:02 <notmyname> https://review.openstack.org/#/c/337297/
21:31:09 <mattoliverau> I saw some more comments on it, and cschwede_ has raised a new patchset, so I'll take a look at it today
21:31:22 <mattoliverau> now that I'm back from vacation
21:31:30 <notmyname> thanks
21:31:35 <notmyname> and timburke has a +2 on it already
21:31:40 <cschwede_> i hope it lands... got some good feedback so far, and i think usability improved a lot in the last 1 or 2 weeks, thanks to the reviews
21:31:42 <notmyname> so that sounds like it's likely to be in the release
21:31:57 <mattoliverau> we'll make it fit ;)
21:33:08 <notmyname> but..
21:33:23 <notmyname> jsut to ask the hard questions (since it's easier before landing it than after)...
21:33:39 <notmyname> *should* it land in this release? are there any big concerns about the impact?
21:34:01 <notmyname> I'm not implying it shouldn't, but I want to make sure we're ok with landing this a week before ocata
21:34:40 <notmyname> (also i'm going to ask the same question about the global ec patch, so get ready kota_ ;-)
21:34:46 <clayg> i always like landing big stuff right *after* the release - I feel like that worked well for encryption
21:34:59 <clayg> I feel like with EC we found some sad stuff right *after* it hit master
21:35:04 <cschwede_> well - that's up to all reviewers, isn't it? i'm biased obviously ;)
21:35:07 <notmyname> however, this one has been discussed for a long time
21:35:09 <kota_> notmyname: probably it's not big impact but I'm worried a bit the increase part power looks conflicted with mine
21:35:12 <clayg> but there's lots of "pressure" out there
21:35:26 <tdasilva> i'm seeing a probe test failure with that one atm
21:35:30 <clayg> I'm acctually done with my release already - I feel master + the EC handoffs_only mode are a great release
21:35:35 <tdasilva> so i'd like to understand why...
21:35:39 <clayg> we got the suffix hashing fix - bunch of other good stuff
21:35:54 <cschwede_> if there are concerns, postpone it and discuss at the PTG
21:35:57 <acoles> tdasilva: with which, global EC or part power?
21:36:02 <tdasilva> part power
21:36:06 <acoles> k
21:36:21 <notmyname> clayg: ok it's not like we won't have a great release if we don't land either one of these. just easier to ask now instead of later
21:36:22 <clayg> cschwede_: we can totally think "this should be on master" - and have a different opinon about "this should be in a situation to recieve/require backports"
21:36:32 <clayg> we're learning a bit about how to manage longer lived releases
21:36:54 <notmyname> yeah, good point clayg. stuff that goes into this release will have backports for a year
21:37:05 <clayg> notmyname: I think it's a great point!  I hope I can keep a cool/level head next time it's my big feature that's about to go in weeks before an openstack release
21:37:15 <notmyname> but historically, backports aren't a huge burden for us
21:37:43 <clayg> notmyname:  I think we mostly just keep people up with master
21:38:05 <notmyname> I just want to make sure we're comfortable with it and that someone isn't able to say "well I thought this was terrible and rushed in"
21:38:09 <clayg> after the PTG i'm working on our LTS strategy for swift releases - so I may thinking about and complaining about backports more
21:38:17 <tdasilva> notmyname: i'm afraid they will become more and more (at least to red hat)
21:38:28 <notmyname> tdasilva: joys of getting older ;-)
21:38:34 <clayg> ah - afaik people that have looked at it like it a lot :D
21:39:13 <cschwede_> well, more eyes won't hurt. and if there are concerns we should discuss them, maybe at the PTG if it's too close to the release
21:39:16 <notmyname> yeah, from what I can tell, it's a good design, it's been thoroughly reviewed, and it's had a lot of the bad beaten out of it
21:39:31 <clayg> wfm!
21:39:49 <notmyname> so unless there's a "no! stop everything I've got a big issue", then we should plan on merging it based on mattoliverau's final review
21:39:59 <clayg> last chance for someone to have a rush of conscience and save us all from our optimistic selfs?
21:40:04 <notmyname> yep!
21:40:30 <clayg> wooooo!
21:40:50 <notmyname> ok, now let's have the same exact discussion about kota_'s global ec patch https://review.openstack.org/#/c/219165/
21:40:55 <clayg> notmyname: i guess tdasilva is still looking at thing with probetests?
21:40:58 <notmyname> kota_: first up, what's the status?
21:41:04 <notmyname> clayg: tdasilva: ah yes
21:42:02 <kota_> notmyname: i had great feedback from clayg and I'm filling small bits it should be addressed in the patch, mainly unittests for reconstructor and thought or quorum number
21:42:09 <notmyname> ok
21:42:11 <kota_> and it looks like acoles leaves comments
21:42:14 <notmyname> yeah
21:42:17 <kota_> while I was asleep
21:42:21 <notmyname> have you had a chance to look at his -1 yet?
21:42:25 <kota_> not yet check out
21:42:33 <notmyname> ah. sleep. seems like a bad habit ;-)
21:42:44 <acoles> kota_: sorry it took me so long to get to reviewing it
21:43:02 <kota_> so i'm planning I will push a new patch set to address both clayg's annd acoles's comments today
21:43:12 <notmyname> kota_: you said it will (likely?) conflict with the part power patch?
21:43:24 <kota_> acoles: no worries and thanks! it looks great idea
21:43:27 <clayg> i'm going to continue to advocate this is very much something we want.  something we should land on master.  something that is important to the direction of the project that should.   and good to merge duplication support ahead of other needed features for global-ec
21:43:27 <kota_> notmyname: yup
21:43:33 <kota_> notmyname: gerrit says
21:44:07 <notmyname> anyone want to fight clay on how awesome this replicated ec idea is? ;-)
21:44:08 <kota_> notmyname: i didn't see in detail yet, I'll try to look at if I have more time after addressing the comments
21:44:30 <clayg> ... but there is little reason to include this change in octocat - it's not a deliverable feature - you can *use* it to run a global ec cluster - you can *play* with it (and maybe a few other hacks) to *test* (and hopefully validate) our current thinking on how swift should approach global-ec
21:44:48 <clayg> I want to land it before the PTG so we can talk about next steps
21:45:13 <notmyname> clayg: so to summarize your position, it's a great idea but not a release blocker (nor is a great marketing thing in a release [and that matters])
21:45:15 <kota_> clayg: +1
21:45:16 <acoles> +1 I think that if it's labelled experimenta;
21:45:19 <clayg> that is my opinion and I recognize it may not be ideal - kota_ has had this patch up for a long time and he has responsibilities to advocate for it as he should
21:45:34 <acoles> oops...then makes sense to land right after release
21:46:07 <notmyname> yeah, but "for the release" is literally 1 or 2 days different than "before the ptg". essentially, before ptg == in the release
21:46:22 <notmyname> granted, a few days *does* matter right at the end
21:46:39 <notmyname> I'd prefer to cut the ocata release as early next week as possible
21:46:46 <clayg> notmyname: my hope is we can +2 the change and after you cut the octocat tag flip +a
21:46:47 <notmyname> but we've got until thursday
21:47:06 <clayg> I don't think it "should* be in the release - I want it to be right at the begining of the cycle
21:47:13 <clayg> i want to *deliver global-ec* during P
21:47:28 <clayg> duplicate support is step 1 merged day 1 of opening P up for development
21:47:31 <clayg> head start!
21:47:40 <mattoliverau> we could merge it at PTG as well
21:48:09 <notmyname> ok, a different way to phrase it then is to actively review it and land it as quickly as possible, but don't consider a release deadline as a factor in when to land it
21:48:19 <notmyname> ...and so if it's in ocata, ok. and if it's not, ok
21:48:36 <mattoliverau> nice wordsmithing :)
21:48:38 <clayg> yeah w/e - point is - i want it on master and not in the release - but... i wouldn't *cry* if it's in the release - but I have some slightly larger concerns maybe if it's in octocat - i'm maybe a little more loosy goosy merging it to development of P
21:48:45 <notmyname> and if it's in, it's not like we can say "swift supports global ec" yet. still a lot of work
21:49:21 <kota_> notmyname: true
21:50:23 <notmyname> kota_: you'll be looking at the review comments today, and perhaps taking a look at resolving some merge conflicts
21:50:36 <notmyname> we'll land part power asap courtesy of timburke and tdasilva and mattoliverau
21:51:14 <cschwede_> hmm, not asap - don't rush it if there are concerns
21:51:15 <notmyname> and if we get kota_'s patch in, we'll celebrate it with him. and if we don't, we'll have the *exact same* conversation in atlanta about it
21:51:22 <notmyname> cschwede_: right :-)
21:51:39 <kota_> notmyname: yes, I'll take to work for ready that it *can be* merged with the highest priority
21:51:50 <notmyname> kota_: perfect. thanks
21:51:55 <notmyname> ok, I'm feeling ok about what the next week looks like and what's expected to be in the release
21:52:06 <notmyname> any questions from anyone? anyone not clear on what's happening?
21:52:35 <notmyname> ok, good :-)
21:52:45 <notmyname> last, brief topic
21:52:49 <notmyname> #topic proposed logo
21:53:00 <notmyname> I put this in irc earlier...
21:53:16 <notmyname> http://d.not.mn/swift_draft_logo.png is from the foundation as a rev 2 of their proposed swift logo
21:53:28 <joeljwright1> is it a swift yet?
21:53:34 <notmyname> joeljwright1: close
21:53:39 <notmyname> more like a martin, actually ;-)
21:53:54 <notmyname> timburke countered with http://imgur.com/a/62naP (which personally I like better because it *does* look like a swift)
21:54:07 <notmyname> so what do you think?
21:54:34 <notmyname> also, ignore the hairline white lines in both. that's not in the final, I'm told
21:54:45 <jrichli> timburke version +1
21:55:12 <acoles> I was about to say the big white space in the middle is odd, so did timburke fill that
21:55:21 <cschwede_> timburke +1
21:55:36 <joeljwright1> timburke: +1
21:55:39 <clayg> those poor design folks
21:55:46 <notmyname> FWIW, swift vs swallow vs martin http://www.bbc.co.uk/nature/22527420
21:55:54 <mattoliverau> timburke: +1
21:56:03 <notmyname> clayg: yeah, I don't think they knew what they were getting in to
21:56:17 <clayg> notmyname: I think the martin is fine - the entire thing is so crazy - Lidia does Swift's official logos - end of story
21:56:20 <notmyname> ok, I'm hearing a lot of votes for timburke's filled-in version
21:56:42 <acoles> yes, timburke +1
21:57:00 <notmyname> ok, I'll respond with that feedback. thanks everyone, and especially timburke :-)
21:57:12 <jrichli> i still  haven't seen the other teams logo from ML
21:57:38 <notmyname> look for "new mascot design" http://lists.openstack.org/pipermail/openstack-dev/2017-February/thread.html
21:57:49 <notmyname> the ironic one had a lot of drama
21:57:54 <jrichli> thx
21:57:58 <notmyname> but there are others in the feb and jan archives
21:58:11 <notmyname> I think that covers it for this week
21:58:16 <notmyname> thank you everyone for coming
21:58:22 <notmyname> and thank you for working on swift
21:58:25 <notmyname> #endmeeting