21:00:33 <notmyname> #startmeeting swift
21:00:34 <openstack> Meeting started Wed Apr 18 21:00:33 2018 UTC and is due to finish in 60 minutes.  The chair is notmyname. Information about MeetBot at http://wiki.debian.org/MeetBot.
21:00:35 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
21:00:37 <openstack> The meeting name has been set to 'swift'
21:00:42 <notmyname> who's here for the swift team meeting?
21:00:44 <mattoliverau> o/
21:00:45 <m_kazuhiro> o/
21:00:47 <tdasilva> o/
21:00:49 <kota_> hi
21:00:50 <acoles> o/
21:00:57 <rledisez_> o/
21:01:10 <clayg> rledisez_: !!!
21:01:11 <torgomatic> o/
21:01:33 <timburke_> o/
21:01:36 <notmyname> welcome
21:01:47 <notmyname> agenda for this week is the same as last week:
21:01:49 <notmyname> #link https://wiki.openstack.org/wiki/Meetings/Swift
21:02:06 <notmyname> but we have a whole week's worth of new info about each topic! :-)
21:02:43 <notmyname> https://review.openstack.org/#/c/555245/ is still open and it fixes a high-priority bug
21:02:44 <patchbot> patch 555245 - swift - Fix versioned writes error with url-encoded object...
21:02:58 <notmyname> still needs reviewers, and it should land before the next swift release.
21:03:12 <notmyname> as we get closer to a release at some point, I'll bug people more about it specifically
21:03:27 <notmyname> (so in this case, there is *not* a week's worth of more info on it)
21:03:35 <clayg> Lol
21:04:14 <timburke_> thanks for the fix kota_! sorry that i haven't gotten to reviewing it :-(
21:04:23 <notmyname> ok... feature/s3api, feature/deep, and torgomatic's consistency engine work. all of these are wrapped up together this week
21:04:29 <notmyname> let's see how I can untangle them for the meeting
21:04:52 <notmyname> first up, I'll give a summary of where we are with feature/deep (and acoles can correct me where I'm off)
21:05:01 <acoles> ok!
21:05:16 <notmyname> good news with feature/deep is that we deployed it to a test cluster at the end of last week and ran it on a 145M row DB
21:05:30 <notmyname> bad news is that it didn't work perfectly
21:05:39 <mattoliverau> :(
21:05:42 <notmyname> timburke_ did a lot of great work to get that tested and diagnosed
21:05:59 <notmyname> the major issue that came up is related to DB replication
21:06:39 <notmyname> ie replication during the sharding process obviously needs some work so that a lot of DB rows aren't moved unnecessarily
21:06:51 <notmyname> timburke_: is that a resaonable, albeit brief, summary?
21:06:53 <timburke_> notmyname: idk what you're talking about -- that was *fantastic* news! *way* better to have it fail now
21:07:02 <acoles> lol
21:07:17 <notmyname> yes, it was fantastic to see it running and finding issues before we merge it :-)
21:08:07 <timburke_> but yeah, definitely a lurking issue with db replication -- i saw my nice beautiful tiny db (that should eventually only have shard ranges) take on ~12M object rows
21:08:20 <notmyname> if you remember from a while back, we'd set a goal of proposing feature/deep to master this week
21:08:20 <acoles> notmyname: specifically we are now planning to stop replicating objects between peers when they could be moved to shards. before we still had some replication happening as well as sharding
21:08:44 <notmyname> timburke_: acoles: right. thanks for the clarification
21:09:06 <notmyname> so regarding a merge proposal to master, the short answer is "it's not happening this week"
21:09:45 <notmyname> acoles and clayg and timburke_ have been working hard on the replication issues (and other things) in order to get something that works
21:10:07 <notmyname> our new goal is to have a feature/deep proposal to master next week
21:10:27 <notmyname> in part, that's going to be based on later tests this week of the fixes that have been proposed running in a real cluster
21:11:10 <mattoliverau> Nice work timburke_ and acoles, sorry if it was something I missed. Is there anything I can do to help?
21:11:12 <notmyname> in general, there's a couple of assurances I want to give to everyone: first, and most importantly, we won't merge something to master that doesn't work
21:11:41 <notmyname> nobody is going to propose a merge to master (much less approve it) in order to meet a date we set for ourselves
21:11:44 <acoles> mattoliverau: nope it's nothing you missed
21:12:11 <notmyname> and secondly, again, it's great we're finding this stuff now
21:12:33 <acoles> mattoliverau: https://review.openstack.org/#/c/549678/ is where we're at and it is quite possibly more in your original direction of thinking
21:12:34 <patchbot> patch 549678 - swift (feature/deep) - Stop replicating object rows when container could ...
21:13:03 <timburke_> it was also an idea i'd had knocking around in my head since the ptg -- once we know where the rows should go, we should be trying to get them there asap, not kick them over to another server to deal with them
21:13:20 <notmyname> so for everyone else, we will need everyone's help to review the merge to master. and it looks like that's coming quickly. the goal is next week
21:13:39 <mattoliverau> Ok, I'll review it today and try and get up to speed.
21:13:40 <notmyname> torgomatic has already started looking at it in order to be able to better review it
21:13:41 <acoles> mattoliverau: IIRC you were 'usync only' once sharding and we're now looking at 'nothing'
21:13:51 <mattoliverau> Kk
21:14:35 <notmyname> which brings me to the consistency engine work that torgomatic was working on (rooted at patch 555563)
21:14:36 <patchbot> https://review.openstack.org/#/c/555563/ - swift - Multiprocess object replicator
21:14:53 <mattoliverau> Well handoffs sharding to the right place is good too, so data does move, just to the right place. Ie a new form of replication
21:15:00 <notmyname> that stuff is still open, still a great improvement to swift, but on hold until after feature/deep lands
21:15:13 <acoles> mattoliverau: yep
21:15:59 <timburke_> mattoliverau: yeah, that was my key takeaway too -- and in particular, we're *not harming durability* by doing that
21:15:59 <notmyname> any questions on feature/deep or torgomatic's patch chain?
21:16:48 <notmyname> tdasilva: as we mentioned last week (I think it was clayg's idea), I'd still like to set up a video chat around the time of a merge proposal so that we can all talk about the general design and to help with our review process
21:17:03 <tdasilva> sounds good
21:17:06 <mattoliverau> timburke_: +1
21:17:39 <kota_> +1
21:17:43 <notmyname> rledisez_: kota_: m_kazuhiro: does all of that make sense? sound good?
21:17:59 <notmyname> kota_: ah, you answered my question before I finished typign :-)
21:18:01 <mattoliverau> +1 to the video as well
21:18:02 <rledisez_> notmyname: looks ok to me
21:18:07 <m_kazuhiro> +1
21:18:20 <notmyname> acoles: clayg: timburke_: anything to add about feature/deep right now?
21:18:26 <kota_> for video call ;)
21:18:30 <acoles> notmyname: not from me
21:19:13 <notmyname> ok, that brings us to the topic of feature/s3api
21:19:28 <notmyname> I've talked to nearly all of you about this topic during the last week
21:19:48 <notmyname> let me paste the summary in here so that we're all talking about the same thing and also so that it is recorded in meeting logs
21:20:27 <notmyname> feature/s3api is ready to be proposed to master. this is a simple import of swift3's functionality and code into swift's repo
21:20:44 <notmyname> if we do that, the advantage is that we have s3 support as a first-class part of swift. IMO this is very good for adoption and marketing of swift
21:21:22 <notmyname> it's also good because it means we can remove "cruft" from swift3 and stop having to worry about compatibility matrixes (which version of swift vs swift version of swift3)
21:21:39 <notmyname> and we'll get s3 testing with patches to swift
21:21:42 <notmyname> that is all great
21:21:48 <notmyname> however
21:21:57 <notmyname> we originally wanted to make some breaking changes to s3api so that we could enable more functionality in the future. for example, we need a better way to map buckets to accounts and containers so that we can support public ACLs and more of the bucket api
21:22:12 <notmyname> if we do that as we land s3api, ie if it's included in the first merge to master, then we've got the advantage of "here's a new thing named something new that doesn't completely work with the old" but that's ok, because it's all new
21:22:18 <notmyname> if we don't do that breaking work now, then we may (probably will?) have to worry more about migrations and support in the future when we get around to actually doing that work
21:22:59 <notmyname> so we could still try to do that breaking-change work to get the better bucket mapping. and do that before trying to import swift3 code
21:23:15 <notmyname> or we could delay that work and merge what we have now
21:23:27 <notmyname> if we delay, it's not a delay until a specific later date. it's a delay to "unscheduled", despite a clean merge being ready now
21:23:53 <notmyname> so... that gets to the question. as I see it, we have 3 options. which do you prefer
21:23:59 <notmyname> (1) community merges feature/s3api now as a swift3 import, and we accept the impact it has to feature/deep work
21:24:06 <notmyname> (2) we designate a couple of people in the community to review and merge feature/s3api into master as it is now. we agree to let them do this without wider community input. ie it's like a normal patch to master with 2 cores and all the rest of us decide to be ok with the added s3 capability. this likely implies that those focusing in feature/deep will keep doing so and delegate authority on
21:24:06 <notmyname> feature/s3api to others
21:24:12 <notmyname> (3) we delay merging feature/s3api until an undetermined date in the future. essentially this means unscheduling the work and accepting that future work will never be as simple as it is right now. at least we'll have merge conflicts and at most we'll have the additional refactoring work too
21:24:27 <notmyname> ok, copy/paste wall done :-)
21:24:46 <notmyname> as i said, I talked to most of you during the past week about this
21:24:51 <notmyname> here's the consensus:
21:25:29 <notmyname> we should do option 2. and whoever the review champions are should keep in mind that they can call out to the rest of us if soemthing big is found
21:26:38 <zaitcev> and the designated champions are...
21:27:00 <notmyname> so here is the actual question for this meeting to everyone here: do you support this agreement. ie are we all ok with this as the way forward as a group?
21:27:20 <notmyname> zaitcev: good question. let's make sure everyone agrees first :-)
21:27:31 <mattoliverau> So option 2 with a little bit of 1 (if something big is found)
21:27:38 <notmyname> mattoliverau: correct
21:27:53 <timburke_> +1
21:28:12 <mattoliverau> If we knew how it worked it could be a great time to use the meeting vote option :p
21:28:21 <mattoliverau> But it's +1 from me
21:28:41 <clayg> lol @ mattoliverau
21:28:46 <clayg> notmyname: sounds like a great plan!
21:28:54 <m_kazuhiro> +1
21:29:05 <kota_> let me correct the difference of 1 and 2
21:29:35 <notmyname> #vote do you support the community designating reviewers for feature/s3api in order to merge it now, knowing that it will cause future pain for migrations whenever we get around to scheduling the bucket mapping work? yes, no
21:29:45 <notmyname> yeah, that didn't work :/
21:29:48 <tdasilva> i know i asked again, but just to try to clarify...if we named it swift3 for now with an eye on one day having s3api as a new middleware with all the work we want, would that help at all? or not really?
21:30:01 <tdasilva> s/asked again/asked before
21:30:13 <notmyname> tdasilva: IMO, not really. the hard part is because it will be in swift, and swift must have a migration plan
21:30:23 <notmyname> ...plan for everything we have in swift
21:30:51 <notmyname> we may need to rename it in the future. that may be part of a future breaking change plan
21:31:38 <notmyname> kota_: for (1) it means we'd all review it and delay feature/deep work. for (2), it means we'd treat it more like a normal patch to swift and we'd have 2 +2s and land it
21:31:48 <timburke_> ....but knowing us, we'll still end up supporting the old name indefinitely :-/
21:32:08 <clayg> timburke_: that's the spirit!
21:32:43 <kota_> notmyname: make sense, and the voting message concrete me to catch up. 2 has still the pain on the migration issue for un-developed features.
21:33:12 <notmyname> kota_: correct. both (1) and (2) have pain for future un-developed features
21:34:39 <notmyname> in my opinion, because the future work for refactoring is completely unscheduled, we should land what we have available now. the benefits of having upstream s3 api support (technical and otherwise) outweigh future migration work that we haven't yet started to work on
21:34:56 <clayg> notmyname: well said!
21:35:01 <clayg> notmyname: one in the hand two in the bush!
21:35:08 <tdasilva> +1
21:36:07 <notmyname> any other comments on this particular question? if not, let's move on to choosing who the reviewers will be
21:36:20 <notmyname> ie move forward with option 2 as I've presented it
21:36:33 * notmyname pauses in case anyone is typing something long
21:37:08 <acoles> I'm ok with us moving forwards with option 2, there's no ideal solution
21:37:41 <notmyname> ok
21:38:23 <m_kazuhiro> I like option 2 :)
21:38:24 <notmyname> now for the big question! zaitcev_ said it... "who are the designated reviewers for feature/s3api?"
21:38:51 <zaitcev_> Some kind of migration from swift3 is needed anyway, I'm afraid. Currently we're trying to wash our hands by making a middleware under a different name.
21:40:32 <notmyname> not everyone volunteering at once :-)
21:40:43 <notmyname> so, obviously, feature/deep is taking a lot of time
21:40:57 <notmyname> the point of option 2 is to impact that work as little as possible
21:41:10 <notmyname> which means at least acoles, clayg, and timburke cannot review feature/s3api
21:41:24 <kota_> zaitcev_: at the point of migration from swift3, i added a few changes to support the migration (e.g. https://review.openstack.org/#/c/557623/)
21:41:24 <patchbot> patch 557623 - swift (feature/s3api) - Change sysmeta name from swift3 to s3api (MERGED)
21:41:56 <kota_> at feature/s3api, so I'd keep the migration path as possible.
21:42:00 <notmyname> torgomatic and mattoliverau are questionable because they're somewhat involved with feature/deep
21:42:26 <notmyname> kota_ is the primary author of feature/s3api, so probably shouldn't be one of the reviewers (but we may reconsider that)
21:42:52 <notmyname> sooo... who's left?
21:43:03 <kota_> notmyname: it's what i thought but can be volunteer to fix the patch/ answer some questions
21:43:45 <notmyname> of all of us here in the meeting, we've got rledisez_, m_kazuhiro, zaitcev_, tdasilva, and myself
21:43:50 <zaitcev_> the answer is obvious, we still have myself, tdasilva, cschwede.
21:43:59 <zaitcev_> oh, yeah, Romain
21:44:21 <notmyname> has anyone seen cschwede? I didn't count him because I haven't seen him around in a long time
21:44:38 <zaitcev_> He's stuck in meetings whole day every day.
21:44:39 <mattoliverau> And my time is currently limited with a bunch going on.. but if no one else will I'll find some time. But it might not be the fastest reviews
21:45:03 <tdasilva> notmyname: did you have a schedule in mind?
21:45:30 <notmyname> the goal of reviewing feature/s3api merge is (1) is it a faithful import of swift3 (2) does it make any major changes that cause for concern (3) is it maintainable code
21:45:34 <tdasilva> i.e., do you want to see it merged before feature/deep or is it completely orthogonal?
21:45:42 <notmyname> (any other major criteria for review of this feature branch?)
21:46:08 <notmyname> I would hope it would land before feature/deep
21:47:26 <notmyname> I *could* start specifically asking people :-)
21:48:30 <mattoliverau> Ok, I'll start reviewing it today, maybe between tdasilva, kota_, zaitcev, m_kazuhiro and anyone else we can get enough eyes to get a consensus.
21:48:48 <notmyname> mattoliverau: thank you
21:49:02 <notmyname> but I'd also like one other person specifically (instead of "and a few other people...")
21:49:26 <tdasilva> notmyname: can we putdown redhat for now?
21:49:27 <m_kazuhiro> I will try to review it.
21:49:51 <notmyname> tdasilva: redhat in general? :-)
21:50:39 <tdasilva> heh, i mean: tdasilva, cschwede or zaitcev
21:50:53 <notmyname> yeah, that's what I meant :-)
21:50:54 <notmyname> thanks
21:50:59 <notmyname> tdasilva: m_kazuhiro: thanks
21:51:11 <notmyname> ok, here's the summary
21:51:14 <zaitcev> I have it on my todo list, but I am afraid to volunteer, given that I've not yet done it
21:52:03 <notmyname> mattoliverau, tdasilva (or zaitcev or cschwede) and m_kazuhiro will review feature/s3api merge to master. when they have approved it, we are all ok with that code merging to master without further review
21:52:33 <notmyname> kota_: this means you or I need to propose the actual merge to master from feature/s3api. I can do that or help you if you want to
21:52:36 <zaitcev> that... sounds like no change from the usual practices, actually.
21:52:48 <zaitcev> oh, yeah, that would be the best
21:53:04 <zaitcev> then I can comment on that review for the merge as a whole
21:53:11 <notmyname> zaitcev: it's the same as normal patches to master. it's a bit different than what we normally do for bigger stuff on feature branches
21:53:28 <kota_> either is fine to me. and i can continue to work on it to address issues from the reviewers anyway
21:53:37 <notmyname> any objections from anyone? this is your chance to say something
21:53:52 <notmyname> kota_: I think we need to review an actual merge proposal to master. not a large patch chain
21:54:41 <kota_> ah, just a merge commit
21:54:55 <notmyname> all right. let's do it
21:55:27 <notmyname> kota_: I can make a merge commit to master right after this meeting so it will be ready after you have breakfast
21:55:50 <kota_> notmyname: thanks ;)
21:56:06 <notmyname> thank you to everyone for helping me work though this and for helping us all figure out the cost/benefit tradeoffs for our ongoing work
21:56:24 <acoles> notmyname: thank you!
21:57:00 <notmyname> that gets us to the end of this week's meeting. maybe by the next meeting we'll have feature/s3api merged and feature/deep proposed to master!
21:57:11 <notmyname> thanks for coming and for your work on swift
21:57:15 <notmyname> #endmeeting