21:00:27 <notmyname> #startmeeting swift
21:00:28 <openstack> Meeting started Wed Mar 21 21:00:27 2018 UTC and is due to finish in 60 minutes.  The chair is notmyname. Information about MeetBot at http://wiki.debian.org/MeetBot.
21:00:29 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
21:00:31 <openstack> The meeting name has been set to 'swift'
21:00:34 <notmyname> who's here for the swift meeting?
21:00:35 <timburke> o/
21:00:36 <m_kazuhiro> o/
21:00:39 <mattoliverau> o/
21:00:42 <kota_> morning
21:00:42 <kei-ichi> o/
21:00:47 <rledisez> hi o/
21:01:16 <notmyname> clayg: tdasilva: cschwede: acoles: ping
21:01:22 <acoles> I'm here
21:01:28 <clayg> 0/
21:01:54 <notmyname> agenda for this week is at ...
21:01:56 <notmyname> #link https://wiki.openstack.org/wiki/Meetings/Swift
21:02:34 <tdasilva> o/
21:02:37 <torgomatic> o/
21:02:44 <notmyname> #topic bugs, oh my!
21:02:51 <notmyname> https://bugs.launchpad.net/swift/+bug/1755554
21:02:52 <openstack> Launchpad bug 1755554 in OpenStack Object Storage (swift) "Percent signs in object names cause trouble for versioned_writes" [High,Confirmed]
21:03:08 <notmyname> last week kota_ and timburke suggested they may have a chance to look at this bug
21:03:31 <notmyname> kota_: timburke: did you look at it since the last meeting?
21:03:50 <timburke> i think i said i could look at it, but probably not that week ;-)
21:03:57 <kota_> sorry i didn't. but i think this week I can have it rather than the last week.
21:03:57 <timburke> i was right on that account
21:03:58 <notmyname> but now it's a new week!
21:04:43 <notmyname> ok, thanks
21:05:07 <mattoliverau> timburke and how's this week looking?
21:05:22 <notmyname> IMO it's an important bug that I'd like to see closed in the next release we do, but I don't think we'll be releasing in the next few days or anythign like that :-)
21:05:50 <timburke> mattoliverau: *shrug* who knows? we'll see where feature/deep goes
21:06:12 <notmyname> kota_: thanks. I'll bring it up again next week to ask about it again
21:06:40 <kota_> notmyname: ok
21:06:43 <notmyname> speaking of "next release", that's a great segue into the next topic...
21:06:49 <notmyname> #topic feature/deep plans
21:06:59 <notmyname> #link https://wiki.openstack.org/wiki/Swift/PriorityReviews
21:07:19 <notmyname> I updated that page with some feature/deep
21:07:34 <notmyname> ...and hit enter too early...
21:07:41 <notmyname> I updated that page with some feature/deep scheduling info
21:07:44 <kota_> lol
21:08:25 <kota_> oh feature/deep is close to the final landing proposal.
21:08:29 <mattoliverau> Oh exciting
21:08:49 <zaitcev> Well, it is.
21:09:08 <notmyname> "where do these dates come from?" you ask. great question
21:10:12 <notmyname> full disclosure, there's some internal dates/goals we're trying to hit at swiftstack related to container sharding. these dates are somewhat related to that, but the upstream work is driving them (more that the other way around)
21:11:15 <notmyname> so talking with acoles and timburke and clayg about what can be done and what's a thing that can be released that will actually help users got us to these date goals
21:11:41 <notmyname> the scope is not to have feature/deep support automatically sharding every container in a cluster and all new containers that are created
21:11:46 <zaitcev> Thank goodness someone found a business goal for the sharding after all these years.
21:12:03 <clayg> zaitcev: you a fan of the big containers?
21:12:11 <notmyname> zaitcev: I think we've all seen pain around big containers from customers :-)
21:12:40 <notmyname> I think it will be wonderful to have everything automatically sharded all the time. but it's not a reasonable thing to have in an initial release
21:13:36 <notmyname> the initial goal is to have tools to identify big containers (or container shards) via recon and then shard a container via an "expert operator"
21:14:17 <notmyname> later, we can make sure we automatically start sharding big stuff, but that will force us to solve the leader election problem (which container sets the shard ranges in a distributed system)
21:14:52 <notmyname> also, I like initially having it triggered by hand so that an operator can monitor it without having the whole cluster automatically do something with little recourse
21:15:48 <notmyname> so an existing swift cluster operator will be able to identify big containers and run some bin/ scripts to start it sharding. later, recon stats may identify a particular shard of that container is getting big, and the process can repeat
21:16:04 <notmyname> does all that make sense? what do you think? mattoliverau, zaitcev, tdasilva, kota_, rledisez
21:16:31 <zaitcev> Sounds good to me.
21:16:45 <tdasilva> notmyname: so the whole sharding process will be manual?? meaning even to create new shards requires operator?
21:16:55 <rledisez> I totally agree to not have auto sharding at first
21:17:22 <timburke> tdasilva: baby steps -- we'll get the human out of the loop eventually
21:17:30 <notmyname> tdasilva: correct. an operator will run a 'find-and-write-the-shard-ranges' script to kick it off
21:17:44 <mattoliverau> Sounds like a good reduced scope and first step. Then slowly turn on the smarts as we go.
21:17:46 <kota_> sounds reasonable. do it manually, then make it as automatic. that wey is well as what tiering also will do.
21:18:02 <kota_> way
21:18:08 <zaitcev> You know even in kernel hugepages are not automatically enabled.
21:18:34 <notmyname> in addition to not making it automatic, it means we don't yet have to solve distributed consensus for which container replica gets to choose the shard ranges
21:18:36 <tdasilva> timburke: i understand the reasoning, was more really just trying to understand the actual process...
21:18:43 <notmyname> ok :-)
21:19:03 <zaitcev> wait, I thought 0-th replica was always the master.
21:19:07 <mattoliverau> Besides having control before handing it completely over feels better, until we know we trust this sharding thing :)
21:19:13 <kota_> one thing, i'm worried about the schedule. s3api should be later than the feature/deep proposal... or not?
21:19:16 <acoles> tdasilva: when shards grow large, they will need to be manually sharded in exact same way as original container was
21:19:20 <timburke> zaitcev: gets messy when you rebalance :-/
21:19:53 <acoles> tdasilva: human kicks off the process but daemon takes care of the shard sharding itself
21:20:40 <mattoliverau> Step 1 no leader election, step 2 basic election, step 3 full leader election (if we ever need it)
21:20:56 <notmyname> kota_: yeah, that schedule conflict is tricky. I was hoping that with a reduced scope for s3api, the merge would be simpler and require less time from everyone. and that we'd be able to get it done before feature/deep
21:21:10 <acoles> as timburke says, the concern with fully automatic sharding is we need a robust leader election, and choosing node index 0 is not robust enough currently without further work
21:21:27 <tdasilva> acoles: thanks, don't want to take time from this meeting, we can probably continue on #openstack-swift later..
21:21:30 <acoles> but FWIW all probe tests *do* run fully automatically
21:22:04 <tdasilva> :D
21:22:19 <acoles> we're just proposing that for production the auto sharding should be off by default
21:22:24 <mattoliverau> And there's an auto shard option just will be turned off by default with a warning.
21:22:30 <kota_> notmyname: appreciated your polite consideration
21:22:32 <notmyname> kota_: but depending on how the s3api work goes, it may in fact become a conflict
21:22:35 <mattoliverau> Or what acoles said
21:23:49 <kota_> notmyname: this (or the next) week, I'll try to figure out the conflict with feature/deep and feature/s3api
21:23:59 <notmyname> kota_: thank you. that will be very helpful
21:24:11 <notmyname> so with those proposed dates about feature/deep, I'd like to hold off the next swift release until feature/deep merges
21:24:31 <timburke> fwiw, the only conflicts (right now) between the two feature branches are in .gitreview (for obvious reasons), .zuul.yaml, and test/functional/__init__.py
21:24:38 <notmyname> it also means that in a few weeks (ie April 16), we'll need a lot of help from everyone for reviewing a merge to master
21:24:43 <kota_> i don't think we have much conflict because s3api doesn't touch existing Swift master code so much.
21:25:03 <kota_> but... the serious problem is lack of reviewers.
21:25:24 <kota_> it looks like no reviews on my s3api patches in the last week...
21:25:46 <notmyname> note, the April 16 "propose to master" date for feature/deep is what I'd expect the earliest to be. and a May 8 "merged to master" is what I'd hope the latest to be
21:26:15 <notmyname> kota_: that is an excellent point
21:26:19 <kota_> timburke: thx your confirmation on the conflict!
21:26:32 <notmyname> I think that means we can do one of two things about feature/s3api
21:26:47 <notmyname> (1) hold off on it until *after* feature/deep merges
21:27:11 <notmyname> (2) ask people to review feature/s3api and hope it doesn't impact ongoing work to finish feature/deep
21:27:26 <notmyname> kota_: what would you prefer?
21:28:22 <kota_> (2) is. but I don't know how much hard for others to find their time.
21:28:30 <notmyname> ok, thank you.
21:28:35 <notmyname> what does everyone else think?
21:28:48 <kota_> I'd try to pick up easy ones as possible to ask someone.
21:28:59 <kota_> when asking.
21:29:41 <zaitcev> Well... I was going to look at s3api. Seemed simpler :-)
21:30:40 <notmyname> the current goal for feature/s3api is to simply bring in the swift3 codebase and resolve test/doc/utils/dependency duplication. *not* to add major new functionality
21:31:28 <kota_> yes
21:31:38 <notmyname> kota_: we can check how others feel right now, but in my opinion, if you merge stuff to the feature branch on your own, that's ok because the scope of feature/s3api isn't huge
21:32:03 <notmyname> ...and will be reviewed as a whole when merged to master
21:32:30 <tdasilva> kota_: fwiw, i started reviewing and got side tracked :(
21:33:03 <kota_> notmyname: got it. try to merge quick with my own decision on the feature branch.
21:33:05 <notmyname> furthermore, I do not think feature/s3api should need approval from everyone in order to land on master. again, the scope and code is rather constrained
21:33:21 <notmyname> kota_: that's my opinion. what does everyone else think?
21:33:33 <mattoliverau> +1
21:33:41 <timburke> yeah, i think i'm fine with all of that
21:35:01 <kota_> ok. cleanup can be progressed by myself. the docs, i'd want helps still though.
21:35:26 <kota_> but part of stuff can be done with the strategy
21:35:46 <notmyname> that's understandable. but I don't think we need to write a bunch of new docs just to import swift3
21:36:13 * notmyname admits he hasn't looked at what docs exist for swift3... maybe it's nothing
21:36:24 <kota_> notmyname: oh
21:37:22 <notmyname> kota_: but, yes, please ask for docs help where you need help. we should definitely help out with that
21:37:23 <kota_> is it ok that docs is just imported from swift3 too...?
21:38:30 <tdasilva> kota_: are you ok if I upload new patchset to your docs patch?
21:38:31 <notmyname> I think so. but docs probably take more integration with the existing docs tree. I think that should be relatively simple, though. at least simpler than writing whole new docs
21:38:47 <mattoliverau> Yeah, they should at least be the basis, let's not rewrite everything if we don't have too.
21:38:56 <notmyname> mattoliverau: right. exactly
21:39:37 <acoles> kota_: notmyname : sorry I was distracted, but I agree that merging own patches on feature branch is ok
21:39:42 <notmyname> mattoliverau: tdasilva: will either of you be able to help kota_ with this feature branch? IMO, if the three of you say it's good, I'm happy to see it land
21:40:55 <tdasilva> btw, kota is not really rewriting all the docs, but they do need some fixups, instructions how to enable and stuff...it's a good effort
21:41:10 <kota_> yup
21:41:11 <notmyname> tdasilva: ah, good point. thanks for the clarification
21:41:13 <kota_> e.g. https://review.openstack.org/#/c/552853/2/swift/common/middleware/s3api/s3api.py
21:41:14 <patchbot> patch 552853 - swift (feature/s3api) - Update S3api Docs
21:41:18 <tdasilva> notmyname: yeah, i rreally want to help kota and will do what i can to help
21:41:24 <notmyname> tdasilva: thanks
21:42:22 <mattoliverau> I'll see what I can do, I'm also trying to keep up with sharding reviews and py3, but knowing the s3api scope I'll try and help where I can
21:42:43 <notmyname> mattoliverau: yeah, that makes sense
21:42:48 <kota_> thank you tdasilva, mattoliverau and notmyname
21:43:05 <tdasilva> mattoliverau: yeah, i'm atm putting s3api ahead of py3
21:43:12 <tdasilva> just cause py3 is more longer term
21:43:23 <notmyname> so to sum up all of that ... any more questions or concerns about feature/deep timeline goals and feature/s3api work?
21:43:33 <timburke> but eventually -- we're gonna have *all* of the 3s!
21:43:45 <mattoliverau> Lol
21:43:54 <mattoliverau> Eventually
21:44:24 <zaitcev> nobody make a joke about that please
21:44:33 <notmyname> ok, moving on. thanks for working through it
21:44:39 <notmyname> #topic LOSF update
21:45:02 <notmyname> this morning (US time), we had a meeting with rledisez about their LOSF work
21:45:09 <notmyname> recording is at
21:45:11 <notmyname> #link https://bluejeans.com/s/pnB9K
21:45:29 <tdasilva> zaitcev: you will eventually get it how funny it is to make 'eventually' jokes ;)
21:45:57 <kota_> nice. I'll check out it.
21:46:00 <notmyname> the very quick summary is: it's going well with initial tests and OVH is starting some bigger scale tests soon
21:46:48 <notmyname> and rledisez and alex will be working on getting their code upstream. I'll set up a feature branch for that (but later, not immediately)
21:47:06 <mattoliverau> Cool!
21:47:24 <notmyname> there are 3 patches that are needed upstream, though, that are proposed already. these are listed on the priority review page
21:47:35 <notmyname> rledisez: what did I miss? any further updates there?
21:48:38 <rledisez> no, it seems complete. about the 3 patches, 2 already got one +2, one need some feedback as it's quite big (tests are not completely passing yet, but feedbacks welcome)
21:49:20 <mattoliverau> I started looking at one the other day, I'll try and get back to it today.
21:50:14 <rledisez> thx mattoliverau
21:50:17 <notmyname> one concern I shared this morning is that OVH is starting to deploy this code in their production clusters (which is really cool), but it's not been integrated upstream. this means that there's a pre-existing migration concern when we actually get to review the LOSF code
21:50:36 <notmyname> this is why I want to get the LOSF code reviewable in the open ASAP
21:50:56 <torgomatic> it sounds like migration isn't really a huge deal; you stand up a new node running LOSF for objects and let replication do its job
21:51:18 <notmyname> in the short term (next few days? next week?) i hope rledisez and alex will be able to share updates to design docs
21:51:20 <torgomatic> if they need to migrate from LOSF format 1 to LOSF format 2, then it's the same procedure
21:51:24 <rledisez> but we tried to anticipate that, the on-disk format is versioned so we can evolve it without breaking things (hopefuly)
21:51:45 <notmyname> yep. sounds great :-)
21:52:13 <mattoliverau> rledisez: good thinking :)
21:52:38 <notmyname> last topic for today...
21:52:45 <notmyname> #topic slogging testing
21:52:52 <kei-ichi> Thank you for giving me a time. About unittest modification policy for slogging.
21:52:55 <notmyname> kei-ichi: you had another question about slogging for us?
21:53:01 <kei-ichi> Yes, I found some dependency between slogging and swift. How should I remove swift dependency ? Dependency is like following.
21:53:08 <kei-ichi> ex1: from initial process of unittest, slogging unittest uses swift directory like '../swift/' .
21:53:13 <kei-ichi> ex2: swift itself is imported from slogging code for many purposes.(raise swift exception, to utilize swift test utility, ...)
21:53:19 <kei-ichi> ex3: unittest uses configuration file in swift repository...
21:53:26 <kei-ichi> I think it is not good that those dependency is remained. So is it okay to copy those kind of files from swift to slogging ?
21:53:38 <kei-ichi> (In our heat-dashboard(split out from Horizon as Heat GUI), there was same problems between Heat-GUI part and Horizon. In this case, we copied all dependent code to heat-dashboard basically.)
21:54:24 <notmyname> I think that's totally up to you. your example 1 seems a bit awkward and perhaps could be improved, but slogging importing swift code seems normal to me
21:54:40 <notmyname> but generally, I don't think you need our permission to update slogging :-)
21:54:51 <notmyname> kei-ichi: what do you think is best for slogging and maintaining it?
21:56:01 <kei-ichi> notmyname thanks! I think I agree to your proposal.
21:56:07 <torgomatic> FWIW, in ProxyFS (another project using Swift code), we started off importing a bunch of code from Swift, and then any time it changed and broke things, we'd copy those functions into the ProxyFS codebase
21:56:09 <torgomatic> https://github.com/swiftstack/ProxyFS/blob/development/pfs_middleware/pfs_middleware/swift_code.py
21:56:30 <clayg> swiftlib
21:56:31 <kei-ichi> Basically I remove dependency, but maybe keep using swift itself.
21:56:53 <notmyname> clayg: swoslo?
21:57:01 <mattoliverau> Lol
21:57:06 <notmyname> also, no ;-)
21:57:06 <clayg> 👍
21:57:13 <clayg> 👎
21:57:14 <torgomatic> As long as the licenses are compatible, I don't think there's a problem copying code over.
21:57:15 <zaitcev> oslo.swift
21:57:26 <zaitcev> ^_^
21:57:34 <kei-ichi> torogomatic thanks! I'll look at that.
21:57:35 <mattoliverau> Except for potential maintenance
21:57:48 <timburke> i wonder if swift-on-file's experience might offer guidance...
21:57:55 <notmyname> mattoliverau: and that's up the an individual project to decide what they want to do
21:57:56 <kei-ichi> That wil be really helpful for me :)
21:58:06 <mattoliverau> Oh yeah I agree
21:58:21 <kota_> swift3 import many of swift ;-)
21:58:25 <kota_> imports
21:58:51 <notmyname> I think we've covered everything that needs covering this week, and we're just about out of our allotted time.
21:58:56 <notmyname> anyone have anything else to bring up?
21:58:57 <zaitcev> Oi
21:59:15 <zaitcev> PUT+POST is ready, for real this time. Even tests.
21:59:18 <kota_> but not locate the code in the repo like ex1 kei-ichi suggested...
21:59:21 <notmyname> zaitcev: ah yes!
21:59:36 <notmyname> zaitcev: thanks for updating that
21:59:42 <kei-ichi> Thanks everyone !! current my opinion. Swift itself -> keep using. Other utility kind of files or direct reference to filesystem -> copy from swift.
21:59:49 <notmyname> I'll put it back on the priority reviews page
22:00:16 <notmyname> thanks everyone for coming. thank you for your work on swift
22:00:21 <notmyname> #endmeeting