21:00:45 #startmeeting swift 21:00:46 Meeting started Wed Dec 13 21:00:45 2017 UTC and is due to finish in 60 minutes. The chair is notmyname. Information about MeetBot at http://wiki.debian.org/MeetBot. 21:00:47 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 21:00:49 The meeting name has been set to 'swift' 21:00:51 yay! wooooo! 21:00:53 who's here for the swift team meeting? 21:00:57 . 21:01:31 hi 21:01:35 o/ 21:01:46 here 21:01:49 tdasilva: cschwede: ? 21:01:50 o/ 21:01:55 clayg: ? 21:02:01 yeah i'm here 21:02:58 agenda for this week is 21:02:59 #link https://wiki.openstack.org/wiki/Meetings/Swift 21:03:11 only a couple of things on it, and follow-ups from last week 21:03:19 #topic symlink status 21:03:25 patch 232162 21:03:25 https://review.openstack.org/#/c/232162/ - swift - Symlink implementation. 21:03:31 is it merged yet? 21:03:44 not yet but close to land, imo 21:03:48 (I mean, obviously not, but what do we need?) 21:03:53 acoles: looks like you added a question to the meeting agenda 21:04:24 hi 21:04:42 notmyname: yes, about one last change that we should maybe squash in before symlink merges 21:05:05 cleong: welcome 21:05:31 w.r.t. how container sync translates symlink sysmeta to x-symlink-target 21:06:20 symlinks is going to be so awesome 21:06:28 I have proposed that the symlink middleware should be deployed in container sync internal client pipeline (a) to avoid leaking symlink implementation detail to container sync and (b) to make it more explicit that symlinks are handled 21:06:33 https://review.openstack.org/#/c/527126/ 21:06:34 patch 527126 - swift - Use symlink in container-sync internal client pipe... 21:06:38 ^ this is a pretty good idea 21:07:07 TBH it was that way at some point - jrichli had a similar patch squashed in, then it changed 21:07:12 using the public api interface (which we have to maintain) is a minor reduction of coupling between components 21:07:52 the reason i am pressing the point is that deciding on the approach NOW avoids an upgrade issue later if we delay the change 21:08:15 @acoles or we carry the cruft 21:08:38 clayg: yes. so lets just call it one way or the other 21:08:42 acoles: the pipeline change could just keep importing from symlink mw until some undetermined time in the future after the sun goes out 21:09:03 one possible concern i had is it could cause operational failure, tho. 21:09:12 kota_: +++ 21:09:28 kota_: how so? 21:09:36 exactly, it's tradeoff thing to leak the code into container-sync or configuration at internal-client. 21:09:38 you HAVE to add symlink to your pipeline (AND your container sync internal client pipeline) *after* you upgrade 21:09:58 I think it's roughly the same thing with encryption and container sync yes? 21:10:06 the case I minded is operator sets the symlink to *proxy pipeline* but forget to set it in the internal-client pipeline. 21:10:08 optional feature - and you have to add it to proxies and also to container sync 21:10:15 clayg: yes, same with encryption 21:10:21 clayg: yes, same thing with encryption 21:10:36 kota_: +1 if you don't put it in internal-client pipeline with acoles change you just double copy objects instead of sync'ing symlinks 21:11:07 kota_: not trivial to fix... but... I think we have some precedence... swift is sadly not known for being easy to configure 21:11:07 ask for a new feature to be enabled by putting it in two places doesn't seem too bad 21:11:14 so it's why I just thought it can be acceptable just one line leak to the container-sync 21:11:34 notmyname: I agree it's not bad - but maybe having container sync import mw code isn't so bad *either* 21:11:47 acoles: just needs the decision hammer - squash or not - either way symlinks is going to be awesome 21:12:07 what else uses internal-client.conf? will they need to know about symlinks, too? 21:12:09 sorry I'm late. 21:12:14 kota_: I think your judgement is reasonable 21:12:23 timburke: expiring objects? no? 21:12:39 timburke: reconciler... probably? shit... 21:12:52 welcome m_kazuhiro! 21:13:00 admittedly, I haven't done a deep examination of the symlink patch. but thinking of it from the perspective of docs (or just explaining it), it seems that explicitly adding it to two places is simplest. otherwise we've got additional things to carry in our head forever about interdependencies of code 21:13:17 kota_: o/ 21:13:22 notmyname: seems pretty justifiable stance to tae 21:13:43 notmyname: "you want symlinks great - put it in all your piplines and the code will do the right thing! forget it in some pipeline and you screwed up" 21:13:44 notmyname: fair point 21:14:01 i remember it was huge pain to unerstand all of the coupling of SLO with container sync 21:14:29 yeah, it would be different if it were an existing feature (like DLO) 21:14:33 ... it has the extra complexity of out of order segment & manifest syncing - but the coupling there is bananas (again it's "just one line" but the ramifications were huge ) 21:14:42 perhaps, I can make a follow up to add warning and sample config for internal-client pipeline for container-sync 21:15:07 notmyname: I think long term the "answer" might be making symlinks NOT optional 21:15:08 warning if you missed to configure the pipeline. 21:15:15 notmyname: timburke is trying to do the same thing for SLO I think 21:15:23 yep, I like that too :-) 21:15:24 notmyname: because "swift api" should be "swift api" 21:15:26 kota_: good idea, like we do for encryption in the sample config 21:15:29 yep 21:15:43 kota_: yeah, that sounds like a good idea 21:15:45 it's nice to add things as "experimental" or "optional" but at some point it's like ... yeah you need this thing in your swift 21:15:53 clayg: makes me think of https://review.openstack.org/#/c/504472/ ... 21:15:54 patch 504472 - swift - Shorten typical proxy pipeline. 21:16:09 timburke: that's pretty cool! 21:16:12 so, make it an explicit thing to be added now, then later make it required everywhere 21:16:16 *starred* 21:16:18 timburke: yeah, I *so* want that 21:16:30 kota_: I think having commented sections in internal-client.conf-sample with note about when they might need to be uncommented would be great 21:17:00 acoles: sure. that's what I thought. 21:17:01 m_kazuhiro: do you have any *reservations* about merging in acoles file work on the container-sync pipeline? notmyname wants to make the call to squash and we *__ALL__* want symlinks to merge! 21:17:15 acoles: you already have all that in your patch tho yeah? 21:17:40 clayg: IIRC I have plenty of doc but not the comments in the actual conf sample 21:18:12 but it has occurred to me since that doing that for encryption also would be good 21:18:28 clayg: What reservation means? 21:18:38 hesitations or concerns 21:18:57 and perhaps also in proxy-server.conf sample, add notes that 'if you enable this here you may want to enable it in internal client config too' 21:20:00 m_kazuhiro: kota_: I feel like that conversation went pretty quickly. let me know when you've caught up and are comfortable 21:20:32 notmyname: no objection from me. it's fair enough and make sense what way we should go. 21:20:38 rledisez: do you have any concerns, operator-wise, with adding the symlink entry into both proxy configs and also internal-client configs? 21:20:42 clayg: notmyname: I see. I don't have reservations. 21:21:18 notmyname: please go a head the meeting 21:21:45 waiting on rledisez now :-) (he said he was here for the meeting) 21:21:48 notmyname: nope. if documented and set by default in a consistent way, no concerns at all 21:21:52 great! 21:22:07 ok, so it sounds like everyone is good with an explicit symlinks mention 21:22:18 who's going to squash the patch? 21:22:27 acoles: ? m_kazuhiro: ? kota_: ? 21:22:50 I can 21:23:00 notmyname: just talked with m_kazuhiro in internal chat about that 21:23:20 acoles: thanks! 21:23:29 acoles: Thank you very much! 21:23:48 everything sounds good. is there anything else to bring up on the symlinks patch in this meeting? 21:23:58 m_kazuhiro: no problem, thank you for driving symlinks to completion 21:24:03 I'll bring it up next week and expect it to be on master by then :-) 21:24:05 and kota_ , tdasilva too 21:24:19 :-) 21:24:27 all right, if nothing else, let's move on 21:24:33 #topic SLO data segments patch 21:24:39 patch 365371 21:24:39 https://review.openstack.org/#/c/365371/ - swift - Add support for data segments to SLO and Segmented... 21:25:08 it also has not landed yet 21:25:20 and it has a pre-req: https://review.openstack.org/#/c/512804/ 21:25:20 IIRC timburke was going to look at it too 21:25:21 patch 512804 - swift - Add base64decode function to common/utils 21:25:24 ah right 21:25:40 (pretty painless pre-req, though) 21:26:20 I'll look at the pre-req one this week 21:26:32 is there anyone else who can too? (looks like an easy review) 21:26:38 did I just curse it? 21:27:41 wooo... such commitment ;-) 21:27:55 i'd be ok doing the *pre-req* :P 21:28:02 thanks :-) 21:28:16 timburke: you still ok with reviewing the data segments one? 21:28:28 I know you were pretty involved, but IMO it's ok 21:28:54 i could take a time to look at them. 21:29:08 ok, thanks kota_ 21:29:08 (expecting symlink will land soon) 21:29:21 #topic open discussion 21:29:22 yeah -- i'm still a little hesitant to +A in light of how many of the recent patchsets i pushed, but i'm looking at it now 21:29:32 anything else to bring up? 21:29:42 acoles: you're getting back into container sharding, right? 21:30:15 * mattoliverau waits for acoles answer 21:30:17 notmyname: yes, ramping up again and in particular working on delete 21:30:24 \o/ 21:30:49 there's a new trello card with some thoughts on managing to delete a sharded container without having to fix sysmeta deletion etc 21:31:20 acoles: I'll take a look and maybe comment today 21:31:22 #link https://trello.com/b/z6oKKI4Q/container-sharding 21:31:31 so we'll warm up feature/deep again (I know mattoliverau did some work while we went quiet - thanks mattoliverau ) 21:31:52 not as much as I'd have liked.. but works got me busy :( 21:31:56 any other topics from anyone? 21:31:59 oh yeah, https://review.openstack.org/#/c/478611/ would probably be nice... 21:32:00 patch 478611 - python-swiftclient - Allow for object uploads > 5GB from stdin. 21:32:04 https://trello.com/c/SDCjvOyt/135-dont-delete-shards-when-marking-root-as-deleted 21:32:09 but have a new manager so might be able to get more swift time in 21:32:17 yay! 21:33:01 * kota_ wanted to get s3api progress if tdasilva is here but it doesn't seem he is here, today. 21:33:03 i might try to get to that this week, though maybe peoplewould be more interested in me actually finishing the bimodal-access docs i'd mentioned last week... 21:33:25 kota_: I'll put it on next week's agenda 21:33:31 kota_: i was thinking about that recently, too :-) 21:33:37 notmyname: :-) 21:33:43 timburke: ^^^ 21:34:31 looks like that's it, then 21:34:41 thanks everyone for coming. thanks for working on swift! 21:34:45 #endmeeting