21:00:09 <notmyname> #startmeeting swift 21:00:10 <openstack> Meeting started Wed Oct 19 21:00:09 2016 UTC and is due to finish in 60 minutes. The chair is notmyname. Information about MeetBot at http://wiki.debian.org/MeetBot. 21:00:11 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 21:00:13 <openstack> The meeting name has been set to 'swift' 21:00:19 <notmyname> who's here for the swift team meeting? 21:00:21 <ntata> Hello 21:00:22 <jrichli> o/ 21:00:24 <bkeller`> o/ 21:00:26 <joeljwright> hello 21:00:29 <pdardeau> hi 21:00:29 <sgundur> hi 21:00:31 <mmotiani_> o/ 21:00:32 <kota_> hey yo 21:00:39 <hurricanerix> o/ 21:00:57 <tdasilva> hi 21:01:01 <acoles> here 21:01:05 <kmARC> o- 21:01:07 <nadeem> hi 21:01:07 <kmARC> o/ 21:01:15 <clayg> ＼(＾O＾)／ 21:01:27 <joeljwright> clayg explodes into the meeting! 21:01:33 <jrichli> lol 21:01:35 <notmyname> :-) 21:01:40 <mattoliverau> o/ 21:01:45 <notmyname> welcome everyone 21:01:50 <cutforth> o/ 21:02:00 <notmyname> just a few things to go over this week 21:02:01 <notmyname> #link https://wiki.openstack.org/wiki/Meetings/Swift 21:02:16 <notmyname> #topic barcelona summit next week 21:02:27 <notmyname> summit is next week 21:02:37 <mattoliverau> \o/ 21:02:49 <notmyname> I've taken the items from the etherpad and grouped them 21:03:16 <notmyname> I'm not totally happy with it yet, but I've pushed up a schedule. still is likely to be adjusted 21:03:25 <clayg> wow, next week - i just can't seem to wrap my head around it - it feels *so* soon 21:03:27 <notmyname> basically, here's the etherpads for the week: 21:03:29 <notmyname> https://etherpad.openstack.org/p/BCN-ops-swift 21:03:29 <notmyname> https://etherpad.openstack.org/p/BCN-swift-community-feedback 21:03:29 <notmyname> https://etherpad.openstack.org/p/BCN-swift-working-session-1 21:03:31 <notmyname> https://etherpad.openstack.org/p/BCN-swift-working-session-2 21:03:33 <notmyname> https://etherpad.openstack.org/p/BCN-swift-working-session-3 21:03:34 <clayg> i bet when the ptg shows up I'll *really* be freaked out? 21:03:35 <notmyname> https://etherpad.openstack.org/p/BCN-swift-working-session-4 21:03:37 <notmyname> https://etherpad.openstack.org/p/BCN-swift-working-session-5 21:03:56 <notmyname> clayg: especially since that's only 4 months away, not 6 21:04:01 <clayg> notmyname: that doesn't seem nessecary 21:04:06 <mattoliverau> yeah short cycle this one 21:04:22 <notmyname> first thing to note is the ops session is part of the ops track 21:04:52 <clayg> OMG look at all this stuff? 21:05:13 <joeljwright> notmyname: I think I missed the boat on your organisation (only added a new topic to the etherpad a couple of hours ago) 21:05:23 <notmyname> we'll use the 40 min in the ops track on tuesday, and then have a "part 2" as one of our fishbowl sessions on thursday morning 21:05:36 <notmyname> joeljwright: ok, pm it to me, and I'll make sure to address it 21:05:48 <joeljwright> cool, thanks 21:06:13 <notmyname> so please try to be at the tuesday ops session 21:06:33 <notmyname> our fishbowl sessions are "ops feedback (part 2)" and "community feedback" 21:06:53 <notmyname> the community feedback one being our own retrospective for how we as a community are doing and what we need to adjust, if anything 21:07:19 <notmyname> the working sessions are arranged in blocks of time, like the last several summits 21:07:23 <mattoliverau> great I like that, an overflow for more ops and then just community discussion 21:07:48 <notmyname> on thursday, we start with the 2 fishbowl sessions and then have 3 blocks of time (separated by breaks or lunch) 21:08:04 <notmyname> it's 2+3+1 blocks of time (40 minute blocks) 21:08:18 <mattoliverau> I'll be the really jetlagged guy at the ops session :P I oly arrive Monday after over 30 hours in transit 21:08:42 <notmyname> then on friday, we have scheduled time (2 chunks of 2 40 min blocks each), then lunch, then the free-form community meeting from 2-6pm 21:09:01 <notmyname> so basically, ops stuff on tuesday, swift-specific stuff on thursday and friday 21:09:16 <clayg> w/e werx - thanks notmyname 21:09:35 <mattoliverau> cools 21:09:59 <notmyname> like I said, I think I want to tweak the agenda for each block some from what I've written down (some important topics will be tricky since some people won't be there) 21:10:05 <notmyname> but in general, this is our starting point 21:10:29 <mattoliverau> And we can obviously go add stuff (prep) to the etherpads 21:10:40 <notmyname> also, if you saw on the ML an hour so so ago, the foundation has pushed out draft versions of their logo (mascot) for each project 21:10:50 <notmyname> I've got the draft version for swift, and I'll be sharing that next week 21:11:03 <notmyname> so we can give feedback. they want to have it final by the PTG in february 21:11:14 <kota_> cool 21:11:29 <notmyname> ok, I think that's all I have on the summit agenda. any questions or other comments about next week 21:12:04 <jrichli> in the schedule, the link is to the #2 etherpad even for some that are later 21:12:48 <notmyname> there are 3 scheduled 40 minute sessions that should link to the #2 etherpad 21:13:09 <jrichli> oh, ok. 21:13:20 <kota_> notmyname: added a few comments to https://etherpad.openstack.org/p/BCN-swift-working-session-5 21:13:21 <notmyname> but I think I added the right time blocks in the top of each etherpad 21:13:26 <notmyname> kota_: thanks 21:13:30 <clayg> notmyname: one benifit of not kicking ou the PTL every few months - they get sorta good at this shit 21:13:33 <jrichli> ok, i missed that. will look again 21:13:45 <kota_> some of my topics there but can't attend at Fri. 21:13:49 <notmyname> oh, also on the #5 etherpad I added a section at the bottom for the "whole afternoon discussion stuff" 21:13:52 <kota_> sorry 21:13:56 <acoles> can we just meet notmyname for breakfast and follow him around each day? 21:13:59 <notmyname> since there's not a place to link to it on the schedule 21:14:14 <notmyname> kota_: good to know. yeah, I didn't work with individual schedules yet 21:14:24 <notmyname> acoles: sounds good to me. :-) 21:15:13 <notmyname> also, in case you haven't been to barcelona before, the culture there is very well adjusted to typical dev schedules ;-) 21:15:36 <notmyname> breakfast is normally about 9-10am and lunch is 3-4 ish and dinner starts at 9 or 10pm 21:15:57 <notmyname> unfortunately, the conference schedule doesn't really reflect that :-) 21:16:08 <mattoliverau> Yeah, I got hungry waiting for dinner when I was there.. thank god for tapas 21:16:18 <notmyname> but don't expect to wake up at 7 or 8 and easily find an open coffee shop to grab a pastry or anything 21:16:26 <kota_> how about siesta? :-) 21:16:33 <notmyname> kota_: of course ;-) 21:16:44 <acoles> notmyname: mathiasb said on the etherpad that he's not there on friday but that's when the keymaster work is scheduled 21:16:59 <acoles> afaik mathiasb is working on the keymaster 21:17:15 <clayg> why is no one hanging around on Friday? 21:17:18 <notmyname> acoles: yeah, that's one thing I need to adjust for 21:17:21 <notmyname> clayg: good question! 21:17:28 <kmARC> Altho I haven't really worked on it yet, but I work together with mathiasb and I will be there on friday 21:17:31 <notmyname> acoles: gotta do the same for kota 21:17:33 <clayg> notmyname: i think we should try to make room for folks while there there 21:17:50 <mattoliverau> kmARC: you have a week to get up to speed :P 21:17:50 <acoles> kmARC: great! 21:17:51 <clayg> i'm somewhat distressed about doing a big block of hummingbird when some of our rax players are missing 21:18:01 <notmyname> clayg: yeah. eg might swap keymaster for golang stuff 21:18:08 <notmyname> right. exactly what I was thinking 21:18:30 <clayg> kk 21:18:46 <clayg> so it's good that we discussed this - notmyname will square it 21:18:46 <notmyname> anything else on the summit next week? 21:18:56 <mattoliverau> we could try and at least trick nadeem into conference calling ;) 21:19:12 <nadeem> I am open to attend the conference call 21:19:17 <clayg> woooo! 21:19:30 <acoles> at 9am CET?? ;) 21:19:30 <notmyname> maybe. I'm not a big fan of that, but maybe 21:19:41 <notmyname> acoles: yeah, that's one reason 21:20:30 <notmyname> ok, moving on to the next topic 21:20:40 <notmyname> #topic big bugs and open patches 21:20:50 <notmyname> thanks for landing the EC fix! 21:20:52 <notmyname> yay 21:21:01 <clayg> acoles: is the bomb 21:21:06 <kota_> acoles is great 21:21:06 <clayg> he has other patches up too 21:21:19 <notmyname> we've got https://bugs.launchpad.net/swift/+bug/1624088 and https://review.openstack.org/#/c/376630/ to address it 21:21:20 <openstack> Launchpad bug 1624088 in OpenStack Object Storage (swift) "EC missing durable can prevent reconstruction" [High,Confirmed] 21:21:21 <clayg> tdasilva: has some good fixes in play too - and has been doing a bunch of great reviews - tdasilva is the bomb 21:21:22 <kota_> clayg:^^ 21:21:25 <acoles> like a rash 21:21:37 <clayg> kota_'s huge impact as well 21:21:40 <clayg> so many kudos! 21:21:55 <notmyname> everyone is awesome :-) 21:22:15 <notmyname> so we've landed the EC corruption fix and the backports for it 21:22:33 <notmyname> and I think it would be good to tag a release with this 21:22:36 <clayg> it was really an ssync protocol bug 21:22:55 <notmyname> but tdasilva had mentioned that https://bugs.launchpad.net/swift/+bug/1624088 should be fixed first 21:22:56 <openstack> Launchpad bug 1624088 in OpenStack Object Storage (swift) "EC missing durable can prevent reconstruction" [High,Confirmed] 21:22:59 <acoles> notmyname: I think we need this first, we're only half done on that bug.. https://review.openstack.org/#/c/387655 21:23:01 <clayg> but far to easy to trigger with ec reconstruction - i feel so bad :'( 21:23:29 <notmyname> acoles: ack. that's exactly why I was bringing it up :-) 21:23:47 <notmyname> so yeah, to summarize: what's open (bugs or patches) that needs to land before we tag it for release? 21:23:48 <clayg> acoles: the corruption fix before the optomistic reconstructor fix 21:23:59 <acoles> notmyname: it's different from the one you mentioned 21:24:04 <notmyname> yes 21:24:14 <notmyname> so finish https://review.openstack.org/#/c/387655 first 21:24:14 <clayg> i guess we should go ahead and make the auditor fix criticial - idk 21:24:25 <notmyname> clayg: what's the bug link? 21:24:26 <acoles> clayg: +1 IMHO 21:24:32 <acoles> https://bugs.launchpad.net/swift/+bug/1633647 21:24:33 <openstack> Launchpad bug 1633647 in OpenStack Object Storage (swift) "bad fragment data not detected in audit" [High,Confirmed] 21:24:33 <clayg> are we really going to backport the optimistic reconstructor fix? 21:24:53 <acoles> notmyname: ^^ 21:25:29 <notmyname> clayg: do you mean https://bugs.launchpad.net/swift/+bug/1624088 ? 21:25:31 <openstack> Launchpad bug 1624088 in OpenStack Object Storage (swift) "EC missing durable can prevent reconstruction" [High,Confirmed] 21:25:53 <clayg> yeah 21:26:04 <clayg> it's weird all this stuff popped up at the same time 21:26:21 <clayg> i'd like to see newton have some of this fixed 21:26:29 <clayg> it's going to be around for awhile 21:26:36 <notmyname> I'm ok with backporting it 21:26:53 <notmyname> but I agree that the corruption bug takes priority 21:26:57 <notmyname> ie https://bugs.launchpad.net/swift/+bug/1633647 21:26:58 <openstack> Launchpad bug 1633647 in OpenStack Object Storage (swift) "bad fragment data not detected in audit" [Critical,Confirmed] 21:27:13 <clayg> I personally haven't cut the SwiftStack Newton release *because* of these issues - we're waiting until after the summit before we start to vet the 2.10+ release 21:27:42 <clayg> but i'm not sure i'm going to go back and repackage our 2.9 - folks that are running EC will just upgrade 21:28:26 <clayg> but when I saw the user survey that it's so common to use os packages - makes me think more about the official openstack releases 21:28:40 <clayg> *idk* - dunno? dunno... backport everything. idk 21:28:47 <acoles> Ideally I'd like to backport the auditor fix for bad frags to mitaka like we did the first patch 21:29:09 <acoles> I do fear that backport of patch https://review.openstack.org/#/c/387655 is not going to be trivial tho 21:29:25 <clayg> acoles: what about optimistic GETS? probably not right? but we could backport the reconstructor fix to Newton? 21:29:46 <clayg> acoles: *right* - test fallout it nuts here! 21:29:47 <kota_> acoles: yeah, not so trivial.. 21:30:11 <notmyname> we should backport what we can, but one of the advantages of having good upgrade paths is that we don't have to backport everything 21:30:18 <acoles> clayg: I'm more concerned with th auditor for newton because we know we have at least one user who needs it and wasn;t planning on upgrade from mitaka 21:30:25 <kota_> my frastration is xenial still provides liberasurecode 1.1.x ... :/ 21:30:28 <acoles> s/newton/mitaka/ 21:30:36 <notmyname> kota_: bah! 21:30:44 <acoles> argh auditor -> mitaka 21:30:58 <acoles> optimistic reconstructor -> newton is ok with me 21:31:06 <clayg> acoles: ok, so well try to backport auditor as far as we can 21:31:09 <clayg> i have to bounce early today 21:31:29 <clayg> i'll async stuff later this afternoon. i'm on tomorrow 21:31:48 <acoles> clayg: k, I should have a little more time for it tomorrow 21:32:06 <acoles> clayg: push anything you might get done 21:32:10 <acoles> kota_: ^^ 21:32:11 <notmyname> to to back up just a bit and to summarize... 21:32:14 <kota_> i can take a look for the new version today. 21:32:29 <kota_> and also able to work tommorow 21:32:32 <notmyname> https://bugs.launchpad.net/swift/+bug/1633647 is important and in progress. when it lands on master, we'll see what's possible to backport 21:32:33 <openstack> Launchpad bug 1633647 in OpenStack Object Storage (swift) "bad fragment data not detected in audit" [Critical,Confirmed] 21:32:59 <notmyname> https://bugs.launchpad.net/swift/+bug/1624088 is of seconday importance (but already has a patch with one +2) and should also land before cutting a release 21:33:00 <openstack> Launchpad bug 1624088 in OpenStack Object Storage (swift) "EC missing durable can prevent reconstruction" [High,Confirmed] 21:33:07 <notmyname> similarly, backport as possible 21:33:23 <notmyname> then after those land, tag a release on master 21:33:27 <acoles> notmyname: good summary, agree 21:33:29 <mattoliverau> +1 21:33:35 <kota_> k 21:33:41 <notmyname> ok, good :-) 21:33:53 <notmyname> last question I have about backports is patch https://review.openstack.org/#/c/385779/ 21:33:57 <notmyname> from cschwede 21:34:08 <notmyname> I'm not sure why this one needs to be backported. did I forget why? 21:34:46 <tdasilva> notmyname: it's important for our downstream newton release 21:34:53 <acoles> notmyname: was this where someone in channel reported seeing huge IO in a fresh empty cluster? 21:34:54 <notmyname> tdasilva: ah ok. got it 21:35:00 <notmyname> acoles: yeah, that's it 21:35:10 <notmyname> specifically, it seems, red hat 21:35:29 <tdasilva> yeah, has to do with the tripleo default installation 21:35:40 <clayg> *huge* IO? really? 21:35:43 <tdasilva> running *everything* under the sun on the controller nodes 21:35:48 <clayg> where they *saveing* that IO for something else? 21:35:52 <notmyname> ok, I'll recheck that one because of the gate failures. then land it later 21:35:56 <clayg> oic 21:36:03 <clayg> so they *were* saving that io for something else 21:36:18 <acoles> clayg: ok s/huge/non-zero/ ;) 21:36:30 <notmyname> #topic open discussion 21:36:32 <clayg> *yawn* 21:36:36 <notmyname> anything else to bring up this week? 21:36:45 <notmyname> obviously, there will be no meeting next week 21:36:59 <notmyname> or rather, lots of meetings, just not in IRC 21:37:10 <mattoliverau> :) 21:38:01 <acoles> mattoliverau: must be about time for you to start your journey ;) 21:38:06 * notmyname is calling it 21:38:11 <notmyname> thanks everyone for working on swift 21:38:19 <mattoliverau> acoles: lol, not quite but soon :) 21:38:20 <notmyname> looking forward to seeing (most of) you next week 21:38:25 <notmyname> #endmeeting