21:00:53 <notmyname> #startmeeting swift
21:00:53 <notmyname> hello, everyone. welcome to the swift team meeting
21:00:53 <openstack> Meeting started Wed Jul  1 21:00:53 2015 UTC and is due to finish in 60 minutes.  The chair is notmyname. Information about MeetBot at http://wiki.debian.org/MeetBot.
21:00:54 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
21:00:56 <openstack> The meeting name has been set to 'swift'
21:00:58 <notmyname> who's here?
21:01:02 <mattoliverau> o/
21:01:02 <peluse> ugh
21:01:02 <minwoob_> o/
21:01:04 <cschwede> o/
21:01:04 <jrichli> hello
21:01:05 <cutforth> o/
21:01:05 <kota_> o/
21:01:05 <ho> o/
21:01:07 <torgomatic> who's on first
21:01:12 <dmorita> o/
21:01:15 <notmyname> torgomatic: right
21:01:16 <acoles> \o
21:01:26 <redbo> ¯\_(ツ)_/¯
21:01:45 <hurricanerix> o/
21:02:06 <notmyname> thanks for coming
21:02:06 <notmyname> let's get started
21:02:11 <notmyname> #link https://wiki.openstack.org/wiki/Meetings/Swift
21:02:17 <notmyname> #topic hackathon
21:02:25 <notmyname> #link https://www.eventbrite.com/e/swift-hackathon-tickets-17308818141
21:02:35 <notmyname> a reminder that the hackathon is coming up in august
21:02:45 <notmyname> looks like all the tickets have been taken
21:03:02 <ho> great!
21:03:09 <notmyname> please let me know if there are other people we need to talk to
21:03:09 <notmyname> I'm working with jrichli on the space we have and who's signed up
21:03:14 <notmyname> any questions about the hackathon?
21:03:15 <minwoob_> Can't wait!
21:03:24 <minwoob_> :)
21:03:33 <peluse> did anyone else have to pay for their hackathon ticket?  :)
21:03:33 <notmyname> the hotel room block info is on the invite
21:03:42 <notmyname> peluse: that was just for you ;-)
21:03:52 * peluse is such a sucker
21:04:00 <mattoliverau> lol
21:04:18 <hurricanerix> notmyname: is there a etherpad for the agenda?
21:04:41 <notmyname> hurricanerix: no, not yet. normally we use a flip chart on the first day ;-)
21:04:50 <notmyname> or whiteboard
21:05:07 <notmyname> hurricanerix: if you want to write some stuff down, I think that would be great
21:05:09 <jrichli> oh, we are gonna need one of those?  ;-)
21:05:16 <notmyname> I'm not too worried about a schedule yet
21:05:32 <hurricanerix> notmyname: ok, i was just curious to see what everyone else was interested in ahead of time.  =)
21:05:47 <notmyname> hurricanerix: you know, swift stuff
21:06:15 <timburke> and swift *client* stuff!
21:06:20 <notmyname> that counts too!
21:06:28 <joeljwright> :)
21:06:30 <timburke> suuure it does...
21:06:40 <notmyname> hurricanerix: if you start one, let's link to it and we'll write stuff down there
21:06:55 <notmyname> if there are any questions that come up, or that people ask you, please let me know if I can help
21:07:10 <notmyname> ok, next up..
21:07:16 <notmyname> #topic swiftclient issues
21:07:36 <notmyname> this one I think has just been resolved, but I want to talk about what happened and what didn't happen
21:08:10 <notmyname> if you look on this dashboard: http://goo.gl/qHus5v
21:08:34 <notmyname> you'll see that there are a lot of patches that are ready to land for swiftclient that haven't landed
21:08:41 <notmyname> 6
21:08:58 <notmyname> I'll get to the reason in just a bit
21:09:20 <notmyname> we've been in this state for over a week, and that's not good
21:09:39 <notmyname> timburke has done a great job and finding and fixing the gate issue that's blocking all of them
21:10:05 <notmyname> but I'd rather not see it take a week or more before someone looks into it. we can do better
21:10:19 <notmyname> so here's what happened:
21:10:51 <notmyname> global-requirements updated some dependencies to include a ";python==<version>" after the version classifier for dependencies
21:11:01 <notmyname> and that new format isn't parseable by pbr<1.2
21:11:45 <notmyname> however, because of the way dependencies are installed in the gate, by the time we had the opportunity to say we wanted to install pbr>=1.2, we had to parse the requirements file, which had the weire semicolon stuff in it
21:11:47 <notmyname> so it broke
21:12:32 <notmyname> the fix is a patch that timburke jsut submitted to update the version of hacking in our requirements file. the older version of hacking had pbr<1.0. so it didn't work
21:12:46 <notmyname> so far we're lucky we haven't hit it in swift yet, but we need to do the same patch there too
21:12:53 <redbo> All this pain has been worth it for whatever it is that pbr does.
21:12:56 <joeljwright> timburke: good catch!
21:13:16 <notmyname> redbo: oh, I'm sure someone somewhere is getting pbr to make them breakfast every morning ;-)
21:14:20 <notmyname> so, yes it's really annoying the way dependencies are found int he gate. and yes it's irritating that we have to update a dependency on hacking just to get a new version of pbr so the gate can install global (not local) requirments. and all of this with us not doing anything new with hacking
21:14:32 <notmyname> I won't argue with any of that
21:14:58 <redbo> would it help if we got rid fo pbr?  I bet it would help if we got rid of pbr.
21:15:14 <notmyname> but I did want to bring up that (1) it's fixed and there's a change we need in swift too and (2) it's not good it took us a week or two before we even started looking at it
21:15:50 <lifeless> ++ - I've been helping projects all week do their analysis. I mailed about the issue to the dev list to give folks a heads up
21:15:57 <lifeless> and would have been delighted to help swift too
21:16:26 <notmyname> lifeless: connecting the ML message (which I did see) to the exact issue we were seeing wasn't obvious. which is why we didn't know the solution until today
21:16:28 <lifeless> (e.g. http://lists.openstack.org/pipermail/openstack-dev/2015-June/068031.html )
21:16:30 <notmyname> lifeless: but thanks for your offer
21:16:48 <timburke> notmyname: nit: in 37 mins (supposedly) it will be fixed. the one patch that will actually pass the gate still needs to get through it :)
21:16:58 <notmyname> timburke: ok :-)
21:17:02 <lifeless> notmyname: ah - thats a shame. I'll have a look at some recent swift logs I guess to see what you saw that was different
21:17:30 <notmyname> lifeless: mostly probably our own ignorance of how all the pieces are put together. it's not exactly a simple system ;-)
21:17:43 <lifeless> notmyname: was it swift or swiftclient failing ?
21:17:55 <mattoliverau> nice work timburke, and yeah fail on 2. I for one will try and keep a better look at the state of the gate in the future.
21:17:58 <notmyname> lifeless: swiftclient, but swift has the same issue still (just lucky to not yet be affected)
21:18:13 <hurricanerix> notmyname: https://etherpad.openstack.org/p/swift-midcycle-aug-2015
21:18:20 <notmyname> hurricanerix: thanks
21:18:44 <mattoliverau> So its been a fun week of gate bugs then :(
21:19:01 <cschwede> exactly - there was also a broken gate due to swift func tests this week - the thing with the maxheaders and a python update
21:19:10 <notmyname> cschwede: yeah
21:19:13 <lifeless> ah yeah, http://logs.openstack.org/46/197646/1/check/check-swiftclient-dsvm-functional/9dca238/console.html#_2015-07-01_17_16_55_731 is a slightly different error message
21:19:31 <notmyname> ok, so let's move on to more interesting ongoing work. lot's more tocover today :-)
21:19:39 <notmyname> #topic ongoing work
21:19:43 <lifeless> ok so future ref: I'm very happy to be pinged on anything weird being thrown out of pip install / setup.py etc.
21:19:48 <notmyname> lifeless: thanks
21:19:53 <notmyname> jrichli: acoles: anything to cover on encryption work?
21:19:58 <mattoliverau> lifeless: thanks man
21:20:03 <jrichli> patch 163806 got merged to the feature branch - yay!
21:20:03 <patchbot> jrichli: https://review.openstack.org/#/c/163806/
21:20:17 <kota_> great!
21:20:18 <jrichli> clayg gave me some good feedback in the trivial keymaster review and has been helping me get things moving in feature/crypto
21:20:25 <notmyname> good
21:20:40 <jrichli> we need the real footers support in place of fake_footers patch 165517
21:20:40 <patchbot> jrichli: https://review.openstack.org/#/c/165517/
21:20:59 <jrichli> I will need some guidance on what approach to take. acoles was going to tackle this one, but he is working other things now.
21:21:05 <notmyname> jrichli: who's working on that?
21:21:07 <notmyname> ah ok
21:21:18 <notmyname> acoles: any ETA on when you'll be "back"? :-)
21:21:55 <acoles> notmyname: err, hopefully a week or so, i'm starting to get some time for upstream back
21:22:02 <notmyname> great :-)
21:22:10 <acoles> notmyname: oh, but then i have vacation
21:22:12 <jrichli> yay
21:22:13 <acoles> sorry!
21:22:23 * acoles #MustWorkHarder
21:22:26 <jrichli> well, good for you, tho! you deserve it
21:22:36 <mattoliverau> acoles: but working upsream is like a vacation :P
21:22:38 <notmyname> there's no vacation in swift!
21:22:42 <notmyname> mattoliverau: nice!
21:23:02 <acoles> notmyname: i'm trying to progress fast-post when i can, and yeah sorry that i havent had much time for encryption stuff
21:23:07 <tdasilva> tell that to his kids
21:23:10 <notmyname> jrichli: ok, what do you need this week on encryption?
21:23:17 <jrichli> having these patches land is important for crypto too: https://review.openstack.org/#/c/134347/ and https://review.openstack.org/#/c/156923/
21:23:22 <acoles> mattoliverau: heh. what can i say in response
21:23:24 <jrichli> so reviews on them
21:23:25 <acoles> :)
21:23:31 <mattoliverau> :)
21:24:02 <notmyname> jrichli: looks like they both have a merge conflict.
21:24:17 <notmyname> tdasilva: can you or ppai update those?
21:24:36 <tdasilva> yes
21:24:39 <notmyname> thanks
21:24:40 <tdasilva> will do
21:24:52 <notmyname> ok, on to EC testing
21:25:07 <notmyname> peluse: kota_: minwoob_: I know each of you have been looking at EC stuff
21:25:15 <notmyname> what's the status there?
21:25:46 <peluse> yeah, I'm on my 3rd rewrite of the GET path ability to tolerate duplicate FI's from the nodes (and subsequently request available alternates identified in headers)
21:25:53 <peluse> will post what I have by EOW, working on not...
21:26:01 <peluse> sounded so easy :)
21:26:06 <peluse> but its getting close
21:26:17 <notmyname> ok, great
21:26:22 <notmyname> our current EC bugs are at https://bugs.launchpad.net/swift/+bugs?field.tag=ec
21:26:31 <minwoob_> I have 3 open patches right now. They're failing on gate-swift-python34
21:26:32 <notmyname> looks like several are in progress now
21:26:40 <minwoob_> Other than that, they should be ready to be merged.
21:26:51 <peluse> yeah, and then it can be extended to potentially ask for specific FI's even w/o a .dirable for other strange cases (but that will be a different patch)
21:26:52 <notmyname> minwoob_: that's expected right now, and that doesn't vote on the gate. so it shouldn't be a blocker for anything
21:27:27 <kota_> I was not working so much in the last week because I got cold and was in bed most of the time.
21:27:37 <kota_> just reviewed a few stuff.
21:27:42 <notmyname> kota_: sorry to hear that. hope you're feeling better
21:27:52 <mattoliverau> kota_: know that feeling, hope your felling better
21:28:26 <mattoliverau> lol
21:28:46 <kota_> notmyname:mattoliverau: thanks :)
21:28:56 <notmyname> we've got to get the EC patches in soon so that we can have it usefully usable for people. please keep track of the bugs and patches for them
21:29:03 <notmyname> I'll start he EC bugfix patches too
21:29:06 <notmyname> *star
21:29:13 <notmyname> so they show up on the review dashboard
21:29:31 <notmyname> any questions on EC work?
21:29:44 <notmyname> ok, moving on then
21:29:44 <minwoob_> Okay. I will ask the cores who have reviewed mine, to revisit them soon.
21:29:50 <notmyname> minwoob_: thansk
21:29:54 <minwoob_> Np.
21:30:13 <notmyname> torgomatic has been working on some ring data placement improvements and has a fun new ring tool.
21:30:15 <mattoliverau> minwoob_: I'll look this morning
21:30:21 <notmyname> torgomatic: can you share a summary?
21:30:33 <notmyname> https://review.openstack.org/#/c/197396/
21:30:51 <torgomatic> two parts:
21:31:23 <torgomatic> part the first: there's a tool that simulates a bunch of rebalances/set-weights/etc. and shows you how many parts changed at each step as well as how many rebalances it took to converge
21:31:42 <notmyname> that sounds great
21:31:45 <torgomatic> this is useful with the second patch (Patch Jr.), which fixes a bug in the ring builder
21:32:00 <torgomatic> basically, with overload > 0, it'd move the stuff that needed to move *and* some stuff that didn't
21:32:17 <torgomatic> so it took more rebalances to converge and moved more partitions along the way
21:32:34 <torgomatic> and... uh... it doesn't do that now
21:32:41 <torgomatic> (with patch 2, I mean)
21:32:41 <patchbot> torgomatic: https://review.openstack.org/#/c/2/
21:32:54 <notmyname> silly patchbot
21:32:56 <torgomatic> uh
21:33:10 <mattoliverau> lol
21:33:26 <notmyname> so rebalance with your patch, especially with unbalanced clusters using overload, less data movement now?
21:33:31 <torgomatic> so that's what it does, and that's why it's dependent on the ring builder analyzer is so you can actually see it improve
21:33:37 <torgomatic> notmyname: excellent summary
21:33:40 <notmyname> yay
21:33:46 <torgomatic> *only* with overload > 0
21:34:27 <torgomatic> overload == 0 doesn't seem to move extra stuff as of master today, and the changes would end up just being a bunch of +0 and *1 operations for overload==0
21:34:36 <notmyname> I'm really excited about several of the recent proposed and landed patches. 1+ object server/port, async container updates, ring balance fixes, ec patches. good stuff going on :-)
21:35:09 <notmyname> torgomatic: thanks for the update
21:35:16 <notmyname> any questions from anyone on it?
21:35:29 <mattoliverau> +1 nice work torgomatic
21:35:50 <notmyname> ok, next up
21:35:57 <torgomatic> mattoliverau: go put that on the patch in gerrit ;)
21:36:15 <notmyname> cschwede: I saw https://review.openstack.org/#/c/196755/ for "add notifications middleware" in my email. looks interesting. tell us more
21:36:19 <mattoliverau> torgomatic: ok ok, I've only just woke up :P
21:36:25 <cschwede> yup
21:36:28 <cschwede> it's a simple middleware that sits in the proxy pipeline, sending notifications on PUT/POST/COPY/DELETE. Designed to be used with OpenStack Zaqar, but anything that understands a POST request with a JSON body should be usable
21:37:05 <notmyname> cschwede: ah interesting that it's using zaqar. nice
21:37:19 <cschwede> it’s related to one of the recently merged specs
21:37:26 <notmyname> so anything that needs to be built on a durable notification queue can use it?
21:37:33 <cschwede> https://github.com/openstack/swift-specs/blob/master/specs/in_progress/notifications.rst
21:37:46 <notmyname> eg policy migrations? tape storage? auto tiering? maybe new contaienr sync?
21:38:19 <cschwede> notmyname: basically, yes. but it’s non blocking (for now); meaning that if the backends fails, swift continues and only logs errors
21:38:20 <notmyname> cschwede: so what's needed there or what's next? is that the first of several patches? is it all ready for review and merge now?
21:39:03 <cschwede> i think it’s ready for review right now. would like to get some initial feedback on it (already got some from Pete and Hisashi; thanks!)
21:39:47 <notmyname> ok
21:40:06 <torgomatic> yeah, definitely good that it sheds load like that
21:40:11 <notmyname> anything else going on that hasn't been covered yet? (before we move on to bugs and specs)?
21:40:17 <notmyname> anything else to report on?
21:40:46 <torgomatic> otherwise I could see a small Zaqar getting hammered by a Swift cluster, crashing, and then Swift saving the load up for when the Zaqar service gets repaired and kncking it over again
21:40:47 <torgomatic> *knocking
21:41:13 <notmyname> ok, moving on to some bug tracking
21:41:22 <notmyname> #topic bugs
21:41:30 <cschwede> torgomatic: exactly. but with the logs and txn_ids an operator might manually send notifications later, if that’s a hard requirement
21:41:47 <notmyname> we've already covered the EC bugs. looks like those are being worked through (great!)
21:42:04 <notmyname> there are 68 bugs that are in "new" state on swiftclient https://bugs.launchpad.net/python-swiftclient/+bugs?field.searchtext=&orderby=-importance&field.status%3Alist=NEW&assignee_option=any&field.assignee=&field.bug_reporter=&field.bug_commenter=&field.subscriber=&field.structural_subscriber=&field.tag=&field.tags_combinator=ANY&field.has_cve.used=&field.omit_dupes.used=&field.omit_dupes=on&field.affects_me.used=&field.has_patch.us
21:42:04 <notmyname> ed=&field.has_branches.used=&field.has_branches=on&field.has_no_branches.used=&field.has_no_branches=on&field.has_blueprints.used=&field.has_blueprints=on&field.has_no_blueprints.used=&field.has_no_blueprints=on&search=Search
21:42:07 <notmyname> bah!
21:42:16 <notmyname> #link https://goo.gl/uO4b7l
21:42:17 <notmyname> there
21:42:49 <notmyname> those need to be looked at and marked as confirmed or needs more info or invalid
21:42:54 <cschwede> some really old stuff there…
21:42:59 <notmyname> I'd appreciate help from anyone on triaging them
21:43:01 <notmyname> cschwede: yup
21:43:30 <joeljwright> notmyname: I can take a look at some, but not until after July 10th
21:43:32 <notmyname> joeljwright: is there anything (beyond the gate fix that timburke did) that needs special attention today?
21:43:34 <notmyname> ok
21:44:04 <joeljwright> there are quite few patches sitting without review for some time
21:44:36 <notmyname> torgomatic: does your gate plunger script do swiftcleint patches too?
21:44:42 <joeljwright> I'll try ot get people chased up on those too when I get through the software release I'm working on atm
21:44:54 <notmyname> joeljwright: thanks! that would be great!
21:44:55 <torgomatic> notmyname: don't think so; the job names are different
21:45:00 <torgomatic> not too hard to fix though
21:45:11 <notmyname> torgomatic: ok. we'll need to recheck a lot of them after timburke's patch lands
21:45:26 <torgomatic> notmyname: heh, looks like it tries to, but fails miserably :D :(
21:46:10 <notmyname> joeljwright: timburke: I'll work on getting all the swiftclient patches rechecked (ie I'll bug torgomatic about his script) after the gate issue is fixed for us
21:46:23 <joeljwright> notmyname: :D
21:46:31 <mattoliverau> deligation power :)
21:46:41 <timburke> several (including my own) have fallen off of the review dashboard's "Older open patches" section
21:46:43 <notmyname> anything else to cover on swiftclient right now?
21:47:07 <notmyname> timburke: yeah, I'd like to improve the dashboard to be better about patches falling off
21:47:19 <notmyname> timburke: seems it still needs work
21:47:30 <notmyname> timburke: I'd be happy to work on that
21:48:08 <notmyname> ok, last item on the agenda (looks like we actually made it through the whole thing this week!)...
21:48:16 <notmyname> #topic specs
21:48:26 <notmyname> here are the specs that have not landed
21:48:28 <notmyname> #link https://review.openstack.org/#/q/status:open+project:openstack/swift-specs,n,z
21:49:17 <notmyname> there's 3 with a +1 already
21:49:27 <notmyname> those are probably the quickest place to make improvements
21:49:44 <notmyname> ho: looks like I owe you a review on the oslo config spec
21:50:00 <notmyname> mattoliverau: have you seen any activity this week on the large container spec?
21:50:13 <mattoliverau> nope
21:50:16 <zaitcev> there's a spec?
21:50:26 <mattoliverau> but its just information on the current POC, so make sense.
21:50:31 <notmyname> https://review.openstack.org/#/c/139921/
21:50:38 <zaitcev> oh 139921
21:51:00 <notmyname> cschwede: anything on your container alias spec?
21:51:07 <mattoliverau> But I should patch it with more new data.
21:51:12 <notmyname> mattoliverau: ok
21:51:32 <cschwede> notmyname: no, not yet
21:51:41 <mattoliverau> I was hoping to patch after the summit version lands.
21:51:43 <notmyname> torgomatic: can you look at the keymaster discussion spec this week? https://review.openstack.org/#/c/183014/
21:51:49 <torgomatic> notmyname: sure
21:51:58 <notmyname> mattoliverau: yeah, that would be good. get something landed
21:51:59 <cschwede> i need to answer tdasilva’s questions though
21:52:00 <notmyname> torgomatic: thanks
21:52:02 <notmyname> cschwede: ok
21:52:17 <ho> notmyname: as for the oslo config spec, i don't use global oslo conf on my rbac patch. so i might set the spec to abandoned.
21:52:35 <notmyname> ho: ah, interesting. ok
21:53:12 <notmyname> anything else on specs to cover int he meeting this week?
21:53:27 <notmyname> #topic open discussion
21:53:38 <notmyname> anything else to bring up?
21:54:06 <notmyname> I've got a picture of contrib activity I worked on this weekend
21:54:29 <notmyname> http://d.not.mn/contrib_activity.png
21:54:45 <kota_> interesting
21:54:57 <notmyname> i'm slowly working on ways to see how and where we are active as a group to see how we can continue to grow and maintain a healthy community
21:55:12 <zaitcev> What are the color.
21:55:12 <cschwede> notmyname: nice work! are these just random colors or do they mean anything?
21:55:19 <notmyname> colors are random
21:55:45 <notmyname> the "activity period" in that rendering is 14 days. so there is a 14 day long bar after a commit lands
21:56:08 <zaitcev> I'm pretty much spending all my time on a genious plan to destroy Swift, so not much in the way of contribution at all.
21:56:14 <notmyname> :-)
21:56:26 <redbo> same
21:56:52 <notmyname> right now I expect to have a swift meeting next week, but I've been called to check in for jury duty. so who knows what will happen
21:57:03 <notmyname> I'll update the wiki page as soon as I hear something
21:57:12 <notmyname> anything else?
21:57:40 <mattoliverau> cschwede: did an awesome job of getting the MAXHEADERS patch up and fixing the gate last week!
21:57:59 <notmyname> oh yeah!
21:58:01 <mattoliverau> thought he deserved some kudos
21:58:03 <notmyname> I mean to bring that up
21:58:15 <cschwede> mattoliverau: notmyname: thx :)
21:58:16 <notmyname> any concerns backporting that patch to kilo and juno?
21:58:23 <notmyname> cschwede: thanks for doing it! great work
21:58:43 <notmyname> cschwede and timburke are my heroes this week for fixing gate issues
21:59:05 <mattoliverau> not, we should do it
21:59:19 <notmyname> I agree. I'll land those this afternoon
21:59:22 <mattoliverau> *nope (back porting)
21:59:41 <notmyname> ok, thanks everyone for coming
21:59:48 <notmyname> see you in #openstack-swift
21:59:52 <notmyname> #endmeeting