19:01:16 <notmyname> #startmeeting swift
19:01:16 <openstack> Meeting started Wed Aug 20 19:01:16 2014 UTC and is due to finish in 60 minutes.  The chair is notmyname. Information about MeetBot at http://wiki.debian.org/MeetBot.
19:01:18 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
19:01:20 <openstack> The meeting name has been set to 'swift'
19:01:30 <notmyname> who's here for the swift meeting?
19:01:32 <mattoliverau> o/
19:01:34 <kota_> o/
19:01:34 <torgomatic> hi
19:01:35 <tdasilva> hi
19:01:36 <portante> o/
19:01:40 <peluse> yo
19:01:41 <cutforth> present and accounted for
19:01:42 <cschwede_> o/
19:02:17 <notmyname> look! it's all my favorite people :-)
19:02:27 <dfg> notmyname: wrong channel
19:02:35 <notmyname> lol
19:02:37 <notmyname> #link https://wiki.openstack.org/wiki/Meetings/Swift
19:02:42 <notmyname> today's agenda ^
19:03:00 <notmyname> and, honestly, I'm hungry and this meeting is keeping me from lunch :-)
19:03:12 <notmyname> #topic hackathon
19:03:18 <notmyname> tdasilva: any updates for us?
19:03:31 <tdasilva> we have 20 people already signed up
19:03:43 <notmyname> tdasilva: how many spots are open?
19:03:46 <peluse> awesome
19:03:47 <tdasilva> 15
19:03:51 <notmyname> cool
19:04:23 <tdasilva> we are now trying to plan a cool even to take everybody out one night
19:04:24 <tdasilva> :)
19:04:28 <notmyname> nice
19:04:40 <notmyname> can you #link the url please?
19:04:48 <tdasilva> #link https://swift-hackathon.eventbrite.com
19:04:51 <tdasilva> is that right?
19:04:55 <notmyname> yup. thanks
19:05:08 <mjseger> tdasilva: can we at least get in 1 kimballs run?
19:05:23 <tdasilva> haha...that's an idea
19:05:29 <tdasilva> we are walking distance from there
19:05:43 <mjseger> I know, I used to be back at DEC
19:06:00 <notmyname> any questions about the hackathon from anyone?
19:06:39 * clayg back from googling kimball farm
19:06:50 <mjseger> ;)
19:06:56 <tdasilva> if people are looking for hotels to book, I've heard the hampton Inn is really nice
19:07:07 <tdasilva> pretty new
19:07:14 <tdasilva> just a bit farther away
19:07:23 <notmyname> thanks
19:07:31 <clayg> notmyname: are you and sam already signed up?  I think timing wise I could probably make it.
19:07:35 * notmyname adds getting flights/hotels to the todo list
19:07:44 <notmyname> tdasilva: is a car needed?
19:07:53 <tdasilva> yes!
19:07:56 <mjseger> isn't the westford regency close?  no idea about prices
19:08:18 <tdasilva> yeah...the westford regency is closer
19:08:22 <tdasilva> but it's an older building
19:09:03 <torgomatic> clayg: I have a hackathon slot, but no flights or hotels or anything yet
19:09:08 <notmyname> ditto
19:09:30 <mjseger> I'm driving from CT
19:09:37 <notmyname> if anyone has questions, feel free to bug tdasilva about it. ;-)
19:09:44 <notmyname> let's move on
19:09:49 <peluse> will there be a beer bus?
19:10:00 <clayg> peluse's head is in the right place
19:10:02 <peluse> :)
19:10:02 <tdasilva> oh....you set the bar really high peluse
19:10:07 <tdasilva> not sure we can match it
19:10:24 <notmyname> :-)
19:10:26 <peluse> I have no doubt it will be as productive as ever!
19:10:32 <notmyname> #topic TC gap analysis follow-up
19:10:50 <notmyname> thanks to everyone who did research and wrote up their findings since last week
19:10:53 <notmyname> #link https://etherpad.openstack.org/p/swift_gap_scratchpad
19:11:18 <notmyname> we've now got an overview of all of the oslo libraries, and you can read them there
19:12:12 <notmyname> this week I'l write up what the change to versioning and releases would mean, so we have all our info in one place
19:12:24 <notmyname> next week I'll be at the san antonio ops meetup, and I'll bring up this topic with the attendees
19:12:29 <peluse> yeah, thanks guys.  I'll go take a peek too
19:13:10 <notmyname> are there any questions about specific findings or additional comments? other things you might have found that you wanted to bring up?
19:14:00 <cschwede_> notmyname: for oslo.i18n, it might make sense to use it earlier to simplify things for the translation team. i could propose a patch for this
19:14:21 <notmyname> cschwede_: ok, thanks
19:14:48 <portante> dfg: has rack space thought about the impact oslo stuff might have on production?
19:15:03 <notmyname> I think that if there are particular libraries that offer obvious benefit (eg perhaps i18n) then we don't need to wait for permission to act on it. and then review as normal
19:15:28 <notmyname> portante: rick's comments on logging were good (he's on the cloud files team)
19:15:51 * portante goes an looks
19:16:24 <dfg> portante: rick is checking it out-
19:16:43 <tdasilva> rick == hurricanerix??
19:16:48 <notmyname> tdasilva: yes
19:16:48 <portante> thanks that is hurricanerix
19:17:06 <notmyname> sorry, his nick didn't tab-complete so I didn't use it :-)
19:18:10 <mattoliverau> Thats when you just butcher it for the meeting logs to to archive :P
19:18:27 <notmyname> next week I'll give a report from the ops summit and I hope we'll be able to have enough info to move forward in some direction. I expect the "Real" conversation to happen in paris
19:18:51 <notmyname> thanks again to everyone who wrote stuff up there. it's very very helpful
19:19:06 <notmyname> and speaking of paris...
19:19:13 <notmyname> #topic upcoming releases
19:19:18 <notmyname> what are we doing until then?
19:19:31 <mattoliverau> nice segway :)
19:19:35 <notmyname> here's my current plan for swift releases over the next few months
19:20:06 <notmyname> next monday (aug 25) cut an RC for 2.1.0
19:20:18 <notmyname> the current WIP changelog is at https://review.openstack.org/#/c/115167/
19:20:44 <notmyname> then do the final 2.1.0 release on sept 1 (the monday following)
19:21:09 <notmyname> which means that if there is stuff that needs to land in 2.1.0, get it landed this week (we'll get to specific patches in a moment)
19:21:45 <notmyname> then, for the 2.next release (either 2.2 or 2.1.1) that is in the integrated juno release, RC cut on oct 6 and the final release close to the oct 16 juno date
19:21:55 <notmyname> any questions or concerns with that plan?
19:22:09 <notmyname> or does it sound good?
19:22:13 <peluse> sounds good
19:22:20 <mattoliverau> sounds good to me
19:23:06 <notmyname> great
19:23:22 <notmyname> I just added a section to https://wiki.openstack.org/wiki/Swift/PriorityReviews for patches that need to land this week
19:23:50 <notmyname> ok, next topic
19:23:57 <notmyname> #topic sad story
19:24:05 <notmyname> this just came up this morning
19:24:53 <notmyname> on the openstack-dev mailing list, someone said he proposed a patch to swiftclient on github and it was rejected. he was asking if the ML was the place to put it, and was told to look at the gerrit workflow
19:25:08 <notmyname> his response was:
19:25:10 <notmyname> Thank you for your answer.
19:25:10 <notmyname> That workflow seems a huge job for me.
19:25:10 <notmyname> I leave this patch up to you.
19:25:40 <notmyname> I don't blame him for that, but it's sad to see the workflow actively discourage contributors. especially for the client
19:25:55 <notmyname> the message is http://lists.openstack.org/pipermail/openstack-dev/2014-August/043556.html
19:25:57 <notmyname> #link http://lists.openstack.org/pipermail/openstack-dev/2014-August/043556.html
19:26:48 <notmyname> so....while we can't change the workflow (or certainly won't in this meeting), I'd love to see someone look at what he was doing and see if it makes sense. if so, respond on the ML and submit the patch to gerrit
19:27:13 <notmyname> it seems to be about running swiftclient on windows. so that limits who knows how to do that
19:27:16 <notmyname> any volunteers?
19:27:31 <cschwede_> notmyname: i just have a running windows vm, will look into this
19:27:33 <clayg> notmyname: "That workflow seems a huge job for me.  I leave this patch up to you."
19:27:43 <notmyname> clayg: ya. exactly :-)
19:27:48 <notmyname> cschwede_: awesome! thanks!
19:28:13 <notmyname> #topic EC status
19:28:22 <notmyname> peluse: tsg: EC updates?
19:28:59 <peluse> ony my end, have been reviewing Ian's notes on the whole issues with partial PUT failures, reconsutrctor cleanup, etc., and preparing an alternate 2 pahse proposal for reivew
19:29:28 <notmyname> #link https://trello.com/b/LlvIFIQs/swift-erasure-codes
19:29:41 <tsg> notmyname: on my side, PyECLib/liberasurecode 1.0 is close to being done
19:29:45 <peluse> also getting close with the base replicator framework that I think I will propose it to land with placeholder/sstubs for some of the other items that are already on the list (the call in from ssync, the cleanup of old files, etc) just so we have the daemon in place to build on
19:29:58 <notmyname> tsg: good to hear
19:30:02 <tsg> notmyname: expect 1.0 by hopefully by end of this month
19:30:03 <peluse> tsg:  and also you landed the EC policy base support which is huge!
19:30:22 <tsg> yep, that's what I was about to mention - EC policy patch merged!
19:30:23 <peluse> argh, reconstructor I mean.  still  sick
19:30:28 <peluse> brain not working at full capacity
19:30:56 <tsg> notmyname: 4 more patches under review (quorum_size, PUT trailers/footers, PUT, GET)
19:30:59 <peluse> torgomatic:  what are thoghts on the 100 continue headers that you and tsg have been looking at for trailer support?
19:31:25 <notmyname> s/trailer/footer/ ;-)
19:31:32 <torgomatic> peluse: I think it'll work, but it'll need contributions upstream in multiple places... at a minimum, mod_wsgi and eventlet
19:32:00 <peluse> torgomatic: how feasible do you think that is and when you you think its time to pull the trigger on those?
19:32:02 <tsg> I am working on adding adding support for custom headers to the 100-continue responses in eventlet wsgi module
19:32:04 <torgomatic> we can start with a hacky, encapsulation-destroying way that works with just eventlet though, and then retrofit the good stuff on once it becomes available
19:32:23 <notmyname> the guy I was talking to who said he knew mod_wsgi basically said we shouldn't use HTTP. next up would be to talk about it on the mod_wsgi mailing list
19:32:55 <torgomatic> if mod_wsgi doesn't want this stuff in it, then I guess you can't have storage nodes using mod_wsgi and use EC
19:33:17 <peluse> ugh
19:33:31 <torgomatic> we're getting outside of what WSGI mandates and into the rest of HTTP, so it may be tricky
19:33:33 <notmyname> no, I don't think it's at that point
19:34:00 <notmyname> torgomatic: are you working on the hacky way this week?
19:34:51 <peluse> torgomatic:  would be good to see even a rough description of the hacky wacky thing, maybe in a discussion card on trello if you can
19:35:03 <torgomatic> notmyname: probably, once I get this ring stuff banged into shape
19:35:05 <notmyname> (for the rest of those who aren't up to speed ont he problem, the issue is figuring out how to handle object overwrites with an EC policy. you can't partially succeed and destroy the existing data. full discussion and plan is on trello)
19:35:13 <notmyname> torgomatic: got it. thanks
19:35:18 <torgomatic> peluse: I'll throw a gist together, or maybe submit to Gerrit... it's easier to see in python than english
19:35:22 <notmyname> thanks
19:35:46 <notmyname> for anyone not yet participating in the EC work, the trello board is up to date with what needs to be done.
19:35:47 <peluse> notmyname:  yeah, that's one part of it.  the other though is genereic footer support and the obj length issue dealing with rolling upgrades
19:36:01 <torgomatic> also how to get the original object's MD5/Etag stored on EC fragment archives when the client does not send it, given that the proxy is the only one who knows it, but it learns it after the headers have gone out
19:36:09 <torgomatic> ^^ the bigger issue IMO
19:36:09 <peluse> I believe that's ^ what's really driving tsg/torgomatic right now (correct me if I'm wrong)
19:36:13 <peluse> yes
19:36:15 <peluse> :)
19:36:24 <notmyname> ya, all of that :-)
19:36:29 <notmyname> so it's a Big Deal for EC
19:36:52 <torgomatic> tl;dr: HTTP 1.1 lets us do all this stuff, but WSGI semantics are a subset of HTTP semantics, so life is hard
19:37:14 <peluse> torgomatic/tsg/notmyname:  I think it may make sense to have a sync up phone call end of this week on this
19:37:49 <notmyname> I'd certainly encourage everyone to take a look at the outstanding work for EC. there's not a ton that's individually huge pieces of work
19:38:04 <notmyname> so jump in!
19:38:05 <torgomatic> peluse: sure, send emails and we'll figure it out
19:38:21 <peluse> https://trello.com/b/LlvIFIQs/swift-erasure-codes
19:38:21 <notmyname> any other questions or updates on the EC work to discuss?
19:38:35 <peluse> link for trello as a reminder ^
19:38:41 <notmyname> (again)
19:38:43 <tsg> torgomatic: ack.  The alternative to 100-continue could be to shove a "X-Trailer-Version" hint in object HEAD responses - can I get your critique on that?
19:39:12 <torgomatic> tsg: PUT doesn't always do a HEAD first, so that won't work
19:39:35 <torgomatic> er, the proxy code for handling an object PUT doesn't typically perform a HEAD request to the object servers, to be more specific
19:39:36 <tsg> torgomatic: true .. wanted to see if we can make it a default in the EC case
19:39:39 <peluse> torgomatic:  stupid question, can't we make it do that for the EC case
19:40:01 <torgomatic> eh, maybe, but if we can get it in one request, that'd be lots nicer
19:40:06 <tsg> I take that back .. this is "generic footer support" :)
19:40:13 <tsg> "generic" being the keyword
19:40:22 <peluse> ugh
19:40:48 <peluse> maybe for a first pass we reconsider generic footers and just go with EC specific, something to keep in the back of our minds
19:41:19 <notmyname> peluse: I'd like to see the ideas fleshed out for generic support. better to get it right up front if that's possible
19:41:36 <peluse> yup, agree... thus the 'back of our minds' comment :)
19:41:37 <tsg> torgomatic: I assume we can't make HEAD the default given the overhead?
19:41:37 <notmyname> IOW, keep torgomatic and tsg talking about it :-)
19:42:08 <torgomatic> tsg: yeah, the overhead is a problem, plus there's the race condition... since we don't keep TCP connections open for multiple HTTP requests, the obj server could change out between HEAD and PUT
19:42:16 <clayg> HEAD's can be tricky what with the serial digging into handoffs
19:42:17 <tsg> we have been talking about it here https://review.openstack.org/#/c/111642/
19:42:20 <torgomatic> it's small, but it goes away if we do it all in one request
19:42:45 <clayg> yeah eafp
19:43:11 <tsg> torgomatic, clayg:  makes sense
19:44:01 <peluse> tsg, I think you meant here https://review.openstack.org/#/c/111562/
19:44:33 <tsg> peluse: thanks - clipboard buffer mismatch :D
19:44:38 <tsg> torgomatic: I will come up with a suggestion for change to eventlet wsgi module in the way we discussed on gerritt and see how that goes
19:45:00 <notmyname> tsg: torgomatic: thanks for working on it
19:45:05 <notmyname> let's move on to patches
19:45:08 <notmyname> #topic patches
19:45:15 <notmyname> #link https://wiki.openstack.org/wiki/Meetings/Swift
19:45:25 <notmyname> doh
19:45:30 <notmyname> #link https://wiki.openstack.org/wiki/Swift/PriorityReviews
19:45:50 <notmyname> note the new 2.1.0 section for stuff that needs to land this week
19:46:16 <peluse> the blank section?
19:46:23 <notmyname> I haven't seen anything that would be strictly required to land this week, but I'd like to see a few things
19:46:26 <notmyname> peluse: ya :-)
19:46:34 <peluse> OK, just making sure :)
19:46:36 <notmyname> peluse: well, I only added it about 10 minutes ago
19:47:03 * notmyname is trying to remember what looked like good stuff to land this week
19:47:54 <notmyname> ph yeah
19:47:54 <notmyname> https://review.openstack.org/#/c/114120/
19:47:55 <cschwede_> https://review.openstack.org/#/c/91788/ is merged :) removing that (keystone v3 in swiftclient)
19:48:02 <notmyname> cschwede_: thanks
19:48:12 <alexpilotti> notmyname: I can help w the swiftclient patch
19:48:38 <notmyname> alexpilotti: which one?
19:48:44 <alexpilotti> notmyname: fixing OpenStack on WIndows is what we do all the time at CLoudbase :-)
19:49:00 <alexpilotti> notmyname: you have more than one? :-)
19:49:10 <notmyname> alexpilotti: ah, awesome. cschwede_ said he'd look in to it too. you two should talk :-)
19:49:20 <alexpilotti> notmyname: I just saw it here: https://github.com/openstack/python-swiftclient/pull/14/files
19:49:35 <notmyname> alexpilotti: ya, we had discussed it a few minutes ago
19:49:49 <alexpilotti> notmyname cschwede_: the patch looks good, I can test it out and send it in as ec
19:50:19 <cschwede_> alexpilotti: ok, nice, i will review it then. thanks!
19:50:55 <alexpilotti> notmyname: I cannot commit it in the name of the guy who sent the email, should I thank him in the commit message?
19:50:57 <notmyname> ok, the other things I'm looking forward too are the performance improvement patches (like torgomatic's splice() ones)
19:51:10 <notmyname> alexpilotti: ya, that's fine, I think
19:51:17 * torgomatic likes performance improvements
19:51:39 * peluse will put review of the splice() deal on his list of things to do...
19:52:02 <notmyname> ...and there are several smaller ones that I think can be knocked out pretty quickly (eg several from mattoliverau)
19:52:25 <notmyname> anyway, review! review! review! ;-)
19:52:40 <notmyname> mjseger: you wanted to bring up a performance issue you've been working on
19:52:46 <mjseger> sure
19:52:51 <notmyname> mjseger: somethign with the container replicator?
19:53:06 <mjseger> yes, see https://bugs.launchpad.net/swift/+bug/1332025
19:53:07 <uvirtbot> Launchpad bug 1332025 in swift "Useless journal file generation in db-replicator may suffer disk I/O performance in ext4 file system" [Undecided,New]
19:53:15 <notmyname> cschwede_: alexpilotti: https://bugs.launchpad.net/python-swiftclient/+bug/1359360
19:53:16 <uvirtbot> Launchpad bug 1359360 in python-swiftclient "generate Windows exe" [Undecided,New]
19:53:18 <mattoliverau> alexpilotti: you should add a 'Co-Authored-By:' to the patch.
19:53:41 <notmyname> mjseger: is that only on ext4 or does it affect xfs too?
19:53:44 <mjseger> but the simple statement is I've found a few places with sql updates are unnecessarily being done
19:54:00 <notmyname> mjseger: the fastest thing to do is to do nothing at all
19:54:05 <mjseger> I think it's mostly an ext4 thing but it's still extra i/o
19:54:11 <notmyname> ok
19:54:31 <mjseger> the fix is to see if an update is really needed and if not, don't make the sql call
19:54:42 <mjseger> I have 2 very small patches I can submit shortly
19:55:19 <mjseger> bottom line is on an ext4 system that's idle, there's a constant 1.5 MB/sec I/O, mostly the ext4 journalling daemon!
19:55:20 <notmyname> mjseger: cool.
19:55:30 <mjseger> the writes slow down the reads
19:55:33 <notmyname> mjseger: sounds like a great improvement
19:55:49 <notmyname> thanks for working on it
19:55:53 <mjseger> w/o the patch GETs go about 15MB/sec and with it they go about 70!!!!!!!!!
19:56:06 <mjseger> yowzer!!!
19:56:33 <mjseger> I'm done unless anyone has any questions.
19:56:34 <notmyname> mjseger: cool. are there specific questions before you submit the patch?
19:56:36 <alexpilotti> mattoliverau: ok!
19:56:37 <notmyname> ok :-)
19:56:48 <notmyname> #topic open discussion
19:56:55 <notmyname> anything else to bring up this week?
19:57:08 <mjseger> my only question is are the patched in the best places and that can be dealt with as part of the code review process, right?
19:57:09 <notmyname> or we can be done (and I can eat lunch!)
19:57:29 <notmyname> mjseger: yes. definitely the code review is the best place
19:58:49 <mattoliverau> if people are happy with the abandon reviews section in the dash can people review it here: https://review.openstack.org/#/c/109159/
19:59:03 <notmyname> mjseger: what's the status of your script?
19:59:07 <notmyname> err
19:59:09 <notmyname> mattoliverau: ^^
19:59:20 <mjseger> you mean my patches?
19:59:28 <mjseger> going to submit shortly
19:59:38 <peluse> mattgriffin:  will do
19:59:38 <notmyname> mjseger: no, sorry. mattoliverau's script for abandoned patches
19:59:44 <mjseger> ;)
19:59:55 <notmyname> heh. everyone is having problems with talking to mattoliverau
20:00:20 <peluse> argh
20:00:25 <mattoliverau> its seems to be working nicely, I get emails for expiring patches, and there is a list of ones the script thinks it show be abandoned
20:00:36 <notmyname> mattoliverau: cool
20:00:37 <mattoliverau> *should
20:00:51 <notmyname> and we're at time...
20:00:57 <notmyname> thanks everyone for coming!
20:00:59 <notmyname> #endmeeting