19:00:30 <notmyname> #startmeeting swift
19:00:30 <openstack> Meeting started Wed Dec 17 19:00:30 2014 UTC and is due to finish in 60 minutes.  The chair is notmyname. Information about MeetBot at http://wiki.debian.org/MeetBot.
19:00:31 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
19:00:33 <openstack> The meeting name has been set to 'swift'
19:00:43 <notmyname> hello, everyone
19:00:48 <notmyname> who's here for the swift meeting?
19:00:49 <gvernik> hello
19:00:50 <mattoliverau> o/
19:00:55 <kota_> o/
19:00:56 <cutforth> hola
19:01:17 <torgomatic> hi
19:01:57 <notmyname> #link https://wiki.openstack.org/wiki/Meetings/Swift
19:02:20 <notmyname> just a few things to go through today
19:02:37 <notmyname> acoles should be back online soon for the last topic
19:02:41 <lpabon> hi
19:02:45 <notmyname> first up. 2.2.1
19:02:52 <notmyname> #topic swift 2.2.1
19:03:16 <notmyname> RC for the release has been cut
19:03:18 <notmyname> #link https://github.com/openstack/swift/blob/master/CHANGELOG
19:03:22 <notmyname> there's the changelog
19:03:47 <notmyname> it's a relatively small release from the perspective of end-user features. but there's some nice improvements for ops in there
19:03:59 <notmyname> plan is to finalize the release at the end of the week
19:04:12 <notmyname> so if there's anything you find, please let everyone know sooner than later
19:04:18 <notmyname> two small notes about the release
19:04:38 <notmyname> first, the tagging has slightly changed due to setuptools now enforcing pep 440
19:04:55 <notmyname> previously the RC would be tagged with <version>rc<rc rev>
19:05:00 <notmyname> eg 2.2.1rc1
19:05:07 <notmyname> then we'd finalize to 2.2.1
19:05:23 <notmyname> but, pep440 normalizes this to 2.2.1c1
19:05:28 <notmyname> note the s/rc/c/
19:05:42 <clarkb> notmyname: to make things even more confusing pep440 may be updated to normalize rc to c in the future because people are complaining
19:05:48 <clarkb> but we will let you know if that happens
19:05:59 <notmyname> clarkb: yay
19:06:07 <clarkb> er sorr c to rc
19:06:13 <clarkb> basically go back to what we were doing before...
19:06:17 <notmyname> wow
19:06:27 <mattoliverau> lol, woo!
19:06:31 <torgomatic> here comes the new pep, same as the old pep
19:06:32 <notmyname> so since we want the tarball to reflect the name of the tag, that means that we have to normalize the tag name according to pep440
19:07:01 <notmyname> so basically, right now you'll see these tags: 2.2.1rc1 and 2.2.1c1
19:07:20 <notmyname> the rc is there because we tried it and it broke things (surprise!) so we retagged
19:07:33 <notmyname> assuming pep440 stays the same, then future rc tags will be of the "c" form
19:07:46 <notmyname> so that's a thing. now you know
19:08:00 <notmyname> ok, for the other "issue" with 2.2.1
19:08:05 <notmyname> not really an issue
19:08:19 <notmyname> there were/are a lot of nice patches in progress that aren't in the 2.2.1 release
19:08:30 <notmyname> not because of lack of trying. and not because they aren't good
19:09:07 <notmyname> so I expect we'll have another release (2.2.2 or 2.3) in the middle to end of january. which should include those patches
19:10:09 <notmyname> why didn't we just wait on 2.2.1 for those patches? a few reasons. we don't know exactly when they'll land. we have a bunch of good stuff already. and, socially, it's nice to have a release right at the end of the year (ie in a marketing "we're here" sense)
19:10:26 <acoles> hi sorry i'm late
19:10:32 <notmyname> acoles: welcome
19:10:47 * acoles ran into fog on commute
19:11:03 <notmyname> any questions about the 2.2.1 release? any concerns or outstanding issues?
19:11:54 <notmyname> ok. let's move on
19:12:04 <notmyname> #topic ec status
19:12:14 <notmyname> peluse: clayg: torgomatic: any updates for the week on EC?
19:12:26 <torgomatic> I've been doing other things this week
19:13:47 <mattoliverau> i knew it torgomatic must be santa :P
19:13:55 <notmyname> heh
19:14:05 <torgomatic> ho ho oh crap nothing works what have I done
19:14:11 <torgomatic> ^ my week
19:14:13 <torgomatic> :)
19:14:30 <mattoliverau> lol
19:14:46 <notmyname> #topic priority reviews
19:15:02 <notmyname> a few things I wanted to ask about
19:15:08 <notmyname> #link https://wiki.openstack.org/wiki/Swift/PriorityReviews
19:15:38 <notmyname> tdasilva: anything new on the fsync dirs patch from ppai https://review.openstack.org/#/c/126923/ ?
19:15:54 <notmyname> tdasilva: looks like we're waiting on feedback from the author there
19:16:16 <tdasilva> notmyname: I was talking to ppai today and he has been running some benchmark numbers and also trying out different approaches that were suggested
19:16:25 <notmyname> ok
19:16:57 <tdasilva> notmyname: I asked him to post those results so people can see and comment
19:17:05 <tdasilva> which I think he will do soon...
19:17:06 <notmyname> thanks
19:17:08 <notmyname> tdasilva: also looks like you just pushed a new patch this mronign fro versioned writes middleware
19:17:25 <tdasilva> notmyname: yeah, but now running into issues with in-process func tests :(
19:17:43 <tdasilva> notmyname: still trying to figure things out...might ask for clayg's help later
19:17:48 <notmyname> tdasilva: ok. do you need anything from us or are you still working through that
19:17:49 <notmyname> ok
19:18:58 <tdasilva> notmyname: still working on it, hopefully I can find a solution soon...
19:19:19 <notmyname> there are 2 patches about data placement that I'm following closely: https://review.openstack.org/#/c/140478/ to warn in ring builder and https://review.openstack.org/#/c/140879/ from torgomatic
19:19:29 <notmyname> and torgomatic is working on another idea right now too
19:20:20 <notmyname> these are ones that are affecting some prod clusters today (also customers), so they're more important in my mind :-)
19:20:54 <notmyname> and clayg's patch for large containers https://review.openstack.org/#/c/141019/ is very nice, since I know that's hit prod clusters before too
19:21:17 <notmyname> mattoliverau: ^^ you might find some sympathy for that one in the cloud files team
19:21:30 <mattoliverau> :)
19:22:29 <notmyname> I updated the shortlinks to review dashboards with something that's more useful (to me)
19:22:55 <notmyname> http://goo.gl/JL65l7 and http://goo.gl/r2mxbe
19:23:00 <notmyname> use them as you wish :-)
19:23:11 <notmyname> any other specific patches to get a status on or ask questions about in here?
19:23:42 <acoles> notmyname: the xlo manifest versioning ones
19:23:59 <notmyname> acoles: ya, getting there with the next topic (yours) :-)
19:24:08 <acoles> ok cool
19:24:33 <notmyname> ok, then let's move on...
19:24:44 <notmyname> #topic versioned writes and manifests
19:24:47 <tdasilva> notmyname: just a quick question on priority reviews
19:24:49 <notmyname> acoles: want to vie
19:24:53 <notmyname> err
19:24:55 <notmyname> tdasilva: ok
19:25:02 <tdasilva> any update on migration middleware?
19:25:06 <clayg> weeeeee!
19:25:14 <notmyname> tdasilva: what clayg said
19:25:15 <notmyname> ;-)
19:25:26 <mattoliverau> notmyname: no negitive feedback filter... seems to have patches with neg feedback.. unless it's just me
19:25:56 <tdasilva> gvernik: ^^^?
19:26:29 <notmyname> mattoliverau: I'm not seeing that, but if so let's figure it out
19:27:15 <clayg> oh i like "in the gate queue"
19:27:17 <mattoliverau> notmyname: could just be my crazy bad english locale :P
19:27:20 <clayg> makes me feel like we're doing stuff
19:27:53 <notmyname> tdasilva: gvernik: unfortunately, I think the reality is that it hasn't gotten enough support to get through. and I don't think we (I) have handled that well. but we've been saying "let's get consensus" for a while, and it hasn't happened
19:27:55 <clayg> notmyname: i don't really get why we have two of them
19:28:31 <notmyname> clayg: use whatever works for you. the "other" one I like because it gives me a sense of current activity vs "review this stuff now"
19:29:09 <notmyname> gvernik: tdasilva: clayg: do you think that ^ is a fair sumary of the migration middleware patch?
19:29:54 <tdasilva> notmyname: yeah, sounds good...I just wanted to offer my help in case it is needed...
19:30:20 <clayg> notmyname: I think I've said enough on the subject - I tried to offer two solutions to either a) do something I can get on board with or b) steamroll over me with no hard feelings
19:30:50 <notmyname> clayg: ya, just including you since you've been vocal in the past :-)
19:31:08 <clayg> i'm always vocal - probably a good reason to exclude me as much as include
19:31:08 <tdasilva> clayg, notmyname: i liked the 'solution' we came out at the conference...it seemed reasonable...
19:31:14 <notmyname> lol
19:31:54 <notmyname> tdasilva: definitely you can keep working with gvernik on it. if getting a fresh set of eyes and new life in it helps, so much the better for everyone
19:32:26 <tdasilva> notmyname: ok. I'll ping gvernik later, he doesn't seem to be around now.
19:32:28 <notmyname> ok...any more on priority reviews before talking about versioning + manifests?
19:32:33 <notmyname> tdasilva: ok, thanks
19:32:48 <notmyname> acoles: ok, want to give a summary of the current state and the outstanding question?
19:33:22 <acoles> yeah, so there are two reviews pending, one to make SLO manifests be versioned the other to stop DLO manifests being versioned
19:33:34 <acoles> and the docs say manifests aren't versioned
19:33:40 <tdasilva> hehe
19:33:42 <clayg> tdasilva: I wonder if you can trick cschwede into helping out?  I think container-alias would be a pretty nice to have feature and a low hanging fruit for a consumer of a new migration/alternative-source framework - I'd be interested to help with the plugin-entry-points work - i tried doing it once for daemons and I keep thinking I want to do it for diskfile and probably storage policies too
19:34:07 <clayg> acoles: heh
19:34:12 <acoles> so i just wondered if collective memory knows why the doc says that?
19:34:24 <acoles> and any opinion on which way we go?
19:34:41 <acoles> and is there a reason for dlo to be different from slo?
19:34:49 <notmyname> acoles: IIRC the doc says no versioning for manifests because versioning ends up doing a copy. which squashes manifests into one object (or errors if it's too large for a single object)
19:34:50 * acoles just wants to know how to vote
19:34:51 <clayg> i bet it went like "What happens if you add x-version-location to a container and then put a manifest in it?" "Dunno, let's say it's unsupported"
19:35:09 <acoles> clayg: :D
19:35:34 <acoles> notmyname: good point
19:35:45 <clayg> notmyname: you're probably onto something with the COPY squash behavior being problimatic
19:36:03 <tdasilva> since the DLO is also a content-length=0 obj, would it make sense to version it?
19:36:19 <tdasilva> i mean the DLO manifest
19:36:23 <notmyname> if the copy/squash did something different, I think it would be possible (probably still confusing). but changing copy/squash is probably a Hard Thing
19:36:35 <acoles> tdasilva: its the header that points to the segment container that counts
19:36:37 <notmyname> tdasilva: I'd prefer to keep them the same
19:37:07 <acoles> right now dlo and slo manifests get copied to the versions container, but get messed up on way back
19:37:23 <acoles> sounds like we should fix it so the manifests dont even go there?
19:37:31 <notmyname> acoles: manifests get copied? wouldn't they be squashed?
19:37:42 <clayg> if you overwrite a xlo manifest in a versioned container it should probably just create a copy of the manifest - and leave squash out of it
19:38:11 <notmyname> this probably gets really confusing for tdasilva
19:38:13 <tdasilva> I believe in the DLO case it is copying the manifest
19:38:15 <notmyname> the middleware patch
19:38:16 <clayg> I think there's a flag you can sent to manifests (for get's anyway) that says (just give me the manifest) - if the version copy did that I think it'd be fine
19:38:27 <notmyname> clayg: ya, for both DLO and SLO
19:38:33 <acoles> no the manifest gets copied, but on way back headers are lost so it no longer behaves like a manifest
19:38:33 <clayg> messed up on the way back?
19:38:39 <notmyname> ?multipart_manifest=get IIRC
19:38:46 <clayg> acoles: sounds like a but with versions
19:39:03 <acoles> clayg: xlo headers aren't copied back from versions container
19:39:09 <clayg> acoles: no headers/metadata should get added/removed in the copy?
19:39:17 <clayg> acoles: yah just sounds like a bug
19:39:33 <clayg> like I said - i'm *pretty* sure someone one just said it's to confusing, unsupported
19:39:40 <acoles> clayg: i *think* only user meta gets copied by default
19:39:43 <clayg> and now you're all going trying to make sense of it
19:40:00 <tdasilva> hehe...like notmyname said, I need to make sense of it
19:40:08 <tdasilva> for the middleware
19:40:10 <clayg> just make a purposal that says how you think it should work and if that makes sense and is implementable we can do that
19:40:16 <acoles> clayg: nah, i think i have heard enough, the doc wins!
19:40:21 <notmyname> tdasilva: also really affects where in the pipeline the version middleware will go
19:40:35 <notmyname> before or after xLO middleware
19:40:43 <tdasilva> notmyname: yes, i've been putting before
19:40:51 <clayg> tdasilva: I think should consider versions and manifest mostly green field - try to think about use cases - docs say it doesn't work so anyone relying on existing behavior is probably in for bad time regardless
19:40:57 <notmyname> ...if only we had some sort of pipeline solver tool....
19:42:12 <clayg> ok, so we have a plan?  continue to ignore it?
19:42:14 <notmyname> ya, clayg is right. figure out the sanest end-user story. what makes sense. it's not supported now.
19:42:32 <notmyname> so do we risk breaking clients one way or another?
19:42:46 <notmyname> if so, don't do that one. if not, do what makes sense
19:43:00 <clayg> notmyname: probably, but I think they're already off the rails - and we don't even have any tests to understand the current behavior - must less how we'd be changing it
19:43:26 <notmyname> acoles: tdasilva: what are you wanting to implement? what do you expect?
19:43:31 <tdasilva> clayg: good point, I think one of the tests we have is actually wrong
19:43:34 <acoles> notmyname: nothing works now, you don't ever get a sane manifest back from the versions container
19:43:36 <torgomatic> so (a) find out what happens now, (b) figure out if it's sane enough to keep around, and (c) do something sane
19:43:38 <notmyname> possible to version manifests? or not possible?
19:43:41 <clayg> acoles: probably just thinks its funny no one knows how it works
19:43:54 <clayg> tdasilva: is acctually worried he's going to "break" something with the version middleware
19:44:01 <notmyname> ah, yes. ok. it doesn't work now at all (because headers are stripped)
19:44:40 <clayg> torgomatic: i like that - it's broadly applicable - liek the scientifici method
19:44:40 <notmyname> so what do we want?
19:44:51 <torgomatic> TIME TRAVEL!
19:44:55 <notmyname> we know what happens now: it breaks
19:45:00 <notmyname> what do we want to happen?
19:45:04 <clayg> notmyname: i'm not goig to implement it - but i'll review it - let's see what tdasilva comes up with
19:45:09 <acoles> notmyname: i'm not proposing to go make manifests versionable. tdasilva's patch stops them leaking out to versions container, so that seems good based on this discussion
19:45:13 <clayg> notmyname: I want tdasilva to figure it out and tell me what to think
19:45:19 <notmyname> heh
19:45:24 <notmyname> tdasilva: it's all up to you!
19:45:24 <tdasilva> hehe
19:45:41 <notmyname> tdasilva is the one writing the actual code, so he's got all the votes right now :-)
19:45:41 <tdasilva> ok...i'll keep working on it and try to come up with a proposal to you guys
19:45:47 <notmyname> great!
19:45:54 <notmyname> acoles: thanks for bringing it up
19:46:02 <acoles> yeah, tell us tdasilva
19:46:03 <clayg> acoles: not copying something in a versioned container out to the version location when you overwrite it seems like an interesting choice :\
19:46:06 <clayg> but maybe
19:46:24 * acoles crawls back under stone
19:46:28 <notmyname> clayg: no! you don't get to make comments. tdasilva will tell you what to think!
19:46:38 <clayg> right!
19:46:40 <tdasilva> lol
19:46:49 <notmyname> #topic other stuff
19:46:54 <tdasilva> it's usually more the other way around
19:46:56 <notmyname> any thing else to bring up this week?
19:47:09 <notmyname> no meetings for the next 2 weeks
19:47:12 <cipher> I heard the at-rest encryption design was approved.  Has there been any implementation for that yet?
19:47:39 <notmyname> next meeting is jan 7
19:47:55 <ferndoggy> cipher good question
19:48:03 <notmyname> cipher: http://specs.openstack.org/openstack/swift-specs/specs/in_progress/at_rest_encryption.html
19:48:05 <ferndoggy> I also wish to know about swift encryption
19:48:29 <notmyname> what we're doing on specs is approving them not when they are 100% complete. but when we agree on something
19:48:44 <notmyname> keep them as living documents of stuff people are designing and working on
19:49:19 <notmyname> so the fact that the spec has been approved isn't quite accurate. more accurate is that the ideas as presented are agreed upon
19:49:51 <notmyname> the more pressing question is "now that there is something agreed upon, who's working on it?"
19:50:34 <notmyname> there were a _lot_ of people active on it so far https://review.openstack.org/#/c/123220/
19:51:13 <clayg> notmyname: not it!
19:51:17 <notmyname> torgomatic was working on it, but he's doing some placements stuff and EC right now
19:51:23 <notmyname> (ie before encryption)
19:51:40 <torgomatic> well, I worked on the spec... I might have typed "import pycrypto" at one point, but that's about the size of it
19:51:56 <notmyname> ferndoggy: cipher: so that's the status :-)
19:52:03 <torgomatic> and that's not even the crypto lib we're going to end up using, so...
19:52:23 <cipher> what lib will we end up using?
19:52:46 <torgomatic> cryptography, I imagine. It's already in the global requirements.txt.
19:52:51 <notmyname> https://github.com/pyca/cryptography
19:52:58 <torgomatic> yeah, that one :)
19:53:12 <cipher> ok, thanks
19:53:31 <ferndoggy> thank you
19:53:34 <notmyname> cipher: just because of your nick, are you interested in working on it? ;-)
19:53:48 <cipher> perhaps
19:54:38 <cipher> I am new to the topic, but will look over the spec and history.
19:54:54 <notmyname> cipher: cool. and please ask any questions in #openstack-swift
19:54:56 <ajiang> I assume that will be some shape of encryption in the L release?
19:54:56 <notmyname> we're nearly out of time. anything else?
19:55:21 <notmyname> ajiang: I'm not assuming anything on dates of implementation at this point
19:55:41 <clayg> ajiang: yeah didn't everyone just get done saying not a line of code has been typed?
19:55:56 <notmyname> EC is more of a priority than encryption right now
19:56:03 <clayg> ^ for me anyways
19:56:15 <clayg> dunno what everyone else has a bee in their bonnet about
19:57:30 <notmyname> ok, I'm calling it. thanks everyone for coming! have a great holiday!
19:57:32 <ajiang> yeah, my question is more like a project manager questions
19:57:41 <ajiang> thanks
19:57:50 <notmyname> #endmeeting