19:00:27 <notmyname> #startmeeting swift
19:00:28 <openstack> Meeting started Wed Mar 12 19:00:27 2014 UTC and is due to finish in 60 minutes.  The chair is notmyname. Information about MeetBot at http://wiki.debian.org/MeetBot.
19:00:29 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
19:00:31 <openstack> The meeting name has been set to 'swift'
19:00:37 <notmyname> who's here?
19:00:39 <portante> o/
19:00:47 <acoles-> hello
19:01:01 <torgomatic> .
19:01:05 <lpabon> hi
19:01:12 <peluse> yo
19:01:34 <notmyname> all right. thanks for coming
19:01:44 <notmyname> #link https://wiki.openstack.org/wiki/Meetings/Swift
19:01:47 <notmyname> agenda there ^
19:01:50 <notmyname> jsut a few things
19:02:08 <notmyname> but first, thanks to daylight savings time, this is now the current meeting time
19:02:16 <notmyname> glad you found it :-)
19:02:23 <peluse> didn't change for me :)
19:02:31 <acoles-> nor me:)
19:03:09 <portante> show offs
19:03:12 <zaitcev> Anyone who has to file flight plans knows about UTC
19:03:13 <portante> ;)
19:03:20 <notmyname> well, just for those of us in most of the US where the government is legislating daylight to account for 19th century farming techniques ;-)
19:04:01 <notmyname> ok, we;ve all been busy with stuff, so let' get started :-)
19:04:06 <notmyname> first up
19:04:13 <notmyname> #topic storage policies
19:04:30 <notmyname> this, as everyone knows is a big deal. and we're very close :-)
19:04:41 <notmyname> torgomatic: peluse: updates?
19:05:03 <clayg> is this the right place?
19:05:04 <peluse> on my end:  ssync, ready I think for final review thanks to review by clayg yesterday.  Need to get through some Jenkins hurdles though
19:05:32 <torgomatic> still working on the reconciler with clayg; I think the daemon that consumes the queue is probably working, and now it's time to work on putting things in that queue
19:05:38 <peluse> also on ym end:  acct rollup (head) patch to report usage.  Ready to go after I rebase once a dependency from torgomatic is done
19:05:38 <notmyname> peluse: looks like jenkins has been having some trouble lately.
19:06:01 <notmyname> torgomatic: that's option A. are you still going to do option B too?
19:06:16 <peluse> and finally for me... one helper patch for torgaomatic that is ready once Jenkins works
19:07:17 <notmyname> peluse: where do you need us to start looking at reviews?
19:07:20 <notmyname> which patch?
19:07:32 <peluse> I'll list them...
19:07:44 <peluse> ssync:  https://review.openstack.org/#/c/65347/
19:07:48 <peluse> ready now
19:07:53 <zaitcev> throw on wiki maybe?
19:08:02 <torgomatic> notmyname: yeah, probably once this one is done
19:08:04 <zaitcev> We used to have "priority review list"
19:08:05 <notmyname> torgomatic: ok
19:08:19 <notmyname> zaitcev: ya, it's still there. but I haven't updated it in a week or two
19:08:25 <peluse> and two others that have dependencies on reconclier stuff that torgomatic is doing so those should probably hold off more reivews
19:08:26 <notmyname> basically, https://review.openstack.org/#/q/status:open+project:openstack/swift+branch:feature/ec,n,z ;-)
19:08:33 <notmyname> peluse: ok
19:08:49 <peluse> but they've all been reviewd at least by one core already and are pretty solid I think
19:08:58 <clayg> torgomatic: it *was* working in reconciler-4 - I still need to merge with reconciler-5
19:09:00 <notmyname> peluse: torgomatic: what do you need? do you have any blockers?
19:09:26 <clayg> notmyname: I bet they're all in trello
19:09:41 <peluse> I just have ssync that is ready now for final review (Jenkins still churning on it for some unrleated flakiness though)
19:09:41 <clayg> I'm going to take a stab at reviewing peluse's 409 patch
19:09:45 <notmyname> https://trello.com/b/LlvIFIQs/swift-erasure-codes-and-storage-policies
19:09:52 <notmyname> clayg: thansk
19:09:53 <peluse> clayg:  its very small
19:10:04 <peluse> https://review.openstack.org/#/c/79731/
19:11:10 <notmyname> anything else to report or questions on these patches or ongoing work?
19:11:10 <peluse> other than that we need to rebase EC which I'll do after the meeting and then also I can run more tests on a real cluster
19:11:16 <clayg> zaitcev: did anyone look at the account/container-backend refactor for you yet?
19:11:28 <notmyname> clayg: zaitcev: hang on. we'll get there
19:11:46 * portante hangs his head in shame ...
19:12:15 <notmyname> anything else on the storage policy patches?
19:12:28 <peluse> dont think so
19:12:39 <zaitcev> I'm mostly nibbing them on the sides still. Looked at ssync, it seemed fine.
19:12:47 <clayg> torgomatic: peluse: can we re-order the stuff in trello so the sp stuff is up top and ec is at the bottom?
19:12:47 <notmyname> ok, so for the fun part of planning these thing...schedules
19:12:56 <clayg> ick
19:13:01 <torgomatic> clayg: I don't have a problem with that
19:13:13 <peluse> yup, want me to do it?
19:13:20 <notmyname> we're getting close to the openstack icehouse integrated release. which means we'll need a swift release there (ie 1.14)
19:13:25 <clayg> peluse: yes please
19:13:32 <notmyname> the goal is to include storage policies in that release
19:13:33 <notmyname> but
19:13:46 <peluse> cool, I'll make a seprate column for EC on treollo for now so its super clear
19:13:56 * clayg hugs peluse
19:13:56 <notmyname> we cannot (and will not knowingly) include it if it's not done
19:14:07 <notmyname> it's done when it's done
19:14:24 <peluse> sooooo close... we can't not get it done
19:14:24 <notmyname> but, it's also very good for us as a community to have it in the icehouse release
19:15:07 <notmyname> why? because all the non-technical parts of the community and ecosystem will be able to talk about it and use it
19:15:39 <torgomatic> peluse: that sounds like a challenge ;)
19:15:55 <peluse> indeed!
19:15:57 <portante> right ec is easy once sp is done
19:16:04 <zaitcev> really
19:16:07 <peluse> yipes, I didn't say that!
19:16:11 <portante> ;)
19:16:17 <torgomatic> it'll be done in the future... if it has to be done within the next N seconds, I recommend accelerating to relativistic speeds
19:16:17 <peluse> :)
19:16:29 <portante> warp speed mr. sulu
19:17:40 <notmyname> please, if there is anything that is blocking any part of this, let me or others know so we can help get it done
19:17:51 <notmyname> and then there's the merge-to-master will will take all of us
19:18:25 <notmyname> we've got about 4 weeks until the icehouse drop-dead date
19:18:46 <zaitcev> if we discuss it at hackathon, we need to come into the room with general knowledge of the code
19:18:48 <peluse> will do chief
19:18:51 <notmyname> any other questions around this topic?
19:19:27 <notmyname> ok. moving on
19:19:49 <notmyname> #topic "plugable back ends" ie DBBrokers
19:20:02 <notmyname> zaitcev: clayg: portante: ok now you can talk about it :-)
19:20:23 <zaitcev> ok... I thoght Acowles' sysmeta was next but I'll take it
19:20:37 <notmyname> it's an unordered list :-)
19:20:48 <acoles-> zaitcev: go for it
19:21:16 <clayg> zaitcev: I do sorta skim over the patch from time to time trying to let is soak in, I think I'm sorta getting my head around it (maybe)
19:21:42 <zaitcev> Paul was going to trade a review of 47713 for reviews on SP, so I tried to keep up.
19:21:47 <peluse> zaitcev:  I owe it another once over for sure since your last rebase
19:22:19 <peluse> zaitcev:  and I do greatly apprecaite the SP reviews!
19:22:48 <peluse> will take another look first thing tomrrow (today is booked with meetings)
19:22:53 <zaitcev> Initially I was really timid in refactoring transformations but it was pointed several times that resulting code was junk. Current PBE still contains things that I had to explain in comments.
19:23:19 <zaitcev> But generally each revision looks nicer, but harder to prove that it does not regress
19:23:37 <peluse> zaitcev:  what kind of regression testing have you done/plan on doing?
19:24:36 <zaitcev> peluse: Honestly, only functional tests and using the cluster, but I do not have real users. Like it seems like there's not even 1 manifested objects (they are all small).
19:24:39 <portante> this is where we might want to push on unit and functional tests ahead of it to help with the regressions
19:25:03 <clayg> portante: oh you know what would help with that - coverage reporting on functional tests!
19:25:06 <notmyname> portante: ya
19:25:14 <portante> yes
19:25:34 <portante> for sp, ec and pbe it would help us breath easier
19:25:45 <peluse> is someone working on that?  (coverage reporting for functional tests)
19:25:58 <portante> yes
19:26:13 <notmyname> portante: what's the link?
19:26:18 <portante> unfortunately, I am not driving it too hard, given my other work requirements
19:26:19 <portante> sec
19:26:22 <clayg> https://review.openstack.org/#/c/66108/
19:26:26 <notmyname> thanks clayg
19:26:31 <portante> yeah, thanks!
19:26:32 <clayg> oh uh....
19:26:34 <clayg> link https://review.openstack.org/#/c/66108/
19:26:39 <clayg> i don't know how this place works
19:26:48 <notmyname> #link https://review.openstack.org/#/c/66108/
19:27:33 <notmyname> as portante said, that has the very nice benefit of helping out with the other stuff going on too
19:28:07 <notmyname> zaitcev: do you have any more specific questions or requests about PBE?
19:28:07 <zaitcev> I'll have a closer look.
19:28:25 <notmyname> zaitcev: what, specifically, do you need from us? jsut reviews?
19:29:00 <portante> clayg, torgomatic, chmouel, notmyname : thanks for taking to look at it so far
19:29:03 <zaitcev> notmyname: Yes, reviews. If I drag it to hackathon, it won't have time because SP and EC are more important.
19:29:32 <notmyname> well, I hope SP and (most of ) EC is done by then!
19:29:40 <zaitcev> notmyname: I am available at any time, but just beating it with inline comments would do fine too.
19:29:46 <notmyname> zaitcev: ok, cool
19:30:13 <notmyname> #topic object sysmeta
19:30:17 <notmyname> acoles-: you're up
19:30:19 <acoles-> ok
19:30:33 <acoles-> i just wanted to explain the motivation behind the patch i put up today
19:30:37 <acoles-> #link https://review.openstack.org/#/c/79991/
19:30:59 <notmyname> this is the extension to the account+container sysmeta alreayd out, right?
19:31:00 <acoles-> there's a couple of mware patches pending that are stashing metadata against objects...
19:31:12 <acoles-> migration and encryption
19:31:35 <acoles-> the original sysmeta patch dropped support for objects...
19:32:11 <acoles-> 'cos the POST semantics are hard to get right. so this patch at least adds sysmeta for PUTs to objects, as a start
19:32:30 <torgomatic> how's it keep that stuff up-to-date on object POST? read-modify-write?
19:33:09 <clayg> post-as-copy is so weird
19:33:20 <acoles-> torgomatic: POST handled by copying existign sysmeta to .meta
19:33:21 <swifterdarrell> torgomatic: "Support for modification of object system metadata with a POST request requires further work as discussed in the blueprint.”
19:33:22 <clayg> it works, because "X-Newest"
19:33:34 <acoles-> clayg: yes!
19:33:43 <notmyname> well, that was my next question
19:33:43 <clayg> i was being snarky
19:33:58 * peluse writes himself a note to go lookup what post as copy is one of these days...
19:33:59 <notmyname> acoles-: should this wait until post-as-copy is merged back into a "fast post"?
19:34:03 <clayg> acoles-: did you test it with fast post?
19:34:14 <notmyname> peluse: default today is that a post is rewritten as a COPY
19:34:28 * clayg dreams of the return of fast-post
19:34:30 <acoles-> clayg: yes, test there for both fast and post-as-copy
19:34:33 <peluse> hmmm, just like is sounds eh?
19:34:39 <notmyname> peluse: pretty close :-0
19:34:45 <acoles-> notmyname: is post-as-copy going away?
19:35:00 <clayg> acoles-: one can hope
19:35:03 <notmyname> acoles-: well, there's been talk of unifying it back to the fast post methods
19:35:17 <torgomatic> I had a thing at one point that I thought might get rid of post-as-copy, but then I learned ssync collapses .data and .meta files together when it replicates, so now I'm not sure what to do about that
19:35:19 <notmyname> but I don't know if anyone has started typing on it
19:35:26 <clayg> acoles-: i don't think anyone is working on it - yeah torgomatic has an idea
19:36:14 <notmyname> acoles-: initially, I don't like the idea of supporting object sysmeta on PUT and not POST (I haven't looked at the patch yet). would getting rid of post-as-copy make it easier?
19:36:15 <clayg> we should just give all metadata timestamps X-Object-Timestamp-Meta-Foo: XXXX and let the reciever suss it out with the on disk data
19:36:26 <acoles-> notmyname: clayg: torgomatic: ok. i need to understand that direction more then because post as copy is particularly nasty to work around
19:36:59 <notmyname> acoles-: perhaps you could do the post-as-copy removal and then the object sysmeta? :-)
19:37:13 * clayg knows acoles- is the man for the job!
19:37:19 <acoles-> Apart from today's patch I also dumped some ideas for POST into wiki...
19:37:27 <acoles-> #link https://blueprints.launchpad.net/swift/+spec/object-system-metadata
19:37:41 <notmyname> acoles-: I bet you could even get some comments from clayg and torgomatic and others if you were doing the typing :-)
19:38:02 <acoles-> and I'm happy to try coding it up but I'd value feedback on the wiki
19:38:19 * clayg reading now
19:38:29 <notmyname> acoles-: cool. thanks
19:38:48 <acoles-> ...'cos i'm newer than most of you to the code :)
19:39:00 <notmyname> acoles-: is that a good place to stop the obj sysmeta conversation for this meeting? or do you have something else on it?
19:39:22 <acoles-> no more. just a heads up and plea for feedback. thanks
19:39:26 <notmyname> kk
19:39:29 <clayg> acoles-: consider awareness raised!
19:39:30 <notmyname> #topic open discussion
19:39:31 <zaitcev> okay
19:39:42 <notmyname> acoles-: thanks for spending some time reviewing the migration middleware
19:40:17 <acoles-> notmyname: np
19:40:22 <peluse> question:  ssync is listed in conf file samples as 'not production' - are these plans for when this will be consisdreed production, icehouse maybe?
19:40:24 <notmyname> also, everyone note that there's been some discussion about getting rid of blueprints and using text files in a git repo (managed via gerrit) instead
19:40:25 <zaitcev> so, about that container alias thing. Looks like Clay thinks benefit/cost is not good enough
19:40:43 <notmyname> peluse: I think it's mostly waiting on RAX to say "yup, we're running it everywhere now"
19:40:56 <clayg> word from RAX is that that ssync is doing pretty good for them!
19:41:06 <peluse> OK, yeah I asked gholt and he said they use it in productions clusters now so
19:41:08 <notmyname> clayg: ya, it's running in 1 to 2 of their clusters now
19:41:13 <clayg> I play it in dev from time to time - it's real easy with vagrant-swift-all-in-one
19:41:15 <peluse> was wondering if that criteria is met
19:41:19 <zaitcev> just make it default for Icehouse and see what happens
19:41:33 <swifterdarrell> zaitcev: haha
19:41:38 <notmyname> zaitcev: clayg: need discussion in here (ie additional to gerrit) on container alias middleware?
19:41:39 <clayg> zaitcev: well there's that idea...
19:41:59 <peluse> kidding aside, I think it'd be good to put a stake in the ground
19:42:00 <clayg> sure!  I don't like how it can hide data.
19:42:21 <clayg> ok, well let's put the stake in juno and see if we can some more testing on it
19:42:31 <notmyname> peluse: I don't think we'll do it for icehouse, but probably sometime after that
19:42:46 <zaitcev> I don't see how alias hides, since the original is still available
19:42:53 <notmyname> peluse: eg circa 1.15 or 1.16
19:42:59 <peluse> cool... thanks
19:43:11 <clayg> zaitcev: yeah it's *available* but not *accessible* (until you remove the alias)
19:43:48 <clayg> zaitcev: and by "available" I mean mostly counting up to your byte counts, but when you head the container you see different byte counts... it's just sorta very strange
19:44:00 <zaitcev> clayg: I don't understand that code, then. I thought adding an alias did absolutely nothing to the original container.
19:44:22 <clayg> zaitcev: it makes it so that when you make reqeusts to that container you see something else
19:44:25 <clayg> it's a little liar
19:44:35 <notmyname> clayg: also known as a symlink? ;-)
19:44:36 <acoles-> clayg: zaitcev: its tricky because there's no way to atomically test for empty container and set alias pointer, so there's always a risk of an object getting hidden in the aliased container
19:44:51 <clayg> notmyname: NO!  symlinks can't have *real* data in the them too!
19:45:05 <notmyname> oh yikes. ya, that does sound scary
19:45:05 <clayg> you can have a symlink that has data IN it point you somewhere else so you can't get to the data
19:45:08 <peluse> whats the use case for container aliasing?
19:45:24 <clayg> peluse: something about --os-storage-url is too complicated
19:45:27 <zaitcev> wait, alias used to check if the aliased container had counts and rejected setting alias
19:45:42 <clayg> zaitcev: yeah i suggested that as a possible fix up to my concern
19:45:52 <clayg> zaitcev: but it cirtianly doesn't work that way
19:46:22 <acoles-> zaitcev: um, i commented that the check wasn't guaranteed if you have concurrent object PUT
19:46:22 <clayg> I think anytime you go to pull alias out of sysmeta you should check the container_info and bail if it's non-zero
19:46:26 <peluse> OK, found use cases:  https://github.com/cschwede/swift-containeralias
19:47:14 <clayg> I just think the benifit/complexity ratio is off - most of the benifit could be got at other ways
19:47:22 <clayg> it's a cool hack - but maybe it doesn't belong in core
19:47:53 * clayg can be overruled, and would change his mind if there was a use-case that was unsolveable w/o this particular hack
19:48:03 <notmyname> sounds like good feedback
19:48:34 <notmyname> anything else that needs to be brought up in the meeting today?
19:49:33 <notmyname> ok
19:49:41 <notmyname> thank you, all of you, for your hard work
19:49:55 <notmyname> sign up for the summit if you haven't already
19:50:10 <notmyname> your ATC code is only valid in the early reg period
19:50:24 <notmyname> #endmeeting