19:01:27 <notmyname> #startmeeting swift
19:01:28 <openstack> Meeting started Wed Dec 11 19:01:27 2013 UTC and is due to finish in 60 minutes.  The chair is notmyname. Information about MeetBot at http://wiki.debian.org/MeetBot.
19:01:29 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
19:01:31 <openstack> The meeting name has been set to 'swift'
19:01:56 <notmyname> agenda this week is light. https://wiki.openstack.org/wiki/Meetings/Swift
19:02:19 <notmyname> who's here?
19:02:28 <peluse> hola
19:02:36 <torgomatic> avast, ye salty sea dog
19:02:38 <portante> o/
19:02:40 <acoles> here
19:02:47 <cschwede> here
19:02:54 <lincolnt> here
19:03:33 <notmyname> cschwede: now that I know we're online at the same time, I had to add a .mailmap entry for you recently (you had multiple addresses). please review and offer a patch if I chose the wrong one
19:03:43 <notmyname> great, let's get started
19:03:52 <notmyname> #topic 1.11.0 status update
19:04:14 <notmyname> RC for 1.11.0 has been cut. tons of great stuff in it (See the changelog)
19:04:23 <notmyname> it's currently living on the milestone-proposed branch
19:04:34 <portante> great
19:04:51 <portante> will it be final tomorrow?
19:04:53 <notmyname> if nothing comes up in the next ~20 hours, we'll have our final 1.11.0 release
19:05:15 <notmyname> the only thing that came up so far was the spninx versioning requirement change
19:05:45 <notmyname> since without it, new versions of sphinx (which -infra has installed) break
19:06:08 <notmyname> anything else know about anything else for 1.11?
19:06:37 <portante> not from red hata
19:06:41 <portante> s/a//
19:07:01 <notmyname> acoles: cschwede: anything from our EU contingent? :-)
19:07:14 <portante> acoles: wake up!
19:07:19 <acoles> no
19:07:20 <notmyname> heh
19:07:20 <portante> we know its late. :)
19:07:21 <cschwede> no :)
19:07:42 <acoles> that was no to notmyname :)
19:07:42 <notmyname> ok. if anything sees anything troubling, please let me know ASAP
19:08:02 <notmyname> #topic python-swiftclient
19:08:10 <notmyname> moving on to the next thing...
19:08:32 <notmyname> the client has had some patches languish and needs some lovin
19:08:51 <notmyname> there are a couple of patches I'd like to see land in it https://wiki.openstack.org/wiki/Swift/PriorityReviews
19:09:05 <notmyname> here's the plan I'm working towards:
19:09:35 <notmyname> (1) land those last 2 patches (2) cut the last 1.X rev (3) land the cert checking patch (4) cut the 2.0 rev
19:10:08 <notmyname> as to the other patches outstanding for python-swiftclient, most are stylistic changes or py3k changes (neither of which I consider high priority)
19:10:25 <notmyname> any objections to this plan?
19:10:48 <portante> none from red hat
19:11:23 <cschwede> I think https://review.openstack.org/#/c/59673/  (one of those 2 last patches) needs only another opinion from someone else, should be easy to merge.
19:12:11 <notmyname> ya, I think both are fairly easy patches to review
19:12:56 <notmyname> I'll look for activity on these an start nagging if nothing has happened by the end of the week
19:13:06 <notmyname> anything else on python-swiftclient?
19:13:53 <notmyname> ok. moving on...
19:14:02 <notmyname> #topic get_diskfile
19:14:11 <portante> is gholt here?
19:14:13 <notmyname> this is a change that portante has.
19:14:16 <notmyname> portante: got a link?
19:14:19 <portante> sec
19:14:21 <notmyname> doesn't look like it
19:14:34 <portante> https://review.openstack.org/#/c/60629/
19:14:45 <notmyname> thanks
19:14:46 <portante> I don't look like that link, but there it is
19:15:06 <portante> gholt has placed a -1 on that patch, are there any other concerns with it?
19:15:36 * torgomatic is entirely neutral
19:15:56 <peluse> not sure I fully understand his concern
19:15:58 <torgomatic> as long as storage policy index can make it down to the filesystem in a manner that nobody hates too much, I'm on board
19:16:16 <notmyname> portante: to summarize what I've heard on it, the get_diskfile change is something that woudl be expected to move over time, so this isn't good or bad, per se
19:16:38 <portante> okay, but that is true for lots of the code
19:16:44 <notmyname> portante: ie anyone writing a DiskFile would be expected to know what versions of swift they are compatible with
19:16:50 <notmyname> portante: sure :-)
19:17:01 <zaitcev> Frankly I don't see why an implementation cannot conform to the old style of parameters and then simply ignore device and partition.
19:17:01 <portante> certainly, that is true today
19:17:20 <portante> it can, this is to line things up with twhat is coming with storage policies
19:17:42 <portante> so that new parameters from the environment that are implementation specific land as kwargs only
19:17:59 <peluse> and BTW, I have the patch almost ready that removes DATADIR and replaces with method func that takes index in and returns obj dir
19:18:11 <portante> this is a very tight change, and one which all the other out of the tree consumers have to adjust to anyways today
19:18:18 <portante> peluse: great
19:18:19 <peluse> yup
19:18:30 <peluse> yup to your first comment, not the 2nd one :)
19:19:06 <peluse> do we need to 'policiy'ize' ASYNCDIR as well?
19:19:29 <notmyname> peluse: before we get there, let's finish up with portante's patch
19:19:30 <peluse> asyndir-1, asyncdir-2, etc
19:19:46 <peluse> sorry, jumping ahead
19:20:08 <portante> notmyname: not much more we can discuss
19:20:16 <portante> gholt is not here, so it is kinda mute
19:20:18 <portante> moot
19:20:41 <portante> I am happy to move on to the next topic and just work this as best we can via -swift
19:20:54 <notmyname> portante: I'm on board with a,c,o, **kwargs. seems like it needs to go through normal review and land when it gets 2 +2s
19:21:43 <notmyname> ok, moving on...to "open"
19:21:49 <notmyname> #topic open discussion
19:21:57 <notmyname> peluse: asyncdirs?
19:22:11 <peluse> yes, question for portante mostly I guess, and torgomatic
19:22:31 <peluse> just one for all policies or a dir per policy like we are doing with objects
19:22:43 <lincolnt> What about the topic on the agenda: gatekeeper middleware (https://review.openstack.org/51228) ?
19:22:44 <portante> I don't think we need to do it for asyncdirs, or quarantine for that matter, as the hash will always be different
19:22:54 <peluse> cool
19:23:08 <torgomatic> lincolnt: I snuck that in there at about UTC 16:59:59, so we can just do that in open discussion :)
19:23:09 <peluse> makes sense
19:23:19 <notmyname> lincolnt: whoops. sorry. I didn't reload my agenda page :-)
19:23:27 <notmyname> lincolnt: I'll come back to that
19:23:28 <torgomatic> there's plenty of time
19:23:31 <lincolnt> ok
19:23:37 <zaitcev> notmyname: Do you think we need more reviewers and if yes, are there any good ideas. I basically do nothing while I'm working on mem_backend for a,c and I feel like reviews pile up.
19:23:59 <notmyname> ok, let me organize this
19:24:03 <notmyname> #topic async dirs
19:24:20 <peluse> got my answer already :)
19:24:22 <notmyname> peluse: clayg: torgomatic: peluse: any resolution here? yes or now?
19:24:24 <notmyname> ok
19:24:50 <peluse> answer is no (no need tomake changes there)
19:25:11 <notmyname> peluse: at least for now. they may be needed at a later time
19:25:30 <peluse> I have it almost done so may submit with the patch and reviwers can decide, easy to back out
19:25:34 <notmyname> ok
19:26:11 <notmyname> #topic review backlog
19:26:37 <notmyname> zaitcev: in general, ya, I'm somewhat worried about the review backlog
19:26:57 <notmyname> part of that has to do with the amount of reviews being done
19:27:40 <notmyname> however, we have seen some recent movement as some of the core devs who weren't participating as much in the past did some more reviews (swifterdarrell)
19:28:18 <torgomatic> someone should set up a graph of review-queue depth
19:28:20 <portante> but for a healthy community we also need to be sure folks from different companies are reviewing others folks work
19:28:21 <torgomatic> :)
19:28:46 <notmyname> portante: yes, indeed
19:28:52 <portante> it is not just about review backlog, it is also about getting the community mind share on changes
19:29:09 <peluse> hear that
19:29:15 <zaitcev> I agree with portante in abstract, but it's a luxury for me. I just grab any reviews I can complete, grab next, and it never ends. Can't even think what company backs what.
19:29:40 <notmyname> aside from one or two changes recently, I don't think this is actually a problem we face currently
19:29:55 <notmyname> portante: do you disagree?
19:30:31 <portante> it would be nice to see the data on the commits and how they were approved
19:30:41 <portante> but not that concerned right now
19:30:46 <notmyname> ok
19:31:14 <zaitcev> I think the only case when out-of-band sit-in-same-building authority forced a dubious patch in was Alex's DB cursors thing... And even in that case I doubt anyone would've caught that. BTW, IIRC it was ultimately fixed by a dude from HP.
19:31:24 <portante> we have partners who are not participating in the swift community because they don't feel it is open enough
19:31:26 <notmyname> if anyone has ideas on how to do reviews differently, please let me know. let's be free with trying new things
19:31:29 <portante> we are trying to change that
19:32:15 <peluse> that's funny - not participating because they don't think its open enough is kinda part of the problem
19:32:17 <notmyname> portante: every time I hear that complaint I ask for data and I haven't gotten any yet
19:32:21 <notmyname> peluse: +1
19:32:36 <portante> agreed, but it is what it is
19:32:42 <zaitcev> I do wonder if I could drum up more support or critique for the backends if I were physically in SF
19:33:11 <notmyname> portante: I'd also be happy to talk to anyone about that if it would help
19:33:11 <zaitcev> I used to work in RH office in Sunnyvale, which is somewhat close.
19:33:28 <portante> yes, and I have recommended that
19:33:41 <notmyname> zaitcev: yes, we all need to get on your db backends reviews (myself included)
19:34:01 <zaitcev> Tangentially re. community, remember this dude who posted a Ceph back-end
19:34:12 <notmyname> yes
19:34:13 <peluse> notmyname:  maybe a peridic 'review-fest' online event or something...
19:34:23 <clayg> zaitcev: I'm looking at the ceph one, but I'm not expert - it takes time
19:34:47 <zaitcev> We told him "go to a separate repo". I'm wondering if he understood it right re. making sure he uses stable APIs and like a welcome test case, and not just giving him a cold shoulder.
19:34:49 <clayg> zaitcev: same for db backends - I don't have a strong vision for what it *should* look like - everything you've got seems sane
19:35:06 <zaitcev> https://review.openstack.org/60215 - Babu Shanmugam
19:36:06 <clayg> zaitcev: I think chmouel should be able to keep in on the right page
19:36:23 <cschwede> zaitcev: i think chmouel is in contact with him
19:36:30 <peluse> same company right?
19:36:30 <clayg> cschwede: beat ya!
19:36:34 <clayg> enovance
19:36:55 <zaitcev> okay
19:36:56 <notmyname> ya, I'm happy that it came from a company that has already been very good at working with upstream swift
19:37:34 <notmyname> but it did get jumped on pretty quickly. not incorrectly, but it may have been seen as harsh or sudden. I'm glad to hear chmouel and cschwede can help guide it
19:37:48 <notmyname> ok, let's move on to lincolnt's topic
19:38:00 <notmyname> #topic gatekeeper middleware
19:38:22 <torgomatic> alright, so there's this patch here that adds a new, mandatory middleware: https://review.openstack.org/51228
19:38:24 <notmyname> lincolnt: what do you have?
19:38:51 <lincolnt> We just noticed it, wanted to discuss, might be related to our https://wiki.openstack.org/wiki/MetadataSearch topic
19:38:57 <tomesh> so why this middlware must be the first one?
19:39:04 <acoles> its my patch
19:39:04 <lincolnt> tomesh and I here, form HP
19:39:09 <lincolnt> from
19:39:18 <clayg> i thoguth it had to be the second after catch_errors
19:39:18 <notmyname> oh, sorry, it is torgomatic's topic, not lincolnt
19:39:30 <tomesh> sure, the second one sorry.
19:39:34 <torgomatic> yeah, I should have put my name on it
19:39:38 * torgomatic is lazy sometimes
19:40:11 <torgomatic> anyhow, when the (mandatory) gatekeeper middleware isn't in the pipeline, this patch adds it
19:40:13 <tomesh> my concern is with the metadata search middleware
19:40:28 <torgomatic> tomesh: I don't think this is related to search, sorry
19:40:55 <tomesh> so we offer search on both custom metadata as well as system metadata
19:40:58 <lincolnt> We will be capturing additional system metadata on requests to add to the TBD metadata DB for searching
19:41:15 <torgomatic> and when gatekeeper isn't second in the pipeline, this patch moves it
19:41:32 <torgomatic> that's what I want to get thoughts on: the automatic moving of middlewares
19:41:48 <torgomatic> IMO, if an operator writes their pipeline in a particular order, we should respect that
19:41:57 <tomesh> I agree
19:41:57 <torgomatic> like if someone sticks mempeek at index 0 to look for memory leaks
19:42:29 <torgomatic> but I don't want to send acoles off on a snipe hunt if other folks think that gatekeeper should get relocated
19:42:29 <zaitcev> torgomatic: I still cannot find it, what's the review #
19:42:39 <torgomatic> zaitcev: https://review.openstack.org/51228
19:42:40 <clayg> 51228
19:42:58 <torgomatic> the code in question is in swift/proxy/server.py
19:43:01 <lincolnt> https://review.openstack.org/#/c/51228/
19:43:37 <portante> I believe the proxy server, and really any WSGI servers needs control of the beginning and end of the pipeline
19:43:57 <torgomatic> so basically what I'm after here is: should Swift re-order middleware or not?
19:44:00 <portante> catch_errors is the one that I think needs to be mandatory
19:44:16 <clayg> portante: i like to turn off catch_errors in dev sometimes (so lazy)
19:44:18 <portante> I don't think it should reorder middleware that is definable in the configuration file
19:44:25 <notmyname> torgomatic: in some cases I can see that it should (eg cache after ratelimit)
19:44:34 <portante> clayg: sure, so have a proxy-serves section switch for that
19:44:49 <notmyname> in other cases, no. eg I have my own custom middleware where I want/need it
19:44:54 <acoles> portante: sure, but if gatekeeper is config definable and someone screws up then sysmeta gets leaked
19:45:05 <clayg> I think that dynamic pipelines patch had a good idea in this regard - you can either ask for the pipeline to be auto ordered or you can say explicitly what you want
19:45:24 <lincolnt> So, related to metadata search: How can new middleware like ours grab new system metadata like (say) x-container-sysmeta-target-container-pointer if it strips it off before our middleware later in the pipe sees it?
19:45:29 <notmyname> ya, I want alpha_ori to resurrect that patch
19:45:48 <portante> but I don't think we are talking about arbitrary middleware, we seem to be talking about how to define the environment that middleware lives in in a WSGI server for swift
19:46:17 <portante> so if we are trying to prevent headers, why wouldn't we foorce that to the beginning of the pipeline?
19:46:36 <acoles> lincolnt: gatekeeper just strips from incoming client request (which is why it needs to be at start of pipe)
19:46:38 <torgomatic> portante: because I might want to profile the whole middleware chain, including gatekeeper
19:46:39 <portante> and how would the administrator know their middleware isn't at the beginning>
19:46:47 <torgomatic> and if you force gatekeeper in front of my profiler, I can't do that
19:47:11 <portante> so then we need to do the profiling differently
19:47:23 <portante> but how many other middlewares will we want to force to the front there?
19:47:31 <portante> profiling seems to be internal type stuff
19:47:47 <portante> so are we talking about internal mechanism or general middleware?
19:48:25 <torgomatic> I'm talking about general stuff... if I type out a pipeline with a bunch of stuff in it in a particular order, I want the proxy to just do it
19:48:35 <portante> and it will
19:48:51 <torgomatic> not if we move gatekeeper around
19:49:06 <portante> why think of it as middleware?
19:49:38 <notmyname> two modes: (1) explicit and warn if it's not up front and (2) auto-manage where it moves things
19:49:38 <portante> torgomatic: there is lots of code in WSGI handling that won't get profiled with profiling middleware
19:49:40 <torgomatic> more concretely, if I have "pipeline = thing-one thing-two catch_errors gatekeeper", the proposed patch will turn that into "pipeline = catch_errors gatekeeper thing-one thing-two"
19:49:41 <notmyname> how abotu that ^
19:50:23 <portante> torgomatic: in this proposed scenario catch_errors and gatekeeper are not middleware
19:50:26 <tomesh> right. why does the getkeeper need to be second
19:50:27 <torgomatic> notmyname: Some people, when confronted with a problem, think "I'll make it configurable!" Now they have 2^N problems.
19:50:32 <portante> they would be part of WSGI
19:50:53 <notmyname> torgomatic: sure sure :-)
19:51:12 <portante> tomesh: catch_errors is a generic error handling thing for the pipeline, also adds txid
19:51:25 <tomesh> sure
19:51:34 * clayg search for gatekeeper in pep333
19:51:40 <portante> gatekeeper is about preventing user specified reserved headers from entering in to the WSGI handling
19:51:42 <tomesh> my question is why do we force getkeeper to be the second middleware
19:51:58 <tomesh> I understand
19:52:00 <torgomatic> portante: ...but they're not? WSGI stuff isn't usually subject to me goofing it up, but these modules are
19:52:00 <portante> so that catch-errors can be first
19:52:04 <tomesh> and I support this functionality
19:52:14 <portante> torgomatic: what?
19:52:19 <tomesh> but why does it NEEDs to be second?
19:52:34 <notmyname> make gatekeeper first but let it whitelist earlier middleware?
19:53:12 <torgomatic> portante: eventlet.wsgi or Apache or whatever is outside the scope of things I can fix by patching Swift, but gatekeeper isn't, so I want it to be examinable
19:53:28 <portante> tomesh: so that no other middleware will be affected by user attempts to set reserved headers
19:54:05 <acoles> notmyname: worth considering, but issue concerns catch_errors too
19:54:07 <tomesh> but if we have a middleware that needs to see the reserved headers
19:54:46 <portante> tomesh: the point is that those middlewares need to be assured that if a reserved header is present it cames from other middleware and not set by the user
19:54:56 <acoles> tomesh: the reserved headers should never be set by a clinet
19:54:59 <acoles> client
19:55:00 <portante> torogmatic: why isn't gatekeeper?
19:55:25 <lincolnt> We want our metadata search middleware to be in front of gatekeeper so we catch system metadata that gatekeeper (with this patch) would strip off, e.g. the example x-container-sysmeta-target-container-pointer
19:55:25 <lincolnt> (my name example)
19:55:43 <torgomatic> portante: because it lives in the Swift source tree, and gets packaged with Swift
19:55:44 <portante> lincolnt: why?
19:56:16 <portante> torgomatic: how much of swift do you want examinable?
19:56:18 <portante> just gatekeeper?
19:56:26 <portante> what about ther est of the wsgi code in common?
19:56:41 <portante> why would gatekeeper be any different from that
19:56:45 <lincolnt> Isn't it a use case that the general mecahnism being added, to store any new system metadata, might be metadata that someone could want to search on? Like custom metadata
19:56:47 <notmyname> I don't think we've got a consensus here (or will get one in the next 4 minutes). so let's move the discussion to #openstack-swift and the patch review
19:56:56 <portante> okay
19:57:06 <torgomatic> sure
19:57:14 <notmyname> but now we all know what's at stake, so go review the code :-)
19:57:22 <acoles> ok. btw thanks folks for the reviews and interest
19:57:35 <notmyname> acoles: I think you've got a lot of interest :-)
19:57:50 <portante> and have done a good job with this
19:57:55 <notmyname> #topic open discussion (redux)
19:58:05 <notmyname> anything else to bring up in the last 3 minutes?
19:58:14 <peluse> EC
19:58:15 <notmyname> s/2/3
19:58:21 <peluse> doubt can do it in 3 min thogh :)
19:58:25 <notmyname> heh
19:58:38 <peluse> we can pick it up on regular IRC
19:58:40 <notmyname> ok
19:58:43 <tsg> works
19:58:53 <notmyname> next meeting is scheduled for Dec 25. I propose we skip it
19:59:03 <peluse> how come?
19:59:04 <peluse> :)
19:59:07 <notmyname> any objections?
19:59:11 <torgomatic> none here
19:59:22 <peluse> sounds good
19:59:23 <notmyname> if so, have the meeting without anyone else ;-)
19:59:38 <portante> none here, so we'll need again in the new year, happy new year!
19:59:40 <zaitcev> international audience
19:59:49 <notmyname> ok, off to -swift and gerrit for EC and gatekeeper discussions
19:59:53 <notmyname> thanks for coming
19:59:55 <notmyname> #endmeeting