19:00:36 <notmyname> #startmeeting swift
19:00:37 <openstack> Meeting started Wed Jan 22 19:00:36 2014 UTC and is due to finish in 60 minutes.  The chair is notmyname. Information about MeetBot at http://wiki.debian.org/MeetBot.
19:00:39 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
19:00:41 <openstack> The meeting name has been set to 'swift'
19:00:45 <notmyname> who's here?
19:00:50 <dfg> hello
19:00:55 <peluse> here
19:00:55 <gholt> o/
19:01:04 <torgomatic> o/
19:01:17 <chmouel> hey there
19:01:19 <cschwede> o/
19:01:26 <notmyname> #link https://wiki.openstack.org/wiki/Meetings/Swift
19:01:41 <notmyname> not too many topics on the agenda today
19:01:48 <notmyname> first, though...
19:01:54 <notmyname> #topic swift 1.12 release
19:02:03 <notmyname> today the RC was cut for 1.12.0
19:02:04 <notmyname> #link https://wiki.openstack.org/wiki/Meetings/Swift
19:02:25 <gholt> Oh, I should add the SOS thing to the Agenda I guess, sorry.
19:02:26 <notmyname> so it's in milestone-proposed now, and assuming nothing major comes up, we'll do a final release on that early next week
19:02:43 <chmouel> i think tristant added the swiftclient ssl thing but can't see it here (or was it discussed last meeting>
19:02:46 <chmouel> ?
19:02:47 <portante> notmyname: will that require a gate job to cut it?
19:03:09 <notmyname> portante: no. just a tag push
19:03:15 <portante> oh good
19:03:44 <notmyname> anything else on 1.12? (beyond SOS from gholt)?
19:03:59 <portante> added the current gate job issue to the list in case folks care
19:04:07 <chmouel> it was cleaned up before we discussed it https://wiki.openstack.org/w/index.php?title=Meetings/Swift&oldid=39969
19:04:16 <notmyname> chmouel: ah, cool
19:04:32 <notmyname> #topic gate status
19:04:52 <notmyname> so I think we can all easily agree that the gate has been broken for quite a while
19:05:02 <notmyname> we discussed some options yesterday in IRC
19:05:18 <notmyname> torgomatic: did you want to lead this part for any follow-up today?
19:06:13 <notmyname> ...or not? ;-)
19:06:13 <torgomatic> just thinking about ways to have a working, fast gate that's upstream of the openstack gate
19:06:24 <torgomatic> if that's even a thing folks are interested in
19:06:31 <clayg> ohai
19:06:45 * portante is interested
19:06:49 * chmouel is listening
19:06:56 <notmyname> the foundation is meeting this week, and I know it's a topic there. I expect something to come from that, but I don't know what
19:07:01 <portante> torgomatic: what are you thinking about, 2nd gerrit server?
19:07:31 <torgomatic> portante: pretty much
19:07:53 <portante> from that, trickle into openstack?
19:08:01 <torgomatic> one can configure Gerrit to cherry-pick instead of merge, which would result in a linear history
19:08:02 <portante> automagically
19:08:20 <torgomatic> and that linear history is ideal for feeding upstream into openstack gerrit as a series of dependent changes
19:08:31 <chmouel> so why is openstack gerrit is actually needed?
19:09:21 <torgomatic> chmouel: that is an excellent question :)
19:09:26 <portante> is the trickle mech automated or does it require babysitting?
19:09:42 <torgomatic> portante: it'd have to be automated; that's computer work, not human work
19:09:47 <notmyname> and what happens if someone contributes to the openstack gerrit system? we'd have 2 systems to review?
19:10:04 <portante> auto -2 on the openstack gerrit one?
19:10:05 <chmouel> torgomatic:  so the question is more about simply splitting from openstack CI, right?
19:10:14 <chmouel> question/discussion
19:10:17 <portante> chmouel: does it?
19:10:28 <torgomatic> notmyname: that's the sticky problem; you don't want to have two code-review systems active simultaneously
19:10:33 <portante> the patches are still integrated continuously
19:11:08 <torgomatic> chmouel: pretty much that's what I want to get away from, yeah... 48+ hours for an attempted merge, with a success probability like a coin flip... bleh
19:11:47 <chmouel> torgomatic: sure we all know that the CI had issues lately
19:12:22 <chmouel> but i'm not sure if swift gerrit to openstack gerrit to github make sense (if i understand correctly what are you trying to do)
19:12:57 <torgomatic> from a technical standpoint I'm pretty sure it's possible
19:13:27 <torgomatic> it's the non-technical stuff that I'm unsure of
19:13:36 <chmouel> sure :)
19:13:36 <notmyname> that's the hard part, right?
19:14:01 <chmouel> so basically that would make swift out of openstack if we don't use openstack ci, right?
19:14:05 <notmyname> I don't think its feasible to fork from openstack-ci
19:14:22 <portante> chmouel: my guess is that once folks see that it works, we won't need to funnel to openstack gerrit, but to a queue of patches to be merged, pulled from a series of repos that are ready
19:14:34 <notmyname> right. what chmouel said. which is something I don't think we want (certainly not something I want)
19:14:45 <chmouel> neither am i
19:14:56 <portante> notmyname: I don't follow why that is
19:14:57 <torgomatic> chmouel: well, everything would still pass openstack CI
19:15:09 <portante> this is all about tooling
19:15:22 <portante> don't we each have patches in our own queues?
19:15:23 <chmouel> torgomatic: but upstream really is the swift gerrit
19:15:31 <torgomatic> chmouel: right
19:15:40 <portante> what torgomatic seems to be saying is that we just merge them in order ahead o tfime
19:15:48 <notmyname> inverting that order "who is upstream" is the problem
19:15:59 <dfg> is work being done on improving the gerrit gate jobs?
19:16:09 <portante> dfg: yes
19:16:16 <notmyname> dfg: sort of
19:16:22 <chmouel> dfg: plenty and hopefuly i like to think it will get better
19:16:23 <portante> will they be successful enough, jury is out
19:16:25 <chmouel> take your pick ;)
19:16:52 <notmyname> and I think that the current situation is by its nature temporary (either it gets fixed or the project dies)
19:16:53 * torgomatic just misses the days of a 10-minute test run with a high SNR
19:17:01 <notmyname> torgomatic: ++
19:17:27 <torgomatic> it's to the point where a patch has to fail like 5 times before I even bother looking at the logs now
19:17:37 <chmouel> yeah right i'm sure swift is not the only project being annoyed by the gate status and if everyone start to use their own upstream i'm not sure that would make sense
19:17:38 <torgomatic> (as happened with my JSON one)
19:17:56 <chmouel> torgomatic: what about requesting to openstack ci not to have the integrated tests?
19:18:15 <notmyname> didn't we propose something similar to that back in december at the openstack team meeting?
19:18:19 <torgomatic> chmouel: I brought that up at a meeting once (not opting out of *all* of them, but the mysql-vs-postgres ones, and the lots-of-neutron-ops one)
19:18:34 <notmyname> torgomatic: how'd that work for you? ;-)
19:18:35 <torgomatic> folks acted like I took a dump on a kitten
19:18:38 <notmyname> lol
19:19:09 <chmouel> i'm sure that a better tradeoff than a split to another ci (which IMO would basically seen as swift breaking up from openstack)
19:19:10 <peluse> nice
19:19:26 <dfg> imo we should just try to get the current tests fixed. the idea of the intergrated tests is a good one. but they need to work better. so we should get our PTL to put pressure on the people running that thing
19:19:48 <portante> dfg: the PTLs seem to be on board, other developers are not
19:19:57 <notmyname> dfg: always :-)
19:19:58 <portante> all neutron jobs were pulled from the gate
19:20:04 <portante> because they were all failing
19:20:04 <notmyname> ah, really?
19:20:06 <clayg> chmouel: I think we need to push harder for removing jobs that don't make sense against swift
19:20:11 <portante> but nobody is jumping in to work on them
19:20:14 * chmouel had actually fixed some of the tests failures
19:20:26 <portante> clayg: I don't think that will help the problem
19:20:27 <notmyname> that would explain the recent rise in the pass rate chance
19:20:28 <torgomatic> portante: I still see blahblah-neutron-large-ops in the current gate, fwiw
19:20:31 <portante> it is more systemic
19:20:47 <portante> torgomatic: I meant neutron changes
19:20:51 <portante> the tests are still there
19:20:58 <torgomatic> portante: ah, okay then
19:21:10 <notmyname> wait, so neutron code is untested going into neutron then?
19:21:13 <portante> but they could not make a change to neutron to get any fixes to the existing problems without creating more problems, from what I can tell
19:21:15 <chmouel> let just send a pull request to remove those jobs and start a discussion on this?
19:21:37 <portante> but doing that won't help the gate for swift
19:21:41 <clayg> chmouel: portante says there's no amout of jobs that could be removed to help the issues with failing jobs
19:21:42 <notmyname> I think pulling unnecessary tests from swift will help
19:21:48 <portante> because there are still jobs ahead of ours that will fail
19:21:51 <notmyname> portante: it's not a complete solution, but it is better
19:22:08 <clayg> portante: this is the merge queue stack thing?
19:22:12 <chmouel> well yeah for merging, i was thinking for at least the rechecks
19:22:14 <torgomatic> at least when a Swift job hits head-of-queue, it would have a better chance of passing
19:22:15 <portante> yes
19:22:18 <portante> clayg: yes
19:22:27 <portante> torgomatic: true
19:22:29 <chmouel> torgomatic: +1
19:22:50 <portante> start at 109 and then wait
19:22:52 <notmyname> chmouel: I'd like to be the one to propose those changes (to remove some tests). can you point me to the right direction after the meeting?
19:22:58 <portante> that is still 40+ hrs
19:23:32 <chmouel> notmyname: yep perfect, (i will have to go just right in 10mn but will catch uop with you tomo about it)
19:23:48 <clayg> and ci would really like to see any change merged into any openstack project pass integrated test
19:23:51 <notmyname> chmouel: ok, sounds good. actually jsut send an email then. i'll be traveling tomorrow
19:24:19 <torgomatic> clayg: yeah, anything we do would have to have swift changes going through integrated tests
19:24:35 <chmouel> I am not sure what time is the swift meeting over but it would be nice if we can take a decision about  swiftclient and the ssl thing
19:24:47 <chmouel> since that get swiftclient removed  from debian
19:24:49 <notmyname> passing integrated tests is a good thing. as a first step we can push more on removing the unnecessary tests. later I'd want to see eg a daily integrated test
19:25:44 <notmyname> but I don't want to jump to "let's use alt-gerrit" yet (although it's an interesting idea to explore)
19:25:57 <notmyname> so for chmouel's time we should move to that topic?
19:26:23 <torgomatic> sure, we've probably talked enough about this for now
19:26:27 <notmyname> kk
19:26:31 <chmouel> tks
19:26:33 <torgomatic> I'm sure it'll come up again as long as the gate stays horrid
19:26:34 <notmyname> #topic swiftclient + ssl
19:26:36 <notmyname> chmouel: go
19:26:47 <chmouel> well you probably all heard about this already
19:26:52 <chmouel> swiftclient not checking for certificates
19:26:53 <notmyname> assume not
19:27:03 <chmouel> so swiftclient is not checking for certificates .... :)
19:27:06 <chmouel> by default
19:27:16 <chmouel> there is a review sent last summer (!!!) to fix that
19:27:31 <chmouel> which went back and forth with the assumtion that we change the default behavior of swiftclient
19:27:43 <chmouel> until it was actually tested (thanks swifterdarrell) and it wasn't working
19:27:57 <chmouel> tristant was working with thomas on this but AFAIK it's not working still yet
19:28:05 <clayg> neat!
19:28:08 <chmouel> and having to do our own imp of SSL is not something I really like about
19:28:29 <chmouel> I would like to go for requests and let the lib handle
19:28:32 <chmouel> all those things
19:28:35 <torgomatic> the certificate CN-vs-hostname checking in there is some seriously scary code
19:28:35 <notmyname> so what's the plan going forward?
19:28:46 <torgomatic> I thought requests couldn't turn off SSL compression
19:28:58 <chmouel> yep i was coming to that
19:29:08 <chmouel> so the two feature for request that we need is :
19:29:10 <chmouel> SSL compression
19:29:17 <chmouel> 100-continue thing
19:29:29 <chmouel> i think the SSL compression is something dear to our HP friends and the glance users
19:29:36 <chmouel> and there is a hack for that
19:29:40 <chmouel> i could not figure out how to use it properly
19:29:44 <chmouel> but may try again
19:30:02 <chmouel> (it's only supported in requests/urllib3 properly from python3.2+)
19:30:17 <chmouel> as for the 100-continue
19:30:26 <notmyname> and expect 100-continue?
19:30:36 <chmouel> well my question was, is that something really needed?
19:30:52 <notmyname> it's terribly useful, especially when doing larger objects
19:31:10 <portante> fail PUT on disk allocation failure (fallocate)
19:31:14 <notmyname> it's not a requirement of the API, but very useful
19:31:28 <clayg> portante: or too big content-length even
19:31:41 <portante> clayg: yes, thanks
19:31:55 <chmouel> i guess we can try to see urllib3/requests to have that there but if we can't are we keeping the status quo like that or just try to go with requests still?
19:32:06 <notmyname> ya, it just means that the client will send the body before checking for 100 continue, even if the request would fail
19:33:33 * portante bbiab
19:33:37 <chmouel> i personally think if there was a vote to go with requests and see if we can the expect-100-cont in requests
19:34:38 <notmyname> so a rewrite to use requests, and the vote to try to get expect-10-continue support in requests either before or after the rewrite?
19:34:43 <torgomatic> IMO, having an upstream library responsible for all the certificate validation stuff is the way to go, so +1 from me on requests or whatever
19:35:13 <chmouel> notmyname: yep that's it
19:36:09 <notmyname> #startvote should we rewrite swiftclient using requests? vote no, yesbefore, yesafter
19:36:10 <openstack> Begin voting on: should we rewrite swiftclient using requests? Valid vote options are vote, no, yesbefore, yesafter.
19:36:11 <openstack> Vote using '#vote OPTION'. Only your last vote counts.
19:36:22 <notmyname> yesbefore == do usptream work first
19:36:25 <chmouel> #vote yesafter
19:36:37 <torgomatic> #vote yesbefore
19:36:37 <notmyname> yesafter == do upstream work after a swiftclient rewrite
19:36:41 <clayg> is chmouel offering to do the requests rewrite?
19:36:55 <clayg> because then he can do it however he wants...
19:36:59 <notmyname> :-)
19:37:04 <cschwede> #vote yesafter
19:37:04 <clayg> IMHO
19:37:09 <chmouel> clayg: heh :) yeah i would not mind or see if tristan would not mind either ;)
19:37:12 <notmyname> clayg: ya, but we'll still have to review it
19:37:24 <chmouel> which is proabbly the hardest part ^
19:37:33 <clayg> lol
19:37:42 <notmyname> #vote yesafter
19:38:11 * chmouel is at least going to start log a bug on urllib3 to support the feature
19:38:13 <notmyname> anyone else want to register their opinion?
19:38:21 <notmyname> of course the real vote is the code review :-)
19:38:32 <notmyname> chmouel: that sounds good
19:38:33 <portante> #vote yesbefore
19:39:11 <notmyname> chmouel: and to be clear, the reason for using requests is to get ssl cert checking. and the reason for that, beyond "it's good security" is that debian did something?
19:39:13 <clayg> notmyname: my perferred option isn't really available for voting
19:39:44 <chmouel> notmyname: since swiftlclient at this time is inherently not secure they dropped it
19:40:01 <chmouel> zigo: ^^
19:40:03 <clayg> chmouel: hmph - blame httplib
19:40:15 <chmouel> clayg: i'll blame python2.x and python3 devs :(
19:40:27 <notmyname> ya, did thye drop python too? ;-)
19:41:07 <notmyname> last call before ending the vote (this is just to register your current thoughts, not on "this is the way it must be done")
19:41:36 <notmyname> #endvote
19:41:37 <openstack> Voted on "should we rewrite swiftclient using requests?" Results are
19:41:38 <openstack> yesafter (3): chmouel, cschwede, notmyname
19:41:39 <openstack> yesbefore (2): portante, torgomatic
19:41:50 <notmyname> ok, good split. no real consensus
19:42:00 <gholt> Heheh
19:42:09 <peluse> perfect
19:42:11 <chmouel> :)
19:42:22 <peluse> my vote is for sale BTW
19:42:28 <notmyname> of course only 5 people voted....
19:42:30 <clayg> if you can patch upstream you can monkey patch in the review until the upstream change is resolved
19:42:30 * koolhead17 looks around
19:42:35 <notmyname> I want to move on to the gatekeeper/sos thing while we have time
19:42:47 <clayg> gholt: does swiftly do cert validation?
19:42:58 <gholt> No it sure doesn't :(
19:43:00 <notmyname> #topic gatekeeper/sos issue (1.12 final blocker?)
19:43:10 <chmouel> sorry i have to go
19:43:13 <notmyname> gholt: debian will drop it! oh noes
19:43:18 <notmyname> chmouel: thanks for the summary
19:43:26 <gholt> I'll steal chmouel's work when he's done. ;)
19:43:33 <notmyname> lol
19:43:34 <chmouel> gholt: hah :)
19:43:42 <notmyname> I just heard about this gatekeeper issue this morning. gholt/dfg you got details?
19:44:10 <gholt> It's a small fix luckily. But sos wants to use relative location headers and swob doesn't allow that.
19:44:35 <notmyname> gatekeeper breaks that or we need a patch to swob?
19:44:35 <dfg> well i need to patch sos and swift but older versions of sos'll have a bug with this new release
19:44:36 <gholt> We never noticed before because swob didn't get in front of (or behind?) sos, so sos had the last word.
19:44:45 <notmyname> ah, ok
19:44:55 <gholt> gatekeeper breaks it just because it happens to use swob
19:45:09 <notmyname> rather than the wsgi environ directly?
19:45:22 <dfg> no- that gets overridden
19:45:22 <chmouel> #link https://github.com/kennethreitz/requests/issues/713
19:45:29 <chmouel> about the 100 continue in requests
19:45:32 <notmyname> thanks
19:45:44 <dfg> from what i remember eventlet does a .location or something? it was kinda weird
19:45:59 * clayg goes to clone sos
19:46:35 <dfg> ya- i think just noting this in SOS will be sufficient. i don't think there will be a lot of affect users because as far as I know there aren't too many of us
19:46:57 <notmyname> dfg: and if gatekeeper is explicitly in the pipeline to the right of sos, it won't get moved. so that's a mitigation strategy until a patch lands, right?
19:47:01 <gholt> Essentially an upgrade to swift 1.12 will require an upgrade sos, right?
19:47:35 <dfg> gholt: ya
19:47:44 <dfg> and noting that in SOS should be enough imo
19:47:56 <gholt> And in the swift 1.12 release notes maybe?
19:48:01 <dfg> and assuming that the swift patch gets backported to 1.12
19:48:02 <portante> so: catch_errors sos gatekeeper
19:48:12 <portante> explicitly in config file is the answer?
19:48:25 <dfg> portante: no- i think sos has to be after auth
19:48:48 <notmyname> dfg: if the swift patch gets submitted soon, then we can review and include it in 1.12
19:48:53 <dfg> i'll have it today
19:49:03 <notmyname> ok
19:49:06 <dfg> i'd be done if it wasn't for this stupid meeting :p
19:49:14 <gholt> Hah
19:49:28 <glange> wow
19:49:48 <dfg> so then it'll just be 2 weeks for reviews and 3-5 days for gate tests and we'll be golden :)
19:49:55 <clayg> I still don't quite follow the problem - maybe patches will make it clearer
19:49:58 <gholt> That's our dfg, damned f---ing grumpy.
19:50:03 <dfg> haha
19:50:09 <peluse> dfg:  how's you get the fast track?
19:50:12 <clayg> hey on the subject of gatekeeper - i think i made a horrible mistake
19:50:24 <dfg> clayg: its much less of a deal than it has become
19:50:34 <clayg> I approved the change w/o docs because I was planning on writing docs - which I did - https://review.openstack.org/#/c/65817/
19:50:40 <clayg> but then it turns out I hate writing docs
19:50:51 <portante> clayg: I'd be happy to help
19:50:54 <dfg> and your horrible writing skills...
19:50:56 <dfg> :)
19:51:07 <portante> not that I am any better
19:51:08 <clayg> dfg: yes THAT even more!
19:51:21 <clayg> portante: yeah so - can you just please change of my id?
19:51:40 <portante> sure
19:51:57 <portante> change your identity, I have friends that can help you do that
19:52:02 <clayg> I personally don't care EVAR if any core wants to post over my change if it makes it better/them like it more
19:52:02 <notmyname> I'd like to see the docs soon, but since those are "released" as soon as they land (ie they're published to swift.openstack.org), it doesn't need to block 1.12
19:52:31 <clayg> but particually with docs - I wonder if anyone really cares to go back and fix and its/it's instead of crowdsourcing it?
19:52:52 <portante> folks pulling 1.12 looking for the docs won't find them, no?
19:52:55 <notmyname> clayg: somebody will want ATC status and do the change
19:53:06 <clayg> notmyname: ok, that was option 2) - I eff'd up merging before there were docs in the tree
19:53:43 <clayg> i was going to ask alister to do it - but then I realized there wasn't a great place to really talk about it in detail/prose - so it seemed like a larger documentation effort
19:53:59 <clayg> including finding a bunch of other middleware that wasn't autodoc'd etc
19:54:11 <notmyname> so we've got to get in dfg's patch for 1.12. and if the docs stuff gets approved we can include it too.
19:55:12 <notmyname> we're almost out of time, so I want to make sure we've covered the basics in here
19:56:16 <notmyname> gholt: dfg: anything else important on this topic?
19:56:46 <gholt> I'm good with it. Mainly wanted to get the knowledge of it out there.
19:56:55 <notmyname> thanks
19:56:59 <dfg> ya- i'm fine
19:57:06 <notmyname> #topic other
19:57:12 <notmyname> anything else in 2.5 minutes?
19:57:40 <gholt> General note on ssync: Testing has been going okay, with one bug/needed-optimization I'll be submitting soon.
19:57:55 <notmyname> great.
19:58:05 <peluse> also, fyi, gholt:  ssync policy patch is up for review if you'd like to a gander
19:58:10 <clayg> gholt: asides from the proxy/lb stuff for container-sync - any more going on there/coming down the pipe?
19:58:22 <notmyname> progress is going well on storage policies as well. I've nearly got a blog post done with a "progress update"
19:58:45 <peluse> coo, torgomatic, quick Q for you on the other channel
19:58:48 <clayg> gholt: for container-sync i mean - i'm sure you've got pleanty of other tricsk up your sleave - but I remember you saying that first container-sync patch was the start of series
19:59:02 <gholt> Er, uh, I don't think we have anything crazy going on other than container-sync and ssync/index-db/etc paths.
19:59:33 <gholt> I think container-sync will be "done" with the http proxies set patch
19:59:43 <clayg> gholt: nice!
20:00:08 <notmyname> cool
20:00:14 <notmyname> ok, time's up
20:00:17 <notmyname> #endmeeting