21:00:23 <notmyname> #startmeeting swift
21:00:23 <openstack> Meeting started Wed Feb  3 21:00:23 2016 UTC and is due to finish in 60 minutes.  The chair is notmyname. Information about MeetBot at http://wiki.debian.org/MeetBot.
21:00:24 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
21:00:27 <openstack> The meeting name has been set to 'swift'
21:00:50 <notmyname> good morning everyone.
21:00:56 <notmyname> who'es here for the swift meeting?
21:00:59 <mattoliverau> o/
21:01:01 <jrichli> yo
21:01:01 <gmmaha> o/
21:01:02 <torgomatic> .
21:01:02 <sgundur> hi
21:01:03 <kota_> hi
21:01:04 <minwoob> o/
21:01:04 <cutforth> o/
21:01:04 <m_kazuhiro> o/
21:01:05 <ho_away> o/
21:01:06 <wbhuber> o/
21:01:07 <takashi> hi
21:01:09 <pdardeau> o/
21:01:09 <cschwede> o/
21:01:12 <timburke> o/
21:01:19 <hseipp> Hi
21:01:20 <akle> akle
21:01:29 <notmyname> welcome
21:01:47 <notmyname> I'm currently visiting mattoliverau in australia
21:01:58 <mattoliverau> \o/
21:02:03 <notmyname> gives a very real perspective of all the different time zones we work in :-)
21:02:09 <acoles> here
21:02:33 <siva_krishnan> o/
21:02:42 <clayg> whoa
21:02:52 <notmyname> so that being said, this week while I'm at a conference, I haven't kept up with IRC too well.
21:03:00 <mattoliverau> +1
21:03:04 <notmyname> so let's get started on the agenda :-)
21:03:08 <notmyname> #link https://wiki.openstack.org/wiki/Meetings/Swift
21:03:14 <notmyname> #topic hackathon
21:03:24 <notmyname> in a few weeks we've got the hackathon in bristol
21:03:29 <notmyname> I'm looking forward to seeing everyone
21:03:36 <notmyname> jrichli: you put together an etherpad
21:03:44 <notmyname> #link https://etherpad.openstack.org/p/swift-hackathon-feb-2016
21:04:12 <notmyname> jrichli: thanks. please share about it
21:04:23 <jrichli> yes - thought we could add topics and ideas similar to the last hackathon
21:04:56 <jrichli> that's really about it :-)
21:05:12 <notmyname> heh, ok :-)
21:05:19 <notmyname> I think it's great
21:05:20 <mattoliverau> can i add topics :P
21:05:33 <jrichli> mattoliverau: please do!
21:05:33 <notmyname> it's something we can use to seed our conversation topics in bristol
21:05:44 <notmyname> yes! everyone add topics you want to bring to the hackathon
21:06:28 <acoles> 'topic' is a candy bar in UK, so bring lots :)
21:06:40 <clayg> nom nom
21:06:42 <notmyname> interesting
21:07:35 <notmyname> #topic auditor watchers^Hbug
21:07:41 <notmyname> #link https://bugs.launchpad.net/swift/+bug/1183656
21:07:42 <openstack> Launchpad bug 1183656 in OpenStack Object Storage (swift) "object auditors don't finish" [High,Confirmed]
21:07:42 <clayg> lol
21:08:04 <notmyname> clayg reminded us of this last week, and it seems that there is some comments on it
21:08:07 * clayg will endeavor to keep his trap shut
21:08:24 <notmyname> I don't remember...was someone starting to investigate a solution?
21:08:37 <notmyname> was it onovy?
21:09:20 <notmyname> hmm...doesn't seem to be around right now
21:09:27 <notmyname> anyone know anything more on this?
21:09:49 <acoles> I asked Gerry to share some numbers from our public cloud just as a reference point, so he added a comment
21:10:00 <notmyname> cool, thanks
21:10:05 <clayg> I hadn't seen Gerry's notes - that's amazing - you guys are the very best
21:10:46 <acoles> shame those particular auditors won't be running too many more times :(
21:10:59 <notmyname> acoles: is that where the bugs at inside of hpe? "yup it's slow, here's numers" or is there any work on a patch?
21:11:06 <notmyname> acoles: isn't it already all turned off?
21:11:41 <clayg> notmyname:  my reading of the comment was just "here's some numbers" - not "we're working on it" or even "we care about this"
21:11:46 <acoles> notmyname: no work on a patch sorry, I just remember last week there was some discussion of 'how much of a real problem is this in a largish cluster'
21:11:51 <notmyname> right
21:12:02 <torgomatic> I mean, it told us what we needed to know. Is saving just the device enough? No, because audits take a couple weeks on real loads, so if we save the partition then we lose many fewer hours' progress on restart
21:12:04 <notmyname> clayg: I'm very optimistic ;-)
21:12:17 <torgomatic> so now we know what level to save at; no pontificating needed.
21:12:28 <torgomatic> :)
21:12:38 <clayg> all over the but the typing
21:12:53 <notmyname> no now that pontification is done, we get to the real question...
21:13:07 <notmyname> who's doing the "think real hard. type in the answer" part?
21:13:32 <clayg> onovy and peterlisak sorta said they might
21:13:52 <clayg> but since the status of the bug is currently unassigned that's probably the most honest representation of reality
21:13:55 <notmyname> clayg: thanks for remembering that. :-)
21:14:14 <notmyname> yeah
21:14:40 <notmyname> so when they're back online, let's bring it up and see what's going on. if I don't, someone else can
21:14:49 <clayg> k
21:14:55 <notmyname> ok, next candy bar
21:14:55 <acoles> i'll try to think about it. finding typing bandwidth might be harder.
21:15:13 <notmyname> #topic patch 273073
21:15:13 <patchbot> notmyname: https://review.openstack.org/#/c/273073/ - swift - Insert versioned_writes in correct pipeline position
21:15:24 <notmyname> acoles: this one is your topic
21:15:40 * notmyname will now only think in terms of meeting candy bars
21:16:33 <clayg> does this patch need discussion or reviews?
21:16:49 <notmyname> acoles: ?
21:16:52 <acoles> if you dont specify it in config then versioned writes can end up in the wrong place
21:17:00 <acoles> ok, just wanted to flag it up because its hopefully an easy review and also not sure who sets it explicitly or who is relying on the code to place it in the pipeline
21:17:19 * clayg pulls our our pipeline solver
21:17:33 <acoles> and ti would be nice to get it fixed because other bugs vary according to where that middleware is in the pipeline :)
21:17:41 <acoles> s/ti/it/
21:17:43 * notmyname really wants to get the pipeline solver back as a thing for swift
21:18:30 <notmyname> ok. small review, important bug fix (or footgun removal). go review it
21:18:47 <acoles> so I have a dependent patch to fix an SLO bug that can't land until we know that versioned writes will be in right place in a test cluster
21:19:12 <acoles> notmyname: yeah, but also a headsup to check what you ship wrt config
21:19:18 <notmyname> yes
21:19:30 <clayg> so it looks like we're still letting it flop in auto - although i'm currently working on a new swift release so that may change
21:19:33 <acoles> clayg: just review i hope
21:19:47 <notmyname> #topic swift and high-latency media
21:19:53 <notmyname> cschwede: this is your topic. what's up?
21:19:59 <cschwede> there were some discussions at the last summit regarding tapes (or high latency media in general). ibm and bdt are working on this, and they just released a middleware prototype
21:20:01 <notmyname> #link http://lists.openstack.org/pipermail/openstack-dev/2016-February/085576.html
21:20:07 <cschwede> so i simply wanted to give a heads up on this topic and hopefully raise some interest
21:20:17 <cschwede> more details are in http://lists.openstack.org/pipermail/openstack-dev/2016-February/085576.html, and hseipp and akle are here today in case you have any questions
21:20:25 <clayg> cschwede: i saw a notice on the ML - so sweet new stuff is out, yeah!?
21:20:33 <notmyname> cool. I love to see people building stuff around swift :-)
21:20:43 <notmyname> akle: hseipp: thanks for working on this :-)
21:20:50 <clayg> ... i haven't really looked at it - does it add some new api goodness to do like "is it ready, is it ready, is it ready, ..."
21:20:51 <hseipp> cschwede: yes and thanks for getting us up here
21:21:29 <akle> you are welcome. would be great if we could talk about
21:21:33 <cschwede> clayg: well, kinda. the middleware adds some new verbs and triggers backend works
21:22:00 <notmyname> interesting
21:22:07 <cschwede> notmyname: yep, might be something completely new for swift. fresh ideas!
21:22:14 <notmyname> that would be good stuff to look at on a long plane ride home
21:22:25 <clayg> new *VERBS*
21:22:34 <clayg> is one of them GREP (cc redbo)
21:22:41 <notmyname> clayg: next up they'll have a grep one... yeha :-)
21:22:59 <cschwede> no grep. well, verbs is the wrong term in this case
21:23:12 <clayg> hseipp: akle: would either of you possibly be at the hack-a-thon in a couple of weeks?
21:23:21 <clayg> notmyname: do you think we could a call in something?
21:23:29 <cschwede> but doing a post and sending a MIGRATE, RECALL along with it
21:23:37 <notmyname> akle: hseipp: is there anything specific you need from us in the community right now?
21:23:41 <cschwede> or a GET STATUS
21:24:00 <akle> unfortunately can't attend this time
21:24:04 <hseipp> clayg: unfortunately I can't make it to this hackathon
21:24:15 <clayg> GET STATUS!?
21:24:19 <clayg> hrmmm...
21:24:27 <ho_away> i see using query parameter
21:24:29 <cschwede> clayg: send a GET, „get“ the STATUS of the RECALL
21:24:49 <cschwede> ho_away: yep. verb was the wrong term. it’s using POST & GET for now
21:24:55 <clayg> a'ight
21:26:07 <notmyname> akle: hseipp: is there anything specific you need from us in the community right now?
21:26:10 <cschwede> so i think it would be nice to have a look at this, talk about this during the hackathon (maybe doing a conf call) and see if we can get to a common implementation that is usable with different backends/vendors
21:26:18 <hseipp> notmyname: at this stage we primarily see for comments and feedback - there is also a Wiki page linked to leave comments
21:26:27 <hseipp> s/see/seek/
21:26:29 <clayg> I had sort of imagined it might be more RESTful, maybe just adding a COPY sort of notion to move it into a storage policy that could do listings (maybe HEAD) but you had to "COPY" it somewhere to do a restore, with some sort of trigger to test if it was ready - I guess I need to dig in to understand some of the implementation considerations/trade-offs and history/alternatives
21:26:37 <notmyname> hseipp: ok, great
21:27:01 <ho_away> do we need to have HLM stub for testing?
21:27:04 <clayg> cschwede: that's a great goal!  you're the champ!
21:27:12 <notmyname> next up (testing!)
21:27:23 <notmyname> #topic in-process testing with fast-post
21:27:32 <hseipp> ho_away: this is one of the next things on our to-do list, for now we tried to put as much into the readme as possible
21:27:34 <notmyname> acoles: what's going on here?
21:27:53 <acoles> I'd like to have a gate job that runs func tests against a fast-post configured service
21:27:55 <clayg> ho_away: I think we all get our own LHC for testing
21:28:00 <ho_away> hseipp: thanks!
21:28:16 <acoles> currently we have 3 func tests runs in the gate, all use post-as-copy
21:28:31 <cschwede> clayg: we should do the next hackathon there and ask for some resources… that would be fun!
21:28:31 <clayg> acoles: *see* everyone uses post-as-copy!
21:28:33 <notmyname> acoles: that sounds like a great idea
21:28:43 <acoles> so this patch adds a tox env to do in process testing with fast-post, patch 274086
21:28:43 <patchbot> acoles: https://review.openstack.org/#/c/274086/ - swift - Enable in-process func tests to optionally use fas...
21:29:01 <clayg> didn't someone +A that yet?
21:29:04 <notmyname> optionally?
21:29:20 <clayg> it just adds an env flag to make it so you can run the in-process functests with a different config option - and we already have some of those
21:29:24 <acoles> question is do I swap gate-tox-func to run that new env, or add a new fourth func test run?
21:29:29 <clayg> ... only difference with this one is that it's useful
21:29:35 <clayg> acoles: ah
21:29:41 <clayg> acoles: yeah that's a discussion
21:29:42 <acoles> clayg: think its waiting for a +A
21:30:03 <notmyname> acoles: IMO replace the existing, since I don't think the current 2 different functests really do anything different anyway
21:30:12 <acoles> and also, headsup that this means new patches will have to work with fast-post enabled!
21:30:17 <clayg> acoles: don't care new job or change the other in-proc job - i don't really care about the in-proc job - i think it'd be fine to make it test fast-post
21:30:35 <clayg> acoles: oh oh oh - that'll be wonderful!
21:30:47 <notmyname> can we get another core to commit to review this asap?
21:30:48 <clayg> acoles: can the name of the job somehow reflect that it's using fast-post?
21:30:55 <clayg> i have no idea how to openstack-ci
21:30:59 <acoles> well, apart from patches to EC, but thats another story, and maybe another gate job :)
21:31:36 <acoles> clayg: good idea, I will have to look at -infra again, maybe the job name simply takes the tox env name, so "yes"
21:31:48 <acoles> -infra is not my strength
21:31:53 <clayg> acoles: probably be nice with in-proc (since we have full control over the env) if we tested as many storage policies as we think we can get away with configuring - do functests still only run against a single policy (but it's configurable?)
21:31:56 <acoles> (do I have any? :)
21:32:39 <acoles> clayg: you can set the policy for func tests, SWIFT_TEST_POLICY IIRC
21:32:54 <mattoliverau> I'll take a look at the testing patch. Especially if it helps acoles.
21:32:56 <acoles> clayg: and we could add an EC ring/policy to the in-process test setup
21:33:08 <notmyname> mattoliverau: thanks
21:33:14 <acoles> mattoliverau: thanks, there are some notes in review comments about how to verify it
21:33:42 <acoles> clayg: so we could even make func-in-process-fast-post use an EC policy!
21:33:47 <mattoliverau> acoles: awesome thanks.. I'll have a play with it during boring talks :)
21:34:01 <acoles> mattoliverau: has notmyname not spoken yet then :D:D
21:34:03 <notmyname> mattoliverau: there's one at 2:15 today that will be terribly boring ;-)
21:34:08 <notmyname> acoles: right!
21:34:19 <notmyname> perfect chance for code reviews
21:34:33 <mattoliverau> notmyname: lol, I need to listen so I can heckle
21:34:43 <ho_away> lol
21:34:55 <notmyname> oh, so mattoliverau gave his first ever public presentation yesterday (on container sharding in swift). he did great
21:35:13 <jrichli> have the link yet?
21:35:14 <mattoliverau> Well first ever solo one
21:35:35 <clayg> mattoliverau: congrats!
21:35:42 <ho_away> cool!
21:35:51 <clayg> mattoliverau: also thanks for looking at fix-rings-again2
21:35:53 <notmyname> acoles: have you done anything yet to update the gate jobs in -infra yet for the fast post stuff? or just waiting for this patch to land first?
21:35:57 <acoles> mattoliverau yay
21:36:32 <acoles> notmyname: not yet, I will let y'all know when I have a patch up for -infra gate job change, and yeah it will depend on this one
21:36:43 <notmyname> ok
21:37:06 <notmyname> acoles: thanks for this. I'm very happy about our fast-post future
21:37:20 <clayg> it's so bright I've gotta wear shades
21:37:26 <notmyname> #topic patch 202411
21:37:26 <patchbot> notmyname: https://review.openstack.org/#/c/202411/ - swift - Add functional test for access control (RBAC) with...
21:37:26 <acoles> thanks for the feedback
21:37:40 * acoles goes to make reconciler fast-post proof...
21:37:41 <clayg> crap
21:37:51 <ho_away> i got a +2 on it now so i want to have more reviews :-)
21:38:03 <clayg> i sorta got a keystone-swift-all-in working (thanks ho_away !) but I haven't looped back to to that change yet
21:38:16 <notmyname> ok, who's volunteering to be another reviewer?
21:38:22 <acoles> not me :)
21:38:25 <clayg> well that was mostly just acoles getting exahusted
21:38:31 <mattoliverau> sounds like clayg is
21:38:35 <clayg> s/mostly/partly/g
21:38:48 <clayg> mattoliverau: I'm trying to get more keystone friendly if that's what your suggesting
21:39:19 <clayg> ho_away: doesn't want *me* to review it - that's practiacally zero chance it makes it through that with a +A - clayg is stone cold heartless -1'er
21:39:35 <notmyname> clayg: cschwede: kota_: tdasilva: torgomatic: ?
21:39:57 <ho_away> clayg: lol
21:40:00 <torgomatic> I'm in the middle of untangling get_*_info, but after I can take a look
21:40:03 <cschwede> notmyname: looking at it right now (but no rating this evening)
21:40:16 <kota_> I also need to get more keystone friendly, once i tried it i got failed to set the keystone env :/
21:40:17 <acoles> clayg: ho_away took a bunch of -1s from me and is still standing
21:40:42 <clayg> kota_: try the keystone branch on swiftstack/swift-all-in-one - it's getting better all the time!
21:40:57 <clayg> acoles: ho_away is a rockstar
21:40:57 <kota_> clayg: \o/
21:41:03 <notmyname> ok, so between torgomatic cschwede clayg and kota_, it *should* be either landed or have other patch sets by next week ;-)
21:41:06 <clayg> kota_: yeah that's also mostly ho_away
21:41:07 <acoles> clayg: +1
21:41:34 <ho_away> torgomatic: cshwede: thanks!
21:41:37 <notmyname> ho_away: the keystone work on the vSAIO is really imporant, IMO. thanks
21:41:57 <acoles> kota_: write a script to feed keystone with random combinations of options until you get a 200 ;)
21:42:04 <notmyname> #topic open discussion
21:42:12 <notmyname> anything else to bring up in the meeting this week?
21:42:25 <awelleck> what does everyone think about https://blueprints.launchpad.net/python-swiftclient/+spec/optional-query-strings-on-requests
21:42:27 <clayg> anyone else noticed the 204 content-length thing!?
21:42:48 <clayg> I was like "i'll be damned i guess we suck"; then I was like WTF this wasn't in 2616?
21:43:19 <torgomatic> I love the tempest thing; there's tests in there that assert we have content-length in there, so we can't take it out to meet spec without first doing a ridiculous multi-repo song-and-dance
21:43:24 <notmyname> awelleck: yup. I think you'll also find a lot of agreement from timburke
21:44:09 <notmyname> clayg: that sounds like one of those things I marked as read this week. what's going on?
21:44:10 <acoles> clayg: link? haven't seen it
21:44:14 <clayg> awelleck: why is this a spec instead of a patch?
21:44:22 <timburke> do it! standardize all the things!
21:44:27 <awelleck> notmyname: ok thanks, ill work on it
21:44:29 <clayg> wait - that's not a spec - that's a blueprint - wtf are we supposed do with that?
21:44:32 <notmyname> awelleck: great thanks
21:44:45 <zaitcev> convert to spec by using cp(1)
21:44:53 <zaitcev> well, curl(1)
21:45:00 <notmyname> clayg: be nice. awelleck probably just found some openstack "here's how you're supposed to do it" docs :-)
21:45:07 <clayg> this is the rfc 7230 tempest thing -> https://review.openstack.org/#/c/275652/
21:45:22 <acoles> I don't get notified of blueprints - is there a way for that to happen?
21:45:41 <clayg> notmyname: awelleck: sry, if you remember where you found such a doc point me out it so I can hate on it
21:45:49 <clayg> don't shoot the messenger and all that
21:46:02 <awelleck> clayg: not sure
21:46:09 <acoles> clayg: oic. thaks
21:46:12 <clayg> awelleck: anyway - sounds broken and annoying - not sure where to put my +A/please-fix - but thanks for point it out
21:46:30 <clayg> i think we could make it an open bug - those we know how to deal with
21:46:42 <notmyname> acoles: no. there is no blueprint notification thing (for new ones. you have to subscribe to each one. yet another reason they are terrible)
21:46:55 <notmyname> clayg: so what do we do with the tempest thing?
21:47:01 <acoles> notmyname: ok. awelleck ^^ take note
21:47:04 <clayg> blueprint - more like BOOprint
21:47:35 <clayg> notmyname: oh idk, i htink they're going to spin their wheels getting tempest team happy only to realize this is for swift api next
21:47:38 <acoles> actually looks like ben martin wrote the blueprint
21:47:38 <clayg> ... as we do
21:47:50 <clayg> MOAR booprints!?
21:48:35 <clayg> notmyname: at first i was feeling bad because the language is pretty clear - but then I remembered we're older than 7230 and they can go pound out a time machine
21:49:32 <notmyname> I don't know. maybe dropping content-length on 204 isn't a breaking thing. idk
21:50:01 <notmyname> we've made similar changes before with standard headers, IIRC
21:50:12 * notmyname doesn't remember specifics
21:51:19 <notmyname> clayg: how'd you find this? do you monitor tempest patches? was it on the ML?
21:52:04 <clayg> notmyname: I think there's a bug associated with it?  i get bugs
21:52:10 <notmyname> ah ok
21:52:34 <notmyname> anything else to bring up at the last minute this week?
21:52:39 <notmyname> from anyone
21:52:46 <clayg> acoles: so what's the short and skinny on the bad with versioned_writes?
21:52:59 <clayg> it should be... before/after dlo/slo?
21:53:18 <timburke> i'd still like some reviews on adding delete markers to versioned_writes - https://review.openstack.org/#/c/214922/
21:54:05 <timburke> clayg: yeah, after both, apparently
21:54:31 <acoles> clayg: should be after slo
21:54:56 <jlhinson> patch 218490 has one +2, needs one more
21:54:57 <patchbot> jlhinson: https://review.openstack.org/#/c/218490/ - swift - Automatic refresh of memcache config settings
21:55:11 <acoles> clayg: but if you have config'd' dlo/slo but not versioned-writes then it gets inserted before them
21:55:49 <acoles> clayg: if you have not config'd dlo nor versioned-writes then it gets inserted after
21:56:02 <acoles> clayg: so as always, not even a simple bug
21:56:17 <notmyname> conference stuff is starting here, so I need to close the meeting and physically move
21:56:33 <notmyname> thank you everyone for attending and working on swift. you are what makes swift awesome
21:56:36 <notmyname> #endmeeting