Monday, 2017-04-03

*** SkyRocknRoll has quit IRC00:04
*** m_kazuhiro has joined #openstack-swift00:52
m_kazuhirogood morning00:52
mattoliveraum_kazuhiro: morning00:53
m_kazuhiromattoliverau: morning!00:53
*** dja has quit IRC01:04
*** jamielennox|away is now known as jamielennox01:06
*** stradling has quit IRC01:22
*** rcernin has joined #openstack-swift01:58
kota_morning02:13
*** vint_bra has joined #openstack-swift02:39
*** klrmn has quit IRC02:41
*** rcernin has quit IRC02:45
*** catintheroof has joined #openstack-swift02:55
*** klrmn has joined #openstack-swift03:03
*** vint_bra has quit IRC03:12
mattoliveraukota_: morning03:15
kota_mattoliverau: o/03:16
*** dja has joined #openstack-swift03:16
*** dja has quit IRC03:38
*** klrmn has quit IRC03:45
*** chsc has joined #openstack-swift03:53
*** chsc has joined #openstack-swift03:53
*** dja has joined #openstack-swift03:55
*** dja has quit IRC04:01
*** Dinesh_Bhor has joined #openstack-swift04:01
*** gkadam has joined #openstack-swift04:03
*** catintheroof has quit IRC04:10
*** chsc has quit IRC04:12
*** psachin has joined #openstack-swift04:26
*** rcernin has joined #openstack-swift04:57
*** rcernin has quit IRC05:18
*** gkadam has quit IRC05:20
*** gkadam has joined #openstack-swift05:21
*** rcernin has joined #openstack-swift05:31
*** IRCFrEAK has joined #openstack-swift05:51
*** IRCFrEAK has left #openstack-swift05:53
*** adriant has quit IRC05:55
*** ChubYann has quit IRC06:20
*** cshastri has joined #openstack-swift06:25
*** sputnik13 has quit IRC06:32
*** sputnik13 has joined #openstack-swift06:34
openstackgerritMatthew Oliver proposed openstack/swift master: Add container sharding to Swift containers  https://review.openstack.org/42390606:34
openstackgerritMatthew Oliver proposed openstack/swift master: Container sharding docs  https://review.openstack.org/43832206:34
*** tesseract has joined #openstack-swift06:34
*** hoonetorg has quit IRC06:37
*** dja has joined #openstack-swift06:41
*** hseipp has joined #openstack-swift06:42
*** oshritf has joined #openstack-swift06:47
openstackgerritMerged openstack/swift master: Fix intermittent test_unlink_* failures  https://review.openstack.org/45159306:55
*** dja has quit IRC06:57
*** pcaruana has joined #openstack-swift06:57
*** jaosorior has joined #openstack-swift06:57
*** geaaru has joined #openstack-swift07:15
*** amoralej|off is now known as amoralej07:20
acolesgood morning07:51
mattoliverauacoles: morning o/08:00
acolesmattoliverau: o/ welcome to (your) winter time :)08:00
*** openstackgerrit has quit IRC08:03
*** jordanP has joined #openstack-swift08:06
*** cbartz has joined #openstack-swift08:15
*** openstackgerrit has joined #openstack-swift08:24
openstackgerritMahati Chamarthy proposed openstack/swift master: Limit number of revert tombstone SSYNC requests  https://review.openstack.org/43957208:24
mattoliverauacoles: thanks and it's been raining ll day so kinda feels like it too08:30
mattoliverauacoles: I meged your sharding config changes into the parent. Still want to play with misplaced objects.08:30
mattoliveraualso thanks so much for all that reviewing08:31
acolesmattoliverau: :( rain08:31
acolesmattoliverau: ok cool. I ran out of time last week re misplaced objects, I think the test is sufficient to demonstrate the problem but I had no time to look t how easy it might be to workaround. Also the test could probably be fleshed out some more to cover more cases/more objects/more pivots??? consider it "scaffolding"!08:33
*** openstackgerrit has quit IRC08:33
acolesgot me thinking about probe tests for this stuff08:33
mattoliverauacoles: yeah totally, I figure scaffolding it a great start.08:34
acolesmattoliverau: I hate to add to your gloomy day but it is beautiful spring sunshine here :P08:34
mattoliverauyeah, I've been meaning to make a basic probe test that would get us containers in the sharding and sharding state so we can start testing more edge cases (there is a trello card for it)08:35
mattoliverauacoles: lol, damn you :P08:35
acolesmattoliverau: actually I think this past weekend has been what we get to call "summer" in its entireity08:35
mattoliverauWe've had a hurrican (way up north) but the weather has pushed alot of rain08:35
acolesmattoliverau: I should check out the trello board, hope to give this more time over coming weeks08:36
mattoliveraulol08:36
mattoliverauI try and update it with things as we think of them, or as people (you and tim) bring up in reviews.08:36
acolesgood idea08:36
acoleswith big features like this its easy to lose track of comments/todos on gerrit as the patch versions roll on, best to keep a list somewhere else08:37
acolesmattoliverau: looks like you and kota were making progress on patch 448480?08:38
mattoliverauyeah, there are things I said 'great idea' too in earlier patchsets that I think I may have already forgotten about.. so yeah can see how easy that is to happen.08:38
patchbothttps://review.openstack.org/#/c/448480/ - swift - Container drive error results double space usage o...08:38
mattoliverauyup, I wrote some probetests if you need an easy way to play with it.08:39
kota_acoles: o/08:39
acoleshi kota_08:39
mattoliverauthere should be a paste linked in the review.08:40
* kota_ was concentrating another patch08:40
mattoliverauA spend a bit of this arvo looking at the composit ring stuff (want to have more of a play with that too)08:40
mattoliverauacoles: seeing as you have a UUID patch for the ringbulder (re:composit rings) did you want to add a rebalance timestamp or something (last rebalance) at the same time.. that is if we want to compare when last rebalanced too.. tho maybe we only want to check if different no necessarily newer.08:40
kota_oops, sounds thunder in Tokyo08:40
acolesmattoliverau: oic, a probe test?08:41
mattoliverau2, one to demonstraight the problem. the other to play with a edgecase that kota_ and I talked about in channel last Friday.08:41
acolesmattoliverau: ok.08:42
mattoliverauit should apply to the review patch.08:42
mattoliverauk need to go do baby night rutine things now, bath, story and bed.. bbl.08:44
acolesmattoliverau: rebalance timestamp or some other way to compare the state of rings might be useful (e.g. md5 of replica2part2dev). I'm hoping we can make progress (i.e. land something) and follow on with those ideas. TBH I am not convinced how much we should care about re-composing rings that have not changed now it is just an offline process, but it would be a nice check to have08:44
acolesmattoliverau: k have a good evening08:44
mattoliverauacoles: yeah, good point (re:being offline). Still want to make it as fool proof as possible, cause I can be pretty foolish ;)08:45
acoleshehe08:45
* acoles going for coffee08:45
mahaticacoles: mattoliverau o/08:49
mahatickota_: o/08:49
mattoliveraumahatic: o/08:52
*** gkadam_ has joined #openstack-swift08:52
*** gkadam has quit IRC08:53
kota_mahatic: o/08:53
*** mtreinish has quit IRC08:53
*** mtreinish has joined #openstack-swift08:57
acolesmahatic: welcome back, did you have a good break?09:00
mahaticacoles: thanks. yup, did some hiking in the hills. away from the Bangalore summer :)09:03
acolesnice09:03
*** szaher has quit IRC09:25
*** szaher has joined #openstack-swift09:30
cbartzAnyone here who can review patch 414232 ? Thx....09:31
patchbothttps://review.openstack.org/#/c/414232/ - swift - Exclude containers for account quota09:31
*** szaher has quit IRC09:33
*** szaher has joined #openstack-swift09:39
onovyfunny bug: https://bugs.launchpad.net/swift/+bug/167897609:51
openstackLaunchpad bug 1678976 in OpenStack Object Storage (swift) "POST with Range header corrupts data without fast-POST" [Undecided,New]09:51
*** silor has joined #openstack-swift09:55
*** sams-gleb has joined #openstack-swift09:59
*** m_kazuhiro has quit IRC10:00
*** szaher has quit IRC10:01
*** silor has quit IRC10:03
*** silor has joined #openstack-swift10:04
*** szaher has joined #openstack-swift10:04
*** rcernin has quit IRC10:05
*** sams-gleb has quit IRC10:05
*** oshritf has quit IRC10:05
*** sams-gleb has joined #openstack-swift10:06
*** hoonetorg has joined #openstack-swift10:07
*** oshritf has joined #openstack-swift10:07
*** szaher has quit IRC10:10
*** sams-gleb has quit IRC10:10
*** szaher has joined #openstack-swift10:11
*** rcernin has joined #openstack-swift10:24
*** oshritf has quit IRC10:31
*** psachin has quit IRC10:34
*** oshritf has joined #openstack-swift10:50
*** psachin has joined #openstack-swift10:51
*** oshritf has quit IRC10:52
*** dja has joined #openstack-swift10:56
*** oshritf has joined #openstack-swift11:01
*** kei_yama has quit IRC11:04
*** oshritf has quit IRC11:10
*** oshritf has joined #openstack-swift11:32
*** ma9_ has joined #openstack-swift11:45
*** dja has quit IRC11:48
*** oshritf has quit IRC11:48
*** openstackgerrit has joined #openstack-swift11:49
openstackgerritOndřej Nový proposed openstack/swift master: Remove Range header from non-GET requests  https://review.openstack.org/45272611:49
*** oshritf has joined #openstack-swift11:52
openstackgerritOndřej Nový proposed openstack/swift master: Remove Range header from non-GET requests  https://review.openstack.org/45272611:54
*** sams-gleb has joined #openstack-swift11:56
*** psachin has quit IRC11:58
*** hseipp has quit IRC12:05
*** psachin has joined #openstack-swift12:11
*** silor1 has joined #openstack-swift12:15
*** silor has quit IRC12:19
*** silor1 is now known as silor12:19
*** vint_bra has joined #openstack-swift12:27
*** vint_bra has quit IRC12:29
*** jaosorior is now known as jaosorior_brb12:36
*** amoralej is now known as amoralej|lunch12:54
*** adu has quit IRC13:13
openstackgerritAlistair Coles proposed openstack/swift master: Fix UnicodeDecodeError in reconstructor _full_path function  https://review.openstack.org/45275013:14
acolesclayg: ^^ I think you found a different bug to rledisez13:15
*** gkadam_ is now known as gkadam13:15
acolesrledisez: what version were you running when you saw bug 1678018 ?13:15
openstackbug 1678018 in OpenStack Object Storage (swift) "ssync_sender encoding issue" [Undecided,Confirmed] https://launchpad.net/bugs/167801813:15
*** gkadam has quit IRC13:17
*** stradling has joined #openstack-swift13:18
rledisezacoles: it was on swift 2.12. in the past we also ran 2.7 with the policy13:18
*** catintheroof has joined #openstack-swift13:31
*** jaosorior_brb is now known as jaosorior13:33
*** ma9_1 has joined #openstack-swift13:37
*** ma9_ has quit IRC13:39
*** stradling has quit IRC13:48
*** vint_bra has joined #openstack-swift14:15
*** amoralej|lunch is now known as amoralej14:20
*** cshastri has quit IRC14:33
*** thurloat[m] has joined #openstack-swift14:36
*** chlong has joined #openstack-swift14:45
*** Guest56308 has joined #openstack-swift15:01
*** stradling has joined #openstack-swift15:01
*** Guest56308 has quit IRC15:06
*** chsc has joined #openstack-swift15:07
*** klrmn has joined #openstack-swift15:15
*** oshritf has quit IRC15:20
*** tesseract has quit IRC15:31
notmynamegood morning15:42
notmynamethe TC has https://review.openstack.org/#/c/451524/ on the agenda for their meeting tomorrow. /cc tdasilva15:43
notmynamepatch 45152415:43
patchbothttps://review.openstack.org/#/c/451524/ - governance - Use case analysis for Golang addition to Openstack15:43
*** jordanP has quit IRC15:43
notmynameacoles: do you remember when joel gets back?15:45
acolesnotmyname: I believe he is back - I saw him leave some review comments  late last week15:45
notmynameok15:45
notmynameoh! you had time change this past weekend, right?15:46
acolesunless he did that from a mountain side somewhere15:46
acolesnotmyname: nope we've been on BST for a whole week now :)15:46
acolesnotmyname: however mattoliverau did just change clocks15:46
notmynameah15:47
*** mvk has quit IRC15:47
*** pcaruana has quit IRC15:53
*** sams-gleb has quit IRC15:54
*** sams-gleb has joined #openstack-swift15:55
*** sams-gleb has quit IRC15:59
*** stradling has quit IRC16:00
*** _JZ_ has joined #openstack-swift16:05
*** JimCheung has joined #openstack-swift16:07
*** psachin has quit IRC16:10
*** klrmn has quit IRC16:13
*** ma9_1 has quit IRC16:17
*** ma9_ has joined #openstack-swift16:18
*** d0ugal has quit IRC16:19
*** ma9_ has quit IRC16:22
*** sams-gleb has joined #openstack-swift16:24
openstackgerritAlistair Coles proposed openstack/swift master: sharding: Test and refactor for _misplaced_objects  https://review.openstack.org/45189616:28
*** ma9_ has joined #openstack-swift16:28
acolesmattoliverau: ^^ the test now passes, I figured out how to get the list_iter_objects to return the full row info16:29
acolesmattoliverau: the interface into the broker is a bit crufty, we might want to make a variant of list_objects_iter that returns a dict like you use generate_list_objects to produce, save on an iteration at same time16:32
tdasilvanotmyname: ack, got an email from ttx16:35
*** stradling has joined #openstack-swift16:35
notmynametdasilva: anything I should know?16:36
tdasilvanotmyname: it was just the meeting agenda16:36
tdasilvahe sent it to tc list and cc'd me on it16:36
*** cbartz has left #openstack-swift16:39
*** ma9_ has quit IRC16:55
*** stradling has quit IRC16:57
openstackgerritOpenStack Proposal Bot proposed openstack/swift master: Updated from global requirements  https://review.openstack.org/8873616:59
*** chlong has quit IRC17:05
*** stradling has joined #openstack-swift17:09
*** klrmn has joined #openstack-swift17:13
*** d0ugal has joined #openstack-swift17:15
*** chsc has quit IRC17:16
*** chlong has joined #openstack-swift17:20
*** jaosorior is now known as jaosorior_away17:29
*** chosafine has joined #openstack-swift17:46
*** stradling has quit IRC17:49
*** jamielennox is now known as jamielennox|away17:50
*** SkyRocknRoll has joined #openstack-swift17:51
*** chosafine has quit IRC17:53
*** amoralej is now known as amoralej|off17:57
*** d0ugal has quit IRC17:59
*** ChubYann has joined #openstack-swift18:02
openstackgerritChristian Schwede proposed openstack/swift master: Add commands to ring-builder to change region / zone  https://review.openstack.org/29199018:14
*** silor1 has joined #openstack-swift18:17
*** silor has quit IRC18:20
*** silor1 is now known as silor18:20
*** d0ugal has joined #openstack-swift18:20
*** tonanhngo has joined #openstack-swift18:43
*** silor1 has joined #openstack-swift18:49
*** silor has quit IRC18:49
*** silor1 is now known as silor18:49
*** silor has quit IRC18:53
*** JimCheung has quit IRC18:54
*** JimCheung has joined #openstack-swift18:55
*** d0ugal has quit IRC19:23
mgagnewhat's the best way to bulk delete a container with millions of objects? :D19:25
*** d0ugal has joined #openstack-swift19:25
*** stradling has joined #openstack-swift19:28
tdasilvamgagne: one way is to use the bulk delete api: https://docs.openstack.org/developer/swift/middleware.html#bulk-delete ;)19:30
mgagnetdasilva: thanks, will have to check if bulk is used. The openstackclient is used and multiple objects are passed. But at first glance, it looks like it doesn't translate to bulk :-/19:32
mgagnelooks like "native" swift client has more options at the CLI19:34
*** d0ugal has quit IRC19:36
tdasilvamgagne: yeah, using either cli is a different approach that can also be useful because you could potentially be spreading the load across many proxies19:36
tdasilvaas opposed to the bulk api where you are sending a request to one proxy and that one proxy is sending all delete requests to the storage servers19:37
mgagnetrue, will check that, thanks =)19:37
*** chlong has quit IRC19:37
mgagnewe aren't using the bulk API and so far, it's gonna take weeks to delete =)19:38
*** chsc has joined #openstack-swift19:39
*** chlong has joined #openstack-swift19:51
*** jamielennox|away is now known as jamielennox20:05
claygsteps to fix a bug in swift #1 report the bug #2 understand what's broke #3 write the trivial and obvious fix #4 ship it - when we try to do #1-3 at the sametime things get gummed up in boring and useless ways - please report issues as early as possilbe and make sure the bug is triaged before commiting to writing tests/change - if you happen to have a patch that womm you can attach it to the bug report to help with the triage.20:36
notmynameI sense frustration20:36
notmynamebut you're not wrong!20:37
*** zaitcev has joined #openstack-swift20:37
*** ChanServ sets mode: +v zaitcev20:37
*** dja has joined #openstack-swift20:42
*** stradling has quit IRC20:43
*** mvk has joined #openstack-swift20:52
*** sams-gleb has quit IRC20:55
*** d0ugal has joined #openstack-swift21:01
*** dja has quit IRC21:16
*** mattoliverau has quit IRC21:24
*** darrenc has quit IRC21:24
*** darrenc has joined #openstack-swift21:24
*** matt6434 has joined #openstack-swift21:25
*** dja has joined #openstack-swift21:44
*** dja has quit IRC21:49
claygnotmyname: nah, it's fine - but I don't feel great about setting a bug from in progress to incomplete - or debating internally if my notes from investigation should go on the bug report or the patch - and I feel like I've done that twice in the two days back from vacation21:53
claygI think the solution is to stop writing bugs - process of not having any bugs to fix is more similar to vacation ;)21:54
*** dja has joined #openstack-swift22:12
*** rcernin has quit IRC22:12
zaitcev"In one of our installations, we are using Fernet tokens with Keystone V3. In that context, I was wondering if we need the keystone revocation thread in RGW to run by default." -- Ceph users, they do curious things.22:19
*** matt6434 is now known as mattoliverau22:22
*** ChanServ sets mode: +v mattoliverau22:22
mattoliverauMorning22:24
*** stradling has joined #openstack-swift22:25
*** stradling has quit IRC22:27
*** chlong has quit IRC22:38
*** geaaru has quit IRC22:38
*** catintheroof has quit IRC22:44
mattoliverauacoles: ta, yeah, that's a good point. that would clean it up alot.22:51
*** vint_bra has quit IRC22:57
*** adriant has joined #openstack-swift23:01
*** _JZ_ has quit IRC23:06
notmynamegood morning mattoliverau23:07
notmynamemattoliverau: I was wondering what's going on with stable gate tests23:08
mattoliveraunotmyname: not too much, I have it running, but haven't checked it much.. I'll make sure I spend some time today spending a few cycles playing with it some more :)23:10
notmynamemattoliverau: https://review.openstack.org/#/c/448854/ seems to be passing the gate now, but wasn't there another issue too? (sorry, I had a 4 day weekend, so my memory is a little fuzzy)23:10
notmynamemattoliverau: yeah, thanks. I was looking at other backports, and that's when I remembered something something something big problem matt's fixing it23:10
mattoliveraulol23:10
mattoliveraunotmyname: so tldr, ther doesn't seem to be any big problems, as each intermittent error is mostly different. so I think it's just flaky testing edge cases in general.23:12
mattoliverauand sometimes it takes a while on an SAIO to hit one.23:12
notmynameyeah, but timburke had mentioned that they'd gotten a lot more frequent lately. i think that was the only warning flag23:13
mattoliverauBut I want to keep an eye, because as we solve each one (over time) we can learn the courses and teach us all how to write better tests :)23:13
mattoliveraunotmyname: yeah, the periodic jobs in gerrit seem to hit alot more. though that do run once a day.23:14
notmynameyeah, if it's so rare in the saio, I'd expect it to be even more rare in a once-daily job23:14
mattoliverauso what 1 a week (pulling a number out of there air) is alot more regualar that what I'm seeing.23:15
mattoliverau^ so what you said :P23:15
mattoliveraunotmyname: is it a higher contended bunch of servers, causing more slips in say dealing with timestamps then what happens to us locally, or the way nodepool builds test nodes?23:16
mattoliverauhmm, maybe I could go build a node pool image, and then run some tests on that.23:16
mattoliverauor however they do that nowdays23:16
mattoliverauI might ask Josh in our meeting today :)23:17
mattoliverau^ I mean the Australian rpc standup23:18
mattoliverauk, email read, /me is going to  grab a coffee. bbs23:19
*** tonanhngo has quit IRC23:24
*** kei_yama has joined #openstack-swift23:36
*** tonanhngo has joined #openstack-swift23:37
*** tonanhngo has quit IRC23:41

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!