| *** baojg has joined #openstack-swift | 02:03 | |
| *** rcernin has quit IRC | 02:45 | |
| *** rcernin has joined #openstack-swift | 02:47 | |
| *** m75abrams has joined #openstack-swift | 03:09 | |
| *** psachin has joined #openstack-swift | 03:31 | |
| *** psachin has quit IRC | 03:34 | |
| *** psachin has joined #openstack-swift | 03:37 | |
| *** psachin has quit IRC | 03:55 | |
| *** e0ne has joined #openstack-swift | 05:41 | |
| *** e0ne has quit IRC | 06:11 | |
| *** tesseract has joined #openstack-swift | 07:08 | |
| *** rcernin has quit IRC | 07:11 | |
| *** pcaruana has joined #openstack-swift | 07:31 | |
| *** tkajinam has quit IRC | 08:30 | |
| *** gkadam has joined #openstack-swift | 08:44 | |
| *** tesseract has quit IRC | 08:58 | |
| *** e0ne has joined #openstack-swift | 09:08 | |
| *** ccamacho has joined #openstack-swift | 09:13 | |
| *** tesseract has joined #openstack-swift | 09:20 | |
| *** csmart has quit IRC | 09:22 | |
| *** altlogbot_0 has quit IRC | 09:24 | |
| *** altlogbot_0 has joined #openstack-swift | 09:26 | |
| *** csmart has joined #openstack-swift | 09:33 | |
| *** m75abrams has quit IRC | 09:49 | |
| *** gkadam has quit IRC | 11:02 | |
| *** gkadam has joined #openstack-swift | 11:17 | |
| *** gkadam has quit IRC | 11:42 | |
| *** gkadam has joined #openstack-swift | 11:44 | |
| *** tdasilva has joined #openstack-swift | 12:11 | |
| *** ChanServ sets mode: +v tdasilva | 12:11 | |
| *** tdasilva has quit IRC | 12:16 | |
| *** tdasilva has joined #openstack-swift | 12:16 | |
| *** ChanServ sets mode: +v tdasilva | 12:16 | |
| *** mvkr has quit IRC | 12:22 | |
| *** BjoernT has joined #openstack-swift | 13:18 | |
| *** mvkr has joined #openstack-swift | 13:20 | |
| *** BjoernT_ has joined #openstack-swift | 13:29 | |
| *** BjoernT has quit IRC | 13:30 | |
| *** BjoernT has joined #openstack-swift | 13:33 | |
| *** BjoernT_ has quit IRC | 13:34 | |
| *** mvkr has quit IRC | 13:34 | |
| *** mvkr has joined #openstack-swift | 13:47 | |
| *** e0ne has quit IRC | 15:30 | |
| *** gyee has joined #openstack-swift | 15:39 | |
| openstackgerrit | Tim Burke proposed openstack/python-swiftclient stable/stein: Fix SLO re-upload https://review.opendev.org/673321 | 15:55 |
|---|---|---|
| openstackgerrit | Tim Burke proposed openstack/python-swiftclient stable/rocky: Fix SLO re-upload https://review.opendev.org/673322 | 15:56 |
| openstackgerrit | Tim Burke proposed openstack/python-swiftclient stable/queens: Fix SLO re-upload https://review.opendev.org/673323 | 15:56 |
| timburke | i really shoulda written up a bug for ^^^ | 15:59 |
| timburke | i realized that we probably ought to get that backported for the sake of distros -- bionic, for example, ships 3.5.0 | 15:59 |
| timburke | (which was also the release that introduced the bug) | 16:00 |
| *** mvkr has quit IRC | 16:01 | |
| *** tdasilva has quit IRC | 16:14 | |
| *** tdasilva has joined #openstack-swift | 16:15 | |
| *** ChanServ sets mode: +v tdasilva | 16:15 | |
| *** tesseract has quit IRC | 16:28 | |
| *** e0ne has joined #openstack-swift | 17:14 | |
| *** rovanleeuwen has joined #openstack-swift | 17:19 | |
| rovanleeuwen | Hello, I have a question: We had an container node that was on-line longer then the tombstone time-out and we started that up. Now we have lots of files in the listings that are actually not there. Is there a way to clean this mess we made? | 17:20 |
| *** e0ne has quit IRC | 17:20 | |
| rovanleeuwen | I meant off-line longer then tombstone time-out obviously | 17:20 |
| *** aj11 has joined #openstack-swift | 17:21 | |
| timburke | rovanleeuwen, ouch :-( yeah, long maintenance windows usually require that you either increase the reclaim age (at least for a while) or wipe the drives before re-introducing them to the cluster | 17:56 |
| timburke | fortunately, the situation you're in with objects appearing in listings that don't have data on-disk (which we sometimes call "ghost listings") isn't too bad to recover from: you can still issue deletes for the affected objects -- you'll get back a 404, but the object-server will still go update the container to mark the name as deleted | 17:56 |
| rovanleeuwen | Any easy way to easily find the files and clean them? It looks like the swift-account-audit will show you the error with the files with errors. I guess I can do some bash magic to issue deletes for those. But a option to that command to force a clean would be nice :) | 17:58 |
| timburke | i was just about to mention how the hard part becomes identifying the objects that were affected :-) you might want to spin up a special proxy server with a particularly-high request_node_count and long conn_timeout and node_timeout | 17:59 |
| timburke | you might also be able to do some digging through your proxy logs to find object names that were deleted | 17:59 |
| timburke | rledisez, alecuyer: if you get a chance, i feel like your particular perspective might be valuable on https://review.opendev.org/#/c/672186/ -- i feel like i'm changing our contract a decent bit and i think you'd both have a pretty good feel for whether this would be preferable for users | 18:05 |
| patchbot | patch 672186 - swift - Ignore 404s from handoffs for objects when calcula... - 6 patch sets | 18:05 |
| timburke | fwiw, tdasilva and i talked a bit about it last week, see http://eavesdrop.openstack.org/irclogs/%23openstack-swift/%23openstack-swift.2019-07-23.log.html#t2019-07-23T14:28:42 | 18:06 |
| tdasilva | rledisez, timburke: I thought that this etherpad was also a nice way to visualize which behavior changed and which remained the same: https://etherpad.openstack.org/p/swift-fail-responses | 18:09 |
| rledisez | thx for the link tdasilva | 18:13 |
| rledisez | is there something missing for situation 5? I don't understand it | 18:13 |
| rledisez | situation 6 might look a bit weird, but it clearly traduce the "eventually consistent" aspect of swift, i'm okay with that | 18:14 |
| rledisez | timburke: fpr sure it's gonna hurt my SLA, but it's for the best, right? :) | 18:15 |
| timburke | for 5 -- we may not actually have responses from primaries, because we *never made the requests* since the nodes were error-limited. so that situation is specifically for "all primaries are error-limited, all handoffs respond 404" | 18:17 |
| timburke | yeah, the SLA-aspect was part of my concern... | 18:17 |
| rledisez | timburke: ok, error limited, got it. | 18:18 |
| rledisez | just joking about SLA, it would feel like hiding the issue. is somebody monitoring 404? (we don't as we assumed it was an user issue) | 18:19 |
| timburke | we had a customer complain about incorrect status codes during a stress test -- they knew the data existed and had for a decent period of time. they were concerned about how putting load on the cluster to the point that timeouts started popping, nodes got error-limited, etc. would cause inconsistencies in what had been settled data | 18:24 |
| timburke | i guess in part i'm wondering about how concerned you'd be if there was a sudden up-tick in 503s after pushing that change out | 18:25 |
| timburke | but maybe you already have ratelimiting in place such that your customers aren't running into this problem? | 18:26 |
| rledisez | we don't do ratelimiting on GET. we do some on PUT/POST to protect the clusters. our 5xx levels is pretty low. would it make sense to make this new behavior configurable at least for few versions so that we can try it on one proxy, check what happens, etc… | 18:29 |
| timburke | rledisez, i can see about doing that. what's your thinking on the default for it? old-but-maybe-wrong behavior to minimize upgrade disruption? or new-but-more-error-ful behavior so we can get people where we "want" them faster? (cc clayg) | 18:34 |
| rledisez | I would say the new behavior is the default, with a note/warning in the changelog | 18:35 |
| timburke | 👍 | 18:36 |
| rledisez | I'm assuming everybody is reading the changelog, but I also know i'm probably wrong. good way to teach them they should :) | 18:36 |
| rledisez | totally unrelated subject: | 18:38 |
| rledisez | do you know where the requests dispatching to workers is done for the proxy? I would say eventlet, but i'm not confortable yet with wsgi.py :) | 18:38 |
| rledisez | I'm having the issue that it does not dispatch the requests equally (eg: round-robin). So, even if there are workers doing nothing, some workers might be handling multiple requests. | 18:38 |
| rledisez | It matters with EC, because one worker is running on one core, so the proxy-server can get CPU-bound while the system is mostly idle. | 18:38 |
| rledisez | hum, I think I get it. there is no "dispatch code", the kernel is auto-magically doing it | 18:53 |
| *** rovanleeuwen has quit IRC | 18:54 | |
| timburke | fwiw, we make GreenAsyncPiles to make requests and gather responses in each of Controller.make_requests and ResumingGetter._get_source_and_node in proxy/controllers/base.py and ECObjectController._get_or_head_response in proxy/controllers/obj.py | 19:07 |
| timburke | how it gets scheduled from there, i'm not so sure. sounds like you found out it's in kernel-land :-) | 19:08 |
| openstackgerrit | Tim Burke proposed openstack/swift master: Give ECAppIter greenthreads a chance to wrap up https://review.opendev.org/665773 | 19:11 |
| timburke | tdasilva, clayg: what do you guys think about the idea of having hardlink-to-symlink validation stop at the symlink? it feels a little weird (to me) that it's perfectly valid to have a symlink point to a name that doesn't exist yet, but you can't make a hardlink point to that symlink | 19:16 |
| clayg | yeah that sounds weird... that traversal code is a bit brittle - IIRC it's reused on PUT (validation) & GET | 19:18 |
| timburke | and given a symlink that *does* point at something, a hardlink pointing at it won't have any stronger guarantees for having done the extra work up front -- you could still swap out the target and the hardlink would 200 | 19:18 |
| clayg | yeah I'm pretty sure there's a test for something very close to that - so I'm kind of surprised by the situation you described... have you attempted a fix? | 19:19 |
| clayg | 100% agree there's no guarntees once the hardlink points to a symlink - at that point no further validation seems needed | 19:19 |
| clayg | from my perspective!!! 🤷♂️ | 19:20 |
| *** e0ne has joined #openstack-swift | 19:22 | |
| *** aj11 has quit IRC | 19:24 | |
| timburke | "I'm kind of surprised by the situation you described" -- what, that last part, where the hardlink-PUT fails? yeah, it responds 404 as it tries to follow the symlink, and _validate_etag_and_update_sysmeta() returns any error it hits during validation | 19:44 |
| clayg | yeah i guess that's the part that was surprising/undesirable - but thinking about how that code works for GET/PUT I guess it makes sense how we got here - if you drop a comment I'll figure out someway to get some tests together and make them pass | 20:02 |
| openstackgerrit | Merged openstack/swift stable/pike: Fix the pep8 test will raise "Unknown test found in profile: B109" https://review.opendev.org/673131 | 20:41 |
| *** e0ne has quit IRC | 20:46 | |
| *** tdasilva has quit IRC | 20:59 | |
| *** tdasilva has joined #openstack-swift | 20:59 | |
| *** ChanServ sets mode: +v tdasilva | 20:59 | |
| timburke | rledisez, if i'm adding a config option anyway... i wonder whether it ought to control the account/container behavior change from https://review.opendev.org/#/c/667411/ too... | 21:22 |
| patchbot | patch 667411 - swift - Return 503 when primary containers can't respond (MERGED) - 2 patch sets | 21:22 |
| timburke | there was much more obvious bad behavior there -- we'd cache a container's non-existence when it was actually just overloaded :-( | 21:22 |
| timburke | which would prevent further writes | 21:23 |
| *** baojg has quit IRC | 21:53 | |
| *** BjoernT has quit IRC | 21:59 | |
| *** mvkr has joined #openstack-swift | 22:06 | |
| *** zaitcev has joined #openstack-swift | 22:12 | |
| *** ChanServ sets mode: +v zaitcev | 22:12 | |
| zaitcev | Anyone noticed recently that proxy-server double-quotes in its log? | 22:22 |
| zaitcev | Jul 26 20:30:35 rhev-a24c-01 proxy-server[9123]: 127.0.0.1 127.0.0.1 27/Jul/2019/00/30/35 PUT /v1/AUTH_test/%25EF%25BF%25BD%25EF%25BF%25BD%25EF%25BF%25BD..... | 22:22 |
| zaitcev | Look at all these %25 | 22:22 |
| zaitcev | Same transaction: Jul 26 20:30:35 rhev-a24c-01 container-server[9036]: 192.168.50.1 - - [27/Jul/2019:00:30:35 +0000] "PUT /a2/196445/AUTH_test/%EF%BF%BD%EF%BF%BD%EF%BF%BD%EF | 22:23 |
| timburke | zaitcev, i'm even more concerned about the fact that there's a bunch of REPLACEMENT CHARACTERs... | 22:38 |
| timburke | the double-quoting is a pretty long-standing wart that even prompted notmyname to write https://github.com/openstack/swift/commit/11e81cfc8 | 22:40 |
| *** tkajinam has joined #openstack-swift | 22:54 | |
| *** rcernin has joined #openstack-swift | 23:02 | |
| openstackgerrit | Tim Burke proposed openstack/swift master: Ignore 404s from handoffs for objects when calculating quorum https://review.opendev.org/672186 | 23:18 |
| timburke | rledisez, i may need your help on the UpgradeImpact wording in ^^^ | 23:19 |
| timburke | as i tried to come up with an explanation there, i was really struggling to come up with a compelling reason to use the option -- it kinda feels like a cop-out to me | 23:20 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!