| *** gyee has quit IRC | 00:03 | |
| clayg | i love how no one even suggests a test for p 742549 (poor container sync) | 00:32 |
|---|---|---|
| patchbot | https://review.opendev.org/#/c/742549/ - swift - Actually interpolate error message - 1 patch set | 00:32 |
| *** mvkr has quit IRC | 02:17 | |
| *** baojg has quit IRC | 02:17 | |
| *** baojg has joined #openstack-swift | 02:18 | |
| *** mvkr has joined #openstack-swift | 02:30 | |
| *** rcernin has quit IRC | 03:28 | |
| *** rcernin has joined #openstack-swift | 03:43 | |
| *** rcernin has quit IRC | 03:47 | |
| *** rcernin has joined #openstack-swift | 03:49 | |
| *** m75abrams has joined #openstack-swift | 05:35 | |
| openstackgerrit | Merged openstack/swift master: Actually interpolate error message https://review.opendev.org/742549 | 08:11 |
| *** rcernin has quit IRC | 08:56 | |
| *** tristan888 has joined #openstack-swift | 10:06 | |
| *** tristan888 has quit IRC | 12:07 | |
| *** baojg has quit IRC | 12:53 | |
| *** baojg has joined #openstack-swift | 12:54 | |
| *** jadonn has quit IRC | 12:58 | |
| *** baojg has quit IRC | 13:35 | |
| *** baojg has joined #openstack-swift | 13:37 | |
| * ktsuyuzaki noticed he missed the meeting in this week because of holiday week in Japan, sorry. | 13:49 | |
| *** m75abrams has quit IRC | 14:04 | |
| *** baojg has quit IRC | 15:48 | |
| *** baojg has joined #openstack-swift | 15:49 | |
| openstackgerrit | Tim Burke proposed openstack/swift master: Allow replication servers to handle all request methods https://review.opendev.org/740867 | 16:11 |
| openstackgerrit | Tim Burke proposed openstack/swift master: Allow direct and internal clients to use the replication network https://review.opendev.org/735751 | 16:11 |
| openstackgerrit | Tim Burke proposed openstack/swift master: wip: Allow probe tests to run with separate replication network https://review.opendev.org/741723 | 16:11 |
| clayg | it seems like rolling upgrades with encryption has some difficulties p 742033 | 16:12 |
| patchbot | https://review.opendev.org/#/c/742033/ - swift - py3: Work with proper native string paths in crypt... - 3 patch sets | 16:12 |
| timburke | hmm... we forgot about that for the v1->v2 transition... | 16:14 |
| timburke | or we just decided to accept that some 500s on read were acceptable so long as the upgrade window was reasonably short | 16:15 |
| timburke | and of course, it only affects new data written since the upgrade began. so *that's* good (or, about as good as we could hope for, anyway) | 16:17 |
| *** gyee has joined #openstack-swift | 16:17 | |
| timburke | hmm... since py2 was already writing things in this format, i suppose we could continue calling it v2... the only time that you really want the certainty of knowing that data was written after this fix is if you've also got a bad pipeline so you're affected by https://bugs.launchpad.net/swift/+bug/1813725 | 16:52 |
| openstack | Launchpad bug 1813725 in OpenStack Object Storage (swift) "Object encryption key truncated sometimes when used with Barbican" [Medium,Confirmed] | 16:52 |
| timburke | clayg, i wonder if maybe we should've had a sense of major/minor versions for encryption key ids... the variations that we have today are all trying for the same core idea; in most cases, all three versions likely produce the same value... but at the same time, i really like that we have a way of saying "this was written *after* this fix was applied; you don't need to waste time checking for x y and z" | 17:01 |
| timburke | upgrade step 1: turn on https://github.com/openstack/swift/blob/master/swift/common/middleware/read_only.py :P | 17:02 |
| openstackgerrit | Tim Burke proposed openstack/swift master: Allow direct and internal clients to use the replication network https://review.opendev.org/735751 | 17:16 |
| openstackgerrit | Tim Burke proposed openstack/swift master: wip: Allow probe tests to run with separate replication network https://review.opendev.org/741723 | 17:16 |
| clayg | yeah there's not a whole lot that can be done now is there; even if we wrote "version 2.1" it still wouldn't be in (1, 2) - stupid old code | 17:40 |
| openstackgerrit | Tim Burke proposed openstack/swift master: container-sharding: Stable overlap order https://review.opendev.org/742535 | 17:41 |
| timburke | clayg, so... what do we want to do about it? is it ok to bump versions? or should we punt for now and say we're just making py3 behave like py2, so keep it v2? | 17:46 |
| timburke | there'll still be an opportunity for old proxies to 500 because of writes during the upgrade, though | 17:47 |
| clayg | even if we don't change the version ? | 17:47 |
| timburke | yup -- think about seongsoocho's cluster, that's py3 today. during the upgrade, both old and new proxies will be able to read already-written data, but newly written data will only be readable by new proxies (if there's non-ascii in the path) | 17:48 |
| timburke | py2 -> py3-with-latest should be smooth at least, though | 17:50 |
| clayg | well then blowing up on a version check seems more reasonable than blowing up unexpected metadata formatting | 17:50 |
| timburke | the downside, of course, is that the version check (currently) blows up on *all* paths, even ascii-only ones | 17:51 |
| timburke | maybe there's a reasonable middle ground? write down that it's v2 if all ascii, v3 if not? | 17:52 |
| clayg | is there | 18:00 |
| clayg | let me look more closely at the data format and see if i have any ideas | 18:00 |
| clayg | nm, i still don't know what a "wsgi string" is; I'm not going to be any help 😞 | 18:23 |
| openstackgerrit | Merged openstack/swift stable/ussuri: py3: Stop munging RAW_PATH_INFO https://review.opendev.org/742338 | 18:25 |
| timburke | clayg, sorry, that's on me for not explaining it enough/well enough. i'm debating about whether we should apply http://paste.openstack.org/show/796261/, basically -- so if the v2 and v3 paths would be the same on both py2 and py3, write it down as v2 | 19:09 |
| clayg | heh, I don't think it's your fault I don't know what a wsgi string is 😁 | 19:10 |
| timburke | i made up the term, i think it's on me to explain it ;-) | 19:11 |
| clayg | heh | 19:11 |
| timburke | anyway, doing something similar for the v1/v2 split seems *way* trickier/more dangerous because of the copy pipeline placement problem | 19:11 |
| clayg | i'm not familiar with "the copy pipeline placement problem" | 19:12 |
| clayg | the paste seems good and bad - on one hand if it's the best we can do it's probably "good enough" for us in that most of our ML dataset is ascii | 19:13 |
| clayg | the bad parts are 1) it doesn't work in general and 2) there's ambiguity about version/format going forward | 19:13 |
| clayg | i.e. more cruft & less obvious for "only just barely works in this one cluster" that we happen to care about disproportionately - not a great trade off | 19:14 |
| clayg | i'm remembering something similar about quoted-roots - were we wanted to get to the new format, but ended up just deprecating the old despite always still writing it 🤔 | 19:15 |
| timburke | "the copy pipeline placement problem": we don't have good checks on pipeline ordering (i should rebase p 635040 ...) which makes it easy to have "bad" pipelines. middleware auto-insertion makes this worse -- in particular, in bug 1813725 i found that if you didn't explicitly place copy *and* you only had one proxy-logging middleware (at the end of the pipeline, after encryption), then when users copy data around, the crypto meta continues | 19:18 |
| openstack | bug 1813725 in OpenStack Object Storage (swift) "Object encryption key truncated sometimes when used with Barbican" [Medium,Confirmed] https://launchpad.net/bugs/1813725 | 19:18 |
| timburke | pointing at the old location | 19:18 |
| patchbot | https://review.opendev.org/#/c/635040/ - swift - Include some pipeline validation during proxy-serv... - 5 patch sets | 19:18 |
| timburke | as for quoted-root, looks like we intentionally chose *not* to continue writing the old meta: https://github.com/openstack/swift/blob/2.25.0/swift/container/sharder.py#L673-L676 | 19:22 |
| timburke | i think i'd originally had it writing down both, though | 19:22 |
| clayg | oh, good there was some sort of graceful degradation - that's good that we didn't have to write both | 19:23 |
| clayg | so it's a similar problem yeah? 1) we need new py3 proxies to read old py2 data 2) we want to have less variety in what we write down | 19:24 |
| clayg | what would it take to write down something that old py2 proxies would read? | 19:25 |
| timburke | just gotta say that it's v2. we could do that in all cases, but it'd make old py3 proxies sad. maybe i add a config option? write_metadata_version = 2|3 ? default to 2 this cycle, 3 the next, and hope to get rid of it... someday? | 19:29 |
| timburke | if we're going that route, i'm inclined to have it *really* write the old (bad) format on py3 when you set it to 2. should ensure a smooth upgrade for everyone, assuming nobody's running a mixed py2/py3 environment in perpetuity | 19:32 |
| clayg | yeah there's some ambiguity in the v2 format? it's writes one set of bytes on py2 and a different set on py3 | 19:35 |
| timburke | yup | 19:35 |
| clayg | the only thing i'm sure of we should introduce code that provides a way for future us to write data in backwards compatible format | 19:35 |
| clayg | i.e. only blow up on version < 2 or something | 19:36 |
| clayg | you said major minor - which might make sense with a write_metadata_version tunable for when we want to cross major versions | 19:36 |
| timburke | all right, i'll see what it's like to add a config option to continue writing down the bad until everybody's upgraded. my plan for the default seem reasonable? 2 for now, put something on my calendar to remind me to switch it to 3 in like 6mos or a year? that way you can upgrade without needing to change configs first | 19:39 |
| clayg | i'm not sure i follow well enough to say | 19:42 |
| timburke | i'm a little surprised that nobody noticed or complained about the break in https://github.com/openstack/swift/commit/43103319d more than a year ago... should i try to support v1, too? 🤮 | 19:43 |
| clayg | it seems like maybe the default that makes sense depends on python you're running somehow? as log as new code can read everything - what's the driver to start writing down in a different format? | 19:43 |
| timburke | wanting to minimize the ambiguity that gets written down | 19:45 |
| clayg | maybe it was just early enough we didn't have a lot of encryption | 19:45 |
| clayg | can v2 just grow a new meta-meta that says what the encoding is? | 19:45 |
| timburke | still has the same trouble of "what's the driver to start writing down in a different format," no? | 19:54 |
| *** renich has joined #openstack-swift | 21:03 | |
| openstackgerrit | Merged openstack/swift master: Allow replication servers to handle all request methods https://review.opendev.org/740867 | 21:23 |
| clayg | hugokuo: lp bug #1778002 is going to be annoying... ssync has some cheeky assumptions about durablity https://github.com/openstack/swift/blob/master/swift/obj/ssync_receiver.py#L265 | 21:27 |
| openstack | Launchpad bug 1778002 in OpenStack Object Storage (swift) "EC non-durable fragment won't be deleted by reconstructor. " [High,Confirmed] https://launchpad.net/bugs/1778002 | 21:27 |
| clayg | technically there's not currently a protocol to send a non-durable frag archive 🤔 | 21:27 |
| openstackgerrit | Tim Burke proposed openstack/swift master: crypto: Add config option to support rolling upgrades https://review.opendev.org/742756 | 21:57 |
| timburke | clayg, see what you think of ^^^ -- i'd be happy to squash it in, too | 21:57 |
| *** dsariel has joined #openstack-swift | 22:11 | |
| *** dsariel has quit IRC | 22:12 | |
| timburke | ugh -- that doesn't actually work right, it starts using the wrong path when making the key... gonna take me a bit to get it right, i think :-( | 22:18 |
| *** dsariel has joined #openstack-swift | 22:21 | |
| *** dsariel has quit IRC | 22:22 | |
| openstackgerrit | Tim Burke proposed openstack/swift master: crypto: Add config option to support rolling upgrades https://review.opendev.org/742756 | 22:57 |
| openstackgerrit | Tim Burke proposed openstack/swift master: crypto: Add config option to support rolling upgrades https://review.opendev.org/742756 | 23:28 |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!