Wednesday, 2023-01-04

opendevreviewTim Burke proposed openstack/swift master: CI: pin tox at the project level
reid_gHello, Looking at this bug's upgrade impact. "Depending on the order in which shared libraries are loaded, your servers may already be reading and writing zlib CRCs, even with old liberasurecode. In that case, no special action is required and WRITING LEGACY CRCS DURING THE UPGRADE WILL CAUSE AN OUTAGE." How does writing legacy crcs cause an outage if liberasurecode 1.6.2 can read/write both?13:47
reid_gI don't think it affects us but was just curious.13:47
reid_gWe are going to add a systemd override to swift-proxy, swift-object, swift-object-reconstructor systemd services to add the env var to enable legacy crcs on servers upgrade to 20.04 before we start swift services. We're on swift 2.25.213:52
timburkereid_g, libec 1.6.2 can read both zlib crcs and "legacy" crcs, and write one or the other16:56
timburkeold libec (1.5.0 or earlier) can read and write just one of the two, and which one depends on library-load order -- if liberasurecode gets loaded before libz, it only knows how to use legacy crcs; if libz gets loaded before liberasurecode, it only knows how to use zlib crcs16:56
timburkethe potential outage comes from rolling upgrades: if an upgraded node computes and writes EC fragments into the cluster and an old node goes to read it, the old node won't be able to. the impact of that varies:16:57
timburkeif an upgraded proxy writes (default) zlib crcs to a bunch of older objects-servers that are using legacy crcs, the fragments will later get quarantined when the object-server goes to read them16:57
timburkeif an upgraded reconstructor uses the env var to ensure it writes legacy crcs when rebuilding frags but pushes them to older object-servers that are using zlib crcs (because that's how the library load-order worked out), these will similarly be quarantined eventually16:57
timburkethere's also situations like an upgraded container-reconciler moving data into an erasure-coded policy -- even if all of the object-servers have already been upgraded, any older proxies wanting to read the data may hit checksum errors and return 503s16:57
timburkethe first two start to look like data loss, though with enough operator intervention it should still be recoverable; the last one is at least limited to "just" being an availability issue16:59
timburkemy recommendation would be to use the script from on a few fragments from various nodes in the cluster to determine whether you've got legacy or zlib crcs17:01
timburkein practice, i'd expect you to get a single answer on that -- if there are a mix, it'd indicate that library-load order is causing incompatibilities even with the old version17:03
timburkeonce you have an answer to whether legacy crcs are in use, you know whether you ought to set the env var (legacy crcs are used) or not (zlib crcs are used). once all nodes are upgraded, you can remove the env var and always write zlib crcs17:05
reid_gOk. That is what I thought but the wording in that post confused me a little. We are currently writing legacy on our Ubuntu 18.04 servers. After upgrading to 20.04 they were writing zlib.17:58
timburke👍 got it -- so yeah, adding the env var via systemd files makes perfect sense, and as long as all writers get it, the upgrade should go smoothly18:43
opendevreviewTim Burke proposed openstack/swift master: CI: pin tox at the project level
opendevreviewTim Burke proposed openstack/swift stable/zed: CI: pin tox at the project level
timburkealmost meeting time!20:54
timburke#startmeeting swift21:00
opendevmeetMeeting started Wed Jan  4 21:00:02 2023 UTC and is due to finish in 60 minutes.  The chair is timburke. Information about MeetBot at
opendevmeetUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.21:00
opendevmeetThe meeting name has been set to 'swift'21:00
timburkewho's here for the swift meeting?21:00
timburkewhoa, it's a clayg!21:03
timburkewell, may as well get started21:03
timburkeonly one topic, really21:04
timburke#topic busted gate21:04
claygyeah, when the gate is busted there isn't really much else to talk about 🤷21:04
clayghow'd it go fighting with tox *today* !?  😁21:04
timburkeover the holidays, a new version of tox came out that caused a bunch of failures21:04
timburkei spent a bit of time making our tox.ini on master work with new tox, and things were good21:06
timburke...but then a few days went by, another tox was released, and new failures were seen21:06
timburkeso my current approach is to just pin tox<421:08
timburkethat looks most promising, and is currently working its way through the gate21:11
timburkeat some point in the future, we can consider trying tox>=4 again, but i'm in no rush21:11
timburkemeanwhile, i'll get some similar patches up against stable ('cause i'm currently getting a bunch of emails every night about how our stable branches have failing gates)21:12
timburkethat's all i've got21:12
timburke#topic open discussion21:12
timburkeanything else we should talk about this week?21:12
claygall good 👍21:16
timburkeall right, let's call it early21:18
timburkethanks for coming clayg! happy new year!21:18
opendevmeetMeeting ended Wed Jan  4 21:18:50 2023 UTC.  Information about MeetBot at . (v 0.1.4)21:18
opendevmeetMinutes (text):
opendevreviewMerged openstack/swift master: CI: pin tox at the project level
opendevreviewTim Burke proposed openstack/liberasurecode master: Replace Flat XOR link with an archived version
opendevreviewTim Burke proposed openstack/swift stable/yoga: CI: pin tox at the project level

Generated by 2.17.3 by Marius Gedminas - find it at!