| opendevreview | Tim Burke proposed openstack/swift master: Improve some logging when aborting workers https://review.opendev.org/c/openstack/swift/+/823439 | 00:12 |
|---|---|---|
| kota | a happy new year, it's working start day of 2022 to me. | 01:07 |
| opendevreview | Tim Burke proposed openstack/swift master: reconstructor: Abort just the changed policies https://review.opendev.org/c/openstack/swift/+/823473 | 04:42 |
| acoles | kota: happy new year | 09:34 |
| ozzzo_work | timburke__: How does the 2013 Swift upgrade procedure apply to kolla-ansible? When I go into the containers I can see the swift services running, for example in the swift_account_server container, I see swift-account-server processes running, but the swift-init client doesn't see them: | 17:52 |
| ozzzo_work | (swift-account-server)[swift@sde6-swift1 /]$ swift-init account status No account-server running | 17:52 |
| timburke__ | ozzzo_work, i'm not real familiar with kolla, but i'd expect you just need to stop the containers that are running the old version and start containers running the new version | 18:06 |
| timburke__ | again, it's the proxies where things will get hairiest -- i'm not sure whether kolla would include the healthcheck middleware in its pipelines or not, and even if it *does*, there's the question of whether your load balancer is configured to check it | 18:15 |
| ozzzo_work | ty timburke__ : I'll try asking in the kolla channel about that | 18:34 |
| kota | good morning | 20:57 |
| timburke__ | o/ | 20:57 |
| kota | timburke__: o/ | 20:58 |
| *** timburke__ is now known as timburke | 20:58 | |
| timburke | #startmeeting swift | 21:00 |
| opendevmeet | Meeting started Wed Jan 5 21:00:16 2022 UTC and is due to finish in 60 minutes. The chair is timburke. Information about MeetBot at http://wiki.debian.org/MeetBot. | 21:00 |
| opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 21:00 |
| opendevmeet | The meeting name has been set to 'swift' | 21:00 |
| timburke | who's here for the swift meeting? | 21:00 |
| kota | hi | 21:00 |
| timburke | if memory serves, mattoliver is out this week | 21:01 |
| timburke | clayg and acoles may or may not be around | 21:01 |
| kota | oic | 21:02 |
| clayg | Thanks for the ping! | 21:02 |
| timburke | maybe it'll work with zaitcev too!? :P | 21:02 |
| timburke | any way, the agenda's at https://wiki.openstack.org/wiki/Meetings/Swift | 21:02 |
| timburke | #topic object updater fairness | 21:03 |
| timburke | we ran with a bunch of updater improvements over the holidays, and they seem to be working well! | 21:03 |
| zaitcev | okay, good. Although your cloud is private, isn't it? So the holiday is actually not the peak load for you guys, right? | 21:04 |
| timburke | true -- i'd say we had a lighter-than-usual load over the holiday | 21:05 |
| timburke | but still, clearing a billion asyncs is impressive! there was another brief spike of 200-300M asyncs, but my understanding was that we could keep our previous tuning just fine and they cleared within a day or two | 21:06 |
| timburke | the skips in https://review.opendev.org/c/openstack/swift/+/820962 seemed to do a good job of protecting the container servers | 21:07 |
| timburke | probably still a good idea to address the deterministic-hashing issue that clayg spotted before we land it, though | 21:08 |
| timburke | #topic memcache refresh | 21:09 |
| clayg | Can someone do that for me? I’m super duper lazy. | 21:09 |
| timburke | clayg, i'll try to get to it | 21:09 |
| clayg | Hahaha I didn’t think that would work! | 21:10 |
| timburke | stuff falling out of memcache is still hurting us, so i'm interested in getting something ship-able | 21:11 |
| timburke | i've got two main efforts: the one for shard ranges is simpler/cheaper: https://review.opendev.org/c/openstack/swift/+/736802 | 21:11 |
| timburke | the one for container info isn't *so* bad: https://review.opendev.org/c/openstack/swift/+/821921 | 21:12 |
| timburke | but it's got a dependency on a refactor: https://review.opendev.org/c/openstack/swift/+/821920 | 21:12 |
| zaitcev | Just think that maybe now we could sustain the load from Gnocchi. | 21:13 |
| zaitcev | Except that Gnocchi is fortunately dead because it was killing Ceph just as well as Swift. | 21:13 |
| zaitcev | (sorry that was about the updater; will try to keep up) | 21:14 |
| timburke | no worries :) | 21:14 |
| clayg | timburke: those are great! I’ll dig in more. Can’t wait to see those new stats in prod. | 21:16 |
| timburke | zaitcev, fwiw, you'll definitely want sharded containers ahead of worrying too much about those updater improvements -- and if we can get the set of updating shard ranges to hang around in memcache, the updater improvements may become moot | 21:16 |
| timburke | the get_container_info refactor has an interesting side-effect: middleware can no longer interfere with the info cached and returned. i don't think it will impact anything, but seems worth calling out | 21:18 |
| timburke | (previously, a left-in-the-pipeline middleware could make a get_container_info call with its local "app" and middleware to the right of it could impact results. with the patch, everybody gets a reference to the actually proxy-server app and uses *that*) | 21:20 |
| clayg | Moddleware can manipulate memcaxge directly. It’s a “gotcha” at worst. | 21:22 |
| timburke | #topic tempurl signature logging | 21:23 |
| timburke | #link https://bugs.launchpad.net/swift/+bug/1685798 | 21:23 |
| timburke | mattoliver took a stab at an alternate approach in https://review.opendev.org/c/openstack/swift/+/822585 | 21:23 |
| clayg | Is Matt driving this? | 21:24 |
| clayg | I’d love to see it closed. | 21:24 |
| timburke | where we'd stick a new key in the request environment and middlewares could update that with whatever headers/params they want to obscure | 21:25 |
| timburke | i got worried that you could bounce out (due to ratelimit, say) before we get to tempurl, and so the env wouldn't get updated | 21:26 |
| timburke | (and it also bugged me that we had to make the same update *for every request*) | 21:27 |
| timburke | so i tried an approach similar to register_swift_info in https://review.opendev.org/c/openstack/swift/+/823432 | 21:27 |
| clayg | Let’s wait and see what Matt thinks then. It looks like mostly code refactoring/quality of life - not correctness. | 21:29 |
| clayg | Should we do the obvious fix and iterate on the technical investment? I think Matt pointed out correctly: we’ll want this the next time we have to obscure something. | 21:29 |
| timburke | sounds good. on the whole, i think i like the approach more than the original patch at https://review.opendev.org/c/openstack/swift/+/817476 -- it became trivial to fix up s3api | 21:30 |
| timburke | that's everything i've got | 21:31 |
| clayg | OIC, we’ll https://review.opendev.org/c/openstack/swift/+/823432 looks pretty thin. | 21:31 |
| timburke | #topic open discussion | 21:31 |
| timburke | what else should we talk about this week? | 21:31 |
| clayg | Oh, but that just builds on Matt’s. You two crack me up. | 21:32 |
| timburke | clayg, yeah, both matt's patch and mine need tests for the new middleware api | 21:32 |
| clayg | So… we’ll keep waiting on closing the CVE 🤣 | 21:33 |
| timburke | it's been 4 years, what's another? :P | 21:34 |
| timburke | all right, i think i'll call it then | 21:36 |
| timburke | thank you all for coming, and thank you for working on swift! | 21:36 |
| timburke | #endmeeting | 21:36 |
| opendevmeet | Meeting ended Wed Jan 5 21:36:56 2022 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 21:36 |
| opendevmeet | Minutes: https://meetings.opendev.org/meetings/swift/2022/swift.2022-01-05-21.00.html | 21:36 |
| opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/swift/2022/swift.2022-01-05-21.00.txt | 21:36 |
| opendevmeet | Log: https://meetings.opendev.org/meetings/swift/2022/swift.2022-01-05-21.00.log.html | 21:36 |
| clayg | 👍 | 21:37 |
| opendevreview | Tim Burke proposed openstack/swift master: reconstructor: Abort just the changed policies https://review.opendev.org/c/openstack/swift/+/823473 | 23:55 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!