21:00:25 <timburke> #startmeeting swift
21:00:25 <openstack> Meeting started Wed Oct 21 21:00:25 2020 UTC and is due to finish in 60 minutes.  The chair is timburke. Information about MeetBot at http://wiki.debian.org/MeetBot.
21:00:26 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
21:00:28 <openstack> The meeting name has been set to 'swift'
21:00:34 <timburke> who's here for the swift meeting?
21:00:43 <acoles> o/
21:00:45 <kota_> hi
21:00:51 <seongsoocho> o/
21:00:59 <clayg> 🥳
21:01:00 <rledisez> hi o/
21:01:55 <timburke> first order of business
21:02:00 <timburke> #topic acoles is back!
21:02:08 <timburke> 🎉
21:02:26 <acoles> hi everyone, it's nice to be here again
21:02:27 <kota_> amazing
21:03:02 <timburke> if there are no objections, i'd like to give him commit rights again
21:03:51 <rledisez> timburke: +2
21:04:28 <clayg> i didn't realize we'd retired him
21:05:48 <timburke> excellent -- 'cause i jumped the gun a little and already did it :-)
21:06:17 <timburke> i figure, we can use all the help we can get -- thanks for coming back acoles, and so glad to have you back!
21:06:28 <timburke> #topic ptg
21:06:41 <timburke> and he has good timing, as we've got the PTG next week!
21:06:55 <kota_> yey
21:07:02 <timburke> topics continue to gathre on the etherpad
21:07:05 <timburke> #link https://etherpad.opendev.org/p/swift-ptg-wallaby
21:07:16 <seongsoocho> \o/
21:07:43 <timburke> ...and i should be sure to set aside sometime for some ops feedback and community feedback
21:09:07 <clayg> excellent
21:09:09 <timburke> i don't really have much of a plan for next week, but as long as we're all together, thinking and chatting about swift, i know good things will come outof it :-)
21:10:18 <timburke> i'll try to order items by level of interest again, probably on friday -- so put yourself down for topics that you care about! it'll give us a good way to prioritize conversations
21:11:26 <timburke> and since we'll have the PTG together, i figure no irc meeting is necessary next week
21:11:47 <acoles> timburke: how does it work - is all discussion over irc?
21:12:05 <mattoliverau> nah all zoom or whatever
21:12:08 <timburke> video chats over zoom
21:12:18 <acoles> ah ok
21:12:24 <clayg> it was not too bad!  not as good as whiteboard 😞
21:13:07 <timburke> though we can also use jitsi via https://meetpad.opendev.org/swift-ptg-wallaby
21:13:21 <mattoliverau> We probably still need to figure out how to white board things
21:13:49 <seongsoocho> clayg:   maybe you can use jamboard as a white board https://jamboard.google.com/
21:14:04 <clayg> worth a shot!!!  💪
21:14:06 <timburke> fingers crossed that in another six months, things have quieted down enough that we can actually all be in the same physical room again
21:14:09 <mattoliverau> Nice
21:14:23 <clayg> timburke: that would be GREAT - miss everyone 😢
21:15:21 <timburke> i've got just one more "announcement"-type item
21:15:32 <timburke> #topic review.openstack.org security incident
21:15:44 <timburke> you all may have noticed that gerrit was down yesterday
21:15:57 * acoles not my fault
21:16:05 <timburke> there's a nice write-up of what happened at
21:16:07 <timburke> #link http://lists.opendev.org/pipermail/service-announce/2020-October/000011.html
21:16:15 <mattoliverau> Lol, acoles comes back and then.. hmm
21:16:29 <kota_> lol
21:17:18 <clayg> it wasn't so bad - just a bummer day
21:17:24 <timburke> i want to applaud the infra team's transparency and quick response; considering the scope of what needed to be checked, losing a day seems remarkably fast
21:17:36 <mattoliverau> +100
21:18:28 <timburke> one of the action items was that "We should audit all changes for projects since 2020-10-01."
21:18:45 <timburke> they already compiled the commits for us at
21:18:48 <timburke> #link https://static.opendev.org/project/opendev.org/gerrit-diffs/
21:19:22 <timburke> and i've already confirmed swift, python-swiftclient, and liberasurecode commits look correct and expected
21:19:54 <mattoliverau> Nice work, thanks timburke
21:20:07 <timburke> (pyeclib and swift-bench didn't even have enough activity to show up)
21:20:25 <timburke> are there other projects that i'm forgetting about that i should check?
21:22:45 <timburke> if anyone has other projects that they have a vested interest in, that's where you can go for first-hand data; expect most temas to put out statements on the mailing list as well (i certainly intend to, probably shortly after this meeting)
21:23:58 <timburke> all right, on to updates!
21:24:00 <timburke> #topic updates
21:24:16 <timburke> we've got two patches that i think mostly just need reviews
21:24:27 <timburke> #link https://review.opendev.org/#/c/706653/
21:24:27 <patchbot> patch 706653 - swift - Let developers/operators add watchers to object au... - 37 patch sets
21:24:34 <timburke> #link https://review.opendev.org/#/c/754242/
21:24:34 <patchbot> patch 754242 - swift - Fix a race condition in case of cross-replication - 6 patch sets
21:24:54 <timburke> i'd meant to look at both and wound up looking at neither :-(
21:25:55 <timburke> does anyone have anything else they'd like to discuss about them? or is it mostly a matter of us finding time to review them?
21:26:20 <timburke> i know zaitcev put a +2 on the replication patch -- thanks!
21:26:47 <zaitcev> I'm worried that I missed some kind of deadlock there, but I cannot find any. I looked at all instances where lock_path is invoked for various paths.
21:27:00 <zaitcev> So, what the heck, seems okay. +2
21:27:31 <zaitcev> I owed Romain for looking at Dark Data
21:27:34 <acoles> did patch 706653 have a predecessor - it rings a bell, and I see torgomatic is an author
21:27:34 <patchbot> https://review.opendev.org/#/c/706653/ - swift - Let developers/operators add watchers to object au... - 37 patch sets
21:27:45 <zaitcev> Yes.
21:28:02 <zaitcev> patch 212824
21:28:03 <patchbot> https://review.opendev.org/#/c/212824/ - swift - Let developers/operators add watchers to object audit - 21 patch sets
21:28:15 <zaitcev> See, this is "alternative"
21:28:38 <zaitcev> acoles: I worked on Sam
21:29:00 <acoles> @zaitcev thanks
21:29:09 <zaitcev> 's patch first, but ended with a trouble trying to debug things, so I threw my toys down and forked it.
21:30:12 <timburke> the main change (zaitcev, correct me if i'm wrong) was that sam's patch had a much more adversarial attitude towards the watchers, and so enforced process isolation
21:30:29 <zaitcev> I created a logger for Sam's patch that actually worked, before the fork.
21:30:41 <timburke> where this patch runs it all in the same process -- and if a watcher breaks the audit, you need to fix itor drop it
21:31:57 <zaitcev> To an extent, we still isolate, as much as Python allows. But a plugin can run out of file descriptors, for example.
21:32:56 <acoles> ok, so IIRC Sam's used a queue so watchers got notifications on 'best-effort' basis. Is this more guaranteed to notify watchers?
21:33:40 <zaitcev> This is just a method call, no queue.
21:35:02 <timburke> ...or call os._exit(0), or os.kill(os.getpid(), signal.SIGKILL), or time.sleep(1<<30)...
21:37:27 <zaitcev> So, anyway.
21:38:12 <zaitcev> If Alistair finds a moment to look, it would be good. But for the extra lock for replicators too.
21:39:08 <timburke> sounds like we mainly need to get some reviews on them. i'll get the replicator guy squared away -- i've probably been overthinking my dev setup, and promise to leave *some* sort of review by the end of the week
21:40:11 <acoles> I'll try but can't promise
21:40:16 <timburke> zaitcev, do you know if david is planning to be at the ptg? i remember seeing audit watchers on the agenda
21:40:38 <clayg> noice
21:40:38 <timburke> do we think that'll mostly be a session to discuss and land it?
21:41:26 <zaitcev> timburke: he was interested in it, but I don't know if he can do it.
21:42:17 <timburke> no worries; just figured i'd check
21:42:37 <timburke> all right
21:42:42 <timburke> #topic open discussion
21:42:51 <timburke> what else should we bring up this week?
21:43:05 <acoles> is there any kind of sign up for ptg or just show up?
21:43:22 <timburke> acoles, https://october2020ptg.eventbrite.com/
21:43:35 <acoles> thanks
21:43:37 <timburke> though i'm sure you could also just show up and we'd make itwork ;-)
21:43:49 <acoles> i want my t shirt
21:44:07 <kota_> lol
21:44:12 <timburke> ...who wants to tell him?
21:44:25 <zaitcev> lol
21:44:49 <acoles> awww
21:45:09 <timburke> 🤔 i *should* talk to notmyname, investigate doing a small run of shirts...
21:45:38 <clayg> 😬 cutting it close now
21:45:39 <timburke> also, i wonderif we've still got some swiftstack shirts stashed somewhere -- collector's items, now!
21:45:44 <mattoliverau> Yeah, even if it's in red bubble or something so we pay for our own shirts, just need the design.
21:45:45 <clayg> hahah
21:45:58 <timburke> clayg, oh, almost certainly for a future event
21:46:31 <clayg> oic
21:47:16 <kota_> zoom list windows showing guys who wear same shirts should look great on the virtual event, so +1
21:47:30 <acoles> i think I have a pile of those hpe iron on badges somewhere...
21:48:20 <zaitcev> I have my yellow-orange "Swift Justice" shirt.
21:48:24 <kota_> Justice!
21:48:43 <timburke> i wore mine for acoles's first day back
21:49:02 <timburke> oh! one other patch i'd like to call attention to -- https://review.opendev.org/#/c/758636/
21:49:03 <patchbot> patch 758636 - swift - Add option to REPLICATE to just invalidate hashes - 2 patch sets
21:50:29 <timburke> i've got this memory of talking about doing something like that as a *pre*-sync check, to try to feel out when there might be a newly-assigned primary
21:53:45 <zaitcev> Curious. Does that help performance any, or what does it do?
21:53:49 <timburke> still needs tests (and some fixups, looks like), but i wanted to have people start thinking about how we could make replication smarter if we had a cheap way to check remote suffix counts
21:55:57 <timburke> so the idea specific to that patch is that when you're moving data as part of a rebalance (and so, suffixes are probably fairly dense, say 3k or so), the sender doesn't really need to wait for the rehash to clean up data; it just needs to make sure everybody got invalidated
21:57:54 <timburke> but *in the future* i've got this thought like we could get rid of handoffs-first and instead have partition sync jobs in a priority queue of sorts, weighted by some combination of how many suffixes are in the partition (if it's a handoff) and when it was last successfully synced
21:59:09 <clayg> timburke: nice!
21:59:40 <clayg> skipping the re-hash IO on the post-rsync invalidate step seems like a quite reasonable enhancement to me
22:00:57 <timburke> all right, we're about at time. oughta let kota_ and mattoliverau start their day, and acoles end his :-)
22:01:08 <timburke> thank you all for coming, and thank you for working on swift!
22:01:14 <mattoliverau> Kk, so great to have you back acoles
22:01:16 <timburke> acoles, again, greatto have you back!
22:01:22 <timburke> #endmeeting