21:00:05 <timburke> #startmeeting swift
21:00:06 <openstack> Meeting started Wed Apr  3 21:00:05 2019 UTC and is due to finish in 60 minutes.  The chair is timburke. Information about MeetBot at http://wiki.debian.org/MeetBot.
21:00:07 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
21:00:09 <openstack> The meeting name has been set to 'swift'
21:00:20 <timburke> who's here for the swift meeting?
21:00:34 <alecuyer> Hello o/
21:00:35 <mattoliverau> o/
21:00:35 <kota_> hi
21:00:40 <rledisez> hi o/
21:01:39 <timburke> not too much on the agenda for this week
21:02:06 <timburke> #link https://wiki.openstack.org/wiki/Meetings/Swift
21:02:18 <timburke> first up, pyeclib
21:02:26 <timburke> #topic pyeclib release
21:03:34 <timburke> we had one! there were a few hiccups since it includes more than just python code, but with a little help from fungi et al. we have a 1.6.0 pacakge on pypi and tarballs.openstack.org!
21:04:03 <fungi> i did almost nothing
21:04:06 <fungi> you fixed the job!
21:04:21 <timburke> you re-enqueued it; i can't do that :-)
21:04:38 * fungi is a button-pushing monkey
21:05:06 <mattoliverau> lol
21:05:07 <timburke> that's all i really wanted to call out about that, unless people have questions
21:05:11 <mattoliverau> nice work everyone
21:05:44 <timburke> so, on to updates
21:05:55 <timburke> #topic py3
21:06:16 <timburke> zaitcev, mattoliverau, much to report?
21:07:19 <zaitcev> timburke: busy with customer cases, so no progress on my side. Some of what you've done set slo back a bit, it needs something like 10 tests fixed now (was just 1 remaining). But it's not a bad thing if the code is more correct.
21:07:26 <mattoliverau> not too much from me this week, I've been pulled into some deployment bugs.. but regarding swift (which is a nice change) so putting in the extra effort to promote swift from the inside ;)
21:07:56 <mattoliverau> I plan get back to the staticweb patch today and I'll try and review some more :)
21:08:46 <tdasilva> o/
21:08:52 <tdasilva> sorry i'm late
21:08:54 <mattoliverau> tdasilva: o/
21:09:01 <timburke> sounds good. on my end, i took a stab at porting s3token (thanks for reviewing it, zaitcev!) and proposed a few changes to move things like docs builds etc to use py3
21:09:47 <timburke> #topic losf
21:10:51 <timburke> rledisez, alecuyer, kota_: how's it going? i haven't seen many patches this week, but i suppose that doesn't always reflect whether progress is being made :-)
21:11:27 <alecuyer> I had to work on other topics at OVH so not as much time as I would like, but today I finished making the RPC functions work over HTTP
21:11:29 <kota_> sorry, not so much from myside, because of beginning of April that is the starting month for Japanese company. that makes me to be tied to some common internal work.
21:11:37 <alecuyer> (the streaming ones were missing)
21:12:06 <kota_> alecuyer: nice, RPC work.
21:12:20 <alecuyer> So I'm testing this and will push it shortly. And I also need to try to get a gate job to compile the golang code and setup a configuration to use LOSF (not diskfile), so the functest can actually exercise LOSF
21:13:05 <timburke> is there anything the rest of us can/should be doing to help out? should someone help figure out the gate job fun?
21:13:39 <alecuyer> That would be nice to help get a job started, then I think I should try to make the steps below that work
21:14:04 <kota_> I'm planning that, sorry it didn't get the progress yet.
21:14:22 <timburke> oh perfect, i was just about to ask for volunteers
21:14:26 <timburke> thanks kota_!
21:14:49 <kota_> my pleasure
21:15:02 <timburke> #topic Denver
21:15:22 <timburke> so we're less than a month out from the Summit
21:15:27 <mattoliverau> \o/
21:15:35 <kota_> wow
21:16:12 <timburke> i still need to make an etherpad for topics for us
21:16:20 <timburke> but who all's coming and when?
21:16:44 <mattoliverau> We can put all that on the etherpad for reference :)
21:16:48 <rledisez> o/ I land on sunday, and leave the next synday
21:16:53 <mattoliverau> but I'll be arriving on the Friday
21:17:18 <timburke> (i feel like i've asked that before, but (1) i forgot the answers and (2) plans may have settled a bit more since then)
21:17:18 <mattoliverau> I'll help at OUI and then leave the Saturday night after the PTG
21:17:27 <alecuyer> I'll be there from sunday to sunday
21:17:52 <kota_> from Sunday to the next Sunday.
21:18:19 <kota_> I heard from m_kazuhiro, he will attend only PTG.
21:18:19 <timburke> i'll be there wed to sun
21:18:27 <m_kazuhiro> I'll be there May 1-5 (only PTG)
21:18:38 <tdasilva> from Tue to Sun
21:18:44 <kota_> oh, m?
21:18:58 <kota_> m_kazuhiro: o/
21:19:05 <kota_> typo :/
21:19:06 <mattoliverau> timburke: want me to go create an etherpad and paste the link in the swift channel? Then people can update that with topics and their availability?
21:19:10 <m_kazuhiro> o/ sorry I'm late.
21:19:29 <timburke> sure, thanks mattoliverau!
21:20:25 <timburke> oh, and i need to respond to one of the mailing list emails and sign us up for a team photo...
21:21:21 <timburke> i think that's all i had for Denver -- i look forward to seeing you all there!
21:21:34 <kota_> b
21:21:34 <timburke> #topic open discussion
21:22:08 <timburke> anyone have anything to bring up?
21:23:12 <zaitcev> Naah. Well, auditiors hang, and I'm looking into it.
21:23:19 <zaitcev> No, updaters hang
21:23:36 <timburke> zaitcev, what version? i feel like we've seen that recently...
21:26:22 <rledisez> zaitcev: we had issues like that, but this commit fixed it: https://github.com/openstack/swift/commit/dc8da5bb194d0a544fb8065aa9a4c7484f605715
21:26:30 <rledisez> (shipped with 2.16)
21:26:36 <timburke> yeah, that one :-) https://bugs.launchpad.net/swift/+bug/1722951
21:26:37 <openstack> Launchpad bug 1722951 in Ubuntu Cloud Archive pike "Container updater may be stuck and not make progress" [High,Triaged]
21:27:06 <timburke> and then there was a follow-up bug (https://bugs.launchpad.net/swift/+bug/1804627) that got fixed in https://github.com/openstack/swift/commit/4809884
21:27:08 <openstack> Launchpad bug 1804627 in OpenStack Object Storage (swift) "Selection of select eventlet.hub instead of poll in wsgi servers" [Undecided,Fix released]
21:27:19 <zaitcev> rledisez: thanks... I thought we crossed internal pipes across os.fork and that caused hangs, but maybe not.
21:27:53 <timburke> (at least, i *think* those were related...)
21:28:09 <zaitcev> well, updater is not wsgi, so probably not
21:28:33 <zaitcev> Anyway, we can talk this outside of the meeting. Thanks for the hints, timburke, rledisez!
21:28:46 <timburke> zaitcev, oh yeah, https://github.com/openstack/swift/commit/6d16079 / https://bugs.launchpad.net/swift/+bug/1710328
21:28:47 <openstack> Launchpad bug 1710328 in OpenStack Object Storage (swift) "object server deadlocks when a worker thread logs something" [High,Fix released] - Assigned to Samuel Merritt (torgomatic)
21:29:24 <kota_> assigned to torgomatic :/
21:30:10 <timburke> kota_, but "Fix released" :-D
21:30:17 <timburke> so it's been a couple months since the quadiron patches were proposed -- anyone have review bandwidth to look at that? chain ends at https://review.openstack.org/#/c/635605/
21:30:18 <kota_> oh yeah
21:30:53 <timburke> i mostly just feel bad that the author asked, "hey, should i propose this?" and then we never got back to it...
21:33:26 <timburke> all right, i'll try to pick that up
21:33:52 <zaitcev> Me too, once I find an opening from updaters and py3
21:34:22 <timburke> thanks zaitcev! there's always so much to fit in
21:35:10 <timburke> i know i talked about async bulk deletes a week or two ago -- i got some fun results for https://review.openstack.org/#/c/648263/
21:36:51 <timburke> had an S3 client that was deleting ~1TB of data. originally (and in part because the client wasn't using markers when listing data to delete), it took ~48hrs to complete
21:37:16 <timburke> on a modest 3 node / 45 disk cluster
21:37:22 <kota_> 48hrs!?
21:37:52 <timburke> they'd list 1k objects, pipe them to a delete-multiple request, then list again
21:37:54 <alecuyer> 1TB of small objects? :-)
21:37:58 <timburke> but with no marker :-(
21:38:11 <zaitcev> Well it least it was a multi-delete :-)
21:38:27 <timburke> so once async pendings started piling up, they'd get back *the same objects*
21:38:48 <timburke> and issue another multi-delete. putting more load on the container.
21:39:04 <timburke> anyway, with the patch, we got that down to ~2hrs
21:39:43 <timburke> so, definitely promising. i've got this feeling like it's got a list of caveats as long as my arm though
21:40:54 <timburke> that's all i've got. last call for anything else to bring up
21:42:03 <mattoliverau> #link https://etherpad.openstack.org/p/swift-ptg-train
21:42:12 <timburke> thanks mattoliverau!
21:43:12 <timburke> all right, i'm calling it
21:43:28 <timburke> thank you all for coming, and thank you for working on swift!
21:43:30 <timburke> #endmeeting