21:01:00 <notmyname> #startmeeting swift 21:01:00 <openstack> Meeting started Wed Jun 6 21:01:00 2018 UTC and is due to finish in 60 minutes. The chair is notmyname. Information about MeetBot at http://wiki.debian.org/MeetBot. 21:01:01 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 21:01:03 <openstack> The meeting name has been set to 'swift' 21:01:05 <notmyname> who's here for the swift team meeting? 21:01:15 <m_kazuhiro> o/ 21:01:17 <alecuyer> o/ 21:01:18 <tdasilva> hi 21:01:19 <rledisez> o/ 21:01:20 <kota_> hello 21:01:33 <acoles> hi 21:01:33 <mattoliverau> o/ 21:01:48 <notmyname> welcome, all 21:01:59 <notmyname> I'm not sure if this meeting will be really short or really long 21:02:07 <notmyname> #link https://wiki.openstack.org/wiki/Meetings/Swift 21:02:15 <notmyname> there's nothing specific on the agenda for this week 21:02:19 <notmyname> so... 21:02:24 <notmyname> #topic open discussion 21:02:33 <notmyname> what's up? anything to bring up this week in the meeting? 21:03:02 <rledisez> LOSF, here it is: https://review.openstack.org/#/c/571763/ ;) 21:03:02 <kota_> lol, agenda is only open discussion 21:03:06 <notmyname> alecuyer: when can we see more LSOF code in gerrit? 21:03:09 <notmyname> oh nice! 21:03:26 <notmyname> that... is a really big patch ;-) 21:03:32 <alecuyer> it is here, Romain kicked me until I pushed it ;) I am not happy with everything in there.. but, 21:03:38 <tdasilva> passes probe tests ;) 21:03:49 <notmyname> tdasilva: lol 21:04:01 <alecuyer> it applies to master, and it does run 21:04:05 <rledisez> i guess something is wrong then ;) 21:04:15 <tdasilva> rledisez: lol 21:04:20 <alecuyer> the patch is not so big 21:04:31 <alecuyer> I mean, there are generated files there 21:04:45 <notmyname> it's only half as big as sharding. it's tiny! ;-) 21:04:48 <acoles> rledisez: alecuyer I started reviewing https://review.openstack.org/#/c/447129/10 today 21:05:02 <alecuyer> acoles: ok, thanks! 21:05:31 <rledisez> acoles: thx! it needs some more work, but an opinion on the way it does things is really welcomed 21:05:54 <alecuyer> notmyname: it touches much less existing files than before. And the kvfile.py is down from about 4000 lines (diskfile.py copy) to about a 1000 (inherits from classes in diskfile) 21:06:14 <acoles> rledisez: it seems heading the right way, I'll post some comments tomorrow I hope 21:07:19 <rledisez> acoles: if you see a better way to patch tests, i'll take it. for now i'm not really happy with the way I did it 21:07:32 <mattoliverau> Wow nice work, /me will definitely be a coffee before I starting tho 21:07:43 <mattoliverau> *need 21:07:54 <kota_> mattoliverau: +1, me too 21:09:47 <notmyname> rledisez: alecuyer: this is great to see. I know you have been working on the migration in your clusters, but you weren't planning on sharing that custom work 21:10:11 <notmyname> in general, how does this work in the sense of migrating to it (or away from it) 21:10:37 <rledisez> we only planned one way: to it ;) 21:10:40 <notmyname> heh 21:11:03 <notmyname> I've been thinking on the ordering of things, and LSOF vs getting rid of rsync is the big question. seems to me that with an rsync replacement, we may be able to more easily move to a different on-disk format 21:11:33 <rledisez> the migration is done in-place. sending new write to losf, but reading both from losf and filesystem. and a process scan the filesystem to write to losf 21:12:04 <rledisez> the downside of that migration process is that it disable the REPLICATE command, so no rebalance/reconstruction while a server is being converted 21:12:15 <rledisez> I let alecuyer talk more in details if needed 21:12:44 <alecuyer> well, what your said is correct ;) 21:12:47 <rledisez> alecuyer: do you think it would be possible to improve the in-place migration? notmyname: would you be interrested with that. 21:12:51 <rledisez> ? 21:13:33 <torgomatic_> I'm sort of surprised you don't just use replication to do it. Take a node out of the ring, reconfigure it for LOSF, put it back in, and wait... 21:14:53 <rledisez> torgomatic_: the issue is the time it would take. the concerned cluster basically does not work anymore, at least for anything related to replication. it is so slow that it timeout fro everytwhere (think 5, even 10 or 20 minutes for a REPLICATE request) 21:15:08 <notmyname> honestly, my personal concern is coordinating the work so that we don't do too much and once and not make progress on anything. we know the LSOF needs migration, but also that there's other planned work ("tsync" that may help). but LSOF is obviously further along 21:15:08 <alecuyer> Well it could be improved, probably REPLICATE could be supported but I don't know if it's a good general purpose tool 21:15:14 <alecuyer> for migration 21:16:02 <alecuyer> notmyname: yes. I would be happy to get feedback before I work further on this version, if we can agree on a way to do things 21:16:13 <alecuyer> (lots of work still, tests… etc) 21:16:31 <torgomatic_> I mean, in-place migration is definitely going to be better; I was just being lazy and considering ways to avoid work ;) 21:17:10 <torgomatic_> also, at some point I got the LOSF RPC server up and running, and I made a branch in vagrant-swift-all-in-one with the necessary dependencies, so that may be a good starting point for anyone else wanting to poke at it 21:17:13 <notmyname> alecuyer: that's why I'm glad you've pushed code before it's "perfect" so we can work on figuring it out together 21:17:14 <torgomatic_> #link https://github.com/swiftstack/vagrant-swift-all-in-one/tree/losf 21:17:28 <notmyname> torgomatic_: nice, thanks 21:18:24 <alecuyer> notmyname: yes, far from perfect which made it hard to push ;) but here it is and I'll work to make it better 21:18:25 <kota_> nice 21:18:44 <alecuyer> torgomatic_: there are some changes in the golang code in this version, but the way to build it has not changed 21:19:19 <alecuyer> and ill look at the vagrant :) 21:20:45 <rledisez> an other interesting question is "grpcio==1.3.3" (can't upgrade grpcio because of eventlet) 21:20:54 <notmyname> ah right 21:22:04 <torgomatic_> is there a library out there that only does the gRPC encoding and decoding, but leaves the IO up to the user? 21:22:04 <tdasilva> rledisez: do you know if the evenlet issue has been fixed in a recent version of grpcio? 21:22:11 <alecuyer> they have a patch to make grpc work with gevent https://github.com/grpc/grpc/issues/4629 but not eventlet 21:22:18 * torgomatic_ is not familiar with the gRPC ecosystem 21:23:00 <alecuyer> torgomatic_do you mean, use protobuf and something else for network communication ? 21:24:31 <torgomatic_> alecuyer: yes 21:24:45 <tdasilva> rledisez: nevermind, was under false impression... 21:25:33 <torgomatic_> like I said, though, I don't know much about this ecosystem, so that may be an absolutely terrible idea 21:26:10 <alecuyer> torgomatic_: we could do that, they're separate libraries. So it's not a terrible idea at all. I started with grpc at a time where it worked fine… then it broke after 1.3.1. 21:26:25 <alecuyer> On the other hand, I also wonder if we need eventlet on the object servers 21:26:36 <notmyname> alecuyer: eventually, we definitely want to remove that, too 21:26:47 <notmyname> again, just a matter of figuring out the right order to do stuff :-) 21:26:57 <alecuyer> right :) 21:28:12 <notmyname> is this a good summary? alecuyer has proposed the remainder of the LSOF code, but it's definitely a WIP. alecuyer will look at torgomatic_'s saio work to get it running. torgomatic_ will likely be looking at the patch, but others should too. especially to think about migrations 21:28:40 <notmyname> there's the higher level question of LSOF migrations and how that should be ordered against tsync work (ie remove rsync) 21:29:12 <alecuyer> I think it's a good summary. I would add, before thinking about migrations, that I would like to have input on the general design, if there are things you feel should be done differently 21:29:16 <kota_> sounds good to me. 21:29:28 <notmyname> alecuyer: right! 21:29:57 <alecuyer> I do have a few specific questions, but maybe I can put that in the review and we discuss this outside the meeting 21:30:15 <notmyname> yes 21:30:30 <notmyname> does anyone else have questions for alecuyer and rledisez about LSOF right now? 21:30:32 <rledisez> notmyname: what is the status of tsync? is there something in progress (eg: https://github.com/swiftstack/tsync). nothing started yet? how big do you think it will be? (tsync is quite small, but not complete) 21:31:40 <notmyname> rledisez: there's lots of good ideas for tsync, but nothing really done right now 21:31:42 <acoles> alecuyer: review discussion can be hard to follow - gerrit is not great for that. another idea might be an etherpad. also not great, but discussion persists there beyond each patchset. 21:31:43 <mattoliverau> Nice work guys, and thanks alecuyer for pushing it up as it is! 21:32:32 <alecuyer> acoles: right, I will make an etherpad page 21:32:48 <rledisez> notmyname: ack 21:33:13 <notmyname> rledisez: see also https://etherpad.openstack.org/p/swift-rebalance from the ideas wiki 21:33:33 <notmyname> ideas wiki: https://wiki.openstack.org/wiki/Swift/ideas 21:34:09 <rledisez> yeah, I remember that, nicely analyzed :) 21:34:56 <tdasilva> acoles, alecuyer we could also have a video meetup if folks like that idea 21:35:02 <alecuyer> finally and I'll stop tallking about this unless you have questions :) there's a readme to get it running https://review.openstack.org/#/c/571763/2/README.LOSF 21:35:23 <notmyname> tdasilva: I feel like we should spend some time looking at the code first. but it's definitely a good idea 21:36:13 <notmyname> is there anything else to bring up in the meeting today? 21:36:20 <notmyname> or shall we call it done? 21:36:22 <tdasilva> notmyname: yep, agree. 21:36:24 <kota_> wait 21:36:30 <notmyname> kota_: what's up? 21:36:30 <kota_> just a small note 21:36:32 <kota_> https://review.openstack.org/#/c/564700/ 21:36:38 <kota_> #link https://review.openstack.org/#/c/564700/ 21:37:03 <kota_> i recently has been working on setting up s3api compatibility check with timburke 21:37:23 <kota_> and it's close to get successful with any gates. 21:37:38 <kota_> and I like to add it as the gate job (even it's non-voting) 21:37:56 <kota_> if you have any opinions to that, please let me know at the patch. 21:38:07 <kota_> it will creates like... 21:38:22 <kota_> http://logs.openstack.org/00/564700/27/check/swift-tox-func-s3api-ceph-s3tests-tempauth/2742951/output/ceph-s3-summary.log 21:39:25 <notmyname> I like it. thanks 21:39:34 <kota_> i think it's worth to know which s3 apis are supported at the commit. 21:39:48 <mattoliverau> +1 nice work 21:39:53 <kota_> just a notification from me :) 21:40:16 <notmyname> kota_: oh, I moved https://review.openstack.org/#/c/571561/ to swift from swift3. so it puts the s3 compat info in the docs 21:40:37 <kota_> yup 21:40:38 <timburke> ugh... that reminds me: i've got a whole bunch of patches i need to move... 21:40:45 <notmyname> speaking of patches (and our review change last week)... 21:40:47 <kota_> notmyname: is it ready to review? 21:40:59 <notmyname> does anyone have any patches that need extra eyes (ie more than 1 +2)? 21:41:01 <notmyname> kota_: yes 21:41:03 <kota_> notmyname: i think you added workflow -1 at the swift3 for some reasons. 21:41:19 <notmyname> kota_: correct. in part because of the planned move. in part because I was being silly 21:41:36 <notmyname> kota_: that is, I had reasons that I've changed my mind on since then 21:41:50 <kota_> got it. I'll take a look. 21:42:24 <notmyname> thanks 21:42:45 <notmyname> I'm not seeing anyone else raise other issues to discuss this week 21:42:50 <m_kazuhiro> I think patch 517389 is ready for review. 21:43:08 <notmyname> ah, sorry. I didnt' move patchbot in here 21:43:28 <m_kazuhiro> https://review.openstack.org/#/c/517389/ 21:43:36 <m_kazuhiro> Oops, merge conflict... 21:44:03 <m_kazuhiro> Sorry I will fix it soon. But I think the patch can be reviewed. 21:44:13 <notmyname> thanks 21:44:45 <notmyname> I think we're done, then 21:44:59 <notmyname> thanks everyone for coming. and thanks for your work on swift 21:45:07 <notmyname> #endmeeting