21:00:03 #startmeeting swift 21:00:04 Meeting started Wed Feb 3 21:00:03 2021 UTC and is due to finish in 60 minutes. The chair is timburke_. Information about MeetBot at http://wiki.debian.org/MeetBot. 21:00:05 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 21:00:07 The meeting name has been set to 'swift' 21:00:14 who's here for the swift meeting? 21:00:19 o/ 21:00:23 o/ 21:00:37 o/ 21:00:41 o/ 21:01:04 o/ 21:01:26 o/ 21:01:37 as usual, the agenda's at 21:01:39 #link https://wiki.openstack.org/wiki/Meetings/Swift 21:01:51 first up 21:01:59 #topic vagrant swift all in one 21:02:10 #link https://github.com/swiftstack/vagrant-swift-all-in-one 21:02:54 i know clayg and i have been pretty heavy users of this tooling to set up our dev environments 21:03:34 ๐Ÿด IT WHILE IT'S ๐Ÿ”ฅ 21:03:45 but in the near future, it's likely to go away (as part of some long-standing cleanup necessary to get rid of the old swiftstack github account) 21:03:47 me and alecuyer (and some other colleagues) used it a lot too. very useful 21:04:43 timburke_: update as of today - there's a non-zero chance it will just end up living at github.com/nvidia/vagrant-swift-all-in-one ๐Ÿคทโ€โ™‚๏ธ 21:04:54 everyone here that uses it should still fork it now just in case 21:05:39 i do wonder if it'd be good for us to have something like it moved in-tree... 21:06:03 virtualbox yes, vagrant maybe, chef NO (like HELL NO) 21:06:03 could probably re-use a lot of the ansible playbooks we've already got for setting up probe tests 21:06:11 timburke_: SOLD 21:06:44 it's so handy that i'm wondering if there is swift devs that does not use it :D 21:08:10 I really love this tool :-) 21:08:20 rledisez: apparently mattoliverau has "better things" that he keeps to himself 21:08:21 it's the sort of thing i'm always a bit torn about -- it's way easier to stand up an environment, but there's definite value in having a range of setups that we each use. vsaio's definitely got some opinions baked in 21:08:41 lol, I don't know about better :P 21:09:31 I just have a dodgy bash script I wrote back when I needed to deploy a bunch on rackspace cloud that sets SAIO the way I want it (and am now used too). But it doesn't do all the things vsaio does. (like s3api). 21:09:58 fwiw, i remember notmyname linking https://gist.github.com/notmyname/40b8131963346676dd18817aeb5ef799 a while back if anyone wanted to go virtualbox with no vagrant ;-) 21:10:13 I always follow our in-tree manual to set up SAIO. 21:10:42 zaitcev: you are a hero ๐Ÿ˜ 21:11:17 https://knowyourmeme.com/memes/no-take-only-throw 21:11:18 ๐Ÿค” i wonder if we could generate some bits of the manual based on in-tree playbooks.... 21:11:47 "IT dog, how do you automate?" "No automate" "Only type" 21:12:01 timburke_: the super weird thing about vsaio is the configuration - like it's opinionated, but supports *some* options (which in a few cases are like... flip this ONE config option to false ๐Ÿคจ) 21:12:29 py3? yes/no 21:13:06 right, then there's other options that are super useful - maybe when we port to in-tree ansible we can square that up to something more sensible 21:13:19 (i *still* wish i had good tooling for working with mixed py2/py3 development) 21:13:29 it'd be nice to have a change that requires some crazy config (servers-per-port) and you can include the saio stuff for reviewers to check it out! 21:15:00 my dodgy one is: https://github.com/matthewoliver/simple_saio but ignore the ubuntu in the readme. I'm not sure what it really supports. I mainly using centos SAIOs atm and Opensuse probably works, well it did a few months ago. 21:15:00 IMHO a common dev and CI ansible setup would be great 21:15:15 one thing that always bothered me is that it seems there is in the code stuff that are specific so that ir works in SAIO. I'm wondering if the one server/vm is still relevant in a world where docker is everywhere. I can imagine creating a real cluster just with some docker-compose file 21:15:19 but yeah, vsaio + ansible would be nicer. 21:15:31 I have a some chef experience, and I didn't enjoy it :P 21:15:49 anyway, i guess we covered what needed to be said. namely, vsaio is a repo that (might) go away, so if you want to keep using it, it's probably a good idea to fork it sooner rather than later 21:16:16 rledisez: i think that's quite reasonable - the trick is porting probetests ๐Ÿค” 21:16:42 i think it was also (vaguely) what notmyname was trying to do with runway 21:17:09 s/might/probably/ go away - s/sooner rather than later/like right now... during this meeting/ 21:17:32 oh right runway!!! that one might already be gone ๐Ÿค” 21:17:48 yeah, that one's already gone 21:18:50 https://gitlab.com/nvidia/proxyfs-ci/runway 21:19:05 we can keep thinking about how best to do dev envs and how similar they need to be to CI vs prod vs some other crazy thing, but i think we should probably keep moving 21:19:38 #topic sharding in train 21:19:59 zaitcev, i haven't seen patches yet, how's it going? 21:20:04 I put together a stack of 18 patches, they pass unit tests. 21:20:16 The 19th was much too hard, so I gave up. 21:20:16 \o/ 21:20:37 didn't sinatra do a song about "sharding in the rain"? 21:20:38 Unfortunately, I must focus on RBAC this week. 21:21:01 no worries, and thank you for taking on the RBAC work! 21:21:14 i just wanted to check in, make sure you weren't blocked 21:21:17 So, I'm tempted to throw them into Gerrit in one big stack, just so they're not locked in my laptop. 21:21:52 After we talked about it last week, I wanted to feed them in batches of 4 or 5, to let review them easier. 21:21:58 that's fine by me. i'll try to get through them quickly once they're up 21:22:07 I replaced Change-ID at least 21:23:01 if possible, try to include the cherry-picked sha in the commit message; makes it a little easier for me to compare master vs stable 21:23:24 Yes, I changed old Change-Id with Cherry-Picked-From. 21:23:52 That's all 21:24:07 #topic eventlet and ssl 21:24:22 #link http://lists.openstack.org/pipermail/openstack-discuss/2021-January/020100.html 21:25:12 i was catching up on mailing list recently and saw zigo has been having trouble with eventlet and ssl in swift-proxy 21:25:39 timburke_: I found out that the issue is dnspython 2.0. 21:25:47 i wanted to check if anyone has a ssl-enabled-keystone handy to try to repro 21:26:12 oh, curious 21:26:13 timburke_: The issue is swift-proxy connecting to Keystone to check credentials... 21:26:14 Ironically I don't 21:26:33 I created a different region for Keystone and set Swift to talk using that region. 21:26:39 So the problem is not a swift-proxy binding over SSL. 21:26:47 timburke_: we do have that, swift talking to a Keystone over SSL 21:26:51 ...i guess maybe dnspython imports ssl before eventlet's monkey-patched it? 21:27:12 timburke_: I can't tell, but it's definitively a monkey paching issue between dnspython and eventlet. 21:27:34 i know i've seen similar recursion errors before, and it's been a matter of not monkey-patching early enough 21:27:42 The same issue happens when Neutron tries to tell Nova (over the Nova API) that a VM port is up. 21:28:29 Well, I'd prefer if there was a strong movement to get out of this madness. 21:28:44 Monkey patching is a terrible idea. 21:29:21 It has numerous times, and still bites hard... 21:30:08 https://github.com/eventlet/eventlet/issues/619 <--- The issue has been opened since 25th of June ... 21:30:30 https://github.com/rthalley/dnspython/blob/v2.0.0/dns/query.py#L48 i guess? i don't see an ssl import on 1.16.0 (at a quick glance, anyway) 21:31:29 yeah, monkey-patching's... not great. one more reason we ought to look at that PUT+POST(+POST) patch again... it bugs me that we're so tied to eventlet 21:32:58 My guess, is that we do eventlet monkey patching early, but then dnspython does monkey patching *after*, and then accessing stuff on the SSLContext object breaks hard. 21:33:05 (I'm not sure, just double-guessing) 21:33:56 oh -- it does its own monkey patching or something, is that it? ick 21:34:34 Isn't what you've just linked does? 21:34:55 (ie: creating an SSLSocket object...) 21:35:17 does anyone have bandwidth to try to repro/fix the issue? having a pin on a two-year-old version of dnspython doesn't seem sustainable 21:35:47 It's also currently completely broken in both Fedora and Debian (both have dnspython 2.0.x). 21:36:28 I'm trying to push to revert to 1.16.0, but I'm not sure I'll be successful. 21:36:31 timburke_: does anyone remember why we depend on dnspython? 21:36:46 clayg: eventlet does depend on it ... 21:37:18 cname_lookup 21:37:35 python-eventlet (master)$ cat setup.py | grep dns 21:37:35 'dnspython >= 1.15.0, < 2.0.0', 21:38:17 so we *do* use it for cname_lookup, but the bigger issue seems to be that if you have it installed for the sake of something else, it'll break ssl in eventlet-ified processes 21:38:54 Indirectly, yes. 21:39:07 We use keystoneauth, which calls requests, which calls urllib3. 21:39:35 Urllib3 access the SSL socket SSLContext.options object, and when it does ... big crash ! 21:40:59 I believe the issue is because this: 21:40:59 https://github.com/eventlet/eventlet/blob/master/eventlet/green/ssl.py#L449 21:40:59 isn't in use because of dnspython overriding the eventlet monkey patching. 21:41:10 I may be wrong, but so far, that's where I am... 21:41:56 all right, i'll look into it. the dnspython tip was useful, looks like i might have a repro now! 21:42:20 What's the intention behind this: 21:42:20 https://github.com/rthalley/dnspython/blob/v2.0.0/dns/query.py#L58 21:42:20 ? 21:42:20 #topic orphaned shard ranges 21:42:39 Ok, thanks. 21:42:50 timburke_: Feel free to ping me anytime and we can discuss this later. 21:43:21 zigo, i think it's just trying to stub out enough to prevent NameErrors and the like in case ssl isn't available 21:43:52 * zigo will try to patch this out, just to see if it continues to work ... 21:45:03 i know we picked up https://review.opendev.org/c/openstack/swift/+/771086 recently to prevent us from running into this orphaned-shard situation... 21:46:13 is https://review.opendev.org/c/openstack/swift/+/770529 still viable for cleaning up any orphans that may already be on disk? or should we abandon that? 21:46:23 yup, and that stops them being created. 21:46:42 There is work on getting the new shrink code working. 21:47:15 acoles: has a chain starting: https://review.opendev.org/c/openstack/swift/+/771885 21:47:19 yeah, maybe i should change the agenda item to cover shrinking generally ;-) 21:47:23 yeah 21:48:16 the start of the chain will allow the root to provide the final shard acceptor as it's self when collapsing. (to keep it root driven) 21:49:09 acoles recently did an awesome job of simlifying that with a auditing state 21:49:16 I prefer the root driven shrink & delete approach rather than the shard self-determination in https://review.opendev.org/c/openstack/swift/+/770529 21:49:33 +1 21:49:59 later on that chain is the new compact shard-manage-shard-ranges command 21:50:10 which reworks how shrinking works in the sharder. 21:50:30 in general the shrinking and overlap repair is coming along nicely, but we uncovered a couple of bugs along the way 21:50:49 Once all these peices are done, orphan shards will be dealt with. initially manually, but its moving in the right direction for automatically too. 21:50:58 mattoliverau: has fixed one https://review.opendev.org/c/openstack/swift/+/773832 21:51:57 I'm about to push another fix, these are both bug fixes in addition to the new feature in swift-manage-shard-ranges commands 21:52:11 so its been an interesting journey :) 21:52:31 :) 21:52:33 cool! sounds like things are moving right along. blocked on anything? 21:54:03 timburke_: not enough hours in a day ! :) 21:54:04 not right now, just testing, reviews, coding.. and confidence we've fixed all the bugs :P 21:54:38 there's the other bug fix https://review.opendev.org/c/openstack/swift/+/774002 21:54:52 all right then 21:54:59 #topic relinker 21:55:06 note: these bugs would not impact *sharding*, just shrinking of shards 21:55:07 i'm still working on some relinker enhancements, and wanted to call out a couple things 21:55:17 first, my patches are (mostly) in a single chain now, so it's easier to try out all the improvements at once by checking out the end of the chain 21:56:12 second, the first couple patches in the chain make it so you can point the relinker at a config file to read most (but not quite all) of the cli flags 21:57:01 that was mostly because i realized we've got a bunch of options that should already be in you [DEFAULT] section of your object-server.conf (--swift-dir, --devices, --skip-mount-check) 21:57:36 with https://review.opendev.org/c/openstack/swift/+/772419, --user 21:58:40 i was mainly wondering if the config file seemed like as reasonable idea to everyone; i can't think of any other one-off tools we have in the swift repo that would go read a conf file... 21:59:15 the config file is still optional tho right? you can still use cli ops if need be. 21:59:57 So I think it makes sense to allow a config, esp when most of the options may already be set. But also it optional so wont stop how others may already be using the tool. 21:59:59 yup -- existing cli tooling should all still work 22:00:21 great job 22:00:42 with that in mind, should i make sure that all config options get CLI args? 22:00:44 timburke_:+1 I think it is very reasonable 22:01:41 +1 was for having a conf file. not sure if you *must* expose all the options if they are beyond current cli 22:02:04 if the defaults are sensible 22:02:04 I think we're at time :( 22:02:20 yeah, i was noticing that, too ;-) 22:02:27 thank you all for coming, and thank you for working on swift! 22:02:30 #endmeeting