| *** jkulik has quit IRC | 00:02 | |
| *** jkulik has joined #openstack-meeting | 00:02 | |
| *** jmasud has joined #openstack-meeting | 00:04 | |
| *** egallen has quit IRC | 00:06 | |
| *** pescobar has quit IRC | 00:07 | |
| *** pescobar has joined #openstack-meeting | 00:08 | |
| *** jmasud has quit IRC | 00:10 | |
| *** jmasud has joined #openstack-meeting | 00:14 | |
| *** jmasud has quit IRC | 00:16 | |
| *** eharney has quit IRC | 00:29 | |
| *** jmasud has joined #openstack-meeting | 00:35 | |
| *** jmasud has quit IRC | 00:41 | |
| *** vishalmanchanda has quit IRC | 00:41 | |
| *** eharney has joined #openstack-meeting | 00:43 | |
| *** jmasud has joined #openstack-meeting | 01:11 | |
| *** jmasud has quit IRC | 01:12 | |
| *** jawad_axd has quit IRC | 01:19 | |
| *** mlavalle has quit IRC | 01:35 | |
| *** rcernin has quit IRC | 01:43 | |
| *** armstrong has quit IRC | 02:02 | |
| *** rcernin has joined #openstack-meeting | 02:20 | |
| *** rcernin has quit IRC | 02:24 | |
| *** rcernin has joined #openstack-meeting | 02:28 | |
| *** rcernin has quit IRC | 02:31 | |
| *** rcernin has joined #openstack-meeting | 02:32 | |
| *** bcafarel has quit IRC | 02:37 | |
| *** armax has quit IRC | 03:14 | |
| *** armax has joined #openstack-meeting | 03:23 | |
| *** armax has quit IRC | 03:27 | |
| *** bcafarel has joined #openstack-meeting | 03:33 | |
| *** yamamoto has quit IRC | 04:14 | |
| *** yamamoto has joined #openstack-meeting | 04:14 | |
| *** yamamoto has quit IRC | 04:19 | |
| *** yamamoto has joined #openstack-meeting | 04:19 | |
| *** yamamoto has quit IRC | 04:30 | |
| *** yamamoto has joined #openstack-meeting | 04:43 | |
| *** jmasud has joined #openstack-meeting | 04:47 | |
| *** yamamoto has quit IRC | 04:59 | |
| *** yamamoto has joined #openstack-meeting | 05:03 | |
| *** yamamoto has quit IRC | 05:03 | |
| *** yamamoto has joined #openstack-meeting | 05:03 | |
| *** jmasud has quit IRC | 05:05 | |
| *** yamamoto has quit IRC | 05:08 | |
| *** jmasud has joined #openstack-meeting | 05:24 | |
| *** evrardjp has quit IRC | 05:33 | |
| *** evrardjp has joined #openstack-meeting | 05:33 | |
| *** dsariel has joined #openstack-meeting | 05:44 | |
| *** ociuhandu has joined #openstack-meeting | 05:44 | |
| *** ociuhandu has quit IRC | 05:49 | |
| *** vishalmanchanda has joined #openstack-meeting | 06:00 | |
| *** gyee has quit IRC | 06:08 | |
| *** jmasud has quit IRC | 06:14 | |
| *** servagem has quit IRC | 06:18 | |
| *** servagem has joined #openstack-meeting | 06:33 | |
| *** mahatic has joined #openstack-meeting | 06:48 | |
| *** rpittau|afk is now known as rpittau | 06:49 | |
| *** icey has quit IRC | 06:52 | |
| *** icey has joined #openstack-meeting | 06:54 | |
| *** yamamoto has joined #openstack-meeting | 07:07 | |
| *** yamamoto has quit IRC | 07:18 | |
| *** yamamoto has joined #openstack-meeting | 07:21 | |
| *** yamamoto has quit IRC | 07:22 | |
| *** ralonsoh has joined #openstack-meeting | 07:31 | |
| *** clayg has quit IRC | 07:35 | |
| *** dklyle has quit IRC | 07:35 | |
| *** clayg has joined #openstack-meeting | 07:35 | |
| *** yasufum has joined #openstack-meeting | 07:37 | |
| *** zbr4 has quit IRC | 07:38 | |
| *** slaweq has joined #openstack-meeting | 07:43 | |
| *** yamamoto has joined #openstack-meeting | 07:58 | |
| *** yamamoto has quit IRC | 08:12 | |
| *** lbragstad_ has joined #openstack-meeting | 08:18 | |
| *** lbragstad has quit IRC | 08:22 | |
| *** jmasud has joined #openstack-meeting | 08:25 | |
| *** zbr has joined #openstack-meeting | 08:26 | |
| *** tosky has joined #openstack-meeting | 08:45 | |
| *** ociuhandu has joined #openstack-meeting | 08:47 | |
| *** rcernin has quit IRC | 08:52 | |
| *** rcernin has joined #openstack-meeting | 09:15 | |
| *** slaweq has quit IRC | 09:17 | |
| *** slaweq has joined #openstack-meeting | 09:19 | |
| *** ociuhandu has quit IRC | 09:32 | |
| *** e0ne has joined #openstack-meeting | 09:38 | |
| *** rcernin has quit IRC | 09:45 | |
| *** ociuhandu has joined #openstack-meeting | 09:49 | |
| *** ociuhandu has joined #openstack-meeting | 09:49 | |
| *** yoctozepto has quit IRC | 09:50 | |
| *** yoctozepto has joined #openstack-meeting | 09:51 | |
| *** ykatabam has joined #openstack-meeting | 09:54 | |
| *** rcernin has joined #openstack-meeting | 09:55 | |
| *** ykatabam has quit IRC | 10:11 | |
| *** kevinz has quit IRC | 10:26 | |
| *** icey has quit IRC | 10:31 | |
| *** icey has joined #openstack-meeting | 10:32 | |
| *** yasufum has quit IRC | 10:40 | |
| *** yasufum has joined #openstack-meeting | 10:43 | |
| *** belmoreira has joined #openstack-meeting | 10:46 | |
| *** oneswig has joined #openstack-meeting | 10:49 | |
| oneswig | #startmeeting scientific-sig | 11:00 |
|---|---|---|
| openstack | Meeting started Wed Nov 18 11:00:36 2020 UTC and is due to finish in 60 minutes. The chair is oneswig. Information about MeetBot at http://wiki.debian.org/MeetBot. | 11:00 |
| openstack | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 11:00 |
| *** openstack changes topic to " (Meeting topic: scientific-sig)" | 11:00 | |
| openstack | The meeting name has been set to 'scientific_sig' | 11:00 |
| oneswig | greetings | 11:00 |
| janders | g'day oneswig | 11:02 |
| janders | how are things? | 11:02 |
| oneswig | Hi janders - going well thanks. I have again not put any thought into preparing for the SIG meeting, alas :-( | 11:02 |
| oneswig | Was just thinking about Lustre and Manila again - seems like a lot of people would be interested in this | 11:03 |
| janders | yeah... I was never ever a fan of having NFS gateways in between | 11:03 |
| verdurin | Morning. We certainly would be. Some new hardware specifically for Secure Lustre testing will be arriving soon. | 11:04 |
| oneswig | Does take away much of the performance advantage | 11:04 |
| oneswig | Hello verdurin, how timely! | 11:04 |
| * janders is looking up a picture that illustrates the issue very well | 11:04 | |
| *** icey has quit IRC | 11:05 | |
| oneswig | With the SIG spanning the globe, it's hard to bring every party to one place and time. | 11:05 |
| oneswig | janders: your former colleagues at CSIRO were using BeeGFS, right? Were they also using Lustre? | 11:06 |
| janders | parallel filesystem / native: https://www.travelweekly.com.au/wp-content/uploads/2019/05/Qantas-Dreamliner.png | 11:07 |
| janders | parallel filesystem + NFS re-export: https://c8.alamy.com/comp/FJCA88/cook-transport-low-loader-truck-taking-a-wide-load-consisting-of-a-FJCA88.jpg | 11:08 |
| janders | I reckon it's not far off... | 11:08 |
| oneswig | ha ha! Good analogy | 11:08 |
| janders | oneswig I they were exploring Lustre towards the end of my time with CSIRO | 11:08 |
| oneswig | janders: anyone there who would be a good contact for a discussion on this? | 11:09 |
| *** yasufum has quit IRC | 11:10 | |
| *** lpetrut has joined #openstack-meeting | 11:10 | |
| oneswig | There's been some interesting talk recently about DDN Lustre and Kubernetes CSI | 11:12 |
| oneswig | An improbable pairing that apparently works | 11:13 |
| janders | oneswig not sure :( | 11:17 |
| janders | regarding k8s + Lustre - not entirely surprised | 11:17 |
| oneswig | Lustre's a user-space driver so may containerise well. | 11:18 |
| *** rcernin has quit IRC | 11:18 | |
| oneswig | The question always arises, doing the development is one thing, supporting and sustaining it another matter altogether | 11:19 |
| *** rcernin has joined #openstack-meeting | 11:19 | |
| janders | agreed | 11:23 |
| janders | but I think there is soemething in containers directly consuming filesystems, pending a reasonable security model | 11:23 |
| janders | feels like one of the ways of the future (superfast object being another one) | 11:24 |
| oneswig | I think it has good potential too. | 11:25 |
| oneswig | janders: this might interest you: https://www.stackhpc.com/sc20-top500.html | 11:26 |
| *** icey has joined #openstack-meeting | 11:26 | |
| *** rcernin has quit IRC | 11:27 | |
| oneswig | We got a machine deployed with 1274 bare metal nodes using Ironic, and into the top 100 (just) | 11:27 |
| janders | that is awesome! :) | 11:27 |
| janders | can I re-post this on #openstack-ironic? :) | 11:28 |
| oneswig | It's getting redeployed for production now. There's going to be some mixed baremetal and virt, which will be implemented *somewhat* like SuperCloud | 11:28 |
| oneswig | janders: of course :-) | 11:28 |
| janders | ...and the idea lives on - fantastic! | 11:28 |
| janders | I am very glad to hear this | 11:28 |
| oneswig | The method isn't an exact copy of your approach, but hypervisors will exist in the overcloud Ironic. | 11:28 |
| verdurin | Looks nice. Does the main text refer to the machine at the Other Place? | 11:30 |
| verdurin | Not the UM6P one? | 11:30 |
| *** baojg has quit IRC | 11:30 | |
| oneswig | verdurin: They are somewhat similar and borrow from each other. | 11:31 |
| *** baojg has joined #openstack-meeting | 11:31 | |
| oneswig | The telemetry graph, I don't think it says, is from a day of LINPACK benchmarking. You can see the carbon footprint of HPC | 11:32 |
| *** rcernin has joined #openstack-meeting | 11:33 | |
| *** slaweq has quit IRC | 11:39 | |
| *** yamamoto has joined #openstack-meeting | 11:39 | |
| *** icey has quit IRC | 11:41 | |
| oneswig | I got the free pass to Supercomputing but must admit I've yet to use it. Anyone followed the keynotes or other parts? | 11:42 |
| *** icey has joined #openstack-meeting | 11:43 | |
| verdurin | I watched a few bits last week, have lacked the time to look at anything this week. | 11:43 |
| *** e0ne has quit IRC | 11:44 | |
| *** slaweq has joined #openstack-meeting | 11:45 | |
| *** rcernin has quit IRC | 11:46 | |
| oneswig | alas a similar situation here. | 11:48 |
| oneswig | Any other business to raise? | 11:48 |
| janders | same here with Kubecon :( | 11:48 |
| *** jmasud has quit IRC | 11:48 | |
| janders | I think we're good | 11:48 |
| janders | it was great to chat! :) | 11:48 |
| oneswig | Likewise, thanks janders verdurin | 11:49 |
| verdurin | Yes, bye. | 11:49 |
| *** icey has quit IRC | 11:49 | |
| oneswig | I will follow up with ideas on Lustre | 11:49 |
| oneswig | until next time | 11:49 |
| oneswig | #endmeeting | 11:49 |
| *** openstack changes topic to "OpenStack Meetings || https://wiki.openstack.org/wiki/Meetings/" | 11:49 | |
| janders | till then! | 11:49 |
| openstack | Meeting ended Wed Nov 18 11:49:48 2020 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 11:49 |
| openstack | Minutes: http://eavesdrop.openstack.org/meetings/scientific_sig/2020/scientific_sig.2020-11-18-11.00.html | 11:49 |
| openstack | Minutes (text): http://eavesdrop.openstack.org/meetings/scientific_sig/2020/scientific_sig.2020-11-18-11.00.txt | 11:49 |
| janders | 935848 | 11:49 |
| openstack | Log: http://eavesdrop.openstack.org/meetings/scientific_sig/2020/scientific_sig.2020-11-18-11.00.log.html | 11:49 |
| *** rcernin has joined #openstack-meeting | 11:52 | |
| *** ociuhandu has quit IRC | 11:52 | |
| *** rcernin has quit IRC | 11:55 | |
| *** icey has joined #openstack-meeting | 11:55 | |
| *** ociuhandu has joined #openstack-meeting | 11:58 | |
| *** e0ne has joined #openstack-meeting | 12:18 | |
| *** icey has quit IRC | 12:23 | |
| *** icey has joined #openstack-meeting | 12:24 | |
| *** oneswig has quit IRC | 12:25 | |
| *** ociuhandu has joined #openstack-meeting | 12:30 | |
| *** yamamoto has quit IRC | 12:43 | |
| *** baojg has quit IRC | 12:48 | |
| *** baojg has joined #openstack-meeting | 12:49 | |
| *** icey has quit IRC | 12:50 | |
| *** masahito has joined #openstack-meeting | 13:03 | |
| *** yamamoto has joined #openstack-meeting | 13:05 | |
| *** yamamoto has quit IRC | 13:05 | |
| *** yamamoto has joined #openstack-meeting | 13:05 | |
| *** yamamoto has quit IRC | 13:10 | |
| *** yamamoto has joined #openstack-meeting | 13:26 | |
| *** yamamoto has quit IRC | 13:26 | |
| *** yamamoto has joined #openstack-meeting | 13:27 | |
| *** yasufum has joined #openstack-meeting | 13:27 | |
| *** yasufum has quit IRC | 13:31 | |
| *** yamamoto has quit IRC | 13:31 | |
| *** yasufum has joined #openstack-meeting | 13:32 | |
| *** ociuhandu has quit IRC | 13:32 | |
| *** e0ne has quit IRC | 13:51 | |
| *** masahito has quit IRC | 13:52 | |
| *** abishop has joined #openstack-meeting | 13:57 | |
| *** abishop has left #openstack-meeting | 13:58 | |
| *** masahito has joined #openstack-meeting | 13:58 | |
| *** e0ne has joined #openstack-meeting | 14:01 | |
| *** thgcorrea has joined #openstack-meeting | 14:04 | |
| *** lajoskatona has joined #openstack-meeting | 14:05 | |
| *** macz_ has joined #openstack-meeting | 14:08 | |
| *** macz_ has quit IRC | 14:12 | |
| *** mbuil has quit IRC | 14:12 | |
| *** ociuhandu has joined #openstack-meeting | 14:23 | |
| *** ociuhandu has quit IRC | 14:28 | |
| *** ociuhandu has joined #openstack-meeting | 14:28 | |
| *** TrevorV has joined #openstack-meeting | 14:32 | |
| *** yasufum has quit IRC | 14:34 | |
| *** yasufum has joined #openstack-meeting | 14:43 | |
| *** yasufum has quit IRC | 14:52 | |
| *** yasufum has joined #openstack-meeting | 14:53 | |
| *** lbragstad_ is now known as lbragstad | 14:59 | |
| *** bcafarel has quit IRC | 14:59 | |
| *** bcafarel has joined #openstack-meeting | 15:04 | |
| *** dsariel has quit IRC | 15:05 | |
| *** dsariel has joined #openstack-meeting | 15:05 | |
| *** dsariel has quit IRC | 15:10 | |
| *** ociuhandu has quit IRC | 15:11 | |
| *** dsariel has joined #openstack-meeting | 15:11 | |
| *** yasufum has joined #openstack-meeting | 15:19 | |
| *** lpetrut has quit IRC | 15:21 | |
| *** mlavalle has joined #openstack-meeting | 15:25 | |
| *** dklyle has joined #openstack-meeting | 15:37 | |
| *** armax has joined #openstack-meeting | 15:39 | |
| *** ociuhandu has joined #openstack-meeting | 15:40 | |
| *** ociuhandu_ has joined #openstack-meeting | 15:59 | |
| *** ociuhandu has quit IRC | 16:02 | |
| *** jmasud has joined #openstack-meeting | 16:12 | |
| *** macz_ has joined #openstack-meeting | 16:13 | |
| *** ociuhandu_ has quit IRC | 16:22 | |
| *** ociuhandu has joined #openstack-meeting | 16:22 | |
| *** belmoreira has quit IRC | 16:27 | |
| *** jmasud has quit IRC | 16:28 | |
| *** masahito has quit IRC | 16:33 | |
| *** ociuhandu has quit IRC | 16:37 | |
| *** yasufum has quit IRC | 16:37 | |
| *** ociuhandu has joined #openstack-meeting | 16:45 | |
| *** lajoskatona has left #openstack-meeting | 17:02 | |
| *** ociuhandu_ has joined #openstack-meeting | 17:03 | |
| *** ociuhandu has quit IRC | 17:06 | |
| *** ociuhandu_ has quit IRC | 17:07 | |
| *** e0ne has quit IRC | 17:21 | |
| *** e0ne has joined #openstack-meeting | 17:22 | |
| *** e0ne has quit IRC | 17:33 | |
| *** yamamoto has joined #openstack-meeting | 17:33 | |
| *** yamamoto has quit IRC | 17:37 | |
| *** baojg has quit IRC | 17:44 | |
| *** baojg has joined #openstack-meeting | 17:45 | |
| *** dsariel has quit IRC | 17:46 | |
| *** e0ne has joined #openstack-meeting | 17:57 | |
| *** ralonsoh has quit IRC | 18:31 | |
| *** e0ne has quit IRC | 18:36 | |
| *** rpittau is now known as rpittau|afk | 18:39 | |
| *** bbowen has quit IRC | 19:25 | |
| *** sluna has quit IRC | 19:27 | |
| *** sluna has joined #openstack-meeting | 19:27 | |
| *** baojg has quit IRC | 19:46 | |
| *** baojg has joined #openstack-meeting | 19:47 | |
| *** vishalmanchanda has quit IRC | 19:49 | |
| *** gyee has joined #openstack-meeting | 20:10 | |
| *** jmasud has joined #openstack-meeting | 20:13 | |
| *** ykatabam has joined #openstack-meeting | 20:23 | |
| *** jmasud has quit IRC | 20:26 | |
| *** acoles has joined #openstack-meeting | 20:27 | |
| *** jmasud has joined #openstack-meeting | 20:28 | |
| *** jmasud has quit IRC | 20:35 | |
| *** jmasud has joined #openstack-meeting | 20:36 | |
| *** jmasud has quit IRC | 20:36 | |
| *** ricolin has quit IRC | 20:40 | |
| *** TrevorV has quit IRC | 20:54 | |
| *** alecuyer has joined #openstack-meeting | 20:55 | |
| *** zaitcev has joined #openstack-meeting | 20:56 | |
| *** slaweq has quit IRC | 20:58 | |
| timburke | #startmeeting swift | 21:00 |
| openstack | Meeting started Wed Nov 18 21:00:15 2020 UTC and is due to finish in 60 minutes. The chair is timburke. Information about MeetBot at http://wiki.debian.org/MeetBot. | 21:00 |
| openstack | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 21:00 |
| *** openstack changes topic to " (Meeting topic: swift)" | 21:00 | |
| openstack | The meeting name has been set to 'swift' | 21:00 |
| timburke | who's here for the swift meeting? | 21:00 |
| zaitcev | o/ | 21:00 |
| alecuyer | o/ | 21:00 |
| rledisez | hi o/ | 21:00 |
| acoles | o/ | 21:00 |
| *** bbowen has joined #openstack-meeting | 21:00 | |
| mattoliverau | o/ | 21:01 |
| timburke | agenda's at https://wiki.openstack.org/wiki/Meetings/Swift | 21:02 |
| timburke | #topic no meeting next week | 21:02 |
| *** openstack changes topic to "no meeting next week (Meeting topic: swift)" | 21:02 | |
| timburke | thanksgiving is next week in the US and i know i'm going to be out all week. i expect things will probably be pretty quiet anyway | 21:03 |
| mattoliverau | Kk | 21:03 |
| clayg | this is where the party is?! | 21:03 |
| timburke | so unless someone else would like to chair it, i propose we skip next week's meeting | 21:03 |
| clayg | yeah, party somewhere else next week | 21:04 |
| mattoliverau | Skipping is fine for me | 21:04 |
| timburke | clayg, iirc you'll be around much of the start of the week; would you mind dropping a reminder in -swift on the end of your Tuesday (and maybe again early Wednesday)? | 21:05 |
| mattoliverau | I'll do it if you like | 21:06 |
| timburke | that works too! thanks mattoliverau | 21:06 |
| timburke | (mainly i proposed clayg do it because i know his timezone offset better ;-) | 21:06 |
| timburke | #topic RBAC community effort | 21:07 |
| *** openstack changes topic to "RBAC community effort (Meeting topic: swift)" | 21:07 | |
| timburke | zaitcev, you added this; what should we know/discuss? | 21:07 |
| clayg | i set a reminder! (but I mean.. I had a reminder go off fo this meeting this afternoon too) | 21:07 |
| mattoliverau | Lol | 21:07 |
| zaitcev | timburke: mainly ask if you know about it along the TC/PTL communications and what your thoughts are, and let you know that I am going to do something about it, most likely add a reader role. | 21:08 |
| zaitcev | timburke: there's a group called "Pop-Up Something Or Other", officially formed at OpenStack (or Open Infra now?). | 21:09 |
| zaitcev | I started looking at keystoneauth.py and found that our unit tests are not water-tight. I was able to add obvious bugs and tests still passed. | 21:10 |
| timburke | i've not had much communication about it, but if i do i'll be sure to direct them your way. i love the idea of a dedicated reader role | 21:10 |
| zaitcev | What, you do? I thought it was some meaningless make-do invented by Keystone people. | 21:11 |
| zaitcev | Okay, tell me one thing. If a token with a reader role tries to GET an object with _no_ ACLs, is it permitted or not? | 21:11 |
| *** rcernin has joined #openstack-meeting | 21:13 | |
| timburke | now that i've got an ops hat i occasionally put on, i'd love to have something like a read-only reseller admin role. i don't think *this* is *that*, but it seems like a start | 21:13 |
| zaitcev | ok. I thought so. | 21:14 |
| zaitcev | Thanks. | 21:14 |
| zaitcev | We can move on to the next item, as far as I'm concerned. | 21:14 |
| timburke | as to the specific question, i'm inclined to say yeah -- it seems similar to the account acls tempauth allows (if less fine-grained) | 21:16 |
| zaitcev | got it | 21:16 |
| timburke | #topic async deletes of SLO segments | 21:16 |
| *** openstack changes topic to "async deletes of SLO segments (Meeting topic: swift)" | 21:16 | |
| timburke | so we merged this recently; i just wanted to give an update now that i've actually seen it in use | 21:17 |
| timburke | tl;dr: it looks to be working great! | 21:17 |
| zaitcev | Good | 21:18 |
| timburke | max delete request time went down from something like 25mins to generally around 10s | 21:18 |
| mattoliverau | nice | 21:18 |
| mattoliverau | are things getting deleted in a timely manner async wise? hows the general task queue size? | 21:19 |
| timburke | the expirer queue would fill and drain in pretty nice waves | 21:19 |
| zaitcev | I'm amazed. Often bumping things to be async does nothing because you still need to use the same amount of resources over long term. | 21:19 |
| *** thgcorrea has quit IRC | 21:19 | |
| timburke | heh. *client* delete request timing ;-) | 21:19 |
| zaitcev | Also, I thought you were looking for a better compatibility with S3. Although maybe I confuse it with some other segment-deletion patch. | 21:20 |
| clayg | and it seems like the expirer's ratelimiting tasks_per_second give us all the knob we need - we can stuff tombstones fast enough they go to async_pending, or slow enough to barely keep up with the next wave of DELETEs | 21:21 |
| timburke | so the queue depth definitely got larger than i was expecting, but the expirers could generally keep up. there was certainly some fairly linear growth during the 48hrs or so worth of deletes, but once the client requests stopped, it cleared pretty quickly | 21:21 |
| zaitcev | Aww | 21:21 |
| zaitcev | To me it does not sound good. There's still a chance for clients to kill clusters. Well, it can be disable on public clouds. | 21:22 |
| zaitcev | Anyway, seems like a net positive. | 21:22 |
| timburke | zaitcev, it's tied to s3api, but somewhat loosely. certainly, i don't expect most s3 clients to have a 25min timeout for deletes, where they likely would at least have a 30s timeout | 21:22 |
| clayg | zaitcev: I dont' think that generally s3 clients expect MPU deletes to be pretty fast even with a bunch a bunch of segments in the original MPU - pretty sure AWS is totally async in this regard - and we're close with the +segments container, but not exactly the same in that we still count your segment bytes until we're all cleared out | 21:23 |
| zaitcev | I see. | 21:23 |
| zaitcev | I'm a little curious what we do if clusters can't keep up. But let's kick this can down the road. | 21:23 |
| clayg | zaitcev: you can disable the option and it shows up in /info - that would put backpressure to clients again same as before | 21:24 |
| timburke | turn on ratelimit and only let clients send so many deletes at a time ;-) | 21:24 |
| timburke | that's all i really had there, just wanted to share ('cause it was pretty cool to see it work so well!) | 21:25 |
| timburke | #topic audit watchers | 21:26 |
| *** openstack changes topic to "audit watchers (Meeting topic: swift)" | 21:26 | |
| mattoliverau | thanks for sharing! | 21:26 |
| clayg | timburke's next crazy idea is to queue s3api async mpu deletes on overwrite! | 21:26 |
| timburke | #link https://review.opendev.org/#/c/706653/ | 21:26 |
| patchbot | patch 706653 - swift - Let developers/operators add watchers to object au... - 40 patch sets | 21:26 |
| zaitcev | There was no change. I think watchers are ready to go, in the sense that I see no show-stopers. We may want to fine-tune it. | 21:26 |
| mattoliverau | I'll be reviewing this patch today. I've been a little distracted as of late | 21:26 |
| zaitcev | I've not yet shipped them, but I ran some tests. | 21:27 |
| timburke | i know i meant to loop around to doing another review, too. hopefully we'll have it merged by the next meeting :-) | 21:27 |
| zaitcev | mattoliverau: many thanks and sharding is of course more important for Swift overall, but this is just so close. I want this monkey off my back. | 21:27 |
| timburke | zaitcev, how'd the tests go? anything interesting to fall out of them? | 21:27 |
| zaitcev | timburke: I found that my cluster would be absolutely, 100% clean of dark data, if not for garbage that I created when testing broken PUT+POST. | 21:28 |
| mattoliverau | totally understand, this is higher priority in my opinion, still a bunch of sharding to do, this is so close, we should land it :) And you've done awesome work here | 21:28 |
| timburke | 🎉 | 21:29 |
| timburke | all right, mattoliverau and i will plan to review it over the next couple weeks | 21:29 |
| zaitcev | But I think maybe we can find good uses for the watchers that I am not foreseeing, like checking if SLOs are missing segments or whatnot. | 21:29 |
| timburke | that is *definitely* one of the use-cases i want to try out. and even going the other way, at least for s3api segments | 21:30 |
| timburke | #topic ssync tracebacks | 21:30 |
| zaitcev | oh, right. segments what miss manifests | 21:31 |
| *** slaweq has joined #openstack-meeting | 21:31 | |
| *** openstack changes topic to "ssync tracebacks (Meeting topic: swift)" | 21:31 | |
| *** jmasud has joined #openstack-meeting | 21:31 | |
| timburke | so i noticed today that something like 75% of the tracebacks i'm seeing in prod are ssync receiver bombing out trying to read a chunked request body | 21:32 |
| timburke | acoles pushed up https://review.opendev.org/763205 and it's working its way through the gate now | 21:32 |
| patchbot | patch 763205 - swift - ssync: don't log tracebacks for client disconnects - 2 patch sets | 21:32 |
| zaitcev | Interesting reviews for ssync, I think I can take a look. It's a targeted SsyncClientDisconnected, not "except Exception" :-) | 21:33 |
| timburke | i've also (more rarely) seen issues where ssync gets and unexpected blank, and proposed https://review.opendev.org/744270 | 21:33 |
| patchbot | patch 744270 - swift - ssync: Tolerate more hang-ups - 4 patch sets | 21:33 |
| timburke | as much as anything, i just wanted to point the frequent ssync errors and mention that we're working to get the noise down | 21:34 |
| timburke | rledisez, alecuyer i'm sure you guys have noticed this problem, too :-) | 21:35 |
| rledisez | timburke: right, ssync is by far the biggest logger when it comes to error | 21:35 |
| rledisez | that's good you're working on that, thx :) | 21:36 |
| alecuyer | yep we see these often | 21:36 |
| acoles | reading the comments in ssync code, I inferred that the traceback logging was originally intended to illuminate code/protocol errors, but the disconnects were getting caught and logged in the same way | 21:36 |
| clayg | ☝️ | 21:36 |
| timburke | #topic s3api, +segments container, and ACLs | 21:37 |
| *** openstack changes topic to "s3api, +segments container, and ACLs (Meeting topic: swift)" | 21:37 | |
| timburke | so i've had s3api users that set up acls on a container, verify that other users can upload ok, but then get very confused when the acl-allowed user can't do multipart uploads | 21:39 |
| timburke | i proposed https://review.opendev.org/763106 as a start, but i've still got a couple concerns | 21:40 |
| patchbot | patch 763106 - swift - s3api: Clone ACLs when creating +segments container - 1 patch set | 21:40 |
| timburke | first, i think the acl-allowed user will still run into trouble if the +segments container doesn't already exist (since they won't have permission to create new containers) | 21:41 |
| clayg | oh yeah, i had that checked out - it looked solid | 21:41 |
| mattoliverau | would you also need to propergate ACL changes when ACLs are changed on the parent container/bucket? | 21:42 |
| timburke | i might be able to work around that by doing a auth-check first then creating the container as a pre-auth'ed request | 21:43 |
| timburke | and then, yeah -- keeping the ACLs in sync is a definite problem. i'm still not sure how to approach that one | 21:43 |
| timburke | (fwiw, the idea for ALOs would be to have all access to the segment container pre-authed and based on the acls set for the main container) | 21:44 |
| clayg | rather that worry about keeping ACLs in sync - can we just work on ... see timburke gets it | 21:44 |
| mattoliverau | yeah, another win for ALO | 21:45 |
| clayg | timburke: whatever the workflow that triggered this - for writing they must have had perms to create +segments - because the problem was "my ACLs were working then some s3client transparently MPU and on some downloads no worky" | 21:46 |
| clayg | i think they have like admin workers doing ingest and then they just want them published for the ML jobs | 21:47 |
| timburke | so i guess my questions are, should a user who's allowed to write objects into the container be able to create the segments container? and if i do the auth-check/pre-auth'ed request to make that possible, would that be "good enough" to merge as a band-aid until i can write ALOs? | 21:47 |
| clayg | I think the band-aid you have is good enough until we have ALO - I think i'd actually be *more* worried about hacking in a pre-authed request - piggybacking on "don't set the wrong spid" with "don't set the wrong acls" seems totally reasonable | 21:48 |
| timburke | clayg, maybe that was it -- for some reason, i could've sworn that i'd seen someone talking about how they were allowed to upload normal objects but not MPUs | 21:48 |
| clayg | if there was any authz concerns with the existing code you're not making it worse | 21:48 |
| timburke | true | 21:48 |
| clayg | ah, you might be right - we've probably had "the wrong ACLs" break both ways - but John said he's happy with "well set it correctly initially" | 21:49 |
| mattoliverau | my initial feeling is, well that's what pre-auth would be for, so long as we're sure the user was authed (via ACLs) first. And they can't say delete the container or any objects there. | 21:49 |
| mattoliverau | but this is morning Matt who is definitely under caffinated :P | 21:50 |
| clayg | honestly I'm amazed the +segments auto-vivify works as well as it does w/o hand-holding; and I guess maybe there's more hand holding going on than I realize (SRE is always dealing with tickets and slack messages to fix ACLs) | 21:50 |
| timburke | all right, maybe i'll just leave the patch as-is then, and write a script to go compare acls between containers across the whole cluster (since we don't have *that* many containers) | 21:51 |
| clayg | mattoliverau: you might be right - if we say "you can upload objects" you can probably create the +segments container to create the objects 🤔 | 21:51 |
| clayg | I just don't like to think to hard when it comes to s3api and swift ACLs - I feel like users are mixing metaphors because they can make it work, and not really thinking about security | 21:51 |
| clayg | we should work on less swift ACLs and more s3api compatible security policies! | 21:52 |
| clayg | zaitcev: amirite!? | 21:52 |
| timburke | my biggest worry is how it smells like a privilege escalation | 21:53 |
| timburke | all right, that's all i've got | 21:54 |
| timburke | #topic open discussion | 21:54 |
| *** openstack changes topic to "open discussion (Meeting topic: swift)" | 21:54 | |
| *** jmasud has quit IRC | 21:55 | |
| timburke | anything else we ought to bring up this week? | 21:55 |
| timburke | if anybody feels like reviewing some client code, https://review.opendev.org/#/c/758500/ seems like a nice usability improvement | 21:57 |
| patchbot | patch 758500 - python-swiftclient - Allow tempurl times to have units - 1 patch set | 21:57 |
| timburke | i don't really feel like memorizing how may seconds are i a day ;-) | 21:57 |
| acoles | never enough I feel ;) | 21:57 |
| timburke | all right | 21:59 |
| timburke | thank you all for coming, and thank you for working on swift! | 21:59 |
| timburke | #endmeeting | 22:00 |
| *** openstack changes topic to "OpenStack Meetings || https://wiki.openstack.org/wiki/Meetings/" | 22:00 | |
| openstack | Meeting ended Wed Nov 18 22:00:03 2020 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 22:00 |
| clayg | 🎉 | 22:00 |
| openstack | Minutes: http://eavesdrop.openstack.org/meetings/swift/2020/swift.2020-11-18-21.00.html | 22:00 |
| acoles | bye | 22:00 |
| openstack | Minutes (text): http://eavesdrop.openstack.org/meetings/swift/2020/swift.2020-11-18-21.00.txt | 22:00 |
| openstack | Log: http://eavesdrop.openstack.org/meetings/swift/2020/swift.2020-11-18-21.00.log.html | 22:00 |
| *** patchbot has left #openstack-meeting | 22:00 | |
| *** acoles has left #openstack-meeting | 22:00 | |
| *** yamamoto has joined #openstack-meeting | 22:01 | |
| *** baojg has quit IRC | 22:03 | |
| *** baojg has joined #openstack-meeting | 22:03 | |
| *** jmasud has joined #openstack-meeting | 22:08 | |
| *** zaitcev has left #openstack-meeting | 22:10 | |
| *** yamamoto has quit IRC | 22:10 | |
| *** yamamoto has joined #openstack-meeting | 22:11 | |
| *** slaweq has quit IRC | 22:16 | |
| *** rh-jlabarre has joined #openstack-meeting | 22:25 | |
| *** vkmc has quit IRC | 22:26 | |
| *** vkmc has joined #openstack-meeting | 22:26 | |
| *** johnsom has quit IRC | 22:27 | |
| *** tinwood_ has joined #openstack-meeting | 22:28 | |
| *** rh-jelabarre has quit IRC | 22:28 | |
| *** tinwood has quit IRC | 22:28 | |
| *** johnsom has joined #openstack-meeting | 22:29 | |
| *** jmasud has quit IRC | 22:38 | |
| *** manpreet has quit IRC | 22:43 | |
| *** jmasud has joined #openstack-meeting | 22:49 | |
| *** jmasud has quit IRC | 22:50 | |
| *** baojg has quit IRC | 23:08 | |
| *** baojg has joined #openstack-meeting | 23:09 | |
| *** number80 has quit IRC | 23:13 | |
| *** number80 has joined #openstack-meeting | 23:26 | |
| *** yamamoto has quit IRC | 23:32 | |
| *** raildo_ has quit IRC | 23:42 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!
