| *** tosky has quit IRC | 00:01 | |
| *** ianychoi has joined #openstack-meeting | 00:14 | |
| *** macz_ has quit IRC | 00:15 | |
| *** zbr has quit IRC | 00:22 | |
| *** zbr has joined #openstack-meeting | 00:22 | |
| *** bbowen has quit IRC | 00:36 | |
| *** bbowen has joined #openstack-meeting | 00:37 | |
| *** mlavalle has quit IRC | 00:47 | |
| *** jmasud has joined #openstack-meeting | 00:52 | |
| *** kevinz has joined #openstack-meeting | 01:03 | |
| *** jmasud has quit IRC | 01:08 | |
| *** jamesmcarthur has quit IRC | 01:13 | |
| *** jamesmcarthur has joined #openstack-meeting | 01:15 | |
| *** icey has quit IRC | 01:15 | |
| *** icey has joined #openstack-meeting | 01:16 | |
| *** jmasud has joined #openstack-meeting | 01:19 | |
| *** jamesmcarthur has quit IRC | 01:36 | |
| *** jamesmcarthur has joined #openstack-meeting | 01:38 | |
| *** dklyle has quit IRC | 01:40 | |
| *** jamesmcarthur has quit IRC | 01:43 | |
| *** armax has quit IRC | 01:47 | |
| *** jamesmcarthur has joined #openstack-meeting | 01:48 | |
| *** dklyle has joined #openstack-meeting | 01:48 | |
| *** baojg has joined #openstack-meeting | 01:52 | |
| *** armax has joined #openstack-meeting | 02:08 | |
| *** tinwood has quit IRC | 02:08 | |
| *** armax has quit IRC | 02:10 | |
| *** tinwood has joined #openstack-meeting | 02:11 | |
| *** macz_ has joined #openstack-meeting | 02:16 | |
| *** macz_ has quit IRC | 02:21 | |
| *** manpreet has quit IRC | 02:44 | |
| *** jmasud has quit IRC | 02:44 | |
| *** dklyle has quit IRC | 02:45 | |
| *** armax has joined #openstack-meeting | 02:48 | |
| *** armax has quit IRC | 02:54 | |
| *** jamesmcarthur has quit IRC | 02:57 | |
| *** jamesmcarthur has joined #openstack-meeting | 02:57 | |
| *** rcernin has quit IRC | 02:57 | |
| *** jamesmcarthur has quit IRC | 03:03 | |
| *** rcernin has joined #openstack-meeting | 03:18 | |
| *** rcernin has quit IRC | 03:21 | |
| *** rcernin has joined #openstack-meeting | 03:21 | |
| *** jamesmcarthur has joined #openstack-meeting | 03:29 | |
| *** psachin has joined #openstack-meeting | 03:33 | |
| *** jmasud has joined #openstack-meeting | 03:35 | |
| *** jmasud has quit IRC | 03:37 | |
| *** ricolin_ has joined #openstack-meeting | 03:39 | |
| *** manpreet has joined #openstack-meeting | 03:46 | |
| *** ricolin_ has quit IRC | 03:54 | |
| *** jamesmcarthur has quit IRC | 04:07 | |
| *** armstrong has joined #openstack-meeting | 04:21 | |
| *** rcernin has quit IRC | 04:35 | |
| *** rcernin has joined #openstack-meeting | 04:35 | |
| *** jmasud has joined #openstack-meeting | 04:41 | |
| *** gyee has quit IRC | 05:08 | |
| *** vishalmanchanda has joined #openstack-meeting | 05:11 | |
| *** evrardjp has quit IRC | 05:20 | |
| *** evrardjp has joined #openstack-meeting | 05:24 | |
| *** evrardjp has quit IRC | 05:33 | |
| *** evrardjp has joined #openstack-meeting | 05:35 | |
| *** jmasud has quit IRC | 05:37 | |
| *** jmasud has joined #openstack-meeting | 05:40 | |
| *** rcernin_ has joined #openstack-meeting | 05:42 | |
| *** rcernin has quit IRC | 05:42 | |
| *** jmasud has quit IRC | 05:49 | |
| *** jmasud_ has joined #openstack-meeting | 05:49 | |
| *** jamesmcarthur has joined #openstack-meeting | 06:04 | |
| *** jamesmcarthur has quit IRC | 06:09 | |
| *** e0ne has joined #openstack-meeting | 06:31 | |
| *** e0ne has quit IRC | 06:49 | |
| *** timburke__ has quit IRC | 06:52 | |
| *** dsariel has joined #openstack-meeting | 06:55 | |
| *** jmasud_ has quit IRC | 07:18 | |
| *** ralonsoh has joined #openstack-meeting | 07:19 | |
| *** rcernin_ has quit IRC | 07:28 | |
| *** armstrong has quit IRC | 07:40 | |
| *** e0ne has joined #openstack-meeting | 07:54 | |
| *** e0ne has quit IRC | 07:54 | |
| *** e0ne has joined #openstack-meeting | 07:55 | |
| *** e0ne has quit IRC | 07:55 | |
| *** slaweq has joined #openstack-meeting | 07:59 | |
| *** diablo_rojo__ has quit IRC | 08:01 | |
| *** slaweq has quit IRC | 08:04 | |
| *** rcernin_ has joined #openstack-meeting | 08:06 | |
| *** slaweq has joined #openstack-meeting | 08:10 | |
| *** jmasud has joined #openstack-meeting | 08:19 | |
| *** e0ne has joined #openstack-meeting | 08:21 | |
| *** e0ne has quit IRC | 08:22 | |
| *** rpittau|afk is now known as rpittau | 08:25 | |
| *** rcernin_ has quit IRC | 08:26 | |
| *** tosky has joined #openstack-meeting | 08:39 | |
| *** e0ne has joined #openstack-meeting | 09:00 | |
| *** rfolco has joined #openstack-meeting | 09:05 | |
| *** jamesmcarthur has joined #openstack-meeting | 09:06 | |
| *** dasp_ has quit IRC | 09:16 | |
| *** dasp has joined #openstack-meeting | 09:17 | |
| *** e0ne has quit IRC | 09:20 | |
| *** e0ne has joined #openstack-meeting | 09:21 | |
| *** e0ne has quit IRC | 09:25 | |
| *** e0ne has joined #openstack-meeting | 09:26 | |
| *** e0ne has quit IRC | 09:26 | |
| *** jamesmcarthur has quit IRC | 09:28 | |
| *** dmacpher has joined #openstack-meeting | 10:17 | |
| *** ociuhandu has joined #openstack-meeting | 10:40 | |
| *** jmasud has quit IRC | 10:47 | |
| *** jmasud has joined #openstack-meeting | 10:49 | |
| *** oneswig has joined #openstack-meeting | 10:59 | |
| oneswig | #startmeeting scientific-sig | 11:00 |
|---|---|---|
| openstack | Meeting started Wed Jan 13 11:00:08 2021 UTC and is due to finish in 60 minutes. The chair is oneswig. Information about MeetBot at http://wiki.debian.org/MeetBot. | 11:00 |
| openstack | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 11:00 |
| *** openstack changes topic to " (Meeting topic: scientific-sig)" | 11:00 | |
| openstack | The meeting name has been set to 'scientific_sig' | 11:00 |
| oneswig | Apologies, no set agenda for today (been busy!) | 11:00 |
| *** dh3 has joined #openstack-meeting | 11:03 | |
| oneswig | Wrangling with diskimage-builder in the other terminal | 11:04 |
| dh3 | o/ my other meeting was cancelled so I thought I'd stick my head in | 11:06 |
| oneswig | Hello dh3, nice to see you. | 11:07 |
| oneswig | I don't think there's anything on my agenda for this week | 11:07 |
| oneswig | Any news to report from you? | 11:07 |
| dh3 | things are ticking over in our "new normal" | 11:08 |
| oneswig | Good to hear. | 11:08 |
| oneswig | The very phrase "ticking over" is evoking all kinds of envy :-) | 11:09 |
| oneswig | (been a busy return to work) | 11:09 |
| *** PeteC61 has joined #openstack-meeting | 11:10 | |
| dh3 | There is plenty going on but not much firefighting which is a good place to be :) | 11:10 |
| oneswig | Happy to hear it, for many reasons | 11:11 |
| PeteC61 | It is only Jan | 11:11 |
| oneswig | Hello PeteC61, nice to see you | 11:12 |
| oneswig | Jan indeed. 2021 is just getting started. | 11:12 |
| oneswig | Next week we have our team "design summit" to plan our R&D activities | 11:13 |
| oneswig | An unusually high number of exciting things to work on... | 11:14 |
| dh3 | are you thinking more infra/deployment tools, or user-facing features? | 11:14 |
| oneswig | Well it covers everything but infrastructure is going to be "the big enchilada" this time round | 11:15 |
| *** b1airo has joined #openstack-meeting | 11:17 | |
| oneswig | Often the priorities we set are akin to new years resolutions in their enduring effect, but we've adopted the 10% time approach to R&D to change that | 11:18 |
| oneswig | Do you have that approach - setting aside some time in the week to focus on R&D and improvement? | 11:19 |
| b1airo | o/ hello and happy new year. i just remembered I'd opened irccloud but then not actually logged in :-) | 11:19 |
| dh3 | That's an interesting one, I'd expect to see some overlap between "I want to work on this" and "I have to work on this to keep my customers happy" | 11:19 |
| dh3 | No official 10% time here but recently we gained an official "no meetings" afternoon each week which is heading in that direction, I think. | 11:20 |
| b1airo | sounds blissful! | 11:20 |
| PeteC61 | That is my plan. In the interim, if something is of potential interest and is proposed, we can potentially make space. | 11:20 |
| oneswig | Hi b1airo, happy new year | 11:21 |
| oneswig | #chair b1airo | 11:21 |
| openstack | Current chairs: b1airo oneswig | 11:21 |
| PeteC61 | always looking to hear how others are progressing though. This can helkp direct us to what else is out there that is working for others. | 11:21 |
| b1airo | interesting culture conversation i've stumbled into by the sounds of it | 11:22 |
| b1airo | i've got some tech/openstack investigations underway if anyone wants to talk shop... | 11:24 |
| oneswig | In some ways it's the challenge of getting beyond chapter 1 in "the phoenix project" | 11:25 |
| oneswig | b1airo: go ahead! | 11:25 |
| b1airo | first, let's start with multi-tenant options for Spectrum Scale. we're designing an OpenStack cloud/hpc hybrid thing at the moment (as you know oneswig), and i'm trying to ensure i've got an up-to-date picture of parallel filesystem options that could fit in and service multiple tenants from shared storage infrastructure | 11:27 |
| b1airo | I have a pretty good idea of what's possible with Lustre, but I'm still trying to figure out if Spectrum Scale even has sufficient isolation controls | 11:29 |
| b1airo | CephFS is obviously a possibility, but I'm a bit wary about it from several perspectives - maturity, supportability, and write/iops performance for scratch storage - we'll probably do other things with it at more modest scale and learn from there | 11:32 |
| oneswig | Have you evaluated Weka b1airo? They have a multi-tenancy story as well. | 11:35 |
| b1airo | my next one is wondering whether anyone has current experience architecting (and running!) Ceph all-flash pools. thinking about choice of data devices and WAL/DB devices etc, also CPUs. wanting to make sure i've learned from someone else's mistakes :-) | 11:35 |
| *** ociuhandu has quit IRC | 11:36 | |
| oneswig | On that subject, this looks like interesting kit https://h20195.www2.hpe.com/v2/GetPDF.aspx/a50000084enw.pdf | 11:36 |
| dh3 | I know Spectrum Scale/GPFS has Cinder drivers etc but not clear if it can present the shared filesystem as a volume (assuming you want to access the same data from both OpenStack and HPC) | 11:37 |
| dh3 | Our Ceph is predominantly HDD (NVME for journals) so the only advice I have is to make sure you use devices with high endurance (DWPD) | 11:39 |
| b1airo | and finally (for now), we've got a partner who will be deploying a cluster (and bunch of related side machines/services) into this environment and they need pretty tight integration with other on-prem infrastructure like the AD domain controllers, so they their own private networking space routed between their on-prem networks and our cloud env. there are plenty of ways of skinning that cat outside of | 11:39 |
| b1airo | OpenStack, but I'm wondering if there is already some relevant routing functionality in Neutron that could make the whole thing more software defined and repeatable for other future tenants | 11:39 |
| b1airo | and finally (for now), we've got a partner who will be deploying a cluster (and bunch of related side machines/services) into this environment and they need pretty tight integration with other on-prem infrastructure like the AD domain controllers, so they want own private networking space routed between their on-prem networks (multiple sites) and our cloud env. there are plenty of ways of skinning that cat | 11:40 |
| b1airo | outside of OpenStack, but I'm wondering if there is already some relevant routing functionality in Neutron that could make the whole thing more software defined and repeatable for other future tenants | 11:40 |
| *** jmasud has quit IRC | 11:42 | |
| b1airo | yep oneswig, that could be a reasonable choice of box. I believe the DL325 is the platform on which HPE offer their Spectrum Scale (ECE based) product/solution too | 11:44 |
| b1airo | though as we will have other general purpose "bulk storage" HDD based pools i was leaning towards scattering NVMe around the cluster. but perhaps it would make more sense to try and get at least 4 dedicated nodes, even if they start out only 1/4 or 1/2 populated from a storage and memory perspective | 11:46 |
| b1airo | dh3: on endurance, that's an interesting point... are you referring to the data devices or WAL/DB? i suspect i've previously been too conservative on this front. some of the all-flash ceph product marketing i've read recently is using QLC based drives with what seem to be very read focused performance | 11:49 |
| b1airo | (data drives that is, often optane for WAL/DB though) | 11:50 |
| oneswig | What's the write endurance quoted for optane? It's certainly a lot quicker and consistent for writes | 11:52 |
| dh3 | I was imagining for the WAL, the DB too assuming your data churn is high enough to justify it. I wonder if the speed differential between Optane and NVME would make it worth it. | 11:53 |
| b1airo | dh3: there is also a Manila driver for Scale/GPFS, however it requires the filesystem to be exported via NFS from GPFS protocol nodes, so it's not a real parallel filesystem option - we do require something high performance | 11:53 |
| b1airo | believe it also requires a shared flat provider network over which all tenants mount their shares, so that immediately limits the practical usage of it | 11:54 |
| dh3 | oneswig we had Intel in to talk about Optane and they didn't put a number on it, just to say "lots" or "plenty" (shades of Rolls-Royce and "adequate horsepower") | 11:55 |
| b1airo | optane is insanely high IOPs both R & W, and very high endurance, but yes i do agree that it could be overly expensive depending on how much you need for DB duties (which in itself seems to be something that changes whereever you look - 4% of data capacity in one place, 6% elsewhere, some GBs per TB number elsewhere, or just 30GB or 300GB per OSD if you dig into mailing list and bugs that detail how things | 11:59 |
| b1airo | work (currently) with RocksDB | 11:59 |
| oneswig | b1airo: the integration with multi-tenant networking could present a significant challenge that HPC-class storage simply doesn't face in it's home turf | 12:00 |
| oneswig | ah, we are out of time | 12:00 |
| dh3 | b1airo I'm interested to know what you end up with, we haven't (yet) found the performance motivation to have anything all-flash | 12:00 |
| oneswig | We did this - but it's out of date now: https://www.stackhpc.com/ceph-on-the-brain-a-year-with-the-human-brain-project.html | 12:01 |
| oneswig | I think this image is still relevant for NVME though - https://www.stackhpc.com/images/julia-nvme-sustained-writes.png | 12:01 |
| oneswig | Better close the meeting - thanks all | 12:03 |
| oneswig | #endmeeting | 12:03 |
| b1airo | to make matters more complicated i've just discovered there's a hybrid Optane memory option now too | 12:03 |
| *** openstack changes topic to "OpenStack Meetings || https://wiki.openstack.org/wiki/Meetings/" | 12:03 | |
| openstack | Meeting ended Wed Jan 13 12:03:46 2021 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 12:03 |
| openstack | Minutes: http://eavesdrop.openstack.org/meetings/scientific_sig/2021/scientific_sig.2021-01-13-11.00.html | 12:03 |
| openstack | Minutes (text): http://eavesdrop.openstack.org/meetings/scientific_sig/2021/scientific_sig.2021-01-13-11.00.txt | 12:03 |
| openstack | Log: http://eavesdrop.openstack.org/meetings/scientific_sig/2021/scientific_sig.2021-01-13-11.00.log.html | 12:03 |
| oneswig | oops, sorry b1airo, messages crossed | 12:03 |
| b1airo | (the H10 if you're interested) | 12:04 |
| b1airo | no worries | 12:04 |
| oneswig | got a link? | 12:04 |
| b1airo | and re endurance for Optane: "The first-generation Optane DC P4800X launched with a 30 DWPD write endurance rating, later increased to 60 DWPD. The new P5800X further increases write endurance to 100 DWPD. Capacities will range from 400GB to 3.2TB." | 12:05 |
| b1airo | https://www.storagereview.com/review/intel-optane-memory-h10-review | 12:05 |
| b1airo | guess i should find the Ceph IRC too | 12:06 |
| oneswig | Good information, thanks b1airo | 12:06 |
| dh3 | interesting reading, thanks, see you soon | 12:06 |
| oneswig | Until next time! | 12:06 |
| b1airo | cya | 12:06 |
| *** oneswig has quit IRC | 12:06 | |
| *** dh3 has quit IRC | 12:10 | |
| *** raildo has joined #openstack-meeting | 12:16 | |
| *** armax has joined #openstack-meeting | 12:24 | |
| *** armstrong has joined #openstack-meeting | 12:38 | |
| *** ociuhandu has joined #openstack-meeting | 12:45 | |
| *** rh-jelabarre has joined #openstack-meeting | 12:53 | |
| *** bbowen has quit IRC | 12:53 | |
| *** ociuhandu has quit IRC | 12:55 | |
| *** ociuhandu has joined #openstack-meeting | 12:56 | |
| *** bbowen has joined #openstack-meeting | 12:56 | |
| *** ociuhandu has quit IRC | 12:56 | |
| *** vishalmanchanda has quit IRC | 13:01 | |
| *** ociuhandu has joined #openstack-meeting | 13:01 | |
| *** jamesmcarthur has joined #openstack-meeting | 13:26 | |
| *** jamesmcarthur has quit IRC | 13:30 | |
| *** e0ne has joined #openstack-meeting | 13:55 | |
| *** lajoskatona has joined #openstack-meeting | 14:00 | |
| *** liuyulong has joined #openstack-meeting | 14:01 | |
| liuyulong | #startmeeting neutron_l3 | 14:02 |
| openstack | Meeting started Wed Jan 13 14:02:17 2021 UTC and is due to finish in 60 minutes. The chair is liuyulong. Information about MeetBot at http://wiki.debian.org/MeetBot. | 14:02 |
| openstack | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 14:02 |
| *** openstack changes topic to " (Meeting topic: neutron_l3)" | 14:02 | |
| openstack | The meeting name has been set to 'neutron_l3' | 14:02 |
| *** liuyulong has quit IRC | 14:03 | |
| slaweq | hi | 14:03 |
| *** liuyulong has joined #openstack-meeting | 14:04 | |
| liuyulong | #startmeeting neutron_l3 | 14:04 |
| openstack | liuyulong: Error: Can't start another meeting, one is in progress. Use #endmeeting first. | 14:04 |
| haleyb | hi | 14:04 |
| liuyulong | Sorry, I lost the internet connection. | 14:04 |
| liuyulong | hi | 14:04 |
| liuyulong | The meeting is started? | 14:04 |
| haleyb | looks like it | 14:04 |
| lajoskatona | Hi | 14:05 |
| * haleyb runs to get a snack quickly | 14:05 | |
| *** liuyulong has quit IRC | 14:05 | |
| *** liuyulong has joined #openstack-meeting | 14:06 | |
| liuyulong | Lost again... | 14:06 |
| liuyulong | #topic Bugs | 14:07 |
| *** openstack changes topic to "Bugs (Meeting topic: neutron_l3)" | 14:07 | |
| liuyulong | We should have at least 3 list from our bug deputy. | 14:07 |
| *** liuyulong has quit IRC | 14:08 | |
| *** liuyulong has joined #openstack-meeting | 14:09 | |
| liuyulong | #link http://lists.openstack.org/pipermail/openstack-discuss/2021-January/019602.html | 14:09 |
| liuyulong | #chair haleyb | 14:09 |
| openstack | Current chairs: haleyb liuyulong | 14:09 |
| liuyulong | #chair slaweq | 14:09 |
| openstack | Current chairs: haleyb liuyulong slaweq | 14:09 |
| liuyulong | Sorry, I've lost the network connection 3 times... | 14:10 |
| liuyulong | #link http://lists.openstack.org/pipermail/openstack-discuss/2021-January/019781.html | 14:10 |
| liuyulong | I have one RFE need the team to pay attention. | 14:10 |
| liuyulong | #link https://bugs.launchpad.net/neutron/+bug/1911126 | 14:10 |
| openstack | Launchpad bug 1911126 in neutron "[RFE][L3] add ability to control router SNAT more granularly" [Wishlist,New] - Assigned to LIU Yulong (dragon889) | 14:10 |
| liuyulong | #link https://review.opendev.org/c/openstack/neutron-specs/+/770540 | 14:10 |
| liuyulong | I've uploaded the spec for it. | 14:11 |
| slaweq | liuyulong: I have this on my todo list to read through it | 14:11 |
| slaweq | I will probably add it to this week's drivers meeting | 14:11 |
| *** dsariel has quit IRC | 14:11 | |
| liuyulong | haleyb, would you mind take over the meeting chair. I'm afraid I will lose the connection again... | 14:11 |
| *** dsariel has joined #openstack-meeting | 14:12 | |
| liuyulong | slaweq, thank you. I will catch the meeting. | 14:12 |
| liuyulong | #link http://lists.openstack.org/pipermail/openstack-discuss/2020-December/019572.html | 14:12 |
| haleyb | i'm back | 14:14 |
| haleyb | So i don't see any new l3 bugs from last weeks report | 14:16 |
| haleyb | any other bugs to discuss? | 14:17 |
| haleyb | #topic On demand agenda | 14:18 |
| *** openstack changes topic to "On demand agenda (Meeting topic: neutron_l3)" | 14:18 | |
| *** liuyulong has quit IRC | 14:18 | |
| haleyb | Any other topics to discuss? | 14:19 |
| slaweq | I don't have anything | 14:19 |
| haleyb | Or reviews that need attention? | 14:19 |
| *** liuyulong has joined #openstack-meeting | 14:19 | |
| *** liuyulong has quit IRC | 14:20 | |
| haleyb | ok, i guess short meeting, liu can just ping anyone in neutron channel if we missed it | 14:20 |
| haleyb | #endmeeting | 14:20 |
| *** openstack changes topic to "OpenStack Meetings || https://wiki.openstack.org/wiki/Meetings/" | 14:20 | |
| openstack | Meeting ended Wed Jan 13 14:20:49 2021 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 14:20 |
| openstack | Minutes: http://eavesdrop.openstack.org/meetings/neutron_l3/2021/neutron_l3.2021-01-13-14.02.html | 14:20 |
| openstack | Minutes (text): http://eavesdrop.openstack.org/meetings/neutron_l3/2021/neutron_l3.2021-01-13-14.02.txt | 14:20 |
| openstack | Log: http://eavesdrop.openstack.org/meetings/neutron_l3/2021/neutron_l3.2021-01-13-14.02.log.html | 14:20 |
| slaweq | o/ | 14:20 |
| slaweq | thx | 14:20 |
| lajoskatona | o/ | 14:22 |
| *** vishalmanchanda has joined #openstack-meeting | 14:35 | |
| *** TrevorV has joined #openstack-meeting | 14:41 | |
| *** jamesmcarthur has joined #openstack-meeting | 14:44 | |
| *** armax has quit IRC | 14:48 | |
| *** armax has joined #openstack-meeting | 14:50 | |
| *** belmoreira has joined #openstack-meeting | 14:57 | |
| *** ociuhandu_ has joined #openstack-meeting | 14:58 | |
| *** ociuhandu has quit IRC | 15:02 | |
| *** macz_ has joined #openstack-meeting | 15:13 | |
| *** macz_ has quit IRC | 15:17 | |
| *** psachin has quit IRC | 15:28 | |
| *** ociuhandu_ has quit IRC | 15:35 | |
| *** ociuhandu has joined #openstack-meeting | 15:35 | |
| *** ricolin has quit IRC | 15:38 | |
| *** jamesmcarthur has quit IRC | 15:45 | |
| *** dklyle has joined #openstack-meeting | 15:46 | |
| *** jamesmcarthur has joined #openstack-meeting | 15:46 | |
| *** ricolin has joined #openstack-meeting | 15:47 | |
| *** jamesmcarthur has quit IRC | 15:51 | |
| *** jamesmcarthur has joined #openstack-meeting | 15:54 | |
| *** macz_ has joined #openstack-meeting | 16:02 | |
| *** ociuhandu_ has joined #openstack-meeting | 16:11 | |
| *** ociuhandu has quit IRC | 16:11 | |
| *** e0ne has quit IRC | 16:15 | |
| *** armstrong has quit IRC | 16:27 | |
| *** jamesmcarthur has quit IRC | 16:32 | |
| *** jamesmcarthur has joined #openstack-meeting | 16:34 | |
| *** dsariel has quit IRC | 16:34 | |
| *** slaweq has quit IRC | 16:34 | |
| *** dsariel has joined #openstack-meeting | 16:34 | |
| *** slaweq has joined #openstack-meeting | 16:36 | |
| *** jamesmcarthur has quit IRC | 16:38 | |
| *** adrianc has quit IRC | 16:41 | |
| *** tosky has quit IRC | 16:41 | |
| *** adrianc has joined #openstack-meeting | 16:42 | |
| *** tosky has joined #openstack-meeting | 16:42 | |
| *** jmasud has joined #openstack-meeting | 16:46 | |
| *** jamesmcarthur has joined #openstack-meeting | 16:50 | |
| *** jmasud has quit IRC | 16:57 | |
| *** jmasud has joined #openstack-meeting | 16:57 | |
| *** gyee has joined #openstack-meeting | 17:00 | |
| *** ociuhandu_ has quit IRC | 17:06 | |
| *** ociuhandu has joined #openstack-meeting | 17:09 | |
| *** e0ne has joined #openstack-meeting | 17:11 | |
| *** ociuhandu_ has joined #openstack-meeting | 17:13 | |
| *** ociuhandu has quit IRC | 17:16 | |
| *** ociuhandu_ has quit IRC | 17:17 | |
| *** ociuhandu has joined #openstack-meeting | 17:22 | |
| *** ociuhandu has quit IRC | 17:27 | |
| *** jamesmcarthur has quit IRC | 17:29 | |
| *** jamesmcarthur has joined #openstack-meeting | 17:30 | |
| *** baojg has quit IRC | 17:41 | |
| *** baojg has joined #openstack-meeting | 17:41 | |
| *** mlavalle has joined #openstack-meeting | 17:57 | |
| *** timburke has joined #openstack-meeting | 17:59 | |
| *** rpittau is now known as rpittau|afk | 18:05 | |
| *** baojg has quit IRC | 18:11 | |
| *** baojg has joined #openstack-meeting | 18:11 | |
| *** timburke_ has joined #openstack-meeting | 18:13 | |
| *** PeteC61 has quit IRC | 18:15 | |
| *** timburke has quit IRC | 18:15 | |
| *** baojg has quit IRC | 18:18 | |
| *** baojg has joined #openstack-meeting | 18:19 | |
| *** ralonsoh has quit IRC | 18:20 | |
| *** e0ne has quit IRC | 18:39 | |
| *** jmasud has quit IRC | 18:46 | |
| *** lajoskatona has quit IRC | 18:48 | |
| *** belmoreira has quit IRC | 19:03 | |
| *** manpreet has quit IRC | 19:06 | |
| *** jamesmcarthur has quit IRC | 19:06 | |
| *** jamesmcarthur_ has joined #openstack-meeting | 19:11 | |
| *** jmasud has joined #openstack-meeting | 19:13 | |
| *** jmasud has quit IRC | 19:36 | |
| *** baojg has quit IRC | 19:39 | |
| *** baojg has joined #openstack-meeting | 19:39 | |
| *** jamesmcarthur_ has quit IRC | 19:54 | |
| *** jamesmcarthur has joined #openstack-meeting | 19:54 | |
| *** jamesmcarthur has quit IRC | 20:14 | |
| *** armstrong has joined #openstack-meeting | 20:20 | |
| *** slaweq has quit IRC | 20:41 | |
| *** acoles has joined #openstack-meeting | 20:58 | |
| timburke_ | #startmeeting swift | 21:00 |
| openstack | Meeting started Wed Jan 13 21:00:11 2021 UTC and is due to finish in 60 minutes. The chair is timburke_. Information about MeetBot at http://wiki.debian.org/MeetBot. | 21:00 |
| openstack | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 21:00 |
| *** openstack changes topic to " (Meeting topic: swift)" | 21:00 | |
| openstack | The meeting name has been set to 'swift' | 21:00 |
| timburke_ | who's here for the swift meeting? | 21:00 |
| mattoliverau | o/ | 21:00 |
| seongsoocho | o/ | 21:00 |
| *** e0ne has joined #openstack-meeting | 21:00 | |
| rledisez | o/ | 21:00 |
| acoles | o/ | 21:01 |
| dsariel | o/ | 21:01 |
| *** e0ne has quit IRC | 21:01 | |
| kota_ | o/ | 21:01 |
| timburke_ | as usual, the agenda's at https://wiki.openstack.org/wiki/Meetings/Swift | 21:01 |
| timburke_ | first up | 21:02 |
| timburke_ | #topic reconciler/ec/encryption | 21:02 |
| *** openstack changes topic to "reconciler/ec/encryption (Meeting topic: swift)" | 21:02 | |
| timburke_ | #link https://bugs.launchpad.net/swift/+bug/1910804 | 21:02 |
| openstack | Launchpad bug 1910804 in OpenStack Object Storage (swift) "Encryption doesn't play well with processes that copy cleartext data while preserving timestamps" [Undecided,New] | 21:02 |
| timburke_ | so i had a customer report an issue with an object that would consistently 503 | 21:03 |
| clayg | ohai | 21:03 |
| timburke_ | digging in more, we found that they had 11 frags of it for an 8+4 policy... but those had 3 separate sets of crypto meta between them | 21:04 |
| timburke_ | ...and no set of crypto meta had more than 7 frags | 21:04 |
| clayg | lawl | 21:04 |
| acoles | (I had to think about this at first) meaning frags have been encrypted with three different body keys...for same object!!! | 21:05 |
| timburke_ | root cause was traced back to a couple issues: (1) we deploy with encryption in the reconciler pipeline and (2) we have every (container?) node running a reconciler | 21:06 |
| timburke_ | (well, that and the fact that it was moved to an EC policy. if it were going to a replicated policy, any replica regardless of crypto meta would be capable of generating a client response) | 21:07 |
| timburke_ | i've got a fix up to pull encryption out of the reconciler pipeline if it was misconfigured -- https://review.opendev.org/c/openstack/swift/+/770522 | 21:08 |
| timburke_ | but i wanted to raise awareness of the issue so no one else finds themselves in this situation | 21:09 |
| timburke_ | also worth noting: i think you could run into a similar issue *without encryption* if your EC backend is non-deterministic | 21:10 |
| *** zaitcev has joined #openstack-meeting | 21:12 | |
| timburke_ | the open source backends are deterministic as i recall (that is, the frag outputs only depend on the EC params from swift.conf and the input data), but i don't know the details of shss, for example | 21:12 |
| timburke_ | does anyone have any questions about the bug or its impact? | 21:13 |
| timburke_ | all right | 21:14 |
| mattoliverau | Nice investigation! | 21:14 |
| timburke_ | #topic SSYNC and non-durable frags | 21:15 |
| *** openstack changes topic to "SSYNC and non-durable frags (Meeting topic: swift)" | 21:15 | |
| timburke_ | #link https://bugs.launchpad.net/swift/+bug/1778002 | 21:15 |
| openstack | Launchpad bug 1778002 in OpenStack Object Storage (swift) "EC non-durable fragment won't be deleted by reconstructor. " [High,Confirmed] | 21:15 |
| timburke_ | i know acoles (and clayg?) has been working on this problem a bit lately, though i'm not sure where things stand | 21:15 |
| kota_ | shss might be impacted. i'll check it. | 21:15 |
| acoles | I just got my probe test working! | 21:16 |
| timburke_ | \o/ | 21:16 |
| acoles | background: we noticed some partitions were never cleaned up on handoffs | 21:16 |
| acoles | turned out they had non-durable data frags on them , so the dir would not be deleted | 21:16 |
| acoles | but reconstructor/ssync does not sync non-durable frags | 21:17 |
| acoles | :( | 21:17 |
| acoles | so https://review.opendev.org/c/openstack/swift/+/770047 should fix that | 21:17 |
| acoles | by (a) sync'ing non-durables (they could still be useful data) and (b) then removing non-durables on the handoff | 21:18 |
| clayg | https://bugs.launchpad.net/swift/+bug/1778002 has been around for awhile - anyone doing EC rebalances has probably noticed it | 21:18 |
| openstack | Launchpad bug 1778002 in OpenStack Object Storage (swift) "EC non-durable fragment won't be deleted by reconstructor. " [High,Confirmed] | 21:18 |
| zaitcev | Hrm. I never noticed because I have excess space. | 21:20 |
| timburke_ | i think we mainly noticed because we monitor handoffs as part of our rebalances | 21:21 |
| acoles | the commit message on the patch details the various changes needed to get the non-durables yielded to ssync and then have ssync sync them | 21:22 |
| timburke_ | acoles, are there any questions that might need answering, or is this something that everyone should just anticipate getting better Real Soon Now? | 21:22 |
| acoles | review always welcome, but there's no specific issue I have in mind for feedback | 21:23 |
| timburke_ | excellent | 21:23 |
| acoles | I'm about to push a new patchset - and I have one more test to write | 21:23 |
| timburke_ | #topic cleaning up shards when root DB is deleted and reclaimed | 21:25 |
| *** openstack changes topic to "cleaning up shards when root DB is deleted and reclaimed (Meeting topic: swift)" | 21:25 | |
| timburke_ | meanwhile, mattoliverau has picked up | 21:25 |
| timburke_ | #link https://bugs.launchpad.net/swift/+bug/1911232 | 21:25 |
| openstack | Launchpad bug 1911232 in OpenStack Object Storage (swift) "empty shards fail audit with reclaimed root db " [Undecided,Confirmed] - Assigned to Matthew Oliver (matt-0) | 21:25 |
| timburke_ | how's that going? | 21:26 |
| mattoliverau | Yeah things are moving along. I have https://review.opendev.org/c/openstack/swift/+/770529 | 21:26 |
| mattoliverau | it's not fixed yet, just worked on a probe test that shows he problem. | 21:26 |
| acoles | a very good place to start :) | 21:27 |
| mattoliverau | In an ideal world we'd have shrinking and autosharding so shards with nothing in them was suppose to collapse into the root before reclaim_age | 21:27 |
| mattoliverau | but we don't have that, and there is still an edge case where their not getting cleaned up. | 21:28 |
| mattoliverau | I'll have another pathset up today that should have an initial version of a fix. Currently still on my laptop as it needs some debugging and tests | 21:28 |
| mattoliverau | keep an eye out for that and then please review and we can make sure we don't leave any pesky shards around :) | 21:29 |
| timburke_ | sounds good | 21:29 |
| timburke_ | #topic s3api and allowable clock skew | 21:30 |
| *** openstack changes topic to "s3api and allowable clock skew (Meeting topic: swift)" | 21:30 | |
| timburke_ | i've had some clients getting back RequestTimeTooSkewed errors for a while -- not real common, but it's a fairly persistent problem | 21:31 |
| timburke_ | i'm fairly certain it's that they retry a failed request verbatim, rather than re-signing with the new request time | 21:31 |
| timburke_ | eventually, given the right retry/backoff options, the retry goes longer than 5mins and they get back a 403 | 21:32 |
| zaitcev | so, there's nothing we can do, right? | 21:33 |
| timburke_ | i *think* AWS has an allowable skew of more like 15mins (though can't remember whether i read it somewhere or determined it experimentally) | 21:33 |
| zaitcev | That's what I remember, too. | 21:34 |
| timburke_ | so i proposed a patch to make it configurable, with a default of (what i recall as being) AWS's limit | 21:34 |
| timburke_ | #link https://review.opendev.org/c/openstack/swift/+/770005 | 21:34 |
| zaitcev | It was mentioned in the old Developer's Guide. But that document is gone, replaced with API Reference. | 21:34 |
| timburke_ | i wanted to check if anyone had concerns about increasing this default value (it would of course be called out in release notes later) | 21:35 |
| kota_ | should we extend the default value too? | 21:35 |
| * kota_ said same thing :P | 21:36 | |
| timburke_ | kota_, yeah, the patch as written increases the timeout from 5mins to 15mins (if you don't explicitly set a value) | 21:36 |
| timburke_ | ok, seems like we're generally ok with it :-) | 21:37 |
| timburke_ | #topic relinker | 21:38 |
| *** openstack changes topic to "relinker (Meeting topic: swift)" | 21:38 | |
| timburke_ | i found a couple issues recently that might be good to know about if anyone's planning a part-power increase (or two) soon | 21:39 |
| timburke_ | #link https://bugs.launchpad.net/swift/+bug/1910589 | 21:39 |
| openstack | Launchpad bug 1910589 in OpenStack Object Storage (swift) "Multiple part power increases leads to misplaced data" [Undecided,New] | 21:39 |
| timburke_ | ^^^ characterizes something i think i mentioned last week, but hadn't gotten a clean repro for | 21:39 |
| timburke_ | rledisez, do you think you might have time to review https://review.opendev.org/c/openstack/swift/+/769855 (which should address it)? | 21:40 |
| zaitcev | Christian is no longer around essentially, we have to do without. | 21:40 |
| clayg | 😢 I hope he's doing well tho! 😁 | 21:41 |
| *** vishalmanchanda has quit IRC | 21:41 | |
| rledisez | timburke_: absolutely, I'll do that this week | 21:41 |
| timburke_ | thanks! only thing worth calling out (i think) is that the state file format changed in such a way that any old state files will just be discarded | 21:42 |
| rledisez | not a big deal. don't upgrade if you're relinking, and worst case sceniario, it restart from zero | 21:43 |
| timburke_ | but that should only really be a concern if someone is doing a swift upgrade mid-part-power-increase, which doesn't seem like a great plan anyway | 21:43 |
| *** baojg has quit IRC | 21:43 | |
| clayg | hahaha | 21:43 |
| timburke_ | the other one i noticed is a little thornier | 21:43 |
| timburke_ | #link https://bugs.launchpad.net/swift/+bug/1910470 | 21:44 |
| openstack | Launchpad bug 1910470 in OpenStack Object Storage (swift) "swift-object-relinker does not handle unmounted disks well" [Undecided,New] | 21:44 |
| *** baojg has joined #openstack-meeting | 21:44 | |
| timburke_ | essentially, on master, if the relinker hits an unmounted disk, you get no feedback about it at all | 21:44 |
| timburke_ | i've got a patch that at least has us log the fact that the disk is getting skipped -- https://review.opendev.org/c/openstack/swift/+/769632 | 21:45 |
| timburke_ | but it doesn't exit with a non-zero status code or anything | 21:45 |
| seongsoocho | So now, is it safe to increase the partition power only once until the patch is applied? | 21:46 |
| rledisez | seongsoocho: from production experience, it is. we did it on multiple clusters with the current status of the relinker | 21:47 |
| timburke_ | seongsoocho, yes, increasing it once will definitely be fine. once it's been increased, you could go manually clear the state files -- then it would be safe to do it again | 21:47 |
| rledisez | but you should care about the last bug mentioned by timburke_, ensure your rights are ok (root:root) on unmounted disk to avoid bad surprises | 21:48 |
| timburke_ | they'd be named something like /srv/node/*/.relink.*.json | 21:48 |
| seongsoocho | aha. ok thanks :) | 21:48 |
| rledisez | at some point, it would be useful to have a recon option that return the values of the relink.json and tells when one is missing (eg: because unmounted) | 21:49 |
| timburke_ | good thought! | 21:49 |
| *** TrevorV has quit IRC | 21:50 | |
| timburke_ | all right, i mostly wanted to raise awareness on those -- i'll let you know if i get a good idea on a better resolution for that second one | 21:50 |
| timburke_ | #topic open discussion | 21:50 |
| *** openstack changes topic to "open discussion (Meeting topic: swift)" | 21:50 | |
| timburke_ | what else should we talk about this week? | 21:51 |
| acoles | OMM I'm seeing this test fail in virtualenvs (e.g. tox -e py36) but not outside virtualenv: 'nosetests ./test/unit/common/test_manager.py:TestManagerModule.test_verify_server' - anyone else noticed that? I'm baffled | 21:52 |
| acoles | AFAICT the test is asserting that swift-Object-server is not on my PATH | 21:53 |
| acoles | note the capital 'O' | 21:53 |
| clayg | does it always fail? | 21:53 |
| acoles | inside virtualenv yes - I mean, I just noticed in last 20mins | 21:54 |
| clayg | are you on a case insenstive file system 🤣 | 21:54 |
| acoles | vsaio and macos both the same | 21:54 |
| acoles | apart from it failing, I don't like that a unit test is making assertions about what I might have on my PATH | 21:55 |
| clayg | oh, i just tried venv in my vsaio and it worked 🤷♂️ | 21:55 |
| acoles | if no-one else has noticed I'll dig some more | 21:55 |
| clayg | py3.8 tho | 21:55 |
| zaitcev | acoles: so, if you comment verify_server, does it fail? | 21:56 |
| zaitcev | or, well | 21:56 |
| zaitcev | it's a test, so it's a little artificial | 21:56 |
| acoles | py2.7 fails | 21:57 |
| zaitcev | se | 21:58 |
| zaitcev | a second | 21:58 |
| timburke_ | maybe related to https://review.opendev.org/c/openstack/swift/+/769848 ? | 21:58 |
| zaitcev | yes, that one | 21:58 |
| timburke_ | i should go review that... or maybe acoles should ;-) | 21:59 |
| *** rcernin has joined #openstack-meeting | 21:59 | |
| zaitcev | Yeah | 21:59 |
| timburke_ | all right | 21:59 |
| acoles | don't think so, in my virtualenvs, '$ which swift-Object-server' actually finds a match | 21:59 |
| acoles | :/ | 22:00 |
| timburke_ | O.o | 22:00 |
| zaitcev | Maybe just back out the whole thing. It's an option. But I hoped that just backing out the effects in the decorator, and _only_ screw with the exit code, would let use preserve it. | 22:00 |
| zaitcev | Oh | 22:00 |
| timburke_ | zaitcev, seems likely to be reasonable | 22:00 |
| timburke_ | acoles, well where did it come from!? | 22:01 |
| timburke_ | anyway, we're at time | 22:01 |
| acoles | I think there's a case-insensitivity thing going on in my virtualenvs ?!? v weird | 22:01 |
| timburke_ | thank you all for coming, and thank you for working on swift! | 22:01 |
| timburke_ | there's a lot going on, and i'm excited to see it all happening | 22:01 |
| timburke_ | #endmeeting | 22:01 |
| *** openstack changes topic to "OpenStack Meetings || https://wiki.openstack.org/wiki/Meetings/" | 22:01 | |
| openstack | Meeting ended Wed Jan 13 22:01:50 2021 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 22:01 |
| openstack | Minutes: http://eavesdrop.openstack.org/meetings/swift/2021/swift.2021-01-13-21.00.html | 22:01 |
| openstack | Minutes (text): http://eavesdrop.openstack.org/meetings/swift/2021/swift.2021-01-13-21.00.txt | 22:01 |
| openstack | Log: http://eavesdrop.openstack.org/meetings/swift/2021/swift.2021-01-13-21.00.log.html | 22:01 |
| *** dsariel has left #openstack-meeting | 22:02 | |
| *** zaitcev has left #openstack-meeting | 22:03 | |
| *** acoles has left #openstack-meeting | 22:14 | |
| *** jmasud has joined #openstack-meeting | 22:18 | |
| *** jmasud has quit IRC | 22:20 | |
| *** baojg has quit IRC | 22:52 | |
| *** baojg has joined #openstack-meeting | 22:53 | |
| *** timburke_ has quit IRC | 23:11 | |
| *** timburke_ has joined #openstack-meeting | 23:11 | |
| *** rh-jelabarre has quit IRC | 23:17 | |
| *** yamamoto has joined #openstack-meeting | 23:32 | |
| *** jmasud has joined #openstack-meeting | 23:33 | |
| *** baojg has quit IRC | 23:33 | |
| *** baojg has joined #openstack-meeting | 23:34 | |
| *** jmasud has quit IRC | 23:37 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!
