14:01:45 <whoami-rajat> #startmeeting cinder 14:01:45 <opendevmeet> Meeting started Wed Jan 24 14:01:45 2024 UTC and is due to finish in 60 minutes. The chair is whoami-rajat. Information about MeetBot at http://wiki.debian.org/MeetBot. 14:01:45 <opendevmeet> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 14:01:45 <opendevmeet> The meeting name has been set to 'cinder' 14:01:48 <whoami-rajat> #topic roll call 14:01:52 <rosmaita> o/ 14:01:53 <jungleboyj> o/ 14:01:54 <simondodsley> o/ 14:02:08 <akawai> o/ 14:02:17 <inori> o/ 14:02:42 <gireesh> hi 14:03:14 <Saikumar> o/ 14:03:14 <msaravan> Hi 14:03:25 <whoami-rajat> #link https://etherpad.opendev.org/p/cinder-caracal-meetings 14:03:52 <crohmann> o/ 14:04:57 <whoami-rajat> let's get started 14:05:02 <whoami-rajat> #topic announcements 14:05:04 <whoami-rajat> first, Dalmatian release schedule 14:05:10 <whoami-rajat> #link https://lists.openstack.org/archives/list/openstack-discuss@lists.openstack.org/thread/KS2WP242E4EQZ2GANROQMS4OHPQQ3ZJZ/ 14:05:19 <whoami-rajat> #link https://review.opendev.org/c/openstack/releases/+/906050 14:06:27 <jayaanand> o/ 14:06:36 <whoami-rajat> the D release schedule is proposed, If you are interested you can take a look and plan accordingly 14:06:56 <whoami-rajat> next, RDO at CentOS Connect and FOSDEM 14:07:05 <whoami-rajat> #link https://lists.openstack.org/archives/list/openstack-discuss@lists.openstack.org/thread/JVBVRUVZDEGQYSIBWVWQI4CBOYKHNQ3U/ 14:07:16 <whoami-rajat> Opportunities to learn more about RDO in centos connect and FOSDEM in Brussels, BE 14:07:20 <whoami-rajat> centos connect: January 31st and February 1st 14:07:21 <whoami-rajat> FOSDEM: February 2nd and 3rd 14:07:46 <whoami-rajat> this is in person event so people who are attending can keep a look out for RDO related sessions if you are eager to learn more about it 14:08:46 <whoami-rajat> finally we have some upcoming milestones 14:08:47 <whoami-rajat> Non-client library freeze: February 22nd, 2024 (R-6 week) 14:08:47 <whoami-rajat> Client library freeze: February 29th, 2024 (R-5 week) 14:08:47 <whoami-rajat> Caracal-3 milestone: February 29th, 2024 (R-5 week) 14:08:47 <whoami-rajat> 2024.1 Caracal final release: April 3rd, 2024 14:08:47 <whoami-rajat> 2024.2 'D' virtual PTG ( https://openinfra.dev/ptg/ ): April 8th-12th, 2024 14:10:14 <whoami-rajat> that's all for announcements 14:10:21 <whoami-rajat> there are some cinder related questions on the ML 14:10:28 <whoami-rajat> which to my understanding were answered 14:11:03 <whoami-rajat> but good to see people raising deployment specific issues and other members helping them out 14:12:42 <rosmaita> \o/ 14:13:01 <whoami-rajat> anyone has anything else for announcements ? 14:16:08 <whoami-rajat> looks like not 14:16:15 <whoami-rajat> let's get to the topics 14:16:29 <whoami-rajat> #topic Ceph caps for Cinder / Glance 14:16:42 <whoami-rajat> #link https://lists.openstack.org/archives/list/openstack-discuss@lists.openstack.org/message/5SVYDYOXWMG4TJKWEA6BFMPZZGC3Q5CS/ 14:17:38 <whoami-rajat> I've been busy for the past couple of days but i will make a note to reply to it today 14:17:51 <whoami-rajat> and please ping me tomorrow if i don't 14:17:54 <whoami-rajat> sorry for the delay here 14:18:16 <crohmann> thanks a bunch whoami-rajat! I will do the patches, but I'd rather have the proper caps down before I even push anything 14:18:46 <whoami-rajat> I can check and provide the info about cinder caps, but do we have info from glance and nova team about their caps? 14:19:06 <whoami-rajat> or you are planning to just do a cinder specific doc effort 14:19:25 <crohmann> no. I'd rather have them all down. For operators it's just "a list of users + caps" 14:20:32 <whoami-rajat> okay, i will add a reply from cinder side on it (and the team can correct me if something is not accurate) 14:20:55 <crohmann> Could you maybe drag someone from Glance / Nova into the conversation on the ML to have this done all in one go? 14:21:24 <crohmann> not one reply ... I mean in one effort / a few days. 14:22:38 <rosmaita> crohmann: i will put something on the glance agenda, though i think the meeting may be cancelled this week 14:22:40 <whoami-rajat> I can mention to ask the teams for their input and the benefit it will have on operators 14:23:38 <rosmaita> crohmann: it would probably help to raise a bug and add all appropriate projects to it 14:24:06 <crohmann> I will do that then. whoami-rajat do you want to hold off your reply until there is a bug then or use the ML firsT? 14:24:17 <whoami-rajat> crohmann, sure, let me know once it's filed 14:24:38 <crohmann> k. 14:25:02 <whoami-rajat> thanks crohmann for keeping on top of it, anything else on this topic? 14:25:50 <crohmann> not frome me. 14:26:07 <whoami-rajat> okay, so we can move to the next topic 14:26:10 <whoami-rajat> #topic Any update on the performance issues with cinder-backup and the chunked driver (e.g. S3) 14:26:17 <whoami-rajat> #link https://bugs.launchpad.net/cinder/+bug/1918119 14:27:11 <whoami-rajat> crohmann, that's you again 14:27:13 <crohmann> This is me again asking for a follow up / current state. 14:27:35 <whoami-rajat> i don't like to ping zaitcev again and again but only he can answer it 14:28:06 <crohmann> I'd like to know if this is treated as a real issue (S3 not being usably fast for production). 14:28:25 <rosmaita> well, it's not not being treated as a real issue 14:28:56 <rosmaita> pete is working on a few different backup items, and one of them has caused some slowdown due to database differences 14:29:22 <rosmaita> so it's still on his radar, just probably not much progress this week 14:29:51 <whoami-rajat> I haven't heard a lot of people use S3 for backups, most of the user survey answers mentions Ceph but happy to be corrected about it 14:29:53 <rosmaita> he's in utc-6 time zone 14:29:56 <crohmann> I did not mean to put this on him personally. The question was more of general nature about getting cinder-backup using the chunked driver to a different performance level 14:30:35 <crohmann> whoami-rajat: Yeah. But if you look at the referenced but, Cern wanted to use S3, but it's simply too slow due to implementation inefficiencies. 14:30:41 <crohmann> s/but/bug/ 14:30:52 <crohmann> They even spoke about this in their talk ;-) 14:31:22 <rosmaita> crohmann: slow because of the chunked driver, or slow on the S3 side? 14:31:23 <crohmann> And it's not just "S3", but all non-rbd drivers that base on the chunked driver 14:32:20 <crohmann> Since rbd is just a "rbd export | rbd import" with no data hitting Python at all it's a whole different ball game 14:33:17 <crohmann> the "chunked driver" on the other hand does actively read "chunks" from the source volume and mangles them until they end up being uploaded to some (remote) storage. Be it S3, GCP or others 14:33:25 <crohmann> s/GCP/GCS/ 14:34:13 <crohmann> see https://bugs.launchpad.net/cinder/+bug/1918119 for some of the analysis we did to narrow down where the time is spent 14:34:25 <whoami-rajat> so the improvement is not specific to S3 but any driver inheriting from chunked driver, this makes the effort more generalized 14:34:52 <crohmann> Yes whoami-rajat 14:35:27 <zaitcev> The performance of the chunkdriver is in my QC goals and literally is my only over-arching bug. So it's either that or my neck. 14:35:31 <crohmann> The S3 driver is simply the last bit of the processing (the upload bit) to remote storage. The heavy-lifing was already done to the data at that point 14:36:04 <rosmaita> sounds like zaitcev is motivated to work on this! 14:36:29 <crohmann> Yeah. I did not mean to pull on the grass to make it grow any faster (as they say in German). 14:37:03 <rosmaita> i have not heard that expression before, but i like it! 14:37:31 <crohmann> In any case - zaitcev please let me know if you require any more input, testing, ideas ... 14:37:43 <crohmann> That's all I wanted to discuss regarding this topic. 14:37:43 <zaitcev> crohmann: sure, thanks 14:37:57 <simondodsley> new Cinder motto - we don't pull on the grass... 14:38:03 <zaitcev> Speaking of backup, we need to drum up +2 for https://review.opendev.org/c/openstack/cinder/+/886584 14:38:07 <jungleboyj> LOL 14:38:56 <crohmann> (Yeah. I put that one and some more cinder-backup related quick fixed in the review list of this meeting) 14:39:06 <zaitcev> oh 14:39:22 <rosmaita> i will look at 886584 today 14:40:15 <whoami-rajat> i think i reviewed it earlier, will take a look at the update 14:40:44 <whoami-rajat> okay, anything else on this topic? 14:42:32 <whoami-rajat> i will take the silence as no, let's move on to the next one then 14:42:37 <whoami-rajat> #topic some CI updates 14:42:39 <whoami-rajat> rosmaita, that's you 14:42:45 <rosmaita> hello 14:43:12 <rosmaita> current word on the street is that CI jobs have having odd timeouts 14:43:30 <rosmaita> i guess one theory is that swapping is causing slowdowns 14:43:55 <rosmaita> so, a patch merged recently to enable zswap on devstack runs 14:44:08 <rosmaita> #link https://review.opendev.org/c/openstack/devstack/+/890693 14:44:35 <rosmaita> (that patch had a typo in the flag name, so there's a followup in the gate now: https://review.opendev.org/c/openstack/devstack/+/906504 ) 14:44:50 <rosmaita> i think the plan is that nova will enable this on a few jobs to see what happens 14:45:11 <rosmaita> i don't know if we want to get in on that action or not? 14:45:34 <rosmaita> there are some limitations noted on the original patch's commit message 14:46:03 <whoami-rajat> if it can make the situation worse, then maybe better to wait for nova to publish their findings 14:46:05 <rosmaita> anyway, just wanted to bring this up in case anyone is interested in this 14:46:20 <whoami-rajat> if it can only improve the situation, i don't have any issues 14:46:42 <whoami-rajat> but I need to do a bit of reading on the concepts used here 14:46:47 <simondodsley> seems to me zswap could only make things better 14:47:39 <rosmaita> well, it's a bit weird because we're using RAM to store swap pages that are swapped out because of lack of RAM, it's just that they're compressed 14:48:05 <rosmaita> so i think it's going to depend on what the pages are like, i.e., how compressible they are 14:48:13 <rosmaita> but maybe i am being too naive 14:48:19 <simondodsley> the nice thing about ZSWAP is that it can dynamically shrink when not needed, unlike SWAP 14:48:52 <whoami-rajat> zswap basically trades CPU cycles 14:48:53 <whoami-rajat> for potentially reduced swap I/O. This trade-off can also result in a 14:48:53 <whoami-rajat> significant performance improvement if reads from the compressed cache are 14:48:53 <whoami-rajat> faster than reads from a swap device. 14:49:24 <whoami-rajat> that's what i got from the docs 14:49:39 <rosmaita> it will be interesting to see what happens 14:50:01 <whoami-rajat> i don't know how to determine the performance of reading from compressed cache 14:50:04 <whoami-rajat> but let's see 14:50:25 <rosmaita> clarkb has said that a lot of time is wasted in swap preparation (writing out zeros), so this should be faster 14:51:55 <rosmaita> anyway, that's all from me 14:52:56 <whoami-rajat> thanks for bringing this up rosmaita , we can keep an eye on it for any improvements in the gate situation (we surely need it) 14:53:17 <whoami-rajat> that's all the topics we had for today 14:53:32 <whoami-rajat> let's move to open discussion 14:53:36 <whoami-rajat> #topic open discussion 14:54:02 <inori> Hi, I'd like to highlight a patch I've been working on: https://review.opendev.org/c/openstack/cinder/+/896077 14:54:44 <inori> It recieved a +2 from whoami-rajat last week, I'm now seeking additional reviews from other core reviewers to merge it. Your input would be highly valuable. 14:56:31 <inori> So could you please take a moment to review the patch at your earliest convenience? 14:57:38 <whoami-rajat> it's a Fujitsu feature sitting for a long time, good to get that in 14:57:43 <whoami-rajat> s/for/from 14:58:19 <rosmaita> i'll take a look, though i share zaitcev's surprise that CLI gives you a speedup 14:58:36 <rosmaita> but if that's what Fujitsu is seeing in their testing, what the heck 14:58:42 <inori> Thx for your time. 14:58:58 <rosmaita> we value empirical evidence in the cinder project 15:00:12 <zaitcev> I'll look at it again. 15:00:35 <inori> thank you zaitcev 15:00:48 <whoami-rajat> i didn't see any performance numbers on the patch but if Fujitsu claims it for their driver, i can believe that 15:00:53 <whoami-rajat> okay we are out of time 15:01:01 <whoami-rajat> please take a look at the review request section 15:01:06 <whoami-rajat> thanks everyone for joining 15:01:07 <whoami-rajat> #endmeeting