| whoami-rajat | #startmeeting cinder | 14:01 |
|---|---|---|
| opendevmeet | Meeting started Wed Jan 24 14:01:45 2024 UTC and is due to finish in 60 minutes. The chair is whoami-rajat. Information about MeetBot at http://wiki.debian.org/MeetBot. | 14:01 |
| opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 14:01 |
| opendevmeet | The meeting name has been set to 'cinder' | 14:01 |
| whoami-rajat | #topic roll call | 14:01 |
| rosmaita | o/ | 14:01 |
| jungleboyj | o/ | 14:01 |
| simondodsley | o/ | 14:01 |
| akawai | o/ | 14:02 |
| inori | o/ | 14:02 |
| gireesh | hi | 14:02 |
| Saikumar | o/ | 14:03 |
| msaravan | Hi | 14:03 |
| whoami-rajat | #link https://etherpad.opendev.org/p/cinder-caracal-meetings | 14:03 |
| crohmann | o/ | 14:03 |
| whoami-rajat | let's get started | 14:04 |
| whoami-rajat | #topic announcements | 14:05 |
| whoami-rajat | first, Dalmatian release schedule | 14:05 |
| whoami-rajat | #link https://lists.openstack.org/archives/list/openstack-discuss@lists.openstack.org/thread/KS2WP242E4EQZ2GANROQMS4OHPQQ3ZJZ/ | 14:05 |
| whoami-rajat | #link https://review.opendev.org/c/openstack/releases/+/906050 | 14:05 |
| jayaanand | o/ | 14:06 |
| whoami-rajat | the D release schedule is proposed, If you are interested you can take a look and plan accordingly | 14:06 |
| whoami-rajat | next, RDO at CentOS Connect and FOSDEM | 14:06 |
| whoami-rajat | #link https://lists.openstack.org/archives/list/openstack-discuss@lists.openstack.org/thread/JVBVRUVZDEGQYSIBWVWQI4CBOYKHNQ3U/ | 14:07 |
| whoami-rajat | Opportunities to learn more about RDO in centos connect and FOSDEM in Brussels, BE | 14:07 |
| whoami-rajat | centos connect: January 31st and February 1st | 14:07 |
| whoami-rajat | FOSDEM: February 2nd and 3rd | 14:07 |
| whoami-rajat | this is in person event so people who are attending can keep a look out for RDO related sessions if you are eager to learn more about it | 14:07 |
| whoami-rajat | finally we have some upcoming milestones | 14:08 |
| whoami-rajat | Non-client library freeze: February 22nd, 2024 (R-6 week) | 14:08 |
| whoami-rajat | Client library freeze: February 29th, 2024 (R-5 week) | 14:08 |
| whoami-rajat | Caracal-3 milestone: February 29th, 2024 (R-5 week) | 14:08 |
| whoami-rajat | 2024.1 Caracal final release: April 3rd, 2024 | 14:08 |
| whoami-rajat | 2024.2 'D' virtual PTG ( https://openinfra.dev/ptg/ ): April 8th-12th, 2024 | 14:08 |
| whoami-rajat | that's all for announcements | 14:10 |
| whoami-rajat | there are some cinder related questions on the ML | 14:10 |
| whoami-rajat | which to my understanding were answered | 14:10 |
| whoami-rajat | but good to see people raising deployment specific issues and other members helping them out | 14:11 |
| rosmaita | \o/ | 14:12 |
| whoami-rajat | anyone has anything else for announcements ? | 14:13 |
| whoami-rajat | looks like not | 14:16 |
| whoami-rajat | let's get to the topics | 14:16 |
| whoami-rajat | #topic Ceph caps for Cinder / Glance | 14:16 |
| whoami-rajat | #link https://lists.openstack.org/archives/list/openstack-discuss@lists.openstack.org/message/5SVYDYOXWMG4TJKWEA6BFMPZZGC3Q5CS/ | 14:16 |
| whoami-rajat | I've been busy for the past couple of days but i will make a note to reply to it today | 14:17 |
| whoami-rajat | and please ping me tomorrow if i don't | 14:17 |
| whoami-rajat | sorry for the delay here | 14:17 |
| crohmann | thanks a bunch whoami-rajat! I will do the patches, but I'd rather have the proper caps down before I even push anything | 14:18 |
| whoami-rajat | I can check and provide the info about cinder caps, but do we have info from glance and nova team about their caps? | 14:18 |
| whoami-rajat | or you are planning to just do a cinder specific doc effort | 14:19 |
| crohmann | no. I'd rather have them all down. For operators it's just "a list of users + caps" | 14:19 |
| whoami-rajat | okay, i will add a reply from cinder side on it (and the team can correct me if something is not accurate) | 14:20 |
| crohmann | Could you maybe drag someone from Glance / Nova into the conversation on the ML to have this done all in one go? | 14:20 |
| crohmann | not one reply ... I mean in one effort / a few days. | 14:21 |
| rosmaita | crohmann: i will put something on the glance agenda, though i think the meeting may be cancelled this week | 14:22 |
| whoami-rajat | I can mention to ask the teams for their input and the benefit it will have on operators | 14:22 |
| rosmaita | crohmann: it would probably help to raise a bug and add all appropriate projects to it | 14:23 |
| crohmann | I will do that then. whoami-rajat do you want to hold off your reply until there is a bug then or use the ML firsT? | 14:24 |
| whoami-rajat | crohmann, sure, let me know once it's filed | 14:24 |
| crohmann | k. | 14:24 |
| whoami-rajat | thanks crohmann for keeping on top of it, anything else on this topic? | 14:25 |
| crohmann | not frome me. | 14:25 |
| whoami-rajat | okay, so we can move to the next topic | 14:26 |
| whoami-rajat | #topic Any update on the performance issues with cinder-backup and the chunked driver (e.g. S3) | 14:26 |
| whoami-rajat | #link https://bugs.launchpad.net/cinder/+bug/1918119 | 14:26 |
| whoami-rajat | crohmann, that's you again | 14:27 |
| crohmann | This is me again asking for a follow up / current state. | 14:27 |
| whoami-rajat | i don't like to ping zaitcev again and again but only he can answer it | 14:27 |
| crohmann | I'd like to know if this is treated as a real issue (S3 not being usably fast for production). | 14:28 |
| rosmaita | well, it's not not being treated as a real issue | 14:28 |
| rosmaita | pete is working on a few different backup items, and one of them has caused some slowdown due to database differences | 14:28 |
| rosmaita | so it's still on his radar, just probably not much progress this week | 14:29 |
| whoami-rajat | I haven't heard a lot of people use S3 for backups, most of the user survey answers mentions Ceph but happy to be corrected about it | 14:29 |
| rosmaita | he's in utc-6 time zone | 14:29 |
| crohmann | I did not mean to put this on him personally. The question was more of general nature about getting cinder-backup using the chunked driver to a different performance level | 14:29 |
| crohmann | whoami-rajat: Yeah. But if you look at the referenced but, Cern wanted to use S3, but it's simply too slow due to implementation inefficiencies. | 14:30 |
| crohmann | s/but/bug/ | 14:30 |
| crohmann | They even spoke about this in their talk ;-) | 14:30 |
| rosmaita | crohmann: slow because of the chunked driver, or slow on the S3 side? | 14:31 |
| crohmann | And it's not just "S3", but all non-rbd drivers that base on the chunked driver | 14:31 |
| crohmann | Since rbd is just a "rbd export | rbd import" with no data hitting Python at all it's a whole different ball game | 14:32 |
| crohmann | the "chunked driver" on the other hand does actively read "chunks" from the source volume and mangles them until they end up being uploaded to some (remote) storage. Be it S3, GCP or others | 14:33 |
| crohmann | s/GCP/GCS/ | 14:33 |
| crohmann | see https://bugs.launchpad.net/cinder/+bug/1918119 for some of the analysis we did to narrow down where the time is spent | 14:34 |
| whoami-rajat | so the improvement is not specific to S3 but any driver inheriting from chunked driver, this makes the effort more generalized | 14:34 |
| crohmann | Yes whoami-rajat | 14:34 |
| zaitcev | The performance of the chunkdriver is in my QC goals and literally is my only over-arching bug. So it's either that or my neck. | 14:35 |
| crohmann | The S3 driver is simply the last bit of the processing (the upload bit) to remote storage. The heavy-lifing was already done to the data at that point | 14:35 |
| rosmaita | sounds like zaitcev is motivated to work on this! | 14:36 |
| crohmann | Yeah. I did not mean to pull on the grass to make it grow any faster (as they say in German). | 14:36 |
| rosmaita | i have not heard that expression before, but i like it! | 14:37 |
| crohmann | In any case - zaitcev please let me know if you require any more input, testing, ideas ... | 14:37 |
| crohmann | That's all I wanted to discuss regarding this topic. | 14:37 |
| zaitcev | crohmann: sure, thanks | 14:37 |
| simondodsley | new Cinder motto - we don't pull on the grass... | 14:37 |
| zaitcev | Speaking of backup, we need to drum up +2 for https://review.opendev.org/c/openstack/cinder/+/886584 | 14:38 |
| jungleboyj | LOL | 14:38 |
| crohmann | (Yeah. I put that one and some more cinder-backup related quick fixed in the review list of this meeting) | 14:38 |
| zaitcev | oh | 14:39 |
| rosmaita | i will look at 886584 today | 14:39 |
| whoami-rajat | i think i reviewed it earlier, will take a look at the update | 14:40 |
| whoami-rajat | okay, anything else on this topic? | 14:40 |
| whoami-rajat | i will take the silence as no, let's move on to the next one then | 14:42 |
| whoami-rajat | #topic some CI updates | 14:42 |
| whoami-rajat | rosmaita, that's you | 14:42 |
| rosmaita | hello | 14:42 |
| rosmaita | current word on the street is that CI jobs have having odd timeouts | 14:43 |
| rosmaita | i guess one theory is that swapping is causing slowdowns | 14:43 |
| rosmaita | so, a patch merged recently to enable zswap on devstack runs | 14:43 |
| rosmaita | #link https://review.opendev.org/c/openstack/devstack/+/890693 | 14:44 |
| rosmaita | (that patch had a typo in the flag name, so there's a followup in the gate now: https://review.opendev.org/c/openstack/devstack/+/906504 ) | 14:44 |
| rosmaita | i think the plan is that nova will enable this on a few jobs to see what happens | 14:44 |
| rosmaita | i don't know if we want to get in on that action or not? | 14:45 |
| rosmaita | there are some limitations noted on the original patch's commit message | 14:45 |
| whoami-rajat | if it can make the situation worse, then maybe better to wait for nova to publish their findings | 14:46 |
| rosmaita | anyway, just wanted to bring this up in case anyone is interested in this | 14:46 |
| whoami-rajat | if it can only improve the situation, i don't have any issues | 14:46 |
| whoami-rajat | but I need to do a bit of reading on the concepts used here | 14:46 |
| simondodsley | seems to me zswap could only make things better | 14:46 |
| rosmaita | well, it's a bit weird because we're using RAM to store swap pages that are swapped out because of lack of RAM, it's just that they're compressed | 14:47 |
| rosmaita | so i think it's going to depend on what the pages are like, i.e., how compressible they are | 14:48 |
| rosmaita | but maybe i am being too naive | 14:48 |
| simondodsley | the nice thing about ZSWAP is that it can dynamically shrink when not needed, unlike SWAP | 14:48 |
| whoami-rajat | zswap basically trades CPU cycles | 14:48 |
| whoami-rajat | for potentially reduced swap I/O. This trade-off can also result in a | 14:48 |
| whoami-rajat | significant performance improvement if reads from the compressed cache are | 14:48 |
| whoami-rajat | faster than reads from a swap device. | 14:48 |
| whoami-rajat | that's what i got from the docs | 14:49 |
| rosmaita | it will be interesting to see what happens | 14:49 |
| whoami-rajat | i don't know how to determine the performance of reading from compressed cache | 14:50 |
| whoami-rajat | but let's see | 14:50 |
| rosmaita | clarkb has said that a lot of time is wasted in swap preparation (writing out zeros), so this should be faster | 14:50 |
| rosmaita | anyway, that's all from me | 14:51 |
| whoami-rajat | thanks for bringing this up rosmaita , we can keep an eye on it for any improvements in the gate situation (we surely need it) | 14:52 |
| whoami-rajat | that's all the topics we had for today | 14:53 |
| whoami-rajat | let's move to open discussion | 14:53 |
| whoami-rajat | #topic open discussion | 14:53 |
| inori | Hi, I'd like to highlight a patch I've been working on: https://review.opendev.org/c/openstack/cinder/+/896077 | 14:54 |
| inori | It recieved a +2 from whoami-rajat last week, I'm now seeking additional reviews from other core reviewers to merge it. Your input would be highly valuable. | 14:54 |
| inori | So could you please take a moment to review the patch at your earliest convenience? | 14:56 |
| whoami-rajat | it's a Fujitsu feature sitting for a long time, good to get that in | 14:57 |
| whoami-rajat | s/for/from | 14:57 |
| rosmaita | i'll take a look, though i share zaitcev's surprise that CLI gives you a speedup | 14:58 |
| rosmaita | but if that's what Fujitsu is seeing in their testing, what the heck | 14:58 |
| inori | Thx for your time. | 14:58 |
| rosmaita | we value empirical evidence in the cinder project | 14:58 |
| zaitcev | I'll look at it again. | 15:00 |
| inori | thank you zaitcev | 15:00 |
| whoami-rajat | i didn't see any performance numbers on the patch but if Fujitsu claims it for their driver, i can believe that | 15:00 |
| whoami-rajat | okay we are out of time | 15:00 |
| whoami-rajat | please take a look at the review request section | 15:01 |
| whoami-rajat | thanks everyone for joining | 15:01 |
| whoami-rajat | #endmeeting | 15:01 |
| opendevmeet | Meeting ended Wed Jan 24 15:01:07 2024 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 15:01 |
| opendevmeet | Minutes: https://meetings.opendev.org/meetings/cinder/2024/cinder.2024-01-24-14.01.html | 15:01 |
| opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/cinder/2024/cinder.2024-01-24-14.01.txt | 15:01 |
| opendevmeet | Log: https://meetings.opendev.org/meetings/cinder/2024/cinder.2024-01-24-14.01.log.html | 15:01 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!