14:00:40 <whoami-rajat> #startmeeting cinder 14:00:40 <opendevmeet> Meeting started Wed Aug 24 14:00:40 2022 UTC and is due to finish in 60 minutes. The chair is whoami-rajat. Information about MeetBot at http://wiki.debian.org/MeetBot. 14:00:40 <opendevmeet> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 14:00:40 <opendevmeet> The meeting name has been set to 'cinder' 14:00:48 <geguileo> hi! o/ 14:00:51 <eharney> hi 14:00:55 <whoami-rajat> #topic roll call 14:01:33 <jungleboyj> o/ kind of ... multi-tasking. 14:01:37 <rosmaita> o/ 14:01:46 <enriquetaso> hi 14:01:56 <felipe_rodrigues> hi 14:02:01 <HappyStacker> hi 14:02:35 <whoami-rajat> #link https://etherpad.openstack.org/p/cinder-zed-meetings 14:02:46 <HelenaDantas[m]> o/ 14:03:02 <luizsantos[m]> o/ 14:03:04 <raffaelacunha[m]> o/ 14:04:28 <whoami-rajat> good turnout today 14:04:30 <whoami-rajat> let's get started 14:04:35 <whoami-rajat> #topic announcements 14:04:49 <whoami-rajat> i think the first two announcements are by rosmaita , would you like to do it? 14:05:42 <rosmaita> sure 14:05:51 <whoami-rajat> great 14:06:15 <rosmaita> as agreed last week, we proposed to EOL cinderlib on stable/train and stable/ussuri 14:06:29 <rosmaita> #link https://lists.openstack.org/pipermail/openstack-discuss/2022-August/030013.html 14:06:47 <rosmaita> the release team is OK with the patches, but have put a hold to give people time to respond 14:06:56 <rosmaita> haven't heard anything in a week so far 14:07:26 <rosmaita> what will happen is that they will tag the branches with 'train-eol' or 'ussuri-eol' 14:07:31 <rosmaita> and then delete the branches 14:07:40 <rosmaita> so if you need them, you can always checkout the tag 14:07:51 <rosmaita> next item 14:08:12 <rosmaita> we agreed at the midcycle that third-party CI should run on all os-brick changes 14:08:25 <rosmaita> so there's an announcement to the ML: 14:08:36 <rosmaita> #link https://lists.openstack.org/pipermail/openstack-discuss/2022-August/030014.html 14:08:49 <rosmaita> no responses to that one either 14:09:08 <rosmaita> hopefully the driver maintainers have seen it 14:09:15 <whoami-rajat> to all vendors in the meeting ^ 14:09:30 <rosmaita> yes, we should point out to any new people 14:09:38 <geguileo> rosmaita: I realized something we'll have to merge and backport in Cinder for cinderlib... 14:10:04 <geguileo> rosmaita: because afaik the primary user of cinderlib is oVirt to attach RBD volumes, and with the removal of the keyring, we broke it 14:10:20 <geguileo> rosmaita: I have a patch to fix this in Cinder (will only work with Cinderlib) 14:10:26 <geguileo> so no security issue for Cinder 14:10:32 <geguileo> https://review.opendev.org/c/openstack/cinder/+/846054 14:10:36 <rosmaita> i thought we had merged that, or am i thinking of something else? 14:10:44 <geguileo> it's now failing on mypy, but it should be ok otherwise 14:10:59 <geguileo> rosmaita: I thought we had already merged it as well 14:11:16 <geguileo> rosmaita: but it turns out we didn't, so iirc right now cinderlib is broken for RBD 14:11:40 <geguileo> rosmaita: oooooh, wait, wait, we merged that 14:11:59 <geguileo> rosmaita: this is a second patch, for when the config is not under /etc/ceph (which is usually the case in cinderlib) 14:12:06 <rosmaita> ah, ok 14:12:17 <geguileo> anyway, just wanted to give the fyi on that one 14:12:39 <geguileo> we can continue, and once CI says it's ok I'll go around with the usual begging for reviews 14:12:42 <rosmaita> guess we need to prioritize getting cinderlib master open for zed development 14:12:46 <rosmaita> ok 14:12:58 <rosmaita> final thing i wanted to say to driver maintainers 14:13:03 <rosmaita> you may know this already 14:13:35 <rosmaita> but when we send stuff to the openstack-discuss mailing list about the cinder project, we put '[cinder]' in the subject line so you can filter easily 14:13:58 <rosmaita> for stuff that impacts driver maintainers, we also put '[cinder][drivers]' in the subject line 14:14:06 <rosmaita> so you should *really* pay attention to those 14:14:10 <whoami-rajat> rosmaita, would be good to get a current status on cinderlib Zed development (maybe later), I'm kind of not updated there 14:14:32 <rosmaita> ok, let's see if there's time during open discussion 14:14:40 <whoami-rajat> ack 14:14:51 <rosmaita> (i am not completely updated myself, tbh) 14:15:27 <rosmaita> though now i remember, so let's talk later 14:15:33 <whoami-rajat> I'm sure more updated than me ! 14:15:38 <geguileo> whoami-rajat: the tldr; is that it's broken 14:15:38 <whoami-rajat> sounds good 14:16:03 <whoami-rajat> geguileo, :( so it's in the state i last remember 14:16:17 <geguileo> whoami-rajat: yeah, I've been a bit busy with the unit tests mess 14:16:30 <rosmaita> busy indeed! 14:16:33 * whoami-rajat will pay more attention to cinderlib after M-3 14:17:01 <whoami-rajat> don't forget quotas! 14:17:17 <geguileo> whoami-rajat: oh I haven't!! 14:17:37 <whoami-rajat> :D 14:17:41 <rosmaita> geguileo has nightmares about quotas 14:17:42 <geguileo> (I have nightmares with it now) 14:17:47 <rosmaita> :) 14:17:50 <geguileo> rosmaita: you know me so well!! 14:18:27 <whoami-rajat> so we can discuss about cinderlib if we get time after the topics 14:18:58 <whoami-rajat> rosmaita, anything else from the first two announcements? 14:19:06 <whoami-rajat> s/from/for 14:20:18 <rosmaita> nope, i think that's all 14:20:44 <whoami-rajat> ok, thanks for the announcements 14:20:47 <whoami-rajat> next, os-brick release this week 14:20:58 <whoami-rajat> we've the os-brick release deadline tomorrow 14:21:11 <whoami-rajat> I've created an etherpad from which most of the changes have merged 14:21:16 <whoami-rajat> #link https://etherpad.opendev.org/p/zed-os-brick-patches 14:21:26 <whoami-rajat> I've left a -1 on the release patch for the meantime 14:21:46 <whoami-rajat> not all the patches are release critical but good to have them merged 14:22:08 <geguileo> this one would be good to have merged: https://review.opendev.org/c/openstack/os-brick/+/854281 14:22:29 <geguileo> that together with a cinder patch can prevent unit tests not returning to the shell 14:22:51 <whoami-rajat> geguileo, yep, i was reviewing it but the meeting started, will finish it after the meeting 14:22:56 <whoami-rajat> that sounds great! 14:22:59 <geguileo> thanks 14:23:00 <rosmaita> we should probably merge it, we're supposed to be supporting py 3.9, and that seems to be where it happens 14:23:16 <geguileo> rosmaita: this is a different blocking ;-) 14:23:28 <geguileo> there are 2 kinds of blockings rofl 14:23:42 <geguileo> because, why not! 14:24:06 <geguileo> this is a weird interaction of the garbage collector calling the close method of the wrapper 14:24:09 <rosmaita> geguileo is laughing because otherwise he would be crying 14:24:26 <geguileo> rosmaita: yup 14:25:06 <whoami-rajat> there is also a topic dedicated to UTs by geguileo later during the meeting 14:25:42 <whoami-rajat> next, Antelope PTG Registration 14:26:06 <whoami-rajat> by this time everyone might be aware that the Antelope PTG in october is going to be virtual 14:26:25 <whoami-rajat> but some people might not have registered before thinking it's in person and forgot about it later 14:26:40 <whoami-rajat> so for a reminder, please register for the upcoming PTG, it's virtual and free 14:26:41 <whoami-rajat> #link https://www.eventbrite.com/e/project-teams-gathering-october-2022-tickets-374118026087 14:27:55 <whoami-rajat> that's all for announcements 14:28:15 <whoami-rajat> let's move on to topics 14:28:28 <whoami-rajat> #topic Our Unit Tests are a broken clock 14:28:31 <whoami-rajat> nice heading 14:28:34 <whoami-rajat> geguileo, that's you 14:28:44 <geguileo> thanks 14:28:59 <geguileo> so basically it's by pure luck that our unit tests run successfully 14:29:07 <geguileo> we have all kicks of madness going on there 14:29:21 <geguileo> and it's easy to see by running tox -epy39 -- --random 14:29:40 <geguileo> if you can run that 3 times without a failure, you should play the lottery, because your luck is out of this world 14:30:02 <whoami-rajat> lol 14:30:06 <geguileo> besides the 10 to 15 seconds tests we have that abishop is working on 14:30:35 <geguileo> I have found that our test runs may just get stuck 14:30:48 <abishop> I think my patches to eliminate timed delays all merged 14:31:09 <geguileo> abishop: didn't know if you had more you were working on or not, thanks 14:31:21 <geguileo> today I hit the blocking issue at the gate https://zuul.opendev.org/t/openstack/build/73706405b8cb472982f4f09b056d3d09 14:31:31 <geguileo> that's on py38 14:31:40 <geguileo> and it shows as the CI job timing out 14:32:06 <geguileo> I have a whole series of patches (16 or so) fixing unit test issues 14:32:13 <geguileo> links on the etherpad 14:32:41 <geguileo> I just want to point out 4 things here for reviewers and coders 14:33:26 <geguileo> - Look at the results of the unit tests jobs when new UTs are being added to confirm there are no >1sec new test 14:34:05 <geguileo> - Make sure that data in 'ddt.data(...)' are static values, otherwise those tests are not executed 14:34:19 <geguileo> example: https://review.opendev.org/c/openstack/cinder/+/854262) 14:34:22 <enriquetaso> oh 14:35:05 <geguileo> - Use self.mock_object and self.patch and NEVER, EVER replace things with something.another = mock.Mock() 14:35:25 <geguileo> Because when that happens on a library you'll break tests that come afterwards 14:36:10 <geguileo> - Preferrably don't use global variables (or class attributes) with data that is going to be used in the tests unless they are not going to be modified 14:36:34 <geguileo> otherwise even if you usually use copy and deepcopy, you will eventually forget somewhere and make your tests flaky 14:36:43 <geguileo> as success will depend on the order in which they are run 14:36:59 <geguileo> A lot of examples of these last 2 in the patches I have proposed 14:37:23 <geguileo> One of the patches proposes adding --random to how we run stestr 14:37:33 <geguileo> to try to detect these sooner 14:38:22 <geguileo> oh, and for anyone that uses 'stestr run --load-list XYZ' to reproduce a gate job tests in the same order 14:38:30 <geguileo> they should know that that is currently broken in stestr 14:39:14 <geguileo> I have proposed a patch to fix it: https://github.com/mtreinish/stestr/pull/331 14:39:50 <geguileo> and that's all I wanted to complain about 14:40:33 <whoami-rajat> there are a series of patches on the etherpad that geguileo mentioned fixes our UTs so please review 14:40:37 <rosmaita> i don't think it counts as complaining when you have posted all those patches fixing stuff 14:41:14 <whoami-rajat> rosmaita, +1 14:41:21 <whoami-rajat> thanks for all the work on this geguileo ! 14:41:41 <geguileo> fingers cross that those are the only big issues for a while 14:42:11 <geguileo> s/cross/crossed 14:43:06 <tosky> geguileo: just one question: stestr is pinned in requirements.git for each release 14:43:34 <geguileo> tosky: I don't know WHEN they broke that feature 14:43:54 <tosky> so what does it mean for older releases? I guess the unit test fixes are still valid to be backported, but the older branches won't benefit much 14:44:01 <tosky> or at least partially 14:44:17 <geguileo> in my opinion for older branches I wouldn't backport the fixes 14:44:25 <geguileo> unless they are problematic for another backport 14:44:26 <tosky> uhm, the "when it broke" is a good question, I hope stestr maintainers could help 14:44:42 <geguileo> tosky: well, the feature that broke is not related to all those fixes 14:44:57 <geguileo> tosky: I think I wasn't clear on what is broken in stestr 14:45:17 <geguileo> so when you run tox for unit tests, or when it runs at the gate and it fails 14:45:23 <geguileo> but you run it locally and it doesn't 14:45:34 <geguileo> it's usually because tests are executed in a different order 14:45:56 <geguileo> so it's a problem with some test that is not mocking things properly and breaks later tests 14:46:10 <geguileo> the way to reproduce this locally is not download the subunit file from the gate 14:46:21 <geguileo> see the worker number that has failed in the logs 14:46:35 <geguileo> extract that worker tests into a file 14:46:38 <tosky> oh, but then wouldn't it make sense to backport them then? 14:46:43 <geguileo> and tell stestr to load that file and run it in that order 14:47:05 <geguileo> and that is the part that's broken in stestr, that it doesn't preserve the order of the list 14:47:15 <geguileo> so you can't reproduce the issues at the gate 14:47:32 <geguileo> so it doesn't affect our gates 14:47:40 <geguileo> (the stestr bug) 14:47:51 <geguileo> only our avility to reproduce the issues to fix things 14:48:15 <eharney> i suspect we'll want to backport some, like https://review.opendev.org/c/openstack/cinder/+/854259/ 14:48:38 <geguileo> eharney: yeah, probably the generic 3 generic ones 14:48:39 <eharney> many of them probably could be ignored for backport until obviously needed, as they've sat and run in the gates for years, mostly working 14:48:42 <geguileo> (first 3) 14:48:54 <geguileo> eharney: +1 14:49:05 <geguileo> if it's not broken, don't touch it ;-) 14:52:19 <geguileo> if there's nothing else, we can continue or finish 14:52:40 <whoami-rajat> we've open discussion :) 14:52:44 <whoami-rajat> so that's all the topics had for today, let's move to open discussion 14:52:48 <whoami-rajat> #topic open discussion 14:53:03 <whoami-rajat> one thing i wanted to mention, we've the feature freeze next week so make sure to review the feature patches 14:53:11 <whoami-rajat> #link https://etherpad.opendev.org/p/cinder-zed-features 14:54:16 <whoami-rajat> rosmaita, would you like to provide an update on cinderlib or we can discuss that later? 14:55:00 <rosmaita> real quick, here's what's going on 14:55:23 <rosmaita> current cinderlib master is running gate CI against cinder and os-brick stable/yoga 14:55:51 <rosmaita> the patch to open it to zed development will run it against cinder/os-brick master 14:55:54 <rosmaita> and that is failing 14:56:04 <rosmaita> mostly due to database changes, i believe 14:56:56 <rosmaita> so if you propose a patch to master right now, i think the CI will be green 14:57:06 <rosmaita> but that's not a "real" green CI 14:58:25 <rosmaita> sorry, got distracted for a minute 14:58:56 <rosmaita> this is the patch opening cinderlib for zed development 14:58:59 <rosmaita> #link https://review.opendev.org/c/openstack/cinderlib/+/848846 15:00:13 <rosmaita> anyway, we could merge the rbd keyring fix to master now and backport it as necessary 15:00:31 <rosmaita> it would just be a little weird 15:00:32 <geguileo> rosmaita: oh, that's a cinder patch 15:00:42 <geguileo> but it's for cinderlib 15:00:43 <rosmaita> oh, then there's no problem wiht that 15:00:46 <geguileo> yup 15:00:59 <rosmaita> ok, good, because i wasn't liking what i suggested there 15:01:08 <rosmaita> can we delete the last 10 lines from teh meeting log? 15:01:48 <whoami-rajat> doesn't look doable but we can end meeting quickly 15:02:18 <whoami-rajat> we're already out of time 15:02:22 <rosmaita> yes, please do, before i say anything else stupid 15:02:34 <whoami-rajat> thanks for the update rosmaita 15:02:37 <whoami-rajat> and thanks everyone for attending 15:02:44 <whoami-rajat> #endmeeting