14:00:12 <rosmaita> #startmeeting cinder
14:00:13 <openstack> Meeting started Wed May 27 14:00:12 2020 UTC and is due to finish in 60 minutes.  The chair is rosmaita. Information about MeetBot at http://wiki.debian.org/MeetBot.
14:00:14 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
14:00:16 <openstack> The meeting name has been set to 'cinder'
14:00:20 <jungleboyj> o/
14:00:22 <rosmaita> #link https://etherpad.openstack.org/p/cinder-victoria-meetings
14:00:22 <rosmaita> #topic roll call
14:00:29 <rajinir> hi
14:00:30 <Liang__> o/
14:00:31 <eharney> howdy
14:00:31 <tosky> hi
14:00:37 <enriquetaso> hi
14:01:40 <e0ne> hi
14:01:52 <rosmaita> hello everyone
14:01:55 <rosmaita> let's get started
14:02:03 <rosmaita> #topic announcements - follow-up from last meeting
14:02:08 <m5z> hi =]
14:02:13 <rosmaita> after my pep talk about how exciting it would be to be liaison with the Interoperability Working Group
14:02:21 <rosmaita> i not only had no takers
14:02:28 <rosmaita> but i completely forgot to attend the meeting last friday
14:02:36 <rosmaita> however, they are meeting at the very first PTG session at 1300 UTC on 1 June
14:02:43 <rosmaita> so i will attend that
14:02:49 <rosmaita> and so i don't forget
14:02:55 <rosmaita> #action rosmaita attend Interop WG PTG session at 1300 UTC on 1 June
14:03:09 <rosmaita> i think that was the only open task from last week
14:03:17 <rosmaita> #topic announcements - Virtual PTG Logistics
14:03:26 <rosmaita> the virtual PTG is next week!
14:03:33 <rosmaita> #link http://lists.openstack.org/pipermail/openstack-discuss/2020-May/015054.html
14:03:40 <rosmaita> go ahead and open that link
14:03:53 <rosmaita> first link in that email is for registration
14:04:01 <rosmaita> if you haven't yet registered, open that link now so that you don't forget
14:04:12 <rosmaita> it's free, but it's also the only way you will get the zoom passwords to join
14:04:21 <rosmaita> we'll be using zoom (instead of Jitsi Meet) so that the sessions can be recorded
14:04:28 <whoami-rajat> hi
14:04:32 <lseki> hi
14:04:35 <rosmaita> everyone is using zoom, not just cinder
14:04:50 <rosmaita> the Cinder team will meet Tuesday-Friday 1300-1600 UTC in the *Diablo* room in zoom
14:04:58 <whoami-rajat> we didn't set it to cinder meeting ?
14:05:04 <rosmaita> so make sure you mark some time on your calendar
14:05:13 <whoami-rajat> and no reminder? i see less people
14:05:32 <rosmaita> i wonder if there's been a netsplit
14:05:37 <whoami-rajat> or i disconnected, then i'm sorry for above messages
14:05:47 <rosmaita> geguileo can confirm that i sent a reminder
14:05:50 <rosmaita> :)
14:06:05 <jungleboyj> Me too.  :-)
14:06:16 <rosmaita> whoami-rajat: you can look at the meeting log to catch up
14:06:17 <whoami-rajat> then i disconnected, sorry
14:06:20 <rosmaita> np
14:06:33 <jungleboyj> rosmaita:  Don't forget to include whoami-rajat   :-)
14:06:39 <jungleboyj> He doesn't like that.
14:07:05 <rosmaita> maybe that's why he didn't see the reminder -- i need to check my copy & paste
14:07:16 <whoami-rajat> :) reminders are really great
14:07:29 <rosmaita> ok, so that email has all the info you need
14:07:38 <rosmaita> #link http://lists.openstack.org/pipermail/openstack-discuss/2020-May/015054.html
14:07:53 <rosmaita> (for anyone who missed it the first time)
14:08:11 <rosmaita> #topic announcements - need a volunteer to be PTGbot handler
14:08:19 <rosmaita> first, i will say that this is way more fun than being interop WG liaison
14:08:28 <rosmaita> i tried to do this in Shanghai, but didn't always remember to update things, so i'd like someone to please commit to do this
14:08:38 <rosmaita> basically, all you have to do is keep the "now" and "next" topics for cinder updated
14:08:44 <e0ne> rosmaita: I can help with ptgbot
14:08:44 <rosmaita> you just do it over IRC, so not very strenuous
14:08:52 <rosmaita> sold!
14:09:01 <rosmaita> you can be as creative as you like to enhance the PTG experience for all attendees
14:09:05 <e0ne> :)
14:09:11 <rosmaita> but handling "now" and "next" is all that's required as far as i'm concerned
14:09:15 <rosmaita> thanks, e0ne
14:09:24 <rosmaita> that was easy
14:09:31 <rosmaita> #topic announcements - virtual happy hour at virtual PTG
14:09:38 <rosmaita> i will supply virtual beverages of your choice, if you want a real beverage, you will have to bring your own
14:09:48 <rosmaita> given the time zone spread of our team, i'm not sure this will be possible
14:09:51 <e0ne> hm... it's not fair!
14:09:57 <rosmaita> but if there's interest, i can put together a doodle poll
14:10:03 <rosmaita> or we can just hold it the last half hour of one of the cinder meeting slots
14:10:10 <rosmaita> it would be nice to have some informal virtual face-to-face time
14:10:16 <rosmaita> any thoughts?
14:10:26 <eharney> good plan
14:11:13 <rosmaita> anyone else interested?
14:11:39 <whoami-rajat> rosmaita++
14:11:52 <rosmaita> ok, i will put together a doodle poll later today
14:11:59 <rosmaita> look for an announcement on the ML
14:12:04 <e0ne> +1
14:12:11 <rosmaita> #action rosmaita doodle poll for virtual happy hour
14:12:20 <rosmaita> #topic announcements - Cinder virtual PTG schedule
14:12:31 <rosmaita> apologies, i got sidetracked on some other stuff this morning
14:12:38 <rosmaita> i'll get that done later today and send an announcement to the ML
14:12:51 <rosmaita> so if you have a topic you want to add, there is still time!
14:13:04 <rosmaita> #link https://etherpad.openstack.org/p/cinder-victoria-ptg-planning
14:13:14 <rosmaita> #action rosmaita - preliminary PTG schedule
14:13:37 <rosmaita> oh yeah, that reminds me, no Cinder meeting next week because we will all be at the PTG
14:14:08 <rosmaita> #action rosmaita cancel next week's cinder meeting
14:14:28 <rosmaita> that's all about the PTG -- any general questions or comments?
14:15:30 <rosmaita> ok, final announcement
14:15:38 <rosmaita> #topic announcements - cinder coresec
14:15:45 <rosmaita> I've asked whoami-rajat to join the cinder core security team
14:15:50 <rosmaita> he will replace jungleboyj, whose time priorities have shifted away from coresec
14:15:56 <rosmaita> but, many thanks to jungleboyj for his past work on the coresec team
14:16:02 <rosmaita> i will send an announcement to the ML
14:16:08 <rosmaita> mostly as a courtesy to the openstack vulnerability management team
14:16:14 <rosmaita> they prefer the coresec team to be small
14:16:22 <rosmaita> but since we are keeping it the same size as it is now, i don't think they'll have any objections
14:16:30 <jungleboyj> :-)  Welcome.  Thanks for whoami-rajat for helping out.  I was never good with security.
14:16:38 <eharney> sounds good to me
14:17:07 <rosmaita> it will extend our time zone reach so that we can work on security issues all around the clock
14:17:34 <whoami-rajat> thanks :)
14:17:35 <rosmaita> ok, on to the regular part of the meeting
14:17:43 <rosmaita> #topic gate check
14:17:50 <rosmaita> we're going to need to merge something to the Extended Maintenance branches early next week
14:17:55 <rosmaita> for cinder and os-brick
14:18:01 <rosmaita> so i thought i'd take a look at the last time something had merged there
14:18:07 <rosmaita> #link https://etherpad.opendev.org/p/cinder-gate-check-May2020
14:18:24 <rosmaita> turns out it was quite a while ago for ocata and pike
14:18:33 <rosmaita> so i proposed some DNM test patches to see what would happen
14:18:49 <rosmaita> and they are not looking good
14:19:15 <rosmaita> so i may need to grab some people for help if it's not clear what's going on
14:19:44 <rosmaita> one thing i still need to check is: compare the gate jobs with the check jobs
14:19:52 <rosmaita> because sometimes there are extra gate jobs
14:20:11 <rosmaita> if there's a difference, i may have to change these DNM patches to something to be merged
14:20:23 <rosmaita> right now, all they do is remove a '.' from a docstring
14:20:23 <smcginnis> Shouldn't be.
14:20:59 <rosmaita> i noticed releasenotes is a gate job but not a check job on one of these branches
14:21:10 <smcginnis> Gate jobs should be only what is in check, minus the non-voting jobs.
14:21:17 <smcginnis> If that's only running in gate, we should fix that.
14:21:38 <rosmaita> ok, smcginnis i may grab you later today so we can get these sorted out
14:21:52 <rosmaita> we will need the check & gate to be running smoothly
14:22:38 <rosmaita> i see that eharney has already noticed a specific problem
14:22:41 <rosmaita> thanks!
14:23:13 <eharney> on gate things...
14:23:30 <eharney> i've noticed for a bit now that the lio-barbican job seems to fail more often than others, but i haven't watched it enough to figure out why
14:23:58 <rosmaita> tempest-grenade also seems to be a culprit
14:24:06 <eharney> just to say, if anyone happens to notice anything in particular there, it'd be helpful
14:24:10 <whoami-rajat> yeah, it's been failing a lot recently
14:25:24 <rosmaita> ok, that's all from me ... stable cores, please keep a lookout, i will ping you in IRC if there's something that needs approval
14:25:47 <rosmaita> looks like we will continue the testing motif here
14:26:03 <rosmaita> #topic bump (WIP) of ceph in devstack-plugin-ceph (also for older branches)
14:26:11 <rosmaita> tosky: that's you
14:26:47 <tosky> vkmc is working to update the version of ceph deployed by devstack-plugin-ceph, which is ancient right now
14:26:53 <tosky> that will affect also the older branches
14:27:13 <tosky> #link https://review.opendev.org/#/c/676722/
14:27:39 <tosky> I created a test job against cinder-tempest-plugin, as we enable more services (namely c-bak)
14:27:47 <tosky> and it fails https://review.opendev.org/#/c/729766/
14:28:03 <tosky> probably related to the bump (the job usually passes)
14:28:07 <tosky> so please take a look at that
14:28:19 <eharney> do we have a general sense that the new version of ceph (nautilus?) will successfully run on whatever older nodes are used for some stable branches?
14:28:20 <tosky> especially people with ceph knowledge
14:28:26 <e0ne> tosky, vkmc: we shouldn't change ceph  version in stable branches
14:28:32 <tosky> vkmc if you are around and you want to comment
14:28:42 <vkmc> sure
14:29:24 <vkmc> so, we can specify the ceph version to be used in the stable branches
14:29:34 <rosmaita> oh, good
14:29:58 <tosky> I'd like to add that the failure happens in master, so that's a separate issue
14:30:25 <vkmc> not sure how it is for you, but in manila stable/queen ands stable/rocky we are running xenial with luminous
14:30:47 <vkmc> and stable/stein, stable/ussuri and stable/train we will bump to bionic with nautilus
14:30:49 <eharney> presumably the distro checks will need centos8 added soon as well
14:31:03 <vkmc> eharney, I'm planning to do so in a follow up patch
14:31:03 <e0ne> can we just create sable branches and leave them as is?
14:31:10 <vkmc> right now devstack doesn't work on centos8 at all
14:31:41 <tosky> vkmc: it does
14:31:42 <tosky> vkmc: https://review.opendev.org/#/c/685048/
14:31:46 <eharney> i thought it was pretty close if not done
14:31:57 * vkmc checks
14:32:20 <vkmc> tosky, that's great news, I was hitting various issues about two weeks ago
14:32:23 <vkmc> mostly networking
14:32:25 <tosky> (and if you want to vote for the review above ^^ which finally adds an LVM/lio/centos8 job, feel free to :)
14:32:45 <tosky> uhm, that job is from 3 weeks ago, strange
14:32:45 <vkmc> were there any changes on devstack recently to make this work?
14:33:12 <eharney> https://review.opendev.org/#/c/688614/
14:33:24 <vkmc> hmm ok, I'll rerun plain devstack and then see what changes we need to make the plugin work as well
14:33:44 <vkmc> eharney, 2 weeks ago
14:33:45 <vkmc> xD
14:34:01 <vkmc> awesome, so this is great news
14:34:01 <tosky> vkmc: one month (it was cherry-picked to ussuri)
14:34:04 <tosky> and we need to solve the c-bak failure
14:34:06 <vkmc> do you have ci based on centos8?
14:34:29 <vkmc> tosky, I haven't check c-bak issues again... do you have any clue on what's going on with that?
14:34:39 <eharney> we very much need a cinder centos8 ceph CI job :)
14:34:49 <vkmc> eharney, do you have a centos7 right now?
14:34:56 <eharney> i think not
14:35:00 <vkmc> k k
14:35:10 <whoami-rajat> the failure here https://review.opendev.org/#/c/729766/ is a timeout issue during restore on a step that i've seen before as some sort of deadlock
14:35:16 <vkmc> ok so I was planning to follow up con centos8 support
14:36:03 <tosky> I'd like to close with "please take a look at the other pending reviews on devstack-plugin-ceph", there are a few which are cinder-related too
14:36:16 <vkmc> ++
14:36:38 <vkmc> I want this change to move forward, so let's sync offline on any blockers you are seeing for cinder and any follow up work we expect to do
14:36:38 <rosmaita> ok, thanks, tosky and vkmc
14:36:52 <vkmc> thanks
14:37:16 <rosmaita> #topic volume local cache
14:37:21 <rosmaita> LiangFang: that's you
14:37:35 <LiangFang> sorry to bother you all every week
14:37:52 <rosmaita> we will eventually have to review this stuff if we want you to stop!
14:37:58 <LiangFang> if there's other topics, please go ahead first:)
14:38:08 <rosmaita> nothing else on the agenda
14:38:14 <rosmaita> go ahead
14:38:42 <LiangFang> I find tempest will fail because nova fail
14:38:58 <LiangFang> nova fail because it depends on os-brick
14:39:21 <LiangFang> it install os-brick like: pip install os-brick
14:39:26 <LiangFang> not pulling from repo
14:39:59 <LiangFang> it seems os-brick patch need to be merged first, and need to go into pipy
14:40:07 <tosky> that's the default behavior (libraries from pip), unless you add os-brick to required-projects (which adds it to LIBS_FROM_GIT)
14:41:05 <eharney> adding a Depends-On: from the nova change for the os-brick change doesn't fix that?
14:41:16 <rosmaita> we may need a patch to nova that adds an extra job to .zuul.yaml in nova that adds os-brick to required-projects
14:41:24 <rosmaita> a DNM patch
14:41:53 <rosmaita> but it would show us that the changes to brick are working with nova before we cut a release of os-brick for the "normal" nova job to use
14:42:02 <tosky> I forgot to comment on the review, but as it's a new optional feature, you may need to add a new configuration variable and only enable the tests when it's set
14:43:11 <rosmaita> tosky: would changing the required-projects and adding a Depends-on allow us to test the change with nova before merging it in os-brick?
14:43:18 <LiangFang> rosmaita: I will try that, thanks
14:44:24 <tosky> rosmaita: it should, but isn't it the idea of the DNM review you proposed above?
14:45:01 <rosmaita> the DNM would use os-brick master
14:45:33 <rosmaita> i think we need both
14:45:55 <rosmaita> anyway, LiangFang let us know what happens and we can further investigate
14:46:06 <eharney> fwiw, this has been a long-standing issue with os-brick CI coverage that could probably use a fresh look  (outside of this particular effort)
14:46:43 <rosmaita> eharney: "this" ==
14:47:04 <eharney> that we don't have enough jobs that test os-brick head, and we notice breakages after we do a release
14:47:11 <rosmaita> gotcha
14:47:28 <eharney> nova of course wants to test against a release, but something needs to test against head before we do a release
14:47:28 <rosmaita> i can propose some periodic jobs for that
14:47:57 <rosmaita> i will action myself to at least get a prototype patch up
14:48:26 <rosmaita> #action rosmaita prototype zuul job to test os-brick head against other projects before we cut a release
14:48:55 <rosmaita> LiangFang: anything else?
14:49:09 <LiangFang> no, thanks, i will try
14:49:18 <rosmaita> excellent, ty
14:49:26 <rosmaita> #topic Open Discussion
14:49:52 <enriquetaso> Hi o/
14:49:55 <tosky> eharney: the jobs with -src- in the name tests os-brick head
14:50:01 <tosky> they are defined inside os-brick
14:50:07 <enriquetaso> I'd like to ask for reviews on https://review.opendev.org/#/c/597148/
14:50:08 * tosky shuts up
14:50:25 <eharney> tosky: i think there are still gaps, but i need to go find some notes to see what they actually are now
14:50:52 <rosmaita> i'll take a look at the -src jobs before putting up a patch
14:51:06 <rosmaita> just so no one missed it
14:51:17 <rosmaita> enriquetaso would like reviews on:
14:51:24 <rosmaita> #link https://review.opendev.org/#/c/597148/
14:51:38 <enriquetaso> thanks
14:51:56 <rosmaita> very exciting, encrypted volume support for NFS
14:53:03 <eharney> someone needs to go write the glance_store patch to fail sensibly when encrypted nfs volumes show up
14:53:14 <eharney> i think whoami-rajat had this on his list
14:53:37 <whoami-rajat> eharney, yep
14:53:38 <eharney> currently, when we land this, bad things will happen in that path
14:54:06 <enriquetaso> #link https://review.opendev.org/#/c/697335/
14:54:19 <tosky> enriquetaso: do we need a new job in cinder-tempest-plugin instead of https://review.opendev.org/#/c/726924/ ?
14:55:00 <whoami-rajat> enriquetaso, that's different but also important
14:55:41 <enriquetaso> tosky, i think so. 726924 is just a test anyway
14:55:53 <whoami-rajat> also just as a reminder, the image volume cache feature doesn't work for nfs and is fixed here
14:55:58 <whoami-rajat> #link https://review.opendev.org/#/c/726471/
14:56:02 <enriquetaso> whoami-rajat, yep :P
14:56:07 <eharney> i was hoping that someone who knows how to write devstack code correctly would fix up 697335
14:56:36 <rosmaita> looks like the NFS driver is an endless source of merriment
14:56:46 <enriquetaso> whoami-rajat, cinder-tempest-plugin-lvm-lio-barbican fails with a weird error in https://review.opendev.org/#/c/726471/
14:56:56 <enriquetaso> looks like isn't related to the fix
14:57:06 <enriquetaso> rosmaita++
14:57:08 <whoami-rajat> eharney, it's fixed and i tested that on glance_store gate as well
14:57:45 <eharney> whoami-rajat: it does gross things  (that i wrote)
14:57:56 <whoami-rajat> eharney, https://review.opendev.org/#/c/329825/
14:58:22 <whoami-rajat> eharney, yeah, there was a time issue which i've left comments on the main patch to fix up
14:58:25 <enriquetaso> maybe if I removed the WIP from the title the devstack core team would review 697335
14:58:50 <eharney> the failure in 726471 is something bad going on in the LIO iscsi code
14:58:54 <eharney> i was just looking at a similar one earlier today
14:59:04 <eharney> that definitely needs some investigation
14:59:04 <whoami-rajat> enriquetaso, this error is everywhere!
14:59:20 <enriquetaso> :(
14:59:48 <whoami-rajat> enriquetaso, also do the changes i mentioned in the comments please
15:00:02 <rosmaita> enriquetaso: i think that's a good idea, remove the WIP from 697335  after you address whoami-rajat's comments
15:00:19 <rosmaita> i think WIP + -1 is preventing devstack-core from looking at it
15:00:31 <eharney> 697335 was marked WIP because it does some messy stuff moving files around that i wouldn't want to merge
15:00:38 <enriquetaso> sure
15:00:40 <jungleboyj> Time check.
15:00:44 <eharney> i guess people can go with it if they want, but i assume it's not right
15:00:57 <rosmaita> ok, out of time, make way for horizon team
15:01:01 <rosmaita> #endmeeting