16:00:06 <smcginnis> #startmeeting Cinder
16:00:10 <smcginnis> ping dulek duncant eharney geguileo winston-d e0ne jungleboyj jgriffith thingee smcginnis hemna xyang1 tbarron scottda erlon rhedlind jbernard _alastor_ bluex karthikp_ patrickeast dongwenjuan JaniceLee cFouts Thelo vivekd adrianofr mtanino karlamrhein diablo_rojo jay.xu jgregor lhx_ baumann rajinir wilson-l reduxio wanghao thrawn01 chris_morrell watanabe.isao,tommylikehu mdovgal ildikov wxy
16:00:14 <openstack> Meeting started Wed May 17 16:00:06 2017 UTC and is due to finish in 60 minutes.  The chair is smcginnis. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:00:15 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
16:00:16 <smcginnis> viks ketonne abishop s
16:00:17 <xyang1> Hi
16:00:18 <openstack> The meeting name has been set to 'cinder'
16:00:18 <geguileo> hi!
16:00:21 <tommylikehu> hi
16:00:22 <lhx_> hi
16:00:23 <rawanh> Hi !
16:00:28 <hemna> mep
16:00:45 <arnewiebalck_> hi
16:00:50 <jungleboyj> o/
16:01:00 <DuncanT> Hi
16:01:06 <smcginnis> DuncanT! :)
16:01:15 <Swanson> Hello.
16:01:15 <jungleboyj> It's DuncanT !!!
16:01:17 <tommylikehu> hey DuncanT :)
16:01:26 <abishop> o/
16:01:33 * DuncanT waves
16:01:33 * smcginnis imagines DuncanT walking into the Cheers bar
16:01:38 <DuncanT> :-)
16:01:41 <diablo_rojo> Hello :)
16:01:47 <Swanson> I thought DuncanT was dead.
16:01:59 * mdbooth can't imagine the Cheers bar serves anything DuncanT would drink?
16:02:28 <scottda> hi
16:02:30 <patrickeast> o/
16:02:32 <DuncanT> mdbooth: It's a real bar now, isn't it?
16:02:39 <DuncanT> mdbooth: I shall have to go check
16:02:45 <jungleboyj> mdbooth:  Ah, he isn't that picky when he is thirsty.  ;-)
16:02:46 <smcginnis> DuncanT: It's actually in Boston.
16:02:47 <mdbooth> DuncanT: +1 ;)
16:02:56 <smcginnis> But I heard the inside isn't anything like the TV show.
16:03:00 <smcginnis> Anyway...
16:03:06 <smcginnis> #topic Announcements
16:03:09 <xyang> smcginnis: 80+/90+ degrees here in Boston this week now that the summit is over:)
16:03:09 <jungleboyj> smcginnis:  That was what I heard too.  :-(
16:03:19 <smcginnis> xyang: Wow!
16:03:24 <smcginnis> #link https://etherpad.openstack.org/p/cinder-spec-review-tracking Review focus
16:03:36 <smcginnis> I've seen some progress being made for some of the new drivers.
16:03:44 <xyang> smcginnis: switching from heat to AC:)
16:03:52 <smcginnis> Let's give them some attention before the last couple days before the P-2 milestone.
16:04:00 <ildikov> xyang: can confirm that, weather got awesome by the time I flew out... :/ :)
16:04:09 <smcginnis> Too hot for me. :)
16:04:13 <xyang> ildikov :)
16:04:37 <smcginnis> Also wanted to bring up there is an intermittent issue with py27 tests completely falling apart.
16:04:51 <e0ne> hi
16:04:53 <smcginnis> Some work being done, mostly by eharney, to try to troubleshoot that.
16:05:00 <smcginnis> But if anyone has any ideas, we could use the help.
16:05:19 <tommylikehu> smcginnis:  would like to have a try
16:05:20 <jungleboyj> ildikov: 90 degrees is awesome?
16:05:37 <smcginnis> tommylikehu: That would be great. And very appreciated.
16:05:48 <smcginnis> I'd love to know what's going on there and get rid of those failures.
16:05:52 <smcginnis> #topic Reproducible DB deadlocks when doing parallel deletions
16:06:00 <hemna> rm -rf tests
16:06:03 <hemna> fixed!
16:06:03 <smcginnis> arnewiebalck_: Hey
16:06:07 <smcginnis> hemna: ;)
16:06:11 <arnewiebalck_> smcginnis: hi
16:06:21 <smcginnis> #link https://bugs.launchpad.net/cinder/+bug/1685818
16:06:23 <openstack> Launchpad bug 1685818 in Cinder "Parallel deletion triggers DB deadlocks " [Medium,Triaged]
16:06:30 <smcginnis> #link https://bugs.launchpad.net/cinder/+bug/1613947
16:06:35 <openstack> Launchpad bug 1613947 in Cinder "Cinder Volume deadlock in quota_reserve and reservation_commit" [Medium,New]
16:06:40 <arnewiebalck_> as mentioned during the summit, help with this prob would be appreciated
16:06:52 <smcginnis> Speaking of failures I'd love to see go away. :)
16:07:00 <jungleboyj> smcginnis: ++
16:07:08 <smcginnis> I still can't get it to happen with LVM.
16:07:17 <smcginnis> arnewiebalck_: You've seen this with ceph, right?
16:07:22 <arnewiebalck_> smcginnis: yes
16:07:31 <hemna> can we reproduce this with the fake driver ?
16:07:31 <arnewiebalck_> smcginnis: couldn’t reproduce on devstack
16:07:43 <smcginnis> arnewiebalck_: Oh really? Devstack with ceph?
16:07:45 <arnewiebalck_> smcginnis: on devstack with LVM
16:07:50 <smcginnis> Ah, OK.
16:07:53 <jungleboyj> I brought this up at my Scrum today and have opened a work item to look at this on our Ceph cluster.
16:08:00 <arnewiebalck_> smcginnis: we can hook a devstack to ceph
16:08:19 <arnewiebalck_> but since it is 100% reproducible on our test instance (and in prod) ...
16:08:19 <smcginnis> For folks with access to other external backends, would be great if you could try and see if you can hit this.
16:08:34 <smcginnis> Repro steps in the bug report.
16:08:45 <arnewiebalck_> some other folks apparently see the saem issue as the bug got some “upvotes” :)
16:08:48 <jungleboyj> smcginnis arnewiebalck_  So, I am hoping I can recreate it and then we can work together on debugging.
16:08:49 <smcginnis> My guess is the external communication changes the timing, but just a hunch.
16:08:56 <smcginnis> jungleboyj: ++
16:08:58 <hemna> can we reproduce this with the fake driver ?
16:09:02 <arnewiebalck_> jungleboyj: that’d be great
16:09:13 <jungleboyj> :-)
16:09:28 <arnewiebalck_> any suggestions what I could try to gather more information
16:09:34 <arnewiebalck_> I know which lock it is
16:09:47 <smcginnis> hemna: That could make it a lot easier to troubleshoot. Haven't tried that.
16:09:50 <arnewiebalck_> as when I remove it, the problem goes away :-D
16:10:09 <hemna> since this isn't really related to the driver, but a timing issue
16:10:18 <hemna> fake driver would just noop
16:10:32 <smcginnis> I'm guessing it's a timing issue, but we need more data I think.
16:10:42 <rawanh> Hey guys, I'm wondering if you have plans for NVME in the coming pike release ?
16:11:39 <smcginnis> rawanh: No one has submitted a driver, if that's what you're asking.
16:11:57 <smcginnis> arnewiebalck_: OK, anything else on this? Or just putting up the "help wanted" sign?
16:11:58 <xyang> e0ne: any comment?
16:12:10 <jungleboyj> rawanh: I have gotten questions about that but no one has proposed to add it.
16:12:16 <xyang> e0ne: I thought you and Michal did some work on that
16:12:20 <smcginnis> We're almost to open discussion. Let's stay on topic.
16:12:27 <hemna> rawanh, not sure what cinder has to do to support NVME
16:12:35 <e0ne> xyang, rawanh: IMO, it will be done not earlier than Q :(
16:12:38 <jungleboyj> smcginnis:  Sorry.
16:12:41 <hemna> it's just a local attached storage.  you could use lvm on it.
16:12:42 <arnewiebalck_> smcginnis: no, that’s basically it, I simply don’t know the quota/reservation part ofthe ode, so debugging is slow
16:12:57 <jungleboyj> smcginnis:  Do we have anyone who knows that code well?
16:12:59 <smcginnis> arnewiebalck_: OK, thanks again for sticking with this. It's appreciated.
16:12:59 <xyang> e0ne: I thought you have a driver somewhere already?
16:13:05 <e0ne> hemna: there was an idea to use nmve over fabric
16:13:09 <smcginnis> jungleboyj: Not sure, but if they do, please step up. :)
16:13:20 <e0ne> xyang: we stucked with snapshots impl and CI
16:13:26 <smcginnis> We're almost to open discussion. Let's stay on topic please.
16:13:29 <xyang> e0ne: ok
16:13:37 <smcginnis> #topic Bug smash event taking place
16:13:47 <smcginnis> #link https://etherpad.openstack.org/p/OpenStack-Bug-Smash-Pike-Suzhou-Bug-List Bug list
16:13:48 <arnewiebalck_> so no suggestions what to I could try to get further to the bottom of this?
16:14:17 <smcginnis> arnewiebalck_: Hoping others (Jay?) can repro and help troubleshoot.
16:14:18 <rawanh> as @e0ne commited before in your cinder, nova and os-brick, but you've used spdk with it, is there a reason why it hasn't been merged to openstack then ?
16:14:18 <hemna> e0ne, nvme over fabric kinda doesn't make sense.   the entire purpose of nvme is fast local attached storage.
16:14:28 <smcginnis> Seriously?
16:14:32 <arnewiebalck_> smcginnis: ok, thx
16:14:39 * jungleboyj is laughing
16:15:02 * smcginnis starts tapping his kickban stick on the table...
16:15:03 <smcginnis> :P
16:15:12 <xyang> smcginnis: are you traveling to Suzhou this time?
16:15:16 <e0ne> hemna: some customers want use it with over RDMA
16:15:21 <smcginnis> No, couldnt' go this time.
16:15:23 <jungleboyj> arnewiebalck_ smcginnis  Right.  I will try to fit that in soon.
16:15:26 <smcginnis> It started yesterday.
16:15:39 <arnewiebalck_> jungleboyj: great, thx
16:15:42 <smcginnis> So there are a team of devs actively working on bugs and looking for feedback.
16:15:48 <hemna> e0ne, couldn't you do it with just LIO exporting over FC?
16:15:54 <jungleboyj> arnewiebalck_:  Welcome!
16:15:55 <smcginnis> So please help out with reviews and and suggestions if you can.
16:15:58 <hemna> e0ne, anyway side convo.
16:16:09 <smcginnis> That etherpad has what they are currently looking at.
16:16:16 <e0ne> hemna: I'm not sure what is faster
16:16:25 <tommylikehu> smcginnis: I can do some help when you guys are sleeping
16:16:27 <smcginnis> If we can help move any of those along, I'm sure it will be very much appreciated.
16:16:41 <smcginnis> tommylikehu: Just don't forget to sleep too. ;)
16:16:48 <jungleboyj> smcginnis:  Link?
16:16:49 <tommylikehu> smcginnis:  :)
16:17:05 <tommylikehu> #link https://etherpad.openstack.org/p/OpenStack-Bug-Smash-Pike-Suzhou-Bug-List
16:17:10 <smcginnis> jungleboyj: Etherpad linked above between all the NVMe comments. :)
16:17:15 <jungleboyj> tommylikehu:  Thank you.
16:17:31 <smcginnis> OK, the moment you've all been waiting for...
16:17:36 <smcginnis> #topic Open discussion
16:17:40 <tommylikehu> lol
16:17:45 <smcginnis> NVMe, NVMe, NVMe!!
16:17:48 <jungleboyj> smcginnis:  Oh, there it is.  Missed that with all the excitement.
16:18:08 <smcginnis> Nothing now? I'm disappointed. :D
16:18:09 <jungleboyj> What should we talk about now?
16:18:17 <hemna> versions
16:18:30 <smcginnis> hemna: Oh yeah, your PSA> :)
16:18:35 <hemna> heh
16:19:11 <hemna> Ok so for driver developers, It's a good idea when you make major changes, new features available in your drivers, to make sure you bump the VERSION string in your driver.
16:19:21 <hemna> that version string is dumped into the cinder log file at startup
16:19:37 <hemna> that is used by folks later on to debug issues in deployments.
16:20:00 <hemna> and can be put into bug reports to help driver maintainers know what the codebase is of the driver.
16:20:03 <lhx_> http://status.openstack.org/openstack-health/#/job/periodic-cinder-py27-with-oslo-master
16:20:14 <lhx_> there are some error tests
16:20:26 <lhx_> but I couldn't find the reason
16:20:27 <hemna> so I've been -1'ing reviews on drivers that don't update their VERSION string and add a comment block, when new features are added and major refactors are done.
16:20:31 <tommylikehu> hemna:  any guideline  or links ?
16:20:32 <lhx_> someone help?
16:20:40 <hemna> tommylikehu, no, not really.
16:20:53 <hemna> maybe add something to the wiki ?
16:21:11 <tommylikehu> hemna:  that would be great :)
16:21:21 <hemna> it's just good s/w development practice to version your code for support later.   that's all really.
16:21:47 <sivn_> Can we talk about the new driver "veritas access driver" https://review.openstack.org/#/c/409914/
16:21:49 <hemna> remember, most deployments of openstack are on very old releases.
16:21:51 <smcginnis> hemna: Adding a comment to the wiki could help.
16:21:56 <DuncanT> hemna: Even for fairly minor bug fixes it is worth bumping the version...
16:22:04 <hemna> DuncanT, yup.
16:22:12 <DuncanT> hemna: It's one way to see if a fix really worked
16:22:13 <hemna> if you take a look at the 3PAR driver, that's what I've done over the years.
16:22:14 <Swanson> I'm going to bump it on every push.
16:22:20 <jungleboyj> DuncanT:  ++
16:22:22 <Swanson> Major only.
16:22:23 <tommylikehu> hemna: thanks
16:22:28 <hemna> it makes it so much easier for people trying to deal with support issues.
16:22:51 <hemna> https://github.com/openstack/cinder/blob/master/cinder/volume/drivers/hpe/hpe_3par_fc.py#L66-L113
16:22:53 <hemna> for example.
16:23:31 <smcginnis> Gets nicely formatted in the output too if you format it right: https://docs.openstack.org/developer/cinder/drivers.html
16:23:49 <hemna> oh nice
16:23:55 <hemna> https://docs.openstack.org/developer/cinder/drivers.html#hpe3parfcdriver
16:24:07 <lhx_> smcginnis, have you seen this?http://status.openstack.org/openstack-health/#/job/periodic-cinder-py27-with-oslo-master
16:24:14 <tommylikehu> that's clear, just a feature history and announce the current version
16:24:37 <smcginnis> lhx_: Yes, see beginning of meeting.
16:25:03 <jungleboyj> For reviewers, please enforce the versioning.
16:25:12 <hemna> anyway, just my PSA.   If anyone objects to me -1 reviews for not bumping versions for new features/major changes.....speak now :)
16:25:23 <hemna> times up.
16:25:24 <hemna> :P
16:25:32 <jungleboyj> @!
16:25:33 <pewp> jungleboyj (╯°□°)╯︵ ┻━┻
16:25:38 <hemna> @!h
16:25:38 <pewp> hemna (/ .□.) ︵╰(゜Д゜)╯︵ /(.□. )
16:25:38 <Swanson> First they came for the micro versions and I said nothing. Now they've come for the versions.
16:25:40 <lhx_> smcginnis, you discussed too quick to catch it, lol
16:25:54 <hemna> smcginnis, ok I'm done.
16:26:16 <xyang> hemna: ok, I have a comment here:)
16:26:46 <smcginnis> lhx_: If you have any ideas there, we could use it.
16:26:59 <smcginnis> lhx_: Just at the point of trying to figure out what's even happing there.
16:27:22 <xyang> hemna: we initially tried to bump version on every change, but ran into so many merge conflicts that it was almost impossible to merge all patches.  So the compromise now is to only bump version once but with description of the change for major update
16:27:44 <xyang> hemna: I can show you how it look like later and see if it looks ok to you
16:27:46 <jungleboyj> Ahhhh Shots Fired, Shots Fired!
16:27:48 <DuncanT> xyang: Stop changing your driver so much!
16:27:48 <smcginnis> xyang: That's because Helen puts up 5 different conflicting patches at the same time. :D
16:28:01 <jungleboyj> smcginnis: ++
16:28:02 <hemna> xyang we had that issue for a while too, but found that in the long run it was really worth the effort.
16:28:02 <xyang> DuncanT: need to fix bugs:)
16:28:09 <smcginnis> :0
16:28:11 <xyang> smcginnis: yes:)
16:28:17 <lhx_> smcginnis, sorry, have no idea, I'm looking into it
16:28:34 <smcginnis> Oh, another announcement I wanted to bring up, just to raise awareness.
16:28:38 <xyang> hemna: help with reviews and may be that won't be a problem to merge them:)
16:28:43 <hemna> :)
16:28:44 <smcginnis> There's a push to move in-tree tempest tests into their own repo.
16:28:54 <smcginnis> It fixes some issues by having them separate.
16:28:54 <DuncanT> xyang: Stop writing bugs in the first place?
16:29:03 <hemna> smh
16:29:07 <smcginnis> So we can't incude temptest tests with the code change itself.
16:29:07 <hemna> like a tempest repo?
16:29:10 <xyang> DuncanT: right, then we will be all out of job:)
16:29:23 <smcginnis> But I don't think we ever have, so not really too big of a deal.
16:29:32 <smcginnis> hemna: Yep, something like a cinder-tempest repo.
16:29:41 <hemna> smcginnis, wasn't the entire point of having in tree tempest tests was that cinder folks could write them?
16:29:47 <smcginnis> Problem with entry points and pbr I think.
16:30:13 <smcginnis> hemna: We'll at least still have full control over what gets let in, it just will have to be a separate patch to a separate repo.
16:30:17 <hemna> yuhk, I can see this getting lost in the weeds.
16:30:35 <hemna> we already are overloaded with repos to do reviews on.  :(
16:30:43 <smcginnis> We could enforce having it with a depends-on tag if we really want to make sure the tests are ready before merging code.
16:31:00 <smcginnis> hemna: Yeah, I pushed back at first.
16:31:11 <eharney> would the cinder-tempest repo have branches?
16:31:17 <hemna> I mean, we aren't even writing them now.
16:31:18 <smcginnis> hemna: But it does solve some problems, and I was definiteyl in the minority.
16:31:31 <smcginnis> eharney: That would definitely be my preference.
16:31:47 <smcginnis> I don't buy the branchles arguments.
16:31:52 <eharney> i think it would have to
16:31:54 <smcginnis> *branchless
16:32:21 <smcginnis> OK, anything else?
16:32:26 <jungleboyj> The future is *less
16:32:26 <sivn_> Can we have a discussion on the new drivers please?
16:32:39 <smcginnis> sivn_: What needs to be discussed?
16:32:45 <tommylikehu> sivn_:  which one?
16:32:46 <sivn_> Veritas driver is ready
16:32:52 <sivn_> Can we talk about the new driver "veritas access driver" https://review.openstack.org/#/c/409914/
16:32:56 <smcginnis> sivn_: What needs to be discussed?
16:33:10 <sivn_> It needs merge
16:33:16 <sivn_> +2 needed
16:33:25 <hemna> it's not even passing jenkins
16:33:32 <smcginnis> sivn_: Again, what needs to be discussed?
16:33:34 <hemna> sivn_, the cinder channel would be a better place to discuss
16:33:35 <sivn_> We have taken care of all review comments
16:34:05 <sivn_> Its because of python-27 failure
16:34:11 <hemna> sivn_, there are questions from eharney not answered
16:34:35 <hemna> sivn_, it's not ready.
16:34:47 <sivn_> Do u have any other issues?
16:35:02 <sivn_> We will take care of them
16:35:10 <sivn_> In the next commit
16:35:34 <smcginnis> Feedback is in the comments on the review. One more time - what needs to be discussed?
16:36:58 <smcginnis> OK, thanks everyone. Please feel free to jump in on troubleshooting the py27 test failures.
16:37:01 <smcginnis> #endmeeting