15:03:27 <e0ne> #startmeeting cinder-testing
15:03:27 <openstack> Meeting started Wed Jul 13 15:03:27 2016 UTC and is due to finish in 60 minutes.  The chair is e0ne. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:03:28 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
15:03:31 <openstack> The meeting name has been set to 'cinder_testing'
15:03:48 <e0ne> #chair scottda
15:03:50 <openstack> Current chairs: e0ne scottda
15:04:11 <openstackgerrit> Merged openstack/cinder-specs: Migrate volume between backends in an async way  https://review.openstack.org/312853
15:04:14 <scottda> e0ne: thanks
15:04:26 <e0ne> scottda: np
15:05:43 <scottda> So, I'm trying to help get this through: https://review.openstack.org/#/c/325895
15:05:53 <scottda> It seems stuck
15:06:38 <patrickeast> :( yea merging stuff into devstack and devstack-gate is slow
15:06:51 <scottda> If anyone knows bkapilov, I'd like to help figure out what to do to get this moving.
15:07:49 <scottda> I've been testing geguileo 's HA patches, and I see dulek has also....just manual testing so far.
15:08:25 <patrickeast> scottda: i'll have the pure ci running A/A HA soon-ish too
15:08:32 <scottda> I think that is a bit tricky, and maybe we can share testing ideas
15:08:46 <scottda> patrickeast: That's great.
15:08:52 <patrickeast> one change blocking the way https://review.openstack.org/#/c/341222/
15:09:23 <patrickeast> well, blocking the way without a forked devstack-gate
15:09:37 <scottda> OK. Anyone who can, please review #link https://review.openstack.org/#/c/341222/
15:10:01 <patrickeast> we can test on os-brick now too with multi-node stuff, got https://review.openstack.org/#/c/336742/ in yesterday
15:10:49 <patrickeast> which kind of lead into the topic of maybe something to discuss at the mid-cycle
15:10:56 <scottda> #info PatrickEast is the Man
15:10:59 <patrickeast> do we have some way for a 3rd party ci to run periodic jobs?
15:11:16 <patrickeast> i don't really want to have to run a bunch of different HA multi-node configs for each patchset
15:11:24 <patrickeast> but could easily do a nightly job for now
15:11:36 <patrickeast> not sure how to convey results to the rest of us though
15:12:10 <scottda> patrickeast: Maybe infra has insight into that. Don't they already have infrastructure for periodic jobs?
15:12:29 <patrickeast> ah yea, should ask
15:12:38 <patrickeast> i think there are ones like that.... somewhere...
15:13:01 <scottda> I already put Testing on the mid-cycle agenda, so please add any topics to that.
15:13:39 <scottda> dulek: You around? Any news on multi-node testing patches?
15:14:28 <scottda> BTW, multi-backend experimental job was approved, but needs dendencies...
15:14:54 <scottda> #link https://review.openstack.org/#/c/330678/ for an example of How to Do experimental job
15:15:03 <DuncanT> If we're going to be doing periodic jobs, we're going to need a decent dashboard to collate the results
15:15:40 <Guest61430> we were talking about using periodic jobs for our CI as well. Would be good if there is a common way for everyone to do it
15:15:43 <Guest61430> grr
15:15:47 <cFouts> hi
15:16:31 <scottda> DuncanT: Looks like zuul has a periodic queue..I'm not sure if it all works and reports anything
15:16:48 <scottda> hahah..we all know Guestxxx is jgriffith
15:16:53 <DuncanT> There was no dashboard last time I looked
15:16:54 <erlon> scottda: http://abregman.com/2016/03/05/openstack-infra-jenkins-jobs/
15:17:04 <jgriffith> scottda: it is?
15:17:08 <erlon> scottda: very helpful to first timers
15:17:29 <karthikp_> scottda: I believe we are having disagreements with infra CI team on whether to having c-vol running on primary
15:17:33 <scottda> jgriffith: oh, sorry, thought it might be :)
15:17:46 <karthikp_> scottda: https://review.openstack.org/#/c/327619/
15:18:11 <jgriffith> scottda: well ya never know, but I haven't had the split personality from multiple logins in a while.  ZNC has been working well :)
15:20:21 <scottda> karthikp_: Yeah, that seems to be a big cause of slowness for these patches...how to reconcile the disagreements more efficiently
15:22:28 <scottda> OK, well, we don't need to spend too much time here. Please populate the Test topic for the mid-cycle with things to discuss next week.
15:22:42 <scottda> Anyone have anything specific to bring up now?
15:23:30 <patrickeast> oh wait
15:23:42 <patrickeast> still trying to get https://review.openstack.org/#/c/312305/ merged
15:23:43 <erlon> scottda: hey, just to updated about the create_from tests on tempest
15:24:12 <patrickeast> make sure to review if anyone from cinder is opposed (or likes) the change to image-cache by default
15:24:26 <erlon> scottda: there where indeed some backends failing this
15:24:28 <karthikp_> scottda: the only way I know is asking them to review our change ..buy spamming their channel :)
15:24:44 <openstackgerrit> Nitin Madhok proposed openstack/cinder: Fixes consistency snapshot creation  https://review.openstack.org/340563
15:24:57 <erlon> scottda: I need someone (jgriffith ) to have a look on this fix: https://review.openstack.org/#/c/336092/
15:25:29 <erlon> jgriffith: :)
15:25:46 <karthikp_> scottda: Let me see what Ian and Clark(from CI team) have to say again!
15:26:00 <scottda> karthikp_: That'd be great, thanks.
15:26:01 <erlon> jgriffith: not sure if was what you suggested
15:26:10 <jgriffith> erlon: let me have a look...
15:27:37 <jgriffith> erlon: yep, I like having it in the driver like that rather than the manager
15:27:56 <scottda> patrickeast: Will you please make sure your patches are on the Review section of https://etherpad.openstack.org/p/Cinder-testing
15:28:18 <patrickeast> scottda: will do
15:28:19 <erlon> jgriffith: good, thanks
15:28:27 <jgriffith> erlon: thank you!
15:28:43 <jgriffith> erlon: if you push up a rebase I'm almost done reviewing and I'll +2 it
15:29:43 <erlon> jgriffith: ok, Ill updated it, I think we should wait a few more CIs to run this to make sure at least the CIs workign pass
15:30:04 <erlon> jgriffith: Ill let you know when I finish the CI list
15:30:20 <jgriffith> erlon: for sure... but the catch is a few of the CI's (mine included) won't run til Gate gives a +1
15:31:09 <erlon> jgriffith: yep, I had several spins of this patch, only this last day is in conflict
15:31:30 <erlon> jgriffith: the other patches where +1
15:31:52 <jgriffith> erlon: yeah, silly merge conflicts. :)
15:33:57 <scottda> I'm figuring we'll use ceph for 2-node HA testing. Anyone who has good scripts/local.conf for that, please let me know
15:34:00 <openstackgerrit> Eric Harney proposed openstack/cinder: Cascade + force volume delete parameters  https://review.openstack.org/337814
15:34:15 <scottda> Or let me know if you've better ideas than ceph
15:35:17 <patrickeast> seems like ceph would be the easiest
15:35:32 <erlon> scottda: wouldn't that work with NFS?
15:36:32 <scottda> erlon: Yeah, NFS would work as well. But we've lots of ceph testing already, and it seems more supported
15:36:41 <eharney> NFS likely won't give good coverage, Ceph would probably be a better option for HA testing
15:36:44 <erlon> scottda: hmm
15:37:00 <openstackgerrit> Ivan Kolodyazhny proposed openstack/os-brick: Local attach feature in RBD connector  https://review.openstack.org/328297
15:37:04 <jgriffith> scottda: why ceph?  Just curious
15:37:10 <jgriffith> scottda: sorry I'm not fully up to speed
15:37:22 <jgriffith> scottda: any reason you wouldn't use LVM?
15:37:34 <patrickeast> jgriffith: iirc we need something that can be remotely managed, single storage backend, from two nodes
15:37:37 <scottda> jgriffith: Well, we start with just the need for shared storage for Active-active c-vol
15:37:38 <patrickeast> that we can run in gate
15:38:12 <jgriffith> patrickeast: scottda sure... but shouldn't we be starting with making the ref work?
15:38:18 <scottda> jgriffith: I actually have "faked" AA with 2 nodes running LVM, but they don't actually sync aything
15:38:26 <jgriffith> patrickeast: scottda so you can do that by running the tgt code on a seperate node
15:39:02 <jgriffith> patrickeast: scottda ie... break out the LVM driver so that JUST the LVM driver runs on a node, and has a rest interface for c-vol to talk to it
15:39:25 <patrickeast> jgriffith: yea, we can definitely make it work
15:39:32 <jgriffith> patrickeast: scottda I guess nobody's really interested in putting forth that kind of effort though :(
15:39:39 <scottda> jgriffith: So, yeah, we can make the ref work with 2-node LVM in the gate, and just give the illusion that they are shared. That actually might be a better first step.
15:39:42 <patrickeast> but "best" and "easiest" are not in sync on this :(
15:40:04 <patrickeast> jgriffith: my concern is we would spend more time initially debugging our remote-lvm backend and not the HA problems
15:40:13 <scottda> jgriffith: I'm willing to put the effort in, just need to figure out a good path ("best" and/or "easiest")
15:40:26 <patrickeast> jgriffith: long term, its probably best to get iscsi multinode ha goodness in gate
15:40:36 <jgriffith> patrickeast: scottda yeah
15:40:46 <jgriffith> patrickeast: scottda maybe I can work something up for mid-cycle
15:40:53 <jgriffith> otherwise no reason to slow your progress
15:41:06 <jgriffith> I just think we're one step closer to the death of LVM at this point
15:41:16 <scottda> jgriffith: OK, I'll keep playing around as well, and we can sync up next week on this.
15:41:38 <eharney> jgriffith: i looked long ago at writing a driver for targetd.  That's a service which already exists as the "separate LVM+iSCSI service" piece that you can remotely manage
15:41:40 <patrickeast> friday hackathon project!
15:41:48 <eharney> jgriffith: just FYI if you start looking into that area
15:42:31 <jgriffith> eharney: perfect
15:42:52 <jgriffith> eharney: if I get to it this week and you have time I'll run some ideas by you
15:43:00 <eharney> jgriffith: sure
15:43:03 <jgriffith> eharney: I'm assuming you'll be in town next week?
15:43:08 <eharney> jgriffith: yes
15:43:13 <jgriffith> eharney: great
15:43:31 <scottda> I'll put something on the agenda to figure this out
15:45:49 <scottda> Anything else? or take a break before Cinder meeting?
15:47:10 <dulek> scottda: Ups, missed the meeting.
15:47:43 <dulek> scottda: Multinode patches are in review. Currently these are focused on Grenade, but it is easy to add anything you need to devstack-gate.
15:47:47 <dulek> I can help with that.
15:48:04 <dulek> So - status is "stuck on reviews for 2 weeks".
15:48:18 <scottda> dulek: Cool, thanks. Yeah, that seems common
15:48:39 <scottda> I had sent out a request on the |ML for help with our testing efforts, but no response.
15:49:13 <scottda> Maybe I'll ping some of my company's people to see if I can conscript a volunteer
15:50:18 <scottda> alright, let's take a break and talk about more test stuff at the mid-cycle. Thanks everyone.
15:50:31 <scottda> #endmeeting