16:00:01 <smcginnis> #startmeeting Cinder
16:00:02 <openstack> Meeting started Wed Jun 29 16:00:01 2016 UTC and is due to finish in 60 minutes.  The chair is smcginnis. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:00:03 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
16:00:05 <openstack> The meeting name has been set to 'cinder'
16:00:06 <smcginnis> ping dulek duncant eharney geguileo winston-d e0ne jungleboyj jgriffith thingee smcginnis hemna xyang tbarron scottda erlon rhedlind jbernard _alastor_ vincent_hou kmartin patrickeast sheel dongwenjuan JaniceLee cFouts Thelo vivekd adrianofr mtanino yuriy_n17 karlamrhein diablo_rojo jay.xu jgregor baumann rajinir wilson-l reduxio wanghao
16:00:11 <eharney> hi
16:00:12 <cFouts> hi
16:00:13 <jgregor> Hello!
16:00:14 <Swanson> Hello
16:00:14 <_alastor_> ol
16:00:14 <sheel> hi
16:00:15 <smcginnis> #chair DuncanT
16:00:15 <openstack> Current chairs: DuncanT smcginnis
16:00:16 <adrianofr> hi
16:00:18 <jseiler> hi
16:00:36 <scottda> hola
16:00:42 <flip214> hi
16:00:43 <smcginnis> I probably have to get going part way through the meeting, so DuncanT has been nice enough to offer to take over for me.
16:00:47 <DuncanT> Hi
16:00:48 <erlon> Oi!
16:00:54 <jungleboyj> o/
16:00:56 <smcginnis> #topic Announcements
16:01:06 <yuriy_n17> hi
16:01:07 <ntpttr> hi
16:01:09 <smcginnis> Just the usual stuff today...
16:01:19 <patrickeast> o/
16:01:20 <diablo_rojo> Hello :)
16:01:21 <smcginnis> #link https://etherpad.openstack.org/p/cinder-spec-review-tracking Review focus
16:01:34 <smcginnis> I'd love to see some more of those drivers out of the way.
16:01:39 <smcginnis> We've gotten a couple through.
16:01:45 <smcginnis> I think there are a few more that are close.
16:01:46 <alyson_> hi
16:01:51 <xyang> hi
16:01:55 <smcginnis> And probably a few others that aren't going to make it.
16:02:22 <zhanghao> hi
16:02:34 <DuncanT> Still waiting on CI for several of the new drivers
16:02:39 <smcginnis> If you have a driver up for review - please make sure you are responding to review feedback and making sure CI is running.
16:02:43 <smcginnis> DuncanT: +1
16:02:55 <smcginnis> That seems to be the biggest challenge for most of them.
16:03:07 <smcginnis> #link https://etherpad.openstack.org/p/newton-cinder-midcycle Midcycle planning
16:03:27 <smcginnis> Please add any midcycle topics to the etherpad.
16:03:31 <rajinir> o/
16:03:43 <smcginnis> And if you are going and haven't yet - please add your info to the etherpad for planning.
16:03:57 <smcginnis> scottda: You need all non-US folks' info, right>
16:04:36 <_alastor_> Since the tintri site has been down for the past week, I've been building a tool to replace it.  It's just a simple cmdline one to find the status of a Third party CI, but I'm taking feature requests
16:04:40 <scottda> Yeah, security flagged it last time as a US state department rule...
16:04:46 <scottda> But mainly for certain countries.
16:04:54 <scottda> Sorry about that.
16:05:06 <smcginnis> _alastor_: Awesome!
16:05:16 <smcginnis> scottda: Have you been getting everything you need on that front?
16:05:26 <scottda> smcginnis: Yes, seems ok so far.
16:05:31 <smcginnis> scottda: OK, good.
16:05:42 <tbarron> hi
16:05:45 <smcginnis> _alastor_: Would love to see that once you have it running. I think DuncanT would too.
16:06:00 * smcginnis marks tbarron tardy...
16:06:01 <smcginnis> :)
16:06:10 <tbarron> :)
16:06:18 <DuncanT> _alastor_: Can we chat after the meeting please? :-)
16:06:26 <_alastor_> DuncanT: sure
16:06:38 <smcginnis> #topic Blueprint discussion
16:06:43 <smcginnis> Just one on the list this week.
16:06:53 <smcginnis> #link https://blueprints.launchpad.net/cinder/+spec/feature-classification Feature classification blueprint
16:07:21 <smcginnis> Would like feedback from everyone if this is something desired.
16:07:34 <smcginnis> It's being done for Nova, but I'm not so sure about it.
16:07:39 <jungleboyj> *click*
16:07:47 <smcginnis> Or at least don't have a clear picture in my head of what this means for us.
16:08:22 <xyang> smcginnis: we probably should wait and see how it works in Nova first
16:08:49 <smcginnis> xyang: Yeah, that might be good to wait for it to be fully adapted there first and be able to see.
16:08:56 <DuncanT> We only have 2 completely optional features (CGs and replication) but there are a few other bits (supported QoS options etc) that are probably interesting to somebody choosing storage
16:09:13 <eharney> it would be good to have a list of a few examples for this blueprint
16:09:38 <tbarron> don't we have more variation within a feature across backends than nova would?
16:09:51 <jungleboyj> tbarron: Potentially.
16:10:05 <jungleboyj> xyang: ++ Seems good to wait and see how this plays out for Nova.
16:10:15 <xyang> I think our support matrix looks much simpler compared to Nova’s
16:10:22 <smcginnis> Not sure if a spec would be overkill, but I would like more specifics.
16:10:24 <e0ne> hi
16:10:27 <xyang> just look at the current ones in wiki
16:10:31 <tbarron> and i worry about "eventual goal is to automate this list from some theird party CI reporting mechanism" when the 3rd party CI isn't yet reliable
16:10:52 <jungleboyj> smcginnis: Doesn't this go along with other work that has been discussed in providing an automated way to show what drivers support what features?
16:10:52 <scottda> Yeah, seems weird. These features actually work: xxxx, These features suck: xxxx
16:10:54 <smcginnis> tbarron: And part of that automation I planned with the interface checks I've added.
16:11:00 <smcginnis> jungleboyj: Yep
16:11:14 <smcginnis> scottda: Hah, we'll make sure that's how it gets published. ;)
16:11:16 <jungleboyj> smcginnis: Ok, so I am surprised that you aren't sure about it.  What am I missing?
16:11:18 <xyang> smcginnis: did you already have some driver interface check?
16:11:29 <tbarron> smcginnis: yeah, get that going first
16:11:30 <smcginnis> xyang: Just the base functionality so far.
16:11:49 <smcginnis> But it should now be easy to extend that to generate the support matrix.
16:12:10 <smcginnis> There's a new compliance job that verifies minimum required interface for all drivers.
16:12:15 <diablo_rojo> smcginnis: THanks to your fancy driver interface work :)
16:12:31 <smcginnis> Of course that doesn't verify they actual work right, but it at least makes sure all the right methods are there.
16:13:09 <diablo_rojo> So this matrix will be backed by a CI run that tests all the drivers for the different features?
16:13:33 <smcginnis> diablo_rojo: Not yet.
16:13:50 <diablo_rojo> smcginnis: I mean like, is that the plan?
16:13:51 <smcginnis> I'll ask for more details in the bp, but for now I'm just going to leave it as is.
16:13:55 <smcginnis> Thanks for all the input.
16:13:55 <scottda> It'd be kind of embarrassing to list features we've tested vs. features we've merged, but not really tested and don't have good automted tests
16:14:05 <smcginnis> scottda: Very true!
16:14:23 <smcginnis> Speaking of which...
16:14:26 <smcginnis> #topic Create from volume/snapshot fixes
16:14:31 <smcginnis> erlon: Hey
16:14:37 <erlon> Hey
16:14:42 <erlon> So, Im trying to make a generic fix for all clone from volume/snapshot bugs.
16:14:53 <erlon> https://review.openstack.org/#/c/333037/
16:15:05 <flip214> erlon: what is the bug? that the size in the volume is not respected?
16:15:15 <flip214> but why does this change *remove* all the size changing code?
16:15:16 <erlon> Just a scratch of the idea
16:15:20 <erlon> flip214: yes
16:15:22 <jgriffith> erlon: I was going to let you finish before bomarding you with questions :)
16:15:41 <erlon> jgriffith: thanks! :)
16:15:48 <erlon> The ideia of the fix is to extend the volume (if it is bigger than the source) in the manager, not in the driver like some drivers are fixing the problem now.
16:16:09 <erlon> flip214: so, remove the fix in the drivers
16:16:18 <flip214> erlon: the disadvantage is that it does an extra manage=>driver "context switch", which might affect performance
16:16:20 <erlon> The problem is that there is some drivers that does not clone the volume using the smaller size, ie, they already clone the volume using the right size (like hemma's, LVM, may be others), so, an extend in the manager would break those drivers.
16:16:22 <flip214> understood.
16:16:33 <smcginnis> erlon: Sorry, I haven't looked, but I remember last time we discussed this there were some drivers that could more efficiently create the new volume at the larger size.
16:16:39 <smcginnis> erlon: Does this account for that?
16:17:01 <erlon> flip214: why is this context switch? it doesnt seem to me that there is a context switch
16:17:18 <smcginnis> erlon: The hang up with a generic fix was it wasn't easy to tell if the new volume was the right size or not without changing the driver method.
16:17:22 <flip214> erlon: "context switch" as in a REST or other API call to the underlying storage.
16:17:24 <erlon> smcginnis: yes
16:17:38 <flip214> if the new size already gets sent along with the snapshot-restore it's more efficient
16:17:39 <jgriffith> flip214: shouldn't be it's still in the manager
16:18:02 <erlon> flip214: there's no context switch, it happens in the context of the task_flow
16:18:12 <erlon> flip214: it doesnt leave the task
16:18:14 <jgriffith> erlon: can I make a suggestion?  I don't want to interrupt you if you have more to say.
16:18:18 <smcginnis> erlon: OK, sounds good from a high level. I'll try to take a look later today. More likely tomorrow though.
16:18:33 <flip214> erlon: "context switch" as in an additional REST call to the storage driver that's being used to actually provide storage.
16:18:34 <erlon> jgriffith: go ahead
16:18:40 <eharney> the general idea here does fit how i would expect things to work
16:19:04 <erlon> One idea is to make those drivers always to create the volume of the size of the source, then a generic extend() would work for all. Other option could be have a flag in the driver/volume, so manager knows how the driver behaves in  clone operations.
16:19:07 <jgriffith> erlon: so I have some concerns about pushing more and more in to the manager.  This has created some issues for us in the past
16:19:21 <jgriffith> erlon: that being said, I definitely see the advantage here
16:19:34 <flip214> +1 for the flag
16:19:47 <jgriffith> erlon: I'd suggest we put the logic in the base driver
16:19:59 <jgriffith> erlon: and NOT the manager
16:20:16 <jgriffith> erlon: or in tflow manager for that matter
16:20:16 <e0ne> jgriffith: it makes sense,
16:20:21 <erlon> jgriffith: mhm, I have to connect the dots in this approach
16:20:43 <jgriffith> erlon: so the beauty of that is that a driver can choose to override the call if they have a better way of doing it
16:20:43 <smcginnis> #link https://review.openstack.org/#/c/333037/ Proposed change
16:20:53 <patrickeast> jgriffith: +1 , i think base driver is the right place for this
16:21:00 <erlon> jgriffith: the implementation in flow_manager seems more straight forward to me right now
16:21:06 <flip214> ah, so that would be a "restore-snapshow-with-new-size" API?
16:21:17 <jgriffith> erlon: that's becuase you've never had to update or fix a bug in that layer :)
16:21:32 <smcginnis> ;)
16:22:03 <erlon> jgriffith: ok, :), Ill figure out how to do it, Ill ping you if I have any blocker
16:22:52 <flip214> erlon: thanks for the explanation!
16:23:01 <jgriffith> erlon: I'm happy to help and talk through it with you if you like
16:23:02 <erlon> flip214: no problem!
16:23:31 <erlon> jgriffith: thanks, ill spent some time thinking and get back to you
16:23:56 <smcginnis> erlon: Anything else at this point?
16:23:56 <erlon> smcginnis: I think for me that is it
16:24:00 <smcginnis> erlon: OK, thanks!
16:24:05 <smcginnis> #topic Proposed cinder.conf changes for defining/configuring backends
16:24:12 <smcginnis> patrickeast: You're up
16:24:16 <patrickeast> hey
16:24:35 <patrickeast> ok so, i put some proposals up for adjusting how we allow for defined backends
16:24:44 <patrickeast> https://blueprints.launchpad.net/cinder/+spec/shared-backend-config
16:24:48 <patrickeast> ^ has links to code and spec
16:24:49 <smcginnis> #link https://review.openstack.org/#/c/330767/ Spec
16:25:06 <smcginnis> #link https://blueprints.launchpad.net/cinder/+spec/shared-backend-config Blueprint
16:25:26 <diablo_rojo> I've already looked over the spec and I think it sounds like it could help simplify the config file a lot
16:25:34 <patrickeast> i bring it up at the meeting mainly to get some eyes on it and as a heads up that folks don't like using DEFAULT for volume backends
16:25:50 <eharney> i've _really_ wanted to fix this for a while now
16:25:54 <patrickeast> i know for myself and several others it has been a big pain for deployment tooling and just explaining how it works
16:26:02 <DuncanT> We've talked about doing this before, good to see it moving ahead
16:26:04 <eharney> but the approach i had in mind was the same one that Gorka suggested in the spec
16:26:12 <e0ne> eharney :+
16:26:12 <geguileo> patrickeast: I'd rather we used the [DEFAULT] section
16:26:16 <patrickeast> ah yea, so i thought about that first
16:26:27 <smcginnis> I thought there were issues with that.
16:26:28 <diablo_rojo> It would be good to get a little separation of volume backends options frm the more general options
16:26:34 <patrickeast> and the main reason i didn't go down that path initially is that imo it makes thing *more* confusing
16:26:37 <smcginnis> Something with oslo_config or something?
16:26:43 <geguileo> diablo_rojo: But then [DEFAULT] does not mean default
16:26:45 <jungleboyj> patrickeast: Yeah, DEFAULT isn't good.
16:26:52 <e0ne> IMO, we should implement such feature in oslo.config first
16:26:56 <patrickeast> trying to explain that sometimes DEFAULT does what you want and sometimes it doesnt, depending on some other config option
16:26:58 <DuncanT> I'm not so sure about gorka's approach - there are some things that are system wide and don't make sense to be inherited per backend
16:27:00 * geguileo wonder how DEFAULT section would be confusing...
16:27:02 <eharney> everyone expects DEFAULT to mean "default", pretty much anything else is confusing
16:27:06 <diablo_rojo> geguileo: Well it does, its just a more generic default.
16:27:06 <smcginnis> geguileo: We did discuss this in Tokyo and decided to make it explicit in its own section.
16:27:10 <erlon> e0ne: +1
16:27:12 <patrickeast> is just as bad as explaining when config items should go into backend sections or DEFAULT
16:27:14 <jungleboyj> diablo_rojo: ++
16:27:36 <diablo_rojo> Driver defaults and other defaults. Kinda.
16:27:46 <patrickeast> e0ne: yea, i looked at that too a little bit, i think maybe as a longer term solution to simplify the code we use
16:27:56 <geguileo> patrickeast: The idea would be that DEFAULT worked for everything
16:28:00 <geguileo> patrickeast: Including backends
16:28:02 <DuncanT> I'd call it 'backend_shared' rather than 'backend_defaults' to remove a little of the confusion, but that's bikeshedding
16:28:06 <patrickeast> geguileo: right, i get that
16:28:14 <geguileo> So no explanation necessary
16:28:18 <jgriffith> eharney: patrickeast hehe... I tried to fix it and FAILED... kudos patrickeast
16:28:19 <jungleboyj> We need to have consistent behavior.  People have been confused about the driver sections and DEFAULT for a long time.
16:28:42 <eharney> jungleboyj: i'd rather fix it to have predictable behavior, we just have to be consistent w/ handling upgrades
16:28:50 <DuncanT> Changing the semantics of DEFAULT is a serious upgrade pain point
16:28:51 <eharney> consistent but confusing is not a huge win
16:28:57 <jungleboyj> eharney: Good point.
16:29:06 <patrickeast> geguileo: so the issue is that its not always the case... i've spent literally hours explaining to doc folks, deployers, and guys working on deployment tooling when config options need to go where
16:29:20 <patrickeast> geguileo: and its like a cross release thing, backwards compatability aside
16:29:33 <smcginnis> I think one of the points for a new config section is if something was in DEFAULT and not being applied, then you upgrade and suddenly that behavior changes, that could be bad.
16:29:37 <patrickeast> geguileo: having to learn which version of openstack means DEFAULT does something different is confusing
16:29:53 <patrickeast> geguileo: so a new section that only exists in newer ones is, imo, less cofusing
16:29:59 <patrickeast> confusing*
16:30:03 <patrickeast> does that make sense?
16:30:07 <jungleboyj> smcginnis: That is going to take thought to deal with.
16:30:08 <DuncanT> Can we deprecate the non-multi-backend style config file this release? So that we've option sin the future, whatever we decide for this/ It's a terrible idea to use it anyway
16:30:13 <geguileo> patrickeast: I think it's as confusing, but I don't have as much experience as others, so...
16:30:18 <jgriffith> DuncanT: +1
16:30:21 <eharney> DuncanT: please
16:30:26 <jgriffith> there's a standard deprecation process for config options
16:30:32 <patrickeast> DuncanT: yea there is a patch up for that already
16:30:38 <jungleboyj> I was going to ask if there was a way to start migrating people and deprecating the options that need to be changed.
16:30:42 <jgriffith> we could also put some logic in something like a cinder-manage tool or something maybe
16:30:45 <DuncanT> patrickeast: There is? Awesome
16:30:51 <patrickeast> DuncanT: the shared config part is kind of a secondary patchset
16:31:02 <patrickeast> DuncanT: https://review.openstack.org/#/c/335135/
16:31:04 <geguileo> DuncanT: +1
16:31:30 <eharney> if we made the new config section [GENERIC] or something instead of default, and had it override [DEFAULT], we could just lean toward not having a [DEFAULT] section at all as we move forward
16:31:43 <eharney> would that be less confusing than DEFAULT + backend_defaults?
16:31:44 <DuncanT> patrickeast: I'll go review that straight after the meeting, since it really needs to be in this release (because I odn't have a time machine to put it in the last one or the one before)
16:32:10 <patrickeast> eharney: dunno, we can try it out
16:32:20 <patrickeast> anyone going to the ops meetup?
16:32:25 <patrickeast> we need a focus group XD
16:32:52 <smcginnis> #link https://review.openstack.org/#/c/335135/ Non-multibackend deprecation
16:33:00 <jungleboyj> eharney: Would need to put thought into the name.  GENERIC is confusing.
16:33:02 <DuncanT> eharney: What about options that are in DEFAULT now and need ot be common across backends? (like RPC driver options)
16:33:07 <jgriffith> eharney: I like your idea
16:33:27 <smcginnis> [SERVICE] [DRIVERS] [backenda]...
16:33:46 <patrickeast> so one other option, not sure if i put it up as a alternative in the spec, is to just give up and do a versioned cinder conf
16:33:49 <smcginnis> I thinked we be the only project NOT using default though.
16:33:50 <jgriffith> eharney: the only problem is in the naming, since every project uses "DEFAULT" it could lead to bad user exp
16:33:52 <patrickeast> we roll out a v2 cinder.conf format
16:34:00 <patrickeast> and just do whatever we want to make it nice
16:34:01 <jgriffith> smcginnis: Jinx!
16:34:04 <smcginnis> :)
16:34:10 <eharney> jgriffith: that's why i liked the other idea of adding an option to change the behavior of DEFAULT.
16:34:16 <smcginnis> patrickeast: XML based
16:34:19 <smcginnis> JK!
16:34:21 <patrickeast> noooo
16:34:29 <jgriffith> eharney: yeah, that might be mo-betta
16:34:46 <DuncanT> Do other projects all use an inheritted DEFAULT?
16:34:52 <smcginnis> [DEFAULT] use_new_style_thang=True
16:35:09 <patrickeast> alright, so i'll put up a patch that does that and we can play around with it
16:35:16 <eharney> the problem with something like "backend_defaults" is that i think options in there that aren't really for backends/drivers will still affect things
16:35:17 <DuncanT> Or at least nova and manilla?
16:35:26 <e0ne> DuncanT: AFAIK, oslo.config doesn't support it. so there is no generic implementation
16:35:27 <patrickeast> luckily this code is all pretty small... so we can try different things
16:35:33 <smcginnis> DuncanT: Shoot, gotta go. Can you take over?
16:35:41 <DuncanT> Yup
16:35:46 <smcginnis> DuncanT: Thanks!
16:35:57 <patrickeast> eharney: ah, so the implementation is only on the volume/configuration.py in theory only volume drivers are using it
16:36:02 <smcginnis> All action items now assigned to me I guess. :)
16:36:07 <patrickeast> eharney: or at least that is what is currently proposed
16:36:27 <eharney> patrickeast: deployers won't know which options work there and which don't
16:36:27 <patrickeast> geguileo: i think its worth trying out the approach you are proposing
16:36:47 <geguileo> patrickeast: But do you think it will be better for users?
16:36:55 <patrickeast> eharney: true, and i guess saying anything that worked in the multi-backend stanza doesn't really help much
16:37:10 <geguileo> patrickeast: Because what I think makes more sense may not be what users think
16:37:11 <patrickeast> geguileo: ehh not sure, i'm not convinced either way is really great
16:37:15 <patrickeast> but both are better than what we have now
16:37:16 <geguileo> lol
16:37:21 <geguileo> patrickeast: +1
16:37:40 <geguileo> I totally agree, either way is better than what we have now
16:38:05 <DuncanT> Given we've got enthusiastic agreement for a change, shall we move on?
16:38:12 <patrickeast> ok so i don't wanna take up too much more time, i'll code up a couple of alternatives (all this is a small amount of changes)
16:38:16 <patrickeast> DuncanT: +1
16:38:29 <patrickeast> i'll maybe put something on the agenda for the midcycle and we can pick a flavor
16:38:30 <DuncanT> #topic Third party CI
16:39:34 <DuncanT> So I put this one on the agenda. Sadly HPE has temporarily taken up most of my time with non-cinder work and that will be true for several more months, so I've not been able to do any chacing
16:39:37 <DuncanT> chasing
16:40:04 <DuncanT> Some CIs fail instantly on every patch, some fail on most patches, some seem to have vanished entirely
16:40:30 <DuncanT> Several driver patches recently have 3 or more rechecks needed for a pass, every revision
16:40:45 <e0ne> :(
16:40:57 <DuncanT> I'm hoping _alastor_'s work can help highlight these cases, but something clearly needs sorting
16:41:31 <patrickeast> i wonder if its a symptom of the infrastructure getting worse, or just not being maintained?
16:41:55 <jgriffith> In my case it's just a matter of shit quit working :(
16:41:57 <DuncanT> I'm not sure, I was kind of hoping some of the CI maintainers might chip in here
16:42:09 <patrickeast> so for mine, i think there is a problem with iscsi... somewhere..
16:42:10 <diablo_rojo> jgriffith: Lol
16:42:14 <jgriffith> haven't figured out exactly why... but a rewrite/redesign is in progress on my side
16:42:16 <patrickeast> not sure if its my testbed or in cinder/brick
16:42:20 <Swanson> jgriffith,  Shit quit working and network reshuffling here.
16:42:22 <DuncanT> jgriffith: Cinder shit broke or random supporting shit broke?
16:42:23 <_alastor_> DuncanT: I haven't gotten rechecks yet.  But I can certainly add it.  Right now I've got features like "is-reporting within x days" and "number of reports within x days" and the like
16:42:28 <fernnest> ours seems to broke on one test case.
16:42:29 <Swanson> jgriffith, plus smcginnis rebooted something.
16:42:38 <jgriffith> DuncanT: nothing to do with Cinder in my case
16:42:46 <_alastor_> DuncanT: also failure ratio
16:42:48 <DuncanT> jgriffith: Thanks, that's useful data
16:42:49 <jgriffith> DuncanT: upgraded OpenStack, shit went flakie
16:42:51 <jungleboyj> IBM is converging all of our CIs to one lab and monitoring it more closely, so hopefully our results will be getting better.
16:42:58 * jgriffith needs to stop using the s word
16:43:15 <DuncanT> jgriffith: Schoolboy error. Just run kilo and be happy ;-)
16:43:22 <jgriffith> DuncanT: I'm seeing things like threads dieing, and my nova instances puking
16:43:29 <jgriffith> ie the instances running CI
16:43:59 <jgriffith> puking as in shutting down and going in to Error state...
16:44:13 <DuncanT> jgriffith: Yuck.
16:44:22 <patrickeast> ouch, what configuration are you using?
16:44:30 <jgriffith> yeah, I have obviously screwed something up
16:44:48 <jgriffith> patrickeast: I actually switched over to using "real" bits
16:44:50 <jgriffith> RDO
16:44:56 <jgriffith> should've stuck with devstack :)
16:45:02 <patrickeast> oh interesting
16:45:13 <patrickeast> i'm using mitaka RDO for mine right now and its pretty good so far
16:45:14 <jgriffith> honestly though, I'm sure it's something I have screwed up
16:45:58 <patrickeast> DuncanT: so, whats the plan to help bring up our 3rd party ci stats?
16:46:09 <patrickeast> DuncanT: just poke people to fix their stuff?
16:46:39 <_alastor_> Weakest-link style gameshow?
16:46:40 <DuncanT> patrickeast: To start with. Some name & shame once we have the tooling. That has been surprisingly effective in the past
16:46:55 <patrickeast> DuncanT: definitely effective
16:47:17 <patrickeast> _alastor_: hah, we could vote one driver off every week at the meetings?
16:47:31 <_alastor_> Oooooh, I like that.  Scary though
16:47:40 <DuncanT> patrickeast: Pick the worse performing two and make them fight for it
16:47:44 <patrickeast> lol
16:47:50 <jungleboyj> I am sorry, you are the weakest link .... Goodbye.
16:47:54 <erlon> haha, I remember those times
16:48:07 <DuncanT> patrickeast: The video revenue can pay for the cinder social at the summit
16:48:09 <jungleboyj> Naming and shaming has always been effective.
16:48:18 * DuncanT has no shame
16:48:22 <jungleboyj> Some people need shaming to get their execs' attention.
16:48:26 <patrickeast> in the past we've had efforts to improve the 3rd party ci infrastructure
16:48:36 <jungleboyj> DuncanT: So we have heard.  ;-)
16:48:42 <patrickeast> with the openstackci converged stuff being our golden ticket...
16:48:46 <patrickeast> are there still issues with that?
16:48:54 <patrickeast> is there more work we can do there?
16:49:11 <DuncanT> jungleboyj: Yup. Emailing execs directly with email with 'I'll be shaming you very publicly in two weeks by pulling you driver and posting about why everywhere' was very, very effective
16:49:28 <jungleboyj> DuncanT: ++
16:49:32 <DuncanT> patrickeast: I think maintaining a CI system is just more effort htan many companies thing
16:49:51 <_alastor_> DuncanT: +1
16:49:54 <jungleboyj> DuncanT: Yep, it is.
16:49:54 <patrickeast> DuncanT: yea, but if we lower the bar more... maybe more won't require shame
16:50:02 <patrickeast> like, if it wasn't such a huge pain
16:50:02 <fernnest> DuncanT: +1,
16:50:35 <DuncanT> patrickeast: The previous improvements came when all the people struggling started talking to each other
16:50:42 <_alastor_> I'm surprised more drivers aren't just lying about their results
16:50:52 <patrickeast> _alastor_: it probably would be easier XD
16:50:54 <_alastor_> I think akerr/amead brought that up a few weeks ago
16:51:01 <DuncanT> patrickeast: I don't know how else to encourage more of that, other than lighting a fire under people.
16:51:28 <DuncanT> _alastor_: I tried to do some scraping of logs, but there are too many different formats etc
16:51:35 <patrickeast> DuncanT: yea, i guess maybe we should try and re-kindle some of the like 3rd party working group meeting advertising
16:51:48 <patrickeast> to try and get folks with broken ci's to work together to unbreak them
16:51:54 <patrickeast> (myself included)
16:51:59 <_alastor_> DuncanT: I think the easiest way is to push a change that's obviously going to break stuff, but will pass normal Jenkins
16:52:13 <_alastor_> DuncanT: And see who reports success :)
16:52:26 <jungleboyj> _alastor_: That is an interesting idea.  We have found issues that way in the past.
16:52:40 <DuncanT> _alastor_: challenging but probably interesting, yes
16:52:55 <diablo_rojo> _alastor_: I like that idea.
16:53:20 <diablo_rojo> DuncanT: Just delete a bunch of important things. Simple.
16:53:26 <DuncanT> I'd really like to expand what we test in third party (require LVM as well as vendor backend & test migration, for example) but that's pointless if we don't have working CIs
16:53:30 <jgriffith> _alastor_: pass jenkins but break stuff?  Wouldn't that mean something horribly wrong with Jenkins... or you mean actually manually break every single driver individually?
16:53:54 <patrickeast> DuncanT: ah yea, thats something up on my backlog
16:53:56 <jgriffith> _alastor_: and break them in some way that their unit tests still work?
16:54:05 <DuncanT> jgriffith: It should be easy enough to remove/patch a method from every driver except the LVM one
16:54:09 <patrickeast> DuncanT: i'm planning to add instructions for multi-backend and multi-node 3rd party CI setups
16:54:11 <_alastor_> jgriffith: I can't remember specifics, but I remember ameade doing something that caused that perfect storm
16:54:17 <jgriffith> DuncanT: famous last words :)
16:54:19 <DuncanT> jgriffith: And monkey path it to pass unit tests
16:54:48 <patrickeast> i think you can just delete driver files... right?
16:54:50 <patrickeast> just leave lvm?
16:55:29 <diablo_rojo> patrickeast: Thats what I was thinking
16:55:32 <DuncanT> jgriffith: It's never going to get merged... we can make it a coding contest, see how cruel and unusual people can be. Maybe get somebody to sponser a small prize?
16:55:33 <jgriffith> patrickeast: *just* delete drivers, and unit tests?
16:55:45 <diablo_rojo> DuncanT: +1
16:55:47 <jgriffith> DuncanT: sounds fun :)
16:55:49 <patrickeast> jgriffith: no need for unit tests to smoke check which ci's are actually running the code
16:55:59 <jgriffith> patrickeast: no no... the problem is jenkins won't pass
16:56:13 <patrickeast> jgriffith:
16:56:15 <patrickeast> oh yea
16:56:17 <jgriffith> patrickeast: I thought the lead in was Jenkins (or I guess Zuul now) passes
16:56:20 <DuncanT> jgriffith: Add a call to 'cinder.utils.explode' in every create method, and monkey path it out in the unit tests should do the job though
16:56:26 <jgriffith> LOL
16:57:15 <patrickeast> could add code into the volume manager to just raise an exception when it loads a driver other than lvm
16:57:31 <patrickeast> or loads it from the config
16:57:56 <DuncanT> Oh, the other thing I wanted to bring up is I intend to update the 'how to submit a driver' wiki page to ask you to put your CI name in the commit message for a new driver... I'm fed up of trying to figure it by hand. Any complaints?
16:57:56 <scottda> I know what will work, but I'm not going to tell you all, so I can win the contest.
16:57:59 <Swanson> As an aside could someone put a bullet in the brain of the Cisco PNR CI? Insta fail every time.
16:58:05 <patrickeast> DuncanT: +1
16:58:12 <jungleboyj> Two minute warning.
16:58:26 <DuncanT> Swanson: I'll ask -infra to ban it
16:58:27 <patrickeast> DuncanT: it should include a link to their wiki page on https://wiki.openstack.org/wiki/ThirdPartySystems
16:58:36 <jungleboyj> DuncanT: I think that is a great idea.
16:58:39 <DuncanT> patrickeast: That too, yes
16:59:18 <diablo_rojo> patrickeast: +1
16:59:29 <DuncanT> Ok, that's time, thanks folks
16:59:36 <jungleboyj> patrickeast: +1
16:59:47 <geguileo> Thanks!
16:59:52 <DuncanT> #endmeeting