16:00:08 <smcginnis> #startmeeting Cinder
16:00:08 <openstack> Meeting started Wed Sep 21 16:00:08 2016 UTC and is due to finish in 60 minutes.  The chair is smcginnis. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:00:09 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
16:00:12 <openstack> The meeting name has been set to 'cinder'
16:00:14 <smcginnis> Agenda: https://wiki.openstack.org/wiki/CinderMeetings#Next_Cinder_Team_meeting
16:00:16 <fernnest__> hi
16:00:17 <jgregor> Hello!
16:00:19 <erlon> hey
16:00:20 <jseiler_> hi
16:00:23 <_alastor_> hey
16:00:27 <smcginnis> ping dulek duncant eharney geguileo winston-d e0ne jungleboyj jgriffith thingee smcginnis hemna xyang1 tbarron scottda erlon rhedlind jbernard _alastor_ bluex patrickeast dongwenjuan JaniceLee cFouts Thelo vivekd adrianofr mtanino yuriy_n17 karlamrhein diablo_rojo jay.xu jgregor baumann rajinir wilson-l reduxio wanghao thrawn01 chris_morrell stevemar watanabe.isao,tommylike.hu
16:00:30 <Swanson> hallo
16:00:34 <adrianofr_> hi
16:00:35 * bswartz on nom nom
16:00:39 <smcginnis> bswartz: ;)
16:00:40 <geguileo> Hi!
16:00:41 <xyang> hi
16:00:45 <diablo_rojo> Hello :)
16:00:45 <rajinir> o/
16:00:46 <cFouts> hi
16:00:52 <e0ne> hi
16:00:55 <baumann> Hello
16:01:00 <hemna> yough
16:01:02 <eharney> hi
16:01:05 <akerr> o/
16:01:10 <smcginnis> #topic Summit Prep
16:01:15 <jungleboyj> Howdy.
16:01:24 <smcginnis> #link https://etherpad.openstack.org/p/ocata-cinder-designsummit-planning Summit planning etherpad
16:01:25 <patrickeast> hi
16:02:01 <smcginnis> We have 4 fishbowls, 4 working rooms, and a half day contributor meetup.
16:02:06 <e0ne> let's propose topics and discuss them on the next meeting?
16:02:12 <smcginnis> Please start adding topic ideas to the etherpad.
16:02:39 <smcginnis> As we get closer we will figure out what goes where and if we need to prioritize any topics over others.
16:03:06 <smcginnis> e0ne: Yeah, we can discuss here once we have more added. Sounds good.
16:03:41 <smcginnis> Any questions on the summit so far?
16:04:00 <e0ne> are we planning any cross-project discussions?
16:04:05 <diablo_rojo> Do we want to make a list like we do for midcycles about who is going?
16:04:06 <smcginnis> e0ne: Yes
16:04:10 <bswartz> last design summit ever...
16:04:19 <e0ne> bswartz: :(
16:04:22 <smcginnis> bswartz: :[
16:04:33 <smcginnis> e0ne: There is another etherpad somewhere for that.
16:04:48 <smcginnis> e0ne: It was mentioned in the ML, but I don't have the link handy at the moment.
16:04:51 <hemna> and I'm missing this one.
16:05:05 <diablo_rojo> hemna :'(
16:05:07 <smcginnis> hemna: :{
16:05:09 <jungleboyj> hemna: :-(
16:05:31 <hemna> no free booze for me!
16:05:48 <jungleboyj> hemna: We will get you some booze.
16:05:48 * smcginnis looks up California's alcohol shipping laws
16:05:50 <scottda> kickstarter fund to buy hemna booze?
16:05:56 <smcginnis> scottda: ;)
16:05:59 <jungleboyj> scottda: +2
16:06:08 <hemna> :P
16:06:09 <TommyLikeHu_> :)
16:06:14 <smcginnis> OK...
16:06:18 <smcginnis> #topic Newton Retrospective
16:06:18 <e0ne> #link https://etherpad.openstack.org/p/ocata-cross-project-sessions
16:06:18 <_alastor_> scottda: +1
16:06:21 <geguileo> hemna: :-(
16:06:24 <smcginnis> e0ne: Thanks!
16:06:43 <smcginnis> So trying this out to see if it's any value.
16:07:07 <smcginnis> #link https://etherpad.openstack.org/p/cinder-newton-retrospective
16:07:09 <hemna> smcginnis, getting all agile on us ?
16:07:16 <smcginnis> Please feel free to add any thoughts there.
16:07:20 <smcginnis> hemna: ;)
16:07:29 <smcginnis> hemna: Just wait until the daily standups start.
16:07:30 <smcginnis> :D
16:07:46 <smcginnis> Nah, this is bordering on my edge of too much process.
16:08:10 <Swanson> I don't have enough material for daily standup.
16:08:15 <bswartz> (obligatory dilbert) http://dilbert.com/strip/2016-09-19
16:08:22 <smcginnis> But thought it might be good to give folks a place to give feedbak and see if there's anything we should pay attention to.
16:08:41 <jungleboyj> bswartz: Nice!
16:08:48 <smcginnis> bswartz: Hah!
16:08:54 <hemna> hehe
16:09:05 <smcginnis> Anyway, just putting that out there.
16:09:09 <hemna> so....privsep needs work
16:09:19 <hemna> there are still things about it that are super annoying
16:09:21 <smcginnis> If there's enough feedback collected and it makes sense we can discuss things at the summit.
16:09:22 <bswartz> IMO a retro isn't worth much if you don't already have a proposal in mind to change something
16:09:29 <hemna> (Newton retro...)
16:09:47 <smcginnis> hemna: Agree
16:09:55 <jungleboyj> hemna: ++
16:10:01 <scottda> bswartz: How do we know what to change without the retro?
16:10:08 <e0ne> hemna: +1
16:10:08 <hemna> and since I won't be in BCN, can someone hunt Angus down if he's there
16:10:20 <diablo_rojo> hemna +1
16:10:22 <hemna> I can make a list of stuff that needs discussing
16:10:29 <hemna> :(
16:10:31 * hemna has a sad
16:10:32 <bswartz> scottda: well enough time will have passed that people will have their ideas already
16:10:35 <scottda> hemna: I think we'll have a cinder-nova session.
16:10:35 <e0ne> hemna: :(
16:10:36 <smcginnis> hemna: Maybe we should add privsep to the cross project etherpad.
16:10:42 <akerr> bswartz never comes to our retros either, so don't feel bad smcginnis
16:10:46 <hemna> smcginnis, yah we should
16:10:51 <diablo_rojo> hemna, I can try to find him, another set of ears to help absorb info would be helpful
16:10:51 <smcginnis> akerr: :)
16:10:51 <scottda> hemna: Privsep issues alone warrant that. Plus multi-attach and new API
16:10:59 <jungleboyj> hemna: Delegate to diablo_rojo .
16:11:02 <bswartz> I suppose the etherpad will tell us if there's enough content to discuss
16:11:14 <smcginnis> bswartz: Yep, that's what I'm thinking.
16:11:18 <diablo_rojo> Do we even know if he will be there?
16:11:32 <smcginnis> Not sure, but I would think likely.
16:11:45 <smcginnis> #topic Ocata testing and stabilization
16:11:45 <hemna> I hope so
16:12:00 <smcginnis> #link https://etherpad.openstack.org/p/cinder-ocata-testing Ocata testing
16:12:24 <smcginnis> So as much as reasonable I would like to make the focus of Ocata be testing and stabilization.
16:12:27 <e0ne> smcginnis: I'll update this etherpad with tempest backup status
16:12:37 <e0ne> smcginnis: +1
16:12:45 <smcginnis> Not that we won't allow things in, just we need to be selective.
16:12:52 <Swanson> No tiramisu?
16:12:53 <smcginnis> e0ne: Oh, tempest.
16:12:56 <smcginnis> Side note before I forget.
16:12:57 <e0ne> but does it mean that no new drivers or bp will be merged?
16:12:57 <bswartz> +10
16:13:26 <geguileo> smcginnis: I understand HA A/A work will be allowed to continue, right?
16:13:26 <akerr> is that +2 in binary?
16:13:31 <smcginnis> If you run your CI with tox -e all_plugin -- --concurrency=1 volume you're broken right now.
16:13:32 <e0ne> we did some progress with backup tests this release. I have to sync with out QA team
16:13:42 <smcginnis> geguileo: Right!
16:13:51 <bswartz> #lijnk https://releases.openstack.org/ocata/schedule.html
16:13:53 <geguileo> :-)
16:13:59 <smcginnis> e0ne: Still allow drivers and limited in flight features I think.
16:14:06 <bswartz> ^ look how short this cycle is
16:14:10 <e0ne> smcginnis: good to know
16:14:11 <smcginnis> bswartz: lijnk?
16:14:17 <bswartz> make the deadline for drivers super early
16:14:20 <TommyLikeHu_> link
16:14:23 <smcginnis> bswartz: :)
16:14:25 <bswartz> s/lijnk/link/
16:14:29 <hemna> whoa
16:14:36 <smcginnis> #link https://releases.openstack.org/ocata/schedule.html Ocata schedule
16:14:40 <e0ne> there could be some BP's that fixes stability
16:14:42 <hemna> RC1 Jan 30!
16:14:52 <smcginnis> Super short this time around.
16:15:01 <TommyLikeHu_> short ..
16:15:07 <smcginnis> So realistically, we don't really have time for any big new features anyway.
16:15:07 <akerr> smcginnis: the tempest command has changed from tox to "tempest run" but we've been running the all-plugin stuff without issue
16:15:14 <bswartz> hemna: don't forget winter holidays taking away 3 weeks of productivity
16:15:15 <hemna> smcginnis, +1
16:15:27 <hemna> if we were ever going to have a bug fix release, this is a great chance to do that
16:15:38 <smcginnis> akerr: My issue with the way they changed that is they hard coded the regex arg to be first.
16:15:44 <smcginnis> hemna: +1
16:16:03 <Swanson> Let's see, 3 weeks of vacation left, last week of the year holidays, Turkey Day. Yeah. Ocata is dead to me.
16:16:13 <e0ne> hemna: +2
16:16:28 <smcginnis> Swanson: That really will be interesting with the holidays in there.
16:16:29 <_alastor_> Swanson: +1
16:16:45 <smcginnis> Literally just a few weeks for Ocata when you count holidays and such.
16:17:20 <smcginnis> scottda started a testing etherpad for Newton. I used that as a basis to start https://etherpad.openstack.org/p/cinder-ocata-testing
16:17:42 <smcginnis> Let's capture test coverage areas and get folks signed up to work on them.
16:17:47 <hemna> smcginnis, should we triage bugs and target them for O ?
16:18:02 <hemna> smcginnis, we should also look at the nova volume bugs again too
16:18:03 <Swanson> So a short release changing anything with how long we are supporting previous releases?
16:18:09 <smcginnis> This will be both for implementing tests and doing manual testing for things we can't easily automate yet.
16:18:18 <smcginnis> hemna: Yeah, good call.
16:18:20 <hemna> I'd like to see if we can start tackling some of those (nova volume based bugs)
16:18:35 <bswartz> swanson: that would be a good TC question
16:18:42 <smcginnis> If nothing else from O, I'd love to see that bug count go down.
16:19:01 <scottda> Might be nice to get a volunteer to be bug czar
16:19:01 <hemna> I'd like to get the cinder/brick/local_dev stuff out of Cinder this release.
16:19:03 <scottda> not it!
16:19:05 <smcginnis> Swanson: That is an interesting question.
16:19:05 <xyang> smcginnis: is the new driver merge deadline the same as before, Ocata-2?
16:19:19 <smcginnis> xyang: Yes. One second...
16:19:39 <smcginnis> #link https://review.openstack.org/#/c/374260/ Cinder Ocata deadlines
16:19:57 <diablo_rojo> eharney is normally the bug czar I thought
16:19:59 <smcginnis> Pretty much the same deadlines in relation to overall deadlines as before.
16:20:43 <bswartz> scottda: appointing a czar is what you do when you want to completely ignore a problem
16:20:44 <scottda> eharney: is the guy who does the most bug triage. Nothing official, to my knowledge..
16:20:55 <scottda> bswartz: Not true. Nova has done a great job with this
16:21:40 <scottda> Nova bug czar reports at weekly meetings, and seeks help with volunteers to take triage shifts.
16:21:44 <smcginnis> I do like the idea of having folks focus on certain areas.
16:21:59 <erlon> smcginnis: +1
16:21:59 <bswartz> depends on the czar having actual powers I guess
16:22:00 <smcginnis> But that only works as well as the ability for those folks to actually focus on it.
16:22:05 <TommyLikeHu_> +1
16:22:54 <smcginnis> Well that's pretty much it on the meeting agenda.
16:22:59 <smcginnis> #topic Open discussion
16:23:02 <smcginnis> Anything else?
16:23:13 <jungleboyj> smcginnis: Did you have something for me?
16:23:21 <bswartz> is there a PTL election?
16:23:27 <smcginnis> jungleboyj: Oh right.. why you're screwed. :)
16:23:31 <smcginnis> bswartz: ?
16:23:34 * jungleboyj sighs
16:23:39 <e0ne> bswartz: we elected smcginnis
16:23:40 <jungleboyj> It has been a long enough couple weeks man.
16:23:42 <e0ne> :)
16:23:46 <jungleboyj> e0ne: ++
16:23:51 <bswartz> congrats smcginnis!
16:23:58 <smcginnis> bswartz: Oh.. no, no one else put there name in this time, so you're all stuck with me again.
16:24:06 <smcginnis> bswartz: Thanks :)
16:24:09 <bswartz> lol
16:24:15 * jungleboyj sings "For he's a jolly good fellow!"
16:24:22 <smcginnis> jungleboyj: So back to your problem.
16:24:22 <TommyLikeHu_> +1000
16:24:35 <smcginnis> jungleboyj: DB backports.
16:25:03 <smcginnis> jungleboyj: masterbasing is not good due to DB migration backports.
16:25:05 <jungleboyj> smcginnis: Ah, ok.
16:25:26 <smcginnis> In order to backport a DB change we use our placeholders.
16:25:50 <smcginnis> Which is what they are intended for on stable releases so there's room to migrate up a number.
16:26:01 <smcginnis> But in the master branch we reserve those numbers ahead of time.
16:26:16 <smcginnis> Which means we need to actually modify an existing DB migration number.
16:26:29 <smcginnis> Which if you are based off of master you will have already migrated through.
16:26:38 <smcginnis> So changes there will not get picked up.
16:27:22 <smcginnis> So either we can't reserve migrations (and possibly can't backport changes to fix things) or you're kind of in trouble doing CD from master.
16:27:24 <jungleboyj> smcginnis: Is that really an issue if we are just staying on master?
16:27:31 <smcginnis> jungleboyj: Yes
16:27:41 <smcginnis> So placeholders will be added.
16:27:51 <DuncanT> smcginnis: We should probably add a devref about these placeholder migrations
16:27:53 <smcginnis> So you end up with something like 080_placeholder.py
16:28:04 <smcginnis> Initially it is basically a noop.
16:28:15 <smcginnis> But then we find something that we need to backport to fix.
16:28:44 <smcginnis> We need to modify 080_placeholder to be 080_add_critical_column.py and change it's content.
16:29:06 <smcginnis> But in the meantime, sqlalchemy will have migrated to version 081 and done the noop for 080.
16:29:27 <smcginnis> So changing it doesn't get picked up when continuously deploying from master.
16:29:36 <eharney> that doesn't sound right to me
16:29:54 <bswartz> is there a specific bug you guys have in mind?
16:29:57 <smcginnis> Because the migration is already at a later version. It thinks it already applied that previous version. But now that previous version is something different.
16:29:59 <eharney> i thought for this to work right, the fix in master would be a new 085_fix, then it would be backported as 080_fix in stable/newton
16:30:15 <smcginnis> eharney: I think that's the only way it would work.
16:30:16 <jungleboyj> Oh, so it gets skipped because we have told the database the work has already been done.
16:30:50 <eharney> smcginnis: but that doesn't sound like it works, specifically for the reason you were mentioning about CD
16:30:53 <geguileo> eharney: I thought that as well
16:31:01 <smcginnis> eharney: But then we have the issue of a previous version stable release applying that change in 080. Then it gets upgraded to the new version and applies 085. In this example it would fail because it would try to add the column twice.
16:31:18 <eharney> smcginnis: i think the migration 085 is supposed to know how to be a no-op if the new column already exists
16:31:23 <jungleboyj> Ok, I need to drop into another meeting.  Will follow up afterwards.
16:31:40 <DuncanT> eharney: +1
16:31:42 <eharney> otherwise i don't see how this really is useful
16:31:42 <smcginnis> jungleboyj: K, talk later.
16:31:54 <geguileo> eharney: +1
16:32:00 <smcginnis> eharney: Maybe, but I don't think we handle things correctly right now.
16:32:21 <geguileo> smcginnis: I haven't seen any DB backports yet...
16:32:26 <smcginnis> It certainly makes backports trickier.
16:32:27 <eharney> i think we do, we just haven't used it
16:32:29 * geguileo probably missed them
16:32:37 <smcginnis> geguileo: Yeah, I don;t think we've ever done them before.
16:32:46 <smcginnis> There's a patch out there now for one though.
16:32:51 <eharney> what patch is that?
16:32:57 <smcginnis> This will be the first time we've done it as far as I know.
16:33:11 <smcginnis> looking...
16:33:18 <geguileo> I think during stable release backports we have to make sure that the migration in master is capable of doing the no-op like eharney mentioned
16:33:31 <smcginnis> https://review.openstack.org/#/c/355214/
16:33:37 <smcginnis> So maybe that's the answer.
16:33:50 <smcginnis> We just need to be aware of whether something potentially could be a backport.
16:34:02 <DuncanT> We should probably make *all* future migrations no-op capable, really
16:34:04 <smcginnis> And write the migration in a way that can recognize if it was already applied.
16:34:20 <smcginnis> Which also means we can't just cherry pick a change.
16:34:25 <DuncanT> It's easier than guessing about backports
16:34:53 <eharney> i think an argument could be made that that bug is not severe enough to backport it given the pain...
16:35:13 <geguileo> DuncanT: For some features it doesn't make sense to make them no-op capable, right?
16:35:21 <geguileo> DuncanT: Only for bug-fixes
16:35:43 <eharney> but if we want to, i don't see why we couldn't do what i was saying above
16:35:55 <DuncanT> geguileo: Depends if we think the feature will ever be backported
16:36:43 <smcginnis> eharney: Yeah, at this point maybe it's not worth it.
16:36:43 <DuncanT> geguileo: The cost of putting a check in the migration is neglegable, and we could even automate a check that it is no-op capable
16:36:45 <geguileo> DuncanT: Even if it doesn't have the no-op initially we can have another patch that modifies that migration and adds the no-op feature when we want to do the backport
16:36:58 <eharney> smcginnis: i suspect few people have hit that problem
16:37:04 <smcginnis> eharney: I wish we would have held off on reserving placeholders. Then it wouldn't have been an issue.
16:37:08 <DuncanT> geguileo: Modifying database migrations is high risk
16:37:34 <smcginnis> eharney: It's at least an existing issue that has been there for some time, so what's one more release. :]
16:37:48 <smcginnis> eharney: And there is a manual work around if a customer really needs it.
16:37:50 <geguileo> DuncanT: We would be just adding the no-op check... I'm missing the high risk, but I'm probably just tired  :-(
16:38:17 * dulek creeps in very late.
16:38:26 <DuncanT> geguileo: The problem happens if you ever subtly change the resulting schema
16:39:00 <dulek> I've seen DB backports discussion - point is that DB migrations should be idempotent. In case of backporting a schema change you have same migration as two numbers.
16:39:23 <DuncanT> geguileo: There's already a risk of that with the backport... I wonder if we should try to have some more solid validation of the db at the start of a release? Even a side-script?
16:39:29 <smcginnis> dulek: That's what the placeholders are supposed to be for. We were using them wrong the last few releases though.
16:39:56 <smcginnis> dulek: We were merging them at the end of a cycle, which really made them useless. Should be first thing in the cycle.
16:40:06 <dulek> Yup.
16:40:23 <dulek> See this: https://review.openstack.org/#/c/109660/
16:40:33 <dulek> And this: https://review.openstack.org/#/c/112107/
16:41:21 <dulek> Migrations 248 and 234 in Nova are exactly the same thing, so regardless of if you're doing CD or not, it works.
16:41:27 <dulek> But they need to be idempotent.
16:41:30 <eharney> dulek: right
16:42:14 <smcginnis> So that looks to be the answer.
16:42:17 <dulek> I guess being idempotent is what you've called "noop capable"? ;)
16:42:27 <smcginnis> We'll just have to be aware of that for anything that may be backported.
16:42:44 <smcginnis> Or as geguileo said, add the noop checking once we realize we need to backport.
16:43:20 <eharney> in this particular case where it just does an alter to extend the field, is it not idempotent anyway?
16:43:28 <bswartz> it's too late once you realize you need to backport
16:44:23 <smcginnis> bswartz: I think adding a check to an existing migration is probably OK. It won't matter to new deployments, so it's just getting it in place for upgrades.
16:44:31 <smcginnis> eharney: I think you're right there.
16:44:41 <eharney> bswartz: good point, it would force us to say that you can only upgrade to the newer release after commit X which is not really how it should work
16:45:05 <bswartz> I wouldn't commit the change in master until the upgrade step is idempotent if you EVER want to backport that change
16:45:31 <bswartz> you can't fix it later without creating a discontinuity in the upgrade path, as eharney points out
16:45:33 <smcginnis> As long as we merge the master change before the backport, I would be very surprised if an older stable release would try to upgrade to some interim master version.
16:45:47 <smcginnis> Fair enough.
16:45:53 <DuncanT> smcginnis: It definitely happens
16:46:06 <smcginnis> One more thing to add to the list of gotchas to watch for when reviewing.
16:46:23 <smcginnis> OK, anything else?
16:46:30 <DuncanT> smcginnis: Testing for idemotency is automateable
16:46:39 <TommyLikeHu_> +1
16:47:14 <smcginnis> Added as a test item here: https://etherpad.openstack.org/p/cinder-ocata-testing
16:48:41 <smcginnis> Going once...
16:49:06 <smcginnis> OK, thanks everyone.
16:49:18 <TommyLikeHu_> cool
16:49:28 <smcginnis> #endmeeting