17:00:07 <ildikov> #startmeeting cinder-nova-api-changes
17:00:08 <openstack> Meeting started Thu Feb  9 17:00:07 2017 UTC and is due to finish in 60 minutes.  The chair is ildikov. Information about MeetBot at http://wiki.debian.org/MeetBot.
17:00:09 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
17:00:11 <openstack> The meeting name has been set to 'cinder_nova_api_changes'
17:00:20 <ildikov> scottda DuncanT ameade cFouts johnthetubaguy jaypipes takashin alaski e0ne jgriffith tbarron andrearosa hemna erlon mriedem gouthamr ebalduf patrickeast smcginnis diablo_rojo gsilvis  xyang1 raj_singh lyarwood
17:00:22 <scottda> hey
17:00:27 <smcginnis> o/
17:00:57 <ildikov> hi :)
17:01:00 <breitz> o/
17:01:06 * johnthetubaguy lurks
17:01:16 <ildikov> let's wait a bit before starting
17:01:30 <mriedem1> o/
17:02:29 <hemna> mep
17:03:22 <ildikov> ok, let's deep dive in the today's meeting
17:03:26 <smcginnis> jgriffith: metadata metadata metadata
17:04:04 <ildikov> I would like to touch on the items we might be able to get done before the PTG and also PTG planning as mriedem1 was suggesting too earlier today
17:04:50 <ildikov> we have a patch up for review in devstack-gate that will allow us to start testing Cinder V3 in the placement API job in Nova
17:05:04 <mriedem> link?
17:05:34 <ildikov> #link https://review.openstack.org/#/c/326585/
17:05:45 <mriedem> oh that
17:06:10 <ildikov> mriedem: yep, this is the dependency one on the infra side
17:06:37 <mriedem> yeah i was expecting to see some DNM hack patch to do some immediate testing
17:06:46 <ildikov> mriedem: I think we don't have anything further prepared yet. Do you think we should try to put together something before this one gets merged?
17:06:54 <mriedem> i'm also wondering what the project-config change looks like, because we could stack that on top of this in d-g to hack it out for now
17:07:08 <mriedem> because then the cinder v3 in nova patch can depend on that for testing
17:07:10 <jgriffith> smcginnis POOF@
17:07:59 <smcginnis> jgriffith: ;)
17:08:23 <mriedem> sdague: is localrc_set in devstack-tools?
17:08:36 <ildikov> mriedem: I would guess we need to set the catalog_info to V3 at some point
17:08:44 <sdague> mriedem: currently, it's using a devstack shell function
17:08:56 <ildikov> mriedem: scottda and jgriffith were playing with Tempest tests more recently
17:09:00 <sdague> but the theory is once we're running everything through that
17:09:08 <sdague> we can flip that to dstools
17:09:20 <sdague> so it will be doing raw local.conf sets
17:09:22 <smcginnis> breitz got the tests to pass as well.
17:09:57 <sdague> also, https://review.openstack.org/#/c/430368/ for local.conf pass through from job definitions
17:11:05 <mriedem> ok i'll need my hand held on that one
17:11:17 <mriedem> we'll add the local_conf builder to the placement job definition
17:11:19 <ildikov> smcginnis: breitz: great news! :)
17:11:43 <mriedem> but then i'm not sure what after that
17:12:17 <ildikov> mriedem: you mean from config perspective?
17:12:22 <mriedem> yes
17:12:28 <mriedem> well, from job template in p-c
17:12:33 <mriedem> i'll ask in the review
17:12:50 <ildikov> sdague: thanks for the update
17:13:35 <ildikov> mriedem: I'm not sure at this point either, I couldn't even successfully point Nova to Cinder V3 yet in my local Tempest tests... :/
17:14:10 <scottda> ildikov: I'd be glad to help with that.
17:14:19 <mriedem> sdague: left my question on usage in https://review.openstack.org/#/c/430368/
17:14:21 <scottda> ildikov: I just used cinder v3 in nova.conf...
17:14:28 <scottda> ildikov: And ran tempest.scenario
17:14:31 <mriedem> maybe that can be worked into the commit message or a comment on how to use the builder or something
17:14:57 <ildikov> scottda: I added that there too, but according to the logs it didn't pick that up for some reason as the URL wasn't V3
17:15:17 <mriedem> so what are the tempest tests?
17:15:29 <mriedem> this is just normal tempest with cinder v3 right?
17:15:32 <ildikov> scottda: I will double check the config
17:15:59 <scottda> mriedem: Yes, just tempest/scenario tests
17:16:05 <ildikov> mriedem: as a first step I think we would like to test V3 overall
17:16:38 <ildikov> mriedem: as we don't have anything in Nova yet to use the new attach/detach API that will be a next step
17:17:00 <smcginnis> ildikov: +1
17:17:21 <smcginnis> Then we need to figure out the best way to handle the workflow branching based on the available microversion I think.
17:17:28 <mriedem> so, https://review.openstack.org/#/c/420201/
17:17:53 <mriedem> can't we create a devstack DNM patch that configures nova for cinder v3 and then depend on that nova change to test it out?
17:18:05 <mriedem> or stack a nova patch on top that depends on a devstack patch that configures nova for cinder v3?
17:18:15 <mriedem> or is this being tested some other way?
17:18:35 <scottda> mriedem: Sure, I'll add a DNM patch as a dependency
17:18:36 <mriedem> also, https://review.openstack.org/#/c/425785/ is unreleased so i don't know how you're testing it without using LIBS_FROM_GIT
17:19:10 <scottda> mriedem: It's not tested in the gate, just manually.
17:19:17 <mriedem> so locally
17:19:18 <mriedem> ok
17:19:44 <mriedem> i can also poke at that
17:20:10 <mriedem> so i think a good goal by the PTG is have this at least working with some DNM patch dependencies since we don't have the local.conf stuff in yet
17:20:19 <mriedem> "this" meaning the nova cinder v3 patch
17:20:22 <scottda> mriedem: The lingering question that smcginnis pointed out it how to get cinder api version info and store/use in nova code.
17:20:34 <scottda> s/it/is
17:20:53 <scottda> One example being that ^^^ POC
17:21:04 <smcginnis> I think we can hack a proof of concept, but it will be up to the nova team to decide how best to integrate that.
17:21:12 <mriedem> we store it in the client i think
17:21:12 <ildikov> just wanted to ask whether we have a good understanding on how to handle the Cinder microversions in Nova?
17:21:15 <mriedem> that's what we do for glance
17:21:37 <mriedem> well glance is different since it's config-based right now
17:21:45 <mriedem> so you're either using glance v1 or v2 (v2 at this point)
17:21:48 <mriedem> and we have a client for each
17:22:19 <mriedem> i think glance v2 has something like microversions, but nova only ever cares about base v2
17:22:43 <mriedem> so i imagine we'll need to handle cinder like how we handle microversions in our placement client
17:23:28 <mriedem> for our placement client we don't do version discovery today, we just handle the 404 if request fails and fallback to older behavior
17:23:38 <mriedem> at some point we need to do version discovery so we don't need to dump errors in the logs
17:23:57 <mriedem> but e.g. https://github.com/openstack/nova/blob/21e36a2c4c280ff3b1d42b7bee16ffe3f235281d/nova/scheduler/client/report.py#L205
17:24:09 <mriedem> https://github.com/openstack/nova/blob/21e36a2c4c280ff3b1d42b7bee16ffe3f235281d/nova/scheduler/client/report.py#L287
17:24:33 <mriedem> that's a method in our client that's doing a 1.1 GET request
17:24:47 <mriedem> if we did version discovery we'd know if we could make that request or not and not log errors on failure
17:25:04 <mriedem> same for 1.4 https://github.com/openstack/nova/blob/21e36a2c4c280ff3b1d42b7bee16ffe3f235281d/nova/scheduler/client/report.py#L257
17:25:09 <johnthetubaguy> we could just hard depend on the v3 version we need right now?
17:25:17 <johnthetubaguy> maybe thats what you are saying here
17:25:21 <mriedem> the code that's using that stuff is handling a fallback scenario in ocata
17:25:24 <sdague> mriedem: replied on expected usage
17:25:27 <ildikov> I think the direction with the Cinder client now is to get the max version supported
17:25:45 <mriedem> ildikov: cinder client in nova, or python-cinderclient/
17:25:46 <mriedem> ?
17:25:49 <scottda> ildikov: That's not hard-coded in the client..
17:25:54 <mriedem> whew
17:25:57 <scottda> ildikov: There's a patch to do that in the CLI
17:26:13 <ildikov> scottda: I didn't mean it's hardcoded
17:26:13 <mriedem> the python api binding user of cinderclient needs to opt in
17:26:18 <mriedem> which is nova in this case
17:26:53 <mriedem> so i think when we setup the client, like in scott's patch, we get the version and build the Client object that way, and when we do that, we know the highest supported v3 version, right?
17:27:03 <scottda> mriedem: Yes
17:27:07 <mriedem> and we know the specific microversions that we need when making requests, 3.27 it looks like
17:27:13 <johnthetubaguy> so the base v3 micro-version will not work with the current Nova code, thats the thing that surprized me I guess.
17:27:25 <scottda> mriedem: And then use a method or Global to refer to something like IS_NEW_API_SUPPORTED
17:27:27 <mriedem> johnthetubaguy: it's not backward compatible?
17:27:34 <johnthetubaguy> I believe its not
17:27:35 <ildikov> yep, exactly, so if the microversion is higher than 3.27 we use the new attach/detach flow
17:27:43 <ildikov> if not than we fall back to the old one
17:27:45 <scottda> johnthetubaguy: What do you mean?
17:27:47 <mriedem> scottda: smcginnis: why isn't v3 compatible with v2?
17:27:56 <scottda> I believe it is
17:28:19 <scottda> IF I use cinder v3 in nova.conf, old api methods work
17:28:20 <johnthetubaguy> oh, I thought we said the old binding API would fail on the base v3 version?
17:28:22 <smcginnis> Yes, base v3 is completely compatible with v2.
17:28:25 <scottda> and tempest.scenario passes
17:28:45 <johnthetubaguy> OK, so I was clearly miss understanding what we said before, my mistake
17:28:52 <ildikov> scottda: what was the solution to the Tempest failures you saw earlier with V3?
17:28:53 <smcginnis> It's just tricky once we start adding in microversion support and having to do different flows based on that.
17:28:59 <ildikov> scottda: I'm just curious
17:29:02 <johnthetubaguy> so we can just follow the usual micro-version patterns it would seem.
17:29:22 <scottda> ildikov: It was a combination of too much concurency with too small a VOLUME_BACKING_SIZE
17:29:31 <scottda> and failed for v2 as well as v3
17:29:34 <mriedem> right, if we can do 3.27, we do that, if not we fallback to v3 (the old attach/detach)
17:29:37 <scottda> nothing to do with API version
17:29:39 <ildikov> scottda: ah, ok
17:29:59 <mriedem> ^ is essentially what i was saying we're doing with placement today
17:30:07 <johnthetubaguy> mriedem: +1 that makes sense to me
17:30:08 <mriedem> we allow a fallback in ocata even though placement is required,
17:30:10 <mriedem> however,
17:30:12 <smcginnis> mriedem: +1
17:30:16 <mriedem> in pike i think we plan on dropping the fallback
17:30:39 <mriedem> so we're going to require a minimum placement microversion in pike, like how the Ironic virt driver requires a minimum Ironic microversion for their code
17:30:43 <johnthetubaguy> yeah, after a few cycles we drop the fallback, like we did with glance v1
17:30:47 <mriedem> johnthetubaguy: yup
17:30:59 <ildikov> mriedem: although we ask for the version in the current PoC as opposed to fall back in case of errors
17:31:19 <mriedem> ildikov: we're going to have to fallback to the old way if 3.27 isn't available
17:31:30 <ildikov> mriedem: yep, we will
17:31:31 <mriedem> i wouldn't be surprised if v3 itself won't be available
17:31:40 <mriedem> i.e. all of our grenade jobs
17:31:51 <mriedem> not available / not configured to use
17:31:54 <johnthetubaguy> yeah
17:31:59 <mriedem> because of the endpoint shenanigans you guys love to do
17:32:07 <scottda> I'm trying to fix that...
17:32:14 <scottda> We're discussing at the PTG
17:32:18 <ildikov> mriedem: :)
17:32:22 <scottda> Consistent Versioned Endpoints.
17:32:30 <mriedem> scottda: like drop v3 and do it all in v2.x ?
17:32:37 <scottda> Unfortunately, not possible to schedule a time and place for such discussions...
17:32:40 <mriedem> or volume endpoint just means, the thing you want
17:32:47 <scottda> not sure
17:32:53 <mriedem> scottda: you're not supposed to ask about scheduling things at the ptg
17:32:56 <mriedem> it will just work
17:32:58 <scottda> not sure what is possible, without breaking consumers of the catalog
17:33:02 <scottda> ha
17:33:10 <mriedem> it would be a monday or tuesday talk,
17:33:15 <scottda> yes
17:33:15 <mriedem> probably arch wg or something like that, idk
17:33:18 <mriedem> there are 20 wg's
17:33:25 <scottda> yeah, actually on the arch wg etherpad ATM
17:33:25 <mriedem> api wg
17:33:38 <mriedem> ok,
17:33:40 <mriedem> so first things first,
17:33:55 <mriedem> we get the cinder v3 in nova change passing some testing so we know things are cool
17:34:02 <mriedem> and worry about the microversion / fallback dance after that
17:34:08 <mriedem> when we actually need to start requesting 3.27
17:34:45 <scottda> #action scottda will post DNM patch for nova to use cinder v3
17:35:31 <ildikov> I like this scenario
17:35:31 <scottda> I think regular tempest/scenario running will then test cinder v3
17:35:34 <mriedem> actually isn't https://review.openstack.org/#/c/420201/ self-testing?
17:35:43 <mriedem> since you're hard-coding the default config?
17:36:13 <scottda> yah, actually it is.
17:36:20 <mriedem> and tempest jobs are failing...
17:36:45 <ildikov> it should be, yes
17:36:54 <scottda> yeah, I'll have a look at the failures more closely
17:37:11 <scottda> mriedem: Which tests run tempest/scenario for nova?
17:37:29 <mriedem> lots
17:37:32 <scottda> neutron-full?
17:37:34 <scottda> ok
17:37:35 <mriedem> test_volume_boot_pattern
17:37:37 <mriedem> is a big one
17:37:47 <ildikov> not just Tempest, even pep8 is failing there, so might be something other than V3
17:37:48 <scottda> yeah, sorry, I meant which jenkins job
17:37:57 <mriedem> http://logs.openstack.org/01/420201/7/check/gate-tempest-dsvm-neutron-full-ubuntu-xenial/68f7c49/console.html.gz#_2017-02-01_00_55_26_080245
17:38:02 <mriedem> ^ is a failure for a known fixed issue
17:38:04 <mriedem> just recheck your change
17:38:36 <mriedem> i rechecked it
17:38:36 <scottda> OK, I'll clean up test failures. That patch is a bit neglected.
17:39:08 <scottda> http://logs.openstack.org/01/420201/7/check/gate-tempest-dsvm-neutron-full-ubuntu-xenial/68f7c49/console.html.gz#_2017-02-01_00_46_19_145600
17:39:15 <scottda> Seems to be passing
17:39:22 <scottda> test_volume_boot_pattern
17:43:30 <smcginnis> mriedem: Did we want to try to set a Nova-Cinder PTG time?
17:43:34 <mriedem> i guess i never appreciated that we reconstruct the cinderclient on every call
17:43:37 <smcginnis> Or just trust that it will all come together?
17:43:44 <mriedem> smcginnis: we want to schedule it
17:43:45 <scottda> mriedem: I hate that we do that as well
17:43:47 <mriedem> so wed or thurs
17:44:12 <smcginnis> I have no set schedule for those days, so really whatever you would like to do, we can schedule around it.
17:44:18 <mriedem> smcginnis: wed or thurs morning work for you? 2 hour block?
17:44:27 <mriedem> 10-noon on wed?
17:44:39 <mriedem> or we could do it wed afternoon
17:44:45 <smcginnis> I was going to say 9, but yeah, 10 is probably better so everyone gets some coffee in first.
17:44:52 <mriedem> let the teams flush out whatever else on the first vertical team morning
17:44:56 <scottda> 10-noon wed will be pretty much first session of the week
17:45:00 <mriedem> right
17:45:07 <mriedem> so maybe thursday is better
17:45:07 <smcginnis> Morning is probably better than after lunch.
17:45:13 <mriedem> let's do thursday then
17:45:14 <smcginnis> Thrusday at 10
17:45:16 <mriedem> done
17:45:21 <ildikov> +1
17:45:28 <smcginnis> I'll add it to our etherpad.
17:45:35 <scottda> #info cinder and nova will meet Thurs at 10:00 at the PTG
17:45:50 <scottda> cinder people in nova room, and nova people in the cinder room
17:45:52 <scottda> :)
17:45:53 <johnthetubaguy> oh, its different auth token for every call, as its the user token, so its a different client, I guess
17:46:13 <ildikov> scottda: :)
17:46:37 <smcginnis> scottda: The jokes on you - we'll all just be in one big room all week. :P
17:46:41 <scottda> johnthetubaguy: Yes, but some novaclient calls (like attach) require 4 cinderclient calls. That could use only one instnace of cinderclient
17:47:05 <scottda> smcginnis: Ha. The Big Tent
17:47:06 <johnthetubaguy> scottda: oh, in that sense, good point
17:47:10 <smcginnis> scottda: LOL
17:47:23 <mriedem> johnthetubaguy: yeah i guess we do the same for neutron
17:47:24 * smcginnis pictures a circus tent in a big parking lot in ATL.
17:47:42 <mriedem> we don't do that for the placement (scheduler report client)
17:47:57 <ildikov> smcginnis: LOL
17:49:06 <smcginnis> OK, so devstack patch in flight for long term, scottda to recheck to get passing v3 results in the short term.
17:49:07 <hemna> no sliders
17:49:24 <smcginnis> Proof of concept will be worked for nova side of calling microversions.
17:49:30 <ildikov> we can use the Cinder-Nova etherpad to draft a few items to talk about at the PTG, along with the Nova spec
17:49:34 <smcginnis> And possible cleanup of multiple sessions.
17:49:49 <smcginnis> Anything else we need to figure out today?
17:49:51 <smcginnis> ildikov: +1
17:50:19 <ildikov> mriedem: the remove check_attach patch was hitting a non-related error, Jenkins looks good on it now
17:50:39 <ildikov> mriedem: johnthetubaguy: what do I need to do to get that thing finally done? :)
17:50:44 <johnthetubaguy> I should clean up that nova spec
17:50:57 <johnthetubaguy> or did we have a new one now?
17:51:06 <smcginnis> johnthetubaguy: Oh right. Not that I've seen.
17:51:27 <johnthetubaguy> I can get that tidied up
17:51:28 <ildikov> johnthetubaguy: I don't think we have a new one for Nova
17:51:39 <johnthetubaguy> the spec
17:51:41 <mriedem> johnthetubaguy: it's still yours
17:51:58 <ildikov> johnthetubaguy: we need to get the current one in shape based on the comments from mriedem and others
17:52:19 <smcginnis> We've at least finalized the Cinder side, so hopefully that helps with the nova spec.
17:52:51 <ildikov> we also have a PoC patch: https://review.openstack.org/#/c/330285/
17:53:07 <ildikov> jgriffith: made the attach/detach parts working
17:53:22 <smcginnis> #link https://review.openstack.org/#/c/330285/ Nova PoC for new attach calls
17:53:24 <ildikov> still have many things to figure out, but at least we have a starting point
17:53:37 <ildikov> smcginnis: thanks :)
17:53:58 <smcginnis> ildikov: Just trying to make the meeting logs look pretty. ;)
17:54:20 * ildikov is thankful for that :)
17:54:44 <smcginnis> OK, I gotta run. Folks on the lunch train are starting to look angry. :)
17:54:47 <ildikov> johnthetubaguy: so you can look into the PoC above too to see what we hacked so far to make things work
17:55:11 <ildikov> smcginnis: I think we're mainly good for today
17:55:18 <ildikov> smcginnis: enjoy lunch :)
17:55:40 <johnthetubaguy> ildikov: gives me a good excuse to fix that
17:55:56 <johnthetubaguy> well, I mean look at those PoC patches, both ways
17:56:04 <ildikov> mriedem: johnthetubaguy: the PoC depends on the Remove check_attach patch as it seemed reasonable to order things
17:56:17 * johnthetubaguy nods
17:56:45 <ildikov> and rebasing is a nightmare, so I would love to see that merged as it's considered to be a bugfix
17:57:24 <johnthetubaguy> we are free to merge that now, I think, give we are open for pike
17:57:37 <ildikov> mriedem: johnthetubaguy: so pretty please review it :)
17:57:50 <mriedem> umm,
17:57:53 <ildikov> mriedem: johnthetubaguy: or tell me who from the cores would willing to look into that change
17:57:56 <mriedem> johnthetubaguy: i'd hold off until we release pike
17:58:00 <johnthetubaguy> yeah, its going to take a few hours to get my head back into that again
17:58:05 <mriedem> in case we need to bump the service version for anything in the next week
17:58:15 <johnthetubaguy> mriedem: yeah, thats true, service version
17:58:15 <mriedem> s/pike/ocata/
17:58:27 <johnthetubaguy> hehe, yeah ocata
17:58:28 <ildikov> mriedem: ;)
17:58:36 <mriedem> so basically can't land that until we cut rc2
17:58:39 <mriedem> which is 2/16 at the latest
17:59:02 <johnthetubaguy> yeah
17:59:07 <ildikov> mriedem: johnthetubaguy: if you could give it a review run earlier rather than later that would still be good
17:59:20 <johnthetubaguy> we could be ready to +W in the PTG session, I suppose
17:59:34 <ildikov> so any issues it might has could be fixed by the time it can get merged
18:00:31 <ildikov> ok, please review it, so I can fix the issues, I don't want to spend time on this at the PTG
18:00:55 <ildikov> it should be more straightforward than that
18:01:29 <ildikov> ok, we're out of time and I think we also have plans
18:02:08 <ildikov> we can double check the progress next week and maybe try to add items to the PTG session agenda so we surely talk about all the items we see important now
18:02:23 <ildikov> anything else from anyone today?
18:03:57 <ildikov> alright, thanks everyone
18:04:29 <ildikov> #endmeeting