00:01:11 <cyeoh> #startmeeting #nova api
00:01:12 <openstack> Meeting started Fri Oct 24 00:01:11 2014 UTC and is due to finish in 60 minutes.  The chair is cyeoh. Information about MeetBot at http://wiki.debian.org/MeetBot.
00:01:13 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
00:01:15 <openstack> The meeting name has been set to '_nova_api'
00:01:33 <eliqiao> hi Chris, hi all
00:01:37 <gmann> hi
00:01:37 <oomichi> hi!
00:01:38 <cyeoh> Hi who's here today?
00:01:39 <alex_xu> hi
00:02:14 <cyeoh> ok let's get started
00:02:19 <cyeoh> #topic v2.1 on v3
00:02:33 <cyeoh> probably should just call it v2.1 work now :-)
00:02:57 <cyeoh> anything that we need to discuss about it? Any urgent reviews? Anything missing?
00:03:21 <eliqiao> see from http://etherpad.openstack.org/p/v2_1_WorkList, there are share test case requires?
00:03:22 <oomichi> nothing special from me, that seems nice progress.
00:04:02 <alex_xu> no more, just reminder I list the unittest sharing task on the working list, if anyone work on it, need mark name after the task.
00:04:09 <alex_xu> # link https://etherpad.openstack.org/p/v2_1_WorkList
00:04:27 <cyeoh> yea so some tests were kept separate because it was too hard to do at the time.
00:04:34 <cyeoh> But we should clean them up when we have time
00:04:44 <cyeoh> alex_xu: thanks for making that list
00:04:49 <alex_xu> cyeoh, np
00:04:56 <alex_xu> should we share integrated test?
00:05:19 <cyeoh> I think for the moment stay away from admin_only_action_common though because I'm waiting for the rest to merge so I can split those up into individual extensions
00:05:26 <cyeoh> its left over icehouse work...
00:05:43 <cyeoh> alex_xu - I think its worth putting on the todo list but is low priority?
00:05:49 <cyeoh> Can see where Jay's work goes too
00:05:57 <cyeoh> as we'd want v2.1 to use that
00:06:13 <alex_xu> yes, Jay is working on separated integrated test in separated file
00:06:19 <alex_xu> #link https://review.openstack.org/#/c/123296/
00:06:45 <alex_xu> Jay's work is on v2 api integrated test, we should stop it?
00:07:07 <cyeoh> well we might want to suggest doing the shared work first
00:07:23 <cyeoh> I think it should be fairly straightforward, just a bit boring ;-)
00:08:12 <cyeoh> anything else on v2.1?
00:08:21 <eliqiao> besides, in v2.1 we change return webob.Response(status_int=202) to wsgi decorator, this makes some trouble unit test case share , we need to add an extra private function _check_status to check the controller return value, we need to do this in each seperated unit test file, how about define it a v21/v2 unittest class,  and make all unit test case shared from them?
00:08:38 <eliqiao> an example #link https://review.openstack.org/#/c/119013/5/nova/tests/api/openstack/compute/contrib/test_floating_ip_dns.py
00:09:28 <cyeoh> eliqiao: yea I was thinking about that the other day. We'd need a common base class for all the api unittests
00:09:34 <cyeoh> or maybe a mixin
00:09:40 <cyeoh> not sure what the best way to go is
00:09:41 <oomichi> eliqiao: that is a nice idea, I prefer it.
00:09:48 <alex_xu> +1
00:09:59 <eliqiao> yes, cool, that will be great
00:10:26 <cyeoh> eliqiao: would you like to put it on the worklist?
00:10:39 <eliqiao> cyeoh: sure,
00:10:43 <gmann> eliqiao: nice idea
00:11:10 <eliqiao> thx.
00:11:37 <cyeoh> ok. anything else on v2.1?
00:11:45 <eliqiao> nothing from me.
00:11:55 <alex_xu> that's all from me
00:12:02 <cyeoh> #topic microversions
00:12:20 <cyeoh> so I have updated the spec:
00:12:46 <cyeoh> #link https://review.openstack.org/127127
00:13:06 <oomichi> cyeoh: thanks, that is very important spec now:)
00:13:15 <cyeoh> please have a look at it when you have time
00:13:36 <cyeoh> also I have a bit of prototype code:
00:13:52 <cyeoh> https://github.com/cyeoh/microversions_poc
00:13:52 <oomichi> cyeoh: great, will you push it to gerrit?
00:14:04 <oomichi> ah, nice.
00:14:16 <cyeoh> oomichi: so this prototype code isn't linked to Nova - it seemed easier to experiment in a standalone environment
00:14:21 <cyeoh> was easier to debug :-)
00:14:46 <oomichi> yeah, that is readable :-)
00:14:56 <cyeoh> anyway I'd appreciate any feedback
00:15:10 <cyeoh> oomichi: I did add to the spec that we'd need versioning for validation schema
00:15:34 <cyeoh> I think its possible, just wasn't sure if you had a preference on how it was implemented
00:15:40 <oomichi> cyeoh: nice work, I will put some comments on that.
00:15:52 <cyeoh> oomichi: cool, thanks!
00:16:10 <oomichi> cyeoh: so how to put some comments for PoC?
00:16:14 <jaypipes> it's not the function we need to version, though.. it's the expected payload and response
00:16:36 <cyeoh> jaypipes: it ends up being the same thing
00:17:05 <jaypipes> this just seems to me like we're going to be super-decorating things even more than they already are.
00:17:08 <cyeoh> jaypipes: and we also need to version semantics - eg payload/response may be the same, but behavior different
00:17:24 <alex_xu> emm... the versioned json-schema can't be done versioned function
00:17:25 <cyeoh> I think its better than having lots of if/else
00:18:29 <cyeoh> jaypipes: so I think the main thing is we get the version information down into the api controller (which we do by attaching the version object to the request object)
00:18:50 <cyeoh> jaypipes: after that we can look at a case by case basis as to whether if/else or decorator works better
00:19:25 <oomichi> current json-schema can be extended based on loaded extensions. so I'd like to dig new versioned idea for knowing the differences.
00:19:48 <cyeoh> oomichi: cool.
00:19:49 <jaypipes> why do we want to attach a version object to a request object?
00:19:55 <jaypipes> sorry, I'm totally not getting this.
00:20:17 <cyeoh> jaypipes: so we don't have to change the signature of all the api methods
00:20:54 <alex_xu> Can I selling my propose about versioning json-schema https://github.com/soulxu/nova-v3-api-doc/commit/2d8489650ba221a3019b7db548ef8ce7a8a186dd ?
00:20:55 <cyeoh> wsgi will interpret the version header and convert it to a version object. It will then pass it down to the method before calling it
00:21:53 <cyeoh> jaypipes: the api methods can then pull out the version request information when deciding how to behave
00:22:07 <cyeoh> jaypipes: and when it comes down to it, a version header request is basically part of the request.
00:23:25 <cyeoh> alex_xu: I will have to have another look at that - it does need to be something that we can roll out on an extension by extension basis
00:24:02 <alex_xu> cyeoh, thanks
00:24:12 <oomichi> for one versioning point, current v2.0 provide deployers with a right for choosing extensions. but current microversioning idea seems to provide it for clients also by accept header.
00:24:17 <jaypipes> friggin extensions... :(
00:24:19 <cyeoh> alex_xu: so IIRC one concern I had about the proposal was that perhaps not all body information was convertible to a flat dict
00:25:04 <cyeoh> oomichi: well only in the sense that they can choose what version of the api they want to interact with
00:25:19 <jaypipes> cyeoh: what is the point of that though?
00:25:28 <alex_xu> cyeoh, other case I think it is [{}, {}], but I think it's fine, not very hard to implement it
00:25:33 <cyeoh> oomichi: and they don't get to define what configuration that is
00:25:50 <cyeoh> oomichi: without help from the deployer
00:26:08 <cyeoh> jaypipes: sorry I don't understand your question?
00:27:03 <oomichi> cyeoh: yeah, I know. I meaned about compatible changes. compatible changes don't hurt clients, so it seems unnecessary to choose Y version from clients.
00:27:10 <jaypipes> I'm looking at your example, and I just don't see what the point of having a version 1.2.3 that does add() by adding two numbers together, and another add() at version 2.0.0 that adds the numbers together and tacks on another 100...
00:27:32 <jaypipes> cyeoh: I guess I have to see a realistic example to see what benefit this approach gives the user of our APIs.
00:27:42 <cyeoh> jaypipes: so that was just to demonstrate that it acts differently depending on version
00:28:13 <cyeoh> a nova api example could be a change in semantic behaviour of the api
00:28:14 <jaypipes> because frankly, if something changes semantics, we should not have the API controller (the same object) doing two completely different things for the same resource.
00:28:20 <jaypipes> it just is madness.
00:28:30 <cyeoh> jaypipes: um, for different versions its not
00:28:43 <cyeoh> unless you want to get stuck with the same semantic behaviour forever
00:28:56 <jaypipes> cyeoh: having different semantics for the same resource operation within the exact same API controller object is madness.
00:29:23 <jaypipes> we are setting these modules up to be something that can never be controlled and will grow without being able to shink.
00:29:43 <cyeoh> jaypipes: thats not true, we can drop old versions
00:29:52 <cyeoh> just like we can drop v2 eventually
00:29:58 <jaypipes> I don't want to have an API controller that has eleventy different ways of doing the same thing.
00:30:05 <oomichi> jaypipes: that meens we need to have different API URL for the same resource?
00:30:32 <jaypipes> oomichi: not sure I follow you...
00:30:36 <cyeoh> jaypipes: if we want to be able to make backwards incompatible changes then we do need controllers to be able to do that
00:30:44 <jaypipes> oomichi: I'm talking about the code structure of the controller modules.
00:31:11 <jaypipes> cyeoh: why? why not just have a separate controller class for a new major version of the API?
00:31:16 <oomichi> ah, OK. I maybe misunderstood.
00:31:26 <cyeoh> jaypipes: that was v3
00:31:31 <jaypipes> cyeoh: no...
00:31:38 <cyeoh> that approach got explicitly rejected because of code duplication
00:32:09 <jaypipes> fair point.
00:32:22 <jaypipes> I guess I was looking for a solution that was:
00:32:44 <jaypipes> a) more declarative w.r.t. *how* an API changed from one microversion to another, and
00:33:15 <jaypipes> b) handled things like translation of response or request payload by looking at a JSONSchema object and appliying a changeset to it.
00:33:56 <cyeoh> so I think with jsonchema input validation and response schema (from tempest) we effectively have a
00:34:28 <cyeoh> for b) that was along the lines of how the original v2.1/v3 worked, but it got rejected for being too complex
00:34:45 <cyeoh> eg. doing translations got rejected
00:34:56 <cyeoh> so here we are now with something a lot simpler.
00:35:12 <oomichi> we have some prototype based on b) before the Atlanta summit, but that was denied now.
00:35:16 <jaypipes> it's not about validation. it's about being able to have a JSONSchema diff be the driver of automated translation of request for API resource at version X.Y.Z to be done for a server on API version X.Y.Z+N
00:35:28 <cyeoh> jaypipes: yes thats what we want
00:35:35 <cyeoh> sorry thats what we had, not want
00:35:56 <cyeoh> though it was just between v2.1 and v3 the idea is it would extend into microversions too
00:36:13 <cyeoh> but that was considered to complicated - I don't think we'd get it through nova-specs now
00:36:19 <jaypipes> the reason it was too complex was because of the stupid API extensions.
00:36:39 <cyeoh> api extensions didn't really complicate it anymore. And like it or not we're stuck with API extensions for now
00:36:44 <cyeoh> because we need to support v2
00:37:09 <jaypipes> honestly, I'd almost rather just make v2 read only, get a decent API framework in place, and work on a new API.
00:37:16 <jaypipes> I can't stand the cruft that's currently in there.
00:37:26 <jaypipes> s/read only/immutable
00:37:52 <jaypipes> prblem is all the damn product managers constantly pushing for new features and unwilling to wait for a proper cleanup or refactoring.
00:37:53 <cyeoh> jaypipes: that's not going to happen though - we won't get permission to do that - and it will end up just halting api changes for a couple of cycles
00:38:41 <cyeoh> I'm just desperate to be able to do some backwards incompatible api changes - we desperately need tasks - and that is a big improvement for users
00:39:18 <jaypipes> yeah. and we could have had it done if our contributor community had the guts to stand up to product managers and customers that want their cake and eat it too.
00:39:22 <cyeoh> lack of a tasks api causes all sorts of issues (we get lots of bug reports because of it)
00:39:52 <jaypipes> actually, the tasks *internal* API should and can be worked on separately from the front-end rest api.
00:40:14 <cyeoh> jaypipes: true but I don't think it'll happen until people see that the API changes are possible
00:40:19 <jaypipes> problem is we got distracted with crap like cells and server groups and all sorts of pointless scheduler crud.
00:40:46 <cyeoh> looks like cells will be big in kilo too, hopefully can get some tasks work still done in Kilo
00:41:20 <jaypipes> I doubt cells will be big in kilo. it's a pile of turds and technical debt and one of the only guys who knew it well enough has left.
00:41:20 <cyeoh> anyway I think we just have to work with what is possible (eg what we can get approved). I've had enough of fighting impossible fights :-)
00:41:43 <jaypipes> yeah, I know you have history here. :) sorry :
00:42:21 <cyeoh> np, just trying to design api microversions to what I think we can get through
00:43:00 <cyeoh> and get it implemented early in kilo (its not a lot of code)
00:43:31 <jaypipes> ok
00:43:34 <cyeoh> I want to leave room for tasks to get done in kilo as well (at least a first demonstration part - say server create)
00:43:53 <jaypipes> tell me though.. how do I get rid of API extensions using this microversioning framework?
00:44:26 <jaypipes> it really is the extensions that kill everything.
00:44:34 <cyeoh> jaypipes: so first you need to run the debate about what is in and out - eg what is our fixed API going to be.
00:44:44 <cyeoh> jaypipes: and then we'll do a major version bump
00:44:57 <cyeoh> jaypipes: and a request from then onwards will only expose the fixed api
00:45:04 <jaypipes> for instance, given our current (horrible) API extension framework, how might I change os-floating-ips to just /servers/$server/floating-ips?
00:46:12 <jaypipes> cyeoh: but the major version bump would need to remove *all* the API extensions at once.... otherwise, each removal of an API extension would be another major API version increment, because it's backwards-incompatible
00:46:25 <jaypipes> cyeoh: thus my stance that API extensions are the root of all API evil. :)
00:46:39 <cyeoh> internally I think we'd implement both (so those asking for earlier versions get still get the old one), but we'd only expose the ones appropriate for the version requested
00:47:10 <cyeoh> jaypipes: so underneath it all we'd still use plugins. We'd just require specific ones to always be loaded - eg what we currently have for core
00:47:19 <jaypipes> cyeoh: and if we're ghoing to need to do one giant removal of all API extensions in order to actually do a major vesion increment, then we're back to square one, with nobody willing to make the hard decisiosn, and we'll keep on limping our API along.
00:47:34 <oomichi> how about JSON-Home? JSON-Home can provide each API URL is stable/deprecated.
00:47:35 <cyeoh> jaypipes: we can gradually increase the size of core.
00:47:50 <cyeoh> jaypipes: doesn't need to be big bang.
00:47:55 <oomichi> as common way.
00:48:11 <jaypipes> cyeoh: and what if I propose to split the API into a user API and an operator API?
00:48:43 <cyeoh> jaypipes: listening on different ports?
00:48:55 <jaypipes> no... entirely different endpoints.
00:49:20 <cyeoh> jaypipes: I'd need to look into the implementation details of that - I think there was a similar request on the operators list recently
00:49:25 <jaypipes> but because of API extensions, there would be old extensions that pointed to a different API endpoint...
00:49:28 <cyeoh> by the catalyst guys
00:50:44 <cyeoh> jaypipes: so I think that may be true of some image location things
00:51:01 <cyeoh> (though I'd like to get rid of proxying in the long term anyway)
00:51:12 <jaypipes> sure
00:51:36 <cyeoh> but I don't think we have a lot of apis that point to others?
00:51:59 <jaypipes> I guess I'm just tired of being handcuffed when it comes to developing new things in the *right* way... and not stuck using an API framework that is brittle, encourages vendorization, and isn't dicoverable
00:51:59 <cyeoh> I guess some of the admin stuff would need their own resource paths rather than piggy backing on the user ones
00:52:25 <jaypipes> cyeoh: all of volume, image, and network point to other APIs...
00:52:50 <cyeoh> well discoverability is definitely on our list. We want json-home and the json schema - its how much devel bandwidth we have in Kilo, and microversion needs to take priority
00:52:55 <jaypipes> cyeoh: with a ton of junk in the code to "translate" from Nova APIs point of view (for differtent API versions and API extensions)
00:53:25 <cyeoh> jaypipes: well if we want to support old versions like v2 we have no choice
00:53:41 <cyeoh> and I think its clear the community will not support big bang upgrades to the Nova API
00:54:24 <cyeoh> anyway before we run out of time. I just want to cover the summit briefly
00:54:39 <cyeoh> #topic summit
00:55:04 <cyeoh> so I won't be able to make it to summit, or attend remotely this time - since I couldn't travel I'm on vacation instead
00:55:09 <cyeoh> (with family)
00:55:53 <cyeoh> oomichi, jaypipes: there's a microversions session scheduled, so I wanted to make sure you at least understood what I was proposing even if you're not happy with it
00:56:16 <cyeoh> oomichi, jaypipes: so please feel free to ping/email me anytime to ask me questions about it.
00:56:40 <cyeoh> I'll be on leave from 31st October to 7th of November inclusive.
00:56:42 <oomichi> cyeoh: OK, I got it. will do it soon after reading the spec.
00:56:53 <cyeoh> oomichi: ok, thanks
00:57:10 <jaypipes> cyeoh: I will, and forgive my harshness tonight please... just been frustrated this evening trying to uncruft the "unit" tests around resource tracker and I'm unfortunately taken my frustration out on you. sorry :(
00:57:44 <cyeoh> jaypipes: no probs. Don't look at the nova api unittests - lots of cleanup needed there :-) (Which we're going to work on)
00:58:31 <cyeoh> so I think we should probably cancel the next two api meetings as well. The first because its likely people will be travelling (or about to). And the second because its summit week.
00:58:37 <cyeoh> does that sound ok?
00:59:09 <oomichi> cyeoh: OK for me.
00:59:16 <alex_xu> ok for me
00:59:20 <alex_xu> and  just reselling my propose again http://soulxu.github.io/blog/2014/09/12/one-option-for-nova-api/  .... :)
00:59:23 <gmann> yes
00:59:36 <cyeoh> alex_xu: I will have another look at it :)
00:59:48 <alex_xu> cyeoh, thank you!
01:00:13 <cyeoh> ok we have 1 minute left. Anything else?
01:00:48 <cyeoh> ok, thanks everyone for coming. cya!
01:00:48 <oomichi> good for me, thanks:-)
01:00:56 <gmann> Thanks all
01:01:00 <cyeoh> #endmeeting