00:01:11 #startmeeting #nova api 00:01:12 Meeting started Fri Oct 24 00:01:11 2014 UTC and is due to finish in 60 minutes. The chair is cyeoh. Information about MeetBot at http://wiki.debian.org/MeetBot. 00:01:13 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 00:01:15 The meeting name has been set to '_nova_api' 00:01:33 hi Chris, hi all 00:01:37 hi 00:01:37 hi! 00:01:38 Hi who's here today? 00:01:39 hi 00:02:14 ok let's get started 00:02:19 #topic v2.1 on v3 00:02:33 probably should just call it v2.1 work now :-) 00:02:57 anything that we need to discuss about it? Any urgent reviews? Anything missing? 00:03:21 see from http://etherpad.openstack.org/p/v2_1_WorkList, there are share test case requires? 00:03:22 nothing special from me, that seems nice progress. 00:04:02 no more, just reminder I list the unittest sharing task on the working list, if anyone work on it, need mark name after the task. 00:04:09 # link https://etherpad.openstack.org/p/v2_1_WorkList 00:04:27 yea so some tests were kept separate because it was too hard to do at the time. 00:04:34 But we should clean them up when we have time 00:04:44 alex_xu: thanks for making that list 00:04:49 cyeoh, np 00:04:56 should we share integrated test? 00:05:19 I think for the moment stay away from admin_only_action_common though because I'm waiting for the rest to merge so I can split those up into individual extensions 00:05:26 its left over icehouse work... 00:05:43 alex_xu - I think its worth putting on the todo list but is low priority? 00:05:49 Can see where Jay's work goes too 00:05:57 as we'd want v2.1 to use that 00:06:13 yes, Jay is working on separated integrated test in separated file 00:06:19 #link https://review.openstack.org/#/c/123296/ 00:06:45 Jay's work is on v2 api integrated test, we should stop it? 00:07:07 well we might want to suggest doing the shared work first 00:07:23 I think it should be fairly straightforward, just a bit boring ;-) 00:08:12 anything else on v2.1? 00:08:21 besides, in v2.1 we change return webob.Response(status_int=202) to wsgi decorator, this makes some trouble unit test case share , we need to add an extra private function _check_status to check the controller return value, we need to do this in each seperated unit test file, how about define it a v21/v2 unittest class, and make all unit test case shared from them? 00:08:38 an example #link https://review.openstack.org/#/c/119013/5/nova/tests/api/openstack/compute/contrib/test_floating_ip_dns.py 00:09:28 eliqiao: yea I was thinking about that the other day. We'd need a common base class for all the api unittests 00:09:34 or maybe a mixin 00:09:40 not sure what the best way to go is 00:09:41 eliqiao: that is a nice idea, I prefer it. 00:09:48 +1 00:09:59 yes, cool, that will be great 00:10:26 eliqiao: would you like to put it on the worklist? 00:10:39 cyeoh: sure, 00:10:43 eliqiao: nice idea 00:11:10 thx. 00:11:37 ok. anything else on v2.1? 00:11:45 nothing from me. 00:11:55 that's all from me 00:12:02 #topic microversions 00:12:20 so I have updated the spec: 00:12:46 #link https://review.openstack.org/127127 00:13:06 cyeoh: thanks, that is very important spec now:) 00:13:15 please have a look at it when you have time 00:13:36 also I have a bit of prototype code: 00:13:52 https://github.com/cyeoh/microversions_poc 00:13:52 cyeoh: great, will you push it to gerrit? 00:14:04 ah, nice. 00:14:16 oomichi: so this prototype code isn't linked to Nova - it seemed easier to experiment in a standalone environment 00:14:21 was easier to debug :-) 00:14:46 yeah, that is readable :-) 00:14:56 anyway I'd appreciate any feedback 00:15:10 oomichi: I did add to the spec that we'd need versioning for validation schema 00:15:34 I think its possible, just wasn't sure if you had a preference on how it was implemented 00:15:40 cyeoh: nice work, I will put some comments on that. 00:15:52 oomichi: cool, thanks! 00:16:10 cyeoh: so how to put some comments for PoC? 00:16:14 it's not the function we need to version, though.. it's the expected payload and response 00:16:36 jaypipes: it ends up being the same thing 00:17:05 this just seems to me like we're going to be super-decorating things even more than they already are. 00:17:08 jaypipes: and we also need to version semantics - eg payload/response may be the same, but behavior different 00:17:24 emm... the versioned json-schema can't be done versioned function 00:17:25 I think its better than having lots of if/else 00:18:29 jaypipes: so I think the main thing is we get the version information down into the api controller (which we do by attaching the version object to the request object) 00:18:50 jaypipes: after that we can look at a case by case basis as to whether if/else or decorator works better 00:19:25 current json-schema can be extended based on loaded extensions. so I'd like to dig new versioned idea for knowing the differences. 00:19:48 oomichi: cool. 00:19:49 why do we want to attach a version object to a request object? 00:19:55 sorry, I'm totally not getting this. 00:20:17 jaypipes: so we don't have to change the signature of all the api methods 00:20:54 Can I selling my propose about versioning json-schema https://github.com/soulxu/nova-v3-api-doc/commit/2d8489650ba221a3019b7db548ef8ce7a8a186dd ? 00:20:55 wsgi will interpret the version header and convert it to a version object. It will then pass it down to the method before calling it 00:21:53 jaypipes: the api methods can then pull out the version request information when deciding how to behave 00:22:07 jaypipes: and when it comes down to it, a version header request is basically part of the request. 00:23:25 alex_xu: I will have to have another look at that - it does need to be something that we can roll out on an extension by extension basis 00:24:02 cyeoh, thanks 00:24:12 for one versioning point, current v2.0 provide deployers with a right for choosing extensions. but current microversioning idea seems to provide it for clients also by accept header. 00:24:17 friggin extensions... :( 00:24:19 alex_xu: so IIRC one concern I had about the proposal was that perhaps not all body information was convertible to a flat dict 00:25:04 oomichi: well only in the sense that they can choose what version of the api they want to interact with 00:25:19 cyeoh: what is the point of that though? 00:25:28 cyeoh, other case I think it is [{}, {}], but I think it's fine, not very hard to implement it 00:25:33 oomichi: and they don't get to define what configuration that is 00:25:50 oomichi: without help from the deployer 00:26:08 jaypipes: sorry I don't understand your question? 00:27:03 cyeoh: yeah, I know. I meaned about compatible changes. compatible changes don't hurt clients, so it seems unnecessary to choose Y version from clients. 00:27:10 I'm looking at your example, and I just don't see what the point of having a version 1.2.3 that does add() by adding two numbers together, and another add() at version 2.0.0 that adds the numbers together and tacks on another 100... 00:27:32 cyeoh: I guess I have to see a realistic example to see what benefit this approach gives the user of our APIs. 00:27:42 jaypipes: so that was just to demonstrate that it acts differently depending on version 00:28:13 a nova api example could be a change in semantic behaviour of the api 00:28:14 because frankly, if something changes semantics, we should not have the API controller (the same object) doing two completely different things for the same resource. 00:28:20 it just is madness. 00:28:30 jaypipes: um, for different versions its not 00:28:43 unless you want to get stuck with the same semantic behaviour forever 00:28:56 cyeoh: having different semantics for the same resource operation within the exact same API controller object is madness. 00:29:23 we are setting these modules up to be something that can never be controlled and will grow without being able to shink. 00:29:43 jaypipes: thats not true, we can drop old versions 00:29:52 just like we can drop v2 eventually 00:29:58 I don't want to have an API controller that has eleventy different ways of doing the same thing. 00:30:05 jaypipes: that meens we need to have different API URL for the same resource? 00:30:32 oomichi: not sure I follow you... 00:30:36 jaypipes: if we want to be able to make backwards incompatible changes then we do need controllers to be able to do that 00:30:44 oomichi: I'm talking about the code structure of the controller modules. 00:31:11 cyeoh: why? why not just have a separate controller class for a new major version of the API? 00:31:16 ah, OK. I maybe misunderstood. 00:31:26 jaypipes: that was v3 00:31:31 cyeoh: no... 00:31:38 that approach got explicitly rejected because of code duplication 00:32:09 fair point. 00:32:22 I guess I was looking for a solution that was: 00:32:44 a) more declarative w.r.t. *how* an API changed from one microversion to another, and 00:33:15 b) handled things like translation of response or request payload by looking at a JSONSchema object and appliying a changeset to it. 00:33:56 so I think with jsonchema input validation and response schema (from tempest) we effectively have a 00:34:28 for b) that was along the lines of how the original v2.1/v3 worked, but it got rejected for being too complex 00:34:45 eg. doing translations got rejected 00:34:56 so here we are now with something a lot simpler. 00:35:12 we have some prototype based on b) before the Atlanta summit, but that was denied now. 00:35:16 it's not about validation. it's about being able to have a JSONSchema diff be the driver of automated translation of request for API resource at version X.Y.Z to be done for a server on API version X.Y.Z+N 00:35:28 jaypipes: yes thats what we want 00:35:35 sorry thats what we had, not want 00:35:56 though it was just between v2.1 and v3 the idea is it would extend into microversions too 00:36:13 but that was considered to complicated - I don't think we'd get it through nova-specs now 00:36:19 the reason it was too complex was because of the stupid API extensions. 00:36:39 api extensions didn't really complicate it anymore. And like it or not we're stuck with API extensions for now 00:36:44 because we need to support v2 00:37:09 honestly, I'd almost rather just make v2 read only, get a decent API framework in place, and work on a new API. 00:37:16 I can't stand the cruft that's currently in there. 00:37:26 s/read only/immutable 00:37:52 prblem is all the damn product managers constantly pushing for new features and unwilling to wait for a proper cleanup or refactoring. 00:37:53 jaypipes: that's not going to happen though - we won't get permission to do that - and it will end up just halting api changes for a couple of cycles 00:38:41 I'm just desperate to be able to do some backwards incompatible api changes - we desperately need tasks - and that is a big improvement for users 00:39:18 yeah. and we could have had it done if our contributor community had the guts to stand up to product managers and customers that want their cake and eat it too. 00:39:22 lack of a tasks api causes all sorts of issues (we get lots of bug reports because of it) 00:39:52 actually, the tasks *internal* API should and can be worked on separately from the front-end rest api. 00:40:14 jaypipes: true but I don't think it'll happen until people see that the API changes are possible 00:40:19 problem is we got distracted with crap like cells and server groups and all sorts of pointless scheduler crud. 00:40:46 looks like cells will be big in kilo too, hopefully can get some tasks work still done in Kilo 00:41:20 I doubt cells will be big in kilo. it's a pile of turds and technical debt and one of the only guys who knew it well enough has left. 00:41:20 anyway I think we just have to work with what is possible (eg what we can get approved). I've had enough of fighting impossible fights :-) 00:41:43 yeah, I know you have history here. :) sorry : 00:42:21 np, just trying to design api microversions to what I think we can get through 00:43:00 and get it implemented early in kilo (its not a lot of code) 00:43:31 ok 00:43:34 I want to leave room for tasks to get done in kilo as well (at least a first demonstration part - say server create) 00:43:53 tell me though.. how do I get rid of API extensions using this microversioning framework? 00:44:26 it really is the extensions that kill everything. 00:44:34 jaypipes: so first you need to run the debate about what is in and out - eg what is our fixed API going to be. 00:44:44 jaypipes: and then we'll do a major version bump 00:44:57 jaypipes: and a request from then onwards will only expose the fixed api 00:45:04 for instance, given our current (horrible) API extension framework, how might I change os-floating-ips to just /servers/$server/floating-ips? 00:46:12 cyeoh: but the major version bump would need to remove *all* the API extensions at once.... otherwise, each removal of an API extension would be another major API version increment, because it's backwards-incompatible 00:46:25 cyeoh: thus my stance that API extensions are the root of all API evil. :) 00:46:39 internally I think we'd implement both (so those asking for earlier versions get still get the old one), but we'd only expose the ones appropriate for the version requested 00:47:10 jaypipes: so underneath it all we'd still use plugins. We'd just require specific ones to always be loaded - eg what we currently have for core 00:47:19 cyeoh: and if we're ghoing to need to do one giant removal of all API extensions in order to actually do a major vesion increment, then we're back to square one, with nobody willing to make the hard decisiosn, and we'll keep on limping our API along. 00:47:34 how about JSON-Home? JSON-Home can provide each API URL is stable/deprecated. 00:47:35 jaypipes: we can gradually increase the size of core. 00:47:50 jaypipes: doesn't need to be big bang. 00:47:55 as common way. 00:48:11 cyeoh: and what if I propose to split the API into a user API and an operator API? 00:48:43 jaypipes: listening on different ports? 00:48:55 no... entirely different endpoints. 00:49:20 jaypipes: I'd need to look into the implementation details of that - I think there was a similar request on the operators list recently 00:49:25 but because of API extensions, there would be old extensions that pointed to a different API endpoint... 00:49:28 by the catalyst guys 00:50:44 jaypipes: so I think that may be true of some image location things 00:51:01 (though I'd like to get rid of proxying in the long term anyway) 00:51:12 sure 00:51:36 but I don't think we have a lot of apis that point to others? 00:51:59 I guess I'm just tired of being handcuffed when it comes to developing new things in the *right* way... and not stuck using an API framework that is brittle, encourages vendorization, and isn't dicoverable 00:51:59 I guess some of the admin stuff would need their own resource paths rather than piggy backing on the user ones 00:52:25 cyeoh: all of volume, image, and network point to other APIs... 00:52:50 well discoverability is definitely on our list. We want json-home and the json schema - its how much devel bandwidth we have in Kilo, and microversion needs to take priority 00:52:55 cyeoh: with a ton of junk in the code to "translate" from Nova APIs point of view (for differtent API versions and API extensions) 00:53:25 jaypipes: well if we want to support old versions like v2 we have no choice 00:53:41 and I think its clear the community will not support big bang upgrades to the Nova API 00:54:24 anyway before we run out of time. I just want to cover the summit briefly 00:54:39 #topic summit 00:55:04 so I won't be able to make it to summit, or attend remotely this time - since I couldn't travel I'm on vacation instead 00:55:09 (with family) 00:55:53 oomichi, jaypipes: there's a microversions session scheduled, so I wanted to make sure you at least understood what I was proposing even if you're not happy with it 00:56:16 oomichi, jaypipes: so please feel free to ping/email me anytime to ask me questions about it. 00:56:40 I'll be on leave from 31st October to 7th of November inclusive. 00:56:42 cyeoh: OK, I got it. will do it soon after reading the spec. 00:56:53 oomichi: ok, thanks 00:57:10 cyeoh: I will, and forgive my harshness tonight please... just been frustrated this evening trying to uncruft the "unit" tests around resource tracker and I'm unfortunately taken my frustration out on you. sorry :( 00:57:44 jaypipes: no probs. Don't look at the nova api unittests - lots of cleanup needed there :-) (Which we're going to work on) 00:58:31 so I think we should probably cancel the next two api meetings as well. The first because its likely people will be travelling (or about to). And the second because its summit week. 00:58:37 does that sound ok? 00:59:09 cyeoh: OK for me. 00:59:16 ok for me 00:59:20 and just reselling my propose again http://soulxu.github.io/blog/2014/09/12/one-option-for-nova-api/ .... :) 00:59:23 yes 00:59:36 alex_xu: I will have another look at it :) 00:59:48 cyeoh, thank you! 01:00:13 ok we have 1 minute left. Anything else? 01:00:48 ok, thanks everyone for coming. cya! 01:00:48 good for me, thanks:-) 01:00:56 Thanks all 01:01:00 #endmeeting