00:01:19 #startmeeting Nova API 00:01:20 Meeting started Fri Apr 4 00:01:19 2014 UTC and is due to finish in 60 minutes. The chair is cyeoh. Information about MeetBot at http://wiki.debian.org/MeetBot. 00:01:21 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 00:01:23 The meeting name has been set to 'nova_api' 00:01:31 Hi - who do we have here today? 00:01:34 o/ 00:01:36 hi 00:01:49 hi 00:02:15 Hi 00:02:22 hi 00:02:42 ok let's get started 00:02:47 #topic vNext 00:03:02 jaypipes: would you like to talk about what you have here? 00:03:10 cyeoh: ok, sure, thx. 00:03:19 #link http://docs.oscomputevnext.apiary.io/ 00:03:27 #link https://github.com/jaypipes/openstack-compute-api 00:03:51 so, I've been brainstorming about ideas for the compute API **if backwards compatibility was no issue whatsoever** 00:04:09 thus, calling it vNext and not getting into any discussion about what number version it is :) 00:04:52 yea so I think when we bump the major version we should feel free to break backwards compatibility 00:04:54 I am still working on the README and apiblueprint documentation, but some of the major concepts that I'm going for are documented at the top of the docs.apiary link above 00:05:39 one of the more controversial concepts I presume will be the complete lack of API extensions. 00:05:55 as well as the removal of anything in the Nova API that is for deployers or operators. 00:06:17 jaypipes: yea I suspect that's something that would need a lot of discussion with deployers/operators 00:06:25 some of the less controversial stuff is probably the lack of XML and the use of JSON-Home and JSONSchema docs for discovery of the API 00:06:36 especially the lack of API extensions because I think they're the ones that want that flexibility 00:07:04 cyeoh: so my initial thought on the operator/deployer API stuff is that operators could continue to use the v2/3 API for operator actions if that were desired 00:07:27 but long term v2/v3 would be dropped wouldn't it? 00:07:32 (like super long term) 00:07:33 cyeoh: yup 00:07:45 so they are going to want some sort of migration path. 00:08:08 cyeoh: which would give us time to decide whether a separate RESTful API for operators is appropriate (or a non-RESTful API or tooling set, ect) 00:08:36 cyeoh: of course. that's not the intention of the vNext API discussions, though. :) 00:08:43 ok, that's reasonable 00:09:00 I think removing XML is no brainer - its already removed from the V3 API and on its way out from V2 00:09:07 V2.1 would not support XML either 00:09:26 sure, I figured there's be little disagreement there. 00:09:31 The JSON-Home/JSON schema docs is how I'd like to head for the V3 API 00:09:39 I think we almost enough in place now 00:09:45 cyeoh: are you familiar with APIBlueprint? 00:09:57 jaypipes: no, have you got a link? 00:10:06 cyeoh: that first link above is it :) 00:10:25 cyeoh: it is the formatted representation of this: https://github.com/jaypipes/openstack-compute-api/blob/master/apiary.apib 00:11:01 cyeoh: basically, it allows one to use Markdown to document the API contract (request and reponse payloads, API routes, etc) 00:11:14 cyeoh: but in a machine readable format that is also human-readable. 00:11:57 hrm interesting. Are you suggesting its something we autogenerate instead of using WADL etc? 00:12:07 cyeoh: so all of the JSON-Home and JSONSchema documents are actually stored in the APIBlueprint Markdown document, which is machine parseable, and thus you see that the request and response payload and object model are expressable in a single document 00:12:31 cyeoh: writing the API spec in APIBlueprint allows you to generate the API spec, yes. 00:12:48 cyeoh: including generating all of the JSONSChema docs that are embedded in the apiblueprint markdown document. 00:13:34 jaypipes: ok, its definitely something we should look at closer then - docs are one of the worst parts of the current API (and its not really the docs team fault) 00:13:36 cyeoh: as a neat side-effect, you can actually hit the oscomputevnext.apiary.com endpoint and test HTTP requests against the APIBlueprint contract. 00:14:36 that sounds great 00:14:50 cyeoh: so in a way, the single APIBlueprint is self-validating, because Markdown is either valid or not, so you can validate the API spec as well as the content of the API specification itself... kinda cool. 00:15:44 anyway... thx for letting me introduce the vNext stuff. It's not done yet, and I very much welcome forks and pull requests on there. feel free to critique, add, delete, etc 00:16:00 so your desire to pull out operator or "internal" type functionality from the REST API - does that come from wanting to make the API simpler for ordinary users to understand? 00:16:02 it's a big playground for ideas on compute API stuff 00:16:16 cyeoh: yes, that is exactly the point. 00:16:25 ok. 00:16:34 cyeoh: there's a reason we don't expose database schema migrations in the REST API... it just doesn't make sense 00:16:45 I really like the idea of having really long term planning for the API 00:16:57 cyeoh: likewise, although I'm sure amazon has internal operator APIs, it's not exposed in the same AWS API everyone else uses... 00:17:38 Just looking at the philosophy/style section I think there's a few things we should integrate into V3 - some i'd like to do like having UUIDs for everything but hard because of $INTERNALS_REASONS 00:18:04 cyeoh: again, I'm not at all opposed to operation API actions... just not keen on them being in the same API as users of the cloud. I feel it is confusing and makes the API documentation and usage mddy 00:18:06 muddy... 00:18:22 yea, it may be as simple as separating them in the documentation? 00:18:46 cyeoh: eh... I'd prefer an entirely separate API really. 00:19:17 if normal users can't discover it does it make a difference? 00:19:28 cyeoh: operators and admins and users just don't view the cloud in the same way, and having a single control API for different audiences doesn't make as much sense to me as having separate ones for the operator and the admins/users 00:20:15 ok 00:20:34 cyeoh: I think it would be simpler in the code to have a clean break, but hey, this is a community consensus thing... there's more than one side to this debate... 00:20:49 depending on the timeframe for a stable V3 I'd like us to consider some of the ideas you have here http://docs.oscomputevnext.apiary.io/ 00:20:53 totally just want to put things on the table and play around with ideas without any boundaries 00:21:38 yea I think we need to work out where we want to get to in the long term, and then consider how to get there separately with the extra constraints like maintenance overhead 00:22:07 anyway thank you very much for joining today to talk about it. Are you planning on submitting a summit session about this? 00:22:45 yep, absolutely. 00:22:57 I'll even bring some beer to the session ;) 00:23:14 excellent :-) was there anything else you want to talk about vNext or we'll move on to the next topic 00:23:25 nope, good on my end, thx 00:23:57 #topic progress on v2 on v3 API POC 00:24:12 so I have submitted a nova-specs patch 00:24:19 #link https://review.openstack.org/#/c/84695/ 00:24:40 this won't merge until we've had some discussions about it at summit 00:24:58 but I'd encourage everyone to have a look at it to make sure we haven't missed anything 00:25:39 ken1ohmichi: I was wondering if you had any more feedback about it - anything you think we should add? 00:26:14 cyeoh: Nova has a lot of APIs. it is better to consider how to manage the progress. 00:26:40 ken1ohmichi: right, and how much we want to do for the POC? 00:27:04 ken1ohmichi: would it be useful to have a spreadsheet like we have for the tempest tests? 00:27:07 cyeoh: now 6 APIs are PoC targets. 00:27:29 cyeoh: I think it is a good idea. 00:28:08 cyeoh: now we are managing Tempest tasks on spreadsheet also. https://docs.google.com/spreadsheet/ccc?key=0AmYuZ6T4IJETdEVNTWlYVUVOWURmOERSZ0VGc1BBQWc#gid=3 00:28:49 cyeoh: I feel it is beter thatv2.1 tasks also will be managed like that:] 00:29:18 ken1ohmichi: I think that would be really good - is that something you have time to do? 00:29:56 cyeoh: sorry, do you mean? 00:30:25 ken1ohmichi: I'm wondering if you could setup the spreadsheet for the v2.1 tasks - and we'll track things that way 00:30:42 cyeoh: oh, I will do it:-) 00:30:50 ken1ohmichi: excellent, thx! 00:31:01 #action ken1ohmichi to set up v2.1 tasks tracking spreadsheet 00:31:20 I guess if we implement any more v2.1 APIs we should concentrate on the core ones 00:31:39 (things defined as core in the V3 API as they are most likely the most important) 00:31:52 cyeoh: agree 00:32:12 so one other thing came up in the discussions around V2.1 00:32:29 and that was if for v2.1 it would be possible to put the input validation into a mode where it didn't reject requests 00:32:38 but instead just logged invalid requests 00:32:47 this would presumably be done using a config flag 00:33:11 cyeoh: we need https://review.openstack.org/#/c/78111/ for PoC. 00:33:14 the reasoning behind this was that some cloud providers would want to do this, so they'd roll out v2.1 to a few clients 00:33:46 ken1ohmichi: ok I'll review that one today 00:33:59 cyeoh: thanks! 00:34:04 and the clients would run their apps as normal 00:34:18 if they were misusing the API because of the relaxed input validation 00:34:35 it would still work with v2.1, but the cloud providers would see the logs and be able to tell them 00:34:49 also they'd get a better handle on how many people are misusing the API 00:35:18 I think the main downsides would be they'd likely get an internal server error when they did misuse the API (but perhaps thats ok) 00:35:26 and we'd need to think carefully about any potential security issues 00:35:51 ken1ohmichi: I think it'd be possible though for the validator to have a mode where it just didn't raise an exception, but logged instead when there was a problem? 00:35:53 I guess many people misuse APIs because Nova unit tests also did it. 00:36:08 ken1ohmichi: yes I'm guessing its actually quite common! 00:36:47 cyeoh: that is an interesting idea. I feel we can do it. 00:36:58 with some option. 00:37:21 ken1ohmichi: ok cool, just wanted to check with you we weren't promising the impossible :-) 00:37:44 cyeoh: now checking:-) 00:38:08 so I talked to russellb about the -2's on the v2-on-v3-api patches, and he has very kindly removed the -2's 00:38:15 cyeoh: in that case will Nova auto recover the wrong input? 00:38:39 GMann_: so it's likely it will crash and burn in some way and raise some random exception 00:38:50 in which case its likely that it will return a 500 Internal Server Error 00:39:09 ok 00:39:12 but as long as cloud operators know that's going to happen I think its ok. 00:39:28 ok, Got it. Thx 00:39:38 its really just a special mode so everyone can get a better handle on how much pain a v2 to v2.1 upgrade will cause 00:40:07 re: the v2-on-v3-api patches, please ensure that the WIP flag is put back on if you do rebasing 00:40:31 so the patches don't end up skewing the stats on the nova queue 00:40:54 and so they don't get accidentally get merged. I think its ok to put a big "DO NOT MERGE" in the commit message for now too 00:41:45 cyeoh: but I cannot mark the other developers patches. 00:42:04 cyeoh: for WIP. 00:42:11 hrm 00:42:17 I thought we could mark other people's patches WIP? 00:42:44 cyeoh: is not enough to include "WIP" as the title? 00:42:46 or maybe you need to be nova-core to do it, I appear to have the perms to do it 00:43:01 ken1ohmichi: that will help, but unfortunately the stats scripts don't look for that. 00:43:27 if you can't mark other people's patches WIP, just make sure you do your own, and I'll go through the patch series every now and then fix up any that are missing 00:44:13 For those who don't know, I did a big rebase this week of all the v2-on-v3-api patches so we have one single series of patches 00:44:22 the end one is this one: https://review.openstack.org/#/c/83256/ 00:44:26 cyeoh: thanks, will ping you when needing it. 00:44:51 which is ken1ohmichi's patch which makes v2.1 appear as /v2 so if you pull that one you should get them all 00:45:11 if you do add any more POC patches, please make sure you insert it somewhere appropriate in the patch series and rebase any dependencies 00:45:34 there is an ordered list at the end of this etherpad: https://etherpad.openstack.org/p/NovaV2OnV3POC 00:45:35 cyeoh: sure, will take care of it:) 00:45:47 to help us make sure we keep things in the right order 00:45:49 ken1ohmichi: thanks! 00:46:21 cyeoh: re: option for log instead of exception: enough to do it, just set some option at https://github.com/openstack/nova/blob/master/nova/api/validation/validators.py#L69 00:46:56 ken1ohmichi: excellent, so that'd just be a single point where we can LOG instead of raising an exception... 00:47:02 cyeoh: will add it to *v3* tasks. 00:47:13 which brings up my other question about v2 on v3 - 00:47:28 nothing from me:) 00:47:38 so when an invalid V2 request comes in 00:47:50 it will get translated and then hit the v3 validator 00:47:59 which will produce a v3 specific error message 00:48:15 cyeoh: right. 00:48:20 is it going to be possible to translate the error message on the way back so say instead of saying invalid imageID 00:48:24 it can say invalid image_id? 00:48:30 oops other way around ;-) 00:48:51 I suspect that's going to be rather tricky, but maybe possible? 00:49:09 cyeoh: I am not sure now.. 00:49:46 we might have to delay the generation of the string message till later, and perhaps raise an object with the raw info 00:49:55 anyway I will play around with a few ideas 00:50:10 but I thought I should flag it as a potential issues 00:50:23 cyeoh: and I feel it is not matter if outputing image_id instead of imageId as v2 BadRequest because v2.0 msg is not consistent. 00:50:41 ken1ohmichi: heh, that is sort of true :-) 00:50:52 I guess I was thinking if it was possible we should try to get it right. 00:51:44 ok was anything else about the v2 on v3 POC that people wanted to talk about? 00:52:00 cyeoh: if doing it, we need to pass v2 info to validation code. that would make a little durty code. 00:52:27 ken1ohmichi: yea I"m not sure how we'd do it cleanly yet. 00:52:41 though we have the name mappings in the decorators already 00:52:51 cyeoh: yes, right. 00:53:10 anyway I will have a play and see if its possible, otherwise it may have to be something we have to live with 00:53:26 cyeoh: thanks:) 00:53:48 #topic progress on API response validation in tempest 00:54:01 is there anything we need to talk about here? Other than needing to do more reviews :-) 00:54:17 cyeoh; yes need more reviews:) 00:54:22 heh :-) 00:54:30 I've been slack this week. Will try to get more done soon 00:54:46 cyeoh: great 00:54:55 mrda: are you around atm? 00:55:22 ok we'll go on to the v3 API work then 00:55:29 #topic Ongoing V3 API related work 00:55:51 so I think we have consensus in #openstack-nova that we can continue to merge V3 related work that we missed in Icehouse 00:56:10 great! 00:56:17 so things like missing extensions, moving policy checks up to the API layer etc 00:56:29 but we do need to still put blueprints specs in 00:56:51 cyeoh: so we can restore these patches 00:56:53 so I'll try to work on those over the next few days, but everyone should feel free to work on them too if they'd like to :-) 00:57:11 ivanzhu: yep, restore them, but we'll need to get the blueprint re-approved. 00:57:26 cyeoh: great 00:57:46 OK, validation taks also remain. will restore them. 00:58:05 ivanzhu: the policy check ones scare me so much though I'm tempted to suggest we need 3 +2 votes on them though 00:58:22 ivanzhu: purely from the point of view that if we get it wrong, we introduce a security bug 00:58:33 ken1ohmichi: that's be great, thx! 00:58:46 ken1ohmichi: the validation patches really clean up the V3 API code 00:59:04 #topic Open Discussion 00:59:10 ok we have like 1 minute left :-) 00:59:18 is there anything else? 00:59:30 cyeoh: before we merge the policy check , i think we should first change the policy https://review.openstack.org/#/c/76829/ 01:00:26 tianst20: ok I will have to look at that 01:00:43 thanks , 01:00:54 tianst20: you will have to chase up jog0 to remove the -2 01:01:21 ok we're out of time, thank you everyone for attending. 01:01:26 #endmeeting