00:01:19 <cyeoh> #startmeeting Nova API
00:01:20 <openstack> Meeting started Fri Apr  4 00:01:19 2014 UTC and is due to finish in 60 minutes.  The chair is cyeoh. Information about MeetBot at http://wiki.debian.org/MeetBot.
00:01:21 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
00:01:23 <openstack> The meeting name has been set to 'nova_api'
00:01:31 <cyeoh> Hi - who do we have here today?
00:01:34 <jaypipes> o/
00:01:36 <ken1ohmichi> hi
00:01:49 <masayukig> hi
00:02:15 <GMann_> Hi
00:02:22 <ivanzhu> hi
00:02:42 <cyeoh> ok let's get started
00:02:47 <cyeoh> #topic vNext
00:03:02 <cyeoh> jaypipes: would you like to talk about what you have here?
00:03:10 <jaypipes> cyeoh: ok, sure, thx.
00:03:19 <jaypipes> #link http://docs.oscomputevnext.apiary.io/
00:03:27 <jaypipes> #link https://github.com/jaypipes/openstack-compute-api
00:03:51 <jaypipes> so, I've been brainstorming about ideas for the compute API **if backwards compatibility was no issue whatsoever**
00:04:09 <jaypipes> thus, calling it vNext and not getting into any discussion about what number version it is :)
00:04:52 <cyeoh> yea so I think when we bump the major version we should feel free to break backwards compatibility
00:04:54 <jaypipes> I am still working on the README and apiblueprint documentation, but some of the major concepts that I'm going for are documented at the top of the docs.apiary link above
00:05:39 <jaypipes> one of the more controversial concepts I presume will be the complete lack of API extensions.
00:05:55 <jaypipes> as well as the removal of anything in the Nova API that is for deployers or operators.
00:06:17 <cyeoh> jaypipes: yea I suspect that's something that would need a lot of discussion with deployers/operators
00:06:25 <jaypipes> some of the less controversial stuff is probably the lack of XML and the use of JSON-Home and JSONSchema docs for discovery of the API
00:06:36 <cyeoh> especially the lack of API extensions because I think they're the ones that want that flexibility
00:07:04 <jaypipes> cyeoh: so my initial thought on the operator/deployer API stuff is that operators could continue to use the v2/3 API for operator actions if that were desired
00:07:27 <cyeoh> but long term v2/v3 would be dropped wouldn't it?
00:07:32 <cyeoh> (like super long term)
00:07:33 <jaypipes> cyeoh: yup
00:07:45 <cyeoh> so they are going to want some sort of migration path.
00:08:08 <jaypipes> cyeoh: which would give us time to decide whether a separate RESTful API for operators is appropriate (or a non-RESTful API or tooling set, ect)
00:08:36 <jaypipes> cyeoh: of course. that's not the intention of the vNext API discussions, though. :)
00:08:43 <cyeoh> ok, that's reasonable
00:09:00 <cyeoh> I think removing XML is no brainer - its already removed from the V3 API and on its way out from V2
00:09:07 <cyeoh> V2.1 would not support XML either
00:09:26 <jaypipes> sure, I figured there's be little disagreement there.
00:09:31 <cyeoh> The JSON-Home/JSON schema docs is how I'd like to head for the V3 API
00:09:39 <cyeoh> I think we almost enough in place now
00:09:45 <jaypipes> cyeoh: are you familiar with APIBlueprint?
00:09:57 <cyeoh> jaypipes: no, have you got a link?
00:10:06 <jaypipes> cyeoh: that first link above is it :)
00:10:25 <jaypipes> cyeoh: it is the formatted representation of this: https://github.com/jaypipes/openstack-compute-api/blob/master/apiary.apib
00:11:01 <jaypipes> cyeoh: basically, it allows one to use Markdown to document the API contract (request and reponse payloads, API routes, etc)
00:11:14 <jaypipes> cyeoh: but in a machine readable format that is also human-readable.
00:11:57 <cyeoh> hrm interesting. Are you suggesting its something we autogenerate instead of using WADL etc?
00:12:07 <jaypipes> cyeoh: so all of the JSON-Home and JSONSchema documents are actually stored in the APIBlueprint Markdown document, which is machine parseable, and thus you see that the request and response payload and object model are expressable in a single document
00:12:31 <jaypipes> cyeoh: writing the API spec in APIBlueprint allows you to generate the API spec, yes.
00:12:48 <jaypipes> cyeoh: including generating all of the JSONSChema docs that are embedded in the apiblueprint markdown document.
00:13:34 <cyeoh> jaypipes: ok, its definitely something we should look at closer then - docs are one of the worst parts of the current API (and its not really the docs team fault)
00:13:36 <jaypipes> cyeoh: as a neat side-effect, you can actually hit the oscomputevnext.apiary.com endpoint and test HTTP requests against the APIBlueprint contract.
00:14:36 <cyeoh> that sounds great
00:14:50 <jaypipes> cyeoh: so in a way, the single APIBlueprint is self-validating, because Markdown is either valid or not, so you can validate the API spec as well as the content of the API specification itself... kinda cool.
00:15:44 <jaypipes> anyway... thx for letting me introduce the vNext stuff. It's not done yet, and I very much welcome forks and pull requests on there. feel free to critique, add, delete, etc
00:16:00 <cyeoh> so your desire to pull out operator or "internal" type functionality from the REST API - does that come from wanting to make the API simpler for ordinary users to understand?
00:16:02 <jaypipes> it's a big playground for ideas on compute API stuff
00:16:16 <jaypipes> cyeoh: yes, that is exactly the point.
00:16:25 <cyeoh> ok.
00:16:34 <jaypipes> cyeoh: there's a reason we don't expose database schema migrations in the REST API... it just doesn't make sense
00:16:45 <cyeoh> I really like the idea of having really long term planning for the API
00:16:57 <jaypipes> cyeoh: likewise, although I'm sure amazon has internal operator APIs, it's not exposed in the same AWS API everyone else uses...
00:17:38 <cyeoh> Just looking at the philosophy/style section I think there's a few things we should integrate into V3 - some i'd like to do like having UUIDs for everything but hard because of $INTERNALS_REASONS
00:18:04 <jaypipes> cyeoh: again, I'm not at all opposed to operation API actions... just not keen on them being in the same API as users of the cloud. I feel it is confusing and makes the API documentation and usage mddy
00:18:06 <jaypipes> muddy...
00:18:22 <cyeoh> yea, it may be as simple as separating them in the documentation?
00:18:46 <jaypipes> cyeoh: eh... I'd prefer an entirely separate API really.
00:19:17 <cyeoh> if normal users can't discover it does it make a difference?
00:19:28 <jaypipes> cyeoh: operators and admins and users just don't view the cloud in the same way, and having a single control API for different audiences doesn't make as much sense to me as having separate ones for the operator and the admins/users
00:20:15 <cyeoh> ok
00:20:34 <jaypipes> cyeoh: I think it would be simpler in the code to have a clean break, but hey, this is a community consensus thing... there's more than one side to this debate...
00:20:49 <cyeoh> depending on the timeframe for a stable V3 I'd like us to consider some of the ideas you have here http://docs.oscomputevnext.apiary.io/
00:20:53 <jaypipes> totally just want to put things on the table and play around with ideas without any boundaries
00:21:38 <cyeoh> yea I think we need to work out where we want to get to in the long term, and then consider how to get there separately with the extra constraints like maintenance overhead
00:22:07 <cyeoh> anyway thank you very much for joining today to talk about it. Are you planning on submitting a summit session about this?
00:22:45 <jaypipes> yep, absolutely.
00:22:57 <jaypipes> I'll even bring some beer to the session ;)
00:23:14 <cyeoh> excellent :-) was there anything else you want to talk about vNext or we'll move on to the next topic
00:23:25 <jaypipes> nope, good on my end, thx
00:23:57 <cyeoh> #topic progress on v2 on v3 API POC
00:24:12 <cyeoh> so I have submitted a nova-specs patch
00:24:19 <cyeoh> #link https://review.openstack.org/#/c/84695/
00:24:40 <cyeoh> this won't merge until we've had some discussions about it at summit
00:24:58 <cyeoh> but I'd encourage everyone to have a look at it to make sure we haven't missed anything
00:25:39 <cyeoh> ken1ohmichi: I was wondering if you had any more feedback about it - anything you think we should add?
00:26:14 <ken1ohmichi> cyeoh: Nova has a lot of APIs. it is better to consider how to manage the progress.
00:26:40 <cyeoh> ken1ohmichi: right, and how much we want to do for the POC?
00:27:04 <cyeoh> ken1ohmichi: would it be useful to have a spreadsheet like we have for the tempest tests?
00:27:07 <ken1ohmichi> cyeoh: now 6 APIs are PoC targets.
00:27:29 <ken1ohmichi> cyeoh: I think it is a good idea.
00:28:08 <ken1ohmichi> cyeoh: now we are managing Tempest tasks on spreadsheet also. https://docs.google.com/spreadsheet/ccc?key=0AmYuZ6T4IJETdEVNTWlYVUVOWURmOERSZ0VGc1BBQWc#gid=3
00:28:49 <ken1ohmichi> cyeoh: I feel it is beter thatv2.1 tasks also will be managed like that:]
00:29:18 <cyeoh> ken1ohmichi: I think that would be really good - is that something you have time to do?
00:29:56 <ken1ohmichi> cyeoh: sorry, do you mean?
00:30:25 <cyeoh> ken1ohmichi: I'm wondering if you could setup the spreadsheet for the v2.1 tasks - and we'll track things that way
00:30:42 <ken1ohmichi> cyeoh: oh, I will do it:-)
00:30:50 <cyeoh> ken1ohmichi: excellent, thx!
00:31:01 <cyeoh> #action ken1ohmichi to set up v2.1 tasks tracking spreadsheet
00:31:20 <cyeoh> I guess if we implement any more v2.1 APIs we should concentrate on the core ones
00:31:39 <cyeoh> (things defined as core in the V3 API as they are most likely the most important)
00:31:52 <ken1ohmichi> cyeoh: agree
00:32:12 <cyeoh> so one other thing came up in the discussions around V2.1
00:32:29 <cyeoh> and that was if for v2.1 it would be possible to put the input validation into a mode where it didn't reject requests
00:32:38 <cyeoh> but instead just logged invalid requests
00:32:47 <cyeoh> this would presumably be done using a config flag
00:33:11 <ken1ohmichi> cyeoh: we need https://review.openstack.org/#/c/78111/ for PoC.
00:33:14 <cyeoh> the reasoning behind this was that some cloud providers would want to do this, so they'd roll out v2.1 to a few clients
00:33:46 <cyeoh> ken1ohmichi: ok I'll review that one today
00:33:59 <ken1ohmichi> cyeoh: thanks!
00:34:04 <cyeoh> and the clients would run their apps as normal
00:34:18 <cyeoh> if they were misusing the API because of the relaxed input validation
00:34:35 <cyeoh> it would still work with v2.1, but the cloud providers would see the logs and be able to tell them
00:34:49 <cyeoh> also they'd get a better handle on how many people are misusing the API
00:35:18 <cyeoh> I think the main downsides would be they'd likely get an internal server error when they did misuse the API (but perhaps thats ok)
00:35:26 <cyeoh> and we'd need to think carefully about any potential security issues
00:35:51 <cyeoh> ken1ohmichi: I think it'd be possible though for the validator to have a mode where it just didn't raise an exception, but logged instead when there was a problem?
00:35:53 <ken1ohmichi> I guess many people misuse APIs because Nova unit tests also did it.
00:36:08 <cyeoh> ken1ohmichi: yes I'm guessing its actually quite common!
00:36:47 <ken1ohmichi> cyeoh: that is an interesting idea. I feel we can do it.
00:36:58 <ken1ohmichi> with some option.
00:37:21 <cyeoh> ken1ohmichi: ok cool, just wanted to check with you we weren't promising the impossible :-)
00:37:44 <ken1ohmichi> cyeoh: now checking:-)
00:38:08 <cyeoh> so I talked to russellb about the -2's on the v2-on-v3-api patches, and he has very kindly removed the -2's
00:38:15 <GMann_> cyeoh: in that case will Nova auto recover the wrong input?
00:38:39 <cyeoh> GMann_: so it's likely it will crash and burn in some way and raise some random exception
00:38:50 <cyeoh> in which case its likely that it will return a 500 Internal Server Error
00:39:09 <GMann_> ok
00:39:12 <cyeoh> but as long as cloud operators know that's going to happen I think its ok.
00:39:28 <GMann_> ok, Got it. Thx
00:39:38 <cyeoh> its really just a special mode so everyone can get a better handle on how much pain a v2 to v2.1 upgrade will cause
00:40:07 <cyeoh> re: the v2-on-v3-api patches, please ensure that the WIP flag is put back on if you do rebasing
00:40:31 <cyeoh> so the patches don't end up skewing the stats on the nova queue
00:40:54 <cyeoh> and so they don't get accidentally get merged. I think its ok to put a big "DO NOT MERGE" in the commit message for now too
00:41:45 <ken1ohmichi> cyeoh: but I cannot mark the other developers patches.
00:42:04 <ken1ohmichi> cyeoh: for WIP.
00:42:11 <cyeoh> hrm
00:42:17 <cyeoh> I thought we could mark other people's patches WIP?
00:42:44 <ken1ohmichi> cyeoh: is not enough to include "WIP" as the title?
00:42:46 <cyeoh> or maybe you need to be nova-core to do it, I appear to have the perms to do it
00:43:01 <cyeoh> ken1ohmichi: that will help, but unfortunately the stats scripts don't look for that.
00:43:27 <cyeoh> if you can't mark other people's patches WIP, just make sure you do your own, and I'll go through the patch series every now and then fix up any that are missing
00:44:13 <cyeoh> For those who don't know, I did a big rebase this week of all the v2-on-v3-api patches so we have one single series of patches
00:44:22 <cyeoh> the end one is this one: https://review.openstack.org/#/c/83256/
00:44:26 <ken1ohmichi> cyeoh: thanks, will ping you when needing it.
00:44:51 <cyeoh> which is ken1ohmichi's patch which makes v2.1 appear as /v2 so if you pull that one you should get them all
00:45:11 <cyeoh> if you do add any more POC patches, please make sure you insert it somewhere appropriate in the patch series and rebase any dependencies
00:45:34 <cyeoh> there is an ordered list at the end of this etherpad: https://etherpad.openstack.org/p/NovaV2OnV3POC
00:45:35 <ken1ohmichi> cyeoh: sure, will take care of it:)
00:45:47 <cyeoh> to help us make sure we keep things in the right order
00:45:49 <cyeoh> ken1ohmichi: thanks!
00:46:21 <ken1ohmichi> cyeoh: re: option for log instead of exception: enough to do it, just set some option at https://github.com/openstack/nova/blob/master/nova/api/validation/validators.py#L69
00:46:56 <cyeoh> ken1ohmichi: excellent, so that'd just be a single point where we can LOG instead of raising an exception...
00:47:02 <ken1ohmichi> cyeoh: will add it to *v3* tasks.
00:47:13 <cyeoh> which brings up my other question about v2 on v3 -
00:47:28 <ken1ohmichi> nothing from me:)
00:47:38 <cyeoh> so when an invalid V2 request comes in
00:47:50 <cyeoh> it will get translated and then hit the v3 validator
00:47:59 <cyeoh> which will produce a v3 specific error message
00:48:15 <ken1ohmichi> cyeoh: right.
00:48:20 <cyeoh> is it going to be possible to translate the error message on the way back so say instead of saying invalid imageID
00:48:24 <cyeoh> it can say invalid image_id?
00:48:30 <cyeoh> oops other way around ;-)
00:48:51 <cyeoh> I suspect that's going to be rather tricky, but maybe possible?
00:49:09 <ken1ohmichi> cyeoh: I am not sure now..
00:49:46 <cyeoh> we might have to delay the generation of the string message till later, and perhaps raise an object with the raw info
00:49:55 <cyeoh> anyway I will play around with a few ideas
00:50:10 <cyeoh> but I thought I should flag it as a potential issues
00:50:23 <ken1ohmichi> cyeoh: and I feel it is not matter if outputing image_id instead of imageId as v2 BadRequest because v2.0 msg is not consistent.
00:50:41 <cyeoh> ken1ohmichi: heh, that is sort of true :-)
00:50:52 <cyeoh> I guess I was thinking if it was possible we should try to get it right.
00:51:44 <cyeoh> ok was anything else about the v2 on v3 POC that people wanted to talk about?
00:52:00 <ken1ohmichi> cyeoh: if doing it, we need to pass v2 info to validation code. that would make a little durty code.
00:52:27 <cyeoh> ken1ohmichi: yea I"m not sure how we'd do it cleanly yet.
00:52:41 <cyeoh> though we have the name mappings in the decorators already
00:52:51 <ken1ohmichi> cyeoh: yes, right.
00:53:10 <cyeoh> anyway I will have a play and see if its possible, otherwise it may have to be something we have to live with
00:53:26 <ken1ohmichi> cyeoh: thanks:)
00:53:48 <cyeoh> #topic progress on API response validation in tempest
00:54:01 <cyeoh> is there anything we need to talk about here? Other than needing to do more reviews :-)
00:54:17 <ken1ohmichi> cyeoh; yes need more reviews:)
00:54:22 <cyeoh> heh :-)
00:54:30 <cyeoh> I've been slack this week. Will try to get more done soon
00:54:46 <ken1ohmichi> cyeoh: great
00:54:55 <cyeoh> mrda: are you around atm?
00:55:22 <cyeoh> ok we'll go on to the v3 API work then
00:55:29 <cyeoh> #topic Ongoing V3 API related work
00:55:51 <cyeoh> so I think we have consensus in #openstack-nova that we can continue to merge V3 related work that we missed in Icehouse
00:56:10 <ken1ohmichi> great!
00:56:17 <cyeoh> so things like missing extensions, moving policy checks up to the API layer etc
00:56:29 <cyeoh> but we do need to still put blueprints specs in
00:56:51 <ivanzhu> cyeoh: so we can restore these patches
00:56:53 <cyeoh> so I'll try to work on those over the next few days, but everyone should feel free to work on them too if they'd like to :-)
00:57:11 <cyeoh> ivanzhu: yep, restore them, but we'll need to get the blueprint re-approved.
00:57:26 <ivanzhu> cyeoh: great
00:57:46 <ken1ohmichi> OK, validation taks also remain. will restore them.
00:58:05 <cyeoh> ivanzhu: the policy check ones scare me so much though I'm tempted to suggest we need 3 +2 votes on them though
00:58:22 <cyeoh> ivanzhu: purely from the point of view that if we get it wrong, we introduce a security bug
00:58:33 <cyeoh> ken1ohmichi: that's be great, thx!
00:58:46 <cyeoh> ken1ohmichi: the validation patches really clean up the V3 API code
00:59:04 <cyeoh> #topic Open Discussion
00:59:10 <cyeoh> ok we have like 1 minute left :-)
00:59:18 <cyeoh> is there anything else?
00:59:30 <tianst20> cyeoh: before we merge the policy check , i think we should first  change the policy  https://review.openstack.org/#/c/76829/
01:00:26 <cyeoh> tianst20: ok I will have to look at that
01:00:43 <tianst20> thanks ,
01:00:54 <cyeoh> tianst20: you will have to chase up jog0 to remove the -2
01:01:21 <cyeoh> ok we're out of time, thank you everyone for attending.
01:01:26 <cyeoh> #endmeeting