16:00:15 <elmiko> #startmeeting api wg
16:00:15 <openstack> Meeting started Thu Jun  4 16:00:15 2015 UTC and is due to finish in 60 minutes.  The chair is elmiko. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:00:16 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
16:00:18 <openstack> The meeting name has been set to 'api_wg'
16:00:19 <elmiko> there we go
16:00:23 <cdent> slow bot
16:00:35 <elmiko> i think i had spaces in front that messed it up
16:00:41 <ryansb> probably
16:00:41 <elmiko> #topic agenda
16:00:48 <elmiko> #link https://wiki.openstack.org/wiki/Meetings/API-WG#Agenda
16:01:34 <elmiko> #topic previous meeting action items
16:01:51 <elmiko> etoews is out, so we'll skip his action
16:02:02 <miguelgrinberg> just finished my review of  https://review.openstack.org/#/c/177397/ a few minutes ago
16:02:08 <elmiko> awesome
16:02:12 <miguelgrinberg> lots of things I did not like
16:02:49 <elmiko> anything specific you'd like to bring up?
16:02:58 <sigmavirus24> o/
16:03:18 <miguelgrinberg> they have optional components in the URL, in the middle
16:03:49 <ryansb> o.O
16:04:02 <miguelgrinberg> something like /artifacts/{type_version}/something_else
16:04:09 <miguelgrinberg> and type_version can be omitted
16:04:29 <cdent> that's confusing
16:04:33 <elmiko> agreed
16:04:43 * sigmavirus24 sees if any of them are around
16:04:44 <miguelgrinberg> I suggested a different structure, let's see how it goes
16:04:47 <cdent> and not very URI
16:04:53 <sdague> o/
16:05:12 <sigmavirus24> o/ sdague
16:05:13 <cdent> what is a "type version"
16:05:54 <miguelgrinberg> it defines the version of the artifact structure
16:05:59 <miguelgrinberg> or so I understand
16:06:01 <sigmavirus24> cdent: so artifacts are a very generic abstraction of "things"
16:06:12 <sigmavirus24> a type is a specialization of sorts, e.g., an image
16:06:15 <sigmavirus24> or a heat template
16:06:37 <sdague> so... this version thing seems like it would be better done in a psuedo content negotiation model
16:06:40 <sigmavirus24> I think the type_version is the artifact's version (since all artifacts are semantically versioned)
16:06:53 <stevelle> ^ that
16:06:54 <cdent> sdague++
16:07:27 <miguelgrinberg> I proposed to move the optional stuff to the end, but content negotiation would work too
16:07:31 <cdent> (it's getting a bit boring agreeing with you so much sdague, can you say some stupid stuff every now and again just to keep me on my toes?)
16:07:50 <sdague> artifact_type seems to be completely mime types, so I don't understand why that's in the uri at all
16:07:57 <sigmavirus24> cdent: having never met you, I'm not convinced you and sdague aren't the same person
16:08:03 <sdague> heh
16:08:08 <miguelgrinberg> sdague: they define it as a semver
16:08:15 <elmiko> sigmavirus24: lol
16:08:27 <sdague> miguelgrinberg: no, that's type_version
16:08:29 <sigmavirus24> Has anyone ever seen cdent and sdague in the same room together? I doubt it ;)
16:08:38 <sdague> The `artifact_type` constant should unambiguously identify the
16:08:38 <sdague> artifact type, so the values of this constants should be unique among all the
16:08:38 <sdague> artifact types defined by the active plugins.
16:08:39 <miguelgrinberg> sdague: ah, sorry, right
16:08:41 <cdent> I think I saw sdague at summit, but I may have been imagining it, talking to myself
16:08:48 <sigmavirus24> Oh I could be wrong
16:08:56 <sigmavirus24> They might allow versioning of artifact plugin types
16:09:25 <sdague> right, that seems to be what it is. But that really seems like it should just be mime types
16:09:29 <sigmavirus24> jaypipes: might know more about artifacts since he's their bossman =P
16:09:36 <sdague> with Accept headers
16:09:36 <sigmavirus24> sdague: I don't disagree :D
16:09:37 <stevelle> they allow versioning the types as well as the artifacts, yes
16:09:48 <sigmavirus24> stevelle: right, I was confused for a few minutes there
16:10:06 * edleafe arrives late and sits in the back row
16:10:42 <elmiko> sounds like miguelgrinberg is on top of this, do we need another action item or does this seem on track?
16:10:58 <elmiko> or would other folks want to checkout #link https://review.openstack.org/#/c/177397/ as well?
16:11:06 <cdent> is it worth getting more people on the review or will that just seem like piling on?
16:11:09 <peterstac> o/
16:11:16 <elmiko> cdent: yea, that's my concern too
16:11:29 <sigmavirus24> cdent: I think one or two other people to back up miguelgrinberg's concerns is fine
16:11:32 <sigmavirus24> I think 10 people is piling on
16:11:55 <miguelgrinberg> I think a couple more is not going to hurt
16:11:57 <sigmavirus24> ativelkov: has been working on artifacts
16:11:58 <elmiko> cdent, sdague, would either of you be able to take a look at that review as well?
16:12:11 <cdent> yeah, I've just put in my think about this queue
16:12:16 <ativelkov> hi folks
16:12:19 <elmiko> cool
16:12:22 <sdague> elmiko: yes, I can.
16:12:24 <elmiko> ativelkov: hi
16:12:31 <elmiko> awesome, thanks
16:12:31 <sigmavirus24> so if y'all have questions, ativelkov can answer them very well
16:12:52 <elmiko> #action cdent and sdague to review https://review.openstack.org/#/c/177397/ and comment
16:13:01 <elmiko> ok, next action item
16:13:19 <elmiko> stevelle, and ryansb, do you guys have anything to talk about for the filtering guideline?
16:13:51 <stevelle> Nothing except to call for more reviews on the updates
16:13:58 <elmiko> #link https://review.openstack.org/#/c/177468/
16:14:07 <ryansb> elmiko: I don't think so
16:14:16 <ryansb> yeah, what stevelle said
16:14:20 <elmiko> ok, cool. is that the correct link?
16:14:29 <ryansb> yup
16:14:52 <elmiko> #topic merge process status
16:15:02 <elmiko> #link https://review.openstack.org/#/c/186836/
16:15:15 <elmiko> so, it looks like we are getting good acceptance of the process that etoews put together
16:15:37 <elmiko> i wonder if we should post something to the ml about this, or can we merge maybe early next week?
16:16:39 <sigmavirus24> send something to the ML
16:16:41 <sigmavirus24> just to be safe
16:16:44 <sdague> posting to the ML never hurts
16:16:56 <sdague> and more communication the better on a lot of this, so there is less surprise to people
16:16:57 <sigmavirus24> well, sometimes it does ;)
16:17:03 <elmiko> ok, i won't freeze it though, since it's not really a guideline. does that make sense?
16:17:12 <sigmavirus24> or if people are surprised, then we can say "you should have read the ML"
16:17:14 <ryansb> sounds reasonable
16:17:15 <sigmavirus24> elmiko: yep
16:17:17 <elmiko> #action elmiko make post to ml about the merge process review
16:17:38 <elmiko> #topic guideline freeze
16:17:51 <elmiko> the review chain starting with #link https://review.openstack.org/#/c/179365/
16:18:02 <elmiko> seems to have good reviews and i think it's mostly ready for merge
16:18:25 <elmiko> should we freeze the main 3 and make a post, following the new process, or are these small enough to skip that?
16:19:33 <cdent> they are small, but they make statements about response codes that seem to stir people up
16:19:33 <elmiko> i'm kinda leaning towards freeze and post, but i wanted to get other's opinions
16:19:40 <cdent> yeah
16:19:45 <sdague> yeh, freeze and post seems safe
16:19:52 <elmiko> k, sounds good
16:20:04 <elmiko> #action elmiko freeze https://review.openstack.org/#/c/179365/ chain and post to ml
16:20:19 <elmiko> #topic guidelines
16:20:33 <elmiko> #link https://review.openstack.org/#/q/status:open+project:openstack/api-wg,n,z
16:20:42 <elmiko> are there any other guidelines we should be looking at?
16:21:46 <cdent> I think we should send the 501 stuff out for a global fight or it's going to linger and never actually be followed
16:22:12 <cdent> each time someone outside the group comes along they say "but wait, no!" and drop a -1 on it
16:22:20 <elmiko> this one https://review.openstack.org/#/c/183456/ ?
16:22:42 <cdent> yeah
16:22:58 <sigmavirus24> I agree with cdent
16:23:07 <elmiko> hmm, what's the next step here. sdague is there room to update this?
16:23:09 <sigmavirus24> I thought we were supposed to be telling people that they were doing it wrong though
16:23:15 * sigmavirus24 kids
16:23:17 <miguelgrinberg> what I think the problem is here is that we are not talking about specific examples
16:23:22 <cdent> I'm starting to think that maybe sigmavirus24 is me too
16:23:27 <elmiko> lol
16:23:37 <sigmavirus24> who's to say that I'm not?
16:23:41 <cdent> exactly!
16:23:55 <sdague> elmiko: so beyond merge conflicts, I think there were a couple of minor wording corrections, right?
16:24:16 <cdent> The reason I bring this up now is because it is one of the main areas where there are a lot of implementations are using 501 to mean "that functionality is not currently configured"
16:24:22 <elmiko> sdague: yea, but it's picked up a few -1s that look like content issues
16:24:23 <sdague> I can make those changes for sure, I don't know where the balance of the 501 fight is right now though, as I'm still getting my hread around it
16:24:30 <sdague> elmiko: yeh, those are fixable
16:24:36 <sdague> I can do that later today
16:24:39 <cdent> and making the guidline not to do that tests whether are guidelines are enforceable
16:24:42 <elmiko> sdague: awesome, thanks
16:24:53 <cdent> s/are/our/
16:24:59 <elmiko> #action sdague review and repost #link https://review.openstack.org/#/c/183456/
16:25:21 <miguelgrinberg> sdague: if you could list examples it would be awesome. I can follow your reasoning all the way until you say 400, then it does not make sense to me, so looking at specific examples might help
16:25:45 <sdague> miguelgrinberg: so, honestly, 400 is to me the default answer when nothing else fits
16:26:00 <jaypipes> sorry I'm late guys.
16:26:05 <sdague> because, you know, the http codes were designed around clients and servers before there were dynamic applications
16:26:06 <miguelgrinberg> sdague: so I want to see why 404 does not fit better
16:26:18 <jaypipes> sigmavirus24: I still have to do the review on the API of the artifacts, sorry :(
16:26:25 <jaypipes> and I'm not the boss man :)
16:26:51 <sigmavirus24> you're sort of a boss, man ;)
16:26:52 <sdague> miguelgrinberg: so, honestly, I don't have a hugely strong opinion on it, it's mostly the practical bit from the mailing list scuff between cdent and I
16:27:12 <miguelgrinberg> sdague: but you understand the push back on 400 correct?
16:27:12 <sdague> I think if anything beyond 400 has multiple meanings, it gets weird on client side coding
16:27:17 <sdague> miguelgrinberg: yes
16:27:36 <cdent> 404 should only be used when there's nothing responding on the URL
16:27:58 <sdague> miguelgrinberg: http://lists.openstack.org/pipermail/openstack-dev/2015-May/063396.html is the crux of my practical argument around 400 fall back
16:28:32 <miguelgrinberg> cdent: so I'd like to see an example where a service that is not configured still can have a valid URL
16:28:43 <sdague> miguelgrinberg: oh, that's easy
16:29:13 <sdague> POST volume-attach to /servers/ID/action
16:29:18 <sdague> for a docker driver based cloud
16:30:08 <sdague> a lot of these are things where the more RPCish stuff is being done for features that not all backend drivers support
16:30:33 <sdague> http://docs.openstack.org/developer/nova/support-matrix.html
16:30:51 <sdague> POST stop /severs/ID/action to an ironic based cloud
16:31:31 <miguelgrinberg> sdague: okay, so side effect of using actions instead of resources. Good example.
16:31:50 <sdague> yeh, I can include that in the updated text
16:31:55 <elmiko> would it help to have some of the example codified in the guideline?
16:31:58 <elmiko> k, thanks sdague
16:32:16 <cdent> when do we write the guideline that says /servers/ID/action is a bad idea?
16:32:24 <miguelgrinberg> sdague: I think it does help to justify the 400 with examples
16:32:25 <sdague> yeh, I'm totally happy adding more rationale to the docs, I think that's huge part of the value
16:32:29 * elmiko backs away slowly
16:32:42 <sdague> cdent: lets see if we can get past this 501 thing first :)
16:32:43 * cdent pokes elmiko
16:32:47 <cdent> :)
16:32:50 <elmiko> hehe
16:33:05 <sdague> and drink a lot of sake in tokyo and convince ourselves the action fight is worth fighting
16:33:14 <sdague> because I'm still mixed on that one
16:33:20 <elmiko> i think at some point we should circle back around on the http guideline and format it more closely to the template,
16:33:21 <miguelgrinberg> :)
16:33:24 <cdent> yeah, "worth" is a good question
16:33:35 <elmiko> we could add examples to each sub-section
16:33:39 <cdent> worth putting off
16:33:50 <sdague> elmiko: yeh, examples would be great
16:34:11 <elmiko> ok, any other guidelines we should be looking at, or might be ready for a freeze?
16:34:17 <sdague> I think every rule should end up with a why (rationale), what (here is the edict), and how (examples)
16:34:23 <elmiko> sdague: +1
16:34:50 <elmiko> i think part of the issue is that the http guideline started before we agreed on a format, so we'll need to clean it up at some point.
16:35:02 <sdague> yeh, no worries, I like that it's evolving that way
16:35:08 <elmiko> cool
16:35:46 <elmiko> ok, moving along
16:35:52 <elmiko> #topic APIImpact
16:36:02 <elmiko> #link https://review.openstack.org/#/q/status:open+AND+(message:ApiImpact+OR+message:APIImpact),n,z
16:36:21 <elmiko> anything there that folks would like to point out?
16:37:10 <elmiko> i know this one, #link https://review.openstack.org/#/c/187443/
16:37:23 <elmiko> got discussed in channel a bit, not sure if it needs any eyeballs or not
16:37:55 <elmiko> looks like they dropped the PATCH ideas, so maybe not
16:39:35 <elmiko> ok then
16:39:40 <miguelgrinberg> elmiko: seems pretty clean design
16:39:42 <elmiko> #topic open discussion
16:39:51 <ryansb> Yeah, they seem like they're fine
16:39:56 <ryansb> (the barbican review)
16:39:59 <elmiko> miguelgrinberg: yea, i think they hashed out all the ideas with you in irc
16:40:12 <cdent> I had dropped on the agenda earlier in the week but it went missing:
16:40:41 <cdent> there's a downstream request that all apis that list time oriented data support some form of changes-since, akin to what nova has on at least some stuff
16:40:59 <cdent> a) is changes-since well and proper?
16:41:04 <cdent> b) is it a portable concept?
16:41:38 <elmiko> so, would this be a request that could be made for changes-since?
16:41:47 <elmiko> (i'm not quite following the usage)
16:42:26 <cdent> I'm not sure I quite follow it either because I'm not clear on what the existing implementation is doing
16:42:33 <elmiko> lol, ok
16:42:33 <sdague> it's a filter
16:42:39 <miguelgrinberg> cdent: is the idea to get the entities that changed since a time, or the individual changes on them?
16:42:45 <elmiko> ah, that makes sense
16:42:47 <stevelle> why just "changes-since" instead of proper filtering on sortable property?
16:43:27 <sdague> because changes-since is a baked in contruct in some of the sqla apis in projects
16:43:35 <cdent> what does the existing thing do?
16:43:47 <stevelle> the construct is updated_at and also supports created_at iirc
16:44:09 <sdague> right, so it turns into an sql filter around updated_at
16:44:21 <sdague> and returns everything updated_at > changes-since
16:44:30 <stevelle> so what I'm asking is why not allow < or <= as well
16:44:30 <sdague> for various data slices
16:44:32 <miguelgrinberg> this can be integrated into ryansb guideline on filtering I think
16:44:58 <sdague> stevelle: could be, this is legacy stuff that's been there for a while
16:45:09 <nikhil_k> o/
16:45:13 <elmiko> miguelgrinberg: +1
16:45:15 <cdent> miguelgrinberg: yeah that probably gets to the core of my question: is changes-since is an implementation dependent use of generic sorting
16:45:19 <sdague> and I agree, the filtering guide probably is an appropriate place to talk about it
16:45:32 <cdent> sounds like it is
16:45:33 <nikhil_k> changes-since is very non-scalable (my 2 cents)
16:45:36 <miguelgrinberg> cdent: I see it as a query on a field of type timestamp
16:45:45 <sdague> nikhil_k: why is it non scalable?
16:45:46 <miguelgrinberg> or datetime if you are Pythonic
16:46:29 <nikhil_k> If your database is N years old and your tenant has X thousand rows, this may result into a gigantic scan + join query
16:46:49 <sdague> nikhil_k: no, it's scanning updated_at, which you've got an index on
16:46:54 <sdague> so it's super scalable
16:47:04 <miguelgrinberg> you should index your datetime fields
16:47:24 <sdague> right, exactly, if you don't... well good luck
16:47:27 <nikhil_k> right, but it won't return just the indexed rows
16:47:29 <edleafe> It actually helps scalability, as you don't pull data that hasn't changed
16:47:36 <sdague> nikhil_k: yes it will
16:47:43 <stevelle> nikhil_k: explain the join bit
16:47:49 <nikhil_k> how about joins on other tables?
16:47:56 <nikhil_k> One example
16:48:04 <sdague> nikhil_k: well, don't crazy overnormalize your tables :)
16:48:17 <nikhil_k> I can say without numbers on Rackspace public cloud that
16:48:29 <edleafe> nikhil_k: why is that worse than any other WHERE clause?
16:48:32 <woodster_> elmiko: (thanks for the barbican mention above btw, just caught up)
16:48:46 <sdague> edleafe: right, exactly
16:48:52 <nikhil_k> The database is at least 5 years old and we have to purge the whole thing to keep only 90 days worth data to let the query run using chages-since
16:48:52 <elmiko> woodster_: np, i know you had some questions about it
16:49:16 <nikhil_k> The worst case scenario is in image properties table
16:49:34 <nikhil_k> well, images have other tables that join over images one (like locations)
16:49:34 <sdague> nikhil_k: so I'd say that is a schema design problem
16:49:56 <sdague> I don't think that changes-since is inherently non scalable, any more than any other criteria
16:50:19 <nikhil_k> So, for example a tenant asks for changes-since 2 years. All images that are public, shared (And visible) and self creates would be retrieved
16:50:45 <miguelgrinberg> nikhil_k: using pagination, correct? you are not going to get all in one batch
16:50:55 <sdague> right, exactly
16:50:55 <nikhil_k> But it would be a mandatory thing for api that don't scale with it (unless I read the proposal wrond)
16:50:58 <nikhil_k> wrong(
16:51:11 <sdague> the fact that you aren't paging is a different problem
16:51:25 <nikhil_k> miguelgrinberg: pagination is another big issue.
16:51:48 <miguelgrinberg> what's the problem?
16:51:55 <nikhil_k> we can set page sizes on clients, servers and apis that nova proxies
16:51:58 <elmiko> does the change-since topic need a post on ML, or can we debate it in the filtering review?
16:52:11 <elmiko> also, <10 min. left
16:52:12 <nikhil_k> so, the latency of fetching can be inconsistent
16:52:20 <cdent> I think on the review elmiko
16:52:31 <sdague> yeh, the filtering review should be fine
16:52:32 <cdent> I hadn't realized raising the issue would be so interesting
16:52:42 <elmiko> cdent: k, cool.
16:52:55 <cdent> I was just trying to determine the extent to which it is worth considering for elsewhere
16:53:01 <elmiko> #action cdent add changes-since language to filtering review
16:53:01 <sigmavirus24> nikhil_k: so I'd argue that if we need to support changes-since that we should be able to change the response we provide
16:53:08 <elmiko> cdent: is that ok ^^?
16:53:09 <sigmavirus24> so that we aren't joining tables if we dont' need to
16:53:27 <cdent> my takeaway is that the specifics of using that term are not germane, there are better more generic ways
16:53:30 <cdent> yup elmiko
16:53:35 <sdague> so, I think we're exposing design issues in some of the projects in the process, which is good. In getting towards consistent APIs we're going to realize a lot of projects were just reflecting up very different db schema
16:53:36 <nikhil_k> I see, also pagination can't be really trusted
16:53:48 <edleafe> cdent: it's pretty generic for data polling
16:53:49 <ryansb> well it's possible to do it as two queries, one with fewer/no joins and then a load of just the new records
16:54:50 <sdague> sigmavirus24: yeh, I think there is a lot of useful db optimization on a bunch of projects that makes all this work better as well, fortunately we can do db migrations to help on that as well :)
16:55:17 <elmiko> a little side-note, i had an action item from summit to contact the CPLs and do some outreach
16:55:31 <sigmavirus24> sdague: glance hasn't had a good recent history with migrations =P
16:55:44 <elmiko> i've shared the nova-inspired liaison responsibilities, and merge process review with most of the listed CPLs
16:55:46 <miguelgrinberg> if there are no other topics, would you guys want to chat about HTTP caching?
16:56:07 <cdent> miguelgrinberg: in 5 minutes? ;)
16:56:12 <nikhil_k> sdague: +1 on db optimization,  input would be appreciated. I would in general be hesitent on changes-since. we never had good experience with it
16:56:25 <miguelgrinberg> cdent: I'm a dreamer :)
16:56:27 <elmiko> we are still looking for CPLs from congress, designate, magnum, mistral, murano, rally, triple-o, and zaqar. if you know anyone, let me know =)
16:56:55 <elmiko> miguelgrinberg: go for it, we'll carry over into openstack-api for those who are interested
16:57:21 <miguelgrinberg> just wanted to test the waters and see what you guys think. I really like to propose that we start adding caching headers to all APIs
16:57:36 <nikhil_k> miguelgrinberg: ++
16:57:38 <miguelgrinberg> a middleware that does the basic support would be nice
16:57:48 <elmiko> i really like the middleware idea
16:57:49 <miguelgrinberg> maybe setting etags on all responses, and interpreting the conditional requests
16:58:04 <sdague> so, I was talking with krotscheck about that the other day
16:58:11 <krotscheck> eh? wha?
16:58:18 <sdague> I'm not sure that etags middleware is going to do us a ton of good
16:58:27 <sdague> because you'll have to compute all the resources
16:58:41 <miguelgrinberg> sdague: yes, it only saves bandwidth
16:58:49 <miguelgrinberg> you still have to run the server side handler
16:58:55 <sdague> right, and bw is rarely the concern here.
16:59:13 <sdague> If we can get clients to not make calls they don't need to, that's a huge win
16:59:21 <miguelgrinberg> so I'm thinking we have to add caching to our ersponses, so we might as well do the etags, since it does not hurt
16:59:43 <sdague> but I don't think you can solve that with middleware, it has to be deeper in the project about what the cache-control really is for things
16:59:46 <elmiko> sorry, almost out of time, can we take this to openstack-api?
16:59:51 <sdague> yeh
16:59:51 <miguelgrinberg> but even adding a basiline cache-control header will be helpful
16:59:55 <elmiko> thanks everyone!
17:00:01 <elmiko> #endmeeting