| *** alex_klimov has quit IRC | 00:04 | |
| *** Apoorva has quit IRC | 00:54 | |
| *** openstack has joined #openstack-api | 01:22 | |
| *** openstack has joined #openstack-api | 01:37 | |
| *** openstack has quit IRC | 01:52 | |
| *** openstack has joined #openstack-api | 01:53 | |
| openstackgerrit | Alex Xu proposed openstack/api-wg: Add guideline of api microverion bump https://review.openstack.org/187896 | 03:20 |
|---|---|---|
| openstackgerrit | Alex Xu proposed openstack/api-wg: Add guideline for api microversion specification https://review.openstack.org/187112 | 03:20 |
| *** krotscheck has quit IRC | 04:19 | |
| *** gilliard has quit IRC | 04:22 | |
| *** krotscheck has joined #openstack-api | 04:25 | |
| *** gilliard has joined #openstack-api | 04:34 | |
| *** ryansb has quit IRC | 05:36 | |
| *** ryansb has joined #openstack-api | 05:45 | |
| *** ryansb has quit IRC | 05:45 | |
| *** ryansb has joined #openstack-api | 05:45 | |
| *** markdstafford has joined #openstack-api | 06:00 | |
| *** flaper87 has quit IRC | 06:33 | |
| *** flaper87 has joined #openstack-api | 06:33 | |
| *** terrylhowe has quit IRC | 06:54 | |
| *** woodster_ has quit IRC | 07:00 | |
| *** alex_klimov has joined #openstack-api | 07:52 | |
| openstackgerrit | Matthew Gilliard proposed openstack/api-wg: s/call/request/ - This isn't RPC https://review.openstack.org/187991 | 07:55 |
| *** lucasagomes has joined #openstack-api | 08:09 | |
| *** salv-orlando has quit IRC | 09:08 | |
| *** cdent has joined #openstack-api | 09:21 | |
| *** e0ne has joined #openstack-api | 09:44 | |
| *** salv-orlando has joined #openstack-api | 10:09 | |
| *** salv-orlando has quit IRC | 10:17 | |
| *** salv-orlando has joined #openstack-api | 10:20 | |
| *** salv-orlando has quit IRC | 10:21 | |
| *** salv-orlando has joined #openstack-api | 10:23 | |
| *** woodster_ has joined #openstack-api | 11:00 | |
| *** lucasagomes is now known as lucas-hungry | 11:27 | |
| *** e0ne is now known as e0ne_ | 11:39 | |
| *** e0ne_ has quit IRC | 11:49 | |
| *** lucas-hungry is now known as lucasagomes | 12:07 | |
| *** e0ne has joined #openstack-api | 12:13 | |
| *** e0ne is now known as e0ne_ | 13:01 | |
| *** e0ne_ is now known as e0ne | 13:04 | |
| *** terrylhowe has joined #openstack-api | 13:49 | |
| *** sigmavirus24_awa is now known as sigmavirus24 | 13:55 | |
| *** e0ne is now known as e0ne_ | 13:59 | |
| *** e0ne_ has quit IRC | 14:04 | |
| *** e0ne has joined #openstack-api | 14:06 | |
| gilliard | Hi. alex_xu | 14:28 |
| alex_xu | gilliard: hi :) | 14:28 |
| gilliard | About your microversion bump devref... | 14:29 |
| alex_xu | yup | 14:29 |
| gilliard | I was looking at a review earlier, something to do with live migration. The exact thing doesn't matter, but it needed an API error for the case where there was an rpc timeout internally in nova. | 14:30 |
| gilliard | This is an internal server error, which we can catch and return to the user. | 14:31 |
| alex_xu | gilliard: emm...good point | 14:32 |
| gilliard | We have a large number of things which can go wrong in nova, so I think 500 is sometimes a valid response code. Let me find the exact patch for you. | 14:33 |
| gilliard | https://review.openstack.org/#/c/168916/ | 14:33 |
| alex_xu | gilliard: so if we decided to turn the timeout error to some specific error code, we need bump the microversion | 14:34 |
| gilliard | Yes. Although I don't know what else we might change it to... | 14:34 |
| gilliard | There's nothing in the HTTP spec for "Expected server-side error" ;) But it's definitely in the 5xx range imo | 14:35 |
| gilliard | Like "we knew this might happen", as opposed to "something totally unexpected went wrong" | 14:35 |
| alex_xu | gilliard: yea, I need rephase the word, that is good point | 14:36 |
| alex_xu | gilliard: actulay it is nova trust other component, but when the behavior isn't expected, the nova return 500 | 14:38 |
| sdague | so, as folks here might have perspectives on this as well - this is a draft post about trying to explain the changes in the Nova API in a more accessible way - https://dague.net/?p=4461&shareadraft=baba4461_557061c963266 as part of our communications plan. If anyone wants to comment on concepts or confusing bits from an outside perspective, it would be appreciated. I'll do typos and grammar nits later | 14:38 |
| alex_xu | sdague: cool! | 14:38 |
| *** jxstanford has joined #openstack-api | 15:00 | |
| cdent | is this a 0:00 week for the meeting? | 15:05 |
| cdent | ah, my timezoning is brokenated | 15:09 |
| elmiko | 16:00 | 15:09 |
| elmiko | so, like 50 minutes | 15:10 |
| elmiko | i'm updating the agenda now =) | 15:10 |
| *** annegentle has joined #openstack-api | 15:13 | |
| ryansb | the new ical files are *great* | 15:13 |
| ryansb | the tool they wrote for it lets you generate your own files with just meetings you care about, so it's 1000x more useful | 15:14 |
| *** gmann has quit IRC | 15:14 | |
| cdent | I think my calendar got horkenated during a migration | 15:18 |
| *** gmann has joined #openstack-api | 15:19 | |
| elmiko | ryansb: are you talking about the red hat calendar stuff? | 15:22 |
| elmiko | or is this something openstack specific? | 15:22 |
| ryansb | elmiko: no, I mean the openstack one | 15:22 |
| ryansb | 1 sec, I'll link you | 15:22 |
| elmiko | interesting... yea, please =) | 15:22 |
| *** notmars has joined #openstack-api | 15:23 | |
| ryansb | ok, so the project is openstack-infra/yaml2ical | 15:23 |
| ryansb | and it grabs meetings from openstack-infra/irc-meetings | 15:24 |
| ryansb | you can get the Big Giant Master ical here http://eavesdrop.openstack.org/irc-meetings.ical | 15:24 |
| elmiko | neat | 15:24 |
| ryansb | or the list http://eavesdrop.openstack.org/ | 15:24 |
| ryansb | since we start with "A", we're top o' the list | 15:25 |
| elmiko | i ususally just grep through that latter list | 15:25 |
| elmiko | hehe | 15:25 |
| sdague | ryansb: yeh, I need to go play with that | 15:25 |
| *** peterstac has joined #openstack-api | 15:43 | |
| *** alex_klimov has quit IRC | 15:50 | |
| *** annegentle has quit IRC | 16:08 | |
| *** Apoorva has joined #openstack-api | 16:19 | |
| *** pballand has joined #openstack-api | 17:00 | |
| sdague | miguelgrinberg: so, I think my only concern with trying to do the caching stuff in Liberty, is there is going to be a bunch of stuff coming out of the API group in Liberty as is and projects are going to need time to digest and see what they are going to change in their existing stuff | 17:00 |
| miguelgrinberg | sdague: so what I'm thinking is that for a project that does not want to get bothered with caching, a middleware that sets the safe headers to prevent caching would be useful | 17:01 |
| *** nikhil_k is now known as nikhil-afk | 17:01 | |
| sdague | and it feels like just getting best practices written down would be good this cycle, and get folks used to digesting that, then attack something like that next cycle as a priority | 17:01 |
| cdent | that's what I was going to say: a first step is preventing caching for everything | 17:01 |
| miguelgrinberg | cdent: +1 | 17:02 |
| cdent | but (as usual) I agree with sdague that we don't want to do too much too quickly, or else no one will do anything | 17:02 |
| miguelgrinberg | but writing that in a guideline would imply projects have to code all this on their own | 17:02 |
| miguelgrinberg | versus saying here, add this middleware and you have something to start from | 17:02 |
| cdent | I'm cool with that but doing caching correctly (the part that actually allows caching to happen) requires really deep knowledge that a generic middleware will never be able to have, so that caveat will need to be present from the start | 17:03 |
| sdague | yeh, I'm a bit scared if it's done as the middleware bit first, it will stop there | 17:03 |
| sdague | and it seems like actually looking through what's protocol cacheable, what projects are returning, and where there are some mismatches (like GET /servers/ID_NOT_YET_CREATED - 404 and cachable by spec) | 17:04 |
| sdague | requires some effort. | 17:05 |
| miguelgrinberg | sdague: it's really hard to decipher, at least for me, how a cache should work in the default case, when no caching headers are included in the response | 17:05 |
| miguelgrinberg | it seems it isn't well defined | 17:05 |
| sdague | so, the spec actually does say what's cachable unless otherwise defined | 17:06 |
| sdague | I have a review up for that | 17:06 |
| cdent | there's what the spec says and what happens out in the world and the overlap is confused | 17:06 |
| miguelgrinberg | yes, it does, but it does not cover the "how" | 17:06 |
| sdague | cdent: sure | 17:06 |
| miguelgrinberg | without any specific guidance, how does a cache know for how long to cache a response? | 17:06 |
| miguelgrinberg | and for who? | 17:07 |
| sdague | so, I'll tell you right now, chrome caches a 301 for eternity | 17:07 |
| sdague | which has made my life a pain, as flushing that bit of it's cache is.. fun | 17:08 |
| cdent | (and ignores vary headers, which burns my hide) | 17:08 |
| miguelgrinberg | that very much confirms that we should be concerned in adding a no-cache baseline | 17:09 |
| *** lucasagomes has quit IRC | 17:10 | |
| sdague | yeh, though it would suck to add that to 404s that were valid and were mallformed so never would work | 17:10 |
| sdague | because I want those cached forever | 17:10 |
| sdague | and get the load off the server | 17:10 |
| sdague | or even the versions doc on / | 17:11 |
| *** salv-orlando has quit IRC | 17:12 | |
| miguelgrinberg | anyway, it seems we can only write a fairly generic guideline on caching, which will mostly say follow the RFC, and when in doubt, add no-cache headers. | 17:14 |
| miguelgrinberg | and maybe then we follow up with a middleware at some point | 17:15 |
| cdent | that seems like a starting point, which is far better than nothing | 17:16 |
| sdague | yep, sure | 17:16 |
| *** nikhil-afk is now known as nikhil_k | 17:17 | |
| *** notmars has quit IRC | 17:19 | |
| *** e0ne has quit IRC | 17:22 | |
| cdent | caring about stuff seem to be a recipe for going quickly insane | 17:29 |
| *** notmars has joined #openstack-api | 17:38 | |
| *** salv-orlando has joined #openstack-api | 17:40 | |
| krotscheck | I'm sorry, but I disagree with "Disable caching for everything is a good first step". | 17:41 |
| krotscheck | It's basically saying "Doing nothing is good!" | 17:41 |
| krotscheck | (Sorry I'm late to the conversation) | 17:41 |
| cdent | that's not quite accurate krotscheck | 17:42 |
| cdent | it protects against things being cached unexpectedly | 17:42 |
| cdent | which is an improvement over the current situation | 17:42 |
| krotscheck | The goal is to improve performance. If I am a client who's polling the services, I cause load on the server no matter what the cache settings are. | 17:42 |
| krotscheck | So adding something that checks the response for etag matching may not reduce load on the server, but it _does_ reduce load on the wire and any clients that are using it. | 17:43 |
| krotscheck | And while I agree that ultimately, the service itself should perform sane etag checking and provide a code path that minimizes server load, adding a middleware that does this kind of etag caching is only an improvement. It doesn't add _more_ load to the server when it comes to how clients use it. | 17:44 |
| krotscheck | cdent: Also, it's not the API's job to define how clients choose to use the API. | 17:45 |
| cdent | oh sure krotscheck, I _completely_ agree with that last statement, but I'm not clear on how that's germane? | 17:45 |
| krotscheck | You're basically removing my ability to make sane, performant choices. | 17:45 |
| cdent | ah, okay, perhaps | 17:46 |
| krotscheck | So hypothetically speaking, let's say we have an etag middleware that checks for oslo's created_at and updated_at fields in the response body, and generates an etag off of that. | 17:46 |
| krotscheck | Now I'm a client who's polling a resource. | 17:47 |
| krotscheck | I'm going to invoke server load on every request, period. | 17:47 |
| krotscheck | because I'm polling. | 17:47 |
| krotscheck | Adding this middleware allows me to be awesome about my own performance, and I don't really care about what the backend implementation is. | 17:48 |
| krotscheck | So if at some point in the future the implementation becomes awesome for the API, I am not impacted, because I'm already making awesome choices. | 17:48 |
| krotscheck | But it doesn't make things _worse_ for the API. | 17:48 |
| cdent | miguelgrinberg what do you think about ^ | 17:49 |
| miguelgrinberg | sorry, missed this discussion. I agree with krotscheck, and I have actually implemented something like this before. Not on openstack and not as a middleware, but I've done it. | 17:51 |
| miguelgrinberg | krotscheck: I used an MD5 of the whole response body for the etag | 17:51 |
| miguelgrinberg | I did not have access to created/updated_at, which when available could be used to add a last-modified header in addition to the etag | 17:52 |
| krotscheck | miguelgrinberg: I'm a little hesitant about doing that, because search results could be large, and md5'ing something like that could actually have a substantial performance impact. | 17:54 |
| krotscheck | Also, search results really should never cache, period. | 17:54 |
| krotscheck | (aaactually, I think I may be able to argue my way out of that one) | 17:54 |
| krotscheck | (But I digress) | 17:54 |
| miguelgrinberg | never say never :) | 17:54 |
| miguelgrinberg | If you have timestamps, then that could be used for the etag as well, as you suggest. | 17:55 |
| krotscheck | miguelgrinberg: I would love it if we could ask the database to see whether individual records in the search result set have changed, easily :) | 17:55 |
| krotscheck | miguelgrinberg: I'm pondering an etag middleware that accepts two parameters: created_fieldname and updated_fieldname. | 17:56 |
| krotscheck | If those don't exist, it does nothing. | 17:56 |
| miguelgrinberg | do we have any API that names these differently? | 17:56 |
| miguelgrinberg | created_at/updated_at seem pretty safe | 17:57 |
| krotscheck | miguelgrinberg: No idea, but I do not put it beyond openstack's governance model to allow special snowflakes. | 17:57 |
| miguelgrinberg | though adding a way to override those defaults does not hurt | 17:57 |
| miguelgrinberg | so why no caching on search results? | 17:57 |
| miguelgrinberg | if you can get a good etag on them it's no different | 17:57 |
| krotscheck | miguelgrinberg: Weeellll I dunno anymore. | 17:57 |
| cdent | I think the concern with this line of thinking is that it only works in situations where you have clear resources and unfortunately there are plenty of requests that result in responses that can't clearly claim nice resource orientation | 17:58 |
| cdent | and it is those requests which need caching-armor of some kind | 17:58 |
| krotscheck | cdent: That's why I'm thinking that etags are only generated if it detects the existence of timestamps. | 17:58 |
| krotscheck | The resource implicitly opts-in to cache support. | 17:58 |
| miguelgrinberg | that works for me | 17:59 |
| cdent | would the bad stuff implicitly opt in to no-cache-ing | 17:59 |
| krotscheck | A different approach would be to allow cache-expiry per regex-matched path. | 17:59 |
| krotscheck | cdent: If they don't have those timestamps, they don't get etag and/or last-modified support. Simple. | 18:00 |
| miguelgrinberg | in that case they get no-cache headers | 18:00 |
| cdent | what miguelgrinberg is saying is what I'm asking | 18:00 |
| cdent | because that represents the concerns that sdague was mentioning earlier | 18:00 |
| cdent | sorry, but I gott run, dinner | 18:00 |
| cdent | biab | 18:00 |
| krotscheck | miguelgrinberg: As for search results, I feel that the response structure of search results vary more than simple resource requests. Some API's return arrays of objects, others return an object with a results: field. | 18:01 |
| krotscheck | kk | 18:01 |
| *** annegentle has joined #openstack-api | 18:01 | |
| miguelgrinberg | krotscheck: right, so that's why I just hashed the whole thing when I implemented this in the past | 18:01 |
| krotscheck | miguelgrinberg: Right | 18:02 |
| miguelgrinberg | it's a baseline implementation, the API is free to generate a better etag, then the middleware is not going to contest it | 18:02 |
| krotscheck | miguelgrinberg: Though, if a search endpoint wants to start caching results, and they respond with an object, they _could_ just add the modified_at field to the result instance. | 18:02 |
| krotscheck | Also, any middleware we build should definitely check for api-set cache headers and deactivate itself if that logic's already been provided. | 18:03 |
| miguelgrinberg | krotscheck: ah, that's not a bad idea, they just put the newest of all the timestamps in the collection | 18:03 |
| miguelgrinberg | krotscheck: yup, if the server sets its own caching, then the middleware becomes a pass-thru | 18:04 |
| miguelgrinberg | actually, not sure you can come up with a good updated_at value to represent a page of a collection | 18:05 |
| miguelgrinberg | since individual results can come and go | 18:05 |
| krotscheck | miguelgrinberg: Hrm. Maybe if, for search results, the query parameters are added? | 18:06 |
| krotscheck | The etag then would become a statement of "For this set of query parameters, the most recent modified_at date is X" | 18:06 |
| krotscheck | Oh, I get what you're saying. | 18:07 |
| krotscheck | If an older record is deleted, shifting some things around, then you're still saying the results aren't fresh when they really are. | 18:07 |
| miguelgrinberg | krotscheck: it can't be done I think. Hashing the whole thing works, but I can't think of anything more subtle that is safe. | 18:09 |
| *** notmars has quit IRC | 18:14 | |
| krotscheck | miguelgrinberg: Nothing short of determining the ordered ID's in the result set plus the last time those resources are modified. We'd have to see what's faster. | 18:14 |
| krotscheck | miguelgrinberg: i.e. is it faster to just hash the whole thing, or to iterate through it all to generate some kind of a hasheable key? | 18:15 |
| miguelgrinberg | good question | 18:15 |
| krotscheck | miguelgrinberg: Lemme write a quick test script. | 18:15 |
| krotscheck | I'm really curious now. | 18:17 |
| miguelgrinberg | awesome | 18:17 |
| miguelgrinberg | I'm going to guess MD5 of the whole thing is faster, but also very curious to find out what your experiment tells us | 18:18 |
| *** annegentle has quit IRC | 18:21 | |
| *** annegentle has joined #openstack-api | 18:22 | |
| *** notmars has joined #openstack-api | 18:26 | |
| *** cdent has quit IRC | 18:29 | |
| *** cdent has joined #openstack-api | 18:33 | |
| *** elmiko is now known as _elmiko | 18:53 | |
| *** Apoorva has quit IRC | 19:04 | |
| *** annegentle has quit IRC | 19:07 | |
| *** salv-orlando has quit IRC | 19:15 | |
| *** salv-orlando has joined #openstack-api | 19:23 | |
| krotscheck | miguelgrinberg: I actually think the larger overhead is going to be in how long it takes to convert the data into a hasheable data type. | 19:24 |
| * krotscheck is trying to figure out how to hash a list :/ | 19:24 | |
| miguelgrinberg | krotscheck: can you hash the response body? (i.e. the JSON blob) | 19:25 |
| krotscheck | miguelgrinberg: Yep. Trying to figure out if that should be part of the algorithm or not. Does the response body arrive pre-encoded in a middleware's post handler? | 19:27 |
| *** elmiko has joined #openstack-api | 19:27 | |
| krotscheck | Actually, that's a good question. Would I have to deserialize the response body in the middleware? | 19:27 |
| miguelgrinberg | the middleware gets the response as a byte sequence | 19:28 |
| miguelgrinberg | it's the rendered JSON in our case | 19:28 |
| krotscheck | looks like the webob response has a json property where the data arrives deserialized. | 19:31 |
| cdent | assuming the response is a webob | 19:36 |
| cdent | I think you need to assume that middleware is pure WSGI | 19:37 |
| cdent | and nothing else | 19:37 |
| cdent | in which case miguelgrinberg is correct: it's a bytesequence | 19:37 |
| elmiko | cdent: +1 | 19:37 |
| cdent | why am I still here? It's nearly 9pm | 19:38 |
| * cdent grumbles | 19:38 | |
| elmiko | heh, go relax cdent! | 19:38 |
| cdent | I keep getting distracted by weird stuff | 19:38 |
| elmiko | i know the feeling | 19:38 |
| cdent | the latest: https://bugs.launchpad.net/gnocchi/+bug/1462076 | 19:39 |
| openstack | Launchpad bug 1462076 in Gnocchi "file-based tooz locking unreliable under concurrent measurement posts" [Undecided,New] | 19:39 |
| cdent | but yeah, I'm going to bail now that I've at least reported that problem and avoid trying to fix it today | 19:39 |
| cdent | have a good evening all, keep fighting the good fight etc | 19:39 |
| * cdent waves | 19:39 | |
| elmiko | take care | 19:39 |
| *** cdent has quit IRC | 19:39 | |
| *** e0ne has joined #openstack-api | 19:55 | |
| *** Apoorva has joined #openstack-api | 20:10 | |
| krotscheck | miguelgrinberg: http://paste.openstack.org/show/263825/ | 20:33 |
| krotscheck | That's assuming a webob.response, so there's no additional deserialization step. | 20:34 |
| krotscheck | Full hash is "just hash the raw body). | 20:34 |
| krotscheck | Selective hash is "Loop over the results and read that" | 20:34 |
| krotscheck | Payload is # of parameters per object. Result size is the humber of objects in the result set. | 20:35 |
| krotscheck | Also, that's running in pydevd, not regular python, so the performance is way worse, but the relationship is pretty clear. Selectively going through a result set for particular properties is WAY more expensive (by a factor of 1000) than just hashing the result body. | 20:36 |
| krotscheck | You can see the script I wrote here: http://paste.openstack.org/show/263910/ | 20:37 |
| *** e0ne has quit IRC | 21:05 | |
| *** notmars has quit IRC | 21:09 | |
| *** notmars has joined #openstack-api | 21:10 | |
| *** elmiko has quit IRC | 21:11 | |
| *** notmars has quit IRC | 21:25 | |
| *** salv-orlando has quit IRC | 21:35 | |
| *** salv-orlando has joined #openstack-api | 21:41 | |
| *** notmars has joined #openstack-api | 21:41 | |
| *** _elmiko is now known as elmiko | 21:47 | |
| miguelgrinberg | krotscheck: just got back from lunch, taking a look at your test | 21:56 |
| krotscheck | miguelgrinberg: Incidentally, I was wrong about the performance difference, it's actually an order of magnitutde greater. | 21:57 |
| miguelgrinberg | so my guess was based on the fact that MD5 is all native code, while selectively looking for data in each result goes at Python speed | 21:58 |
| *** sigmavirus24 is now known as sigmavirus24_awa | 22:13 | |
| *** notmars has quit IRC | 22:14 | |
| *** annegentle has joined #openstack-api | 22:35 | |
| *** annegentle has quit IRC | 22:40 | |
| *** annegentle has joined #openstack-api | 22:42 | |
| krotscheck | miguelgrinberg: That makes sense. It probably drops to the c-lib | 22:43 |
| krotscheck | miguelgrinberg: With that in mind, it may make the most sense to just md5 _everything_ instead of trying some wacky modified date lookup. | 22:44 |
| miguelgrinberg | yes, that's what I did in previous projects, and always worked well for me | 22:45 |
| miguelgrinberg | and you do this only if the response doesn't have the etag already in it, as this gives the server the option to generate its own more efficient etags | 22:45 |
| *** Apoorva has quit IRC | 23:03 | |
| *** Apoorva has joined #openstack-api | 23:04 | |
| *** annegentle has quit IRC | 23:08 | |
| *** salv-orlando has quit IRC | 23:53 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!