| *** sudorandom has joined #openstack-marconi | 00:14 | |
| *** reed has quit IRC | 00:23 | |
| *** reed_ has joined #openstack-marconi | 00:24 | |
| *** reed_ has quit IRC | 00:25 | |
| *** lakspace_ has joined #openstack-marconi | 00:31 | |
| *** amitgandhi has joined #openstack-marconi | 00:32 | |
| *** amitgandhi has quit IRC | 00:39 | |
| *** ayoung has quit IRC | 00:53 | |
| *** ayoung has joined #openstack-marconi | 00:59 | |
| *** nosnos has joined #openstack-marconi | 01:00 | |
| *** malini_afk is now known as malini | 01:19 | |
| *** amitgandhi has joined #openstack-marconi | 01:35 | |
| *** flwang has quit IRC | 01:37 | |
| *** amitgandhi has quit IRC | 01:40 | |
| *** amitgandhi has joined #openstack-marconi | 02:36 | |
| *** jcru has joined #openstack-marconi | 02:39 | |
| *** jcru has quit IRC | 02:40 | |
| *** amitgandhi has quit IRC | 02:40 | |
| *** oz_akan_ has quit IRC | 02:40 | |
| *** oz_akan_ has joined #openstack-marconi | 02:41 | |
| *** oz_akan_ has quit IRC | 02:44 | |
| *** oz_akan_ has joined #openstack-marconi | 02:45 | |
| *** amitgandhi has joined #openstack-marconi | 02:46 | |
| *** amitgandhi has quit IRC | 02:50 | |
| *** malini is now known as malini_afk | 03:00 | |
| *** amitgandhi has joined #openstack-marconi | 03:36 | |
| *** amitgandhi has quit IRC | 03:41 | |
| *** jburkhar1 has joined #openstack-marconi | 03:50 | |
| *** ekarlso has quit IRC | 03:52 | |
| *** jburkhart has quit IRC | 03:52 | |
| *** ekarlso has joined #openstack-marconi | 03:54 | |
| *** fandikurnia01 has joined #openstack-marconi | 03:58 | |
| *** flwang has joined #openstack-marconi | 04:13 | |
| *** amitgandhi has joined #openstack-marconi | 04:47 | |
| *** amitgandhi has quit IRC | 04:52 | |
| *** amitgandhi has joined #openstack-marconi | 05:37 | |
| *** amitgandhi has quit IRC | 05:42 | |
| *** amitgandhi has joined #openstack-marconi | 06:38 | |
| *** amitgandhi has quit IRC | 06:42 | |
| *** openstackgerrit has quit IRC | 06:46 | |
| *** openstackgerrit has joined #openstack-marconi | 06:46 | |
| *** cpallares has quit IRC | 07:01 | |
| *** amitgandhi has joined #openstack-marconi | 07:39 | |
| *** amitgandhi has quit IRC | 07:43 | |
| *** fandikurnia01 has quit IRC | 08:02 | |
| *** fandikurnia01 has joined #openstack-marconi | 08:03 | |
| *** flaper87|afk is now known as flaper87 | 08:08 | |
| *** amitgandhi has joined #openstack-marconi | 08:39 | |
| *** amitgandhi has quit IRC | 08:44 | |
| *** flwang has quit IRC | 09:01 | |
| *** yassine has joined #openstack-marconi | 09:05 | |
| *** nosnos has quit IRC | 09:26 | |
| *** nosnos has joined #openstack-marconi | 09:27 | |
| *** nosnos has quit IRC | 09:31 | |
| *** amitgandhi has joined #openstack-marconi | 09:40 | |
| *** amitgandhi has quit IRC | 09:45 | |
| *** amitgandhi has joined #openstack-marconi | 10:41 | |
| *** amitgandhi has quit IRC | 10:45 | |
| *** haomaiwang has quit IRC | 10:54 | |
| *** haomaiwang has joined #openstack-marconi | 10:55 | |
| *** haomaiwang has quit IRC | 10:59 | |
| *** fandikurnia01 has quit IRC | 11:12 | |
| *** jamieh__ has joined #openstack-marconi | 11:18 | |
| *** amitgandhi has joined #openstack-marconi | 11:41 | |
| *** amitgandhi has quit IRC | 11:46 | |
| *** haomaiwang has joined #openstack-marconi | 11:54 | |
| *** vkmc has joined #openstack-marconi | 11:56 | |
| *** tedross has joined #openstack-marconi | 12:39 | |
| *** amitgandhi has joined #openstack-marconi | 12:42 | |
| *** amitgandhi has quit IRC | 12:47 | |
| *** amitgandhi has joined #openstack-marconi | 12:52 | |
| *** aniuskad has joined #openstack-marconi | 12:53 | |
| *** aniuskad has left #openstack-marconi | 12:54 | |
| *** amitgandhi has quit IRC | 12:56 | |
| *** amitgandhi has joined #openstack-marconi | 13:42 | |
| *** amitgandhi has quit IRC | 13:47 | |
| * flaper87 wonders where Alejandro is | 13:49 | |
| *** openstackgerrit has quit IRC | 13:53 | |
| *** openstackgerrit has joined #openstack-marconi | 13:53 | |
| *** aniuskad has joined #openstack-marconi | 14:04 | |
| *** jcru has joined #openstack-marconi | 14:10 | |
| *** russellb is now known as rustlebee | 14:13 | |
| * flaper87 shakes marconi channel | 14:23 | |
| *** amitgandhi has joined #openstack-marconi | 14:24 | |
| *** amitgandhi has quit IRC | 14:26 | |
| *** amitgandhi has joined #openstack-marconi | 14:27 | |
| *** malini_afk is now known as malini | 14:38 | |
| *** kgriffs_afk is now known as kgriffs | 14:42 | |
| flaper87 | kgriffs: malini morning :) | 14:42 |
|---|---|---|
| kgriffs | o/ | 14:42 |
| * flaper87 is reviewing partition patches | 14:43 | |
| malini | hello!! | 14:49 |
| *** flwang has joined #openstack-marconi | 14:56 | |
| kgriffs | flaper87: Comments here, just minor stuff - https://review.openstack.org/#/c/52389/ | 15:07 |
| flaper87 | kgriffs: awesome, thanks for reviewing that | 15:07 |
| kgriffs | yw | 15:07 |
| kgriffs | flaper87: u see this? http://arstechnica.com/information-technology/2013/11/amazon-wades-into-big-data-streams-with-kinesis/ | 15:17 |
| flaper87 | kgriffs: damn! | 15:22 |
| flaper87 | I keep looking at that diagram and reading Marconi everywhere in it! | 15:25 |
| kgriffs | yeah | 15:27 |
| kgriffs | esp given a high-perf queue flavor + push streaming to workers | 15:27 |
| kgriffs | a couple observations | 15:27 |
| kgriffs | First, they invented a new protocol and everything. They didn't reference, e.g., Storm. | 15:28 |
| kgriffs | Second, they missed an opportunity to create a generic "worker pool as a service" | 15:28 |
| kgriffs | I suppose you get some performance benefits from tightly defining what producers and workers can and can't do | 15:29 |
| flaper87 | I don't expect AWS to use any existing technology / protocol | 15:29 |
| flaper87 | because they can't control it | 15:29 |
| flaper87 | brb, meeting | 15:29 |
| kgriffs | yep, this is par for the course wrt Amazon | 15:29 |
| kgriffs | Amazon is the new Microsoft | 15:30 |
| * kgriffs opinions aren't necessarily those of his employer. | 15:37 | |
| kgriffs | :p | 15:37 |
| *** alcabrera has joined #openstack-marconi | 15:42 | |
| alcabrera | Good morning, everyone. :) | 15:43 |
| kgriffs | 0/ | 15:45 |
| alcabrera | kgriffs: yo! :D | 15:46 |
| *** rustlebee is now known as drumkilla | 15:50 | |
| *** drumkilla is now known as rustlebee | 15:50 | |
| *** rustlebee is now known as drumkilla | 15:51 | |
| kgriffs | alcabrera. malini: when flaper87 is back, I'd like to spend some time triaging bugs | 15:57 |
| * flaper87 back | 15:58 | |
| flaper87 | kgriffs: LOL | 15:58 |
| flaper87 | alcabrera: goooooooood morning!! | 15:58 |
| malini | we can set all the proxy related bugs to Invalid, rt? | 15:58 |
| flaper87 | malini: I'd say yes, alcabrera ? | 15:59 |
| malini | alcabrera is busy discussing requests module with amitgandhi | 15:59 |
| *** tedross has quit IRC | 16:01 | |
| * alcabrera catche sup | 16:02 | |
| alcabrera | *catches up | 16:02 |
| alcabrera | flaper87: heeey! :D | 16:03 |
| alcabrera | malini: yup - `del proxy` | 16:03 |
| malini | I dont have the privilege to 'Won't Fix' | 16:03 |
| malini | somebody else change thtose, plz | 16:04 |
| kgriffs | alcabrera: you should have the ACL | 16:04 |
| kgriffs | I am assuming it is tied to being a member of Marconi Core | 16:04 |
| kgriffs | protip: ls-n.net is *not* the link shortner you are looking for | 16:05 |
| flaper87 | kgriffs: alcabrera it's | 16:05 |
| alcabrera | I'll "Won't Fix"-ify in just a moment. :) | 16:06 |
| * alcabrera has the ACL powerz | 16:06 | |
| kgriffs | crap. stupid ln-s.net won't shorten my launchpad link | 16:06 |
| * kgriffs tries goo.gl | 16:06 | |
| kgriffs | http://goo.gl/Q8RwQS | 16:06 |
| alcabrera | kgriffs: "俺の話しを聞け！更新 ～Vol.104 俺の話しを聞け！～" - definitely not the url shortener. :P | 16:07 |
| flaper87 | kgriffs: you shortened the ln-s.net link | 16:07 |
| flaper87 | :D | 16:07 |
| kgriffs | lol | 16:07 |
| kgriffs | http://goo.gl/Yc4kau | 16:07 |
| kgriffs | let's first go down and update status on triaged bugs, then we can look at some news ones | 16:08 |
| kgriffs | let's see | 16:08 |
| kgriffs | homedoc fix | 16:08 |
| kgriffs | I don't see Mr. Vollero in the chanell | 16:08 |
| kgriffs | channel | 16:09 |
| flaper87 | doesn't seem to be around! | 16:09 |
| kgriffs | ok | 16:10 |
| kgriffs | afaik this work is still pending? | 16:11 |
| kgriffs | actually | 16:11 |
| flaper87 | are we going to update the home doc at some point or replace it with the discoverable API ? | 16:11 |
| kgriffs | I thought I remembered seeing a patch | 16:11 |
| malini | I dont see anything here https://review.openstack.org/#/q/status:open+project:openstack/marconi,n,z | 16:13 |
| kgriffs | flaper87: mmm, good question. Let's discuss that at the meeting. My first thought is to see if there is a way we can extend the home doc rather than inventing our own one-off format | 16:13 |
| flaper87 | kgriffs: kk, kinda update it dynamically | 16:14 |
| kgriffs | hmm, I was just reading Sam Harwell's comment on that bug | 16:15 |
| kgriffs | you guys see that, at the bottom? | 16:15 |
| flaper87 | nope | 16:15 |
| kgriffs | "is flawed due to the simple fact that URI construction from the listed templates using that RFC will always fail when used with RBAC." | 16:15 |
| kgriffs | I don't follow his logic | 16:15 |
| *** yassine has quit IRC | 16:16 | |
| flaper87 | ah, sorry, I do see his comment, I thought you sent one. | 16:16 |
| kgriffs | RBAC doesn't come into play here, perhaps he is referring to keystone/catalog auth | 16:16 |
| kgriffs | (for those following along at home: https://bugs.launchpad.net/marconi/+bug/1245656) | 16:17 |
| kgriffs | even then, the RFC rules still work fine, don't they? | 16:17 |
| flaper87 | mmh, I don't follow that part either | 16:17 |
| flaper87 | :/ | 16:17 |
| flaper87 | I think they do! | 16:18 |
| kgriffs | If I query "http://marconi.example.com/v1/204824" then get back a URI with an absolute path, I would construct the next uri like so | 16:19 |
| kgriffs | "http://marconi.example.com/v1/queues" | 16:19 |
| kgriffs | if my path is relative, then I would do | 16:19 |
| *** tedross has joined #openstack-marconi | 16:19 | |
| kgriffs | "http://marconi.example.com/v1/204824/queues" | 16:19 |
| kgriffs | flaper87: btw, Rackspace keystone currently is hard-coded to put the project ID in the URI in the catalog | 16:20 |
| flaper87 | :P | 16:20 |
| kgriffs | so, today we just have a rewrite rule in uwsgi to strip it | 16:20 |
| flaper87 | I was a bit confused | 16:20 |
| flaper87 | kk | 16:20 |
| alcabrera | middleware solves everything (wrt to hard-coded Project Id). :P | 16:21 |
| flaper87 | but yeah, the base url is the one that returns the API spec | 16:21 |
| kgriffs | right | 16:21 |
| flaper87 | somehow I read middleware as middleman | 16:21 |
| flaper87 | @.@ | 16:21 |
| kgriffs | and the bug is that we return an absolute path, but didn't include the "v1" part | 16:21 |
| flaper87 | it's Friday after all | 16:21 |
| flaper87 | exactly | 16:21 |
| kgriffs | so, for a stock deployment | 16:21 |
| kgriffs | client queries "https://marconi.example.com/v1" | 16:22 |
| flaper87 | also, the home generation can be pulled out of the version and be completely dynamically generated once we get the API layer done | 16:22 |
| kgriffs | get's back, e.g., "queues/{queue-name}" | 16:22 |
| kgriffs | and would construct a bogus URI; | 16:22 |
| flaper87 | 'of the version package' | 16:22 |
| flaper87 | yeah | 16:22 |
| kgriffs | "https://marconi.example.com/queues/my-queue" | 16:22 |
| kgriffs | can somebody sanity-check me on this? I want to make sure I am understanding RFC 3986 correctly | 16:23 |
| kgriffs | could be as simple as testing with urllib | 16:23 |
| flaper87 | FWIW, I follow your thinking and I think it's correct | 16:24 |
| kgriffs | oops, not urllib | 16:24 |
| kgriffs | urlparse | 16:24 |
| alcabrera | I mix those two up, as well, since urlparse and urllib methods were decoupled a bit between py2 and py3 | 16:25 |
| malini | kgriffs: anytime you mention RFC, a few of us are going to get lost :( | 16:25 |
| kgriffs | heh | 16:26 |
| alcabrera | Request For Cookies 3986. :D | 16:26 |
| malini | alcabrera: tht's a lot better :D | 16:26 |
| flaper87 | and 3986 is the number of cookies being requested | 16:26 |
| alcabrera | exactly | 16:27 |
| malini | RFC is a sure way to kill the triaging session.. | 16:28 |
| flaper87 | hahah | 16:29 |
| malini | I am trying to read the Section 5.2.2 & my poor brain cells are confused | 16:29 |
| flaper87 | I started marking other bugs as Won't fix | 16:30 |
| malini | " A non-strict parser may ignore a scheme in the reference | 16:30 |
| malini | -- if it is identical to the base URI's scheme. | 16:30 |
| malini | " | 16:30 |
| flaper87 | I mean, proxy bugs | 16:30 |
| flaper87 | LD | 16:30 |
| flaper87 | :D | 16:30 |
| malini | wth does that mean? | 16:30 |
| malini | My vote is to come to this bug after all else | 16:30 |
| flaper87 | alcabrera: I assume all those proxy bugs can be marked as such | 16:30 |
| alcabrera | The parser can ignore "https://", for example, if it matches with the scheme for the Host, "https://marconi.blah.ly" | 16:31 |
| alcabrera | flaper87: yup - WF | 16:31 |
| kgriffs | ok, let's come back to this bug | 16:31 |
| kgriffs | Let me write up a counter-point to Sam's argument | 16:32 |
| kgriffs | and test it with urlparse | 16:32 |
| kgriffs | next up | 16:32 |
| kgriffs | "Autoreconnect not handled in all cases" | 16:32 |
| flaper87 | mmh | 16:33 |
| kgriffs | So, this one has stalled. I need to get back to that | 16:33 |
| kgriffs | I've got a patch halfway-done | 16:33 |
| flaper87 | ok | 16:33 |
| flaper87 | you should pick that bug | 16:34 |
| flaper87 | assign it to you | 16:34 |
| kgriffs | it's already assigned | 16:34 |
| flaper87 | mmh | 16:34 |
| kgriffs | this one is unassigned | 16:34 |
| kgriffs | Layer 3 errors in MongoDB driver not handled gracefully | 16:34 |
| flaper87 | wait, I'm looking at the wrong bug | 16:34 |
| flaper87 | https://bugs.launchpad.net/marconi/+bug/1214974 | 16:34 |
| flaper87 | ah, nevermind | 16:34 |
| kgriffs | heh | 16:34 |
| flaper87 | I was indeed looking at the wrong bug | 16:34 |
| flaper87 | :D | 16:34 |
| flaper87 | It's friday after all! | 16:34 |
| kgriffs | so, I think we should wait on that until the autoreconnect is done. | 16:35 |
| flaper87 | +1 | 16:35 |
| kgriffs | between Mondays and Fridays, it's a wonder anything productive gets done! | 16:35 |
| kgriffs | ok, next | 16:35 |
| kgriffs | "Message body size validation is slow" | 16:35 |
| kgriffs | related: "total size for all message bodies is not validated" | 16:36 |
| kgriffs | zyuan starting looking at these and thinking about what to do there | 16:36 |
| alcabrera | The slowness bug - that's because we reserialize? | 16:36 |
| kgriffs | alcabrera: yes | 16:37 |
| alcabrera | gotcha | 16:37 |
| kgriffs | ideally, json.load would just count things as it went | 16:37 |
| zyuan | kgriffs: i did some hack to simplejson | 16:38 |
| kgriffs | even if we say, we will include whitespace in the count | 16:38 |
| zyuan | so, it's possible | 16:38 |
| kgriffs | we still have a problem because we have to size individual objects independently | 16:38 |
| kgriffs | like, the size of each message in the array of posted messages | 16:38 |
| zyuan | just, it's a question whether openstack requirements.txt can accept a hacked simplejson | 16:38 |
| kgriffs | so, I am starting to wonder if this is all worth it after all | 16:38 |
| kgriffs | zyuan: exactly. More likely we would have to get the patch merged upstream into simplejson and then it becomes "standard" anyway | 16:39 |
| kgriffs | alternatively, we fork simplejson and include it directly in marconi | 16:39 |
| kgriffs | but I am not exactly jumping up and down with joy to do that | 16:40 |
| zyuan | and, besides, currently our integer overflow checking slowed simplejson down | 16:40 |
| kgriffs | zyuan: that is another good point. I suspect getting a patch into simplejson for just that wouldn't be too difficult | 16:40 |
| kgriffs | but | 16:41 |
| kgriffs | I guess that is a different bug | 16:41 |
| zyuan | -- so... if it's not openstack, based on "my ways" i'll fork simplejson and get it solved. :) | 16:41 |
| kgriffs | has one been registered for the integer overflow thing? | 16:41 |
| zyuan | kgriffs: i'm not sure whether they care | 16:41 |
| zyuan | as same as json in python std lib, they both populate bignum | 16:42 |
| kgriffs | zyuan: alternate ways to check overflow? Could we just do it after parsing the JSON? | 16:42 |
| zyuan | it's just database, msgpack, etc uses fixed size integers | 16:42 |
| zyuan | kgriffs: if i do, i'll just get this checked within simplejson C code... | 16:43 |
| zyuan | kgriffs: no. if so, we have have to walk down through the json tree | 16:43 |
| zyuan | this is costly | 16:43 |
| zyuan | pypy's famous quote: O(1) is not doing nothing | 16:44 |
| kgriffs | remind be the original problem that overflow check was trying to solve? | 16:44 |
| kgriffs | zyuan: LOL. So true. | 16:45 |
| kgriffs | btw, created a bug for this just so we can track what happens with it | 16:45 |
| kgriffs | https://bugs.launchpad.net/marconi/+bug/1251693 | 16:45 |
| zyuan | integer generated from simplejson/json != that used by backend | 16:45 |
| zyuan | while marconi defines data type based on what backend have | 16:45 |
| kgriffs | that rings a bell | 16:46 |
| kgriffs | backend was raising an error when trying to persist big ints | 16:46 |
| kgriffs | but, some backends may support bignum | 16:47 |
| zyuan | kgriffs: this is almost impossible | 16:47 |
| kgriffs | is it not possible to catch raised errors from the backend after the fact? | 16:47 |
| zyuan | (<del>if they do, i'll use msgpack intead</del>) | 16:48 |
| zyuan | kgriffs: then you're going to run into teansaction problem | 16:48 |
| zyuan | msg insertion must success as a whole | 16:49 |
| kgriffs | good point | 16:49 |
| kgriffs | what is the performance hit from that hook? | 16:49 |
| kgriffs | (the validation hook) | 16:49 |
| zyuan | no sure. basically, it requires an C-> Python -> C context switch | 16:50 |
| zyuan | while all other types of parsing are done within C | 16:50 |
| zyuan | an intersting thing is, if we just pure python json in pypy, it's not an issue | 16:51 |
| zyuan | because pypy optimize pure python very well | 16:51 |
| zyuan | it's just cpython is slow if you use C and python at the same time... | 16:51 |
| zyuan | anyway, even with the hook, simplejson is much faster than pure python json under cpython | 16:52 |
| zyuan | so this "bug" is merely, i think, we can do better | 16:52 |
| alcabrera | zyuan: IIRC, 2.7 and 3.x python use simplejson as the implementation in the stdlib, so they're equivalent in perf. | 16:53 |
| alcabrera | in 2.6, it *is* really slow | 16:54 |
| zyuan | alcabrera: no | 16:54 |
| zyuan | 2.7 is still pure python | 16:54 |
| alcabrera | ah, I see. | 16:54 |
| alcabrera | found it - you're right. :) | 16:54 |
| kgriffs | ok | 16:55 |
| zyuan | as well as py33? | 16:55 |
| zyuan | http://morepypy.blogspot.com/2011/10/speeding-up-json-encoding-in-pypy.html | 16:56 |
| alcabrera | hmmmm | 16:56 |
| alcabrera | http://hg.python.org/cpython/file/f39fac22ca9a/Modules/_json.c | 16:56 |
| zyuan | ok | 16:56 |
| zyuan | not a loadable module | 16:56 |
| kgriffs | "Note that CPython by default uses the optimized C extension" | 16:57 |
| zyuan | already a part of the intepreter | 16:57 |
| kgriffs | so why is simplejson faster? | 16:57 |
| zyuan | now they are the same thing i belive | 16:57 |
| zyuan | bring py27 to py26? | 16:58 |
| kgriffs | they do behave a little differently with unicode | 16:58 |
| kgriffs | if I set ensure_ascii=False then simplejson always returns unicode | 16:58 |
| zyuan | i see | 16:58 |
| kgriffs | whereas the stdlib only returns unicode if some of the strings it serializes are unicode | 16:58 |
| kgriffs | it's annoying, actually | 16:59 |
| kgriffs | I prefer simplejson | 16:59 |
| kgriffs | regarding perf, I guess someone should benchmar it | 16:59 |
| alcabrera | http://paste.openstack.org/show/53172/ | 16:59 |
| zyuan | (i don't think so) | 16:59 |
| alcabrera | done ^^ (it's a lazy benchmark, admittedly) | 16:59 |
| kgriffs | hmm | 17:00 |
| openstackgerrit | Flavio Percoco proposed a change to openstack/python-marconiclient: Bootstrap Messages support https://review.openstack.org/52389 | 17:02 |
| kgriffs | alcabrera: can you test py2.6 as well? | 17:04 |
| kgriffs | flaper87: It seems that simplejson isn't faster than the stdlib after all! | 17:05 |
| alcabrera | kgriffs: sure thing. I just happen to have py2.6 installed. :) | 17:05 |
| flaper87 | I was just looking at alcabrera bench | 17:05 |
| *** aniuskad has quit IRC | 17:05 | |
| flaper87 | mmh | 17:05 |
| flaper87 | that's interesting | 17:05 |
| flaper87 | well, that removes a dependency from our requirements.txt | 17:05 |
| flaper87 | :D | 17:05 |
| kgriffs | I do prefer it's unicode handling, but we can work around that I suppose | 17:05 |
| zyuan | i'm in favor of the "sometimes not unicode" way | 17:06 |
| zyuan | it's just fine to use str as unicode in python2, they have the same interface | 17:07 |
| zyuan | -- duck typing | 17:07 |
| *** reed has joined #openstack-marconi | 17:07 | |
| zyuan | str also supports encode, decode, etc | 17:07 |
| zyuan | unlike that in python 3 | 17:07 |
| zyuan | anyway. if we support py26 as well, we may need to preserve simplejson for 26 as least | 17:08 |
| zyuan | if kurt insist to have "always unicode", then preserve it to 27 as well | 17:08 |
| zyuan | py33, not needed | 17:09 |
| kgriffs | nah, it isn't a big deal. | 17:09 |
| kgriffs | I mean, always having unicode returned in py2 | 17:10 |
| flaper87 | plus, I expect 26 to disapear soon | 17:10 |
| kgriffs | alcabrera: be sure to bench loads as well | 17:10 |
| kgriffs | flaper87: srsly? | 17:10 |
| kgriffs | What's the RHEL story there? | 17:10 |
| zyuan | ^^ a fact that i don't like, always unicode | 17:10 |
| * flaper87 hides | 17:11 | |
| flaper87 | ghghghgh | 17:11 |
| alcabrera | I'll be adding loads to the benchmark now. | 17:11 |
| alcabrera | Here's the dumps: http://paste.openstack.org/show/53173/ | 17:11 |
| kgriffs | flaper87: maybe docker will save us. :p | 17:12 |
| alcabrera | IIRC, Fedora (next) ships with py3 by default. I wonder if that's related? ;) | 17:12 |
| alcabrera | py3 as default, I should say. | 17:12 |
| flaper87 | huhauhauhauahua | 17:12 |
| kgriffs | alcabrera: I just did a quick test and simplejson is faster for loads | 17:12 |
| kgriffs | while stdlib is fater for dumps | 17:12 |
| kgriffs | WAH?! | 17:13 |
| flaper87 | well, yeah, there're still some envs to support but I've this feeling | 17:13 |
| flaper87 | :P | 17:13 |
| kgriffs | srsly, if people can deploy openstack with docker, then doesn't it make it less important to support py2.6 ? | 17:13 |
| flaper87 | alcabrera: that's certainly related | 17:13 |
| flaper87 | anyway, I'll have to step out in a bit | 17:14 |
| flaper87 | lets move on! | 17:14 |
| flaper87 | next bug | 17:14 |
| alcabrera | +1 - let's move away from this one. :D | 17:14 |
| kgriffs | actually, let's stop here and pick up again next week | 17:15 |
| alcabrera | ( kgriffs: on py 3.3.2, json.loads is faster than simplejson.loads ) | 17:15 |
| kgriffs | btw, don't forget about the mtg on Monday | 17:15 |
| kgriffs | https://wiki.openstack.org/wiki/Meetings/Marconi#Agenda | 17:15 |
| alcabrera | w00t - looking forward to the meeting. | 17:15 |
| kgriffs | Add stuff to the agenda and tag with ur name if there is other stuff to discus | 17:15 |
| kgriffs | It was recently suggested that all the projects stop spamming openstack-dev with meeting reminders | 17:16 |
| alcabrera | interesting | 17:16 |
| kgriffs | Part of the discussion to reduce noise on the list in general | 17:17 |
| kgriffs | (FWIW) | 17:17 |
| kgriffs | so, let's get closure on the size issue and the simplejson issue | 17:17 |
| kgriffs | first, I feel myself being won over to just specifying a max size for the message posting overall, including whitespace | 17:18 |
| kgriffs | and removing the individual message size limit | 17:18 |
| kgriffs | this is basically what we are already doing | 17:18 |
| kgriffs | just formalize that | 17:19 |
| kgriffs | flaper87, alcabrera, zyuan, megan_w, malini: thoughts? | 17:19 |
| alcabrera | +1 | 17:19 |
| alcabrera | it's simple, it's fast, and it's easy to reason about | 17:19 |
| alcabrera | len(body_content) < max_size | 17:19 |
| kgriffs | we just say, hey clients, you can't post a serialized array of messages where the resulting text is greater than, e.g., 256 K | 17:20 |
| flaper87 | mmh | 17:20 |
| kgriffs | by extension, that means no individual message could be > 256 K | 17:20 |
| alcabrera | and most clients will already be using a quality JSON module (regardless of language) that'll mini-fy the JSON output. | 17:20 |
| flaper87 | you mean, have a max_size for the whole post operation (bulk and single) instead of max_size per message? | 17:20 |
| flaper87 | ok, yeah, you meant that | 17:21 |
| kgriffs | flaper87: that's correct | 17:21 |
| kgriffs | it also has the nice side-effect of encouraging clients to not pretty-print by default | 17:21 |
| kgriffs | reducing overall traffic going to a marconi deployment | 17:21 |
| flaper87 | mhh, but what's the benefit there? When the request gets to Marconi, it's already in memory | 17:22 |
| flaper87 | why shouldn't we process it if we already have it ? | 17:22 |
| kgriffs | couple things | 17:22 |
| kgriffs | first, you could also restrict this in the web server | 17:22 |
| kgriffs | in fact, just check content-length | 17:22 |
| flaper87 | (genuine question, btw) | 17:22 |
| kgriffs | then it isn't in memory | 17:22 |
| kgriffs | we only read content-length bytes anyway | 17:22 |
| kgriffs | second benefit is we don't end up with arbitrarily large documents in the backend storage | 17:23 |
| flaper87 | but for 1) it won't be marconi to enforce the limit but the web server, right? | 17:23 |
| kgriffs | keeps those sane | 17:23 |
| kgriffs | flaper87: good question; the web server could enforce the limit | 17:23 |
| flaper87 | but we will end up with large documents | 17:23 |
| kgriffs | and we could also check it in the transport | 17:24 |
| flaper87 | I can send 1 document of 50kbs and another one of 206 | 17:24 |
| kgriffs | flaper87: yes, but no single document could exceed the overall max | 17:24 |
| flaper87 | ooook | 17:24 |
| flaper87 | gotcha | 17:24 |
| kgriffs | so, assuming the overall max is something sane, then we are ok | 17:24 |
| flaper87 | got your point - again, it's Friday in Italy, I don't know about the US but we become very dumb on Fridays - | 17:25 |
| kgriffs | so, idk whether the web server should be expected to enforce the limit or we add a second-tier check in the transport as wel | 17:25 |
| *** cindy_ has joined #openstack-marconi | 17:25 | |
| flaper87 | I like the idea | 17:25 |
| kgriffs | flaper87: LOL | 17:25 |
| flaper87 | so, +1 from me | 17:25 |
| kgriffs | flaper87: In the US we get very dump on Mondays | 17:25 |
| kgriffs | s/dump/dumb | 17:26 |
| alcabrera | lol | 17:26 |
| flaper87 | I think it all comes down to food | 17:26 |
| flaper87 | We have pasta like everyday 10 times per day | 17:26 |
| kgriffs | folks, any advantages to doing the sanity-check in the marconi app itself? | 17:26 |
| flaper87 | then on saturdays we're kinda dead | 17:26 |
| kgriffs | seems like it may make life a little easier for ops | 17:26 |
| flaper87 | and then we have a 10 hours lunch on sundays | 17:26 |
| alcabrera | flaper87: soo much foooood. :) | 17:27 |
| kgriffs | i mean, they don't have to make a rule that only applies to POSTing messages | 17:27 |
| kgriffs | (in their web server config) | 17:27 |
| flaper87 | alcabrera: isn't that what life is about? I mean, eating | 17:27 |
| flaper87 | +1 for having that in Marconi | 17:27 |
| kgriffs | flaper87: WAH?! | 17:27 |
| flaper87 | that's part of Marconi's API | 17:27 |
| kgriffs | flaper87: You need to invite me over for lunch | 17:27 |
| kgriffs | flaper87: I agree. | 17:27 |
| kgriffs | ok, so the proposal is: | 17:27 |
| alcabrera | I favor having the check in marconi | 17:28 |
| flaper87 | kgriffs: you're more than welcome! I'll make sure you'll have the best pasta ever! | 17:28 |
| kgriffs | 1. Redefine max message size in the spec (v1.1 I guess) | 17:28 |
| flaper87 | alcabrera: you too, so, stop crying! | 17:28 |
| flaper87 | :D | 17:28 |
| kgriffs | 2. Do a quick content-length check in the WSGI transport driver | 17:28 |
| flaper87 | zyuan: and you too! | 17:28 |
| flaper87 | :D | 17:28 |
| * kgriffs is drooling all over his keyboard | 17:28 | |
| malini | I have no friends :'( | 17:29 |
| flaper87 | malini: YOU TOOOO! | 17:29 |
| flaper87 | sorrrryyyyy!!! | 17:29 |
| flaper87 | :D | 17:29 |
| malini | what a relief!! | 17:29 |
| flaper87 | kgriffs: LOOOL | 17:29 |
| flaper87 | kgriffs: +1 for the proposal | 17:29 |
| alcabrera | flaper87: hahaha, awesome. :D | 17:29 |
| flaper87 | it sounds good! | 17:29 |
| kgriffs | zyuan: sound good? ^^^ | 17:29 |
| kgriffs | ok, so we can basically do the implementation now | 17:30 |
| kgriffs | and formalize it in the docs for API v1.1 | 17:30 |
| kgriffs | in other words, we won't "fix" those bugs until 1.1 | 17:30 |
| flaper87 | +1 | 17:30 |
| kgriffs | something like that | 17:30 |
| kgriffs | I always expected 1.0 to be a bit messy | 17:31 |
| kgriffs | (it always is) | 17:31 |
| kgriffs | ok, let's run with that | 17:31 |
| zyuan | sry, did not follow | 17:31 |
| zyuan | i see. np | 17:33 |
| kgriffs | zyuan: basically we are just formalizing what you are doing already | 17:33 |
| kgriffs | we remove the per-message size option from config | 17:34 |
| zyuan | no problem. i read conversation | 17:34 |
| kgriffs | and just check content-length header for max overall size | 17:34 |
| kgriffs | kewl | 17:34 |
| zyuan | on this https://bugs.launchpad.net/marconi/+bug/1251693 | 17:34 |
| flaper87 | kk, gtg guys | 17:34 |
| kgriffs | that will essentially "fix" those two bugs, no? | 17:34 |
| flaper87 | ttyl | 17:34 |
| kgriffs | flaper87: cheers | 17:34 |
| zyuan | i think we need a tiny patch to choose from json and simplejson | 17:34 |
| kgriffs | flaper87: one thing to ponder - in the client doing a seamless integration with swift | 17:35 |
| kgriffs | (for large payloads) | 17:35 |
| alcabrera | flaper87: take care! :) | 17:36 |
| *** flaper87 is now known as flaper87|afk | 17:36 | |
| kgriffs | zyuan: gochya. I think we will be fine getting rid of simplejson | 17:36 |
| kgriffs | it should balance out performance-wise, since one is faster serializing while the other is faster deserializing | 17:36 |
| kgriffs | and, like you said, we get the side benefit of pypy goodness | 17:37 |
| kgriffs | alcabrera: any objections? | 17:37 |
| zyuan | i mean, py26 still need simplejson | 17:37 |
| zyuan | and, it's just good for now, always good for pypy | 17:38 |
| alcabrera | I'm fine with simplejson being eliminated, all the moreso since python 3.3+ and pypy exhibit faster json behavior in all cases. | 17:38 |
| zyuan | ...then why we still spport py26? | 17:38 |
| alcabrera | zyuan: legacy systems. D: (also, openstack still supports 2.6) | 17:39 |
| zyuan | then why decrease its performance intentionally? | 17:39 |
| alcabrera | hmmm... | 17:39 |
| kgriffs | zyuan: why does py26 need it? | 17:39 |
| zyuan | because pure python json is slow in py26 | 17:39 |
| zyuan | 10x slow | 17:40 |
| kgriffs | oh well | 17:40 |
| kgriffs | we should just tell people to use python 2.7 | 17:40 |
| zyuan | :) | 17:40 |
| kgriffs | it is faster in other ways besides json | 17:40 |
| zyuan | you know, YaoMing face | 17:40 |
| kgriffs | if people complain it is slow, we just say "deploy on py27) | 17:40 |
| alcabrera | kgriffs: I was thinking that, too, and felt kind of mean, but kind of practical, too. :P | 17:40 |
| alcabrera | Moving Python forward. ;) | 17:41 |
| kgriffs | I think Rackspace is currently deployed on py26? | 17:41 |
| zyuan | no comment. agree with silence | 17:41 |
| zyuan | kgriffs: i hope they don't? | 17:41 |
| *** cindy_ is now known as cpallares | 17:41 | |
| alcabrera | kgriffs: I think that's correct. I'll check with mpanetta and oz_akan_. | 17:41 |
| kgriffs | I don't mind being 2.6 *compatible* but saying that it won't give you the best experience | 17:41 |
| alcabrera | +1 | 17:42 |
| kgriffs | alcabrera: kk. We really need to upgrade. If it is because of RHEL, they should try docker | 17:42 |
| kgriffs | ok | 17:42 |
| kgriffs | who wants to remove simplejson then? | 17:42 |
| kgriffs | (also from requirements.txt) | 17:42 |
| kgriffs | malini: we will need to watch out for performance regressions - not a big deal with 2.6 but we shouldn't see any for 2.7 | 17:43 |
| kgriffs | and indeed, things should be faster with pypy | 17:43 |
| alcabrera | I'm fine with dropping simplejson | 17:44 |
| alcabrera | It would require two changes in the code, namely, 'import simplejson as json' -> 'import json' | 17:44 |
| kgriffs | ok, so you or zyuan want to do it? | 17:45 |
| kgriffs | (do the work) | 17:45 |
| alcabrera | I'll do it. | 17:46 |
| alcabrera | Patch incoming in 10 min. :D | 17:46 |
| kgriffs | thanks! | 17:46 |
| alcabrera | kgriffs: do we have a bug open for this? | 17:48 |
| alcabrera | just double-checking, since I've been multiplexing eom:bastion, book shopping, and benchmarking while keeping up with the simplejson discussion. ;) | 17:48 |
| openstackgerrit | Alejandro Cabrera proposed a change to openstack/marconi: refactor: drop simplejson requirement https://review.openstack.org/56672 | 17:51 |
| kgriffs | alcabrera: mmm, there isn't a bug for just the json change | 17:51 |
| alcabrera | kk | 17:51 |
| alcabrera | The patch is now available. | 17:52 |
| *** flwang has quit IRC | 18:01 | |
| *** whenry has quit IRC | 18:12 | |
| *** whenry has joined #openstack-marconi | 18:46 | |
| *** reed has quit IRC | 18:55 | |
| *** whenry has quit IRC | 19:02 | |
| *** jergerber has joined #openstack-marconi | 19:17 | |
| *** jburkhar1 is now known as jburkhart | 19:26 | |
| *** oz_akan_ has joined #openstack-marconi | 19:34 | |
| *** oz_akan_ has quit IRC | 19:34 | |
| *** tedross has quit IRC | 19:35 | |
| *** oz_akan_ has joined #openstack-marconi | 19:35 | |
| *** tedross has joined #openstack-marconi | 19:54 | |
| *** kgriffs is now known as kgriffs_afk | 20:19 | |
| *** malini is now known as malini_afk | 20:27 | |
| russell_h | are there any docs/examples on how to use python-marconiclient? | 20:40 |
| *** mpanetta has joined #openstack-marconi | 20:41 | |
| *** mpanetta has quit IRC | 20:42 | |
| *** mpanetta has joined #openstack-marconi | 20:43 | |
| alcabrera | russell_h: not yet. flaper87|afk has been hard at work making the implementation work. :) | 20:44 |
| alcabrera | We've got a vision doc of sorts. | 20:44 |
| alcabrera | https://wiki.openstack.org/wiki/Marconi/PythonClient (might be a little dated) | 20:44 |
| russell_h | yeah, I found that, seems pretty reasonable, I was mostly just trying to figure out how to properly construct a client | 20:45 |
| russell_h | the API doesn't appear to match that wiki page exactly right now, but looks pretty workable | 20:45 |
| alcabrera | I'm giving it a look. It's been awhile since I've had a chance to play with the client. | 20:49 |
| alcabrera | russell_h: based on what I'm reading, you bring in a 'marconiclient.queues.v1.client.Client', pass it an instance of oslo.config.cfg.ConfigOpts (or oslo.config.cfg.CONF), and then... | 20:55 |
| *** jamieh__ has quit IRC | 20:56 | |
| alcabrera | That config either indicates which config file to use, and contains 'os_queues_url' under [DEFAULT], or pass in the url to the Client(url=localhost:8000) | 20:56 |
| alcabrera | That should do it. | 20:56 |
| alcabrera | From there, you can do 'c = Client(...)', 'c.queue(<queue_name>)' to inspect the functionality of the client. | 20:57 |
| russell_h | ah, I didn't realize I could pass a url directly | 20:59 |
| russell_h | actually, unless there's some serious magic, that isn't implemented | 21:00 |
| russell_h | https://github.com/openstack/python-marconiclient/blob/master/marconiclient/queues/v1/client.py#L38 | 21:01 |
| russell_h | its set, just never used | 21:01 |
| alcabrera | hmmm | 21:01 |
| alcabrera | good point | 21:02 |
| alcabrera | I'm out. Have a good weekend, everyone! | 21:08 |
| *** alcabrera has quit IRC | 21:09 | |
| *** jamieh__ has joined #openstack-marconi | 21:11 | |
| *** jamieh__ has quit IRC | 21:35 | |
| *** jamieh__ has joined #openstack-marconi | 21:43 | |
| *** reed has joined #openstack-marconi | 21:53 | |
| *** whenry has joined #openstack-marconi | 21:54 | |
| *** drumkilla has quit IRC | 22:19 | |
| *** russellb has joined #openstack-marconi | 22:20 | |
| *** kgriffs_afk is now known as kgriffs | 22:21 | |
| *** amitgandhi has quit IRC | 22:23 | |
| *** amitgandhi has joined #openstack-marconi | 22:24 | |
| *** cpallares has quit IRC | 22:26 | |
| *** amitgandhi has quit IRC | 22:29 | |
| *** tedross has quit IRC | 22:39 | |
| openstackgerrit | A change was merged to openstack/marconi: Update openstack/common/lockutils https://review.openstack.org/56353 | 23:20 |
| *** jcru has quit IRC | 23:20 | |
| *** amitgandhi has joined #openstack-marconi | 23:24 | |
| *** kgriffs is now known as kgriffs_afk | 23:27 | |
| *** amitgandhi has quit IRC | 23:31 | |
| *** amitgandhi has joined #openstack-marconi | 23:34 | |
| *** malini_afk is now known as malini | 23:37 | |
| *** amitgandhi has quit IRC | 23:39 | |
| *** malini is now known as malini_afk | 23:57 | |
| *** vkmc has quit IRC | 23:58 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!