| *** kgriffs is now known as kgriffs_afk | 00:32 | |
| *** nosnos has joined #openstack-marconi | 00:36 | |
| *** oz_akan_ has joined #openstack-marconi | 00:41 | |
| *** amitgandhi has joined #openstack-marconi | 00:58 | |
| *** ayoung has joined #openstack-marconi | 01:13 | |
| *** amitgandhi has quit IRC | 01:38 | |
| *** ayoung is now known as admiyo | 02:16 | |
| openstackgerrit | Oz Akan proposed a change to stackforge/marconi: Adds support for multiple databases for mongodb https://review.openstack.org/45952 | 03:14 |
|---|---|---|
| openstackgerrit | Oz Akan proposed a change to stackforge/marconi: Adds support for multiple databases for mongodb https://review.openstack.org/45952 | 03:27 |
| *** kgriffs_afk is now known as kgriffs | 03:30 | |
| openstackgerrit | Oz Akan proposed a change to stackforge/marconi: Adds support for multiple databases for mongodb https://review.openstack.org/45952 | 03:39 |
| *** oz_akan_ has quit IRC | 03:42 | |
| *** oz_akan_ has joined #openstack-marconi | 03:43 | |
| *** oz_akan_ has quit IRC | 03:47 | |
| *** kgriffs is now known as kgriffs_afk | 04:22 | |
| *** ykaplan has joined #openstack-marconi | 06:47 | |
| *** flaper87|afk is now known as flaper87 | 07:29 | |
| openstackgerrit | Flavio Percoco proposed a change to stackforge/marconi: Move Unit tests under a unit package https://review.openstack.org/45046 | 08:51 |
| *** ykaplan has quit IRC | 09:29 | |
| *** ykaplan has joined #openstack-marconi | 09:35 | |
| openstackgerrit | A change was merged to stackforge/marconi: Move Unit tests under a unit package https://review.openstack.org/45046 | 10:15 |
| *** ykaplan_ has joined #openstack-marconi | 10:32 | |
| *** ykaplan__ has joined #openstack-marconi | 10:33 | |
| *** ykaplan has quit IRC | 10:36 | |
| *** ykaplan_ has quit IRC | 10:37 | |
| *** vkmc has joined #openstack-marconi | 12:02 | |
| *** tedross has joined #openstack-marconi | 12:04 | |
| *** oz_akan_ has joined #openstack-marconi | 12:18 | |
| *** oz_akan_ has quit IRC | 12:22 | |
| *** key4 has quit IRC | 12:45 | |
| *** key4 has joined #openstack-marconi | 12:46 | |
| *** ykaplan_ has joined #openstack-marconi | 12:47 | |
| *** ykaplan__ has quit IRC | 12:51 | |
| *** cthulhup has quit IRC | 12:53 | |
| *** oz_akan_ has joined #openstack-marconi | 12:56 | |
| *** oz_akan_ has joined #openstack-marconi | 12:57 | |
| *** amitgandhi has joined #openstack-marconi | 13:12 | |
| oz_akan_ | hi team | 13:23 |
| zyuan_ | ? | 13:23 |
| oz_akan_ | zyuan_: just said hi, I don't want anything :) | 13:23 |
| zyuan_ | ... morning | 13:23 |
| flaper87 | oz_akan_: zyuan_ yo! Good morning | 13:24 |
| oz_akan_ | good afternoon flaper87 | 13:24 |
| *** malini_afk is now known as malini | 13:31 | |
| *** nosnos has quit IRC | 13:38 | |
| *** kgriffs_afk is now known as kgriffs | 13:51 | |
| *** malini is now known as malini_afk | 13:53 | |
| oz_akan_ | who put all these spaces there.. jenkins is mad about my patch | 13:54 |
| oz_akan_ | :) | 13:54 |
| *** malini_afk is now known as malini | 13:55 | |
| kgriffs | oz_akan_: looks like mmh3 is not in the deps file | 14:00 |
| oz_akan_ | right I will add it there | 14:00 |
| kgriffs | requirements.txt | 14:01 |
| kgriffs | thanks! | 14:01 |
| oz_akan_ | pylru too | 14:01 |
| kgriffs | ah, yes | 14:01 |
| oz_akan_ | it seems pylru is slower than the one I had used earlier | 14:01 |
| kgriffs | oh yeah? | 14:01 |
| kgriffs | by how much? | 14:02 |
| oz_akan_ | I will submit with pylru later after some benchmark, we may change if we think it is still slow | 14:02 |
| kgriffs | oic | 14:02 |
| kgriffs | it isn't in the patch yet | 14:02 |
| * kgriffs looks | 14:02 | |
| oz_akan_ | patch 4 must have it | 14:02 |
| oz_akan_ | https://review.openstack.org/#/c/45952/ | 14:02 |
| oz_akan_ | kgriffs: quick question | 14:05 |
| oz_akan_ | I have this : message = list(self._col[db_id].find(query).limit(1).hint([('_id', 1)])) | 14:05 |
| oz_akan_ | and pep8 says it is too long | 14:05 |
| oz_akan_ | line too long (80 > 79 characters) | 14:05 |
| oz_akan_ | what shall I do? | 14:05 |
| * kgriffs looking | 14:06 | |
| oz_akan_ | there are no spaces nothing, that I can split it | 14:07 |
| oz_akan_ | I could make something like col = self._col[db_id] and then col.find ... | 14:07 |
| oz_akan_ | but it seems kind of silly | 14:07 |
| zyuan_ | split it into two lines | 14:09 |
| zyuan_ | split at '(' | 14:09 |
| kgriffs | I would do collection = self._col[db_id] on the previous line. HACKING actually encourages breaking up statement compositions anyway, for readability | 14:09 |
| kgriffs | you can also do hint as a separate step to make the line even shorter | 14:09 |
| kgriffs | cursor = foo | 14:09 |
| kgriffs | cursor = foo.hint(…) | 14:09 |
| kgriffs | sorry | 14:09 |
| kgriffs | cursor = cursor.hint(...) | 14:09 |
| zyuan_ | i don't overwrite variable unless i have to... | 14:10 |
| kgriffs | i don't mind that style myself, as long as it is done on immediately successive lines | 14:11 |
| kgriffs | and it isn't overdone | 14:11 |
| kgriffs | (more than 2 or three statements in a row starts to look ugly) | 14:12 |
| oz_akan_ | kgriffs: zyuan_ thanks | 14:14 |
| openstackgerrit | Oz Akan proposed a change to stackforge/marconi: Adds support for multiple databases for mongodb https://review.openstack.org/45952 | 14:21 |
| oz_akan_ | lets see what jenkins is going to say | 14:23 |
| kgriffs | oz_akan_: just for fun, I converted your lru algorithm to use a closure | 14:34 |
| kgriffs | def make_lru(user_function, maxsize=1024): | 14:34 |
| kgriffs | # Link layout: [PREV, NEXT, KEY, RESULT] | 14:34 |
| kgriffs | root = [None, None, None, None] | 14:34 |
| kgriffs | ctx = {'root': root} | 14:34 |
| kgriffs | user_function = user_function | 14:34 |
| kgriffs | cache = {} | 14:34 |
| kgriffs | last = root | 14:34 |
| kgriffs | for i in range(maxsize): | 14:34 |
| kgriffs | key = object() | 14:34 |
| kgriffs | cache[key] = last[1] = last = [last, root, key, None] | 14:34 |
| kgriffs | root[0] = last | 14:34 |
| kgriffs | def lookup(*key): | 14:34 |
| kgriffs | root = ctx['root'] | 14:34 |
| kgriffs | link = cache.get(key) | 14:34 |
| kgriffs | if link is not None: | 14:34 |
| kgriffs | link_prev, link_next, _, result = link | 14:34 |
| kgriffs | link_prev[1] = link_next | 14:34 |
| kgriffs | link_next[0] = link_prev | 14:34 |
| kgriffs | last = root[0] | 14:34 |
| kgriffs | last[1] = root[0] = link | 14:34 |
| kgriffs | link[0] = last | 14:34 |
| kgriffs | link[1] = root | 14:34 |
| kgriffs | return result | 14:34 |
| kgriffs | result = user_function(*key) | 14:34 |
| kgriffs | root[2] = key | 14:34 |
| kgriffs | root[3] = result | 14:34 |
| kgriffs | oldroot = root | 14:34 |
| kgriffs | root = ctx['root'] = root[1] | 14:34 |
| kgriffs | root[2], oldkey = None, root[2] | 14:34 |
| kgriffs | root[3], oldvalue = None, root[3] | 14:34 |
| kgriffs | del cache[oldkey] | 14:35 |
| kgriffs | cache[key] = oldroot | 14:35 |
| kgriffs | return result | 14:35 |
| kgriffs | return lookup | 14:35 |
| kgriffs | oops | 14:35 |
| kgriffs | meant to paste this: http://paste.openstack.org/show/46657/ | 14:35 |
| kgriffs | :p | 14:35 |
| oz_akan_ | hehe | 14:35 |
| oz_akan_ | it came really slow, as if you are typinh | 14:35 |
| kgriffs | on my MBP it yielded 20% speedup | 14:35 |
| oz_akan_ | versus pylru ? | 14:35 |
| kgriffs | http://paste.openstack.org/show/46658/ | 14:36 |
| oz_akan_ | or vs not having closure? | 14:36 |
| oz_akan_ | wow | 14:36 |
| kgriffs | same test using pylru is 2.25 µs per loop | 14:36 |
| kgriffs | btw, any particular reason that you accept a variable list of key(s) instead of just a single key? | 14:38 |
| oz_akan_ | so we have 1.65 vs 2.25 | 14:38 |
| kgriffs | *key | 14:38 |
| oz_akan_ | I didn't really write that code, I found online | 14:39 |
| kgriffs | ah, ok | 14:39 |
| kgriffs | let me see if I can make it is even better. ;) | 14:39 |
| oz_akan_ | :) at this point making it faster magic to me | 14:40 |
| *** cppcabrera has joined #openstack-marconi | 14:41 | |
| cppcabrera | Good morning! | 14:41 |
| kgriffs | o/ | 14:42 |
| oz_akan_ | cppcabrera: moving here | 14:46 |
| oz_akan_ | http://pep8online.com/s/yypKkZlq | 14:46 |
| oz_akan_ | I can't make it correcr | 14:46 |
| cppcabrera | I heard some pretty cool news. eventlet is looking to support python 3.3 soon: https://github.com/eventlet/eventlet/issues/6 (kgriffs, flaper87) | 14:48 |
| oz_akan_ | https://bugs.launchpad.net/marconi/+bug/1223916 about limiting number of queues per project | 14:52 |
| kgriffs | cppcabrera: oh, interesting. looks like somebody has a vested interest in making that happen! | 14:54 |
| openstackgerrit | Oz Akan proposed a change to stackforge/marconi: Adds support for multiple databases for mongodb https://review.openstack.org/45952 | 14:54 |
| kgriffs | oz_akan_: there may already be a bp for that (basically, quotas) | 14:55 |
| kgriffs | https://blueprints.launchpad.net/marconi/+spec/quotas | 14:55 |
| oz_akan_ | let me see | 14:55 |
| oz_akan_ | got it, couldn't see it before | 14:55 |
| oz_akan_ | let me delete then what I created | 14:56 |
| oz_akan_ | yay ! malini thanks pep8 is all good now | 14:58 |
| *** flaper87 is now known as flaper87|afk | 14:58 | |
| cppcabrera | awesome, oz_akan. :) | 14:58 |
| malini | yayy!! | 14:58 |
| *** ykaplan_ has quit IRC | 14:59 | |
| malini | autopep8 shud be part of jenkins ;-) | 14:59 |
| oz_akan_ | yes :) | 14:59 |
| oz_akan_ | a few long line and then jenkins will start to love the code | 15:00 |
| oz_akan_ | I am so close | 15:00 |
| kgriffs | cppcabrera: re the cons of using replication vs. caching | 15:00 |
| kgriffs | "The other half of the problem with this approach is that there's no longer a notion of a "cache miss" for slaves - just stale data." | 15:02 |
| kgriffs | i have an idea for addressing that | 15:02 |
| kgriffs | let me bring up an etherpad | 15:02 |
| openstackgerrit | Oz Akan proposed a change to stackforge/marconi: Adds support for multiple databases for mongodb https://review.openstack.org/45952 | 15:05 |
| cppcabrera | thanks for bringing that up again, kgriffs. Etherpad sounds good to me: +1 | 15:06 |
| *** gordonsim has joined #openstack-marconi | 15:07 | |
| *** flaper87|afk is now known as flaper87 | 15:08 | |
| kgriffs | https://etherpad.openstack.org/queuing-scratch | 15:08 |
| oz_akan_ | sometimes I think if cppcabrera has a very clever auto responder :) | 15:08 |
| * flaper87 is back | 15:09 | |
| kgriffs | addressing "it isn't really cache" that is correct. We are using replication in lieu of cache, not dissimilar to what swift does by replicating the hash rings | 15:09 |
| flaper87 | cppcabrera: wow, eventlet Py3k :D | 15:09 |
| kgriffs | if we use mongodb then shouldn't be an issue with memory usage. Redis would impose a max catalog size, obviously | 15:09 |
| *** ykaplan_ has joined #openstack-marconi | 15:10 | |
| oz_akan_ | if we use redis as master and then replicate date to slaves, we will also have to map data to disk, because we will rely on redis only | 15:11 |
| flaper87 | are we talking about marconi-proxy ? | 15:11 |
| oz_akan_ | and we will have to ensure there are 3 copies healthy all the time | 15:11 |
| oz_akan_ | so redis won't be pure in memory cache anymore | 15:12 |
| oz_akan_ | one concern is the behavior when we have 10-20 proxies | 15:12 |
| kgriffs | btw guys, stuff like pylru is MUCH faster under pypy | 15:13 |
| oz_akan_ | in case of a massive reboot, it will take a while to replicate data to all slaves | 15:13 |
| oz_akan_ | flaper87: yes | 15:13 |
| kgriffs | it basically means you don't have to try as hard to optimize | 15:13 |
| kgriffs | that's why i REALLY want us to benchmark pypy with gunicorn | 15:13 |
| oz_akan_ | pypy won't work with uwsgi? | 15:14 |
| kgriffs | mmm, good question. | 15:14 |
| kgriffs | i can't remember | 15:14 |
| * kgriffs knows he used to know | 15:14 | |
| oz_akan_ | http://uwsgi-docs.readthedocs.org/en/latest/PyPy.html | 15:14 |
| kgriffs | yay for Alex! | 15:14 |
| oz_akan_ | http://uwsgi-docs.readthedocs.org/en/latest/PyPy_benchmarks.html | 15:14 |
| kgriffs | sweeeeeeeet | 15:14 |
| kgriffs | we REALLY need to try that! | 15:14 |
| cppcabrera | however... (http://morepypy.blogspot.com/2013/03/so-you-want-to-try-pypy.html) | 15:15 |
| cppcabrera | This pypy blog recommends using gunicorn + pypy over uwsgi + pypy | 15:15 |
| flaper87 | can I say something about the proxy ? | 15:15 |
| flaper87 | :D | 15:15 |
| flaper87 | :D | 15:15 |
| flaper87 | :D | 15:15 |
| flaper87 | o/ | 15:15 |
| kgriffs | the uwsgi patch came in more recently than that article | 15:15 |
| cppcabrera | no, flaper87. :P | 15:15 |
| flaper87 | o/ | 15:15 |
| flaper87 | o/ | 15:15 |
| flaper87 | o/ | 15:15 |
| cppcabrera | lol | 15:15 |
| flaper87 | PLEAAAAAASEEEEEE | 15:15 |
| flaper87 | :( | 15:15 |
| cppcabrera | go on~ :D | 15:15 |
| flaper87 | w000t | 15:15 |
| flaper87 | so, I was thinking this morning that we could structure Marconis tree similar to swifts | 15:16 |
| flaper87 | and have: proxy, queues, notification | 15:16 |
| flaper87 | right under marconi/marconi/ | 15:16 |
| flaper87 | and, I was thinking about kgriffs proposal | 15:16 |
| flaper87 | and I think it does make sense to support multiple storage for the proxy as well | 15:17 |
| flaper87 | ok, I said it! No, feel free to insult me :D | 15:17 |
| kgriffs | you mean, organize the tree by resource rather than driver type? | 15:19 |
| flaper87 | https://etherpad.openstack.org/marconi-tree | 15:20 |
| oz_akan_ | my fear is that if we focus on trying to support multiple of everything, we will end up with something that doesn't work good enough for anything | 15:20 |
| oz_akan_ | proxy storage, has to be very stable and performant enough. that is the requirement | 15:21 |
| zyuan_ | memcache.py is not py3 compatible... heheh | 15:22 |
| flaper87 | oz_akan_: I agree | 15:22 |
| flaper87 | oz_akan_: my thinking is that we should work on a first version of this proxy and then add support for other things | 15:23 |
| kgriffs | i still think we just do one, and let 3rd-parties maintain their own drivers | 15:23 |
| kgriffs | but that means we need to support stevedore drivers | 15:23 |
| flaper87 | kgriffs: +1 | 15:23 |
| kgriffs | that forces good interface-oriented design anyway | 15:23 |
| oz_akan_ | I see | 15:23 |
| cppcabrera | that works for me - one official version + stevedore. | 15:24 |
| kgriffs | caveat: if we only support one backend officially, it can't be AGPL | 15:24 |
| oz_akan_ | +1 then, we are on the same page for sure | 15:24 |
| flaper87 | TBH, I think it could be an sqlalchemy backend | 15:24 |
| flaper87 | I mean, storage | 15:24 |
| kgriffs | if somebody comes up with an alternative that is insanely great, we can consider pulling it later | 15:24 |
| flaper87 | that would give enough choices to people using the proxy | 15:24 |
| flaper87 | from sqlite and mysql to psql | 15:25 |
| openstackgerrit | Oz Akan proposed a change to stackforge/marconi: Adds support for multiple databases for mongodb https://review.openstack.org/45952 | 15:25 |
| cppcabrera | I'd consider writing an sqlalchemy driver for the proxy as a first, except that I have very little experience actually working with SQLs. :P | 15:26 |
| kgriffs | flaper87: only thing with that is we will likely need in-process lru since sql queries are going to be way slower than, say, redis | 15:26 |
| oz_akan_ | you know what we should use mysql as nosql | 15:26 |
| oz_akan_ | if I am not wrong there is a driver that that doesn't uses sql and it is faster than anything out there | 15:26 |
| oz_akan_ | then, why would someone think about using another sql backend | 15:27 |
| flaper87 | kgriffs: mmh, good point | 15:27 |
| oz_akan_ | if nothing can beat the performance | 15:27 |
| flaper87 | kgriffs: what about using an in-memory instance of sqlalchemy for lru ? | 15:27 |
| flaper87 | ah wait | 15:27 |
| flaper87 | that's stupid | 15:27 |
| flaper87 | nevermind | 15:27 |
| flaper87 | T_T | 15:27 |
| * flaper87 facepalm | 15:27 | |
| kgriffs | :p | 15:27 |
| oz_akan_ | for proxy I want mongo for sure | 15:27 |
| kgriffs | oz_akan_: if it is comparable to redis, then sounds good | 15:27 |
| oz_akan_ | 3 copies of data, same storage with marconi data | 15:28 |
| flaper87 | oz_akan_: mongo is AGPL | 15:28 |
| oz_akan_ | same skill set to manage | 15:28 |
| oz_akan_ | like kgriffs mongo + sqlalchemy | 15:28 |
| kgriffs | if we do mongo, we will need to maintain 2 drivers as we do with the server storage | 15:28 |
| flaper87 | cppcabrera: did you take a look here? https://etherpad.openstack.org/marconi-tree | 15:29 |
| kgriffs | srsly these drivers will be very simple | 15:29 |
| flaper87 | kgriffs: +1 | 15:29 |
| cppcabrera | yup, flaper87. | 15:29 |
| kgriffs | flaper87: oh, so you are just dropping current stuff down under "queue" | 15:29 |
| oz_akan_ | operationally it is not desired to have two different database technology | 15:29 |
| kgriffs | oz_akan_: good point | 15:29 |
| flaper87 | kgriffs: does that make sense? :P | 15:30 |
| cppcabrera | hmm... | 15:30 |
| flaper87 | kgriffs: s/queue/queuing | 15:30 |
| flaper87 | or another name | 15:30 |
| oz_akan_ | https://review.openstack.org/#/c/45952/ finally passed jenkins test, please review when you have time. I need to get rid of this asap | 15:30 |
| kgriffs | i vote for either just redis or mongo + sqlalchemy | 15:30 |
| * flaper87 is pretty bad for names | 15:30 | |
| kgriffs | (just redis because you probably have that anyway; keystone middleware speaks memcached and so does redis, so you don't need memcached as well) | 15:30 |
| kgriffs | it's all sort of queueing related | 15:31 |
| kgriffs | flaper87: let me think about a name there | 15:31 |
| oz_akan_ | I think redis is not stable enough to hold critical mapping information. there the more traditional it is the safer probably | 15:31 |
| flaper87 | kgriffs: what do you think about the structure in general ? | 15:31 |
| oz_akan_ | I mean to store permanently | 15:31 |
| flaper87 | I'd say mongo + sqlalchemy | 15:32 |
| kgriffs | flaper87: i think it is good. | 15:32 |
| kgriffs | marconi-proxy | 15:32 |
| kgriffs | marconi-server (?) | 15:32 |
| kgriffs | and marconi-notifications | 15:32 |
| flaper87 | marconi-queuing | 15:32 |
| flaper87 | :D | 15:32 |
| kgriffs | that last one I guess will just be some set of workers and an API for controlling them | 15:32 |
| cppcabrera | marconi/marconi/{server,proxy,notifications}, I'm gathering. | 15:33 |
| flaper87 | cppcabrera: I'd be ok with that but I think it'll create some confusion | 15:33 |
| flaper87 | like: "What is server?" | 15:33 |
| cppcabrera | hmm... that'd be our current marconi. | 15:34 |
| flaper87 | confusion in terms of, lack of consistency | 15:34 |
| cppcabrera | gotcha - server is pretty ambiguous. | 15:34 |
| flaper87 | yup | 15:34 |
| cppcabrera | even the proxy is a type of server | 15:34 |
| cppcabrera | server -> message_bus|queues | 15:34 |
| cppcabrera | on the topic of proxy storage, I've gathered the following so far: 1) support multiple storage backends (implies stevedore), 2) start with mongo, 3) support sqlalchemy later, 4a) use slave/master replication 4)b use hierarchical caching | 15:35 |
| cppcabrera | *4b) | 15:35 |
| flaper87 | cppcabrera: I'd sawp 2 and 3 | 15:36 |
| * flaper87 is thinking about graduation | 15:36 | |
| kgriffs | only think is, nobody knows how to spell queueing | 15:37 |
| kgriffs | :p | 15:37 |
| flaper87 | LOL | 15:37 |
| kgriffs | I still vote for 4a since it is the fastest option, and if using disk-backed store, the non-hot queues will get paged to disk and most stuff can stay in RAM | 15:38 |
| cppcabrera | 4a with a disk-backed option will be speedy, no doubt, since the queries will be of a key-value nature. | 15:39 |
| kgriffs | i say just leave it up to the catalog driver (cache or whatever) | 15:40 |
| flaper87 | kgriffs: +! | 15:40 |
| flaper87 | kgriffs: +1 | 15:40 |
| flaper87 | up to the catalog driver | 15:40 |
| kgriffs | the mongo one can do replication since that is super nice and easy | 15:40 |
| kgriffs | mysql read-only slaves are a bit of a pain, but doable | 15:41 |
| kgriffs | we can punt on that, but we will need SQLAlchemy in order to graduate | 15:41 |
| kgriffs | (AGPL thing) | 15:41 |
| zyuan_ | flaper87: nosetests became totally unusable; what happened? | 15:42 |
| flaper87 | zyuan_: o.0 | 15:42 |
| flaper87 | why ? | 15:42 |
| flaper87 | :( | 15:42 |
| flaper87 | what's the error? | 15:42 |
| zyuan_ | it just stuck on the first test | 15:42 |
| flaper87 | (btw, make sure you remove all .pyc) | 15:42 |
| zyuan_ | did | 15:42 |
| flaper87 | mmh :( | 15:42 |
| flaper87 | no error ? | 15:42 |
| cppcabrera | I wonder if they'restuck on the server start up... | 15:42 |
| zyuan_ | it just stuck. no progress, can't even break into the test function | 15:43 |
| zyuan_ | which server? | 15:43 |
| cppcabrera | kgriffs, flaper87: I can take the 4a route, but I'll have to read up on slave/master replication to understand what that means for the driver implementation (caveats, e.g.) and what it means for scaling (likely eventual consistency, which is fine). | 15:43 |
| flaper87 | zyuan_: tox -epy27 doesn't work ? | 15:43 |
| flaper87 | zyuan_: I just ran that, it worked perfectly | 15:44 |
| zyuan_ | tox works | 15:44 |
| flaper87 | zyuan_: what command are you running ? | 15:44 |
| zyuan_ | nosetests | 15:44 |
| flaper87 | nosetests tests | 15:45 |
| flaper87 | that works for me | 15:45 |
| flaper87 | Ran 350 tests in 8.345s | 15:45 |
| flaper87 | OK (SKIP=162) | 15:45 |
| zyuan_ | still stuck | 15:45 |
| flaper87 | holymoly, 350 tests already ? | 15:45 |
| cppcabrera | lol | 15:45 |
| zyuan_ | ddt | 15:45 |
| zyuan_ | there is no that much actually | 15:46 |
| flaper87 | zyuan_: man, don't break my wings just like that | 15:46 |
| flaper87 | :D | 15:46 |
| flaper87 | zyuan_: seriously, not sure why it gets stuck in your box | 15:46 |
| flaper87 | cppcabrera: can you try running those tests ? | 15:46 |
| flaper87 | plus, why does tox work and nosetests doesn;t ? | 15:47 |
| flaper87 | mmh | 15:47 |
| kgriffs | zyuan_: are you running on BSD or Linux? | 15:48 |
| zyuan_ | i don't even know where it stucked... | 15:48 |
| zyuan_ | freebsd | 15:48 |
| zyuan_ | i don't think we used any non-portable stuff... | 15:49 |
| cppcabrera | I'm checking it out, flaper87. :) | 15:49 |
| cppcabrera | nosetests-style | 15:49 |
| cppcabrera | from a fresh venv | 15:49 |
| zyuan_ | wait... memcache? | 15:49 |
| flaper87 | memcache ? | 15:50 |
| flaper87 | brb, meeting | 15:52 |
| zyuan_ | do i need to run memcached? | 15:52 |
| *** ykaplan_ has quit IRC | 15:52 | |
| flaper87 | zyuan_: no | 15:52 |
| flaper87 | is it requesting memcache? | 15:52 |
| flaper87 | where? | 15:52 |
| zyuan_ | no | 15:52 |
| flaper87 | mmhh | 15:52 |
| flaper87 | weird | 15:52 |
| flaper87 | ah ok | 15:52 |
| zyuan_ | i'm gussing | 15:52 |
| flaper87 | at least I didn't leave a pdb somewhere | 15:53 |
| flaper87 | :D | 15:53 |
| flaper87 | I hope | 15:53 |
| flaper87 | :D | 15:53 |
| cppcabrera | grin pdb -> None | 15:53 |
| cppcabrera | :) | 15:54 |
| flaper87 | hahaha | 15:54 |
| flaper87 | phew | 15:54 |
| flaper87 | cppcabrera: so, do they work for you? | 15:54 |
| cppcabrera | almost done getting reqs - things seem slow today. :/ | 15:54 |
| zyuan_ | trying a new repo... | 15:55 |
| zyuan_ | doesn't work. maybe my machine's problem | 15:57 |
| *** pycabrera has joined #openstack-marconi | 15:58 | |
| pycabrera | VPNs drive me crazy... | 15:58 |
| pycabrera | flaper87, zyuan: Tests pass. ^^ | 15:59 |
| *** zyuan_ has quit IRC | 15:59 | |
| flaper87 | pycabrera: cool | 15:59 |
| *** cppcabrera has quit IRC | 16:00 | |
| *** pycabrera is now known as cppcabrera | 16:00 | |
| kgriffs | pycabrera: is that your alternate nick? | 16:01 |
| cppcabrera | yup, kgriffs | 16:01 |
| cppcabrera | I decided to resort to an ethernet connection. Wireless VPN gets really silly @ the office. | 16:01 |
| malini | zyuan_ : are you running into 'Could not load 'sqlite': python-memcached' ? | 16:04 |
| cppcabrera | MARCONI_TESTS_CONFIGS_DIR=etc MONGODB_TEST_LIVE=1 nosetests unit -> ArgsAlreadyParsedError: arguments already parsed: cannot register CLI option (after 274 tests) | 16:06 |
| *** zyuan_ has joined #openstack-marconi | 16:07 | |
| zyuan_ | how can i run unit/storage tests only? | 16:07 |
| zyuan_ | i suspect it's only functional tests which have the problem | 16:08 |
| malini | grr..I hate it when folks whine about functional tests without looking at it :'( | 16:08 |
| malini | something else is going one | 16:09 |
| malini | on* | 16:09 |
| zyuan_ | i have no idea why the tests stuck at the first test under functional | 16:09 |
| oz_akan_ | malini: we need to run functional tests on https://review.openstack.org/#/c/45952/ | 16:09 |
| oz_akan_ | current test endpoint has it | 16:09 |
| oz_akan_ | would you be able to run the test today? | 16:09 |
| malini | oz_akan_ : I am trying to figure out how to run them after the refactoring..there is something funny going on | 16:10 |
| malini | flaper87: Can you try running the tests against a remote server | 16:11 |
| malini | I am running into issues there | 16:11 |
| zyuan_ | i'm running unit/transport test only, why i need MARCONI_TESTS_CONFIGS_DIR? | 16:12 |
| oz_akan_ | malini: ok, I am out for lunch | 16:13 |
| oz_akan_ | http://paste.openstack.org/show/46657/ waits for review | 16:14 |
| * flaper87 is half here | 16:15 | |
| flaper87 | malini: what issues ? | 16:15 |
| malini | flaper87: I am running into http://paste.openstack.org/show/46673/ | 16:16 |
| *** ykaplan_ has joined #openstack-marconi | 16:16 | |
| malini | I wud assume, it doesnt need to know abt the storage drivers, since I am testing a remote installation | 16:16 |
| *** ykaplan_ has quit IRC | 16:16 | |
| flaper87 | malini: just ran tests against a remote server, it worked here. Let me take a look at the paste | 16:16 |
| flaper87 | why is it requiring memcached ? | 16:17 |
| flaper87 | mmhh | 16:17 |
| kgriffs | something about keystone middleware? | 16:19 |
| flaper87 | mhhh | 16:19 |
| malini | flaper87: functional-marconi.conf doesn't need any changes (except for limits if remote server has different configs), rt ? | 16:20 |
| zyuan_ | confirmed, it's 226e813 which does not work for me | 16:21 |
| flaper87 | malini: right, plus the remote server info | 16:21 |
| malini | flaper87: Both my conf files arr in ~/.marconi with functional-tests.conf pointing to the remote server | 16:21 |
| flaper87 | try installing python-memcached manually | 16:21 |
| flaper87 | and see what happens | 16:21 |
| malini | ok | 16:21 |
| flaper87 | it should work, I guess | 16:21 |
| flaper87 | zyuan_: could it me related to multiprocessing ? | 16:23 |
| malini | looks like its trying to start a marconi server | 16:24 |
| * flaper87 is just guessing | 16:24 | |
| malini | '2013-09-11 12:23:18.015 5111 INFO marconi.transport.wsgi.driver [-] Serving on host 127.0.0.1:8888 | 16:24 |
| malini | ' | 16:24 |
| flaper87 | malini: set run_server=False | 16:24 |
| flaper87 | malini: in the functional-tests.conf | 16:24 |
| malini | ok | 16:25 |
| flaper87 | malini: did that work ? | 16:27 |
| * flaper87 is confused about that memcached thing | 16:27 | |
| malini | flaper87: http://paste.openstack.org/show/46677/ | 16:28 |
| flaper87 | malini: are you using latest code? | 16:28 |
| flaper87 | malini: there shouldn't be any http.post calls | 16:28 |
| flaper87 | I replaced that with a small http client | 16:28 |
| cppcabrera | that looks outdated, malini. "marconi/tests/functional/wsgi/v1/test_claims.py" no longer exists. | 16:29 |
| cppcabrera | nvm. >.> | 16:29 |
| cppcabrera | It does. | 16:29 |
| * cppcabrera is confused | 16:29 | |
| malini | defd75e1a5b3b5d7c0730151acc86fd18e3e36ae is the latest, rt? | 16:29 |
| flaper87 | cppcabrera: ah, that too | 16:29 |
| flaper87 | malini: remove all .pyc :D | 16:30 |
| flaper87 | malini: find . -name "*.pyc" -delete | 16:30 |
| flaper87 | malini: and yes, that's latest | 16:30 |
| cppcabrera | malini: I've confirmed the existence of 'http.post' in marconi.tests.functional.helpers | 16:34 |
| cppcabrera | That's a bug. | 16:34 |
| malini | https://github.com/stackforge/marconi/blob/master/marconi/tests/functional/helpers.py#L40 | 16:34 |
| malini | cppcabrera: you made it first :( | 16:34 |
| cppcabrera | :) | 16:34 |
| cppcabrera | that should be requests.post | 16:34 |
| malini | flaper87: you probably had auth turned off | 16:34 |
| malini | so never got it :) | 16:35 |
| cppcabrera | kgriffs, flaper87, oz_akan_: "A replica set can have up to 12 members, [1] but only 7 voting members. See non-voting members for more information." - mongo docs | 16:35 |
| cppcabrera | So if we take the slave/master approach, we're limited to 12 proxies, assuming a single master. | 16:35 |
| cppcabrera | Also assuming mongo. | 16:35 |
| kgriffs | ah, darn | 16:36 |
| kgriffs | well, how about redis then? | 16:36 |
| cppcabrera | checking. :) | 16:36 |
| flaper87 | malini: DAMNIT! I'M SO SORRY ABOUT THAT! | 16:37 |
| flaper87 | malini: I'll fix it | 16:37 |
| malini | flaper87: np :) | 16:37 |
| flaper87 | malini: btw, do you have a server ("out there") I can use with auth enabled? | 16:38 |
| flaper87 | I'll configure one, otherwise | 16:38 |
| malini | flaper87: we have one, but you'll need a RAX identity credentials for tht :( | 16:39 |
| flaper87 | malini: no worries, I'll set one up | 16:39 |
| zyuan_ | after process.start() is called, the process becomes 'stopped' | 16:40 |
| zyuan_ | can not break at server's run() method... | 16:41 |
| flaper87 | zyuan_: mmh | 16:44 |
| zyuan_ | confirmed, it stucked at DriverManager | 16:46 |
| zyuan_ | on marconi/marconi/bootstrap.py:60 | 16:46 |
| zyuan_ | stevedore wins | 16:47 |
| flaper87 | that's weird | 16:48 |
| flaper87 | but you know, computers | 16:48 |
| cppcabrera | There are no published limits on the number of slaves in a redis-backed slave/master scheme. (kgriffs, oz_akan_, flaper87) | 16:50 |
| zyuan_ | hope run_server=false can help (me to continue writing code...) | 16:50 |
| flaper87 | zyuan_: you can disable functional tests | 16:51 |
| malini | I can run the tests now <:o) | 16:51 |
| flaper87 | malini: what did you do? | 16:51 |
| flaper87 | zyuan_: re disable tests https://github.com/stackforge/marconi/blob/master/tests/etc/functional-tests.conf#L2 | 16:52 |
| malini | I just replaced the http.post, nothing major | 16:52 |
| flaper87 | malini: kk, I'll fix that bug! sorry about that, totally missed it! | 16:52 |
| flaper87 | cppcabrera: interesting | 16:52 |
| malini | flaper87: np..It was huge patch sets & you missed just one!! | 16:53 |
| malini | oz_akan_ : I am seeing some random 204s & 404s on test env - post messages & claim | 16:53 |
| cppcabrera | redis replication is kind of nifty. The master process is forked, the master:child dumps it's state to an RDB, while the master:parent continues serving requests and building a replication buffer. Once the master:child is done dumping the state, the master:parent starts relaying the dumped RDB to slaves. Once slaves are caught up, the master:parent starts sending data from the replication buffer. | 16:54 |
| cppcabrera | Reminds me a lot of the phased migration approach we had for proxy earlier. | 16:54 |
| flaper87 | zyuan_: but, I'd like to undestand why it is failing on freebsd, it'd be nice if you could dig more | 16:55 |
| cppcabrera | A consequence of the above is that there's additional memory overhead for state replication (repl. buffer). | 16:55 |
| kgriffs | cppcabrera: so, the concern is scale I guess | 16:57 |
| zyuan_ | not too bad, after disabled functional tests, i ran into the "normal" stevedore errors: No 'marconi.transport' driver found, looking for 'wsgi | 16:57 |
| kgriffs | 1000 slaves | 16:57 |
| kgriffs | if we assume queue creation is pretty infrequent | 16:57 |
| kgriffs | then, it shouldn't be a problem methinks | 16:57 |
| cppcabrera | agreed, kgriffs. | 16:58 |
| kgriffs | so, i guess we just hit the master for create/delete queue, and hit the slave for everything else | 16:59 |
| cppcabrera | yup | 16:59 |
| kgriffs | if we get a miss on the slave, check the master | 16:59 |
| kgriffs | if it isn't there either, 404? | 16:59 |
| cppcabrera | that should work | 16:59 |
| kgriffs | i guess that depends on the request. seems like some requests don't check queue's existence | 16:59 |
| kgriffs | like, claim | 16:59 |
| kgriffs | if you stick the proxy in there, then that would slightly change the expected error codes unless the proxy tracks what the server does | 17:00 |
| cppcabrera | the proxy would have to maintain API consistency with the server, which is unfortunate. | 17:01 |
| kgriffs | yeah | 17:01 |
| kgriffs | I guess our functional tests should catch mismatches? | 17:01 |
| cppcabrera | Yup, given a good suite of basic scenarios. | 17:01 |
| kgriffs | btw guys, can we get this merged? | 17:02 |
| kgriffs | https://review.openstack.org/#/c/44340/ | 17:02 |
| kgriffs | flaper87: ^^ | 17:02 |
| zyuan_ | damn! | 17:02 |
| zyuan_ | either me stupid or venv stupid | 17:02 |
| flaper87 | kgriffs: can I review that after dinner? | 17:02 |
| * flaper87 is starving | 17:02 | |
| kgriffs | cppcabrera: ok, can you make sure we have test coverage for that? I think we do. | 17:02 |
| cppcabrera | my last concern is HA w/ redis. I need to investigate the master election protocol, and the deployment process - how one would go about setting this up. | 17:02 |
| kgriffs | flaper87: no changes since you last looked, just rebased | 17:02 |
| cppcabrera | also, data redundancy concerns | 17:03 |
| zyuan_ | i forgot i have to setenv PYTHONPATH . to get stevedore work | 17:03 |
| cppcabrera | meaning - let;'s never lose all the data. :P | 17:03 |
| flaper87 | cppcabrera: I'll read the backlog in a bit and comment | 17:03 |
| kgriffs | cppcabrera: ah, as in how big these catalogs will get | 17:03 |
| flaper87 | kgriffs: kk | 17:03 |
| zyuan_ | problem solved (although i don't know the reason behind...) | 17:03 |
| cppcabrera | thanks: flaper87 | 17:03 |
| cppcabrera | kgriffs: That's another one. | 17:03 |
| flaper87 | then LGTM | 17:03 |
| cppcabrera | How much memory do we need? | 17:03 |
| cppcabrera | partitions should take up almost no space. | 17:03 |
| cppcabrera | It's the catalogue proper that will get bulky over time. | 17:04 |
| cppcabrera | Slowly, though. | 17:04 |
| cppcabrera | But surely. | 17:04 |
| kgriffs | cppcabrera: that patch, can you review? I don't think you saw it since you got back from jury duty | 17:04 |
| cppcabrera | I'll load it up now, kgriffs. :) | 17:04 |
| kgriffs | thanks! | 17:04 |
| kgriffs | flaper87: when you get back, as long as cppcabrera is cool with the patch, you can approve | 17:05 |
| flaper87 | kgriffs: https://review.openstack.org/#/c/44340/9/marconi/storage/mongodb/queues.py | 17:05 |
| zyuan_ | err, kgriffs 's patch conflicts with my utcnow_ts change... | 17:05 |
| flaper87 | kgriffs: active and total run the same query | 17:05 |
| flaper87 | ah no | 17:06 |
| flaper87 | include_claim | 17:06 |
| flaper87 | nevermind | 17:06 |
| * flaper87 asked the same question last time | 17:06 | |
| flaper87 | difference, this time I did it outloud | 17:06 |
| flaper87 | ok, brbr | 17:06 |
| cppcabrera | kgriffs: reviewed - typos strike again. | 17:10 |
| cppcabrera | -1 for typo, +2 for implementation | 17:10 |
| cppcabrera | brb | 17:13 |
| cppcabrera | back | 17:20 |
| openstackgerrit | Kurt Griffiths proposed a change to stackforge/marconi: fix: Requests get slower when queues have a lot of messages https://review.openstack.org/44340 | 17:39 |
| kgriffs | cppcabrera: ^^^ Fixed typo | 17:39 |
| cppcabrera | kgriffs: +2'd | 17:41 |
| cppcabrera | :) | 17:41 |
| kgriffs | thanks! | 17:46 |
| *** admiyo has quit IRC | 17:54 | |
| *** ayoung_ has joined #openstack-marconi | 17:54 | |
| oz_akan_ | malini: I saw your message on random 404s | 17:56 |
| malini | oz_akan: want me try again, so you can watch the logs? | 17:56 |
| *** ayoung_ is now known as ayoung | 17:59 | |
| oz_akan_ | malini: yes pls give me a few minutes | 17:59 |
| oz_akan_ | I am cleaning logs | 18:00 |
| zyuan_ | cppcabrera: on today's stand-up, can you tell the team i'm doing the unit tests stuff? i'm going to stay home this afternoon as well. | 18:01 |
| oz_akan_ | malini: I am ready, could you run? | 18:02 |
| malini | sure | 18:02 |
| cppcabrera | sure thing, zyuan_ | 18:03 |
| zyuan_ | oz_akan_: which is your email adress, ozgur.akan@ or oz.akan@ ? | 18:03 |
| oz_akan_ | oz.akan@ better | 18:03 |
| zyuan_ | cppcabrera: thanks | 18:03 |
| malini | oz_akan_ : its done | 18:03 |
| oz_akan_ | malini: tks let me cehck | 18:03 |
| amitgandhi | zyuan_: your are required in the other room | 18:04 |
| amitgandhi | nvm | 18:04 |
| zyuan_ | hmm, the proxy intrinsically depends on wsgi... | 18:07 |
| cppcabrera | zyuan_: Yes, as it is now. | 18:07 |
| zyuan_ | makes me a little bit worry about how the layout is going to be... | 18:09 |
| zyuan_ | tests/unit/proxy | 18:09 |
| zyuan_ | and i need to import tests/unit/wsgi/base.py | 18:09 |
| oz_akan_ | malini: right claims get random 404 | 18:23 |
| malini | & post messages get random 404 | 18:23 |
| malini | I am seeing 204s on claim | 18:23 |
| malini | despite having messages in the q | 18:24 |
| oz_akan_ | one web server is bad I think | 18:25 |
| kgriffs | cppcabrera: ping | 18:26 |
| cppcabrera | kgriffs: pong | 18:27 |
| kgriffs | oz_akan_, cppcabrera: so, about the repl thing | 18:27 |
| kgriffs | given that mongo has limitations on secondaries | 18:27 |
| kgriffs | and we have some significant unknowns surrounding Redis | 18:27 |
| kgriffs | I think we should go back to the caching plan | 18:28 |
| cppcabrera | +1 | 18:28 |
| kgriffs | so, catalog driver can be redis or mongo,sqlalchemy | 18:28 |
| oz_akan_ | malini: one server was bad, fixed it, can you please run test again? | 18:28 |
| kgriffs | and we will use flaper87's oslo cache | 18:28 |
| oz_akan_ | oh so the idea was using mongo secondaries on each proxy? | 18:28 |
| kgriffs | (in front of the catalog driver) | 18:28 |
| kgriffs | oz_akan_: yes | 18:28 |
| oz_akan_ | got it | 18:29 |
| oz_akan_ | +1 | 18:29 |
| zyuan_ | do we really need that flexibility in proxy? | 18:29 |
| oz_akan_ | on caching plan | 18:29 |
| kgriffs | it may still work within a patition (putting secondaries on web heads) since you won't have very many web heads per partition | 18:29 |
| zyuan_ | i think a domain specific arch is enough... | 18:29 |
| oz_akan_ | zyuan_: +1 | 18:29 |
| malini | oz_akan_ :running now | 18:30 |
| oz_akan_ | kgriffs: proxies will be for all the aprtitions | 18:30 |
| kgriffs | zyuan_: what flexibility are your referring to specifically, the catalog driver? | 18:30 |
| * cppcabrera goes to look up current state of oslo.cache | 18:30 | |
| kgriffs | cppcabrera: it is stalled due to feature freeze | 18:30 |
| kgriffs | but we will be using our copy in the marconi repo anyway | 18:31 |
| zyuan_ | catalog | 18:31 |
| cppcabrera | I remember that part, kgriffs. :) | 18:31 |
| * cppcabrera watches the mailing lists diligently | 18:31 | |
| zyuan_ | for nwo | 18:31 |
| malini | oz_akan_ : looks good so far | 18:32 |
| malini | trying messages & queue tests now | 18:32 |
| kgriffs | oz_akan: would you prefer redis or mongo for the catalog? | 18:32 |
| kgriffs | (caching aside) | 18:33 |
| oz_akan_ | malini: yay! | 18:33 |
| oz_akan_ | kgriffs: mongo | 18:34 |
| kgriffs | ok, I am going to merge that patch. got a "verbal" LGTM from flaper87 already | 18:34 |
| kgriffs | (index patch) | 18:34 |
| oz_akan_ | awesome | 18:34 |
| kgriffs | there it goes! | 18:34 |
| kgriffs | cppcabrera: ok, i say then we do a mongo driver and sqlalchemy driver for catalog backends in the proxy | 18:35 |
| kgriffs | then we can add redis later, similar to what we are doing for marconi-queueing | 18:36 |
| kgriffs | i think a localhost memcached for the proxy box will be fine for our cache | 18:37 |
| kgriffs | you might also add an in-process LRU for good measure, but not required | 18:37 |
| kgriffs | (saying this because oslo.cache already has a memcached driver) | 18:37 |
| cppcabrera | alright - hierarchical caching, mongo catalog master, and some form of local caching. | 18:38 |
| oz_akan_ | kgriffs: what about redis as cache on proxies? hasn't cppcabrera implemented like that at the moment? | 18:38 |
| kgriffs | I plan to work on an LRU backend for oslo.cache that will support expiration | 18:38 |
| cppcabrera | I have implemented that so far. | 18:38 |
| kgriffs | FWIW | 18:38 |
| oz_akan_ | kgriffs: I was planning to replace memcached with redis on web servers after testing the performance | 18:38 |
| kgriffs | mostly i want us to use oslo.cache rather than rolling our own caching library | 18:39 |
| cppcabrera | +1 kgriffs | 18:39 |
| kgriffs | we can start with the memcached driver since redis supports the memcache protocol | 18:39 |
| oz_akan_ | oki doki, sounds good | 18:39 |
| kgriffs | cppcabrera: feel free to write a native redis driver for oslo.cache | 18:39 |
| cppcabrera | I can see the implementation changes now | 18:39 |
| oz_akan_ | where can I read more about oslo.cache? | 18:40 |
| kgriffs | we will keep our drivers in marconi repo until oslo.cache is merged and then port upstream | 18:40 |
| openstackgerrit | A change was merged to stackforge/marconi: fix: Requests get slower when queues have a lot of messages https://review.openstack.org/44340 | 18:40 |
| cppcabrera | oslo.cache.get_multi/set_multi with writes to a primary mongo. | 18:40 |
| cppcabrera | The updates shouldn't be too difficult. | 18:40 |
| cppcabrera | We just need to get oslo.cache into marconi | 18:41 |
| kgriffs | cppcabrera: already there, man! | 18:43 |
| kgriffs | marconi.common.cache | 18:44 |
| cppcabrera | weird... it's not in my tree.. hmm... | 18:45 |
| cppcabrera | nevermind... I'm looking at a branch that needs rebasing | 18:46 |
| cppcabrera | Thanks, kgriffs | 18:46 |
| kgriffs | btw, can you review this latest draft? | 18:47 |
| kgriffs | https://github.com/racker/falcon/pull/176 | 18:47 |
| malini | oz_akan_ : all the tests look good | 18:47 |
| oz_akan_ | malini: thank yo | 18:48 |
| openstackgerrit | Kurt Griffiths proposed a change to stackforge/marconi: perf: Partition messages collection by project https://review.openstack.org/45966 | 18:54 |
| openstackgerrit | Zhihao Yuan proposed a change to stackforge/marconi: feat(mongo): use UNIX timestamp instead of datetime https://review.openstack.org/45760 | 19:09 |
| zyuan_ | so, kurt wrote a per-project message collection patch | 19:10 |
| zyuan_ | oz wrote a hashed message collection patch | 19:10 |
| zyuan_ | hehe | 19:10 |
| kgriffs | zyuan_: actually, oz is hashing to DB | 19:12 |
| kgriffs | not collection | 19:12 |
| zyuan_ | collection | 19:12 |
| kgriffs | so, per mongo cluster you could have, say 4 databases | 19:12 |
| zyuan_ | i reviewed it many times | 19:12 |
| kgriffs | ? | 19:12 |
| kgriffs | let me look | 19:12 |
| kgriffs | ahhhh | 19:13 |
| kgriffs | that is a bug | 19:13 |
| kgriffs | it should be picking DB, not collection | 19:14 |
| kgriffs | locks are per-db, not per-collection | 19:14 |
| kgriffs | oz_akan_: ^^^ | 19:14 |
| zyuan_ | benchmark says everything | 19:14 |
| kgriffs | i'm not saying there isn't a benefit to partitioning collection (hence my patch) | 19:14 |
| kgriffs | but there is also a benefit for partitioning dbs | 19:15 |
| kgriffs | that was the original goal of oz's patch - not sure how we got our wires crossed | 19:15 |
| zyuan_ | and, per-project collection brings a heavy overhead on indexing i guess | 19:15 |
| kgriffs | (goal was avoiding db lock) | 19:15 |
| kgriffs | s/avoiding/mitigating | 19:15 |
| zyuan_ | there is no lock between collections i guess | 19:15 |
| kgriffs | zyuan_: actually, lower overhead on indexing since the indexes are less complex, smaller | 19:15 |
| kgriffs | (pulling project out of the index) | 19:15 |
| zyuan_ | if they have lock on collection, then why mongo does no support cross-collection query? | 19:16 |
| zyuan_ | it doesn't make sense, right? | 19:16 |
| kgriffs | zyuan_: http://docs.mongodb.org/manual/faq/concurrency/ | 19:16 |
| * flaper87 back | 19:16 | |
| zyuan_ | locks are per-db | 19:19 |
| cppcabrera | wb, flaper87! | 19:19 |
| kgriffs | wait | 19:19 |
| * flaper87 waits | 19:19 | |
| kgriffs | looks like https://review.openstack.org/#/c/45952/8/marconi/storage/mongodb/driver.py | 19:19 |
| flaper87 | cppcabrera: danke | 19:19 |
| kgriffs | he is using multiple dbs | 19:19 |
| kgriffs | hmm | 19:19 |
| kgriffs | let me look over the entire patch | 19:20 |
| flaper87 | btw, re that patch | 19:20 |
| flaper87 | do we really need those 2 third-party libs ? | 19:20 |
| flaper87 | I haven't looked at the patch yet | 19:20 |
| flaper87 | so, I may be saying something stupid | 19:20 |
| flaper87 | but, I'm just concerned about the dependencies list and the fact they have to be added to openstack/requirements | 19:20 |
| zyuan_ | kgriffs: he opened two connection to 1 db | 19:20 |
| zyuan_ | team: do not merge https://review.openstack.org/45760 , i haven't tested it after rebasing | 19:22 |
| kgriffs | lets see | 19:22 |
| kgriffs | conn == client | 19:22 |
| zyuan_ | it heavily conflicted with kurt's patch which was just merged | 19:22 |
| kgriffs | so _messageS_databases is a list of DB objects, marconi_m1, marconi_m2, etc. | 19:23 |
| kgriffs | but, this looks wrong | 19:24 |
| kgriffs | https://review.openstack.org/#/c/45952/8/marconi/storage/mongodb/claims.py | 19:24 |
| zyuan_ | kgriffs: ? | 19:24 |
| kgriffs | just a minute | 19:24 |
| zyuan_ | just look at this line | 19:25 |
| zyuan_ | pymongo.MongoClient(options.CFG.uri) | 19:25 |
| zyuan_ | if you want to use multiple db, you have to have multiple uri | 19:25 |
| zyuan_ | 1 URI, 1 db | 19:25 |
| kgriffs | zyuan_: you are confligrating db and server/cluster | 19:26 |
| zyuan_ | kgriffs: you mean? | 19:26 |
| kgriffs | a single MongoDB process can run multiple DBs | 19:26 |
| zyuan_ | ha? | 19:26 |
| kgriffs | so, you can have a single connection, multiple dbs | 19:26 |
| kgriffs | think of db is just a kind of namespace | 19:26 |
| zyuan_ | that's collection... | 19:27 |
| kgriffs | server/db/collection | 19:28 |
| kgriffs | it's not entirely accurate, but helps me thing about it | 19:29 |
| kgriffs | client['mydbname']['mycollectionname'] | 19:29 |
| kgriffs | client = pymongo.MongoClient(options.CFG.uri) | 19:29 |
| * cppcabrera is having a bit of fun benchmarking pypy + gunicorn + marconi-proxy (current) + marconi-mongo on localhost | 19:29 | |
| zyuan_ | the first level is db... | 19:31 |
| zyuan_ | then have we ever used collection? | 19:32 |
| zyuan_ | i see... currently we we 'marconi' db | 19:33 |
| zyuan_ | use* | 19:33 |
| zyuan_ | and 'message' | 19:33 |
| zyuan_ | 'queues', etc... | 19:33 |
| openstackgerrit | Kurt Griffiths proposed a change to stackforge/marconi: perf: Partition messages collection by project https://review.openstack.org/45966 | 19:35 |
| zyuan_ | kgriffs: oz_akan_ 's patch is right | 19:35 |
| zyuan_ | he used multi db | 19:35 |
| *** whenry has quit IRC | 19:35 | |
| zyuan_ | but! he collected collections from multi db into the old self._col | 19:35 |
| kgriffs | flaper87: need a review on this: https://review.openstack.org/#/c/45966/ | 19:35 |
| zyuan_ | which makes it 'looks' like using multiple collections | 19:36 |
| flaper87 | kgriffs: will do | 19:36 |
| flaper87 | kgriffs: does that patch depends on oz_akan_'s ? | 19:36 |
| kgriffs | thanks! I imagine it will take a few iterations to get right | 19:36 |
| kgriffs | no | 19:36 |
| kgriffs | rather, i think we will need to make oz_akan_'s depend on this one | 19:36 |
| kgriffs | (see my comment) | 19:36 |
| flaper87 | (sorry guys, I haven't been part of today's discussions, it was my many meetings day) | 19:36 |
| flaper87 | kgriffs: kk | 19:36 |
| kgriffs | heh, it happens to the best of us | 19:36 |
| kgriffs | (my comment on oz's patch) | 19:37 |
| flaper87 | seriously, meetings destroy my brain, ideas or whatever could happen in there | 19:37 |
| *** whenry has joined #openstack-marconi | 19:39 | |
| cppcabrera | flaper87: meetings slow me down quite a bit, too. That was the case for me yesterday. | 19:41 |
| cppcabrera | GET /v1/queues/{queue}/stats appears to be one of our slowest endpoints. (it's slower than POST /v1/queues/{queue}/messages) | 19:46 |
| cppcabrera | This is not altogether surprising, given that /stats makes multiple DB queries. | 19:47 |
| kgriffs | cppcabrera: https://bugs.launchpad.net/marconi/+bug/1218789 | 19:48 |
| flaper87 | cppcabrera: counts | 19:48 |
| flaper87 | :D | 19:48 |
| * flaper87 should work on that bug | 19:49 | |
| cppcabrera | on my localhost pypy marconi-proxy tests, it's about... (checks) | 19:49 |
| cppcabrera | 35x slower than POST messages. :P | 19:49 |
| *** malini is now known as malini_afk | 19:50 | |
| cppcabrera | This includes pypy priming/warm-up for both the routes. | 19:50 |
| *** gordonsim has quit IRC | 19:50 | |
| cppcabrera | quick note regarding the current marconi-proxy patch: /v1/queues/{queue}/stats is missing. (zyuan, zyuan_) | 19:56 |
| cppcabrera | I accidentally left that one out. :P | 19:57 |
| openstackgerrit | Alejandro Cabrera proposed a change to stackforge/marconi: feat: marconi proxy https://review.openstack.org/43909 | 20:16 |
| cppcabrera | disregard above: rebasing patches | 20:18 |
| oz_akan_ | zyuan_: "from pylru import lrudecorator" | 20:19 |
| oz_akan_ | do I need to chagne it to import pylru ? | 20:19 |
| zyuan_ | oz_akan_: i think os... | 20:20 |
| zyuan_ | and just use pylru.lrudecorator | 20:20 |
| zyuan_ | @pylru.lrudecorator | 20:20 |
| oz_akan_ | ok got it, if that is the os way, must be then | 20:20 |
| oz_akan_ | tks | 20:20 |
| zyuan_ | cppcabrera: i guess i'd better starts with testing the non-marconi api part :) | 20:21 |
| cppcabrera | that'd be best, zyuan_. :) | 20:22 |
| cppcabrera | /v1/partitions and /v1/catalogue | 20:22 |
| cppcabrera | where /v1/catalogue is primarily controlled through /v1/queues/{queue} | 20:23 |
| cppcabrera | s/controlled/affected | 20:23 |
| openstackgerrit | Alejandro Cabrera proposed a change to stackforge/marconi: feat: marconi proxy https://review.openstack.org/43909 | 20:24 |
| openstackgerrit | Alejandro Cabrera proposed a change to stackforge/marconi: feat: marconi proxy (v1, health) https://review.openstack.org/44356 | 20:31 |
| * cppcabrera is learning how to rebase dependent patches | 20:32 | |
| zyuan_ | cppcabrera: easy, all-in-once | 20:37 |
| openstackgerrit | Alejandro Cabrera proposed a change to stackforge/marconi: feat: marconi proxy (v1, health) https://review.openstack.org/44356 | 20:38 |
| openstackgerrit | Alejandro Cabrera proposed a change to stackforge/marconi: feat: marconi-proxy forwarding https://review.openstack.org/44364 | 20:43 |
| oz_akan_ | kgriffs: https://review.openstack.org/#/c/45966/ is very slow | 20:43 |
| oz_akan_ | I commented | 20:43 |
| oz_akan_ | 0.47 ms vs 0.14 sec | 20:44 |
| zyuan_ | ~~ | 20:44 |
| openstackgerrit | Alejandro Cabrera proposed a change to stackforge/marconi: feat: marconi proxy https://review.openstack.org/43909 | 20:47 |
| openstackgerrit | Alejandro Cabrera proposed a change to stackforge/marconi: feat: marconi proxy (v1, health) https://review.openstack.org/44356 | 20:47 |
| kgriffs | oz_akan_: really? that's interesting | 20:47 |
| openstackgerrit | Alejandro Cabrera proposed a change to stackforge/marconi: feat: marconi-proxy forwarding https://review.openstack.org/44364 | 20:47 |
| oz_akan_ | kgriffs: I pasted test results | 20:48 |
| kgriffs | i guess that's what benchmarks are for, after all | 20:48 |
| cppcabrera | there - rebasing + minor bug fixes completed for marconi-proxy. | 20:48 |
| cppcabrera | I'm done spamming this room with review requests. :P | 20:48 |
| oz_akan_ | this is 50 users 40 seconds and 100 users 40 seconds benchmark | 20:48 |
| cppcabrera | +1 for your benchmark efforts, oz_akan_ | 20:48 |
| oz_akan_ | with a single project id | 20:48 |
| kgriffs | hmm, interesting | 20:48 |
| kgriffs | I suspect that at extremely large scale this patch may actually perform better, but that is hard to know until we have the ability to test it | 20:49 |
| kgriffs | let me put this on the backburner | 20:49 |
| oz_akan_ | for one project if it was a bit slower I think it would be acceptable | 20:49 |
| kgriffs | it's probably mostly due to the lookup cost to get the collection to use | 20:49 |
| oz_akan_ | yes lookup is slow, that is why I used lru cache for database mapping | 20:50 |
| oz_akan_ | and using 10 db is slow than using 2-4 | 20:50 |
| oz_akan_ | slower | 20:50 |
| kgriffs | I also use lru cache, but it still adds overhead | 20:51 |
| kgriffs | oz_akan: did you see my comment? | 20:51 |
| kgriffs | i think your perf will do much better if you only partition on dbs, not collections | 20:51 |
| oz_akan_ | on lru ? | 20:51 |
| kgriffs | oz_akan_: no, in gerrit | 20:51 |
| kgriffs | on your patch | 20:51 |
| oz_akan_ | yes, I believe I replied let me check | 20:52 |
| openstackgerrit | Kurt Griffiths proposed a change to stackforge/marconi: chore: Update oslo.config to version 1.2.0a4 https://review.openstack.org/46136 | 20:53 |
| oz_akan_ | zyuan_: thank your last comments | 20:54 |
| oz_akan_ | zyuan_: thanks for your last comments | 20:54 |
| cppcabrera | Regarding oslo 1.2.0a4: http://docs.openstack.org/developer/oslo.config/#a4 (kgriffs, flaper87) | 20:55 |
| cppcabrera | It;s nice seeing the release notes. | 20:55 |
| cppcabrera | "Finished Python 3 support" | 20:55 |
| cppcabrera | woot | 20:55 |
| flaper87 | cppcabrera: that moves us a step forward to support P3K | 20:56 |
| kgriffs | oz_akan_: saw your comment, let me take another look | 20:56 |
| oz_akan_ | tks | 20:56 |
| kgriffs | oic | 20:57 |
| cppcabrera | Jenkins doesn't like the latest oslo.config... | 20:57 |
| cppcabrera | Hmm... | 20:57 |
| kgriffs | sorry, should have finished reading all the changes before i commented | 20:57 |
| kgriffs | i got mixed up since i was thinking about my patch at the same time. :p | 20:57 |
| oz_akan_ | :) | 20:58 |
| cppcabrera | It seems like the embedded server isn't launching for the functional tests with oslo.config-1.2.0a4 | 20:59 |
| cppcabrera | "ConnectionError: HTTPConnectionPool(host='127.0.0.1', port=8888): Max retries exceeded with url: /v1/queues?detailed=True (Caused by <class 'socket.error'>: [Errno 111] Connection refused)" | 20:59 |
| flaper87 | cppcabrera: mmhh | 20:59 |
| kgriffs | oz_akan_: so, a suggestion | 21:00 |
| kgriffs | to simplify your code (DRY) | 21:00 |
| *** malini_afk is now known as malini | 21:00 | |
| kgriffs | make a helper to give you the correct collection given queue_name, project | 21:00 |
| oz_akan_ | apart of self.driver.db_id ? | 21:00 |
| kgriffs | collection = self._collection(queue_name, project) | 21:01 |
| oz_akan_ | I see, I will return collection directyl | 21:01 |
| kgriffs | I just noticed that you have to do self.driver.db_id followed by _col[db_id] all over the place | 21:01 |
| kgriffs | just factor those out into a helper | 21:01 |
| flaper87 | cppcabrera: any idea why ? | 21:01 |
| oz_akan_ | got it | 21:01 |
| kgriffs | cool | 21:01 |
| kgriffs | you could even lru-decorate that helper | 21:02 |
| kgriffs | (may or may not be faster, depends on impact of function-call overhead) | 21:02 |
| oz_akan_ | driver.db_id has the decorator | 21:02 |
| oz_akan_ | hmm | 21:02 |
| kgriffs | i mean, at a higher level | 21:03 |
| kgriffs | move your decorator up to the helper we just discussed | 21:03 |
| kgriffs | so it will memoize the array lookup as well | 21:03 |
| cppcabrera | flaper87: I have no idea. :/ | 21:04 |
| flaper87 | cppcabrera: can you run marconi-server ? | 21:04 |
| kgriffs | oz_akan: p.s. - db_id() can be a free function, maybe part of utils.py | 21:04 |
| oz_akan_ | kgriffs: makes sense | 21:04 |
| oz_akan_ | about db_id I didn't know where to put it | 21:05 |
| oz_akan_ | if you think so, sure I will move it there | 21:05 |
| kgriffs | i think it makes sense | 21:05 |
| cppcabrera | flaper87: I'll try to from the oslo.config++ branch | 21:05 |
| kgriffs | you aren't accessing anything with self | 21:05 |
| kgriffs | and utils is a good place to put things that may be (potentially) used in different modules. | 21:06 |
| oz_akan_ | I was thinking it might be faster like this as it is part of the object... but I understand it doesn't really belong there | 21:07 |
| oz_akan_ | thanks kgriffs I am heading home | 21:08 |
| cppcabrera | kgriffs: when you get a chance, can I get a quick review on https://review.openstack.org/#/c/45439/ ? (needed to advance marconiclient dev) | 21:09 |
| cppcabrera | tiny patch | 21:09 |
| *** oz_akan_ has quit IRC | 21:09 | |
| kgriffs | cppcabrera: the issue is we don't wait long enough for the server to boot | 21:09 |
| kgriffs | I was seeing that too | 21:09 |
| kgriffs | not related to the other patch imo | 21:10 |
| cppcabrera | well... | 21:10 |
| cppcabrera | "error: Could not find required distribution oslo.config>=1.2.0a4" | 21:10 |
| kgriffs | (the update for oslo.config) | 21:10 |
| *** oz_akan_ has joined #openstack-marconi | 21:10 | |
| cppcabrera | I got that from 'git review -d $OSLO_PATCH' | 21:10 |
| kgriffs | oh, interesting | 21:10 |
| kgriffs | hmm | 21:10 |
| cppcabrera | after doing python setup.py develop | 21:10 |
| flaper87 | why are we increasing oslo's version? | 21:10 |
| kgriffs | flaper87: mostly testing it to see if we have any issues, but also py3k support, right? | 21:11 |
| cppcabrera | found the problem | 21:11 |
| cppcabrera | "http://tarballs.openstack.org/oslo.config/oslo.config-1.2.0a>>>>>3<<<<<.tar.gz#egg=oslo.config-1.2.0a4" | 21:11 |
| cppcabrera | One '3' needs to become a '4' | 21:12 |
| kgriffs | oops | 21:12 |
| * kgriffs blushes | 21:12 | |
| flaper87 | kgriffs: sounds good, I was just curious if there was another need bsides those | 21:12 |
| flaper87 | +! | 21:13 |
| flaper87 | +1 | 21:13 |
| *** oz_akan_ has quit IRC | 21:15 | |
| openstackgerrit | Kurt Griffiths proposed a change to stackforge/marconi: chore: Update oslo.config to version 1.2.0a4 https://review.openstack.org/46136 | 21:15 |
| flaper87 | guys, you gotta wait for Jenkins bless :P | 21:18 |
| cppcabrera | +1 flaper87. :) | 21:19 |
| kgriffs | flaper87: yep | 21:19 |
| cppcabrera | jenkins > core | 21:19 |
| kgriffs | hmm, that is odd | 21:20 |
| kgriffs | why are we getting 204 now - shouldn't have anything to do with the config upgrade | 21:20 |
| cppcabrera | pypy succeeds... >.> | 21:22 |
| kgriffs | hmmm | 21:22 |
| flaper87 | try recheck no bug | 21:22 |
| flaper87 | lets see if py27 fails again | 21:23 |
| cppcabrera | python26 succeeds, too | 21:23 |
| flaper87 | if it doesn't then it's random and I'll have to dig more | 21:23 |
| * flaper87 hates random behaviours in functional tests | 21:23 | |
| openstackgerrit | Kurt Griffiths proposed a change to stackforge/marconi: perf: Partition messages collection by project https://review.openstack.org/45966 | 21:25 |
| kgriffs | oz_akan: can you benchmark this? Made a little change to lru just out of curiosity | 21:26 |
| kgriffs | https://review.openstack.org/#/c/45966/ | 21:26 |
| kgriffs | kk folks, I gotta hit the road - I'll be back online in ~1 hour | 21:27 |
| kgriffs | oh darn | 21:27 |
| kgriffs | missed Oz | 21:27 |
| kgriffs | malini: | 21:27 |
| kgriffs | ping | 21:27 |
| kgriffs | :D | 21:27 |
| kgriffs | :D | 21:27 |
| kgriffs | :D | 21:27 |
| flaper87 | kgriffs: have a good one | 21:28 |
| kgriffs | thanks, ttfn | 21:28 |
| * kgriffs throws pop-tarts everywhere and makes a mad dash out the door | 21:28 | |
| cppcabrera | woot | 21:34 |
| cppcabrera | I'm out, as well. | 21:34 |
| cppcabrera | Good night, everyone. :) | 21:34 |
| flaper87 | cppcabrera: night | 21:37 |
| * flaper87 ate all those pop-tarts | 21:37 | |
| *** kgriffs is now known as kgriffs_afk | 21:37 | |
| *** tedross has quit IRC | 21:43 | |
| *** cppcabrera has quit IRC | 21:44 | |
| *** kgriffs_afk is now known as kgriffs | 22:16 | |
| *** oz_akan_ has joined #openstack-marconi | 22:20 | |
| *** oz_akan_ has quit IRC | 22:25 | |
| *** amitgandhi has quit IRC | 22:30 | |
| *** malini is now known as malini_afk | 22:34 | |
| *** flaper87 is now known as flaper87|afk | 22:42 | |
| openstackgerrit | Kurt Griffiths proposed a change to stackforge/marconi: fix(tests): unit tests disabled due to missing __ini__.py https://review.openstack.org/46155 | 23:32 |
| openstackgerrit | A change was merged to stackforge/marconi: fix(tests): unit tests disabled due to missing __ini__.py https://review.openstack.org/46155 | 23:42 |
| openstackgerrit | Kurt Griffiths proposed a change to stackforge/marconi: perf: Partition messages collection by project https://review.openstack.org/45966 | 23:43 |
| *** kgriffs is now known as kgriffs_afk | 23:45 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!