Wednesday, 2013-09-11

*** kgriffs is now known as kgriffs_afk00:32
*** nosnos has joined #openstack-marconi00:36
*** oz_akan_ has joined #openstack-marconi00:41
*** amitgandhi has joined #openstack-marconi00:58
*** ayoung has joined #openstack-marconi01:13
*** amitgandhi has quit IRC01:38
*** ayoung is now known as admiyo02:16
openstackgerritOz Akan proposed a change to stackforge/marconi: Adds support for multiple databases for mongodb
openstackgerritOz Akan proposed a change to stackforge/marconi: Adds support for multiple databases for mongodb
*** kgriffs_afk is now known as kgriffs03:30
openstackgerritOz Akan proposed a change to stackforge/marconi: Adds support for multiple databases for mongodb
*** oz_akan_ has quit IRC03:42
*** oz_akan_ has joined #openstack-marconi03:43
*** oz_akan_ has quit IRC03:47
*** kgriffs is now known as kgriffs_afk04:22
*** ykaplan has joined #openstack-marconi06:47
*** flaper87|afk is now known as flaper8707:29
openstackgerritFlavio Percoco proposed a change to stackforge/marconi: Move Unit tests under a unit package
*** ykaplan has quit IRC09:29
*** ykaplan has joined #openstack-marconi09:35
openstackgerritA change was merged to stackforge/marconi: Move Unit tests under a unit package
*** ykaplan_ has joined #openstack-marconi10:32
*** ykaplan__ has joined #openstack-marconi10:33
*** ykaplan has quit IRC10:36
*** ykaplan_ has quit IRC10:37
*** vkmc has joined #openstack-marconi12:02
*** tedross has joined #openstack-marconi12:04
*** oz_akan_ has joined #openstack-marconi12:18
*** oz_akan_ has quit IRC12:22
*** key4 has quit IRC12:45
*** key4 has joined #openstack-marconi12:46
*** ykaplan_ has joined #openstack-marconi12:47
*** ykaplan__ has quit IRC12:51
*** cthulhup has quit IRC12:53
*** oz_akan_ has joined #openstack-marconi12:56
*** oz_akan_ has joined #openstack-marconi12:57
*** amitgandhi has joined #openstack-marconi13:12
oz_akan_hi team13:23
oz_akan_zyuan_: just said hi, I don't want anything :)13:23
zyuan_... morning13:23
flaper87oz_akan_: zyuan_ yo! Good morning13:24
oz_akan_good afternoon flaper8713:24
*** malini_afk is now known as malini13:31
*** nosnos has quit IRC13:38
*** kgriffs_afk is now known as kgriffs13:51
*** malini is now known as malini_afk13:53
oz_akan_who put all these spaces there.. jenkins is mad about my patch13:54
*** malini_afk is now known as malini13:55
kgriffsoz_akan_: looks like mmh3 is not in the deps file14:00
oz_akan_right I will add it there14:00
oz_akan_pylru too14:01
kgriffsah, yes14:01
oz_akan_it seems pylru is slower than the one I had used earlier14:01
kgriffsoh yeah?14:01
kgriffsby how much?14:02
oz_akan_I will submit with pylru later after some benchmark, we may change if we think it is still slow14:02
kgriffsit isn't in the patch yet14:02
* kgriffs looks14:02
oz_akan_patch 4 must have it14:02
oz_akan_kgriffs: quick question14:05
oz_akan_I have this : message = list(self._col[db_id].find(query).limit(1).hint([('_id', 1)]))14:05
oz_akan_and pep8 says it is too long14:05
oz_akan_ line too long (80 > 79 characters)14:05
oz_akan_what shall I do?14:05
* kgriffs looking14:06
oz_akan_there are no spaces nothing, that I can split it14:07
oz_akan_I could make something like col = self._col[db_id] and then col.find ...14:07
oz_akan_but it seems kind of silly14:07
zyuan_split it into two lines14:09
zyuan_split at '('14:09
kgriffsI would do collection = self._col[db_id] on the previous line. HACKING actually encourages breaking up statement compositions anyway, for readability14:09
kgriffsyou can also do hint as a separate step to make the line even shorter14:09
kgriffscursor = foo14:09
kgriffscursor = foo.hint(…)14:09
kgriffscursor = cursor.hint(...)14:09
zyuan_i don't overwrite variable unless i have to...14:10
kgriffsi don't mind that style myself, as long as it is done on immediately successive lines14:11
kgriffsand it isn't overdone14:11
kgriffs(more than 2 or three statements in a row starts to look ugly)14:12
oz_akan_kgriffs: zyuan_ thanks14:14
openstackgerritOz Akan proposed a change to stackforge/marconi: Adds support for multiple databases for mongodb
oz_akan_lets see what jenkins is going to say14:23
kgriffsoz_akan_: just for fun, I converted your lru algorithm to use a closure14:34
kgriffsdef make_lru(user_function, maxsize=1024):14:34
kgriffs    # Link layout:     [PREV, NEXT, KEY, RESULT]14:34
kgriffs    root = [None, None, None, None]14:34
kgriffs    ctx = {'root': root}14:34
kgriffs    user_function = user_function14:34
kgriffs    cache = {}14:34
kgriffs    last = root14:34
kgriffs    for i in range(maxsize):14:34
kgriffs        key = object()14:34
kgriffs        cache[key] = last[1] = last = [last, root, key, None]14:34
kgriffs    root[0] = last14:34
kgriffs    def lookup(*key):14:34
kgriffs        root = ctx['root']14:34
kgriffs        link = cache.get(key)14:34
kgriffs        if link is not None:14:34
kgriffs            link_prev, link_next, _, result = link14:34
kgriffs            link_prev[1] = link_next14:34
kgriffs            link_next[0] = link_prev14:34
kgriffs            last = root[0]14:34
kgriffs            last[1] = root[0] = link14:34
kgriffs            link[0] = last14:34
kgriffs            link[1] = root14:34
kgriffs            return result14:34
kgriffs        result = user_function(*key)14:34
kgriffs        root[2] = key14:34
kgriffs        root[3] = result14:34
kgriffs        oldroot = root14:34
kgriffs        root = ctx['root'] = root[1]14:34
kgriffs        root[2], oldkey = None, root[2]14:34
kgriffs        root[3], oldvalue = None, root[3]14:34
kgriffs        del cache[oldkey]14:35
kgriffs        cache[key] = oldroot14:35
kgriffs        return result14:35
kgriffs    return lookup14:35
kgriffsmeant to paste this:
oz_akan_it came really slow, as if you are typinh14:35
kgriffson my MBP it yielded 20% speedup14:35
oz_akan_versus pylru ?14:35
oz_akan_or vs not having closure?14:36
kgriffssame test using pylru is 2.25 µs per loop14:36
kgriffsbtw, any particular reason that you accept a variable list of key(s) instead of just a single key?14:38
oz_akan_so we have 1.65 vs 2.2514:38
oz_akan_I didn't really write that code, I found online14:39
kgriffsah, ok14:39
kgriffslet me see if I can make it is even better. ;)14:39
oz_akan_:) at this point making it faster magic to me14:40
*** cppcabrera has joined #openstack-marconi14:41
cppcabreraGood morning!14:41
oz_akan_cppcabrera: moving here14:46
oz_akan_I can't make it correcr14:46
cppcabreraI heard some pretty cool news. eventlet is looking to support python 3.3 soon: (kgriffs, flaper87)14:48
oz_akan_ about limiting number of queues per project14:52
kgriffscppcabrera: oh, interesting. looks like somebody has a vested interest in making that happen!14:54
openstackgerritOz Akan proposed a change to stackforge/marconi: Adds support for multiple databases for mongodb
kgriffsoz_akan_: there may already be a bp for that (basically, quotas)14:55
oz_akan_let me see14:55
oz_akan_got it, couldn't see it before14:55
oz_akan_let me delete then what I created14:56
oz_akan_yay ! malini thanks pep8 is all good now14:58
*** flaper87 is now known as flaper87|afk14:58
cppcabreraawesome, oz_akan. :)14:58
*** ykaplan_ has quit IRC14:59
maliniautopep8 shud be part of jenkins ;-)14:59
oz_akan_yes :)14:59
oz_akan_a few long line and then jenkins will start to love the code15:00
oz_akan_I am so close15:00
kgriffscppcabrera: re the cons of using replication vs. caching15:00
kgriffs"The other half of the problem with this approach is that there's no longer a notion of a "cache miss" for slaves - just stale data."15:02
kgriffsi have an idea for addressing that15:02
kgriffslet me bring up an etherpad15:02
openstackgerritOz Akan proposed a change to stackforge/marconi: Adds support for multiple databases for mongodb
cppcabrerathanks for bringing that up again, kgriffs. Etherpad sounds good to me: +115:06
*** gordonsim has joined #openstack-marconi15:07
*** flaper87|afk is now known as flaper8715:08
oz_akan_sometimes I think if cppcabrera has a very clever auto responder :)15:08
* flaper87 is back15:09
kgriffsaddressing "it isn't really cache" that is correct. We are using replication in lieu of cache, not dissimilar to what swift does by replicating the hash rings15:09
flaper87cppcabrera: wow, eventlet Py3k :D15:09
kgriffsif we use mongodb then shouldn't be an issue with memory usage. Redis would impose a max catalog size, obviously15:09
*** ykaplan_ has joined #openstack-marconi15:10
oz_akan_if we use redis as master and then replicate date to slaves, we will also have to map data to disk, because we will rely on redis only15:11
flaper87are we talking about marconi-proxy ?15:11
oz_akan_and we will have to ensure there are 3 copies healthy all the time15:11
oz_akan_so redis won't be pure in memory cache anymore15:12
oz_akan_one concern is the behavior when we have 10-20 proxies15:12
kgriffsbtw guys, stuff like pylru is MUCH faster under pypy15:13
oz_akan_in case of a massive reboot, it will take a while to replicate data to all slaves15:13
oz_akan_flaper87: yes15:13
kgriffsit basically means you don't have to try as hard to optimize15:13
kgriffsthat's why i REALLY want us to benchmark pypy with gunicorn15:13
oz_akan_pypy won't work with uwsgi?15:14
kgriffsmmm, good question.15:14
kgriffsi can't remember15:14
* kgriffs knows he used to know15:14
kgriffsyay for Alex!15:14
kgriffswe REALLY need to try that!15:14
cppcabrerahowever... (
cppcabreraThis pypy blog recommends using gunicorn + pypy over uwsgi + pypy15:15
flaper87can I say something about the proxy ?15:15
kgriffsthe uwsgi patch came in more recently than that article15:15
cppcabrerano, flaper87. :P15:15
cppcabrerago on~ :D15:15
flaper87so, I was thinking this morning that we could structure Marconis tree similar to swifts15:16
flaper87and have: proxy, queues, notification15:16
flaper87right under marconi/marconi/15:16
flaper87and, I was thinking about kgriffs proposal15:16
flaper87and I think it does make sense to support multiple storage for the proxy as well15:17
flaper87ok, I said it! No, feel free to insult me :D15:17
kgriffsyou mean, organize the tree by resource rather than driver type?15:19
oz_akan_my fear is that if we focus on trying to support multiple of everything, we will end up with something that doesn't work good enough for anything15:20
oz_akan_proxy storage, has to be very stable and performant enough. that is the requirement15:21 is not py3 compatible... heheh15:22
flaper87oz_akan_: I agree15:22
flaper87oz_akan_: my thinking is that we should work on a first version of this proxy and then add support for other things15:23
kgriffsi still think we just do one, and let 3rd-parties maintain their own drivers15:23
kgriffsbut that means we need to support stevedore drivers15:23
flaper87kgriffs: +115:23
kgriffsthat forces good interface-oriented design anyway15:23
oz_akan_I see15:23
cppcabrerathat works for me - one official version + stevedore.15:24
kgriffscaveat: if we only support one backend officially, it can't be AGPL15:24
oz_akan_+1 then, we are on the same page for sure15:24
flaper87TBH, I think it could be an sqlalchemy backend15:24
flaper87I mean, storage15:24
kgriffsif somebody comes up with an alternative that is insanely great, we can consider pulling it later15:24
flaper87that would give enough choices to people using the proxy15:24
flaper87from sqlite and mysql to psql15:25
openstackgerritOz Akan proposed a change to stackforge/marconi: Adds support for multiple databases for mongodb
cppcabreraI'd consider writing an sqlalchemy driver for the proxy as a first, except that I have very little experience actually working with SQLs. :P15:26
kgriffsflaper87: only thing with that is we will likely need in-process lru since sql queries are going to be way slower than, say, redis15:26
oz_akan_you know what we should use mysql as nosql15:26
oz_akan_if I am not wrong there is a driver that that doesn't uses sql and it is faster than anything out there15:26
oz_akan_then, why would someone think about using another sql backend15:27
flaper87kgriffs: mmh, good point15:27
oz_akan_if nothing can beat the performance15:27
flaper87kgriffs: what about using an in-memory instance of sqlalchemy for lru ?15:27
flaper87ah wait15:27
flaper87that's stupid15:27
* flaper87 facepalm15:27
oz_akan_for proxy I want mongo for sure15:27
kgriffsoz_akan_: if it is comparable to redis, then sounds good15:27
oz_akan_3 copies of data, same storage with marconi data15:28
flaper87oz_akan_: mongo is AGPL15:28
oz_akan_same skill set to manage15:28
oz_akan_like kgriffs mongo + sqlalchemy15:28
kgriffsif we do mongo, we will need to maintain 2 drivers as we do with the server storage15:28
flaper87cppcabrera: did you take a look here?
kgriffssrsly these drivers will be very simple15:29
flaper87kgriffs: +115:29
cppcabrerayup, flaper87.15:29
kgriffsflaper87: oh, so you are just dropping current stuff down under "queue"15:29
oz_akan_operationally it is not desired to have two different database technology15:29
kgriffsoz_akan_: good point15:29
flaper87kgriffs: does that make sense? :P15:30
flaper87kgriffs: s/queue/queuing15:30
flaper87or another name15:30
oz_akan_ finally passed jenkins test, please review when you have time. I need to get rid of this asap15:30
kgriffsi vote for either just redis or mongo + sqlalchemy15:30
* flaper87 is pretty bad for names15:30
kgriffs(just redis because you probably have that anyway; keystone middleware speaks memcached and so does redis, so you don't need memcached as well)15:30
kgriffsit's all sort of queueing related15:31
kgriffsflaper87: let me think about a name there15:31
oz_akan_I think redis is not stable enough to hold critical mapping information. there the more traditional it is the safer probably15:31
flaper87kgriffs: what do you think about the structure in general ?15:31
oz_akan_I mean to store permanently15:31
flaper87I'd say mongo + sqlalchemy15:32
kgriffsflaper87: i think it is good.15:32
kgriffsmarconi-server (?)15:32
kgriffsand marconi-notifications15:32
kgriffsthat last one I guess will just be some set of workers and an API for controlling them15:32
cppcabreramarconi/marconi/{server,proxy,notifications}, I'm gathering.15:33
flaper87cppcabrera: I'd be ok with that but I think it'll create some confusion15:33
flaper87like: "What is server?"15:33
cppcabrerahmm... that'd be our current marconi.15:34
flaper87confusion in terms of, lack of consistency15:34
cppcabreragotcha - server is pretty ambiguous.15:34
cppcabreraeven the proxy is a type of server15:34
cppcabreraserver -> message_bus|queues15:34
cppcabreraon the topic of proxy storage, I've gathered the following so far: 1) support multiple storage backends (implies stevedore), 2) start with mongo, 3) support sqlalchemy later, 4a) use slave/master replication 4)b use hierarchical caching15:35
flaper87cppcabrera: I'd sawp 2 and 315:36
* flaper87 is thinking about graduation15:36
kgriffsonly think is, nobody knows how to spell queueing15:37
kgriffsI still vote for 4a since it is the fastest option, and if using disk-backed store, the non-hot queues will get paged to disk and most stuff can stay in RAM15:38
cppcabrera4a with a disk-backed option will be speedy, no doubt, since the queries will be of a key-value nature.15:39
kgriffsi say just leave it up to the catalog driver (cache or whatever)15:40
flaper87kgriffs: +!15:40
flaper87kgriffs: +115:40
flaper87up to the catalog driver15:40
kgriffsthe mongo one can do replication since that is super nice and easy15:40
kgriffsmysql read-only slaves are a bit of a pain, but doable15:41
kgriffswe can punt on that, but we will need SQLAlchemy in order to graduate15:41
kgriffs(AGPL thing)15:41
zyuan_flaper87: nosetests became totally unusable; what happened?15:42
flaper87zyuan_: o.015:42
flaper87why ?15:42
flaper87what's the error?15:42
zyuan_it just stuck on the first test15:42
flaper87(btw, make sure you remove all .pyc)15:42
flaper87mmh :(15:42
flaper87no error ?15:42
cppcabreraI wonder if they'restuck on the server start up...15:42
zyuan_it just stuck. no progress, can't even break into the test function15:43
zyuan_which server?15:43
cppcabrerakgriffs, flaper87: I can take the 4a route, but I'll have to read up on slave/master replication to understand what that means for the driver implementation (caveats, e.g.) and what it means for scaling (likely eventual consistency, which is fine).15:43
flaper87zyuan_: tox -epy27 doesn't work ?15:43
flaper87zyuan_: I just ran that, it worked perfectly15:44
zyuan_tox works15:44
flaper87zyuan_: what command are you running ?15:44
flaper87nosetests tests15:45
flaper87that works for me15:45
flaper87Ran 350 tests in 8.345s15:45
flaper87OK (SKIP=162)15:45
zyuan_still stuck15:45
flaper87holymoly, 350 tests already ?15:45
zyuan_there is no that much actually15:46
flaper87zyuan_: man, don't break my wings just like that15:46
flaper87zyuan_: seriously, not sure why it gets stuck in your box15:46
flaper87cppcabrera: can you try running those tests ?15:46
flaper87plus, why does tox work and nosetests doesn;t ?15:47
kgriffszyuan_: are you running on BSD or Linux?15:48
zyuan_i don't even know where it stucked...15:48
zyuan_i don't think we used any non-portable stuff...15:49
cppcabreraI'm checking it out, flaper87. :)15:49
cppcabrerafrom a fresh venv15:49
zyuan_wait... memcache?15:49
flaper87memcache ?15:50
flaper87brb, meeting15:52
zyuan_do i need to run memcached?15:52
*** ykaplan_ has quit IRC15:52
flaper87zyuan_: no15:52
flaper87is it requesting memcache?15:52
flaper87ah ok15:52
zyuan_i'm gussing15:52
flaper87at least I didn't leave a pdb somewhere15:53
flaper87I hope15:53
cppcabreragrin pdb -> None15:53
flaper87cppcabrera: so, do they work for you?15:54
cppcabreraalmost done getting reqs - things seem slow today. :/15:54
zyuan_trying a new repo...15:55
zyuan_doesn't work. maybe my machine's problem15:57
*** pycabrera has joined #openstack-marconi15:58
pycabreraVPNs drive me crazy...15:58
pycabreraflaper87, zyuan: Tests pass. ^^15:59
*** zyuan_ has quit IRC15:59
flaper87pycabrera: cool15:59
*** cppcabrera has quit IRC16:00
*** pycabrera is now known as cppcabrera16:00
kgriffspycabrera: is that your alternate nick?16:01
cppcabrerayup, kgriffs16:01
cppcabreraI decided to resort to an ethernet connection. Wireless VPN gets really silly @ the office.16:01
malini zyuan_ : are you running into 'Could not load 'sqlite': python-memcached' ?16:04
cppcabreraMARCONI_TESTS_CONFIGS_DIR=etc MONGODB_TEST_LIVE=1 nosetests unit -> ArgsAlreadyParsedError: arguments already parsed: cannot register CLI option (after 274 tests)16:06
*** zyuan_ has joined #openstack-marconi16:07
zyuan_how can i run unit/storage tests only?16:07
zyuan_i suspect it's only functional tests which have the problem16:08
malinigrr..I hate it when folks whine about functional tests without looking at it :'(16:08
malinisomething else is going one16:09
zyuan_i have no idea why the tests stuck at the first test under functional16:09
oz_akan_malini: we need to run functional tests on
oz_akan_current test endpoint has it16:09
oz_akan_would you be able to run the test today?16:09
malinioz_akan_ :  I am trying to figure out how to run them after the refactoring..there is something funny going on16:10
maliniflaper87: Can you try running the tests against a remote server16:11
maliniI am running into issues there16:11
zyuan_i'm running unit/transport test only, why i need MARCONI_TESTS_CONFIGS_DIR?16:12
oz_akan_malini: ok, I am out for lunch16:13
oz_akan_ waits for review16:14
* flaper87 is half here16:15
flaper87malini: what issues ?16:15
maliniflaper87: I am running into
*** ykaplan_ has joined #openstack-marconi16:16
maliniI wud assume, it doesnt need to know abt the storage drivers, since I am testing a remote installation16:16
*** ykaplan_ has quit IRC16:16
flaper87malini: just ran tests against a remote server, it worked here. Let me take a look at the paste16:16
flaper87why is it requiring memcached ?16:17
kgriffssomething about keystone middleware?16:19
maliniflaper87: functional-marconi.conf doesn't need any changes (except for limits if remote server has different configs), rt ?16:20
zyuan_confirmed, it's 226e813 which does not work for me16:21
flaper87malini: right, plus the remote server info16:21
maliniflaper87: Both my conf files arr in ~/.marconi with functional-tests.conf pointing to the remote server16:21
flaper87try installing python-memcached manually16:21
flaper87and see what happens16:21
flaper87it should work, I guess16:21
flaper87zyuan_: could it me related to multiprocessing ?16:23
malinilooks like its trying to start a marconi server16:24
* flaper87 is just guessing16:24
malini'2013-09-11 12:23:18.015 5111 INFO marconi.transport.wsgi.driver [-] Serving on host
flaper87malini: set run_server=False16:24
flaper87malini: in the functional-tests.conf16:24
flaper87malini: did that work ?16:27
* flaper87 is confused about that memcached thing16:27
flaper87malini: are you using latest code?16:28
flaper87malini: there shouldn't be any calls16:28
flaper87I replaced that with a small http client16:28
cppcabrerathat looks outdated, malini. "marconi/tests/functional/wsgi/v1/" no longer exists.16:29
cppcabreranvm. >.>16:29
cppcabreraIt does.16:29
* cppcabrera is confused16:29
malinidefd75e1a5b3b5d7c0730151acc86fd18e3e36ae is the latest, rt?16:29
flaper87cppcabrera: ah, that too16:29
flaper87malini: remove all .pyc :D16:30
flaper87malini: find . -name "*.pyc" -delete16:30
flaper87malini: and yes, that's latest16:30
cppcabreramalini: I've confirmed the existence of '' in marconi.tests.functional.helpers16:34
cppcabreraThat's a bug.16:34
malinicppcabrera: you made it first :(16:34
cppcabrerathat should be requests.post16:34
maliniflaper87: you probably had auth turned off16:34
maliniso never got it :)16:35
cppcabrerakgriffs, flaper87, oz_akan_: "A replica set can have up to 12 members, [1] but only 7 voting members. See non-voting members for more information." - mongo docs16:35
cppcabreraSo if we take the slave/master approach, we're limited to 12 proxies, assuming a single master.16:35
cppcabreraAlso assuming mongo.16:35
kgriffsah, darn16:36
kgriffswell, how about redis then?16:36
cppcabrerachecking. :)16:36
flaper87malini: DAMNIT! I'M SO SORRY ABOUT THAT!16:37
flaper87malini: I'll fix it16:37
maliniflaper87: np :)16:37
flaper87malini: btw, do you have a server ("out there") I can use with auth enabled?16:38
flaper87I'll configure one, otherwise16:38
maliniflaper87: we have one, but you'll need a RAX identity credentials for tht :(16:39
flaper87malini: no worries, I'll set one up16:39
zyuan_after process.start() is called, the process becomes 'stopped'16:40
zyuan_can not break at server's run() method...16:41
flaper87zyuan_: mmh16:44
zyuan_confirmed, it stucked at DriverManager16:46
zyuan_on marconi/marconi/
zyuan_stevedore wins16:47
flaper87that's weird16:48
flaper87but you know, computers16:48
cppcabreraThere are no published limits on the number of slaves in a redis-backed slave/master scheme. (kgriffs, oz_akan_, flaper87)16:50
zyuan_hope run_server=false can help (me to continue writing code...)16:50
flaper87zyuan_: you can disable functional tests16:51
maliniI can run the tests now <:o)16:51
flaper87malini: what did you do?16:51
flaper87zyuan_: re disable tests
maliniI  just replaced the, nothing major16:52
flaper87malini: kk, I'll fix that bug! sorry about that, totally missed it!16:52
flaper87cppcabrera: interesting16:52
maliniflaper87: np..It was  huge patch sets & you missed just one!!16:53
malinioz_akan_ : I am seeing some random 204s & 404s on test env - post messages & claim16:53
cppcabreraredis replication is kind of nifty. The master process is forked, the master:child dumps it's state to an RDB, while the master:parent continues serving requests and building a replication buffer. Once the master:child is done dumping the state, the master:parent starts relaying the dumped RDB to slaves. Once slaves are caught up, the master:parent starts sending data from the replication buffer.16:54
cppcabreraReminds me a lot of the phased migration approach we had for proxy earlier.16:54
flaper87zyuan_: but, I'd like to undestand why it is failing on freebsd, it'd be nice if you could dig more16:55
cppcabreraA consequence of the above is that there's additional memory overhead for state replication (repl. buffer).16:55
kgriffscppcabrera: so, the concern is scale I guess16:57
zyuan_not too bad, after disabled functional tests, i ran into the "normal" stevedore errors: No 'marconi.transport' driver found, looking for 'wsgi16:57
kgriffs1000 slaves16:57
kgriffsif we assume queue creation is pretty infrequent16:57
kgriffsthen, it shouldn't be a problem methinks16:57
cppcabreraagreed, kgriffs.16:58
kgriffsso, i guess we just hit the master for create/delete queue, and hit the slave for everything else16:59
kgriffsif we get a miss on the slave, check the master16:59
kgriffsif it isn't there either, 404?16:59
cppcabrerathat should work16:59
kgriffsi guess that depends on the request. seems like some requests don't check queue's existence16:59
kgriffslike, claim16:59
kgriffsif you stick the proxy in there, then that would slightly change the expected error codes unless the proxy tracks what the server does17:00
cppcabrerathe proxy would have to maintain API consistency with the server, which is unfortunate.17:01
kgriffsI guess our functional tests should catch mismatches?17:01
cppcabreraYup, given a good suite of basic scenarios.17:01
kgriffsbtw guys, can we get this merged?17:02
kgriffsflaper87: ^^17:02
zyuan_either me stupid or venv stupid17:02
flaper87kgriffs: can I review that after dinner?17:02
* flaper87 is starving17:02
kgriffscppcabrera: ok, can you make sure we have test coverage for that? I think we do.17:02
cppcabreramy last concern is HA w/ redis. I need to investigate the master election protocol, and the deployment process - how one would go about setting this up.17:02
kgriffsflaper87: no changes since you last looked, just rebased17:02
cppcabreraalso, data redundancy concerns17:03
zyuan_i forgot i have to setenv PYTHONPATH . to get stevedore work17:03
cppcabrerameaning - let;'s never lose all the data. :P17:03
flaper87cppcabrera: I'll read the backlog in a bit and comment17:03
kgriffscppcabrera: ah, as in how big these catalogs will get17:03
flaper87kgriffs: kk17:03
zyuan_problem solved (although i don't know the reason behind...)17:03
cppcabrerathanks: flaper8717:03
cppcabrerakgriffs: That's another one.17:03
flaper87then LGTM17:03
cppcabreraHow much memory do we need?17:03
cppcabrerapartitions should take up almost no space.17:03
cppcabreraIt's the catalogue proper that will get bulky over time.17:04
cppcabreraSlowly, though.17:04
cppcabreraBut surely.17:04
kgriffscppcabrera: that patch, can you review? I don't think you saw it since you got back from jury duty17:04
cppcabreraI'll load it up now, kgriffs. :)17:04
kgriffsflaper87: when you get back, as long as cppcabrera is cool with the patch, you can approve17:05
zyuan_err, kgriffs 's patch conflicts with my utcnow_ts change...17:05
flaper87kgriffs: active and total run the same query17:05
flaper87ah no17:06
* flaper87 asked the same question last time17:06
flaper87difference, this time I did it outloud17:06
flaper87ok, brbr17:06
cppcabrerakgriffs: reviewed - typos strike again.17:10
cppcabrera-1 for typo, +2 for implementation17:10
openstackgerritKurt Griffiths proposed a change to stackforge/marconi: fix: Requests get slower when queues have a lot of messages
kgriffscppcabrera: ^^^ Fixed typo17:39
cppcabrerakgriffs: +2'd17:41
*** admiyo has quit IRC17:54
*** ayoung_ has joined #openstack-marconi17:54
oz_akan_malini: I saw your message on random 404s17:56
malinioz_akan: want me try again, so you can watch the logs?17:56
*** ayoung_ is now known as ayoung17:59
oz_akan_malini: yes pls give me a few minutes17:59
oz_akan_I am cleaning logs18:00
zyuan_cppcabrera: on today's stand-up, can you tell the team i'm doing the unit tests stuff?  i'm going to stay home this afternoon as well.18:01
oz_akan_malini: I am ready, could you run?18:02
cppcabrerasure thing, zyuan_18:03
zyuan_oz_akan_: which is your email adress, ozgur.akan@ or oz.akan@ ?18:03
oz_akan_oz.akan@ better18:03
zyuan_cppcabrera: thanks18:03
malinioz_akan_ : its done18:03
oz_akan_malini: tks let me cehck18:03
amitgandhizyuan_: your are required in the other room18:04
zyuan_hmm, the proxy intrinsically depends on wsgi...18:07
cppcabrerazyuan_: Yes, as it is now.18:07
zyuan_makes me a little bit worry about how the layout is going to be...18:09
zyuan_and i need to import tests/unit/wsgi/base.py18:09
oz_akan_malini: right claims get random 40418:23
malini& post messages get random 40418:23
maliniI am seeing 204s on claim18:23
malinidespite having messages in the q18:24
oz_akan_one web server is bad I think18:25
kgriffscppcabrera: ping18:26
cppcabrerakgriffs: pong18:27
kgriffsoz_akan_, cppcabrera: so, about the repl thing18:27
kgriffsgiven that mongo has limitations on secondaries18:27
kgriffsand we have some significant unknowns surrounding Redis18:27
kgriffsI think we should go back to the caching plan18:28
kgriffsso, catalog driver can be redis or mongo,sqlalchemy18:28
oz_akan_malini: one server was bad, fixed it, can you please run test again?18:28
kgriffsand we will use flaper87's oslo cache18:28
oz_akan_oh so the idea was using mongo secondaries on each proxy?18:28
kgriffs(in front of the catalog driver)18:28
kgriffsoz_akan_: yes18:28
oz_akan_got it18:29
zyuan_do we really need that flexibility in proxy?18:29
oz_akan_on caching plan18:29
kgriffsit may still work within a patition (putting secondaries on web heads) since you won't have very many web heads per partition18:29
zyuan_i think a domain specific arch is enough...18:29
oz_akan_zyuan_: +118:29
malinioz_akan_ :running now18:30
oz_akan_kgriffs: proxies will be for all the aprtitions18:30
kgriffszyuan_: what flexibility are your referring to specifically, the catalog driver?18:30
* cppcabrera goes to look up current state of oslo.cache18:30
kgriffscppcabrera: it is stalled due to feature freeze18:30
kgriffsbut we will be using our copy in the marconi repo anyway18:31
cppcabreraI remember that part, kgriffs. :)18:31
* cppcabrera watches the mailing lists diligently18:31
zyuan_for nwo18:31
malinioz_akan_ :  looks good so far18:32
malinitrying messages & queue tests now18:32
kgriffsoz_akan: would you prefer redis or mongo for the catalog?18:32
kgriffs(caching aside)18:33
oz_akan_malini: yay!18:33
oz_akan_kgriffs: mongo18:34
kgriffsok, I am going to merge that patch. got a "verbal" LGTM from flaper87 already18:34
kgriffs(index patch)18:34
kgriffsthere it goes!18:34
kgriffscppcabrera: ok, i say then we do a mongo driver and sqlalchemy driver for catalog backends in the proxy18:35
kgriffsthen we can add redis later, similar to what we are doing for marconi-queueing18:36
kgriffsi think a localhost memcached for the proxy box will be fine for our cache18:37
kgriffsyou might also add an in-process LRU for good measure, but not required18:37
kgriffs(saying this because oslo.cache already has a memcached driver)18:37
cppcabreraalright - hierarchical caching, mongo catalog master, and some form of local caching.18:38
oz_akan_kgriffs: what about redis as cache on proxies? hasn't cppcabrera implemented like that at the moment?18:38
kgriffsI plan to work on an LRU backend for oslo.cache that will support expiration18:38
cppcabreraI have implemented that so far.18:38
oz_akan_kgriffs: I was planning to replace memcached with redis on web servers after testing the performance18:38
kgriffsmostly i want us to use oslo.cache rather than rolling our own caching library18:39
cppcabrera+1 kgriffs18:39
kgriffswe can start with the memcached driver since redis supports the memcache protocol18:39
oz_akan_oki doki, sounds good18:39
kgriffscppcabrera: feel free to write a native redis driver for oslo.cache18:39
cppcabreraI can see the implementation changes now18:39
oz_akan_where can I read more about oslo.cache?18:40
kgriffswe will keep our drivers in marconi repo until oslo.cache is merged and then port upstream18:40
openstackgerritA change was merged to stackforge/marconi: fix: Requests get slower when queues have a lot of messages
cppcabreraoslo.cache.get_multi/set_multi with writes to a primary mongo.18:40
cppcabreraThe updates shouldn't be too difficult.18:40
cppcabreraWe just need to get oslo.cache into marconi18:41
kgriffscppcabrera: already there, man!18:43
cppcabreraweird... it's not in my tree.. hmm...18:45
cppcabreranevermind... I'm looking at a branch that needs rebasing18:46
cppcabreraThanks, kgriffs18:46
kgriffsbtw, can you review this latest draft?18:47
malinioz_akan_ : all the tests look good18:47
oz_akan_malini: thank yo18:48
openstackgerritKurt Griffiths proposed a change to stackforge/marconi: perf: Partition messages collection by project
openstackgerritZhihao Yuan proposed a change to stackforge/marconi: feat(mongo): use UNIX timestamp instead of datetime
zyuan_so, kurt wrote a per-project message collection patch19:10
zyuan_oz wrote a hashed message collection patch19:10
kgriffszyuan_: actually, oz is hashing to DB19:12
kgriffsnot collection19:12
kgriffsso, per mongo cluster you could have, say 4 databases19:12
zyuan_i reviewed it many times19:12
kgriffslet me look19:12
kgriffsthat is a bug19:13
kgriffsit should be picking DB, not collection19:14
kgriffslocks are per-db, not per-collection19:14
kgriffsoz_akan_: ^^^19:14
zyuan_benchmark says everything19:14
kgriffsi'm not saying there isn't a benefit to partitioning collection (hence my patch)19:14
kgriffsbut there is also a benefit for partitioning dbs19:15
kgriffsthat was the original goal of oz's patch - not sure how we got our wires crossed19:15
zyuan_and, per-project collection brings a heavy overhead on indexing i guess19:15
kgriffs(goal was avoiding db lock)19:15
zyuan_there is no lock between collections i guess19:15
kgriffszyuan_: actually, lower overhead on indexing since the indexes are less complex, smaller19:15
kgriffs(pulling project out of the index)19:15
zyuan_if they have lock on collection, then why mongo does no support cross-collection query?19:16
zyuan_it doesn't make sense, right?19:16
* flaper87 back19:16
zyuan_locks are per-db19:19
cppcabrerawb, flaper87!19:19
* flaper87 waits19:19
kgriffslooks like
flaper87cppcabrera: danke19:19
kgriffshe is using multiple dbs19:19
kgriffslet me look over the entire patch19:20
flaper87btw, re that patch19:20
flaper87do we really need those 2 third-party libs ?19:20
flaper87I haven't looked at the patch yet19:20
flaper87so, I may be saying something stupid19:20
flaper87but, I'm just concerned about the dependencies list and the fact they have to be added to openstack/requirements19:20
zyuan_kgriffs: he opened two connection to 1 db19:20
zyuan_team: do not merge , i haven't tested it after rebasing19:22
kgriffslets see19:22
kgriffsconn == client19:22
zyuan_it heavily conflicted with kurt's patch which was just merged19:22
kgriffsso _messageS_databases is a list of DB objects, marconi_m1, marconi_m2, etc.19:23
kgriffsbut, this looks wrong19:24
zyuan_kgriffs: ?19:24
kgriffsjust a minute19:24
zyuan_just look at this line19:25
zyuan_if you want to use multiple db, you have to have multiple uri19:25
zyuan_1 URI, 1 db19:25
kgriffszyuan_: you are confligrating db and server/cluster19:26
zyuan_kgriffs: you mean?19:26
kgriffsa single MongoDB process can run multiple DBs19:26
kgriffsso, you can have a single connection, multiple dbs19:26
kgriffsthink of db is just a kind of namespace19:26
zyuan_that's collection...19:27
kgriffsit's not entirely accurate, but helps me thing about it19:29
kgriffsclient = pymongo.MongoClient(options.CFG.uri)19:29
* cppcabrera is having a bit of fun benchmarking pypy + gunicorn + marconi-proxy (current) + marconi-mongo on localhost19:29
zyuan_the first level is db...19:31
zyuan_then have we ever used collection?19:32
zyuan_i see... currently we we 'marconi' db19:33
zyuan_and 'message'19:33
zyuan_'queues', etc...19:33
openstackgerritKurt Griffiths proposed a change to stackforge/marconi: perf: Partition messages collection by project
zyuan_kgriffs: oz_akan_ 's patch is right19:35
zyuan_he used multi db19:35
*** whenry has quit IRC19:35
zyuan_but! he collected collections from multi db into the old self._col19:35
kgriffsflaper87: need a review on this:
zyuan_which makes it 'looks' like using multiple collections19:36
flaper87kgriffs: will do19:36
flaper87kgriffs: does that patch depends on oz_akan_'s ?19:36
kgriffsthanks! I imagine it will take a few iterations to get right19:36
kgriffsrather, i think we will need to make oz_akan_'s depend on this one19:36
kgriffs(see my comment)19:36
flaper87(sorry guys, I haven't been part of today's discussions, it was my many meetings day)19:36
flaper87kgriffs: kk19:36
kgriffsheh, it happens to the best of us19:36
kgriffs(my comment on oz's patch)19:37
flaper87seriously, meetings destroy my brain, ideas or whatever could happen in there19:37
*** whenry has joined #openstack-marconi19:39
cppcabreraflaper87: meetings slow me down quite a bit, too. That was the case for me yesterday.19:41
cppcabreraGET /v1/queues/{queue}/stats appears to be one of our slowest endpoints. (it's slower than POST /v1/queues/{queue}/messages)19:46
cppcabreraThis is not altogether surprising, given that /stats makes multiple DB queries.19:47
flaper87cppcabrera: counts19:48
* flaper87 should work on that bug19:49
cppcabreraon my localhost pypy marconi-proxy tests, it's about... (checks)19:49
cppcabrera35x slower than POST messages. :P19:49
*** malini is now known as malini_afk19:50
cppcabreraThis includes pypy priming/warm-up for both the routes.19:50
*** gordonsim has quit IRC19:50
cppcabreraquick note regarding the current marconi-proxy patch: /v1/queues/{queue}/stats is missing. (zyuan, zyuan_)19:56
cppcabreraI accidentally left that one out. :P19:57
openstackgerritAlejandro Cabrera proposed a change to stackforge/marconi: feat: marconi proxy
cppcabreradisregard above: rebasing patches20:18
oz_akan_zyuan_: "from pylru import lrudecorator"20:19
oz_akan_do I need to chagne it to import pylru ?20:19
zyuan_oz_akan_: i think os...20:20
zyuan_and just use pylru.lrudecorator20:20
oz_akan_ok got it, if that is the os way, must be then20:20
zyuan_cppcabrera: i guess i'd better starts with testing the non-marconi api part :)20:21
cppcabrerathat'd be best, zyuan_. :)20:22
cppcabrera/v1/partitions and /v1/catalogue20:22
cppcabrerawhere /v1/catalogue is primarily controlled through /v1/queues/{queue}20:23
openstackgerritAlejandro Cabrera proposed a change to stackforge/marconi: feat: marconi proxy
openstackgerritAlejandro Cabrera proposed a change to stackforge/marconi: feat: marconi proxy (v1, health)
* cppcabrera is learning how to rebase dependent patches20:32
zyuan_cppcabrera: easy, all-in-once20:37
openstackgerritAlejandro Cabrera proposed a change to stackforge/marconi: feat: marconi proxy (v1, health)
openstackgerritAlejandro Cabrera proposed a change to stackforge/marconi: feat: marconi-proxy forwarding
oz_akan_kgriffs: is very slow20:43
oz_akan_I commented20:43
oz_akan_0.47 ms vs 0.14 sec20:44
openstackgerritAlejandro Cabrera proposed a change to stackforge/marconi: feat: marconi proxy
openstackgerritAlejandro Cabrera proposed a change to stackforge/marconi: feat: marconi proxy (v1, health)
kgriffsoz_akan_: really? that's interesting20:47
openstackgerritAlejandro Cabrera proposed a change to stackforge/marconi: feat: marconi-proxy forwarding
oz_akan_kgriffs: I pasted test results20:48
kgriffsi guess that's what benchmarks are for, after all20:48
cppcabrerathere - rebasing + minor bug fixes completed for marconi-proxy.20:48
cppcabreraI'm done spamming this room with review requests. :P20:48
oz_akan_this is 50 users 40 seconds and 100 users 40 seconds benchmark20:48
cppcabrera+1 for your benchmark efforts, oz_akan_20:48
oz_akan_with a single project id20:48
kgriffshmm, interesting20:48
kgriffsI suspect that at extremely large scale this patch may actually perform better, but that is hard to know until we have the ability to test it20:49
kgriffslet me put this on the backburner20:49
oz_akan_for one project if it was a bit slower I think it would be acceptable20:49
kgriffsit's probably mostly due to the lookup cost to get the collection to use20:49
oz_akan_yes lookup is slow, that is why I used lru cache for database mapping20:50
oz_akan_and using 10 db is slow than using 2-420:50
kgriffsI also use lru cache, but it still adds overhead20:51
kgriffsoz_akan: did you see my comment?20:51
kgriffsi think your perf will do much better if you only partition on dbs, not collections20:51
oz_akan_on lru ?20:51
kgriffsoz_akan_: no, in gerrit20:51
kgriffson your patch20:51
oz_akan_yes, I believe  I replied let me check20:52
openstackgerritKurt Griffiths proposed a change to stackforge/marconi: chore: Update oslo.config to version 1.2.0a4
oz_akan_zyuan_: thank your last comments20:54
oz_akan_zyuan_: thanks for your last comments20:54
cppcabreraRegarding oslo 1.2.0a4: (kgriffs, flaper87)20:55
cppcabreraIt;s nice seeing the release notes.20:55
cppcabrera"Finished Python 3 support"20:55
flaper87cppcabrera: that moves us a step forward to support P3K20:56
kgriffsoz_akan_: saw your comment, let me take another look20:56
cppcabreraJenkins doesn't like the latest oslo.config...20:57
kgriffssorry, should have finished reading all the changes before i commented20:57
kgriffsi got mixed up since i was thinking about my patch at the same time. :p20:57
cppcabreraIt seems like the embedded server isn't launching for the functional tests with oslo.config-1.2.0a420:59
cppcabrera"ConnectionError: HTTPConnectionPool(host='', port=8888): Max retries exceeded with url: /v1/queues?detailed=True (Caused by <class 'socket.error'>: [Errno 111] Connection refused)"20:59
flaper87cppcabrera: mmhh20:59
kgriffsoz_akan_: so, a suggestion21:00
kgriffsto simplify your code (DRY)21:00
*** malini_afk is now known as malini21:00
kgriffsmake a helper to give you the correct collection given queue_name, project21:00
oz_akan_apart of self.driver.db_id  ?21:00
kgriffscollection = self._collection(queue_name, project)21:01
oz_akan_I see, I will return collection directyl21:01
kgriffsI just noticed that you have to do self.driver.db_id followed by _col[db_id] all over the place21:01
kgriffsjust factor those out into a helper21:01
flaper87cppcabrera: any idea why ?21:01
oz_akan_got it21:01
kgriffsyou could even lru-decorate that helper21:02
kgriffs(may or may not be faster, depends on impact of function-call overhead)21:02
oz_akan_driver.db_id has the decorator21:02
kgriffsi mean, at a higher level21:03
kgriffsmove your decorator up to the helper we just discussed21:03
kgriffsso it will memoize the array lookup as well21:03
cppcabreraflaper87: I have no idea. :/21:04
flaper87cppcabrera: can you run marconi-server ?21:04
kgriffsoz_akan: p.s. - db_id() can be a free function, maybe part of utils.py21:04
oz_akan_kgriffs: makes sense21:04
oz_akan_about db_id I didn't know where to put it21:05
oz_akan_if you think so, sure I will move it there21:05
kgriffsi think it makes sense21:05
cppcabreraflaper87: I'll try to from the oslo.config++ branch21:05
kgriffsyou aren't accessing anything with self21:05
kgriffsand utils is a good place to put things that may be (potentially) used in different modules.21:06
oz_akan_I was thinking it might be faster like this as it is part of the object... but I understand it doesn't really belong there21:07
oz_akan_thanks kgriffs I am heading home21:08
cppcabrerakgriffs: when you get a chance, can I get a quick review on ? (needed to advance marconiclient dev)21:09
cppcabreratiny patch21:09
*** oz_akan_ has quit IRC21:09
kgriffscppcabrera: the issue is we don't wait long enough for the server to boot21:09
kgriffsI was seeing that too21:09
kgriffsnot related to the other patch imo21:10
cppcabrera"error: Could not find required distribution oslo.config>=1.2.0a4"21:10
kgriffs(the update for oslo.config)21:10
*** oz_akan_ has joined #openstack-marconi21:10
cppcabreraI got that from 'git review -d $OSLO_PATCH'21:10
kgriffsoh, interesting21:10
cppcabreraafter doing python develop21:10
flaper87why are we increasing oslo's version?21:10
kgriffsflaper87: mostly testing it to see if we have any issues, but also py3k support, right?21:11
cppcabrerafound the problem21:11
cppcabreraOne '3' needs to become a '4'21:12
* kgriffs blushes21:12
flaper87kgriffs: sounds good, I was just curious if there was another need bsides those21:12
*** oz_akan_ has quit IRC21:15
openstackgerritKurt Griffiths proposed a change to stackforge/marconi: chore: Update oslo.config to version 1.2.0a4
flaper87guys, you gotta wait for Jenkins bless :P21:18
cppcabrera+1 flaper87. :)21:19
kgriffsflaper87: yep21:19
cppcabrerajenkins > core21:19
kgriffshmm, that is odd21:20
kgriffswhy are we getting 204 now - shouldn't have anything to do with the config upgrade21:20
cppcabrerapypy succeeds... >.>21:22
flaper87try recheck no bug21:22
flaper87lets see if py27 fails again21:23
cppcabrerapython26 succeeds, too21:23
flaper87if it doesn't then it's random and I'll have to dig more21:23
* flaper87 hates random behaviours in functional tests21:23
openstackgerritKurt Griffiths proposed a change to stackforge/marconi: perf: Partition messages collection by project
kgriffsoz_akan: can you benchmark this? Made a little change to lru just out of curiosity21:26
kgriffskk folks, I gotta hit the road - I'll be back online in ~1 hour21:27
kgriffsoh darn21:27
kgriffsmissed Oz21:27
flaper87kgriffs: have a good one21:28
kgriffsthanks, ttfn21:28
* kgriffs throws pop-tarts everywhere and makes a mad dash out the door21:28
cppcabreraI'm out, as well.21:34
cppcabreraGood night, everyone. :)21:34
flaper87cppcabrera: night21:37
* flaper87 ate all those pop-tarts21:37
*** kgriffs is now known as kgriffs_afk21:37
*** tedross has quit IRC21:43
*** cppcabrera has quit IRC21:44
*** kgriffs_afk is now known as kgriffs22:16
*** oz_akan_ has joined #openstack-marconi22:20
*** oz_akan_ has quit IRC22:25
*** amitgandhi has quit IRC22:30
*** malini is now known as malini_afk22:34
*** flaper87 is now known as flaper87|afk22:42
openstackgerritKurt Griffiths proposed a change to stackforge/marconi: fix(tests): unit tests disabled due to missing
openstackgerritA change was merged to stackforge/marconi: fix(tests): unit tests disabled due to missing
openstackgerritKurt Griffiths proposed a change to stackforge/marconi: perf: Partition messages collection by project
*** kgriffs is now known as kgriffs_afk23:45

Generated by 2.14.0 by Marius Gedminas - find it at!