Wednesday, 2013-08-28

*** openstack has joined #openstack-marconi18:46
kgriffsin that case, you would have to either have atomic id/marker generation (ala autoinc key) or you would need to pipe all inserts for a given queue through a serializing proxy thingy18:47
kgriffsif you wanted to guarantee FIFO18:47
kgriffscurrently, the spec does not require FIFO for multiple producers18:47
kgriffsif you want to spread out a single queue across multiple storage nodes, you unfortunately can't just use a timestamp-based id/field to order by, since it is easy to get race conditions under heavy load18:48
kgriffsBTW, SQS does not guarantee ordering at all18:49
kgriffsmakes their lives a lot easier. :p18:49
torgomatic(just so I'm clear here) so, even though the insert order for one producer is A1, A2, A3, the spec doesn't guarantee getting A1, then A2, then A3 out?18:49
torgomatic(not complaining, just asking)18:49
torgomaticand that's because there's a second producer at the same time?18:49
kgriffswell, it depends on how close both producers are18:50
kgriffsif they interleave with gaps of, say, 1 seconds or so they likely won't get out of order18:50
torgomaticokay, I think I see18:51
*** meganw has quit IRC19:01
*** meganw has joined #openstack-marconi19:01
*** meganw has quit IRC19:06
kgriffstorgomatic: rock on.19:13
openstackgerritKurt Griffiths proposed a change to stackforge/marconi: chore: Add the up-and-coming oslo.cache module
kgriffsguys, need some reviews on these19:14
*** JRow has joined #openstack-marconi19:20
*** JRow has quit IRC19:34
kgriffsAlex_Gaynor: thought you'd like to know that pylru is 47x faster for a lookup under pypy vs. py2719:54
* kgriffs loves pypy19:54
Alex_Gaynorkgriffs: awesome19:54
flaper87kgriffs: pong20:00
* flaper87 is back20:00
acabreraI'm back from the tech talk. :)20:02
*** acabrera is now known as cppcabrera20:02
cppcabreraGot my name back, too. :P20:02
cppcabrerakgriffs: The *only* downside to pylru is that it doesn't support python 3. It's extension building approach is incompatible with py3k.20:03
cppcabreraI do love how speedy it is compared to available lru cache options.20:03
cppcabreraI tried porting it over the weekend, but kept running into some exceptions at import time when I built the source modified with ifdefs to handle py3k.20:04
*** malini is now known as malini_afk20:04
* cppcabrera has little to no extension building experience in python, fyi20:04
*** meganw has joined #openstack-marconi20:07
*** meganw has quit IRC20:07
*** meganw has joined #openstack-marconi20:08
kgriffsflaper87: were you already working on the change to remove the range queries from _list ?20:09
kgriffs(and remove threshold "feature" from gc)20:10
flaper87kgriffs: nope, today I focused on the mapping and then got pulled into thousands of meetings T_T20:11
flaper87kgriffs: I can do it tomorrow20:11
kgriffsok, I couldn't remember who was going to work on it20:12
flaper87and then focus on the test change20:12
flaper87kgriffs: whe never said it :D20:12
flaper87so, either would be great20:12
kgriffsno wonder!20:12
kgriffsI can take it20:12
kgriffsin other news, see my patches to pull in cache from oslo20:12
kgriffsfrom future import oslo.cache20:13
kgriffsflaper87: do you have a favorite Python LRU library?20:13
kgriffsI would like to implement an LRU backend and then create a hierarchical meta-backend20:14
kgriffs(with write-through support)20:14
kgriffscppcabera: can you +2 those oslo update patches?20:20
kgriffscppcabrera: can you review those?20:21
flaper87kgriffs: mmh, TBH, I don't have a preferred. Last time I needed one, I worte it myself20:21
kgriffscppcabrera: i may have missed a namespace change, so need some eyes on it20:21
* cppcabrera goes to check on the growing review queue20:23
cppcabreraIt's been a meetings kind of day. I'm lucky I found a good hour this morning to modularize marconi-proxy. :P20:23
flaper87quick english question20:24
flaper87why do spell checkers tell be I should put a hypem in backend ?20:24
meganwthey do that to me too!  but its wrong.  one word.20:25
kgriffsI suppose you can make it two words20:26
kgriffs"back end"20:26
amitgandhiinteresting, they use with space, without, and with hypen20:26
cppcabrerathe use of hyphens is one of those debated things in English, IIRC. Most of the time, when a word could take a hyphen, it can also be used with a space.20:27
kgriffsbut seems like the colloquial spelling is to remove the space20:27
flaper87I used to write it as backend 'til those stupid spell checkers consued me20:27
meganwgotta stand up to the man on these things20:27
kgriffs"common misspelling"20:28
cppcabrera+1 meganw20:28
cppcabreraI'm all for theatre and catalogue which sets off english (US) spell checkers. ;)20:29
cppcabreraAlso, any time spanish sneaks into my english - yup. :P20:29
kgriffsflaper87: what do you think about cache backends normalizing exceptions, so that the caller doesn't have to have except blocks based on backend-specific types?20:32
flaper87kgriffs: I think that's a good idea, however, IIRC, the api doesn't raise any exception yet20:57
kgriffsI was just looking at the memcached backend20:59
kgriffslooks like it just propagates any exceptions that are raised from python-memcached21:00
flaper87kgriffs: mmh, if it crashes really bad, yup, that's current case. The methods we're using from python-memcache shouldn't raise any exception21:01
flaper87AFAIC :/21:01
flaper87damn, I can't type today21:01
flaper87except for decr and incr21:02
flaper87I agree on normalizing exceptions, though21:02
flaper87like catching bad things, printing a nice log and re-raising a cache-op exception21:03
flaper87or something like that21:03
flaper87kgriffs: did you hit any case where an exception should be raised?21:03
*** meganw has quit IRC21:04
*** meganw has joined #openstack-marconi21:05
*** malini_afk is now known as malini21:05
kgriffsit was with ultramemcache I hit that21:06
kgriffsbut judging by the python-memcached code it handles exceptions for you21:06
*** meganw has quit IRC21:09
*** meganw has joined #openstack-marconi21:09
kgriffslooks like redis-py does raise exceptions, tho21:09
kgriffsflaper87: ^^^21:10
kgriffsso, the cache abstraction should define behavior there.21:10
kgriffsas in, should it fail silently and mark the node as "dead" or what?21:10
flaper87kgriffs: yeah, I guess it should21:14
flaper87it should try to reconnect21:14
flaper87or find a new valid node21:14
flaper87IMHO, what do you think?21:14
torgomaticI'd reconnect, but that's just me21:15
flaper87torgomatic: me too :)21:15
torgomaticif you try to find a new valid node, then you can get inconsistent cache21:15
torgomaticeither from different ideas of "next valid node", or even node A can connect to cachebox1, but node B can't21:16
flaper87torgomatic: btw, did you take a look at swift's column here?21:16
flaper87do my and cppcabrera comments make sense?21:16
torgomaticflaper87: I did; I've been thinking over ways to do that and take advantage of multiple containers for performance, though21:17
torgomaticbut what you've said is correct, and I believe that 1-1 mapping of queue to container will give you the semantics you want, subject to the usual eventual consistency of Swift21:17
flaper87torgomatic: cool21:18
cppcabreratorgomatic: The using multiple containers to handle very large queues idea has come up in general marconi discussions as well. Even in mongo, breaking down a large queue into sub-queues transparently would be a benefit to latency in some cases. If you come with any ways to tackle that, let us know. :)21:19
torgomaticcppcabrera: will do :)21:19
kgriffsguys, a quick approval:
kgriffs(updates oslo)21:19
flaper87kgriffs: I was thinking to give a bit of extra priority to that backend, what do you think? (once we figure out the API thing)21:19
torgomaticif I don't have to order the results of a list-message request in any particular way, it's easy ;)21:19
* flaper87 would love to take care of swift's backend21:19
flaper87it would be an amazing opportunity to get to know it better21:19
kgriffsflaper87: not sure on the priority yet. I'd like to find out what people are most interested in for backends21:21
*** ayoung has quit IRC21:21
flaper87kgriffs: I think it is important for us to support at least 1 other backend != mongodb21:21
flaper87kgriffs: it could be that or a rel db21:21
flaper87I'd love it to be a MB21:22
flaper87but we need to figure out the API issue21:22
flaper87and what's actualyl possible, valid and reasonable21:22
flaper87cool, what aboout putting it on next week's meeting agenda ?21:22
cppcabreraI'm thinking of working on one of two things this Friday: Redis (finish it) or zmq transport (start it). There's a hackday at the office this friday. :)21:22
flaper87kgriffs: -1 some files missing21:23
flaper87cppcabrera: redis :)21:23
flaper87you could start having it in a separate repo21:23
flaper87which will be a proof of marconi's support for third-party modules21:23
kgriffsflaper87: added to agenda21:23
flaper87once we figure out what modules should live in Marconi's tree, we could pull it in21:24
cppcabreraI see, to show how someone else might add a third party moodule, hmm...21:24
flaper87cppcabrera: yup21:24
cppcabreraHow would that plug into stevedore?21:24
flaper87pip install marconi-redis-backend21:24
flaper87cppcabrera: entry_points:
flaper87cppcabrera: you'd have something like: redis =
cppcabreraAhhh, I get it.21:25
flaper87(the packages can be whatever you want)21:25
zyuan_i'll love if it's hyperdex backend :)21:25
cppcabreraentry points uses a module import syntax21:25
cppcabreraSo as long as the entry point entry is installed on the python module path... got it!21:26
flaper87cppcabrera: exactly21:26
cppcabreraAlright, I'll give it a spin.21:26
flaper87cppcabrera: another cool project for your hackday could be the client :P21:26
cppcabreraHahha, that's #3 on my list. :P21:27
cppcabreraI like using hack days to explore the unknown - it's the only reason the client is lower on my list for that day. :)21:27
flaper87cppcabrera: I was joking anyway, redis backend should be fun21:27
flaper87cppcabrera: +121:27
*** malini is now known as malini_afk21:30
*** oz_akan_ has quit IRC21:31
*** oz_akan_ has joined #openstack-marconi21:31
*** oz_akan_ has quit IRC21:36
kgriffsflaper87: re fileutils and excutils, pulled those in automagically, so I didn't think they were needed in openstack-common.conf21:37
flaper87kgriffs: yeah, basically can use that file to know which modules to update21:38
flaper87that script is still stupid, I guess21:38
flaper87the whole process is really dumb21:38
*** meganw has quit IRC21:50
*** meganw has joined #openstack-marconi21:51
*** meganw has quit IRC21:55
cppcabreraI'm out for the night. Take care, guys. :)21:56
*** cppcabrera has quit IRC21:56
*** kgriffs is now known as kgriffs_afk21:59
*** fifieldt_ has joined #openstack-marconi22:00
*** meganw has joined #openstack-marconi22:03
*** meganw_ has joined #openstack-marconi22:03
*** meganw_ has quit IRC22:05
*** meganw_ has joined #openstack-marconi22:05
*** meganw_ has quit IRC22:07
*** meganw_ has joined #openstack-marconi22:07
*** meganw has quit IRC22:07
*** meganw_ has quit IRC22:12
*** flaper87 is now known as flaper87|afk22:20
*** amitgandhi has quit IRC22:23
*** oz_akan_ has joined #openstack-marconi22:42
*** key4 has quit IRC22:44
*** key4 has joined #openstack-marconi22:45
*** oz_akan_ has quit IRC22:47
*** kgriffs_afk is now known as kgriffs22:59
*** fifieldt_ is now known as fifieldt23:03
*** jergerber has quit IRC23:24
*** kgriffs is now known as kgriffs_afk23:34
*** amitgandhi has joined #openstack-marconi23:45

Generated by 2.14.0 by Marius Gedminas - find it at!