Thursday, 2013-08-29

*** kgriffs_afk is now known as kgriffs00:04
*** kgriffs is now known as kgriffs_afk00:52
*** kgriffs_afk is now known as kgriffs00:54
*** malini_afk is now known as malini01:16
*** amitgandhi has quit IRC01:16
*** kgriffs is now known as kgriffs_afk01:20
*** kgriffs_afk is now known as kgriffs01:24
*** kgriffs is now known as kgriffs_afk01:31
*** oz_akan_ has joined #openstack-marconi01:45
*** oz_akan_ has quit IRC01:46
*** oz_akan_ has joined #openstack-marconi01:46
*** kgriffs_afk is now known as kgriffs02:26
*** oz_akan_ has quit IRC02:30
*** oz_akan_ has joined #openstack-marconi02:30
*** oz_akan_ has quit IRC02:35
*** malini is now known as malini_afk02:40
*** kgriffs is now known as kgriffs_afk02:46
*** vkmc has quit IRC03:11
*** oz_akan_ has joined #openstack-marconi03:41
*** oz_akan_ has quit IRC03:46
*** key4 has quit IRC06:58
*** key4 has joined #openstack-marconi06:58
*** flaper87|afk is now known as flaper8707:12
*** _alexr_ has joined #openstack-marconi07:35
*** ykaplan has joined #openstack-marconi09:23
*** ykaplan has quit IRC10:22
*** fifieldt has quit IRC10:29
*** JRow has joined #openstack-marconi11:55
*** jergerber has joined #openstack-marconi12:30
*** oz_akan_ has joined #openstack-marconi12:58
*** oz_akan_ has quit IRC12:59
*** oz_akan_ has joined #openstack-marconi13:00
*** _alexr_ has quit IRC13:25
*** _alexr_ has joined #openstack-marconi13:25
*** _alexr_ has quit IRC13:30
*** _alexr_ has joined #openstack-marconi13:33
*** kgriffs_afk is now known as kgriffs13:33
*** amitgandhi has joined #openstack-marconi13:38
*** amitgandhi has quit IRC13:40
*** amitgandhi has joined #openstack-marconi13:40
*** malini_afk is now known as malini13:51
kgriffsmalini: can you insert some messages into the test db?13:52
kgriffsI just need to know the queue name and project ID13:53
maliniI'll insert to q18 , 80606713:54
malini35K messages is good  ?13:54
malinikgriffs: it shud have 36K messages now14:00
oz_akan_kgriffs: I am working on keystone issue so test environment is not functionally now14:00
oz_akan_malini: ^^14:00
oz_akan_trying to get rid of the middleware to see the performance without it14:01
oz_akan_I will let you know when it is back14:01
malinioz_akan_ : ok..I got back 201s anyways ;)14:02
*** cppcabrera has joined #openstack-marconi14:03
oz_akan_malini:  please don't run anything, as all queries will go to identity14:04
oz_akan_there is no caching14:04
cppcabreraGood morning!14:04
malinioz_akan_ : I wont do anything, till you ask me to14:04
kgriffsi am just executing queries directly on the master db14:04
oz_akan_malini: tis, I will let you know14:05
flaper87malini: morning14:05
flaper87I commented on your patch14:05
maliniflaper87: thanks!!!14:05
oz_akan_kgriffs: sure that is fine, just we won't be able to insert data through marconi for a while14:05
maliniI was just checking that out14:05
flaper87it'd be really great if you could address those comments soon14:05
flaper87malini: I already applied my work to your patch14:05
maliniflaper87: I am working on tht, right now :)14:05
oz_akan_cppcabrera: flaper87 good morning and afternoon14:05
flaper87meaning, I made it depend on yours14:05
flaper87oz_akan_: cppcabrera kgriffs morning guys14:06
maliniflaper87: I am on vacation most of next week & coming back on Friday..I would love to get this merged before I go14:06
cppcabreraI'm going to check out the marconi review queue once I finish sorting through morning email. There's plenty pending there that needs review, last I checked.14:06
flaper87malini: yes, please14:06
cppcabreraI'll give your patch priority, malini.14:06
flaper87I promisse to get my things merged before you come back14:06
flaper87as a wb surprise14:07
flaper87so, I split the test package14:08
flaper87ran tests and it just works14:08
cppcabreraQuick question on a comment for malini's patch, flaper87: why setUp instead of setUpClass for operations that make sense to run once per test suite, rather than once per test case?14:09
cppcabrera+1 flaper87 :D14:09
malinicppcabrera: thanks for asking that :D14:09
cppcabreraSame goes for the tearDown comment. :)14:09
malinilike the headers & stuff, we can just do it once14:10
flaper87cppcabrera: good question, I should've written the motivation of my comment there. Sorry.14:11
cppcabreraNo worries, flaper87, so long as we understand the rationale. :)14:12
flaper87I personally consider the provision of the test environment as something that should run for every test. Meaning, every test should be ensured to have a clean env to work on14:12
flaper87so, example14:12
flaper87If test_a writes some messages into the queue created for the whole test suite and fails, test_b will get those messages too14:13
flaper87and that may break tests in test_b14:13
Alex_GaynorIf anyone has a few minutes, is the first step to getting marconiclient on pypy as well14:13
flaper87Alex_Gaynor: +114:14
cppcabrera+1 Alex_Gaynor14:14
Alex_Gaynor(actually it's the only step, next one after this is adding ito the gate :D)14:14
cppcabreraI see what you're saying flaper87.14:14
maliniflaper87: hmm..but tht implies each claim test, should have its own queue ?14:15
cppcabreraGiven that the three of us have knowledge of the test operations, I would call the setUpClass an optimization, since expensive operations like calling across the network are performed only once. However, with that optimization in place, if a test were to delete one of the queues created during setUpClass by a new contributor, and we failed to catch that during review, what's the cost?14:15
malinicppcabrera: then it's a bug in the test ;)14:16
flaper87malini: that implies each claim test will get an empty queue14:16
flaper87that's part of keeping tests isolated14:16
flaper87we don't want test_b to fail if test_a fails14:16
cppcabreraPUTing a queue doesn't delete its messages/claims, though, if it already exists. :x14:17
cppcabreraI love the idea of test isolation, though. I agree there.14:17
flaper87cppcabrera: that's what tearDown should do, right?14:17
cppcabreraAhh, I see.14:17
cppcabreraI see now.14:17
maliniwhat abt keeping the setupclass for headers etc. , but use setup for each queue creation etc. ?14:17
flaper87malini: +114:18
flaper87that makes sense to me14:18
malinibut, what will happen if we ever have these tests running in parallel ?14:18
cppcabreraflaper87: that's where I advocate using a context manager for creating messages/claims/queues, depending on what's being tested, and setUp/tearDown for consytructing/deleting the primary resource being tested.14:18
cppcabreramalini: ^14:19
flaper87malini: in that case we can create a queue per test14:19
maliniwe'll have multiple tests trying to create the queue etc. - unless we use diff queue names for each test14:19
cppcabrerause the queue name /v1/queues/{uuid.uuid1()}14:19
flaper87cppcabrera: +114:19
maliniok..I'll make those changes14:20
flaper87malini: awesome!14:20
cppcabrerawoot. :)14:21
cppcabreraThen there's the config/base class suggestion14:21
cppcabreraHow do you guys feel about putting that off 'til another patch?14:21
cppcabreramalini, flaper87: ^14:21
malinicppcabrera: why ?14:21
flaper87mmh, I'd prefer having it in the same patch, it's part of system -> functional refactor14:22
flaper87and reduces the LOC to review14:22
cppcabreraYeah, good point flaper87.14:22
* flaper87 lazy14:22
cppcabreraI had forgotten the name of the patch, TBH. :P14:22
maliniflaper87: I dont really understand the whole thing on tht comment..I'll have questions once I get to that14:23
flaper87malini: sure14:24
cppcabreraHanging out in #haskell is fun, flaper87. There's lambdas, type signatures, and -> every where. :P14:30
flaper87cppcabrera: yeaaaah, and haskell's community is pretty nice14:31
flaper87same happens in #rust14:31
cppcabreraSeems like it. :)14:31
cppcabreraOoohh, #rust - young and budding.14:31
cppcabreraI haven't been there yet.14:31
flaper87nice community, nice language as well14:32
kgriffsI've been watching rust, trying to find an excuse to try it. :p14:32
flaper87cppcabrera: mmh, btw, I should probably have mentioned that #rust is in irc.mozilla.(something)14:32
kgriffsflaper87: remind me why we index first by queue name, not by project?14:33
kgriffsisn't project going to be more selective?14:33
cppcabrerathanks for the directions, flaper87. :)14:33
flaper87kgriffs: not a real reason. there was a good reason for queue's id14:33
kgriffswhat do you mean by a "good reason"?14:34
flaper87the reason why queue's id was the first in possition is because we could have used it for filtering and nothing else was needed14:35
flaper87meaning, just using queue_id in the query14:35
kgriffsok. Let me see if we actually use that anywhere14:35
flaper87as for queue_name and project, I honestly don't know which one would be better14:35
flaper87kgriffs: I don't think we're14:35
flaper87I guess queue_name should go first anyway14:36
flaper87the probability of having the queue name N times in the index is lower than the probability of having gazillions of queues under the same project14:37
zyuan_flaper87: i agree14:38
zyuan_we can leave queue name and project in queues14:39
zyuan_but not one messages; messages can just join then together14:39
cppcabrerakgriffs: re: the oslo.cache fork you're adding in - does it add anything to the original oslo.cache package, or is it a verbatim copy for the time being?14:40
kgriffsI will update those patches in a bit; right now I am working on the perf patch14:40
cppcabreraI'm just running a round of morning reviews. ;)14:41
cppcabreraI plan on making that part of my routine.14:41
flaper87cppcabrera: +114:42
flaper87malini: would you be mad if I break your changes in my test split patch?14:44
flaper87(i'm obviously joking)14:44
openstackgerritZhihao Yuan proposed a change to stackforge/marconi: fix: claimed message require claim_id to delete
maliniflaper87: yes 8o|14:44
maliniseriously ;)14:45
zyuan_can you review this ?14:45
cppcabreraAlright, all marconi patches have gotten my review. :D14:50
cppcabreraThe client side needs a little love.14:50
cppcabreraflaper87: Do you have a spare moment to take a look at reverting common lib ( and checking out the message controller (
maliniflaper87, cppcabrera: It does seem to get slower with test level setup14:53
flaper87cppcabrera: yup, I'll review those14:53
flaper87malini: I wouldn't be so worried about that, though14:53
cppcabreramalini: It will. Instead of sending http requests once per test suite, it'll do it once per test cases. As flaper87 it saying, though, that's okay.14:53
flaper87that's the tradeoff we14:53
flaper87that's the tradeoff we've to pay to keep our tests isolated14:54
flaper87ah, it is14:54
cppcabreraIf and when functional test speed becomes a concern, we can parallelize using the uuid.uuid1() method.14:54
cppcabreraUnit tests, OTOH, should be blazing fast. :D14:54
malini& it'll be faster than now, when running locally with tox..14:55
maliniI am running against a remote server14:55
flaper87I'm just waiting for your patch14:55
openstackgerritZhihao Yuan proposed a change to stackforge/marconi: fix: claimed message require claim_id to delete
malinihave a meeting..will be back in an hour15:04
flaper87I already commited my first patch, I just need to rebase and then push for review15:05
*** JRow1 has joined #openstack-marconi15:06
*** JRow has quit IRC15:08
*** meganw has joined #openstack-marconi15:17
zyuan_cppcabrera: client message controller still depends on apiclient?15:22
cppcabreralemme double check, zyuan_. Last I checked, I didn't use any of the common lib to implement message controller.15:23
cppcabreraOops, it does. :P15:23
cppcabreraI borrow one exception class from the common lib.15:23
openstackgerritAlejandro Cabrera proposed a change to stackforge/python-marconiclient: Implement message controller.
cppcabreraThere - the message controller patch no longer depends on common lib.15:27
openstackgerritAlejandro Cabrera proposed a change to stackforge/python-marconiclient: Implement message controller.
openstackgerritAlejandro Cabrera proposed a change to stackforge/python-marconiclient: Implement message controller.
*** _alexr_ has quit IRC15:57
*** malini is now known as malini_afk16:07
*** malini_afk is now known as malini16:08
flaper87cppcabrera: feel free to approve this one16:19
flaper87malini: are you back?16:19
cppcabreraflaper87: I've yet to figure out how to +2 things. :P16:21
cppcabreraI figured it would come up in the interface, but I see nothing.16:21
flaper87cppcabrera: just click review and you'll see it16:21
flaper87ahh wait16:21
flaper87did you get granted with cow powers?16:22
cppcabrera[-1, 0, +1]16:22
cppcabreraI remember seeing an email about it...16:22
* cppcabrera double checks16:22
flaper87ahh nevermind, I know what happened16:22
flaper87lp and gerrit are not synced anymore16:22
flaper87so, this needs to be done in both places16:22
cppcabreraYup, I have the lp membership. :P16:24
cppcabreraNo gerrit, though, which explains the missing -2, +216:24
*** flaper87 is now known as flaper87|afk16:25
*** flaper87|afk is now known as flaper8716:26
flaper87cppcabrera: did you get my last messages?16:27
flaper87cppcabrera: done16:27
flaper87you should see the +2 and approve thing now16:27
cppcabrerayup, I see the full range now, flaper87.16:31
cppcabrerathanks. :)16:31
flaper87cppcabrera: with great power, blah blah blah blah blah16:31
cppcabrerahaha, of course. :D16:31
* cppcabrera approves all the things16:31
flaper87brb, dinner16:36
maliniflaper87: back16:36
flaper87malini: seriously?16:37
maliniwill talk after ur dinner :d16:37
flaper87awesome, thanks16:38
flaper87how far are you from submitting the new patch?16:38
*** whenry has quit IRC16:44
*** flaper87 is now known as flaper87|afk16:49
*** whenry has joined #openstack-marconi16:57
amitgandhido we have a list of the possible internal error codes documented anywhere:
amitgandhiim refferring to the internal code here (not the http response codes)17:23
amitgandhimalini: that link gives the http response codes (in the example it says "code": 1092.  What is 1092 ?17:24
maliniamitgandhi: sorry, I have no clue on tht one17:24
amitgandhikgriffs: cppcabrera: any idea?17:25
kgriffsinternal error codes, like what might be logged?17:25
amitgandhi shows a "code": 1092 element17:26
amitgandhii guess so17:26
kgriffsthat slipped through the cracks. It isn't implemented17:26
kgriffslet me look for a bp17:27
kgriffsif someone is looking for something to do, they can work on it.17:27
amitgandhilol not urgent17:27
amitgandhitheres more important stuff that needs to be worked on =P17:27
zyuan_kgriffs: can i have a review at ?17:36
kgriffsanybody remember what the new commit message syntax is re bugs? It doesn't look like the wiki has been updated.17:42
zyuan_kgriffs: Closes-Bug:17:42
cppcabreraIt's as zyuan_ said.17:46
cppcabreraCloses-Bug: #123456717:46
cppcabreraAlso, Partial-Bug and Related-Bug:17:46
kgriffsoh, then it has been updated17:47
cppcabrerayup. :)17:47
kgriffsthought I remembered it being different than that17:47
amitgandhihas anyone looked into the performance of messages._next_marker()17:47
kgriffszyuan_: ^^^17:47
kgriffsreviewing now17:47
amitgandhithe find_one appears to be consistently slow looking at newrelic graphs17:47
kgriffsamitgandhi: not yet; remind me to look at that next17:48
zyuan_kgriffs: ??17:48
amitgandhithe find() takes about 0.8% of the time, while find_one() is 31% of the time @ 3.8 ms avg17:49
zyuan_kgriffs: it's encode, it can't fail17:49
kgriffsI am almost done reviewing17:50
* amitgandhi wonders if its as simple as order of query (p, q) instead of doing (q, p) to use the correct index?17:50
kgriffsamitgandhi: order in the query doesn't matter, just when defining the index17:52
*** flaper87|afk is now known as flaper8717:56
flaper87malini: ping17:56
maliniflaper87: pong17:56
flaper87malini: news ?17:56
maliniI am almost done with the class level setup stuff.17:56
maliniI split the queue tests into 3 classes, since setup, teardown is different for insert queue, metadata & all the rest of it17:57
cppcabrera+1 malini17:57
malinibut I need help with the config file stuff17:58
cppcabreraWhen I did that crazy test refactor patch long ago, I did that same class splitting for the queues. :)17:58
maliniI think we shud continue supporting an extrenal config file, so tht vendors can run tests against their installations17:59
malinias in , I can run the same tests for a Rackspace implementation of marconi17:59
flaper87malini: we will still support it, my suggestion there is to let oslo.config take care of the path discovery18:00
flaper87instead of doing that ourselves18:00
maliniflaper87: tht sounds cool (except I have no idea how to do that :-$ )18:01
flaper87plus, lets use oslo.config's instance instead of having a "get" method for every parameter we need in the test18:01
flaper87malini: let me help you with that18:01
* flaper87 gets some links18:01
maliniflaper87: I will gladly take that :)18:01
kgriffszyuan: patch approved18:02
zyuan_has anyone used os.fork() in python?18:03
flaper87zyuan_: o/18:03
kgriffscan someone sanity-check
kgriffsit looks a little strange to me18:04
openstackgerritA change was merged to stackforge/marconi: chore: increase coverage in some trivial ways
kgriffsmostly starting with the try block18:04
flaper87malini: this call here
zyuan_kgriffs: what's wrong?18:05
flaper87malini: will look for the config file under /etc/marconi and ~/.marconi18:05
kgriffsseems like the first message is always eaten18:05
kgriffsalso, why claim.pop ?18:05
zyuan_it's not18:05
kgriffs(nevermind claim variable being overloaded)18:05
maliniflaper87: where I do pass in the config file name?18:05
kgriffsif I do next(msgs)18:06
kgriffsthat advances the cursor, right?18:06
openstackzyuan_: Error: "!!" is not a valid command.18:06
kgriffsI don't see that claimed(…) is returning anything special for the first request to the generator18:07
flaper87malini: that was my next question, why do we need other people to pass the config file for functional tests? I understand about letting then specify what server to talk to18:08
flaper87kgriffs: mmh, seems you're right18:08
flaper87it should chain it18:08
maliniflaper87: diff implementations can configure marconi message uplimit yada yada18:08
malini+login details18:08
maliniwhich auth to talk to18:09
flaper87mmh, but should they run their own marconi-server in that case?18:09
flaper87and just let the test know where it is18:10
zyuan_flaper87: no problem with it, see line 6918:10
zyuan_kgriffs: ^^18:10
flaper87ah right18:10
zyuan_manually chained18:10
flaper87man, I was like O.O WHAT DID I DOOOO???18:11
maliniflaper87:  'should they run their own marconi-server in that case?' —>it wont be a local server, but a remote installation..So (no tox + self running marconi server) in this case18:11
flaper87mmh, sorry, still don't get it :/18:12
flaper87functional tests should be used during marconi's development to test it in live environments, I understand the fact we want those test to be able to talk to a remote marconi18:13
maliniflaper87: Lets say Rackspace has a production implementation of Marconi. Its not running on a dev laptop & we need to test it18:13
flaper87and we can specify that easily18:13
flaper87malini: sure, totally get that, what I don18:13
malini'functional tests should be used during marconi's development to test it in live environments' —>this is where we have a gap18:13
flaper87ah fuck18:13
maliniIn my veiw, I should be able to run the same tests, if somebody asks me to test in production18:14
flaper87forget my last, partial message18:14
maliniits not just during development phase18:14
flaper87malini: I guet that, I'm cool with that18:14
flaper87bad phrasing from my side18:14
maliniflaper87: is there a better way to specify vendor configurations, wuthout a functional-tests.conf ? can oslo config handle that?18:15
flaper87malini: yeah, but I think you need a change I added in my local patch18:16
flaper87lets do this18:16
malinilet me just submit what I have now18:16
flaper87lets let that patch land with the config as is18:16
maliniI have the class setups mostly removed (except for one)18:17
flaper87and I'll refactor the config part when adding the in-test marconi execution18:17
malinisounds good18:17
malinihere it comes18:17
openstackgerritMalini Kamalambal proposed a change to stackforge/marconi: Refactor System Tests
maliniflaper87: aargh..forgot the util part18:20
flaper87malini: hehe18:21
maliniur suggestion was to move helpers, http & base to functional/ ?18:21
flaper87I think you can leave it18:21
flaper87for now18:21
flaper87I mean, we have util for the other tests as well18:22
flaper87I'll remove that in my patch18:22
flaper87since we're splitting tests18:22
malinithanks :)18:22
cppcabreraI'll check it next.18:26
flaper87cppcabrera: thanks :)18:26
flaper87I know there are some flake8 / hacking issues but since it's being ignored for now, I wouldn't boder raising those nits18:27
flaper87I guess we'll fix them once it runs under tox18:27
cppcabrera+1 flaper8718:27
cppcabrera+2 - approved; malini. :)18:28
flaper87I guess it is bother18:28
flaper87damn, what's wrong w/ me18:28
cppcabreraToo much work, flaper87. ;)18:29
flaper87cppcabrera: you're right, I know you're right18:29
openstackgerritA change was merged to stackforge/marconi: Refactor System Tests
cppcabreraIthappens to me, too. :)18:29
cppcabreraSo I understand, hehe.18:29
flaper87malini: thanks for the hard work there18:29
flaper87now, I'll do my part18:30
flaper87malini: you'll be around tomorrow, right?18:30
flaper87cppcabrera: yeah, :(18:30
flaper87I need more brain energy18:30
maliniflaper87: yes..I'll be here tomorrow18:31
maliniwe have the hackday, but I would really love to have this whole thing figured out before I leave18:31
maliniits merged <:o) !!!!!!!18:32
cppcabreramalini: Yup, merged after 2 +2s. :]18:32
flaper87malini: cool, I will ask you to review my patches tomorrow before you EOD18:33
flaper87I want your thoughts about them18:33
flaper87just wanted to emphasize that18:33
maliniI will have internet till SUnday morning..SO I can review on saturday too :)18:34
flaper87malini: I'd never ask you to do that, but I'll make sure you get enough emails from gerrit and let it be the bad guy18:34
flaper87kk guys, brb!18:35
openstackgerritKurt Griffiths proposed a change to stackforge/marconi: fix: Requests get slower when queues have a lot of messages
cppcabrerareviewing soon, kgriffs.18:40
cppcabreraGreat commit message, btw.18:40
kgriffsflaper87: whats the "pop" for vs. just doing a lookup?18:41
kgriffs'ttl': claim.pop('t'),18:41
* kgriffs is looking18:41
zyuan_lookup only is faster18:50
zyuan_i hope the code does not depends on the side-effect?18:51
zyuan_flaper87: review~~~
kgriffsIn [49]: %timeit utcnow_ts()19:00
kgriffs1000000 loops, best of 3: 525 ns per loop19:00
kgriffsIn [50]: %timeit utcnow_ts_slow()19:00
kgriffs100000 loops, best of 3: 3.83 us per loop19:00
openstackgerritKurt Griffiths proposed a change to stackforge/marconi: fix: Requests get slower when queues have a lot of messages
openstackgerritAlejandro Cabrera proposed a change to stackforge/marconi: feat: marconi proxy
cppcabrerakgriffs, zyuan: marconi proxy v0.5 is ready for review. I'll be reviewing your mongo opts now, kgriffs.19:09
cppcabreraMaann, I dropped +922 lines on you guys. This is not code review friendly. :P19:10
cppcabreraSorry about that.19:10
cppcabreraquick question: does it matter if we use BSON dates versus python dates when issuing a query through pymongo? (kgriffs)19:19
zyuan_we are not using python date; we used integers19:19
cppcabrerazyuan_: timeutils.utcnow() -> datetime.datetime(2013, 8, 29, 19, 21, 44, 929690)19:22
cppcabreraquery = {..., 'e': {'$lte': timeutils.utcnow()}, ...}19:22
cppcabreraso we are using python dates in mongodb.messages:_removed_expired19:23
zyuan_cppcabrera: ...19:24
cppcabrera"Note that documents can contain native Python types (like datetime.datetime instances) which will be automatically converted to and from the appropriate BSON types."19:24
kgriffsseems like we could just use timestamps there19:24
cppcabreraCool, it's not a problem.19:24
kgriffssave a little room, maybe faster query19:24
cppcabreraThat's from the pymongo docs ^^19:24
cppcabrerawe'd save a bit of space, I'm sure. Sounds like one of those potential optimizations.19:25
zyuan_it does not save space19:30
*** JRow1 has quit IRC19:30
zyuan_where bson datetime type is just a int6419:30
zyuan_milisecods from epoch19:30
cppcabreraAhh, good find zyuan_19:30
kgriffswho knows, it may add up over time? :p19:41
cppcabreraphew, 8x more nanoseconds19:41
zyuan_that's for sure...19:42
kgriffsalso consider the conversion from datetime to bson timestamp19:42
kgriffsmay add another 200 ns19:42
* kgriffs runs for the hills 19:42
zyuan_double vs. struct19:42
cppcabreraThere's a catch, though.19:42
zyuan_what i did in sqlite is to store double19:43
cppcabrerais time.time() UTC-based?19:43
kgriffssrsly though, if you optimize a few of these and shave a few microseconds, it can be significant in terms of throughput19:43
kgriffscppcabrera: unix timestamp19:43
cppcabreraI need to refresh on UNIX timestamp's relationship with timezones. >.>19:43
cppcabreraahh, UTC19:44
zyuan_i used julianday19:44
kgriffsit's just number of seconds since an arbitrary epoch19:44
cppcabreraalright, time.time() is the way to go in the future for another improvement19:44
kgriffsadded to the work items19:46
kgriffsnote that if we use timeutils.utcnow_ts() we should submit a patch to use time.time() as well in that function19:46
kgriffs(see my earlier paste)19:47
* kgriffs thinks it is lame to slow things down for the sake of testability when there's any easy alternative19:47
kgriffssee also:
openstackgerritKurt Griffiths proposed a change to stackforge/marconi: chore: Update openstack.common, add lockutils
cppcabrerahere's the full impact of time.time() vs. running against mongo:
cppcabrerakgriffs ^^19:50
kgriffsthat's ms, right?19:50
cppcabreraThat's seconds.19:51
cppcabrera7.18 seconds vs 6.75 seconds19:51
kgriffsoh, I see, you didn't divide by number19:51
cppcabreraI drop()'d the collection between runs to make the amount of data in the DB didn't affect the test.19:51
kgriffslooks like ~21 microseconds19:52
kgriffsthat's actually significant19:52
kgriffs(difference, per attempt on average)19:53
* cppcabrera loves timeit for microbenchmarks19:53
kgriffsbtw, closures work too19:54
kgriffsmin(timeit.repeat(my_func)) / number19:55
cppcabrerahmmm... I had no idea functions could be passed to timeit.19:57
kgriffscppcabrera: this should be good to go now:
cppcabreraI just +2'd it before coming back to IRC, heh.20:02
cppcabrerame, too.20:05
cppcabrerawb, flaper87. :)20:05
flaper87cppcabrera: you too :D20:05
*** malini is now known as malini_afk20:05
cppcabreraThe hot, pending patches for the day are: (lock utils into marconi), marconi proxy (, fix slowdown as messages are added (, add oslo.cache to marconi ( (flaper87, kgriffs)20:07
flaper87kgriffs: +220:07
openstackgerritA change was merged to stackforge/marconi: feat(wsgi): homedoc now ships relative URIs
cppcabrerasweet - 3 patches to go. marconi-proxy is a beast of a patch. :P20:07
flaper87cppcabrera: mind If I review the proxy patch tomorrow?20:08
cppcabrerathat's cool, flaper87. :)20:08
flaper87I don't think I've the right mind power to review it as it should20:08
cppcabreraI'll be adding unit tests to it later today. I barely have the mind to review my own patch atm.20:08
flaper87cppcabrera: the only thought I've right now is: Can it be split in 2 or more patches?20:08
flaper87if so, it would be really nice to do it20:09
openstackgerritKurt Griffiths proposed a change to stackforge/marconi: fix: Requests get slower when queues have a lot of messages
openstackgerritA change was merged to stackforge/marconi: chore: Update openstack.common, add lockutils
cppcabreraflaper87: I'll try to split it. I think I can make it so that the first patch adds the core catalogue stuff, the second patch forwards things to marconi, and maybe another split.20:10
flaper87cppcabrera: awesome, that would be awesome20:11
cppcabreraI've read the studies on code review, too, and I know there's fierce diminishing returns after about 350~ LOC. :P20:11
cppcabreraI remember seeing it as a hot topic on openstack-dev@ (code review studies)20:12
kgriffsthis one also needs some love20:12
cppcabreraAhh, yes. That one affects our API's behavior.20:13
cppcabreraMakes it a little stricter about claims handling.20:13
kgriffsis_claimed = 'c' in message and message['c']['e'] > now20:13
kgriffsc is always in message now, or am I wrong?20:14
kgriffsyes, we need it to be feature complete (forgot it wasn't in yet)20:14
flaper87kgriffs: c is always present20:14
kgriffszyuan_: ^^^20:14
flaper87(assuming we're talking about mongodb's driver)20:14
kgriffsI'm reviewing it now20:14
zyuan_let'me check20:16
zyuan_'c' always exists; may be None20:17
kgriffsok, second thing20:18
kgriffsmessagenotpermitted and claimnotpermitted don't seem to be appropriate names for the errors20:18
kgriffshow about something more like "DeleteNotPermitted"20:18
zyuan_there base is NotPermitted20:19
zyuan_...out api only has 1 "not permitted" concept20:19
kgriffsyes, I know, but that doesn't prevent us from renaming the child classes20:19
kgriffsyou aren't deleting a claim...20:20
zyuan_(don't remind me java -_-)20:20
kgriffsyou are deleting a message20:20
kgriffsand it isn't permitted, right?20:21
kgriffsdo you need two different error types for the two cases?20:21
zyuan_for wsgi, no20:21
zyuan_but it's good for storage20:21
zyuan_just like we have different types of NotExisting20:22
kgriffsI mean, could you just raise something like DeleteNotPermitted if the message is claimed, but either the caller did not specify a claim ID, or they did but it was invalid?20:22
openstackgerritKurt Griffiths proposed a change to stackforge/marconi: chore: Add the up-and-coming oslo.cache module
zyuan_kgriffs: that's currently what wsgi understand, but i don't think storage should do this20:25
zyuan_those errors are not echo to users anyway20:25
openstackgerritZhihao Yuan proposed a change to stackforge/marconi: fix: claimed message require claim_id to delete
zyuan_someone broke the unittests' config file i guess20:26
amitgandhikgriffs: the task to look into find_one() performance − should that be a bp/bug/or added as a workitem to
zyuan_bug +120:28
kgriffszyuan_: if you want to raise different types, that's fine, but the current names imply that "a message is not permitted" and that a "claim is not permitted"20:29
kgriffs(doesn't make much sense in this context)20:29
zyuan_try rename it...20:30
zyuan_but... v_v malini_afk -_- fix nosetests20:30
openstackgerritZhihao Yuan proposed a change to stackforge/marconi: fix: claimed message require claim_id to delete
zyuan_MessageIsNotClaimedBy ?20:33
amitgandhiperf bug reported:
openstackgerritZhihao Yuan proposed a change to stackforge/marconi: fix: claimed message require claim_id to delete
kgriffsamitgandhi: thanks!20:37
kgriffscan someone pls confirm
flaper87kgriffs: I was looking at it. I guess we could try .find().limit(1) + index hint20:38
flaper87or keeping the marker somewhere else20:38
amitgandhii noticed that we hit q and then p20:38
amitgandhidoes that matter20:38
zyuan_i guess not20:38
amitgandhiie does it screw up the way the index is used20:39
zyuan_the last entry is hard to reach20:39
zyuan_there is just no index on marker20:39
amitgandhiyou would want to filter p and then by q to reduce the scan (idk if mongo is smart enough to rearrange the filter clauses like sql is)20:40
flaper87amitgandhi: yeah, there's a patch for that20:40
flaper87kurt added it20:40
flaper87amitgandhi: good thought, anyway20:40
kgriffsamitgandhi: i'm certain it doesn't matter, otherwise pymongo would require a list of tuples20:41
flaper87kgriffs: I think he means in the index20:42
flaper87not in the query20:42
kgriffsI went ahead an reorded in my patch anyway just to be consistent with the index definitions20:42
flaper87I guess20:42
flaper87well, it makes sense in the index20:42
flaper87kgriffs: and I +1 you for that20:42
flaper87there are more messages in a queue than queues in a project20:43
flaper87well, that's what my random probability calculation says20:43
flaper87kgriffs: btw, what do you think about merging p and q in the same field20:43
flaper87in the message collection20:43
flaper87that should reduce the number of indexes and indexes lookup20:43
flaper87I mean, the size of the index20:44
flaper87and the fields to filter on20:44
flaper87damn, today is not my chatting day20:44
flaper87and that should also make the query faster20:44
* flaper87 has a background thread thinking on further optimizations20:45
kgriffsI thought of that a long time ago, but forgot about it since we were originally querying on the queue ID20:45
flaper87kgriffs: yeah, that didn't make sense before20:45
flaper87it does now though20:45
zyuan_i think it's easier to store an sha256 for q+'+'+p20:45
kgriffsI don't see any reason not to try it unless for some reason we only want to query one of those in isolation20:46
kgriffsI suppose we could use a regex for that (don't tell anyone)20:46
flaper87kgriffs: EXACTLY20:46
kgriffsif it's a prefix pattern, then it can still use the index20:46
zyuan_al message need is unique constrain by queue and project20:46
kgriffsif you need all things in a project20:46
zyuan_need to go the /queues endpoint20:47
flaper87kgriffs: yup20:47
kgriffsfeel free to give that a try20:47
flaper87sure, I will20:47
flaper87lets get yours merged first20:47
zyuan_flaper87: i recommand do this:20:49
kgriffssha256 isn't exactly fast20:50
zyuan_a.digest() # bytes20:50
flaper87yeah, I'd prefer keeping it plain20:50
zyuan_but it's always 32bytes20:50
kgriffstrue, it is a space-time tradeoff20:50
zyuan_short=faster to __hash__20:51
kgriffsalthough if you start having to page then smaller is faster too20:51
kgriffsmaybe one step at a time20:51
kgriffsmerging them as plaintext will already save a little RAM20:52
zyuan_just use update() can save an intermediate string20:52
flaper87the q+p will end up in a utility function20:52
kgriffsthen you have a baseline for performance testing if you want to hash20:52
flaper87which will be easier to update in the future20:52
zyuan_no string contenation is needed20:52
flaper87with some hashing20:52
zyuan_q+p needs allocation20:52
kgriffsyes, but having a single index field vs two should save some overhead20:52
zyuan_while update is just a scan on source20:52
kgriffsalso a little bson overhead20:53
flaper87zyuan_: I know, I just haven't figure out a name for the function20:53
zyuan_hmmm, wait. considering the use case, it may not be short.  ever seen project + queue longer than 32 chars?20:54
kgriffsI imagine we will see a lot in the 40-60 range20:55
kgriffsqueue names may be uuids20:55
kgriffs(36 chars just for that)20:55
openstackgerritAlejandro Cabrera proposed a change to stackforge/marconi: feat: marconi proxy
cppcabrerapatch split part 1: the proxy-specific operations - no forwarding ^^20:57
zyuan_interesting. sha512 is faster than sha25620:57
*** meganw has quit IRC20:57
zyuan_but 2x longer result20:57
flaper87cppcabrera: +1 thanks for that20:57
cppcabrera~644 LOC now. :P20:58
cppcabrerakgriffs: If I do 'pip install falcon' atm, does it include the set_default_route patch?20:58
flaper87kgriffs: btw, we've to answer the question w.r.t "Why falcon and not Pecan" in the next meeting.20:59
flaper87kgriffs: you had some benchmarks about that, right? (different from the ones on the website)20:59
zyuan_cppcabrera: marconi's version restriction ....21:00
cppcabrerazyuan_: that's true, also, falcon's new patch hasn't been pushed to pypi just yet. I verified. :(21:00
kgriffscppcabrera: no21:00
kgriffscppcabrera: will be in 0.1.721:01
kgriffsif anyone wants to help finish that release during a hack day, be my guest!21:02
kgriffsflaper87: there is a complex benchmark that you can run as a command after installing falcon...21:03
kgriffsit doesn't do the same complex things in all the other frameworks, only benchmarks falcon21:03
kgriffsyou can run the simple benchmark21:03
kgriffsthen run the complex one21:03
kgriffsand if the complex one is still a lot faster than the simple one for the other frameworks, you win. :D21:04
kgriffswhich is the case last time I checked21:04
flaper87mmh, ok!21:05
kgriffsthe real question is what would be the perf difference with a real app21:05
flaper87Do we have other reasons / answers for that question?21:05
flaper87kgriffs: yeah21:05
kgriffsso we would really have to spent a fair amoutn of time writing a pecan driver to find out21:05
flaper87my question actually is, Why does it matter so much?21:06
kgriffsit is based on past experience with a proprietary Rackspace infrastructure service21:06
flaper87what's the big issue about having falcon as a library in the global requirements21:06
kgriffsthat framework speed matters in the data plane21:06
flaper87kgriffs: no no, I wasn't talking about the speed21:07
kgriffsyou mean, why does anyone care that we take a dep on falcon?21:07
flaper87kgriffs: yup21:07
kgriffsDoug is on a crusade to standardize web frameworks across openstack projects21:07
kgriffs(and I mean "crusade" in the nicest of ways)21:07
flaper87yeah, I've helped a bit on that,  TBH21:08
kgriffsI am not opposed to the spirit of the idea21:08
flaper87but, mmh, well, I guess that makes contributors life easier21:08
flaper87as well21:08
kgriffsI'm not sure that Pecan was the best choice if he was hoping to get marconi and swift on board21:08
kgriffsI'm not saying this because I wrote falcon21:09
kgriffsI didn't want to write falcon21:09
kgriffsbut it was necessary21:09
flaper87I know, we're just talking about what the best thing for projects is21:09
flaper87or guessing, at least21:09
kgriffsnow, all this being said, I think we really should try to take some time and write a pecan driver at some point21:09
*** oz_akan_ has quit IRC21:10
flaper87kgriffs: kk21:10
openstackgerritAlejandro Cabrera proposed a change to stackforge/marconi: feat: marconi proxy (v1, health)
flaper87I guess we can propose that as part of the incubation process21:10
kgriffsswift might try it as well.21:10
flaper87I mean, we can write that during incubation21:10
kgriffsbut I don't want to put them in the middle of this if they don't want to be21:10
flaper87and decide before being integrated21:10
torgomaticpecan seems to have blocking IO in the web framework --> one thread or process per connection --> something like a 0.00% chance of ever landing in Swift21:10
* notmyname isn't familiar with either pecan or falcon, other than knowing people seem to think they should be used everywhere21:11
notmynametorgomatic: ah. well then21:11
flaper87torgomatic: owww really????21:11
torgomaticnotmyname: this is from a cursory glance, though, not a deep dive21:11
flaper87mmhh, interesting21:11
torgomaticmaybe it can be used together with eventlet or gevent, in which case it'd be usable21:11
torgomaticin Swift, that is21:11
torgomaticbut as it is, it seems like there's no provision for handling many client connections in a single Python process21:12
* torgomatic is happy to be proven wrong, though21:12
kgriffsmy perspective is that it's a web app framework21:13
flaper87I guess we should ask markmacclain21:13
kgriffsthe creators didn't envision it for large-scale web services21:13
kgriffsbut I digress21:13
kgriffsFWIW, I haven't been pushing falcon in the community. Other people have suggested using it in other projects, but I have been trying to steer the discussions in a more collaborative vs. adversarial direction.21:14
kgriffsI'm sure that since OpenStack started using pecan (introduced by Doug, who works at dreamhost, where Pecan was born)21:15
kgriffsthere has been some effort to make it better for web services21:15
kgriffsanyway, I'm speculating so don't quote me on any of this. :p21:16
kgriffsflaper87: one more thing21:16
kgriffsit's hard to make money on a queuing service. So operators will need it to be as efficient as possible.21:17
kgriffsif a pecan driver is elegant and really fast, then sure, why not?21:17
kgriffsbut if not, I think pragmatism has to have it's due21:18
* kgriffs isn't at all opinionated21:19
* ametts loves it when kgriffs says "Don't quote me on this" in a public IRC channel :)21:20
kgriffsif only you knew what I was *really* thinking but didn't say!21:20
flaper87kgriffs: yeah, I completely agree with you. That's why I see it as a "Ok, lets try this thing first and we'll see"21:21
flaper87as part of the incubation period21:21
kgriffsyep, I think that is our answer21:21
flaper87I'm not opinionated but I do wan't what's best for the project21:21
flaper87and I've been appliying the same way of thinking to every other project throughout OpenStack21:21
flaper87so, I expect others to do the same w/ Marconi21:22
kgriffsit isn't "us vs. them", it's just "us"21:22
flaper87TBH, I don't think it'll be an issue, at all21:22
flaper87as long as we're open to try Pecan and pick the one that fits better for marconi21:23
kgriffssure, it only makes sense21:24
openstackgerritAlejandro Cabrera proposed a change to stackforge/marconi: feat: marconi proxy
openstackgerritAlejandro Cabrera proposed a change to stackforge/marconi: feat: marconi proxy (v1, health)
openstackgerritKurt Griffiths proposed a change to stackforge/marconi: chore: Track the up-and-coming oslo.cache module
openstackgerritAlejandro Cabrera proposed a change to stackforge/marconi: feat: marconi-proxy forwarding
kgriffsfolks: ^^^21:31
amitgandhiso many propositions21:31
kgriffsI added a test to make sure namespaces were correct, and found some bugs. :p21:31
kgriffsshould be good to go now21:31
flaper87kgriffs: in the cache lib?21:31
kgriffssince I moved it under marconi.common21:32
flaper87kgriffs: something I should port to the oslo patch?21:32
flaper87ah ok21:32
kgriffsno, ur patch is fine21:32
flaper87kgriffs: btw, it got some extra reviews, in case you want to comment. I'll do that tomorrow21:33
kgriffson that note21:34
kgriffswhat do you think about a decorator that will call _prepare_key for you?21:35
kgriffs(side-by-side comparison of how it might look)21:35
flaper87There are 2 reasons why I'd prefer not having it: The first one is that it assumes there's a key to transform in the first argument, which shouldn't be a big deal since the API defines the methods that way. The second is that there are cases where having the original key is needed
flaper87and third, that it would make sense just for set, unset21:38
kgriffswhy not get21:39
flaper87while the *_many methods will have to call it manually21:39
flaper87kgriffs: and get, sorry21:39
flaper87those are just some thoughts, I'm not compeltely against the decorator21:39
kgriffsincr, add21:40
flaper87I definitely hate calling that method everytime21:40
kgriffslooks like the _many don't fit nicely in that model, I agree21:40
kgriffshmm, well, something to noodle on21:40
kgriffsif you can figure out a nice way to avoid having to call that every time, that would be AWESOME21:41
kgriffsI gotta run21:41
flaper87kgriffs: couldn't agree more!21:41
flaper87kgriffs: take care :)21:41
kgriffsflaper87: pls review the cache patch for marconi21:41
flaper87kgriffs: yes, I will21:42
kgriffshave a good one,21:42
kgriffsu around tomorrow?21:42
flaper87kgriffs: yes, I'll be here21:43
flaper87I'm planning to close the tests patches tomorrow21:43
flaper87and help you with the performance ones21:43
kgriffsI added some work items to the bottom (just brainstorming)21:43
flaper87ah, nice. Awesome. That's helpful21:44
kgriffsthis too21:44
kgriffsits like 7x faster21:44
kgriffstakes it down to nanosecond range21:44
flaper87why is calendar being used in first place?21:45
flaper87I mean, in oslo21:45
kgriffsto reuse utcnow21:45
flaper87ah right21:45
flaper87kk, sounds like something we can contribute back to oslo21:45
flaper87kgriffs: +!21:45
flaper87kgriffs: +121:45
kgriffswould we need to open a bug first?21:45
kgriffsor just submit the patch?21:46
flaper87I'd say just submit a patch with a good description21:46
kgriffswill do21:46
kgriffssomeone may have a better idea of how to "fix" this, but this was the first approach that came to mind21:46
*** flaper87 is now known as flaper87|afk21:51
openstackgerritAlejandro Cabrera proposed a change to stackforge/marconi: feat: marconi-proxy forwarding
cppcabreraFinally got those dependencies right. :P21:56
cppcabreraThree patches: catalogue, v1 + health, full-on forwarding21:56
cppcabreraStill needs unit tests.21:56
cppcabreraBut that's the core with added docstrings @ [644, 70, 160] LOC21:57
kgriffstimeutils patch ^^^21:57
kgriffscppcabrera: cool, thanks21:57
kgriffsI gotta run21:57
kgriffsI'll have to check it out tomorrow21:57
kgriffsthanks everyone for you awesome work21:58
cppcabrerakgriffs: o/21:58
cppcabreraI'm out for the night, too.21:58
cppcabreraTake care everyone.21:59
*** cppcabrera has quit IRC21:59
*** kgriffs is now known as kgriffs_afk21:59
*** kgriffs_afk is now known as kgriffs22:12
*** kgriffs is now known as kgriffs_afk22:13
*** oz_akan_ has joined #openstack-marconi22:21
*** oz_akan_ has quit IRC22:25
*** amitgandhi has quit IRC22:35
*** amitgandhi has joined #openstack-marconi23:39
*** oz_akan_ has joined #openstack-marconi23:52
*** oz_akan_ has quit IRC23:52

Generated by 2.14.0 by Marius Gedminas - find it at!