Monday, 2013-09-09

*** oz_akan_ has joined #openstack-marconi00:42
*** oz_akan_ has quit IRC00:46
*** amitgandhi has joined #openstack-marconi00:52
*** nosnos has joined #openstack-marconi01:22
*** amitgandhi has left #openstack-marconi01:50
*** oz_akan_ has joined #openstack-marconi03:29
*** oz_akan_ has quit IRC03:41
*** oz_akan_ has joined #openstack-marconi03:42
*** oz_akan_ has quit IRC03:46
*** nosnos has quit IRC05:32
*** nosnos has joined #openstack-marconi05:32
*** ykaplan has joined #openstack-marconi06:45
*** nosnos_ has joined #openstack-marconi07:00
*** nosnos has quit IRC07:01
*** flaper87|afk is now known as flaper8707:13
* flaper87 shakes the channel and wakes everyone up09:08
al-maisan*yawn* :-)09:09
flaper87al-maisan: good morning :D09:10
al-maisanGood morning flaper8709:10
al-maisanwe should have our VC today09:10
al-maisanis 14:00 CEST ok?09:11
flaper87Yup, souuunds good to me!09:11
flaper87al-maisan: ^09:11
flaper87G+ ?09:12
al-maisangreat, let's do a Google hangout at 1409:12
flaper87I'm flaper87 there too.09:12
flaper87damn I'm so easy to find09:12
al-maisanthat's handy :)09:12
*** ykaplan has quit IRC09:55
*** ykaplan has joined #openstack-marconi10:27
*** nosnos_ has quit IRC10:39
flaper87Alex_Gaynor: could you take a quick look here again?
*** oz_akan_ has joined #openstack-marconi11:12
*** cthulhup has joined #openstack-marconi11:16
flaper87cthulhup: yo, welcome back!11:17
*** tedross has joined #openstack-marconi11:26
*** cthulhup has quit IRC12:00
*** cthulhup has joined #openstack-marconi12:01
*** ykaplan has quit IRC12:03
*** ykaplan has joined #openstack-marconi12:07
*** oz_akan_ has quit IRC12:26
*** flaper87 has quit IRC12:44
*** flaper87 has joined #openstack-marconi12:44
*** amitgandhi has joined #openstack-marconi13:09
*** rustlebee is now known as russellb13:14
*** oz_akan_ has joined #openstack-marconi13:23
flaper87amitgandhi: will cppcabrera be around today?13:23
*** oz_akan_ has quit IRC13:25
*** oz_akan_ has joined #openstack-marconi13:25
*** malini_afk is now known as malini13:28
flaper87sooooo happy to have you back!13:28
flaper87good morning13:28
flaper87zyuan_: heyyyy, good morning! :)13:29
flaper87zyuan_: how was the conference ?13:29
zyuan_flaper87: great, except the last day13:29
flaper87hat happened? Did someone said C# is better?13:29
zyuan_i was tired on the last day, the last day is all M$ craps13:29
flaper87baahhhh!!! buuuuuuu!!!!13:30
zyuan_i felt like that's too stupid taht i don't know whether should i comment...13:30
zyuan_i recognize visual c++ as a good compiler until...13:31
zyuan_until they told me how they did the optimizer13:31
flaper87 ooohhhh do tell13:32
zyuan_in front of me was Chandler, the LLVM guy13:32
zyuan_and he confirmed twice with the Visual C++ guys, like, are they serious...13:33
zyuan_most of other talks are great13:34
*** tedross has quit IRC13:36
zyuan_noone took the timeutil one?13:42
zyuan_if not, i can do it now13:42
flaper87zyuan_: timeutil ?13:43
*** amitgandhi has quit IRC13:44
zyuan_flaper87: yea, change mongo's datetime type13:44
flaper87zyuan_: ah, erm, no, I don't recall any of us taking it\13:45
flaper87zyuan_: go ahead13:46
zyuan_after the patches being merged last week13:46
zyuan_sqlite driver's converage finally went back to 100%13:46
flaper87zyuan_: +1 for you!!!!13:46
flaper87zyuan_: do you know if cppcabrera will be around today?13:49
flaper87zyuan_: btw, did you see we're now incubated? :D w0000t13:50
flaper87Migration under openstack/* will happen later this week or the next one, I guess. -infra guys still have to figure that out. (Meeting tomorrow)13:50
zyuan_flaper87: see? i know we passed incubation13:51
zyuan_you mean is code is moved?13:51
flaper87zyuan_: no no, I meant if you knew we're and you do :D13:52
maliniflaper87: hola !!!13:57
maliniSorry got distracted with all the stuff happening in office ..& I was out for just a week :D13:58
flaper87malini: hehehehe!! Thought so!13:58
*** ykaplan has quit IRC13:59
flaper87malini: do you have some time to go through test patches?14:03
flaper87I bet you've tons of emails to read14:03
maliniflaper87: I am going thru those right now..14:04
maliniis there any patch you would like to see go first?14:05
flaper87The first one :P (I had to chain them to make this whole refactor work)14:06
flaper87which you already +1'd14:06
* flaper87 dials cppcabrera's direct line to check whether he'll show up today or not.14:07
malinihe won't14:07
malinihe is still in jury duty14:08
flaper87malini: :(14:08
flaper87poor boy!14:08
flaper87do you guys mind if I ninja approve that first patch?14:08
flaper87mmh, I should wait for kurt14:08
flaper87even if he already +2'd14:08
flaper87kgriffs_afk: knock knock, time to +2 some patches :D14:08
zyuan_a lot!14:09
*** ykaplan has joined #openstack-marconi14:15
maliniflaper87: do we have/need a standard abt using % vs .format for string operations ?14:20
flaper87malini: we don't have one but I think we should. We use % dict(...) in other projects14:21
maliniI remember cppcabrera leaning towards .format..But I just cant remember why..14:22
flaper87malini: Py3K comptatibility, most probably14:22
malinithts what I thought too14:22
maliniespecially since its cppcabrera :D14:23
malinido we want to stick with % (or) start using .format ?14:24
*** kgriffs_afk is now known as kgriffs14:25
kgriffshey folks14:25
kgriffsgot a bomb to drop on you14:26
kgriffsflaper87: ping14:26
flaper87malini: I leaning towards .format as well. %-format is old and has some issues14:26
flaper87kgriffs: pong14:26
flaper87kgriffs: good morning14:26
kgriffsgood afternoon!14:26
flaper87kgriffs: shooooot14:26
kgriffsso, like you, leaving one message in each queue has bugged be for a while14:26
kgriffsthis morning on the bus I was thinking about it again and something was tickling the back of my mind14:27
kgriffsso when I got to the office I started doodling on a whiteboard…14:28
kgriffs<who gave that guy a marker?!>14:28
kgriffsanyway, I think I've got a solution14:28
flaper87ohhh, shoot shoot!!!14:28
flaper87I can give you an etherpad if you want14:29
kgriffsmight need it, we'll see14:29
kgriffsthe key is to not increment the side counter when you retrieve it14:29
kgriffsyou increment it after you've written the new message14:29
kgriffsbut, there's a problem with that14:29
kgriffswhich originally stumped my. but the solution was staring me in the face, I just didn't see it before.14:30
kgriffsanyone want to take a guess at the problem?14:30
flaper87by you increment it after, you mean, store the message, increment the counter and store it in the message ?14:32
kgriffsno, let me write it out on the pad14:32
kgriffsmake sense14:34
kgriffs you still have to have the unique index and retry on collisions, as currently implemented14:35
flaper87(worte a comment there)14:36
*** tedross has joined #openstack-marconi14:36
kgriffsso, there may be other problems I haven't thought of yet, but here is the first one I thought of14:38
kgriffsSuppose a parallel request comes in, and so one of the requests "wins" and causes the other to retry14:38
kgriffssuppose the thread that wins crashes before it manages to update the side counter?14:39
zyuan_1. do we do it currenly14:40
zyuan_2. even we do, does it affects invariants?14:40
zyuan_looks like not, if we just skipped some markers14:40
kgriffszyuan_: we don't use a side counter currently, we are discussing the consequences of introducing one14:40
kgriffsand on a retry currently there is a max number of retries before the thread gives up14:41
zyuan_kgriffs: i saw a seperated marker in mongo14:41
*** whenry has joined #openstack-marconi14:41
flaper87kgriffs: what happens if we increment it before? with a findAndModify ?14:41
flaper87that will give us a counter we can use14:42
kgriffsok, when I say "side counter" I mean one that is not stored in the message document itself14:42
flaper87if another insert wins, it doesn't matter, we just increment it again14:42
zyuan_that's... hard to do it in mongo14:42
flaper87(not the side counter, but the one we're using for the marker)14:42
kgriffsflaper87: if we do that, we have the out-of-order insertion/race condition problem14:42
zyuan_because mongo can not query 2 collections atomically14:42
flaper87kgriffs: why out of order?14:43
kgriffsflaper87: let me illustrate on the pad14:43
kgriffsmake sense?14:48
flaper87kgriffs: sorry, got distracted w/ another convo14:58
flaper87kgriffs: I was thinking to call findAndModify once per every message (let me read your proposal carfully)14:58
flaper87kgriffs: btw, can we ninja approve some of the test patches if they get some +1?15:09
flaper87not sure when cppcabrera will be back and the chain is getting bigger and bigger15:09
kgriffsflaper87: sure thing15:10
flaper87kgriffs: awesome, whenever you get a chance :)15:10
* flaper87 back to the pad15:10
*** BigChzy_away has joined #openstack-marconi15:12
*** BigChzy_away is now known as BigChzy15:12
* kgriffs is back from the ether15:14
* flaper87 is back from a black hole15:14
flaper87kgriffs: it makes sense overall, I'm not sure how failure cases should be handled there15:14
kgriffsso, my idea was to have some kind of hall monitor that breaks up the fights.15:15
kgriffsyou would have to set a timestamp when incrementing the counter15:15
kgriffs(last update time)15:15
kgriffsand then have a daemon running that checks for counters that are really stale and increments them15:15
* kgriffs is trying to decide if that introduces another race condition15:16
kgriffsalternatively, a worker after retrying X times will check the timestamp and increment it if it is stale15:16
kgriffslet me write that out to see if there is a race condition15:17
flaper87mmh, I'n not sure about adding yet another process to marconi's stack.15:17
flaper87it seems to me we're moving some of mongodb's logic to marconi15:17
flaper87(btw, we've a meeting today, right?)15:17
zyuan_why python2's datetime has no .timestamp()15:17
kgriffsoh, yeah. I forgot to send an email15:17
kgriffslet me do that15:17
flaper87ykaplan: is here ready to talk about sql backend15:18
kgriffszyuan_; 45 mins15:18
ykaplanflaper87, yes15:21
kgriffszyuan_: time.time()15:25
kgriffsbut we should be using utils.utcnow_ts() since it allows us to override "now" for testability15:26
* kgriffs still thinks that function is poorly named15:26
zyuan_kgriffs: no, i mean the one convert a datetime to timestamp15:26
zyuan_because bson ObjectId only gives you datetime15:27
zyuan_now i get it15:27
zyuan_different for python2 and python315:27
kgriffsyeah, I think it would be great to see some of timeutils get into python propper15:27
zyuan_python3 did it15:27
kgriffsah, so python3 has it. nice15:27
zyuan_but they refuse to backport15:28
malinikgriffs:  I love this on the agenda 'Commit tsung configs in Marconi's tree' ..Was just thinking about it on my way in today :)15:28
zyuan_see? python2 and 3 are two languages15:28
flaper87malini: o/15:30
maliniflaper87: strawberry chocolate pop tarts for you !15:30
flaper87today's agenda is quite big!15:33
kgriffsflaper87: not sure if this happened to you, but I ended up not being able to use since it wouldn't install pbr from there. Maybe my system is messed up in a different way, but it seemed like was the culprit:
kgriffsusing /simple seems to help anyway15:33
flaper87kgriffs: ah yes, indeed it happened to me15:33
kgriffsflaper87: I think we will need a 3rd-party process after all to check and increment stale counters. If you have the request handler do it, you could get into a situation similar to the findAndModify race condition15:37
kgriffsbut, we could simple make it part of marconi-gc15:37
* kgriffs doesn't want to have to come up with a new name15:38
kgriffsactually, we don't need GC any more if we make this change, since we can just use mongo's TTL feature15:39
*** BigChzy has left #openstack-marconi15:40
*** jergerber has joined #openstack-marconi15:41
flaper87kgriffs: mmh, that's true, we don't need a gc anymore!15:43
flaper87kgriffs: marconi-aligner :P15:43
flaper87like the worst name ever15:43
flaper87what if we leave the align process to the process inserting the message ?15:43
flaper87I mean, the counter is aligned when it is >= than the marker used in the message that was just inserted15:44
flaper87we could check if the counter is >= k after calling findAndModify15:45
flaper87and call findAndModify again if it isn't15:46
*** tedross has quit IRC15:49
flaper87 <- read the warning15:50
kgriffsare you proposing this as a way to do findandmodify for each new message, or just when a worker dies and a counter gets "frozen"?15:50
kgriffsflaper87: yes, I understand the scaling implications.15:50
flaper87kgriffs: mmh I'd say when the counter gets frozen15:51
flaper87I mean, aren't we going to call findAndModify for each post operation?15:52
kgriffsi see, we check if it is > the last inserted K and if not then we set it to k+115:52
kgriffsso if two workers do that at the same time, they will set it to the same value15:52
flaper87well, findAndModify happens 1 at a time15:52
kgriffsoh, maybe not findandmodify then15:53
flaper87we can teak the query and say: if counter < x, incr +115:53
kgriffsto many windows15:53
kgriffsyeah, do one will have the query fail15:53
kgriffsso it doesn't get incremented twice15:53
kgriffsi like it15:54
kgriffsI guess that is cool as long as all the messages don't expire in the mean time15:54
kgriffsthen you loose your reference point15:54
kgriffsso, there is a slim chance that a worker could insert a message, die before updating15:55
kgriffsthen several minutes go by, the message expires15:55
kgriffsthen a client posts a new message15:55
kgriffsseems like the only way to detect that is to use a heuristic15:56
kgriffslets see, if we also insert a "last modified" timestamp per counter15:57
kgriffsthen you could do a findandmodify that is like15:58
kgriffsquery: { last_modified: {$lt: now - fudge_sec}}15:59
kgriffsso, only update it if it hasn't been updated for a few seconds15:59
kgriffswe would only do that if the queue is empty (last 'k' is unknown)16:00
flaper87(current status: )16:01
kgriffsalthough, I guess if fudge_sec is like 2, then maybe we can just always do that16:01
*** zyuan has joined #openstack-marconi16:01
*** zyuan has quit IRC16:01
*** zyuan has joined #openstack-marconi16:02
flaper87mmh, yeah, I think we can use the last_modified field for that16:02
flaper87(btw, meeting time)16:02
flaper87ykaplan: ^16:02
*** zyuan_ has quit IRC16:02
*** tedross has joined #openstack-marconi16:06
*** cpritchett has joined #openstack-marconi16:36
zyuanRT @HackerNewsOnion: Obama – "The national technical debt is increasing at an alarming rate. We have to stop writing PHP."16:56
flaper87btw guys, I forgot to mention: Marconi17:04
flaper87damn it17:04
flaper87MArconi's migration under openstack/* will be discussed in the next -infra meeting
flaper87(Meeting is tomorrow)17:04
flaper87kgriffs: ^17:05
*** openstackgerrit has quit IRC17:05
*** openstackgerrit has joined #openstack-marconi17:05
flaper87malini: did you create the bp or you want me to do it ?17:05
maliniflaper87: I'll let you do it17:05
flaper87malini: kk17:05
malinican steal your actions ;)17:05
malinican't *17:05
flaper87I can give them away17:05
flaper87who wants the bp creation action?17:06
flaper87( malini take your chance)17:06
maliniflaper87: we need to have the tempest tests in the next 6 weeks ?17:06
*** cpritchett has quit IRC17:06
flaper87malini: nope, we need to have it 6 weeks before the next release17:06
flaper87which gives us a bunch of weeks17:06
malinitht is good :)17:07
maliniI was getting worried17:07
maliniI'll start looking at tempest though17:08
flaper87malini: what's your nick on lp ?17:09
kgriffsflaper87: I understand the integrated projects are currently in feature freeze, right?17:10
flaper87kgriffs: correct17:10
flaper87all frozen17:10
kgriffsok, I think we should frozen too, with the exception of the proxy which is critical for scaling17:10
kgriffsfor Icehouse we will obviously have to be super strict, tho17:11
flaper87kgriffs: agreed!17:11
flaper87kgriffs: when you get a chance:
flaper87that's the one of the series17:12
flaper87malini: I saw your comments17:12
flaper87malini: can we do the .format as part of a separate patch?17:12
flaper87since we just discussed that17:12
maliniflaper87: sure17:13
flaper87and re setUpClass, the issue is with loading the config instance17:13
flaper87(read the comment in functional/
flaper87that's why I changed that from setUpClass to setUp17:13
flaper87I don't think we should be using setUpClass there17:13
flaper87anyway, brb!17:14
* flaper87 is starving17:14
kgriffsI made it a bug17:14
kgriffsI will make other perf issues bugs as well17:15
maliniflaper87: thanks..makes sense now17:15
maliniSorry I missed tht comment17:15
flaper87malini: no worries :D17:17
maliniThat leaves us with just one, the last comment in
maliniflaper87: can you assign me to tht bp ?17:18
flaper87malini: yup, what's your lp nick ?17:18
flaper87malini: re the last comment, The metadata endpoint belongs to a specific queue:
flaper87malini: assigned to you17:20
maliniflaper87: But the base_url is the same for all queues,eg.  '' + '/queues/queuename'17:21
flaper87malini: replied here as well:
flaper87well, actually, I think you're right there. Plus, queue_url is not being used in that test case17:22
flaper87good catch17:22
flaper87I'll get rid of that17:23
flaper87ok, brb, dinner17:23
flaper87thanks malini17:23
*** ykaplan has quit IRC17:35
kgriffsmalini: have you verified this is fixed?
kgriffsnevermind, just saw the comment17:36
*** cppcabrera has joined #openstack-marconi17:40
openstackgerritA change was merged to stackforge/marconi: Pull actual tests out of marconi/tests
openstackgerritA change was merged to stackforge/marconi: Remove util packages
openstackgerritA change was merged to stackforge/marconi: Use oslo.config for functional tests
cppcabreraSeems I've walked in on a merge-a-thon. :P17:45
cppcabreraHeya, guys.17:45
flaper87cppcabrera: HELLOOOOOOOOOOOOOOOOOOOO17:52
flaper87cppcabrera: so happy to have you BACK!!!!!17:52
*** kgriffs is now known as kgriffs_afk17:53
* flaper87 steps out again17:53
cppcabreraHahaha, glad to be back, flaper87. :)17:54
cppcabreraToday's a day off for me, so I'm just sitting back and catching up on everything I've missed.17:54
*** jergerber has quit IRC17:56
*** kgriffs_afk is now known as kgriffs18:06
*** ayoung is now known as ayoung-afk18:29
flaper87cppcabrera: yo yo yo!!!!19:02
flaper87hope you're ok, enjoy your day off19:02
cppcabrerathanks, flaper87!19:03
flaper87btw, guys, still very experimental / young but, in case you want to give it a try
* flaper87 is already using it to query gerrit from cli19:08
kgriffsflaper87: excellent19:10
flaper87kgriffs: pip install gerrit-cli19:10
* kgriffs does that19:11
flaper871 thing is really annoying, you have to specify the server, I use .ssh/config though19:11
flaper87kgriffs: this is what the command looks like: gerrit reviews list review --projects=stackforge/marconi19:12
flaper87I'll make it read .gerritconfig19:12
flaper87so, if you're in marconi's dir, it'll read that19:12
kgriffsthat would awesome sauce19:14
kgriffs"Incorrect padding"19:14
kgriffsdoesn't like my rsa key19:14
flaper87mmmmhh, you using spaces instead of tabs?19:14
flaper87is it complaining about .ssh/config ?19:14
kgriffsno, doesn't seem to like my identity key19:18
kgriffsit's 2048 RSA, does that matter?19:18
kgriffs(vs. 1024)19:18
flaper87kgriffs: mmh, don't think so. Try ssh port19:20
*** ayoung-afk is now known as ayoung19:21
kgriffsport 29418 ?19:22
flaper87kgriffs: yup19:22
kgriffsstupid mac19:26
kgriffshad to put -p in there19:26
kgriffsssh -p 2941819:26
kgriffsanyway, it works fine19:26
flaper87ah sorry, actually, -p is required in linux as well :D19:27
kgriffsheh, I should have known better19:27
kgriffsflaper87: got a sec to talk about the indexing patch?19:42
maliniflaper87: How do I specify the 'MARCONI_TESTS_CONFIGS_DIR' for the refactored functional tests?19:44
flaper87kgriffs: yup19:48
flaper87malini: export MARCONI_TESTS_CONFIG_DIR=/path/to/configs/dir19:49
kgriffsflaper87: ok, I just added a comment in reply19:49
kgriffsTL;DR it's a good idea to leave 'e' out of the index19:49
flaper87kgriffs: awesome19:50
flaper87I've to admit it's bit weird that leaving fields out of the index actually makes the query faster19:50
openstackgerritKurt Griffiths proposed a change to stackforge/marconi: fix: Requests get slower when queues have a lot of messages
kgriffsflaper87: yes, we may see different characteristics depending on # of messages in the queue19:51
openstackgerritKurt Griffiths proposed a change to stackforge/marconi: fix: Requests get slower when queues have a lot of messages
kgriffsflaper87: ^^ Any other comments?19:56
flaper87kgriffs: yeah, just 119:58
flaper87kgriffs: LGTM19:58
flaper87(waiting for Jenkins to bless it)19:59
kgriffsI will have some folks benchmark it a final time to be sure19:59
kgriffsI'll get malini and/or oz to +1 and then it should be good to merge at you convenience.19:59
kgriffs(+1 after they benchmark)19:59
*** tedross has quit IRC20:01
flaper87kgriffs: awesome, I'll wait for they bless as well then!20:04
kgriffsrock on20:04
*** kgriffs is now known as kgriffs_afk20:06
*** kgriffs_afk is now known as kgriffs20:12
*** tedross has joined #openstack-marconi20:17
*** torgomatic has quit IRC20:31
*** kgriffs is now known as kgriffs_afk20:31
*** russell_h has quit IRC20:31
*** torgomatic has joined #openstack-marconi20:32
*** kgriffs_afk is now known as kgriffs20:33
*** russell_h has joined #openstack-marconi20:34
*** kgriffs is now known as kgriffs_afk20:39
*** cppcabrera has quit IRC20:48
*** oz_akan_ has quit IRC21:08
*** oz_akan_ has joined #openstack-marconi21:08
openstackgerritZhihao Yuan proposed a change to stackforge/marconi: feat(mongo): use UNIX timestamp instead of datetime
openstackgerritZhihao Yuan proposed a change to stackforge/marconi: feat(mongo): use UNIX timestamp instead of datetime
*** tedross has quit IRC21:33
*** malini is now known as malini_afk21:38
*** tedross has joined #openstack-marconi21:49
*** kgriffs_afk is now known as kgriffs21:53
*** oz_akan_ has quit IRC21:58
*** tedross has quit IRC22:14
*** kgriffs is now known as kgriffs_afk22:38
*** flaper87 is now known as flaper87|afk22:51
*** oz_akan_ has joined #openstack-marconi23:09
*** oz_akan_ has quit IRC23:13
*** tedross has joined #openstack-marconi23:14
*** oz_akan_ has joined #openstack-marconi23:20
*** tedross has quit IRC23:39
*** oz_akan_ has quit IRC23:55
*** oz_akan_ has joined #openstack-marconi23:56

Generated by 2.14.0 by Marius Gedminas - find it at!