Monday, 2014-10-13

« Sunday, 2014-10-12 Index Tuesday, 2014-10-14 »
*** zhiyan|afk is now known as zhiyan02:28
*** jeffrey4l has joined #openstack-zaqar02:29
openstackgerritPeng Fei Wang proposed a change to openstack/zaqar: Make the refrence to storage.errors to be consistent  https://review.openstack.org/11655102:30
*** jeffrey4l has quit IRC03:24
*** echevemaster has quit IRC03:36
*** jeffrey4l has joined #openstack-zaqar03:37
*** sgotliv__ has joined #openstack-zaqar04:49
*** sgotliv__ has quit IRC04:50
*** sgotliv has joined #openstack-zaqar04:50
*** exploreshaifali has joined #openstack-zaqar05:52
*** jeffrey4l has quit IRC05:53
*** exploreshaifali has quit IRC05:53
*** rrr has joined #openstack-zaqar05:59
*** rrr has quit IRC06:07
*** jeffrey4l has joined #openstack-zaqar06:08
*** sgotliv has quit IRC07:19
*** tmu has quit IRC08:13
*** tmu has joined #openstack-zaqar08:22
*** tmu has quit IRC08:40
openstackgerritFlavio Percoco proposed a change to openstack/zaqar: Fix ttl and grace story for claims updates  https://review.openstack.org/12751808:44
*** exploreshaifali has joined #openstack-zaqar08:44
*** flaper87|afk is now known as flaper8708:45
flaper87exploreshaifali: in order to run tests you need to install test-requirements.txt08:47
exploreshaifalihi flaper87  :)08:50
flaper87exploreshaifali: hi :D08:50
exploreshaifaliso probably it is needed to add same at #openstack-opw,#openstack-zaqar,#openstack-horizon,#openstack-neutron,#openstack-trove,#openstack-glance08:50
exploreshaifalisorry here https://wiki.openstack.org/wiki/Zaqar/Give_Zaqar_a_try#Running%20test08:51
flaper87exploreshaifali: yeah, we need to replace `pip install ddt mock` with `pip install -r test-requirements.txt`08:52
exploreshaifali:)08:52
flaper87that said, it shouldn't fail if redis is not installed08:53
flaper87something we need to look at08:53
flaper87exploreshaifali: could you please update the wiki page?08:53
exploreshaifaliflaper87: ya sure!08:53
exploreshaifaliflaper87: also I want to know since in this project, we will be changing whole data layer of zaqar08:54
exploreshaifaliso I will be needed to modify transport layer code a bit08:54
*** tmu has joined #openstack-zaqar08:54
exploreshaifalias they both interact with each other08:54
exploreshaifaliflaper87:  as data  data  go to transport layer from storage layer, so after separating the planes we will also need to modify the transport plane, as per the changes made in data plane08:56
flaper87exploreshaifali: mmh, could you elaborate more on the changes you think are needed?08:58
exploreshaifaliflaper87: I am not sure weather they will be needed or not, but may be the way transport layer interact with data palne will change after separating the two planes of storage layer08:59
flaper87exploreshaifali: I don't think they would be needed but we'll figure it out later. Nonetheless, I'd recommend you to always try to support guesses with a bit of research in order to have a clearer idea of what's needed and the likelihood of that guess to be true.09:03
flaper87exploreshaifali: you're doing a great job in your research on Zaqar's code, btw. Keep it up.09:04
*** sgotliv has joined #openstack-zaqar09:04
exploreshaifaliflaper87: Thanks for the complement :)09:04
exploreshaifaliso at present I need to focus only of storage code?09:05
jeffrey4lflaper87, morning sir! Could u read the comments I make on https://bugs.launchpad.net/zaqar/+bug/1373464 when u r free.09:05
flaper87jeffrey4l: gooood morning09:05
flaper87jeffrey4l: ohhh, I wanted to talk with you about that oslo.cfg comment09:05
flaper87I read it... yesterday? I think.09:05
jeffrey4lok.09:05
jeffrey4lwhat do you think?09:05
flaper87jeffrey4l: TL;DR: I disagree :D09:05
jeffrey4l;(09:06
flaper87global objects is exactly what we don't want09:06
flaper87hence all the object passing09:06
flaper87I've a work in progress patch that'll get rid completely of the global CONF09:06
jeffrey4lOK.09:06
flaper87the reason we were (are) using it is because other libraries depended on that09:06
flaper87and we were forced to do so09:06
flaper87that said, the idea is to keep everything as local as possible to avoid side-effects09:07
jeffrey4lok. understand.09:07
flaper87I know it can be confusing since we're kind of in the middle between global and local things09:07
flaper87but the global bit is because there's some technical debt in openstack we're working on fixing09:07
flaper87and lets not talk about that `dynamic_conf` thing :P09:07
jeffrey4lAre u mean debt oslo.conf?09:08
jeffrey4lAre u mean debt in oslo.conf?09:08
flaper87not oslo.conf, I mean openstack :P09:08
flaper87openstack (as in all projects) used to rely in a global object09:08
jeffrey4lYes. I make the comment based on the nova's code.09:09
flaper87jeffrey4l: actually, https://wiki.openstack.org/wiki/Oslo#Why_does_oslo.config_have_a_CONF_object.3F_Global_object_SUCK.2109:09
flaper87read that09:09
flaper87:D09:09
* jeffrey4l is reading09:10
*** exploreshaifali has quit IRC09:11
*** tmu has quit IRC09:14
*** exploreshaifali has joined #openstack-zaqar09:18
exploreshaifaliflaper87: so at present I need to focus only on storage layer's code?09:20
flaper87exploreshaifali: yes, I'd say so!09:20
exploreshaifaliokay :)09:21
exploreshaifaliflaper87: do I need to do run  * pip install test-requirements.txt* for installing test requirements?09:36
exploreshaifalirunning *install requirements.txt* says 'missing destination file  operand after ‘test-requirements.txt’09:37
exploreshaifaliso what will be the destination directory?09:38
exploreshaifalirunning *install test-requirements.txt .* also didn't work09:39
jeffrey4lexploreshaifali, try `pip install -r test-requirements.txt`09:39
*** tmu has joined #openstack-zaqar09:39
exploreshaifalijeffrey4l: cool :)09:39
jeffrey4lflaper87, read the wiki/maillist. I understand why we didn't chose the global CONF. But a another question, is there a good pattern to register opt in runtime? It seems that the zaqar still have the issue on that( the admin_mode registered twice in bootstrap.py)09:47
flaper87jeffrey4l: yeah, that's exactly what I'd like to fix in this patch. I will have to rebase the patch since it hasn't been updated in a bit09:49
*** prashanthr_ has joined #openstack-zaqar09:49
flaper87that said, there's not, the pattern is to register_options when needed09:49
flaper87which most of the time ends up being in the constructor09:49
flaper87I think the overall structure makes sense, we should probably just focus on fixing tests09:50
flaper87ok, gtg now! I'm on the road09:50
flaper87ttyl09:50
jeffrey4lok.09:50
jeffrey4lsee u later.09:50
*** flaper87 is now known as flaper87|afk09:51
*** jeffrey4l has quit IRC10:01
*** bradjones has joined #openstack-zaqar10:06
*** njirap has joined #openstack-zaqar10:14
*** jeffrey4l has joined #openstack-zaqar10:17
*** sgotliv has quit IRC10:24
*** njirap has quit IRC10:32
*** njirap has joined #openstack-zaqar10:45
*** exploreshaifali has quit IRC10:50
*** sgotliv has joined #openstack-zaqar10:57
*** sgotliv has quit IRC11:04
*** sgotliv has joined #openstack-zaqar11:05
*** X019___ has quit IRC11:07
*** exploreshaifali has joined #openstack-zaqar11:18
*** X019___ has joined #openstack-zaqar11:19
*** X019___ has quit IRC11:34
*** exploreshaifali has quit IRC12:33
*** prashanthr_ has quit IRC12:56
*** exploreshaifali has joined #openstack-zaqar13:02
*** jchai has joined #openstack-zaqar13:04
*** sriram has joined #openstack-zaqar13:05
*** ametts has joined #openstack-zaqar13:09
*** sgotliv has quit IRC13:26
*** prashanthr_ has joined #openstack-zaqar13:31
*** malini has joined #openstack-zaqar13:37
*** sgotliv has joined #openstack-zaqar13:40
openstackgerritJeffrey Zhang proposed a change to openstack/zaqar: Fix the doc typo  https://review.openstack.org/12796414:02
*** mpanetta has joined #openstack-zaqar14:30
*** amitgandhinz has joined #openstack-zaqar14:30
*** mpanetta has quit IRC14:31
*** mpanetta has joined #openstack-zaqar14:32
*** X019 has joined #openstack-zaqar14:46
*** jchai is now known as jchai_afk15:10
*** prashanthr_ has quit IRC15:14
*** kgriffs|afk is now known as kgriffs15:26
*** earnThis has joined #openstack-zaqar15:29
*** mpanetta_ has joined #openstack-zaqar15:29
*** mpanetta has quit IRC15:33
*** mpanetta_ has quit IRC15:34
*** mpanetta has joined #openstack-zaqar15:34
*** amitgandhinz has quit IRC15:37
*** njirap has quit IRC15:38
*** sgotliv has quit IRC15:40
*** njirap has joined #openstack-zaqar15:51
*** njirap has quit IRC15:59
*** njirap has joined #openstack-zaqar15:59
*** exploreshaifali has quit IRC16:00
*** amitgandhinz has joined #openstack-zaqar16:04
*** jchai_afk is now known as jchai16:07
*** sgotliv has joined #openstack-zaqar16:30
*** sgotliv has quit IRC16:36
*** exploreshaifali has joined #openstack-zaqar16:38
*** njirap has quit IRC17:04
kgriffsmalini: if you have a sec, could you review this? https://review.openstack.org/#/c/126947/17:12
*** amitgandhinz has quit IRC17:16
*** earnThis has quit IRC17:16
malinisure17:19
*** sgotliv has joined #openstack-zaqar17:19
malinikgriffs: Do you still need this https://github.com/openstack/zaqar/blob/master/zaqar/tests/__init__.py#L23 ?17:24
kgriffsyes, there's one test that I wasn't able to reduce... but let me check17:24
kgriffsok, confirmed17:25
kgriffsthere is a mongo test that hast to weight quite a while to ensure the mongod GC has occurred, since we can't influence that from the app17:25
kgriffss/weight/wait17:26
* kgriffs wonders where that typo came from!17:26
maliniI was surprised it came from you :D17:26
maliniAre you the real kgriffs?17:27
kgriffsshhhh, don't tell anybody!17:27
kgriffs:)17:27
openstackgerritA change was merged to openstack/zaqar: Fix no links property in pool list response  https://review.openstack.org/12346217:34
*** jchai is now known as jchai_afk17:39
openstackgerritA change was merged to openstack/zaqar: Reduce duration of some long-running tests  https://review.openstack.org/12694717:59
*** jchai_afk is now known as jchai18:11
*** amitgandhinz has joined #openstack-zaqar18:18
openstackgerritOpenStack Proposal Bot proposed a change to openstack/zaqar: Updated from global requirements  https://review.openstack.org/12780018:19
*** flaper87|afk is now known as flaper8718:31
flaper87o/18:33
*** mlilenium_ has joined #openstack-zaqar18:43
*** mlilenium_ has left #openstack-zaqar18:43
exploreshaifaliflaper87, do we not have storage layer in zaqarclient?18:47
exploreshaifalihttps://github.com/openstack/python-zaqarclient/tree/master/zaqarclient18:48
flaper87exploreshaifali: nope, the client library maps to the API18:52
flaper87not to the layers18:52
flaper87so, you'll see the resources implemented there that talk to the API endpoints18:52
exploreshaifaliokay, got it:)18:52
exploreshaifaliand once we were talking about separating tht databse18:53
exploreshaifali*the18:53
exploreshaifalione for data layer and one for control layer18:53
exploreshaifaliso from where I can see that database?18:54
exploreshaifalieven after installing zaqar, I am unable to see the database18:54
exploreshaifaliflaper87, ^18:55
flaper87exploreshaifali: mmmh, not sure what you mean :(18:57
exploreshaifaliflaper87, we have common database for both the planes right?18:57
exploreshaifalithere is only single database for whole storage layer18:58
exploreshaifaliand we need to separate it too18:58
*** amitgandhinz has quit IRC19:04
*** amitgandhinz has joined #openstack-zaqar19:07
flaper87kgriffs: arpund?19:23
flaper87exploreshaifali: yep19:23
flaper87around, that is19:23
kgriffso/19:23
flaper87kgriffs: yo bro, how ya' doin' ?19:24
flaper87kgriffs: https://git.openstack.org/cgit/openstack/zaqar/tree/zaqar/tests/queues/storage/base.py#n84819:24
exploreshaifaliflaper87, in order to separate it first I need to look at it and understand it, so how can i access it?19:25
flaper87does that sound correct to you? We're currently setting the ttl for messages that would expire *before* the ttl+grace does19:25
flaper87kgriffs: but, that doesn't sound right19:25
flaper87kgriffs: we should re-set the ttl for messages whom would expire before the *claim*19:25
flaper87it's not a harmful bug, we're extending the life of more messages that we should19:26
flaper87but I think that logic is wrong19:26
flaper87exploreshaifali: are you using mongo ?19:26
exploreshaifaliflaper87, yes19:26
kgriffsflaper87: looking19:26
flaper87exploreshaifali: then I'd recommend the mongodb shell19:26
flaper87exploreshaifali: are you on ubuntu?19:26
exploreshaifaliyes, ubuntu14.0419:27
flaper87just make sure you have the client `mongo` installed19:27
flaper87run it and enjoy19:27
exploreshaifalihaha yup19:27
exploreshaifaliflaper87, so to start this project, i think first task should be dividing the database?19:28
flaper87first task is splitting the configs19:28
exploreshaifaliflaper87, I am unable to make much sense from https://etherpad.openstack.org/p/zaqar-scratch19:29
flaper87exploreshaifali: ah yeah, sorry about that19:29
kgriffsflaper87: seems like your patch to add grace to claim updates was extending message lifetimes if they expired before the claim itself, right?19:29
exploreshaifaliI am trying to understand, but things are not getting clear to me19:29
kgriffsflaper87: whereas originally only messages that would expire before claim_ttl + grace would be updated?19:30
flaper87kgriffs: we already do that: https://git.openstack.org/cgit/openstack/zaqar/tree/zaqar/queues/storage/mongodb/claims.py#n18419:31
flaper87I thought it was my patch but nope, we're getting messages <= message_expiration19:31
flaper87(which has grace already added19:31
flaper87)19:31
flaper87and that should be, imho, claim_expires19:32
kgriffsok, that was my next question19:32
kgriffsah, yes, I thought I remembered this being changed in ur patch19:34
kgriffshttps://review.openstack.org/#/c/127518/3/zaqar/queues/storage/mongodb/claims.py19:34
flaper87yup, I noticed it and changed it but then many tests broke so, before I go ahead and fix them, I wanted to double check with you19:35
flaper87I think the fix on that patch is correct19:35
kgriffsyes, I think it is OK.19:35
flaper87without 200 ?19:36
flaper87:(19:36
kgriffsbut now that has me wondering about the redis driver19:36
* flaper87 listens19:36
kgriffsflaper87: without 200? what?19:36
* kgriffs is confused19:36
flaper87kgriffs: 200 OK19:36
flaper87:P19:36
kgriffsoh19:36
* kgriffs facepalm19:36
kgriffs:)19:36
flaper87#badumps19:36
flaper87:P19:36
kgriffsre the redis driver, I was just wondering if it is updating the message expiration19:37
flaper87oh, mmh, I should probably check that19:37
*** malini has quit IRC19:38
kgriffsok, so looks like claim_messages.lua needs to be updated19:40
kgriffsand also19:41
kgriffsredis/claims.py:34119:41
kgriffsP.S. - Feel free to remove that _msg_would_expire function and just do the comparison inline; I don't think that function is used anywhere else19:42
kgriffsclear as mud?19:42
flaper87sorry, I was distracted19:47
flaper87yeah, clear. I'll update that19:47
kgriffskewl19:47
flaper87also, I just noticed we're using date objects for the expire and timestamps for the claim.expires19:47
flaper87do you recall why?19:47
kgriffssomething to do with mongo TTL index requirements?19:48
flaper87ah right19:48
flaper87yeah19:48
flaper87damn, you've good memory19:48
kgriffslol19:48
kgriffsflaper87: btw, the more I think about the idea of having one redis instance for the msg ID set index, and other nodes for the actual messages, the more I like it, but should talk more at the summit19:50
kgriffsfor example, this would let us use the new redis cluster thingy19:50
flaper87kgriffs: I was thinking today about how bad would it be to *always* require redis to be there19:50
kgriffsi still think if we can get all openstack projects to support redis for cache in addition to memcached, it would be an easier sell19:52
kgriffslets people run one less DB flavor in their deployments19:52
flaper87kgriffs: tooz is adding support for redis as a replacement for zookeper19:53
flaper87not the default but a valid option19:53
kgriffsoh19:53
kgriffswhat about etcd?19:53
flaper87no idea19:53
kgriffsrandom trivia: coreos was founded by some ex-Rackers (Rackspace employees)19:54
kgriffsanyway, seems like if you are going to replace zookeeper, why not just use etcd rather than having to invent your own shared state thingy19:55
kgriffsbut whatever19:55
flaper87kgriffs: it's not replacement, it's another supported backend19:58
flaper87there could be one for etcd19:58
kgriffsi mean, I guess seems like etcd would less work to add as a backend than redis, but I could be wrong19:59
kgriffsaaaanyqay19:59
kgriffsyeah, if redis starts being used more19:59
kgriffsthat would be groovy20:00
kgriffsflaper87: btw, if we use caching more in zaqar, I think we should invest some time to create a tiered cache model - L1 (localhost) and L2 (centralized)20:02
kgriffsyou could even add one more20:03
kgriffsL1 - in-process20:03
kgriffsL2 - shared localhost, between processes20:03
kgriffsL3 - shared between boxes20:03
flaper87boris-42: yeah, my thoughts today were running a bit around that. I was thinking about the notifications thing20:03
flaper87and how useful that could be20:03
exploreshaifaliflaper87, configs are defined inside options.py file, right? https://github.com/openstack/zaqar/blob/master/zaqar/queues/storage/mongodb/options.py#L720:03
boris-42flaper87 what?)20:04
flaper87boris-42: erm, I meant kgriffs20:04
boris-42flaper87 huh=)20:04
*** amitgandhinz has quit IRC20:11
*** amitgandhinz has joined #openstack-zaqar20:12
openstackgerritFlavio Percoco proposed a change to openstack/zaqar: Fix ttl and grace story for claims updates  https://review.openstack.org/12751820:13
*** amitgandhinz has quit IRC20:16
flaper87guys, there're no items in the agenda for today and I think I won't be able to make it to the meeting. Is it ok if we skip it this week?20:16
*** amitgandhinz has joined #openstack-zaqar20:17
flaper87gtg, bbib20:18
*** flaper87 is now known as flaper87|afk20:18
kgriffsttfn20:20
*** kgriffs is now known as kgriffs|afk20:41
*** flaper87|afk is now known as flaper8720:49
exploreshaifaliflaper87, configs that we need to separate very first comes from https://github.com/openstack/zaqar/blob/master/zaqar/queues/storage/mongodb/options.py21:31
exploreshaifaliright?21:31
flaper87exploreshaifali: yup, for mongodb yes21:31
flaper87we need to do this for sqlalchemy too21:32
exploreshaifaliyes21:32
exploreshaifaliso these will be configs specific for mongodb21:32
exploreshaifaliflaper87, and will get initialized when conf object will be initialized?21:34
flaper87kgriffs|afk: malini|afk https://review.openstack.org/#/c/127518/21:34
flaper87exploreshaifali: what do you mean with `conf object` initialized?21:37
flaper87the config options are registered when the driver is instantiated21:37
exploreshaifaliokay, I was talking about conf at line 70 and 34 https://github.com/openstack/zaqar/blob/master/zaqar/queues/storage/mongodb/driver.py21:39
exploreshaifalidoes that options.py is related with this conf or not?21:39
exploreshaifaliflaper87, ^21:39
flaper87yup yup21:40
flaper87ok, gtg now21:41
flaper87ttyt21:41
*** flaper87 is now known as flaper87|afk21:41
exploreshaifaliwhen will we able to meet next?21:41
*** sriram has quit IRC21:49
*** jchai has quit IRC22:02
*** mpanetta has quit IRC22:10
*** amitgandhinz has quit IRC22:10
*** cpallares has joined #openstack-zaqar22:13
*** ametts has quit IRC22:55
*** openstackgerrit has quit IRC23:03
*** openstackgerrit has joined #openstack-zaqar23:04
openstackgerritOpenStack Proposal Bot proposed a change to openstack/zaqar: Updated from global requirements  https://review.openstack.org/12780023:12
*** amitgandhinz has joined #openstack-zaqar23:33
*** openstackgerrit has quit IRC23:33
*** openstackgerrit has joined #openstack-zaqar23:34
*** yfujioka has joined #openstack-zaqar23:39
« Sunday, 2014-10-12 Index Tuesday, 2014-10-14 »

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!