Friday, 2015-04-03

« Thursday, 2015-04-02 Index Saturday, 2015-04-04 »
*** kfox1111 has quit IRC00:50
*** johnbelamaric has joined #openstack-dns00:57
*** johnbelamaric has quit IRC00:58
*** mariusv has quit IRC01:00
*** rektide has quit IRC01:02
*** mariusv has joined #openstack-dns01:02
*** mariusv has quit IRC01:02
*** mariusv has joined #openstack-dns01:02
*** rektide has joined #openstack-dns01:02
*** dcmartin has joined #openstack-dns01:11
*** ericpeterson has joined #openstack-dns01:16
*** ericpeterson has quit IRC01:26
*** ericpeterson has joined #openstack-dns01:29
*** MentalRay has quit IRC01:46
*** MentalRay has joined #openstack-dns01:53
*** elarson has quit IRC02:00
*** MentalRay has quit IRC02:08
*** GonZo2K has joined #openstack-dns02:31
*** mwagner_lap has quit IRC02:46
*** mwagner_lap has joined #openstack-dns02:59
*** ericpeterson has quit IRC03:11
*** chlong has joined #openstack-dns03:48
*** chlong has quit IRC03:49
*** dcmartin has quit IRC04:20
*** MentalRay has joined #openstack-dns04:28
*** richm has quit IRC04:37
*** penick has joined #openstack-dns04:41
*** penick has quit IRC04:42
*** penick has joined #openstack-dns04:46
*** penick has quit IRC04:46
*** GonZoPT has joined #openstack-dns04:50
*** GonZo2K has quit IRC04:52
*** penick has joined #openstack-dns05:15
*** penick has quit IRC05:15
*** penick has joined #openstack-dns05:33
*** penick has quit IRC05:56
*** penick has joined #openstack-dns07:32
*** penick has quit IRC07:40
*** fawadkhaliq has joined #openstack-dns07:52
*** GonZoPT has quit IRC07:58
*** penick has joined #openstack-dns08:02
*** fawadk has joined #openstack-dns08:08
*** MentalRay has quit IRC08:11
*** fawadkhaliq has quit IRC08:12
*** fawadk has quit IRC08:13
*** penick has quit IRC08:19
*** jordanP has joined #openstack-dns08:20
*** fawadkhaliq has joined #openstack-dns08:54
*** untriaged-bot has joined #openstack-dns10:01
untriaged-botUntriaged bugs so far:10:01
untriaged-bothttps://bugs.launchpad.net/designate/+bug/143769910:01
openstackLaunchpad bug 1437699 in Designate "mDNS should Handle Bad File Descriptor" [Medium,New] - Assigned to Ron Rickard (rjrjr)10:01
*** untriaged-bot has quit IRC10:01
*** fawadkhaliq has quit IRC11:22
*** km has quit IRC11:32
openstackgerritGraham Hayes proposed openstack/designate: Fixed sort key to not sort by an un-indexed field by default  https://review.openstack.org/17021311:50
*** richm has joined #openstack-dns12:16
*** ericpeterson has joined #openstack-dns12:20
*** fawadkhaliq has joined #openstack-dns12:23
*** fawadkhaliq has quit IRC12:28
*** lkoranda has quit IRC12:29
*** johnbelamaric has joined #openstack-dns12:31
*** fawadkhaliq has joined #openstack-dns12:46
*** ericpeterson has quit IRC12:46
*** kei_yama has quit IRC12:57
*** jmcbride has joined #openstack-dns13:17
*** elarson has joined #openstack-dns13:18
*** ericpeterson has joined #openstack-dns13:33
*** openstackgerrit has quit IRC13:36
*** openstackgerrit has joined #openstack-dns13:36
*** ChanServ sets mode: +v openstackgerrit13:36
*** jmcbride has quit IRC13:56
*** jmcbride has joined #openstack-dns13:56
*** vinod1 has joined #openstack-dns14:00
*** ericpeterson has quit IRC14:07
*** paul_glass has joined #openstack-dns14:13
*** jordanP has quit IRC14:26
*** openstackgerrit has quit IRC14:39
*** openstackgerrit has joined #openstack-dns14:39
*** ChanServ sets mode: +v openstackgerrit14:39
paul_glasswhat is the difference between a pool_nameserver and a pool_target in designate.conf?14:48
*** fawadkhaliq has quit IRC14:49
mugsiepaul_glass: a target is where we write changes, nameserver is where we read from14:53
timsimA nameserver is the thing that gets polled, so where the change actually resolves, the target is where the change is made. So for BIND9, that'd be where you'd put options like rndc host/port. For powerdns, that'd be  where you put a database connection string, for the agent, that's where you put the host/port that is different than the host/port to poll14:53
mugsie(e.g. with akamai, we write to the REST API, and read from the nameservers)14:54
mugsieor PowerDNS we write the new zone in DB, but could read from totally differnt servers14:54
paul_glassthat makes sense. thanks.14:55
*** dcmartin has joined #openstack-dns15:16
*** dcmartin has quit IRC15:17
*** dcmartin has joined #openstack-dns15:18
*** Kiall has quit IRC15:22
*** Kiall has joined #openstack-dns15:22
openstackgerritPaul Glass proposed openstack/designate: Remove zone wrapper object in Tempest tests  https://review.openstack.org/17053815:30
*** vinod1 has quit IRC15:39
*** ericpeterson has joined #openstack-dns15:41
elarsonI'm trying to create a zone and getting this error. http://paste.openstack.org/show/198143/ is this b/c there aren't any masters defined?15:41
timsimMost likely elarson15:42
elarsonwe should probably update the docs15:42
* elarson looks for a ticket15:42
timsimThe sample config?15:43
timsimOh jk, yeah15:43
elarsontimsim: oh, I was thinking about the docs for creating a domain, but you're right, I should double check the config15:44
timsimYeah you need https://github.com/openstack/designate/blob/master/etc/designate/designate.conf.sample#L268 probably15:44
elarsonthat said, it arguably could default to whatever ns has been configured right?15:44
timsimFor masters? I don't think so, because most of the time that'd be itself...I think15:45
elarsontimsim: you and I should have a rap session about this ;)15:45
* timsim walks 3 steps15:45
* elarson takes 2 steps back15:46
jbrattonis this going to turn in to a dance off?15:46
* Kiall backs away slowly15:47
elarsonalready happened... I won15:49
timsimI would argue that result15:49
elarsondangit!!!15:49
*** penick has joined #openstack-dns15:50
elarsontimsim: so what is this error about then? http://paste.openstack.org/show/198146/15:54
paul_glasselarson: I just ran into that... I think I had the pool_target and pool_nameserver sections swapped.15:55
timsimI think that's this line https://github.com/openstack/designate/blob/master/etc/designate/designate.conf.sample#L26915:55
dcmartincouchdb15:56
elarsonmaybe it is couchdb15:56
elarsonwhow knows15:56
elarsontimsim: I have that uncommented for the target15:56
*** dcmartin has quit IRC15:57
mugsiewell that was odd15:58
*** andrewbogott_afk is now known as andrewbogott16:01
*** untriaged-bot has joined #openstack-dns16:01
untriaged-botUntriaged bugs so far:16:01
untriaged-bothttps://bugs.launchpad.net/designate/+bug/143769916:01
openstackLaunchpad bug 1437699 in Designate "mDNS should Handle Bad File Descriptor" [Medium,New] - Assigned to Ron Rickard (rjrjr)16:01
*** untriaged-bot has quit IRC16:01
*** penick has quit IRC16:04
elarsonhere is a tidbit16:18
timsimcouchdb?16:18
elarsonin oslo.config (and I suspect configparser as well) if you have `foo = bar  # a helpful comment` the value of foo is {'foo': 'bar  # a helpful comment'}16:19
elarsonI suppose that makes sense since you often would have characters after the `=` that you'd want to include16:19
elarsonin any case, knowing is half the battle16:19
*** bharath_ has joined #openstack-dns16:28
openstackgerritPaul Glass proposed openstack/designate: First Tempest tests for recordsets  https://review.openstack.org/16735616:29
*** vinod1 has joined #openstack-dns16:48
bharath_We are trying to setup designate juno (in ebay) and we are running into an issue for a single nova boot ... There is nothing else happening at the same time and so it seems strange that it's waiting on a lock !!! Here's the stack trace:17:08
bharath_Stack trace for create: 2015-04-03 01:28:07.740 11180 TRACE oslo.messaging.rpc.dispatcher Traceback (most recent call last): 2015-04-03 01:28:07.740 11180 TRACE oslo.messaging.rpc.dispatcher   File "/opt/openstack/designate/2015.2.0.r1/lib/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py", line 134, in _dispatch_and_reply 2015-04-03 01:28:07.740 11180 TRACE oslo.messaging.rpc.dispatcher     incoming.message)) 2015-0417:08
bharath_2015-04-03 01:28:07.740 11180 TRACE oslo.messaging.rpc.dispatcher OperationalError: (OperationalError) (1205, 'Lock wait timeout exceeded; try restarting transaction') 'INSERT INTO recordsets (id, version, created_at, tenant_id, domain_id, name, type, ttl) VALUES (%s, %s, %s, %s, %s, %s, %s, %s)' ('772bdc16437343ae874b259e510e6eac', 1, datetime.datetime(2015, 4, 3, 1, 27, 16, 152409), 'a9fac4ab8c314ed786f547a7facfa5ca', '38717:09
bharath_Any pointers would be helpful ?17:10
*** penick has joined #openstack-dns17:28
*** pk_ has joined #openstack-dns17:38
Kiallbharath_: just had a chance to have a quick look, so the underlying bug we talked about a few days back could still be causing that - even with exactly nova boot, depending on your sink handler... The bug effectivly meant a request came into central, central would start a TX and say, insert a record, it would then yield to another thread, which could be a another request from the sink handler etc, which would open a TX and then stall trying to17:38
Kiall bump the serial number up, leading to a deadlocked process which can't recover until the timeout.17:38
KiallIt was a total "how on earth did this bug go unnoticed for so long" - we had that bug in HP Cloud DNS producton for quite some time, and it didn't even show itself there :(17:39
bharath_If it's deadlocked on bumping the serial number and if I do increment_serial = False from the sink handlers, then it shouldn't run into the issue, rite ?17:42
Kiallbharath_: Ehh - I think so, the serial is the only place I know of where we deadlocked.17:44
Kiall(Also - I never tried doing that, so it's possible it won't work..)17:44
bharath_Yup, I tried that and it didn't work :(17:45
KiallOh, I just looked at the paste above and noticed the query..17:45
Kiallthat's an `insert into recordsets` getting a lock wait timeout .. Humm17:45
bharath_yes17:45
bharath_in case of nova delete, it's 'delete from records'17:46
KiallCan you fire one off, and during the 30 second window collect the output of this? SHOW ENGINE INNODB STATUS;17:47
KiallTheres also .. And I always have to go find this query on StackOverflow every time I need it ;)17:49
KiallSELECT r.trx_id waiting_trx_id,  r.trx_mysql_thread_id waiting_thread,17:49
Kiall      r.trx_query waiting_query,17:49
Kiall      b.trx_id blocking_trx_id, b.trx_mysql_thread_id blocking_thread,17:49
Kiall      b.trx_query blocking_query17:49
KiallFROM       information_schema.innodb_lock_waits w17:49
KiallINNER JOIN information_schema.innodb_trx b  ON  b.trx_id = w.blocking_trx_id17:49
KiallINNER JOIN information_schema.innodb_trx r  ON  r.trx_id = w.requesting_trx_id;17:49
*** rrickard has joined #openstack-dns17:52
*** richm has quit IRC17:53
rrickard@kiall: are you working with bharath_?17:55
KiallYea, asked him to gather some details during a lock..17:55
openstackgerritMerged openstack/designate: Remove zone wrapper object in Tempest tests  https://review.openstack.org/17053817:55
Kiallhttp://eavesdrop.openstack.org/irclogs/%23openstack-dns/%23openstack-dns.2015-04-03.log has some details ;)17:56
Kiall(not pasting that big query twice :P)17:56
KiallAbout to hop on another call, be back in a bit.17:57
*** pk_ has quit IRC17:58
rrickard@mugsie: our chat message doesn't show the location of the logs anymore?17:58
rrickardi'm just seeing a message about Zuul being stuck.17:59
Kiallrrickard: something broke while something was broke and the topic has been like that ever since.. lol18:06
*** ChanServ sets mode: +o Kiall18:06
*** Kiall changes topic to "OpenStack Designate - Logged @ http://eavesdrop.openstack.org/irclogs/%23openstack-dns"18:06
*** ChanServ sets mode: -o Kiall18:06
*** richm has joined #openstack-dns18:11
bharath_| 39A9D6         |          19257 | INSERT INTO recordsets (id, version, created_at, tenant_id, domain_id, name, type, ttl) VALUES ('cede4541c31d4ac9a8ed1bb9b25cf990', 1, '2015-04-03 18:09:23.748365', 'a9fac4ab8c314ed786f547a7facfa5ca', '3876b908dffb47a181a09570ef21387f', 'dev-test30-5655.slc01.dev.ebayc3.com.', 'A', NULL) | 39A9D4          |           19255 | NULL18:13
bharath_that's the output for the sql query18:13
bharath_only one row18:13
*** dcmartin has joined #openstack-dns18:18
bharath_http://cryptb.in/Vx6vP (key - 52c4c4ac5d516c415c67eabd330e83af)18:26
*** pk_ has joined #openstack-dns18:44
*** penick has quit IRC19:00
*** pk_ has quit IRC19:01
*** pk_ has joined #openstack-dns19:02
*** pk_ has quit IRC19:06
*** dcmartin has quit IRC19:07
*** MentalRay has joined #openstack-dns19:12
*** penick has joined #openstack-dns19:14
*** dcmartin has joined #openstack-dns19:14
*** dcmartin has quit IRC19:55
*** dcmartin has joined #openstack-dns19:55
*** ericpeterson has quit IRC20:00
*** ericpeterson has joined #openstack-dns20:00
*** ericpete_ has joined #openstack-dns20:01
*** pk_ has joined #openstack-dns20:03
*** ericpeterson has quit IRC20:06
*** pk_ has quit IRC20:07
*** MentalRay has quit IRC20:07
*** dcmartin has quit IRC20:16
*** bharath_ has quit IRC20:30
*** fawadkhaliq has joined #openstack-dns20:39
rrickard@Kiall: we have figured out why we are getting the lock contention.20:40
rrickardwe have Designate sink listening on a topic called 'dns'.20:40
rrickardnova is publishing on 'notifications, dns'.20:41
rrickardDesignate sink is consuming both 'notifications.info' and 'dns.info' even though we want it just consuming 'dns.info' and our config shows just 'dns' topic.20:41
rrickardthis was not a problem in Designate Icehouse, but appeared when we upgraded to Designate Juno.20:42
rrickardso, 2 notifications are consumed and we have lock contention.20:43
*** penick has quit IRC20:56
*** MentalRay has joined #openstack-dns21:02
ekarlsorrickard: how's life at ebay ? :p21:04
*** MentalRay has quit IRC21:13
openstackgerritTim Simmons proposed openstack/designate: WIP: Refactor notify logic into backends  https://review.openstack.org/17061221:22
*** vinod1 has quit IRC21:26
*** paul_glass has quit IRC21:41
*** pk has joined #openstack-dns21:44
*** penick has joined #openstack-dns21:49
*** rrickard has quit IRC21:58
*** andrewbogott is now known as andrewbogott_afk22:07
*** MentalRay has joined #openstack-dns22:07
*** ericpete_ has quit IRC22:08
*** pk has quit IRC22:09
*** penick has quit IRC22:30
*** pk has joined #openstack-dns22:32
openstackgerritKiall Mac Innes proposed openstack/designate: Split out Optional OpenStack services in localrc  https://review.openstack.org/17062022:34
*** penick has joined #openstack-dns22:35
*** pk has quit IRC22:37
*** pk has joined #openstack-dns22:43
*** fawadkhaliq has quit IRC22:48
Kialltimsim: if your still about, https://review.openstack.org/#/c/166191/ should finally be good to turn back on again :)22:48
openstackgerritKiall Mac Innes proposed openstack/designate: DevStack: Akamai/Dyn Support for Jenkins Credentials and Exercises  https://review.openstack.org/16989122:52
Kialland ^ ;)22:53
*** jmcbride has quit IRC22:56
openstackgerritKiall Mac Innes proposed openstack/designate: DevStack: Akamai/Dyn Support for Jenkins Credentials and Exercises  https://review.openstack.org/16989122:56
*** pk has quit IRC23:03
*** MentalRay has quit IRC23:09
boris-42Kiall ekarlso howdy23:11
Kiallheya23:13
boris-42Kiall: how are you?)23:19
KiallGood, but about to head to bed :P23:20
boris-42Kiall: -)23:21
boris-42Kiall: soo I made recently a lib that should be interesting for you=)23:21
boris-42Kiall: https://github.com/boris-42/profimp23:21
KiallOh?23:21
boris-42Kiall: yep to speed up import of python-designateclient23:22
*** pk has joined #openstack-dns23:23
KiallInteresting.. Not something I've considered optomizing before, given it's a once off startup cost - but.. Yea, the CLI starts up every command ;)23:23
Kiall(and all the OS CLIs are terribly slow to "boot" ;))23:23
boris-42Kiall:  http://boris-42.github.io/profimp/designate.html23:25
boris-42Kiall: so this is trace for designate client ^23:26
boris-42Kiall: so now we see how imports are done and where the most time is consumed23:26
boris-42Kiall: so adding in proper places lazy imports can fix this23:26
Kiall208.41 ms ain't bad in fairness!23:27
Kiall(We're dealing with python compile times ;))23:27
KiallAnyway, I'm nearly fast asleep! I'll have a look tomorrow :)23:28
boris-42Kiall: good night=)23:28
boris-42Kiall: so ping me when you find time23:28
Kiall:)23:28
boris-42Kiall: and yep 270 is bad for libs23:28
boris-42Kiall: it should be something like 20ms=)23:28
KiallAt a glance, Most of our time is spent importing PBR (we can probably kill that one) and Keystone (Can't do much about that one, 99% of use cases will need us to load it anyway, so lazy loading just defers it)23:30
Kiallactually, PBR seems (from the graph) to be coming in via Keystone Client23:31
Kiall" from keystoneclient import adapter .. 207.47 ms" out total is 208.41 ms23:31
KiallAnyway, I'm probably reading it wrong at this point of the evening :P23:32
Kiallcya - will check it out properly over the weekend :)23:32
*** penick has quit IRC23:35
boris-42Kiall: -)23:39
*** pk has quit IRC23:53
*** pk has joined #openstack-dns23:55
« Thursday, 2015-04-02 Index Saturday, 2015-04-04 »

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!