Tuesday, 2015-04-21

« Monday, 2015-04-20 Index Wednesday, 2015-04-22 »
*** EricGonc_ has quit IRC00:16
*** openstackgerrit has quit IRC00:22
*** openstackgerrit has joined #openstack-dns00:22
*** ChanServ sets mode: +v openstackgerrit00:22
*** openstack has joined #openstack-dns00:35
*** ChanServ sets mode: +v openstack00:35
*** penick has joined #openstack-dns00:48
*** encld06 has joined #openstack-dns01:25
*** fawadkhaliq has joined #openstack-dns01:25
*** fawadkhaliq has quit IRC01:29
*** mwagner_lap has joined #openstack-dns01:37
*** stanzgy has joined #openstack-dns01:50
*** echoingumesh has quit IRC02:04
*** echoingumesh has joined #openstack-dns02:04
*** echoingumesh has quit IRC02:09
*** encld06 has quit IRC02:13
*** EricGonczer_ has joined #openstack-dns02:17
*** fawadkhaliq has joined #openstack-dns02:39
*** EricGonczer_ has quit IRC02:49
*** EricGonczer_ has joined #openstack-dns03:08
*** EricGonczer_ has quit IRC03:10
*** EricGonczer_ has joined #openstack-dns03:13
*** fawadkhaliq has quit IRC03:22
*** EricGonczer_ has quit IRC03:28
*** echoingumesh has joined #openstack-dns04:23
*** echoingumesh has quit IRC04:29
*** fawadkhaliq has joined #openstack-dns04:30
*** echoingumesh has joined #openstack-dns04:30
*** echoingumesh has quit IRC04:34
*** richm has quit IRC04:38
*** echoingumesh has joined #openstack-dns05:55
*** echoingumesh has quit IRC06:00
*** echoingumesh has joined #openstack-dns06:04
*** echoingumesh has quit IRC07:01
*** echoingumesh has joined #openstack-dns07:02
*** echoingumesh has quit IRC07:07
*** penick has quit IRC07:39
*** jordanP has joined #openstack-dns08:16
*** fawadkhaliq has quit IRC09:11
*** fawadkhaliq has joined #openstack-dns09:25
*** fawadkhaliq has quit IRC09:27
stanzgyhi guys, the performance issue on powerdns with mysql has finally been solved after I moved powerdns db to a dedicated mysql server with optimized options offered by ahu_.09:33
stanzgyThen main performance bottleneck is the MySQL IO performance. When running a script keeping creating records, PDNS's mysql still costs >10MB/s IO after records of a single domain growing to >20k. There is still room for optiziming. I shall file a bug on this.09:38
*** fawadkhaliq has joined #openstack-dns09:38
*** fawadkhaliq has quit IRC09:38
*** fawadkhaliq has joined #openstack-dns09:38
*** untriaged-bot has joined #openstack-dns10:01
untriaged-botUntriaged bugs so far:10:01
untriaged-bothttps://bugs.launchpad.net/designate/+bug/144512710:01
untriaged-bothttps://bugs.launchpad.net/designate/+bug/144628110:01
untriaged-bothttps://bugs.launchpad.net/designate/+bug/144638710:01
untriaged-bothttps://bugs.launchpad.net/designate/+bug/144624210:01
openstackLaunchpad bug 1445127 in Designate "Multiple Pool Manager - every PM is running periodic sync" [Undecided,Confirmed]10:01
untriaged-bothttps://bugs.launchpad.net/designate/+bug/144490110:01
*** untriaged-bot has quit IRC10:01
openstackLaunchpad bug 1446281 in Designate "designate dashboard detail screen does not support horizon juno" [Undecided,In progress] - Assigned to Eric Peterson (ericpeterson-l)10:01
openstackLaunchpad bug 1446387 in Designate "Support Host header in REST queries" [Undecided,New] - Assigned to Cedric Brandily (cbrandily)10:01
openstackLaunchpad bug 1446242 in Designate "Allow to use network label in nova_fixed format" [Undecided,In progress] - Assigned to Cedric Brandily (cbrandily)10:01
openstackLaunchpad bug 1444901 in Designate "Service threadpool size needs to be configurable" [Undecided,Triaged]10:01
*** boris-42 has quit IRC10:05
*** boris-42 has joined #openstack-dns10:08
*** fawadkhaliq has quit IRC10:32
*** fawadkhaliq has joined #openstack-dns10:32
Kiallstanzgy: that depends I guess, is there anything designate can do to make pDNS "happier"? Maybe it's a documentation thing?10:33
openstackgerritKiall Mac Innes proposed openstack/designate: Test change  https://review.openstack.org/17580210:35
KiallPgSQL gate was added last night.. Fingers crossed it works!10:35
*** fawadk has joined #openstack-dns10:36
*** fawadkhaliq has quit IRC10:38
stanzgyKiall: I did some mysql innodb tuning under ahu_'s guidance. But it still costs too much IO. I have noticed that for pDNS table 'record', the index length is about 3x~5x that of the data length. It's a heavy task to maintain these indexes, may be we could remove some useless indexex to optimize db performance.10:39
stanzgyKiall: ;D10:40
KiallAha, that makes sense as a bug for us since we do pdns DB migrations10:40
*** fawadkhaliq has joined #openstack-dns10:41
stanzgyKiall: I shall file a bug with detailed statistics on it tomorrow.10:41
Kiallif you set slow query log to 0, run for a while, then point percona toolkit's pt-query-digest at the slow query log, you can get some great info on additional indexes needed, and it's other tooks like pt-duplicate-key-checker etc to identify useless ones10:41
Kiallstanzgy: please do ;)10:42
Kiall:)*10:42
Kiall(But the slow query log will likely do a MB/s or three on it's own under load, so you may need to scale back the test zone size etc ;)10:42
*** fawadk has quit IRC10:43
stanzgyKiall: There are obvious a dupe index: KEY `nametype_index` (`name`,`type`) and KEY `rec_name_index` (`name`).10:43
stanzgyKiall: IMO, the second one could be removed at least.10:43
KiallYea, that looks like an obvious one alright :) Probably more too!10:44
*** fawadk has joined #openstack-dns10:44
stanzgyKiall: Maybe. I need to catch the bus now, back to work tomorrow :D10:46
*** fawadkhaliq has quit IRC10:48
*** fawadkhaliq has joined #openstack-dns10:49
*** fawadkhaliq has quit IRC10:50
*** fawadkhaliq has joined #openstack-dns10:50
*** fawadk has quit IRC10:51
*** stanzgy has quit IRC10:51
*** fawadk has joined #openstack-dns10:53
*** fawadkhaliq has quit IRC10:55
*** fawadkhaliq has joined #openstack-dns10:55
*** fawadk has quit IRC10:58
*** nihilifer has quit IRC11:03
*** nihilifer has joined #openstack-dns11:03
*** fawadkhaliq has quit IRC11:09
*** kei_yama has quit IRC11:09
*** fawadkhaliq has joined #openstack-dns11:13
*** fawadkhaliq has quit IRC11:15
*** fawadkhaliq has joined #openstack-dns11:16
*** km has quit IRC11:28
*** fawadkhaliq has quit IRC11:30
*** fawadkhaliq has joined #openstack-dns11:32
*** fawadkhaliq has quit IRC11:35
openstackgerritKiall Mac Innes proposed openstack/designate: Ensure addition of TSIG do not trigger a TooBig exception  https://review.openstack.org/17582211:35
*** mwagner_lap has quit IRC11:38
*** jmcbride has quit IRC12:01
openstackgerritKiall Mac Innes proposed openstack/designate: PM: Ensure only one periodic task runs per host  https://review.openstack.org/17583012:09
*** richm has joined #openstack-dns12:10
openstackgerritKiall Mac Innes proposed openstack/designate: Service ThreadPool size should be configurable  https://review.openstack.org/17583512:17
*** mwagner_lap has joined #openstack-dns12:19
openstackgerritKiall Mac Innes proposed openstack/designate: Ensure addition of TSIG do not trigger a TooBig exception  https://review.openstack.org/17582212:21
openstackgerritKiall Mac Innes proposed openstack/designate: Service ThreadPool size should be configurable  https://review.openstack.org/17583512:22
openstackgerritKiall Mac Innes proposed openstack/designate: PM: Ensure only one periodic task runs per host  https://review.openstack.org/17583012:54
openstackgerritKiall Mac Innes proposed openstack/designate: Service ThreadPool size should be configurable  https://review.openstack.org/17583513:03
*** richm has quit IRC13:12
*** richm has joined #openstack-dns13:22
*** busterswt has joined #openstack-dns13:23
*** lkoranda has quit IRC13:24
*** jmcbride has joined #openstack-dns13:24
*** vinod1 has joined #openstack-dns13:31
*** andrewbogott_afk is now known as andrewbogott13:31
*** lkoranda has joined #openstack-dns13:33
*** fawadkhaliq has joined #openstack-dns13:44
*** johnbelamaric has joined #openstack-dns13:45
*** EricGonczer_ has joined #openstack-dns14:06
*** paul_glass has joined #openstack-dns14:09
*** EricGonc_ has joined #openstack-dns14:15
*** andrewbogott is now known as andrewbogott_afk14:16
*** andrewbogott_afk is now known as andrewbogott14:16
*** EricGonczer_ has quit IRC14:17
*** vinod1 has quit IRC14:18
*** vinod1 has joined #openstack-dns14:22
Kiallmugsie / vinod1 / timsim - I dropped a bunch of reviews for a couple of the kilo-rc-potential issues today - https://bugs.launchpad.net/designate/+bugs?field.tag=kilo-rc-potential - should aim to land them (after proper review!) soon14:54
*** encld06 has joined #openstack-dns14:56
*** penick has joined #openstack-dns14:57
paul_glassKiall: so I thought the TEMPEST_CONFIG env var got set in the gate, but it doesn't look like it.14:59
KiallEhh, I believe it expects you to use the default of /etc/tempest/tempest.conf (or was it /etc/tempest.conf?)14:59
paul_glassyeah, tempest.config used: https://github.com/openstack/tempest/blob/master/tempest/config.py#L117415:01
paul_glassbut without importing tempest, I don't have that module dir, here: https://review.openstack.org/#/c/175562/2/functionaltests/common/config.py15:03
openstackgerritDoug Hellmann proposed openstack/python-designateclient: Update README to work with release tools  https://review.openstack.org/17592515:07
*** fawadkhaliq has quit IRC15:07
Kiallpaul_glass: Oh.. Humm, I wonder how you're meant to handle that.. copy+pasting into each service repo seems like it will be prone to getting  out of date15:09
*** penick has quit IRC15:18
paul_glassKiall: for the time being, I figured: keep tempest installed in the gate, and export TEMPEST_CONFIG=/opt/stack/new/tempest/etc/tempest.conf in whichever the relevant gate hook script is.15:20
Kialltempest should still be happening in the gate, did TEMPEST_CONFIG exist before?15:20
KiallOHH15:21
KiallI see it15:21
KiallRe-read the links you have15:21
Kiallgave*15:21
KiallSo, humm.15:21
KiallHave any other projects made the change yet? Might serve as an example for us15:22
paul_glassnot that I'm aware of. I'd need to ask around.15:24
*** fawadkhaliq has joined #openstack-dns15:28
*** fawadkhaliq has quit IRC15:33
*** fawadkhaliq has joined #openstack-dns15:34
*** vinod1 has quit IRC15:43
openstackgerritPaul Glass proposed openstack/designate: Functional tests: minus tempest, plus noauth support  https://review.openstack.org/17556215:47
*** fawadkhaliq has quit IRC15:51
*** fawadkhaliq has joined #openstack-dns15:51
*** jhazen has joined #openstack-dns15:57
openstackgerritMerged openstack/designate: Update PM pooling defaults  https://review.openstack.org/17334915:59
*** untriaged-bot has joined #openstack-dns16:01
untriaged-botUntriaged bugs so far:16:01
untriaged-bothttps://bugs.launchpad.net/designate/+bug/144512716:01
openstackLaunchpad bug 1445127 in Designate "Multiple Pool Manager - every PM is running periodic sync" [Undecided,Confirmed]16:01
untriaged-bothttps://bugs.launchpad.net/designate/+bug/144628116:01
openstackLaunchpad bug 1446281 in Designate "designate dashboard detail screen does not support horizon juno" [Undecided,In progress] - Assigned to Eric Peterson (ericpeterson-l)16:01
untriaged-bothttps://bugs.launchpad.net/designate/+bug/144638716:01
openstackLaunchpad bug 1446387 in Designate "Support Host header in REST queries" [Undecided,New] - Assigned to Cedric Brandily (cbrandily)16:01
untriaged-bothttps://bugs.launchpad.net/designate/+bug/144624216:01
openstackLaunchpad bug 1446242 in Designate "Allow to use network label in nova_fixed format" [Undecided,In progress] - Assigned to Cedric Brandily (cbrandily)16:01
untriaged-bothttps://bugs.launchpad.net/designate/+bug/144490116:01
openstackLaunchpad bug 1444901 in Designate "Service threadpool size needs to be configurable" [Undecided,In progress] - Assigned to Kiall Mac Innes (kiall)16:01
*** untriaged-bot has quit IRC16:01
openstackgerritKiall Mac Innes proposed openstack/designate: PM: Ensure only one periodic task runs per host  https://review.openstack.org/17583016:12
openstackgerritKiall Mac Innes proposed openstack/designate: PM: Ensure only one periodic task runs per host  https://review.openstack.org/17583016:15
timsimKiall: you fixed that like, five seconds before I was going to -1.16:16
mugsietimsim: i just shouted across the desk at him :)16:17
*** vinod1 has joined #openstack-dns16:17
timsimNice.16:17
*** fawadkhaliq has quit IRC16:18
*** eandersson has joined #openstack-dns16:18
*** fawadkhaliq has joined #openstack-dns16:20
*** fawadkhaliq has quit IRC16:23
*** jordanP has quit IRC16:29
timsimKiall: So with 10 and 20 worker threads, I'm still seeing multiple periodic tasks running. 10: http://paste.openstack.org/show/205011/ 2 perodic recoveries start at about the same time 20: http://paste.openstack.org/show/205010/ 4 recoveries start at about the same time16:44
*** vinod1 has quit IRC16:53
openstackgerritMerged openstack/python-designateclient: Update README to work with release tools  https://review.openstack.org/17592517:05
*** fawadkhaliq has joined #openstack-dns17:21
*** fawadkhaliq has quit IRC17:26
*** vinod1 has joined #openstack-dns17:36
*** penick has joined #openstack-dns17:36
*** fawadkhaliq has joined #openstack-dns17:48
*** echoingumesh has joined #openstack-dns17:59
openstackgerritMerged openstack/designate: Ensure addition of TSIG do not trigger a TooBig exception  https://review.openstack.org/17582218:08
*** fawadkhaliq has quit IRC18:18
*** fawadkhaliq has joined #openstack-dns18:18
*** johnbelamaric has quit IRC18:21
openstackgerritMerged openstack/designate: Service ThreadPool size should be configurable  https://review.openstack.org/17583518:32
*** openstackgerrit has quit IRC18:37
*** openstackgerrit has joined #openstack-dns18:37
*** ChanServ sets mode: +v openstackgerrit18:37
*** johnbelamaric has joined #openstack-dns18:39
Kialltimsim: 2 or 4 is better than 10 or 20 :p in all reality, if that's happening, it has to be a race in oslo.concurrency :(18:49
timsimFair enough. The longer term solution should probably take care of that.18:50
*** jmcbride has quit IRC18:50
KiallI'm on my cell, so can't look at the logs.. If you have debug on, do you see multiple tasks acquiring the lock?18:52
timsimYeah I have debug on. Basically you see "16:38:40.840 Acquired file lock "/opt/designate/periodic_recovery" after waiting 0.000s" "16:38:40.926 Acquired file lock "/opt/designate/periodic_recovery" really close to each other about four times, every periodic interval.18:55
*** jmcbride has joined #openstack-dns18:55
KiallIf its really an Oslo.concurrency issue.. We can likely get that fixed before we release...18:56
*** jmcbride has quit IRC18:56
KiallThat certainly sounds like it shouldn't be happening based on your paste there :(18:56
timsimYeah it only happened when you started a lot of them.18:56
timsimThe vm has 2 cpus, and I'm starting like 10 pool managers, so something is bound to go wrong.18:57
*** jmcbride has joined #openstack-dns19:00
KiallLooking at oslo.concurrency.. It's delegating out to https://docs.python.org/2/library/fcntl.html#fcntl.lockf19:01
KiallI'd kinda expect that to work19:01
timsimI could be messing something up, try it out at your leisure, if you don't see it, I'm probably doing something stupid.19:01
*** johnbelamaric has quit IRC19:02
KiallYour logs show it happening ;) Myself and graham were actually debating if a temp fix was even worth it, since all we're really doing is serializing them periodics rather than deduplicating them.. I expect 99% of deploys will have 1 to say, 4 workers, if it's working there, I'd say it's better than nothing19:04
*** paul_glass has quit IRC19:05
Kialltimsim: actually, in http://paste.openstack.org/show/205010/, I don't see the lock being handed out twice concurrently?19:06
KiallLooking at the other set now..19:06
timsimRight, so they don't actually happen at the same time. But it doesn't run just one time per host19:07
KiallOHH.. Okay, now I understand.. Yea, that's not something we can avoid wither bigger changes :(19:08
KiallThey'll pile up, and run serially rather than all running at exactly the same time19:08
timsimYeah, I figured the bigger change later that's mentioned will probably fix that.19:08
*** paul_glass has joined #openstack-dns19:08
Kiall(i.e spead the load they cause over time, rather than having it ALL at once)19:08
Kiallspread*19:08
*** johnbelamaric has joined #openstack-dns19:08
timsimBut I just wanted to bring it up, in case it wasn't intended to work that way19:09
Kialloslo.concurrency doesn't give us a way to say, try obtain the lock, and if you can't, let ius know so we can skip this iteration etc19:09
KiallWhat I really wated to do, was have each worker attempt to grab the lock, and hold it forever, until that thread dies.. but it would have required too many changes for an rc2 fix19:09
Kialluntil that proceess dies*19:10
timsimAh that'd be better. What it's doing now is definitely better. You can still work around it fine in a proper deployment. I'm fine with fixing it completely later.19:12
Kialltimsim: psst - +A needed on https://review.openstack.org/#/c/175326/19:13
Kiall(Similar reviews elsewhere got an explanation that yes, they need to land ;))19:13
Kialltimsim: yea, I think once we pull in tooz (Which we'll need anyway in L (making sure secondary zones respect the SOA refresh interval without piles of different workers checking each once at once), we get this for free:  http://docs.openstack.org/developer/tooz/tutorial/leader_election.html19:21
Kialland http://docs.openstack.org/developer/tooz/tutorial/lock.html ;)19:22
KiallAnyway.. Call time.19:22
timsimWell that's pretty cool.19:22
timsimMore like bedtime19:22
Kialllol.. My last meeting of the day is scheduled for 11:30PM. That one, I rejected. lol19:22
timsimoof. We complain if meetings go until 5 :x19:23
Kiall(If Dominic happens to be here, sorry :P)19:23
*** vinod1 has quit IRC19:35
*** andrewbogott is now known as andrewbogott_afk19:37
*** vinod1 has joined #openstack-dns19:37
*** EricGonc_ has quit IRC19:39
*** ericpeterson has joined #openstack-dns19:48
*** fawadkhaliq has quit IRC19:50
*** echoingumesh has quit IRC19:56
*** echoingumesh has joined #openstack-dns19:57
*** echoingumesh has quit IRC20:01
*** paul_glass1 has joined #openstack-dns20:05
*** EricGonczer_ has joined #openstack-dns20:07
*** echoingumesh has joined #openstack-dns20:07
*** jmcbride has quit IRC20:08
*** paul_glass has quit IRC20:08
*** jmcbride has joined #openstack-dns20:12
*** andrewbogott_afk is now known as andrewbogott20:34
*** echoingumesh has quit IRC20:41
*** openstackgerrit has quit IRC20:52
*** openstackgerrit has joined #openstack-dns20:52
*** ChanServ sets mode: +v openstackgerrit20:52
openstackgerritPaul Glass proposed openstack/designate: Zone ownership tests  https://review.openstack.org/17608021:11
*** vinod1 has quit IRC21:12
*** johnbelamaric has quit IRC21:17
*** iraw_ has joined #openstack-dns21:19
*** iraw_ has quit IRC21:19
*** EricGonczer_ has quit IRC21:25
*** iraw_ has joined #openstack-dns21:26
*** johnbelamaric has joined #openstack-dns21:38
*** johnbelamaric has quit IRC21:42
openstackgerritPaul Glass proposed openstack/designate: Zone ownership tests  https://review.openstack.org/17608021:45
*** mwagner_lap has quit IRC21:47
*** busterswt has quit IRC21:53
*** iraw_ has quit IRC22:02
*** ericpete_ has joined #openstack-dns22:05
*** ericpeterson has quit IRC22:08
*** ericpete_ has quit IRC22:09
*** jmcbride has quit IRC22:11
*** andrewbogott is now known as andrewbogott_afk22:20
openstackgerritPaul Glass proposed openstack/designate: Functional tests to check recordset ownership  https://review.openstack.org/17610522:22
*** paul_glass1 has quit IRC22:23
*** mwagner_lap has joined #openstack-dns22:32
*** km has joined #openstack-dns22:44
*** EricGonczer_ has joined #openstack-dns22:50
*** jmcbride has joined #openstack-dns22:57
*** penick has quit IRC22:58
*** penick has joined #openstack-dns23:02
*** penick has quit IRC23:03
*** jhazen has quit IRC23:09
*** EricGonczer_ has quit IRC23:20
*** kei_yama has joined #openstack-dns23:32
*** penick has joined #openstack-dns23:36
*** penick has quit IRC23:56
« Monday, 2015-04-20 Index Wednesday, 2015-04-22 »

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!