Monday, 2015-11-09

« Sunday, 2015-11-08 Index Tuesday, 2015-11-10 »
*** ccneill_ has quit IRC00:04
*** Guest29885 has quit IRC00:20
*** km has joined #openstack-dns00:21
*** km is now known as Guest3918700:21
*** jmcbride has joined #openstack-dns00:27
*** jasonsb has quit IRC00:47
*** jmcbride has quit IRC01:28
*** rudrajit has joined #openstack-dns01:30
*** rudrajit has quit IRC01:35
*** stanzgy has joined #openstack-dns01:40
*** ducttape_ has joined #openstack-dns01:55
*** ccneill_ has joined #openstack-dns01:59
*** ducttape_ has quit IRC02:07
*** ducttape_ has joined #openstack-dns02:17
*** ccneill_ has quit IRC02:18
*** ducttape_ has quit IRC02:21
*** ducttape_ has joined #openstack-dns02:24
*** richm has joined #openstack-dns02:31
*** richm has quit IRC02:45
*** ducttape_ has quit IRC02:51
*** Jet_ has joined #openstack-dns02:52
*** ducttape_ has joined #openstack-dns02:57
*** jmcbride has joined #openstack-dns02:59
*** jmcbride has quit IRC02:59
*** ducttape_ has quit IRC03:01
*** ducttape_ has joined #openstack-dns03:17
*** jmcbride has joined #openstack-dns03:25
*** ducttape_ has quit IRC03:27
*** ducttape_ has joined #openstack-dns03:28
*** richm has joined #openstack-dns03:44
*** ducttape_ has quit IRC03:55
*** kei_yama has quit IRC04:02
*** kei_yama has joined #openstack-dns04:03
*** ducttape_ has joined #openstack-dns04:05
*** ducttape_ has quit IRC04:05
*** richm has quit IRC04:09
*** ducttape_ has joined #openstack-dns04:12
*** Jet_ has quit IRC04:17
*** ducttape_ has quit IRC04:19
*** Jezogwza_ has quit IRC04:34
*** kei_yama has quit IRC04:59
*** kei_yama has joined #openstack-dns04:59
*** rudrajit has joined #openstack-dns05:28
*** rudrajit has quit IRC05:29
*** rudrajit has joined #openstack-dns05:29
*** murali has joined #openstack-dns05:44
*** Jet_ has joined #openstack-dns05:44
*** murali has quit IRC05:58
*** jmcbride has quit IRC06:04
*** jasonsb has joined #openstack-dns06:33
*** elarson has joined #openstack-dns06:51
*** kei_yama has quit IRC06:59
*** kei_yama has joined #openstack-dns06:59
*** Jet_ has quit IRC07:16
*** kei_yama has quit IRC08:10
*** kei_yama has joined #openstack-dns08:11
*** Guest39187 has quit IRC08:11
*** km has joined #openstack-dns08:11
*** km is now known as Guest2817908:11
*** zigo has quit IRC08:16
*** zigo has joined #openstack-dns08:19
*** nyechiel has joined #openstack-dns08:42
*** jordanP has joined #openstack-dns08:53
*** kei_yama_ has joined #openstack-dns09:02
*** kei_yama has quit IRC09:02
*** rudrajit_ has joined #openstack-dns09:12
*** rudrajit has quit IRC09:15
*** openstack has joined #openstack-dns09:20
*** ChanServ sets mode: +v openstack09:20
*** rudrajit_ has quit IRC09:35
*** rudrajit has joined #openstack-dns09:35
*** bradjones has joined #openstack-dns10:37
*** bradjones has quit IRC10:37
*** bradjones has joined #openstack-dns10:37
*** nyechiel has joined #openstack-dns10:45
*** stanzgy has quit IRC10:48
*** kei_yama_ has quit IRC10:54
*** kei_yama has joined #openstack-dns10:55
*** nyechiel has quit IRC11:14
*** weihan has joined #openstack-dns11:43
*** nyechiel has joined #openstack-dns12:11
*** kei_yama has quit IRC12:13
*** ig0r__ has joined #openstack-dns12:17
*** Guest28179 has quit IRC12:25
*** weihan has quit IRC12:57
*** jordanP has quit IRC12:59
*** slevchenko has quit IRC13:15
*** Diplomat has joined #openstack-dns13:20
DiplomatHello, anyone here? I have a quick question.. is there a way to control designate from the controller ?13:21
*** boris-42 has quit IRC13:28
*** jordanP has joined #openstack-dns13:37
*** rsyed_away is now known as rsyed13:42
*** mwagner has quit IRC13:44
en_austinhello, anyone here?13:47
*** boris-42 has joined #openstack-dns13:59
mugsieen_austin: yup - here14:07
mugsieyour error looks like you have an old version of pip14:07
mugsieDiplomat: from the designate controller?14:08
DiplomatNaw, from OS controller14:08
Diplomatbut I already found an answer14:08
*** mwagner has joined #openstack-dns14:16
DiplomatWould it be possible to update the serial ?14:16
DiplomatAlso is there an easy to way to transfer data from bind? Like.. existing zones14:17
mugsieyeap, there is an import function14:17
DiplomatSweet, what about updating the serial ?14:18
mugsieupdating the serial - just make a change to any of the records .. there was a v1 API for "touch"ing a domain14:18
mugsiebut I don't think that got into the v2 API14:18
DiplomatWell, I'd like to set a specific value there14:18
mugsieah14:18
mugsieno.14:19
mugsiewe assume control of the serial internally14:19
DiplomatHmn, that's not the best.. I think I could create a temporary dns record with an updated serial and then import it14:20
mugsieDiplomat: why do you need to set the serial?14:21
DiplomatClients need that.. because they have some kind of reasons14:21
mugsieah14:21
mugsieclients14:21
* mugsie tries to avoid them :P14:22
Diplomatlol14:22
DiplomatWell, it's not that bad as long as your service is good14:22
*** richm has joined #openstack-dns14:22
DiplomatI think I could update that API a bit for myself14:23
mugsieDiplomat: if you want to send it upstream we could talk about if it makes sense for the project14:32
DiplomatWell, I don't know much sense it would make for most people, but I'm sure there are people who would be interested in it14:33
*** nyechiel has quit IRC14:42
*** nyechiel has joined #openstack-dns14:42
en_austinmugsie: well, i've already solved this :) now i have an another questiion - how can I upgrade from Kilo to Liberty?14:44
en_austinwhat steps should I do?14:44
mugsieshould be shutdown services,  run migrations, restart services with new code, and start the new zone manager service14:45
en_austinwhat is "new zone manager service"?14:49
en_austine.g from Liberty there is a 5 services - API, Central, mDNS, PoolManager and that new service?14:49
*** ducttape_ has joined #openstack-dns14:57
*** ducttape_ has quit IRC15:00
*** ducttape_ has joined #openstack-dns15:00
mugsieen_austin: yup, there is a new "designate-zone-manager" service15:02
*** pglass has joined #openstack-dns15:08
*** _ducttape_ has joined #openstack-dns15:09
*** ducttape_ has quit IRC15:11
*** openstackgerrit has quit IRC15:17
*** openstackgerrit has joined #openstack-dns15:17
*** ChanServ sets mode: +v openstackgerrit15:17
en_austinmugsie: can you give a link to this service FAQ or write some lines here?15:18
en_austinto let me know what that new service is running for?15:18
mugsiesure - it does periodic tasks that relate to zones. (e.g. it checks secondary zones SOA records to make sure we didnt miss a NOTIFY, it clears out zones that are in a "deleted" state, it sends periodic "Exists" events for celiometer)15:19
en_austinbtw, i'm experiencing now some problems with actual DNS provisioning (designate stable/kilo, bind9 backend) - delay from actual API call and actual record provisioning sometimes (not always, of course) is 10+ minutes (while API reports this record as ACTIVE)15:19
en_austinmaybe, upgrading to Liberty will solve this...15:20
mugsieit sould15:20
mugsiethere was some wierdness around updates in Kilo15:20
en_austinit's solved in L? :-)15:20
mugsiewe certainly hope so15:21
en_austinWell, I'll upgrade a production Designate and will report here :D15:21
mugsiedo you have a non default threshold set in the pool manager config file?15:21
en_austinnope. what threshold you are talking abt?15:22
en_austinactually, I have only pool_id in my PoolManager service config section.15:23
en_austin(and cache_driver=noop)15:23
mugsieah15:23
mugsiethat could be it15:23
en_austinhm15:23
en_austinbut why?15:23
mugsiewe need a place to store what servers have the zone active15:24
mugsieand the code that tries to do that without a cache is interesting15:25
mugsieyou can just use another DB on the same servers15:25
mugsieor if you have memcached for anything, you can use that15:25
en_austinwell, I can run memcached for this15:26
en_austinbut how? :)15:26
en_austinor use mysql db (btw, i have one created, but it's unused)15:27
timsimInstall memecached somewhere. Change this line https://github.com/openstack/designate/blob/master/etc/designate/designate.conf.sample#L289 Add this section: https://github.com/openstack/designate/blob/master/etc/designate/designate.conf.sample#L306-L311 replace memcached servers to "127.0.0.1:11211" or something en_austin15:28
*** csoukup has joined #openstack-dns15:30
en_austinso, you think, that cache can help to resolve an issue with non-provisioned records I've described above?15:32
*** ducttape_ has joined #openstack-dns15:32
mugsieen_austin: it might15:34
timsimIt should if you've got more than one nameserver (and probably will even if you don't).15:34
mugsieI know we saw weird stuff using the fake cache driver15:34
timsimThe Pool Manager or MiniDNS logs should have more information as to why you're failing, if that cache driver isn't it.15:34
*** _ducttape_ has quit IRC15:35
*** jmcbride has joined #openstack-dns15:39
en_austinyep, i have 2 NS running15:39
en_austinthere is no "ERROR: xxx" logs in mdns/pm, I've checked :(15:39
*** Diplomat has quit IRC15:40
*** Diplomat has joined #openstack-dns15:44
*** jmcbride has quit IRC15:46
*** Jezogwza_ has joined #openstack-dns15:47
*** jmcbride has joined #openstack-dns15:47
timsimYeah it might not be logged ERROR :/15:50
*** rudrajit has quit IRC15:53
openstackgerritMerged openstack/designate: Use oslo_config PortOpt type for port options  https://review.openstack.org/24125715:53
openstackgerritMerged openstack/python-designateclient: Add functional tests for tlds  https://review.openstack.org/22850915:54
openstackgerritMerged openstack/python-designateclient: Add functional tests for blacklists  https://review.openstack.org/22861615:54
en_austin:(15:55
en_austinwell, will try it anyway.15:55
en_austinbtw, are there a correct way to RENAME a designate-handled zone?15:55
en_austine.g - i'm running xxx.zone in Designate, and I want to rename it to yyy.zone (with all its records, of course)15:56
en_austinare there another way except of export - rename $ORIGIN - import again ?15:56
timsimNope. Name is immutable.15:56
en_austinso, import/export is the only way?15:58
timsimYep, I believe so15:58
*** mlavalle has joined #openstack-dns16:00
*** jordanP has quit IRC16:01
*** rsyed is now known as rsyed_away16:17
*** ccneill_ has joined #openstack-dns16:18
*** ccneill_ is now known as ccneill16:19
*** rudrajit has joined #openstack-dns16:19
*** rudrajit has quit IRC16:23
en_austin2015-11-09 19:31:25.944 2052 ERROR designate DbMigrationError: Storage requires its own database. Please check your config file.16:31
en_austinit's after designate-admin database sync16:31
openstackgerritMerged openstack/python-designateclient: Fix support for --os-endpoint  https://review.openstack.org/24194116:32
openstackgerritMerged openstack/python-designateclient: Fix support for --os-endpoint  https://review.openstack.org/24194116:32
en_austintimsimL it's some new entries in config, I suppose?16:32
en_austintimsim: ^ *16:32
timsimWhere do you see that error en_austin? mdns?16:34
timsimThat should be the `connection` field in the storage:sqlalchemy section.16:34
en_austinaaawh, it's my fault, sorry :( i've misplaced an my old designate config file (and tried to run database-sync on clear github installation) :)16:35
en_austinPM throws:16:36
en_austin2015-11-09 19:35:58.119 2235 WARNING designate.coordination [-] No coordination backend configured, assuming we are the leader. Please configure a coordination backend16:36
en_austinWhat he is expecting here?16:36
en_austin(same for zone-manager)16:36
timsimIt's using the openstack tooz library to provide some coordination services. Namely, pool-manager leader election (to ensure that only one instance runs periodic sync jobs) and zone sharding (zone-manager). So you need one of the drivers that tooz offers. I believe it's supposed to work with memecached, but the recommended path is to run Apache Zookeeper. However, if you've just got a small deployment, memecached ought to16:41
timsim work.16:41
timsimThe config section you need is here: https://github.com/openstack/designate/blob/master/etc/designate/designate.conf.sample#L41916:41
timsimSo if you were going to use memecached, you would put backend_url=memecached://127.0.0.1:11211 or whatever.16:42
en_austinI'm going to use memcached as cache server for pool manager. What do you mean under "small deployment" ? :)16:47
timsimYou can use it for both the pool manager cache, and this coordination backend.16:48
timsimIf you only have one pool manager instance, one zone-manager instance, the coordination driver you choose won't have much effect.16:48
timsimBut if you've got 20, you might want to look into zookeepr.16:49
en_austinI'm running now with 4 APIs, 4 Central's and 4 mDNS's.16:49
timsimpool-managers?16:50
en_austinnow 1 only16:50
mugsiethen without a co-ordination backend will be fine16:50
timsimAre those all separate VMs running one process each?16:50
en_austinbut if I'll upgrade Designate to Liberty and run 4 ZF and 4 PMs with memcached - will i have a degradated performance?16:50
en_austinnope, that's all a single OpenVZ container16:50
mugsienot sure how it will perform to be honest16:51
timsimAnd you're starting 4 processes of each one separately, or using the inbuilt worker processes?16:51
en_austininbuilt workers16:51
timsimWhat kind of performance are you building for? What's ideal for you?16:52
*** Jaan_ has joined #openstack-dns16:55
en_austinmain goal of our designate deployment is to provide a records to clients asap (we're using in another openstack infrastructure)16:55
*** Diplomat has quit IRC16:55
*** Jaan_ is now known as Diplomat16:56
en_austinand we're expecting a 30sec...1min delay between API call (post v2/zones/.../recordsets) and actually record resolving16:56
en_austinthat's why I'm taking care about even several 10min delays between api call and records delivery - each of 'em is very important :(16:58
timsimWell, I would try just using memecached for the coordination, but if you're set on having four processes, you might need to install zookeeper and use the "kazoo" backend, that's the only one I've seen work consistently.16:58
timsim10 minute delays means there is a big problem. That's not regular behavior at all. We've got designate pushing across the pacific ocean much faster than that at pretty high rates.16:59
timsims/much faster/under 10 seconds17:00
en_austinsome debugging shows that all of backends (bind9's) responds with SOA instead of xxxxxxx A 127.0.0.1 (that means that record wasn't delivered yet - that's why I've asked for record delivery issues in this chat above) :)17:04
en_austinI hope that upgrading to Liberty and setting up a cache backend will help.17:05
timsimThere has to be something in that log.17:06
timsimIf you had those handy to paste, we might be able to help more.17:06
en_austinI'll try to find it out - too much logs :((17:06
timsimBetter than not enough ;)17:06
*** nkinder has joined #openstack-dns17:13
mugsietimsim: Kiall federico3 elarson ekarlso updated the designate dashboard - graham.hayes.ie/designate/dashboard is a redirect17:19
timsim:fire:17:19
elarson:badoosh: :docker: :+1:17:20
en_austintimsim: well, right 15min ago i've fired again this issue with non-delivered domain, so I can find a logs :)17:21
en_austinwhich logs I should paste? All (api, central, PM, mDNS) ?17:21
*** nyechiel has quit IRC17:23
timsimPM / mdns17:24
en_austinwell, I've found a big time spread in PM logs...17:27
en_austinwill now paste it.17:27
en_austinhttp://paste.openstack.org/show/vkoT73o4m3C4nU7f5xsJ/17:28
en_austinawh, mdns logs are not for this record (while poolmanager is exactly for that rec - i've grep'd by zoneid)17:30
*** Jezogwza_ is now known as Jezogwza17:32
openstackgerritGraham Hayes proposed openstack/designate: Added HTTP_X_FORWARDED_PROTO support to API  https://review.openstack.org/24172717:38
en_austinhttp://paste.openstack.org/show/9S6Pi7RdTvnAYVBtAifG/17:39
en_austinhere is correct paste (and I've found here some warning timeouts...)17:39
en_austintimsim: is it a helpful logs? :)17:40
timsim`mdns.log.1:Got Timeout while trying to send 'SOA' for 'aps.sw.ru.' to '10.28.0.18:53'. Timeout='30' seconds. Retry='2'`17:40
timsimThat means the mdns server is having trouble talking to '10.28.0.18:53'17:40
en_austinWell, but API received this record in 19:54:49.xxx, and mDNS tried to push it at 20:06:35...17:41
en_austinwhy so?17:41
timsimIt looks like it was successful at 19:54:52.9 in mdns17:42
en_austinbut it wasn't present at bind zonefile (i've grep'd it at 20+ hrs when i was notified about dns failure)17:43
en_austinthat's the problem...17:43
en_austinbtw, i'm running two NSs (and Timeout connection problem is only for one of them)17:44
timsimupdate_status: Calling pool manager for aps.sw.ru. : UPDATE : SUCCESS : 1447088089 on nameserver means that mdns saw it land. meaning it was propagated.17:44
timsimIt looks like the lack of a cache driver may have actually slowed that down. Because that message would have been received by the pool manager, but then it had to "get_serial_number" on those servers, which we see, and that's where it ran into connectivity issues.17:45
en_austinBut actually, how the cache is engaged into this mDNS<->backend conversation?17:46
en_austinWill now enable it and check it out - maybe, it will help...17:46
en_austinand, if mDNS successfully pushed it on backends - why it was missing in slave.xxxxxxxx file at backend (bind9)17:48
en_austinThat's the question...17:48
timsimWhen mdns does that report to say "Calling pool manager for aps.sw.ru. : UPDATE : SUCCESS : 1447088089" pool manager stores that, so that when the second one reports back, it knows that the state was active on the other server, so it now knows it's active on both and the domain can be active.17:48
timsimWhen it goes "ACTIVE" in the API, it means it's resolving. It might be doing something different than you're expecting on the bind server, but ACTIVE status means designate did a dns query and successfully resolved.17:49
*** jmcbride has quit IRC17:52
*** jmcbride has joined #openstack-dns17:52
*** Diplomat has quit IRC17:57
*** jmcbride1 has joined #openstack-dns18:01
en_austinI understand - but it switched to ACTIVE without actually being pushed to backends. In approx 20:25 +/- it appeared at bind's zonefile (and, of course, started to resolve)18:03
*** jmcbride has quit IRC18:03
en_austinOr - there was some black magic that I don't understand :(18:03
*** jmcbride has joined #openstack-dns18:04
*** jmcbride2 has joined #openstack-dns18:04
*** jmcbride1 has quit IRC18:05
en_austinWhen this problem appeared, I've grep'd by zonefile - and there wasn't this record at that moment.18:05
en_austinLater (~5-7min) it appeared into zonefile - and all is OK for now.18:06
mugsieen_austin: I may be wrong ( timsim can correct me) but I don;t think bind rights to the file straight away on a zone transfer in18:08
mugsieit keeps it in memory AFAIK, but it should be resolving18:08
*** jmcbride has quit IRC18:09
mugsieACTIVE says that designate has queried the 2 Bind servers and gotten the correct serial back from both18:09
en_austinookay, but... i've dig'd it two times with 10min delay - first query - "SOA .......", second - correct A record :(18:10
en_austinsome strange problem, I don't know what to do with it :(18:10
en_austinbtw! i have an dig output stored (when it's not resolved)18:11
en_austinaaand... here is an equal SOA with PoolManaager and mDNS's logs!18:12
en_austinhttp://paste.openstack.org/show/RyAas5EuDuL1oqiJoDGV/ -- SOA serial is 1447088089 - which is the same as in logs I've showed you above - maybe, here is a problem root?18:14
*** jmcbride2 has quit IRC18:21
*** jmcbride has joined #openstack-dns18:21
openstackgerritMerged openstack/designate: remove default=None for config options  https://review.openstack.org/24283518:25
en_austinmugsie: any ideas?18:32
mugsiehumm... it should be serving the right data if there is the same serial... when there is a change does the zone serial match the SOA serial when it is marked as active, but not returning the right record data?18:33
*** rsyed_away is now known as rsyed18:35
en_austin>  it should be serving the right data if there is the same serial.18:36
en_austinmaybe, there is some communication problems or smth else... that Designate pushes a SOA update to backend without data (for example, I just assume)18:36
en_austinand, actual data become pushed to backend only at next SOA update18:36
en_austinI don't know if it is even possible - but...18:37
mugsiewhat is the layout of your deployment?18:38
en_austin3 VMs (1 for Designate api+central+mdns+PM+RabbitMQ) and two for BINDs18:38
mugsie2 Bind servers + Designate ? (i.e. is there anything else in front of those Bind servers? ) and are they both set up as pool_nameservers in the designate.conf filke?18:39
openstackgerritMerged openstack/designate: Switch to oslo.reports  https://review.openstack.org/24168418:39
openstackgerritMerged openstack/designate: Fix argument order in assertEqual to (expect, obs)  https://review.openstack.org/23940018:40
en_austinthey are both set up as pool_nameservers, yep18:40
mugsiedesignate pushes a NOTIFY to the bind servers, which should trigger them to get the new data from the miniDNS servers.18:42
*** Jezogwza has quit IRC18:42
mugsiethe serial BIND serves should not change until the new data is on the bind server18:42
*** jasonsb has quit IRC18:45
timsimalso, what version of BIND is that?18:50
openstackgerritMerged openstack/designate: Add functional tests for TLDs  https://review.openstack.org/18352218:54
*** ccneill_ has joined #openstack-dns18:54
en_austinBIND 9.9.4-RedHat-9.9.4-18.el7_1.5 (Extended Support Version)18:55
*** ccneill__ has joined #openstack-dns18:55
*** ccneill has quit IRC18:56
en_austinbtw the problem is float (e.g some instability issue) - it fires twice a day, approximately18:56
en_austinor, once on weekend :(18:56
*** ccneill_ has quit IRC18:59
timsimI think the cache driver will help, but it looks like there might be network instability. Also I have no idea why the A record wouldn't be there if bind was returning the right serial o.O18:59
en_austinWell... I'll try to upgrade Designate from Kilo to Liberty and we'll see - will it helps...18:59
en_austinand will enable a cache driver.19:00
en_austinwill you be here for approx in half an hour? (I am going to update my production designate) :))19:00
*** ducttape_ has quit IRC19:04
timsimYep, I'll be around.19:06
openstackgerritTim Simmons proposed openstack/designate: Handle NO_DOMAIN status  https://review.openstack.org/23077019:08
timsimKiall mugsie rebased ^19:08
mugsietimsim: can you check about travel to Ireland for mid cycle - I need to make a proposal to get office space etc soon enough. cc/ jmcbride ;)19:11
timsimSure19:12
mugsiesweet19:12
mugsiehttp://gph.is/1guxtqM19:14
*** ducttape_ has joined #openstack-dns19:26
en_austinWell, seems I've migrated from Kilo to Liberty :)19:28
*** jasonsb has joined #openstack-dns19:28
en_austinSo, now... Cache, Coordination backend.. Yep? timsim19:28
en_austinand, how to change "127.0.0.1" to actual hostname in "links.self" response? :)19:28
timsimYep, that should work.19:29
timsimThere's a config option for the base uri to build them from, and also a config option to use host headers to build that.19:29
timsimHost headers: https://github.com/openstack/designate/blob/master/etc/designate/designate.conf.sample#L8819:30
*** jasonsb has quit IRC19:30
timsimapi base uri https://github.com/openstack/designate/blob/master/etc/designate/designate.conf.sample#L9119:30
*** james_li has joined #openstack-dns19:31
openstackgerritMerged openstack/designate: Removes zone files when a zone is deleted  https://review.openstack.org/23666219:32
en_austintimsim: i'm receiving a "got lower serial for xxxxx, expected: ..., got: .....' in mdns logs.. how i can return it to in-sync state?19:38
mugsieen_austin: it should do it itself19:39
en_austini dont' understand which serials it's talking about ('cause in logs: 'Expected:'1447097385'. Got:'1447097196'), in bind - SOA is 144709666319:39
en_austino.O19:39
jmcbrideoooh! I'd love to… I mean, we need to attend the 2016 mid-cycle in Ireland. Thanks for the invite mugsie, we'll keep you posted.19:40
en_austin2015-11-09 22:42:31.643 2064 WARNING designate.mdns.notify [req-013170d5-c331-4ba5-a9db-6969e421e05d noauth-user noauth-project - - -] Got Timeout while trying to send 'NOTIFY' for 'xxx.xx.xx' to '127.0.0.1:53'. Timeout='30' seconds. Retry='3'19:43
en_austinbut mDNS is running....19:43
timsimIsn't your dns server on 10.something. It looks like it's trying to notify 127.0.0.119:44
timsimPaste your whole config :)19:44
en_austinhttp://paste.openstack.org/show/bA5MkZ7bp1LXHtWiqPIM/19:46
*** jasonsb has joined #openstack-dns19:46
timsimMove the host/port things in the target sections into the csv of options.19:47
timsimThat's why it's notifying 127.0.0.119:47
timsimWe're working on fixing that, sorry :/19:47
mugsiejmcbride: cool. I am trying to sync on 2 mid cycles, so its a bit hectic19:50
en_austinupdate_status: Calling central's update_status for cf5c1b26-a945-449d-890e-0a1db5a2f339 : SUCCESS19:51
en_austinnice!19:51
en_austintimsim: are there any changelogs (for issues like this)? :)19:51
timsimI wouldn't think that would have changed from kilo->liberty19:52
*** jmcbride1 has joined #openstack-dns19:52
*** jmcbride has quit IRC19:53
en_austinbut it changed and there is no any comments etc even in sample config :(19:55
en_austinok, seems it's stable (create/delete works ok)19:56
timsimIt seems like it was options: host/port in this doc at least http://docs.openstack.org/developer/designate/install/ubuntu-kilo.html but there really should be upgrade docs, sorry mate.19:59
mugsieFYI - http://lists.openstack.org/pipermail/openstack-dev/2015-October/077442.html kicks in tomorrow. cc/ timsim elarson ekarlso federico3 Kiall pglass20:00
ekarlsomugsie: who's doing le grand rename20:01
mugsieekarlso: I am going to start it tomorrow, but I think it may end up being multiple patches20:02
mugsiewe shall see as the week progresses20:02
*** jmcbride1 has quit IRC20:02
*** jasonsb has quit IRC20:04
en_austintimsim: so, how to enable cache via memcached?20:05
en_austinhttps://github.com/openstack/designate/blob/master/etc/designate/designate.conf.sample#L306-L311   only this?20:06
en_austinor I should specify smth in [service:pool_manager] block (in cache_driver option, for example)20:06
timsimYeah, and https://github.com/openstack/designate/blob/master/etc/designate/designate.conf.sample#L28920:06
en_austinexpiration=3600 ?20:08
en_austin(as in default config)20:08
en_austinand what does this 1 hour means? zones will update in 1 hour? :))20:08
timsimProbably that cache entries expire in memecahed in an hour :P20:08
en_austinokaaaay, but what info this cache stores?20:09
timsimStatus of zones on nameservers.20:09
en_austinand - are there any log entries that shows a successful cache connection?20:09
en_austin> Status of zones20:10
en_austinThey will update once an hour? :O20:10
timsimNo.20:10
timsimThe status entry for a zone at a specific serial number will expire from memecached in an hour.20:10
en_austinI don't understand what's this cache for (which role it's playing in whole Designate infrastructure)20:10
timsimIt's just a semi-persistent store of the ACTUAL state of the zone (is it resolving) at a specific serial number on a specific nameserver.20:10
timsimSo that the pool manager can quickly calculate if a zone is live everywhere it needs to be to be considered active for a serial number.20:11
en_austinokay :)20:12
en_austinwill Designate report it's using a memcached (or another cache)? there is no any log entries for "cache" or ":11211" :(20:13
timsimI'm not sure.20:13
en_austinShould it fail if it's not able to connect to cache? :)20:14
*** ccneill has joined #openstack-dns20:15
* timsim sighs20:16
en_austin(answering: or smth went wrong, or it's not using a config params - i've specified an invalid parameters and it's just bypassed)20:16
timsimi don't think it does squawk if it can't connect.20:16
en_austinI just want to assume it's going well and using cache :)20:17
*** ccneill__ has quit IRC20:17
*** jmcbride has joined #openstack-dns20:17
timsimSeems legit20:18
timsimYou should be able to telnet into memecached and see if there are keys getting created.20:18
timsimOr if you have --debug on in pool manager there should be some logs saying "cache hit" "cache miss" etc20:19
*** james_li has quit IRC20:22
en_austin2015-11-09 23:25:42.477 4418 DEBUG oslo_service.service [req-e7cfb1d6-3397-4bd5-919e-addf3d8424b5 - - - - -] memcached_servers              = None log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:226020:26
en_austinare you sure that parameter should be stored in [pool_manager_cache:memcache] memcached_servers ?20:26
timsimPretty sure.20:27
en_austinWell.20:28
en_austin2015-11-09 23:28:01.582 4487 DEBUG designate.pool_manager.service [req-23a5cb47-2d4d-4d43-afce-5b25e0649943 noauth-user noauth-project - - -] For domain XXXXX. : UPDATE on nameserver 10.28.0.18:53 the cache serial is 0 and the actual serial is 1447100861. update_status /opt/designate/designate/pool_manager/service.py:49020:28
en_austin2015-11-09 23:28:01.584 4487 DEBUG designate.pool_manager.service [req-23a5cb47-2d4d-4d43-afce-5b25e0649943 noauth-user noauth-project - - -] Cache miss! Did not retrieve status and serial for domain XXXXXX. on nameserver 10.28.0.17:53 with action UPDATE from the cache. Getting it from the server. _retrieve_statuses20:28
timsimpool_manager_cache:memcache.memcached_servers = ['127.0.0.1:11211']20:30
en_austinAwh...20:30
timsimIs what I see in my debug logs20:30
*** james_li has joined #openstack-dns20:30
en_austin[root@designate designate]# cat designate.conf | grep memcached20:30
en_austinmemcached_servers = 127.0.0.1:1121120:30
en_austinhere should be exact Python list (in config), am I right?20:30
timsimThat should be fine. What section is it in?20:30
en_austin[pool_manager_cache:memcache]20:31
timsimAnd you don't see pool_manager_cache:memcache.memcached_servers = ['127.0.0.1:11211'] when you start pool-manager with debug?20:31
en_austinhttp://paste.openstack.org/show/478385/20:31
en_austinI see it, it's OK here20:32
*** ducttape_ has quit IRC20:32
en_austinnow the question is - why there are only 'cache miss' in logs?20:32
timsimCache misses are expected at first, until you see those update status SUCCESS logs i mdns20:32
timsimin mdns20:32
en_austinWell, will now restart all services (not PM only), will see.20:33
en_austinNo effect - still "cache miss" for every call.20:35
en_austinAwh. It was (memcached) not running - and Designate silently bypassed it. Sorry :)20:37
timsimWhat do the mdns logs look like?20:37
timsimlol, that'd explain it20:38
en_austinnow "cache hit!" logs are here :)20:38
timsimw00t20:38
mugsietimsim / Kiall: the message we should be using on patches we are -2'ing from tomorrow is here - http://graham.hayes.ie/posts/the-big-rename/ (at the bottom)20:39
timsimhttp://replygif.net/i/241.gif20:41
*** ducttape_ has joined #openstack-dns20:43
en_austinwell, seems it's working20:43
en_austinI really appreciate your help, timsim20:44
en_austinthank you :)20:44
en_austinit's 11:44PM, time to drive home...20:44
mugsietimsim: http://replygif.net/134120:44
timsimYou're welcome en_austin20:45
en_austinI'll come back again tomorrow with ZooKeeper & Designate :)20:45
en_austinand couple of more questions... :)20:46
mugsiegood - I am taking notes so I know what is wrong with our docs20:46
en_austinbtw I've enabled a debug mode on prod to show you more logs about non-provisioned records20:46
en_austinbut to tell the truth, I hope that Kilo-Liberty upgrade and cache enabling should help :)20:46
en_austinDesignate is missing a some step-by-step upgrade FAQ20:47
en_austinfor example, Kilo-Liberty upgrade FAQ should contain an info about zone-manager, breaking change in backends config, whatever20:48
mugsie++20:48
*** jmcbride has quit IRC20:52
*** csoukup has quit IRC20:52
*** jmcbride has joined #openstack-dns20:52
*** jasonsb has joined #openstack-dns20:53
openstackgerritEric Larson proposed openstack/designate-specs: Alias Records Spec  https://review.openstack.org/21478820:58
*** jasonsb has quit IRC21:00
en_austinhave a good day, thanks again21:00
en_austinsee you soon ))21:00
*** en_austin has quit IRC21:01
*** jasonsb has joined #openstack-dns21:02
*** ducttape_ has quit IRC21:10
*** jmcbride has quit IRC21:12
*** jmcbride has joined #openstack-dns21:15
*** ducttape_ has joined #openstack-dns21:23
*** jasonsb_ has joined #openstack-dns21:27
timsimKiall: Can I get a procedural +A on this sometime before we freeze: https://review.openstack.org/#/c/230770/21:28
*** jasonsb has quit IRC21:30
*** pglbutt has joined #openstack-dns21:36
*** pglass has quit IRC21:39
*** mwagner has quit IRC21:42
*** rudrajit has joined #openstack-dns21:52
*** rudrajit has quit IRC21:52
*** rudrajit has joined #openstack-dns21:53
openstackgerritTim Simmons proposed openstack/designate-specs: Incremental Zone Transfer (IXFR) Support  https://review.openstack.org/13263821:54
*** jasonsb has joined #openstack-dns21:55
*** jasonsb_ has quit IRC21:59
openstackgerritTim Simmons proposed openstack/designate-specs: Incremental Zone Transfer (IXFR) Support  https://review.openstack.org/13263822:02
*** pglass has joined #openstack-dns22:31
*** pglbutt has quit IRC22:35
*** rsyed is now known as rsyed_away22:35
*** rsyed_away is now known as rsyed22:36
*** jmcbride has quit IRC22:45
*** km has joined #openstack-dns22:49
*** km is now known as Guest3302322:49
*** jmcbride has joined #openstack-dns22:52
*** csoukup has joined #openstack-dns22:53
*** jmcbride1 has joined #openstack-dns22:55
*** jmcbride has quit IRC22:57
*** mwagner has joined #openstack-dns22:58
*** jmcbride1 has quit IRC23:01
*** Jet has joined #openstack-dns23:02
*** csoukup has quit IRC23:06
*** csoukup has joined #openstack-dns23:10
*** kei_yama has joined #openstack-dns23:24
*** jasonsb has quit IRC23:25
*** pglass has quit IRC23:31
*** ducttape_ has quit IRC23:32
*** jmcbride has joined #openstack-dns23:39
*** boris-42 has quit IRC23:58
*** mlavalle has quit IRC23:59
« Sunday, 2015-11-08 Index Tuesday, 2015-11-10 »

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!