Friday, 2015-01-16

rjrjranybody on to help me figure this out?00:05
rjrjrso, one thought i had was to patch central so that when it calls pool_manager, it calls the mock and does nothing.00:07
rjrjrbut then i started thinking, central is probably used in many places in our unit tests...00:07
rjrjri see no way to just clear oslo messaging.  and setting up a client to eat the messages seems like overkill.00:09
KiallHeya - Back, got dragged into a meeting00:25
KiallGive me 5 mins and I can take another look - find anything else out?00:26
*** EricGonczer_ has joined #openstack-dns00:27
openstackgerritGraham Hayes proposed openstack/designate: Add more tests to the central service  https://review.openstack.org/14768800:44
rjrjrjust for grins, i fired up a pool manager instance and pointing the unit tests for central at it.  sure enough, pool manager is being called.00:53
rjrjrKiall: let me know when you are back on.00:55
openstackgerritGraham Hayes proposed openstack/designate: Add more tests to the central service  https://review.openstack.org/14768800:55
Kiallrjrjr: back - for real this time ;)00:56
KiallI think everyone who would walk by and stop for a chat have either just done it, or gone home ;)00:56
rjrjrso, i fired up a pool manager instance manually and ran just the central tests, pointing those tests at the systems rabbit.  sure enough, there are all those wonderful domains being created, updated, etc.00:56
rjrjri'm convinced this is the issue.00:57
rjrjrso, i started thinking about the solution:00:57
rjrjr1) somehow clean out the 'fake' transport00:57
rjrjr2) start using mock objects wherever pool manager is called.  this probably goes throughout all of our unit tests.00:58
rjrjr3) write a server to eat the messages.00:59
rjrjr4) ?00:59
rjrjri looked and our unit tests are not very 'unity'.  we are calling central all over the place.00:59
rjrjrand sure enough, even in pool manager, i'm calling central indirectly by using self.create_domain() from the base test class.01:00
rjrjreven in pool manager unit tests.01:00
rjrjrultimately, it feels like our unit tests could be more compartmentalized than they are now.01:01
rjrjrbut, to do that will require a fair amount of work.01:02
rjrjrso, i'm open to suggestions on any shortcuts you can think of.01:03
*** EricGonczer_ has quit IRC01:04
*** Stanley00 has joined #openstack-dns01:05
KiallLol - I apologize. Again. Alex came back over -_-01:07
rjrjrnp.  just hoping to brainstorm on a the best way to proceed here..01:07
*** EricGonczer_ has joined #openstack-dns01:08
Kiall1 and/or 2 sound like approperate options - both would likely be implemented as fixtures and we can slot in.. The Q is figuring out if this really is out issue :/01:08
KiallHave we managed to prove it is?01:08
openstackgerritMerged openstack/designate: Updated from global requirements  https://review.openstack.org/14702501:09
KiallA quick test would be to remove the self.pool_manager = start_service('pool_manager') from your code, and put it into the base test case - then update all the calls from central -> PM to be call rather than cast.. If that passes, I think that would prove it..01:09
rjrjri fired up a standalone pool manager instance, point the central oslo messaging at the system rabbit, and sure enough, messages flowed to pool manager.  (that in itself is not necessarily the silver bullet) but i did notice those domains are the exact domains we are seeing with the problem.01:09
KiallHumm - No other changes? Because we set oslo.messaging to use the fake driver, so it shouldn't use a real rabbit at all?>01:10
rjrjri changed the unit test for the base to use 'rabbit' and added the password/host.01:10
rjrjrdoes fake driver mean some Python implementation of a queue?01:11
KiallKinda - ish.. https://github.com/openstack/oslo.messaging/blob/master/oslo_messaging/_drivers/impl_fake.py#L9101:11
rjrjrit's a working driver though.  not fake in the sense of our network_api or backend drivers.01:12
KiallOh, yea, It's a working driver that just does everything with dict's and list's rather than an external queue01:13
rjrjrokay.   i'm 100% convinced this is our problem then.01:13
rjrjrso, option 4) run pool_manager in the base test class like we do central.01:14
rjrjrhonestly though, that is taking us away from what unit tests are suppose to be.01:15
KiallWell - That will fix it, if this is the issue, but dig us furthur into the spaghetti unit tests's we already have :/01:15
Kiallsnap.01:15
rjrjri'm concerned if mdns tests run before pool manager tests and the pool manager calls are not mocked.01:16
rjrjri verified the API is not mocked and actually making calls to central.01:16
KiallSo - tests are gatherd into a giant list, sorted randomly, split into smaller lists to match the number of CPU cores, and then executed01:16
Kiallso - if ordering becomes important somewhere, everything will explode01:17
rjrjrpool manager tests have already exploded.01:17
rjrjrLOL01:17
*** richm has quit IRC01:17
Kiall`tox -e py27 -- --until-failure` <-- That's a great one for finding ordering issues ;)01:18
KiallIt just runs over and over and over until something explodes.01:18
rjrjri have all external calls except for the database calls mocked.  so when i call central or mdns, they are mocked.  i need to make the change to not use self.create_domain() since that is really calling central which calls pool_manager ...01:18
rjrjrsorry, was just saying if we decide to tackle this by mocking pool manager calls wherever they are, that could be quite a few places.01:19
rjrjr(mdns, API, central so far001:19
rjrjrfar)01:19
*** rmoe_ has quit IRC01:19
rjrjrbut, that would probably be the most correct way to handle this.01:19
rjrjrnot expediant, but correct. :)01:20
KiallYea, it probably would be - Why don't we be 100% sure first, and start with your option #4 - start PM in the base test... BUT - I'm not sure that alone helps, since they are cast rather call messages..01:21
Kiallso cast, return, test finish, stop PM, 1 message left in queue01:21
rjrjrwe would need to leave the PM up long enough to ensure it consumes the messages before stopping it.01:21
rjrjrmaybe a delay of 5 or 10 seconds just to be sure.01:22
KiallSwapping them -> call and starting PM should give us the guarantee that this really is the issue, and that we're not missing something obvious01:22
*** ryanpetrello has joined #openstack-dns01:22
rjrjri'm convinced this is the issue.01:22
rjrjrbut, if you want me to go through this exercise, i can.01:22
Kiall5-10 seconds per test? The tests task 37 sec right now, that would bring them to like an hour ;)01:23
Kiallrjrjr: if your 100% sure, then no... No need. That change won't merge anyway :) We would need to find a fix without the cast -> call part of the change at a minimun..01:24
rjrjrthere is no real good delay to introduce here.  if we wanted to have this be the solution, we'd want a long enough delay to guarantee the messages are consumed.  but, since this is just a test, i can do this without a delay.01:24
rjrjri'm sure.01:24
rjrjrso, start patching PoolManagerAPI in central and see how much of this we can clean up?01:25
KiallI'm eyeballing a simpler change of guaranteeing the queue is cleared....01:25
rjrjrthen move onto API, mdns, and whatever else is using central.01:25
rjrjrokay.  i'll give you time to think/play.01:25
KiallJust trying to trace through oslo.messaging01:25
KiallIt's unfamilar so hard to follow -_-01:26
rjrjrKiall: the right approach long term would be fix all the unit tests to be standalone with no running service outside of the database.01:27
Kiall150% yes...01:27
KiallBut - We're a ways off that right now..01:27
rjrjrKiall: i was asked to accellerate getting Designate rolled out into production.  my management wants it by February 15th.  i tried the "we're a ways off from that right now" today and didn't get very far. :)01:30
Kialllol - nice.01:30
openstackgerritMerged openstack/designate: Add more unit tests to mdns service  https://review.openstack.org/14766901:30
Kiallrjrjr: the current patchset, are all those tests OK on their own?01:31
*** stanzgy has joined #openstack-dns01:32
rjrjryes.  i have 4 places where i used Equals instead of Equal, but for the most part, it is ready.  i also added flags to the configuration file to disable the 2 timers (sync and recovery)01:32
*** boris-42 has quit IRC01:33
rjrjrbut outside those few changes, what i checked in works.01:33
rjrjr(Equal replaces the deprecated Equals according to PEP)01:33
* Kiall has his fingers crossed for a 2 line fix :D01:33
Kiallor not01:34
*** openstack has joined #openstack-dns01:35
*** ChanServ sets mode: +v openstack01:35
KiallActually - It broke some other tests which rely on / validate notifications.. I think all yours passed01:35
Kiallrunning again with the logs saved this time..01:35
rjrjrk01:35
*** ryanpetrello has quit IRC01:35
*** rmoe has joined #openstack-dns01:36
rjrjrjust as a FYI, the fix for the problem with the one broken PM unit test was to replace the call to a function for side_effect with an iterator (list) instead.01:37
rjrjri had picked up the side_effect function code online and it worked for one method, not the other.  i replace them both with a list instead and now all the unit tests work.  (that code was submitted in the last patch.)01:37
Kiallrjrjr: so - this get's us REALLY close http://paste.openstack.org/show/158194/01:37
Kiall4 failing tests, all unrelated to pools, and all are failing due to notification issues - which rpc.init() in the wrong place will likely do01:38
rjrjrinteresting.   you know a bit more about this stuff than i do. :)01:39
rjrjrso, rpc.init() clears the queue apparently?01:39
KiallI *think* it causes a new instance of the FakeDriver to be created.. Which would in turn happen to clear em01:40
Kiall(I'm still tracing ;))01:40
rjrjri tried creating a new Fixture in an attempt to do the same thing, but it didn't work.  neither did a call to <fixture>.reset()01:41
rjrjri was working with the fixture though, not the driver.01:42
KiallThe Messaging fixture is based around setting it's config up, it doesn't seem to clear etc01:42
*** Stanley00 has quit IRC01:45
KiallI'm stupted as to why it's breaking those tests, but I 100% agree with you now that we've found the real issue.01:49
KiallThis could even be the issue I've been trying to trace for MONTHS.. So NICE :D01:50
rjrjri'm testing now to see which tests are breaking.01:51
rjrjrpersonally, i'm glad this is understood.  it really looked like a database issue earlier this morning.01:51
KiallI've gotta run in 10 mins - trying out 1 last idea...01:51
*** Stanley00 has joined #openstack-dns01:52
Kiall  py27: commands succeeded01:52
Kiall  congratulations :)01:52
Kiallrjrjr: this did it - http://paste.openstack.org/show/158195/01:52
Kiall(Just that + your current patch)01:53
KiallI'm unsure if that will have any other negative side effects though01:53
KiallWe may be better adding a rpc.deinit() to the designate.rpc module, and writing a fixture to call rpc.deinit() as it's cleanup01:53
rjrjrok.  i'll try it out.  let me do some testing and if it looks stable, i'll submit that with the other few changes i have.01:54
KiallI was about to ping endre - since he wrote this code.. but realized it's 3am for him ;)01:54
KiallOkay - Gone!01:54
rjrjrl8r01:54
*** GonZo2000 has quit IRC01:58
*** timfreun1 is now known as timfreund02:06
*** GonZo2000 has joined #openstack-dns02:27
*** GonZo2000 has joined #openstack-dns02:27
*** vinod has joined #openstack-dns02:30
*** GonZo2000 has quit IRC02:30
*** EricGonczer_ has quit IRC02:52
*** mwagner_lap has joined #openstack-dns03:28
*** GonZo2000 has joined #openstack-dns03:32
*** GonZo2000 has joined #openstack-dns03:32
*** GonZo2000 has quit IRC03:51
*** boris-42 has joined #openstack-dns04:43
*** ryanpetrello has joined #openstack-dns05:13
*** ryanpetrello has quit IRC05:24
*** nihilifer has joined #openstack-dns06:54
*** chlong has quit IRC07:20
openstackgerritEndre Karlson proposed openstack/designate: Cleanup dead code and add some tests for coverage  https://review.openstack.org/14768708:27
openstackgerritEndre Karlson proposed openstack/designate: Remove dead server code in storage and add tests  https://review.openstack.org/14768708:28
*** jordanP has joined #openstack-dns09:17
*** GonZo2000 has joined #openstack-dns10:22
*** GonZo2000 has joined #openstack-dns10:22
*** GonZo2000 has quit IRC10:31
*** stanzgy has quit IRC10:50
*** Stanley00 has quit IRC10:53
*** untriaged-bot has joined #openstack-dns11:02
untriaged-botUntriaged bugs so far:11:02
untriaged-bothttps://bugs.launchpad.net/designate/+bug/128944411:02
untriaged-bothttps://bugs.launchpad.net/designate/+bug/141143211:02
untriaged-bothttps://bugs.launchpad.net/designate/+bug/140820211:02
untriaged-bothttps://bugs.launchpad.net/designate/+bug/140962911:02
untriaged-bothttps://bugs.launchpad.net/designate/+bug/141113211:02
*** untriaged-bot has quit IRC11:02
*** kbyrne has quit IRC12:06
*** kbyrne has joined #openstack-dns12:08
rjrjrKiall: lots of fun so far this morning.  So, I wrote an RPCFixture which registers the rpc.cleanup for cleanup.  (basically, this puts us back to the state where we call rpc.init from the base test's setup. except we also now cleanup the rpc transport after the tests are complete.)12:24
rjrjrthis puts us back into the situation where the 4 unit tests in central fail that are testing for a notification.12:25
rjrjrthat is when i notice we are not actually stopping the central service (there is no kill() method for services which is what the ServiceFixture is calling.12:25
rjrjrnow i have it so the RPC transport and central is being started and stopped for every unit test.  except now even more unit tests are failing.  i suspect the way we have the services get the RPC transport is causing this.  we do it at the class level and i'll bet central is still hanging onto the old RPC transport that no longer exists.12:27
ekarlsofun change incoming13:03
openstackgerritEndre Karlson proposed openstack/designate: Switch to o.log and remove unused LOG vars  https://review.openstack.org/14783113:03
ekarlsohirr, about 13x+ files touched :p13:04
ekarlsowhat u up for so early rjrjr ?13:06
ekarlsogo to bed :p13:06
*** EricGonczer_ has joined #openstack-dns13:21
*** EricGonczer_ has quit IRC13:22
*** EricGonczer_ has joined #openstack-dns13:25
*** GonZo2000 has joined #openstack-dns13:30
*** GonZo2000 has quit IRC13:30
*** puck` has quit IRC13:55
*** puck has joined #openstack-dns13:56
eanderssono13:59
ekarlsoeandersson: hey swede!14:02
eanderssonFriday!14:02
eanderssonHey14:02
ekarlso:D14:03
openstackgerritRon Rickard proposed openstack/designate: WIP Add Unit Tests for Pool Manager  https://review.openstack.org/14724114:08
openstackgerritRon Rickard proposed openstack/designate: Ensure Pool Manager Works for Multiple Backend Servers  https://review.openstack.org/14624614:08
rjrjrekarlso: i'd love to.  but, i have to get this done.  3 days of working on unit tests for pool manager.  not fun.14:09
*** richm has joined #openstack-dns14:15
*** betsy has joined #openstack-dns14:18
*** rjrjr has quit IRC14:20
*** rjrjr has joined #openstack-dns14:29
*** ryanpetrello has joined #openstack-dns14:29
*** vinod1 has joined #openstack-dns14:46
*** jmcbride has joined #openstack-dns14:55
openstackgerritRon Rickard proposed openstack/designate: Add Unit Tests for Pool Manager  https://review.openstack.org/14724114:57
*** timsim has joined #openstack-dns15:00
*** jmcbride has quit IRC15:02
*** jmcbride has joined #openstack-dns15:02
*** nihilifer has quit IRC15:23
*** jmcbride has quit IRC15:24
*** jmcbride has joined #openstack-dns15:31
openstackgerritRon Rickard proposed openstack/designate: Pool Manager Throws An Error When No Servers are Defined  https://review.openstack.org/14788615:35
*** f1ller is now known as filler15:36
*** jmcbride has quit IRC15:37
*** jmcbride has joined #openstack-dns15:37
*** paul_glass has joined #openstack-dns15:38
*** vinod has joined #openstack-dns15:43
*** vinod1 has quit IRC15:45
*** EricGonczer_ has quit IRC15:55
*** EricGonczer_ has joined #openstack-dns15:59
openstackgerritMerged openstack/designate: Add more tests to the central service  https://review.openstack.org/14768816:07
*** jordanP has quit IRC16:50
*** rmoe has quit IRC17:02
*** untriaged-bot has joined #openstack-dns17:02
untriaged-botUntriaged bugs so far:17:02
untriaged-bothttps://bugs.launchpad.net/designate/+bug/141170017:02
untriaged-bothttps://bugs.launchpad.net/designate/+bug/141171717:02
untriaged-bothttps://bugs.launchpad.net/designate/+bug/140962917:02
untriaged-bothttps://bugs.launchpad.net/designate/+bug/141113217:02
untriaged-bothttps://bugs.launchpad.net/designate/+bug/141143217:02
untriaged-bothttps://bugs.launchpad.net/designate/+bug/128944417:02
untriaged-bothttps://bugs.launchpad.net/designate/+bug/140820217:02
*** untriaged-bot has quit IRC17:02
*** EricGonczer_ has quit IRC17:09
*** EricGonczer_ has joined #openstack-dns17:09
*** rmoe has joined #openstack-dns17:15
*** eandersson has quit IRC17:38
openstackgerritMerged openstack/designate: Ensure Pool Manager Works for Multiple Backend Servers  https://review.openstack.org/14624617:52
*** shakamunyi has joined #openstack-dns18:02
*** puck has quit IRC18:32
*** puck has joined #openstack-dns18:33
*** vinod has quit IRC18:35
*** GonZo2000 has joined #openstack-dns18:47
openstackgerritMerged openstack/designate: Moved to oslo_utils namespace  https://review.openstack.org/14733118:51
*** vinod has joined #openstack-dns18:59
*** paul_glass has quit IRC19:50
*** timsim has quit IRC20:21
*** timsim has joined #openstack-dns20:48
*** timsim has quit IRC20:50
*** timsim has joined #openstack-dns20:51
*** timsim has quit IRC20:53
*** timsim has joined #openstack-dns20:54
*** csoukup has joined #openstack-dns21:02
ekarlsoKiall: did you see the oslo log change ?21:08
rjrjri am moving on to https://bugs.launchpad.net/designate/+bug/140820221:17
rjrjrKiall: I ended up getting Pool Manager unit tests integrated by just cycling RPC to clear out the messages.  thanks for the help yesterday.21:17
timsimrjrjr: You should probably take a nap first :)21:18
rjrjrtimsim: if you had the day and news i did the past few days, you wouldn't sleep either.  LOL21:18
timsimWell rock on then I guess.21:18
rjrjrlook forward to seeing everyone next week.21:19
rjrjrtimsim: do you know paul_glass?21:19
ekarlsorjrjr: what news ? :P21:20
timsimYep, sit right next to him lol21:20
rjrjrekarlso: internal stuff.  nothing i can share unfortunately. :(21:20
rjrjrtimsim: thank paul_glass for helping with testing Pool Manager.  i've been feeling like a one man army and his testing helped me get some things straightened out.21:22
ekarlsorjrjr: cool stuff I hope ? :p21:22
ekarlsohah21:22
rjrjrekarlso: actually the opposite. :(21:22
timsimWill do!21:22
ekarlso:/21:22
rjrjrtimsim: how is the agent looking?  next week, some eBay guys will be asking questions and I have a feeling the agent will end up being the answer.21:24
ekarlsorjrjr: no mdns ? :p21:25
timsimrjrjr: Alright, I mean, it works. I've been getting my first feedback from vinod the last few days.21:25
timsimekarlso: Agent still uses mdns ;)21:25
ekarlsoah21:25
rjrjrthere is much gnashing of teeth over Designate being a DNS master.21:25
ekarlsotimsim: I thought the agent was instead of mdns ?21:26
timsimekarlso: Nope, it's the "reflection" of mdns. Mdns sends NOTIFYs and AXFRs to the Agent, which takes zone info and applies it to a DNS server.21:26
rjrjri've been hearing concerns about this from internal folks.21:26
ekarlsorjrjr: good timing :p21:27
timsimrjrjr: Yeah we probably couldn't have mdns be our master either.21:27
rjrjrwe are rolling out Designate icehouse initially because of the concerns.21:27
rjrjri was hoping it would be kilo (hence my working on PM), but now I'm hearing that is too big a risk. LOL21:28
*** nkinder_away has quit IRC21:44
*** westy has joined #openstack-dns21:45
*** nkinder_away has joined #openstack-dns21:47
*** csoukup has quit IRC21:57
westyhey all - running into some issues between designate/powerdns. designate looks configured properly- can create servers domains, records, etc. but all remain in PENDING status. none of these are being written to powerdns schema22:02
westyis there a separate sync step that needs to take place?22:03
westyi've tried to curl http://localhost/v1/domains/sync but that doesn't seem to do much22:03
westyusing latest master.. e56eb9e422:04
rjrjrdo the pool manager logs show a problem?22:09
rjrjror the mdns logs?22:09
rjrjra request to create a domain is sent to the API which communicates with Central.  Central updates the database and sends the request to Pool Manager.  Pool Manager communicates directly to the DNS servers for creating the domain through the backend plugin.22:11
*** csoukup has joined #openstack-dns22:11
westymaybe i'm not following the correct instructions.. not running pool manager or mdns as far as I know22:11
westylooking at http://designate.readthedocs.org/en/latest/install/ubuntu.html22:12
rjrjrif you are using the code from the master, pool manager and mdns are requirements.  this is true since the kilo release.22:12
rjrjrif you don't want to use pool manager/mdns (they are still work in progress, but pretty well along) then you'll want the icehouse release.22:12
KiallAh - We've been working through some big changes recently, docs aren't up to date.. Also - Docs have moved to http://docs.openstack.org/developer/designate/22:12
westythat connects the dots -22:13
KiallI should kill / put a notice up on read the docs .org22:13
westyi tried with the icehouse version, following similar instruction, but ran into issues. i'll look at pool manager and mdns. thanks22:14
rjrjrwesty, if you can be specific on the problems you had with icehouse, we can help.  also, we can help if you want to continue using the code from the master.  just let us know.22:14
timsimKiall: our next sprint should probably be docs :P22:15
rjrjrtimsim: good call!22:15
westyrjrjr - thanks. I'll dig into master for a bit more22:16
westyappreciate the help22:16
Kialltimsim: lol - from an internal weekly status update email I sent like two hours ago..22:17
KiallOn Thursday, we held the first "topic sprint " - which we plan to make a monthly occurrence. All the active contributes to designate got together (virtually) and spent half the day working through a single area of weakness in the codebase - test coverage was up this month, and while we didn't make as much progress as I would have hoped, it's only our first round ;) - Next month looks like it'll be a documentation sprint..22:17
timsimlol.22:17
timsimIt was cool to get everyone together yesterday. I'll be interested to see what kind of gains we made once everything lands. Probably like 8% or so?22:19
betsyoh, boy! Docs will be just as much fun as unit tests! :)22:26
Kialltimsim: I'm not sure we made a big dent in the %, but I think we got everyone more familar with writing tests / using mock etc ;)22:31
timsimFor sure. I thought it was productive, either way :)22:31
KiallYep - It was :)22:32
*** EricGonczer_ has quit IRC22:34
*** shakamunyi has quit IRC22:40
*** westy has quit IRC22:55
*** jmcbride1 has joined #openstack-dns22:59
*** ryanpetrello has quit IRC22:59
*** csoukup has quit IRC23:00
*** jmcbride has quit IRC23:02
*** ryanpetrello has joined #openstack-dns23:03
*** openstackgerrit has quit IRC23:19
*** openstackgerrit has joined #openstack-dns23:19
*** ChanServ sets mode: +v openstackgerrit23:19
*** ryanpetrello has quit IRC23:55
*** ryanpetrello has joined #openstack-dns23:59

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!