Friday, 2014-11-14

Kiallonce we shift the HP public cloud towards Designates V2 API, and we we need to rewrite anyway, Rally is the obvious choice.. We don't have the benefit of old code etc00:00
boris-42Kiall: oh JMeter oh party hard=)00:00
boris-42Kiall:  btw we are going to support stuff like benchmarks for specific version00:01
boris-42Kiall: cause now it's a bit tricky =)00:01
ekarlsoKiall: this stinks :P00:02
Kiallekarlso: you broke it, you fix it ;)00:03
ekarlsoKiall: when passing in an endpoint should we rely on it being http://192.168.1.1:9001/v1 then or ?00:03
Kiallehh... it depends on how keystone+you wrote the code I guess?>00:03
ekarlsoKiall: well i'm changing it so that that bit is only done when using u +00:04
ekarlsou +00:04
ekarlsosigh, u + p00:04
Kiallccccccdugckeeifdgddnvdlvnucgkncckijnnnvridfv00:08
Kiall-_-00:08
boris-42Kiall: ^ cat?)00:15
Kiallboris-42: VPN token ;) 1 time password00:15
eanderssonbtw thanks for the help Kialla. :)00:16
ekarlsoKialla ? :p :D00:16
ekarlsosounds like a girl :)00:16
eanderssonI'll try to make some time to contribute back to Designate once I am back in Europe.00:16
eanderssonlol00:16
eandersson*Kiall00:16
Kiall:)00:20
openstackgerritBoris Pavlovic proposed openstack/designate: Add rally job related files  https://review.openstack.org/13440000:24
ekarlsoboris-42: boris-42 fixed tghe stuff btw00:24
boris-42Kiall: ekarlso ^00:24
boris-42ekarlso: nice =)00:24
ekarlsoboris-42: it's now running -,,-00:24
boris-42ekarlso: fucking yeah* =)00:25
ekarlsoboris-42: but for V2 we'll need to make rally use a session -,,-00:25
ekarlsosince it doesn't allow u to pass anything else :P00:25
boris-42ekarlso: lol00:25
boris-42ekarlso:  okay00:25
boris-42)00:25
boris-42ekarlso:  so I made both patches in infra and in designate to init job *00:26
openstackgerritEndre Karlson proposed openstack/python-designateclient: Make token + endpoints work correctly  https://review.openstack.org/13441000:26
ekarlsoboris-42: clients you created are cached no ?00:26
ekarlso^ boris-4200:26
boris-42ekarlso: ya00:27
boris-42ekarlso: during the run of one scenario iteration*00:27
boris-42ekarlso: otherwise you will get buthurt =)00:27
boris-42ekarlso: cause they are not thread safe=)00:27
ekarlso:P00:27
ekarlsonow if Kiall could sign off on it too...00:28
ekarlsoit would be uber00:28
Kiallekarlso: does it actually work? ;)00:28
ekarlsojust sucks none of the other guys are here :p00:28
boris-42ekarlso: lol00:28
ekarlsoKiall: yeah, I tseted with rally & the designate clietn00:28
boris-42ekarlso: it's 3:30 a.m.=)00:28
boris-42ekarlso: so people usually sleep at the moment in Russia=)00:28
ekarlsodesignate --os-token <toen> --os-endpoint http://192.168.27.100:9001 --debug domain-list00:28
ekarlsoworks fine !00:28
boris-42ekarlso: tested on devstack* lol00:29
ekarlsorally too actually :D00:29
boris-42ekarlso: =)00:29
ekarlsoboris-42: it's just 00:30 at Kiall's place so :p00:29
Kialli.e. bedtime00:30
ekarlsoKiall: pffft :p00:30
KiallCya! Goodnight ;)00:31
boris-42Kiall: see you00:31
boris-42ekarlso: hmmm01:01
boris-42ekarlso: how ot enable designate in devstack ?01:02
eanderssonTake a look at this documentation01:04
eanderssonhttp://designate.readthedocs.org/en/latest/devstack.html01:04
boris-42eandersson: danke shoön01:12
eanderssonnp :)01:14
boris-42eandersson: am I right that in designate there is only one WSGI service?01:35
boris-42eandersson: cmd/api.py01:35
*** nosnos has joined #openstack-dns01:43
boris-42ekarlso: pingngng02:04
*** amcrn has quit IRC02:08
eanderssonThere should only be the one afaik02:28
eanderssonWe use Central/Api/Sink02:28
boris-42eandersson: so one more question03:01
boris-42eandersson: where are rpc managers? is there something like https://github.com/openstack/cinder/blob/master/cinder/service.py#L12103:03
*** nosnos has quit IRC03:30
*** nosnos has joined #openstack-dns03:30
*** nosnos has quit IRC03:35
*** richm has quit IRC03:42
*** GonZo2K has quit IRC03:50
*** ryanpetrello has joined #openstack-dns04:06
*** ryanpetrello has quit IRC04:13
*** ryanpetrello has joined #openstack-dns04:18
*** nosnos has joined #openstack-dns04:27
*** ryanpetrello has quit IRC04:28
*** rmoe has quit IRC06:48
*** k4n0 has joined #openstack-dns07:08
*** nosnos has quit IRC08:00
*** nosnos has joined #openstack-dns08:03
*** jordanP has joined #openstack-dns08:30
openstackgerritEndre Karlson proposed openstack/python-designateclient: Make token + endpoints work correctly  https://review.openstack.org/13441008:50
*** GonZo2K has joined #openstack-dns10:05
*** GonZo2K has quit IRC10:07
openstackgerritEndre Karlson proposed openstack/designate: Support secondary zones  https://review.openstack.org/13368210:09
*** nosnos has quit IRC10:43
*** rediskin has joined #openstack-dns10:43
*** nosnos has joined #openstack-dns10:51
*** jmcbride has joined #openstack-dns11:06
*** Madkiss has quit IRC11:21
*** jmcbride has quit IRC11:32
openstackgerritEndre Karlson proposed openstack/designate: Support secondary zones  https://review.openstack.org/13368212:05
*** nosnos has quit IRC12:33
*** EricGonczer_ has joined #openstack-dns12:33
openstackgerritKiall Mac Innes proposed openstack/designate: Add tox check for missing pip requirements  https://review.openstack.org/13451612:37
*** EricGonczer_ has quit IRC12:43
*** cbaesema has quit IRC12:45
*** EricGonczer_ has joined #openstack-dns12:53
*** russellb is now known as rustlebee13:02
*** EricGonczer_ has quit IRC13:03
openstackgerritKiall Mac Innes proposed openstack/designate: WIP: Retry on Database Deadlocks  https://review.openstack.org/13452413:06
*** EricGonczer_ has joined #openstack-dns13:15
*** ryanpetrello has joined #openstack-dns13:20
*** ryanpetrello has quit IRC13:30
*** betsy has joined #openstack-dns13:31
*** ryanpetrello has joined #openstack-dns13:33
boris-42Kiall: hm around ?13:37
KiallYep13:37
openstackgerritMerged openstack/designate: Updated from global requirements  https://review.openstack.org/13438213:39
boris-42Kiall: so I was wokring yesterday on integration osprofiler to designate13:45
boris-42Kiall: and have question regarding to RPC services..13:45
KiallSure13:45
boris-42Kiall: so if we take a look at cinder13:46
boris-42Kiall:  I need something like that https://github.com/openstack/cinder/blob/master/cinder/service.py#L121-L12213:46
boris-42Kiall:  so we can put tracing for all RPC managers methods calls13:46
*** jordanP has quit IRC13:47
KiallDoes it have do be the cls, or is the instance OK? https://github.com/openstack/designate/blob/master/designate/service.py#L6713:47
*** jordanP has joined #openstack-dns13:48
KiallThere's also the create method on that class, which is effectively a factory.. so it could help?13:48
Kiall(I'm not 100% confident everything makes use of that factory, but we can fix that if there are cases which done)13:49
Kiallwhich don't*13:49
KiallAlso.. I kinda figured you would hook into RPC via oslo.messaging?13:51
Kiallbrb13:51
boris-42Kiall: soo there are 2 differnet separated parts13:51
boris-42Kiall: one is done in serialization/desserialization of request (via oslo.messaging stuff)13:52
boris-42Kiall:  so actually it just makes next thing if profiler is initialized in caler (we will pass to request context profling info)13:53
boris-42Kiall: on callee side if we see this info we will init profiler with passed data13:53
boris-42Kiall:  this is part one13:53
boris-42Kiall: part 2 is that I would like to trace all calls of RPC service API13:54
boris-42Kiall: and that is what I am asking about13:54
boris-42Kiall:  so endpoints is instance of what actually?13:55
Kialleffectively, the manager13:56
Kiallboris-42: L105 of https://github.com/openstack/designate/blob/master/designate/service.py should be what you need to wrap13:57
KiallHumm13:57
KiallMaybe13:57
Kialloslo.messaging supports having multiple "endpoints" (historically, called managers in nova/cinder)13:58
Kiallhere's a example of where the endpoint != the service class - https://github.com/openstack/designate/blob/master/designate/mdns/service.py#L3714:05
Kiallthat's currently the only example, and it's only using 1 endpoint but will expand..14:05
boris-42Kiall: hm hm14:06
boris-42Kiall:  so we can actually put on all classes @profiler.trace_cls()14:06
boris-42Kiall:  but it's not enough nice14:06
boris-42Kiall:  so what in service I should wrap?14:07
boris-42Kiall: actually I need to wrap actually class (not base class)14:07
KiallEhh, I'm honestly not 100% sure14:08
Kiallwell - for everything BAR designate-mdns,  L105 of https://github.com/openstack/designate/blob/master/designate/service.py has a reference to the endpoint (aka "manager") cls before it's instantiated..14:08
boris-42Kiall:  seems like I need to sleep more14:10
boris-42Kiall: yep that is what I was actually looking for14:10
KiallYea, But, it work work (I think) for the designate-mdns service - or any service that makes use of multiple oslo.messaging managers?14:11
*** nkinder has quit IRC14:11
boris-42Kiall: yep14:21
boris-42Kiall:  it will14:21
boris-42Kiall: in any case you'll be able to play14:22
boris-42Kiall:  and see how it works14:22
Kiallare you sure? A quick look at the os-profiler code makes me think it wont :) But, yea.. It'll work for most cases and we can figure it out from there.14:22
*** richm has joined #openstack-dns14:22
boris-42Kiall: why it won't lol?14:23
boris-42Kiall: if it is isntance of class and it is instance of class it will14:23
boris-42Kiall:  in cinder and nova it works well lol14:23
Kiallthe trace_cls method iterates the class methods and wraps them, which if your using oslo.messaging's multiple endpoints, means you can't just wrap the service class.. that doesn't actually have any of the RPC methods on it14:24
Kialle.g. for designate-msdns, cls there will be the Service class here: https://github.com/openstack/designate/blob/master/designate/mdns/service.py while this is the "endpoint" https://github.com/openstack/designate/blob/master/designate/mdns/notify.py14:25
Kialland - there can more more than 1 endpoint.. we don't use that (yet), but will14:25
boris-42Kiall:  heh14:26
Kiallhence why I expected os-profiler to hook straight into oslo-messaging, specifically the RPCDispatcher class14:26
Kiallthat's where all incoming RPC calls "begin"14:26
boris-42Kiall: i know but it's nicer to wrap methods that are exactly called14:27
KiallSure, but there could be 10s of different classes making up 1 RPC service :)14:27
boris-42Kiall: huh maybe wrap all of them14:28
boris-42Kiall: like iterate through multple endpoints and wrap them all14:28
KiallYep, but I'm not sure we have a way to do that right now :(14:28
boris-42Kiall:  or we can just add everywhere14:29
boris-42@trace_cls14:29
boris-42^14:29
KiallThat works I guess :)14:29
boris-42okay let me publish something=)14:30
*** EricGonczer_ has quit IRC14:32
* Kiall is scared of seeing the results ;)14:33
boris-42Kiall:  lol14:33
boris-42=)14:33
* boris-42 Kiall: some day I'll stop working on Rally & OSprofiler and start working on improving OpenStack14:34
*** EricGonczer_ has joined #openstack-dns14:35
Kialleandersson: still looking at your issue... turning into a little bit of a nightmare -_-14:43
*** ryanpetrello has quit IRC14:49
*** richm has quit IRC14:57
*** nkinder has joined #openstack-dns14:58
*** richm has joined #openstack-dns15:06
*** ryanpetrello has joined #openstack-dns15:12
*** jmcbride has joined #openstack-dns15:15
*** ryanpetrello has quit IRC15:17
Kialleandersson: I've filed https://bugs.launchpad.net/designate/+bug/1392762 for one part of the issue your seeing.15:23
uvirtbotLaunchpad bug 1392762 in designate "Concurrent requests can cause designate-central to lock up" [High,Triaged]15:23
openstackgerritMerged openstack/python-designateclient: Make token + endpoints work correctly  https://review.openstack.org/13441015:25
openstackgerritMerged openstack/designate: Add rally job related files  https://review.openstack.org/13440015:26
openstackgerritMerged openstack/designate: Add tox check for missing pip requirements  https://review.openstack.org/13451615:26
boris-42Kiall: btw you can use rally for testing/reprodcuing such bugs in gates15:40
*** nkinder is now known as nkinder_afk16:01
*** mwagner_lap has quit IRC16:05
*** paul_glass has joined #openstack-dns16:32
*** f1ller is now known as filler16:33
*** timsim has joined #openstack-dns17:03
*** mwagner_lap has joined #openstack-dns17:12
*** ryanpetrello has joined #openstack-dns17:15
eanderssonThanks Kiall.17:29
eanderssonI am feeling confident that we can mitigate the issue by simply adding more workers and nodes into the mix.17:39
eanderssonFor now anyway :p17:49
*** ryanpetrello has quit IRC17:59
*** paul_glass has quit IRC18:04
*** ryanpetrello has joined #openstack-dns18:06
*** amcrn has joined #openstack-dns18:40
*** jordanP has quit IRC18:41
timsimKiall: Around?18:46
*** ryanpetrello has quit IRC18:47
*** ryanpetrello has joined #openstack-dns18:52
*** paul_glass has joined #openstack-dns18:54
*** paul_glass has quit IRC18:55
ekarlsoKiall: he's out19:01
ekarlsowhat's up ?19:01
timsimI think I may have figured it out...just a minute19:02
*** ryanpetrello has quit IRC19:17
*** ryanpetrello has joined #openstack-dns19:22
*** MasterPiece has joined #openstack-dns19:37
*** MasterPiece has quit IRC19:58
openstackgerritTim Simmons proposed openstack/designate: Add a reverse name column to the domains table  https://review.openstack.org/13438720:18
*** ryanpetrello has quit IRC20:18
openstackgerritTim Simmons proposed openstack/designate: Add a reverse name column to the domains table  https://review.openstack.org/13438720:21
*** harmw_ is now known as harmw20:24
*** jmcbride has quit IRC20:46
*** ryanpetrello has joined #openstack-dns21:00
*** rustlebee is now known as russellb21:03
*** nkinder_afk is now known as nkinder21:08
openstackgerritBetsy Luzader proposed openstack/designate:   Can't Access Default Pool through API  https://review.openstack.org/13465321:14
*** paul_glass has joined #openstack-dns21:15
*** jmcbride has joined #openstack-dns21:19
*** ryanpetrello has quit IRC21:22
*** ryanpetrello has joined #openstack-dns21:25
betsyHmm - notice we have a new jenkins gate21:42
betsynon-voting, but mine failed it21:42
*** ryanpetrello has quit IRC21:42
timsimFrom the log, it doesn't appear to me to have anything to do with your patch: http://logs.openstack.org/53/134653/1/check/gate-rally-dsvm-designate-designate/ed5074a/console.html21:44
openstackgerritTim Simmons proposed openstack/designate: Add a reverse name columns to domains/recordsets  https://review.openstack.org/13438721:57
ekarlsoany volunteers to test v2 cli ?21:57
timsimMaybe next week /yawns21:58
*** EricGonczer_ has quit IRC22:03
*** timsim has quit IRC22:05
betsytimsim: Yeah. I saw the log. I was just wondering when and why it got added22:20
*** mwagner_lap has quit IRC22:25
*** EricGonczer_ has joined #openstack-dns22:25
*** paul_glass has quit IRC22:32
*** EricGonczer_ has quit IRC22:32
*** EricGonczer_ has joined #openstack-dns22:34
openstackgerritBoris Pavlovic proposed openstack/designate: Fix rally gates  https://review.openstack.org/13466222:34
*** EricGonczer_ has quit IRC22:59
*** jmcbride has quit IRC23:03
boris-42GUYS you are so fast on review=)23:05
boris-42https://review.openstack.org/#/c/134662/1 but it was not finished yet*23:05
Kialllol23:06
openstackgerritBoris Pavlovic proposed openstack/designate: Fix rally gates  https://review.openstack.org/13466223:07
Kiallbetsy: that new job runs rally as part of the gate ;) https://github.com/stackforge/rally/23:07
Kiallboris-42: finished? ;)23:08
boris-42Kiall:  I hope so23:08
boris-42Kiall: actually benchmarks are going to fail unit ekarlso fix that issue with client*23:08
boris-42Kiall: so there will be one more patch after fix from ekarlso that adds SLA checks*23:08
boris-42Kiall: but for now we can merge this one23:08
ekarlsowhat issue now boris-42 ?23:08
ekarlsoI thought I fixed it ?!23:08
ekarlso:P23:09
boris-42ekarlso: oh23:09
boris-42ekarlso:  let see23:09
KiallI think he did? https://github.com/openstack/python-designateclient/commit/cdf87e7fcaf2fe444904e785e1588bdb057f97de23:09
boris-42ekarlso: ^_^23:09
boris-42ekarlso: Kiall I didn't know that!=)23:09
Kialltimsim: I spent the last 30 minutes trying to find how to make SQL Alchemy issue the REVERSE() function without doing it in code, and I'm totally shocked it's not supported -_-23:10
boris-42Kiall: lol it happens=)23:12
boris-42Kiall:  btw it's faster to ask author =)23:12
boris-42Kiall: zzzeek is his nickname+)23:12
KiallYea, I've talked to him a few times.. But I was on my phone, no IRC ;)23:13
boris-42Kiall: heh=)23:13
openstackgerritEndre Karlson proposed openstack/designate: MDNS part of Secondary zones  https://review.openstack.org/13368323:13
boris-42ekarlso: Error <class 'designateclient.exceptions.Unknown'>: None uh OH23:25
ekarlsoboris-42: where ?23:27
boris-42ekarlso:  your patch in rally that adds SLAs23:27
boris-42ekarlso:  i re-run check experimental23:27
eanderssonHmm, odd error.23:28
eanderssonResourceClosedError: This result object does not return rows. It has been closed automatically.23:28
eanderssonAnyone seen this when trying to run multiple workers?23:29
boris-42ekarlso: http://logs.openstack.org/39/133839/1/experimental/gate-rally-dsvm-designate-rally/b463f5a/rally-plot/sla.txt23:29
eanderssonfor Central23:29
boris-42ekarlso: race lol?23:29
ekarlsoboris-42: race ? :P23:30
ekarlsoboris-42: I think the reason it is still bork is due to the fact that the fix is not in a "released" version yet of designateclient -,,-23:30
ekarlsooh, nvm that then23:30
ekarlsohmm23:30
ekarlsoboris-42: how to get the full stack ?23:31
boris-42ekarlso:  https://github.com/boris-42/boris-42.github.io/blob/master/d.json#L122-L12423:31
Kialleandersson: I spotted that today while debugging your issue... It actually seems to be caused by the same issue :(23:31
boris-42ekarlso: nope23:31
boris-42ekarlso: cause dsvm is installing from master23:31
boris-42ekarlso: at least it should lol23:31
ekarlsoso it's deleting record.23:31
boris-42ekarlso: seems like IT's race23:32
boris-42ekarlso: part of iterations works part doesn't23:32
boris-42ekarlso: it's not rally or client issue23:32
boris-42Kiall: ^ so I think rally is good for such stuff=)23:32
Kialleandersson: I have a fairly good idea on what the underlying issue is, it's just proving difficult to fix :( I filed a bug for it BTW - https://bugs.launchpad.net/designate/+bug/139276223:32
uvirtbotLaunchpad bug 1392762 in designate "Concurrent requests can cause designate-central to lock up" [High,Triaged]23:32
Kiallboris-42: lol.. I'm sure it's the same damn bug -_-23:32
ekarlso:D23:32
boris-42Kiall:  so I can add SLA for all benchmarks and rally job will fail always..23:33
boris-42Kiall: or add everything except that benchmark23:33
boris-42Kiall:  on monday we will get a better HTML report (so you'll get inside it all traces)23:33
boris-42Kiall:  like this http://logs.openstack.org/44/131844/15/check/gate-rally-dsvm-rally/5001b1a/rally-plot/results.html.gz#/Dummy.dummy_exception_probability#failures23:34
KiallSince it's non voting, I think it's OK to leave the broken run in.. We need to find + fix the issue anyway23:34
boris-42Kiall: cause now you need to download full json and analyze it =(23:34
*** EricGonczer_ has joined #openstack-dns23:34
boris-42Kiall: okay so I am going to add SLA check no failures23:34
*** rediskin has left #openstack-dns23:35
openstackgerritBoris Pavlovic proposed openstack/designate: Fix rally gates  https://review.openstack.org/13466223:37
Kiallboris-42: that the winning patch? ;)23:37
boris-42Kiall: ya it should be23:38
boris-42=)23:38
boris-42Kiall:  I am usually doing for 40 patcheset's lol23:38
boris-42=)23:38
Kialllol...23:38
boris-42Kiall:  like this lol https://review.openstack.org/#/c/129060/23:38
boris-42=)23:38
boris-42Kiall:  not so fast as you guys=)23:38
Kiall;)23:38
KiallOkay.. Gone! It's late.. cyas!23:38
ekarlsoKiall: enjoy : )23:39
boris-42Kiall: =)23:39
boris-42Kiall: see you23:39
eanderssonYep, read it last night Kiall. Thanks.23:39
eanderssonI set up a few extra nodes to mitigate the issue. :p23:40
*** nkinder has quit IRC23:40
boris-42ekarlso: hm so these warning about deleting resources http://logs.openstack.org/62/134662/2/check/gate-rally-dsvm-designate-designate/e08bcb5/console.html may be jsut cause of bug?23:41
*** EricGonczer_ has quit IRC23:41
ekarlsoboris-42: which bug u mean ?23:43
ekarlsothe deadlock one ?23:43
boris-42ekarlso: idk =)23:43
ekarlsoeandersson: u where from blizz no ? ^23:43
boris-42ekarlso:  but for some reason part of iterations works=)23:43
boris-42ekarlso:  part not, as well during cleanup stuff23:43
boris-42ekarlso: some resource can't be deleted cause max retries occur..23:43
ekarlsoboris-42: I think it's due to designate deadlocking23:43
boris-42ekarlso: okay so I merged your patch in rally that adds SLAs23:44

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!