| *** shakamunyi has quit IRC | 00:01 | |
| *** shakamunyi has joined #openstack-dns | 00:03 | |
| *** EricGonczer_ has quit IRC | 00:05 | |
| *** EricGonczer_ has joined #openstack-dns | 00:14 | |
| *** jmcbride has quit IRC | 00:18 | |
| *** EricGonczer_ has quit IRC | 00:19 | |
| *** paul_glass has joined #openstack-dns | 00:20 | |
| *** paul_glass has quit IRC | 00:25 | |
| *** eandersson has quit IRC | 00:39 | |
| *** eandersson has joined #openstack-dns | 00:40 | |
| *** EricGonczer_ has joined #openstack-dns | 00:41 | |
| *** rjrjr has quit IRC | 00:45 | |
| *** stevelle has left #openstack-dns | 00:50 | |
| *** amcrn has quit IRC | 01:04 | |
| *** nkinder has joined #openstack-dns | 01:07 | |
| *** rmoe has joined #openstack-dns | 01:12 | |
| *** shakamunyi has quit IRC | 01:14 | |
| *** GonZoPT has quit IRC | 01:22 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/designate: Updated from global requirements https://review.openstack.org/134092 | 01:23 |
|---|---|---|
| *** jmcbride has joined #openstack-dns | 01:29 | |
| *** jmcbride has quit IRC | 01:30 | |
| *** jmcbride has joined #openstack-dns | 01:31 | |
| *** EricGonczer_ has quit IRC | 01:46 | |
| *** jmcbride has quit IRC | 01:46 | |
| *** nosnos has joined #openstack-dns | 02:06 | |
| *** stevelle has joined #openstack-dns | 02:44 | |
| *** richm has quit IRC | 02:45 | |
| *** nosnos has quit IRC | 03:25 | |
| *** nosnos has joined #openstack-dns | 03:57 | |
| *** pfreund_ has quit IRC | 04:05 | |
| *** CaptTofu has quit IRC | 04:06 | |
| *** pfreund_ has joined #openstack-dns | 04:06 | |
| *** CaptTofu has joined #openstack-dns | 04:07 | |
| *** harmw_ has joined #openstack-dns | 04:08 | |
| *** simonmcc_ has joined #openstack-dns | 04:10 | |
| *** ahu_ has joined #openstack-dns | 04:11 | |
| *** vipul has quit IRC | 04:12 | |
| *** harmw has quit IRC | 04:12 | |
| *** simonmcc has quit IRC | 04:12 | |
| *** filler has quit IRC | 04:12 | |
| *** ahu has quit IRC | 04:12 | |
| *** vipul has joined #openstack-dns | 04:12 | |
| *** simonmcc_ is now known as simonmcc | 04:14 | |
| *** f1ller has joined #openstack-dns | 04:14 | |
| *** alokj has joined #openstack-dns | 05:20 | |
| *** k4n0 has joined #openstack-dns | 06:09 | |
| *** cbaesema has quit IRC | 06:38 | |
| *** cbaesema has joined #openstack-dns | 06:42 | |
| *** rmoe has quit IRC | 07:45 | |
| *** yfujioka has joined #openstack-dns | 09:19 | |
| *** jordanP has joined #openstack-dns | 09:29 | |
| *** MasterPieceF has joined #openstack-dns | 10:15 | |
| *** MasterPiece has quit IRC | 10:15 | |
| *** MasterPieceF is now known as MasterPiece | 10:15 | |
| *** MasterPiece has quit IRC | 10:25 | |
| *** mwagner_lap has quit IRC | 10:51 | |
| openstackgerrit | Endre Karlson proposed openstack/designate: Support secondary zones https://review.openstack.org/133682 | 10:52 |
| *** nosnos has quit IRC | 11:34 | |
| *** nosnos has joined #openstack-dns | 11:34 | |
| Kiall | eandersson: aha, with paris etc I totally forgot about the issue. I can take a look tomorrow, today is a busy day! | 11:35 |
| openstackgerrit | Merged openstack/designate: Move import code to dnsutils https://review.openstack.org/133681 | 11:35 |
| openstackgerrit | Endre Karlson proposed openstack/designate: Support secondary zones https://review.openstack.org/133682 | 11:37 |
| *** nosnos has quit IRC | 11:39 | |
| *** stevelle has left #openstack-dns | 11:54 | |
| *** alokj has quit IRC | 12:25 | |
| *** yfujioka has quit IRC | 12:28 | |
| *** mwagner_lap has joined #openstack-dns | 12:34 | |
| openstackgerrit | Endre Karlson proposed openstack/python-designateclient: Move some useful code outside v1 https://review.openstack.org/134186 | 12:53 |
| openstackgerrit | Endre Karlson proposed openstack/python-designateclient: Move session creation up to shell https://review.openstack.org/133208 | 12:56 |
| openstackgerrit | Endre Karlson proposed openstack/python-designateclient: Move some useful code outside v1 https://review.openstack.org/134186 | 13:11 |
| openstackgerrit | Endre Karlson proposed openstack/python-designateclient: V2 CLI https://review.openstack.org/133676 | 13:11 |
| openstackgerrit | Endre Karlson proposed openstack/python-designateclient: V2 Bindings https://review.openstack.org/134196 | 13:11 |
| ekarlso | Kiall: ^ | 13:11 |
| openstackgerrit | Endre Karlson proposed openstack/python-designateclient: Move session creation up to shell https://review.openstack.org/133208 | 13:11 |
| ekarlso | the session / Move code thing | 13:12 |
| openstackgerrit | Endre Karlson proposed openstack/python-designateclient: Move session creation up to shell https://review.openstack.org/133208 | 13:28 |
| openstackgerrit | Endre Karlson proposed openstack/python-designateclient: V2 Bindings https://review.openstack.org/133675 | 13:29 |
| openstackgerrit | Kiall Mac Innes proposed openstack/designate: Add Server object validations https://review.openstack.org/129625 | 13:39 |
| openstackgerrit | Kiall Mac Innes proposed openstack/designate: Add basic validation functionality to DesignateObjects https://review.openstack.org/127846 | 13:39 |
| openstackgerrit | Kiall Mac Innes proposed openstack/designate: Support Nested/Recursive Object Validations https://review.openstack.org/129895 | 13:39 |
| openstackgerrit | Kiall Mac Innes proposed openstack/designate: Add Domain object validations https://review.openstack.org/129909 | 13:39 |
| *** ryanpetrello has joined #openstack-dns | 13:49 | |
| openstackgerrit | Kiall Mac Innes proposed openstack/designate: Add Server object validations https://review.openstack.org/129625 | 13:52 |
| openstackgerrit | Kiall Mac Innes proposed openstack/designate: Support Nested/Recursive Object Validations https://review.openstack.org/129895 | 13:52 |
| openstackgerrit | Kiall Mac Innes proposed openstack/designate: Add Domain object validations https://review.openstack.org/129909 | 13:52 |
| *** ryanpetrello has quit IRC | 13:54 | |
| *** nkinder has quit IRC | 14:08 | |
| *** k4n0 has quit IRC | 14:10 | |
| *** richm has joined #openstack-dns | 14:12 | |
| *** MasterPiece has joined #openstack-dns | 14:21 | |
| *** MasterPiece has quit IRC | 14:22 | |
| betsy | Hey kiall/mugsie: I’ve got a question for y’all re the pool API | 14:30 |
| *** mikeit has joined #openstack-dns | 14:44 | |
| *** jmcbride has joined #openstack-dns | 14:46 | |
| *** mikeit has quit IRC | 14:48 | |
| openstackgerrit | Merged openstack/designate: Updated from global requirements https://review.openstack.org/134092 | 14:54 |
| *** nkinder has joined #openstack-dns | 14:54 | |
| Kiall | betsy: heya | 14:58 |
| *** ryanpetrello has joined #openstack-dns | 15:04 | |
| betsy | kiall: hey | 15:06 |
| betsy | So, I just want to make sure the pool api is behaving as expected. | 15:07 |
| betsy | the default pool has null for the tenant id | 15:07 |
| betsy | I’m running a vagrant box with no-auth | 15:07 |
| betsy | But when I make an api call to GET all pools, it comes back empty | 15:07 |
| betsy | It’s trying to match where tenant-id is ‘no-auth’, and sense it’s null, it doesn’t match | 15:08 |
| betsy | I can elevate the context to tenant=all in the call to central, but I was wondering if that was the correct thing to do or not | 15:08 |
| betsy | Because then any admin would get all the pools, but then we only have one now anyway | 15:08 |
| betsy | Does that make sense? | 15:09 |
| betsy | ^since | 15:09 |
| betsy | ^ for the first ‘sense’ not the last one ;) | 15:10 |
| Kiall | Sorry - Was AFK :) | 15:11 |
| betsy | np | 15:11 |
| Kiall | So.. Humm.. I guess the Q is, should normal users be able to see pools that don't belong to them? Even shared/default pools? | 15:12 |
| Kiall | I think we talked about them not being able to see them, but I can't remember 100% TBH. | 15:12 |
| betsy | That’s what I thought. But now as an admin on a no-auth system, I can’t see the default pool | 15:13 |
| betsy | Since the admin (me) didn’t create it and it has a null tenant-id | 15:13 |
| betsy | I’m not sure how to fix that tho, other than to change the all-tenant=True | 15:14 |
| betsy | and not sure if that’s the right thing | 15:14 |
| Kiall | Ah, Okay.. I get you... | 15:16 |
| betsy | Of course, it wouldn’t matter at this time, since we only have one pool and if the admin has permission to view pools, he should be able to see it | 15:17 |
| Kiall | Let me re-read the code that does the tenant filtering to see what the "right" thing to do us.. | 15:17 |
| betsy | Ok. Thanks. | 15:17 |
| betsy | I’ll be glad to open and bug and fix it, just wanted to check with you first on what it shoudl be | 15:18 |
| openstackgerrit | Graham Hayes proposed openstack/designate: Added functionality to allow for zone ownership transfers https://review.openstack.org/107822 | 15:19 |
| Kiall | betsy: so.. I think the right way for this to work is, (explaining in a roundabout way..), as an admin user, if I query for a resource which has a tenant_id field, I should by default see resources where tenant_id = admins_tenant_id AND tenant_id = NULL.. while, as a normal user, I only get to see tenant_id = user_tenant_id | 15:22 |
| Kiall | So.. this method would be updated: https://github.com/openstack/designate/blob/master/designate/sqlalchemy/base.py#L117 | 15:22 |
| *** MasterPiece has joined #openstack-dns | 15:24 | |
| betsy | Ok. The _apply_tenant_criteria? | 15:29 |
| betsy | That makes sense | 15:29 |
| betsy | Okay. I’ll open that bug and make the change | 15:29 |
| betsy | Thanks | 15:29 |
| Kiall | Yea, we can do a policy.check(context, 'admin') in there and change that else block to do the `OR tenant_id == NULL` for admin users | 15:29 |
| mugsie | should the pool no just be created with the managed tenant_id ? | 15:29 |
| mugsie | i thought that was part of the reason we added the managed_tenent_id config car | 15:30 |
| betsy | It’s created during the db migration | 15:30 |
| mugsie | it should have access to the managed_tenant_id var in the migration afaik | 15:30 |
| Kiall | mugsie: I think that's stretching the "managed" concept into something it wasn't really intended for | 15:30 |
| mugsie | not really... | 15:31 |
| Kiall | and - Maybe we do want users to be able to see (but not end) "global" / "shared" resources .. NULL makes much more sense than granting access to some other tenant_id that has lots of other stuff in it. | 15:31 |
| mugsie | it is a managed resource | 15:31 |
| Kiall | Managed resources are resources which we manage on behalf of a tenant.. A shared pool isn't a single tenants | 15:32 |
| mugsie | we dont want to give access to pools directly - (as was dicussed in Seattle) | 15:32 |
| mugsie | it is, its part of the admin / service tenant | 15:32 |
| Kiall | Yea.. That's what I remember too, but the concepts used here should map to all global things.. which may end up with something we DO want to show to users.. | 15:32 |
| Kiall | tenant_id=NULL for something which doesn't belong to a tenant is IMO the correct thing. | 15:33 |
| mugsie | and the managed_tentant_id for something that is managed by the admin is right IMO | 15:33 |
| Kiall | managed_tentant_id isn't "admin managed". It's a resource managed by the system itself on behalf of a tenant. I wrote it, I know what the intention behind it was ;) | 15:34 |
| Kiall | besides.. seeing SELECT * FROM table WHERE tenant_id = UUID_B OR tenant_id = UUID_B; vs SELECT * FROM table WHERE tenant_id = UUID_B OR tenant_id IS NULL; - Which one is clearer? | 15:35 |
| betsy | Sorry, mugsie. I have to side with Kiall on this one. :) | 15:35 |
| openstackgerrit | Merged openstack/designate: Add basic validation functionality to DesignateObjects https://review.openstack.org/127846 | 15:35 |
| mugsie | betsy: fair eniough :) | 15:35 |
| mugsie | has to happen some of the time | 15:36 |
| betsy | :D | 15:36 |
| betsy | So, I’ll open the bug and fix it | 15:36 |
| *** timsim has joined #openstack-dns | 15:37 | |
| Kiall | betsy: Cool :) | 15:37 |
| Kiall | timsim: hows the db indexing testing going BTW? | 15:41 |
| *** paul_glass has joined #openstack-dns | 15:41 | |
| timsim | Kiall: Good, the indices I have up now improved performance massively. Not ideal, but it's a start. Went from falling over around 50k zones to staying alive past 700k zones, 2 mil records. Some of the response times were like five seconds, but no API failures. | 15:42 |
| Kiall | also - betsy - re https://review.openstack.org/#/c/133549/ .. I'm not sure I understand the reasoning for exposing "pool_attributes" as a full-fledged resource, rather than linking it into the standard Pool resource? | 15:43 |
| Kiall | timsim: cool - you think there OK to merge as is? (After a rebase + migration renumber).. If so, I'll have a look in the next few mins.. | 15:44 |
| betsy | Well, started wondering that myself after I did it, but I was thinking I need those calls to migrate the server table to the pool_attributes table | 15:44 |
| *** paul_glass1 has joined #openstack-dns | 15:44 | |
| Kiall | Wouldn't the migration would just use the DB directly? | 15:44 |
| betsy | Well, and to make the v1 api server create call work | 15:45 |
| timsim | Kiall: Yep, I believe so. I'll rebase/up the migration number in a bit. | 15:45 |
| Kiall | For that - Can't we "attach" attributes to the Pool object, like we attach records to the RecordSet object? | 15:45 |
| betsy | hmm. let me rethink that | 15:45 |
| *** EricGonczer_ has joined #openstack-dns | 15:47 | |
| betsy | Well, we do still have records CRUD calls in central to support v1 | 15:47 |
| timsim | betsy: Reading the scroll a bit, somewhat unrelated, but maybe still helpful for the future. If you pass an X-Auth-All-Projects:true header you can see the resource you're looking for, for all tenants: https://github.com/openstack/designate/blob/master/designate/api/middleware.py#L72 | 15:47 |
| Kiall | Yea.. I think it's a tradeoff, in that we end up loading the attributes even if they aren't needed that way.. vs having 500+ lines of extra code and a larger central API.. I'm not convinced I actually have my mind made up on which is better ;) | 15:47 |
| *** paul_glass has quit IRC | 15:48 | |
| Kiall | betsy: the record CRUD calls really need to be removed :) | 15:48 |
| betsy | ok. let me look at my imlementation some more on moving the v1 server call | 15:48 |
| Kiall | (Also .. I'm dying to removing piles of code like the record CRUD, and piles of almost-duplicate per-resource code... Hence everytime I see more, I cry a little inside :P) | 15:49 |
| Kiall | Just not sure what the best way to get rid of most of it is yet ;) | 15:50 |
| betsy | ok, ok. I’ll try to reduce that. ;) | 15:50 |
| betsy | timsim: true, but won’t work in a production env. At least I hope not | 15:54 |
| Kiall | heh - Records APIv1 mostly uses the RRSet methods already.. Bonus ;) | 15:54 |
| Kiall | And, I can see why... get record by ID isn't something you can do via the RRSet interfaces... | 15:56 |
| openstackgerrit | Tim Simmons proposed openstack/designate: Add some helpful SQL indices https://review.openstack.org/129675 | 15:56 |
| Kiall | timsim: Q - Why the index_exists method in ^? | 16:00 |
| Kiall | Also.. Do you have a SQL dump of the largest Designate DB you managed? Save me building one up! | 16:01 |
| Kiall | (I'm assuming it was built with test data...) | 16:01 |
| *** EricGonczer_ has quit IRC | 16:02 | |
| timsim | Yeah I do actually. Somewhere. | 16:02 |
| timsim | The index_exists is there just to make sure that the migration doesn't fail if something has gone awry. Ideally you wouldn't need it, but bad thigns happen if that index already exists. If you try and add one that exists, or drop one that doesn't, or really do anything that isn't in absolute ideal conditions, the whole thign blows up. So I had it there as a safeguard. | 16:04 |
| timsim | Plus, how can you not love: [str(x).split('.')[1] for x in index[1:]] | 16:04 |
| Kiall | :) | 16:06 |
| *** EricGonczer_ has joined #openstack-dns | 16:23 | |
| *** jmcbride has quit IRC | 16:24 | |
| *** jmcbride has joined #openstack-dns | 16:26 | |
| timsim | Kiall: Designate DB dump, 623k zones, 1.87 mil records http://964700e4a3d9dbf5b5ba-7a27b8c5d9fcdc26d383a194ab4f0ebe.r14.cf2.rackcdn.com/designatedump.sql | 16:28 |
| timsim | 888 MB ^ | 16:28 |
| Kiall | Cool, pulling it down now :) | 16:28 |
| Kiall | probably going to take a while to import.. Heh.. | 16:35 |
| timsim | Yeah it took like...2 days to load up via the API, haha. Anything is better than that. | 16:39 |
| *** mwagner_lap has quit IRC | 16:42 | |
| Kiall | timsim: running on a VM or metal? | 16:43 |
| timsim | Many VMs. | 16:43 |
| Kiall | Well, the DB, single or traditional replication, or percona cluster etc? :) | 16:44 |
| timsim | 3-node galera cluster. | 16:44 |
| timsim | Percona xtrabackup. | 16:44 |
| openstackgerrit | Endre Karlson proposed openstack/designate-specs: [designateclient] v1 support for keystone sessions https://review.openstack.org/105033 | 16:46 |
| Kiall | timsim: another Q ... Any tuning of the out of the box mysql/percona settings? | 16:49 |
| timsim | Yep, just a sec | 16:49 |
| timsim | They were 8GB vms. https://gist.github.com/TimSimmons/5b7fa48d95d7ee364d38 | 16:50 |
| timsim | Hold, on that might not be right. | 16:51 |
| timsim | Yeah I think that was right. | 16:52 |
| timsim | It wasn't much, just a bit of extra config. | 16:53 |
| Kiall | K.. Still loading into MySQL anyway ;) | 16:53 |
| timsim | Hah. | 16:53 |
| timsim | I did this at some point, not sure if I ever shared, might be of some interest: https://gist.github.com/TimSimmons/4bc5936d9ad8325d3bd2 | 16:54 |
| Kiall | lol @ the ASCII art | 16:54 |
| timsim | I have no idea why I decided to do that. So dumb, it's not searchable! | 16:55 |
| Kiall | ;) | 16:55 |
| mugsie | timsim: #v2/zones?name=*poo.com* | 16:56 |
| mugsie | nice | 16:56 |
| Kiall | SHOW FULL PROCESSLIST; while importing that data == bad idea. | 16:56 |
| timsim | Always my default test domain. | 16:56 |
| mugsie | :D | 16:57 |
| Kiall | I think the import is maxing out my SSD's write performance.. lol | 16:58 |
| *** jmcbride has quit IRC | 16:59 | |
| timsim | Kiall: How do you feel about adding a column for domains/maybe others with the name field reversed? | 17:01 |
| *** rmoe has joined #openstack-dns | 17:01 | |
| Kiall | We talked about that in the past, as a possible speed up for sub/super domain checking as far as I recall | 17:01 |
| mugsie | yup | 17:02 |
| Kiall | At the time I seem to remember the discussion ending with something like "Let's test it and see" | 17:02 |
| timsim | Yep, that was what I was thinking. I was going to try and test what kind of improvement we get | 17:02 |
| timsim | Cool :P | 17:02 |
| Kiall | Yea, It's worth testing .. I'm expecting that once this data loads (still going...), we'll have some more stuff we can fixup in the DB.. e.g. even with your indexes, many queries are resulting in a filesort... | 17:04 |
| timsim | Oh yeah. There's plenty more to do. | 17:05 |
| timsim | There's probably going to have to be a migration that basically drops all the foreign key relationships, unique key constraints, creates a few more indexes, and adds it all back. But trying to run that in production *shudders* | 17:07 |
| *** jmcbride has joined #openstack-dns | 17:11 | |
| Kiall | Yea, a massive DB cleanup really means taking a downtime hit :/ | 17:11 |
| timsim | Nova has ~70 indexes, not counting Unique Constraints: https://github.com/openstack/nova/blob/master/nova/db/sqlalchemy/models.py :O | 17:17 |
| *** betsy has quit IRC | 17:24 | |
| * Kiall is probably going to regret running the latest migrations against your test DB.. | 17:26 | |
| Kiall | Yep.. #42 is a biggie :( | 17:26 |
| Kiall | Aha.. Luckily you had very few MX or SRV records ;) | 17:27 |
| openstackgerrit | Graham Hayes proposed openstack/designate: Added functionality to allow for zone ownership transfers https://review.openstack.org/107822 | 17:28 |
| *** openstackgerrit has quit IRC | 17:34 | |
| *** openstackgerrit has joined #openstack-dns | 17:34 | |
| *** ChanServ sets mode: +v openstackgerrit | 17:34 | |
| timsim | haha. | 17:34 |
| timsim | Yeah that dump definitely had my indices in there already. I wonder what it'd be like to create them on something that large. | 17:35 |
| *** GonZo2K has joined #openstack-dns | 17:36 | |
| *** jordanP has quit IRC | 17:36 | |
| *** paul_glass1 has quit IRC | 17:39 | |
| *** MasterPiece has quit IRC | 17:40 | |
| Kiall | timsim: BTW.. You mentioned some queries/API calls were taking ages with that dataset, which ones in particular? | 17:42 |
| *** rmoe has quit IRC | 17:43 | |
| Kiall | Fetching all zones for T10 (24k zones) via the API is taking 30seconds, with the vast majority of the time being spent in Python (most likely serializing/deserializing for passing over the wire on rabbit..).. | 17:44 |
| *** rmoe has joined #openstack-dns | 17:45 | |
| *** jmcbride has quit IRC | 17:45 | |
| timsim | I believe the first two queries in this doc were the main culprits: https://gist.github.com/TimSimmons/f573c797ddec1b37d2f1 | 17:47 |
| timsim | According to a DBA, the first one can be fixed by reording the unique key constraint (wat). The second one would be using that reverse name column. | 17:48 |
| Kiall | Interestingly, I'm seeing that query complete in 0.01 sec | 17:49 |
| Kiall | (for a different domain that exists in the dataset) | 17:49 |
| timsim | Probably not that one then. :P | 17:50 |
| timsim | Is that the one with the wildcard? or the inner join? | 17:50 |
| Kiall | :D | 17:50 |
| Kiall | Ah yea, Wildcard one is taking 2.3 sec... | 17:50 |
| timsim | Yeah I think that was the main problem. | 17:50 |
| Kiall | inner join was 0.0.1 | 17:50 |
| Kiall | 0.01* | 17:50 |
| timsim | Maybe I had that one because it got executed so many times or something. idk. | 17:51 |
| timsim | The problem with the wildcard is the left wildcard can't use an index. If we flipped it and used a right, it'd be better supposedlyl. | 17:52 |
| Kiall | I suppose that makes sense :) | 17:52 |
| timsim | I was going to try it out this afternoon and see | 17:52 |
| timsim | If that got improved, and the record patch/put thing got resolved, that'd be a huge performance boost. | 17:53 |
| Kiall | Yea, just adding a reversed name now to test | 17:53 |
| Kiall | record patch/put thing? | 17:54 |
| Kiall | mysql> update domains set rname=REVERSE(name); <-- wonder how long this will take ;) | 17:54 |
| timsim | When you PUT to add a record to a recordset, it deletes all of them and recreates them plus the one you added? | 17:55 |
| Kiall | Uhh.. I'm 99% sure I wrote code to avoid doing that :/ | 17:55 |
| Kiall | crud -_- | 17:55 |
| timsim | I could check again, but I'm pretty sure I saw something do like 80 queries to add a record. | 17:56 |
| mugsie | -_- | 17:56 |
| Kiall | Yea.. I did: https://github.com/openstack/designate/blob/master/designate/storage/impl_sqlalchemy/__init__.py#L398 | 17:56 |
| Kiall | I wonder if that's not being triggered somehow | 17:56 |
| Kiall | OO.. The API layer needs something similar or it won't work :/ | 17:57 |
| timsim | Yeah here are the queries https://gist.github.com/TimSimmons/4bc5936d9ad8325d3bd2#file-queries-sql-L233 | 17:57 |
| * mugsie is listening to The Naked and Famous - Punching in a Dream | 17:59 | |
| mugsie | gah - damn scrollback | 17:59 |
| Kiall | timsim: wow.. http://paste.openstack.org/show/132901/ | 18:00 |
| Kiall | (normal vs reversed) | 18:00 |
| timsim | That's....better. | 18:00 |
| Kiall | timsim: slightly | 18:00 |
| Kiall | Probably not enough to be worth it though ;) | 18:00 |
| mugsie | 2.36 vs 0 ? | 18:01 |
| mugsie | thats definitly worth it ;) | 18:01 |
| Kiall | Yea, it's only a wee difference ;) | 18:01 |
| timsim | I'll say | 18:01 |
| *** EricGonc_ has joined #openstack-dns | 18:02 | |
| Kiall | the actual col + index etc I added.. just for reference .. http://paste.openstack.org/show/132902/ | 18:02 |
| *** EricGonczer_ has quit IRC | 18:02 | |
| Kiall | So.. I'd say adding that is a yes ;) But, ideally we hide it inside the SQLAlchemy plugin (it's a very SQL optimization after all) | 18:03 |
| mugsie | yeah, I would agree | 18:03 |
| Kiall | (and.. we can't really use rname as the column name, since that has special meaning in DNS and will be confusing :P) | 18:03 |
| *** openstackgerrit has quit IRC | 18:03 | |
| *** openstackgerrit has joined #openstack-dns | 18:04 | |
| *** ChanServ sets mode: +v openstackgerrit | 18:04 | |
| timsim | Yeah that'd be best | 18:04 |
| mugsie | it also would not nessisarly be relevent in another DB | 18:04 |
| timsim | Postgres wouldn't benefit? | 18:05 |
| mugsie | it would, but mongo might not | 18:05 |
| mugsie | or couchDB | 18:05 |
| * mugsie gets sick think of anyone trying to run this on couchdb | 18:06 | |
| mugsie | thinking* | 18:06 |
| timsim | True, but they wouldn't be using SQLA for that? | 18:06 |
| Kiall | timsim: yea, hence hiding it inside SQLA :) | 18:06 |
| Kiall | inside the SQLA driver | 18:06 |
| timsim | Ok, I've caught up now | 18:06 |
| Kiall | timsim: any chance you created some zones with a massive # of records/recordsets? | 18:06 |
| timsim | Unfortunately not. I didn't want to with that bug that I thought was a known issue :P Vinod and I talked about it, probalby should have those discussions with everyone | 18:07 |
| Kiall | This the "mDNS makes 4.5 billion queries to sever an AXFR" issue? | 18:08 |
| Kiall | (I was hoping to experiment with that exact "bug" ;)) | 18:08 |
| timsim | Nah the records add/delete bit | 18:08 |
| timsim | I'm actually not sure what you're talking about on that one^ | 18:10 |
| mugsie | mDNS does quite a few calls to get the data for an AFXR | 18:11 |
| Kiall | Yea.. Try adding 100 RRSets to a zone, then do an AXFR and watch mdns do 101 or so DB queries ;) | 18:12 |
| Kiall | 102 actually I think.. | 18:12 |
| Kiall | Easy enough fix, just hasn't been done yet | 18:12 |
| timsim | Oop. I haven't tried that. | 18:12 |
| *** jmcbride has joined #openstack-dns | 18:14 | |
| timsim | Our initial test was just Create a ton, add some recordsets. Test the end to end time from API->MiniDNS. | 18:15 |
| timsim | Actually, API->Bind9 via MiniDNS | 18:15 |
| Kiall | http://paste.openstack.org/show/132904/ <-- Happy with that.. Pulling 5k zones down == 5 sec, that's a fairly reasonable time for a call of that size.. 100 zones taking .2 sec | 18:16 |
| timsim | Yeah that isn't bad at all. Did you do anything to optimze that? | 18:18 |
| Kiall | Nope... Designate etc running in a VirtualBox VM.. MySQL running on desktop with SSD with innodb_buffer_pool_size = 12G / key_buffer = 1024M | 18:20 |
| Kiall | (stock mysql.. no cluser / slaves etc | 18:20 |
| *** mwagner_lap has joined #openstack-dns | 18:20 | |
| timsim | Yeah. I haven't had any issue with those gets. If we're going to add the reverse_name column, that should improve filtering performance too. | 18:21 |
| timsim | Kiall: So I'll go ahead and work on that change, unless you want to? | 18:23 |
| Kiall | Go ahead.. :) | 18:23 |
| eandersson | Morning. :) | 18:26 |
| mugsie | eandersson: hey | 18:27 |
| timsim | hey eandersson | 18:28 |
| *** amcrn has joined #openstack-dns | 18:30 | |
| eandersson | I was busy trying to figure out the threading issue in central yesterday. | 18:30 |
| Kiall | timsim: found an issue with your MySQL tuning ;) | 18:30 |
| timsim | Not surpised :P what'cha got? | 18:30 |
| timsim | God I can't speel today. | 18:31 |
| eandersson | hehe | 18:31 |
| Kiall | http://paste.openstack.org/show/132905/ You had your innodb_buffer_pool_size set to 3GB, while that dataset really needs 4.5+GB | 18:31 |
| mugsie | eandersson: which threading issue? I *thought* we got all of them | 18:32 |
| timsim | Ah, figures. Performance did suffer a bit when it got real big. The configurations I put in for everything were intended as a proof of concept before our new DevOp came in. We ended up trying to muder it :P | 18:33 |
| eandersson | I can't figure out why _increment_domain_serial is causing a database deadlock for us. | 18:33 |
| eandersson | https://github.com/openstack/designate/blob/stable/juno/designate/central/service.py#L293 | 18:33 |
| mugsie | eh.... this is sounding familiar... | 18:34 |
| eandersson | I wrote a big bug report with lots of logs before I left Europe, but launchpad decided to error out. | 18:35 |
| eandersson | and never actually published it | 18:35 |
| mugsie | -_- | 18:35 |
| mugsie | ah, launchpad | 18:35 |
| eandersson | OperationalError: (OperationalError) (1205, 'Lock wait timeout exceeded; try restarting transaction') 'UPDATE domains SET updated_at=%s, serial=%s WHERE domains.id = %s AND domains.deleted = %s' (datetime.datetime(2014, 10, 29, 22, 2, 26, 129759), 1414620146, 'bcaea39f389d4336b60de0adcd4513a2', '0') | 18:36 |
| eandersson | TRX HAS BEEN WAITING 38 SEC FOR THIS LOCK TO BE GRANTED: | 18:36 |
| eandersson | At first I thought it could have been related to my Sink changes, but I saw the same thing when simply using the Designate Client to create new records. | 18:36 |
| *** paul_glass has joined #openstack-dns | 18:39 | |
| mugsie | yeah, we found some DB indexing issues at scale recently, but I am not sure if they got back ported to juno | 18:39 |
| eandersson | How does threading work in central? | 18:39 |
| eandersson | I think I added a stupid fix for now, by wrapping that call in a lock :P | 18:40 |
| eandersson | I am going to set up a new dev envrionment today, so that I can troubleshoot it properly. | 18:41 |
| *** GonZo2K has quit IRC | 18:41 | |
| *** paul_glass has quit IRC | 18:42 | |
| *** paul_glass has joined #openstack-dns | 18:43 | |
| Kiall | eandersson: by any chance did you save all that detail? | 18:45 |
| Kiall | Also, were you using percona cluster or galera etc? (I remember asking, but can't remember the answer) | 18:47 |
| *** openstackgerrit has quit IRC | 18:49 | |
| *** openstackgerrit has joined #openstack-dns | 18:49 | |
| *** ChanServ sets mode: +v openstackgerrit | 18:49 | |
| eandersson | Nah, we are pretty much using the same setup as with the devstack. | 18:50 |
| eandersson | for Designate | 18:50 |
| eandersson | PowerDNS with a MySQL DB | 18:50 |
| eandersson | I think the reason this hasn't been discovered is because it requires a script to start multiple instances at the same time | 18:51 |
| eandersson | I discovered it when one of our Devs ran a script to start 7 servers at the same time. | 18:51 |
| Kiall | Well, we've done testing around creating multiple records in a single zone at once (which should be the same I guess), did you manage to save the SHOW INNODB ENGINE STATUS; output? | 18:52 |
| eandersson | I did, but lets see if I can find it. | 18:52 |
| eandersson | I posted it on the pastbin. | 18:52 |
| eandersson | I think I sent the link to you as a private message a few weeks ago. | 18:52 |
| Kiall | I've no PM history with you, so must have been in the room. The room is logged, any chance you remember the date? ;) | 18:53 |
| eandersson | http://paste.openstack.org/show/07TzeLYlhUbKeEeQPDOg/ | 18:53 |
| eandersson | I found some saved logs in my mailbox. :P | 18:54 |
| eandersson | I can't remember unfortuantely. | 18:54 |
| eandersson | That is from when I reproduced it on my dev machine. | 18:54 |
| Kiall | Humm - That only seem to be showing 1 of the TX's .. The second TX it's conflicting with would be useful :( | 18:56 |
| eandersson | I don't think I kept the logs on my laptop. I am in the US now, so I don't have access to my PC unfortunately. | 18:58 |
| eandersson | http://paste.openstack.org/show/132907/ | 18:58 |
| eandersson | I don't think that contains what you need unfortuantely. | 18:58 |
| eandersson | I am working on setting up a new dev environment, so I can probably get you those logs for tomorrow. | 18:59 |
| Kiall | Just dialling into a call, back in 30 | 19:01 |
| *** amcrn_ has joined #openstack-dns | 19:01 | |
| *** amcrn has quit IRC | 19:04 | |
| *** amcrn_ is now known as amcrn | 19:04 | |
| eandersson | ok reproduced it | 19:17 |
| eandersson | This is how you re-produce the issue | 19:18 |
| eandersson | http://paste.openstack.org/show/132921/ | 19:18 |
| *** openstackgerrit has quit IRC | 19:18 | |
| *** openstackgerrit has joined #openstack-dns | 19:18 | |
| *** ChanServ sets mode: +v openstackgerrit | 19:18 | |
| eandersson | Basically really shitty example. :P | 19:19 |
| *** paul_glass1 has joined #openstack-dns | 19:22 | |
| *** paul_glass1 has quit IRC | 19:23 | |
| *** paul_glass has quit IRC | 19:23 | |
| Kiall | eandersson: just off the call, give me 5 and I'll see if I hit it too with that | 19:24 |
| *** paul_glass has joined #openstack-dns | 19:26 | |
| Kiall | eandersson: my test DB is well.. HUGE.. right now.. so it's falling over for other reasons right now ;) | 19:39 |
| eandersson | We are seeing issues with both updating the domain and insert into recordset | 19:43 |
| ekarlso | eandersson: what errors ? | 19:47 |
| Kiall | I'm not sure if my dev env is broke, or if your script is reproducing something much worse than a deadlock.. I can't even connect to mysql right now ;) | 19:47 |
| Kiall | ekarlso: lots of scrollback with detail | 19:47 |
| eandersson | yep | 19:47 |
| eandersson | that is what I was seeing on my dev instance as well | 19:47 |
| Kiall | eandersson: opening the mysql client is failing for you too? | 19:47 |
| Kiall | That should never happen really... | 19:47 |
| eandersson | not in this enviornment | 19:48 |
| eandersson | but it was eating all the resources on my dev environment | 19:48 |
| eandersson | so I couldn't do anything related to the MySQL DB | 19:48 |
| ekarlso | deadlock issues ? sorry it looks like TLDR | 19:49 |
| timsim | ekarlso: Yep. In the Juno release. | 19:50 |
| eandersson | ekarlso: I am basically seeing my Deisgnate DB completely locked up when multiple create record are called at the same time. | 19:50 |
| Kiall | Okay - My env is broke.. a single insert seems to be failing | 19:51 |
| eandersson | Some additional logs if it helps | 19:52 |
| eandersson | http://paste.openstack.org/show/132925/ | 19:52 |
| eandersson | We can only see one query though. So not sure what is locking. | 19:53 |
| Kiall | Into a table with 3 records, an insert is taking me like 45 seconds! Let me fix this up, then I can re-test with out script :) | 19:54 |
| Kiall | your* | 19:54 |
| Kiall | INSERT into* | 19:55 |
| Kiall | can't type at all today -_- | 19:55 |
| eandersson | I am still so jetlagged :p | 19:56 |
| eandersson | hate traveling | 19:56 |
| Kiall | Oh.. Can't read either it seems.. was right the first time.. | 19:56 |
| *** openstackgerrit has quit IRC | 20:04 | |
| *** openstackgerrit has joined #openstack-dns | 20:04 | |
| *** ChanServ sets mode: +v openstackgerrit | 20:05 | |
| Kiall | eandersson: and a rebuild has done the same. I'm really really stumped to be honest. I've not see behaviour like this before | 20:06 |
| eandersson | I think that it only happens once you go into a more production like environment. | 20:07 |
| eandersson | Since it is usually only triggered by DevOps type scripts that create multiple instances. | 20:08 |
| Kiall | We've done lots and lots of testing for that kinda stuff | 20:08 |
| Kiall | eandersson: think I have it.. | 20:10 |
| Kiall | So, the default mysql driver is the C mysql driver.. it can't be monkeypatched by eventlet, so doesn't work async.. | 20:11 |
| eandersson | ah wow | 20:11 |
| Kiall | when the first one opens a TX, eventlet moves over to another "greenthread", which tries to use the DB, and blocks. | 20:11 |
| Kiall | That's why we only see 1 query running | 20:11 |
| Kiall | but 2 TX's open | 20:12 |
| Kiall | From memory, oslo.db (the openstack db lib) works around this.. But.. Clearly it isn't | 20:12 |
| ekarlso | oslo.db bug or smth+ | 20:13 |
| Kiall | ekarlso: possibly... | 20:14 |
| Kiall | I've gotta run, but I've got enough info to debug this properly tomorrow. | 20:17 |
| ekarlso | how to reproduce ? | 20:17 |
| ekarlso | Kiall: beer time ? :P | 20:18 |
| eandersson | I posted a terrible example :P http://paste.openstack.org/show/132921/ | 20:18 |
| Kiall | yep | 20:18 |
| ekarlso | Kiall: -,,- | 20:18 |
| eandersson | Is there a way to limit central to one thread? | 20:18 |
| timsim | ekarlso: http://paste.openstack.org/show/132921/ | 20:18 |
| ekarlso | hmm, 1 worker u mean eandersson ? | 20:18 |
| eandersson | Yea | 20:18 |
| ekarlso | workers=1 | 20:18 |
| eandersson | So that this bug cant be triggered | 20:18 |
| eandersson | ah that simple | 20:18 |
| eandersson | awesome | 20:19 |
| ekarlso | think so | 20:19 |
| eandersson | although that didnt work on the sink afaik | 20:19 |
| eandersson | I'll try it out though | 20:19 |
| timsim | Workers would go under servce:central like this: https://github.com/openstack/designate/blob/master/etc/designate/designate.conf.sample#L126 | 20:20 |
| ekarlso | eandersson: tested on kilo btw? | 20:20 |
| eandersson | Yea. Actually my initial logs were from the master. | 20:20 |
| eandersson | timsim: Unless I am wrong, the notifications/requests are still handled async, even with one worker. | 20:22 |
| eandersson | At least that is what I was seeing with the sink. | 20:22 |
| timsim | Yeah you're right, I wasn't sure that'd actually work. | 20:23 |
| eandersson | My initial fix for this was to add a thread lock in the Sink. Which works. | 20:23 |
| timsim | There might be a way to only let it do one request at a time. | 20:23 |
| eandersson | but the problem is that it wont prevent API requests from causing this. | 20:23 |
| timsim | Yeah that's an issue for sure. | 20:24 |
| eandersson | Yea, that would be perfect. Performance is the least of my concerns for DNS. | 20:24 |
| eandersson | (at least on the management level) | 20:24 |
| ekarlso | what's causing the issue ? | 20:29 |
| eandersson | You mean in our environment? | 20:32 |
| ekarlso | just trying to understand the issue :P | 20:33 |
| eandersson | Basically whenever you try to create 3+ instances (DNS records) at the same time | 20:33 |
| ekarlso | it locks up ? | 20:33 |
| eandersson | yep | 20:33 |
| eandersson | It opens two TX's, but only one query | 20:33 |
| ekarlso | wonder why that happens : | 20:34 |
| eandersson | <Kiall> So, the default mysql driver is the C mysql driver.. it can't be monkeypatched by eventlet, so doesn't work async.. | 20:34 |
| eandersson | Sounds like that is the cause. | 20:34 |
| ekarlso | yeah, that seems to be hanging here too :| | 20:36 |
| ekarlso | is it only on rrset ? | 20:37 |
| ekarlso | or domains too ? | 20:37 |
| eandersson | I haven't tried tbh, but recordsets are much more likely, as it would be rare to create multiple domains that fast. | 20:37 |
| ekarlso | hmm, eandersson we had some deadlock issues earlier this year... | 20:39 |
| ekarlso | eandersson: what versions have you tried ? | 20:42 |
| eandersson | 2014.2 | 20:45 |
| eandersson | and master (the week before the summit) | 20:45 |
| eandersson | I did not see this in 2014.1 | 20:46 |
| ekarlso | but also I see it's doing something stupid it shouldn't need to in master I think | 20:55 |
| ekarlso | or nvm | 20:56 |
| *** EricGonc_ has quit IRC | 21:09 | |
| *** EricGonczer_ has joined #openstack-dns | 21:11 | |
| *** openstackgerrit has quit IRC | 21:19 | |
| *** openstackgerrit has joined #openstack-dns | 21:19 | |
| *** ChanServ sets mode: +v openstackgerrit | 21:19 | |
| *** nkinder has quit IRC | 21:56 | |
| eandersson | So anyone has a clue how to limit Central to one record at a time? :D | 22:07 |
| timsim | Maybe rpc_thread_pool_size=1 http://docs.openstack.org/developer/designate/configuration.html or change this to one https://github.com/openstack/designate/blob/b254e98c78f2cdd0f0f038b22fbed985bd7a4bc0/designate/service.py#L41 | 22:12 |
| timsim | eandersson^ major hack suggestions, lol. | 22:12 |
| timsim | That assumes one thread=one concurrent request. *shrugs* | 22:13 |
| *** ryanpetrello has quit IRC | 22:16 | |
| eandersson | Yea. It's pretty much what I was going for. | 22:19 |
| eandersson | My current solution was to wrap everything in a lock :P | 22:19 |
| *** GonZo2K has joined #openstack-dns | 22:19 | |
| timsim | Yeah, my guess is that Kiall will get a fix up tomorrow ;) | 22:22 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/designate: Updated from global requirements https://review.openstack.org/134382 | 22:27 |
| openstackgerrit | Tim Simmons proposed openstack/designate: WIP: Add a reverse name column to the domains table https://review.openstack.org/134387 | 22:36 |
| *** timsim has quit IRC | 22:40 | |
| *** paul_glass has quit IRC | 22:41 | |
| *** jmcbride has quit IRC | 22:49 | |
| *** boris-42 has joined #openstack-dns | 23:16 | |
| *** nkinder has joined #openstack-dns | 23:17 | |
| *** EricGonczer_ has quit IRC | 23:20 | |
| Kiall | eandersson / ekarlso: to limit central to 1 at a time, you can limit the thread pool size to 1... BUT.. Increasing the # of workers would also resolve (part of) the issue.. The part that's not resolved is, we have generally "ignored" the deadlock situation around the same zone being updated concurrently (since the serial # has to be updated in a single row) because.. Frankly, it virtually never happens. (2 years running HP Cloud DNS, we've seen | 23:27 |
| Kiall | about 4 deadlocks). The fix for this issue, as far as designate is concerned anyway, is to retry operations on deadlocks - which is basically what the DB expects you to do. Beyond that, we're limited by eventlet + MySQL-Python.... Deployers can choose to either use PyMySQL (a pure python mysql implemention, or, they can add `use_tpool = True` to the [database] config section... Either should resolve the issue. | 23:27 |
| Kiall | wow.. longer than I thought that message was | 23:27 |
| Kiall | (also - re "virtually never happens".. I mean has never really happened for our use case @ HP ;)) | 23:28 |
| Kiall | ccccccdugckefldltecuuncdltcfdhguvbuninirudvb | 23:30 |
| boris-42 | Kiall: ekarlso hi guys | 23:31 |
| boris-42 | Kiall ekarlso here is the patch that adds non voting rally job to designate https://review.openstack.org/#/c/134392/ | 23:31 |
| boris-42 | Kiall: ekarlso I am going to make patch in designate as well=) | 23:32 |
| ekarlso | boris-42: coolio | 23:33 |
| ekarlso | Kiall: stop drinking beer and fitch the deadlock issue :P | 23:34 |
| boris-42 | ekarlso: LOL | 23:34 |
| Kiall | boris-42: nice :) I was going to add the same job to Designate after I saw you had it on rally | 23:34 |
| boris-42 | Kiall: the only thing is that we have some issues with designate in Rally | 23:34 |
| Kiall | ekarlso: lol.. I'm Irish, when have you ever known an Irishman to not drink beer? | 23:34 |
| boris-42 | Kiall: for some reason python client stopped workign | 23:34 |
| ekarlso | that's my fault :P | 23:34 |
| Kiall | lol.. ekarlso what did you do? -_- | 23:35 |
| boris-42 | Kiall: oh I need some business trip to Irish | 23:35 |
| boris-42 | Kiall: =) | 23:35 |
| boris-42 | Kiall: in russian only vodka only hardcore | 23:35 |
| ekarlso | Kiall: the whole constructing the client with a token + ep without a auth_url causes it to try discovery and goes bork no url thingie | 23:35 |
| boris-42 | =) | 23:35 |
| Kiall | boris-42: Russian right? | 23:35 |
| boris-42 | Kiall: lol | 23:35 |
| boris-42 | Kiall: actually no=) | 23:35 |
| Kiall | heh.. well, our vodka isn't as good ;) | 23:35 |
| boris-42 | Kiall: I just live here for 15 years=) | 23:35 |
| ekarlso | Kiall: u dont want competitors like boris-42, you'll be under the table :P | 23:36 |
| boris-42 | ekarlso: lol | 23:36 |
| boris-42 | ekarlso: ya something like that=) | 23:36 |
| Kiall | mugsie will take him ;) | 23:36 |
| ekarlso | Kiall: dont forget russians are fed vodka as 40% of their breast feeding -,,- | 23:36 |
| ekarlso | at least :P | 23:36 |
| Kiall | Anyway.. Happy to add the job non-voting, fix the issues, and go from there.. the rally-jobs template though.. I assume that has lots of different jobs? | 23:37 |
| Kiall | ekarlso: too far BTW ;) | 23:37 |
| ekarlso | Kiall: yeah, that one was a little off :p | 23:37 |
| ekarlso | anyways, i'll take a look at the rally + designateclient situation tmrw boris-42 | 23:38 |
| ekarlso | should be a simple fix :) | 23:38 |
| Kiall | Ah.. only 1 is added to the layout, so not an issue if theres more than just designate jobs.. | 23:38 |
| boris-42 | Kiall: ?) | 23:38 |
| boris-42 | Kiall: we can add any amount of jobs to any of projects | 23:38 |
| boris-42 | Kiall: i just don't see for now case | 23:39 |
| boris-42 | Kiall: to do that | 23:39 |
| Kiall | Yea, But I wouldn't want gate-rally-trove or similar running ;) But.. it or similar won't since it's an explicit job name in the zuul layout.. (again, assumes rally-jobs has more than gate-rallt-{project} in it...) | 23:40 |
| Kiall | (in other words, I should have read more before commenting...) | 23:40 |
| boris-42 | Kiall: ahhh | 23:41 |
| boris-42 | Kiall: so lemme try to expalin | 23:41 |
| Kiall | No need, I just misread :) | 23:42 |
| boris-42 | Kiall: we have few differnet job templates | 23:42 |
| boris-42 | Kiall: (for designate, for zaqar, for neutron, and just for nova-netowrk) | 23:42 |
| boris-42 | Kiall: they just have different configured devstack-gate to install required stuff | 23:42 |
| boris-42 | Kiall: so in your case we are adding rally job that is running agaisnt openstack with designate) | 23:43 |
| Kiall | Yep, I see it now :) It originally looked to me like all the $project+rally jobs, but that was just misreading :) | 23:43 |
| boris-42 | Kiall: so basically you can write plugins for Rally inside designate project source | 23:45 |
| Kiall | I am curious though, about what you consider a fail for when it comes to making it voting... Gating on performance - when dealing with cloud instances from different providers like we do - seems unlikely. You only fail the job when the APIs etc explode? | 23:45 |
| ekarlso | boris-42: an evil way of just "fixing" it for now is to pass auth_url down to it | 23:45 |
| eandersson | Kiall: So the reason we didn't see this in 2014.1 is probably because we had multiples Nodes running. | 23:45 |
| boris-42 | ekarlso: why evil?) | 23:45 |
| ekarlso | boris-42: ideally though you'd have Rally support keystone sessions | 23:45 |
| Kiall | ekarlso: or you fix the code ;) | 23:45 |
| ekarlso | Kiall: pffftm it's not my fault, it's ksclient :P | 23:46 |
| Kiall | If it worked a week ago.. It needs to work today. | 23:46 |
| eandersson | We were only running a single instance with one worker for testing purposes, before we went live. | 23:46 |
| boris-42 | Kiall: about different VM from nodepool | 23:46 |
| boris-42 | Kiall: first of all you can bigger values in criteria of success | 23:46 |
| boris-42 | Kiall: like avg duration < 2 * AVG | 23:46 |
| ekarlso | Kiall: well I think the generic.Token stuff would work, obviously it doesn't work like before :( | 23:46 |
| boris-42 | Kiall: that will catch terrible changes. But won't be super preciese | 23:47 |
| ekarlso | Kiall: though I wonder if I could just "hack it" and just pass a v2.Token vs generic.Token since I guess the token payload is the same *goes to test* | 23:47 |
| boris-42 | Kiall: one of the topics that we are going to work is normalization of Rally results | 23:47 |
| Kiall | ekarlso: but it needs to, it's a bug if it doesn't sadly! The client is an API - rally, horizon (at least in HP), and $random peoples code depened on that API.. If if's fundamentally broken and unfixable, we need to back out the recent changes. | 23:47 |
| boris-42 | Kiall: soo you'll multiple all values on some magic number and it will normalize results | 23:48 |
| boris-42 | Kiall: but what I would like to say regression testing is just small part of what you can do with rally in gates.. | 23:49 |
| Kiall | boris-42: Okay, makes sense.. +/-50% is an unlikely to occur thing on a single change purely due to the speed of the slave.. :) | 23:49 |
| ekarlso | Kiall: call me doh, I think i've used the wrong kind of cred -,,- | 23:49 |
| boris-42 | Kiall: like somebody is saying that his patch improves performance => you can easily check that in gates. by running N times recheck | 23:49 |
| boris-42 | Kiall: ya dsvm is run against really different VMs but we will find the way to normalize that=) | 23:50 |
| Kiall | boris-42: absolutely! Having the perf info on hand is the main reason I'd want the rally gate, i.e. not necessarily as something which -1's expect in extreme cases. | 23:50 |
| boris-42 | Kiall: so as well you'll get profiling in gates | 23:50 |
| Kiall | also.. Having another client test out code means ekarlso's compatibility breaks won't happen as often ;) | 23:50 |
| boris-42 | Kiall: when we finish some very last stuff in rally and add it to designate) | 23:51 |
| boris-42 | Kiall: cause getting such traces under load http://boris-42.github.io/ngk.html makes totally sense | 23:51 |
| ekarlso | Kiall: sshhhh | 23:51 |
| ekarlso | someone has to do clientwork ! :P | 23:51 |
| boris-42 | Kiall: e.g. if person is saying that he is fixing some specific part of code, you can just measure it separatly =) | 23:52 |
| boris-42 | Kiall: and then you don't care about absolute values of whole iteration duration=0 | 23:52 |
| Kiall | boris-42: http://boris-42.github.io/ngk.html, that looks like profiling output? tied to os-profiler? | 23:52 |
| boris-42 | Kiall: it's osprofiler output | 23:52 |
| boris-42 | Kiall: bootving VM from nova cli | 23:53 |
| * Kiall loves that kinda info.. The more we have, the better :) | 23:53 | |
| boris-42 | Kiall: yaa | 23:53 |
| boris-42 | Kiall: when I started rally it was the goal to automate profiling + load and reports and gates=) | 23:54 |
| boris-42 | Kiall: so in Kilo it will happen lol=) | 23:54 |
| Kiall | boris-42: I bet when you started you didn't know just how much of a performance difference there was between rackspace and hpcloud, the two providers of of the gate ;) (Let's not get into the details of which one is worse :P) | 23:55 |
| boris-42 | Kiall: I know that ... | 23:55 |
| boris-42 | knew* | 23:55 |
| Kiall | :) | 23:55 |
| boris-42 | Kiall: but it's just one of millions task that have to be resolved | 23:55 |
| ekarlso | Kiall: or dreamhost for that matter :p | 23:55 |
| Kiall | ekarlso: none of the gate runs on dreamhost ;) | 23:56 |
| Kiall | (Unless that changed recently) | 23:56 |
| boris-42 | Kiall: btw we are going to present some basic historical perfomrnace data for all projects | 23:56 |
| ekarlso | :P | 23:56 |
| boris-42 | Kiall: like we will run on same hardware islotaed job | 23:56 |
| boris-42 | Kiall: every day from master deployed openstack | 23:56 |
| boris-42 | Kiall: finally I got servers lol | 23:56 |
| Kiall | boris-42: Thats actually really interesting - Having identical hardware configs for N years of perf testing gives everyone a really good baseline vs .. well.. cloud. | 23:57 |
| boris-42 | Kiall: yep but as well is just 1 small case=) | 23:57 |
| Kiall | 1 consistent case is better than zero, or 100s of inconsistent ;) | 23:58 |
| boris-42 | Kiall: btw did you try to run Rally locally?) | 23:58 |
| boris-42 | Kiall: so we already have a lot =) | 23:58 |
| ekarlso | Kiall: crap | 23:58 |
| Kiall | Yep, ekarlso wrote the designate pieces and we started using it a bit.. We're still a little stuck on JMeter though due to .. well.. it works.. | 23:59 |
| ekarlso | Kiall: so when using token_endpoint.Token ie the thing that does no discovery and all that crap it doesn't even do discovery to discover which version endpoint of designate it should use :p | 23:59 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!