Wednesday, 2014-01-08

« Tuesday, 2014-01-07 Index Thursday, 2014-01-09 »
*** sballe has quit IRC00:15
*** sballe has joined #openstack-dns00:15
*** dguerri has quit IRC00:28
*** dguerri has joined #openstack-dns00:34
*** dguerri has quit IRC00:43
openstackgerritEndre Karlson proposed a change to stackforge/designate: Part 1 of FloatingIP PTR record functionality  https://review.openstack.org/5831600:44
*** nosnos has joined #openstack-dns00:53
*** shawni has joined #openstack-dns01:02
*** shawni has quit IRC01:06
*** rossk has quit IRC01:14
*** rossk has joined #openstack-dns01:15
*** rossk has quit IRC01:15
*** rossk has joined #openstack-dns01:16
*** rossk has quit IRC01:29
*** rossk has joined #openstack-dns01:29
*** CaptTofu has quit IRC01:34
*** CaptTofu has joined #openstack-dns01:34
*** CaptTofu has quit IRC01:35
*** CaptTofu has joined #openstack-dns01:35
*** CaptTofu has quit IRC01:39
*** nosnos has quit IRC01:43
*** nosnos has joined #openstack-dns01:44
*** rossk has quit IRC01:46
*** msisk_ has joined #openstack-dns02:02
*** msisk_ has quit IRC02:04
*** nosnos has quit IRC02:43
*** eankutse has joined #openstack-dns02:57
*** rektide_ is now known as rektide03:10
*** vipul is now known as vipul-away03:48
*** vipul-away is now known as vipul03:52
*** msisk has quit IRC03:55
*** eankutse has quit IRC04:45
*** sballe_ has joined #openstack-dns05:23
*** sballe has quit IRC05:26
*** jmcbride has joined #openstack-dns05:29
*** jmcbride has quit IRC05:33
*** amrit has quit IRC06:18
*** zigo_ is now known as zigo06:46
*** pavelk has joined #openstack-dns08:14
*** CaptTofu has joined #openstack-dns09:17
*** CaptTofu has quit IRC09:19
*** CaptTofu has joined #openstack-dns09:19
*** CaptTofu has quit IRC09:24
kiallartom: humm.. Looks like that index isn't missing.. at least with MySQL..10:21
kiallI think this only affects SQLite .. Since SQLite doesn't natively support ALTER TABLE..10:25
kiallSo, SQLA actually creates a new table, copies data, deletes the old, renames the new10:25
kiallWow - Test run on my laptop is taking AGES..10:27
kiallThis can't be right ;)10:30
kiall13.5s before vs 105.334s after the patchset is applied10:31
openstackgerritKiall Mac Innes proposed a change to stackforge/designate: Ensure SQLite has the correct unique_record index  https://review.openstack.org/6544610:37
ekarlsoartom: you here ?10:47
*** CaptTofu has joined #openstack-dns11:55
ekarlsotror /win 4911:56
kiallekarlso: ?11:57
ekarlsowrong window :)11:57
*** artom_ has joined #openstack-dns12:43
artom_kiall, I'll have to make sure it only migrates the database once for every repo...12:44
artom_ekarlso, connected from home :)12:44
*** sballe_ has quit IRC12:45
*** sballe_ has joined #openstack-dns12:45
artom_kiall, the SSD makes it hard to judge runnning time ;)12:46
kiall;)12:47
*** rjrjr has joined #openstack-dns13:05
*** sballe_ has quit IRC13:06
*** artom_ has quit IRC13:19
*** jmcbride has joined #openstack-dns13:26
*** jmcbride has quit IRC13:33
*** CaptTofu has quit IRC13:36
*** CaptTofu has joined #openstack-dns13:37
*** CaptTofu has quit IRC13:37
*** eankutse has joined #openstack-dns14:10
*** eankutse has quit IRC14:23
*** eankutse has joined #openstack-dns14:23
artomkiall, I suspect the long runtime is because, while it only upgrades the db once per process, many processes are spawned.14:38
artomI added some logging with PID to a file and that's what comes out...14:39
kiallPossibly.. I'll check with a concurrency of 114:39
*** jmcbride has joined #openstack-dns15:02
*** betsy has joined #openstack-dns15:18
*** tsimmons has joined #openstack-dns15:29
*** CaptTofu has joined #openstack-dns15:41
openstackgerritVinod Mangalpally proposed a change to stackforge/designate: Add APIs for managing TLDs  https://review.openstack.org/6548815:47
tsimmonskiall: I was looking at doing some work on https://blueprints.launchpad.net/designate/+spec/bulk-modify-delete-zones (bulk modify/delete zones). Do you see any issue off the top of your head with me starting work on that?15:51
*** sballe has joined #openstack-dns15:51
kiallHey tsimmons15:52
*** sballe has quit IRC15:52
kiallSo, yes/no .. I think we need to start being transactional in how we interact with the designate DB for reliable bulk actions..15:53
kiallBut, I'm not sure how best to get there :)15:53
tsimmonsHm. Would there be some way to confirm at the time of this particular action that it is viable and if not roll it back?15:55
kiallWell, the difficulty is handling that concurrently on all the designate-central's .. So I reckon we'll have to rely on DB transactions for it to really work15:56
kiallAnyway - It's defiantly something we should be doing .. A good start would be updating storage/api.py to use TX's rather than manual rollbacks, then extending those to support batch actions..15:58
kiallThat still leaves issues with powerdns db / bind zone files etc.. But maybe if a TX fails, we resync the whole zone?15:59
tsimmonsSo if another designate central were to try to access the items that were being bulk modified by another? Is that the concurrency you're talking about?16:00
artomkiall, any results with concurrency=1?16:01
kialltsimmons: yea, the risk of conflicting API calls is much higher when updating 100 records etc16:02
kiallartom: haven't had a chance yet16:02
*** msisk has joined #openstack-dns16:02
artomI could move the one-db-per-repo check out of process, to a file most likely...16:03
kiallNo, we do actually need to create 1 per testr process, but.. It "feels" like it's doing much more than that16:05
artomWe do?16:06
*** CaptTofu has quit IRC16:06
*** CaptTofu has joined #openstack-dns16:07
*** CaptTofu has quit IRC16:09
*** CaptTofu has joined #openstack-dns16:09
*** vinod1 has joined #openstack-dns16:12
tsimmonsSo instead of doing a storage.delete_domain for example. Would you want to some sort of more manual database action?16:12
*** eankutse has quit IRC16:16
*** eankutse has joined #openstack-dns16:16
kialltsimmons: sorry, was AFK16:16
kialltsimmons: so, for example .. https://github.com/stackforge/designate/blob/master/designate/storage/api.py#L73-9016:17
kiallWe should be replacing the manual rollbacks there with proper transactions, then, extending storage to support batch actions..16:17
*** msisk has quit IRC16:17
tsimmonsOk, so in general, what would that sort of look like?16:20
kialloff the top of my head, in that file, we'd call a self.storage.begin() / self.storage.commit() / self.storage.rollback() where appropriate, then implement those 3 methods in the SQLA driver to handle the TX, which SQLA has support for etc16:21
kiallSo - That part should be simple enough, after that, we could start adding support for batches to the various methods.. Updating central as we go16:22
*** msisk has joined #openstack-dns16:22
artomI suppose I could run tests from a USB stick...16:26
artomGets rid of my SSD "problem" so I can replicate the issue ;)16:27
kiallartom: sorry - very distracted at the moment ;)16:27
kiallI'll give it a shot in a few16:27
artomBah, regardless, it's not very scalable to run to you every time I change something.16:28
kiall:P16:28
artomThough I don't grok why we need one golden db per process16:29
artomAs opposed to a single global one (per repo)16:29
kiallWe don't have anywhere to create 1 "total" as we've got nowhere to run code before the processes are started16:30
artomHence my file idea.16:31
kiallI'm not sure I follow then :)16:36
artomSomething like this: http://paste.openstack.org/show/60809/16:37
artomWith locking thrown in so that while the the db is being upgraded other process don't use it.16:38
openstackgerritA change was merged to stackforge/designate: Ensure SQLite has the correct unique_record index  https://review.openstack.org/6544616:45
kiallHumm, That would mean all the processes need to know the same filename in advance, which is generally something to avoid..16:45
artomYeah, the filename would be hardcoded.16:46
artomFor example by prepending /tmp to the migrate repo path.16:46
*** sballe has joined #openstack-dns16:48
*** vinod1 has quit IRC16:55
*** jmcbride has quit IRC16:56
kiallMeeting in 2 mins..16:57
kiallAlthough, looks like the previous meeting might run over.. We'll see.16:57
tsimmonsCool16:58
*** vinod has joined #openstack-dns16:59
*** jmcbride has joined #openstack-dns17:00
kiallAnd - started :)17:00
rjrjrwhat channel was the meeting again?17:02
tsimmons#openstack-meeting-alt17:02
rjrjrthanks17:03
*** sballe has quit IRC17:23
*** sballe has joined #openstack-dns17:24
*** jmcbride1 has joined #openstack-dns17:29
*** jmcbride has quit IRC17:30
*** jmcbride has joined #openstack-dns17:39
*** CaptTofu has quit IRC17:40
*** CaptTofu has joined #openstack-dns17:41
*** jmcbride1 has quit IRC17:43
*** CaptTofu has quit IRC17:50
*** CaptTofu has joined #openstack-dns17:50
*** tsimmons has quit IRC18:05
*** jmcbride has quit IRC18:19
*** rossk has joined #openstack-dns18:26
*** CaptTofu has quit IRC18:26
*** vinod has quit IRC18:29
*** CaptTofu has joined #openstack-dns18:34
*** jorgem has joined #openstack-dns18:35
*** eankutse has quit IRC18:36
*** CaptTofu has quit IRC18:58
*** jmcbride has joined #openstack-dns19:36
*** eankutse has joined #openstack-dns19:50
*** vinod has joined #openstack-dns19:58
*** vipul is now known as vipul-away20:02
*** vipul-away is now known as vipul20:02
*** vipul is now known as vipul-away20:17
*** vipul-away is now known as vipul20:36
artomDid some tests on my USB key... (since it's the only "slow" disk I have)20:50
artomhttp://paste.openstack.org/show/60828/20:52
artomI think it's as good as it's going to get...20:53
*** betsy has quit IRC20:54
openstackgerritArtom Lifshitz proposed a change to stackforge/designate: Database fixture  https://review.openstack.org/6207920:56
*** CaptTofu has joined #openstack-dns21:06
openstackgerritEndre Karlson proposed a change to stackforge/designate: Part 1 of FloatingIP PTR record functionality  https://review.openstack.org/5831621:27
*** david-lyle has quit IRC21:27
*** david-lyle has joined #openstack-dns21:28
*** eankutse has quit IRC21:31
*** eankutse has joined #openstack-dns21:31
*** jmcbride has quit IRC22:00
*** vinod has quit IRC22:11
*** eankutse has quit IRC22:24
*** msisk has quit IRC22:36
kiallartom: I'll test on my "slow" PC in the morning, but I expect we can reduce the time by consolidating the old migrations down.. Assuming the "slowness" really is just the cost of testing using the actual migrations, it's an acceptable cost I reckon. We have a bunch of things we can do to reduce the cost anyway.. like:23:00
kiall1) reduce the migrations..23:00
kiall2) Offer 2 test modes, 1 which uses the migrations, 1 which creates the DB from the models.. But add a test to ensure the model generated DB is the same as the migration generated DB23:00
kiallBoth would be followup commits rather than changes to your stuff :)23:01
kiallthe test from in #2 is would be another good thing anyway, regardless.23:01
*** jmcbride has joined #openstack-dns23:30
*** jmcbride has quit IRC23:33
*** jorgem has quit IRC23:42
« Tuesday, 2014-01-07 Index Thursday, 2014-01-09 »

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!