Tuesday, 2015-07-28

*** jwang has quit IRC00:00
*** jwang has joined #openstack-trove00:00
*** _amrith_ is now known as amrith00:22
*** openstackgerrit has quit IRC00:46
*** openstackgerrit has joined #openstack-trove00:47
*** esmute_ has quit IRC00:51
*** annashen has joined #openstack-trove00:54
*** annashen has quit IRC00:59
*** erkules_ has joined #openstack-trove01:04
*** erkules has quit IRC01:07
*** haomaiwang has quit IRC01:22
*** haomaiwang has joined #openstack-trove01:23
*** lexholden has quit IRC01:51
*** annashen has joined #openstack-trove01:55
*** annashen has quit IRC02:00
*** mingdang1 has joined #openstack-trove02:04
*** eglute has quit IRC02:07
*** eglute has joined #openstack-trove02:09
*** eglute has quit IRC02:09
*** eglute has joined #openstack-trove02:10
*** annashen has joined #openstack-trove02:17
openstackgerritPeter Stachowski proposed openstack/trove: Add support for Redis replication  https://review.openstack.org/20327802:25
*** esmute_ has joined #openstack-trove02:27
*** annashen_ has joined #openstack-trove02:56
*** esmute_ has quit IRC02:57
*** esmute_ has joined #openstack-trove02:59
*** annashen_ has quit IRC03:01
pmalikHi, may I possibly ask for some reviews on the following patch sets. All of them have already been reviewed and amended more than once at various points but are now stalling. Thanks. :-P03:10
pmalikhttps://review.openstack.org/#/c/194739/03:10
pmalikhttps://review.openstack.org/#/c/198595/03:10
pmalikhttps://review.openstack.org/#/c/198891/03:10
pmalik^^^^ cp16net vkmc SlickNik edmondk amrith03:10
*** esmute_ has quit IRC03:17
*** coolsvap|away is now known as coolsvap03:32
cp16netpmalik: alright03:34
*** esmute_ has joined #openstack-trove03:41
*** annashen has quit IRC03:45
*** amrith is now known as _amrith_03:45
*** esmute_ has quit IRC03:54
*** annashen has joined #openstack-trove03:57
*** annashen has quit IRC04:01
*** esmute_ has joined #openstack-trove04:08
*** esmute_ has quit IRC04:23
*** sgotliv has joined #openstack-trove04:38
*** esmute_ has joined #openstack-trove04:39
*** sgotliv has quit IRC04:50
*** annashen has joined #openstack-trove04:58
*** annashen has quit IRC05:02
*** esmute_ has quit IRC05:39
*** annashen has joined #openstack-trove05:59
*** albertom has quit IRC06:01
*** annashen has quit IRC06:03
*** albertom has joined #openstack-trove06:09
*** chlong has quit IRC06:17
*** saban has quit IRC06:22
*** pmackinn has quit IRC06:38
*** saban has joined #openstack-trove06:47
-openstackstatus- NOTICE: zuul is stuck and about to undergo an emergency restart, please be patient as job results may take a long time06:47
*** ChanServ changes topic to "zuul is stuck and about to undergo an emergency restart, please be patient as job results may take a long time"06:47
*** annashen has joined #openstack-trove07:00
*** annashen has quit IRC07:05
*** sgotliv has joined #openstack-trove07:23
*** k4n0 has joined #openstack-trove07:24
*** coolsvap is now known as coolsvap|away07:57
*** sgotliv has quit IRC07:58
*** annashen has joined #openstack-trove08:01
*** ChanServ changes topic to "Trove Openstack Database as a Service | Docs https://github.com/openstack/trove | Channel Logs http://eavesdrop.openstack.org/irclogs/ | IRC must read http://sackheads.org/~bnaylor/spew/away_msgs.html"08:04
-openstackstatus- NOTICE: zuul has been restarted and queues restored. It may take some time to work through the backlog.08:04
*** annashen has quit IRC08:05
*** erkules_ is now known as erkules08:16
*** erkules has joined #openstack-trove08:16
*** sgotliv has joined #openstack-trove08:20
*** coolsvap|away is now known as coolsvap08:56
*** sgotliv has quit IRC08:59
*** annashen has joined #openstack-trove09:01
*** annashen has quit IRC09:06
*** exploreshaifali has joined #openstack-trove09:24
*** nexusz99 has quit IRC09:42
*** annashen has joined #openstack-trove10:02
*** lexholden has joined #openstack-trove10:03
*** annashen has quit IRC10:07
*** annashen has joined #openstack-trove11:03
*** annashen has quit IRC11:07
*** sgotliv has joined #openstack-trove11:22
*** coolsvap is now known as coolsvap|away11:37
*** mingdang1 has quit IRC11:38
*** pmackinn has joined #openstack-trove11:52
*** annashen has joined #openstack-trove12:04
*** annashen has quit IRC12:09
*** mingdang1 has joined #openstack-trove12:11
*** radez is now known as radez_g0n312:42
*** coolsvap|away is now known as coolsvap12:45
*** clduser has joined #openstack-trove12:57
*** lexholden has quit IRC13:02
*** annashen has joined #openstack-trove13:05
*** annashen has quit IRC13:09
*** lexholden has joined #openstack-trove13:17
*** tosky has joined #openstack-trove13:27
*** k4n0 has quit IRC13:29
*** sgotliv_ has joined #openstack-trove13:40
*** sgotliv has quit IRC13:41
*** radez_g0n3 is now known as radez14:04
*** annashen has joined #openstack-trove14:06
*** annashen has quit IRC14:11
*** mingdang1 has quit IRC14:35
openstackgerritMatthew Van Dijk proposed openstack/trove: Cluster instances could falsely report 'ready'  https://review.openstack.org/20611514:43
*** nshah has joined #openstack-trove14:44
*** pmackinn has quit IRC15:02
*** pmackinn has joined #openstack-trove15:03
*** annashen has joined #openstack-trove15:06
*** annashen has quit IRC15:11
*** exploreshaifali has quit IRC15:21
*** annashen has joined #openstack-trove15:48
*** annashen has quit IRC15:51
*** sriram_tesora has joined #openstack-trove15:53
*** logan2 has quit IRC15:55
*** pmackinn has quit IRC15:55
*** pmackinn has joined #openstack-trove15:55
*** lexholden has quit IRC16:02
*** coolsvap is now known as coolsvap|dinner16:06
*** annashen has joined #openstack-trove16:07
*** ashleighfarnham has joined #openstack-trove16:12
*** annashen has quit IRC16:12
*** ashleighfarnham has quit IRC16:16
*** ashleighfarnham has joined #openstack-trove16:17
*** logan2 has joined #openstack-trove16:24
*** esmute__ has joined #openstack-trove16:36
*** annashen has joined #openstack-trove16:38
*** tosky has quit IRC16:47
*** tosky has joined #openstack-trove16:47
*** PeterSchwaller has joined #openstack-trove16:55
*** tosky has quit IRC16:55
*** PeterS has quit IRC16:55
*** annashen has quit IRC16:57
*** annashen has joined #openstack-trove16:59
*** coolsvap|dinner is now known as coolsvap|mt17:01
*** coolsvap|mt is now known as coolsvap|mtg17:01
*** exploreshaifali has joined #openstack-trove17:04
*** esmute__ has quit IRC17:28
*** mingdang1 has joined #openstack-trove17:36
*** mingdang1 has quit IRC17:40
*** esmute__ has joined #openstack-trove17:41
*** annashen has quit IRC17:42
*** esmute__ has quit IRC17:46
*** haomaiwang has quit IRC17:51
*** annashen has joined #openstack-trove17:52
*** haomaiwang has joined #openstack-trove17:52
*** annashen has quit IRC17:56
*** tosky has joined #openstack-trove18:00
*** sgotliv_ has quit IRC18:06
openstackgerritPeter Stachowski proposed openstack/trove: Add Redis backup/restore functionality  https://review.openstack.org/18589218:13
*** PeterSchwaller has quit IRC18:14
*** PeterS has joined #openstack-trove18:15
*** imandhan has joined #openstack-trove18:24
*** PeterSchwaller has joined #openstack-trove18:26
*** esmute__ has joined #openstack-trove18:26
*** annashen has joined #openstack-trove18:27
*** PeterS has quit IRC18:29
*** haomaiwang has quit IRC18:29
*** PeterS has joined #openstack-trove18:30
*** haomaiwang has joined #openstack-trove18:30
*** PeterSchwaller has quit IRC18:33
openstackgerritMatthew Van Dijk proposed openstack/trove: MongoDB cluster instances missing 'key'  https://review.openstack.org/20369618:33
cp16netdbennett: you around?18:35
cp16netso i'm working on getting to 3 nodes into the cluster but right now i only have 2.18:36
cp16netthe first node starts up and does the bootstrap-pxc just fine18:36
cp16neti can login and check the wsrep% and it looks good18:36
cp16netthe second node however is not connecting correctly18:37
cp16nethttps://gist.github.com/cp16net/4d3da8cc61d93a82d76118:37
*** bhunter71 has joined #openstack-trove18:38
cp16neti set the ports 3306, 4444, 4567, 4568 to be open on tcp18:39
cp16netdo they need udp as well?18:39
*** tomblank has joined #openstack-trove18:39
cp16netgeorgelorch: ^^? lefred18:42
dbennettcp16net: checking...18:42
cp16netthx18:42
cp16netfrom the docs i dont see anything about udp only tcp18:43
cp16neti can ping the other ip from each instance18:44
dbennettcp16net: can you post the my.cnf from nodes 1 & 2?18:45
cp16netdbennett: https://gist.github.com/cp16net/202d5dcb6beb71d35e9418:48
dbennettcp16net: Also, is this your cluster name: 7db3c646-c619-4bfe-b98c-a630e3b20f17   if so, it is too long,  don't use a UUID,  there is a bug posted here: https://github.com/bloomberg/chef-bcpc/issues/64918:49
cp16netah shit yeah it is18:49
dbennettyup, that's it18:49
cp16netdamn ok18:49
cp16netso whats the length?18:49
cp16net32?18:49
dbennettjust use a random or grab the last 16 characters from the UUID if possible.18:50
cp16netok18:50
dbennettchecking...18:50
cp16netthats what i'll do18:50
dbennettit's not documented :(.    checking the source....18:50
cp16netboooo18:52
cp16net:-P18:52
* dbennett filing Jira issue on wsrep config var lengths :)18:54
cp16net:-D18:56
cp16netdid you find out what the size must be?18:58
cp16net16? or 32?18:58
*** radez is now known as radez_g0n318:59
*** esmute__ has quit IRC19:00
dbennettcp16net:  there is no defined size limit (it's just malloc'd to whatever is configured).   However,  the gmcast protocol packet that contains the cluster name does apparently have a size limit.   atm, to be safe, I'd say keep at or under 1619:05
cp16netok i made it 1619:07
*** esmute__ has joined #openstack-trove19:12
cp16netdbennett: hmm looks like that helped19:16
cp16netbut now another error trying to sync the data it looks like19:16
cp16nethttps://gist.github.com/cp16net/4c71b20cd947f0c501b319:16
cp16netlooks like the local_status_uuid is not matching19:20
cp16netrather the cluster_state_uuid19:20
*** exploreshaifali has quit IRC19:21
cp16netso i see a few posts online about making the wsrep_sst_method=xtrabackup-v2 instead of just xtraback19:23
cp16netxtrabackup*19:23
*** esmute__ has quit IRC19:25
*** annashen has quit IRC19:30
*** radez_g0n3 is now known as radez19:30
*** esmute__ has joined #openstack-trove19:33
* dbennett filed an internal PXC issue to identify user-defined values that a referenced in gmcast messages and validate a maximum (documented) length.19:34
dbennettcp16net: ^^^19:34
cp16netok19:34
cp16netdbennett: how about that next issue could it be that i am using the wrong name for the sst?19:34
dbennettcp16net: checking...19:35
cp16neti just changed it to see if i could get it working19:36
*** radez is now known as radez_g0n319:36
cp16neti changed it on the joiner node and it looked a little cleaner19:36
cp16netbut failed still19:37
*** esmute__ has quit IRC19:38
cp16netahh shit19:39
cp16net i think i see the problemm19:39
dbennettcp16net:  I have seen an issue like this before.   I'm looking through your configs and I'm not sure if the [mysqld] datadir variable is defined in the primary my.cnf file.   Can you sure that is (set to /var/lib/mysql).   This is a requirement for Percona XtraBackup.  I think we have an open ticket on it.19:40
cp16netlooking at the innobackup.backup.log on the donor19:40
cp16netit found a directory .lost+found that it cant touch19:40
cp16netbecause /var/lib/mysql is a volume mounted19:40
dbennettahh,  that could be an issue.19:40
dbennetthowever, do make sure that datadir=/var/lib/mysql is in the primary my.cnf19:41
cp16netit should be19:41
cp16netdatadir = /var/lib/mysql19:41
dbennettI could see it was in one of them,  but you cat'd multiples19:42
cp16netyeah the my.cnf has the base config19:42
cp16netand we load the conf.d/*19:42
dbennettok,  then it may be the ext# lost+found19:42
cp16netcluster.cnf is in there with jsut the overrides for clustering19:42
cp16netyup look slike it....19:42
dbennettyou could try mounting it somewhere else (/mnt or /srv), creating a mysql directory in the volume and then symlinking it to /var/lib19:43
cp16netyeah i think we have a patch to do something like that19:44
georgelorchcp16net, dbennett yeah, that i already a known issue, you should never use the root of a volume as mysql data dir https://bugs.launchpad.net/trove/+bug/137064619:44
openstackLaunchpad bug 1370646 in Trove "Mysql datadir shouldn't be in the root of mounted filesystem" [Medium,Fix released] - Assigned to Simon Chang (changsimon)19:44
cp16netyup19:46
cp16neti guess i need to pull that patch in19:47
cp16neti'm a little behind since i've been workin on all this clustering for a few weeks19:47
cp16netthat should solve this issue19:47
cp16net:-D19:47
*** radez_g0n3 is now known as radez19:47
dbennettgeorgelorch:  I wonder if that's any root file system or just an ext(2,3,4) file system.19:49
georgelorchdbennett, In general, any filesystem from the old MySQL best practices19:50
*** annashen has joined #openstack-trove19:53
dbennettgeorgelorch: good to know!19:54
*** annashen has quit IRC19:57
cp16netyeah the error says that it doesnt have permission to that folder20:06
*** PeterS has quit IRC20:06
*** esmute__ has joined #openstack-trove20:15
*** annashen has joined #openstack-trove20:20
*** esmute__ has quit IRC20:21
*** PeterS has joined #openstack-trove20:23
*** sgotliv_ has joined #openstack-trove20:25
*** esmute__ has joined #openstack-trove20:29
*** PeterS has quit IRC20:33
*** PeterSchwaller has joined #openstack-trove20:33
*** lexholden has joined #openstack-trove20:39
*** esmute__ has quit IRC20:40
*** logan2 has quit IRC20:42
*** logan2 has joined #openstack-trove20:45
*** sriram_tesora has quit IRC20:46
*** esmute__ has joined #openstack-trove20:48
*** cweid has quit IRC20:48
*** esmute__ has quit IRC20:52
*** esmute__ has joined #openstack-trove20:59
openstackgerritSharika Pongubala proposed openstack/python-troveclient: Root enablement for Vertica clusters/instances  https://review.openstack.org/19736020:59
*** saurabhs has joined #openstack-trove21:23
*** esmute__ has quit IRC21:30
*** esmute__ has joined #openstack-trove21:39
*** tomblank has quit IRC21:42
*** esmute__ has quit IRC21:44
*** esmute__ has joined #openstack-trove21:49
*** annashen_ has joined #openstack-trove21:53
*** annashen_ has quit IRC21:58
*** coolsvap|mtg is now known as coolsvap|away22:00
openstackgerritPetr Malik proposed openstack/trove: Implement user functions for Cassandra datastore  https://review.openstack.org/20673922:15
*** haomaiwang has quit IRC22:15
openstackgerritPetr Malik proposed openstack/trove: Improve the guestagent configuration manager  https://review.openstack.org/19859522:16
openstackgerritPetr Malik proposed openstack/trove: Implement user functions for Cassandra datastore  https://review.openstack.org/20673922:17
openstackgerritPetr Malik proposed openstack/trove: Implement Cassandra Configuration Groups  https://review.openstack.org/20674022:17
*** haomaiwang has joined #openstack-trove22:17
SlickNik_amrith_: around?22:18
*** bhunter71 has quit IRC22:24
openstackgerritPetr Malik proposed openstack/trove: Add backup & restore for Cassandra  https://review.openstack.org/20675122:37
openstackgerritPetr Malik proposed openstack/trove: Add generic int-test classes  https://review.openstack.org/20327422:41
*** esmute__ has quit IRC22:44
*** esmute__ has joined #openstack-trove22:55
*** esmute__ has quit IRC22:59
*** haomaiwang has quit IRC23:01
*** haomaiwang has joined #openstack-trove23:02
*** dbennett has quit IRC23:13
*** tosky has quit IRC23:27
*** saurabhs has quit IRC23:35
*** esmute__ has joined #openstack-trove23:46
*** albertom has quit IRC23:46
*** albertom has joined #openstack-trove23:48
*** annashen_ has joined #openstack-trove23:54
*** annashen has quit IRC23:55
*** annashen_ has quit IRC23:59

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!