| *** mcriswell has quit IRC | 00:03 | |
| *** pid1 has quit IRC | 00:07 | |
| *** jawad_axd has joined #openstack | 00:09 | |
| *** jawad_axd has quit IRC | 00:14 | |
| *** jawad_axd has joined #openstack | 00:30 | |
| *** jawad_axd has quit IRC | 00:34 | |
| *** jawad_axd has joined #openstack | 00:50 | |
| *** xaban has quit IRC | 00:51 | |
| *** jawad_axd has quit IRC | 00:55 | |
| *** gyee has quit IRC | 01:03 | |
| *** skyraven has joined #openstack | 01:10 | |
| *** Onionnion has joined #openstack | 01:13 | |
| *** skyraven has quit IRC | 01:15 | |
| *** pbandark has quit IRC | 01:19 | |
| *** jawad_axd has joined #openstack | 01:32 | |
| *** jawad_axd has quit IRC | 01:36 | |
| *** jawad_axd has joined #openstack | 01:52 | |
| *** jawad_axd has quit IRC | 01:57 | |
| *** jawad_axd has joined #openstack | 02:13 | |
| *** jawad_axd has quit IRC | 02:18 | |
| *** brokencycle has quit IRC | 02:22 | |
| *** bobmel has joined #openstack | 02:32 | |
| *** bobmel has quit IRC | 02:32 | |
| *** awalende has joined #openstack | 02:44 | |
| *** awalende has quit IRC | 02:48 | |
| *** shibboleth has quit IRC | 03:01 | |
| *** arnoldoree has quit IRC | 03:04 | |
| *** Onionnion has quit IRC | 03:23 | |
| *** skyraven has joined #openstack | 05:12 | |
| *** skyraven has quit IRC | 05:16 | |
| *** k_mouza has joined #openstack | 05:34 | |
| *** k_mouza has quit IRC | 05:38 | |
| *** iokiwi has joined #openstack | 05:50 | |
| *** kjackal has joined #openstack | 06:12 | |
| *** dsneddon has quit IRC | 06:23 | |
| *** noodle has quit IRC | 06:36 | |
| *** noodle has joined #openstack | 06:44 | |
| *** dsneddon has joined #openstack | 06:52 | |
| *** dsneddon has quit IRC | 06:57 | |
| *** sauvin has joined #openstack | 07:07 | |
| *** sauvin has joined #openstack | 07:11 | |
| *** skyraven has joined #openstack | 07:13 | |
| *** skyraven has quit IRC | 07:17 | |
| *** dsneddon has joined #openstack | 07:28 | |
| *** belmoreira has joined #openstack | 07:33 | |
| *** asidorenko has quit IRC | 07:39 | |
| *** asidorenko has joined #openstack | 07:41 | |
| *** rcernin_ has quit IRC | 07:45 | |
| *** jtomasek has joined #openstack | 07:48 | |
| *** dsneddon has quit IRC | 07:54 | |
| *** maddtux has joined #openstack | 08:03 | |
| *** belmoreira has quit IRC | 08:07 | |
| *** belmoreira has joined #openstack | 08:09 | |
| *** k_mouza has joined #openstack | 08:15 | |
| *** k_mouza has quit IRC | 08:20 | |
| *** kjackal has quit IRC | 08:21 | |
| *** belmoreira has quit IRC | 08:21 | |
| *** kjackal has joined #openstack | 08:23 | |
| *** belmoreira has joined #openstack | 08:35 | |
| *** slaweq has joined #openstack | 08:49 | |
| *** slaweq has quit IRC | 08:54 | |
| *** belmoreira has quit IRC | 09:01 | |
| *** asbalderson has quit IRC | 09:01 | |
| *** asbalderson has joined #openstack | 09:01 | |
| *** slaweq has joined #openstack | 09:05 | |
| *** lpetrut has quit IRC | 09:07 | |
| *** slaweq has quit IRC | 09:10 | |
| *** skyraven has joined #openstack | 09:14 | |
| *** cah_link has joined #openstack | 09:14 | |
| *** cah_link1 has joined #openstack | 09:17 | |
| *** cyberpear has quit IRC | 09:17 | |
| *** cah_link has quit IRC | 09:17 | |
| *** cah_link1 is now known as cah_link | 09:17 | |
| *** cyberpear has joined #openstack | 09:18 | |
| *** skyraven has quit IRC | 09:18 | |
| *** slaweq has joined #openstack | 09:20 | |
| *** donnyd has quit IRC | 09:22 | |
| *** donnyd has joined #openstack | 09:23 | |
| *** jeh has quit IRC | 09:24 | |
| *** jeh has joined #openstack | 09:24 | |
| *** slaweq has quit IRC | 09:25 | |
| *** belmoreira has joined #openstack | 09:37 | |
| *** belmoreira has quit IRC | 09:38 | |
| *** aedc has joined #openstack | 09:40 | |
| *** sshnaidm|off has quit IRC | 09:48 | |
| *** cz3 has quit IRC | 09:56 | |
| *** cz3 has joined #openstack | 09:57 | |
| *** dsneddon has joined #openstack | 10:12 | |
| *** dsneddon has quit IRC | 10:17 | |
| *** kjackal has quit IRC | 10:18 | |
| *** gixx has quit IRC | 10:27 | |
| *** gixx has joined #openstack | 10:27 | |
| *** southquist has joined #openstack | 10:34 | |
| *** xaban has joined #openstack | 10:37 | |
| xaban | Hi, our rabbitmq cluster is broken and we can't get it fixed. We stopped rabibtmq on all nodes, starting nodes one by one, but still they cannot form a cluster. | 10:37 |
|---|---|---|
| osmanlicilegi | xaban: have you tried clearing /var/lib/rabbitmq/mnesia ? | 10:38 |
| xaban | Waiting for Mnesia tables for 30000 ms, 9 retries left and suddenly it fails | 10:38 |
| xaban | osmanlicilegi I have cleared /var/lib/rabbitmq/* | 10:38 |
| xaban | The nodes can telnet to eachother via 5672. | 10:39 |
| osmanlicilegi | xaban: stop all servers, rm /var/lib/rabbitmq/mnesia from all and then start servers one by one | 10:39 |
| xaban | Thats what I did | 10:39 |
| osmanlicilegi | xaban: so it may be a network problem | 10:39 |
| osmanlicilegi | do you have any firewall policy on your gateway? if so you should check it | 10:40 |
| xaban | osmanlicilegi: telnet on port 5672 between the nodes just works | 10:40 |
| yoctozepto | xaban: what method of deployment is this? what version of rabbitmq? what distro? | 10:40 |
| osmanlicilegi | oh sorry for my eyes saw it as "cannot" | 10:40 |
| xaban | yoctozepto: Queens/13, TripleO, containerized | 10:41 |
| xaban | Messages like "node 'rabbit@am6-controller-2' down: connection_closed" | 10:42 |
| xaban | And the other node: ** Node 'rabbit@am6-controller-0' not responding ** | 10:42 |
| xaban | While with tcpdump I see traffic between them | 10:42 |
| osmanlicilegi | xaban: are all containers up and running? | 10:42 |
| yoctozepto | all 3 are up? what does the cluster status say? | 10:42 |
| xaban | They are in starting state, and fail to start because can't connect to eachoter. | 10:43 |
| yoctozepto | odd, they should not fail to start | 10:43 |
| yoctozepto | I mean, the containers/daemons should still be up | 10:44 |
| osmanlicilegi | xaban: have you tried removing all containers and their volumes and then deploing the rabbitmq cluster from strach? | 10:44 |
| yoctozepto | just reporting some problems | 10:44 |
| yoctozepto | ^^ that makes most sense | 10:44 |
| yoctozepto | if it's all broken and hard to guess why | 10:44 |
| yoctozepto | if you cleared mnesia then the data is already gone so nothing to lose | 10:45 |
| xaban | osmanlicilegi: Pacemaker is controller the cluster. I "unmanaged" the cluster from pcs. Stopped the containers on all nodes. Cleared /var/lib/rabbitmq/ on all nodes. | 10:45 |
| xaban | controlling* | 10:45 |
| xaban | Then put the in manage mode again, and start them 1 by 1 | 10:45 |
| yoctozepto | ah yeah, ooo uses pacemaker | 10:45 |
| xaban | Oh yes, sorry | 10:46 |
| *** aedc has quit IRC | 10:47 | |
| yoctozepto | and not even a single one can boot up with zeroed mnesia? | 10:48 |
| xaban | When 1st node is started: [{nodes,[{disc,['rabbit@am6-controller-0']}]}, {running_nodes,['rabbit@am6-controller-0']}, | 10:48 |
| yoctozepto | and does it work? | 10:48 |
| yoctozepto | try connecting some service, posting message | 10:48 |
| osmanlicilegi | imho, deploying form scratch should work | 10:49 |
| osmanlicilegi | xaban: have you had a docker upgrade previously? | 10:50 |
| xaban | For some time we didn't applied any updates | 10:50 |
| osmanlicilegi | if "docker kill rmq && docker rm rmq && docker volume rm rmq && deploy rmq" does not work, maybe the problem is not rmq alone. | 10:53 |
| yoctozepto | osmanlicilegi: I suspect pacemaker playing tricks on it | 10:54 |
| yoctozepto | rmq does not die by itself in such cases | 10:55 |
| xaban | yoctozepto: I guess so yess | 10:55 |
| osmanlicilegi | yoctozepto is right | 10:55 |
| yoctozepto | most likely pacemaker killing it due to healthchecks | 10:55 |
| xaban | Now even pacemaker doesn't try to start the container on 1 of the nodes | 10:55 |
| yoctozepto | healthchecks gone bad | 10:55 |
| yoctozepto | meh, you accurately pinpoint why I avoided tripleo | 10:56 |
| xaban | rabbitmq-bundle-0 was running on controller-0, rabbitmq-bundle-1 on controller-1, rabbitmq-bundle-2 on controller-2 .. now it starts rabbitmq-bundle-1 on controller-2 :/ | 10:56 |
| xaban | and nothing on controller-1 | 10:56 |
| yoctozepto | I can't really see why they went with pacemaker for rabbitmq when it clusters itself properly | 10:57 |
| yoctozepto | I would suggest you also try asking on #tripleo and on the openstack-discuss mailing list | 11:01 |
| xaban | Thanks | 11:02 |
| osmanlicilegi | yoctozepto: +1 for your rmq clustering opinion. using pacemaker to prevent spof can sometimes results as pacemaker becomes spof. | 11:03 |
| *** southquist has quit IRC | 11:04 | |
| *** brokencycle has joined #openstack | 11:07 | |
| yoctozepto | osmanlicilegi: indeed, I would say pacemaker is quite fragile - one bad move and you are fried - i.e. don't use when not necessary | 11:09 |
| yoctozepto | rmq has native clustering and is picky about hostname/ip address mappings | 11:10 |
| yoctozepto | if pacemaker is throwing it around the cluster I can see no good to it | 11:11 |
| yoctozepto | but otherwise have no idea so best ask someone knowledgeable at the aforementioned places | 11:11 |
| *** skyraven has joined #openstack | 11:14 | |
| *** skyraven has quit IRC | 11:19 | |
| *** Lucas_Gray has joined #openstack | 11:24 | |
| *** belmoreira has joined #openstack | 11:31 | |
| *** sshnaidm|off has joined #openstack | 11:32 | |
| *** sshnaidm|off has quit IRC | 11:36 | |
| *** sshnaidm has joined #openstack | 11:36 | |
| *** belmoreira has quit IRC | 11:41 | |
| *** Adri2000 has quit IRC | 11:51 | |
| *** Adri2000 has joined #openstack | 11:52 | |
| *** sshnaidm has quit IRC | 12:00 | |
| *** sshnaidm has joined #openstack | 12:01 | |
| *** spiral has joined #openstack | 12:09 | |
| *** dsneddon has joined #openstack | 12:13 | |
| *** dsneddon has quit IRC | 12:17 | |
| *** Lucas_Gray has quit IRC | 12:20 | |
| *** dudek has joined #openstack | 12:23 | |
| *** belmoreira has joined #openstack | 13:05 | |
| *** skyraven has joined #openstack | 13:15 | |
| *** skyraven has quit IRC | 13:20 | |
| *** damien_r has joined #openstack | 13:40 | |
| *** belmoreira has quit IRC | 13:47 | |
| *** belmoreira has joined #openstack | 14:04 | |
| *** dsneddon has joined #openstack | 14:13 | |
| *** dsneddon has quit IRC | 14:18 | |
| *** damien_r has quit IRC | 14:21 | |
| *** soniya29 has joined #openstack | 14:49 | |
| *** belmoreira has quit IRC | 14:54 | |
| *** belmoreira has joined #openstack | 15:01 | |
| *** soniya29 has quit IRC | 15:10 | |
| *** skyraven has joined #openstack | 15:16 | |
| *** skyraven has quit IRC | 15:21 | |
| *** damien_r has joined #openstack | 15:33 | |
| *** belmoreira has quit IRC | 15:34 | |
| *** damien_r has quit IRC | 15:34 | |
| *** Relih has joined #openstack | 15:37 | |
| *** spiral has quit IRC | 16:18 | |
| *** maddtux has quit IRC | 16:30 | |
| *** shibboleth has joined #openstack | 17:12 | |
| *** skyraven has joined #openstack | 17:17 | |
| *** skyraven has quit IRC | 17:22 | |
| *** k_mouza has joined #openstack | 17:48 | |
| *** dudek has quit IRC | 18:37 | |
| *** dudek has joined #openstack | 18:37 | |
| *** mrechtma__ has joined #openstack | 18:38 | |
| *** xaban has quit IRC | 18:49 | |
| *** mrechtma__ has quit IRC | 18:59 | |
| *** skyraven has joined #openstack | 19:18 | |
| *** skyraven has quit IRC | 19:22 | |
| *** jtomasek has quit IRC | 19:30 | |
| *** mulbc has quit IRC | 19:30 | |
| *** ChrisNBlum has joined #openstack | 19:30 | |
| *** avivgt has quit IRC | 19:35 | |
| *** oncall-pokemon has joined #openstack | 19:38 | |
| *** Onionnion has joined #openstack | 19:39 | |
| *** CrayZee has joined #openstack | 19:58 | |
| *** CrayZee has quit IRC | 19:59 | |
| *** snapiri has joined #openstack | 19:59 | |
| *** dsneddon has joined #openstack | 20:01 | |
| *** dsneddon has quit IRC | 20:06 | |
| *** damien_r has joined #openstack | 20:15 | |
| *** k_mouza has quit IRC | 20:19 | |
| *** damien_r has quit IRC | 20:29 | |
| *** damien_r has joined #openstack | 20:33 | |
| *** damien_r has quit IRC | 20:33 | |
| *** sshnaidm has quit IRC | 20:34 | |
| *** sshnaidm has joined #openstack | 20:36 | |
| *** sshnaidm has quit IRC | 20:36 | |
| *** factor has joined #openstack | 20:50 | |
| *** avivgt has joined #openstack | 21:00 | |
| *** k_mouza has joined #openstack | 21:02 | |
| *** skyraven has joined #openstack | 21:19 | |
| *** k_mouza has quit IRC | 21:20 | |
| *** skyraven has quit IRC | 21:24 | |
| *** renich has joined #openstack | 21:26 | |
| *** k_mouza has joined #openstack | 21:35 | |
| *** k_mouza has quit IRC | 21:40 | |
| *** k_mouza has joined #openstack | 21:42 | |
| *** damien_r has joined #openstack | 21:46 | |
| *** damien_r has quit IRC | 21:50 | |
| *** damien_r has joined #openstack | 21:50 | |
| *** damien_r has quit IRC | 21:56 | |
| *** damien_r has joined #openstack | 22:01 | |
| *** CrayZee has joined #openstack | 22:04 | |
| *** CrayZee is now known as Guest22601 | 22:04 | |
| *** damien_r has quit IRC | 22:05 | |
| *** damien_r has joined #openstack | 22:06 | |
| *** damien_r has quit IRC | 22:06 | |
| *** snapiri has quit IRC | 22:06 | |
| *** Guest22601 has quit IRC | 22:07 | |
| *** tomgray has joined #openstack | 22:23 | |
| *** dudek has quit IRC | 22:25 | |
| *** bobmel has joined #openstack | 22:33 | |
| *** Croata has joined #openstack | 22:35 | |
| *** bobmel has quit IRC | 22:36 | |
| *** bobmel has joined #openstack | 22:36 | |
| *** bobmel has quit IRC | 22:36 | |
| *** bobmel has joined #openstack | 22:36 | |
| *** tomgray has quit IRC | 22:41 | |
| *** bobmel has quit IRC | 22:41 | |
| *** Onionnion has quit IRC | 22:49 | |
| *** gafur has joined #openstack | 23:14 | |
| *** cah_link1 has joined #openstack | 23:18 | |
| *** gafur has quit IRC | 23:19 | |
| *** skyraven has joined #openstack | 23:20 | |
| *** cah_link has quit IRC | 23:21 | |
| *** cah_link1 is now known as cah_link | 23:21 | |
| *** k_mouza has quit IRC | 23:23 | |
| *** skyraven has quit IRC | 23:24 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!