| *** zhouhan has quit IRC | 00:16 | |
| *** djbalrog has quit IRC | 00:22 | |
| *** djbalrog has joined #openvswitch | 00:24 | |
| *** dcbw has quit IRC | 00:56 | |
| *** tbachman has quit IRC | 01:20 | |
| *** tbachman has joined #openvswitch | 01:22 | |
| *** acidfu_ has joined #openvswitch | 01:39 | |
| *** vdasari has quit IRC | 01:40 | |
| *** djbalrog has quit IRC | 01:47 | |
| *** djbalrog has joined #openvswitch | 01:48 | |
| *** dholler has quit IRC | 02:07 | |
| *** dholler has joined #openvswitch | 02:20 | |
| *** ricolin_ has joined #openvswitch | 02:32 | |
| *** ricolin has quit IRC | 02:33 | |
| *** tbachman_ has joined #openvswitch | 02:41 | |
| *** tbachman has quit IRC | 02:41 | |
| *** tbachman_ is now known as tbachman | 02:42 | |
| *** rcernin has quit IRC | 03:20 | |
| *** rcernin has joined #openvswitch | 03:23 | |
| *** djbalrog has quit IRC | 03:27 | |
| *** djbalrog has joined #openvswitch | 03:29 | |
| *** djbalrog has quit IRC | 03:41 | |
| *** djbalrog has joined #openvswitch | 03:43 | |
| *** ricolin_ has quit IRC | 04:24 | |
| *** ricolin has joined #openvswitch | 04:34 | |
| *** ricolin_ has joined #openvswitch | 04:42 | |
| *** ricolin has quit IRC | 04:42 | |
| *** djbalrog has quit IRC | 05:06 | |
| *** djbalrog has joined #openvswitch | 05:10 | |
| *** djbalrog has quit IRC | 05:21 | |
| *** djbalrog has joined #openvswitch | 05:24 | |
| *** djbalrog has quit IRC | 05:56 | |
| *** djbalrog has joined #openvswitch | 05:59 | |
| *** djbalrog has quit IRC | 06:11 | |
| *** djbalrog has joined #openvswitch | 06:14 | |
| *** anilvenk_ has quit IRC | 06:17 | |
| *** slaweq has joined #openvswitch | 06:18 | |
| *** anilvenkata has joined #openvswitch | 06:22 | |
| *** eelco has joined #openvswitch | 06:22 | |
| *** ralonsoh has joined #openvswitch | 06:28 | |
| *** slaweq has quit IRC | 06:39 | |
| *** djbalrog has quit IRC | 06:46 | |
| *** djbalrog has joined #openvswitch | 06:49 | |
| *** ricolin_ is now known as ricolin | 06:50 | |
| *** ricolin has quit IRC | 06:51 | |
| *** ricolin has joined #openvswitch | 06:53 | |
| *** mdgray has joined #openvswitch | 07:14 | |
| *** thaller has quit IRC | 07:31 | |
| *** thaller has joined #openvswitch | 07:31 | |
| *** djbalrog has quit IRC | 07:37 | |
| *** djbalrog has joined #openvswitch | 07:38 | |
| *** djbalrog has quit IRC | 07:51 | |
| *** thaller has quit IRC | 07:53 | |
| *** thaller has joined #openvswitch | 07:53 | |
| *** djbalrog has joined #openvswitch | 07:54 | |
| *** rcernin has quit IRC | 08:04 | |
| *** rcernin has joined #openvswitch | 08:37 | |
| *** rcernin has joined #openvswitch | 08:38 | |
| *** istokes has joined #openvswitch | 08:46 | |
| *** cpaelzer__ has joined #openvswitch | 08:47 | |
| *** cpaelzer has quit IRC | 08:49 | |
| *** istokes has left #openvswitch | 08:53 | |
| *** cpaelzer_ has joined #openvswitch | 08:53 | |
| *** cpaelzer__ has quit IRC | 08:54 | |
| *** rcernin has quit IRC | 09:16 | |
| *** djbalrog has quit IRC | 09:16 | |
| *** djbalrog has joined #openvswitch | 09:19 | |
| *** rcernin has joined #openvswitch | 09:22 | |
| *** psahoo has joined #openvswitch | 09:22 | |
| *** rcernin has quit IRC | 09:36 | |
| *** rcernin has joined #openvswitch | 09:54 | |
| *** rcernin has quit IRC | 09:57 | |
| *** rcernin has joined #openvswitch | 09:57 | |
| *** djbalrog has quit IRC | 10:07 | |
| *** djbalrog has joined #openvswitch | 10:09 | |
| *** anilvenkata has quit IRC | 10:59 | |
| *** ihrachys has quit IRC | 11:02 | |
| *** ihrachys has joined #openvswitch | 11:08 | |
| *** rcernin has quit IRC | 11:16 | |
| *** anilvenkata has joined #openvswitch | 11:19 | |
| *** anilvenkata has quit IRC | 11:20 | |
| *** djbalrog has quit IRC | 11:47 | |
| *** djbalrog has joined #openvswitch | 11:49 | |
| *** anilvenkata has joined #openvswitch | 11:50 | |
| *** anilvenkata has quit IRC | 11:55 | |
| *** anilvenkata has joined #openvswitch | 11:56 | |
| *** bostondriver has joined #openvswitch | 11:56 | |
| *** yamamoto has quit IRC | 11:58 | |
| *** djbalrog has quit IRC | 12:02 | |
| *** djbalrog has joined #openvswitch | 12:03 | |
| *** tfherbert has quit IRC | 12:28 | |
| *** psahoo has quit IRC | 12:31 | |
| *** psahoo has joined #openvswitch | 12:32 | |
| *** vdasari_ has joined #openvswitch | 12:35 | |
| *** vdasari_ has left #openvswitch | 12:35 | |
| *** djbalrog has quit IRC | 12:37 | |
| *** djbalrog has joined #openvswitch | 12:38 | |
| *** dholler has quit IRC | 12:41 | |
| *** yamamoto has joined #openvswitch | 12:43 | |
| *** yamamoto has quit IRC | 12:51 | |
| *** yamamoto has joined #openvswitch | 12:51 | |
| *** djbalrog has quit IRC | 12:52 | |
| *** djbalrog has joined #openvswitch | 12:53 | |
| *** acidfu_ has quit IRC | 13:15 | |
| *** acidfu_ has joined #openvswitch | 13:18 | |
| *** djbalrog has quit IRC | 13:27 | |
| *** anilvenkata has quit IRC | 13:27 | |
| *** djbalrog has joined #openvswitch | 13:29 | |
| *** dcbw has joined #openvswitch | 13:32 | |
| *** djbalrog has quit IRC | 13:42 | |
| *** djbalrog has joined #openvswitch | 13:43 | |
| *** anilvenkata has joined #openvswitch | 13:50 | |
| *** anilvenkata has quit IRC | 13:54 | |
| *** anilvenkata has joined #openvswitch | 13:54 | |
| *** anilvenkata has quit IRC | 13:56 | |
| *** yamamoto has quit IRC | 13:56 | |
| *** spatel has joined #openvswitch | 14:14 | |
| spatel | folk, looking for help to setup ovn / ovsdb cluster with 3 node and something is missing in my config, this is what i did so let me know what is missing here - http://paste.openstack.org/show/805545/ | 14:15 |
|---|---|---|
| spatel | Trying to understand how do i verify my ovsdb clustering is working, does anyone know command to verify or method? | 14:15 |
| *** jangutter_ has joined #openvswitch | 14:15 | |
| *** jangutter has quit IRC | 14:18 | |
| *** jangutter has joined #openvswitch | 14:19 | |
| *** tbachman has quit IRC | 14:20 | |
| *** tbachman has joined #openvswitch | 14:22 | |
| *** jangutter_ has quit IRC | 14:23 | |
| *** numans has quit IRC | 14:28 | |
| imaximets | spatel, something like 'ovs-appctl -t /var/run/ovn/ovnnb_db.ctl cluster/status OVN_Northbound' should show the status of a db cluster. | 14:30 |
| spatel | imaximets thanks for reply is OVN_Northbound is option in this command or something else? | 14:31 |
| *** djbalrog has quit IRC | 14:31 | |
| imaximets | spatel, it's a database schema name. | 14:32 |
| spatel | oh! so i can see long output on node-1 but node-2 and node-3 throwing error | 14:32 |
| *** anilvenk_ has joined #openvswitch | 14:33 | |
| spatel | imaximets here is the output - http://paste.openstack.org/show/805547/ | 14:33 |
| *** djbalrog has joined #openvswitch | 14:34 | |
| *** istokes has joined #openvswitch | 14:34 | |
| imaximets | spatel, node-1 started a cluster and it's the only server in the cluster. Probably node-2 and 3 failed to join the cluster/started incorrectly. You should check logs in /var/log/ovn | 14:35 |
| spatel | let me try to re-join and see.. this is good command to see cluster status :) | 14:35 |
| spatel | imaximets this is the command i run to join cluster, do you see any issue here or missing parameter - http://paste.openstack.org/show/805548/ | 14:38 |
| spatel | why its saying - ovsdb-tool: 'join-cluster' command requires at least 4 arguments, also in end its saying connected should i consider it error or warning | 14:39 |
| imaximets | spatel, yep. join-cluster failed. What is your version of OVS and OVN ? | 14:39 |
| imaximets | spatel, 'connected' is about local unix socket, so irrelevant. | 14:40 |
| imaximets | spatel, 'join-cluster' should not fail. it's an error. | 14:40 |
| spatel | stupid question how do i check OVN version? | 14:41 |
| spatel | ovn-northd 20.03.2 | 14:41 |
| spatel | Open vSwitch Library 2.13.3 | 14:41 |
| spatel | got it | 14:41 |
| *** yamamoto has joined #openvswitch | 14:42 | |
| spatel | imaximets i feel like its new, i build whole environment 2 month ago.. so assuming it does have support | 14:43 |
| *** tbachman_ has joined #openvswitch | 14:44 | |
| *** moldorcoder7 has quit IRC | 14:44 | |
| *** tbachman has quit IRC | 14:45 | |
| *** tbachman_ is now known as tbachman | 14:45 | |
| imaximets | spatel, is it acceptable for you to wipe out /var/run/ovn/*.db ? This will destroy all OVN databases/configuration. | 14:45 |
| spatel | Yes, this is lab i can do whatever you say..:) | 14:45 |
| spatel | do you want me to whip out on all 3 nodes or node-2 and node-3? | 14:46 |
| spatel | wipe* | 14:46 |
| imaximets | spatel, 2 and 3. first seems fine. | 14:47 |
| spatel | ok doing it | 14:47 |
| spatel | imaximets check this out - http://paste.openstack.org/show/805549/ | 14:49 |
| spatel | I didn't see other errors | 14:50 |
| spatel | does cluster join has any debug option to see what is going on | 14:52 |
| imaximets | spatel, could you show your /var/log/ovn/ovsdb-server-nb.log ? | 14:53 |
| spatel | sure.. | 14:53 |
| spatel | imaximets here you go - http://paste.openstack.org/show/805550/ | 14:54 |
| spatel | check last couple lines.. it may contain yesterday logs also | 14:56 |
| imaximets | spatel, it looks like ovsdb thinks that it's a satandalone database, not a clustered one... | 14:57 |
| spatel | hmm | 14:57 |
| spatel | how did you check that from logs? | 15:00 |
| *** yamamoto has quit IRC | 15:00 | |
| imaximets | spatel, I just don't see any attempts to connect to the cluster that should be there. And raft module is not initialized hence the "cluster/status" is not a valid command. | 15:01 |
| spatel | hmm | 15:02 |
| *** moldorcoder7 has joined #openvswitch | 15:02 | |
| spatel | do you think i should wipe out *.db on node-1 also ? | 15:02 |
| imaximets | spatel, you can try, but it should not make a difference. 2 and 3 didn't even try to connect. Some problem with ovn-ctl and how it prepares a database file, most likely. | 15:04 |
| spatel | interesting.. | 15:05 |
| spatel | node-2# ovsdb-tool db-local-address /var/lib/ovn/ovnsb_db.db | 15:05 |
| spatel | ovsdb-tool: /var/lib/ovn/ovnsb_db.db: not a clustered database | 15:05 |
| spatel | let me try to remove all DB and then try | 15:05 |
| spatel | no luck.. | 15:10 |
| spatel | may be my command is wrong.. | 15:11 |
| spatel | to join | 15:11 |
| * imaximets checks how ovn-k8s starts ovn... | 15:12 | |
| spatel | imaximets why k8s ? is this a documentation ? | 15:16 |
| spatel | node-1 # ovsdb-tool db-local-address /var/lib/ovn/ovnnb_db.db | 15:16 |
| spatel | tcp:172.30.40.93:6643 | 15:16 |
| spatel | why i am seeing db-local-address listening on 6643 on node-1 ? | 15:16 |
| spatel | 6642 is for ovsdb-server where is 6643 coming from? | 15:16 |
| spatel | got it - OVN_Northbound cluster remote tcp port is 6643 | 15:19 |
| imaximets | spatel, I just know that ovn-k8s uses this successfully. but it uses ssl, so no help from there. | 15:20 |
| imaximets | spatel, yes, 6643 is default for northbound. | 15:20 |
| spatel | let me check ovn-k8s but i am very surprised there isn't any good doc or guide to deal with this issue | 15:21 |
| imaximets | spatel, as an option, you may try to create a db file before running ovn-ctl. | 15:22 |
| spatel | hmm what do you mean by create a db file? | 15:23 |
| spatel | imaximets am i missing this step? - ovsdb-tool create-cluster /etc/openvswitch/ovnsb_db.db | 15:25 |
| spatel | i didn't use create-cluster command | 15:25 |
| spatel | may be that is my issue.. | 15:25 |
| imaximets | spatel, ovsdb-tool join-cluster /var/lib/ovn/ovnsb_db.db /usr/share/ovn/ovn-nb.ovsschema tcp:<local-ip>:6643 tcp:<remote-ip>:6643 | 15:26 |
| spatel | let me try | 15:26 |
| imaximets | spatel, in theory, I think, you don't need to run any ovsdb-tool commands. But I don't know what is going on. | 15:27 |
| imaximets | spatel, oops i messed up nb and sb. | 15:28 |
| imaximets | ovsdb-tool join-cluster /var/lib/ovn/ovnnb_db.db /usr/share/ovn/ovn-nb.ovsschema tcp:<local-ip>:6643 tcp:<remote-ip>:6643 | 15:28 |
| spatel | :) | 15:29 |
| imaximets | ovsdb-tool join-cluster /var/lib/ovn/ovnsb_db.db /usr/share/ovn/ovn-sb.ovsschema tcp:<local-ip>:6644 tcp:<remote-ip>:6644 | 15:29 |
| spatel | i got error then i realized | 15:29 |
| spatel | node-2# ovsdb-tool join-cluster /var/lib/ovn/ovnnb_db.db /usr/share/ovn/ovn-nb.ovsschema tcp:172.30.40.25:6643 tcp:172.30.40.93:6643 | 15:29 |
| spatel | ovsdb-tool: /usr/share/ovn/ovn-nb.ovsschema: not a valid schema name (use "schema-name" command to find the correct name) | 15:29 |
| imaximets | spatel, delete databases first. | 15:30 |
| spatel | oh ok | 15:30 |
| spatel | rm /var/lib/ovn/ovnnb_db.db | 15:30 |
| spatel | still getting same error | 15:30 |
| spatel | not a valid schema name | 15:30 |
| imaximets | ok. I messed this up too. :) | 15:31 |
| imaximets | ovsdb-tool create-cluster /var/lib/ovn/ovnnb_db.db /usr/share/ovn/ovn-nb.ovsschema tcp:<local-ip>:6643 | 15:32 |
| imaximets | ovsdb-tool join-cluster /var/lib/ovn/ovnnb_db.db OVN_Northbound tcp:<local-ip>:6643 tcp:<remote-ip>:6643 | 15:32 |
| *** psahoo has quit IRC | 15:32 | |
| spatel | doing it | 15:32 |
| imaximets | or not | 15:32 |
| *** jangutter has quit IRC | 15:33 | |
| * imaximets needs some coffee and rest... | 15:33 | |
| spatel | +1 | 15:33 |
| *** jangutter has joined #openvswitch | 15:33 | |
| spatel | imaximets look at this - http://paste.openstack.org/show/805555/ | 15:34 |
| *** eelco has quit IRC | 15:35 | |
| imaximets | spatel, that is fine, but you don't need create-cluster command. :) | 15:35 |
| imaximets | spatel, delete databases on nodes 2 and 3 and execute: | 15:35 |
| imaximets | ovsdb-tool join-cluster /var/lib/ovn/ovnnb_db.db OVN_Northbound tcp:<local-ip>:6643 tcp:<remote-ip>:6643 | 15:36 |
| spatel | ok | 15:36 |
| imaximets | ovsdb-tool join-cluster /var/lib/ovn/ovnsb_db.db OVN_Southbound tcp:<local-ip>:6644 tcp:<remote-ip>:6644 | 15:36 |
| *** jangutter_ has joined #openvswitch | 15:36 | |
| spatel | ovsdb-tool join-cluster /var/lib/ovn/ovnnb_db.db OVN_Northbound tcp:172.30.40.25:6643 tcp:172.30.40.93:6643 | 15:37 |
| spatel | no error | 15:37 |
| spatel | doing it on sb now | 15:37 |
| imaximets | cool | 15:37 |
| spatel | no error | 15:38 |
| spatel | both command ran without errors | 15:38 |
| imaximets | let's try to run ovn-ctl | 15:38 |
| spatel | ovn-ctl ... whole giant command ? | 15:39 |
| spatel | /usr/share/ovn/scripts/ovn-ctl --db-nb-addr=172.30.40.25 --db-nb-create-insecure-remote=yes --db-sb-addr=172.30.40.25 --db-sb-create-insecure-remote=yes --db-nb-cluster-local-addr=172.30.40.25 --db-sb-cluster-local-addr=172.30.40.25 --db-nb-cluster-remote-addr=172.30.40.93 --db-sb-cluster-remote-addr=172.30.40.93 --ovn-northd-nb-db=tcp:172.30.40.93:6641,tcp:172.30.40.25:6641,tcp:172.30.40.177:6641 --ovn-northd-sb-db=tcp:172.30.40 | 15:39 |
| spatel | .93:6642,tcp:172.30.40.25:6642,tcp:172.30.40.177:6642 start_northd | 15:39 |
| *** jangutter has quit IRC | 15:40 | |
| imaximets | yep | 15:40 |
| spatel | ok | 15:40 |
| spatel | done.. let me check status now | 15:40 |
| spatel | ovs-appctl -t /var/run/ovn/ovnnb_db.ctl cluster/status OVN_Northbound | 15:41 |
| spatel | still give same error | 15:41 |
| spatel | node-2# ovsdb-tool check-cluster /var/lib/ovn/ovnnb_db.db | 15:41 |
| spatel | ovsdb-tool: all logs are empty | 15:41 |
| imaximets | ps -aux | grep ovsdb-server ? | 15:42 |
| spatel | http://paste.openstack.org/show/805556/ | 15:42 |
| spatel | ovsdb-server procs - http://paste.openstack.org/show/805557/ | 15:43 |
| imaximets | cat /var/lib/ovn/ovnnb_db.db ? | 15:44 |
| spatel | ok | 15:45 |
| spatel | http://paste.openstack.org/show/805558/ | 15:45 |
| spatel | look like empty DB | 15:46 |
| imaximets | yes, it didn't try to connect to the cluster... | 15:47 |
| spatel | Yes i am running tcpdump on node-1 and not seeing any single packet hit node-1 | 15:47 |
| spatel | very odd | 15:47 |
| imaximets | this should be something really stupid, but I don't understand what it is. | 15:48 |
| spatel | do i need to restart any services etc.. or may be reboot whole box to make it fresh | 15:48 |
| spatel | its very odd its not attempting to connect node-1 | 15:48 |
| spatel | # /usr/share/ovn/scripts/ovn-ctl command doesn't have good debug option also | 15:49 |
| *** anilvenk_ has quit IRC | 15:50 | |
| spatel | oh wait.. i saw some packets on node-1 when i run ovn-ctl (sorry my filter was wrong..) | 15:50 |
| spatel | let me check pcap | 15:50 |
| imaximets | spatel, could you add --ovn-nb-log="-vfile:dbg" ? | 15:51 |
| spatel | let me try that | 15:51 |
| spatel | look at this ngrep / tcpdump output look like node-1 sending all data to node-2 - http://paste.openstack.org/show/805559/ | 15:53 |
| imaximets | spatel, it's a northd from node-2 connects to database on node-1. | 15:55 |
| spatel | not seeing anything good using this command - --ovn-nb-log="-vfile:dbg | 15:55 |
| imaximets | spatel, there should be more logs in a log file | 15:55 |
| spatel | look like node-2 try to attempt node-1 but then no progress, still same error.. | 15:56 |
| spatel | my logs file looks same like before i didn't see any debug logs etc.. | 15:56 |
| imaximets | in /var/log/ovn/ovsdb-server-nb.log ? | 15:56 |
| spatel | same logs.. | 15:57 |
| *** djbalrog has quit IRC | 15:57 | |
| spatel | no extra details.. | 15:57 |
| imaximets | hmm | 15:57 |
| spatel | what is this file.. /var/lib/ovn/ovnnb_db.db.backup5.20.0-987891875 ? | 15:58 |
| spatel | i am seeing lots of data in that file | 15:58 |
| *** elvira has quit IRC | 15:58 | |
| imaximets | spatel, if database file exists before starting ovsdb-process ovn-ctl makes a backup. | 15:58 |
| spatel | oh | 15:59 |
| *** djbalrog has joined #openvswitch | 16:00 | |
| imaximets | spatel, the ultimate option is to open ovn-ctl script and add 'set -x' somewhere at the top... | 16:00 |
| spatel | oh! i like that | 16:00 |
| spatel | look like we ran out of options here.. | 16:06 |
| spatel | i think time to read some documentation | 16:06 |
| spatel | 16:07 | |
| imaximets | the problem is that according to documentation your setup should work. :) | 16:07 |
| spatel | yes yes.. | 16:07 |
| spatel | something is missing here... | 16:07 |
| spatel | even my stupid debug is not working otherwise we can see some magic in logs | 16:10 |
| imaximets | spatel, could you try to just run ovsdb-server directly? i.e. ovsdb-server -vconsole:dbg -vfile:info --log-file=/var/log/ovn/ovsdb-server-nb.log --remote=punix:/var/run/ovn/ovnnb_db.sock --pidfile=/var/run/ovn/ovnnb_db.pid --unixctl=/var/run/ovn/ovnnb_db.ctl --remote=db:OVN_Northbound,NB_Global,connections --private-key=db:OVN_Northbound,SSL,private_key --certificate=db:OVN_Northbound,SSL,certificate --ca-cert=db:OVN_Northbou | 16:12 |
| imaximets | nd,SSL,ca_cert --ssl-protocols=db:OVN_Northbound,SSL,ssl_protocols --ssl-ciphers=db:OVN_Northbound,SSL,ssl_ciphers --remote=ptcp:6641:172.30.40.25 /var/lib/ovn/ovnnb_db.db | 16:12 |
| *** djbalrog has quit IRC | 16:12 | |
| spatel | let me try | 16:12 |
| spatel | should i stop ovsdb-server or kill pid first? | 16:13 |
| *** djbalrog has joined #openvswitch | 16:13 | |
| imaximets | stop everything | 16:13 |
| spatel | ok | 16:13 |
| imaximets | spatel, except for node-1. :) | 16:14 |
| spatel | yeah :)_ | 16:14 |
| spatel | i can see logs are flowing on terminal | 16:15 |
| imaximets | it's because of -vconsole:dbg :) | 16:16 |
| spatel | let me open other terminal and run nbctl | 16:16 |
| imaximets | is there something about raft? | 16:16 |
| spatel | i can see its chatting with node-1 | 16:16 |
| spatel | heartbeat... | 16:16 |
| imaximets | ok. good. cluster/status ? | 16:17 |
| spatel | :) | 16:17 |
| spatel | look like i see lots of data in cat /var/lib/ovn/ovnnb_db.db | 16:17 |
| spatel | let me run that command | 16:17 |
| spatel | imaximets look at that what did you do :) - http://paste.openstack.org/show/805562/ | 16:19 |
| imaximets | ok. it's working. | 16:19 |
| imaximets | but I have no idea what is wrong with ovn-ctl. | 16:19 |
| spatel | hmm on node-2 what is this Vote: unknown ? | 16:20 |
| imaximets | spatel, it's a raft leadership election. Term is 1 and that means that there was no elections yet, so node-2 never voted. | 16:21 |
| spatel | my ovn-northd service is still down.. | 16:22 |
| spatel | i didn't run this command yet - /usr/share/ovn/scripts/ovn-ctl | 16:22 |
| spatel | should i run whole command - /usr/share/ovn/scripts/ovn-ctl..... | 16:22 |
| imaximets | you may try to run northd only by passing --ovn-manage-ovsdb=no | 16:23 |
| imaximets | in this case you, probably, don't need half of other arguments... | 16:24 |
| imaximets | I don't understand what is going on with ovn-ctl, so I can't guarantee anything. :) | 16:25 |
| spatel | you are saying in whole /usr/share/ovn/scripts/ovn-ctl command just add --ovn-manage-ovsdb=no right? | 16:25 |
| imaximets | yes. | 16:26 |
| *** blahdodo has quit IRC | 16:26 | |
| spatel | done.. | 16:26 |
| spatel | node-2# ovn-nbctl show | 16:26 |
| spatel | ovn-nbctl: unix:/var/run/ovn/ovnnb_db.sock: database connection failed () | 16:26 |
| spatel | why i am getting that error? | 16:26 |
| spatel | i think when replication start i should be able to run that command on all 3 nodes right? | 16:27 |
| imaximets | you should be able. Is the ovsdb-server still running on that node? | 16:28 |
| spatel | yes.. whatever command you gave me same still running | 16:28 |
| imaximets | try: ovn-nbctl -vdbg show | 16:29 |
| *** blahdodo has joined #openvswitch | 16:30 | |
| spatel | check this out - http://paste.openstack.org/show/805564/ | 16:30 |
| imaximets | spatel, ok, it wants to talk only with the cluster leader. try adding --no-leader-only option. | 16:31 |
| spatel | look like they didn't selected leader | 16:31 |
| spatel | in /usr/share/ovn/scripts/ovn-ctl ? | 16:31 |
| imaximets | nope, in ovn-nbctl | 16:32 |
| spatel | ok | 16:32 |
| imaximets | ovn-nbctl --no-leader-only show | 16:32 |
| spatel | that works :) | 16:32 |
| spatel | phew! | 16:32 |
| spatel | can you tell me what is going on why these two nodes didn't selected leader yet? | 16:33 |
| spatel | look like they can see each other but still waiting for something.. (should i bring node-3 in?) | 16:33 |
| imaximets | spatel, they have a leader. but leader is on node-1 while you're requesting data from server on node-2 directly. | 16:35 |
| imaximets | ovn-nbctl --db=tcp:172.30.40.93:6641,tcp:172.30.40.25:6641,tcp:172.30.40.177:6641 show | 16:35 |
| spatel | oh wait.. so this cluster isn't active/active? | 16:35 |
| imaximets | it is, but ovn-nbctl wants to talk only to leader in order to avoid some possible issues with "write" transactions. | 16:36 |
| imaximets | "When a client conducts a mix of read and write transactions across more than one server in a cluster, it can see inconsistent results because a read transaction might read stale data whose updates have not yet propagated from the leader. By default, utilities such as ovn-sbctl (in OVN) connect to the cluster leader to avoid this issue." | 16:37 |
| imaximets | https://docs.openvswitch.org/en/latest/ref/ovsdb.7/ | 16:37 |
| imaximets | ovn-controller's will connect to any server, they don't need to talk only with a leader. | 16:38 |
| spatel | hmm | 16:39 |
| spatel | just like mysql galera cluster... all master but better to write only single node | 16:39 |
| spatel | Status: cluster member | 16:41 |
| spatel | Role: follower | 16:41 |
| spatel | Term: 1 | 16:41 |
| spatel | Leader: 6b6e | 16:41 |
| spatel | Vote: unknown | 16:41 |
| spatel | when my node-1 go down assuming node-2 will be leader right? | 16:42 |
| imaximets | 2 node cluster is not ideal. You need odd number of servers to have a good behavior. | 16:44 |
| spatel | let me bring up 3rd node | 16:45 |
| *** istokes has quit IRC | 16:56 | |
| spatel | imaximets i have added 3rd node in | 16:59 |
| spatel | http://paste.openstack.org/show/805565/ | 17:00 |
| imaximets | spatel, ok. if you'll bring node-1 down, one of other nodes should become a leader. | 17:00 |
| spatel | i can see 3 entries in Server: section | 17:01 |
| spatel | let me do that | 17:01 |
| spatel | just kill ovn-northd PID is enough? | 17:01 |
| imaximets | ovsdb-server, not northd. | 17:01 |
| *** djbalrog has quit IRC | 17:01 | |
| spatel | ohhh | 17:01 |
| spatel | let me shutdown ovsdb-server | 17:02 |
| spatel | or i can do iptables -j DROP all connection :) | 17:02 |
| imaximets | that's an option. :) | 17:02 |
| spatel | hmm systemctl stop ovsdb-server.service no impact still seeing PID in proc table | 17:04 |
| *** djbalrog has joined #openvswitch | 17:04 | |
| spatel | may be need to kill it by -9 | 17:04 |
| spatel | hey!!! node-3 is leader now :) | 17:05 |
| imaximets | cool. | 17:05 |
| spatel | Term: 2 | 17:05 |
| spatel | Term is number of nodes? | 17:05 |
| imaximets | no, term is measure of "time". It's kind of a stage for raft. it got incremented due to election. Every election increments term. | 17:07 |
| spatel | explain me why i need to start ovsdb-server using your command ? | 17:07 |
| imaximets | spatel, because ovn-ctl doesn't work for unknown reason. :) It shouldn't be necessary to start ovsdb-server like that. | 17:08 |
| spatel | how do i handle this? | 17:08 |
| spatel | very odd | 17:08 |
| imaximets | need to debug what is actually going on in ovn-ctl. Did 'set -x' show anything interesting? | 17:10 |
| spatel | very interesting now i reboot node-3 and everything came back clean... | 17:10 |
| spatel | look like just one time process to start ovsdb-server with your command line option and then everything looks good, even after reboot it works | 17:11 |
| spatel | I tried set -x but it had massive output so i may try to read line by line to understand whole process | 17:12 |
| imaximets | hmm. it started automatically after reboot and works? | 17:12 |
| spatel | Yes | 17:12 |
| imaximets | weird. | 17:12 |
| spatel | i reboot all 3 nodes one by one and all came back clean and join cluster automatically | 17:12 |
| imaximets | hmm. ok. | 17:12 |
| spatel | that is why i am surprised.. | 17:12 |
| spatel | look like your command help it to setup cluster database and now it has DB so it automatically joining cluster | 17:13 |
| imaximets | very weird. | 17:14 |
| imaximets | the problem is that my commad is just a copy of the 'ps -aux' that you showed my with a slight logging modifications. | 17:14 |
| imaximets | s/my/me | 17:14 |
| spatel | AH!!! hehehe | 17:14 |
| spatel | i think i need to destroy everything and rebuild it again.. | 17:15 |
| spatel | we should go bottom of this.. | 17:15 |
| imaximets | makes sense. | 17:15 |
| spatel | imaximets Thank you so much!! without your help i was lost :) | 17:16 |
| spatel | we should turn this IRC chat into blog to help someone out :) | 17:16 |
| imaximets | spatel, no problem. Have fun. :) | 17:16 |
| spatel | I will blog this out because it has lot to chew!! thank you again.. i will keep you posted on my progress | 17:17 |
| imaximets | ok | 17:17 |
| * imaximets goes afk for today. | 17:17 | |
| spatel | have a good weekend :) | 17:17 |
| *** djbalrog has quit IRC | 17:37 | |
| *** mdgray has quit IRC | 17:38 | |
| *** djbalrog has joined #openvswitch | 17:39 | |
| *** djbalrog has quit IRC | 17:51 | |
| *** djbalrog has joined #openvswitch | 17:53 | |
| *** ktraynor has quit IRC | 18:10 | |
| *** yamamoto has joined #openvswitch | 18:44 | |
| *** acidfu_ has quit IRC | 18:44 | |
| *** yamamoto has quit IRC | 18:52 | |
| *** ralonsoh has quit IRC | 19:49 | |
| *** djbalrog has quit IRC | 20:07 | |
| *** djbalrog has joined #openvswitch | 20:09 | |
| *** djbalrog has quit IRC | 20:21 | |
| *** djbalrog has joined #openvswitch | 20:24 | |
| *** djbalrog has quit IRC | 20:57 | |
| *** fbl_ has joined #openvswitch | 20:58 | |
| *** djbalrog has joined #openvswitch | 20:59 | |
| *** fbl has quit IRC | 21:00 | |
| *** dcbw has quit IRC | 21:01 | |
| *** thaller_ has joined #openvswitch | 21:15 | |
| *** mmichelson_ has joined #openvswitch | 21:15 | |
| *** mmichelson has quit IRC | 21:15 | |
| *** thaller has quit IRC | 21:15 | |
| *** admin0 has joined #openvswitch | 21:24 | |
| *** bostondriver has quit IRC | 21:37 | |
| *** djbalrog has quit IRC | 22:01 | |
| *** djbalrog has joined #openvswitch | 22:02 | |
| *** spatel has quit IRC | 22:10 | |
| *** yamamoto has joined #openvswitch | 22:30 | |
| *** djbalrog has quit IRC | 22:36 | |
| *** djbalrog has joined #openvswitch | 22:38 | |
| *** donhw has quit IRC | 23:03 | |
| *** donhw has joined #openvswitch | 23:19 | |
| *** yamamoto has quit IRC | 23:24 | |
| *** djbalrog has quit IRC | 23:41 | |
| *** djbalrog has joined #openvswitch | 23:44 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!