Thursday, 2016-02-11

*** armax has joined #openstack-neutron-ovn00:10
*** thumpba has joined #openstack-neutron-ovn00:13
*** thumpba has quit IRC00:17
*** aginwala has quit IRC00:42
*** aginwala has joined #openstack-neutron-ovn00:45
*** aginwala has quit IRC00:46
*** aginwala has joined #openstack-neutron-ovn00:46
*** gangil1 has joined #openstack-neutron-ovn00:47
*** gangil has quit IRC00:48
*** gangil1 has quit IRC00:48
*** gangil has joined #openstack-neutron-ovn00:48
*** gangil has joined #openstack-neutron-ovn00:48
*** aginwala has quit IRC01:13
*** aginwala has joined #openstack-neutron-ovn01:20
*** aginwala has quit IRC01:35
*** azbiswas has quit IRC01:39
*** azbiswas has joined #openstack-neutron-ovn01:39
*** azbiswas has quit IRC01:43
*** azbiswas has joined #openstack-neutron-ovn01:57
*** gangil has quit IRC02:07
*** azbiswas_ has joined #openstack-neutron-ovn02:08
*** terryw has joined #openstack-neutron-ovn02:14
*** azbiswas has quit IRC02:15
*** otherwiseguy has quit IRC02:15
*** openstackgerrit has quit IRC02:15
*** openstackgerrit has joined #openstack-neutron-ovn02:24
*** armax has quit IRC02:43
*** jckasper has joined #openstack-neutron-ovn02:53
*** jckasper has quit IRC03:11
*** jckasper has joined #openstack-neutron-ovn03:11
*** jckasper has quit IRC03:15
*** azbiswas_ has quit IRC03:52
*** azbiswas has joined #openstack-neutron-ovn03:53
*** azbiswas has quit IRC03:57
*** azbiswas has joined #openstack-neutron-ovn04:04
*** aginwala has joined #openstack-neutron-ovn04:04
*** jckasper has joined #openstack-neutron-ovn04:30
*** armax has joined #openstack-neutron-ovn04:34
*** aginwala has quit IRC04:34
*** jckasper has quit IRC04:36
*** jckasper has joined #openstack-neutron-ovn04:36
*** jckasper has quit IRC04:40
*** salv-orl_ has joined #openstack-neutron-ovn04:41
*** jckasper has joined #openstack-neutron-ovn04:44
*** salv-orlando has quit IRC04:44
*** jckasper has quit IRC04:45
*** jckasper has joined #openstack-neutron-ovn04:45
*** jckasper has quit IRC04:50
*** armax has quit IRC04:51
*** armax has joined #openstack-neutron-ovn05:05
*** aginwala has joined #openstack-neutron-ovn05:08
*** aginwala has quit IRC05:12
openstackgerritBabu Shanmugam proposed openstack/networking-ovn: DPDK support for OVN
*** armax has quit IRC05:37
*** allan_h has quit IRC05:56
*** aginwala has joined #openstack-neutron-ovn07:02
*** numans has quit IRC07:48
openstackgerritBabu Shanmugam proposed openstack/networking-ovn: Enabling qos support through Logical_Port.options
*** nate_gone has joined #openstack-neutron-ovn08:07
*** nate_gone is now known as njohnston08:08
*** aginwala_ has joined #openstack-neutron-ovn08:09
*** aginwala has quit IRC08:11
*** palexster has joined #openstack-neutron-ovn08:11
*** palexster has quit IRC08:12
*** palexster has joined #openstack-neutron-ovn08:12
*** jckasper has joined #openstack-neutron-ovn08:15
*** jckasper has quit IRC08:20
*** aginwala_ has quit IRC08:43
*** azbiswas has quit IRC08:44
*** aginwala has joined #openstack-neutron-ovn08:44
*** openstackgerrit has quit IRC08:47
*** openstackgerrit_ has joined #openstack-neutron-ovn08:47
*** openstackgerrit_ is now known as openstackgerrit08:48
*** numans has joined #openstack-neutron-ovn08:48
*** aginwala has quit IRC08:49
*** jckasper has joined #openstack-neutron-ovn09:11
*** jckasper has quit IRC09:16
*** yamamoto has quit IRC10:10
*** yamamoto has joined #openstack-neutron-ovn10:10
*** yamamoto has quit IRC10:38
*** salv-orlando has joined #openstack-neutron-ovn10:42
*** azbiswas has joined #openstack-neutron-ovn10:44
*** salv-orl_ has quit IRC10:45
*** roeyc has joined #openstack-neutron-ovn10:48
*** yamamoto has joined #openstack-neutron-ovn10:49
*** yamamoto has quit IRC10:49
*** azbiswas has quit IRC10:50
openstackgerritNuman Siddique proposed openstack/networking-ovn: Neutron ovn northbound db sync tool
*** jckasper has joined #openstack-neutron-ovn11:24
*** jckasper has quit IRC11:29
*** _Mic22 has joined #openstack-neutron-ovn11:30
*** Mic22 has quit IRC11:32
*** Mic22 has joined #openstack-neutron-ovn11:38
*** _Mic22 has quit IRC11:41
*** yamamoto has joined #openstack-neutron-ovn11:50
*** yamamoto has quit IRC11:58
*** ig0r_ has joined #openstack-neutron-ovn12:12
*** rtheis has joined #openstack-neutron-ovn12:38
*** azbiswas has joined #openstack-neutron-ovn12:47
*** azbiswas has quit IRC12:51
*** jckasper has joined #openstack-neutron-ovn13:07
*** jckasper has quit IRC13:11
*** jckasper has joined #openstack-neutron-ovn13:11
*** jckasper has quit IRC13:15
*** jckasper has joined #openstack-neutron-ovn13:48
*** brad_behle has joined #openstack-neutron-ovn14:02
*** palexster1 has joined #openstack-neutron-ovn14:14
*** palexster has quit IRC14:14
*** palexster1 is now known as palexster14:14
Sam-I-Amrussellb: moooorning!14:15
Sam-I-Amrussellb: up for a philosophical discussion on routing in ovn?14:30
*** allan_h has joined #openstack-neutron-ovn14:53
openstackgerritRyan Moats proposed openstack/networking-ovn: (WIP) Support port security API extension
openstackgerritRyan Moats proposed openstack/networking-ovn: (WIP) Support Allowed address pairs
*** ig0r_ has quit IRC15:21
*** azbiswas has joined #openstack-neutron-ovn15:33
azbiswasrussellb: Have you ever encountered "referential integrity violation" in the ovsdb.15:44
Sam-I-Amazbiswas: i've seen it in the gate15:44
azbiswasSam-I-Am: Thanks, do we know the reason behind it - Is it multiple api workers trying to modify the DB at the same time?15:45
Sam-I-Amdont know15:46
Sam-I-Amdo we set workers >1 by default?15:46
azbiswasIn devstack yes - not sure what the gate does15:46
azbiswasI'm not sure if the api workers is the reason behind it either.15:47
Sam-I-Amdefinitely needs some invesigation. i dont see it in the neutron/ml2 jobs.15:47
azbiswasIn our situation, there was 2 api workers operating on the same Logical Switch/ACL association at the same time. The 1st api worker was able to commit its changes correctly, the 2nd api worker failed.15:49
russellbsounds like an expected type of error when there are multiple workers15:54
russellbwe need to be able to handle those errors, it means we need to redo the transaction using the updated db state15:55
*** njohnston has quit IRC15:55
*** HenryG has quit IRC15:55
Sam-I-Ami wonder what the default # of workers is15:55
Sam-I-Ami know increasing rpc workers creates a mess15:55
russellbit's > 115:55
Sam-I-Amis it >1 for neutron/ml2 too?15:55
Sam-I-Ami thought it was based on number of cpus or something15:55
russellbit has changed a few times15:56
Sam-I-Amsurprise :)15:56
russellbneutron defaults to # of CPUs15:57
russellbthe gate cuts that back15:57
russellbbut it's still multiple15:57
azbiswasNeutron retries the transaction if the error returned is RETRY15:57
azbiswasbut in this case it was error15:58
russellboh, i see15:58
russellbcan you send the full error?15:58
azbiswas"details":"Table Logical_Switch column acls row fa55edc2-72de-4d02-8ccc-19f10be2dc1e references nonexistent row 036dc043-f434-48a3-b767-9b68653f3fa3 in table ACL.","error":"referential integrity violation"}15:58
azbiswasThere were 2 create ports on the same provider network at the same time on 2 different api workers16:00
azbiswasThere were 4 existing ports on the provider networks, say A, B, C, D and api worker 1 is adding port X, api worker 2 is adding port Y.16:01
*** HenryG has joined #openstack-neutron-ovn16:02
russellbi see...16:03
azbiswasapi worker 1 tries to commit [DelACL A, AddACL A, DelACL B, AddACL B, DelACL C, AddACL C, DelACL D, AddACL D, DelACL X, AddACL X]16:03
azbiswasand succeeds16:03
azbiswasapi worker 2 tries to commit [DelACL A, AddACL A, DelACL B, AddACL B, DelACL C, AddACL C, DelACL D, AddACL D, DelACL Y, AddACL Y]16:03
azbiswasand fails16:03
azbiswassince api worker doesn't know about X as yet, it doesn't operate on X - that's a separate problem.16:04
azbiswasor maybe the same?16:04
* russellb reading code, not ignoring16:05
russellboops i have a meeting that started 5 minutes ago16:05
azbiswasWe can discuss later - The nonexistent ACL row was deleted by api worker 1 but MAY have been contained in the LSwitch ACL list transaction of api worker 2 - possibly.16:06
russellbyes that would explain it16:11
russellband if that's it i think i know the fix16:11
russellbwe should be calling verify()16:13
azbiswasI'm still trying to wrap my head around why the 2nd api worker LSwitch ACL list would contain an old ACL, they are both deleting the existing A,B,C,D ACLs16:13
russellbto make sure that when we update a column, it previously had the values we think it did16:13
azbiswaswe are calling lswitch.verify() after each add/delete of ACL16:14
russellbhm ok16:14
azbiswasSo each api worker sees its own version of lswitch = idlutils.row_by_value(self.api.idl, 'Logical_Switch', ..)?16:19
*** roeyc has quit IRC16:23
*** thumpba has joined #openstack-neutron-ovn16:26
*** flaviof has quit IRC16:38
*** lrichard has quit IRC16:39
*** lrichard has joined #openstack-neutron-ovn16:39
*** armax has joined #openstack-neutron-ovn16:41
*** salv-orl_ has joined #openstack-neutron-ovn16:41
*** salv-orlando has quit IRC16:44
*** flaviof has joined #openstack-neutron-ovn16:53
*** numans has quit IRC17:13
Sam-I-Amrussellb: moo?17:20
*** regXboi has joined #openstack-neutron-ovn17:24
*** manand has joined #openstack-neutron-ovn17:24
*** manand has quit IRC17:25
russellbazbiswas: yes, each worker has its own local copy of the db, when it does row_by_value(), it's from local cache17:42
russellbverify() is supposed to before saying "before i update this column, make sure it still has the previous value i thought it had"17:42
russellbyou said we're calling verify() after each add/delete, that might be wrong17:43
russellbafter instead of before i mean17:43
russellbi'll have to dig into the code some more17:43
russellbcan someone file a bug about this?17:43
russellbSam-I-Am: hi17:43
Sam-I-Amrussellb: hey17:44
azbiswasI'll file the bug, the verify() is called at each run_idl() call for AddACL and DelACL17:44
russellbin any case, it definitely sounds like we've missed something in proper transaction handling17:45
Sam-I-Amrussellb: i had a long debate with myself yesterday about routing... particularly the provider-private/nat/fip bits. maybe you have some time to discuss at some point?17:46
russellbsure, though i don't know what our plans are for that yet17:47
Sam-I-Amwell, the concern is, do we really NEED plans17:47
Sam-I-Amhence the philosophical part :)17:47
russellbazbiswas and chandrav did a fantastic job on a floating IP proposal (that i've been bad about not responding to yet)17:47
russellbdid it get solved in my sleep?17:48
* azbiswas perks up about floating IP discussion :)17:48
*** igordcard has joined #openstack-neutron-ovn17:48
Sam-I-Amwell, i'm kind of wondering... is the concept of floating ips flawed and just something we've all gotten used to having?17:48
russellbsome people claim that17:50
russellbhowever, it's a well established thing that we're expected to have17:50
Sam-I-Amseems like its there to work around the limits of ipv4 addresses17:50
Sam-I-Amthere is no floating ip for v617:50
Sam-I-Amnat is a hack17:50
russellbare we doing ipv6 only?17:50
russellb(no :-p)17:50
russellbi'm not sure we can get away with *not* supporting nat and floating ips17:51
russellbit's the most common setup17:51
Sam-I-Amwell, maybe, maybe not17:51
Sam-I-Amfor a long time, it was assumed that The Way with neutron was a provider network, private network(s), routers, and floating IPs for vms that need access from the outside world17:52
Sam-I-Ampeople quickly found out the limitations of the network node... a performance bottleneck and single point of failure17:53
* russellb nods17:53
Sam-I-Amalong came the networking guide which outlined the ability to attach vms directly to provider nets, or use a hybrid approach17:53
Sam-I-Amfor the hybrid approach, vms that need to be public have an interface on a provider network, and optionally a private network that may or may not allow snat.17:54
russellbscaling that well really needs "routed networks"17:55
Sam-I-Amdefine routed networks17:55
Sam-I-Amyou mean rather than snatted?17:55
russellbi mean the neutron routed networks spec17:56
Sam-I-Amoh, ehhh....17:57
russellbpart of what you're getting at kind of sounds like this existential debate about how much we want virtual networking in the first place17:57
Sam-I-Amwell, i was hoping that wasnt the case...17:57
russellbmaybe it's not17:58
russellbwith both floating IPs and direct plugging to the public network, we currently assume all IPs are valid on every hypervisor17:59
russellbin larger deployments, that starts to not be great, as i understand it17:59
Sam-I-Amthings like dvr go to great extents to preserve the conventional floating ip logic more or less using layer 2. sure, each compute node can respond to floating IPs for vms that reside on it, but at the expense of complexity and resiliency18:00
russellbgateways help in one way (limit the scope of what nodes need this)18:00
russellbthe other way is to expose more info about the scalable L3 fabric directly to the compute hosts (that's where routed networks comes in)18:00
Sam-I-Amdvr also doesnt solve snat for fixed ips, because what it does is broken.18:00
Sam-I-Amwell, thats sort of where i was going18:01
Sam-I-Amconventional networks either use mct, vrrp, or rely on routing protocols to handle ecmp and failover18:02
Sam-I-Amneutron l3ha is vrrp, and that's all good. its well understood and works. with the proper scheduling of routers, it scales better, but still not going to match hardware routers/nat18:03
azbiswasrussellb: I gotta run, will be following the ovn meeting in read only mode, but I noticed that in the neutron code the verify is done before update while in the networking-ovn code the verify is done after the update.18:05
azbiswas        row.external_ids = {'neutron:lport': self.lport}18:05
azbiswas        acls = getattr(lswitch, 'acls', [])18:05
azbiswas        acls.append(row.uuid)18:05
azbiswas        setattr(lswitch, 'acls', acls)18:05
azbiswas        lswitch.verify('acls')18:05
russellbok, i think we have that backwards18:05
azbiswasi'll try out the change and see if it makes a difference18:06
russellbazbiswas: great thank you!!18:06
russellbhopefully that's all it is ...18:06
* azbiswas nods18:06
*** azbiswas has quit IRC18:06
russellb(OVN meeting in 10 minutes)18:06
Sam-I-Amrussellb: i think what i'm getting at is if we're going to do routing in neutron, it needs to be REAL routing, not some hacky layer2 stuff18:07
Sam-I-Ami was sort of getting the feeling that the nat/fip spec for ovn looked too much like dvr18:07
russellbfrom a high level yes18:07
russellbimplemented quite differently18:07
russellbbut yeah.18:07
Sam-I-Amovn/ovs is already overwhelming for most operators to comprehend, hence why linuxbridge is gaining popularity18:08
Sam-I-Amdvr adds significant complexity that arguably overshadows what it aims to solve18:08
Sam-I-Ami'm afraid that we're going to implement something thats fundamentally broken just to say we have it rather than looking at the bigger picture of what routing in neutron means18:09
*** gangil has joined #openstack-neutron-ovn18:11
*** gangil has joined #openstack-neutron-ovn18:11
russellbright now we just have the east-west part ... as for north/south, the picture is totally unclear to me until there's a proposal for gateways18:12
russellbi hope you can chime in on that, because it sounds like you have a better idea than i do18:12
Sam-I-Amyeah... east-west that doesnt rely on a single node is one of the two things DVR does18:13
russellbthat part seems sane18:13
Sam-I-Amyeah, it makes sense18:13
*** aginwala has joined #openstack-neutron-ovn18:14
Sam-I-Amwhat doesnt make sense is n-s with fips18:14
russellbok, it'd be good to express that view on the floating IP proposal on the ovs dev mailing list18:14
russellb(meeting time)18:15
*** zhouhan has joined #openstack-neutron-ovn18:15
Sam-I-Ami'm trying to make sure i dont sound completely silly. just after a long internal debate, it seemed like we're trying to implement fips because they've just been there.18:15
Sam-I-Amrussellb: yeah, catch you on the other side18:15
Sam-I-Ammestery: i'd also like to get your opinion on these things18:16
*** azbiswas_ has joined #openstack-neutron-ovn18:18
*** chandrav has joined #openstack-neutron-ovn18:21
*** numans has joined #openstack-neutron-ovn18:48
*** arosen12 has left #openstack-neutron-ovn18:54
*** arosen12 has joined #openstack-neutron-ovn18:54
*** ig0r_ has joined #openstack-neutron-ovn18:58
*** ig0r_ has quit IRC19:03
*** azbiswas_ has quit IRC19:04
*** ig0r_ has joined #openstack-neutron-ovn19:08
*** jckasper has quit IRC19:15
*** jckasper has joined #openstack-neutron-ovn19:17
russellbanyone interested in testing this to see if you can reproduce it?
openstackLaunchpad bug 1543795 in networking-ovn "Ping through 2 routers is not working" [Undecided,New]19:19
*** numans has quit IRC19:19
mesteryregXboi: I can try that at the top of the hour19:27
mesteryI have a meeting in 3 minutes19:27
mesteryrussellb not regXboi19:27
chandravrussellb: Is that a valid configuration ?19:27
russellbi think there's a typo in the report19:27
russellb(same address on networks 2 and 3)19:28
russellbassuming that's a typo, seems like it should be valid19:28
chandravHow can network 2 be connected to 2 routers19:28
chandravit has only 1 gateway address (
russellbah, well that's a good point :)19:29
chandravfor a given network there can be only 1 default gateway. I doubt if openstack allows such a configuration19:30
russellbthanks chandrav, i guess i hadn't really thought through what they were doing19:30
chandravi am seeing a problem with provider networks with the latest tip of the tree. the ports connecting the br-provider and br-int do not show up. anyone else seen this issue yet ?19:36
russellbchandrav: it might be a change in behavior you didn't know about19:38
russellbchandrav: the patch ports get created later, once you create a port on the provider network19:39
russellband there's going to be a patch port per VM now19:39
chandravrussellb: thx, i'll look into it19:40
russellband then zhouhan has a nice patch to change it yet again :)19:47
russellbso that we don't create a separate lswitch for every port on a provider network19:47
*** azbiswas has joined #openstack-neutron-ovn19:50
*** aginwala has quit IRC19:56
*** allan_h has quit IRC19:59
*** allan_h has joined #openstack-neutron-ovn20:02
*** aginwala has joined #openstack-neutron-ovn20:04
Sam-I-Amrussellb: moo20:26
Sam-I-Amrussellb: got a q about the devstack plugin20:26
Sam-I-Amrussellb: in vagrant, the controller node needs neither ovn-northd nor ovn-controller20:28
Sam-I-Amovn-northd is on the db node and ovn-controller is only on compute nodes20:28
Sam-I-Amhowever, if you disable both of those services, install_ovn never runs20:28
regXboiSam-I-Am: mestery and I ran into that20:29
Sam-I-Amsounds like we can just ditch that conditional?20:29
Sam-I-Amwe already know the deployment uses networking-ovn at that point20:29
regXboiI was thinking of running install_ovn if q-svc is defined20:29
Sam-I-Amthat might work. really if any q* service is enabled.20:30
regXboiyeah, I think that sounds right20:30
Sam-I-Amdo the conventional neutron agents like dhcp and md need ovn-controller at all?20:31
russellbdhcp does20:31
russellbnot sure about md, i don't remember how that one works to be honest20:31
russellbany agent that creates ports needs ovn-controller20:31
azbiswasl3 does as well20:31
russellbso yep, l3 needs it too20:32
Sam-I-Amazbiswas: yeah. i'm assuming we're not using the conventional l3 agent.20:32
Sam-I-Ambut i figured it would apply20:32
azbiswasright, just threw it out there.20:32
russellbthat's some validation we could put into our devstack plugin actually20:32
russellbif (q-l3 || q-dhcp) && !ovn-controlller, error20:32
russellbif nova-compute && !ovn-controller, error20:33
Sam-I-Amthinking about that20:34
Sam-I-Amfirst thing, if is_ovn_service_enabled q-* then do main loop20:35
Sam-I-Amsince there's no generic ovn service20:36
Sam-I-Ambut we know if there's neutron bits, ovn should probably be there too20:36
Sam-I-Amrussellb: is this even a thing - is_service_enabled ovn ?20:39
Sam-I-Amfrom plugin.sh20:39
russellbhistorical ...20:40
Sam-I-Amok, makes sense.20:40
russellbfirst version didn't support multi-node20:40
russellbit was only "ovn" and it enabled everything20:40
*** aginwala has quit IRC20:40
Sam-I-Amdoes anything still use it?20:40
*** aginwala has joined #openstack-neutron-ovn20:40
russellbno docs or anything no20:41
Sam-I-Amdebating where i might put this sanity check. in here... or in another function... or just in the main loop20:41
Sam-I-Amor if its just a docs thing too20:45
russellbdetecting obviously broken config seems like a good thing if easy enough to do20:45
russellbnot critical though20:46
*** aginwala has quit IRC21:02
*** aginwala has joined #openstack-neutron-ovn21:03
*** jckasper has quit IRC21:09
*** aginwala has quit IRC21:11
*** aginwala has joined #openstack-neutron-ovn21:16
*** allan_h has quit IRC21:24
*** allan_h has joined #openstack-neutron-ovn21:24
Sam-I-Amoops, disabling ovn-controller also yielded another problem21:32
Sam-I-Amyay fun bugs21:32
*** allan_h has quit IRC21:39
*** arosen121 has joined #openstack-neutron-ovn21:47
*** brad_behle has quit IRC21:49
*** allan_h has joined #openstack-neutron-ovn21:51
*** arosen12 has quit IRC21:51
*** aginwala has quit IRC21:54
*** thumpba has quit IRC21:54
*** manand has joined #openstack-neutron-ovn21:57
*** aginwala has joined #openstack-neutron-ovn21:57
*** arosen12 has joined #openstack-neutron-ovn22:00
*** jckasper has joined #openstack-neutron-ovn22:02
*** arosen121 has quit IRC22:03
*** igordcard has quit IRC22:09
*** aginwala_ has joined #openstack-neutron-ovn22:10
*** aginwala has quit IRC22:10
*** rtheis has quit IRC22:10
*** c_z has joined #openstack-neutron-ovn22:29
*** jckasper has quit IRC22:30
*** jckasper has joined #openstack-neutron-ovn22:31
*** jckasper has quit IRC22:35
*** jckasper has joined #openstack-neutron-ovn22:37
c_zI tried going through this tutorial but the second devstack host never finished setting up. just says Waiting for ovn-controller to start ... done.22:37
c_zany thoughts on why this happened?22:38
Sam-I-Amc_z: good question22:40
Sam-I-Amso the controller node hung?22:40
Sam-I-Amare there any firewalls on these nodes?22:41
c_zI checked nova hypervisor-list too, it isn't showing up22:42
c_zI don't think so22:42
Sam-I-Amis the ovsdb listening on the controller port 6640?22:42
*** salv-orlando has joined #openstack-neutron-ovn22:42
c_zhow can I check that?22:42
Sam-I-Amss -lntp | grep 664022:42
c_zyeah looks like it is22:43
Sam-I-Amcan the compute node get there?22:43
*** salv-orl_ has quit IRC22:44
c_zI'm pretty new to this so I apologize if this is something silly22:47
Sam-I-Amwell, it could be something broken in devstack22:49
Sam-I-Amc_z: not sure how updated that blog post is22:51
Sam-I-Amdoes this change anything you did?
c_zI've gone through that one too, same result22:53
Sam-I-Amso the first node worked ok?22:53
c_zand I can boot VMs fine22:54
Sam-I-Amcan you find anything in the logs on the compute node?22:54
Sam-I-Amusually ovn-controller hanging means it can't reach the db for some reason22:54
c_zI honestly don't know what to look for22:54
Sam-I-Amso from the compute node, you can telnet to the controller node on port 6640?22:55
*** aginwala_ has quit IRC22:56
*** aginwala has joined #openstack-neutron-ovn22:56
c_zno, looks like I can't23:00
Sam-I-Ambut its listening on the controller node?23:03
c_zI think so23:05
*** regXboi has quit IRC23:07
Sam-I-Amso, ss -lntp shows 6640 listening?23:07
Sam-I-Amand the ovsdb process is running?23:07
c_zif I do that on the first host it shows 6640 listening23:08
c_zwas I supposed to do it on the second?23:08
Sam-I-Ammight as well23:10
c_zwell on the second it shows nothing23:11
Sam-I-Amon the controller node, what does iptables -S show?23:11
*** jckasper has quit IRC23:13
*** jckasper has joined #openstack-neutron-ovn23:13
*** jckasper has quit IRC23:14
*** jckasper has joined #openstack-neutron-ovn23:14
russellbc_z: so it says waiting for ovn-controller to start, but it *does* say "done." right?23:16
russellband this is a compute node?23:16
russellbthe next command the devstack plugin runs is "ovs-appctl"23:16
russellbcan you see if there is an ovs-appctl process running?23:16
c_zit looks like there is23:21
c_zand yes, it says done after waiting for ovn-controller23:23
c_zbut it never showed the host ip or said completed23:23
russellbso ovs-appctl is hanging then23:24
russellbc_z: try ... sudo ovs-appctl -t ovn-controller list-commands23:25
*** manand has quit IRC23:26
c_zso I tried on both nodes23:27
c_zthe first node it lists commands, but it hangs for the second node23:27
russellbok, well at least it's consistent!!23:27
russellbis ovn-controller actually running?23:27
russellbor did devstack mistakenly think so23:28
russellband if it *is* running, what's in the log23:28
c_zhmmm it looks like it23:29
c_zis there a better way to check than ps aux?23:29
russellbit should be running in one of the screen tabs23:29
russellbscreen -x ... switch to the tab running ovn-controller23:29
russellbmaybe ovn-controller is mis-configured and it's stuck in some startup..23:29
* russellb has to step away ... leave whatever info you have and i'll keep looking later (maybe tomorrow)23:31
c_zlooks like it is attempting to connect over and over23:31
russellbok, so mis-configured somehow23:31
russellbI take it you set SERVICE_HOST in local.conf to the IP of the first node?23:31
c_zyes I did23:32
russellbi'll see if i can reproduce tomorrow, not sure what it23:32
c_zok, thanks for the help!23:32
c_zyou too, Sam-I-Am23:32
c_zit could be user error :P23:33
Sam-I-Amc_z: does it say what its trying to connect to?23:33
c_z2016-02-11T23:34:47Z|00817|reconnect|INFO|tcp: connecting... 2016-02-11T23:34:47Z|00818|reconnect|INFO|tcp: connection attempt failed (Connection refused) 2016-02-11T23:34:47Z|00819|reconnect|INFO|tcp: waiting 8 seconds before reconnect23:33
Sam-I-Amwhat is 2.15?23:35
c_zsorry, that's the ip of the first node23:35
Sam-I-Amok, so back on the controller node...23:35
Sam-I-Amwhat does iptables -S say?23:35
Sam-I-Amalso what distro is this?23:35
c_zubuntu 14.0423:36
c_zhere is the iptables result for the first node
c_zand for the second
Sam-I-Amon the controller node, what is the output of 'ss -lntp' ?23:38
Sam-I-Amwhats the ip of the second node?23:38
Sam-I-Am10.0.2.x smells like a virtualbox network... or an lxc default.23:39
c_zI am doing this on virtualbox... is that a mistake?23:40
Sam-I-Ambut i'm thinking your vms cant actually see each other23:42
c_zyou know, I bet you're right23:42
Sam-I-Amso, since you're using virtualbox, did you know we offer vagrant scripts?23:43
Sam-I-Amits a lot more elegant23:43
c_zhmm, maybe I should go that route23:45
c_zI think I am gonna come back to this tomorrow, I need to take a break haha23:48
c_zthanks for all the help :)23:48
Sam-I-Amno problem23:49
Sam-I-Ambrain hurt sucks23:50
Sam-I-Amits not unusual around here23:50
c_zhaha good to know23:51
*** c_z has left #openstack-neutron-ovn23:51
Sam-I-Amthat looks a lot like an ibm ip23:58
*** chandrav has quit IRC23:59

Generated by 2.14.0 by Marius Gedminas - find it at!