Monday, 2017-07-24

*** yamamoto has joined #openstack-neutron-ovn00:08
*** saphi has joined #openstack-neutron-ovn01:51
*** rpb has quit IRC02:32
saphiHi numans! After install neutron-server + python-networking-ovn (master) + run metadata on compute node. And I create an instance, But Metadata can't create datapath. with an error. Log error here:
*** janki has joined #openstack-neutron-ovn05:25
saphiHi numans06:08
*** saphi has quit IRC06:12
*** yamamoto has quit IRC06:34
*** anilvenkata has joined #openstack-neutron-ovn06:35
*** zefferno has joined #openstack-neutron-ovn06:48
*** fzdarsky has joined #openstack-neutron-ovn07:00
*** saphi has joined #openstack-neutron-ovn07:04
*** janki is now known as janki|lunch07:37
*** yamamoto has joined #openstack-neutron-ovn07:45
*** fzdarsky has quit IRC07:51
*** saphi has quit IRC08:04
*** rpb has joined #openstack-neutron-ovn08:10
*** saphi has joined #openstack-neutron-ovn08:16
*** lucas-afk is now known as lucasagomes08:26
numansHello everyone, networking-ovn CI is broken because of this new job gate-tripleo-ci-centos-7-scenario007-multinode-oooq-puppet which was enabled recently and we need the following patches to be merged - and
*** fzdarsky has joined #openstack-neutron-ovn08:32
lucasagomesnumans, morning08:36
numanslucasagomes, hey08:36
lucasagomesnumans, you sure that's going to fix it, see:
lucasagomesnumans, maybe we should have it as -nv until the job is fixed08:37
numansi agree08:37
numanslucasagomes, you want to submit a patch for -nv ?08:39
lucasagomesnumans, sure, I can put one up in a few08:39
numanslucasagomes, ERROR! the file_name '/opt/stack/new/tripleo-quickstart/config/general_config/featureset028.yml' does not exist, or is not readable08:40
numanslucasagomes, this is the reason for the error08:40
*** janki|lunch is now known as janki08:40
lucasagomesoh right, yeah I just saw that brctl command not found... Maybe we need to fix few other things more08:41
lucasagomesnone seems very big btw, so we can re-enable it to voting soon :D08:41
*** yamamoto has quit IRC09:02
*** fzdarsky has quit IRC09:11
*** yamamoto has joined #openstack-neutron-ovn09:15
*** rpb has quit IRC09:29
saphiHi guys!09:47
*** yamamoto has quit IRC09:47
saphiI'm using openstack pike and python-networking-ovn master  branch.09:48
saphiafter launch instance, the port can't create on the switch with error here
saphiovs_version: "2.7.0"09:50
*** yamamoto has joined #openstack-neutron-ovn10:02
*** yamamoto has quit IRC10:02
*** yamamoto has joined #openstack-neutron-ovn10:33
numanssaphi, hi10:39
saphiI'm using openstack pike and python-networking-ovn master  branch. after launch instance, the port can't create on the switch with error here
saphiHi numans10:40
saphiI installed manual.10:40
numanssaphi, for your other question n metadata, i think you need latest ovs10:40
numanssaphi, for this one -
saphiI will try install openvswitch latest. I think I have to install from source10:42
numanssaphi, can you run ovn-sbctl show and see if all your compute nodes are able to talk to ovn sb db10:42
numanssaphi, you are using ubuntu ?10:42
numanssaphi, if centos i can point you to the rpms for the latest ovs10:42
saphiyes. compute node can comunicate with ovn sb db10:42
numanssaphi, for ubuntu i am not sure if there are any deb files10:42
saphinumans . Please share me those files :D10:43
numanssaphi, you mean for centos ?10:43
saphiyes. numans10:43
saphiI'm trying on both of them.10:43
saphiUbuntu 16.04 and CentOS 710:44
numanssaphi, this would give you latest ovs rpms10:44
numanssaphi, not sure about ubuntu :)10:44
saphiYes. I will try10:45
*** lucasagomes is now known as lucas-hungry10:52
*** saphi has quit IRC11:02
*** anilvenkata has quit IRC11:52
*** anilvenkata has joined #openstack-neutron-ovn11:53
*** lucas-hungry is now known as lucasagomes12:03
*** saphi has joined #openstack-neutron-ovn12:10
*** yamamoto has quit IRC12:19
*** russellb_ has joined #openstack-neutron-ovn12:21
*** yamamoto has joined #openstack-neutron-ovn12:36
*** yamamoto has quit IRC12:36
openstackgerritTerry Wilson proposed openstack/ovsdbapp master: Add RowEvent mechanism from networking-ovn
saphihi numans, I was check communication between compute and controller node. Everything is ok. To clear. I installed nova-compute, ovn-controller on Controller. So I have still this error.13:01
*** mmichelson has joined #openstack-neutron-ovn13:02
anilvenkatanumans, when I tried to create HA router, command is failing with below error13:20
anilvenkata[centos@ovn-devstack devstack]$ openstack router create --ha hr113:20
anilvenkataHttpException: Bad Request (HTTP 400) (Request-ID: req-9c141861-a1f0-4356-8ae2-71c8680cdd3c), Unrecognized attribute(s) 'ha'13:20
anilvenkatanumans, error in neutron server log13:21
anilvenkataJul 24 13:19:05 ovn-devstack neutron-server[31510]: INFO neutron.api.v2.resource [None req-9c141861-a1f0-4356-8ae2-71c8680cdd3c demo admin] create failed (client error): Unrecognized attribute(s) 'ha': HTTPBadRequest: Unrecognized attribute(s) 'ha'13:21
anilvenkatanumans, do u have any idea?13:21
numansanilvenkata, i don't think that's supported13:22
numansanilvenkata, i.e we don't support dvr or ha flags.13:22
anilvenkatanumans, ok13:22
saphihi numans, Did you have any solutions for me?13:30
numanssaphi, regarding ?13:30
numansthe port binding issue ?13:30
saphiyes. Port binding issue.13:30
numanssaphi, is your deployment a  fresh ovn deployment or you changed from ml2ovs to ovn ?13:31
saphifresh deployment.13:31
numanssaphi, could you please raise a bug ? and attach the neutron-server logs ?13:32
numanssaphi, the paste you shared has very less info. so can't comment much13:33
saphibecause I deploy an other lab using devstack. everything is ok. I want to install it manually. But something is not correct.13:33
saphiI saw, ovn-controller can connect to ovs sb db13:34
saphiIt can create tunnel.13:34
saphibut when launch instance, It can't create a port13:34
numanssaphi, check the hostname configured in nova.conf and the hostname configured in ovs db13:35
numanssaphi, on one of the compute node, run "ovs-vsctl get open . external_ids" and see if the hostname displayed matches what you have in nova.conf13:36
numanssaphi, i suspect that's the issue13:36
*** yamamoto has joined #openstack-neutron-ovn13:36
*** yamamoto has quit IRC13:44
*** russellb_ has quit IRC13:44
saphiThanks numans, port binding I fixed. The hostname is not match. in nova and ovsdb13:45
saphiAnd now, the error Unexpected exception in notify_loop: KeyError: 'external_ids', still exist. :)13:46
numanssaphi, may be you need to upgrade your schema14:02
saphiSo I can remove all openvswitch database (old) and restart openvswitch, ovn-northd, ovn-controller?14:03
numanssaphi, you need to either upgradethe db or create a new one14:04
openstackgerritvenkata anil proposed openstack/networking-ovn master: schedule gateway on chassis from config file
saphiHi numans, What command to upgrade the db :D14:13
numanssaphi, please have a look at this utility.14:14
numansif you use this to start ovsdb-servers, it will take care of upgrading the db as well. please have a look at this script if you want to do it yourself. see start_ovsdb function14:15
saphi numans, After upgrade database schema, the error is gone :D14:24
numanssaphi, cool14:24
saphiBut the namespace of ovnmetadata is not created14:25
numansdalvarez, you have any pointers for saphi ?14:25
* dalvarez reading14:25
saphion neutron-server.log14:25
saphietadata service is not ready for port df720792-c157-4355-b9f6-43598889e3cc, check networking-ovn-metadata-agent status/logs.: MetadataServiceReadyWaitTimeoutException14:26
saphi2017-07-24 14:23:37.913 40 INFO neutron.notifiers.nova [-] Nova event response: {u'status': u'completed', u'tag': u'df720792-c157-4355-b9f6-43598889e3cc', u'name': u'network-vif-plugged', u'server_uuid': u'553219ad-73e0-4edc-bcc2-9f7b89789b04', u'code': 200}14:26
dalvarezsaphi, we sent a patch to disable metadata by default and got merged on Friday IIRC14:26
dalvarezsaphi, ops sry it wasn't merged due to some other error14:27
saphiwhy do we disable metadata?14:27
dalvarezsaphi, this is the patch:
dalvarezsaphi, because we depend on OVS 2.814:27
numansdalvarez, i think he is using ovs master14:27
dalvarezsaphi, numans then you need to start networking-ovn-metadata-agent14:28
dalvarezsaphi, if you want metadata you need to enable metadata service in local.conf for devstack14:28
dalvarezand/or start it manually14:29
saphihi dalvarez, I deployed ovn with devstack all in one. And they run successful.14:29
saphiAnd now, I want to deploy it manually.14:29
dalvarezsaphi, you have metadata service enabled in local.conf?14:29
dalvarezsaphi, ps -ef | grep networking-ovn-metadata-agent14:29
saphihere you are14:30
dalvarezsaphi, nice so it's running14:31
dalvarezsaphi, could you share the logs maybe?14:31
saphiI think I get something.14:33
saphiI will try to fix. :D14:33
dalvarezsaphi, what's that? im curious14:35
dalvarezbtw i think i want to rename that ns-metadata-proxy to ovn*, what do you think numans ?14:35
openstackgerritDaniel Alvarez proposed openstack/networking-ovn master: Rename 'ns-metadata-proxy' config dir to 'ovn-metadata-proxy'
saphidalvarez, I installed ovn-metadata in a docker container, After remove  and recreate ovs db. I have to restart container, :)14:49
dalvarezsaphi, nice! :)14:50
saphiI'm looking for OVS 2.8 release14:51
*** mmirecki has joined #openstack-neutron-ovn14:51
*** mlavalle has joined #openstack-neutron-ovn14:51
dalvarezsaphi, should be out soon14:52
*** mmirecki has quit IRC15:00
dalvarezrussellb, ping re. trunk ports in OVN15:02
russellbpong dalvarez15:02
russellbwhat's the issue?15:02
dalvarezrussellb, in networking-ovn, the trunk driver will handle the event when a subport is added to a trunk and that will set the parent port and that's it15:02
dalvarezthen ovn-controller will insert the proper flows to add/strip the vlan tag15:03
dalvarezbut in ML2/OVS the driver handles binding for the subports as well so subports transition from DOWN to ACTIVE when a VM boots15:03
dalvarezin OVN that never happens, subports always stay DOWN15:03
russellbso OVN bug then15:03
russellbfirst question ... do we set the chassis column of the port_binding row for the subport in the ovn southbound db?15:03
russellb(narrowing down where a change is needed)15:04
russellbovn-northd sets the "up" column based on the chassis column being set on a port binding15:05
russellbso i imagine the fix here isn't very big, but not immediately obvious if it's just an ovn-northd fix15:05
russellbor ovn-controller15:05
dalvarezrussellb, looking...15:05
dalvarezrussellb, re. setting the chassis column for the subports: yes we do15:06
russellbOK, cool.15:06
russellbthen sounds like an ovn-northd bug15:06
dalvarezrussellb, ops wait sry it we don't15:06
dalvarezi was looking at the parent port sry15:06
russellbso we have 2 choices to fix this15:07
russellb1) make ovn-controller set the chassis column for child ports15:07
russellb2) make ovn-northd calculate the "up" state based on looking at the parent port status15:07
* russellb flips coin15:07
russellbmy preference is #115:08
dalvarezrussellb, that would do it?15:08
dalvarezah ok yeah we would monitor that in networking-ovn and update the plugin15:08
russellbwe could also work around the issue in networking-ovn15:08
russellbwhen we see a state change for a port, update any child ports as well15:08
russellbbut i think OVN should reflect proper "up" state for child ports15:09
dalvarezrussellb, i thought of that last choice15:09
*** anilvenkata has quit IRC15:09
dalvarezbut there's no straightforward way to tell the childs given a parent port right?15:09
dalvarezit's like the other way around15:09
russellb#2 above is probably the smallest patch15:11
dalvarezrussellb, question, how does ovn-northd sets the 'up' column? it's aware when the interface is present in br-int and updates it accordingly?15:11
russellbit looks at the chassis column of port_binding in ovn southbound15:11
russellbif it's set, up == true15:12
russellbif it's empty, up == false15:12
dalvarezright, im stupid :)15:12
dalvarezrussellb, so in OVN subports aren't plugged to any ovs bridge right?15:14
russellbnot directly15:15
russellbso maybe the ovn-northd fix is the better one15:15
dalvarezrussellb, agree15:15
russellbi've got to run to a dr appt ... back after that and lunch15:15
dalvarezrussellb, ack thanks i'll try to spot the best place in ovn-northd for that15:16
russellblet me know if you run short on time for it15:17
*** zefferno has quit IRC15:23
saphiHi dalvarez. I have an error. ProcessExecutionError: Exit code: 99; Stdin: ; Stdout: ; Stderr: /usr/local/bin/neutron-rootwrap: Unauthorized command: ip netns exec ovnmeta-682c8300-aea1-4e75-8df4-0aedbf7a5150 haproxy -f /var/lib/neutron/ns-metadata-proxy/682c8300-aea1-4e75-8df4-0aedbf7a5150.conf (no filter matched)15:26
saphiI think this error related rootwrap config15:27
dalvarezsaphi, apparently rootwrap filters missing :)15:27
dalvarezbut neutron already had filters for it15:27
saphiI saw this filter: ip_exec: IpNetnsExecFilter, ip, root15:28
dalvarezsaphi, that should do it IMO15:28
dalvarezsaphi, weird...15:28
dalvarezi think we shouldn't be seeing "neutron-rootwrap" but instead, rootwrap-daemon15:28
dalvarezsaphi, :?15:30
saphiI can't understand your mean.15:30
saphiCould you explain it for me clear!15:30
dalvarezsaphi, i mean that we're not using rootwrap anymore but rootwrap-daemon, right?15:30
dalvarezwe == neutron15:31
saphiI think so.15:31
saphibecause other command are using rootwrap and success.15:32
saphiOh no!15:32
saphiRunning command: ['sudo', 'neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ip', 'netns', 'exec', 'ovnme15:32
saphita-682c8300-aea1-4e75-8df4-0aedbf7a5150', 'haproxy', '-f', '/var/lib/neutron/ns-metadata-proxy/682c8300-aea1-4e75-8df4-0aedbf7a5150.conf']15:32
saphiWe're still using neutron-rootwrap15:33
*** janki has quit IRC15:35
dalvarezsaphi, take a look at rootwrap.conf to check what filters it's using15:37
dalvarezrussellb, i think this is the place, right:
*** lucasagomes is now known as lucas-afk16:04
dalvarezrussellb, ping i wrote a patch to northd and now subports transition to up depending on parent port state :)  however this is not enough to transition to ACTIVE In neutron, I think we need to monitor the up->True in networking-ovn and if it's a subport (it's got a parent port) then set the vif_type accordingly and update the plugin16:14
dalvarezrussellb, I gotta go now but i'll submit the patches tomorrow, let me know how it sounds to you?16:15
*** fzdarsky has joined #openstack-neutron-ovn16:48
*** fzdarsky_ has joined #openstack-neutron-ovn16:48
*** fzdarsky_ has quit IRC17:06
*** fzdarsky has quit IRC17:06
*** saphi has quit IRC18:00
*** gtrxcb has quit IRC18:08
*** gtrxcb has joined #openstack-neutron-ovn18:31
*** gtrxcb has quit IRC19:41
*** yamamoto_ has joined #openstack-neutron-ovn21:09
*** yamamoto_ has quit IRC21:20
*** rpb has joined #openstack-neutron-ovn21:41
*** yamamoto has joined #openstack-neutron-ovn22:01
*** tongl has joined #openstack-neutron-ovn22:05
*** mlavalle has quit IRC22:27
*** mmichelson has quit IRC22:28
*** gtrxcb has joined #openstack-neutron-ovn22:57
*** tongl has quit IRC23:15

Generated by 2.15.3 by Marius Gedminas - find it at!