| *** yamamoto has joined #openstack-neutron-ovn | 00:08 | |
| *** saphi has joined #openstack-neutron-ovn | 01:51 | |
| *** rpb has quit IRC | 02:32 | |
| saphi | Hi numans! After install neutron-server + python-networking-ovn (master) + run metadata on compute node. And I create an instance, But Metadata can't create datapath. with an error. Log error here: https://paste.ubuntu.com/25160091/ | 03:45 |
|---|---|---|
| *** janki has joined #openstack-neutron-ovn | 05:25 | |
| saphi | Hi numans | 06:08 |
| *** saphi has quit IRC | 06:12 | |
| *** yamamoto has quit IRC | 06:34 | |
| *** anilvenkata has joined #openstack-neutron-ovn | 06:35 | |
| *** zefferno has joined #openstack-neutron-ovn | 06:48 | |
| *** fzdarsky has joined #openstack-neutron-ovn | 07:00 | |
| *** saphi has joined #openstack-neutron-ovn | 07:04 | |
| *** janki is now known as janki|lunch | 07:37 | |
| *** yamamoto has joined #openstack-neutron-ovn | 07:45 | |
| *** fzdarsky has quit IRC | 07:51 | |
| *** saphi has quit IRC | 08:04 | |
| *** rpb has joined #openstack-neutron-ovn | 08:10 | |
| *** saphi has joined #openstack-neutron-ovn | 08:16 | |
| *** lucas-afk is now known as lucasagomes | 08:26 | |
| numans | Hello everyone, networking-ovn CI is broken because of this new job gate-tripleo-ci-centos-7-scenario007-multinode-oooq-puppet which was enabled recently and we need the following patches to be merged - https://review.openstack.org/#/c/475423/ and https://review.openstack.org/#/c/484943/ | 08:31 |
| *** fzdarsky has joined #openstack-neutron-ovn | 08:32 | |
| lucasagomes | numans, morning | 08:36 |
| numans | lucasagomes, hey | 08:36 |
| numans | morning | 08:36 |
| lucasagomes | numans, you sure that's going to fix it, see: http://logs.openstack.org/67/483767/4/check/gate-tripleo-ci-centos-7-scenario007-multinode-oooq-puppet/fe7df95/console.html#_2017-07-24_00_32_27_127374 | 08:37 |
| lucasagomes | numans, maybe we should have it as -nv until the job is fixed | 08:37 |
| numans | lucasagomes++ | 08:37 |
| numans | i agree | 08:37 |
| numans | lucasagomes, you want to submit a patch for -nv ? | 08:39 |
| lucasagomes | numans, sure, I can put one up in a few | 08:39 |
| numans | lucasagomes, ERROR! the file_name '/opt/stack/new/tripleo-quickstart/config/general_config/featureset028.yml' does not exist, or is not readable | 08:40 |
| numans | lucasagomes, this is the reason for the error | 08:40 |
| *** janki|lunch is now known as janki | 08:40 | |
| lucasagomes | oh right, yeah I just saw that brctl command not found... Maybe we need to fix few other things more | 08:41 |
| lucasagomes | none seems very big btw, so we can re-enable it to voting soon :D | 08:41 |
| *** yamamoto has quit IRC | 09:02 | |
| *** fzdarsky has quit IRC | 09:11 | |
| *** yamamoto has joined #openstack-neutron-ovn | 09:15 | |
| *** rpb has quit IRC | 09:29 | |
| saphi | Hi guys! | 09:47 |
| *** yamamoto has quit IRC | 09:47 | |
| saphi | I'm using openstack pike and python-networking-ovn master branch. | 09:48 |
| saphi | after launch instance, the port can't create on the switch with error here https://paste.ubuntu.com/25161517/ | 09:49 |
| saphi | ovs_version: "2.7.0" | 09:50 |
| *** yamamoto has joined #openstack-neutron-ovn | 10:02 | |
| *** yamamoto has quit IRC | 10:02 | |
| *** yamamoto has joined #openstack-neutron-ovn | 10:33 | |
| numans | saphi, hi | 10:39 |
| saphi | I'm using openstack pike and python-networking-ovn master branch. after launch instance, the port can't create on the switch with error here https://paste.ubuntu.com/25161517/ | 10:40 |
| saphi | Hi numans | 10:40 |
| saphi | I installed manual. | 10:40 |
| numans | saphi, for your other question n metadata, i think you need latest ovs | 10:40 |
| numans | saphi, for this one - https://paste.ubuntu.com/25160091/ | 10:41 |
| saphi | I will try install openvswitch latest. I think I have to install from source | 10:42 |
| numans | saphi, can you run ovn-sbctl show and see if all your compute nodes are able to talk to ovn sb db | 10:42 |
| numans | saphi, you are using ubuntu ? | 10:42 |
| numans | saphi, if centos i can point you to the rpms for the latest ovs | 10:42 |
| saphi | yes. compute node can comunicate with ovn sb db | 10:42 |
| numans | saphi, for ubuntu i am not sure if there are any deb files | 10:42 |
| saphi | numans . Please share me those files :D | 10:43 |
| numans | saphi, you mean for centos ? | 10:43 |
| saphi | yes. numans | 10:43 |
| saphi | I'm trying on both of them. | 10:43 |
| numans | saphi, https://copr.fedorainfracloud.org/coprs/leifmadsen/ovs-master/ | 10:43 |
| saphi | Ubuntu 16.04 and CentOS 7 | 10:44 |
| numans | saphi, this would give you latest ovs rpms | 10:44 |
| numans | saphi, not sure about ubuntu :) | 10:44 |
| saphi | Yes. I will try | 10:45 |
| *** lucasagomes is now known as lucas-hungry | 10:52 | |
| *** saphi has quit IRC | 11:02 | |
| *** anilvenkata has quit IRC | 11:52 | |
| *** anilvenkata has joined #openstack-neutron-ovn | 11:53 | |
| *** lucas-hungry is now known as lucasagomes | 12:03 | |
| *** saphi has joined #openstack-neutron-ovn | 12:10 | |
| *** yamamoto has quit IRC | 12:19 | |
| *** russellb_ has joined #openstack-neutron-ovn | 12:21 | |
| *** yamamoto has joined #openstack-neutron-ovn | 12:36 | |
| *** yamamoto has quit IRC | 12:36 | |
| openstackgerrit | Terry Wilson proposed openstack/ovsdbapp master: Add RowEvent mechanism from networking-ovn https://review.openstack.org/484964 | 12:37 |
| saphi | hi numans, I was check communication between compute and controller node. Everything is ok. To clear. I installed nova-compute, ovn-controller on Controller. So I have still this error. | 13:01 |
| saphi | https://paste.ubuntu.com/25160091/ | 13:02 |
| *** mmichelson has joined #openstack-neutron-ovn | 13:02 | |
| anilvenkata | numans, when I tried to create HA router, command is failing with below error | 13:20 |
| anilvenkata | [centos@ovn-devstack devstack]$ openstack router create --ha hr1 | 13:20 |
| anilvenkata | HttpException: Bad Request (HTTP 400) (Request-ID: req-9c141861-a1f0-4356-8ae2-71c8680cdd3c), Unrecognized attribute(s) 'ha' | 13:20 |
| anilvenkata | numans, error in neutron server log | 13:21 |
| anilvenkata | Jul 24 13:19:05 ovn-devstack neutron-server[31510]: INFO neutron.api.v2.resource [None req-9c141861-a1f0-4356-8ae2-71c8680cdd3c demo admin] create failed (client error): Unrecognized attribute(s) 'ha': HTTPBadRequest: Unrecognized attribute(s) 'ha' | 13:21 |
| anilvenkata | numans, do u have any idea? | 13:21 |
| numans | anilvenkata, i don't think that's supported | 13:22 |
| numans | anilvenkata, i.e we don't support dvr or ha flags. | 13:22 |
| anilvenkata | numans, ok | 13:22 |
| saphi | hi numans, Did you have any solutions for me? | 13:30 |
| numans | saphi, regarding ? | 13:30 |
| numans | the port binding issue ? | 13:30 |
| saphi | yes. Port binding issue. | 13:30 |
| numans | saphi, is your deployment a fresh ovn deployment or you changed from ml2ovs to ovn ? | 13:31 |
| saphi | fresh deployment. | 13:31 |
| numans | saphi, could you please raise a bug ? and attach the neutron-server logs ? | 13:32 |
| numans | saphi, the paste you shared has very less info. so can't comment much | 13:33 |
| saphi | because I deploy an other lab using devstack. everything is ok. I want to install it manually. But something is not correct. | 13:33 |
| saphi | I saw, ovn-controller can connect to ovs sb db | 13:34 |
| saphi | It can create tunnel. | 13:34 |
| saphi | but when launch instance, It can't create a port | 13:34 |
| numans | saphi, check the hostname configured in nova.conf and the hostname configured in ovs db | 13:35 |
| numans | saphi, on one of the compute node, run "ovs-vsctl get open . external_ids" and see if the hostname displayed matches what you have in nova.conf | 13:36 |
| numans | saphi, i suspect that's the issue | 13:36 |
| *** yamamoto has joined #openstack-neutron-ovn | 13:36 | |
| *** yamamoto has quit IRC | 13:44 | |
| *** russellb_ has quit IRC | 13:44 | |
| saphi | Thanks numans, port binding I fixed. The hostname is not match. in nova and ovsdb | 13:45 |
| saphi | And now, the error Unexpected exception in notify_loop: KeyError: 'external_ids', still exist. :) | 13:46 |
| numans | saphi, may be you need to upgrade your schema | 14:02 |
| saphi | So I can remove all openvswitch database (old) and restart openvswitch, ovn-northd, ovn-controller? | 14:03 |
| numans | saphi, you need to either upgradethe db or create a new one | 14:04 |
| openstackgerrit | venkata anil proposed openstack/networking-ovn master: schedule gateway on chassis from config file https://review.openstack.org/486098 | 14:11 |
| saphi | Hi numans, What command to upgrade the db :D | 14:13 |
| numans | saphi, please have a look at this utility. | 14:14 |
| numans | if you use this to start ovsdb-servers, it will take care of upgrading the db as well. please have a look at this script if you want to do it yourself. see start_ovsdb function | 14:15 |
| saphi | numans, After upgrade database schema, the error is gone :D | 14:24 |
| numans | saphi, cool | 14:24 |
| saphi | But the namespace of ovnmetadata is not created | 14:25 |
| numans | dalvarez, you have any pointers for saphi ? | 14:25 |
| * dalvarez reading | 14:25 | |
| saphi | on neutron-server.log | 14:25 |
| saphi | etadata service is not ready for port df720792-c157-4355-b9f6-43598889e3cc, check networking-ovn-metadata-agent status/logs.: MetadataServiceReadyWaitTimeoutException | 14:26 |
| saphi | 2017-07-24 14:23:37.913 40 INFO neutron.notifiers.nova [-] Nova event response: {u'status': u'completed', u'tag': u'df720792-c157-4355-b9f6-43598889e3cc', u'name': u'network-vif-plugged', u'server_uuid': u'553219ad-73e0-4edc-bcc2-9f7b89789b04', u'code': 200} | 14:26 |
| dalvarez | saphi, we sent a patch to disable metadata by default and got merged on Friday IIRC | 14:26 |
| dalvarez | saphi, ops sry it wasn't merged due to some other error | 14:27 |
| saphi | why do we disable metadata? | 14:27 |
| dalvarez | saphi, this is the patch: https://review.openstack.org/#/c/483915/ | 14:27 |
| dalvarez | saphi, because we depend on OVS 2.8 | 14:27 |
| numans | dalvarez, i think he is using ovs master | 14:27 |
| dalvarez | saphi, numans then you need to start networking-ovn-metadata-agent | 14:28 |
| dalvarez | saphi, if you want metadata you need to enable metadata service in local.conf for devstack | 14:28 |
| dalvarez | saphi, https://review.openstack.org/#/c/471140/36/devstack/local.conf.sample | 14:28 |
| dalvarez | and/or start it manually | 14:29 |
| saphi | hi dalvarez, I deployed ovn with devstack all in one. And they run successful. | 14:29 |
| saphi | And now, I want to deploy it manually. | 14:29 |
| dalvarez | saphi, you have metadata service enabled in local.conf? | 14:29 |
| saphi | yes. | 14:29 |
| dalvarez | saphi, ps -ef | grep networking-ovn-metadata-agent | 14:29 |
| saphi | https://paste.ubuntu.com/25162825/ | 14:30 |
| saphi | here you are | 14:30 |
| dalvarez | saphi, nice so it's running | 14:31 |
| dalvarez | saphi, could you share the logs maybe? | 14:31 |
| saphi | I think I get something. | 14:33 |
| saphi | I will try to fix. :D | 14:33 |
| dalvarez | saphi, what's that? im curious | 14:35 |
| dalvarez | btw i think i want to rename that ns-metadata-proxy to ovn*, what do you think numans ? | 14:35 |
| openstackgerrit | Daniel Alvarez proposed openstack/networking-ovn master: Rename 'ns-metadata-proxy' config dir to 'ovn-metadata-proxy' https://review.openstack.org/486637 | 14:37 |
| dalvarez | ^:) | 14:39 |
| saphi | dalvarez, I installed ovn-metadata in a docker container, After remove and recreate ovs db. I have to restart container, :) | 14:49 |
| dalvarez | saphi, nice! :) | 14:50 |
| saphi | I'm looking for OVS 2.8 release | 14:51 |
| *** mmirecki has joined #openstack-neutron-ovn | 14:51 | |
| *** mlavalle has joined #openstack-neutron-ovn | 14:51 | |
| dalvarez | saphi, should be out soon | 14:52 |
| saphi | :) | 14:59 |
| *** mmirecki has quit IRC | 15:00 | |
| dalvarez | russellb, ping re. trunk ports in OVN | 15:02 |
| russellb | pong dalvarez | 15:02 |
| russellb | what's the issue? | 15:02 |
| dalvarez | russellb, in networking-ovn, the trunk driver will handle the event when a subport is added to a trunk and that will set the parent port and that's it | 15:02 |
| dalvarez | then ovn-controller will insert the proper flows to add/strip the vlan tag | 15:03 |
| dalvarez | but in ML2/OVS the driver handles binding for the subports as well so subports transition from DOWN to ACTIVE when a VM boots | 15:03 |
| dalvarez | in OVN that never happens, subports always stay DOWN | 15:03 |
| russellb | OK | 15:03 |
| russellb | so OVN bug then | 15:03 |
| russellb | first question ... do we set the chassis column of the port_binding row for the subport in the ovn southbound db? | 15:03 |
| russellb | (narrowing down where a change is needed) | 15:04 |
| russellb | ovn-northd sets the "up" column based on the chassis column being set on a port binding | 15:05 |
| russellb | so i imagine the fix here isn't very big, but not immediately obvious if it's just an ovn-northd fix | 15:05 |
| russellb | or ovn-controller | 15:05 |
| dalvarez | russellb, looking... | 15:05 |
| dalvarez | russellb, re. setting the chassis column for the subports: yes we do | 15:06 |
| russellb | OK, cool. | 15:06 |
| russellb | then sounds like an ovn-northd bug | 15:06 |
| dalvarez | russellb, ops wait sry it we don't | 15:06 |
| russellb | ah | 15:06 |
| dalvarez | i was looking at the parent port sry | 15:06 |
| russellb | OK | 15:07 |
| russellb | so we have 2 choices to fix this | 15:07 |
| russellb | 1) make ovn-controller set the chassis column for child ports | 15:07 |
| russellb | 2) make ovn-northd calculate the "up" state based on looking at the parent port status | 15:07 |
| * russellb flips coin | 15:07 | |
| russellb | my preference is #1 | 15:08 |
| dalvarez | russellb, that would do it? | 15:08 |
| dalvarez | ah ok yeah we would monitor that in networking-ovn and update the plugin | 15:08 |
| russellb | we could also work around the issue in networking-ovn | 15:08 |
| russellb | when we see a state change for a port, update any child ports as well | 15:08 |
| russellb | but i think OVN should reflect proper "up" state for child ports | 15:09 |
| dalvarez | russellb, i thought of that last choice | 15:09 |
| *** anilvenkata has quit IRC | 15:09 | |
| dalvarez | but there's no straightforward way to tell the childs given a parent port right? | 15:09 |
| dalvarez | it's like the other way around | 15:09 |
| russellb | correct | 15:11 |
| russellb | #2 above is probably the smallest patch | 15:11 |
| dalvarez | russellb, question, how does ovn-northd sets the 'up' column? it's aware when the interface is present in br-int and updates it accordingly? | 15:11 |
| russellb | it looks at the chassis column of port_binding in ovn southbound | 15:11 |
| russellb | if it's set, up == true | 15:12 |
| russellb | if it's empty, up == false | 15:12 |
| dalvarez | right, im stupid :) | 15:12 |
| dalvarez | russellb, so in OVN subports aren't plugged to any ovs bridge right? | 15:14 |
| russellb | correct | 15:15 |
| russellb | not directly | 15:15 |
| russellb | so maybe the ovn-northd fix is the better one | 15:15 |
| dalvarez | russellb, agree | 15:15 |
| russellb | i've got to run to a dr appt ... back after that and lunch | 15:15 |
| dalvarez | russellb, ack thanks i'll try to spot the best place in ovn-northd for that | 15:16 |
| russellb | k | 15:17 |
| russellb | let me know if you run short on time for it | 15:17 |
| *** zefferno has quit IRC | 15:23 | |
| saphi | Hi dalvarez. I have an error. ProcessExecutionError: Exit code: 99; Stdin: ; Stdout: ; Stderr: /usr/local/bin/neutron-rootwrap: Unauthorized command: ip netns exec ovnmeta-682c8300-aea1-4e75-8df4-0aedbf7a5150 haproxy -f /var/lib/neutron/ns-metadata-proxy/682c8300-aea1-4e75-8df4-0aedbf7a5150.conf (no filter matched) | 15:26 |
| saphi | I think this error related rootwrap config | 15:27 |
| dalvarez | saphi, apparently rootwrap filters missing :) | 15:27 |
| dalvarez | but neutron already had filters for it | 15:27 |
| saphi | I saw this filter: ip_exec: IpNetnsExecFilter, ip, root | 15:28 |
| dalvarez | saphi, that should do it IMO | 15:28 |
| dalvarez | saphi, weird... | 15:28 |
| dalvarez | i think we shouldn't be seeing "neutron-rootwrap" but instead, rootwrap-daemon | 15:28 |
| dalvarez | right? | 15:28 |
| saphi | No! | 15:29 |
| dalvarez | saphi, :? | 15:30 |
| saphi | I can't understand your mean. | 15:30 |
| saphi | Could you explain it for me clear! | 15:30 |
| dalvarez | saphi, i mean that we're not using rootwrap anymore but rootwrap-daemon, right? | 15:30 |
| dalvarez | we == neutron | 15:31 |
| saphi | I think so. | 15:31 |
| saphi | because other command are using rootwrap and success. | 15:32 |
| saphi | Oh no! | 15:32 |
| saphi | Running command: ['sudo', 'neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ip', 'netns', 'exec', 'ovnme | 15:32 |
| saphi | ta-682c8300-aea1-4e75-8df4-0aedbf7a5150', 'haproxy', '-f', '/var/lib/neutron/ns-metadata-proxy/682c8300-aea1-4e75-8df4-0aedbf7a5150.conf'] | 15:32 |
| saphi | We're still using neutron-rootwrap | 15:33 |
| *** janki has quit IRC | 15:35 | |
| dalvarez | saphi, take a look at rootwrap.conf to check what filters it's using | 15:37 |
| dalvarez | russellb, i think this is the place, right: https://github.com/openvswitch/ovs/blob/master/ovn/northd/ovn-northd.c#L5911 | 15:37 |
| *** lucasagomes is now known as lucas-afk | 16:04 | |
| dalvarez | russellb, ping i wrote a patch to northd and now subports transition to up depending on parent port state :) however this is not enough to transition to ACTIVE In neutron, I think we need to monitor the up->True in networking-ovn and if it's a subport (it's got a parent port) then set the vif_type accordingly and update the plugin | 16:14 |
| dalvarez | russellb, I gotta go now but i'll submit the patches tomorrow, let me know how it sounds to you? | 16:15 |
| *** fzdarsky has joined #openstack-neutron-ovn | 16:48 | |
| *** fzdarsky_ has joined #openstack-neutron-ovn | 16:48 | |
| *** fzdarsky_ has quit IRC | 17:06 | |
| *** fzdarsky has quit IRC | 17:06 | |
| *** saphi has quit IRC | 18:00 | |
| *** gtrxcb has quit IRC | 18:08 | |
| *** gtrxcb has joined #openstack-neutron-ovn | 18:31 | |
| *** gtrxcb has quit IRC | 19:41 | |
| *** yamamoto_ has joined #openstack-neutron-ovn | 21:09 | |
| *** yamamoto_ has quit IRC | 21:20 | |
| *** rpb has joined #openstack-neutron-ovn | 21:41 | |
| *** yamamoto has joined #openstack-neutron-ovn | 22:01 | |
| *** tongl has joined #openstack-neutron-ovn | 22:05 | |
| *** mlavalle has quit IRC | 22:27 | |
| *** mmichelson has quit IRC | 22:28 | |
| *** gtrxcb has joined #openstack-neutron-ovn | 22:57 | |
| *** tongl has quit IRC | 23:15 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!