| *** rcernin has joined #openstack-neutron | 00:00 | |
| *** Swami has quit IRC | 00:03 | |
| *** mriedem has quit IRC | 00:03 | |
| *** mriedem has joined #openstack-neutron | 00:04 | |
| *** harlowja has quit IRC | 00:04 | |
| *** tommylikehu_ has joined #openstack-neutron | 00:08 | |
| *** liangy has quit IRC | 00:09 | |
| *** Swami has joined #openstack-neutron | 00:10 | |
| openstackgerrit | Armando Migliaccio proposed openstack/neutron-specs: Ocata release postmortem https://review.openstack.org/425990 | 00:13 |
|---|---|---|
| *** tommylikehu_ has quit IRC | 00:13 | |
| openstackgerrit | Armando Migliaccio proposed openstack/neutron-specs: Ocata release postmortem https://review.openstack.org/425990 | 00:13 |
| *** Leo_ has quit IRC | 00:22 | |
| *** fnaval has quit IRC | 00:23 | |
| openstackgerrit | Jaivish Kothari(janonymous) proposed openstack/os-vif: Fix broken Link https://review.openstack.org/425994 | 00:23 |
| *** masaki has joined #openstack-neutron | 00:25 | |
| *** mlavalle has quit IRC | 00:27 | |
| *** abhiraut has quit IRC | 00:27 | |
| *** john-davidge has joined #openstack-neutron | 00:28 | |
| *** john-davidge has quit IRC | 00:32 | |
| *** tommylikehu_ has joined #openstack-neutron | 00:33 | |
| *** abhiraut has joined #openstack-neutron | 00:33 | |
| openstackgerrit | Jaivish Kothari(janonymous) proposed openstack/os-vif: Removing Deprecated hacking Check https://review.openstack.org/425996 | 00:36 |
| *** rajinir has quit IRC | 00:36 | |
| openstackgerrit | Swaminathan Vasudevan proposed openstack/neutron: DVR: Add static routes to FIP namespace https://review.openstack.org/308068 | 00:37 |
| *** boden has quit IRC | 00:40 | |
| *** abhiraut has quit IRC | 00:41 | |
| *** MasterOfBugs has quit IRC | 00:42 | |
| *** PramodJ has joined #openstack-neutron | 00:42 | |
| *** pramodrj07 has quit IRC | 00:42 | |
| *** iyamahat_ has joined #openstack-neutron | 00:42 | |
| *** iyamahat has quit IRC | 00:42 | |
| *** MasterOfBugs has joined #openstack-neutron | 00:42 | |
| *** jose-phillips has quit IRC | 00:43 | |
| *** thorst_ has joined #openstack-neutron | 00:45 | |
| *** hfu has joined #openstack-neutron | 00:47 | |
| *** radhikam has quit IRC | 00:47 | |
| *** claudiub has quit IRC | 00:47 | |
| *** abhiraut has joined #openstack-neutron | 00:48 | |
| *** hfu has quit IRC | 00:49 | |
| openstackgerrit | YAMAMOTO Takashi proposed openstack/neutron-vpnaas: gate_hook: Add "tempest" case https://review.openstack.org/426000 | 00:51 |
| *** rcernin has quit IRC | 00:51 | |
| *** numans_ has joined #openstack-neutron | 00:52 | |
| *** belharar_ has quit IRC | 00:53 | |
| *** numans has quit IRC | 00:53 | |
| *** rcernin has joined #openstack-neutron | 00:54 | |
| *** thorst_ has quit IRC | 00:55 | |
| *** rcernin has quit IRC | 00:55 | |
| *** rcernin has joined #openstack-neutron | 00:55 | |
| *** abhiraut has quit IRC | 00:58 | |
| *** abhiraut has joined #openstack-neutron | 01:01 | |
| *** abhiraut has quit IRC | 01:01 | |
| *** ventifus has quit IRC | 01:04 | |
| *** ijw has quit IRC | 01:05 | |
| openstackgerrit | Nate Johnston proposed openstack/neutron-fwaas: Use neutron-lib definition of neutron-fwaas API https://review.openstack.org/421472 | 01:13 |
| Sukhdev_ | ijw: Ping | 01:21 |
| *** leitan has joined #openstack-neutron | 01:22 | |
| *** mriedem is now known as mriedem_afk | 01:23 | |
| openstackgerrit | YAMAMOTO Takashi proposed openstack/neutron-lib: Add a space after comma https://review.openstack.org/420991 | 01:25 |
| openstackgerrit | YAMAMOTO Takashi proposed openstack/neutron-lib: Add FIP64 extension from networking-midonet https://review.openstack.org/416951 | 01:25 |
| openstackgerrit | YAMAMOTO Takashi proposed openstack/neutron-lib: Add router-interface-fip extension from networking-midonet https://review.openstack.org/410001 | 01:25 |
| openstackgerrit | YAMAMOTO Takashi proposed openstack/neutron-lib: Migrate logging-resource extension from networking-midonet https://review.openstack.org/407974 | 01:25 |
| *** ijw has joined #openstack-neutron | 01:26 | |
| *** leitan has quit IRC | 01:27 | |
| *** thorst_ has joined #openstack-neutron | 01:27 | |
| *** iranzo has joined #openstack-neutron | 01:28 | |
| *** iranzo has quit IRC | 01:28 | |
| *** iranzo has joined #openstack-neutron | 01:28 | |
| *** manheim has quit IRC | 01:30 | |
| *** ijw has quit IRC | 01:30 | |
| openstackgerrit | Nate Johnston proposed openstack/neutron: IPtables enhancement for co-existence of SG and FWaaS v2 drivers https://review.openstack.org/348177 | 01:39 |
| *** iranzo has quit IRC | 01:40 | |
| *** liangy has joined #openstack-neutron | 01:44 | |
| openstackgerrit | Nate Johnston proposed openstack/neutron: IPtables enhancement for co-existence of SG and FWaaS v2 drivers https://review.openstack.org/348177 | 01:45 |
| *** Swami_ has joined #openstack-neutron | 01:45 | |
| *** Swami_ has quit IRC | 01:46 | |
| *** tommylikehu_ has quit IRC | 01:47 | |
| *** Swami has quit IRC | 01:48 | |
| *** iwamoto has joined #openstack-neutron | 01:48 | |
| *** tommylikehu_ has joined #openstack-neutron | 01:48 | |
| *** tommylikehu_ has quit IRC | 01:53 | |
| *** tbachman has quit IRC | 01:54 | |
| *** Sukhdev_ has quit IRC | 01:54 | |
| *** tbachman has joined #openstack-neutron | 01:55 | |
| *** tbachman has quit IRC | 01:55 | |
| *** tbachman has joined #openstack-neutron | 01:55 | |
| *** tbachman has quit IRC | 01:59 | |
| *** hichihara has joined #openstack-neutron | 02:01 | |
| openstackgerrit | Merged openstack/neutron: Transition qos notification driver into qos driver https://review.openstack.org/396651 | 02:01 |
| *** hichihara has quit IRC | 02:03 | |
| *** tbachman has joined #openstack-neutron | 02:04 | |
| *** hfu has joined #openstack-neutron | 02:05 | |
| *** tommylikehu_ has joined #openstack-neutron | 02:07 | |
| openstackgerrit | Merged openstack/neutron: adds support for vhost user reconnect. https://review.openstack.org/344997 | 02:08 |
| *** tbachman_ has joined #openstack-neutron | 02:08 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/neutron: Updated from global requirements https://review.openstack.org/423645 | 02:09 |
| *** tbachman has quit IRC | 02:09 | |
| *** tbachman_ is now known as tbachman | 02:09 | |
| openstackgerrit | Merged openstack/neutron: Use unique subnetpools in SubnetPoolPrefixDbObjectTestCase https://review.openstack.org/425409 | 02:09 |
| *** MasterOfBugs has quit IRC | 02:11 | |
| *** dims_ has joined #openstack-neutron | 02:14 | |
| *** PramodJ has quit IRC | 02:14 | |
| *** dims has quit IRC | 02:14 | |
| *** jhershbe has joined #openstack-neutron | 02:15 | |
| *** thorst_ has quit IRC | 02:18 | |
| *** thorst_ has joined #openstack-neutron | 02:20 | |
| *** tommylikehu_ has joined #openstack-neutron | 02:21 | |
| *** tommylikehu_ has quit IRC | 02:22 | |
| *** tommylikehu_ has joined #openstack-neutron | 02:22 | |
| openstackgerrit | Kevin Benton proposed openstack/neutron: Stop passing datapath as 'may_exist' argument https://review.openstack.org/426032 | 02:27 |
| *** john-davidge has joined #openstack-neutron | 02:29 | |
| *** yamahata has quit IRC | 02:29 | |
| *** iyamahat_ has quit IRC | 02:29 | |
| *** kevo has quit IRC | 02:32 | |
| *** john-davidge has quit IRC | 02:33 | |
| *** baoli has joined #openstack-neutron | 02:35 | |
| *** gvrangan has quit IRC | 02:35 | |
| kevinbenton | yamamoto: here :) | 02:35 |
| kevinbenton | yamamoto: i also respun https://review.openstack.org/#/c/419232/ . I remember why i had removed the fixture before | 02:36 |
| kevinbenton | yamamoto: the deepcopy it was using broke trying to deepcopy a weakref | 02:36 |
| *** fzdarsky_ has joined #openstack-neutron | 02:36 | |
| kevinbenton | yamamoto: i fixed it to use a regular copy | 02:36 |
| yamamoto | ah, it makes sense. | 02:36 |
| yamamoto | i'll look | 02:36 |
| kevinbenton | yamamoto: i also have some ovs related things if you have a sec | 02:37 |
| kevinbenton | questions mainly | 02:37 |
| yamamoto | sure | 02:38 |
| kevinbenton | yamamoto: first, i found a small bug. not sure if it was really impacting anything or not. https://review.openstack.org/#/c/426032/ | 02:38 |
| *** jhershbe has quit IRC | 02:38 | |
| kevinbenton | yamamoto: second, what leads to these "datapath invalid exceptions" http://logs.openstack.org/41/425841/1/check/gate-tempest-dsvm-platform-centos7-nv/19326f0/logs/screen-q-agt.txt?level=DEBUG#_2017-01-26_19_53_42_410 | 02:39 |
| *** fzdarsky|afk has quit IRC | 02:39 | |
| kevinbenton | yamamoto: third, I noticed in that line out_group=4294967295,out_port=4294967295, are both unsigned 32 bit int wrap-arounds of -1 (the invalid port) is that a problem? | 02:40 |
| *** absubram has joined #openstack-neutron | 02:43 | |
| yamamoto | "datapath invalid" is from ryu. it usually means the bridge disconnected. | 02:43 |
| *** baoli has quit IRC | 02:45 | |
| kevinbenton | yamamoto: the openflow connection? or the bridge was removed? | 02:45 |
| kevinbenton | yamamoto: ah, which brings me to my last question | 02:46 |
| kevinbenton | yamamoto: 2017-01-26T19:53:25.306Z|00814|rconn|ERR|br-tun<->tcp:127.0.0.1:6633: no response to inactivity probe after 5 seconds, disconnecting | 02:46 |
| kevinbenton | yamamoto: http://logs.openstack.org/41/425841/1/check/gate-tempest-dsvm-platform-centos7-nv/19326f0/logs/openvswitch/ovs-vswitchd.txt.gz | 02:46 |
| kevinbenton | yamamoto: corresponds to that same time | 02:46 |
| *** dims_ has quit IRC | 02:47 | |
| yamamoto | openflow has special constant like 0xffffffff. meaning any/all or something like that. | 02:47 |
| kevinbenton | yamamoto: is it possible there is a blocking issue where we don't respond to a probe on the OF channel? | 02:47 |
| *** d-bark has joined #openstack-neutron | 02:47 | |
| kevinbenton | yamamoto: ok | 02:47 |
| *** absubram has quit IRC | 02:48 | |
| yamamoto | sure, if there's a blocking issue, it would explain those messages. | 02:49 |
| *** absubram has joined #openstack-neutron | 02:49 | |
| kevinbenton | yamamoto: it seems that command blocked everything | 02:50 |
| *** dims has joined #openstack-neutron | 02:50 | |
| kevinbenton | yamamoto: agent freezes for 20 seconds when it issues here http://logs.openstack.org/41/425841/1/check/gate-tempest-dsvm-platform-centos7-nv/19326f0/logs/screen-q-agt.txt?level=DEBUG#_2017-01-26_19_53_18_848 | 02:50 |
| kevinbenton | yamamoto: and during that freeze we get the inactivity probes | 02:50 |
| *** tommylikehu_ has quit IRC | 02:52 | |
| *** tommylikehu_ has joined #openstack-neutron | 02:52 | |
| kevinbenton | yamamoto: what i guess we can't see is if there is a hard freeze in some part of RYU or if RYU is properly yielding out and then ovsdb native is experiencing a block | 02:53 |
| *** TMM has quit IRC | 02:54 | |
| *** TMM has joined #openstack-neutron | 02:54 | |
| yamamoto | sure, but i don't have any idea where it is. it can be even kernel/scheduler issue. | 02:55 |
| *** ijw has joined #openstack-neutron | 02:55 | |
| *** yuanying has quit IRC | 02:56 | |
| yamamoto | iwamoto: do you have any idea? i vaguely remember kakuma investigated similar pause, but it was long ago. | 02:56 |
| kevinbenton | i need to step away for a bit. yamamoto, iwamoto: send me any pointers if you do thing of possible causes. | 02:58 |
| *** tbachman has quit IRC | 02:58 | |
| iwamoto | hi all | 02:59 |
| iwamoto | i need more context to ask kakuma-san | 02:59 |
| *** ijw has quit IRC | 02:59 | |
| yamamoto | iwamoto: no more context than what you can see in the backlog :-) | 03:01 |
| *** tommylikehu_ has quit IRC | 03:02 | |
| yamamoto | mysterious pause of ovs-agent causes ovs disconnect. | 03:02 |
| iwamoto | turning on ryu debug logging might reveal something | 03:04 |
| *** tommylikehu_ has joined #openstack-neutron | 03:06 | |
| *** kuromagi^ has joined #openstack-neutron | 03:07 | |
| *** baoli has joined #openstack-neutron | 03:08 | |
| *** kuromagi has quit IRC | 03:10 | |
| kevinbenton | iwamoto: what's the best way to do that? | 03:10 |
| kevinbenton | iwamoto: regular logging level config? | 03:10 |
| iwamoto | kevinbenton: ryu logging was too verbose and armando changed that a while ago | 03:11 |
| iwamoto | kevinbenton, 5743ed4679d43c325c5b3a2a4955088e35018c97 | 03:12 |
| *** yamahata has joined #openstack-neutron | 03:13 | |
| iwamoto | kakuma-san says increasing ovs timeout value helped :) | 03:16 |
| *** numans_ has quit IRC | 03:22 | |
| *** numans has joined #openstack-neutron | 03:22 | |
| *** leitan has joined #openstack-neutron | 03:23 | |
| iwamoto | sudo ovs-vsctl set Controller br-int inactivity_probe=30000 | 03:24 |
| *** leitan has quit IRC | 03:28 | |
| *** mriedem_afk is now known as mriedem | 03:30 | |
| *** yarkot has joined #openstack-neutron | 03:31 | |
| *** pbandark has joined #openstack-neutron | 03:32 | |
| *** tbachman has joined #openstack-neutron | 03:37 | |
| *** thorst_ has quit IRC | 03:40 | |
| *** psahoo has joined #openstack-neutron | 03:41 | |
| *** tommylikehu_ has quit IRC | 03:41 | |
| *** amotoki has quit IRC | 03:43 | |
| *** gouthamr has quit IRC | 03:43 | |
| *** liangy has quit IRC | 03:47 | |
| *** kevo has joined #openstack-neutron | 04:12 | |
| *** nicolasbock has quit IRC | 04:17 | |
| *** mtreinish has quit IRC | 04:22 | |
| *** mtreinish has joined #openstack-neutron | 04:23 | |
| *** john-davidge has joined #openstack-neutron | 04:29 | |
| *** mdnadeem has joined #openstack-neutron | 04:30 | |
| *** nyechiel_ has joined #openstack-neutron | 04:32 | |
| *** john-davidge has quit IRC | 04:34 | |
| *** nyechiel_ has quit IRC | 04:45 | |
| *** iyamahat has joined #openstack-neutron | 04:46 | |
| *** tommylikehu_ has joined #openstack-neutron | 04:46 | |
| *** tommylikehu_ has quit IRC | 04:49 | |
| *** ayogi has joined #openstack-neutron | 04:50 | |
| *** radhikam has joined #openstack-neutron | 04:50 | |
| *** baoli_ has joined #openstack-neutron | 04:53 | |
| *** tommylikehu_ has joined #openstack-neutron | 04:55 | |
| *** baoli has quit IRC | 04:55 | |
| *** absubram has quit IRC | 04:55 | |
| *** reedip_ has joined #openstack-neutron | 04:56 | |
| *** udesale has joined #openstack-neutron | 04:58 | |
| openstackgerrit | Reedip proposed openstack/neutron-fwaas: TrivialFix: Added link for modindex https://review.openstack.org/426055 | 04:59 |
| *** iyamahat has quit IRC | 05:01 | |
| *** yamahata has quit IRC | 05:01 | |
| iwamoto | kevinbenton, yamamoto: http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22error%20Datapath%20invalid%5C%22 | 05:04 |
| iwamoto | not good | 05:04 |
| *** armax has quit IRC | 05:05 | |
| iwamoto | is there a bug report? | 05:05 |
| *** tommylikehu_ has quit IRC | 05:06 | |
| *** nyechiel_ has joined #openstack-neutron | 05:08 | |
| yamamoto | iwamoto: dunno | 05:08 |
| *** ratailor has joined #openstack-neutron | 05:12 | |
| *** reedip_ has quit IRC | 05:17 | |
| *** janki has joined #openstack-neutron | 05:19 | |
| *** ataraday_ has joined #openstack-neutron | 05:23 | |
| *** leitan has joined #openstack-neutron | 05:25 | |
| *** gvrangan has joined #openstack-neutron | 05:27 | |
| *** tbachman has quit IRC | 05:29 | |
| *** leitan has quit IRC | 05:29 | |
| *** shausy has joined #openstack-neutron | 05:29 | |
| *** jchhatbar has joined #openstack-neutron | 05:30 | |
| *** janki has quit IRC | 05:33 | |
| *** psahoo has quit IRC | 05:35 | |
| openstackgerrit | Yaohua Yan proposed openstack/neutron-fwaas: Deleting firewall in DB is directly operated by plugin, instead of agents' callback https://review.openstack.org/426067 | 05:41 |
| openstackgerrit | Ghanshyam Mann proposed openstack/neutron-fwaas: Switch to decorators.idempotent_id https://review.openstack.org/426069 | 05:41 |
| *** david-lyle has quit IRC | 05:46 | |
| *** psahoo has joined #openstack-neutron | 05:51 | |
| *** karthiks has joined #openstack-neutron | 05:53 | |
| *** nmathew has joined #openstack-neutron | 05:55 | |
| *** Jack_V has joined #openstack-neutron | 05:56 | |
| *** thorst_ has joined #openstack-neutron | 05:56 | |
| *** wolverineav has joined #openstack-neutron | 05:57 | |
| *** absubram has joined #openstack-neutron | 05:58 | |
| *** jchhatbar is now known as janki | 06:00 | |
| *** thorst_ has quit IRC | 06:01 | |
| *** nagarjung has joined #openstack-neutron | 06:01 | |
| *** wolverineav has quit IRC | 06:02 | |
| *** absubram has quit IRC | 06:02 | |
| kevinbenton | iwamoto, yamamoto: the bug is manifesting in different ways | 06:03 |
| kevinbenton | I think there is a bug under ovs timeouts | 06:03 |
| kevinbenton | Setting the inactivity probe high may be a reasonable workaround | 06:04 |
| kevinbenton | But it seems like that's just working around the issue that something is blocking the agent completely | 06:04 |
| *** gvrangan has quit IRC | 06:06 | |
| yamamoto | i agree | 06:08 |
| *** karthiks has quit IRC | 06:09 | |
| yamamoto | making log more verbose might help to isolate where it's blocking | 06:09 |
| *** haleyb has quit IRC | 06:09 | |
| iwamoto | if something is holding eventlet threads, it can be other than ryu-related | 06:11 |
| iwamoto | but we can try | 06:12 |
| openstackgerrit | Ghanshyam Mann proposed openstack/neutron-dynamic-routing: Switch to decorators.idempotent_id https://review.openstack.org/426086 | 06:12 |
| *** pgadiya has joined #openstack-neutron | 06:14 | |
| iwamoto | "no response to inactivity probe" shows a lot of hits. and aren't necessarily cause test failures | 06:15 |
| kevinbenton | iwamoto: but if we get a disconnect during a command it may lead to a failure | 06:16 |
| *** ranjithd has quit IRC | 06:16 | |
| kevinbenton | iwamoto: like the data path invalid one | 06:16 |
| *** bobmel has quit IRC | 06:16 | |
| *** ranjithd has joined #openstack-neutron | 06:17 | |
| yamamoto | maybe we can improve recovery | 06:18 |
| kevinbenton | yamamoto: does ryu have hierarchical exceptions so we could tell what errors are connection related? | 06:19 |
| yamamoto | it has some hierarchy but probably not useful for this purpose | 06:21 |
| *** sridharg has joined #openstack-neutron | 06:22 | |
| *** claudiub has joined #openstack-neutron | 06:25 | |
| *** claudiub|2 has joined #openstack-neutron | 06:28 | |
| *** john-davidge has joined #openstack-neutron | 06:30 | |
| *** claudiub has quit IRC | 06:31 | |
| *** wolverineav has joined #openstack-neutron | 06:33 | |
| *** Sukhdev has joined #openstack-neutron | 06:34 | |
| *** john-davidge has quit IRC | 06:34 | |
| *** iyamahat has joined #openstack-neutron | 06:36 | |
| *** gcheresh_ has joined #openstack-neutron | 06:38 | |
| *** wolverineav has quit IRC | 06:38 | |
| *** psahoo has quit IRC | 06:40 | |
| *** donghao has joined #openstack-neutron | 06:41 | |
| *** horms has joined #openstack-neutron | 06:42 | |
| *** iyamahat has quit IRC | 06:47 | |
| *** moshele has joined #openstack-neutron | 06:48 | |
| *** wolverineav has joined #openstack-neutron | 06:50 | |
| *** gvrangan has joined #openstack-neutron | 06:50 | |
| *** horms has quit IRC | 06:51 | |
| *** eezhova has joined #openstack-neutron | 06:52 | |
| *** pgadiya_ has joined #openstack-neutron | 06:53 | |
| *** wolverineav has quit IRC | 06:54 | |
| *** sputnik13 has quit IRC | 06:55 | |
| *** sputnik13 has joined #openstack-neutron | 06:56 | |
| *** pgadiya has quit IRC | 06:57 | |
| *** moshele has quit IRC | 06:57 | |
| *** karthiks has joined #openstack-neutron | 06:58 | |
| *** psahoo has joined #openstack-neutron | 07:00 | |
| *** gcheresh_ has quit IRC | 07:02 | |
| openstackgerrit | Merged openstack/neutron: Fix a bad docstring in provisioning blocks module https://review.openstack.org/425308 | 07:04 |
| *** yamamoto has quit IRC | 07:05 | |
| *** radhikam has quit IRC | 07:06 | |
| *** amotoki has joined #openstack-neutron | 07:07 | |
| *** hfu has quit IRC | 07:09 | |
| *** Sukhdev has quit IRC | 07:10 | |
| *** adriant has quit IRC | 07:11 | |
| kevinbenton | iwamoto: how does RYU respond to those probes. is it using eventlet? | 07:15 |
| kevinbenton | iwamoto: to maintain a responder? | 07:15 |
| *** hfu has joined #openstack-neutron | 07:17 | |
| *** tobias_fiberdata has joined #openstack-neutron | 07:20 | |
| *** wolverineav has joined #openstack-neutron | 07:21 | |
| *** rcernin has quit IRC | 07:24 | |
| *** leitan has joined #openstack-neutron | 07:26 | |
| *** wolverineav has quit IRC | 07:26 | |
| *** haleyb has joined #openstack-neutron | 07:26 | |
| iwamoto | kevinbenton, it's done around ryu.controller.ofp_handler.OFPHandler iirc. RyuApp is a eventlet thread | 07:26 |
| *** donghao has quit IRC | 07:26 | |
| *** andreas_s has joined #openstack-neutron | 07:26 | |
| *** bbzhao has quit IRC | 07:27 | |
| *** korzen has joined #openstack-neutron | 07:27 | |
| *** bzhao has joined #openstack-neutron | 07:28 | |
| *** leitan has quit IRC | 07:30 | |
| *** gkadam has joined #openstack-neutron | 07:30 | |
| *** wolverineav has joined #openstack-neutron | 07:31 | |
| *** eezhova has quit IRC | 07:36 | |
| *** wolverineav has quit IRC | 07:36 | |
| *** gkadam has quit IRC | 07:42 | |
| *** gkadam has joined #openstack-neutron | 07:45 | |
| *** gkadam has quit IRC | 07:46 | |
| *** gkadam has joined #openstack-neutron | 07:47 | |
| *** ralonsoh has joined #openstack-neutron | 07:50 | |
| *** rubasov has joined #openstack-neutron | 07:55 | |
| openstackgerrit | Rodolfo Alonso Hernandez proposed openstack/neutron: Enforce port QoS policies for all ports. https://review.openstack.org/425280 | 07:55 |
| *** donghao has joined #openstack-neutron | 07:56 | |
| *** donghao has quit IRC | 07:56 | |
| *** thorst_ has joined #openstack-neutron | 07:57 | |
| openstackgerrit | Thomas Morin proposed openstack/neutron: Refactor OVSCookieBridge to always apply extension cookie https://review.openstack.org/425756 | 07:58 |
| *** bfernando has joined #openstack-neutron | 07:59 | |
| *** wolverineav has joined #openstack-neutron | 08:01 | |
| *** yamamoto has joined #openstack-neutron | 08:01 | |
| *** thorst_ has quit IRC | 08:01 | |
| *** nyechiel_ has quit IRC | 08:02 | |
| *** amotoki has quit IRC | 08:05 | |
| *** wolverineav has quit IRC | 08:05 | |
| *** horms has joined #openstack-neutron | 08:08 | |
| *** mickeys has quit IRC | 08:14 | |
| *** mickeys has joined #openstack-neutron | 08:14 | |
| *** eezhova has joined #openstack-neutron | 08:15 | |
| *** john-davidge has joined #openstack-neutron | 08:16 | |
| *** fzdarsky_ is now known as fzdarsky | 08:16 | |
| *** gkadam has quit IRC | 08:16 | |
| *** john-davidge has quit IRC | 08:17 | |
| *** john-davidge has joined #openstack-neutron | 08:17 | |
| *** jpena|off is now known as jpena | 08:19 | |
| *** wolverineav has joined #openstack-neutron | 08:20 | |
| *** wolverineav has quit IRC | 08:25 | |
| *** gkadam has joined #openstack-neutron | 08:26 | |
| *** panda|Zz is now known as panda | 08:26 | |
| *** manheim has joined #openstack-neutron | 08:27 | |
| *** donghao has joined #openstack-neutron | 08:27 | |
| *** wolverineav has joined #openstack-neutron | 08:30 | |
| *** donghao has quit IRC | 08:31 | |
| openstackgerrit | Yushiro FURUKAWA proposed openstack/neutron-specs: (Operator-only) Logging API for security groups https://review.openstack.org/203509 | 08:31 |
| *** pmannidi has quit IRC | 08:32 | |
| *** jpena is now known as jpena|off | 08:32 | |
| *** nyechiel_ has joined #openstack-neutron | 08:34 | |
| *** wolverineav has quit IRC | 08:35 | |
| *** horms has quit IRC | 08:37 | |
| *** gcheresh_ has joined #openstack-neutron | 08:38 | |
| *** jpena|off is now known as jpena | 08:39 | |
| *** nyechiel_ has quit IRC | 08:41 | |
| *** manheim has quit IRC | 08:47 | |
| *** wolverineav has joined #openstack-neutron | 08:50 | |
| *** iranzo has joined #openstack-neutron | 08:50 | |
| *** iranzo has quit IRC | 08:50 | |
| *** iranzo has joined #openstack-neutron | 08:50 | |
| *** matrohon has joined #openstack-neutron | 08:50 | |
| *** wolverineav has quit IRC | 08:54 | |
| *** yamamoto has quit IRC | 08:58 | |
| *** zzzeek has quit IRC | 09:00 | |
| *** wolverineav has joined #openstack-neutron | 09:00 | |
| *** rmart04 has joined #openstack-neutron | 09:01 | |
| *** zzzeek has joined #openstack-neutron | 09:01 | |
| *** ygbo has joined #openstack-neutron | 09:01 | |
| *** iwamoto has quit IRC | 09:03 | |
| openstackgerrit | Daniel Alvarez proposed openstack/neutron: Include port_security check in fullstack tests https://review.openstack.org/425812 | 09:03 |
| dalvarez | ajo bcafarel kevinbenton ^ may you have a look please? thanks a lot | 09:04 |
| *** wolverineav has quit IRC | 09:04 | |
| bcafarel | no more evil whitespace left? :) | 09:05 |
| *** TMM has quit IRC | 09:06 | |
| *** claudiub has joined #openstack-neutron | 09:08 | |
| *** kevo has quit IRC | 09:09 | |
| *** wolverineav has joined #openstack-neutron | 09:10 | |
| *** claudiub|2 has quit IRC | 09:11 | |
| *** manheim has joined #openstack-neutron | 09:14 | |
| *** wolverineav has quit IRC | 09:14 | |
| *** Miouge has joined #openstack-neutron | 09:15 | |
| *** jlibosva has joined #openstack-neutron | 09:15 | |
| *** gkadam has quit IRC | 09:15 | |
| *** manheim has quit IRC | 09:16 | |
| Miouge | ajo: I can’t figure out why gate-neutron-dsvm-functional-ubuntu-xenial fails on https://review.openstack.org/#/c/425218 | 09:17 |
| Miouge | Any tips? | 09:17 |
| *** neiljerram has joined #openstack-neutron | 09:17 | |
| *** jlibosva has quit IRC | 09:18 | |
| *** oreillyd_ has joined #openstack-neutron | 09:20 | |
| *** masaki has quit IRC | 09:20 | |
| matrohon | Hi all | 09:21 |
| *** masaki has joined #openstack-neutron | 09:21 | |
| *** efoley has joined #openstack-neutron | 09:22 | |
| matrohon | we have this bug on kolla : | 09:22 |
| matrohon | https://bugs.launchpad.net/kolla/+bug/1655794 | 09:22 |
| openstack | Launchpad bug 1655794 in kolla "dnsmasq dhcp lease is not cleaned up after instance termination" [Undecided,New] | 09:22 |
| matrohon | I'd like to know if there is a corresponding bug on neutron | 09:22 |
| *** masaki has quit IRC | 09:23 | |
| *** gkadam has joined #openstack-neutron | 09:24 | |
| *** moshele has joined #openstack-neutron | 09:26 | |
| *** jlibosva has joined #openstack-neutron | 09:26 | |
| *** leitan has joined #openstack-neutron | 09:27 | |
| *** vijaykc4 has joined #openstack-neutron | 09:27 | |
| *** donghao has joined #openstack-neutron | 09:28 | |
| *** manheim has joined #openstack-neutron | 09:28 | |
| *** wolverineav has joined #openstack-neutron | 09:29 | |
| *** lucas-afk is now known as lucasagomes | 09:32 | |
| *** leitan has quit IRC | 09:32 | |
| *** tmorin has quit IRC | 09:32 | |
| *** donghao has quit IRC | 09:33 | |
| *** wolverineav has quit IRC | 09:34 | |
| *** amotoki has joined #openstack-neutron | 09:35 | |
| *** karthiks has quit IRC | 09:35 | |
| *** horms has joined #openstack-neutron | 09:36 | |
| *** mickeys has quit IRC | 09:39 | |
| *** wolverineav has joined #openstack-neutron | 09:39 | |
| *** gkadam has quit IRC | 09:41 | |
| *** mhickey has joined #openstack-neutron | 09:41 | |
| ralonsoh | Miouge: this error in neutron.tests.functional.agent.test_dhcp_agent.DHCPAgentOVSTestCase.test_bad_address_allocation is unrelated with your patch | 09:41 |
| *** gkadam has joined #openstack-neutron | 09:41 | |
| Miouge | It sounds like that, but is it why Jenkins is voting -1? | 09:41 |
| ralonsoh | Miouge: because there is an error, that's all | 09:42 |
| *** wolverineav has quit IRC | 09:44 | |
| ralonsoh | Miouge: comment in your patch that the error in gate-neutron-dsvm-functional-ubuntu-xenial is unrelated, and be specific with the test failing | 09:44 |
| Miouge | ralonsoh: ok, thank you for the help | 09:44 |
| *** amotoki has quit IRC | 09:45 | |
| *** hfu has quit IRC | 09:47 | |
| *** hfu has joined #openstack-neutron | 09:48 | |
| *** hfu has quit IRC | 09:49 | |
| *** hfu has joined #openstack-neutron | 09:49 | |
| *** hfu has quit IRC | 09:49 | |
| *** rubasov has quit IRC | 09:50 | |
| *** hfu has joined #openstack-neutron | 09:50 | |
| *** hfu has quit IRC | 09:50 | |
| *** hfu has joined #openstack-neutron | 09:51 | |
| *** hfu has quit IRC | 09:51 | |
| *** karthiks has joined #openstack-neutron | 09:51 | |
| *** baoli_ has quit IRC | 09:51 | |
| *** hfu has joined #openstack-neutron | 09:52 | |
| *** hfu has quit IRC | 09:52 | |
| *** claudiub has quit IRC | 09:52 | |
| *** gabor_antal has joined #openstack-neutron | 09:52 | |
| ralonsoh | moshele: ping | 09:52 |
| moshele | ralonsoh: pong | 09:53 |
| ralonsoh | moshele: about https://review.openstack.org/#/c/409526/ | 09:53 |
| *** iranzo has quit IRC | 09:53 | |
| ralonsoh | moshele: in function "discover_devices" | 09:54 |
| ralonsoh | moshele: if you have two devices (dev1, dev2), both with the same pci address | 09:54 |
| *** tommylikehu_ has joined #openstack-neutron | 09:54 | |
| *** claudiub has joined #openstack-neutron | 09:54 | |
| *** horms has quit IRC | 09:54 | |
| ralonsoh | moshele: in "_create_emb_switch", you will execute two times this code | 09:55 |
| *** hishh has joined #openstack-neutron | 09:55 | |
| ralonsoh | moshele: and in both cases you'll have "embedded_switch = EmbSwitch(phys_net, dev_name, exclude_devices)" with the same info | 09:55 |
| ralonsoh | moshele: because, in your case, both PF can see all VFs | 09:55 |
| *** nplanel has joined #openstack-neutron | 09:56 | |
| moshele | ralonsoh: the dev_name is different | 09:56 |
| ralonsoh | moshele: I know | 09:56 |
| ralonsoh | moshele: but you'll have a list of all VFs with dev1 and another list with all VFs with dev2 | 09:56 |
| ralonsoh | moshele: how can you distinguish which VF belongs to each PFs?? | 09:57 |
| moshele | ralonsoh: by the mac | 09:57 |
| ralonsoh | moshele: reading the VF mac gives you this info? how? | 09:58 |
| *** thorst_ has joined #openstack-neutron | 09:58 | |
| moshele | ralonsoh: because I now what is the mac I want to set on the vm so I just need to find a vf that has that mac (but in the this case on both PFs) | 09:59 |
| *** yamamoto has joined #openstack-neutron | 09:59 | |
| moshele | ralonsoh: I mean in this case I am looking on both PFs | 09:59 |
| moshele | ralonsoh: it is a wired implementation of SR-IOV in one of our legacy card :) | 10:00 |
| ralonsoh | moshele: is there a way to use the code you implemented but having two separate lists? or that's not what you are looking for? | 10:00 |
| ralonsoh | moshele: one list with VFs for dev1 and other with VFs of dev2 | 10:01 |
| *** davidsha has joined #openstack-neutron | 10:01 | |
| moshele | ralonsoh: sorry I don't follow | 10:02 |
| ralonsoh | moshele: I mean, when you build "embedded_switch = EmbSwitch(phys_net, dev_name, exclude_devices)" | 10:02 |
| *** absubram has joined #openstack-neutron | 10:02 | |
| *** thorst_ has quit IRC | 10:02 | |
| ralonsoh | moshele: "embedded_switch" contains all VFs, then remove those ones that doesn't belong to this device | 10:03 |
| moshele | ralonsoh: but the EmbSwitch is per PF | 10:03 |
| moshele | ralonsoh: you want me to create an EmbSwitch that take 2 PFs? | 10:04 |
| ralonsoh | moshele: that's the problem: EmbSwitch is PER pci_address | 10:04 |
| ralonsoh | moshele: self.pci_slot_map[pci_slot] = embedded_switch | 10:05 |
| *** ramishra has quit IRC | 10:05 | |
| *** TMM has joined #openstack-neutron | 10:05 | |
| *** absubram has quit IRC | 10:07 | |
| moshele | ralonsoh: so EmbSwitch represent a PF and I extend the mapping so that self.pci_slot_map will hold list of EmbSwitch | 10:08 |
| *** yamamoto has quit IRC | 10:08 | |
| *** yamamoto has joined #openstack-neutron | 10:08 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/neutron: Updated from global requirements https://review.openstack.org/423645 | 10:08 |
| *** ramishra has joined #openstack-neutron | 10:09 | |
| *** wolverineav has joined #openstack-neutron | 10:09 | |
| ralonsoh | moshele: I know what you want, but because both PFs have the same pci_address and you execute, for example, "clear_max_rate" filtering by pci_address | 10:10 |
| *** horms has joined #openstack-neutron | 10:10 | |
| ralonsoh | moshele: and embedded_switch.get_pci_device(pci_slot) will give you all VFs from both PFs, you don't need this change | 10:10 |
| *** gkadam has quit IRC | 10:12 | |
| *** gkadam has joined #openstack-neutron | 10:12 | |
| *** wolverineav has quit IRC | 10:13 | |
| *** psahoo has quit IRC | 10:14 | |
| *** gcheresh_ has quit IRC | 10:14 | |
| moshele | ralonsoh: yes but remember that if I have 3 VF on PF1 and 2 VF on PF2 I will have 5 VF on each PF. so if I will reset the rate on pci_address_x which is VF 3. it indeed will do it on both PF1 and PF2 but VF3 on PF2 is not in used | 10:15 |
| sean-k-mooney | moshele: in linux netdevs to pci functions is a many to many relation ship with 1:1 being the most common. your legacy card hand sevarl phyicl fucntion netdevs associated with a single pci address. i have also seen the revices where a singel netdev spaned two pci endpoints and the embeded switch formed a lag/bond between them and passed that to the pf netdev | 10:15 |
| sean-k-mooney | nova tracks sriov allocation as a special case of pci-passhtough so it will not account for your legacy pf correctly as they both had the same pci address | 10:17 |
| *** jlibosva has quit IRC | 10:17 | |
| *** d-bark has quit IRC | 10:17 | |
| *** yamamoto has quit IRC | 10:17 | |
| *** tommylikehu_ has quit IRC | 10:18 | |
| ralonsoh | moshele: and take a look at your implementation. if you execute "_clear_rate" for a pci_address (pci_slot), you'll execute this twice in the same VF | 10:18 |
| *** wolverineav has joined #openstack-neutron | 10:19 | |
| *** vijaykc4 has quit IRC | 10:19 | |
| *** yamamoto has joined #openstack-neutron | 10:20 | |
| moshele | sean-k-mooney: yes we won't be able to do PF passhtough with those card, but with the pci regex support I can do VF passhtough https://review.openstack.org/#/c/199488/ | 10:21 |
| sean-k-mooney | moshele: why would you need the regex support for the connect x3 | 10:22 |
| sean-k-mooney | you ould have used the vendorid and product id with the pci adress | 10:23 |
| sean-k-mooney | regex support just meand you have less entries in the pci whitelist | 10:23 |
| *** wolverineav has quit IRC | 10:23 | |
| sean-k-mooney | actully for vf you just needed the vendor id and proct id of the vf which is different form the pf | 10:24 |
| moshele | sean-k-mooney: so I can use the second port for different for different physnet and also less entries in the pci whitelist | 10:24 |
| openstackgerrit | Artur Korzeniewski proposed openstack/neutron: Addressing L3 HA keepalived failures in functional tests https://review.openstack.org/420693 | 10:24 |
| sean-k-mooney | moshele: yes but you could have done that previously with out the regex just with more whitelist entriese so there is no hard dependecy on the regex support | 10:25 |
| sean-k-mooney | moshele: its just a nice to have that make deployment eaisier | 10:25 |
| *** yamamoto has quit IRC | 10:25 | |
| moshele | sean-k-mooney: that true | 10:25 |
| sean-k-mooney | looking at the code it looks like some vendor specific details have crept in | 10:27 |
| *** karthiks has quit IRC | 10:27 | |
| sean-k-mooney | the sriov nic agent was never designed to work with intel nics originaly | 10:27 |
| sean-k-mooney | it now does but you are assuming the embeed swtich is per pf. | 10:28 |
| sean-k-mooney | that does not have to be the case | 10:28 |
| sean-k-mooney | it can be per defvice | 10:28 |
| *** wolverineav has joined #openstack-neutron | 10:28 | |
| moshele | what do you mean per device ? | 10:29 |
| *** gvrangan has quit IRC | 10:29 | |
| sean-k-mooney | per phyical pci addin card | 10:29 |
| moshele | are you talking about multi-pf? | 10:30 |
| sean-k-mooney | yes the embbed switch can span across multiple pfs | 10:30 |
| sean-k-mooney | openstack cant configur that but i am talking about at the silicon level | 10:31 |
| *** murphy_zhao has quit IRC | 10:33 | |
| *** wolverineav has quit IRC | 10:33 | |
| sean-k-mooney | using the flowdirector api im pretry sure you can directly switch traffic between 2 VFs allocated from different PFs via the embeded switch | 10:33 |
| sean-k-mooney | prvided they are form the same addin card | 10:34 |
| *** vijaykc4 has joined #openstack-neutron | 10:35 | |
| sean-k-mooney | if they are not from the same addin card they have to comunicate via the tor | 10:35 |
| sean-k-mooney | the match api intel was working on would have exposed to the linux kernel i belive melenox were working on a similar switchdev? api in the linux kernel? | 10:36 |
| *** vanzyla has joined #openstack-neutron | 10:36 | |
| *** psahoo has joined #openstack-neutron | 10:37 | |
| *** rossella_ has joined #openstack-neutron | 10:39 | |
| moshele | we have VF LAG, where you put both PFs in bond and also the traffic of the VF can go out from 2 VFs on different PFs although we passhtough one of them to the guest | 10:39 |
| *** karthiks has joined #openstack-neutron | 10:39 | |
| *** tmorin has joined #openstack-neutron | 10:39 | |
| *** tommylikehu_ has joined #openstack-neutron | 10:40 | |
| *** tommylikehu_ has quit IRC | 10:42 | |
| moshele | sean-k-mooney: now I am thinking if the agnet_required deprecation was a mistake :) | 10:42 |
| moshele | s/agnet_required/agent_required | 10:43 |
| sean-k-mooney | moshele: hehe perhaps. or as the agent evolves we may require drivers to abstract so vendor specifics | 10:43 |
| sean-k-mooney | i dont know about ye but we have a lot of silicon feature in currnt and past nics that arnt enabled in the linux driver | 10:44 |
| sean-k-mooney | e.g. you have to use the source forge driver, dpdk driver or windows driver to use them | 10:44 |
| sean-k-mooney | for example vlan trunking for sriov vf can only be enabled on intel niantic cards via the dpdk driver not the linux one... | 10:45 |
| sean-k-mooney | s/niantic/x520/ | 10:47 |
| moshele | sean-k-mooney: so maybe we should define abstract driver and work pluggable drivers for that agent | 10:47 |
| *** manheim has quit IRC | 10:47 | |
| moshele | sean-k-mooney: as I remember your dpdk driver requires other data than pci_address | 10:48 |
| sean-k-mooney | moshele: i think that would be a good work item for pike. it would all others such as at&t who developed an sriov agent based on the dpdk driver to intergrate upstream in a common agent | 10:48 |
| moshele | sean-k-mooney: cool is it opensource can you point me to source code? | 10:49 |
| openstackgerrit | Artur Korzeniewski proposed openstack/neutron: Add support for Keepalived VRRP health check https://review.openstack.org/273546 | 10:49 |
| sean-k-mooney | moshele: yes it is opensouce i have not dug too deeply into how it works myself but ill grab the link its on github | 10:50 |
| *** jhershbe has joined #openstack-neutron | 10:50 | |
| sean-k-mooney | moshele: this is the deamon its self https://github.com/att/vfd | 10:52 |
| *** manheim has joined #openstack-neutron | 10:52 | |
| sean-k-mooney | i taught there was some openstack intergration code somewhate but i could be wrong | 10:53 |
| moshele | sean-k-mooney: I remember that code, someone in the SR-IOV sub meeting wanted to add support for this. So I suggested spec and POC code | 10:54 |
| moshele | sean-k-mooney: I never heard from them since... | 10:54 |
| sean-k-mooney | yes i think it provides an alternive command line that basicly the agent could invoke to do the configuration of the vfs | 10:54 |
| sean-k-mooney | i have heard things on and off every so often but nothing since the winter break | 10:55 |
| moshele | sean-k-mooney: do you have some emails/names? maybe they started something | 10:56 |
| *** dimak_ has joined #openstack-neutron | 10:56 | |
| *** yamamoto has joined #openstack-neutron | 10:57 | |
| *** vijaykc4 has quit IRC | 10:57 | |
| sean-k-mooney | backends like snabb switch are built around a similar concept in that they use the sriov silocon features in the nic and the embedded switch which they then expose as vhost-user interfaces to the guest. | 10:57 |
| sean-k-mooney | am not to hand | 10:57 |
| *** ociuhandu has quit IRC | 10:58 | |
| sean-k-mooney | ralonsoh: do you? | 10:58 |
| ralonsoh | sean-k-mooney: one sec | 10:58 |
| sean-k-mooney | moshele: ralonsoh perhaps we should ping them on the mailing list and ask if they have something they could share. | 10:59 |
| *** nmathew has quit IRC | 11:01 | |
| *** udesale has quit IRC | 11:01 | |
| *** dimak_ has quit IRC | 11:02 | |
| *** mvk has quit IRC | 11:06 | |
| *** panda is now known as panda|lunch | 11:07 | |
| *** hishh has quit IRC | 11:08 | |
| *** dsneddon is now known as dsneddon_afk | 11:08 | |
| *** mars[m] has quit IRC | 11:10 | |
| *** vcn[m] has quit IRC | 11:10 | |
| *** pavel_bondar[m] has quit IRC | 11:10 | |
| *** hishh has joined #openstack-neutron | 11:10 | |
| *** hfu has joined #openstack-neutron | 11:13 | |
| *** sambetts|afk is now known as sambetts | 11:14 | |
| *** horms has quit IRC | 11:14 | |
| *** pavel_bondar[m] has joined #openstack-neutron | 11:15 | |
| *** vijaykc4 has joined #openstack-neutron | 11:17 | |
| *** jhershbe has quit IRC | 11:17 | |
| *** jchhatbar has joined #openstack-neutron | 11:21 | |
| *** yamamoto has quit IRC | 11:22 | |
| *** janki has quit IRC | 11:24 | |
| *** jchhatbar is now known as janki | 11:24 | |
| *** chlong has joined #openstack-neutron | 11:24 | |
| *** hishh has quit IRC | 11:24 | |
| *** manheim has quit IRC | 11:26 | |
| *** vijaykc4 has quit IRC | 11:27 | |
| *** vijaykc4 has joined #openstack-neutron | 11:27 | |
| *** vijaykc4 has quit IRC | 11:28 | |
| *** leitan has joined #openstack-neutron | 11:28 | |
| *** vijaykc4 has joined #openstack-neutron | 11:28 | |
| *** donghao has joined #openstack-neutron | 11:29 | |
| *** vijaykc4 has quit IRC | 11:29 | |
| *** vanzyla has quit IRC | 11:30 | |
| *** leitan has quit IRC | 11:33 | |
| *** donghao has quit IRC | 11:33 | |
| *** murphy_zhao has joined #openstack-neutron | 11:34 | |
| *** tbachman has joined #openstack-neutron | 11:35 | |
| openstackgerrit | Merged openstack/neutron: [Live Migration] Extend ml2_port_binding table https://review.openstack.org/404293 | 11:36 |
| *** murphy_zhao has quit IRC | 11:40 | |
| *** tmorin has quit IRC | 11:40 | |
| des_consolado | morning all - I'm quite confused with how to follow this http://docs.openstack.org/newton/networking-guide/config-bgp-dynamic-routing.html | 11:41 |
| *** tbachman has quit IRC | 11:41 | |
| *** gcheresh_ has joined #openstack-neutron | 11:41 | |
| des_consolado | I've used kolla-ansible to deploy and I've got networking working to have instances talking to the outside world. I haven't yet re-configured or customised any services like this though. | 11:41 |
| des_consolado | so on my controller node where neutron-server is running I can easily change the neutron.conf and restart the neutron-server container and I presume that deals with that part of things | 11:43 |
| des_consolado | But about this BGP agent... Well I'm running all services in containers, I guess I need to set up a new container for this agent... | 11:44 |
| openstackgerrit | Artur Korzeniewski proposed openstack/neutron: Add support for Keepalived VRRP health check https://review.openstack.org/273546 | 11:45 |
| *** moshele has quit IRC | 11:53 | |
| *** murphy_zhao has joined #openstack-neutron | 11:53 | |
| *** thorst_ has joined #openstack-neutron | 11:59 | |
| openstackgerrit | Rodolfo Alonso Hernandez proposed openstack/neutron: Enforce port QoS policies for all ports. https://review.openstack.org/425280 | 11:59 |
| *** manheim has joined #openstack-neutron | 12:01 | |
| openstackgerrit | Sam Betts proposed openstack/neutron: Remove baremetal notification from nova notifier https://review.openstack.org/424247 | 12:01 |
| *** bobmel has joined #openstack-neutron | 12:01 | |
| *** leitan has joined #openstack-neutron | 12:02 | |
| *** thorst_ has quit IRC | 12:03 | |
| *** catintheroof has joined #openstack-neutron | 12:04 | |
| *** mvk has joined #openstack-neutron | 12:04 | |
| *** nicolasbock has joined #openstack-neutron | 12:04 | |
| *** bfernando has quit IRC | 12:05 | |
| *** manheim has quit IRC | 12:05 | |
| *** bobmel has quit IRC | 12:06 | |
| *** manheim has joined #openstack-neutron | 12:08 | |
| korzen | Hi haleyb, I have address the comments in https://review.openstack.org/#/c/420693 | 12:10 |
| *** vcn[m] has joined #openstack-neutron | 12:10 | |
| *** mars[m] has joined #openstack-neutron | 12:10 | |
| korzen | haleyb, can you also take a look at: https://review.openstack.org/#/c/273546 | 12:11 |
| *** panda|lunch is now known as panda | 12:14 | |
| *** jpena is now known as jpena|lunch | 12:17 | |
| *** obondarev has joined #openstack-neutron | 12:17 | |
| *** dimak_ has joined #openstack-neutron | 12:18 | |
| *** sridharg has quit IRC | 12:19 | |
| *** lucasagomes is now known as lucas-hungry | 12:21 | |
| *** ganeshkaila has joined #openstack-neutron | 12:22 | |
| *** yamamoto has joined #openstack-neutron | 12:22 | |
| ganeshkaila | Hi @here, the SSH Tunnel 'ssh 8899:ip:80 user@ip' among the openstack instances is getting failed with the error 'bind: Address cannot be assigned'. It is getting succeed when I give command 'ssh -4 8899:ip:80 user@ip' which means forcing ssh to use ipv4. Is there a possibility to make the instances to use ipv4 all over? | 12:25 |
| openstackgerrit | Ann Taraday proposed openstack/neutron: [WIP] Use writer for objects create https://review.openstack.org/426212 | 12:26 |
| *** moshele has joined #openstack-neutron | 12:27 | |
| *** moshele has quit IRC | 12:29 | |
| *** amarao has joined #openstack-neutron | 12:29 | |
| *** yamamoto has quit IRC | 12:31 | |
| *** jhershbe has joined #openstack-neutron | 12:32 | |
| *** ranjithd has quit IRC | 12:33 | |
| *** dimak_ has quit IRC | 12:40 | |
| *** limao has joined #openstack-neutron | 12:40 | |
| openstackgerrit | Maxime Guyot proposed openstack/neutron: Apply QoS policy on network:router_gateway https://review.openstack.org/425218 | 12:40 |
| *** jhershbe has quit IRC | 12:40 | |
| *** sdague has joined #openstack-neutron | 12:40 | |
| *** psahoo has quit IRC | 12:42 | |
| *** thorst__ has joined #openstack-neutron | 12:43 | |
| *** limao_ has joined #openstack-neutron | 12:44 | |
| *** limao has quit IRC | 12:44 | |
| *** tommylikehu_ has joined #openstack-neutron | 12:46 | |
| *** ociuhandu has joined #openstack-neutron | 12:46 | |
| *** sdague_ has joined #openstack-neutron | 12:52 | |
| *** tbachman has joined #openstack-neutron | 12:54 | |
| *** andreas_s has quit IRC | 12:55 | |
| *** limao_ has quit IRC | 12:56 | |
| *** tmorin has joined #openstack-neutron | 12:58 | |
| *** ratailor has quit IRC | 13:03 | |
| *** catintheroof has quit IRC | 13:05 | |
| openstackgerrit | Thomas Morin proposed openstack/neutron: Refactor OVSCookieBridge to always apply extension cookie https://review.openstack.org/425756 | 13:05 |
| *** trown|outtypewww is now known as trown | 13:05 | |
| *** hishh has joined #openstack-neutron | 13:07 | |
| *** obondarev has quit IRC | 13:10 | |
| *** catintheroof has joined #openstack-neutron | 13:16 | |
| *** iranzo has joined #openstack-neutron | 13:16 | |
| openstackgerrit | Thomas Morin proposed openstack/neutron: Refactor OVSCookieBridge to always apply extension cookie https://review.openstack.org/425756 | 13:19 |
| *** tommylikehu_ has quit IRC | 13:19 | |
| *** lucas-hungry is now known as lucasagomes | 13:19 | |
| *** hishh has quit IRC | 13:20 | |
| *** tommylikehu_ has joined #openstack-neutron | 13:20 | |
| *** ayogi has quit IRC | 13:20 | |
| *** jpena|lunch is now known as jpena | 13:22 | |
| *** mdnadeem has quit IRC | 13:23 | |
| *** sdague has quit IRC | 13:25 | |
| *** jlibosva has joined #openstack-neutron | 13:25 | |
| *** pbandark has quit IRC | 13:26 | |
| *** nmathew has joined #openstack-neutron | 13:26 | |
| tmorin | ralonsoh: I've removed the factory function and moved things in br_cookie (https://review.openstack.org/#/c/425756/ ) - thanks for your feedback! | 13:28 |
| *** jhershbe has joined #openstack-neutron | 13:30 | |
| *** pgadiya_ has quit IRC | 13:34 | |
| *** jhershbe has quit IRC | 13:35 | |
| ralonsoh | tmorin: I'll review it now. Thank you | 13:35 |
| tmorin | ralonsoh: many thanks! | 13:36 |
| *** jheroux has joined #openstack-neutron | 13:40 | |
| *** thorst__ is now known as thorst_ | 13:42 | |
| *** tommylikehu_ has quit IRC | 13:49 | |
| *** gouthamr has joined #openstack-neutron | 13:53 | |
| *** trown is now known as trown|brb | 13:57 | |
| *** nmathew has quit IRC | 13:58 | |
| *** ihrachys has joined #openstack-neutron | 14:01 | |
| *** nagarjung has quit IRC | 14:05 | |
| *** trown|brb is now known as trown | 14:09 | |
| *** ratailor has joined #openstack-neutron | 14:09 | |
| *** cleong has joined #openstack-neutron | 14:10 | |
| *** ratailor has quit IRC | 14:11 | |
| *** shausy has quit IRC | 14:11 | |
| *** sdague_ is now known as sdague | 14:12 | |
| *** absubram has joined #openstack-neutron | 14:14 | |
| *** kriskend has joined #openstack-neutron | 14:15 | |
| *** markvoelker has joined #openstack-neutron | 14:16 | |
| *** wolverineav has joined #openstack-neutron | 14:16 | |
| *** kriskend has quit IRC | 14:16 | |
| *** kriskend has joined #openstack-neutron | 14:16 | |
| *** wolverin_ has joined #openstack-neutron | 14:18 | |
| *** absubram has quit IRC | 14:18 | |
| *** murphy_zhao has quit IRC | 14:19 | |
| *** anilvenkata has joined #openstack-neutron | 14:19 | |
| *** gcheresh_ has quit IRC | 14:20 | |
| *** wolverineav has quit IRC | 14:20 | |
| dims | kevinbenton : dasm : ataraday : jlibosva : ajo : a couple of jobs have a high rate of failure in the gate queue - http://tinyurl.com/zwpv823 | 14:20 |
| ajo | dims looking thanks | 14:21 |
| dims | kevinbenton : dasm : ataraday : jlibosva : ajo : if you see http://status.openstack.org/zuul/ you can see we are trailing 12 hours because of these | 14:21 |
| dims | thanks ajo | 14:21 |
| ajo | ouch | 14:21 |
| jlibosva | dims: those data are from gate or check queue? | 14:22 |
| ajo | dims do we know which specific failures are hitting us?, I will investigate otherwise | 14:22 |
| dims | jlibosva you can see the 2 lines one from check and one from gate | 14:22 |
| dims | more worried about the gate than the check honestly | 14:22 |
| jlibosva | dims: oh, sorry. I'm stupid, it's written there in brackets :) | 14:23 |
| dims | ajo : nope. got this graph from sdague on infra channel :) | 14:23 |
| ajo | when the load is higher we fail more | 14:23 |
| ajo | (race conditions, timeouts?) | 14:23 |
| jlibosva | there is still this issue with memory depletion | 14:23 |
| jlibosva | don't know if it's relevant but since full jobs run more tests, it might be related | 14:23 |
| *** ranjithd has joined #openstack-neutron | 14:24 | |
| ajo | jlibosva I checked that, 17 failures in the last 24h | 14:24 |
| ajo | I wouldn't expect that to be a 50% failure rate | 14:24 |
| ajo | there must be something else | 14:24 |
| *** amuller has joined #openstack-neutron | 14:24 | |
| jlibosva | ajo: for full job? | 14:24 |
| *** absubram has joined #openstack-neutron | 14:24 | |
| ajo | jlibosva I don't know time to look for failures, I'll start that | 14:24 |
| *** jperry has joined #openstack-neutron | 14:24 | |
| ajo | jlibosva my thought on timeouts is, if it's happening in functional, nothing stops it from happening in full (under the hood) | 14:25 |
| ajo | it's a bad indicative | 14:25 |
| jlibosva | ajo: oh, I'm not talking about timeouts in ovsdb | 14:25 |
| ajo | jlibosva oh, sorry, what did you mean with "jlibosva:ajo: for full job?" = | 14:26 |
| jlibosva | ajo: I thought you were checking hits for oom-killer: https://bugs.launchpad.net/neutron/+bug/1656386 | 14:26 |
| openstack | Launchpad bug 1656386 in neutron "Memory leaks on Neutron jobs" [Critical,Confirmed] - Assigned to Darek Smigiel (smigiel-dariusz) | 14:26 |
| jlibosva | ajo: I asked whether you limited those hits to the neutron-full jobs only | 14:26 |
| *** zhhuabj has quit IRC | 14:26 | |
| jlibosva | ok, I see you commented there :) | 14:27 |
| *** kriskend has quit IRC | 14:27 | |
| *** kriskend has joined #openstack-neutron | 14:27 | |
| *** kriskend has quit IRC | 14:29 | |
| *** kriskend has joined #openstack-neutron | 14:29 | |
| *** iranzo has quit IRC | 14:30 | |
| ajo | dims, is there anywhere where I can see the specific gate failures for neutron ? | 14:30 |
| *** baoli has joined #openstack-neutron | 14:30 | |
| *** karthiks has quit IRC | 14:30 | |
| ajo | (see which test failed, and logs, etc) | 14:30 |
| dims | ajo : this help? http://status.openstack.org/elastic-recheck/data/integrated_gate.html#gate-tempest-dsvm-neutron-full-ubuntu-xenial | 14:31 |
| *** kbringard has joined #openstack-neutron | 14:36 | |
| *** efoley has quit IRC | 14:37 | |
| *** Dinesh_Bhor has quit IRC | 14:37 | |
| ajo | dims, oh, right! | 14:37 |
| ajo | thanks | 14:37 |
| *** efoley has joined #openstack-neutron | 14:38 | |
| kevinbenton | dims: several of the resets we were hitting yesterday were libvirt issues | 14:38 |
| kevinbenton | dims: the only bug in the gate queue that I know of that is specific to neutron jobs is the oom killer | 14:38 |
| *** zhhuabj has joined #openstack-neutron | 14:39 | |
| dasm | kevinbenton: did your fix solve it? | 14:39 |
| *** efoley has quit IRC | 14:39 | |
| *** tlian has joined #openstack-neutron | 14:39 | |
| dasm | swappiness one? | 14:39 |
| kevinbenton | https://review.openstack.org/#/c/425961/ | 14:39 |
| ajo | kevinbenton do the oom kills happen only for neutron jobs? | 14:39 |
| *** efoley has joined #openstack-neutron | 14:39 | |
| kevinbenton | Still waiting for infra core | 14:39 |
| ajo | https://etherpad.openstack.org/p/neutron-gate-failures-2017-01-27 | 14:39 |
| ajo | I'm going over those | 14:39 |
| ajo | let's see what we find | 14:40 |
| ajo | if anybody want`s to take that list on reverse order, feel free ^ :) | 14:40 |
| kevinbenton | ajo: it looks like it might | 14:40 |
| dasm | ajo: i'll go in reverse order | 14:40 |
| ajo | thanks dasm | 14:40 |
| kevinbenton | ajo: but that's probably just because those jobs run extra services (neutron) to push memory usage higher | 14:41 |
| ajo | kevinbenton yikes :S | 14:41 |
| ajo | kevinbenton what do you mean? | 14:41 |
| kevinbenton | ajo: memory usage of neutron procs is not as bad as some of the others | 14:41 |
| kevinbenton | ajo: the neutron jobs run more services than non neutron | 14:41 |
| ajo | kevinbenton yes, that's what I saw, compared to cinder we're nothing | 14:42 |
| ajo | ah | 14:42 |
| openstackgerrit | Henry Gessau proposed openstack/neutron: [WIP] Operation Pig Bristle, part 7 https://review.openstack.org/409557 | 14:42 |
| openstackgerrit | Henry Gessau proposed openstack/neutron: [WIP] Operation Pig Bristle, part 5 https://review.openstack.org/409556 | 14:42 |
| openstackgerrit | Henry Gessau proposed openstack/neutron: [WIP] Operation Pig Bristle, part 4 https://review.openstack.org/409555 | 14:42 |
| openstackgerrit | Henry Gessau proposed openstack/neutron: [WIP] Operation Pig Bristle, part 3 https://review.openstack.org/409554 | 14:42 |
| openstackgerrit | Henry Gessau proposed openstack/neutron: Decouple hook and func registration from CommonDbMixin https://review.openstack.org/409553 | 14:42 |
| openstackgerrit | Henry Gessau proposed openstack/neutron: [WIP] Operation Pig Bristle, part 6 https://review.openstack.org/407837 | 14:42 |
| kevinbenton | ajo: a neutron server, neutron agents, etc | 14:42 |
| ajo | HenryG++ | 14:42 |
| *** janzian has joined #openstack-neutron | 14:42 | |
| HenryG | ajo: Just keeping my patches rebased with master. How are you my friend? | 14:43 |
| openstackgerrit | Rodolfo Alonso Hernandez proposed openstack/neutron: QoS NIC bandwidth manager for SR-IOV backend https://review.openstack.org/401254 | 14:43 |
| ajo | HenryG: good :D, I'm good :) | 14:44 |
| otherwiseguy | kevinbenton, saw that you pinged me in #openstack-meeting last night? | 14:44 |
| HenryG | ajo: Are going to the Boston summit? | 14:44 |
| kevinbenton | otherwiseguy: something seems to block up everything in the ovs agent | 14:45 |
| kevinbenton | otherwiseguy: so eventlet doesn't yield | 14:46 |
| otherwiseguy | Ok. I'll take a look at removing eventlet. :p | 14:46 |
| kevinbenton | otherwiseguy: subsequently ryu and ovsdb native stops answering | 14:46 |
| otherwiseguy | hmm. | 14:46 |
| kevinbenton | Ovs's phone calls | 14:47 |
| kevinbenton | And we get the disconnects | 14:47 |
| *** dansmith is now known as superdan | 14:47 | |
| ajo | HenryG not sure about boston, only Atlanta for now | 14:47 |
| kevinbenton | At least that's one situation I've observed in the gate | 14:47 |
| otherwiseguy | kevinbenton, got a quick link to log where it happens? | 14:47 |
| HenryG | ajo: No worries. Any time you are near Boston, let me know and we'll meet for drinks. | 14:48 |
| ajo | HenryG I'd prefer not having to travel 4 times a year :D , my wife would hate me if she has to take care of kids for 1 month/year total while I'm abroad | 14:48 |
| ajo | HenryG oohh, so you're in boston now, ack :) I didn't know | 14:48 |
| ajo | thanks HenryG I'll eventually be around, :) | 14:48 |
| ajo | otherwiseguy kevinbenton is it related to the ovs timeouts? | 14:49 |
| kevinbenton | ajo: yes | 14:49 |
| kevinbenton | otherwiseguy: one sec | 14:50 |
| kevinbenton | 2017-01-26T19:53:25.306Z|00814|rconn|ERR|br-tun<->tcp:127.0.0.1:6633: no response to inactivity probe after 5 seconds, disconnecting | 14:52 |
| kevinbenton | http://logs.openstack.org/41/425841/1/check/gate-tempest-dsvm-platform-centos7-nv/19326f0/logs/openvswitch/ovs-vswitchd.txt.gz | 14:52 |
| kevinbenton | agent freezes for 20 seconds when it issues here http://logs.openstack.org/41/425841/1/check/gate-tempest-dsvm-platform-centos7-nv/19326f0/logs/screen-q-agt.txt?level=DEBUG#_2017-01-26_19_53_18_848 | 14:52 |
| ajo | kevinbenton : http://logs.openstack.org/93/404293/44/gate/gate-tempest-dsvm-neutron-full-ubuntu-xenial/2197257/logs/screen-q-agt.txt.gz?#_2017-01-27_09_15_02_712 | 14:52 |
| ajo | one of the unidentified gate failures ^ | 14:52 |
| ajo | does that ring a bell? | 14:52 |
| kevinbenton | otherwiseguy: you can see the ovs side disconnective from ovs agent | 14:53 |
| ajo | when are we supposed to have guru meditation tracebacks there? | 14:53 |
| kevinbenton | otherwiseguy: in the ovs agent you see it try to issue an ofctl command with everything blocks up for 20 seconds | 14:53 |
| *** iyamahat has joined #openstack-neutron | 14:56 | |
| *** mlavalle has joined #openstack-neutron | 14:56 | |
| *** armax has joined #openstack-neutron | 14:56 | |
| *** murphy_zhao has joined #openstack-neutron | 14:57 | |
| kevinbenton | ajo: that's just agent being stopped, no? | 14:57 |
| ajo | kevinbenton hmm, yes, it looks like that ':D | 14:57 |
| ajo | now I just realized | 14:57 |
| ajo | thanks :D | 14:57 |
| ajo | It's the end of execution of tests | 14:58 |
| *** jlibosva1 has joined #openstack-neutron | 15:00 | |
| haleyb | is that from the monkey patching in eventlet_utils ? more curious than anything | 15:02 |
| *** jlibosva has quit IRC | 15:02 | |
| *** tbachman has quit IRC | 15:02 | |
| *** anilvenkata has quit IRC | 15:03 | |
| *** korzen has quit IRC | 15:03 | |
| *** tbachman has joined #openstack-neutron | 15:04 | |
| otherwiseguy | kevinbenton, that looks nasty. researching. | 15:05 |
| *** efoley_ has joined #openstack-neutron | 15:05 | |
| kevinbenton | otherwiseguy: iwamoto had mentioned that someone had run into this and increasing the timeout for the heartbeat was enough to paper over the issue | 15:06 |
| *** absubram has quit IRC | 15:06 | |
| kevinbenton | otherwiseguy: so maybe we do that for now... | 15:08 |
| *** efoley has quit IRC | 15:09 | |
| openstackgerrit | Anindita Das proposed openstack/neutron: [WIP] Integration of Allocation/Endpoints OVO https://review.openstack.org/367810 | 15:10 |
| openstackgerrit | Victor Morales proposed openstack/neutron: Integrate NetworkSegment OVO https://review.openstack.org/385178 | 15:11 |
| *** yamahata has joined #openstack-neutron | 15:12 | |
| *** jamesdenton has joined #openstack-neutron | 15:14 | |
| prometheanfire | john-davidge: updated the pd bug | 15:15 |
| john-davidge | prometheanfire: Thanks! | 15:16 |
| dasm | kevinbenton: seatbelts fastened? sdague gave +W for swappiness :) | 15:16 |
| john-davidge | prometheanfire: Does the rule get added if you remove and re-add the router interface? or not at all? | 15:16 |
| kevinbenton | dasm: yep :) | 15:17 |
| *** anilvenkata has joined #openstack-neutron | 15:19 | |
| *** gouthamr has quit IRC | 15:19 | |
| *** claudiub|2 has joined #openstack-neutron | 15:20 | |
| ajo | otherwiseguy I started this yesterday: https://review.openstack.org/#/c/425623/ but it's probably useless I guess | 15:21 |
| *** claudiub|3 has joined #openstack-neutron | 15:23 | |
| *** claudiub has quit IRC | 15:23 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/neutron: Updated from global requirements https://review.openstack.org/423645 | 15:24 |
| prometheanfire | john-davidge: now my pd subnet's VMs are all missing their addresses :| | 15:24 |
| john-davidge | prometheanfire: :/ | 15:24 |
| prometheanfire | -A neutron-l3-agent-scope -i qr-7bc14490-05 -j MARK --set-xmark 0x4000000/0xffff0000 | 15:24 |
| prometheanfire | might be another problem though (not related), checking | 15:25 |
| *** jaosorior has joined #openstack-neutron | 15:25 | |
| *** claudiub|2 has quit IRC | 15:25 | |
| john-davidge | prometheanfire: no sign of a mangle rule starting -d <prefix> ? | 15:25 |
| prometheanfire | nope | 15:26 |
| prometheanfire | like I said, this is having the patch applied to newton | 15:26 |
| prometheanfire | so maybe something there? | 15:27 |
| prometheanfire | l3 agent hands requests the subnet via pd right? | 15:28 |
| otherwiseguy | kevinbenton, so the ofctl request is the last thing that is made before things freeze, at 19:53:18, we get the 5 sec inactivity probe failure at 19:53:25, so something weird happened between :18 and :20. The DbListCommand looks like it finishes. | 15:28 |
| otherwiseguy | kevinbenton, is this fairly easy to reproduce/does it happen a lot? | 15:28 |
| *** Leo_ has joined #openstack-neutron | 15:29 | |
| kevinbenton | otherwiseguy: not easy to reproduce | 15:29 |
| kevinbenton | otherwiseguy: does seem to happen frequently | 15:29 |
| kevinbenton | otherwiseguy: http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22error%20Datapath%20invalid%5C%22 | 15:29 |
| *** crose has joined #openstack-neutron | 15:34 | |
| ajo | dasm , I need to leave: feel free to check my incomplete stuff if you had time: https://etherpad.openstack.org/p/neutron-gate-failures-2017-01-27 | 15:35 |
| dasm | ajo: sure. i'm on it | 15:35 |
| ajo | mane of them seem nova related | 15:35 |
| ajo | many | 15:35 |
| ajo | dims ^ | 15:35 |
| prometheanfire | john-davidge: ya, my dibbler died | 15:36 |
| ajo | that's early conclusion, let's let dasm dig up further, | 15:36 |
| dims | ajo : thanks for looking, i will take a peek after the release team meeting | 15:38 |
| john-davidge | prometheanfire: Ah, ok :) Hopefully that's all it was | 15:40 |
| prometheanfire | that, and I upgraded the upstream router, breaking pd both up and down stream :P | 15:41 |
| *** gouthamr has joined #openstack-neutron | 15:41 | |
| prometheanfire | https://community.ubnt.com/t5/EdgeMAX/ipv6-Prefix-delegation-for-a-other-Router-Fritzbox/m-p/1535886#M106803 | 15:42 |
| *** gkadam has quit IRC | 15:42 | |
| prometheanfire | that's how to get PD working on and edgerouter | 15:42 |
| prometheanfire | need to patch after every upgrade... | 15:42 |
| openstackgerrit | Merged openstack/neutron-fwaas: TrivialFix: Added link for modindex https://review.openstack.org/426055 | 15:42 |
| *** marst has joined #openstack-neutron | 15:47 | |
| *** princenana has joined #openstack-neutron | 15:48 | |
| *** absubram has joined #openstack-neutron | 15:49 | |
| otherwiseguy | kevinbenton, if it was super frequent, I'd just try forcing ovs-ofctl instead of native just to see if it is something in ryu or the ovsdb stuff that is getting hung. | 15:49 |
| otherwiseguy | kevinbenton, trying to find out how the ryu stuff actually works wrt its event loop/responding to inactivity probes, etc. | 15:50 |
| otherwiseguy | kevinbenton, I've never looked at the actual code of the the openflow side of neutron :p | 15:51 |
| *** fzdarsky is now known as fzdarsky|afk | 15:51 | |
| *** amotoki has joined #openstack-neutron | 15:51 | |
| *** nplanel has quit IRC | 15:54 | |
| *** jlibosva1 is now known as jlibosva | 15:55 | |
| *** amotoki has quit IRC | 15:56 | |
| *** mvk has quit IRC | 15:58 | |
| *** nplanel has joined #openstack-neutron | 15:59 | |
| *** dave-johnston has joined #openstack-neutron | 15:59 | |
| *** ijw has joined #openstack-neutron | 16:00 | |
| jlibosva | kevinbenton: http://logs.openstack.org/12/425812/2/check/gate-neutron-dsvm-functional-ubuntu-xenial/70f2d8a/logs/dsvm-functional-logs/neutron.tests.functional.agent.ovsdb.test_impl_idl.ImplIdlTestCase.test_post_commit_vswitchd_completed_no_failures.txt.gz#_2017-01-27_09_15_18_513 | 16:03 |
| *** murphy_zhao has quit IRC | 16:05 | |
| jlibosva | kevinbenton: http://logs.openstack.org/12/425812/2/check/gate-neutron-dsvm-functional-ubuntu-xenial/70f2d8a/testr_results.html.gz | 16:05 |
| *** efoley__ has joined #openstack-neutron | 16:06 | |
| *** TMM has quit IRC | 16:06 | |
| openstackgerrit | Bertrand Lallau proposed openstack/neutron-fwaas: Fix scale issue by using cast instead of fanout https://review.openstack.org/426287 | 16:06 |
| *** efoley_ has quit IRC | 16:09 | |
| *** donghao has joined #openstack-neutron | 16:11 | |
| *** ventifus has joined #openstack-neutron | 16:11 | |
| *** kriskend has quit IRC | 16:12 | |
| *** efoley__ has quit IRC | 16:12 | |
| jlibosva | kevinbenton: ihrachys otherwiseguy maybe we could enable debug on ovsdb server side for functional jobs? before the timeout there are some debug logs from python ovs - tcp:127.0.0.1:6640: received unexpected reply message | 16:13 |
| openstackgerrit | Bertrand Lallau proposed openstack/neutron-fwaas: Fix scale issue in case of services_sync_needed https://review.openstack.org/424551 | 16:13 |
| *** donghao has quit IRC | 16:13 | |
| otherwiseguy | jlibosva, that message is repeated *a lot* throughout the logs. I remember tracking it down at some point and now I find that I can't remember what the cause was. | 16:14 |
| * otherwiseguy thinks | 16:14 | |
| kevinbenton | jlibosva: for this one | 16:14 |
| kevinbenton | http://logs.openstack.org/12/425812/2/check/gate-neutron-dsvm-functional-ubuntu-xenial/70f2d8a/logs/dsvm-functional-logs/neutron.tests.functional.agent.ovsdb.test_impl_idl.ImplIdlTestCase.test_post_commit_vswitchd_completed_no_failures.txt.gz#_2017-01-27_09_15_18_513 | 16:14 |
| kevinbenton | jlibosva: something blocked the agent for 10 seconds | 16:15 |
| *** gkadam has joined #openstack-neutron | 16:15 | |
| jlibosva | kevinbenton: I was looking at http://logs.openstack.org/17/413017/23/check/gate-telemetry-dsvm-integration-gnocchi-ubuntu-xenial/f36e699/logs/screen-q-agt.txt#_2017-01-26_19_10_28_078 | 16:15 |
| *** iranzo has joined #openstack-neutron | 16:15 | |
| *** iranzo has quit IRC | 16:15 | |
| *** iranzo has joined #openstack-neutron | 16:15 | |
| openstackgerrit | Bertrand Lallau proposed openstack/neutron-fwaas: Fix scale issue by using cast instead of fanout https://review.openstack.org/426287 | 16:15 |
| *** fnaval has joined #openstack-neutron | 16:16 | |
| *** liangy has joined #openstack-neutron | 16:16 | |
| otherwiseguy | jlibosva, ah yes, the unexpected reply messages can happen if a transaction is destroyed before receiving a reply. I think this happens when we do a verify() call and it fails because a column has been updated while we are trying to set and have to retry? | 16:17 |
| jlibosva | kevinbenton: where do you see it was blocked (in the link you sent). The test doesn't even have an agent. It just sets up the ovsdb API and adds a bridge | 16:17 |
| *** trevormc has joined #openstack-neutron | 16:18 | |
| kevinbenton | jlibosva: that period of time where it is only showing _check_child_processes outputs | 16:18 |
| otherwiseguy | in a test environment when we have lots of things adding bridges and deleting them, we'd definitely see a lot of of those messages. | 16:18 |
| kevinbenton | jlibosva: 10 seconds or so | 16:18 |
| *** HenryG has quit IRC | 16:19 | |
| kevinbenton | jlibosva: followed by "neutron.agent.ovsdb.native.vlog [-] 10000-ms timeout __log_wakeup" | 16:19 |
| *** nplanel has quit IRC | 16:20 | |
| *** HenryG has joined #openstack-neutron | 16:20 | |
| kevinbenton | otherwiseguy: that native.vlog uses that custom ovs poller class | 16:20 |
| kevinbenton | it's supposed to be eventletized | 16:21 |
| kevinbenton | https://github.com/openvswitch/ovs/blob/master/python/ovs/poller.py | 16:21 |
| *** eezhova has quit IRC | 16:21 | |
| *** tbachman has quit IRC | 16:22 | |
| *** kriskend has joined #openstack-neutron | 16:22 | |
| prometheanfire | john-davidge: for now, ignore my -1 til I get pd working again | 16:22 |
| john-davidge | prometheanfire: ok sure, let me know if its still not working after that | 16:23 |
| prometheanfire | ya | 16:23 |
| otherwiseguy | kevinbenton, yeah, i wanted to be able to hook the python ovs idl lib's logs into our logging. it's a weird bastardization of python logging to make it match the rest of ovs's logging. | 16:24 |
| otherwiseguy | the ovs lib *should* be eventlet friendly, but ... | 16:24 |
| *** tbachman has joined #openstack-neutron | 16:25 | |
| otherwiseguy | and the native ovsdb stuff I tried to just avoid any kind of decision on an event loop but tried to make it eventlet friendly with threads, etc. which are monkey patched. but i've always been mildly uncomfortable just because of how much I hate eventlet :p | 16:26 |
| kevinbenton | otherwiseguy, jlibosva: this patch was basically added because of a hard block wasn't it? https://github.com/openstack/neutron/commit/c13d722f3913137945c27fcc74371d3316129f30 | 16:29 |
| *** teclator has quit IRC | 16:29 | |
| *** teclator_ has joined #openstack-neutron | 16:29 | |
| jlibosva | kevinbenton: yes | 16:29 |
| dalvarez | jlibosva, kevinbenton, otherwiseguy i hit that issue today and i think it's bug 1627106 in case it may help | 16:29 |
| openstack | bug 1627106 in neutron "TimeoutException while executing test_post_commit_vswitchd_completed_no_failures" [High,In progress] https://launchpad.net/bugs/1627106 - Assigned to Miguel Angel Ajo (mangelajo) | 16:29 |
| *** david-lyle has joined #openstack-neutron | 16:29 | |
| jlibosva | dalvarez: yeah, it is :) | 16:30 |
| jlibosva | kevinbenton: also | 16:30 |
| jlibosva | kevinbenton: I think we started seeing the error after we fixed a bug in that patch | 16:30 |
| kevinbenton | jlibosva: why is it blocking? is the connection already dead at that point? | 16:30 |
| jlibosva | kevinbenton: which was that timeout was in miliseconds or something | 16:30 |
| dalvarez | https://review.openstack.org/#/c/418120/ maybe some other eyes please? | 16:31 |
| *** ralonsoh has quit IRC | 16:31 | |
| dalvarez | jlibosva, otherwiseguy https://review.openstack.org/#/c/425812/2 if you guys have some time pls :))))) <3 | 16:32 |
| jlibosva | kevinbenton: I don't know, by that time it just caused that all tests get stuck and the whole job got killed by global timeout | 16:32 |
| jlibosva | kevinbenton: https://github.com/openstack/neutron/commit/a01ca102 | 16:32 |
| *** thiagolib has joined #openstack-neutron | 16:34 | |
| jlibosva | kevinbenton: iirc the ovs should have some mechanism of reconnecting in case something goes south | 16:35 |
| jlibosva | but by that time of gate getting stuck, it didn't reconnect, so this is more like a workaround for this issue, which we might be still suffering | 16:36 |
| *** panda is now known as panda|bbl | 16:37 | |
| jlibosva | kevinbenton: also a good hint could be https://bugs.launchpad.net/neutron/+bug/1646107 - I think on ovs 2.6 it is more reproducible (or the xenial default) | 16:37 |
| openstack | Launchpad bug 1646107 in neutron "Functional job times out on xenial" [Medium,Confirmed] | 16:37 |
| kevinbenton | ack | 16:37 |
| *** annegentle has joined #openstack-neutron | 16:40 | |
| *** janki has quit IRC | 16:40 | |
| otherwiseguy | kevinbenton, jlibosva: that timeout has always made me a bit uncomfortable since we have a queue for requests and responses that we read one at a time and if we timeout on a get(), it seems like we could get out of sync if we got a late result. I need to work out how to thoroughly test that. | 16:42 |
| prometheanfire | well, can confirm client side is good, looks like I hosed the upstream server, will have to redo that | 16:42 |
| *** tbachman has quit IRC | 16:43 | |
| openstackgerrit | Kevin Benton proposed openstack/neutron: sanity check https://review.openstack.org/426302 | 16:45 |
| *** absubram has quit IRC | 16:47 | |
| * otherwiseguy stares at functional test console for Kevin's sanity check | 16:49 | |
| *** Brenda has quit IRC | 16:49 | |
| *** gkadam has quit IRC | 16:49 | |
| kevinbenton | otherwiseguy: of course that won't be the problem | 16:49 |
| kevinbenton | otherwiseguy: that would be too easy :) | 16:49 |
| *** jlibosva has quit IRC | 16:49 | |
| kevinbenton | otherwiseguy: but if i didn't do it, it would be the problem | 16:49 |
| dasm | otherwiseguy: is it very exciting? | 16:49 |
| kevinbenton | otherwiseguy: it's a quantum bug :) | 16:49 |
| * dasm is thinking also about looking at console :) | 16:49 | |
| *** iranzo has quit IRC | 16:50 | |
| *** kbringard has quit IRC | 16:50 | |
| otherwiseguy | kevinbenton, I can't begin to express how much I hate debugging eventlet-related things. That probably means I just need to get better at it. But I find the monkey patching approach in general hard to stomach. | 16:50 |
| * otherwiseguy tries to adopt a more open mind | 16:51 | |
| *** eezhova has joined #openstack-neutron | 16:52 | |
| otherwiseguy | kevinbenton, there seem to be quite a few passed functional tests at this point. | 16:58 |
| *** anilvenkata has quit IRC | 16:59 | |
| *** crose has quit IRC | 17:00 | |
| *** slaweq has joined #openstack-neutron | 17:02 | |
| *** mhickey has quit IRC | 17:02 | |
| *** matrohon has quit IRC | 17:04 | |
| *** lnicolas1 has quit IRC | 17:05 | |
| *** rajinir has joined #openstack-neutron | 17:05 | |
| kevinbenton | otherwiseguy: yeah | 17:05 |
| kevinbenton | otherwiseguy: the problem with eventlet is only takes one uncooperative function somewhere that blocks | 17:06 |
| kevinbenton | otherwiseguy: to bring the whole system to a halt | 17:06 |
| *** rossella_ has quit IRC | 17:07 | |
| openstackgerrit | Henry Gessau proposed openstack/neutron: [WIP] Operation Pig Bristle, part 7 https://review.openstack.org/409557 | 17:07 |
| openstackgerrit | Henry Gessau proposed openstack/neutron: [WIP] Operation Pig Bristle, part 5 https://review.openstack.org/409556 | 17:07 |
| openstackgerrit | Henry Gessau proposed openstack/neutron: [WIP] Operation Pig Bristle, part 4 https://review.openstack.org/409555 | 17:07 |
| openstackgerrit | Henry Gessau proposed openstack/neutron: [WIP] Operation Pig Bristle, part 6 https://review.openstack.org/407837 | 17:07 |
| *** kbringard has joined #openstack-neutron | 17:10 | |
| *** iyamahat has quit IRC | 17:11 | |
| *** yamahata has quit IRC | 17:11 | |
| *** fzdarsky|afk is now known as fzdarsky | 17:12 | |
| *** slaweq has quit IRC | 17:12 | |
| openstackgerrit | Bertrand Lallau proposed openstack/neutron-fwaas: Fix scale issue in case of services_sync_needed https://review.openstack.org/424551 | 17:18 |
| *** Leo_ has quit IRC | 17:22 | |
| *** Leo_ has joined #openstack-neutron | 17:23 | |
| *** Jeffrey4l_ has quit IRC | 17:24 | |
| openstackgerrit | Sindhu Devale proposed openstack/neutron: RouterPort OVO integration https://review.openstack.org/424154 | 17:24 |
| openstackgerrit | Sindhu Devale proposed openstack/neutron: Router, RouterPort and FloatingIP to OVO https://review.openstack.org/307964 | 17:24 |
| openstackgerrit | Shashank Kumar Shankar proposed openstack/neutron: Integration of Port Binding Level OVO https://review.openstack.org/382037 | 17:24 |
| *** Jeffrey4l_ has joined #openstack-neutron | 17:24 | |
| *** eezhova has quit IRC | 17:27 | |
| *** eezhova has joined #openstack-neutron | 17:27 | |
| *** annegentle has quit IRC | 17:29 | |
| *** rmart04 has quit IRC | 17:29 | |
| openstackgerrit | Kevin Benton proposed openstack/neutron: sanity check https://review.openstack.org/426302 | 17:30 |
| *** annegentle has joined #openstack-neutron | 17:30 | |
| *** slaweq has joined #openstack-neutron | 17:33 | |
| *** iyamahat has joined #openstack-neutron | 17:34 | |
| *** iyamahat has quit IRC | 17:34 | |
| *** nlahouti has joined #openstack-neutron | 17:35 | |
| *** tbachman has joined #openstack-neutron | 17:35 | |
| openstackgerrit | Merged openstack/neutron: Add missing module-level SUBNET ref https://review.openstack.org/417672 | 17:36 |
| *** kevo has joined #openstack-neutron | 17:37 | |
| *** ygbo has quit IRC | 17:38 | |
| *** Leo_ has quit IRC | 17:39 | |
| *** liangy has quit IRC | 17:41 | |
| *** Leo_ has joined #openstack-neutron | 17:42 | |
| *** mriedem is now known as mriedem_beach | 17:42 | |
| *** ociuhandu has quit IRC | 17:43 | |
| *** jpena is now known as jpena|off | 17:46 | |
| openstackgerrit | Kevin Benton proposed openstack/neutron: sanity check https://review.openstack.org/426302 | 17:48 |
| *** Swami has joined #openstack-neutron | 17:51 | |
| *** trown is now known as trown|lunch | 17:51 | |
| kevinbenton | otherwiseguy: that timeout change makes less sense to me than before | 17:51 |
| kevinbenton | otherwiseguy: the "run" function is running in its own python native thread | 17:52 |
| openstackgerrit | Swaminathan Vasudevan proposed openstack/neutron: DVR: Create router to fip namespace connection based on gateway state https://review.openstack.org/283757 | 17:53 |
| kevinbenton | otherwiseguy: so who cares if it blocks forever | 17:53 |
| *** ijw has quit IRC | 17:54 | |
| *** ijw has joined #openstack-neutron | 17:54 | |
| *** radhikam has joined #openstack-neutron | 17:55 | |
| *** lucasagomes is now known as lucas-afk | 17:55 | |
| *** yamahata has joined #openstack-neutron | 17:56 | |
| *** ijw_ has joined #openstack-neutron | 17:56 | |
| *** ijw has quit IRC | 17:56 | |
| kevinbenton | otherwiseguy: ah, nevermind | 17:57 |
| *** harlowja has joined #openstack-neutron | 18:02 | |
| *** oreillyd_ has quit IRC | 18:02 | |
| *** hfu has quit IRC | 18:02 | |
| ihrachys | armax: we probably want to include https://review.openstack.org/#/c/425023/ into rc1 | 18:02 |
| *** chlong has quit IRC | 18:03 | |
| openstackgerrit | Aliaksandr Dziarkach proposed openstack/neutron: Pluggable IPAM rollback mechanism is not robust https://review.openstack.org/390594 | 18:06 |
| *** mickeys has joined #openstack-neutron | 18:07 | |
| *** sambetts is now known as sambetts|afk | 18:07 | |
| *** jose-phillips has joined #openstack-neutron | 18:08 | |
| *** tbachman has quit IRC | 18:10 | |
| *** ataraday_ has quit IRC | 18:11 | |
| *** tbachman has joined #openstack-neutron | 18:12 | |
| *** ijw_ has quit IRC | 18:12 | |
| *** donghao has joined #openstack-neutron | 18:14 | |
| *** Leom has joined #openstack-neutron | 18:15 | |
| *** Leo_ has quit IRC | 18:15 | |
| *** manheim has quit IRC | 18:16 | |
| *** mvk has joined #openstack-neutron | 18:16 | |
| *** amarao has quit IRC | 18:17 | |
| *** donghao has quit IRC | 18:18 | |
| openstackgerrit | Brian Haley proposed openstack/neutron: Update in-tree code to use new neighbour functions https://review.openstack.org/420209 | 18:19 |
| *** tbachman has quit IRC | 18:20 | |
| *** gvrangan has joined #openstack-neutron | 18:21 | |
| ihrachys | kevinbenton: we fixed 'RuntimeError: Failed to start 2 workers.' functional test failure already right? | 18:23 |
| *** tbachman has joined #openstack-neutron | 18:24 | |
| kevinbenton | ihrachys: i think so with the psutils patch | 18:24 |
| ihrachys | ok makes sense. I will ignore those in my triaging then. | 18:24 |
| *** jose-phillips has quit IRC | 18:26 | |
| openstackgerrit | Thomas Morin proposed openstack/neutron: Fix WaitTimeout string representations https://review.openstack.org/425924 | 18:29 |
| *** ranjithd has quit IRC | 18:32 | |
| *** vhosakot has joined #openstack-neutron | 18:32 | |
| *** davidsha has quit IRC | 18:33 | |
| kevinbenton | ihrachys, armax: time for end of cycle dhcp agent bugs :) https://bugs.launchpad.net/neutron/+bug/1659919 | 18:34 |
| openstack | Launchpad bug 1659919 in neutron "dhcp agent race with port update and network create" [Undecided,New] | 18:34 |
| * ihrachys rages in and merges all the things | 18:35 | |
| openstackgerrit | Kevin Benton proposed openstack/neutron: Always acquire network.id lock in dhcp port update https://review.openstack.org/426339 | 18:38 |
| *** david-lyle has quit IRC | 18:39 | |
| *** john-davidge has quit IRC | 18:39 | |
| openstackgerrit | Kevin Benton proposed openstack/neutron: sanity check https://review.openstack.org/426302 | 18:42 |
| *** abhiraut has joined #openstack-neutron | 18:43 | |
| *** slaweq has quit IRC | 18:45 | |
| *** dave-johnston has quit IRC | 18:45 | |
| *** iranzo has joined #openstack-neutron | 18:46 | |
| openstackgerrit | Victor Morales proposed openstack/neutron: Integrate Security Groups OVO https://review.openstack.org/284738 | 18:46 |
| *** slaweq has joined #openstack-neutron | 18:47 | |
| *** slaweq has quit IRC | 18:52 | |
| *** slaweq has joined #openstack-neutron | 18:52 | |
| *** abhiraut has quit IRC | 18:53 | |
| *** abhiraut has joined #openstack-neutron | 18:53 | |
| *** ijw has joined #openstack-neutron | 18:56 | |
| *** beagles is now known as beagles-biab | 18:58 | |
| *** liangy has joined #openstack-neutron | 19:00 | |
| *** gcheresh_ has joined #openstack-neutron | 19:01 | |
| *** manheim has joined #openstack-neutron | 19:08 | |
| *** ranjithd has joined #openstack-neutron | 19:09 | |
| *** abhiraut has quit IRC | 19:10 | |
| *** absubram has joined #openstack-neutron | 19:10 | |
| *** obondarev has joined #openstack-neutron | 19:11 | |
| *** abhiraut has joined #openstack-neutron | 19:11 | |
| *** donghao has joined #openstack-neutron | 19:15 | |
| *** radhikam has quit IRC | 19:15 | |
| *** absubram has quit IRC | 19:15 | |
| *** hishh has joined #openstack-neutron | 19:16 | |
| *** Sukhdev has joined #openstack-neutron | 19:18 | |
| *** absubram has joined #openstack-neutron | 19:18 | |
| *** radhikam has joined #openstack-neutron | 19:19 | |
| *** slaweq has quit IRC | 19:19 | |
| *** donghao has quit IRC | 19:19 | |
| *** kbaegis has joined #openstack-neutron | 19:21 | |
| kbaegis | Hi all. Having some issue with the stable/newton odl plugin | 19:21 |
| kbaegis | My devstack VM comes up and stacks fine, but about 80% of the time I lose network connectivity from br-ext to the Virtualbox networking stack | 19:23 |
| openstackgerrit | boden proposed openstack/neutron-lib: Remove devref modindex ref https://review.openstack.org/426368 | 19:24 |
| *** tbachman has quit IRC | 19:26 | |
| openstackgerrit | Kevin Benton proposed openstack/neutron: Always acquire network.id lock in dhcp port update https://review.openstack.org/426339 | 19:26 |
| kevinbenton | haleyb: ping | 19:26 |
| *** trown|lunch is now known as trown | 19:27 | |
| *** obondarev has quit IRC | 19:28 | |
| *** gcheresh_ has quit IRC | 19:30 | |
| *** Matias has quit IRC | 19:32 | |
| *** Matias has joined #openstack-neutron | 19:33 | |
| haleyb | kevinbenton: pong | 19:35 |
| kevinbenton | haleyb: can you take a quick look at https://review.openstack.org/426339 | 19:36 |
| kevinbenton | haleyb: traced a random gate failure down to it | 19:36 |
| *** ikeyman has joined #openstack-neutron | 19:36 | |
| haleyb | kevinbenton: why was that done like that? to try and save on a lock? seems like a good change | 19:39 |
| *** jheroux has quit IRC | 19:43 | |
| *** tbachman has joined #openstack-neutron | 19:47 | |
| *** ijw_ has joined #openstack-neutron | 19:47 | |
| *** jose-phillips has joined #openstack-neutron | 19:49 | |
| *** ijw has quit IRC | 19:49 | |
| openstackgerrit | Isaku Yamahata proposed openstack/neutron: Kill the metadata proxy process unconditionally https://review.openstack.org/411566 | 19:51 |
| *** tbachman has quit IRC | 19:52 | |
| *** d-bark has joined #openstack-neutron | 19:53 | |
| *** david-lyle has joined #openstack-neutron | 19:53 | |
| *** david-lyle has quit IRC | 19:53 | |
| *** david-lyle has joined #openstack-neutron | 19:54 | |
| dasm | armax: ihrachys: fyi. i'm taking 2 weeks PTO, starting from Monday. Will be back around Feb 13 | 19:57 |
| ihrachys | ack | 19:57 |
| armax | l | 19:57 |
| armax | k | 19:57 |
| dasm | armax: ihrachys i'll be checking irc/ml but won't be able to release rc1 | 19:57 |
| *** crose has joined #openstack-neutron | 19:58 | |
| armax | ack | 19:58 |
| armax | thanks for the heads up | 19:58 |
| dasm | np. have fun with last days of Ocata ;) | 19:58 |
| *** manheim has quit IRC | 19:58 | |
| *** Jack_V has quit IRC | 19:58 | |
| *** manheim has joined #openstack-neutron | 19:59 | |
| *** beagles-biab is now known as beagles | 20:00 | |
| *** Leom_ has joined #openstack-neutron | 20:01 | |
| *** david-lyle has quit IRC | 20:01 | |
| *** david-lyle has joined #openstack-neutron | 20:01 | |
| *** tbachman has joined #openstack-neutron | 20:01 | |
| *** manheim has quit IRC | 20:01 | |
| *** Leom has quit IRC | 20:03 | |
| *** crose has quit IRC | 20:06 | |
| *** nlahouti has quit IRC | 20:08 | |
| *** ikeyman has quit IRC | 20:10 | |
| *** dsneddon_afk is now known as dsneddon | 20:12 | |
| *** MasterOfBugs has joined #openstack-neutron | 20:12 | |
| *** Miouge has quit IRC | 20:15 | |
| *** pramodrj07 has joined #openstack-neutron | 20:15 | |
| *** lnicolas has joined #openstack-neutron | 20:15 | |
| *** Leo_ has joined #openstack-neutron | 20:17 | |
| *** Leom_ has quit IRC | 20:17 | |
| ihrachys | armax: kevinbenton: ok I think I have a list of issues that happened Jan26+ at https://etherpad.openstack.org/p/neutron-upstream-ci line 20+ | 20:18 |
| * armax looks | 20:18 | |
| *** MasterOfBugs has quit IRC | 20:18 | |
| ihrachys | what pops out is, multiple tests in test_dhcp_agent fails with device XXX does not exist | 20:18 |
| armax | kill dhcp | 20:18 |
| armax | make it disabled by default and deprecate it as of now | 20:19 |
| armax | problem solved | 20:19 |
| ihrachys | they fail a bit differently, but I think it's the same issue | 20:19 |
| armax | DHCP is so yesterday anyway | 20:19 |
| *** amarao has joined #openstack-neutron | 20:19 | |
| ihrachys | then we have a bunch of tests that just fail after test case timeout - absolutely random tests, from db migration to ovs agent | 20:19 |
| ihrachys | there is a fix for a symptom of those in https://review.openstack.org/#/c/425924/ but that won't resolve the root cause | 20:20 |
| ihrachys | and finally we have ovs native (?) timeout on adding bridge, and metadata proxy startup failure | 20:21 |
| ihrachys | that seems all there was in last 1d+ | 20:21 |
| ihrachys | I will have a look at dhcp agent one, seems weird it's only that test, maybe we do something wrong in service setup | 20:21 |
| ihrachys | and it may be as well the same ovs add-bridge timeout issue, though I see at least one failure that complains about tap missing, not bridge | 20:22 |
| *** abhiraut has quit IRC | 20:24 | |
| *** gvrangan has quit IRC | 20:27 | |
| *** slaweq has joined #openstack-neutron | 20:28 | |
| *** panda|bbl is now known as panda | 20:28 | |
| *** jose-phillips has quit IRC | 20:29 | |
| *** Leo_ has quit IRC | 20:30 | |
| *** kevo has quit IRC | 20:30 | |
| *** Leo_ has joined #openstack-neutron | 20:31 | |
| openstackgerrit | Thomas Morin proposed openstack/neutron: Refactor OVSCookieBridge to always apply extension cookie https://review.openstack.org/425756 | 20:32 |
| ihrachys | ok, all dhcp failures are actually add-bridge-timeout failures | 20:33 |
| *** dsneddon is now known as dsneddon_afk | 20:34 | |
| ihrachys | kevinbenton: otherwiseguy: http://logs.openstack.org/43/408143/9/check/gate-neutron-dsvm-functional-ubuntu-xenial/8926e44/logs/dsvm-functional-logs/neutron.tests.functional.agent.test_dhcp_agent.DHCPAgentOVSTestCase.test_agent_mtu_set_on_interface_driver.txt.gz#_2017-01-27_02_41_16_452 does it ring a bell? | 20:37 |
| haleyb | i'm seeing a failure in the rally py35 tests now, timeout deleting dhcp port, what's with the rash of dhcp issues? | 20:38 |
| haleyb | http://logs.openstack.org/09/420209/7/check/gate-rally-dsvm-py35-neutron-neutron-ubuntu-xenial/cc405f8/logs/screen-q-svc.txt.gz?#_2017-01-27_19_24_53_668 | 20:39 |
| *** Leo_ has quit IRC | 20:40 | |
| *** Leo_ has joined #openstack-neutron | 20:40 | |
| openstackgerrit | John Perkins proposed openstack/neutron: Integrate oslo.config validator https://review.openstack.org/426397 | 20:41 |
| *** vnovakov has joined #openstack-neutron | 20:41 | |
| *** hishh has quit IRC | 20:43 | |
| openstackgerrit | Anindita Das proposed openstack/neutron: [WIP] Live Migration New Port Bindings Extension https://review.openstack.org/414251 | 20:46 |
| openstackgerrit | Thomas Morin proposed openstack/neutron: Refactor OVSCookieBridge to always apply extension cookie https://review.openstack.org/425756 | 20:46 |
| *** claudiub has joined #openstack-neutron | 20:47 | |
| openstackgerrit | John Perkins proposed openstack/neutron: Integrate oslo.config validator https://review.openstack.org/426397 | 20:47 |
| slaweq | haleyb: hello | 20:47 |
| haleyb | slaweq: hi | 20:47 |
| slaweq | haleyb: I'm checking what is reason of issue https://bugs.launchpad.net/neutron/+bug/1657260 and I think I found it | 20:47 |
| openstack | Launchpad bug 1657260 in neutron "Established connection don't stops when rule is removed" [Undecided,New] - Assigned to Slawek Kaplonski (slaweq) | 20:47 |
| slaweq | it's missing CT zone in iptables config | 20:48 |
| slaweq | and conntrack is called with "-w 1" parameter | 20:48 |
| slaweq | that don't remove entry from conntrack | 20:48 |
| openstackgerrit | John Perkins proposed openstack/neutron: Integrate oslo.config validator https://review.openstack.org/426397 | 20:48 |
| *** armax has quit IRC | 20:49 | |
| slaweq | if I manually run this conntrack -D command on my host it works fine when it was without "-w 1" | 20:49 |
| slaweq | but I'm not an expert of linuxbridge and iptables driver so I want to ask if it makes sense for You? :) | 20:49 |
| *** claudiub|3 has quit IRC | 20:49 | |
| *** amuller has quit IRC | 20:49 | |
| *** fzdarsky has quit IRC | 20:50 | |
| *** gvrangan has joined #openstack-neutron | 20:50 | |
| haleyb | slaweq: so is the zone not created? this was only with linuxbridge agent not ovs, right? looking at bug again | 20:51 |
| slaweq | for me it looks like zone is not created | 20:51 |
| *** gcheresh_ has joined #openstack-neutron | 20:51 | |
| slaweq | and in fact there is no code to create zone in IptablesFirewallDriver class | 20:52 |
| slaweq | it is only in OVSHybrid driver which is not used by linuxbridge | 20:52 |
| *** jose-phillips has joined #openstack-neutron | 20:52 | |
| *** wolverin_ is now known as wolverineav | 20:53 | |
| haleyb | slaweq: i can't remember if that was intentional, it's been a while since that CT code was added | 20:54 |
| *** hishh has joined #openstack-neutron | 20:54 | |
| haleyb | but maybe finding why -w is used when there is no zone is quicker to get this fixed | 20:54 |
| slaweq | haleyb: I don't know it too and I wanted to ask if I should try to add support for zones in Iptables Driver and linuxbridge agent or maybe (what IMHO should be easier) to remove this zone from conntrack command in case if zone is not used | 20:55 |
| slaweq | what You think? | 20:55 |
| *** sdague has quit IRC | 20:56 | |
| haleyb | slaweq: let me see if i can find the change that added it, think it was kevinbenton, he adds all the cool stuff :) | 20:57 |
| slaweq | https://review.openstack.org/#/c/118274/ | 20:57 |
| slaweq | it was probably this one | 20:57 |
| slaweq | it's quite old patch | 20:58 |
| haleyb | slaweq: yes, and that was reverted and re-added i think | 20:58 |
| *** jose-phillips has quit IRC | 20:58 | |
| slaweq | so if it's that then this feature was not working for LB agent for quite long time :) | 20:58 |
| slaweq | haleyb: I found it from "git blame" so it's in current code with this commit id | 20:59 |
| *** Leom has joined #openstack-neutron | 20:59 | |
| *** Leo_ has quit IRC | 21:00 | |
| haleyb | slaweq: so the zones are required due to overlapping IP, and of course that commit message says "Currently this patch only works for OVS agent" | 21:01 |
| slaweq | yes | 21:01 |
| slaweq | and it cause bug with LB agent :/ | 21:01 |
| slaweq | I think it should be quite easy to add zones to IptablesFirewallDriver class (but I might be wrong) | 21:02 |
| slaweq | what You think? is it good idea to try to go this way to fix this issue? | 21:02 |
| *** sam_s has quit IRC | 21:02 | |
| *** liangy has quit IRC | 21:03 | |
| *** haplo37_ has quit IRC | 21:05 | |
| haleyb | slaweq: i don't know if it's easy or not from looking, but it might not be required, i'm thinking if it's partly due to DVR on computes | 21:06 |
| slaweq | haleyb: ok, I will try to do something with it :) | 21:06 |
| slaweq | to fix this issue | 21:06 |
| slaweq | can You maybe update status of this bug on launchpad? or it's not necessary? | 21:07 |
| kevinbenton | slaweq, haleyb: adding zones to linuxbridge shouldn't be needed | 21:07 |
| *** sdague has joined #openstack-neutron | 21:07 | |
| *** vhosakot has quit IRC | 21:08 | |
| haleyb | kevinbenton: was it partly due to dvr? i'm blanking | 21:08 |
| kevinbenton | haleyb: it was because when two nodes communicated on the same host | 21:08 |
| *** vhosakot has joined #openstack-neutron | 21:08 | |
| kevinbenton | haleyb: it would cross iptables twice | 21:08 |
| *** thiagolib has quit IRC | 21:08 | |
| kevinbenton | haleyb: once on the filtering bridge between the sender and OVS | 21:09 |
| kevinbenton | haleyb: and once again on the filtering bridge between the receiver and OVS | 21:09 |
| slaweq | kevinbenton: hello, so I should change IpConntrackManager class? | 21:09 |
| haleyb | kevinbenton: ah, so the CT entries were "crossing" so to speak, with the wrong one getting hit | 21:09 |
| kevinbenton | haleyb: well what would happen is when one side would generate a TCP reset, iptables would immediately removed the tracked connection on the first cross | 21:10 |
| kevinbenton | haleyb: then when it would hit iptables on the receiver, it wouldn't match an active connection so the TCP RST would be dropped | 21:10 |
| kevinbenton | haleyb: and the instance would never get the reset | 21:10 |
| kevinbenton | slaweq: yeah, is it easy to tell from there if there is no conntrack zone set? | 21:11 |
| haleyb | kevinbenton: so it sounds like in this case, the zone isn't in use, but conntrack -D is using the argument when it shouldn't be | 21:11 |
| kevinbenton | haleyb: yeah, sounds that way | 21:11 |
| slaweq | haleyb: exactly | 21:11 |
| *** cleong has quit IRC | 21:13 | |
| kevinbenton | haleyb, slaweq: however, upon re-explaining that, it occured to me that this can still happen with linuxbridge if the instances are on two different networks | 21:13 |
| kevinbenton | :) | 21:13 |
| kevinbenton | so we probably should add zone support in | 21:13 |
| *** eezhova has quit IRC | 21:13 | |
| kevinbenton | if they are on two networks communicating via a router, a single packet will flow through iptables twice (once on each network bridge) | 21:13 |
| kevinbenton | slaweq: i would proceed with the easy fix for now so we can back-port it | 21:14 |
| kevinbenton | then we should probably add CT support to the regular iptables driver linux bridge uses | 21:14 |
| slaweq | kevinbenton: ok, so I will try to make easy fix to conntrack manager now | 21:14 |
| slaweq | thx for help | 21:15 |
| kevinbenton | no prob | 21:15 |
| kevinbenton | ihrachys, otherwiseguy: does OVS support instant messenger lingo now? | 21:15 |
| kevinbenton | maybe 'k' was meant to be an acknowledgement | 21:15 |
| ihrachys | :) | 21:16 |
| haleyb | slaweq: does a grep -e "Assigned CT" in the log show just a single zone in use? | 21:16 |
| *** donghao has joined #openstack-neutron | 21:16 | |
| *** haplo37_ has joined #openstack-neutron | 21:16 | |
| slaweq | haleyb: You mean in linuxbridge agent log? | 21:16 |
| haleyb | slaweq: yes, i see a different zone for each port in the ovs agent log | 21:16 |
| ihrachys | kevinbenton: one other thing in the same log that I can't figure out is - we see that timeout happening for the bridge @ ...:16, but then later in the run, just before deleteBridge operation, @ :17, we have another attempt to add the bridge | 21:17 |
| kevinbenton | haleyb: to answer your earlier question about my dhcp lock patch, yeah, the original thought was to save an unnecessary lock | 21:17 |
| ihrachys | here: http://logs.openstack.org/43/408143/9/check/gate-neutron-dsvm-functional-ubuntu-xenial/8926e44/logs/dsvm-functional-logs/neutron.tests.functional.agent.test_dhcp_agent.DHCPAgentOVSTestCase.test_agent_mtu_set_on_interface_driver.txt.gz#_2017-01-27_02_41_17_544 | 21:17 |
| haleyb | slaweq: or maybe there is no assigned zone of course | 21:17 |
| ihrachys | note it happens after (re?) connect to ovsdb tcp port | 21:17 |
| ihrachys | a second later | 21:18 |
| slaweq | haleyb: there is no such entry in log file | 21:18 |
| *** vhosakot has quit IRC | 21:18 | |
| *** kevo has joined #openstack-neutron | 21:18 | |
| *** leitan has quit IRC | 21:19 | |
| ihrachys | which probably correspond to https://github.com/openstack/neutron/blob/master/neutron/agent/ovsdb/native/connection.py#L145 | 21:19 |
| slaweq | but when I removed SG rule I saw that conntrack was called with "-w 1" for one port and for "-w 2" for second port from same network | 21:19 |
| kevinbenton | ihrachys: this may be what otherwiseguy was worried about | 21:19 |
| slaweq | soe it looks that conntrack manager assigns zone "per port" | 21:19 |
| kevinbenton | ihrachys: us using timeout on the poller may leave the previous command in the queue | 21:19 |
| ihrachys | yea | 21:19 |
| ihrachys | that's what I was about to suggest | 21:19 |
| kevinbenton | slaweq: yeah, zones are per port | 21:20 |
| kevinbenton | slaweq: the original per network zones were for overlapping IP issues | 21:20 |
| *** donghao has quit IRC | 21:20 | |
| kevinbenton | slaweq: the per port zones were added later for the double cross issue and TCP resets | 21:20 |
| ihrachys | kevinbenton: and also, shouldn't we see connecting messages to the port before first attempt to add the bridge? | 21:20 |
| *** eezhova has joined #openstack-neutron | 21:20 | |
| ihrachys | or do we reuse some connection pool in ovs lib? | 21:20 |
| kevinbenton | ihrachys: connection is stored on class level attribute IIRC in ovs lib | 21:21 |
| kevinbenton | ihrachys: so it is re-used | 21:21 |
| *** panda is now known as panda|zZ | 21:24 | |
| *** nlahouti has joined #openstack-neutron | 21:24 | |
| *** dsneddon has joined #openstack-neutron | 21:25 | |
| *** armax has joined #openstack-neutron | 21:26 | |
| *** eezhova_ has joined #openstack-neutron | 21:28 | |
| *** eezhova has quit IRC | 21:29 | |
| *** tbachman has quit IRC | 21:31 | |
| *** absubram has quit IRC | 21:33 | |
| *** abhiraut has joined #openstack-neutron | 21:33 | |
| *** Jeffrey4l_ has quit IRC | 21:35 | |
| openstackgerrit | Henry Gessau proposed openstack/neutron: [WIP] Operation Pig Bristle, part 7 https://review.openstack.org/409557 | 21:36 |
| openstackgerrit | Henry Gessau proposed openstack/neutron: [WIP] Operation Pig Bristle, part 5 https://review.openstack.org/409556 | 21:36 |
| openstackgerrit | Henry Gessau proposed openstack/neutron: [WIP] Operation Pig Bristle, part 4 https://review.openstack.org/409555 | 21:36 |
| openstackgerrit | Henry Gessau proposed openstack/neutron: [WIP] Operation Pig Bristle, part 3 https://review.openstack.org/409554 | 21:36 |
| openstackgerrit | Henry Gessau proposed openstack/neutron: Decouple hook and func registration from CommonDbMixin https://review.openstack.org/409553 | 21:36 |
| openstackgerrit | Henry Gessau proposed openstack/neutron: [WIP] Operation Pig Bristle, part 6 https://review.openstack.org/407837 | 21:36 |
| *** gcheresh_ has quit IRC | 21:40 | |
| *** vhosakot has joined #openstack-neutron | 21:40 | |
| *** Jeffrey4l_ has joined #openstack-neutron | 21:41 | |
| *** nplanel has joined #openstack-neutron | 21:41 | |
| *** tbachman has joined #openstack-neutron | 21:42 | |
| *** catintheroof has quit IRC | 21:44 | |
| *** catintheroof has joined #openstack-neutron | 21:45 | |
| kevinbenton | slaweq, haleyb: yeah, i remember now that when the zones were originally added it was per network to solve the overlapping IP issue | 21:47 |
| kevinbenton | which linuxbridge didn't have at all | 21:47 |
| kevinbenton | Then when they were changed to per port to deal with the multiple table hits, I forgot to consider that linux bridge would have that issue as well | 21:48 |
| openstackgerrit | Swaminathan Vasudevan proposed openstack/neutron: DVR: Add forwarding routes based on address_scopes https://review.openstack.org/355062 | 21:49 |
| *** catintheroof has quit IRC | 21:49 | |
| *** ijw_ has quit IRC | 21:51 | |
| *** absubram has joined #openstack-neutron | 21:55 | |
| *** gvrangan has quit IRC | 21:57 | |
| kevinbenton | ihrachys: i wonder if the issue is that a reconnect wont' occur while in that block() function | 21:59 |
| kevinbenton | ihrachys: so whenever a disconnect happens, it hangs the ongoing connection until we throw the timeout error | 22:00 |
| kevinbenton | ihrachys: what is the magic incantation to make otherwiseguy appear? :) | 22:00 |
| otherwiseguy | i'm here! | 22:01 |
| *** absubram has quit IRC | 22:01 | |
| * otherwiseguy reads the scrollback | 22:02 | |
| kevinbenton | otherwiseguy: ihar observed a late re-issuance of the same command | 22:02 |
| kevinbenton | otherwiseguy: and it sort of sounded like exactly like what you were worried could happen | 22:03 |
| kevinbenton | otherwiseguy: the timeout left a command on the queue or something | 22:03 |
| *** rmart04 has joined #openstack-neutron | 22:03 | |
| kevinbenton | otherwiseguy: http://logs.openstack.org/43/408143/9/check/gate-neutron-dsvm-functional-ubuntu-xenial/8926e44/logs/dsvm-functional-logs/neutron.tests.functional.agent.test_dhcp_agent.DHCPAgentOVSTestCase.test_agent_mtu_set_on_interface_driver.txt.gz#_2017-01-27_02_41_16_452 | 22:04 |
| kevinbenton | otherwiseguy: notice that test-brca7476e2 addbridge gets called again towards the end right before deletebridge is issued | 22:04 |
| *** rmart04 has left #openstack-neutron | 22:04 | |
| otherwiseguy | i wonder what caused the connection dropped (protocol error) | 22:05 |
| ihrachys | otherwiseguy: could it be something left by a previous test? | 22:06 |
| ihrachys | otherwiseguy: I started looking at logs of other native tests but haven't got far yet | 22:06 |
| ihrachys | I mean those executed by this test worker | 22:06 |
| *** trown is now known as trown|outtypewww | 22:08 | |
| *** Leom has quit IRC | 22:08 | |
| *** Leo_ has joined #openstack-neutron | 22:08 | |
| kevinbenton | ihrachys: that reminds me. each test worker is a separate process | 22:08 |
| kevinbenton | ihrachys: right? | 22:08 |
| *** vhosakot_ has joined #openstack-neutron | 22:08 | |
| otherwiseguy | ihrachys, I mean, I guess it's possible. It obviously *shouldn't* be...and all of the actual creation of ovsdb protocol messages is done by OVS's lib, so why there would be one beginning with a 'k' instead of valid JSON is a mystery. | 22:09 |
| ihrachys | kevinbenton: yea | 22:10 |
| kevinbenton | otherwiseguy: we can't log the full message on a failure like that? | 22:10 |
| kevinbenton | ihrachys: doesn't that mean each function process has its own ovsdb monitor? | 22:10 |
| ihrachys | probably. what are you up for? | 22:11 |
| *** jose-phillips has joined #openstack-neutron | 22:11 | |
| kevinbenton | which one is the one we have to tell ovs to connect to | 22:12 |
| kevinbenton | and listen locally | 22:12 |
| kevinbenton | is that the native OFCTL replacement? | 22:12 |
| *** vhosakot has quit IRC | 22:12 | |
| ihrachys | it should be native | 22:12 |
| kevinbenton | how do we pick port numbers for that? | 22:12 |
| otherwiseguy | kevinbenton, the native.vlog messages are generated by the OVS lib itself. | 22:13 |
| ihrachys | define that | 22:13 |
| *** gvrangan has joined #openstack-neutron | 22:13 | |
| ihrachys | otherwiseguy: can we make the library log more? | 22:13 |
| *** vishwanathj has joined #openstack-neutron | 22:13 | |
| otherwiseguy | ihrachys, more than "2017-01-27 02:41:16.452 27330 WARNING neutron.agent.ovsdb.native.vlog [-] tcp:127.0.0.1:6640: error parsing stream: line 0, column 1, byte 1: invalid keyword 'k'" | 22:13 |
| otherwiseguy | ? | 22:13 |
| *** ihrachys_ has joined #openstack-neutron | 22:14 | |
| *** vishwanathj has quit IRC | 22:14 | |
| otherwiseguy | I can look at the code and see if it has access to anything else. | 22:14 |
| ihrachys_ | ok folks I gotta run. shoot anything relevant my way, I will check later in the evening. need to get out of the bay before the Traffic hits the road | 22:15 |
| *** vishwanathj has joined #openstack-neutron | 22:15 | |
| *** ihrachys_ has quit IRC | 22:15 | |
| *** abhiraut has quit IRC | 22:15 | |
| *** thorst_ has quit IRC | 22:16 | |
| *** abhiraut has joined #openstack-neutron | 22:16 | |
| *** vishwanathj has quit IRC | 22:17 | |
| *** ihrachys has quit IRC | 22:17 | |
| *** vhosakot_ has quit IRC | 22:17 | |
| kevinbenton | ok | 22:18 |
| *** limao has joined #openstack-neutron | 22:23 | |
| otherwiseguy | kevinbenton, It looks like we could probably add the logging of the current block of data being processed. There's a chance it wouldn't give anything new since the parse failure is showing at byte 1. | 22:24 |
| kevinbenton | otherwiseguy: yeah, i was thinking maybe if we could see the whole like it would be more obvious what leaked onto that | 22:24 |
| kevinbenton | whole line* | 22:25 |
| kevinbenton | whole blob | 22:25 |
| kevinbenton | whatever you want to call it :) | 22:25 |
| *** gouthamr has quit IRC | 22:27 | |
| *** armax has quit IRC | 22:27 | |
| *** kriskend has quit IRC | 22:28 | |
| *** baoli has quit IRC | 22:28 | |
| otherwiseguy | Might be able to monkeypatch the ovs lib's jsonrpc.py's recv() function and have it dump self.input if it returns an error. Not sure if they'd want a patch adding the output to a debug log or not. | 22:28 |
| *** amarao has quit IRC | 22:29 | |
| otherwiseguy | But since this doesn't reproduce easily, just having a testing patch that monkeypatches it might not be that helpful. | 22:29 |
| otherwiseguy | kevinbenton, ^ | 22:30 |
| *** slaweq_ has quit IRC | 22:30 | |
| *** PramodJ has joined #openstack-neutron | 22:31 | |
| *** Leom has joined #openstack-neutron | 22:32 | |
| *** Leo_ has quit IRC | 22:33 | |
| *** pramodrj07 has quit IRC | 22:34 | |
| otherwiseguy | kevinbenton, weird, "error parsing stream" happens in a very concentrated group in the last 7 days: http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22error%20parsing%20stream%5C%22 | 22:35 |
| *** limao has quit IRC | 22:40 | |
| *** limao has joined #openstack-neutron | 22:40 | |
| *** abhiraut has quit IRC | 22:40 | |
| *** abhiraut1 has joined #openstack-neutron | 22:40 | |
| otherwiseguy | kevinbenton, wrt the AddBridgeCommand being called twice, the first one looks like it was told to TRY_AGAIN righ before the connection dropped. The situation that I was worried about is Req sent/processed off the queue, immediately it waits for a response. The response is delayed and times out. Then the response arrives in the queue. Then another command is called, and the queued result gets sent to it. | 22:41 |
| *** jaosorior has quit IRC | 22:42 | |
| otherwiseguy | Apparently things are not working well when there is a disconnection. The OVS lib has a state machine to handle reconnects transparently in the background, but who knows... | 22:43 |
| *** hishh has quit IRC | 22:46 | |
| *** ijw has joined #openstack-neutron | 22:47 | |
| *** ijw has quit IRC | 22:47 | |
| *** ijw has joined #openstack-neutron | 22:47 | |
| *** abhiraut1 has quit IRC | 22:47 | |
| *** nplanel has quit IRC | 22:47 | |
| *** mriedem_beach is now known as mriedem | 22:48 | |
| *** kbaegis has quit IRC | 22:48 | |
| *** kbaegis has joined #openstack-neutron | 22:49 | |
| *** kbaegis has quit IRC | 22:49 | |
| *** gouthamr has joined #openstack-neutron | 22:51 | |
| *** princenana has quit IRC | 22:51 | |
| *** Swami_ has joined #openstack-neutron | 22:53 | |
| *** kbringard has quit IRC | 22:53 | |
| *** Swami has quit IRC | 22:57 | |
| *** jamesdenton has quit IRC | 22:58 | |
| *** Swami_ has quit IRC | 23:00 | |
| *** jperry has quit IRC | 23:01 | |
| *** tbachman has quit IRC | 23:01 | |
| *** limao has quit IRC | 23:01 | |
| *** limao has joined #openstack-neutron | 23:02 | |
| *** slaweq_ has joined #openstack-neutron | 23:04 | |
| openstackgerrit | Merged openstack/neutron: Handle attempt to enable br_netfilter in namespace https://review.openstack.org/423777 | 23:05 |
| *** annegentle has quit IRC | 23:06 | |
| *** greghaynes has quit IRC | 23:10 | |
| *** Leom has quit IRC | 23:10 | |
| *** Leo_ has joined #openstack-neutron | 23:11 | |
| *** abhiraut has joined #openstack-neutron | 23:11 | |
| *** nplanel has joined #openstack-neutron | 23:14 | |
| *** abhiraut has quit IRC | 23:16 | |
| *** marst has quit IRC | 23:20 | |
| *** tbachman has joined #openstack-neutron | 23:21 | |
| *** limao has quit IRC | 23:22 | |
| *** limao has joined #openstack-neutron | 23:23 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/neutron: Updated from global requirements https://review.openstack.org/423645 | 23:23 |
| *** trevormc has quit IRC | 23:24 | |
| openstackgerrit | Slawek Kaplonski proposed openstack/neutron: Clear conntrack entries without zones if CT zones are not used https://review.openstack.org/426429 | 23:29 |
| *** eezhova_ has quit IRC | 23:30 | |
| slaweq | haleyb kevinbenton: ^^ I just make some patch to not use CT zones in Iptables driver | 23:31 |
| slaweq | if You will have a while it would be nice if You can have a look on it :) | 23:31 |
| *** greghaynes has joined #openstack-neutron | 23:37 | |
| *** abhiraut has joined #openstack-neutron | 23:42 | |
| *** nplanel has quit IRC | 23:42 | |
| *** Swami has joined #openstack-neutron | 23:43 | |
| *** limao has quit IRC | 23:44 | |
| *** jamesden_ has joined #openstack-neutron | 23:44 | |
| *** Swami has quit IRC | 23:44 | |
| *** limao has joined #openstack-neutron | 23:44 | |
| *** pramodrj07 has joined #openstack-neutron | 23:44 | |
| *** Swami has joined #openstack-neutron | 23:44 | |
| *** PramodJ has quit IRC | 23:48 | |
| *** Swami_ has joined #openstack-neutron | 23:48 | |
| openstackgerrit | Aradhana Singh proposed openstack/neutron: Refactoring config options for ml2 config opts https://review.openstack.org/340228 | 23:49 |
| *** Swami has quit IRC | 23:52 | |
| *** john-davidge has joined #openstack-neutron | 23:52 | |
| *** janzian has quit IRC | 23:52 | |
| *** greghaynes has quit IRC | 23:52 | |
| *** tbachman has quit IRC | 23:55 | |
| *** john-davidge has quit IRC | 23:56 | |
| *** wolverineav has quit IRC | 23:57 | |
| *** wolverineav has joined #openstack-neutron | 23:58 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!