| *** br7k has joined #openstack-neutron | 00:02 | |
| *** zhanglong has joined #openstack-neutron | 00:05 | |
| *** br7k has quit IRC | 00:06 | |
| sean-k-mooney | o/ just signing of for the day but just tought i would ask why neutron enable N322 in flake8 http://logs.openstack.org/58/640258/4/check/openstack-tox-pep8/37c4c53/job-output.txt.gz#_2019-03-04_21_47_31_735680 | 00:09 |
|---|---|---|
| *** itlinux has joined #openstack-neutron | 00:09 | |
| sean-k-mooney | we do not use that in nova and i also dont think i would enable it in os-vif as i dont think useing assert_called_once_with is generally good pratice if the arguments the method are invoked with are not relevent | 00:10 |
| *** itlinux_ has joined #openstack-neutron | 00:13 | |
| sean-k-mooney | assert_called_once is a valid method https://docs.python.org/3/library/unittest.mock.html#unittest.mock.Mock.assert_called_once | 00:14 |
| sean-k-mooney | so https://github.com/openstack/neutron/blob/master/neutron/hacking/checks.py#L55-L76 apears to be incorrect | 00:14 |
| *** itlinux has quit IRC | 00:15 | |
| *** wolverineav has joined #openstack-neutron | 00:16 | |
| sean-k-mooney | ah its because they only exist for py3.6 on which is what im usingbut i guess they are noops in python 2.7 and 3.5 so asser_called_once_with is the python 2/3 compatiable method https://github.com/testing-cabal/mock/blob/release-0.8-maint/docs/mock.txt#L104 | 00:26 |
| *** openstackgerrit has joined #openstack-neutron | 00:26 | |
| openstackgerrit | Kailun Qin proposed openstack/neutron master: [doc] Add network segment ranges into admin guide https://review.openstack.org/639110 | 00:26 |
| *** abaindur has quit IRC | 00:28 | |
| *** cshen has joined #openstack-neutron | 00:29 | |
| *** abaindur has joined #openstack-neutron | 00:31 | |
| *** abaindur has quit IRC | 00:31 | |
| *** abaindur has joined #openstack-neutron | 00:32 | |
| *** cshen has quit IRC | 00:33 | |
| *** sapd1 has quit IRC | 00:42 | |
| *** wolverineav has quit IRC | 00:47 | |
| *** cheng1 has joined #openstack-neutron | 00:48 | |
| *** markvoelker has joined #openstack-neutron | 00:48 | |
| *** abaindur has quit IRC | 00:48 | |
| *** abaindur has joined #openstack-neutron | 00:49 | |
| *** cheng1 has quit IRC | 00:54 | |
| eandersson | Can you scale up the dhcp agent? | 00:55 |
| *** wolverineav has joined #openstack-neutron | 00:58 | |
| *** macza has joined #openstack-neutron | 00:59 | |
| *** ileixe has joined #openstack-neutron | 01:00 | |
| openstackgerrit | Doug Wiegley proposed openstack/neutron master: Fix a couple of bw burst calc unit tests for macs https://review.openstack.org/640908 | 01:00 |
| eandersson | > Could not release DHCP leases for these IP | 01:00 |
| eandersson | This seems to be slowing things down | 01:01 |
| *** macza has quit IRC | 01:03 | |
| eandersson | In fact the dhcp-agent looks to be deadlocked | 01:11 |
| *** br7k has joined #openstack-neutron | 01:14 | |
| *** br7k has quit IRC | 01:19 | |
| *** cheng1 has joined #openstack-neutron | 01:21 | |
| zhanglong | chinamobile | 01:21 |
| zhanglong | o sry | 01:21 |
| *** markvoelker has quit IRC | 01:22 | |
| *** macza has joined #openstack-neutron | 01:35 | |
| eandersson | sean-k-mooney, do you have any advise on tuning something like neutron_num_sync_threads for the dhcp agent? | 01:35 |
| *** macza has quit IRC | 01:39 | |
| *** ileixe has quit IRC | 01:40 | |
| *** njohnston has quit IRC | 01:41 | |
| *** abaindur has quit IRC | 01:46 | |
| *** dave-mccowan has joined #openstack-neutron | 01:46 | |
| *** gouthamr has quit IRC | 01:48 | |
| *** dmellado has quit IRC | 01:50 | |
| *** ileixe has joined #openstack-neutron | 01:51 | |
| *** ileixe has quit IRC | 01:52 | |
| *** sapd1 has joined #openstack-neutron | 01:52 | |
| *** ileixe has joined #openstack-neutron | 01:53 | |
| openstackgerrit | Kailun Qin proposed openstack/neutron master: Support Network Segment Range CRUD as extensions https://review.openstack.org/624708 | 01:54 |
| openstackgerrit | Kailun Qin proposed openstack/neutron master: Use network segment ranges for segment allocation https://review.openstack.org/624709 | 01:54 |
| openstackgerrit | Kailun Qin proposed openstack/neutron master: Devstack plugin for network-segment-range api https://review.openstack.org/638386 | 01:54 |
| *** wolverineav has quit IRC | 01:57 | |
| *** abaindur has joined #openstack-neutron | 02:00 | |
| openstackgerrit | Doug Wiegley proposed openstack/neutron master: DNM: test ci https://review.openstack.org/640920 | 02:01 |
| openstackgerrit | Doug Wiegley proposed openstack/neutron master: Use dynamic lazy mode for fetching security group rules https://review.openstack.org/637407 | 02:08 |
| *** macza has joined #openstack-neutron | 02:11 | |
| *** hongbin has joined #openstack-neutron | 02:13 | |
| *** macza has quit IRC | 02:15 | |
| *** markvoelker has joined #openstack-neutron | 02:19 | |
| *** br7k has joined #openstack-neutron | 02:27 | |
| *** abaindur has quit IRC | 02:27 | |
| *** ircuser-1 has joined #openstack-neutron | 02:28 | |
| *** abaindur has joined #openstack-neutron | 02:28 | |
| *** zhanglong has quit IRC | 02:29 | |
| *** cshen has joined #openstack-neutron | 02:29 | |
| *** br7k has quit IRC | 02:31 | |
| *** abaindur has quit IRC | 02:32 | |
| *** wolverineav has joined #openstack-neutron | 02:33 | |
| *** cshen has quit IRC | 02:34 | |
| *** s1061123 has quit IRC | 02:34 | |
| *** zhanglong has joined #openstack-neutron | 02:36 | |
| *** s1061123 has joined #openstack-neutron | 02:37 | |
| *** sapd1 has quit IRC | 02:37 | |
| *** zhanglong has quit IRC | 02:45 | |
| openstackgerrit | Zhao Qikun proposed openstack/neutron master: This is a test https://review.openstack.org/640927 | 02:45 |
| *** gouthamr has joined #openstack-neutron | 02:50 | |
| *** zhanglong has joined #openstack-neutron | 02:50 | |
| *** markvoelker has quit IRC | 02:51 | |
| *** dmellado has joined #openstack-neutron | 02:55 | |
| dougwig | eandersson: how many do you have now? | 02:56 |
| eandersson | We had it set to default 4 | 02:58 |
| eandersson | Raised it to 8 | 02:58 |
| eandersson | We got stuck in some sort of deadlock, but currently believe that it was caused by too high load on the neutron-rpc/api server. | 02:58 |
| *** wolverineav has quit IRC | 03:00 | |
| eandersson | Still scary as the dhcp-agent had over 1k messages queued in RabbitMQ | 03:01 |
| *** wolverineav has joined #openstack-neutron | 03:01 | |
| *** wolverineav has quit IRC | 03:06 | |
| *** itlinux_ has quit IRC | 03:06 | |
| *** itlinux has joined #openstack-neutron | 03:12 | |
| dougwig | if you can identify the expensive operation, you can deploy more worker nodes. for us, it was api calls, so an additional couple neutron controllers that did nothing but rpc and dhcp did the trick. they're stateless, so they can do nothing but run those background threads. | 03:13 |
| *** yamamoto has joined #openstack-neutron | 03:17 | |
| dougwig | i'm not sure when we set our api workers to the number of cores and rpc workers to 1, but both are terrible values. :) | 03:18 |
| *** dsneddon has quit IRC | 03:19 | |
| *** itlinux has quit IRC | 03:21 | |
| *** itlinux has joined #openstack-neutron | 03:25 | |
| *** wolverineav has joined #openstack-neutron | 03:31 | |
| *** tbachman has quit IRC | 03:33 | |
| *** rkukura has quit IRC | 03:34 | |
| *** dsneddon has joined #openstack-neutron | 03:35 | |
| *** yamamoto has quit IRC | 03:35 | |
| *** yamamoto has joined #openstack-neutron | 03:35 | |
| *** cheng1 has quit IRC | 03:37 | |
| *** dsneddon has quit IRC | 03:40 | |
| *** dave-mccowan has quit IRC | 03:42 | |
| openstackgerrit | Merged openstack/neutron-fwaas master: add python 3.7 unit test job https://review.openstack.org/638259 | 03:46 |
| *** zhanglong has quit IRC | 03:47 | |
| *** markvoelker has joined #openstack-neutron | 03:48 | |
| openstackgerrit | Yushiro FURUKAWA proposed openstack/neutron-fwaas master: Migrate to stestr as unit tests runner https://review.openstack.org/505526 | 04:02 |
| openstackgerrit | Yushiro FURUKAWA proposed openstack/neutron-fwaas master: Update mailinglist from dev to discuss https://review.openstack.org/632416 | 04:03 |
| *** wolverineav has quit IRC | 04:07 | |
| *** ramishra has joined #openstack-neutron | 04:09 | |
| *** dsneddon has joined #openstack-neutron | 04:15 | |
| *** dsneddon has quit IRC | 04:20 | |
| *** markvoelker has quit IRC | 04:22 | |
| *** cshen has joined #openstack-neutron | 04:30 | |
| *** cshen has quit IRC | 04:34 | |
| *** toshiiw has joined #openstack-neutron | 04:37 | |
| *** sridharg has joined #openstack-neutron | 04:41 | |
| openstackgerrit | Merged openstack/neutron-fwaas master: Remove midonet job https://review.openstack.org/634855 | 04:47 |
| *** armax has quit IRC | 04:48 | |
| *** br7k has joined #openstack-neutron | 04:49 | |
| *** dsneddon has joined #openstack-neutron | 04:50 | |
| *** br7k has quit IRC | 04:53 | |
| *** dsneddon has quit IRC | 04:57 | |
| *** wolverineav has joined #openstack-neutron | 05:00 | |
| *** wolverineav has quit IRC | 05:06 | |
| *** janki has joined #openstack-neutron | 05:08 | |
| *** hongbin has quit IRC | 05:09 | |
| *** markvoelker has joined #openstack-neutron | 05:19 | |
| *** ratailor has joined #openstack-neutron | 05:23 | |
| *** dsneddon has joined #openstack-neutron | 05:30 | |
| *** cheng1 has joined #openstack-neutron | 05:31 | |
| *** rkukura has joined #openstack-neutron | 05:36 | |
| eandersson | dougwig, yea lowered the number to match our physical cores | 05:37 |
| eandersson | and moved some stuff around | 05:37 |
| eandersson | looking a lot better now | 05:37 |
| eandersson | but for sure a lot more cpu usage now than before | 05:37 |
| dougwig | yeah, the short-term answer is going to be to throw more boxes at it, unless you start seeing db contention, and then more parallel will start to hurt you. | 05:38 |
| eandersson | btw changing the neutron-api vip from rr to least connections made the cpu load a lot more even | 05:38 |
| dougwig | i believe that. settles out the long-running stuff. | 05:39 |
| *** dsneddon has quit IRC | 05:40 | |
| eandersson | hopefully the dhcp "deadlock" was related to extreme load | 05:40 |
| eandersson | will have to add some extra monitoring to make sure we catch that sooner | 05:40 |
| *** cheng1 has quit IRC | 05:48 | |
| *** markvoelker has quit IRC | 05:51 | |
| *** pcaruana has joined #openstack-neutron | 05:52 | |
| *** lajoskatona has joined #openstack-neutron | 05:53 | |
| *** cheng1 has joined #openstack-neutron | 05:58 | |
| *** pcaruana has quit IRC | 06:07 | |
| *** dsneddon has joined #openstack-neutron | 06:10 | |
| *** sm806 has quit IRC | 06:17 | |
| *** sm806 has joined #openstack-neutron | 06:18 | |
| *** dims has quit IRC | 06:24 | |
| *** dims has joined #openstack-neutron | 06:26 | |
| *** sm806 has quit IRC | 06:31 | |
| *** sm806 has joined #openstack-neutron | 06:31 | |
| *** gcheresh_ has joined #openstack-neutron | 06:33 | |
| *** itlinux has quit IRC | 06:34 | |
| *** dims has quit IRC | 06:36 | |
| *** dims has joined #openstack-neutron | 06:37 | |
| *** cheng1 has quit IRC | 06:40 | |
| *** jlibosva has joined #openstack-neutron | 06:46 | |
| *** cshen has joined #openstack-neutron | 06:46 | |
| *** ksambor has joined #openstack-neutron | 06:48 | |
| *** markvoelker has joined #openstack-neutron | 06:48 | |
| *** wolverineav has joined #openstack-neutron | 06:48 | |
| *** Luzi has joined #openstack-neutron | 06:51 | |
| *** wolverineav has quit IRC | 06:53 | |
| openstackgerrit | LIU Yulong proposed openstack/neutron master: More accurate agent restart state transfer https://review.openstack.org/640797 | 06:57 |
| *** sridharg has quit IRC | 06:57 | |
| *** mosulica has joined #openstack-neutron | 06:59 | |
| *** dayou has quit IRC | 07:01 | |
| *** zhanglong has joined #openstack-neutron | 07:01 | |
| *** dayou has joined #openstack-neutron | 07:01 | |
| *** sridharg has joined #openstack-neutron | 07:02 | |
| *** cheng1 has joined #openstack-neutron | 07:06 | |
| *** ccamposr has joined #openstack-neutron | 07:10 | |
| f0o | Morning | 07:10 |
| *** ccamposr__ has joined #openstack-neutron | 07:10 | |
| *** slaweq has joined #openstack-neutron | 07:11 | |
| *** ccamposr has quit IRC | 07:14 | |
| *** wolverineav has joined #openstack-neutron | 07:14 | |
| *** wolverineav has quit IRC | 07:14 | |
| *** hamdyk has joined #openstack-neutron | 07:18 | |
| *** br7k has joined #openstack-neutron | 07:21 | |
| *** markvoelker has quit IRC | 07:22 | |
| *** aojea has joined #openstack-neutron | 07:23 | |
| *** br7k has quit IRC | 07:25 | |
| *** yamamoto has quit IRC | 07:34 | |
| *** ralonsoh has joined #openstack-neutron | 07:34 | |
| *** yamamoto has joined #openstack-neutron | 07:36 | |
| *** dayou has quit IRC | 07:37 | |
| *** snapiri has joined #openstack-neutron | 07:57 | |
| *** dayou has joined #openstack-neutron | 07:58 | |
| *** abaindur has joined #openstack-neutron | 07:59 | |
| *** abaindur has quit IRC | 08:04 | |
| *** abaindur has joined #openstack-neutron | 08:04 | |
| *** tesseract has joined #openstack-neutron | 08:09 | |
| rubasov | mlavalle, slaweq: good morning, I just read the neutron team meeting log from last evening | 08:14 |
| slaweq | hi rubasov | 08:14 |
| rubasov | mlavalle, slaweq: thanks for taking up the min-bw topic on it | 08:14 |
| slaweq | I wanted to ping You :) | 08:14 |
| rubasov | mlavalle, slaweq: I think we only have one change that is still due until ff | 08:15 |
| rubasov | mlavalle, slaweq: this: https://review.openstack.org/574783 | 08:15 |
| *** luksky has joined #openstack-neutron | 08:16 | |
| *** awalende has joined #openstack-neutron | 08:16 | |
| rubasov | mlavalle, slaweq: thanks in advance if you can look at it | 08:16 |
| *** tssurya has joined #openstack-neutron | 08:16 | |
| slaweq | rubasov: sure | 08:16 |
| rubasov | mlavalle, slaweq: also let me cross-check with you please | 08:16 |
| *** sapd1 has joined #openstack-neutron | 08:16 | |
| *** rpittau|afk is now known as rpittau | 08:16 | |
| slaweq | rubasov: so all other patches from https://review.openstack.org/#/q/topic:minimum-bandwidth-allocation-placement-api+status:open aren't really needed to have BP done? | 08:16 |
| rubasov | mlavalle, slaweq: we have a feq feature related bugfixes: those have two more weeks, right? | 08:17 |
| rubasov | mlavalle, slaweq: like this: https://review.openstack.org/638714 | 08:17 |
| rubasov | mlavalle, slaweq: and what is the deadline for docs? rc1? | 08:18 |
| *** markvoelker has joined #openstack-neutron | 08:18 | |
| slaweq | rubasov: I don't think there is any strict deadline for docs | 08:18 |
| slaweq | maybe amotoki will know ^^ :) | 08:18 |
| *** pcaruana has joined #openstack-neutron | 08:18 | |
| rubasov | slaweq: because the networking guide patch is still wip | 08:19 |
| rubasov | slaweq: plus lajoskatona has a few more tests on that gerrit topic, but IIUC those also don't have strict deadlines | 08:19 |
| slaweq | ok, so the most important for now is this one Your patch | 08:20 |
| slaweq | and other things can land a bit later | 08:20 |
| slaweq | thx for confirmation | 08:20 |
| rubasov | slaweq: yep, that's correct | 08:20 |
| slaweq | I will review this Your patch today | 08:20 |
| rubasov | slaweq: thanks for taking the time | 08:20 |
| slaweq | rubasov: sure, np :) | 08:21 |
| *** tkajinam has quit IRC | 08:22 | |
| *** pcaruana has quit IRC | 08:25 | |
| *** rubasov has quit IRC | 08:33 | |
| *** br7k has joined #openstack-neutron | 08:34 | |
| *** pcaruana has joined #openstack-neutron | 08:37 | |
| *** br7k has quit IRC | 08:38 | |
| *** pcaruana has quit IRC | 08:44 | |
| *** markvoelker has quit IRC | 08:51 | |
| *** jpena|off is now known as jpena | 08:52 | |
| *** pcaruana has joined #openstack-neutron | 09:01 | |
| *** macza has joined #openstack-neutron | 09:01 | |
| amotoki | rubasov: slaweq: there is no deadline for the documentation. hopefully it is nice if the doc patch lands till RC-1 (i.e., stein branch cut), but we can backport it even after that. | 09:01 |
| slaweq | amotoki: that's what I though, thx for confirmation :) | 09:03 |
| *** macza has quit IRC | 09:05 | |
| *** moshele has joined #openstack-neutron | 09:09 | |
| *** toshiiw has quit IRC | 09:10 | |
| *** abaindur has quit IRC | 09:13 | |
| *** panda|ruck|off is now known as panda|ruck|flu | 09:13 | |
| *** lemko has joined #openstack-neutron | 09:23 | |
| *** cheng1 has quit IRC | 09:34 | |
| *** br7k has joined #openstack-neutron | 09:45 | |
| *** markvoelker has joined #openstack-neutron | 09:48 | |
| *** br7k has quit IRC | 09:49 | |
| *** cheng1 has joined #openstack-neutron | 09:49 | |
| *** moshele has quit IRC | 09:52 | |
| *** zhanglong has quit IRC | 10:03 | |
| *** rubasov has joined #openstack-neutron | 10:05 | |
| *** yamamoto has quit IRC | 10:07 | |
| *** cheng1 has quit IRC | 10:08 | |
| openstackgerrit | Merged openstack/networking-ovn master: Switch released jobs to OVS 2.11 https://review.openstack.org/640800 | 10:14 |
| *** salmankhan has joined #openstack-neutron | 10:20 | |
| *** davidsha has joined #openstack-neutron | 10:21 | |
| *** markvoelker has quit IRC | 10:22 | |
| *** luksky has quit IRC | 10:22 | |
| *** yamamoto has joined #openstack-neutron | 10:42 | |
| *** macza has joined #openstack-neutron | 10:51 | |
| *** macza has quit IRC | 10:55 | |
| *** br7k has joined #openstack-neutron | 10:56 | |
| *** cheng1 has joined #openstack-neutron | 10:56 | |
| *** br7k has quit IRC | 11:00 | |
| *** luksky has joined #openstack-neutron | 11:02 | |
| *** cheng1 has quit IRC | 11:05 | |
| openstackgerrit | YAMAMOTO Takashi proposed openstack/networking-midonet master: TEST FOR CI DO NOT REVIEW https://review.openstack.org/199387 | 11:07 |
| openstackgerrit | YAMAMOTO Takashi proposed openstack/networking-midonet master: DNM Testing legacy jobs on bionic https://review.openstack.org/639990 | 11:08 |
| openstackgerrit | LIU Yulong proposed openstack/neutron master: More accurate agent restart state transfer https://review.openstack.org/640797 | 11:17 |
| *** markvoelker has joined #openstack-neutron | 11:19 | |
| *** janki has quit IRC | 11:22 | |
| slaweq | dalvarez: ajo: hi, can You maybe take a look at https://bugs.launchpad.net/networking-ovn/+bug/1818626 when You will have some time? | 11:27 |
| openstack | Launchpad bug 1818626 in networking-ovn "Functional python 3.5 job is failing on Ubuntu Bionic" [Undecided,New] | 11:27 |
| slaweq | it's not urgent yet but please keep in mind that there is such issue :) | 11:27 |
| slaweq | thx in advance | 11:27 |
| dalvarez | lucasagomes: ^ | 11:29 |
| *** sapd1 has quit IRC | 11:29 | |
| * lucasagomes looks | 11:29 | |
| lucasagomes | dalvarez, slaweq oh yeah... we are currently looking for the 3.5 version of Python, perhaps that's not needed ? I mean we can run functional tests with "python3" (be it 3.5, 3.6, 3.7) | 11:32 |
| * lucasagomes check how other projects are handling it | 11:32 | |
| openstackgerrit | Slawek Kaplonski proposed openstack/neutron master: Migrate neutron-functional job to zuul v3 syntax https://review.openstack.org/619742 | 11:43 |
| slaweq | lucasagomes: yes, that would be the best option IMO | 11:45 |
| openstackgerrit | Lucas Alvares Gomes proposed openstack/networking-ovn master: Switch default functional tests to Python3 https://review.openstack.org/640988 | 11:45 |
| lucasagomes | slaweq, ^ I will rebase ur testing patch on top of it | 11:45 |
| slaweq | lucasagomes: thx a lot :) | 11:46 |
| openstackgerrit | Lucas Alvares Gomes proposed openstack/networking-ovn master: DNM Testing legacy jobs on bionic https://review.openstack.org/639673 | 11:46 |
| * lucasagomes took a note to convert the functional to native zuul | 11:46 | |
| dalvarez | lucasagomes++ :) | 11:46 |
| *** cheng1 has joined #openstack-neutron | 11:46 | |
| *** markvoelker has quit IRC | 11:51 | |
| *** cheng1 has quit IRC | 11:54 | |
| *** diliprenkila has joined #openstack-neutron | 11:57 | |
| *** br7k has joined #openstack-neutron | 12:04 | |
| diliprenkila | Hi all, I am running neutron-server on neutron-server (14.0.0.0b1) and neutron-dhcp-agent (14.0.0.0b1) and still they aren't using same rpc version. It seems strange to me. Here is the detailed log https://etherpad.openstack.org/p/5yZ756ncTC | 12:04 |
| *** br7k has quit IRC | 12:08 | |
| *** ratailor has quit IRC | 12:13 | |
| *** ociuhandu has quit IRC | 12:14 | |
| *** mmethot has quit IRC | 12:15 | |
| *** awalende has quit IRC | 12:15 | |
| *** awalende has joined #openstack-neutron | 12:16 | |
| *** dave-mccowan has joined #openstack-neutron | 12:19 | |
| openstackgerrit | Javier Peña proposed openstack/networking-sfc master: Fix unit tests on latest neutron https://review.openstack.org/640999 | 12:31 |
| *** jpena is now known as jpena|lunch | 12:31 | |
| *** boden has joined #openstack-neutron | 12:32 | |
| *** sm806 has quit IRC | 12:33 | |
| *** sm806 has joined #openstack-neutron | 12:34 | |
| *** cshen has quit IRC | 12:35 | |
| *** zhanglong has joined #openstack-neutron | 12:36 | |
| *** cshen has joined #openstack-neutron | 12:39 | |
| *** cheng1 has joined #openstack-neutron | 12:40 | |
| *** mriedem has joined #openstack-neutron | 12:41 | |
| *** yamamoto has quit IRC | 12:44 | |
| frickler | diliprenkila: check your versions of neutron-lib, iiuc the rpc version is defined there | 12:44 |
| *** yamamoto has joined #openstack-neutron | 12:46 | |
| *** tosky has joined #openstack-neutron | 12:48 | |
| *** markvoelker has joined #openstack-neutron | 12:49 | |
| *** yamamoto has quit IRC | 12:52 | |
| *** zhanglong has quit IRC | 12:57 | |
| *** yamamoto has joined #openstack-neutron | 12:58 | |
| diliprenkila | frickler: where can i find versions in neutron-lib ? | 13:00 |
| *** njohnston has joined #openstack-neutron | 13:01 | |
| *** mchlumsky has joined #openstack-neutron | 13:01 | |
| *** mmichelson has joined #openstack-neutron | 13:03 | |
| *** panda|ruck|flu is now known as panda|ruck|lunch | 13:03 | |
| frickler | diliprenkila: depends on how you installed things, either something like the python-neutron-lib pkg e.g. for UCA or check the output of "pip freeze" | 13:08 |
| *** yamamoto has quit IRC | 13:14 | |
| *** janki has joined #openstack-neutron | 13:17 | |
| *** cheng1_ has joined #openstack-neutron | 13:19 | |
| *** markvoelker has quit IRC | 13:21 | |
| *** cheng1 has quit IRC | 13:22 | |
| boden | frickler what are you guys looking for? rpc versioning? | 13:24 |
| *** br7k has joined #openstack-neutron | 13:24 | |
| *** zhanglong has joined #openstack-neutron | 13:28 | |
| *** br7k has quit IRC | 13:28 | |
| diliprenkila | boden: we are loooking for rpc versioning | 13:28 |
| boden | diliprenkila I'm no expert, but thought that was defined in the rpc Target creation as per https://github.com/openstack/oslo.messaging/blob/master/oslo_messaging/target.py#L52. so for neutron that would be something like http://codesearch.openstack.org/?q=%5C.Target%5C(&i=nope&files=.py&repos=neutron | 13:30 |
| diliprenkila | boden: i have same version of python-neutron-lib on neutron-server and neutron-agents Installed: 1.20.0-0ubuntu1~cloud0 | 13:30 |
| boden | but maybe I'm wrong | 13:30 |
| *** sm806 has quit IRC | 13:31 | |
| *** sm806 has joined #openstack-neutron | 13:32 | |
| *** oanson has joined #openstack-neutron | 13:36 | |
| *** ivve has joined #openstack-neutron | 13:38 | |
| *** jpena|lunch is now known as jpena | 13:40 | |
| *** diliprenkila has quit IRC | 13:41 | |
| openstackgerrit | Slawek Kaplonski proposed openstack/neutron master: Fix handling no qos_queues while removing min bw limit rule https://review.openstack.org/640874 | 13:42 |
| *** sapd1 has joined #openstack-neutron | 13:43 | |
| *** sapd1 has quit IRC | 13:47 | |
| f0o | boden: hi, had any chance to look at that bug report from me? | 13:50 |
| f0o | boden: or know any pointers in debugging this> | 13:50 |
| *** panda|ruck|lunch is now known as panda|ruck|flu | 13:50 | |
| boden | f0o I asked the plugin team to look... I think they will get to it in the next few days | 13:50 |
| *** mchlumsky has quit IRC | 13:51 | |
| f0o | awesome thanks :) | 13:52 |
| *** mchlumsky has joined #openstack-neutron | 13:53 | |
| *** yamamoto has joined #openstack-neutron | 13:53 | |
| *** ratailor has joined #openstack-neutron | 13:56 | |
| *** cheng1_ has quit IRC | 13:58 | |
| *** yamamoto_ has joined #openstack-neutron | 14:01 | |
| *** yamamoto has quit IRC | 14:03 | |
| *** mmethot has joined #openstack-neutron | 14:05 | |
| haleyb | slaweq: can you take a look at https://review.openstack.org/#/c/623066/ ? | 14:08 |
| *** janki has quit IRC | 14:12 | |
| *** janki has joined #openstack-neutron | 14:12 | |
| *** dklyle has joined #openstack-neutron | 14:13 | |
| *** sapd1 has joined #openstack-neutron | 14:18 | |
| *** markvoelker has joined #openstack-neutron | 14:18 | |
| *** zhanglong has quit IRC | 14:19 | |
| *** lemko has quit IRC | 14:22 | |
| slaweq | haleyb: looking | 14:23 |
| slaweq | haleyb: +2 | 14:31 |
| haleyb | slaweq: thanks! | 14:31 |
| *** goldyfruit has joined #openstack-neutron | 14:39 | |
| goldyfruit | Hi guys | 14:42 |
| goldyfruit | I would like to know if there is a side effect of having a big value for agent_boot_time parameter ? | 14:44 |
| goldyfruit | We are facing flows issue with DVR and increasing agent_boot_time solved our issue | 14:44 |
| goldyfruit | More information about it: https://bugs.launchpad.net/neutron/+bug/1794991 | 14:44 |
| openstack | Launchpad bug 1794991 in neutron "Inconsistent flows with DVR l2pop VxLAN on br-tun" [Undecided,New] - Assigned to Bruca Lock (lockshi) | 14:44 |
| *** hongbin has joined #openstack-neutron | 14:47 | |
| *** njohnston has quit IRC | 14:50 | |
| *** njohnston has joined #openstack-neutron | 14:50 | |
| *** markvoelker has quit IRC | 14:52 | |
| *** pcaruana has quit IRC | 14:57 | |
| *** munimeha1 has joined #openstack-neutron | 14:59 | |
| *** awalende has quit IRC | 14:59 | |
| *** ratailor has quit IRC | 15:00 | |
| *** awalende has joined #openstack-neutron | 15:00 | |
| *** lajoskatona has quit IRC | 15:00 | |
| *** cheng1 has joined #openstack-neutron | 15:01 | |
| *** Luzi has quit IRC | 15:02 | |
| *** tbachman has joined #openstack-neutron | 15:02 | |
| *** awalende has quit IRC | 15:04 | |
| openstackgerrit | Slawek Kaplonski proposed openstack/neutron-fwaas master: Add new upgrade check https://review.openstack.org/637203 | 15:04 |
| *** tbachman_ has joined #openstack-neutron | 15:05 | |
| *** tbachman has quit IRC | 15:07 | |
| *** tbachman_ is now known as tbachman | 15:07 | |
| *** hamdyk has quit IRC | 15:16 | |
| openstackgerrit | Bence Romsics proposed openstack/neutron master: Fail placement sync if _get_rp_by_name() fails https://review.openstack.org/641034 | 15:22 |
| *** sapd1 has quit IRC | 15:25 | |
| *** mlavalle has joined #openstack-neutron | 15:25 | |
| openstackgerrit | Nate Johnston proposed openstack/networking-bagpipe master: Run pep8 tests under python 3 https://review.openstack.org/641041 | 15:27 |
| *** tbachman has quit IRC | 15:28 | |
| *** openstackgerrit has quit IRC | 15:28 | |
| *** openstackgerrit has joined #openstack-neutron | 15:32 | |
| openstackgerrit | Roman Safronov proposed openstack/neutron-tempest-plugin master: Add logging of the test steps in NetworkWritableMtuTest https://review.openstack.org/640821 | 15:32 |
| *** gcheresh_ has quit IRC | 15:36 | |
| bcafarel | ralonsoh: slaweq: https://review.openstack.org/#/c/640999/ I guess now we can assume OVSAgentBridge to always have 3 parameters now? (in master) | 15:38 |
| bcafarel | not sure if other stadium projects got impacted by it | 15:38 |
| slaweq | bcafarel: yes | 15:39 |
| *** sm806 has quit IRC | 15:39 | |
| *** sm806 has joined #openstack-neutron | 15:40 | |
| haleyb | goldyfruit: so when you increased agent_boot_time your flows/fdb were always correctly populated? | 15:40 |
| ralonsoh | bcafarel, slaweq: you are right, we should not change this API | 15:40 |
| slaweq | ralonsoh: or at least check stadium projects which are impacted :/ | 15:41 |
| ralonsoh | bcafarel, slaweq: I think I should propose a change to make the third parameter a kwarg | 15:41 |
| ralonsoh | slaweq, I'll check it now | 15:41 |
| slaweq | ralonsoh: http://codesearch.openstack.org/?q=OVSAgentExtensionAPI&i=nope&files=&repos= | 15:41 |
| openstackgerrit | Merged openstack/neutron master: Delete port binding level for deleted bindings https://review.openstack.org/634276 | 15:41 |
| *** cshen has quit IRC | 15:41 | |
| slaweq | it looks that also bagpipe and fwaas may be broken because of that | 15:41 |
| *** pcaruana has joined #openstack-neutron | 15:42 | |
| ralonsoh | slaweq, fwaas is only mocking it, but bagpipe will need this new parameter | 15:42 |
| slaweq | yes, right :) | 15:43 |
| slaweq | I just looked quickly to projects in which it was found :) | 15:43 |
| bcafarel | he, was doing the same in another tab :) (and same conclusion) | 15:43 |
| bcafarel | fwaas periodic passed, bagpipe http://logs.openstack.org/periodic/git.openstack.org/openstack/networking-bagpipe/master/openstack-tox-py35-with-neutron-lib-master/e8aa898/testr_results.html.gz | 15:44 |
| *** alisanhaji has joined #openstack-neutron | 15:44 | |
| openstackgerrit | LIU Yulong proposed openstack/neutron master: More accurate agent restart state transfer https://review.openstack.org/640797 | 15:45 |
| openstackgerrit | Kailun Qin proposed openstack/neutron master: [doc] Add network segment ranges into admin guide https://review.openstack.org/639110 | 15:45 |
| ralonsoh | bcafarel, slaweq: IMO, we should make this parameter optional | 15:45 |
| ralonsoh | bcafarel, slaweq: only qos is using it by calling request_phy_brs() | 15:46 |
| slaweq | ralonsoh: yes, that's IMO good idea | 15:46 |
| * bcafarel reads the original review | 15:46 | |
| *** armax has joined #openstack-neutron | 15:46 | |
| ralonsoh | slaweq, perfect, thanks! | 15:46 |
| openstackgerrit | Merged openstack/networking-ovn master: Run TripleO scenario003 in gate pipeline https://review.openstack.org/638422 | 15:47 |
| *** markvoelker has joined #openstack-neutron | 15:48 | |
| *** ksambor has quit IRC | 15:49 | |
| bcafarel | ralonsoh: sounds great :) | 15:49 |
| openstackgerrit | Nate Johnston proposed openstack/networking-bagpipe master: Migrate bagpipe legacy tempest jobs to zuulv3 and py3 https://review.openstack.org/641057 | 15:50 |
| liuyulong | https://bugs.launchpad.net/neutron/+bug/1818614 | 15:51 |
| openstack | Launchpad bug 1818614 in neutron "Various L3HA functional tests fails often" [Critical,Confirmed] | 15:51 |
| liuyulong | https://bugs.launchpad.net/neutron/+bug/1818613 | 15:51 |
| openstack | Launchpad bug 1818613 in neutron "Functional qos tests fails often" [Critical,Confirmed] | 15:51 |
| liuyulong | slaweq, I also noticed such issue. IMO, we can mark them as duplicated. | 15:51 |
| liuyulong | http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%20%5C%22line%20690%2C%20in%20wait_until_true%5C%22 | 15:52 |
| openstackgerrit | Doug Wiegley proposed openstack/networking-sfc master: DNM: ci test https://review.openstack.org/641058 | 15:52 |
| slaweq | liuyulong: hi | 15:52 |
| liuyulong | The issue was seen since 2019-2-25. | 15:52 |
| slaweq | liuyulong: yes, those are I think duplicates of https://bugs.launchpad.net/neutron/+bug/1818614 and https://bugs.launchpad.net/neutron/+bug/1818613 | 15:52 |
| openstack | Launchpad bug 1818614 in neutron "Various L3HA functional tests fails often" [Critical,Confirmed] | 15:52 |
| *** sri_ has quit IRC | 15:52 | |
| openstack | Launchpad bug 1818613 in neutron "Functional qos tests fails often" [Critical,Confirmed] | 15:52 |
| *** rubasov has quit IRC | 15:52 | |
| slaweq | ahh, sorry | 15:53 |
| slaweq | You pasted those already :) | 15:53 |
| slaweq | if You already reported those issues before, please mark one of them as duplicate of the other and will be fine | 15:53 |
| liuyulong | slaweq, yes, problem are all from that " File "neutron/common/utils.py", line 690, in wait_until_true" | 15:53 |
| liuyulong | No, I just checked the LP bug you filed there. | 15:54 |
| dougwig | is someone actively working on them? | 15:54 |
| slaweq | liuyulong: so what duplicates You are then talking about? | 15:54 |
| slaweq | dougwig: are You asking about those 2 functional tests issues? | 15:54 |
| dougwig | yeah, and a similar looking issue with fullstack. | 15:55 |
| slaweq | dougwig: I don't think there is anyone working on them already | 15:55 |
| liuyulong | dougwig, no, I can indeed reproduce it with low problities. | 15:55 |
| slaweq | I have them on list to discuss on CI meeting in few minutes | 15:55 |
| liuyulong | slaweq, problem are all from that " File "neutron/common/utils.py", line 690, in wait_until_true" | 15:55 |
| liuyulong | slaweq, this the main reason why I think they are duplicated. | 15:56 |
| slaweq | liuyulong: this is only helper which checks some condition | 15:56 |
| slaweq | those 2 issues are different | 15:56 |
| slaweq | one is related to l3ha routers and second to ovs and min bw limit qos rules | 15:56 |
| slaweq | liuyulong: You should check what is the predicate given as argument to wait_until_true() there | 15:57 |
| dougwig | liuyulong: that's a generic function that waits from something to be done. in the case of races or deadlocks, it'd be a common wait point. that doesn't necessarily make them related. | 15:57 |
| dougwig | slaweq: ok, let me add some elastic recheck queries for them, since we're hitting them so often. | 15:57 |
| slaweq | dougwig: sure, thx | 15:58 |
| slaweq | we are hitting them a lot recently | 15:58 |
| liuyulong | the root cause may be related to the OVS mechanism, IMO, l3 also needs L2 functionalites. | 15:59 |
| slaweq | liuyulong: but in case of those L3 failures it's always waiting for router to be master | 15:59 |
| slaweq | it's functional tests, so there is no L2 agent involved in that | 16:00 |
| slaweq | I guess it's some issue with keepalived | 16:00 |
| slaweq | but I may be wrong | 16:00 |
| *** tbachman has joined #openstack-neutron | 16:00 | |
| openstackgerrit | Rodolfo Alonso Hernandez proposed openstack/neutron master: Make "phys_brs" argument in OVSAgentExtensionAPI optional https://review.openstack.org/641064 | 16:01 |
| liuyulong | Let's make bold assumptions, verify them carefully. | 16:01 |
| *** cheng1 has quit IRC | 16:01 | |
| liuyulong | I will paste some issues I met. | 16:02 |
| *** mosulica has quit IRC | 16:07 | |
| *** wwriverrat has quit IRC | 16:08 | |
| openstackgerrit | boden proposed openstack/neutron master: remove neutron.db.api references https://review.openstack.org/633283 | 16:15 |
| *** wwriverrat_ has joined #openstack-neutron | 16:17 | |
| liuyulong | http://logs.openstack.org/47/638647/7/check/neutron-functional/ce2f9fd/logs/testr_results.html.gz, this is the mainly issue I met recently. I can reproduce these failure locally with 10-20% probability. | 16:17 |
| *** markvoelker has quit IRC | 16:22 | |
| boden | mlavalle ping if you get a min... no rush at all | 16:22 |
| mlavalle | boden: after the ci meeting | 16:23 |
| boden | cool | 16:23 |
| *** liuyulong is now known as liuyulong_zzz | 16:27 | |
| dougwig | slaweq: looking through all the various bugs and failures, the common element is applying some kind of feature change to a port and waiting for it to complete (ha, qos, etc.) | 16:33 |
| dougwig | have we had any port related changes in that area in the last bit? | 16:34 |
| slaweq | dougwig: ha is applied not to port but to the router | 16:34 |
| dougwig | router, or the router's ports? | 16:34 |
| dougwig | maybe it's higher level, then. we do something, expect a change, and never get it. | 16:35 |
| *** imacdonn has quit IRC | 16:35 | |
| dougwig | or they're all just separate and new. | 16:36 |
| *** imacdonn has joined #openstack-neutron | 16:36 | |
| dougwig | but they all have a similar feel. | 16:36 |
| *** janki has quit IRC | 16:41 | |
| slaweq | dougwig: router | 16:41 |
| *** luksky has quit IRC | 16:41 | |
| *** ramishra has quit IRC | 16:41 | |
| *** janki has joined #openstack-neutron | 16:41 | |
| slaweq | dougwig: can You maybe join openstack-meeting channel? We are discussing about it now | 16:41 |
| *** rubasov has joined #openstack-neutron | 16:41 | |
| *** tssurya has quit IRC | 16:45 | |
| *** jpena is now known as jpena|brb | 16:46 | |
| openstackgerrit | Doug Wiegley proposed openstack/networking-sfc master: DNM: ci test https://review.openstack.org/641058 | 16:48 |
| *** br7k has joined #openstack-neutron | 16:48 | |
| *** ivve has quit IRC | 16:58 | |
| *** macza has joined #openstack-neutron | 16:58 | |
| *** davidsha has quit IRC | 17:00 | |
| slaweq | njohnston: what You wanted to talk about? | 17:00 |
| njohnston | OK, the one item I wanted to talk about in the neutron-ci meeting is this: | 17:00 |
| *** wolverineav has joined #openstack-neutron | 17:00 | |
| njohnston | Now that tripleo has shifted to use OVN by default, I think we should make the networking-ovn-tempest-dsvm-ovs-release job voting so our changes do not break them. I pushed up a change to that effect: https://review.openstack.org/#/c/640812/ | 17:00 |
| njohnston | The main issue I see is that the failures for the OVN job are high enough to be of concern. We would need to get those down before we make it voting. See http://grafana.openstack.org/d/Hj5IHcSmz/neutron-failure-rate?panelId=26&fullscreen&orgId=1 | 17:00 |
| njohnston | But once we do that I wanted to see if there were any objections to moving that job to voting/gating? | 17:01 |
| slaweq | yes, I was thinking about it today even as I saw Your patch | 17:01 |
| slaweq | and I think that we should first try to lower failure rate a bit first | 17:01 |
| dougwig | +1 | 17:01 |
| dougwig | first, we have to stop the bleeding. :) | 17:02 |
| slaweq | and also this is question to mlavalle - how he see it, in Denver we agreed to add non-voting jobs from stadium | 17:02 |
| njohnston | Yes, I just wanted to raise the change to make sure there were not any concerns except those related to failure rate | 17:02 |
| slaweq | but IMO for networking-ovn there is no big risk that project will not have community which will fix potential issues | 17:02 |
| dougwig | what's our long-term primary plugin going to be? linuxbridge, ovs, ovn, or other? | 17:03 |
| slaweq | so I'm ok to switch it to be voting job once it will be more stable | 17:03 |
| slaweq | dougwig: that's good question ;) | 17:03 |
| slaweq | dougwig: for now I think it's ovs still (at least for upstream) | 17:03 |
| slaweq | dougwig: but e.g. in TripleO default now is ovn :) | 17:03 |
| mlavalle | yes and we can have a conversation on this in Denver | 17:04 |
| *** macza_ has joined #openstack-neutron | 17:04 | |
| slaweq | mlavalle: sounds good | 17:04 |
| *** wolverineav has quit IRC | 17:04 | |
| slaweq | njohnston: I think that for now we should focus on stabilizing this job a bit | 17:04 |
| njohnston | OK, I will leave the proposed change up but I will leave the -W on it | 17:05 |
| slaweq | will You have some time to take a look at failures and maybe check if there is something what we (or ovn) team can fix? | 17:05 |
| dougwig | i agree. but getting long-term clarity on the bigger plugin picture will also make this an easy decision, as to how many resources we need to devote to it. | 17:05 |
| slaweq | dougwig: default for neutron is now ovs and I think that will not change soon | 17:06 |
| *** br7k has quit IRC | 17:06 | |
| mlavalle | yes | 17:06 |
| slaweq | that's from U/S perspective :) | 17:06 |
| dougwig | i bring it up because we also support linuxbridge (and a lot of operators like lb), and i'm wondering if ovn is moving in the same direction. | 17:07 |
| slaweq | tbh from what is in u/s I think that linuxbridge isn't developed too much - most of the patches are related to ovs (and sriov sometimes) drivers IMO | 17:08 |
| *** macza has quit IRC | 17:08 | |
| slaweq | linuxbridge is in kind of "maintenance" mode IMHO - at least it looks like that for me | 17:08 |
| dougwig | maybe i'm just daydreaming about a set of CI jobs that are just one plugin and one version of python. :) | 17:08 |
| slaweq | dougwig: yes, You are dreaming :) | 17:09 |
| dougwig | the maintenance load is high right now. | 17:09 |
| dougwig | haha. | 17:09 |
| slaweq | but I have that dream too :D | 17:09 |
| openstackgerrit | Manjeet Singh Bhatia proposed openstack/neutron-tempest-plugin master: add assert to ping test https://review.openstack.org/640847 | 17:09 |
| slaweq | ok, I need to go now, sorry guys but my daughers are waiting for me :) | 17:09 |
| slaweq | one last thing I wanted to raise on meeting | 17:10 |
| slaweq | we have first bug for os-ken: https://storyboard.openstack.org/#!/story/2005142 :) | 17:10 |
| slaweq | it's some issue with pep8 | 17:10 |
| slaweq | so gate failure in fact | 17:10 |
| slaweq | if someone have some cycles to look at this, would be great :) | 17:10 |
| slaweq | mlavalle: hongbin ^^ :) | 17:10 |
| mlavalle | slaweq: ack | 17:11 |
| slaweq | thx mlavalle | 17:11 |
| -openstackstatus- NOTICE: Gerrit is being restarted for a configuration change, it will be briefly offline. | 17:11 | |
| slaweq | ok, see You later | 17:11 |
| *** jpena|brb is now known as jpena | 17:12 | |
| *** br7k has joined #openstack-neutron | 17:13 | |
| *** armax has quit IRC | 17:13 | |
| openstackgerrit | Elod Illes proposed openstack/networking-odl master: Use latest Oxygen release instead of snapshot https://review.openstack.org/641088 | 17:14 |
| *** jistr|sick is now known as jistr | 17:14 | |
| openstackgerrit | sean mooney proposed openstack/neutron master: do not skip ports with unset or invalid ofport https://review.openstack.org/640258 | 17:16 |
| openstackgerrit | Doug Wiegley proposed openstack/networking-sfc master: DNM: ci test https://review.openstack.org/641058 | 17:18 |
| *** br7k has quit IRC | 17:19 | |
| *** markvoelker has joined #openstack-neutron | 17:19 | |
| *** ileixe has quit IRC | 17:20 | |
| *** cshen has joined #openstack-neutron | 17:21 | |
| *** iballou has joined #openstack-neutron | 17:22 | |
| *** yamamoto_ has quit IRC | 17:22 | |
| *** luksky has joined #openstack-neutron | 17:23 | |
| *** yamamoto has joined #openstack-neutron | 17:24 | |
| *** yamamoto has quit IRC | 17:24 | |
| *** yamamoto has joined #openstack-neutron | 17:25 | |
| *** cshen has quit IRC | 17:26 | |
| *** rkukura has quit IRC | 17:27 | |
| *** panda|ruck|flu is now known as panda|ruck|off | 17:27 | |
| *** yamamoto has quit IRC | 17:29 | |
| *** sridharg has quit IRC | 17:29 | |
| *** aojea has quit IRC | 17:34 | |
| *** itlinux has joined #openstack-neutron | 17:35 | |
| openstackgerrit | Nate Johnston proposed openstack/neutron-fwaas master: Migrate to stestr as unit tests runner https://review.openstack.org/505526 | 17:42 |
| *** tbachman has quit IRC | 17:44 | |
| *** tbachman has joined #openstack-neutron | 17:44 | |
| openstackgerrit | Vivekanandan Narasimhan proposed openstack/networking-odl master: Re-enable support for multiple router-associations to a single bgpvpn for opendaylight https://review.openstack.org/614431 | 17:48 |
| *** br7k has joined #openstack-neutron | 17:49 | |
| goldyfruit | haleyb: yes ! | 17:51 |
| *** rkukura has joined #openstack-neutron | 17:51 | |
| *** markvoelker has quit IRC | 17:51 | |
| *** rpittau is now known as rpittau|afk | 17:52 | |
| *** br7k has quit IRC | 17:54 | |
| openstackgerrit | boden proposed openstack/neutron master: use payloads for ROUTER_GATEWAY events https://review.openstack.org/638274 | 17:56 |
| *** br7k has joined #openstack-neutron | 17:58 | |
| sean-k-mooney | otherwiseguy: o/ | 18:00 |
| sean-k-mooney | regarding https://review.openstack.org/#/c/640258 | 18:01 |
| sean-k-mooney | if you add a port to ovs that does not yet exist the ovs-db allowcation will success because we use --no-wait and the ovs-vswithd will rais a dataplane error resulting in -1 for the ofport in the ovsdb untill the corresponding kernel netdev is created | 18:02 |
| sean-k-mooney | at which point an ofport will be assigned to the port when it is attached to the dataplane | 18:03 |
| sean-k-mooney | this is then updated in the ovsdb and is detected by the ovs monior | 18:03 |
| otherwiseguy | sean-k-mooney: that's really odd. just posted me doing a killall ovs-vswitchd and --nowait port add and it shows [] as ofport (which is what I remember it always doing). | 18:06 |
| sean-k-mooney | otherwiseguy: try this | 18:07 |
| sean-k-mooney | ubuntu@os-vif-1:/opt/repos/nova$ sudo ovs-vsctl add-br test | 18:07 |
| sean-k-mooney | ubuntu@os-vif-1:/opt/repos/nova$ sudo ovs-vsctl --no-wait add-port test fake-port | 18:07 |
| sean-k-mooney | ubuntu@os-vif-1:/opt/repos/nova$ sudo ovs-vsctl list interface fake-port | 18:07 |
| sean-k-mooney | it will result in the ofport being -1 | 18:07 |
| otherwiseguy | ofport is has min:0 max:1 which basically means a nullable integer. | 18:07 |
| otherwiseguy | that's because that is an error, though. w/o type=internal, that actually fails to add the port if ovs-vswitchd is up, right? | 18:08 |
| sean-k-mooney | yes | 18:09 |
| sean-k-mooney | we are addign the port before it exisit so we expect it the dataplane to rais that error and set the ofport to -1 | 18:09 |
| otherwiseguy | I mean, it's recoverable if you set stuff up right later, but the neutron code has always relied on the interface actually existing. it goes to a lot of trouble to ignore the port until vswitchd is done plugging it in (if I remember correctly because it tries to do ip commands on it very quickly) | 18:11 |
| sean-k-mooney | otherwiseguy: this is what actully happens http://paste.openstack.org/show/747300/ | 18:11 |
| *** br7k has quit IRC | 18:11 | |
| sean-k-mooney | otherwiseguy: nova allway plugs the interface before the vm is created | 18:11 |
| otherwiseguy | and non-vm ports? | 18:12 |
| sean-k-mooney | in the hybrid plug case we create a veth pair that is added to the bridge in non hybridge plug libvirt adds teh port only when it created the tap | 18:12 |
| sean-k-mooney | otherwiseguy: nova does not create non vm ports so i assume the l3 and dhcp aggents create the veth pair or us ovs internal ports | 18:13 |
| otherwiseguy | sean-k-mooney: right, but if we remove the check_ofport, what is to keep them from running ip commands *before* the interfaces exist? | 18:15 |
| otherwiseguy | and failing | 18:15 |
| * otherwiseguy looks for the wait code | 18:15 | |
| sean-k-mooney | otherwiseguy: im putting in a check to prevent the call to the security group agent when ofport is -1 or [] | 18:15 |
| sean-k-mooney | so when the ovs monitor detect the ofport change it triggers the addtion of the port to the updated list that is passed tread_device_added_or_updated | 18:17 |
| sean-k-mooney | onsec ill get a link | 18:17 |
| otherwiseguy | sean-k-mooney: Ok, I see that the check_ofport method is now separate from the _ovsdb_retry that was added. | 18:18 |
| otherwiseguy | which just checks on result == [] | 18:18 |
| sean-k-mooney | sorry its this line that wires it up https://github.com/openstack/neutron/blob/master/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py#L1801 | 18:19 |
| otherwiseguy | so since _check_ofport isn't part of the retrying/waiting behavior, I'm less worried anyway. :D | 18:19 |
| sean-k-mooney | he i basically moved the check to here https://review.openstack.org/#/c/640258/5/neutron/agent/securitygroups_rpc.py@157 | 18:20 |
| sean-k-mooney | if port and (not port.ofport or port.ofport == -1): | 18:20 |
| sean-k-mooney | continue | 18:20 |
| mlavalle | slaweq: I read the channel logs with the rubasov feedback. So we are in syncg | 18:20 |
| sean-k-mooney | that will skip calling the iptables or openflow firewall dirver if the ofport has not been set | 18:21 |
| sean-k-mooney | the next time process_network_ports is called after the netdev is added the firewall rules will be installed | 18:21 |
| mlavalle | rubasov: while we want to merge the networking guide chapter aap, no har deadline there | 18:21 |
| *** njohnston has quit IRC | 18:24 | |
| *** salmankhan has quit IRC | 18:26 | |
| *** jamesdenton has quit IRC | 18:27 | |
| *** jamesdenton has joined #openstack-neutron | 18:29 | |
| *** ccamposr__ has quit IRC | 18:31 | |
| sean-k-mooney | otherwiseguy: i updated the patch with more context and ill work on moving the check into the firwall driver instead. if you have any other suggestions let me know | 18:32 |
| *** jpena is now known as jpena|off | 18:33 | |
| *** mriedem has quit IRC | 18:37 | |
| *** wolverineav has joined #openstack-neutron | 18:38 | |
| *** mriedem has joined #openstack-neutron | 18:39 | |
| openstackgerrit | boden proposed openstack/neutron master: use callback payloads for AGENT OVS_RESTARTED events https://review.openstack.org/636421 | 18:40 |
| *** itlinux has quit IRC | 18:42 | |
| *** br7k has joined #openstack-neutron | 18:46 | |
| *** itlinux has joined #openstack-neutron | 18:47 | |
| *** markvoelker has joined #openstack-neutron | 18:48 | |
| eandersson | > Timeout in RPC method dhcp_ready_on_ports. Waiting for 48 seconds before next attempt. If the server is not down, consider increasing the rpc_response_timeout option as Neutron server(s) may be overloaded and unable to respond quickly enough. | 18:51 |
| eandersson | We are seeing this extensively after upgrading to Rocky. We thought it was due to cpu load at first, but after getting the load down. We are still seeing it frequently. | 18:51 |
| eandersson | Where are these requests originating from? | 18:53 |
| dougwig | how many rpc_workers do you have running? | 18:54 |
| *** yamamoto has joined #openstack-neutron | 18:55 | |
| dougwig | eandersson: ^^ | 18:55 |
| eandersson | 20x2 | 18:56 |
| eandersson | and 8x2 dhcp threads | 18:56 |
| dougwig | rpc and not api, right? | 18:56 |
| eandersson | 16x2 api | 18:56 |
| dougwig | and rabbit isn't getting behind? | 18:56 |
| eandersson | api_workers = 16 | 18:57 |
| eandersson | rpc_workers = 20 | 18:57 |
| eandersson | rabbits are almost idle | 18:57 |
| eandersson | The only issue we had was a "dhcp-agent" deadlock, but at that point there were 0 messages consumed from the dhcp queues | 18:58 |
| *** br7k has quit IRC | 18:58 | |
| dougwig | what's the cpu load on your controllers look like? | 18:58 |
| dougwig | and so you've got 36 or so neutron-server processes running on each controller? | 18:59 |
| eandersson | 14-16 | 18:59 |
| eandersson | We used to have 36 | 18:59 |
| eandersson | (with peaks much higher) | 18:59 |
| dougwig | wait, 16+20 should be 36. how can it be fewer? did you lower those settings? | 19:00 |
| dougwig | are any of the processes dying, due to exceptions or whatever? | 19:02 |
| eandersson | nova and neutron are both on the same controller in this case | 19:03 |
| eandersson | no exceptions | 19:03 |
| dougwig | alright, neutron forks one process per defined api and rpc worker, so if the number of those processes don't match what's in your config file for them, that's the first mystery to solve. | 19:04 |
| eandersson | sorry, I was replying about laod | 19:04 |
| eandersson | not the process count on the controller | 19:04 |
| eandersson | load is at 14-16 now | 19:04 |
| dougwig | oh, 14-16 is the load? | 19:04 |
| dougwig | gotcha. how many physical cores? | 19:04 |
| eandersson | 16 physical (32 with hyperthreading) | 19:04 |
| eandersson | So in my opinion the load is perfect at the momenty | 19:05 |
| dougwig | well, except that rpc processing is starving. :) | 19:05 |
| dougwig | how many hypervisors? | 19:06 |
| eandersson | but only dhcp rpcs for some reason? | 19:07 |
| eandersson | ~300 | 19:07 |
| *** yamamoto has quit IRC | 19:07 | |
| eandersson | neutron-server has not had a single missed RPC that I can see in the logs. | 19:08 |
| eandersson | Other agents are also not complaning | 19:09 |
| eandersson | We can add another neutron controller I guess to verify that it isn't just load related | 19:09 |
| dougwig | that is super weird. | 19:10 |
| dougwig | if that's easy, it'd at least rule out load. | 19:10 |
| *** ivve has joined #openstack-neutron | 19:12 | |
| eandersson | We are gonna try adding another neutron node | 19:14 |
| dougwig | i feel like that last statement could almost be a team slogan. :) | 19:16 |
| *** hamdyk has joined #openstack-neutron | 19:16 | |
| eandersson | haha | 19:17 |
| *** br7k has joined #openstack-neutron | 19:20 | |
| *** markvoelker has quit IRC | 19:21 | |
| *** cshen has joined #openstack-neutron | 19:23 | |
| *** armax has joined #openstack-neutron | 19:26 | |
| *** sm806 has quit IRC | 19:27 | |
| *** cshen has quit IRC | 19:27 | |
| *** sm806 has joined #openstack-neutron | 19:27 | |
| openstackgerrit | Merged openstack/networking-ovn master: Switch default functional tests to Python3 https://review.openstack.org/640988 | 19:28 |
| *** cshen has joined #openstack-neutron | 19:30 | |
| *** lajoskatona has joined #openstack-neutron | 19:31 | |
| *** cshen has quit IRC | 19:34 | |
| *** wolverineav has quit IRC | 19:41 | |
| *** wolverineav has joined #openstack-neutron | 19:42 | |
| *** wolverineav has quit IRC | 19:43 | |
| *** wolverineav has joined #openstack-neutron | 19:43 | |
| eandersson | We added two more neutron-servers and unfortunately still seeing the RPC error | 19:45 |
| *** wolverineav has quit IRC | 19:45 | |
| *** lajoskatona has left #openstack-neutron | 19:45 | |
| *** wolverineav has joined #openstack-neutron | 19:46 | |
| *** wolverineav has quit IRC | 19:46 | |
| *** wolverineav has joined #openstack-neutron | 19:46 | |
| *** dave-mccowan has quit IRC | 19:55 | |
| eandersson | We are seeing the following line (when debug is enabled) being super spammy | 19:55 |
| eandersson | > Checking segment: NetworkSegment | 19:55 |
| *** ceryx has joined #openstack-neutron | 19:55 | |
| eandersson | o/ ceryx | 19:55 |
| ceryx | hi! | 19:55 |
| *** itlinux has quit IRC | 19:56 | |
| eandersson | With 4 neutron-servers servers we are seeing 44128 checking segment requests over 10 minutes on a single control node. | 19:57 |
| eandersson | Is there any way we can reduce those? | 19:58 |
| eandersson | nvm we have only had deubg on for 5 minutes, so it's much higher than that | 19:58 |
| eandersson | 11k per minute (times 4) | 19:59 |
| *** dave-mccowan has joined #openstack-neutron | 20:02 | |
| mlavalle | boden: ping | 20:05 |
| boden | mlavalle howdy | 20:05 |
| mlavalle | boden: sorry, got derailed earlier | 20:05 |
| boden | mlavalle np... so we discussed targetting some consumption patches.. there are a few that are low hanging fruit | 20:07 |
| boden | they have no dependencies | 20:07 |
| boden | https://review.openstack.org/#/c/633283/ https://review.openstack.org/#/c/637004/ https://review.openstack.org/#/c/637304/ | 20:08 |
| boden | there are some others, but I'm getting them up to date now | 20:08 |
| *** itlinux has joined #openstack-neutron | 20:08 | |
| mlavalle | boden: at first glance, these 3 look fine. I'll take a look at them today | 20:09 |
| boden | thanks | 20:09 |
| openstackgerrit | Hamdy Khader proposed openstack/neutron master: Add Smart NIC representor port to integration bridge https://review.openstack.org/586252 | 20:13 |
| *** goldyfruit has quit IRC | 20:14 | |
| *** itlinux has quit IRC | 20:15 | |
| *** markvoelker has joined #openstack-neutron | 20:19 | |
| *** iballou has quit IRC | 20:19 | |
| *** br7k has quit IRC | 20:20 | |
| *** br7k has joined #openstack-neutron | 20:21 | |
| openstackgerrit | Rodolfo Alonso Hernandez proposed openstack/neutron master: [WIP] Add a more robust method to check OVSDB values in BaseOVSTestCase https://review.openstack.org/641117 | 20:24 |
| *** br7k has quit IRC | 20:25 | |
| openstackgerrit | Brian Haley proposed openstack/neutron master: Fix some pylint E128 warnings https://review.openstack.org/641119 | 20:27 |
| eandersson | Anyone know what would be making check_segment_for_agent (ml2 mech_agent) calls? | 20:27 |
| eandersson | Trying to figure out who is sending all these rpc requests. | 20:27 |
| *** igordc has joined #openstack-neutron | 20:28 | |
| *** ralonsoh has quit IRC | 20:29 | |
| *** wolverineav has quit IRC | 20:31 | |
| *** wolverineav has joined #openstack-neutron | 20:32 | |
| slaweq | mlavalle: thx for checking channel logs for info from rubasov :) CI meeting was really long today and I later totally forgot about it | 20:36 |
| *** wolverineav has quit IRC | 20:37 | |
| *** mmichelson has quit IRC | 20:37 | |
| mlavalle | slaweq: no problem, daughters have priority much higher than the stupid PTL. I know my place ;-) | 20:37 |
| slaweq | mlavalle: LOL | 20:37 |
| slaweq | I never though about it this way :) | 20:37 |
| openstackgerrit | Nate Johnston proposed openstack/os-ken master: Fix pep8 errors in os-ken https://review.openstack.org/641122 | 20:39 |
| mlavalle | slaweq: well, that's the way it is. don't even doubt it if you don't want to be slapped by wife | 20:42 |
| *** br7k has joined #openstack-neutron | 20:43 | |
| slaweq | mlavalle: LOL, fortunatelly my wife can't use irc :P | 20:43 |
| slaweq | so she will not read this :D | 20:43 |
| *** br7k has quit IRC | 20:47 | |
| *** jlibosva has quit IRC | 20:50 | |
| *** markvoelker has quit IRC | 20:52 | |
| dougwig | slaweq: looking at that qos patch you posted earlier, at ovs_lib.py, those new methods would very much not enjoy being parallel'ed. do the fullstack or functional tests run in parallel, against a single OVS ? | 20:54 |
| *** br7k has joined #openstack-neutron | 20:55 | |
| slaweq | dougwig: yes, it runs in parallel generally | 20:55 |
| slaweq | I think that we have 4 workers in gate ci always | 20:55 |
| slaweq | but each test should work on own bridge/ports | 20:56 |
| dougwig | "should" ? | 20:56 |
| *** prometheanfire has joined #openstack-neutron | 20:56 | |
| prometheanfire | http://logs.openstack.org/71/640671/3/check/cross-neutron-py27/0fd85d4/testr_results.html.gz | 20:56 |
| prometheanfire | it looks like updates are causing some issues, I think it's pyroute2/sqlalchemy that are doing it | 20:57 |
| prometheanfire | but it'd be nice to have another person look as well | 20:57 |
| prometheanfire | https://review.openstack.org/640671 is the review | 20:57 |
| prometheanfire | maybe alembic as well | 20:58 |
| *** salmankhan has joined #openstack-neutron | 20:59 | |
| slaweq | dougwig: IIRC separate bridges are created | 20:59 |
| slaweq | but I may be wrong or maybe there is some bug in some tests :) | 20:59 |
| *** wolverineav has joined #openstack-neutron | 21:02 | |
| eandersson | What causes > Provisioning complete for port | 21:03 |
| eandersson | We are seeing ~4k per minute | 21:03 |
| *** pcaruana has quit IRC | 21:10 | |
| *** wolverineav has quit IRC | 21:11 | |
| wwriverrat_ | Would love to have discussion with anyone willing regarding multi-segment per host support re: http://lists.openstack.org/pipermail/openstack-discuss/2019-March/003467.html | 21:12 |
| openstackgerrit | Slawek Kaplonski proposed openstack/neutron master: Store journal log in functional tests results https://review.openstack.org/641127 | 21:17 |
| dougwig | eandersson: i'm guessing that's here: neutron/db/provisioning_blocks.py, which is referenced by dhcp | 21:18 |
| *** dsneddon has quit IRC | 21:18 | |
| prometheanfire | I don't think anyone has done testing with sqlalchemy-1.3.0? | 21:22 |
| openstackgerrit | boden proposed openstack/neutron master: use payloads for PORT AFTER_CREATE events https://review.openstack.org/634461 | 21:26 |
| *** br7k has quit IRC | 21:27 | |
| *** br7k has joined #openstack-neutron | 21:28 | |
| eandersson | dougwig, looks related to the DHCP Scheduler | 21:29 |
| *** wolverineav has joined #openstack-neutron | 21:29 | |
| *** janki has quit IRC | 21:29 | |
| *** cshen has joined #openstack-neutron | 21:30 | |
| eandersson | Is network_auto_schedule required? | 21:31 |
| *** salmankhan has quit IRC | 21:32 | |
| *** cshen has quit IRC | 21:34 | |
| *** br7k has quit IRC | 21:34 | |
| *** br7k has joined #openstack-neutron | 21:36 | |
| *** numans has quit IRC | 21:38 | |
| *** ivve has quit IRC | 21:44 | |
| *** goldyfruit has joined #openstack-neutron | 21:45 | |
| *** alisanhaji has quit IRC | 21:46 | |
| *** wolverineav has quit IRC | 21:47 | |
| *** br7k has quit IRC | 21:48 | |
| *** wolverineav has joined #openstack-neutron | 21:48 | |
| *** markvoelker has joined #openstack-neutron | 21:49 | |
| *** mchlumsky has quit IRC | 21:50 | |
| *** wolverineav has quit IRC | 21:53 | |
| *** hamdyk has quit IRC | 21:57 | |
| *** goldyfruit has left #openstack-neutron | 21:58 | |
| *** br7k has joined #openstack-neutron | 22:01 | |
| *** br7k has quit IRC | 22:12 | |
| *** wolverineav has joined #openstack-neutron | 22:12 | |
| *** br7k has joined #openstack-neutron | 22:12 | |
| *** br7k has quit IRC | 22:20 | |
| *** markvoelker has quit IRC | 22:22 | |
| mlavalle | boden: done with the 3 patches you gave me earlier. Thanks! | 22:23 |
| boden | mlavalle thanks! | 22:24 |
| *** br7k has joined #openstack-neutron | 22:24 | |
| *** mvkr has quit IRC | 22:25 | |
| *** boden has quit IRC | 22:30 | |
| *** mmethot has quit IRC | 22:34 | |
| *** sapd1 has joined #openstack-neutron | 22:36 | |
| *** br7k has quit IRC | 22:39 | |
| *** br7k has joined #openstack-neutron | 22:39 | |
| *** wolverineav has quit IRC | 22:47 | |
| *** dsneddon has joined #openstack-neutron | 22:51 | |
| *** munimeha1 has quit IRC | 22:52 | |
| *** tkajinam has joined #openstack-neutron | 22:54 | |
| *** binh_ has quit IRC | 22:55 | |
| *** binh_ has joined #openstack-neutron | 22:55 | |
| *** dsneddon has quit IRC | 22:56 | |
| *** iyamahat has joined #openstack-neutron | 22:58 | |
| *** br7k has quit IRC | 22:59 | |
| *** yamamoto has joined #openstack-neutron | 23:05 | |
| *** iyamahat has quit IRC | 23:08 | |
| *** br7k has joined #openstack-neutron | 23:09 | |
| *** yamamoto has quit IRC | 23:10 | |
| *** igordc has quit IRC | 23:10 | |
| *** br7k has quit IRC | 23:10 | |
| *** dsneddon has joined #openstack-neutron | 23:10 | |
| *** br7k has joined #openstack-neutron | 23:12 | |
| dougwig | eandersson: not if you always specify a network for instances. | 23:14 |
| *** dsneddon has quit IRC | 23:15 | |
| *** br7k has quit IRC | 23:16 | |
| *** markvoelker has joined #openstack-neutron | 23:19 | |
| prometheanfire | for those neutron devs that wish to test with sqlalchemy-1.3.0 I've made a patch to depend on to aid you. https://review.openstack.org/641151 | 23:20 |
| *** wolverineav has joined #openstack-neutron | 23:24 | |
| *** dsneddon has joined #openstack-neutron | 23:25 | |
| *** wolverineav has quit IRC | 23:30 | |
| *** cshen has joined #openstack-neutron | 23:30 | |
| *** br7k has joined #openstack-neutron | 23:31 | |
| *** cshen has quit IRC | 23:35 | |
| *** br7k has quit IRC | 23:38 | |
| *** wolverineav has joined #openstack-neutron | 23:38 | |
| *** tesseract has quit IRC | 23:42 | |
| *** markvoelker has quit IRC | 23:52 | |
| *** wolverineav has quit IRC | 23:59 | |
| *** wolverineav has joined #openstack-neutron | 23:59 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!