| *** jckasper has quit IRC | 00:01 | |
| *** jckasper has joined #openstack-neutron-ovn | 00:01 | |
| *** azbiswas has quit IRC | 00:03 | |
| *** azbiswas has joined #openstack-neutron-ovn | 00:04 | |
| *** azbiswas has quit IRC | 00:04 | |
| *** azbiswas has joined #openstack-neutron-ovn | 00:04 | |
| *** azbiswas has quit IRC | 00:05 | |
| *** jckasper has quit IRC | 00:06 | |
| russellb | Sam-I-Am: thanks for helping debug that! | 00:10 |
|---|---|---|
| Sam-I-Am | russellb: no problem | 00:19 |
| Sam-I-Am | i bet that was it though | 00:19 |
| Sam-I-Am | bleh, another failed vagrant | 00:20 |
| *** salv-orlando has quit IRC | 00:25 | |
| *** aginwala has quit IRC | 00:29 | |
| *** aginwala has joined #openstack-neutron-ovn | 00:32 | |
| *** aginwala has quit IRC | 00:33 | |
| *** aginwala has joined #openstack-neutron-ovn | 00:33 | |
| *** zhouhan has quit IRC | 00:58 | |
| *** dlundquist has joined #openstack-neutron-ovn | 01:00 | |
| *** allan_h has quit IRC | 01:09 | |
| *** aginwala has quit IRC | 01:10 | |
| *** aginwala has joined #openstack-neutron-ovn | 01:14 | |
| *** aginwala has quit IRC | 01:15 | |
| *** aginwala has joined #openstack-neutron-ovn | 01:15 | |
| *** aginwala has quit IRC | 01:43 | |
| *** yamamoto_ has joined #openstack-neutron-ovn | 01:44 | |
| *** aginwala has joined #openstack-neutron-ovn | 01:50 | |
| *** yamamoto_ has quit IRC | 02:04 | |
| *** gangil has quit IRC | 02:16 | |
| *** aginwala has quit IRC | 02:26 | |
| openstackgerrit | Merged openstack/networking-ovn: DPDK support for OVN https://review.openstack.org/275103 | 02:30 |
| *** aginwala has joined #openstack-neutron-ovn | 02:30 | |
| *** arosen12 has quit IRC | 02:46 | |
| *** arosen121 has joined #openstack-neutron-ovn | 02:46 | |
| *** yamamoto_ has joined #openstack-neutron-ovn | 02:51 | |
| *** salv-orlando has joined #openstack-neutron-ovn | 02:53 | |
| *** salv-orlando has quit IRC | 02:55 | |
| *** aginwala has quit IRC | 02:57 | |
| *** aginwala has joined #openstack-neutron-ovn | 03:00 | |
| *** aginwala has quit IRC | 03:02 | |
| *** azbiswas has joined #openstack-neutron-ovn | 03:05 | |
| *** gangil has joined #openstack-neutron-ovn | 03:08 | |
| *** gangil has joined #openstack-neutron-ovn | 03:08 | |
| *** arosen121 has quit IRC | 03:13 | |
| *** azbiswas has quit IRC | 03:13 | |
| *** arosen12 has joined #openstack-neutron-ovn | 03:13 | |
| *** jckasper has joined #openstack-neutron-ovn | 03:27 | |
| *** jckasper has quit IRC | 04:16 | |
| *** jckasper has joined #openstack-neutron-ovn | 04:16 | |
| *** jckasper has quit IRC | 04:19 | |
| *** jckasper has joined #openstack-neutron-ovn | 04:19 | |
| *** chandrav has joined #openstack-neutron-ovn | 04:26 | |
| *** jckasper has quit IRC | 04:37 | |
| *** salv-orlando has joined #openstack-neutron-ovn | 04:37 | |
| *** jckasper has joined #openstack-neutron-ovn | 04:38 | |
| *** salv-orlando has quit IRC | 04:42 | |
| *** chandrav has quit IRC | 05:02 | |
| *** chandrav has joined #openstack-neutron-ovn | 05:06 | |
| *** chandrav has quit IRC | 05:09 | |
| *** toor has quit IRC | 05:09 | |
| *** toor has joined #openstack-neutron-ovn | 05:09 | |
| *** chandrav has joined #openstack-neutron-ovn | 05:10 | |
| *** stac has quit IRC | 05:11 | |
| *** stac has joined #openstack-neutron-ovn | 05:17 | |
| *** jckasper has quit IRC | 05:21 | |
| *** jckasper has joined #openstack-neutron-ovn | 05:21 | |
| *** aginwala has joined #openstack-neutron-ovn | 05:25 | |
| *** allan_h has joined #openstack-neutron-ovn | 05:27 | |
| *** jckasper_ has joined #openstack-neutron-ovn | 05:31 | |
| *** jckasper has quit IRC | 05:32 | |
| *** jckasper_ has quit IRC | 05:38 | |
| *** jckasper has joined #openstack-neutron-ovn | 05:39 | |
| *** jckasper has quit IRC | 05:53 | |
| *** chandrav has quit IRC | 05:53 | |
| *** jckasper has joined #openstack-neutron-ovn | 05:53 | |
| *** salv-orlando has joined #openstack-neutron-ovn | 05:54 | |
| *** salv-orlando has quit IRC | 05:56 | |
| *** jckasper has quit IRC | 05:57 | |
| *** aginwala has quit IRC | 06:01 | |
| *** aginwala has joined #openstack-neutron-ovn | 06:04 | |
| *** numans has joined #openstack-neutron-ovn | 06:09 | |
| *** gangil has quit IRC | 06:13 | |
| *** aginwala has quit IRC | 06:25 | |
| *** numans has quit IRC | 06:26 | |
| *** numans has joined #openstack-neutron-ovn | 06:28 | |
| *** aginwala has joined #openstack-neutron-ovn | 06:35 | |
| *** arosen12 has quit IRC | 06:45 | |
| *** numans has quit IRC | 06:53 | |
| *** armax has quit IRC | 06:54 | |
| *** gangil has joined #openstack-neutron-ovn | 06:56 | |
| *** gangil has joined #openstack-neutron-ovn | 06:56 | |
| *** aginwala has quit IRC | 06:58 | |
| *** aginwala has joined #openstack-neutron-ovn | 07:00 | |
| openstackgerrit | Babu Shanmugam proposed openstack/networking-ovn: Enabling qos support through Logical_Port.options https://review.openstack.org/265798 | 07:03 |
| *** salv-orlando has joined #openstack-neutron-ovn | 07:12 | |
| *** numans has joined #openstack-neutron-ovn | 07:15 | |
| *** salv-orlando has quit IRC | 07:23 | |
| *** allan_h has quit IRC | 07:29 | |
| *** gangil has quit IRC | 07:33 | |
| *** aginwala has quit IRC | 08:17 | |
| *** salv-orlando has joined #openstack-neutron-ovn | 08:28 | |
| *** aginwala has joined #openstack-neutron-ovn | 08:31 | |
| *** salv-orlando has quit IRC | 08:35 | |
| *** aginwala has quit IRC | 08:36 | |
| *** ig0r_ has quit IRC | 08:45 | |
| *** salv-orlando has joined #openstack-neutron-ovn | 09:05 | |
| *** igordcard has joined #openstack-neutron-ovn | 09:11 | |
| *** aginwala has joined #openstack-neutron-ovn | 09:14 | |
| *** aginwala has quit IRC | 09:18 | |
| *** igordcard has quit IRC | 09:42 | |
| *** salv-orlando has quit IRC | 09:52 | |
| *** salv-orlando has joined #openstack-neutron-ovn | 10:03 | |
| *** ig0r_ has joined #openstack-neutron-ovn | 10:13 | |
| *** openstackgerrit has quit IRC | 10:32 | |
| *** openstackgerrit has joined #openstack-neutron-ovn | 10:32 | |
| *** aginwala has joined #openstack-neutron-ovn | 10:35 | |
| *** aginwala has quit IRC | 10:40 | |
| *** salv-orl_ has joined #openstack-neutron-ovn | 10:41 | |
| *** numans has quit IRC | 10:42 | |
| *** salv-orlando has quit IRC | 10:45 | |
| *** numans has joined #openstack-neutron-ovn | 11:30 | |
| *** numans has quit IRC | 11:34 | |
| *** numans has joined #openstack-neutron-ovn | 11:36 | |
| *** ig0r_ has quit IRC | 11:48 | |
| *** ig0r_ has joined #openstack-neutron-ovn | 11:49 | |
| *** aginwala has joined #openstack-neutron-ovn | 11:53 | |
| *** jckasper has joined #openstack-neutron-ovn | 11:55 | |
| *** aginwala has quit IRC | 11:58 | |
| *** jckasper has quit IRC | 12:00 | |
| *** jckasper has joined #openstack-neutron-ovn | 12:06 | |
| *** chandrav has joined #openstack-neutron-ovn | 12:08 | |
| *** yamamoto_ has quit IRC | 12:12 | |
| *** jckasper has quit IRC | 12:16 | |
| *** jckasper has joined #openstack-neutron-ovn | 12:16 | |
| *** jckasper has quit IRC | 12:21 | |
| *** rtheis has joined #openstack-neutron-ovn | 12:25 | |
| *** aginwala has joined #openstack-neutron-ovn | 12:35 | |
| *** aginwala has quit IRC | 12:39 | |
| *** aginwala has joined #openstack-neutron-ovn | 12:48 | |
| *** aginwala has quit IRC | 12:53 | |
| *** salv-orl_ has quit IRC | 12:58 | |
| *** thumpba has joined #openstack-neutron-ovn | 13:13 | |
| *** thumpba has quit IRC | 13:14 | |
| *** thumpba has joined #openstack-neutron-ovn | 13:14 | |
| *** thumpba has quit IRC | 13:19 | |
| *** aginwala has joined #openstack-neutron-ovn | 13:40 | |
| *** aginwala has quit IRC | 13:44 | |
| Sam-I-Am | russellb: moo | 13:45 |
| russellb | ribbit | 13:45 |
| * russellb testing QoS integration | 13:47 | |
| Sam-I-Am | oh boy | 13:47 |
| Sam-I-Am | russellb: i was trying to disable ovn-controller on the vagrant controller node because nothing really needs it. however, this breaks a lot of the devstack plugin. | 13:48 |
| russellb | so fix it! :-p | 13:49 |
| Sam-I-Am | seems to me that in a prod environment, there's actually no reason for any ovn stuff on the controller node, including ovs | 13:49 |
| russellb | yep makes sense | 13:49 |
| russellb | just never tried to configure that | 13:49 |
| Sam-I-Am | however, its nice to have ovs on the vagrant controller node to make testing a bit easier | 13:49 |
| Sam-I-Am | in other words, the controller node can see all the network stuffs | 13:49 |
| openstackgerrit | Russell Bryant proposed openstack/networking-ovn: Enabling qos support through Logical_Port.options https://review.openstack.org/265798 | 13:53 |
| Sam-I-Am | mestery: ping | 13:55 |
| mestery | Sam-I-Am: pong | 13:55 |
| Sam-I-Am | mestery: can you read back a few mins here and let me know what you think? | 13:55 |
| * mestery reads and scales | 13:56 | |
| Sam-I-Am | i'd have to modify devstack to get ovs on a node without any normal ovn bits | 13:56 |
| mestery | Sam-I-Am: I am in favor of not running ovn-controller on the control node for devstack, how hard is that turning out to be? | 13:57 |
| Sam-I-Am | mestery: in theory its easy. in practice... not easy. | 13:57 |
| mestery | lol | 13:57 |
| Sam-I-Am | so, here's my thinking... | 13:57 |
| Sam-I-Am | as i mentioned to russell, in a prod environment, the controller node really won't have any interaction with the ovs/ovn networks... in other words, it really doesn't need ovs at all. | 13:58 |
| * mestery nods | 13:58 | |
| Sam-I-Am | which is sort of what happens if you disable ovn-controller. ovs isnt deployed, and when it goes to do the provider network bits, it breaks | 13:58 |
| russellb | "when it goes to do the provider network bits" | 13:59 |
| russellb | when what does the provider network bits | 13:59 |
| russellb | why is devstack doing provider network bits on a node not running ovn-controller? | 13:59 |
| russellb | "because devstack" ? | 13:59 |
| Sam-I-Am | because of the vagrant provider setup script that runs on every node to configure br-provider | 14:00 |
| russellb | ok, i see | 14:00 |
| Sam-I-Am | yeah, its a holdover from when the controller ran n-cpu | 14:00 |
| Sam-I-Am | reality is, i can probably just pop the eth2 (what used to br-provider) interface on the provider network and "see" all of the stuff on the compute node provider network | 14:01 |
| mestery | Sam-I-Am: This is tied to our Vagrant stuff as well right? | 14:01 |
| Sam-I-Am | the only thing that won't work is your Magic Workaround for accessing private networks | 14:01 |
| Sam-I-Am | well, wont work from the controller node | 14:01 |
| russellb | was Mic22's idea, his magic workaround :) | 14:03 |
| Sam-I-Am | lol | 14:04 |
| Mic22 | what are you talking about? | 14:04 |
| russellb | Mic22: he was referencing the use of an ovs port with type=internal to put your hypervisor on an ovn logical network | 14:04 |
| russellb | Mic22: and i'm trying to give you credit for the idea, even though i added it to our docs :) | 14:05 |
| russellb | he called it my workaround :) | 14:05 |
| russellb | that's all.. | 14:05 |
| *** chandrav has quit IRC | 14:07 | |
| Mic22 | I imagined you were talking about this but I don't get how is this related to a provider net | 14:07 |
| Sam-I-Am | so... | 14:07 |
| Sam-I-Am | right now i think most people would run that workaround on the controller node, so they can play with stuff | 14:08 |
| Sam-I-Am | if we remove ovs from the controller node in vagrant, which is supposed to mimic a minimal prod arch, we cant do the hack there. | 14:08 |
| Sam-I-Am | thats really the only reason. outside of that, the controller still has an interface on the same ip network as the provider net and should "see" vms on the provider net just fine | 14:09 |
| Sam-I-Am | just... not vms on a private net | 14:09 |
| Mic22 | the reason why i needed this hack was to have a bunch of bm to proxy traffinc in the vm net | 14:09 |
| Sam-I-Am | mestery russellb so if we want vagrant to lean more toward this minimal prod-like ref arch, we should remove ovs from the controller | 14:09 |
| Mic22 | well if you have all your BM inside the same L2 segment you can of course just use a provider network | 14:10 |
| Mic22 | and everything would work | 14:10 |
| Sam-I-Am | yeah | 14:10 |
| Mic22 | but in a deployment where you have separate providers I don't see how can you achieve the same with provider nets | 14:11 |
| Sam-I-Am | the vagrant deployment only supports 1 provider net | 14:12 |
| Sam-I-Am | so... | 14:13 |
| Mic22 | so yes ! | 14:13 |
| Sam-I-Am | aight, i'll chuck ovs from the controller node and see what happens | 14:13 |
| *** salv-orlando has joined #openstack-neutron-ovn | 14:13 | |
| Mic22 | it will work even better because you don't have the overhead of tunneling | 14:13 |
| Sam-I-Am | you can all throw rotten tomatoes at my patch if you dont like it | 14:14 |
| Mic22 | not saying this, I was just trying to get your idea. Actually in my deployment on the controller i don't have ovs so i cannot agree more | 14:15 |
| Sam-I-Am | oh, interesting. | 14:15 |
| Sam-I-Am | i'm trying to make vagrant deploy the ref arch | 14:15 |
| Sam-I-Am | at least as close as i can get | 14:15 |
| Sam-I-Am | because that'll eventually go into the networking guide as the deployment scenario | 14:15 |
| Mic22 | the only problem I encountered is related to SSL for the northd | 14:16 |
| Mic22 | but i simply got rid of it | 14:16 |
| openstackgerrit | Numan Siddique proposed openstack/networking-ovn: Neutron ovn northbound db sync tool https://review.openstack.org/277805 | 14:16 |
| Mic22 | dunno if it can be useful for you but this https://patchwork.ozlabs.org/patch/581435/ patch separate ovn dbs from ovs you can take the scripts out and just run the dbs needed | 14:17 |
| Mic22 | i have ovs installed but I'm simply not running it | 14:17 |
| Sam-I-Am | hmmm | 14:18 |
| *** jckasper has joined #openstack-neutron-ovn | 14:37 | |
| Sam-I-Am | russellb: did you see my crazy e-mail to the dev list? | 14:46 |
| russellb | sure did, via my cell phone last night at least | 14:46 |
| mestery | Sam-I-Am: I'm reading now | 14:49 |
| Sam-I-Am | welp, time to dust off the old resume... | 14:50 |
| *** jckasper has quit IRC | 14:50 | |
| *** jckasper has joined #openstack-neutron-ovn | 14:51 | |
| mestery | lol | 14:54 |
| *** jckasper has quit IRC | 14:55 | |
| Sam-I-Am | vagrant, round 34 | 15:06 |
| *** yamamot__ has joined #openstack-neutron-ovn | 15:09 | |
| *** jckasper has joined #openstack-neutron-ovn | 15:14 | |
| *** jckasper has quit IRC | 15:20 | |
| *** jckasper has joined #openstack-neutron-ovn | 15:20 | |
| openstackgerrit | Russell Bryant proposed openstack/networking-ovn: doc: Note DPDK support in the features doc. https://review.openstack.org/279599 | 15:22 |
| *** jckasper has quit IRC | 15:22 | |
| *** jckasper has joined #openstack-neutron-ovn | 15:22 | |
| *** jckasper has quit IRC | 15:27 | |
| openstackgerrit | Numan Siddique proposed openstack/networking-ovn: Neutron ovn northbound db sync tool https://review.openstack.org/277805 | 15:27 |
| *** jckasper has joined #openstack-neutron-ovn | 15:27 | |
| openstackgerrit | Russell Bryant proposed openstack/networking-ovn: doc: Rename doc to "install". https://review.openstack.org/279603 | 15:28 |
| Sam-I-Am | russellb: hey now | 15:29 |
| Sam-I-Am | am i reading this right? | 15:29 |
| russellb | Sam-I-Am: thoughts? | 15:29 |
| Sam-I-Am | ubuntu can has packages? | 15:29 |
| russellb | yes | 15:30 |
| russellb | well | 15:30 |
| Sam-I-Am | in X? or cloud-archive? | 15:30 |
| russellb | no. | 15:30 |
| russellb | well... | 15:30 |
| russellb | the packaging stuff is maintained in ovs itself | 15:30 |
| russellb | it has been updated | 15:30 |
| russellb | i'm not sure if it has been built in any distro yet | 15:30 |
| russellb | step 1 is done though | 15:30 |
| russellb | you can build ovn packages from the ovs repo :) | 15:30 |
| Sam-I-Am | yeah | 15:31 |
| russellb | which means once a version new enough ships in a distro, it should include ovn as well | 15:31 |
| russellb | in theory | 15:31 |
| russellb | (same state as the RHEL/Fedora bits) | 15:31 |
| Sam-I-Am | well, given how quickly ovn/ovs moves along, it probably makes sense to either a) tell people they need to build locally and optionally create packages for their particular version of a distro or b) produce packages for a handful of versions and stuff them in a repo on ovs.org or whatever. however, with all the kernel deps i'm not sure thats possible. | 15:33 |
| Sam-I-Am | russellb: did my ovs-dev post make any sense? | 15:42 |
| *** jckasper has quit IRC | 15:44 | |
| russellb | there are pacakges for the kernel module as well | 15:44 |
| *** jckasper has joined #openstack-neutron-ovn | 15:45 | |
| russellb | and to be honest, i haven't digested your post yet | 15:45 |
| russellb | i'm sure it makes sense though :) | 15:45 |
| Sam-I-Am | lol | 15:45 |
| Sam-I-Am | i recommend tums :) | 15:45 |
| Sam-I-Am | doesnt the kernel package depend on a particular kernel ver? | 15:45 |
| russellb | and by "haven't digested" i mean, haven't read | 15:45 |
| russellb | yes | 15:45 |
| russellb | so kind of a pain | 15:45 |
| Sam-I-Am | seems like we should just run with the 'build it locally, heres how you can build packages if you want to roll this on multiple nodes' | 15:46 |
| russellb | ++ for now | 15:46 |
| Sam-I-Am | i dont need to reinvent the wheel for packaging... ovs has that licked. | 15:46 |
| russellb | we want people testing ovs master right now anyway | 15:46 |
| Sam-I-Am | we'll just need to provide some hints on things like init scripts | 15:47 |
| * russellb nods | 15:47 | |
| russellb | those are included in the packages at least | 15:47 |
| Sam-I-Am | for kicks i might try to install ovs from homebrew packages to see what happens | 15:48 |
| Sam-I-Am | sort of... kick off devstack with ovn, but dont have it install ovs... or something. | 15:48 |
| *** jckasper has quit IRC | 15:49 | |
| openstackgerrit | Russell Bryant proposed openstack/networking-ovn: Fix several ovsdb verify() issues. https://review.openstack.org/279615 | 15:49 |
| *** aginwala has joined #openstack-neutron-ovn | 15:54 | |
| *** aginwala has quit IRC | 15:58 | |
| *** jckasper has joined #openstack-neutron-ovn | 16:00 | |
| *** salv-orlando has quit IRC | 16:01 | |
| *** gangil has joined #openstack-neutron-ovn | 16:02 | |
| *** gangil has joined #openstack-neutron-ovn | 16:02 | |
| *** azbiswas has joined #openstack-neutron-ovn | 16:16 | |
| *** gangil has quit IRC | 16:19 | |
| *** gangil1 has joined #openstack-neutron-ovn | 16:19 | |
| *** ig0r_ has quit IRC | 16:20 | |
| *** gangil1 has quit IRC | 16:20 | |
| azbiswas | russellb: I saw that you proposed the fix https://review.openstack.org/#/c/279615/ for DB Integrity Error | 16:20 |
| russellb | azbiswas: yeah, saw a bug report so went ahead and threw up a patch | 16:21 |
| azbiswas | I was testing with a similar patch last night, the DB errors seem to be gone. I did run into RETRYs and sometimes Timeouts | 16:21 |
| russellb | RETRYs would definitely be expected | 16:21 |
| russellb | timeouts may be worth looking into some more | 16:21 |
| azbiswas | Yes that's good | 16:21 |
| azbiswas | But the timeouts could become a problem | 16:21 |
| russellb | yes | 16:21 |
| russellb | need to figure out where that's happening exactly | 16:22 |
| russellb | maybe we have a timeout set too low | 16:22 |
| azbiswas | I was trying to get to the DB corruption with my patch so I was a little bullish in my scenario | 16:22 |
| russellb | :) | 16:23 |
| azbiswas | I was creating upto 200 ports on a provider network | 16:23 |
| azbiswas | Let's say there are M existing ports, the M+1th port results in M+1 DelACLs + (M+1)*C AddACLs where C is the number of security group rules | 16:24 |
| azbiswas | it's a bunch of transactions that may need to retried in case verify returns RETRY | 16:24 |
| azbiswas | I'm not sure if that is resulting in timeout though | 16:25 |
| russellb | are you getting RuntimeError("OVS transaction timed out") ? | 16:25 |
| russellb | or something else? | 16:25 |
| azbiswas | raise RuntimeError("OVS transaction timed out") | 16:25 |
| azbiswas | RuntimeError: OVS transaction timed out | 16:25 |
| azbiswas | 2016-02-12 09:06:31.482 TRACE neutron.api.v2.resource RuntimeError: OVS transaction timed out | 16:25 |
| russellb | yep, ok | 16:25 |
| russellb | that timeout defaults to 1 minute | 16:26 |
| azbiswas | I'll try out your exact patch and see if that makes a difference | 16:26 |
| russellb | means we're taking longer than 1 minute to successfully complete a given transaction ... | 16:26 |
| azbiswas | Right | 16:26 |
| russellb | that's a lot :/ | 16:26 |
| azbiswas | Let me try out your specific patch - mine had a couple of difference | 16:26 |
| russellb | ok | 16:27 |
| azbiswas | minor ones | 16:27 |
| *** gangil has joined #openstack-neutron-ovn | 16:27 | |
| *** gangil has joined #openstack-neutron-ovn | 16:27 | |
| *** gangil has quit IRC | 16:27 | |
| russellb | i moved things a little more than necessary | 16:27 |
| russellb | but addressed some places other than acls | 16:27 |
| russellb | acls seems to be where we're most likely to hit it | 16:27 |
| *** gangil has joined #openstack-neutron-ovn | 16:27 | |
| *** gangil has joined #openstack-neutron-ovn | 16:27 | |
| russellb | but there were some others | 16:27 |
| azbiswas | right I saw, anyways will update you on the patch | 16:28 |
| russellb | thanks for the help! | 16:28 |
| azbiswas | what about the M+1 * C ACLs updates - can that be improved? | 16:28 |
| russellb | probably. there's a couple things to that ... | 16:28 |
| russellb | we may be deleting/adding more than is actually changing. however, the IDL is supposed to automatically work it down to only things that changed, so the resulting ovsdb transaction should be what we really wanted | 16:29 |
| russellb | i haven't looked at how that's implemented, it could be that we could cut it down in a more efficient way before going into the ovsdb code | 16:29 |
| russellb | if we actually *need* to change that many rows, then that's a different story | 16:30 |
| russellb | there's a good change we really do need to change that many rows though | 16:30 |
| russellb | the default security group rules for neutron include "remote_group" | 16:30 |
| russellb | which is a little bit painful to implement | 16:30 |
| azbiswas | The NB ovs db logs show a lot of transactions - these are optimized out as well? | 16:31 |
| russellb | if you have a network with 200 ports | 16:31 |
| *** numans has quit IRC | 16:31 | |
| russellb | when you add port 201, you really do need to update the rules for the other 200 ports to allow traffic from port 201 | 16:31 |
| russellb | so maybe that's what you're seeing ... | 16:31 |
| russellb | we've talked a bit about making that more efficient, but haven't identified a clear path forward | 16:32 |
| russellb | zhouhan in particular was looking into that | 16:32 |
| *** armax has joined #openstack-neutron-ovn | 16:32 | |
| azbiswas | Is there a bug filed for that? | 16:32 |
| russellb | good question ... i don't think so | 16:32 |
| russellb | i think it would require work in OVN | 16:33 |
| russellb | when these timeouts occurred, were you doing several operations at once, loading the system? | 16:33 |
| russellb | or do you get to where it takes that long just for 1 operation? | 16:33 |
| russellb | (bbl, getting some lunch) | 16:37 |
| azbiswas | I am running with multiple api workers, but 1 operation (add port to provider network) at a time from the CLI. | 16:39 |
| azbiswas | It start coughing at around 150 ports | 16:39 |
| *** allan_h has joined #openstack-neutron-ovn | 16:41 | |
| russellb | ouch | 16:52 |
| russellb | that seems pretty bad | 16:52 |
| russellb | azbiswas: would be good to know if it's just neutron-server eating up the CPU, or if other services are eating the CPU as well (ovsdb-server, ovn-northd, ovn-controller, ...) | 16:53 |
| russellb | or it's just some combination of all of them, or what | 16:53 |
| russellb | in any case, sounds like something we should improve | 16:54 |
| azbiswas | Yes, I am going to try out your patch and see if that makes a difference. And then figure out where the CPU blockage is. | 16:54 |
| azbiswas | But either way, the fix is correct | 16:54 |
| azbiswas | it prevents the DB corruption | 16:54 |
| russellb | yay! | 16:55 |
| azbiswas | other problems already existing | 16:55 |
| azbiswas | existed | 16:55 |
| russellb | we have a couple of patches that will help the performance of this scenario on the OVN side | 16:55 |
| russellb | pending patches i mean | 16:55 |
| azbiswas | NB improvement? | 16:55 |
| russellb | if it looks like ovn-northd and/or ovn-controller are part of the problem anyway | 16:55 |
| azbiswas | yeah true | 16:56 |
| azbiswas | maybe i should run it without ovn-controller to eliminate that part | 16:56 |
| azbiswas | that could be the bottleneck in my simple experiment | 16:57 |
| azbiswas | will keep you updated - the test will take some time to run | 16:57 |
| russellb | sounds good | 16:57 |
| *** yamamot__ has quit IRC | 16:59 | |
| *** yamamoto_ has joined #openstack-neutron-ovn | 17:00 | |
| mestery | Sam-I-Am: Easy one https://review.openstack.org/#/c/279599/ | 17:07 |
| russellb | mestery: Easy one https://review.openstack.org/#/c/279603/ | 17:13 |
| russellb | :) | 17:13 |
| Sam-I-Am | and we try vagrant again... devstack requirements transient failure i think | 17:17 |
| *** openstackgerrit has quit IRC | 17:17 | |
| *** openstackgerrit has joined #openstack-neutron-ovn | 17:17 | |
| openstackgerrit | Merged openstack/networking-ovn: doc: Note DPDK support in the features doc. https://review.openstack.org/279599 | 17:19 |
| *** aginwala has joined #openstack-neutron-ovn | 17:25 | |
| openstackgerrit | Merged openstack/networking-ovn: doc: Rename doc to "install". https://review.openstack.org/279603 | 17:29 |
| *** aginwala has quit IRC | 17:39 | |
| Sam-I-Am | russellb: heres a weird one for ytou | 18:02 |
| Sam-I-Am | russellb: does plugin.sh run 4 times? | 18:02 |
| russellb | more than once, at least, yes | 18:02 |
| russellb | it runs at various "stages" | 18:02 |
| russellb | see the bottom where it figures out what stage we're in and then runs the right stuff | 18:02 |
| Sam-I-Am | i moved my snippet to configure env vars OVN_NB_DB and OVN_SB_DB up into plugin.sh and i get 4 sets of entries | 18:02 |
| Sam-I-Am | and its outside of the main loop... just near the bottom of the script | 18:03 |
| russellb | yep, if it's global, it'll run several times | 18:03 |
| Sam-I-Am | damn | 18:03 |
| *** chandrav has joined #openstack-neutron-ovn | 18:03 | |
| Sam-I-Am | guess i could move it into the main loop under post-config | 18:04 |
| Sam-I-Am | i forgot its run for the various stack/unstack/post-config stages | 18:07 |
| Sam-I-Am | for not changing much, this has become one of the more frustrating vagrant/devstack patches | 18:08 |
| Sam-I-Am | sort of hit a lot of logic problems removing ovn-controller and now ovs from the controller node | 18:08 |
| *** jckasper has quit IRC | 18:11 | |
| *** jckasper has joined #openstack-neutron-ovn | 18:11 | |
| *** aginwala has joined #openstack-neutron-ovn | 18:12 | |
| *** jckasper has quit IRC | 18:12 | |
| *** jckasper has joined #openstack-neutron-ovn | 18:12 | |
| *** terryw is now known as otherwiseguy | 18:14 | |
| *** jckasper has quit IRC | 18:14 | |
| *** jckasper has joined #openstack-neutron-ovn | 18:15 | |
| *** salv-orlando has joined #openstack-neutron-ovn | 18:16 | |
| *** jckasper has quit IRC | 18:17 | |
| *** jckasper has joined #openstack-neutron-ovn | 18:17 | |
| *** aginwala has quit IRC | 18:24 | |
| *** openstack has joined #openstack-neutron-ovn | 18:25 | |
| *** aginwala has joined #openstack-neutron-ovn | 18:28 | |
| azbiswas | russellb: The patch https://review.openstack.org/#/c/279615/ encountered the timeout at the 140 port mark similar to mine. Unfortunately I wasn't monitoring what was going on at that time. | 18:31 |
| russellb | azbiswas: OK, well at least it's reproducible :) | 18:31 |
| azbiswas | I think we should merge it though, since it's not hitting the DB corruption. This is probably a pre-existing condition. | 18:32 |
| russellb | agree | 18:32 |
| russellb | even if it *did* cause a performance regression, i don't think we have a choice | 18:32 |
| russellb | this is required | 18:32 |
| azbiswas | I'll see what the update from my folks are - they are trying out the patch in a real scenario. | 18:33 |
| azbiswas | yes - agree this is necessary for correctness | 18:33 |
| russellb | azbiswas: can you comment on the patch with your successful test report? | 18:34 |
| azbiswas | Yes doing that now and will provide another update after i talk with my folks. | 18:34 |
| *** aginwala has quit IRC | 18:35 | |
| *** aginwala has joined #openstack-neutron-ovn | 18:37 | |
| *** openstackgerrit has quit IRC | 18:47 | |
| *** openstackgerrit has joined #openstack-neutron-ovn | 18:47 | |
| *** jckasper has quit IRC | 18:48 | |
| *** jckasper has joined #openstack-neutron-ovn | 18:48 | |
| *** jckasper has quit IRC | 18:49 | |
| *** jckasper has joined #openstack-neutron-ovn | 18:49 | |
| *** jckasper has quit IRC | 18:51 | |
| *** jckasper has joined #openstack-neutron-ovn | 18:52 | |
| openstackgerrit | Russell Bryant proposed openstack/networking-ovn: doc: Document HOST_IP setting requirement. https://review.openstack.org/279723 | 18:54 |
| *** jckasper has quit IRC | 18:54 | |
| *** jckasper has joined #openstack-neutron-ovn | 18:54 | |
| *** numans has joined #openstack-neutron-ovn | 19:07 | |
| *** numans has quit IRC | 19:17 | |
| *** aginwala has quit IRC | 19:22 | |
| *** yamamoto_ has quit IRC | 19:34 | |
| *** aginwala has joined #openstack-neutron-ovn | 19:36 | |
| *** azbiswas has quit IRC | 19:58 | |
| *** gangil has quit IRC | 19:59 | |
| *** aginwala has quit IRC | 20:00 | |
| *** aginwala has joined #openstack-neutron-ovn | 20:02 | |
| *** aginwala has quit IRC | 20:03 | |
| *** aginwala has joined #openstack-neutron-ovn | 20:07 | |
| *** aginwala has quit IRC | 20:12 | |
| *** azbiswas has joined #openstack-neutron-ovn | 20:12 | |
| *** aginwala has joined #openstack-neutron-ovn | 20:13 | |
| openstackgerrit | Merged openstack/networking-ovn: doc: Document HOST_IP setting requirement. https://review.openstack.org/279723 | 20:42 |
| openstackgerrit | Russell Bryant proposed openstack/networking-ovn: Enabling qos support through Logical_Port.options https://review.openstack.org/265798 | 20:46 |
| *** yamamoto_ has joined #openstack-neutron-ovn | 20:48 | |
| *** gangil has joined #openstack-neutron-ovn | 20:51 | |
| *** gangil has joined #openstack-neutron-ovn | 20:51 | |
| *** yamamoto_ has quit IRC | 20:52 | |
| *** aginwala has quit IRC | 20:57 | |
| *** aginwala has joined #openstack-neutron-ovn | 20:59 | |
| *** salv-orlando has quit IRC | 21:03 | |
| azbiswas | russellb: My tests are showing that each port is deleting and re-adding the same set of ACLs for each new port add in that provider network. There is no remote_group in this case. | 21:08 |
| azbiswas | Basically the entire set of ACLs in the DB are getting changed for each port add | 21:08 |
| azbiswas | Am wondering if we can optimize: | 21:09 |
| azbiswas | txn.add(self._ovn.delete_acl( | 21:09 |
| azbiswas | utils.ovn_name(port['network_id']), | 21:09 |
| azbiswas | port['id'])) | 21:09 |
| azbiswas | sg_ports_cache = {} | 21:09 |
| azbiswas | subnet_cache = {} | 21:09 |
| azbiswas | self._add_acls(context, port, txn, | 21:09 |
| azbiswas | sg_ports_cache=sg_ports_cache, | 21:09 |
| azbiswas | subnet_cache=subnet_cache) | 21:09 |
| azbiswas | in _update_port_in_ovn | 21:09 |
| russellb | that's what happens in Python | 21:11 |
| russellb | is that what actually gets sent to ovsdb-server as well? | 21:11 |
| azbiswas | I'll have to investigate that - am noticing the entire DB ACL uuids changing | 21:11 |
| russellb | a logical switch has an acls column | 21:12 |
| russellb | which is a set of UUIDs | 21:12 |
| azbiswas | right | 21:12 |
| russellb | you'll always see a new set of UUIDs sent | 21:12 |
| russellb | no way to optimize that really | 21:12 |
| azbiswas | what about the actual ACLs themselves | 21:13 |
| russellb | that's not true, there is, but requires a new ovsdb feature | 21:13 |
| azbiswas | The ACL table | 21:13 |
| azbiswas | the uuids in that table seems to completely change as well | 21:13 |
| azbiswas | is that expected? | 21:14 |
| russellb | no | 21:14 |
| azbiswas | I'll write up a script to compare | 21:14 |
| russellb | my expectation was that if an ACL didn't actually change, it would get excluded from the transaction | 21:14 |
| russellb | i left a comment in the code saying that | 21:14 |
| russellb | if it's not true, then we definitely have room for optimization here! | 21:15 |
| azbiswas | right, that was my thought as well, but let me write the script | 21:15 |
| azbiswas | and see | 21:15 |
| russellb | ok :) | 21:15 |
| *** yamamoto_ has joined #openstack-neutron-ovn | 21:18 | |
| *** yamamoto_ has quit IRC | 21:22 | |
| *** aginwala has quit IRC | 21:31 | |
| *** s3wong has joined #openstack-neutron-ovn | 21:36 | |
| -openstackstatus- NOTICE: The infrastructure team is taking gerrit offline for maintenance this afternoon, beginning at 22:00 utc. We should have it back online around 23:00 utc. http://lists.openstack.org/pipermail/openstack-dev/2016-February/086195.html | 21:46 | |
| *** yamamoto_ has joined #openstack-neutron-ovn | 21:48 | |
| azbiswas | russellb: I ran my script - as I suspected with my eyeball test, all the ACL uuids have changed after 1 port was added | 21:49 |
| russellb | ok, sounds like it's not behaving like i expected | 21:49 |
| russellb | i'm leaving now for the weekend though | 21:50 |
| azbiswas | can I take over this investigation | 21:50 |
| russellb | of course :) | 21:50 |
| russellb | want to file a bug and then assign it to yourself? | 21:50 |
| azbiswas | yes will do | 21:50 |
| russellb | thanks! have a nice weekend! | 21:50 |
| azbiswas | have a good weekend | 21:50 |
| *** yamamoto_ has quit IRC | 21:52 | |
| *** aginwala has joined #openstack-neutron-ovn | 21:54 | |
| -openstackstatus- NOTICE: Gerrit is offline for maintenacne until 23:00 utc | 22:05 | |
| *** ChanServ changes topic to "Gerrit is offline for maintenacne until 23:00 utc" | 22:05 | |
| *** rtheis has quit IRC | 22:30 | |
| *** aginwala has quit IRC | 22:38 | |
| *** aginwala has joined #openstack-neutron-ovn | 22:42 | |
| *** yamamoto_ has joined #openstack-neutron-ovn | 22:48 | |
| *** yamamoto_ has quit IRC | 22:52 | |
| *** jckasper has quit IRC | 23:02 | |
| *** jckasper has joined #openstack-neutron-ovn | 23:03 | |
| *** jckasper has quit IRC | 23:06 | |
| *** jckasper has joined #openstack-neutron-ovn | 23:06 | |
| -openstackstatus- NOTICE: Gerrit is offline for maintenance, ETA updated to 23:30 utc | 23:06 | |
| *** ChanServ changes topic to "Gerrit is offline for maintenance, ETA updated to 23:30 utc" | 23:06 | |
| Sam-I-Am | moo? | 23:09 |
| *** aginwala has quit IRC | 23:13 | |
| *** jckasper has quit IRC | 23:16 | |
| *** allan_h has quit IRC | 23:16 | |
| *** jckasper has joined #openstack-neutron-ovn | 23:16 | |
| *** aginwala has joined #openstack-neutron-ovn | 23:19 | |
| *** jckasper has quit IRC | 23:21 | |
| *** jckasper has joined #openstack-neutron-ovn | 23:22 | |
| *** jckasper has quit IRC | 23:27 | |
| *** jckasper has joined #openstack-neutron-ovn | 23:28 | |
| *** azbiswas has quit IRC | 23:30 | |
| *** azbiswas has joined #openstack-neutron-ovn | 23:31 | |
| *** jckasper has quit IRC | 23:32 | |
| *** yamamoto_ has joined #openstack-neutron-ovn | 23:33 | |
| *** azbiswas has quit IRC | 23:36 | |
| *** yamamoto_ has quit IRC | 23:37 | |
| -openstackstatus- NOTICE: Gerrit is offline for maintenance, ETA updated to 23:59 utc | 23:41 | |
| *** ChanServ changes topic to "Gerrit is offline for maintenance, ETA updated to 23:59 utc" | 23:41 | |
| *** salv-orlando has joined #openstack-neutron-ovn | 23:44 | |
| *** armax has quit IRC | 23:44 | |
| *** armax has joined #openstack-neutron-ovn | 23:44 | |
| *** jckasper has joined #openstack-neutron-ovn | 23:52 | |
| *** jckasper has quit IRC | 23:53 | |
| *** salv-orlando has quit IRC | 23:53 | |
| *** jckasper has joined #openstack-neutron-ovn | 23:54 | |
| *** jckasper has quit IRC | 23:58 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!