Thursday, 2016-01-21

openstackgerritAaron Rosen proposed openstack/networking-ovn: Disable more IPv6 tests.
*** chandrav has joined #openstack-neutron-ovn00:15
*** chandrav has quit IRC00:44
*** flaviof has quit IRC01:02
*** salv-orlando has quit IRC01:09
*** gangil has quit IRC01:14
*** gangil has joined #openstack-neutron-ovn01:16
*** gangil has joined #openstack-neutron-ovn01:16
*** gangil has quit IRC01:24
openstackgerritAaron Rosen proposed openstack/networking-ovn: random test
*** gangil has joined #openstack-neutron-ovn01:54
*** gangil has joined #openstack-neutron-ovn01:54
*** gangil has quit IRC01:58
*** gangil has joined #openstack-neutron-ovn02:03
*** gangil has joined #openstack-neutron-ovn02:03
*** gangil has quit IRC02:07
*** gangil has joined #openstack-neutron-ovn02:10
*** gangil has joined #openstack-neutron-ovn02:10
*** arosen has quit IRC02:13
*** roeyc has quit IRC02:17
*** gangil has quit IRC02:18
*** fzdarsky has quit IRC02:19
*** yamamoto has joined #openstack-neutron-ovn02:23
*** fzdarsky has joined #openstack-neutron-ovn02:25
*** yamamoto has quit IRC02:37
*** yamamoto has joined #openstack-neutron-ovn02:37
*** roeyc has joined #openstack-neutron-ovn02:41
*** yamamoto has quit IRC02:46
*** roeyc has quit IRC02:49
*** gangil has joined #openstack-neutron-ovn03:07
*** gangil has joined #openstack-neutron-ovn03:07
*** yamamoto_ has joined #openstack-neutron-ovn03:39
*** armax has quit IRC03:42
openstackgerritli,chen proposed openstack/networking-ovn: HOST_IP is missing in computenode-local.conf.sample
*** armax has joined #openstack-neutron-ovn04:20
*** chenli has joined #openstack-neutron-ovn04:34
chenlihello guys,    I have build a 2 node devstack environment with ovn enabled followed by
chenliThen I created 10 instances04:35
chenliBut all instances on the second node do not get IP address from DHCP04:35
chenlianyone can help me ?04:35
*** gangil has quit IRC04:42
*** mamulsow has quit IRC04:48
*** mamulsow has joined #openstack-neutron-ovn04:49
*** gangil has joined #openstack-neutron-ovn04:51
*** gangil has joined #openstack-neutron-ovn04:51
*** mamulsow has quit IRC04:54
*** roeyc has joined #openstack-neutron-ovn05:33
*** numans has joined #openstack-neutron-ovn06:17
*** chandrav has joined #openstack-neutron-ovn06:22
*** chandrav has quit IRC07:17
*** stac- has quit IRC07:23
*** fzdarsky has quit IRC07:27
*** fzdarsky has joined #openstack-neutron-ovn07:27
*** chandrav has joined #openstack-neutron-ovn07:33
*** stac has joined #openstack-neutron-ovn07:47
*** numans has quit IRC07:52
*** roeyc has quit IRC07:57
*** roeyc has joined #openstack-neutron-ovn07:59
*** gangil has quit IRC08:15
*** numans has joined #openstack-neutron-ovn08:23
*** ajo_ is now known as ajo08:24
*** chandrav has quit IRC08:28
*** stac has quit IRC08:31
*** chandrav has joined #openstack-neutron-ovn08:32
*** armax has quit IRC08:36
*** stac has joined #openstack-neutron-ovn08:40
*** chandrav has quit IRC09:33
*** openstackgerrit has quit IRC10:02
*** openstackgerrit has joined #openstack-neutron-ovn10:02
*** roeyc has quit IRC10:06
*** roeyc has joined #openstack-neutron-ovn10:12
*** roeyc has quit IRC10:44
*** ajo has quit IRC11:24
*** ajo has joined #openstack-neutron-ovn11:25
*** yamamoto_ has quit IRC11:31
*** chenli has quit IRC12:46
*** chenli has joined #openstack-neutron-ovn12:47
*** flaviof has joined #openstack-neutron-ovn13:01
*** chenli has quit IRC13:09
*** rtheis has joined #openstack-neutron-ovn13:33
*** yamamoto has joined #openstack-neutron-ovn13:48
*** yamamoto has quit IRC13:49
*** yamamoto has joined #openstack-neutron-ovn13:52
*** dslevin has quit IRC13:54
* mestery yawns and sips coffee14:10
Sam-I-Ami have not coffeed yet14:10
mesterySam-I-Am: Get on that man!14:11
Sam-I-Amovsing before coffee is dangerous14:11
Sam-I-Amyeah, i'm slackin'14:11
* Sam-I-Am looks at pile of things14:11
Sam-I-Amgates are still broken14:13
Sam-I-Amthink we need to figure out how to make the ovn gate actually test provider nets14:16
mesterySam-I-Am: The Vagrant updates I have out for review do that, so it's possible14:16
mesteryBut yes, we need to do that14:16
Sam-I-Ambut last time i looked into that, i got deep into neutron-legacy and almost jumped off my roof14:17
*** stac has quit IRC14:18
Sam-I-Amit looks easy... tell ovn there's a public net, put a flat neutron net there, drop an ip on br-ex (or whatever) ... everything else is the same14:18
Sam-I-Amright now the gate puts qg* into br-ex which bypasses... a lot of things14:19
Sam-I-Amwhich is how i came across the larger neutron gate problem14:19
Sam-I-Amthe only reason it works at all is because the l3 agent has external_network_bridge=br-ex14:19
Sam-I-Amshould be fun when that option finally goes away14:19
Sam-I-Amif you look at the nets, the pub net is type vxlan... yet it works... because of that option14:20
Sam-I-Amthats when i spent too long in the devstack code and went blind14:20
*** yamamoto has quit IRC14:28
russellbmisc idea ... a bug tag for bugs that require work in OVN itself (not just networking-ovn)14:33
russellbbut i can't think of a good name for it14:33
russellbnaming is hard.14:34
*** stac has joined #openstack-neutron-ovn14:34
mesteryrussellb: ovn-ovs? ovn-github? ovn-upstream?14:35
russellbi even made it an official tag in launchpad.14:36
russellbwatch out!14:37
Sam-I-Amrussellb: any more suggestions for my ovn gate logic patch?14:38
russellbnot right now, no14:39
Sam-I-Amok. didnt know if we wanted to do anything re: dougs comment14:39
russellbi commented14:39
Sam-I-Ami think i fixed your comment14:39
Sam-I-Amunless you commented again14:39
russellbi suggested we remove tests from the regex14:39
russellbas suggested by doug14:39
*** dslev has joined #openstack-neutron-ovn14:39
Sam-I-Ami did that14:39
Sam-I-Ami'm quick. like a rabbit, but not like rabbitmq.14:39
*** regXboi has joined #openstack-neutron-ovn14:41
russellbheh, +114:54
Sam-I-Amthat file is a mess though14:55
Sam-I-Amrussellb: what impact does the mtu on the system-level bridge interface have?14:56
Sam-I-Amif any14:57
russellbnot sure14:57
Sam-I-Amcan one set the mtu of an ovs bridge?14:57
Sam-I-Ami'm trying my mtu experiments on ovs now14:57
Sam-I-Amseems like an ovs bridge uses the mtu of the smallest interface on the bridge, which makes sense14:58
*** yamamoto has joined #openstack-neutron-ovn15:03
mesteryrussellb: Looks like blp and han provided nice feedback on your OVN provider patches! Yay!15:12
russellbmestery: yep, i need to re-spin them today15:12
*** shettyg has joined #openstack-neutron-ovn15:19
*** mamulsow has joined #openstack-neutron-ovn15:23
regXboirussellb: ping - shouldn't ovn-controller be dropping a PID file in /usr/local/var/run/openvswitch so that ovs-appctl will work?15:38
*** fzdarsky has quit IRC15:39
russellbregXboi: is it not?15:39
*** fzdarsky has joined #openstack-neutron-ovn15:39
russellboh, we're not running with --pid-file15:39
russellbsame for ovn-northd15:39
russellbwanna patch devstack/ ?15:40
russellb.. and put it in the queue of stuff we can't merge yet :(15:40
mesteryMan, we gotta get the dsvm jobs fixed15:40
mesteryso many patches15:40
regXboisure, I'm on it15:40
russellbi have some ideas to chase, but long todo list too15:40
mamulsowrussellb: let me know when you want to start looking at ovn-controller again15:42
russellbmamulsow: main question i have right now is what impact patch #2 has vs just patch #115:43
russellbperf wise15:43
russellbon an idle system15:43
mamulsowI cleaned up our environment last night, but the latest ovn-controller from your branch is intermittently spinning at 100% with nothing in the cloud15:43
mamulsowother ovn-controller nodes were all 0%15:44
mamulsowhaven't had a chance to look into where it's spinning its time, but I can do that now15:44
mamulsowspending its time*15:44
*** fzdarsky has quit IRC15:45
mamulsowyeah, on an idle system the latest build from your ovn-controller-perf branch is spinning at 100%15:46
russellbok thanks15:49
russellbweird, that was supposed to make it do *less*15:50
*** fzdarsky has joined #openstack-neutron-ovn15:52
mamulsowoh, I think it is doing less work, but isn't sleeping at the end of each poll interval15:56
openstackgerritRyan Moats proposed openstack/networking-ovn: Run northd and ovn-controller with --pidfile
regXboirussellb: ^^^^^15:58
*** numans has quit IRC15:58
regXboinow to restack with it and verify :)15:59
russellbmamulsow: it should be waiting on poll_block(), in theory, unless it's constantly waking back up because we're skipping code that should be handling something making poll wake up immediately...16:01
regXboimestery: you should have made something else out of that16:01
russellbseems like i should be able to reproduce that if so16:01
russellbmamulsow: oh, yes, it's spinning in a trivial test setup for me, oops16:02
russellbi'll figure it out16:02
mamulsowcool, I like easy problems :)16:03
russellbalso, reminder for everyone interested, OVN meeting is today in about 2 hours (see topic)16:04
russellbthis patch makes my laptop very sad :-p16:06
*** yamamoto has quit IRC16:11
*** yamamoto has joined #openstack-neutron-ovn16:11
*** yamamoto has quit IRC16:11
regXboiand yes, that patch worked16:14
russellbgreat thanks regXboi16:15
regXboisc68cal: where are we w.r.t all dsvm jobs getting unwedged?16:15
regXboiugh - wrong handle16:16
regXboiSam-I-Am: ^^^^16:16
Sam-I-AmregXboi: good question16:17
Sam-I-Ami think its this patch -
Sam-I-Amwhich was failing due to the nova-net pub/priv thing16:19
regXboiyeah but that patch failed the recheck16:22
regXboiI think16:22
regXboiyeah, it did16:22
regXboiso this is now waiting on ... what?16:22
russellbmamulsow: i just dropped that 2nd patch from the branch for now ...16:22
mamulsowso I added some debug statements and started it (patch #2) again and it's working great right now16:23
mamulsowthe environment has about 20 routers and several VMs16:23
regXboiSam-I-Am: I'm watching -infra now16:24
russellbhm, well something is broken with it .. it made ovn-controller spin at 100% even without creating any resources16:24
regXboiso I see what's going on16:24
mamulsowother ovn-controllers are at about 10 - 30%, patched one is at 0%16:24
Sam-I-AmregXboi: the talk is actually in -qa16:24
Sam-I-Amas i later found out16:24
Sam-I-Amso many irc16:24
mamulsowyeah, I'll try creating/deleting some stuff to see if I can get it to hit the 100% case again16:25
mamulsowwhen it works it works great though :)16:25
mamulsowrussellb: I created one network and it started busy spinning16:28
mamulsowit's no longer waiting at poll_block16:28
mamulsowovsdb_changed = 016:28
mamulsowboth before and after16:28
mamulsowI didn't catch it when I made the change, but I assume it was true once16:29
russellbsomething in the true path clears an fd thats' causing poll to wake up16:29
russellbnot sure what, needs a closer look16:29
russellbmamulsow: mestery btw, there's some config things you could consider to isolate your VMs from ovn-controller (and other openstack agents, like nova-compute)16:59
russellbyou can configure nova to exclude a CPU from the set of CPUs VMs are allowed to use16:59
russellbso then you'd have 1 CPU free for everything else16:59
russellbjust a random thought16:59
mamulsowsounds like a good idea16:59
russellbprobably worthwhile though17:00
russellbas ovn-controller is going to spike as you load the system with create/deletes, same with nova-compute i bet17:00
russellbto some degree anyway17:00
russellbbut i'd definitely like to get ovn-controller down to where it's truly idling when the env is idling, that's a bug IMO17:00
russellbwe should be able to do that soonish17:00
russellbbut like i said, you'll still see spikes that are normal (as it calculates new state in response to changes)17:01
mesteryThanks for the advice russellb, mamulsow I think we should look at implementing this.17:02
mamulsowyeah, these boxes have 56 cores and are mostly idle, I wasn't so much concerned about the CPU usage itself, but I was worried that it appeared OVN controller wasn't able to keep up with the requests it was getting17:02
mamulsowbut yes, that does seem like a good thing to do anyway17:03
russellbi don't think it's not keeping up17:04
russellbi think it's actually keeping up way too aggressively, heh17:04
*** flaviof is now known as flaviof_afk17:05
*** gangil has joined #openstack-neutron-ovn17:05
*** gangil has joined #openstack-neutron-ovn17:05
* mestery heads out for lunch and will be back in an hour or so17:06
*** salv-orlando has joined #openstack-neutron-ovn17:10
*** roeyc has joined #openstack-neutron-ovn17:12
*** roeyc has quit IRC17:13
*** roeyc has joined #openstack-neutron-ovn17:15
*** flaviof_afk is now known as flaviof17:15
*** arosen has joined #openstack-neutron-ovn17:16
*** rtheis has quit IRC17:28
*** armax has joined #openstack-neutron-ovn17:28
*** rtheis has joined #openstack-neutron-ovn17:29
*** rtheis has quit IRC17:33
mamulsowwell, I put in a dumb hack that's working pretty well, added a 'rerun' variable that gets set to true any time ovsdb_changed so it runs twice through the true case17:34
mamulsowobviously it would be better to find the fd that's not getting cleared17:34
mamulsowbut this is working well for me now17:35
mamulsowother interesting insight from that is that a second run through the true case clears the fd17:37
regXboiok, 270417 has merged - let's see if that makes a difference or not17:40
Sam-I-AmregXboi: woohoooo17:43
*** rtheis has joined #openstack-neutron-ovn17:43
* regXboi watches jobs in zuul and prepares rechecks17:45
Sam-I-Amtrigger happy?17:46
regXboiI like to think of it is "being prepared" :)17:47
regXboirussellb: I may be a bit distracted at the start of today's IRC meeting17:53
* regXboi has a rescheduled scrum call running at the same time and has to channel mestery while on it17:53
Sam-I-AmregXboi: you need more ram17:53
regXboiSam-I-Am: I could use more parallel processing17:53 ?17:54
regXboihas somebody registered that?17:54
Sam-I-Amlooks that way17:55
* regXboi sighs17:56
Sam-I-Ammy brain is full17:56
Sam-I-Ami need to increase the mtu17:56
Sam-I-Amdid you read my mtu ramblings from the weekend?17:56
regXboiwhere were they?17:57
* regXboi was chasing getting ovn running most of last weekend17:57
* regXboi queues up to read later17:58
Sam-I-Ami am not responsible for drain bramage resulting from it17:58
Sam-I-Amtl;dr - i think i figured out the primary problem we're having, at least for phys nets with mtu = 150017:58
Sam-I-Amthe next step is seeing what happens with phys net mtu > 1500. i.e., do the 'middle things' use it, or do we need to do something else17:59
regXboiok, I read the tl;dr - /me now weeps17:59
Sam-I-Amthe next step is 'what happens in ovn'17:59
Sam-I-AmregXboi: it should make you happy18:00
regXboino it makes me weep18:00
Sam-I-Amweep for joy?18:00
regXboino weep18:00
regXboibut that's for a little later - time for scrum call now18:00
*** flaviof is now known as flaviof_afk18:05
*** flaviof_afk is now known as flaviof18:05
arosenjoin #openvswitch18:09
arosendoh :(18:09
*** shettyg1 has joined #openstack-neutron-ovn18:11
*** shettyg has quit IRC18:12
*** chandrav has joined #openstack-neutron-ovn18:14
*** numans has joined #openstack-neutron-ovn18:16
*** zhouhan has joined #openstack-neutron-ovn18:18
*** azbiswas has joined #openstack-neutron-ovn18:19
mesterylol arosen :)18:36
*** rtheis has quit IRC18:49
*** rtheis has joined #openstack-neutron-ovn18:50
numansrussellb, for multiple entries of same mac in Logical_Port.addresses, it needs a fix right ? (
*** rtheis has quit IRC18:54
*** numans has quit IRC18:58
*** rtheis has joined #openstack-neutron-ovn19:09
*** roeyc has quit IRC19:13
*** roeyc has joined #openstack-neutron-ovn19:22
*** roeyc has quit IRC19:23
*** roeyc has joined #openstack-neutron-ovn19:29
*** roeyc has quit IRC19:42
*** salv-orlando has quit IRC19:42
regXboiso, it looks like the dsvm stuff is somewhat uncorked19:52
regXboiand maybe totally uncorked19:55
arosenwhat was it?20:01
regXboiok... good news - 269121 passed dsvm :) - so if somebody wants to pull the W+1 on it - I'm going to recheck my other patch20:01
arosena change in ovn ?20:01
arosenor openstack ;)20:01
regXboino - it was changes needed in openstack20:01
arosenwhich one?20:02
regXboi270417 was one of them20:02
regXboiI don't remember the other20:02
regXboibut thanks for rechecking 270879 - that was where I was going20:03
arosenah this keystone thing?20:03
regXboithat was part of it yes20:03
regXboiextra delays leading to race conditions in nova-net20:03
arosendo we understand the race conditions?20:04
regXboium... yes - it's nova networks20:04
arosenwhat's the race condition?20:04
arosenit seems like the tempest test did:20:04
regXboinova networks API does *not* hang around and wait for a valid response code before continuing20:04
arosendidn't wait long enough for it to be deleted (and have the ports cleaned up)20:04
regXboiit's async20:04
arosenbefore doing delete network.20:04
russellbarosen: that's what i was thinking too20:05
russellbarosen: and looping in ml2 would mask it20:05
russellbseems worthy of a dev list post20:05
regXboiI filed a bug a while back against nova about that problem20:05
arosenwell i wonder if it's a bug in nova then or tempest.20:05
* regXboi goes and looks for it20:05
russellbarosen: that'd be a bug in ... tempest and neutron, depending on perspective20:06
regXboiyeah... take a look at
openstackLaunchpad bug 1497740 in OpenStack Compute (nova) "nova API proxy to neutron should avoid race-ful behavior" [Medium,Confirmed]20:06
russellb1) tempest needs to make sure it's a valid time to delete a network before trying to delete it ...20:06
russellb2) neutron shouldn't mask things in a stupid way20:06
regXboithat's w.r.t to floating ips in specific, but I can believe the same problem is elsewhere20:06
aroseni'll recheck all the patches up there.20:06
regXboiarosen, I was going to do that, but if you have a quicker way :)20:07
regXboibut anyway, russellb, you want to pull the +A lever on 269121?20:08
arosenregXboi: i just do it manually.20:08
arosendone anyways.20:08
regXboiarosen: ok, thx20:08
arosencould write a script to do it ;)20:08
russellbregXboi: done20:08
regXboimahalo - /me now goes to zuul page to watch the various patches20:08
regXboi178826 just merged20:09
russellbthat merged a long time ago right?20:09
regXboihmm, that's not what I saw just now - let me double check20:10
regXboiyeah this merged back on 1/1320:10
russellbarosen: you think we should try to work on a tempest patch?20:13
*** rtheis has quit IRC20:13
russellbi probably don't have time this week though, maybe we should record it in a bug for now20:14
regXboirussellb: I was trying out master this morning with the northd and controller processes in debug mode and I'm seeing a bunch of row_event messages in q-svc log - is that expected?20:14
russellbregXboi: sounds lik eit20:14
arosenrussellb:   we can. I want to look at nova as well and see if the issue could be there too.20:15
regXboiso if northd and controller are logging in info, they don't report row events?20:15
arosenit seems like after nova marks a vm as deleted the vm's ports could still be around20:15
russellbarosen: i think vm delete is an async request20:15
arosenor it could be tempest not waiting on the vm to be deleted (and checking that it's deleted).20:15
russellbarosen: is tempest doing anything to wait for a vm to go deleted?20:15
russellbyeah, need to look at tempest code, i haven't yet20:15
arosenlet me look :)20:15
russellbi'm still guessing20:15
russellbi have a big pile of ovn patches i'm trying to revise20:15
*** rtheis_ has joined #openstack-neutron-ovn20:17
arosenit looks like tempest does wait.20:17
*** rtheis__ has joined #openstack-neutron-ovn20:18
*** rtheis has joined #openstack-neutron-ovn20:20
*** rtheis has quit IRC20:20
*** rtheis has joined #openstack-neutron-ovn20:20
*** rtheis_ has quit IRC20:21
*** rtheis__ has quit IRC20:22
*** azbiswas has quit IRC20:28
*** azbiswas has joined #openstack-neutron-ovn20:29
arosenlet me play around with nova and see what it does if i put a huge sleep in port_delete in neutron.20:29
aroseni'll dig into it.20:29
arosengoing to grab a quick lunch though bbl20:29
*** azbiswas has quit IRC20:33
russellbsounds good20:34
* russellb revising his provider network fixes for ovn20:34
*** chandrav has quit IRC20:40
*** gangil has quit IRC20:45
*** chandrav has joined #openstack-neutron-ovn20:53
*** dslev has quit IRC21:00
*** manand has joined #openstack-neutron-ovn21:05
*** salv-orlando has joined #openstack-neutron-ovn21:11
russellbi don't think our job is fixed21:13
russellbsame errors happening on the rechecks21:13
*** roeyc has joined #openstack-neutron-ovn21:14
*** zhouhan has quit IRC21:16
*** gangil has joined #openstack-neutron-ovn21:19
*** gangil has joined #openstack-neutron-ovn21:19
*** dslev has joined #openstack-neutron-ovn21:19
mamulsowrussellb: nothing urgent, by FYI, I've been doing some scale testing with your latest (I believe it's patch #1 level now) and see some interesting log messages21:21
russellbonly on the patched host?21:21
mamulsowI've got them all patched now, I could switch one back though to test21:22
russellbarosen: looking at one of the latest NetworkInUse failures, and i dug into the specific test code ...21:23
russellb100     def _delete_server(self, server):21:23
russellb101         self.servers_client.delete_server(server['id'])21:23
russellb102         waiters.wait_for_server_termination(self.servers_client, server['id'])21:23
russellbso it waits for the server to be terminated, in theory21:23
openstackgerritMerged openstack/networking-ovn: Make master networking-ovn work with stable/liberty
mesterymamulsow: Now that ^^^ has merged, we can likely move away from my private fork we're using for testing.21:24
mesterymamulsow: I assume regXboi tested this on stable/liberty even :)21:24
mamulsowmestery: nice!21:24
regXboiI'm pretty sure I did - it all blurs together now21:25
russellbseems we're passing some now at least21:26
mesteryrussellb: progress?21:27
russellbyes progress of some sort21:27
russellbmamulsow: re: log messages, looks "normal" if ovn-controller is under load21:28
russellbthe INFO stuff is a little noisy under high load it seems21:28
mamulsowyeah, it was about 600 routers, 600 networks/subnets at the time, and still in the process of creating things21:29
*** chandrav has quit IRC21:29
russellbmamulsow: so ovn-controller is just pegged recalculating state as things are changing21:31
russellbyou've basically hit a bottleneck21:31
*** azbiswas has joined #openstack-neutron-ovn21:31
mamulsowit's not *too* noisy, it's adding a few lines every 5 seconds or so, a lot better than many of the other logs on this system21:31
russellb1) you may want to make the nova config change i suggested earlier to reserve a CPU for system stuff21:31
russellb2) we need to improve ovn-controller to help this bottleneck21:31
russellb3) even if it's pegged, it's likely still processing everything, it's just a question of how long it's taking21:32
russellband if it can keep up with the rate of change you want to inflict on the env21:32
mamulsowyep, 1.4 second poll interval is not bad21:32
russellbyeah, so that means it's re-adjusting to new state in 1.4 seconds or so21:32
mamulsowas long as that doesn't get into the 30 second range21:32
russellbwhich should apply all changes that have happened in the last 1.4 seconds or whatever21:32
russellbsomething like that21:33
russellbi'd also verify other ways, like look at how long it takes ports to come up21:33
russellbright now we have it so that neutron port state won't be up until OVN says it's up21:33
mamulsowso is this applying just the things that changed since the last time or is it trying to apply everything21:33
russellband OVN won't say it's up until ovn-controller has done its thing for the new port21:33
russellbright now ovn-controller does a full calculation of state every time.  blp said in our meeting today that he'd try to prioritize working on incrememntal updates now that we know this is a bottleneck21:34
russellbit calculates full state, and then in practice only applies differences21:34
russellbbut there's lots of room to be smarter21:34
mamulsowcool, sounds good21:34
mamulsowso far this is still looking pretty good even as it's working now21:35
russellbok great21:35
mamulsowI'll let you know how it looks when I get to 4k routers21:35
russellbwe'll keep workikng on improvements too of course21:35
russellbthis is hugely helpful21:35
*** chandrav has joined #openstack-neutron-ovn21:36
mamulsowhugely helpful for me too :)21:38
mamulsowthanks so much for your help so far, definitely seeing gains with your patch21:38
russellbgreat, that was low hanging fruit21:41
russellbthat patch is going to change, ben told me he had some suggestions21:41
russellbbut it's a test env after all :)21:41
russellbarosen: from my digging through the neutron code, it should be deleting ports before the instance is marked as deleted in the db, so as long as all tests are waiting like the one i looked at earlier, it should be OK...21:43
russellbdigging through *nova* code21:43
mesteryrussellb: neutron mid-cycle in Rochester, MN at IBM:
mesteryA chance to hack on OVN perhaps?21:43
russellbi thought that wasn't happening for some reason21:44
mesteryIt wasn't ... and then armax and dougwig made it happen :P21:44
russellbhow do i get there, snowmobile?21:44
mesteryAnd whiskey. Whiskey keeps you warm.21:44
russellbmestery: i put it on my calendar to think about at least21:45
Sam-I-Ami thought the neutron mid-cycle was just for a few things... 2 weeks ago21:45
russellbbut what a terrible place to go in Feb :-p21:45
Sam-I-Ammarch sounds like a 3/4 cycle21:45
Sam-I-Amor the week prior21:45
* russellb trolling21:46
russellbmestery: 4 full days?21:47
russellbor more likely 2 full and 2 partial, depending on travel schedule ...21:47
Sam-I-Amrussellb: got time for some questions?21:48
mesteryYeah, that's what I was thinking21:48
mestery2 full, 2 partial21:48
mesteryI may even drive home a couple nights for kids activities (shhhh, don't tell)21:48
Sam-I-Ammestery: aren't you up that way?21:48
mesterySam-I-Am: About 1.5 hours away21:48
* regXboi gets in queue behind Sam-I-Am:21:49
regXboispeak for yourself - it's more like 6 hours21:49
Sam-I-AmregXboi: at least you can avoid air travel21:49
russellbquick questions at least21:50
russellbi need to leave in a few minutes21:50
regXboithe beauty of being in the middle of the country - anything from about OKC to Denver to Fargo to Chicago can be a drive21:50
Sam-I-Amrussellb: doh, how about tomorrow?21:50
russellbi'll be around21:50
Sam-I-Amrussellb: i guess the quick question is - has someone thought about mtu considerations in ovn.21:51
regXboirussellb: pointer to what AddLogicalPortCommand actually does vis a vis OVS21:51
Sam-I-Amadds a logical port21:51
russellbSam-I-Am: yes, but i'm not sure OVN needs to do anything21:51
* regXboi looking for a secret decoder ring in terms of the protocol, Sam-I-Am21:51
russellbsame issue as with vxlan or gre today21:52
Sam-I-Amor should i say &addlogicalportcommand :)21:52
russellbregXboi: AddLogicalPortCommand creates a new row in the Logical_Port table in the OVN_Northbound db21:52
russellbwell, and adds it to the ports column of a logical switch in the Logical_Switch table21:52
regXboirussellb: ok, does the client do anything in addition to sending the new row?21:52
*** azbiswas has quit IRC21:52
regXboidoes it pull the table first?21:52
regXboidoes it pull the table afterwards?21:53
russellbis the schema docs21:53
russellbthe client actually has a local cache of the entire db21:53
russellbthat it receives updates for from the db21:53
regXboiasync updates or sync updates?21:53
russellbit gets a dump at startup, and then async updates over time21:53
regXboiok, so that likely isn't it21:53
russellbdepending on what you mean by sync or async21:53
regXboiwell, would it refresh the cache *before* making the update21:54
regXboier the add I mean21:54
regXboiok, so that's not it21:54
regXboiok, I'll likely need to add some instrumentation into ovn-controller to trace the code21:54
Sam-I-Amrussellb: well, it might need to pull in an mtu config option. i'm mainly concerned about the mismatch problem we're discovering in the ml2 drivers. mtus can (and should) change to account for overhead, as long as the change happens in something that can emit icmp pmtu messages.21:54
Sam-I-Amrussellb: i dont know enough about ovs and the l3 implementation to know if it'll do this21:55
regXboibut that will be part of tomorrow's headache21:55
russellbSam-I-Am: *nods*, what would it do with an mtu config option21:55
russellbright now we rely on the admin configuring the dhcp server opt for it21:55
Sam-I-Amrussellb: that only helps traffic outbound from a vm21:55
Sam-I-Amits the easy part of this21:56
russellbthen i don't understand the hard part :)21:56
Sam-I-Aminbound traffic21:56
Sam-I-Amhost outside openstack sends packet with max mtu, packet needs to traverse a tunnel to get to the vm. what happens there? with the ml2 drivers, its just discarded because the disparity occurs on a layer-2 device that can't speak icmp for ptmu discovery.21:57
*** azbiswas has joined #openstack-neutron-ovn21:58
Sam-I-Amtl;dr on my mtu experiments over the weekend - we need to do mtu changes to account for tunnels in the layer-3 device that routes between the provider net and overlay net21:59
*** dslev has quit IRC21:59
openstackgerritMerged openstack/networking-ovn: Run northd and ovn-controller with --pidfile
regXboirussellb: ok I see now - I'll revisit my instrumentation run with that in mind22:01
*** s3wong has joined #openstack-neutron-ovn22:01
Sam-I-Amrussellb: the other side of the coin is making sure that if the phys net supports a large mtu, that large mtu is implemented in all the necessary places22:01
russellbSam-I-Am: ok, this is probably an area i don't understand well know to know what should change, but definitely be interested in feedback22:01
russellbi need to go for now though22:01
Sam-I-Amrussellb: yeah, thats why we need a bit more time22:02
Sam-I-Ami can fill you in on All the Things22:02
Sam-I-Amthe goal - get it right the first time :)22:02
*** shettyg1 has quit IRC22:02
Sam-I-Ambecause its a mess in neutron now22:02
russellbok cool22:10
russellbi think my brain is fried for today22:10
regXboirussellb: oh I ack that22:10
russellbhave a nice evening everyone22:10
mesteryyou too russellb22:10
*** dslev has joined #openstack-neutron-ovn22:14
openstackgerritKyle Mestery proposed openstack/networking-ovn: Vagrant: Completely redo the Vagrant configuration
*** chandrav has quit IRC22:19
*** regXboi has quit IRC22:19
*** chandrav has joined #openstack-neutron-ovn22:25
openstackgerritMerged openstack/networking-ovn: Devstack: cleanup datapath
openstackgerritMerged openstack/networking-ovn: devstack: Move tox install.
openstackgerritMerged openstack/networking-ovn: Deployment: Update with OVN DB requirements
*** dslev has quit IRC22:34
* mestery loves it22:35
*** chandrav has quit IRC22:40
openstackgerritMerged openstack/networking-ovn: Updated from global requirements
*** chandrav has joined #openstack-neutron-ovn22:43
*** chandrav has quit IRC22:53
*** jckasper has joined #openstack-neutron-ovn23:00
*** chandrav has joined #openstack-neutron-ovn23:01
*** roeyc has quit IRC23:05
*** chandrav has quit IRC23:06
*** roeyc has joined #openstack-neutron-ovn23:16
openstackgerritMatthew Kassawara proposed openstack/networking-ovn: Modify docs build environment
openstackgerritAaron Rosen proposed openstack/networking-ovn: Add missing call to self._process_l3_update/delete()
openstackgerritMerged openstack/networking-ovn: Vagrant: Completely redo the Vagrant configuration
*** azbiswas has quit IRC23:55
*** rtheis has quit IRC23:57

Generated by 2.14.0 by Marius Gedminas - find it at!