Monday, 2016-01-25

*** thumpba has joined #openstack-neutron-ovn00:15
*** thumpba has quit IRC00:20
*** dslev has joined #openstack-neutron-ovn00:34
*** chenli has joined #openstack-neutron-ovn00:38
*** dslev has quit IRC01:05
*** lrichard has quit IRC01:14
*** fzdarsky__ has joined #openstack-neutron-ovn02:12
*** fzdarsky_ has quit IRC02:15
*** thumpba has joined #openstack-neutron-ovn02:18
*** yamamoto has joined #openstack-neutron-ovn02:20
*** thumpba has quit IRC02:23
*** yamamoto has quit IRC02:31
*** thumpba has joined #openstack-neutron-ovn02:54
*** yamamoto_ has joined #openstack-neutron-ovn03:23
*** mamulsow has quit IRC03:30
*** gangil has joined #openstack-neutron-ovn03:52
*** gangil has joined #openstack-neutron-ovn03:52
*** dslev has joined #openstack-neutron-ovn04:01
*** dslev has quit IRC04:06
*** thumpba has quit IRC05:59
*** gangil has quit IRC06:00
*** gangil has joined #openstack-neutron-ovn06:08
*** gangil has joined #openstack-neutron-ovn06:08
*** salv-orlando has joined #openstack-neutron-ovn07:18
*** fzdarsky__ has quit IRC07:34
*** fzdarsky has joined #openstack-neutron-ovn07:36
*** palexster has joined #openstack-neutron-ovn08:25
*** gangil has quit IRC08:31
*** thumpba has joined #openstack-neutron-ovn08:31
*** thumpba has quit IRC08:36
*** salv-orlando has quit IRC09:04
*** flaviof has quit IRC09:22
*** chenli has quit IRC09:34
*** salv-orlando has joined #openstack-neutron-ovn10:05
*** salv-orlando has quit IRC10:12
*** salv-orlando has joined #openstack-neutron-ovn10:22
*** thumpba has joined #openstack-neutron-ovn10:33
*** thumpba has quit IRC10:39
*** yamamoto_ has quit IRC11:12
*** roeyc has joined #openstack-neutron-ovn11:34
*** yamamoto has joined #openstack-neutron-ovn11:58
*** yamamoto_ has joined #openstack-neutron-ovn11:59
*** yamamoto has quit IRC12:03
*** yamamoto_ has quit IRC12:17
*** rtheis has joined #openstack-neutron-ovn12:19
*** yamamoto has joined #openstack-neutron-ovn12:26
*** thumpba has joined #openstack-neutron-ovn12:50
*** thumpba has quit IRC12:55
*** salv-orlando has quit IRC13:02
*** yamamoto has quit IRC13:03
*** yamamoto has joined #openstack-neutron-ovn13:04
*** yamamoto has quit IRC13:08
*** lrichard has joined #openstack-neutron-ovn13:12
*** yamamoto has joined #openstack-neutron-ovn13:17
*** flaviof has joined #openstack-neutron-ovn13:22
*** dslev has joined #openstack-neutron-ovn13:52
*** dslev_ has joined #openstack-neutron-ovn13:53
*** gongysh has joined #openstack-neutron-ovn13:53
*** salv-orlando has joined #openstack-neutron-ovn13:57
*** dslev has quit IRC13:57
*** yamamoto has quit IRC14:02
*** thumpba has joined #openstack-neutron-ovn14:10
openstackgerritMerged openstack/networking-ovn: Vagrant: Fix issue with boxes
openstackgerritMerged openstack/networking-ovn: Vagrant: Adjust HOST_IP for compute nodes
*** gongysh has quit IRC14:34
russellbtempest totally broken for us right now, fix is:
*** shettyg has joined #openstack-neutron-ovn15:16
*** shettyg has quit IRC15:17
*** dslev_ has quit IRC15:45
Sam-I-Amrussellb: moo?15:49
Sam-I-Amrussellb: got some time today?15:50
Sam-I-Amexcellent. think friday ran out on us.15:50
Sam-I-Amare you east coast?15:51
*** regXboi has joined #openstack-neutron-ovn15:51
Sam-I-Amhere comes trouble15:51
Sam-I-Amoh hai regXboi15:52
regXboiSam-I-Am: not trouble yet - I think I've figured out my headache, which is that I'm missing a log15:53
regXboiwhat is happening is A->B->C and I'm looking at the logs of A and C, without the log of B15:54
regXboiwhich makes understanding the process somewhat challenging15:54
regXboiotherwise, morning all16:02
*** salv-orl_ has joined #openstack-neutron-ovn16:05
Sam-I-AmregXboi: ramping this mtu testing up was sort of similar16:06
Sam-I-Am'crap i should have tested X here too'16:06
*** salv-orlando has quit IRC16:08
russellbyes i'm east coast16:12
*** shettyg has joined #openstack-neutron-ovn16:12
Sam-I-Amrussellb: whens good for you today?16:14
mesteryrussellb: Sad about the tempest brokeness :(16:42
*** flaviof has quit IRC16:59
*** ajo has quit IRC17:20
russellbmestery: all your fault!17:24
russellbSam-I-Am: i have a call at 1, so 2pm or later (eastern)17:24
Sam-I-Amrussellb: lets do 2 eastern17:27
*** ajo has joined #openstack-neutron-ovn17:33
*** azbiswas has joined #openstack-neutron-ovn17:35
*** azbiswas_ has joined #openstack-neutron-ovn17:40
*** azbiswas has quit IRC17:43
*** azbiswas_ has quit IRC17:48
*** azbiswas_ has joined #openstack-neutron-ovn17:49
*** shettyg has quit IRC18:03
russellbmestery: could use a review on this once jenkins signs off
* mestery looks18:07
russellbnetworking-ovn busted on that18:07
mesteryWAiting on Jenkins18:09
*** gangil has joined #openstack-neutron-ovn18:09
*** gangil has joined #openstack-neutron-ovn18:09
*** chandrav has joined #openstack-neutron-ovn18:30
*** salv-orl_ has quit IRC18:35
*** shettyg has joined #openstack-neutron-ovn18:40
*** pcaruana has joined #openstack-neutron-ovn18:51
regXboirussellb: does it make sense to add code to the devstack/plugin to treat ovsdb-server log the same way as the other logs?  Right now, there is only one copy kept around which caused me some grief after a restack :(18:54
russellbregXboi: yes, and ovs-vswitchd while we're at it18:54
regXboirussellb: ok, I'll spin a patch out in the near future to address both issues18:54
russellback thanks18:54
*** gangil has quit IRC18:55
Sam-I-Amrussellb: can we push our chat 30 minutes?18:57
russellbSam-I-Am: yes, i'm reading your dev posts now anyway18:57
Sam-I-Amrussellb: oh, they're really exciting.19:00
Sam-I-Amthe one from this past friday is the most 'accurate' since i had a real lab with jumbo frames and didn't have to munge a bunch of stuff19:00
*** openstackgerrit has quit IRC19:02
*** openstackgerrit has joined #openstack-neutron-ovn19:02
*** pcaruana has quit IRC19:32
*** salv-orlando has joined #openstack-neutron-ovn19:35
*** pcaruana has joined #openstack-neutron-ovn19:36
Sam-I-Amrussellb: how goes the reading?19:39
russellbwell, i read it19:39
russellbi can't say i have my head wrapped around it all though19:39
Sam-I-Amwhere can i help?19:40
russellbi probably just have to spend more time understanding things ... most helpful thing probably is to spoon feed me/ovn any recommendations you have19:40
*** pcaruana has quit IRC19:40
russellblike "add option X to make life easier"19:41
*** mamulsow has joined #openstack-neutron-ovn19:41
Sam-I-Ami'm betting that ovn with the neutron l3 agent is going to experience at least some of these problems19:41
Sam-I-Ambut i'm particularly curious about ovn w/ native l319:41
Sam-I-Amthats what i'm testing here in a bit19:41
russellbyes, native L3 is what i care about most19:41
russellbl3 agent is a temp hack anyway19:41
russellbok cool19:41
Sam-I-Amalso things like dhcp... any namespacey stuff19:42
mamulsowso jumping in the middle of a conversation I didn't see the beginning of, but..19:42
mamulsowanything in particular you're looking for in the native l3 routing?19:43
mamulsowor dhcp stuff19:43
*** pcaruana has joined #openstack-neutron-ovn19:43
mamulsowI've been hammering on that stuff pretty hard19:43
Sam-I-Ammamulsow: in conventional neutron, we have a problem with mtu when VMs reside on overlay networks19:43
Sam-I-Ampeople have been throwing random fixes at it for a while, but none of them actually work, so i ran some experiments to see whats up19:44
mamulsowhmm, so we were seeing some issues that seemed to be MTU related, but I bumped MTU to 9000 on all our physical layer stuff and haven't seen MTU issues since then19:45
*** salv-orlando has quit IRC19:46
Sam-I-Ammamulsow: this wouldn't be performance issues... it would be more like things breaking issues19:46
Sam-I-Amlet's say your physical layer has mtu 900019:47
mamulsowyes, I'm not 100% sure it was MTU caused, but when the compute nodes had MTU 1500 we were seeing dropped traffic and weird cases of only being able to get 1 way communication19:47
Sam-I-Ama packet from the outside world with an mtu of 9000 heads to the VM... which has a smaller mtu due to the overlay protocol overhead19:47
Sam-I-Amohhh.... thats sort of smells like mtu problems19:47
Sam-I-Ammtu disparities are normal... e.g., the internet doesnt have a 9000 mtu, yet hosts with 9000 mtu seem to communicate fine over it.19:48
Sam-I-Amthe problem is where the disparity occurs19:48
Sam-I-Amnegotiating mtu is a layer-3 operation that requires icmp... so if the disparity happens on a purely layer-2 device (such as a switch, bridge, or veth pair), packets with a larger mtu are silently dropped.19:49
mamulsowso for our tests VMs had MTU of 1500 and physical layer had MTU 1500, so with encap overhead we should have been splitting any large packets, but for some cases it seemed like packets were getting silently dropped19:50
mamulsowbut it's entirely possible I had something misconfigured that was causing that issue19:52
Sam-I-Amthis is what i'm looking to find out19:52
Sam-I-Amso your fix was just increasing the phys net mtu to 9000?19:53
mamulsowyep, and haven't seen issues since then19:53
Sam-I-Amok, thats also what my conventional ovs testing reveals19:53
Sam-I-Amdid you try just setting the vm's mtu to 1500 minus overlay overhead?19:54
Sam-I-Amthe vm side is the easy part. we need them to know the mtu of their overlay link.19:54
Sam-I-Amthe not so easy side is what comes in from the outside world19:55
mamulsownot this time, but in the past I believe we did that with conventional OVS and got good results19:55
mamulsowie. VM MTU 1450 with vxlan encapsulation over physical layer at 150019:56
mamulsowI believe that worked well19:56
*** pcaruana has quit IRC19:56
Sam-I-Amit sort of works19:56
*** flaviof has joined #openstack-neutron-ovn19:56
mamulsowbut then we had the problem of what if a customer uploads an image with a different MTU19:56
Sam-I-Amhopefully people are using dhcp... or they're aware that mtu is 'a thing' and they can obtain the value from the api (or something)19:57
Sam-I-Ammamulsow: can you see in the code for the native l3 agent whether or not the ports plugged into it (assuming one on either network) account for the mtu of those networks?20:00
Sam-I-Amor if ovn/ovs generate icmp fragmentation-needed or packet-too-large messages in general20:01
mamulsownot off the top of my head, but I can go check20:02
mamulsowI suspect that it is not, give what we've been seeing20:02
Sam-I-Amso far, my testing with conventional ovs reveals that ovs itself Just Works with any arbitrary mtu20:04
Sam-I-Amits all the non-ovs stuff thats broken20:04
Sam-I-Amwe'll always have a router between a provider net and a private net... and that's where the mtu change needs to occur, with icmp messages20:06
*** lrichard has quit IRC20:07
Sam-I-Amthe mtu value for the private net should be calculated from the mtu value of the phys net it rides on20:07
Sam-I-Ammamulsow: is there a way to glean mtu values of ovs ports?20:08
*** lrichard has joined #openstack-neutron-ovn20:11
*** flaviof has quit IRC20:13
*** flaviof_ has joined #openstack-neutron-ovn20:13
*** flaviof_ has quit IRC20:14
*** flaviof has joined #openstack-neutron-ovn20:15
russellbSam-I-Am: i like "just works"!20:20
Sam-I-Amrussellb: yeah, hence why i'm curious whats going on inside20:20
russellbi wouldn't know..20:21
Sam-I-Aminside ovs?20:21
russellbyes, i don't know ovs internals that well20:25
russellbSam-I-Am: thanks for digging into this in so much detail, i do really appreciate it20:27
Sam-I-Amrussellb: can you point me at someone for ovs internals?20:29
Sam-I-Amrussellb: its a long-standing issue that needs resolution... lets get it right the first time with ovn :)20:29
russellbovs discuss mailing list is probably the best audience20:32
*** salv-orlando has joined #openstack-neutron-ovn20:32
*** rtheis has quit IRC20:38
*** chandrav has quit IRC20:38
*** thumpba has quit IRC20:46
openstackgerritRyan Moats proposed openstack/networking-ovn: Make OVS log multi-stack friendly
regXboirussellb: ^^^^^ (there you go)20:55
russellback thanks20:55
*** yamamoto has joined #openstack-neutron-ovn20:57
russellbregXboi: i thought you were going to do ovsdb-server as well?21:00
russellband i'm not actually sure what this is doing really21:00
regXboirussellb - this patch was for this one log21:00
russellbwhat about running ovsdb-server and ovs-vswitchd in screen with the rest?21:01
russellbrun_process ovsdb-server ..21:01
regXboirussellb: well, now that's an interesting idea - I was thinking of that in a separate patch set21:01
regXboibut -1 this and I'll merge it all togethre21:01
russellbthanks for working on it21:02
regXboisure. no worries21:07
regXboiI got caught by it not working that way over the weekend and I don't want to lose work in the future21:07
*** manand has joined #openstack-neutron-ovn21:07
regXboirussellb: I see why I got confused - it looks like both ovs-vswitchd and ovsdb-server log to the same place21:09
regXboiwhich I can also fix in the next PS21:09
Sam-I-Amyeah... they can (and probably should) be different21:11
Sam-I-Amdistros usually package them as such21:12
*** gongysh has joined #openstack-neutron-ovn21:30
*** chandrav has joined #openstack-neutron-ovn21:47
*** roeyc has quit IRC21:50
*** roeyc has joined #openstack-neutron-ovn21:51
*** Guest66973 has joined #openstack-neutron-ovn21:56
*** gongysh has quit IRC22:01
*** regXboi has quit IRC22:01
*** gongysh has joined #openstack-neutron-ovn22:01
*** roeyc has quit IRC22:01
*** roeyc has joined #openstack-neutron-ovn22:02
*** roeyc has quit IRC22:03
*** salv-orl_ has joined #openstack-neutron-ovn22:06
*** roeyc has joined #openstack-neutron-ovn22:06
*** gongysh has quit IRC22:07
*** salv-orlando has quit IRC22:09
*** yamamoto has quit IRC22:10
*** flaviof has quit IRC22:22
*** s3wong has joined #openstack-neutron-ovn22:34
*** chandrav has quit IRC23:26
*** yamamoto_ has joined #openstack-neutron-ovn23:58

Generated by 2.14.0 by Marius Gedminas - find it at!