Tuesday, 2015-09-22

*** salv-orlando has quit IRC00:24
*** salv-orlando has joined #openstack-neutron-ovn02:28
*** salv-orlando has quit IRC02:32
*** arosen has quit IRC03:44
*** armax has quit IRC03:55
*** salv-orlando has joined #openstack-neutron-ovn05:26
*** salv-orlando has quit IRC05:33
*** subscope has quit IRC07:03
*** salv-orlando has joined #openstack-neutron-ovn07:03
*** salv-orlando has quit IRC09:56
*** salv-orlando has joined #openstack-neutron-ovn10:47
*** regXboi has joined #openstack-neutron-ovn12:46
*** salv-orlando has quit IRC13:18
*** shettyg has joined #openstack-neutron-ovn14:11
*** shettyg has quit IRC14:50
*** armax has joined #openstack-neutron-ovn15:07
*** shettyg has joined #openstack-neutron-ovn15:10
russellbthat demo with sflow is really cool15:15
mesteryrussellb: +1000!15:52
russellblove the potential it demonstrates, at least15:52
russellbnow to replicate it using all open source software :)15:52
russellbannnnnd go15:52
*** arosen has joined #openstack-neutron-ovn16:45
openstackgerritAaron Rosen proposed openstack/networking-ovn: Test remove oslo_db_api.wrap_db_retry  https://review.openstack.org/22641516:47
openstackgerritAaron Rosen proposed openstack/networking-ovn: Test remove oslo_db_api.wrap_db_retry  https://review.openstack.org/22641616:47
openstackgerritAaron Rosen proposed openstack/networking-ovn: Test remove oslo_db_api.wrap_db_retry  https://review.openstack.org/22641716:48
openstackgerritAaron Rosen proposed openstack/networking-ovn: Test remove oslo_db_api.wrap_db_retry  https://review.openstack.org/22641816:48
openstackgerritAaron Rosen proposed openstack/networking-ovn: Test: normal ci run  https://review.openstack.org/22641916:48
openstackgerritAaron Rosen proposed openstack/networking-ovn: Test: normal ci run  https://review.openstack.org/22642016:48
openstackgerritAaron Rosen proposed openstack/networking-ovn: Test: normal ci run  https://review.openstack.org/22642116:49
openstackgerritAaron Rosen proposed openstack/networking-ovn: Test: normal ci run  https://review.openstack.org/22642216:49
russellbarosen: ha, parallel CI runs?  :)16:50
arosenyup, :)16:50
aroseni wanna get to the end of this SAVE POINT mysql bug16:50
russellbi want to get security groups working16:51
russellbseems to be close ... current issue seems to be an ovn issue, not plugin16:51
arosenIt seems like if I remove the retry decorator  I don't see any failures and the CI passes each time.16:52
russellbarosen: well that's frustrating16:52
arosenAt least it did https://review.openstack.org/#/c/224364/216:52
russellbseems odd to have to do the retry decorator down in our code anyway ... wish that could be hidden16:52
* russellb dives into plugin.py to try to add provider network support17:14
arosenrussellb: coolio, I just gave a quick review on your security group patch.17:24
russellbarosen: thanks!!17:25
arosenyou bet, nice job getting the security group support into ovn and the plugin :)17:26
russellbi just did the neutron side, heh17:30
russellbquite the team effort on this stuff through several layers17:30
arosenyea i'm sure. I bet it's fun to dig into all layers of it.17:31
russellbit also hurts my head17:31
russellbbut yeah :)17:31
russellbarosen: good feedback, much appreciated17:34
arosensure thing, thanks.17:34
openstackgerritAaron Rosen proposed openstack/networking-ovn: Test retry on request not deadlock  https://review.openstack.org/22649418:49
openstackgerritAaron Rosen proposed openstack/networking-ovn: Test retry on request not deadlock  https://review.openstack.org/22649518:49
openstackgerritAaron Rosen proposed openstack/networking-ovn: Test retry on request not deadlock  https://review.openstack.org/22649618:49
openstackgerritAaron Rosen proposed openstack/networking-ovn: Test retry on request not deadlock  https://review.openstack.org/22649718:49
*** jimchou has joined #openstack-neutron-ovn18:49
*** asuvvari has joined #openstack-neutron-ovn19:22
switchcaderussellb: hoping to look at the ACL conntrack issue a little more this afternoon20:15
russellbswitchcade: oh, hi!  cool, thanks20:15
russellbswitchcade: i'm sorta stuck :/20:16
switchcadeone alarm bell I see is that ingress pipeline ct(commit) flows aren't hit20:16
switchcadeit would be super helpful if we could get per-packet traces through the openflow pipeline20:16
russellbis that possible?20:16
russellbi've use the trace thing before, will that work here?20:17
switchcadeunfortunately since it's coming from the local stack, it has additional conntrack information and I'm not sure if ofproto/trace supports specifying that information or not20:17
russellbah, makes sense20:17
switchcadeI'm not super familiar with trace either.20:17
switchcadeI have a couple of directions to start though, like writing some system-traffic kmod unit tests that look at the behaviour in this case and just see what the behaviour is20:18
russellbi also didn't see anything special in between from-lport and to-lport for a packet that stayed on the same host20:18
russellbso not sure how that's supposed to work20:19
switchcadefor instance, I don't know exactly what OVS will report for a packet that comes from the local stack: Is it still "new", since this is the first packet for the connection? Or is it "established", because part of the linux stack processing involves committing the conntrack entry in zone=020:19
switchcadetwo thoughts I have are: there's a flow which does ct(zone_reg=NXM_NX_REG5). Either that is not hit for the ping traffic, or the zone is not what we expect20:21
switchcadethat's in ingress, I mean20:21
russellbdon't discount the possibility that the output is misleading20:21
switchcadecookie=0x0, duration=18.690s, table=17, n_packets=2, n_bytes=196, priority=100,ip,metadata=0x1 actions=ct(recirc,next_table=18,zone_reg=NXM_NX_REG5[])20:21
russellbbecause i'd restart ovn-controller (which had the side effect of resetting the counts in the flows)20:21
russellband then i'd ping again20:21
russellband there may have been some pre-existing state in conntract at that point20:22
switchcademight be good to do a "conntrack -F" in between testing20:22
switchcadeto flush all conntrack state20:22
russellbok, that was my next thought, "todo - look up how to clear conntrack"  heh20:22
russellbi could also probably set this up for you if that's useful20:23
switchcadeso, one thing that might help is if we could get a fresh openflow dump that only has the ping traffic through it (ideally no arp or anything else)20:23
russellbyeah, that's what i was trying to get20:23
switchcadeif we know there should only be 2 packets: forward and reply, it might be easier to step through flow-by-flow20:23
russellbhard to prevent other random crap20:23
switchcadeyeah, I noticed there was some IPv6 stuff in there20:24
switchcadeand I wasn't sure if ARP was too20:24
russellbwhat's the better way to clear the counters20:24
switchcadeif we manually add flows using ovs-ofctl, will ovn-controller override them?20:24
russellbi'm sure "restarting ovn-controller" isn't it20:24
russellbovn-controller is aggressive about ensuring the flows are what it thinks they should be20:24
switchcadeI see.20:25
switchcadeI don't think there's an explicit way to clear counters.20:25
russellbi guess restarting ovn-controller is just making it reprogram all of them then20:25
switchcadeyeah. I think that adding the existing flow will reset its count20:26
switchcadeis it possible to mock out ovn-controller at all? Say we run ovn-controller, dump the flows, then kill ovn-controller, then send the traffic20:26
switchcadewe could then manually manipulate the flows to remove uninteresting traffic20:27
switchcademaybe this is overthinking it:)20:27
russellbit's possible20:27
russellbjust stop ovn-controller20:27
switchcadehow about stateless ACLs to drop IPv6 traffic?20:27
russellband it'll stay how it is20:27
russellbi could do that,s ure20:27
switchcadeunfortunately I think these suggestions are all more of a "shotgun approach" to debugging20:28
switchcadebut if we can get a clearer picture, maybe it's what we need.20:29
russellbarp should at least be mostly separated, as all the ACL flows i have match ip4, ip6, or both20:30
russellbbut dropping all ipv6 is a good start, because i don't know what that junk is :)20:30
russellb(i know what ipv6 is, i'm not sure what the packets were)20:30
* russellb also distracted ... in an IRC meeting20:36
switchcadeI think the dump you posted previously was fairly clean, but I had a little trouble resolving the flow byte counts20:37
switchcadeseemed like some of the byte counts just suddenly entered a few tables in, so maybe the stats snapshot was not atomic20:38
switchcadeI think what you need to do is clear the flows, set up the flows, send traffic, wait 1-2 seconds, then dump the flows20:38
switchcademake sure that the openflow layer statistics are up to date20:39
russellbcould have been, i was trying to grab it quickly20:39
switchcadeit shouldn't take more than 500ms I think to sync the statistics, but better safe20:39
*** asuvvari has quit IRC20:56
*** asuvvari has joined #openstack-neutron-ovn20:56
*** asuvvari has joined #openstack-neutron-ovn20:57
switchcadein my test environment, the initial ping packet coming from local stack has zone=0,ct_state=+trk+new20:59
switchcadeslowly extending its pipeline to make it more elaborate20:59
switchcadehaven't tried zones yet, could be a bug there21:00
switchcade(although the latest zone code is different from what's present on justin's OVN ACL branch)21:00
russellbswitchcade: yeah, i'm using that branch ..21:01
russellbmeeting over yay21:01
*** asuvvari has quit IRC21:02
switchcadeI'll spend a little longer on this, but my main priority is to get the next version of the patches out so we can merge it and be sure we're not encountering bugs that were already fixed:)21:05
*** asuvvari has joined #openstack-neutron-ovn21:06
*** fzdarsky has joined #openstack-neutron-ovn21:20
russellbswitchcade: makes sense!21:26
russellbswitchcade: i could also just shelve this and wait for your patches to be merged and justin to rebase the OVN ACL code21:27
russellbsince it sounds like that's imminent anyway21:27
russellbi've got other stuff i can work on21:27
openstackgerritAaron Rosen proposed openstack/networking-ovn: Change oslo_db_api decorator to retry on request  https://review.openstack.org/22649721:42
*** salv-orlando has joined #openstack-neutron-ovn21:48
switchcaderussellb: might be a direction to take.21:51
switchcaderussellb: for what it's worth, I've boiled down the OVN pipeline into the basic pieces that interact with conntrack and it seems to work on my WIP development branch (based on the v2 series)21:51
russellbswitchcade: if you think that branch could be missing important fixes, it's not worth wasting anyone's time21:52
switchcadeFlows are here: https://gist.github.com/joestringer/ffc6154c548ba057864321:52
switchcaderussellb: I don't know of any specific related fixes, but I'm also not sure how justin did the zone and table stuff21:52
switchcadeso there's some degree of possibility that it's just a bug in the ct interface with that particular branch21:53
switchcademy test uses a network namespace in substitute for a VM, but the main point in the connection with the local stack I think21:53
switchcadesyntax is a little different there, but it may help in future.21:54
switchcadeanyways, this case works and I see three relevant entries in conntrack: One for local port, one for zone=0 and one for the namespace port21:55
russellband it seems in my case, the state is missing for a zone21:56
switchcadeso the fundamentals appear to be there at least with the latest code. I'll switch to trying to push this closer to upstream and we can revisit this issue when it gets merged21:56
* russellb nods21:56
russellbyeah, revisiting sounds fine21:56
russellbthis branch has the awesome benefit of making it easy to test without building a new kernel :)21:58
russellbhow much of a pain would it be to rebase the kernel changes once your userspace patches merge to master?21:59
russellbI can even have it running in OpenStack CI, which is cool.21:59
switchcadeI have a branch for it, minus IP fragment support22:00
switchcadebut I figured that it's so close to merging upstream that I won't push a fresh version with out-of-tree module until it gets merged22:00
switchcadewhen it goes upstream, I plan to push a version like this again with out-of-tree, but no IP fragment support, then I'll start working on the IP frag backport22:01
switchcadethen hopefully in the next few weeks we can get that merged too22:01
switchcademain thing is that there'll be a few OpenFlow interface changes and I wanted to get them solidified before I hand another build out.22:02
*** jerrygb has joined #openstack-neutron-ovn22:08
*** markmcclain has quit IRC22:08
*** markmcclain has joined #openstack-neutron-ovn22:16
russellbswitchcade: sure, makes sense.  thanks for all of your hard work.  this is great stuff :)22:26
*** asuvvari has quit IRC22:27
*** asuvvari has joined #openstack-neutron-ovn22:28
*** asuvvari has quit IRC22:32
russellbswitchcade: also thanks for hanging out in here :)22:33
*** jimchou has quit IRC22:33
*** fzdarsky has quit IRC22:34
*** fzdarsky has joined #openstack-neutron-ovn22:37
*** asuvvari has joined #openstack-neutron-ovn22:41
*** regXboi has quit IRC22:45
*** asuvvari has quit IRC22:46
*** shettyg has quit IRC22:48
switchcaderussellb: always happy to lurk ;)23:01
*** fzdarsky has quit IRC23:08
*** arosen has quit IRC23:21
*** salv-orlando has quit IRC23:25

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!