Friday, 2015-05-01

*** blp has quit IRC00:14
*** armax has joined #openstack-neutron-ovn01:22
*** russellb changes topic to " -=- OVN meeting Thursdays 1700 UTC #openvswitch"13:26
russellbdocs now published13:27
russellbcould use some more love13:27
mesteryrussellb: Nice!13:37
mesteryrussellb: Back to trying on fedora this morning, was knocked out by this nasty cold I've had for a week13:37
russellbmestery: i'm with you13:37
russellbhad a cold for 2 weeks now, saw dr yesterday though13:37
russellbgot antibiotic ... developed a sinus infection13:37
russellbfun times.13:37
russellband by fun i mean "fun"13:37
russellbmestery: recreating the setup is "working" for me, but I don't think the network is working like I think it is13:38
russellbit might be an illusion13:38
russellbbut in any case, I'm not getting nova errors13:38
mesteryoh man13:38
mesteryI'm likely to go into the doctor in a bit too13:38
mesteryTired of being sick :(13:38
russellbmestery: being a PTL will do that :-p13:39
mesterylol, no kidding, and coupled with 4 kids ...13:40
mesteryneed to get healthy before my insane 2 weeks of travel coming up too13:40
openstackgerritMerged stackforge/networking-ovn: docs: Remove content from usage.rst
russellbtoday was my kid's last day of day care13:41
russellbi'm hoping that cuts down on how much crap he brings home, for his sake and mine :)13:41
openstackgerritMerged stackforge/networking-ovn: devstack: Fix git repo name assumptions.
openstackgerritMerged stackforge/networking-ovn: devstack: Give example of custom ovs git repo.
mesteryI remember that when my daughter (first kid) stopped going to daycare13:41
mesteryIt helped a lot13:41
openstackgerritRussell Bryant proposed stackforge/networking-ovn: docs: Add testing howto doc.
openstackgerritRussell Bryant proposed stackforge/networking-ovn: Move TODO out of published docs.
openstackgerritRussell Bryant proposed stackforge/networking-ovn: docs: Add testing howto doc.
russellbmestery: for your ACK at some point :-)
*** openstackgerrit has quit IRC13:51
mesteryrussellb: Nice! I saw Octavia wants to do the same, so cool!13:51
russellbmakes sense13:51
*** openstackgerrit has joined #openstack-neutron-ovn13:51
russellbotherwiseguy: didn't want to +2 ?14:23
otherwiseguyHad a comment on patch 5, but doesn't really matter one way or the other. Can always just do another patch.14:24
russellboh sorry14:25
otherwiseguyrussellb: no need. it was really just a very small 'would it be more consistent if we...' kind of thing.14:27
*** shettyg has joined #openstack-neutron-ovn14:43
openstackgerritRussell Bryant proposed stackforge/networking-ovn: devstack: Update integration bridge config
*** gsagie_ has joined #openstack-neutron-ovn14:58
gsagie_russellb : any help needed in the multinode devstack?15:01
openstackgerritRussell Bryant proposed stackforge/networking-ovn: Drop version= from setup.cfg.
gsagie_nice job on the docs15:03
russellbre: multi-node, feel free to help if you'd like15:04
russellbthere's not much code needed15:04
russellbit's a bit of code and a lot more testing and docs work15:04
gsagie_np, was just wondering if you got to it, if not i can start on sunday, if you did i can start testing and see if anything needs a fix15:05
russellbthe code needed is to make it possible to configure what services get run locally, only one node should run ovn-northd15:05
russellband on that node, ovsdb-server needs to serve the additional DBs15:05
russellband the other nodes need to know where ovn-northd is running15:05
russellbthat kind of config flexibility15:05
russellbgsagie_: whatever i get done i'll get up on gerrit so you can see what i have15:07
gsagie_ok cool, will continue from there15:08
russellbi need to do some more work to better verify that the OVN datapath is working as expected15:08
russellband document how to check it15:08
russellbgsagie_: you +2'd this patch before, can you re-add?
gsagie_reading changes15:14
otherwiseguyrussellb: the threads I create in that patch are only created after the up event is triggered, then it creates the thread just to run the notification which quickly exits (unless there is some kind of deadlock or something).15:33
otherwiseguy(with that said, I started working on the thread/queue implementation) :p15:33
russellbotherwiseguy: oh nevermind.15:33
russellbwhatever you think15:34
otherwiseguyrussellb: playing with threads while cloud-init builds fail. :p15:34
* mestery back from doctor and trying OVN on fedora again15:44
gsagie_russellb : i think you noticed yesterday that ovn-northd didn't update the port status ?15:47
gsagie_if there is a bug in OVN maybe we should hold with otherwiseguy patch, as the port will never get updated to up in nova15:48
otherwiseguygsagie_: would be fine with me as I'm playing around with threading stuff on it.15:48
otherwiseguySomeone can just click 'rebase' :p15:48
openstackgerritGal Sagie proposed stackforge/networking-ovn: Reduce the number of OVSDB transactions
gsagie_yeah, just think i saw him says something about it yesterday but maybe it works, didn't check it yet myself15:50
gsagie_otherwiseguy, russell the proposed patch with otherwiseguy comment addressed ;)15:51
otherwiseguygsagie_: didn't we talk about you working too much? :p15:51
russellbgsagie_: yeah, in my first devstack install, the ports weren't being set to up, i need to debug it15:56
russellbinstead i re-created my env and documented it along the way15:56
openstackgerritTerry Wilson proposed stackforge/networking-ovn: Add port 'up' notification
otherwiseguydoh, that worked, but I can see it is broken.15:59
openstackgerritTerry Wilson proposed stackforge/networking-ovn: Add port 'up' notification
otherwiseguyBTW, if anyone needs a quick script to play around with creating an ovs port to trigger an 'up' notification based on a neutron port-create $net --name $name: e.g. neutron port-create private myport1 && ./ myport1.16:17
openstackgerritGal Sagie proposed stackforge/networking-ovn: Reduce the number of OVSDB transactions
*** gsagie_ has quit IRC16:37
*** ajo has joined #openstack-neutron-ovn16:45
*** markmcclain has quit IRC16:49
*** HenryG is now known as floccinaucinihil16:54
*** floccinaucinihil is now known as HenryThe8th16:55
*** markmcclain has joined #openstack-neutron-ovn17:03
*** blp has joined #openstack-neutron-ovn17:32
russellbmestery: think you could look over my devstack testing guide?  want to get it published so I can share.  :-)
mesteryrussellb: Ack! I'll do that now.17:46
mesteryrussellb: I'm still getting odd nova errors in my setup which I can't get past :(17:46
russellbmestery: odd, but doesn't sound ovn related :-/17:46
mesteryNope, which is all the more frustrating :(17:46
russellbmestery: if you want to give me access i can dig into it though17:46
*** ajo has quit IRC17:47
mesteryrussellb: Cool, thanks! Give me a few more minutes, I'm trying one more thing17:47
mesteryIn the meantime, I'm looking at your reviews now17:47
russellbi rebuilt my env and it worked again *shrug*17:47
russellbwell, "worked"17:47
russellbbehaved the same way as before anyway17:47
mesteryGood datapoint!17:48
openstackgerritMerged stackforge/networking-ovn: Move TODO out of published docs.
*** shettyg has quit IRC18:03
*** ajo has joined #openstack-neutron-ovn18:05
*** shettyg has joined #openstack-neutron-ovn18:06
mesteryrussellb: So, even before launching a VM, I get this error in my n-cpu log:18:09
mestery2015-05-01 17:54:41.857 ERROR nova.compute.manager [req-35c4e807-46b1-40de-9b29-b9a17e327ec2 None None] No compute node record for host ovn18:09
mesteryrussellb: I suspect that's the root of my problem18:09
russellbthat's the error you were getting on boot too18:09
russellbi'll dig into it, got sidetracked by ovn bug i noticed18:09
mesteryno worries18:10
mesteryI'm getting pulled in 3 directions ATM as well18:10
*** shettyg has quit IRC18:11
russellbsounds about right18:12
russellbwell, fixed an ovn bug making ovn-controller die in my test env18:23
russellbblp: this killed ovn-controller :)
russellbeven though apparently i can't spell in my commit message ... and typoed the config variable name18:25
russellbblp: and I think I figured out why things "worked" before fixes ... if the above killed ovn-controller right away, and I also hadn't updated our code to configure br-int with fail-mode=secure, I think everything on the switch was just happily talking to each other with the default switch config, heh18:32
mesteryrussellb: Getting further with my setup18:33
mesteryNow seeing this: tError: Unable to add port tap0f222df0-7a to OVS bridge br-int: Operation not permitted18:33
mesterylibvirtError: Unable to add port tap0f222df0-7a to OVS bridge br-int: Operation not permitted18:33
russellbprogress indeed18:33
mesteryoh no18:35
mesteryis it apparmor?18:35
mesteryMay  1 18:16:48 localhost kernel: [251495.793655] type=1400 audit(1430504208.416:47): apparmor="DENIED" operation="exec" profile="/usr/sbin/libvirtd" name="/usr/local/bin/ovs-vsctl" pid=8661 comm="libvirtd" requested_mask="x" denied_mask="x" fsuid=0 ouid=018:35
* mestery cries a little18:35
russellbsecurity FTMFL!18:36
* mestery sheds tears of uncertainty 18:36
* mestery boots a VM with APPARMOR TURNED OFF18:36
mesteryit still failed18:37
* mestery reboots his devstack VM18:41
*** ajo has quit IRC18:44
* mestery burned apparmor to the ground18:47
*** shettyg has joined #openstack-neutron-ovn19:02
openstackgerritRussell Bryant proposed stackforge/networking-ovn: devstack: Run ovn-controller with sudo.
russellb2015-05-01T19:21:59.096Z|00015|pipeline|INFO|Dropped 59 log messages in last 60 seconds (most recently, 1 seconds ago) due to excessive rate19:22
mesteryafter burning apparmor to the ground ... MY VMS BOOT!19:24
mesteryand my pings work!19:25
mesteryrussellb: ^^^^^19:25
* mestery dances the happy dance19:25
mesteryHowever, when I dump the flows on br-int, I only see the NORMAL action19:29
mesterywhich explains why it's working19:29
mesteryBut likely not the right thing19:29
russellbmestery: yeah so i was about to break some news to you haha19:29
mesteryubuntu@ovn:~/devstack$ sudo ovs-ofctl dump-flows br-int19:29
mesteryNXST_FLOW reply (xid=0x4):19:29
mestery cookie=0x0, duration=906.883s, table=0, n_packets=352, n_bytes=40363, idle_age=1, priority=0 actions=NORMAL19:29
russellbi just recently put the pieces together that i think our pings are full of lies19:29
russellbgood news is blp really did make things work19:29
russellbbut, issue #1:
russellbissue #2 is the bug for the patch you acked on ovs-dev, that made ovn-controller exit right away19:30
russellbovn-controller never set anything up, so, our VMs are just on ovs in normal mode, haha19:31
mesterywell ... HAPPY FRIDAY! :)19:31
russellbthey totally were able to ping each other though19:31
russellbbut don't look behind the curtain19:32
russellbanyway, like i said, the good news is blp made only a couple of minor fixes and it really does work with latest code :)19:32
mesteryCool! Did you pull the latest code into your repo yet? That's where I had my local.conf pointed19:33
mesteryAlso, we need otherwiseguy to merge your change which I +2'd19:33
russellbovn repo?19:33
russellblet's see19:34
russellbbasically we need the one fix on top of blp's ovn branch19:34
* russellb does that19:34
* otherwiseguy reads history19:34
russellb ovn branch has it now19:35
russellbwhich is just ovn branch plus my 1 fix19:35
otherwiseguy+2 givified19:36
mesterycool russellb!19:36
russellband then you need some ovn-networking fixes19:36
russellball on gerrit19:36
russellboh the joys of stuff under this rapid of dev, heh19:36
russellbneed to run for now, will talk to you all later19:38
*** HenryThe8th has quit IRC19:50
*** ajo has joined #openstack-neutron-ovn19:50
*** ajo has quit IRC19:52
*** ajo has joined #openstack-neutron-ovn19:52
*** HenryG has joined #openstack-neutron-ovn19:54
*** HenryG has quit IRC19:55
*** HenryG has joined #openstack-neutron-ovn19:56
openstackgerritMerged stackforge/networking-ovn: devstack: Update integration bridge config
openstackgerritMerged stackforge/networking-ovn: devstack: Run ovn-controller with sudo.
*** HenryG has quit IRC20:16
openstackgerritKyle Mestery proposed stackforge/networking-ovn: setup.cfg: Remove version
*** HenryG has joined #openstack-neutron-ovn20:21
otherwiseguyrussellb: looks like your global-requirements patch is again out of sync. Crazy.20:31
otherwiseguyrussellb: the suggestion for a minimum version may not be horrible.20:34
otherwiseguymestery: I had to look up what #FF meant. I'm so old.20:36
mesteryopenstackgerrit: lol20:36
mesteryotherwiseguy: lol20:36
mesterythought openstackgerrit, also lol20:36
otherwiseguyI think I *finally* have gotten disk-image-builder to build me a fedora image with a custom cloud-init package based off the latest commit which *looks* like it has ipv6 support so I can finally test it.20:40
otherwiseguyThat dog is better dressed than I ever am.20:42
russellbotherwiseguy: so ... did you see that there's a new version of cirros available (maybe just for testing and not released) that's supposed to have some ipv6 stuff in it?20:44
otherwiseguyrussellb: I did not. I gave up on it when I noticed there was no ping6 command. Figures.20:44
otherwiseguyI notice that launchpad has the latest release of cloud-init at 0.7.6 but ubuntu has a 0.7.7 package built.20:45
russellbotherwiseguy: i know nothing beyond the existence of that ML post20:45
otherwiseguyOne of these ays I'm going to start reading my email.20:46
otherwiseguyand days20:46
russellbit doesn't actually say anything about cloud-init20:46
russellbso *shrug*20:46
otherwiseguyrussellb: I guess at least I know what patches need to be modified for the fedora cloud-init packagers to update to the latest version. :p20:47
mesteryrussellb: Even with your new OVS repo, I still don't see any flows on br-int20:56
russellbmestery: welp.20:59
mesteryrussellb: It's almost quitting time here, the drugs are making me sleepy. Cold medicine and all :)20:59
russellbmestery: yeah, it has been that kinda week..20:59
russellbmestery: we'll get things all sorted and more predictable next week :)20:59
russellbif you feel like looking any more, i'd say 1) is ovn-controller running? and 2) check /usr/local/var/log/openvswitch/ovn-controller.log21:00
russellbfor anything interesting21:00
russellborrrrr we'll figure it out next week21:01
russellbeither is fine with me21:01
mesteryovn-controller not running :(21:02
mesteryMay  1 20:23:20 localhost kernel: [ 5218.260438] ovn-controller[32242]: segfault at 150 ip 000000000040818a sp 00007ffd2fd52150 error 4 in ovn-cont21:03
mesteryThat explains it :)21:03
russellbooooh, fun21:03
russellbi wonder if my patch is broken :-/21:03
russellbmust ... put ... code ... down21:04
mesteryIs your patch on your ovn branch from your github?21:05
mesteryThe last one I see is from blp21:05
russellbit's supposed to be21:05
russellbbut no21:05
russellbit is not.21:05
mesterywell, that was it!21:05
russellbwhat the heck21:07
russellbfixed ...21:08
russellbi made a n00b git mistake21:09
*** ajo has quit IRC21:09
russellbthe old branch was definitely broken21:10
russellbmestery: ^21:10
russellbok need to call it a day/week i think21:11
mesteryrussellb: later and thanks!21:12
*** ajo has joined #openstack-neutron-ovn21:15
*** ajo has quit IRC21:18
*** shettyg has quit IRC22:03

Generated by 2.14.0 by Marius Gedminas - find it at!