Wednesday, 2015-05-27

*** hitalia has quit IRC00:18
*** armax has quit IRC03:54
*** smallbig has quit IRC06:50
*** smallbig has joined #openstack-neutron-ovn06:50
gsagierunning latest code, i am getting the following error on the ovn-controller:07:53
gsagie2015-05-27T07:48:22Z|00008|pipeline|INFO|logical flow for unknown logical datapath 629d4b80-7c4e-492a-8807-09cfc486b168 ovn-controller failed to start07:53
gsagieanyone noticed that too?07:54
*** ezobn has joined #openstack-neutron-ovn07:58
*** gsagie has quit IRC08:02
*** gsagie has joined #openstack-neutron-ovn08:17
gsagiei took the ovn branch from Ben's repository and its working09:20
*** roeyc has joined #openstack-neutron-ovn09:33
gsagieohhh... roeyc !09:42
gsagieYou are back09:42
roeycyea :)09:46
*** roeyc has quit IRC10:05
*** roeyc has joined #openstack-neutron-ovn11:03
*** openstack has joined #openstack-neutron-ovn11:37
*** roeyc has quit IRC12:37
*** roeyc has joined #openstack-neutron-ovn12:41
*** ajo has joined #openstack-neutron-ovn13:46
*** mestery has quit IRC14:33
*** mestery has joined #openstack-neutron-ovn14:34
*** shettyg has joined #openstack-neutron-ovn14:52
*** armax has joined #openstack-neutron-ovn15:01
russellbgsagie: that error message shouldn't cause ovn-controller to not start15:02
russellbgsagie: i'll try running the latest code though15:03
russellbgsagie: we do get that error message a lot, i think it should probably be removed15:04
russellbgsagie: for example,
*** roeyc has quit IRC15:15
*** roeyc has joined #openstack-neutron-ovn15:17
*** gsagie has quit IRC15:26
*** wznoinsk_ has joined #openstack-neutron-ovn15:47
*** wznoinsk_ is now known as wznoinsk15:48
*** ezobn has quit IRC16:18
*** ezobn has joined #openstack-neutron-ovn16:19
*** ezobn has quit IRC16:23
otherwiseguyrussellb: others: So during summit, I was thinking about the OVN port up notification. I think I currently prefer the idea of having a 'leader' defined via config, with only the leader handling notifications. On startup, the leader can then also safely handle sending notifications for neutorn ports marked down that are actually up. Tools like pacemaker could be used start 'clusters' of plugins, ensuring that16:29
otherwiseguy one and only one was the 'leader'. If at some point active-active distributed ovsdb becomes a thing, then revisit. Thoughts?16:29
russellbotherwiseguy: will that work when the typical case is that a single host actually spawns N API workers?16:38
otherwiseguyrussellb: it just spawns API workers and not plugin-side though, right?16:38
otherwiseguyOr is that wrong?16:38
russellbplugin is in the API worker though16:38
russellbit's just multiple processes16:38
russellbi'm tempted to just execute a cleanup /sanity check if the length of the list of tracked ports gets to some large number16:39
russellbsigh i don't know16:40
russellbthere's benefits to your approach too16:40
russellbit feels like something is just fundamentally wrong here16:42
russellbwe need to take a message that's broadcasted and ensure it gets processed once (ideally exactly once, but definitely at least once)16:43
russellband somehow do it in a way that's horizontally scalable too16:44
otherwiseguyrussellb: right. and there are ways to do that. it's just that they are hard. :) And also tend to be expensive.16:44
* russellb just expressing the problem out loud again while i think16:44
* otherwiseguy has about 10 papers open describing such methods16:45
russellbtempting to write a worker that sits next to ovsdb-server and just converts port updates to oslo.messaging messages16:45
russellbso that we actually have a queue all of the neutron-server processes can pull from16:45
otherwiseguyrussellb: yeah, that is definitely the easiest thing I've thought of.16:45
russellbnot thrilled that it requires a new service16:45
russellbbut at least it gets things into the right pattern16:46
russellband like you said, ovsdb-server isn't distributed today anyway16:46
russellbso it's not adding a new single point16:46
otherwiseguyBut it does miss out on the case where the notifier service dies for a bit and comes back up.16:47
otherwiseguyor servers missing notifications, etc.16:47
russellbso it'd have to do a sync of port state on startup16:47
otherwiseguyyou still need that global state that says "we are on item N together"16:48
russellbyes, a server could pull a message from the queue and then fail to process it16:48
otherwiseguyOh, I'm on item H. Give me H-N" :p16:48
russellbanother option would be to favor correctness over scale for now ... have every API worker process every port update, and just ensure that an update when the state is already set is effectively a no-op16:49
russellb(oh, already set to up, moving on then)16:49
otherwiseguyThe question kind of comes to what's 'good enough'. Of course, there are lots of times when something is built on "good enough" then it isn't, and fixing it becomes nearly impossible.16:49
* russellb nods16:50
russellblast resort is saying screw it, and leaving it how it is16:50
russellbi dunno ... i need to get some lunch16:50
russellbwill keep thinking16:50
otherwiseguyrussellb: and then there is the case where a tool outside of us modifies a port... :)16:51
otherwiseguyenjoy the food.16:51
*** roeyc has quit IRC17:02
*** hitalia has joined #openstack-neutron-ovn17:56
russellbotherwiseguy: i did enjoy the food, and i came up with no further useful insight :)18:12
russellbotherwiseguy: on an unrelated note, there's some ovs-dev discussion about what to do with Python 3 support18:12
russellbotherwiseguy: in case you want to weigh in with an opinion18:12
switchcadeby the way, is python2.4 that incompatible with python3?18:30
russellbgenerally, yes18:30
russellbit's a little more complicated than that, of course :)18:30
switchcadeI wouldn't expect that we use anything particularly fancy from python18:30
switchcadebut if the effort for making the scripts work on both is too high, then I can see the reservation18:31
russellbPython 3 is not backwards compat with Python 2, and you generally need things added in later versions of Python 2 (2.7) to stay sane18:31
switchcadeI see.18:31
russellbas they started doing things to make the transition a little easier18:31
switchcadehmm, I may have broken the extract-ofp-fields script for python2.4 in that case..18:32
russellbit's really easy to do18:32
russellb(break 2.4 that is)18:32
russellbPython 2.4 is over 10 years old now  ...18:33
switchcadeyeah, it really needs to die.18:33
russellb+1 :)18:34
switchcadejust searching around, having trouble trying to find a box with it still on there to test18:34
russellbyeah, seems like it's just the xen thing18:34
russellbmaybe CentOS 5?18:34
switchcadeI mean, even Debian hasn't had it for the past two releases18:34
russellbright, even RHEL hasn't had it since 2 releases ago18:35
russellb(RHEL 5, and its derivatives such as CentOS 5)18:35
russellbbooting CentOS 5 is probably easiest way to get it18:35
*** blp has joined #openstack-neutron-ovn18:54
otherwiseguyYeah, the except Exception, e: vs except Exception as e: change is pretty annoying to work around.19:08
otherwiseguyrussellb: Looks like the way to support 2.4 and 3 syntax is except Exception: e = sys.exc_info()[1]. Which is ugly, but maybe we just see if we can make everything happy in the same codebase?19:35
russellbotherwiseguy: yeah, i worry about what else we'll hit, but probably worth it for now19:36
russellbotherwiseguy: i'd really rather avoid having to carry a patch set outside of master19:36
otherwiseguyTesting it is going to be the biggest pita.19:36
* otherwiseguy really doesn't want to install Centos 5 :p19:37
russellbyeah, i don't know how else to test it19:37
russellbluckily we have this cloud thing that should make it easy enough to boot centos 5 ...19:38
russellbassuming there's an image, heh19:38
russellbi assume too much19:38
blpI really hope that Andy gets back and says that installing 2.7 on XenServer is easy.19:38
blpSupporting 2.4 is a pain and it's been that way for a while.19:39
russellbblp: supporting it this long has been very generous :)19:39
otherwiseguyrussellb: blp: and the patch I proposed doesn't really add python 3 support so much as just fix warnings thrown when install is called. still *lots* of work to do for actual python 3 support.19:44
russellbotherwiseguy: the first step in that direction19:45
otherwiseguyso I can certainly see not wanting to break an existing user for just fake python 3 support. :)19:45
russellbright, more patches to come, just trying to figure out what the workflow should be for them19:46
otherwiseguyBut I am committed to actually doing full support. :)19:46
russellbright, we kinda have to19:46
blpHow bad is it to support both Python 2.7 and Python 3?19:47
blpStill painful, or OK?19:48
otherwiseguyAnd I'd like to fix the "built-in json parser is about 60x slower than it should be" issue at some point to (either just use C bindings to existing ovs json parser, or use something like yajl).19:48
otherwiseguyblp: not too bad. there are libraries like 'six' to make it easier to share a single codebase.19:48
russellbyeah, i think 2.7 and 3.4 at the same time is pretty common19:48
otherwiseguyin fact, single source tree seems to be the preferred method of supporting multiple python versions from what I've seen.19:48
blpThe JSON parser is my fault.19:49
russellbblp: don't worry, you have enough karma :)19:49
blpIt was clear that the built-in Python JSON parser wouldn't work, so I rewrote the C version in Python very quickly.19:49
otherwiseguyblp: it works. :) It's just very close to a line-by-line port of the C one. Which is totally valid thing to do, probably makes testing easier, etc.19:50
otherwiseguyI just noticed when profiling that pretty much all of the time was actually spent in the parser.19:50
russellbblp: neat, ovn-controller seg faults on startup in latest ovn branch19:51
otherwiseguyblp: I don't think any human could sanely write a great from-scratch streaming json parser in the time frame you had available. ;)19:51
russellbah, conversion of chassis column did it19:52
* russellb gives play by play of debug and fixing :-p19:53
blpotherwiseguy: Most of the Python code is a line-by-line port of the C version, because I wrote most of it in an afternoon ;-)19:55
otherwiseguy"russellb is at the prompt. he's running the code. Oooooh, SyntaxError was thrown. That's gotta hurt!"19:55
blpTwo-minute penalty for typo!19:55
blp"He's interrupted by IRC. Now his officemate makes a joke!"19:56
* otherwiseguy assigns the announcer a two-minute penalty for using thrown vs raised in a pedantic fit19:56
blp"Oops, someone just asked him a question on the conference call he's mostly ignoring. Will he be able to field it?"19:56
russellbpretty much sounds like my day19:56
russellband he submits a patch!19:59
*** mestery_ has joined #openstack-neutron-ovn20:00
blpIt's too bad we can't just do "#define as ,"20:00
russellbit's too bad the move to Python 3 is a giant pain20:00
russellbbasically, we should have written OpenStack in C20:01
russellbwould cut down on contributors, but probably in a good way20:02
russellbblp: check out this fun log file from the functional test CI job:
*** mestery has quit IRC20:03
russellbseems we hit that condition a lot :)20:03
russellbof course, this job is also completely failing, haven't had time to dig yet ...20:04
blprussellb: Hmm.  That's a lot of unknown datapaths.20:55
*** mestery_ is now known as mestery21:12
*** hitalia has quit IRC23:40

Generated by 2.14.0 by Marius Gedminas - find it at!