| *** hitalia has quit IRC | 00:18 | |
| *** armax has quit IRC | 03:54 | |
| *** smallbig has quit IRC | 06:50 | |
| *** smallbig has joined #openstack-neutron-ovn | 06:50 | |
| gsagie | running latest code, i am getting the following error on the ovn-controller: | 07:53 |
|---|---|---|
| gsagie | 2015-05-27T07:48:22Z|00008|pipeline|INFO|logical flow for unknown logical datapath 629d4b80-7c4e-492a-8807-09cfc486b168 ovn-controller failed to start | 07:53 |
| gsagie | anyone noticed that too? | 07:54 |
| *** ezobn has joined #openstack-neutron-ovn | 07:58 | |
| *** gsagie has quit IRC | 08:02 | |
| *** gsagie has joined #openstack-neutron-ovn | 08:17 | |
| gsagie | i took the ovn branch from Ben's repository and its working | 09:20 |
| *** roeyc has joined #openstack-neutron-ovn | 09:33 | |
| gsagie | ohhh... roeyc ! | 09:42 |
| gsagie | You are back | 09:42 |
| roeyc | yea :) | 09:46 |
| *** roeyc has quit IRC | 10:05 | |
| *** roeyc has joined #openstack-neutron-ovn | 11:03 | |
| *** openstack has joined #openstack-neutron-ovn | 11:37 | |
| *** roeyc has quit IRC | 12:37 | |
| *** roeyc has joined #openstack-neutron-ovn | 12:41 | |
| *** ajo has joined #openstack-neutron-ovn | 13:46 | |
| *** mestery has quit IRC | 14:33 | |
| *** mestery has joined #openstack-neutron-ovn | 14:34 | |
| *** shettyg has joined #openstack-neutron-ovn | 14:52 | |
| *** armax has joined #openstack-neutron-ovn | 15:01 | |
| russellb | gsagie: that error message shouldn't cause ovn-controller to not start | 15:02 |
| russellb | gsagie: i'll try running the latest code though | 15:03 |
| russellb | gsagie: we do get that error message a lot, i think it should probably be removed | 15:04 |
| russellb | gsagie: for example, http://logs.openstack.org/26/178826/7/check/check-tempest-dsvm-networking-ovn/966525f/logs/screen-ovn-controller.txt.gz | 15:04 |
| *** roeyc has quit IRC | 15:15 | |
| *** roeyc has joined #openstack-neutron-ovn | 15:17 | |
| *** gsagie has quit IRC | 15:26 | |
| *** wznoinsk_ has joined #openstack-neutron-ovn | 15:47 | |
| *** wznoinsk_ is now known as wznoinsk | 15:48 | |
| *** ezobn has quit IRC | 16:18 | |
| *** ezobn has joined #openstack-neutron-ovn | 16:19 | |
| *** ezobn has quit IRC | 16:23 | |
| otherwiseguy | russellb: others: So during summit, I was thinking about the OVN port up notification. I think I currently prefer the idea of having a 'leader' defined via config, with only the leader handling notifications. On startup, the leader can then also safely handle sending notifications for neutorn ports marked down that are actually up. Tools like pacemaker could be used start 'clusters' of plugins, ensuring that | 16:29 |
| otherwiseguy | one and only one was the 'leader'. If at some point active-active distributed ovsdb becomes a thing, then revisit. Thoughts? | 16:29 |
| russellb | otherwiseguy: will that work when the typical case is that a single host actually spawns N API workers? | 16:38 |
| otherwiseguy | russellb: it just spawns API workers and not plugin-side though, right? | 16:38 |
| otherwiseguy | Or is that wrong? | 16:38 |
| russellb | plugin is in the API worker though | 16:38 |
| russellb | it's just multiple processes | 16:38 |
| russellb | i'm tempted to just execute a cleanup /sanity check if the length of the list of tracked ports gets to some large number | 16:39 |
| russellb | sigh i don't know | 16:40 |
| russellb | there's benefits to your approach too | 16:40 |
| russellb | it feels like something is just fundamentally wrong here | 16:42 |
| russellb | we need to take a message that's broadcasted and ensure it gets processed once (ideally exactly once, but definitely at least once) | 16:43 |
| russellb | and somehow do it in a way that's horizontally scalable too | 16:44 |
| otherwiseguy | russellb: right. and there are ways to do that. it's just that they are hard. :) And also tend to be expensive. | 16:44 |
| russellb | yeah. | 16:44 |
| * russellb just expressing the problem out loud again while i think | 16:44 | |
| * otherwiseguy has about 10 papers open describing such methods | 16:45 | |
| russellb | tempting to write a worker that sits next to ovsdb-server and just converts port updates to oslo.messaging messages | 16:45 |
| russellb | so that we actually have a queue all of the neutron-server processes can pull from | 16:45 |
| otherwiseguy | russellb: yeah, that is definitely the easiest thing I've thought of. | 16:45 |
| russellb | not thrilled that it requires a new service | 16:45 |
| russellb | but at least it gets things into the right pattern | 16:46 |
| russellb | and like you said, ovsdb-server isn't distributed today anyway | 16:46 |
| russellb | so it's not adding a new single point | 16:46 |
| otherwiseguy | But it does miss out on the case where the notifier service dies for a bit and comes back up. | 16:47 |
| russellb | true. | 16:47 |
| otherwiseguy | or servers missing notifications, etc. | 16:47 |
| russellb | so it'd have to do a sync of port state on startup | 16:47 |
| otherwiseguy | you still need that global state that says "we are on item N together" | 16:48 |
| russellb | yes, a server could pull a message from the queue and then fail to process it | 16:48 |
| otherwiseguy | Oh, I'm on item H. Give me H-N" :p | 16:48 |
| russellb | another option would be to favor correctness over scale for now ... have every API worker process every port update, and just ensure that an update when the state is already set is effectively a no-op | 16:49 |
| russellb | (oh, already set to up, moving on then) | 16:49 |
| otherwiseguy | The question kind of comes to what's 'good enough'. Of course, there are lots of times when something is built on "good enough" then it isn't, and fixing it becomes nearly impossible. | 16:49 |
| * russellb nods | 16:50 | |
| russellb | last resort is saying screw it, and leaving it how it is | 16:50 |
| russellb | i dunno ... i need to get some lunch | 16:50 |
| russellb | will keep thinking | 16:50 |
| otherwiseguy | russellb: and then there is the case where a tool outside of us modifies a port... :) | 16:51 |
| otherwiseguy | enjoy the food. | 16:51 |
| *** roeyc has quit IRC | 17:02 | |
| *** hitalia has joined #openstack-neutron-ovn | 17:56 | |
| russellb | otherwiseguy: i did enjoy the food, and i came up with no further useful insight :) | 18:12 |
| russellb | otherwiseguy: on an unrelated note, there's some ovs-dev discussion about what to do with Python 3 support | 18:12 |
| russellb | otherwiseguy: in case you want to weigh in with an opinion | 18:12 |
| switchcade | by the way, is python2.4 that incompatible with python3? | 18:30 |
| russellb | generally, yes | 18:30 |
| russellb | it's a little more complicated than that, of course :) | 18:30 |
| switchcade | I wouldn't expect that we use anything particularly fancy from python | 18:30 |
| switchcade | but if the effort for making the scripts work on both is too high, then I can see the reservation | 18:31 |
| russellb | Python 3 is not backwards compat with Python 2, and you generally need things added in later versions of Python 2 (2.7) to stay sane | 18:31 |
| switchcade | I see. | 18:31 |
| russellb | as they started doing things to make the transition a little easier | 18:31 |
| switchcade | hmm, I may have broken the extract-ofp-fields script for python2.4 in that case.. | 18:32 |
| russellb | it's really easy to do | 18:32 |
| russellb | (break 2.4 that is) | 18:32 |
| switchcade | :/ | 18:32 |
| russellb | Python 2.4 is over 10 years old now ... | 18:33 |
| switchcade | yeah, it really needs to die. | 18:33 |
| russellb | +1 :) | 18:34 |
| switchcade | just searching around, having trouble trying to find a box with it still on there to test | 18:34 |
| russellb | yeah, seems like it's just the xen thing | 18:34 |
| russellb | maybe CentOS 5? | 18:34 |
| switchcade | I mean, even Debian hasn't had it for the past two releases | 18:34 |
| russellb | right, even RHEL hasn't had it since 2 releases ago | 18:35 |
| russellb | (RHEL 5, and its derivatives such as CentOS 5) | 18:35 |
| russellb | booting CentOS 5 is probably easiest way to get it | 18:35 |
| *** blp has joined #openstack-neutron-ovn | 18:54 | |
| otherwiseguy | Yeah, the except Exception, e: vs except Exception as e: change is pretty annoying to work around. | 19:08 |
| otherwiseguy | russellb: Looks like the way to support 2.4 and 3 syntax is except Exception: e = sys.exc_info()[1]. Which is ugly, but maybe we just see if we can make everything happy in the same codebase? | 19:35 |
| otherwiseguy | http://python3porting.com/differences.html#except | 19:36 |
| russellb | otherwiseguy: yeah, i worry about what else we'll hit, but probably worth it for now | 19:36 |
| russellb | otherwiseguy: i'd really rather avoid having to carry a patch set outside of master | 19:36 |
| otherwiseguy | Testing it is going to be the biggest pita. | 19:36 |
| * otherwiseguy really doesn't want to install Centos 5 :p | 19:37 | |
| russellb | yeah, i don't know how else to test it | 19:37 |
| russellb | luckily we have this cloud thing that should make it easy enough to boot centos 5 ... | 19:38 |
| russellb | assuming there's an image, heh | 19:38 |
| russellb | i assume too much | 19:38 |
| blp | I really hope that Andy gets back and says that installing 2.7 on XenServer is easy. | 19:38 |
| blp | Supporting 2.4 is a pain and it's been that way for a while. | 19:39 |
| russellb | blp: supporting it this long has been very generous :) | 19:39 |
| otherwiseguy | russellb: blp: and the patch I proposed doesn't really add python 3 support so much as just fix warnings thrown when setup.py install is called. still *lots* of work to do for actual python 3 support. | 19:44 |
| russellb | otherwiseguy: the first step in that direction | 19:45 |
| otherwiseguy | so I can certainly see not wanting to break an existing user for just fake python 3 support. :) | 19:45 |
| russellb | right, more patches to come, just trying to figure out what the workflow should be for them | 19:46 |
| otherwiseguy | But I am committed to actually doing full support. :) | 19:46 |
| russellb | right, we kinda have to | 19:46 |
| otherwiseguy | Yeah. | 19:46 |
| blp | How bad is it to support both Python 2.7 and Python 3? | 19:47 |
| blp | Still painful, or OK? | 19:48 |
| otherwiseguy | And I'd like to fix the "built-in json parser is about 60x slower than it should be" issue at some point to (either just use C bindings to existing ovs json parser, or use something like yajl). | 19:48 |
| otherwiseguy | blp: not too bad. there are libraries like 'six' to make it easier to share a single codebase. | 19:48 |
| russellb | yeah, i think 2.7 and 3.4 at the same time is pretty common | 19:48 |
| otherwiseguy | in fact, single source tree seems to be the preferred method of supporting multiple python versions from what I've seen. | 19:48 |
| blp | The JSON parser is my fault. | 19:49 |
| russellb | blp: don't worry, you have enough karma :) | 19:49 |
| blp | It was clear that the built-in Python JSON parser wouldn't work, so I rewrote the C version in Python very quickly. | 19:49 |
| otherwiseguy | blp: it works. :) It's just very close to a line-by-line port of the C one. Which is totally valid thing to do, probably makes testing easier, etc. | 19:50 |
| otherwiseguy | I just noticed when profiling that pretty much all of the time was actually spent in the parser. | 19:50 |
| russellb | blp: neat, ovn-controller seg faults on startup in latest ovn branch | 19:51 |
| otherwiseguy | blp: I don't think any human could sanely write a great from-scratch streaming json parser in the time frame you had available. ;) | 19:51 |
| russellb | ah, conversion of chassis column did it | 19:52 |
| * russellb gives play by play of debug and fixing :-p | 19:53 | |
| blp | otherwiseguy: Most of the Python code is a line-by-line port of the C version, because I wrote most of it in an afternoon ;-) | 19:55 |
| otherwiseguy | "russellb is at the prompt. he's running the code. Oooooh, SyntaxError was thrown. That's gotta hurt!" | 19:55 |
| blp | Two-minute penalty for typo! | 19:55 |
| blp | "He's interrupted by IRC. Now his officemate makes a joke!" | 19:56 |
| russellb | :) | 19:56 |
| * otherwiseguy assigns the announcer a two-minute penalty for using thrown vs raised in a pedantic fit | 19:56 | |
| otherwiseguy | lol | 19:56 |
| blp | "Oops, someone just asked him a question on the conference call he's mostly ignoring. Will he be able to field it?" | 19:56 |
| russellb | pretty much sounds like my day | 19:56 |
| russellb | and he submits a patch! | 19:59 |
| *** mestery_ has joined #openstack-neutron-ovn | 20:00 | |
| blp | It's too bad we can't just do "#define as ," | 20:00 |
| russellb | it's too bad the move to Python 3 is a giant pain | 20:00 |
| russellb | basically, we should have written OpenStack in C | 20:01 |
| russellb | :-p | 20:01 |
| russellb | would cut down on contributors, but probably in a good way | 20:02 |
| russellb | blp: check out this fun log file from the functional test CI job: http://logs.openstack.org/26/178826/7/check/check-tempest-dsvm-networking-ovn/966525f/logs/screen-ovn-controller.txt.gz | 20:03 |
| *** mestery has quit IRC | 20:03 | |
| russellb | seems we hit that condition a lot :) | 20:03 |
| russellb | of course, this job is also completely failing, haven't had time to dig yet ... | 20:04 |
| blp | russellb: Hmm. That's a lot of unknown datapaths. | 20:55 |
| *** mestery_ is now known as mestery | 21:12 | |
| *** hitalia has quit IRC | 23:40 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!