14:01:05 #startmeeting networking 14:01:06 Meeting started Tue Mar 10 14:01:05 2015 UTC and is due to finish in 60 minutes. The chair is mestery. Information about MeetBot at http://wiki.debian.org/MeetBot. 14:01:08 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 14:01:10 hi 14:01:10 The meeting name has been set to 'networking' 14:01:17 Ialoha! 14:01:18 We've got a packed meeting today, so lets get to it! 14:01:20 aloha! 14:01:22 #link https://wiki.openstack.org/wiki/Network/Meetings Agenda 14:01:29 #topic Announcements 14:01:50 We passed FPF last week, and only a handful of BPs had no code. So, the good news is we have a lot to review! The bad news is, we have a lot to review. 14:01:54 hi 14:01:59 #info Feature, String, and Dependency Freeze: March 19 14:02:01 #info RCs: April 9-23 14:02:05 #info Kilo release: April 30 14:02:13 #link https://launchpad.net/neutron/+milestone/kilo-3 14:02:29 Any questions on dates for the end of Kilo? 14:02:50 I also wanted to highlight the OpenStack Client weekly meeting 14:02:59 #info OpenStack Client weekly meeting Thursdays at 19:00 UTC in #openstack-meeting 14:03:09 #link https://wiki.openstack.org/wiki/Meetings/OpenStackClient#Next_Meeting_Agenda 14:03:21 It would be great if we could get someone to represent Neutron there as we look to deprecate the client in Liberty 14:03:47 OK, lets move on as we have a packed agenda today. 14:03:50 #topic Bugs 14:03:53 enikanorov: Hello! 14:04:00 enikanorov_: Also hello 14:04:05 mestery: hi 14:04:19 hi 14:04:25 not much of updates from me this week. we have one critical issue 14:04:26 q-agent-notifier-tunnel-update_fanout_e7932584281b4eaca12007d4db921da1 14:04:29 oops 14:04:31 not this one! 14:04:37 lol 14:04:40 https://bugs.launchpad.net/neutron/+bug/1323658 14:04:40 Launchpad bug 1323658 in OpenStack Compute (nova) "Nova resize/restart results in guest ending up in inconsistent state with Neutron" [Critical,Confirmed] 14:04:55 that is hanging for quite a while now... 14:05:31 enikanorov_: Yes, it's been around for a long while, and unfourtanetly I had no time to dig into this the past week, I should have some time later this week yet. 14:05:44 enikanorov_: what's the gate hit rate in the past 7 days? 14:06:25 salv-orlando: i didn't look at it, sorry 14:06:26 o/ 14:06:56 https://bugs.launchpad.net/neutron/+bug/1382064 14:06:56 Launchpad bug 1382064 in neutron "Failure to allocate tunnel id when creating networks concurrently" [High,In progress] - Assigned to Eugene Nikanorov (enikanorov) 14:06:57 enikanorov_: np. I was just considering whether it was serious enough to preempt our current work and jump on it 14:07:10 o/ 14:07:17 for this one I've updated the fix with oslo.db wrapper introduced in 1.5.0 14:07:29 enikanorov_: Nice! 14:07:30 hopefully this will allow us to merge multiple API/RPC workers patch 14:07:54 that's all from me 14:08:07 enikanorov_: Thanks! Any other bugs anyone wants to bring up with the broader team here? 14:08:58 OK, lets move along. 14:09:01 #topic Docs 14:09:09 emagana: Hi there, you here this morning? 14:09:35 mestery: he's the ops summit right? 14:09:41 markmcclain: Good call 14:09:42 #undo 14:09:43 Removing item from minutes: 14:09:51 Lets move on 14:09:53 I am here! 14:09:59 #topic Docs 14:10:01 emagana: Ha! Hi there! :) 14:10:06 Quick docs update? 14:10:08 sorry,... at the operators meetup 14:10:24 emagana: OK, so no docs update? 14:10:40 gerrit review on the nova-network to neutron migration: https://review.openstack.org/#/c/155947/ 14:11:00 emagana: Nice! 14:11:05 #link https://review.openstack.org/#/c/155947/ 14:11:05 funny, we just discussed at the operators two things: 14:11:26 1. more volunteers for the networking guide! We got four or five volunteers!!!! yeah!! 14:11:38 emagana: awesome 14:11:49 emagana: the nova-net to enutron migration docs patch is on hold, yes? 14:12:07 emagana: That is great! 14:12:12 emagana: pending anne's approval to move ahead once networking docs transistion to rst files? 14:12:12 :-) 14:12:18 2. how positive is nova-network to neutron migration for production cloud operators... These are the bad news... No good feedback yet. 14:12:21 emagana: is this not true? 14:12:45 emagana: what do you mean bad news 14:12:52 anteaya: Yes, it is in hold but just based on the location and format... so, yes you are correct 14:13:03 emagana: what feedback have you gotten? 14:13:21 btw I wasn't expecting _any_ good news on the migrtion to come from the ops meetup 14:13:28 anteaya: I will love to provide it right now but it will consume a lot of time, mestery should I go ahead? 14:13:35 so the fact you haven't gotten any is in line with my expectations 14:13:45 it is kind of important to me 14:13:50 unless mestery disagrees 14:13:55 anteaya: we can discuss offline! 14:13:58 and the point wasn't really to get feedback 14:14:04 it was to inform them of our work 14:14:10 anteaya: Ack 14:14:43 anteaya: bottom line is that even with a easier migration neutron still needs to provide 100% parity and people running IPv6 multihost are not seeing it yet 14:15:08 so this is a neutron parity issue corrent? 14:15:21 based on what you are hearing from those still running nova net? 14:15:35 I knew about the multihost things... but what's specific about ipv6? 14:15:36 carl_baldwin: For IPV6 multihost, we'll need IPV6 support in DVR I believe 14:15:46 armax: ^^^ 14:15:47 and scine when did nova net grow ipv6 ability? 14:15:50 mestery: that is correct! 14:16:05 mestery: ack 14:16:09 anteaya: my recollection as well 14:16:14 mestery: obviously not there yet 14:16:20 Yup 14:16:29 #info Operators asked about IPV6 DVR support at the Ops Mid-Cycle 14:16:35 emagana: so please get some names if you can from your feedback 14:16:37 mestery: there may be issues with DVR and IPv6, but at least from a a testing perspective, what works for legacy routing works for DVR too 14:16:47 as right now what you are saying makes no sense to me 14:17:20 there was an interesting conversation about how Ipv6 works with nova-network and sc68cal had interesting questions about it.. Internet2 project is the operator asking for it 14:17:37 Those blasted internet researchers 14:17:38 ;) 14:18:25 :) 14:18:29 OK, this is all good data emagana, but it looks like you and anteaya should sync up post Ops Meetup. 14:18:35 emagana: that's good to know the requirement. I still wonder however whether is a requirement for neutron or a lack of parity 14:18:47 but I guess that's just pointless speculation 14:18:58 OK, anything else emagana? 14:19:05 salv-orlando: it's a good question 14:19:08 mestery: not really! 14:19:15 yeah, we need to follow what the tc said on this, not just make somtehting up based on what one person said or is doing 14:19:17 emagana: Thanks, and enjoy the rest of the Ops Meetup! 14:19:33 I think we should have a session Ops with Neutron team 14:19:33 #action emagana and anteaya to sync post Ops Meetup around nova-network to neutron migration discussions 14:19:46 emagana: Also makes sense 14:20:13 OK, lets move along, thanks emagana! 14:20:16 mestery: my feeling is that they may do not have all the information and maybe confused on how they work with Neutron ;-) 14:20:21 sure!!! 14:20:46 #action Team to look at having an Ops/Neutron session in Vancouver at the Design Summit 14:20:52 #topic Should the *aas repos be depending on neutron HEAD? 14:20:59 Not sure who put this on the agenda ... 14:21:04 marun, ? 14:21:04 marun added this. 14:21:11 marun: Floor is yorus! 14:21:41 his idea was to pin the unit tests to specific commits of neutron, to prevent them from constantly breaking with breaking neutron commits. this moves the break further away from the break, but also doesn't randomly break the *aas gates. 14:21:46 Same question applies to the vendor repos? 14:21:59 HenryG: Good point 14:22:18 HenryG, even more relevant there since neutron team does not track them 14:22:24 I mainly wanted to raise the profile of this idea. 14:22:28 HenryG: pinning on a specific hash is only feasbile once the decomp is complete 14:22:29 HenryG, dougwig: I deliberately decided to not pint stackforge/vmware-nsx on any neutron commit, because this would make the whole point of 3rd party CI vain. 14:22:33 If Neutron is now a library, well, you normally use a specific version of a library and not just pull it from Git 14:22:45 not sure if that can apply also to the *ass repos 14:22:53 salv-orlando: there should be a difference between pinning and 3rd party ci 14:23:03 doing this would require an automated job (possible via mechanical turk at first) to keep bumping the commit revision. 14:23:08 salv-orlando: 3rd party ci *should* test against latest 14:23:12 salv-orlando: you can have the CI determine whether you can move the pin forward 14:23:26 dougwig: +1 14:23:28 CI may test against HEAD 14:23:31 salv-orlando: but I would expect repos to pin so that development and stabilization could proceed independently 14:23:37 if jenkins and 3rd party test against different commit revisions, people will ignore 3rd party even more than they do today. 14:23:44 dougwig: ++ 14:23:46 we have to eat our own dogfood, not set up those CIs for failure. 14:24:00 dougwig: everyone ignores non-voting ci, except when they have a reason not to 14:24:16 marun: the thing is that the goal of CI for a plugin is to detect breakages induced in your system from the dependent library. And indeed should test against latest, as you say. Now... if I pin neutron I won't be able to test against latest anymore. 14:24:16 dougwig: non-voting is delibertatly non-forcing - it's just communication 14:24:18 heck, i even ignore the voting ones. and as a third-party CI operator, I should know better. 14:24:44 marun: ignore my point, finish first your discussion with dougwig. No need to cross wires. 14:25:15 marun: I see your point here, but I think salv-orlando's point is valid as well and I don't know how to reconcile the two. 14:25:28 it's simply an option 14:25:49 mestery - i would assume that the 3rd party CIs would test against the same revision as jenkins, via the tox.ini setting. 14:25:50 marun: OK 14:25:57 If I had a chance to run two jobs I might be ok with having a "stable" voting job against a pinned neutron 14:26:04 if folks in any given dependent project want to conjoin development and stabilization against dependencies, they can continue to do so. 14:26:07 and another non-voting job against latest neutron 14:26:10 I think it sucks, personally. 14:26:17 i'm really quite torn on this idea. on the one hand, it reduces the gate breakages that happen pretty often. on the other, it just moves the problem away further in time. 14:26:37 I like marun's idea of separating development and stabilization here 14:26:39 It's a solid idea 14:26:54 It does require dedicated resources to stabilize, though. 14:26:54 It does kick the can down the road, but it has a lot of benefits as well 14:26:59 Right 14:27:00 marun: I think marun has a good point. We can work around that. At the end of the day is trying to make people developing *asses easier 14:27:01 It doesn't make less work, it just partitions it. 14:27:12 Yes 14:27:15 hopefully, something like proposal bot can keep that commit hash pretty current. 14:27:28 dougwig: that would be the hope. 14:27:37 dougwig: falling permanently behind isn't an option. 14:27:56 right. by 'hopefully', i meant, 'dougwig is going to make sure there is a bot, for his own sanity.' 14:28:05 I would hope we could experiment with this in the small, here. 14:28:15 With an eye towards doing it for all of openstack in the very distant future. 14:28:22 ++ 14:28:27 We have the same problem between the 'integrated core' 14:28:27 ++ 14:28:33 sounds good 14:28:47 So, lets try it out with LBaaS or VPNaaS, taht would be my proposal 14:28:49 And see how it goes 14:28:58 lbaas can be the guinea pigs. 14:28:59 Thoughts? 14:29:08 i'm curious how it turns out myself. 14:29:08 dougwig++ 14:29:10 #info LBaaS to try out the new pinning approach presented here 14:29:15 nope, I am happy to extend the experiment to stackforge/vmware-nsx 14:29:15 I would like to participate, to see how we can structure things to make it work. 14:29:31 #info stackforge/vmare-nsx to try pinning from a vendor side as well 14:29:34 Getting down and dirty is the only real way to see how to make it work. 14:29:39 marun: Can you work with dougwig and salv-orlando then? 14:29:40 :) 14:29:49 mestery: definitely. 14:29:49 #info marun to work closely with dougwig and salv-orlando on the pinning strategy 14:29:51 Sweet! 14:29:53 sweet. 14:29:56 i think. :) 14:29:59 Consensus! :) 14:29:59 Group hug 14:29:59 heh 14:30:02 nice. 14:30:03 +1 14:30:04 marun++, dougwig++, salv-orlando++ 14:30:11 Shall we move on in the agenda? 14:30:33 marun: Do we need to discuss Xen and OVS support still? 14:30:42 mestery: maybe? 14:30:43 mestery: one of us could be byzantine. You need a fourth person to establish a consensus ;) 14:30:50 #topic The Future of Xen and OVS Support 14:30:53 salv-orlando: lol 14:30:59 marun: Go ahead please! 14:31:16 Does anyone know of dedicated resources maintaining/testing Xen support? 14:31:26 Bob Ball seems to be working on 3rd party CI 14:31:37 But I haven't seen any developers working on the project. 14:31:44 Anyone else? 14:31:44 Me either 14:32:01 The question is, how can we justify Xen support without someone stepping up to work on it? 14:32:05 marun, on FOSDEM, I talked to someone from Xen, and they were not that happy about the idea of voting for all neutron patches at that time. is it changed now? 14:32:26 bobball works with xen 14:32:32 does ops on the ci system 14:32:37 though I also think manual check trigger would be enough 14:32:37 has anyone talked to him? 14:32:38 anteaya: I'm not sure that's enough 14:32:49 oh I'm not saying it would be 14:32:50 anteaya: Yes, he replied to the thread on the ML 14:32:50 anteaya: there was a conversation on the mailing list 14:32:59 mestery: great 14:33:01 anteaya: But only on the issue of 3rd party ci 14:33:07 #link http://lists.openstack.org/pipermail/openstack-dev/2015-March/058211.html 14:33:33 Even if we did have 3rd party CI, I'm not sure that Xen support belongs in-tree. 14:33:37 So, I think unless someone steps up soon and a CI starts running validating patches (especially on the OVS agent), we may need to look at deprecating support for Xen. 14:33:49 There's nothing stopping it from being done out-of-tree. 14:33:57 marun: ++ 14:33:58 The support is mainly at the agent level anyway 14:34:19 marun: Can you reply on that thread again with this info? Lets start moving it in that direction. 14:34:24 mestery, marun: there isn't. the only point is... does nova's xenapi driver still work with nova-network 14:34:33 otherwise we should look at a combined deprecation 14:34:41 and see also what's johnthetubaguy opinion on it is 14:34:44 salv-orlando: Good call, that can be on the ML thread as well 14:34:48 salv-orlando: hmmm, more questions... 14:35:03 * johnthetubaguy wonders if he can help, reads scrollback 14:35:08 I do see Xen CI running against nova nad devstack 14:35:10 But not neutron patches 14:35:41 johnthetubaguy: we realized XenAPI + OVS support has not been maintained for ages in Neutron. We're considering deprecation. 14:35:49 ah, right, gotcha 14:36:06 I think Citrix are working on a CI for that, the current CI is testing nova-network 14:36:36 johnthetubaguy: CI is fine, but in the absence of development resources it's not clear how detected breakage would get fixed 14:36:39 * mestery sees BobBall join the channel now 14:36:42 I think its passing, but it has all the neutron related races libvirt used to have, because its missing the event callback logic in Nova, I think, it could be something else 14:36:46 johnthetubaguy: good. As long as we're still sure the nova-network support still works we can keep our discussion contained to neutron. 14:37:01 BobBall: howdy, XenAPI neutron CI, hows that going? 14:37:05 johnthetubaguy: since I know the guy who did that support I was highly skeptical it worked ;) 14:37:20 salv-orlando: :) 14:37:32 BobBall: thanks 14:37:38 You're right that we're seeing races that we haven't been able to resolve yet - so we don't have a stable environment we can then convert to a CI 14:38:01 BobBall: I think the bigger question is can neutron expect more resources in addition to yourself 14:38:06 BobBall: Did you happen to read my most recent reply on the ml? 14:38:09 Those may be, as John said, similar to those that were seen in libvirt, but we've not yet tested that theory. 14:38:21 since I admire what you do and think you do great work, but you are on ly one person 14:38:46 salv-orlando: my take is, as long as deprecation is reversible, it could be the right thing to do 14:38:53 Absolutely 100% yes in the medium term, but in the short term we don't have additional resources to add on it. 14:39:13 I did marun - but didn't have much more to add. 14:39:40 My perspective is that I agree that a CI is a pre-requisite to being able to ensure continued support of XenAPI+OVS in Neutron 14:39:41 BobBall: Do you think continued support for Xen in Neutron is justified under the current circumstances? 14:39:54 I want to get the quark plugin that we use with XenServer more tested against upstream, etc, but honestly, that effort has not started yet, but its something I want to make happen if possible 14:40:23 I don't want Neutron to drop this support, but I can't justifiably argue for it until we have an automated way of testing it upstream. 14:40:41 marun: its an interesting twist with the nova-network deprecation talk, but that shouldn't change things 14:40:59 I'm sensing we may need to deprecate it in the short term then, or at least move it out of tree into a stackforge repo. 14:41:12 It coudl exist in stackforge in it's current state until someone takes it up and fixes it. Thoughts? 14:41:23 mastery: thats my take on this too 14:41:32 I think if Citrix/the Xen community wanted to provide Xen support for Neutron, they could do so out of tree relatively easily. 14:41:46 In terms of nova-network deprecation it is understood that XenAPI will have to work with Neutron+OVS, including an upstream CI, and we are aware of that as a requirement moving forward. 14:42:02 mestery: I seem to recall that the XenAPI support for OVS is tightly coupled with the OVS agent. I'm not sure if it can be split out, but it's surely worth trying 14:42:11 #info Xen Support in Neutron to move out of tree into stackforge if possible 14:42:17 OK, I think we can close on this subject here now. 14:42:24 BobBall johnthetubaguy, thanks for attending with us here! 14:42:39 Lets move on in the agenda now, 17 minutes left 14:42:43 no worries, sorry its not a better answer, but sounds like a good way forward 14:42:50 perhaps out of tree development allows xen folks to avoid continueous CI posting and they can pin neutron version to stabilize their developemnet 14:42:59 #topic Report on progress of "MTU selection and advertisement" blueprint implementation. 14:43:00 Does anyone know who put this on the agenda? 14:43:01 (though it is tihgtly coupled with ovs-agent) 14:43:13 HenryG: ^^^^ 14:43:32 ijw-ubuntu, I thin? 14:43:51 Hi 14:44:07 I would say the implementation is going fine. It will not be a lot of code. 14:44:13 HenryG: OK, is that the updatE? :) 14:44:22 HenryG: Have we merged the spec patch yet? That's been there for at least 10 days I think 14:44:51 I don't have +A on specs. 14:44:59 HenryG: I'll take care of that post-meeting 14:45:04 Thanks for the update HenryG! 14:45:07 Moving right along ... 14:45:19 #topic Proposal to remove py3* from the default tox envlist for now to address "fail blindness" 14:45:23 gus: This is you I think :) 14:45:33 And maybe dougwig too 14:45:45 well, just +1 on making 'tox' actually pass. Is it controvercial? 14:45:50 #link https://review.openstack.org/137516 14:45:56 the target exists for everyone else right? 14:45:59 ihrachyshka: I think so, thus why gus put it here 14:46:05 markmcclain: Yes, I think so 14:46:25 i am about +20000000 on this. i hate that we put something in broken, to attempt social engineering. when it actual fact, it trains the opposite behavior, and frustrates newbies. 14:46:43 dougwig: agreed 14:47:00 generally we've gotten push back from the folks that really want py3 support now 14:47:03 I am +2 on that patch until it works as well dougwig. 14:47:13 it is like this is all the integrated projects, so i'm not sure if we should do this in neutron (heck yes we should) or if we need to fight a larger battle. 14:47:19 +1 14:47:26 markmcclain, we're like ~3-4 cycles from there I guess. 14:47:35 dougwig: We control our own destiny in neutron, we can win that battle then fight the larger war :) 14:47:40 dougwig: I think we're in control of our own tox.ini 14:47:41 if they want it now, submit the code to support py3. i'd like it to work... that's not my beef. 14:47:50 then here comes a +a 14:47:51 dougwig: Well said 14:47:53 ihrachyshka: I know.. wish we were much closer 14:48:10 markmcclain, we're blocked by eventlet. main obstacle. 14:48:14 I think this change is doing nothing. Just avoiding failures for tests people probably don't even expect to run 14:48:20 ihrachyshka: eventlet has py3 support 14:48:24 so it's harmless, imho 14:48:28 (it's brand new) 14:48:38 I'm +2 on this 14:48:42 markmcclain, well, not for long (it's very fresh, and afaik still bugs are found) 14:49:35 ihrachyshka: so do reckon it's not feasible to achieve 3.x compatibility in Liberty? 14:50:00 salv-orlando, hah, in dreams - maybe 14:50:02 even if it's reachable in two weeks, shouldn't the tox env be added in the patch that enables py3? 14:50:24 if we submitted code to enable something broken for *anything else*, would anyone here approve it? 14:50:26 I don't know why we discuss the matter for 5 mins already 14:50:29 salv-orlando: it's going to take a good deal of work dive into the dependency hole and fix the odd cases where it doesn't work 14:50:31 dougwig: nope 14:50:51 ihrachyshka: because religion. 14:51:35 ok, the patch is +A'd, no way back, can we move on? :) 14:51:38 OK, shall we move on? 14:51:46 please 14:51:47 Lets move on. 14:51:48 ihrachyshka: there is always a way back! 14:51:59 #topic Open Discussion 14:52:00 unless you jump off a cliff. In that case there isn't 14:52:04 8 minutes left, Open discussion time! 14:52:25 I have oslo.log patch for kilo that breaks all CI for split vendors 14:52:38 I wonder whether we're ok with merging it as-is, knowingly breaking them 14:52:45 ihrachyshka: link? 14:52:53 (breakage is because they use neutron.openstack.common.log) 14:53:03 #link https://review.openstack.org/159638 14:53:08 ihrachyshka: can we not leave a shim there? 14:53:18 we need to remove logging from oslo-incubator. 14:53:20 markmcclain: via sys.modules hacks? 14:53:23 ihrachyshk, armax: I think one of the reasons of the decomposition is to not waste time with vendor incompatibilities 14:53:24 markmcclain, libraries are mutual exclusive 14:53:30 markmcclain, because of conflicting options 14:53:31 markmcclain: it may break pylint though ;) 14:53:42 marun: haha 14:53:45 lol 14:53:53 armax: mestery: May I request you to please take a look at https://review.openstack.org/#/c/159056/ for brocade vyatta vendor decomposition and provide your review. 14:54:04 so what do you reckon if we just send a warning on the mailing list, wait 48 hours for vendor repos to adapt, and then merge it? 14:54:20 without going into hacks. 14:54:25 all in all, vendors should follow some kind of guidelines for their code and packages released, and I started a new wiki page for that 14:54:30 #link https://wiki.openstack.org/wiki/Neutron/VendorSplitPackaging#Neutron_Vendor_split:_Packager_perspective 14:54:34 please take a look and review 14:54:48 salv-orlando, I'm ok with sending the email 14:54:52 salv-orlando, ihrachyshka: I think we whould not delete openstack/common log 14:54:58 ihrachyshka: until everyone is moved over 14:55:04 salv-orlando, we just need to review the patch quickly enough because of dep freeze 14:55:11 ihrachyshka: if we can avoid breaking the world, I think we should 14:55:28 Maybe we need a dedicated communication mechanism for intentional breakage? 14:55:36 armax, I already mentioned conflicting options, that's a common issue for oslo graduations 14:55:37 http://lists.openstack.org/pipermail/openstack-dev/2015-March/058610.html (oslo.log information) 14:56:10 amotoki, right, that's what I would expect them to see if they use both versions 14:56:36 this is the "problem" with putting oslo in non-leaf nodes of the dependency tree, really. you start having the same kind of backwards compatibility issues that we have with other neutron entry points that are being used in a library like fashion. 14:56:39 marun, + for communication channel. some [new-tag] for openstack-dev? 14:56:53 ihrachyshka: ok 14:57:24 ihrachyshka: either that or a wiki page/file in tree that lists successive, timestamped breaking changes 14:57:44 ihrachyshka: so when breakage is detected, that is the first thing to check 14:57:51 ihrachyshka: rather than having to search ml 14:58:05 understood. wiki is probably more dynamic for the use case 14:58:06 single entry point for information is nice. 14:58:36 I'll create the page and give a heads-up in ML 14:58:41 ihrachyshka: sounds good 14:58:43 ihrachyshka: this becomes a real pain for having the same out-of-tree code support both kilo and earlier releases. 14:58:45 (and reviews are welcome) 14:59:07 OK, thanks folks! 14:59:19 Don't forget about Kilo-3 reviews :) 14:59:22 See you next week! 14:59:24 #endmeeting