14:00:06 #startmeeting neutron lbaas 14:00:08 Meeting started Thu Jul 17 14:00:06 2014 UTC and is due to finish in 60 minutes. The chair is jorgem. Information about MeetBot at http://wiki.debian.org/MeetBot. 14:00:09 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 14:00:09 \o\ /o/ 14:00:11 The meeting name has been set to 'neutron_lbaas' 14:00:17 o/ 14:00:20 hello 14:00:30 hi 14:00:53 o/ 14:00:56 jorgem, I would like to add one item to the ageand. We talked about enhancing the existing reference driver for juno. Is that still in the plans? 14:00:58 Does anyone have anything they want to add to the agenda? 14:00:59 Morning! 14:01:11 hi 14:01:13 jorgem, lol 14:01:19 sballe_: I am currently doing this 14:01:29 hello 14:01:30 blogan, oh fantastic! 14:01:42 hi 14:02:02 sballe_: yes phil and doug started that I believe 14:02:11 sballe_: you mean the reference implemenatation to work with the new api right? 14:03:09 sballe_: Would you still like me to add that to the agenda? 14:03:14 blogan, I was also wondering if we were adding any other functionaltiy. At some poitn we had talked about adding HA just to make it a little moe usable before we have Octavia and I am not sure what the edn conclusion ended up being 14:03:30 jorgem, no. I'll follow up with blogan 14:03:37 sballe_: I don't think HA was part of that 14:03:54 sballe_: Just API functionality was going to be implemented 14:04:10 Anyone else have agenda items they would like added? 14:04:13 sballe_: no that woudl require way too much time and we need to get this out asap, so I'm just focusing on it being able to do current features 14:04:29 jorgem, ok fair enough. 14:04:32 Here is the current agenda: 14:04:32 • Review Updates 14:04:32 • Discuss blockers and how to mitigate 14:04:32 ◦ Juno release coming quickly! 14:04:32 • Discuss reference implementation 14:04:36 It was my understanding that the reference implementation needed to do things like L7 and TLS so that these features could be adopted. But HA was not part of this, as it doesn't affect the user API. 14:04:51 +1 14:04:51 sbalukoff1: Correct. 14:05:19 sbalukoff1: correct but I won't be implementing TLS or L7 this time, that should be left to the TLS and L7 blueprints to implement 14:05:33 sbalukoff1, ok good enough for me. No HA. 14:05:34 jorgem: TLS: Closing on SubjectAltName and TLS: Conflict Resolution during SNI 14:05:35 Okay, well I think agenda is set then. Let's continue with this topic. 14:05:41 i want both of them to be added 14:05:43 #topic Discuss reference implementation 14:05:44 blogan: True. 14:05:50 we're actually removing some features (the agent, which hurts scalability), and adding others (tls/l7, if/when those features hit), since no feature/api can be released with an implementation of some kind. 14:06:03 apologies for the late addition 14:06:04 /with/without/ 14:06:35 Hi all 14:06:52 vjay: no worries I got it 14:06:55 dougwig: this is true, hopefully people understand that this reference implementation version is only meant for devstack deployments and testing 14:07:35 blogan: Well, *we* understand that. But I'm not sure it's possible to telegraph that message to the rest of the OpenStack community. :P 14:07:41 dougwig: Eventually we were aiming to make Octavia the reference implementation once it becomes operator grade I believe? 14:07:59 blogan, I agree. We had just had some discussion in the past around this and I just wanted us to be on the same page. 14:07:59 Of you build it someone will put it into "prod" 14:08:02 If 14:08:04 sbalukoff +1 -- wonder when we get asked in the irc about scalability bugs 14:08:26 yes, and yes, but blogan's plan, which i 100% agree with, is that we have to get *something* in in order to ship any of this, or we risk getting punted out of juno entirely. 14:08:27 xgerman: Haha! 14:08:36 Yep! 14:08:46 The issue is that today there is no Neutron LBaaS that can used in production/ 14:09:11 well, if we omit the hardware appliances... 14:09:11 apparently the agent haproxy is being used in production somewhere 14:09:15 which is fine. We just need to make sure people understand that 14:09:16 sballe_: correction, there is no fully open-source neutron lbaas that can be used in production. 14:09:33 sballe_: yet folks are doing just that. 14:09:35 which is arguably true of neutron itself right now. 14:09:37 dougwig: +1 14:09:42 dougwig, Yes thanks for corecting me :) 14:09:55 dougwig is good at correcting people 14:10:05 dougwig, our public cloud is using Neutron in production 14:10:07 so we need to add some startup message to the os driver "Don't use that in production" 14:10:25 xgerman: or just inject a header 14:10:38 dougwig: as is our private cloud 14:10:55 ctracey, I agree with you. People will be using it in production but at their own risk 14:10:57 with stock ml2 ova? 14:11:06 ovs? 14:11:16 auto-correct and openstack acronyms do not get along. :) 14:11:22 What's the status on the reference implementation assuming the features we want are exposed? 14:11:23 dougwig, We have our own SDN plugin 14:11:38 blogan: ? ptoohill ? 14:11:42 dougwig, in Atalnta we had like three ralks on nhow to make Neutron work -- that's just not the scope of this meeting ;-) 14:11:58 its in progress atm 14:12:04 dougwig: Rackspace has our own OVS plugin (Quark, I believe) but it is fully open source IIRC :P 14:12:22 well, neither is the ref implementation, so... :-) 14:12:22 Ill be working on some tests for the driver itself, while blogan will be updating managers 14:12:23 jorgem: its close to being in a reviewable state, large effort still required on tests, stats and health monitoring 14:12:48 blogan: Okay cool. No blockers though right? 14:13:23 No blockers i can think of at this point jorgem 14:14:14 Speaking of blockers let's move to that topic 14:14:28 #topic Discuss blockers and mitigation tactics 14:14:40 So, Juno is coming up quite quickly 14:14:57 I'm with blogan...very close to review 14:15:00 * TrevorV_ thinks jorgem brought that up just to get a good segue 14:15:07 For the CLI bits 14:15:11 jorgem, juno-2 right? 14:15:12 a big blocker is getting this reference implementation done because the v2 APi won't get merged until it has a reference implementation workign with it 14:15:31 Well, what is the absolute last day we can get code merged? 14:15:41 I'm a little confused on the cycles 14:15:43 for Juno-2, I think its Monday 14:16:02 next week, Monday? 14:16:05 #link https://wiki.openstack.org/wiki/Juno_Release_Schedule 14:16:27 maybe thats when the code has to be in for review 14:17:00 1) Is sept. 18th the absolute last day? 14:17:16 thought that was for the bps, isnt the 20th last day for BP approval? 14:17:20 I just want us all to be on the same page 14:17:23 I talked to mestery yesterday and he said he was going to be busy Mon, Tues and Wed of next week with Juno-2 related task 14:18:44 which, if so, is a big blocker for TLS stuff if we cant get those approved in time. Im hoping im just misunderstanding the due dates 14:19:07 hey i know people are waiting on the reference implementation, but what can be done in the meantime to speed up everything thats waiting on this? 14:19:24 jorgem can you check with mestery on dates and update us on the ML? 14:19:52 xgerman: lol sure 14:19:57 thanks 14:20:10 #action jorgem to check on Neutron due dates 14:20:42 To blogans question anything else that can be worked on at the moment? 14:20:52 blogan: let's iterate closer over the next few days. I'd be happy to help bang this out. 14:20:59 I'm starting to feel that we won't get work completed in time 14:21:02 one thing people can do is make sure they're able to deploy the new code, and actually make requests to the API. I'm sure I've made some mistakes on that so just doing that would be great. Plus getting a good understanding of the code woudl be beneficial for everyone. 14:21:19 ++ 14:21:23 ++ 14:21:29 +++ 14:21:32 ctracey: sure whenever you got time 14:21:54 also I saw the tempest test not being reviewed -- can we get more eyes on that? 14:22:05 xgerman: link? 14:22:05 #action: ctracey and blogan to work closely over the next few days 14:22:28 if you have trouble deploying the code, hit us up on the lbaas channel. 14:22:40 Can free up most of today and probably all of tomorrow. 14:22:43 I think this list is still being maintained listing all the gerrit stuff that concerns Neutron LBaaS: https://etherpad.openstack.org/p/lbaas_reviews 14:23:00 yep, that's where I ran across that 14:23:07 can someone point to the patch of tempest test? 14:23:13 #link https://etherpad.openstack.org/p/lbaas_reviews 14:23:22 https://review.openstack.org/#/c/106089/ 14:23:28 sbalukoff: thanks that has the tempest link 14:24:00 thanks for keeping this updated dougwig 14:24:16 we should link that form our main wiki 14:24:16 Yes, thanks dougwig! 14:24:26 +1 14:24:27 ctracey, Did you add your review to the page? 14:24:28 yw 14:24:31 xgerman: it is linked 14:24:41 dougwig:10x! 14:24:42 Does anyone want to volunteer on the tempest stuff? 14:24:49 I will stop using browser history then 14:25:13 jorgem: mlavalle is working on that 14:25:32 but yes more people to review it 14:25:36 blogan: Correct but we need eyes reviewing it I believe 14:25:37 I will reach out to HP QA 14:25:38 i've looked at the tempest review, but it is a WIP 14:25:40 jorgem: evish, evgeny and myslef are on this chat together via my nick. evgeny will review 14:25:52 samuelbercovici: awesome thanks 14:26:07 also avishayb is reviewing the core code 14:26:13 sballe_: still working parts of it out 14:26:26 ctracey, ok ping me when you have it in there 14:26:31 #action evgenyf to review https://review.openstack.org/#/c/106089/ 14:26:55 I will definitely today. The members logic is a bit odd. 14:27:05 #action avishayb to continue reviewing core code 14:27:38 That's something I will likely talk to you about blogan 14:28:04 ctracey: okay ill be around most of the day 14:28:15 Okay, I know everyone will be reviewing but I'd like to have certain community members spearhead the effort on certain tasks 14:28:40 #action avishayb to review https://review.openstack.org/#/c/105609 https://review.openstack.org/#/c/105331 https://review.openstack.org/#/c/105610 14:28:45 Again, I just want to make sure we increase our momentum 14:28:45 #action everyone deploy the new code. instructions can be found here: https://wiki.openstack.org/wiki/Neutron/LBaaS/DeployWithDevstack 14:29:04 blogan: +1 14:29:07 +1 14:29:33 what about https://review.openstack.org/#/c/98640. Seems like every one but the core reviewers are chiming in on it. 14:29:36 Any other items related to speeding things up that people want to talk about? 14:29:59 blogan: it would be better to extend the tempest tests and use CI to run end-to-end 14:30:02 +1 crc32 14:30:20 samuelbercovici +1 14:30:21 #action jorgem to ask core reviewers to review https://review.openstack.org/#/c/98640 14:30:31 crc32: i'd wager they're waiting for the debate to calm down. 14:31:07 i don't tend to ping the cores until something has several +1's from the sub team, and no -1's. 14:31:09 samuelbercovici: for testing it out and eploying? 14:31:17 the TLS was stable for a while. I'd wager that my requests for core reviews led to more people to just blind -1 stuff. 14:31:18 *deploying 14:31:25 blogan:yes 14:31:36 jorgem: There is currently SNI issues discussions that should be resolved first 14:31:50 nice segway 14:32:16 crc32: the review monster is a fickle mistress, i will agree. :) 14:32:39 avishayb: asked for reviews on: https://review.openstack.org/#/c/99709/ 14:32:40 #topic TLS issues 14:32:47 TLS: Closing on SubjectAltName and TLS: Conflict Resolution during SNI 14:33:05 I think this is a nice segue 14:33:26 are we discussing TLS spec now? 14:33:44 vjay2: Yes since we were talking about it 14:34:07 vjay2: And it's a major component that needs to be addressed 14:34:27 vjay2: Would you like to start us off? 14:34:42 Yes! 14:34:55 I had raised 2 issues 14:35:01 * dougwig gets a bowl of popcorn. 14:35:12 I'll let you guys decide the conflict resolution but I just want to make sure SSL parsing code is avalble to the API. Also a declaration that x509 parsing isn't being done at SNI Handshake time. 14:35:19 about parsing of the certificates's SAN 14:35:21 #topci TLS Issues: Closing on SubjectAltName 14:35:30 irc://irc.freenode.net:6667/#topci TLS Issues: Closing on SubjectAltName 14:35:38 ugh nm 14:35:59 I think we should not detail about SAN in the spec unless required 14:36:20 there could be backends like NetScaler which might have used SAN 14:36:46 *sorry not used SAN 14:36:54 I disagree, since every major browser on the market uses SAN. :P 14:37:10 And it's technically part of the TLS RFC. 14:37:26 SAN is common 14:37:34 So what is the expecation from the backend if it does not implement NetScaler 14:37:41 s/Netscaler/SAN 14:37:54 if neutron fetches a cert/key, and parses out certain info, and then hands all of that to a driver, the driver is free to throw away what it isn't interested in/can't handle, and do its own thing, isn't it? 14:38:13 dougwig: +1 14:38:18 dougwig: +! 14:38:22 i agree with dougwig 14:38:34 Then where is our disagreement? 14:38:34 I would exactly want that, period 14:39:05 Okay so we have consensus it looks like. Who wants to update the bp? 14:39:11 dougwig+ I agree but you vjay2 are declaring that SAN should not be mentioned in the spec 14:39:15 i think that is what the spec is defining, and if you want to parse out hostname on your own and not the way neutron is specifying, that's entirely possible. 14:39:18 I disagree with dougwig "own thing" scares me that things would work differently with each driver and not produce standatdized results 14:39:49 Now the spec like SAN seems mandatory 14:39:56 xgerman: Isn't that why you have different drivers to begin with? 14:40:02 xgerman: isn't that going to happen with every driver? 14:40:09 there should be an option for the driver to thow error if it could not support it 14:40:12 xgerman: Keyword being different? 14:40:29 vjay: I can mention this point explicitlty in RST. Driver does not have to use SAn that is passed from API leyer 14:40:39 vjay2: Thats what sbuukoff suggested but the Mail thread went in differen't directions. 14:40:51 The ability for the driver to reject certain configurations otherwise valid is going to become more and more important as we add features... 14:41:11 xgerman: well, step back, and the feature being deliver is SNI. it is limiting and slows things down if every driver must implement that exactly the same way with exactly the same features. i mean, we all "know" what SNI means, and you're free to not use a vendor that purposely implemented something that doesn't even come close. 14:41:20 on the other hand to play ball they also need to support a minimum set of features 14:41:26 thanks evgenyf 14:41:31 sbalukoff: +1 14:41:48 dougwig, thanks for the clarification 14:42:17 fine. I sucks that every vender behaves differenty with SNI so I guess we have to accept this is driver specific. 14:42:45 err it. 14:42:50 crc32 wisj w ecould apply some normative pressure... 14:42:59 bp=https://review.openstack.org/#/c/106089/ 14:43:04 Okay, so what updates do we need to make to the spec? 14:43:10 i'm not advocating that every vendor be different. i'm advocating to not pull everything down to the lowest common denominator every time. 14:43:41 wrong one. bp=https://review.openstack.org/#/c/98640/ 14:44:24 It sounds like we'll need to add verbage to the spec that says a driver can choose not to use SAN info in order to get vjay2's approval. 14:44:41 I will do that 14:44:49 dougwig: Agreed. 14:44:57 dougwig: +1 14:44:59 I am fine with that 14:45:13 #action evgenyf to add verbage to the spec that says a driver can choose not to use SAN info in order to get vjay2's approval. (https://review.http://openstack.org/#/c/98640/) 14:45:16 sbalukoff: if the x509 has SAN info, than the backend should throw an unsupported exception 14:45:41 Awesome. Now let's address conflict resolution during SNI. What is the main issue here? 14:45:55 vjay2: is that ok? 14:46:03 Also, for conflict resolution: Any problems with saying SNI-related certs have an order, and that individual drivers should do whatever they think is necessary to resolve conflicting hostnames? (Given that conflicting hostnames are a rarity?) 14:46:28 + 1 14:46:29 yes, SAN in combination with SNI. I dont think SAN is valid for non SNI cases 14:46:32 sbalukoff: +1 14:46:44 #topic TLS Conflict Resolution during SNI 14:46:44 samuelbercovici: +1 14:47:01 sbalukoff: +1 14:47:10 vjay2: Any problem with that? 14:47:18 sbalukoff: +1 . RST is already stating that 14:47:33 so the correct behaviour of a backend is that if he can't match the expected behavior it should throw an unsupported exception 14:47:50 samuelbercovici: +1 14:48:02 samuelbercovici: +1, that's what i've been doing, anyway. 14:48:03 Is order a mandatory to be specified while binding certs for SNI? 14:48:03 samuelbercovici +1 14:48:05 +1 though the community had trouble with that before 14:48:19 vjay2: No the spec already mentioned that. 14:48:20 mandatory == mandatory attribute 14:48:30 the expected behavior in this case is that if SAN exists in the certificate, it should be used. most certificates will not include SAN in them 14:48:35 vjay2: Yes. And in fact, order only has relevance when there's a hostname conflict, which should be a rare situation in any case. 14:49:03 well, the UI might be able to point that out anyway 14:49:10 vjay2: The attribute was mandetory but the back end is free to ignore it. Much like SAN is now agreed to be. 14:49:29 crc32: +1 14:49:34 well, ignore or throw an exception -- not mix the two 14:49:55 crc32: is SAN mandatory? I thooght it can be empty or non-existant 14:49:59 crc32: That is what I dont like. someone could specify order if required 14:50:08 Also, is *.finance.com and abc.finance.com considered conflicting? 14:50:18 the attr is mandetory though. Meaning all requests would trigger an Error. 14:50:50 vjay2: No. abe.finance.com would take priority like PKIX says it should 14:50:56 vjay2: Yes those would be conflicting names. And yes order should be a mandatory attribute. 14:51:04 Heh! 14:51:40 conflict about whether there is conflict. meta. 14:51:52 * TrevorV_ is metatastic 14:51:59 NetScaler will be able to auto resolve this situation and choose abc.finance.com when the request comes for abc.finance.com 14:52:08 samuelbercovici: In the Radware implementation, if *.finance.com comes before abc.finance.com, does *.finance.com win in this case? 14:52:48 And by "comes before" I mean "is ordered before" 14:52:55 It should be most specific first 14:53:08 +1 14:53:17 johnsom: I agree. But I don't know whether Radware does. :) 14:54:00 so what do we follow ordering or http://tools.ietf.org/html/rfc2818? 14:54:18 well, can't we order in the api/driver so that most speciifc is first? 14:54:38 rfc2818+ 14:55:06 xgerman: Sure, but the whole reason we went with order was because it's necessary for Radware's implementation, from what I recall. 14:55:25 we could but the RFC2818 kinda mentions it's braindamage to use *.abc.com if www.abc.com is also specified. 14:56:03 it seems many people are losing connection 14:56:07 I'm fine either way. 14:56:09 sbalukoff: don't remeber by heart..the full logic was in ML. 14:56:29 I don't want to argue anymore. 14:56:36 I seem to recall that was a hard requirement. 14:56:36 +1 14:56:40 jorgem: got disconneted and can't get back in 14:56:41 the problem is lets say a user binds a cert with *.finance.com earlier than abc.finance.com then the behaviour will be different between NetScaler and the backend which honours order 14:56:44 Yeah, I just want to get this done. 14:57:24 sbalukoff: me too 14:57:39 vijay2, dougwig said that we can buy whatever LB we think is best for our use case 14:57:40 So how about this, then: We say the order can be used as a "hint" for name conflict resolution, but it's ultimately up to the driver to decide how to resolve the conflict. 14:57:44 I'm back 14:57:58 Order should be a mandatory attribute for those drivers that need it, but again, those that don't can ignore it. 14:58:12 sbalukoff: i am ok with that 14:58:13 Name conflicts are anticipated to be a rare edge case anyway. 14:58:31 vijay2: does this need to be mentioned int he spec then? 14:58:33 if name conflicts are rare 14:58:34 So yes, different drivers might have slightly different behavior when there are name conflicts. 14:58:41 does it need to be in the spec 14:58:44 svalukoff +1 14:58:51 So be it: I don't think we can do better than that, given vendor implementations are going to vary too much here. 14:59:04 vjay2: Yes. So that we have this decision documented. 14:59:08 yep, and I like things int nhe spec so we are explicit 14:59:20 What is the decision? 14:59:23 I am fine with that 14:59:41 drivers can deal with name conflicts however they choosew 14:59:52 order is a hint to help them 14:59:56 jorgem: Decision is that order is a mandatory attribute. Order is a hint for drivers to use for conflict resolution when hostnames conflict. 15:00:00 jorgem: "sbalukoff> So how about this, then: We say the order can be used as a "hint" for name conflict resolution, but it's ultimately up to the driver to decide how to resolve the conflict." 15:00:01 Awesome. I think we are out of time folks. 15:00:10 o/ 15:00:13 Also, drivers are free to use their own logic for conflict resolution and don't have to use order. 15:00:20 guys, need to evacyuate as we have a siren 15:00:22 bye 15:00:24 o/ 15:00:25 Good discussions today. 15:00:30 bye 15:00:32 samuelbercovici: shit, be safe 15:00:33 sbalukoff: Should order be an optional attr then? 15:00:34 Thanks! 15:00:35 Bye! 15:00:43 #endmeeting