Wednesday, 2014-07-23

*** xgerman has quit IRC00:06
*** vivek-ebay has quit IRC00:07
*** vivek-ebay has joined #openstack-lbaas00:07
*** vivek-ebay has quit IRC00:12
*** mlavalle has quit IRC00:43
*** crc32 has quit IRC01:01
*** fnaval has quit IRC01:04
*** crc32 has joined #openstack-lbaas01:06
*** blogan_ has joined #openstack-lbaas01:06
*** crc32 has quit IRC01:07
*** fnaval has joined #openstack-lbaas01:11
*** dlundquist has quit IRC01:21
*** sbfox has joined #openstack-lbaas01:30
*** TrevorV has quit IRC01:34
*** TrevorV has joined #openstack-lbaas01:35
*** woodster__ has quit IRC01:35
*** sbfox has quit IRC01:44
*** fnaval has quit IRC01:51
*** sbfox has joined #openstack-lbaas01:57
*** fnaval has joined #openstack-lbaas02:15
*** sbfox has quit IRC02:36
*** fnaval has quit IRC03:30
*** ptoohill_ has joined #openstack-lbaas04:07
*** ptoohill_ has quit IRC04:07
*** ptoohill_ has joined #openstack-lbaas04:08
*** HenryG is now known as HenryG_afk04:19
*** sbfox has joined #openstack-lbaas05:50
*** evgenyf has joined #openstack-lbaas06:45
*** sbfox has quit IRC08:01
*** ptoohil__ has joined #openstack-lbaas08:07
*** ptoohill_ has quit IRC08:07
*** ptoohil__ has quit IRC08:12
*** evgenyf has quit IRC08:27
*** evgenyf has joined #openstack-lbaas08:40
*** [1]evgenyf has joined #openstack-lbaas08:53
*** evgenyf has quit IRC08:55
*** [1]evgenyf is now known as evgenyf08:55
*** neeraj has joined #openstack-lbaas08:55
neerajhello all08:55
neeraji am doing a project, in which i need to write load balancer code08:56
neerajwhich file of openstack neutron i should look upon?08:56
*** neeraj has quit IRC09:08
*** evgenyf has quit IRC09:27
*** evgenyf has joined #openstack-lbaas09:41
*** woodster__ has joined #openstack-lbaas12:58
*** fnaval has joined #openstack-lbaas13:11
*** HenryG_afk is now known as HenryG13:24
*** fnaval has quit IRC13:41
*** markmcclain has joined #openstack-lbaas13:56
*** rolledback has joined #openstack-lbaas13:58
*** fnaval has joined #openstack-lbaas14:17
*** ptoohill_ has joined #openstack-lbaas14:30
ptoohillneeraj, there's quite a few files that makes up the load balancer code. Is there a particular fix thats needed?14:30
*** rolledback has quit IRC14:56
*** rolledback has joined #openstack-lbaas15:00
*** blogan_ has quit IRC15:06
*** sballe has joined #openstack-lbaas15:08
*** mlavalle has joined #openstack-lbaas15:13
*** TrevorV_ has joined #openstack-lbaas15:15
*** sbfox has joined #openstack-lbaas15:23
*** sbfox has quit IRC15:26
*** blogan_ has joined #openstack-lbaas15:30
*** blogan_ has quit IRC15:32
sballedougwig, blogan: Is it worth doing a retrospective on what we learned so far about the openstack process? I am especially interested in this BP deadline. What can we do better for the K release.15:33
sballeShould we add that to the agenda for tomororw's meeting or is it better to discuss this at the summit on we have the whole picture of what we learned from Juno15:34
dougwiga talk about the frustration of attempting to join a group with 300 unwritten rules, all of which are too tiny for anyone to remember to write down after they've learned them, you mean?  oh, and the rules change daily.  :)16:21
*** rolledback has quit IRC16:23
*** sbalukoff has quit IRC16:26
blogandougwig you sound a bit....frustrated16:27
dougwignah, it's just a larger hurdle than most anyone wants to admit.16:28
*** xgerman has joined #openstack-lbaas16:29
dougwigblogan: is the top-level review stable yet?  it's been changing daily; i want to start pushing to get these in, but not until it stabilizes.16:31
bloganyou mean the extension?16:33
bloganor the first 3?16:33
dougwigthe extension16:33
blogani think its stable but i'm sure someone will come in with somethign to fix16:34
*** sbfox has joined #openstack-lbaas16:35
*** sbfox has quit IRC16:35
*** sbfox has joined #openstack-lbaas16:39
*** jorgem has joined #openstack-lbaas16:42
dougwigblogan: might want to double-check your reviews; it just got another ripple up rebase.16:50
*** rolledback has joined #openstack-lbaas16:56
*** evgenyf has quit IRC17:00
*** sbfox1 has joined #openstack-lbaas17:33
*** sbalukoff has joined #openstack-lbaas17:34
*** sbfox has quit IRC17:35
*** sbfox has joined #openstack-lbaas17:43
*** sbfox1 has quit IRC17:46
*** sbfox has quit IRC17:49
*** sbfox has joined #openstack-lbaas17:49
*** sballe has quit IRC17:50
*** vivek-ebay has joined #openstack-lbaas17:50
*** fnaval has quit IRC17:53
blogandougwig: i think evgenyf did the same thing you did yesterday18:04
bloganor the day before18:04
*** rohara has quit IRC18:10
*** sbfox has quit IRC18:13
*** sbfox has joined #openstack-lbaas18:18
*** sbfox has quit IRC18:23
dougwigyep, that's why i warned you.18:27
bloganoh ha i didnt see your message18:34
blogani wonder how that happens18:35
dougwigi did a git rebase on my dependent review branch, then merged it into my topic branch, then git review, and "magic" happened.  now i re-clone, git review -d, cherry-pick into topic, git review.18:44
*** crc32 has joined #openstack-lbaas19:02
*** crc32 has quit IRC19:03
*** crc32 has joined #openstack-lbaas19:04
*** sbfox has joined #openstack-lbaas19:10
*** sbfox has quit IRC19:10
*** sbfox has joined #openstack-lbaas19:11
*** vivek-ebay has quit IRC19:20
*** vivek-ebay has joined #openstack-lbaas19:24
dougwiganything for today's hangout, or cancel?19:30
blogani think we decided to cancel until v2 is done19:32
bloganor in a relatively good review state19:32
*** fnaval has joined #openstack-lbaas19:33
*** mestery has quit IRC19:34
*** mestery has joined #openstack-lbaas19:34
*** vivek-ebay has quit IRC19:43
*** rohara has joined #openstack-lbaas20:09
*** vivek-ebay has joined #openstack-lbaas20:18
*** TrevorV_ has quit IRC20:27
*** sbfox has quit IRC20:31
*** vivek-ebay has quit IRC20:39
*** vivek-ebay has joined #openstack-lbaas20:41
*** vivek-ebay has quit IRC20:42
*** vivek-ebay has joined #openstack-lbaas20:43
*** sbfox has joined #openstack-lbaas20:50
*** rolledback has quit IRC20:54
*** Youcef has quit IRC21:01
*** markmcclain1 has joined #openstack-lbaas21:27
rm_workHey guys, who is around that could discuss the Barbican/TLS interaction for a few minutes? sbalukoff crc32 xgerman dougwig blogan enikanorov21:27
dougwigi'm here21:28
*** markmcclain has quit IRC21:28
rm_workSo, I've got all this stuff now around Consumer Registration in Barbican, but… I am not 100% confident that it is actually necessary21:28
rm_workTalking to woodster__ the other day got me thinking21:29
rm_workThe real problem we were trying to solve was that a user's Cert/Key needed to be stored in a way we could access and we didn't want to lose track of it21:29
rm_workbut… every backend that exists right now will store the cert/key in their own way once we pass it to them, right?21:29
rm_workand once Octavia gets going, it will be storing the certs/keys in Barbican but under its own user account21:30
rm_workis there a reason the API will even need to store what Barbican container the cert/key came from?21:30
dougwigthe primary use case that i was thinking of was horizon (or its equivalents) providing an easy way in the cert store interface to put front-and-center all the places that are using said cert, to help the user not screw themselves over.21:30
rm_workwould the user be ABLE to screw themselves over?21:31
*** sballe has joined #openstack-lbaas21:31
xgermandougwig +121:31
rm_workeven if they deleted the cert from barbican, would neutron-lbaas *ever* need to access the cert again after the initial read?21:31
xgermanwell, as a suer it is confusing if you delete something from bbq and it lives on as a zombie21:31
rm_workall of the backends (including Octavia) would store the cert/key on their own terms21:31
crc32rm_work: The initial requirement is we needed a secure repository to store the keys in. Which every one agreed barbican was. :|21:31
rm_workfor *Octavia*21:32
dougwigmaybe it's just me, but i'd envision a future where you could store just a barb id, and not copy the cert/key.  and to get to that point, i'd want to know who was using things.  all the duplication is a maintenance and security headache.21:32
xgermanit was debated that the lb should check during failover if the cert is still there21:32
xgermandougwig +121:32
dougwigas in, octavia on process start would go fetch its key, and not store it anywhere.21:32
xgermanwell, there was the sue case we need to fail ove rand the cert is gone21:33
xgermanthat was the only case we said we might want to store the cert somewhere else21:33
xgerman(+ carlos had some for caching/spped)21:33
dougwigxgerman: right, but if you warn the user where it's used, and it's deleted anyway, the window for accidental failure versus purposeful is sufficiently tiny that i think you can not worry about it.21:33
xgermanyep, i completely agree. I like a single source of truth more then thousand copies we need to kjeep in sync21:34
rm_workright, but none of the backends at this point use Barbican21:35
rm_workas far as I'm aware21:35
rm_workthey all have their own storage mechanisms21:35
rm_workand Octavia will use Barbican, but for this purpose it could be considered a black box21:35
rm_workthe actual neutron-lbaas API receives a request to enable TLS with a barbican containerID for cert/key21:36
rm_workbut once we read the cert/key and pass it to the driver, the driver stores and utilizes that info21:36
rm_workand the neutron-lbaas API itself won't ever be queried again for that info21:36
dougwigchicken/egg.  you'll never get to a single source of truth if you don't include the primitives for people to start using it as such.  you can't wait for the backends to evolve there on their own.21:36
rm_workor at least that is my current understanding21:36
crc32if we don't store the TLS container ids then when ever a cert is being added to an LB we have to reparse the other X509s associated with that loadbalancer for SNI hostname mangling. etc etc. Plus its awkward for a customer to not have a way to retrieve the certificate associated with their loadbalancer.21:36
xgermanyep, but me the user goes in sees all those certificates and wonders where they are used21:37
rm_workAH, so for updates, we need to pass down every cert/key again?21:37
rm_workor at least, we have to assume that it's necessary to do so?21:37
rm_workwhether or not every backend requires that or not21:37
crc32rm_work: yes we need to send a new cert at the minimum the key may be the same though.21:38
*** markmcclain1 has quit IRC21:38
dougwigas a driver author, i'd like to get both the barb id and the cert/key, and i'll pick which i use.  the re-parsing issue i'm not worried about, because provisioning changes are still relatively rare events in comparison to actual load balancing.21:38
crc32xgerman: by not maintaining an association your making this hard to trouble shoot when a user does come back with a matching x509 issue.21:39
rm_workok, so the consensus is we *do* still want to track the cert/key in the neutron-lbaas API?21:40
xgermanyes. I think that would be best21:40
dougwigrm_work: does octavia, or any vendor, need the registration in the short-term?  no, we're all going to copy.  to me, it's purely a long-term thing, with the hope that we stop copying over time.21:40
rm_workI just needed confirmation before I kept rolling ahead with this, as I was starting to be convinced it wasn't strictly necessary21:40
crc32I'm a fan of Single source of truth but I still have to balance it against trouble shooting and RestApi to RestApi performance lag.21:41
rm_workdougwig: i mean, do you forsee a time when A10's appliance actually uses barbican to pull cert/key instead of storing it on its own? :P21:41
xgermanno worries. The main goal is to get to our pie in the sky that BBQ automatically renews the cert, notifies us, we reload it autmagically or ask the nuser to press oem button21:41
crc32rm_work: Not strictly needed unless you need to start trouble shooting what cert and key actually made it to the backend.21:41
dougwigif it's more secure, or provides a better user experience, or if some big customer just wants it centralized, absolutely.21:42
vivek-ebayjust came back21:43
vivek-ebaycatching up21:43
dougwiglet me flip that around; it's easy to see why a big iron appliance with 500k LB's on it would store its own certs.  but if you have a huge pile of soft appliances running those LBs instead...21:43
dougwigthe management/maintenance implies treating the soft appliances like cattle, which means the more you push out of them, the better.21:44
xgermanagreed. Octavia has no need to store the cert locally21:45
xgermanand use the annotated one21:45
rm_workso we'd actually pass through the original Barbican ID to Octavia?21:46
rm_workand Octavia wouldn't store its own copy?21:46
xgermanthat would be a possibility21:46
dougwigthe less state that octavia stores, the easier it will be to shuttle LB's around the nova infrastructure.21:46
xgermanand it hasn't been decided how failover will work -- which is the only use case of storing a copy21:47
dougwignot to mention, it's one less security headache; no sense doing PKI in multiple places if you can just let barbican handle it.21:47
rm_workmy understanding was that: A) The driver interface would only send the raw cert/key, after having decoupled them from Barbican; B) There was not a way for the backends to query upwards to the API layer at a later time to retrieve information21:47
rm_workThinking about it, possibly B is very misinformed21:47
dougwigthe backends are getting the sqlalchemy objects. we can absolutely go uphill.  :)21:48
rm_workactually, was B the thing we were specifically trying to enforce so that the backends don't all store their own truths21:48
rm_workIE, we WANT the backends to query uphill to the API as much as possible for info instead of storing it?21:48
bloganyes and that is what the driver does should it need to get an updated entity21:49
bloganwell what the drivers should do21:49
dougwigi'm going to flip that around again, and say that i want to avoid cache coherency and security problems unless i can't avoid dealing with them.21:49
rm_workok so, different question now -- if we are not going to store a copy of the cert/key on our own in Octavia, would the lack of "hard-delete-prevention" be an issue?21:49
dougwigwhich implies, it'd be awesome if barbican was the source of truth (if it's fast enough for operational concerns.)21:49
rm_worksince we agreed on the ML in the past that storing our own copy of the key in Barbican would be the accepted solution to the problem of the user possibly deleting keys21:50
dougwigif the UI warns the user, which it can do so with registration (a soft lock, as it were), then you've stopped the stupid user case.  the api is power users, so they get what they ask for.  so no, i think it's avoided in the ui.21:50
xgermandougwig +121:50
rm_workso we don't want a hard insurance policy21:50
dougwigi want it, i just didn't get it.  and i can live with a ui-only version of same.21:51
xgermanyep, and the really case where it was trouble was failover which might be solved differently (e.g. having two lbs configured and running)21:51
crc32no the user could delete their original copy of the TLS container thus breaking future lb fail over type breaks.21:52
dougwigright ,but if they do that, even seeing their cert is in use, it's their problem.  we can't protect everyone from everything.21:53
crc32dougwig: I don't want to relie on UI usage. Straight API users can still hurt them selves.21:54
dougwigIMO - API users are power users; they get what they ask for.21:55
rm_workis "sorry, you did this to yourself" a good thing to be saying on the phone to a customer who just had their site unavailable for some duration because we had a failover event in our datacenter?21:55
rm_workeven if it's a power user? :/21:55
rm_workthat's not "Fanatical" :P21:55
xgermanwell, as I said we can debate that when we do failover21:56
xgermanfailove rmight not involve getting the cert from BBQ21:56
*** fnaval has quit IRC21:56
xgermanbecause we have 10 LBs running ready to go21:56
dougwiglike i said, i'd prefer a force flag.  you could always put a layer in front that turned the registrations into an api warning.21:58
dougwigbut without it, there is a path forward without copying, i think21:58
dougwighow many phone calls do you get from people that misuse the API and are surprised when it breaks something?  :)21:59
dougwig(law of averages means probably many.)21:59
crc32I don't like the possability of configuring fail over on a loadbalancer. Then delete some randon TLS id from barbican then poof lb failover is broken. That just sucks.21:59
rm_workwell, I guess this discussion can be tabled and handled during Octavia design/dev22:00
xgermanyeah, but that is a different discussion than this one22:00
rm_workthe original question I had is already answered -- yes, we want to keep the customer's barbican container ID, and yes, we want to register for them22:00
rm_workso thanks for the arguments :P22:00
rm_workI'm re-convinced it's necessary22:01
crc32xgerman: If we make a decision now then the debate won't even be available when we actually talk about failover. This kinda needs to be finialized early.22:01
rm_workwas starting to feel like I wasted the last month22:01
xgermancrc32 I don't think having registration precludes us from decising later that we need to store a copy somewhere22:03
xgermanfor all we know we might offer lbs which don't fail over22:03
*** fnaval has joined #openstack-lbaas22:03
dougwigrm_work: i think what you're doing is quite important for more than lbaas, in the long-term.  that's just my opinion.  without it, we couldn't even have the follow-on conversation.22:03
dougwig(IMO, of course.)22:04
rm_workone more opinion question -- Consumer Registration right now takes a "Service Name" and a "Reference URL"22:05
rm_workwhich for example would be "LBaaS" and "<uuid>"22:05
rm_workok but the question is (sorry, got distracted IRL)22:07
rm_workHow many characters do we allow for each22:07
rm_workI have the "Name" at varchar(36) right now22:08
rm_workand "URL" at varchar(120)22:08
rm_workI had them much larger22:08
rm_workI am running into MySQL Constraint constrains (lol)22:08
rm_work*Constraint constaints22:08
rm_workdamnit i can't type22:08
rm_work*Constraint constraints22:08
rm_workso my concern with the URL being *only* 120 characters is22:09
xgermanwell, if it can't be done22:10
xgermanthough I like longer urls and info like which lb it is22:10
rm_workhttps:// (8) + (x) + /v2.0/lbaas/listeners/C57BD591-25CB-44A6-8E2A-1E2FE99C225D (58)22:10
rm_workleaves x = 5422:11
rm_workalso, the 58 characters is just for LBaaS22:11
rm_workother people's URLs could be longer, like if they use multiple uuids22:11
*** ptoohill_ has quit IRC22:13
*** sballe has quit IRC22:14
xgermanyeah, wonder why we can't have 255 or more...22:16
dougwigDNS max hostname len is 1024, so 120 is already flirting with death22:16
dougwigi would think 256/1500 would be enough, if you can get away with it.22:17
rm_workyeaaaah so22:22
rm_workMySQL Constrains have a limit of 790 or so BYTES22:23
rm_workwhich equates to slightly less than 200 characters22:23
rm_worksince max bytesize per char is 422:23
rm_workvarchar(198) is the max for a constraint22:23
rm_workfunnily enough, this is not a problem in sqlite >_>22:24
*** sballe has joined #openstack-lbaas22:24
rm_workso one solution is to add another column which is a hash of Name+URL22:25
rm_workand just put the constraint on that <_<22:25
xgermancan't you just use a tinyblob or so?22:25
rm_workI guess maybe that's the way to go?22:25
xgermando we really need to index that field?22:26
rm_workit's a unique constraint22:26
xgermanoh, ok22:26
xgermanit's per listener so that's ok22:26
rm_workYeah I think I'll probably ask the Barbican folks first but seems like the hash is maybe the best bet22:27
dougwigit's a low volume API, so you could also just do a get/set instead of a constraint.22:41
dougwig(in a transaction)22:41
*** mlavalle has quit IRC22:58
*** sballe has quit IRC23:04
*** fnaval has quit IRC23:32
rm_workdougwig: right, atomicity was my concern with that23:34
rm_workthough I think that's pretty ugly anyway23:34
rm_work(I did consider it)23:34
dougwigwrap the get/write in a transaction and it's still atomic, without the constraint.  i don't know if our libraries make that easy or not.23:35
dougwiguglier than tiny fields?  :)23:35
rm_worki'm going with hashing23:35
rm_workContainerID+ServiceName+URL -> Hash -> Store in hash_field, put constraint on hash_field23:35
rm_workdougwig: chance of collision should be low… :/23:36
rm_worklike, "i might be more likely to be hit by a comet. while playing volleyball. on a thursday." low23:36
dougwigprobably more likely than that, given api retries/multiple api servers.23:37
*** sbfox has quit IRC23:42
sbalukoffHi guys. Just got caught up on the conversation. I agree that hash seems to make sense to handle the unique constraint.23:46
*** ptoohill_ has joined #openstack-lbaas23:46
*** ptoohill_ has quit IRC23:46
sbalukoffAlso regarding Octavia and storge of TLS certificates:23:46
sbalukoffI'm not a fan of anything which will create a ticking timebomb later.23:46
*** ptoohill_ has joined #openstack-lbaas23:46
sbalukoffThose always end up being the most annoying support cases to solve, so let's not do that.23:46
sbalukoffIf Octavia doesn't have its own store to ensure that failovers (and scaling, when we get there) don't break when a user (stupidly) deletes a key that's in use, then we should force all services using the TLS container to break at the time when the user does a force delete.23:48
sbalukoffI don't see barbican wanting to support those kinds of call-outs on delete...23:48
dougwigsbalukoff: on the spectrum of, 1) users are guaranteed to shoot themselves, and A) users can never shoot themselves, the registration+UI gets you 99% of the way to A), which is why, to me, it starts to become an acceptable tradeoff to not over-complicate for that last 1% (numbers made up out of thin air.)23:48
sbalukoffSo, for now I'm in favor of Octavia using its own (redundant) store for these secrets (which will be barbican, but on an 'octavia system user' account instead of the user's account)...23:50
sbalukoffdougwig: I'm thinking of the difficulty troubleshooting for the operator when the user shoots himself in the foot (which he absolutely will, even with a warning)23:50
sbalukoffI agree that registration is a huge step in the right direction.23:51
dougwigit's not a complete solution, agreed.  i question how much effort to put in, given that barbican is adding an event stream, probably in the same timeframe that octavia will land.23:52
sbalukoffIf we get the event stream, then that will solve the ticking timebomb problem.23:52
sbalukoffAnd, really, we can cross this bridge when we're actually implementing Octavia.23:53
dougwigyep, hook delete, that's the "right" fix.23:53
sbalukoffIt might be worthwhile to have the code in there for Octavia to have its own secret store: I can see some operators still wanting to have it even with a delete hook.23:54
sbalukoff(Mostly, large private organizations who occasionally employ stupid application developers. ;) )23:54
dougwighence, IMO, no.  :)23:55
dougwig(says the guy with his own secret store.)23:55
sbalukoffHe forgot to mention off-by-one errors.23:55
sbalukoffI think it should be a config option to enable/disable a separate secret store for Octavia.23:56
*** sbfox has joined #openstack-lbaas23:56

Generated by 2.14.0 by Marius Gedminas - find it at!