Monday, 2018-08-27

« Sunday, 2018-08-26 Index Tuesday, 2018-08-28 »
*** abaindur has quit IRC00:30
*** hongbin has joined #openstack-lbaas01:03
*** sapd1 has joined #openstack-lbaas01:56
sapd1Does anyone know Vietnam Openinfra Days 2018?01:57
*** ramishra has joined #openstack-lbaas02:09
*** ramishra has quit IRC03:00
*** ramishra has joined #openstack-lbaas03:08
*** hongbin has quit IRC05:27
*** numans has joined #openstack-lbaas05:29
*** reedipb has joined #openstack-lbaas05:39
*** sapd1 has quit IRC06:00
*** pcaruana has joined #openstack-lbaas06:49
*** velizarx has joined #openstack-lbaas06:58
*** dmellado has quit IRC07:00
*** dmellado has joined #openstack-lbaas07:02
*** sapd1 has joined #openstack-lbaas07:04
*** velizarx has quit IRC07:24
*** rcernin has quit IRC07:31
*** velizarx has joined #openstack-lbaas07:36
*** dolly has quit IRC08:04
*** celebdor has joined #openstack-lbaas08:07
*** yboaron has joined #openstack-lbaas08:15
openstackgerritYang JianFeng proposed openstack/octavia master: Add quota support to octavia's l7policy and l7rule  https://review.openstack.org/59062008:29
*** celebdor has quit IRC08:34
*** celebdor has joined #openstack-lbaas08:35
openstackgerritYang JianFeng proposed openstack/octavia master: [WIP] Refactor 'check_quota_met' and 'decrement_quota'  https://review.openstack.org/59666508:38
openstackgerritYang JianFeng proposed openstack/octavia master: [WIP] Refactor 'check_quota_met' and 'decrement_quota'  https://review.openstack.org/59666508:44
*** velizarx has quit IRC09:03
*** velizarx has joined #openstack-lbaas09:09
*** ataraday has joined #openstack-lbaas09:20
*** sapd1 has quit IRC09:26
openstackgerritYang JianFeng proposed openstack/octavia master: [WIP] Refactor 'check_quota_met' and 'decrement_quota'  https://review.openstack.org/59666509:35
ataradayHello everyone! I reviewed this change https://review.openstack.org/#/c/587400/ and saw that there is no ModelMigrationtest in Octavia https://docs.openstack.org/neutron/pike/contributor/testing/template_model_sync_test.html which allows to run migrations against different backends, so it will help to avoid errors in migrations. Do you have any plans on implementing it? neutron-lbaas had it.09:41
*** sapd1 has joined #openstack-lbaas10:01
*** dolly has joined #openstack-lbaas10:13
sapd1johnsom: Which tool you are using to benchmark Octavia Performance?10:15
nmagnezisapd1, https://github.com/johnsom/stressoctaviaapi10:22
sapd1nmagnezi: I mean, Test Performance of Octavia amphora. :D (HaProxy)10:23
*** celebdor has quit IRC10:26
*** pcaruana has quit IRC10:32
*** pcaruana has joined #openstack-lbaas10:32
cgoncalvesdayou, hi. please see my comment in https://review.openstack.org/#/c/595775/ and if you agree please remove +W10:46
*** ataraday has quit IRC10:47
*** celebdor has joined #openstack-lbaas10:51
*** celebdor has quit IRC11:25
openstackgerritYang JianFeng proposed openstack/octavia master: [WIP] Refactor 'check_quota_met' and 'decrement_quota'  https://review.openstack.org/59666511:42
*** rtjure has quit IRC11:44
*** rtjure has joined #openstack-lbaas11:58
*** ktibi has joined #openstack-lbaas12:10
*** celebdor has joined #openstack-lbaas12:37
*** celebdor has quit IRC12:38
*** celebdor has joined #openstack-lbaas12:38
*** rtjure has quit IRC12:43
*** ramishra has quit IRC12:46
*** rtjure has joined #openstack-lbaas12:48
*** amuller has joined #openstack-lbaas12:52
*** yboaron_ has joined #openstack-lbaas13:31
*** yboaron has quit IRC13:34
*** hvhaugwitz has quit IRC13:37
*** hvhaugwitz has joined #openstack-lbaas13:39
*** yboaron_ has quit IRC14:14
johnsomsapd1: I use Tsung and iPerf314:14
*** yboaron_ has joined #openstack-lbaas14:16
*** ramishra has joined #openstack-lbaas14:30
*** ianychoi has quit IRC14:36
*** ianychoi has joined #openstack-lbaas14:42
*** yboaron_ has quit IRC14:48
johnsomcgoncalves I was going to let the zuul yaml thing go on the old branches, but it would be a quick patch to split them into the same format we use for the other branches.....  grin15:14
cgoncalvesjohnsom, yeah. I've just lifted my -1. up to cores :)15:14
johnsomSo you aren't taking the bait and fixing it?  grin15:15
cgoncalvesdidn't you have enough of fishing last Friday? :P15:16
cgoncalvesk, I'll do it15:16
johnsomGot cancelled due to weather sadly.15:16
xgerman_:-(15:17
xgerman_Going through my patches - it seems nothing got merged while I was on vacation15:17
cgoncalvesfixed15:17
xgerman_https://review.openstack.org/#/c/539350/ -- will that still be in R15:18
xgerman_?15:18
johnsomR shipped15:18
xgerman_cgoncalves:  and I need to present something - and I don't like to start pull this PR and15:18
xgerman_ok, then this will be our presentation15:18
xgerman_fortunately we don;t have a limit on backport so I can throw https://review.openstack.org/#/c/587505/ into Pike...15:19
cgoncalvesxgerman_, summit presentation?15:20
xgerman_yep, in order to migrate you might want to run the proxy15:21
xgerman_well, we can maybe backport to older version since it's a gate... not sure if that's ok15:24
dayousigh, just back to computer after a babysitting day15:24
xgerman_it's the first day of school for my kids...15:24
xgerman_so likely will throw a party in a bit :-)15:25
johnsomlol15:25
dayou:-) Lucky man, I should learn cooking, actually I am learning, so my wife won't bother with that15:26
*** velizarx has quit IRC15:27
xgerman_cooking: open package, put in microwave, follow instructions?15:27
dayouNow I make fruit juices for my kids and my wife everyday15:28
cgoncalvesxgerman_, it is not gate stuff but I agree with the relevance of backporting it to queens at least15:29
*** pcaruana has quit IRC15:29
xgerman_yeah, would love to have the gate on R so we can recommend something gated15:29
cgoncalveshaving it in queens too would be nice as we deprecated nlbaas in queens15:31
xgerman_agreed... let's get it merged and then we can backport.15:35
jitekaHello, I've got a question regarding LB building time and spare pool15:44
jitekaFrom what I read on documentation, using spare pool doesn't guarantee anti-affinity server group in case of failover15:44
jitekaIs it something that is currently looked at to have that feature with server group supported ?15:44
jitekaI'm trying to reduce LB creation time that is currently around 3min15:44
jitekaI'm also reviewing our current image cache strategy to see if amphora VM build time could be reduce as it seems to be the principal bottleneck here15:46
xgerman_jiteka: our problem is nova not being able to "migrate" vms accordingly. There is a multi AZ patch though...15:55
*** ktibi_ has joined #openstack-lbaas16:00
*** ktibi has quit IRC16:02
*** pcaruana has joined #openstack-lbaas16:26
*** celebdor has quit IRC17:02
*** velizarx has joined #openstack-lbaas17:02
*** openstackgerrit has quit IRC17:04
*** dayou has quit IRC17:21
*** dayou has joined #openstack-lbaas17:22
*** openstackgerrit has joined #openstack-lbaas17:23
openstackgerritGerman Eichberger proposed openstack/octavia master: Delete zombie amphora when detected  https://review.openstack.org/58750517:23
*** velizarx has quit IRC17:27
johnsomjiteka 3 minutes for a LB build seems long, mine are less than a minute.  We currently use the nova anti-affinity server groups, but we can't use a spares pool with it as we can't add pre-booted instances to the server group.17:39
*** sapd1_ has joined #openstack-lbaas17:45
colin-what flavor typically? just curious17:47
colin-@johnsom17:47
johnsomcolin- Which nova flavor?  1GB RAM, 1vCPU, 2GB disk17:48
colin-got it, apples to apples in that case17:48
johnsomYeah, with the current version of HAProxy and disabled flow logs, you don't get much benefit from giving it more resources.  I think that will change in Stein forward though.17:49
openstackgerritMichael Johnson proposed openstack/octavia master: Add amphora statistics to the admin API  https://review.openstack.org/58503117:51
sapd1_I shared this architecture to my community in Vietnam Openinfra Days 2018 (on August 25) https://docs.google.com/presentation/d/17b_r2flQn2mVOu2Ybdlxnb1gJuQqqE3cQfvtv0vTEhw/edit?usp=sharing17:55
sapd1_:D17:55
sapd1_johnsom: nmagnezi Take your time to check my slides :D17:56
johnsomsapd1_ Interesting.  Thanks for spreading word of Octavia.18:03
sapd1_I want Octavia has  more features than this :D18:04
johnsomsapd1_ Are you planning to upstream this work?18:05
sapd1_johnsom: I added nameserver to /etc/resolv.conf but I can't resolve domain.18:06
sapd1_in amphora18:06
johnsomI have been thinking about gnocchi as well18:06
johnsomsapd1_ Yes, we have done a number of things to disable DNS in the amphora18:06
sapd1_johnsom: I would like. But I have to complete my working in my company.18:07
sapd1_use gnocchi to store metric and monasca to store log. :D18:07
johnsomsapd1_ If you must have DNS, this is the element to disable: https://github.com/openstack/octavia/blob/master/elements/no-resolvconf/finalise.d/99-disable-resolv-conf18:07
johnsomBut it can slow down some operations (boot, etc.) having it enabled18:08
sapd1_johnsom:  visualize by grafana: https://imgur.com/a/B7ydth818:09
sapd1_I'm trying to collect from multi-process.18:09
johnsomYeah, nice graphs18:10
sapd1_johnsom: Ok, I will enable on running amphora.18:11
sapd1_It's working now :D18:12
sapd1_johnsom: I need finish boot from volume feature first. :(18:14
*** sapd1_ has quit IRC18:16
*** fnaval has joined #openstack-lbaas18:20
*** celebdor has joined #openstack-lbaas18:38
*** abaindur has joined #openstack-lbaas18:40
abaindurjohnsom: or rm_work: hey, discussing from last week...18:47
abaindurYou mentioned that in [certificates] section, "ca_certificate = server CA's "server.pem" file. Which file is this exactly? Perhaps my understanding of TLS/self-signed certs isn't complete, but as I understand, there is the CA's private key, and the CA's cert. From here a user can generate their own private key, then use the CA's cer to generate a client cert.18:47
abaindurThis is what I see the create_certificates.sh script creating: First private key for CA (private/cakey.pem), then the CA cert (ca_01.pem). It then generates a client key and client cert (client.pem), using the CA's cert to do so.18:48
abaindurSo in this case, what is 'ca_certificate = server CA's "server.pem"'? Is it not ca_01.pem, the CA's cert?18:48
abaindurForget about the whole client CA side of things, client_ca and client_cert, because I understand those (from a separate run of create_certificates.sh, i set client_ca = ca_01.pem, and client_cert = client.pem).18:50
abaindurso basiclly... i'm still confused as to the difference between ca_certificate and server_ca. One is used to generate the amphora certs, the other is used to validate them. Seems like exactly the same thing from my basic knowledge of certs... wouldnt they both be ca_01.pem generated by the script?18:54
abaindurdevstack and other online examples set them to exact same thing as well18:55
johnsomGive me 5 minutes or so, then I can chat for a bit18:55
abaindurnp18:55
johnsomok, reading19:03
johnsomLet me try re-running that script and look at what I puts out.  I didn't write it and it is auto-magic for me most of the time, so need to refresh my memory19:07
abaindur[root@arjun octcerts2]# ls19:09
abaindurca_01.pem  client.csr  client.key  client-.pem  client.pem  index.txt  index.txt.attr  index.txt.old  newcerts  private  serial  serial.old19:09
abaindur[root@arjun octcerts2]# ls private/19:09
abaindurcakey.pem19:09
abaindurclient.pem is a certificate generated using the ca_01.pem, which is the x509 CA cert. and private/cakey.pem is the CA's private key19:10
abaindurfor server side of things, i think client.pem can be ignored - since the amp certificates are generated by octavia code and injected into the VM. To do this, "ca_certificate" is used here: https://github.com/openstack/octavia/blob/stable/queens/octavia/certificates/generator/local.py#L10219:13
*** pcaruana has quit IRC19:13
johnsomWell, let's split things into two parts, client and server.19:15
johnsomClient:19:15
johnsomthis is the mechanism the controllers uses to identify themselves to the amphora-agent.19:16
abaindurclien_cert and client_ca... yep19:16
johnsomI create a directory "client_ca" and run the script in that directory to create the "client side" certs/keys.19:16
abaindurYep did that. Ran script twice into separate output folders to simulate a different server and client CAs19:17
johnsomeach controller will need a "client.pem" file which has the cert and key appended to each other.19:17
abainduryes, Thats the client.pem file generated by that script. We set that to client_cert.19:18
johnsom[haproxy_amphora] client_cert  points to this "client.pem" file19:18
johnsom[controller_worker] client_ca points to the "ca_01.pem" file from this directory19:18
abaindurYes, we have that. as mentioned the "Client" side of things is self explanatory19:19
johnsomOk, just working through it.19:19
johnsomServer side19:19
johnsomin the server directory, I run the same script19:19
johnsom[haproxy_amphora]  server_cert  points to the "ca_01.pem"19:20
abaindurthats not an option19:21
abaindurdid you mean, server_ca ?19:21
johnsomYes, I did, sorry19:21
abaindurhttps://github.com/openstack/octavia/blob/stable/queens/octavia/common/config.py#L28519:21
abaindurkk19:22
johnsomcut/paste error19:22
abaindurok yep... server_ca == /root/server/ca_01.pem19:22
johnsom[certificates] ca_private_key = private/cakey.pem19:23
johnsom[certificates] ca_certificate = ca_01.pem19:23
johnsomI think.19:23
abaindurright sooo thats what i had asked first. server_ca is the exact same as ca_certificate?19:24
abaindurOne is used to sign the amph server certs... the other is used to verify. They both seem to be ca_01.pem19:25
johnsomYes, I think it is in this case. If not it's ca_certificate = client.pem and ca_private_key = client.key, but I don't think that is the right answer.19:25
johnsomRight19:25
abainduryea it doesnt seem to be client. here client is generated/signed *by* the CA cert. and octavia generates amp client certs separately in the code19:25
johnsomYeah, I think that would maker certs that have an intermediate CA, which we don't need here19:26
abaindurjohnsom: ok great so then seems we are in agreement. i was confused for longest time as to the difference between the two19:26
johnsomThis is certainly an area that could use improvement in Octavia.19:27
johnsomAnd the darn detailed install guide that I never get time to work on....19:27
abaindurWhen the amphora certs expire (the server certs), Octavia will re-gen and take care of it all automatically, right?19:27
abaindurOnly thing we need to monitor for and re-gen, would be the "Client" side of things - the clien_cert and client CA19:28
johnsomYes, the housekeeping process has a thread that rotates those19:28
abaindurAnd to do this, we need to basically admin shut the LB mgmt ports, to trigger a failover?19:28
johnsomRight, there are some issues  with rotating the controller side that I think are documented in the maintenance guide19:28
johnsomhttps://docs.openstack.org/octavia/latest/admin/guides/operator-maintenance.html19:29
*** celebdor1 has joined #openstack-lbaas19:29
johnsomhttps://docs.openstack.org/octavia/latest/admin/guides/operator-maintenance.html#rotating-cryptographic-certificates19:29
johnsomDepending on the version of Octavia you deployed, there are also "failover" APIs to trigger safe failovers19:30
johnsomhttps://developer.openstack.org/api-ref/load-balancer/v2/index.html#failover-a-load-balancer and https://developer.openstack.org/api-ref/load-balancer/v2/index.html#failover-amphora19:30
*** celebdor has quit IRC19:31
johnsomI would like to have an "update amphora config" API, but no one has had time to contribute that either.19:33
*** ramishra has quit IRC19:38
*** amuller has quit IRC19:45
*** celebdor1 has quit IRC19:50
abaindurwhats the difference between failover of a LB vs amphora?20:27
abaindurjohnsom: or would the diff only be when you have amphora in active active or standby configs? If SINGLE topology, is it fundamentally the same?20:27
abaindurI guess the opeator maintenace doc suggests shutting the mgmt port. Is that a deprecated method now and the APIs above are preferred? https://docs.openstack.org/octavia/queens/admin/guides/operator-maintenance.html#rotate-an-amphora20:31
johnsomCorrect, if load balancer failover it sequences the active and standby failovers for you20:32
*** ktibi_ has quit IRC20:32
openstackgerritGerman Eichberger proposed openstack/octavia master: Delete zombie amphora when detected  https://review.openstack.org/58750521:18
*** rcernin has joined #openstack-lbaas21:51
*** fnaval has quit IRC22:21
openstackgerritGerman Eichberger proposed openstack/octavia master: Delete zombie amphora when detected  https://review.openstack.org/58750522:40
*** celebdor1 has joined #openstack-lbaas22:45
*** celebdor1 has quit IRC23:06
« Sunday, 2018-08-26 Index Tuesday, 2018-08-28 »

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!