Wednesday, 2018-10-24

*** salmankhan has quit IRC00:12
*** sapd1 has quit IRC01:16
*** sapd1 has joined #openstack-lbaas01:18
*** hongbin has joined #openstack-lbaas01:19
*** phuoc has joined #openstack-lbaas01:27
openstackgerritIan Wienand proposed openstack/octavia master: [wip] Redirect disk-image-builder logs, make verbose
openstackgerritIan Wienand proposed openstack/octavia-tempest-plugin master: Collect diskimage-builder logs
*** phuoc has quit IRC01:55
*** yamamoto has quit IRC03:21
*** yamamoto has joined #openstack-lbaas03:21
openstackgerritIan Wienand proposed openstack/octavia master: [wip] Redirect disk-image-builder logs, make verbose
*** ramishra has joined #openstack-lbaas03:49
*** yboaron_ has joined #openstack-lbaas04:18
*** hongbin has quit IRC04:25
*** phuoc has joined #openstack-lbaas05:18
*** phuoc has quit IRC05:27
*** ramishra has quit IRC05:34
openstackgerritAkhil jain proposed openstack/octavia master: Add framework for octavia-status upgrade check
*** ccamposr has joined #openstack-lbaas05:37
*** ramishra has joined #openstack-lbaas05:39
*** phuoc has joined #openstack-lbaas05:40
*** phuoc has quit IRC05:40
*** phuoc has joined #openstack-lbaas05:40
*** phuoc_ has joined #openstack-lbaas05:47
*** phuoc has quit IRC05:49
openstackgerritIan Wienand proposed openstack/octavia master: Redirect disk-image-builder logs, make verbose
openstackgerritIan Wienand proposed openstack/octavia-tempest-plugin master: Collect diskimage-builder logs
*** numans has joined #openstack-lbaas06:16
*** raginbajin has quit IRC06:32
*** raginbajin has joined #openstack-lbaas06:34
*** phuoc__ has joined #openstack-lbaas06:41
*** phuoc_ has quit IRC06:44
*** rcernin has quit IRC07:03
*** ivve has joined #openstack-lbaas07:03
*** pcaruana has joined #openstack-lbaas07:05
*** threestrands has quit IRC07:13
*** phuoc has joined #openstack-lbaas07:16
*** phuoc__ has quit IRC07:18
*** ianychoi_ is now known as ianychoi08:03
*** celebdor has joined #openstack-lbaas08:12
*** ramishra has quit IRC08:16
*** ramishra has joined #openstack-lbaas08:17
*** abaindur has quit IRC08:39
*** phuoc_ has joined #openstack-lbaas08:54
*** phuoc has quit IRC08:58
*** salmankhan has joined #openstack-lbaas09:06
*** Emine has joined #openstack-lbaas09:13
openstackgerritAkhil jain proposed openstack/octavia master: Add framework for octavia-status upgrade check
*** phuoc_ has quit IRC09:30
*** yamamoto has quit IRC09:58
*** yamamoto has joined #openstack-lbaas09:59
*** yamamoto has quit IRC10:03
openstackgerritDai Dang Van proposed openstack/octavia-dashboard master: Support the X-Forwarded-Proto insertion header
*** phuoc_ has joined #openstack-lbaas10:39
*** yamamoto has joined #openstack-lbaas10:42
*** phuoc__ has joined #openstack-lbaas11:21
*** phuoc_ has quit IRC11:24
nicolasbock<freenode_emc "nicolasbock hmm yeah it didn't t"> It works now. I had changed the wrong `nova.conf` :(11:51
nicolasbockIn other words, the cpu model is `kvm64` now11:51
nicolasbockI'll see whether I can get the amphora to launch11:51
openstackgerritMerged openstack/octavia master: Add posibilities to set default timeouts
openstackgerritJacky Hu proposed openstack/octavia-dashboard master: Support the X-Forwarded-Proto insertion header
nicolasbockemccormick (IRC): the amphora is coming up and I can ping it!12:07
nicolasbockThanks for the pointer!12:08
*** yamamoto has quit IRC12:11
*** yamamoto has joined #openstack-lbaas12:11
*** yamamoto has quit IRC12:20
*** yamamoto has joined #openstack-lbaas12:20
*** yamamoto has quit IRC12:24
*** ccamposr has quit IRC12:38
*** yamamoto has joined #openstack-lbaas12:55
*** yamamoto has quit IRC13:14
*** ccamposr has joined #openstack-lbaas13:24
tobias-urdincant belive this issue tbh, it's insane, i did a hello world app with flask, serving with gunicorn (to simulate amphora-agent) and copied and reworked the requests code that is done from the controller node13:43
tobias-urdinif i copy the client_ca.pem and server.pem from an amphora and run it on my controller it works13:43
tobias-urdinif i do the opposite and move my server_ca and client.pem for controller into the amphora and run the code it fails with13:44
tobias-urdinrequests.exceptions.SSLError: HTTPSConnectionPool(host='localhost', port=5000): Max retries exceeded with url: / (Caused by SSLError(SSLError("bad handshake: Error([('rsa routines', 'RSA_padding_check_PKCS1_type_1', 'invalid padding'), ('rsa routines', 'rsa_ossl_public_decrypt', 'padding check failed'), ('SSL routines', 'tls_process_key_exchange', 'bad signature')],)",),))13:44
tobias-urdini can replicate the same behaviour with openssl s_server and s_client as well, on both xenial and centos 713:44
tobias-urdinso must be that i created by certs using some option on my ubuntu bionic 18.04 controller machine, or some dependency below python13:45
tobias-urdini deployed py2 and py3 requirements.txt (did pip freeze inside venv on amphora machine) inside a venv on my controller to make sure same python code was executed, and it works...13:46
tobias-urdinso weird13:46
*** yamamoto has joined #openstack-lbaas13:46
*** daikk115 has joined #openstack-lbaas13:54
*** KeithMnemonic has joined #openstack-lbaas13:54
KeithMnemonicjohnsonm: how are you today. did you know Octavia is a brand of car made in the Czech Republic?13:55
daikk115johnsom, hi, could you review the patch related to X-Forwarded-Proto insertion
daikk115we really need this feature :D13:56
openstackgerritVadim Ponomarev proposed openstack/octavia master: Add notifications about changed status to worker
*** yboaron_ has quit IRC14:10
*** phuoc__ has quit IRC14:26
xgerman_@KeithMnemonic we know about Skoda Octavia ;-)14:42
dayouThey also make machine guns, it's a brand with a good story14:47
johnsomKeithMnemonic: Yes, but no worries, we got the trademark for our use and assigned it to the OpenStack foundation.14:49
tobias-urdinjohnsom: any suggestions? i've pinpointed the issue to be inside the amphora on xenial and centos 7 (didnt even get bionic to boot), it's probably below python as well, so something with openssl14:58
tobias-urdinperhaps i should create a patch that tries dual CA's in CI and see if it catches something similar14:59
johnsomI will try to setup a dual CA system today.  I suspect your issue is with how the certs are being created.15:08
xgerman_yeah, especially with the. more strict things they don’t let you use client certs on a server and vice versa…15:10
tobias-urdincrazy tho
tobias-urdincopied out the client_ca.pem and generated server.pem from an amphora and verified on controller15:18
tobias-urdinif do the same but opposite on the amphora, copy the server_ca.crt and client.pem to the amphora and run that code inside the amphora venv it fails15:18
tobias-urdincan also reproduce with openssl s_server and s_client which pretty much confirmes it's not python, like the above does as well15:19
*** Emine has quit IRC15:22
*** celebdor has quit IRC15:24
*** boden has joined #openstack-lbaas15:26
bodenhi. it seems that the recent commit of  breaks some consumers... I'm now seeing all UTs in vmware-nsx (uses Octavia) broken with: oslo_config.cfg.NoSuchOptError: no such option haproxy_amphora in group [DEFAULT]15:30
bodenguessing we need to mock something out, but still under investigation15:30
*** KeithMnemonic has quit IRC15:42
*** pcaruana has quit IRC15:49
*** ramishra has quit IRC16:14
*** ccamposr has quit IRC16:14
*** fnaval has joined #openstack-lbaas16:25
*** celebdor has joined #openstack-lbaas16:28
*** pcaruana has joined #openstack-lbaas16:46
*** velizarx has quit IRC17:01
*** irclogbot_3 has quit IRC17:08
*** irclogbot_3 has joined #openstack-lbaas17:08
*** salmankhan has quit IRC17:09
*** Swami has joined #openstack-lbaas17:18
*** irclogbot_3 has quit IRC17:21
openstackgerritGerman Eichberger proposed openstack/octavia master: Refactor the pluggin of the VIP
*** phuoc has joined #openstack-lbaas17:54
openstackgerritGerman Eichberger proposed openstack/octavia master: Allows failover if port is not deallocated by nova
*** abaindur has joined #openstack-lbaas18:15
*** ivve has quit IRC18:18
*** phuoc has quit IRC18:30
*** raginbajin has quit IRC18:48
*** nmagnezi has quit IRC18:48
*** celebdor has quit IRC18:50
*** openstackgerrit has quit IRC19:06
*** celebdor has joined #openstack-lbaas19:10
*** abaindur has quit IRC19:10
*** abaindur has joined #openstack-lbaas19:11
*** abaindur has quit IRC19:13
*** abaindur has joined #openstack-lbaas19:14
*** abaindur has quit IRC19:18
*** ivve has joined #openstack-lbaas19:20
*** irclogbot_3 has joined #openstack-lbaas19:24
*** nmagnezi- has joined #openstack-lbaas19:56
*** nmagnezi- is now known as nmagnezi19:56
johnsom#startmeeting Octavia20:00
openstackMeeting started Wed Oct 24 20:00:11 2018 UTC and is due to finish in 60 minutes.  The chair is johnsom. Information about MeetBot at
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.20:00
*** openstack changes topic to " (Meeting topic: Octavia)"20:00
openstackThe meeting name has been set to 'octavia'20:00
johnsomHi folks. Welcome to another episode of Octavia this week.20:00
johnsom#topic Announcements20:01
*** openstack changes topic to "Announcements (Meeting topic: Octavia)"20:01
johnsomStein Milestone 1 is this week20:01
johnsomThe release team is changing how milestones are done. We still have the milestone dates, but we are not cutting a milestone release. That is unless we want one.20:02
johnsomOther than that, I don't have any more announcements. Anyone else?20:02
nmagneziNothing on my end20:03
johnsomOk then, moving on20:03
johnsom#topic Brief progress reports / bugs needing review20:03
*** openstack changes topic to "Brief progress reports / bugs needing review (Meeting topic: Octavia)"20:03
johnsomI have been focused on testing and octavia-lib work.20:04
johnsomI am done adding gates that test IPv6 with actual traffic. It's just waiting on patches to merge.20:04
johnsomSpecifically I would really like to see this merge:20:04
xgerman_#link <- needs eyes20:05
johnsomSo we can get started on backporting those20:05
xgerman_and #link
johnsomThe octavia-lib work is coming along, but I realized we need to move forward on the driver lib using an endpoint, so I'm working on that today.  More on that discussion later on the agenda20:06
johnsomI also plan to spend a bit of time on the certificate setup stuff folks are running into trouble with. I will probably start a doc for it that will eventually become part of the install guide.20:07
johnsomAny other updates today?20:07
johnsom#topic RFEs20:08
*** openstack changes topic to "RFEs (Meeting topic: Octavia)"20:08
johnsomI wanted to bring to the team's attention an RFE that came in for "Add notifications about object's status changes"20:09
johnsomWe have had some discussion in the channel about this, but wanted to give the team a heads up to comment on the proposal before we approve it.20:09
johnsomWork has already started it appears.20:10
johnsomAny discussion now on the RFE?20:10
johnsom#topic Octavia-lib and the need for a new Octavia controller process20:11
*** openstack changes topic to "Octavia-lib and the need for a new Octavia controller process (Meeting topic: Octavia)"20:11
johnsomAs I have moved the code over for the provider drivers to use octavia-lib I worked on removing the dependency on octavia.20:11
johnsomSince we don't want to make the Octavia project into a library, we don't really want to add it to global requirements.20:12
johnsomWhich gets into issues with the driver_lib callbacks for the status/stats.20:12
johnsomWe have wanted to make this use an endpoint for the drivers to send the stats/status data back.20:13
johnsomI realized now is the time.20:13
johnsomThe problem comes in that with a provider driver, really the only part of Octavia you need is the API service.20:13
johnsomWhich is implemented as a WSGI application.20:14
johnsomSadly, this means forking out a process that handles these stats/status updates is not practical. Most notably because we can't get shutdown events.20:14
*** abaindur has joined #openstack-lbaas20:15
johnsomSo, the path I am going down is creating another process that will run along side the API service, that will handle the stats/status updates.20:15
johnsomThe updates will pass from octavia-lib to the new process via unix domain sockets.20:15
johnsomThis makes the issues around securing the endpoint easier.20:16
johnsomAny thoughts/comments on that approach?20:16
nmagneziJust to get my head around it20:16
nmagneziWhat are the alternatives?20:16
nmagneziNot saying I disagree, just for thinking about this more20:16
johnsomWell, we declare Octavia is a library and change it's release cycle, then add it to global requirements.20:17
xgerman_yeah, can’t we just send it to an API endpoint?20:17
xgerman_aka API server gets a stats endpoint and we call it a day?20:18
johnsomWe can setup a UDP/TCP endpoint and figure out how to secure it. This would either need the new process or would require the operator to install the HM and make it reachable from the API process.20:18
johnsomxgerman_ So the driver would have to get a keystone token?20:18
nmagnezijohnsom, good point20:19
xgerman_yeah, why not20:19
xgerman_or shared secret or…20:19
*** ivve has quit IRC20:19
*** abaindur has quit IRC20:20
xgerman_so the proviser driver would run in said new process or would it run somehwere else and communicate witj Ocatvia?20:20
johnsomSeems a bit messier, in that it's more setup people installing would have to do. I.e. setup the account in keystone and make sure it has the right RBAC.20:20
*** irclogbot_3 has quit IRC20:21
johnsomIt still runs in the context of the API process as the API server loads it.20:21
xgerman_well, most people install with installers (OSA, TripleO, …) so don’t see that as a problem20:21
johnsomLol, we seem to get a lot of questions about that.....20:22
nmagnezijohnsom, say with go with the suggestion you proposed, unix domain socket, how would that look for drivers who run on a different node such as F5?20:22
xgerman_ok, if it runs on the api server we can just restrict access to thet endpoint to lcoalhost?20:22
johnsomnmagenzi This part of the code has to run on the API server anyway as it's a loaded driver.20:23
johnsomxgerman_ Yeah, that is basically what I an doing with the unix domain socket.20:23
nmagnezijohnsom, true. Just want to know what if the actual provider software needs to pull status of some object and it lives on an external node20:24
xgerman_yeah,  think having a web server restricted to ips (hello loadbalancer) is more scalable20:25
johnsomIf a part of their driver lives on a different node, they will need to figure out how to request that of either their driver in the API node, or via the main Octavia API20:25
xgerman_Adding this to the Ocatvia API makes sense to me… and then haveoctavia-lib access it (maybe keystone-auth, maybe ip-lock)20:26
johnsomI think it needs to be stronger than ip-lock really.20:26
xgerman_well, we cna leve that to the individual operator ;-)20:27
xgerman_just make it pluggable20:27
johnsomI lean towards the process as it separates the stats/status load from the API infrastructure uWSGI, etc.20:27
johnsomI worry about putting that much data down the same path as the API calls.20:28
xgerman_I believe we cna scale api pretty well — and adding some other process complicates things20:29
johnsomWhat are the downsides to adding a process?20:29
xgerman_will it be another service like an o-*20:30
johnsomo-da for driver_agent?  hahaha20:31
nmagneziWorks for me :D20:31
xgerman_yeah, probably minimal downside since people install with OSA, TRIPLE-O… etc. but I am also not sure if that needs to scale independently from the api server since you plan to hitch it to that container/host anyway20:32
xgerman_so could also be an endpoint20:32
xgerman_and all we “save” is some authentication and net traffic20:33
johnsomIn reality it is an endpoint either way, just implementation is different.20:33
xgerman_o-hm can scale independently — with the provider driver tied to o-api…20:33
johnsomSo you would add the requirement of running the o-hm always?20:34
xgerman_nah, I am saying we could just add that to o-api and could be good20:34
johnsomAnd that the API needs to be able to reach an endpoint on the o-hm (not a requirement today)20:35
johnsomI guess you have more faith in the abilities of uWSGI than I do20:35
xgerman_ok, let’s back off20:35
xgerman_our event pipeline is n o-hm… so if we do something else for processing of status we need to replicate that20:36
johnsomIt is similar but simpler than the o-hm code.20:36
xgerman_ok, so o-hm would need to be co-located with o-da for the amphora-driver?20:37
johnsomThat is the code that would move into the new "thing"20:37
emccormickHey, is there a way to specify which version of amphora agent gets installed in the diskimage build?20:38
emccormickI see this here: amphora-agent git /opt/amphora-agent
xgerman_johnsom: this codes lacks all the event stuff 920:38
emccormickbut wouldn't that fetch master by default?20:38
xgerman_e.g. throwing it on the n-lbaas queu20:38
johnsomIf/when we move the amphora-driver to use this path, o-hm would need to send that data over to the amphora-driver, yes.20:38
xgerman_emccormick: can youw aint until our Open Discussion section in our meeting20:39
emccormickOh I'm sorry, I popped in from another window and didn't see the meeting start. My apologies for interrupting20:39
xgerman_no worries :-020:39
xgerman_johnsom: still wrapping my head around that. Yes, we would not want o-hm access the o-api20:40
xgerman_so if we have amphora-driver and venfdor Y — this o-da would run on the api host and the hm host?20:41
johnsomNo, just the API host. Currently o-hm does all of this on its own (plus some)20:42
xgerman_yeah, but we should move that part out of o-hm and into o-da — or not?20:42
johnsomShould at some point to be "clean"-er-ish20:43
johnsomMore than I plan to do in Stein however20:43
xgerman_ok, this is a strike against an o-api endpoint since we then would need to create connectiovity o-hm<->o-api20:43
xgerman_but we could also make that similr to o-cw which get integrated into o-hm as a “library”20:44
johnsomWhich isn't a huge leap given o-hm talks to the other service APIs, but yes20:44
xgerman_mmh, I still  (1) think an extra service complicates our offerings (2) endpoints should be RESY and o-api20:46
johnsomthat is true.20:46
johnsomthat is true was about the library part20:46
xgerman_yeah, got that :-)20:47
xgerman_I know my 2nd part is controversial20:47
*** abaindur has joined #openstack-lbaas20:47
xgerman_and we lost nmagnezi20:47
nmagneziSorry I try to multi-task here O_o20:48
johnsomWell, I don't want to burn all of our time on this. I can post the octavia-lib side today and you can look at it.20:48
johnsomThis would give some time to think about it.20:48
johnsomThough I do want to move fast on this as we have people waiting on octavia-lib.20:49
johnsom#topic Open Discussion20:49
*** openstack changes topic to "Open Discussion (Meeting topic: Octavia)"20:49
johnsomOther topics for today before we run out of time?20:50
*** pcaruana has quit IRC20:50
nmagneziMaybe emccormick :)20:51
johnsomI side messaged him the info he needed.20:51
johnsomWhich for the log is these two variables:20:52
nmagnezithat's very.. efficient20:52
nmagnezi(The side msg)20:52
johnsomHa, easy when I know the answer and where to point in the code20:52
johnsomOk, let's wrap up then.20:53
johnsomPlease give me your thoughts on the octavia-lib discussion in the next day or two.20:53
emccormickthanks for hte info20:54
johnsomAlso, please try to find some time for reviews. We have a lot of open stuff.20:54
*** openstack changes topic to "OpenStack PTG etherpad:"20:54
openstackMeeting ended Wed Oct 24 20:54:52 2018 UTC.  Information about MeetBot at . (v 0.1.4)20:54
openstackMinutes (text):
*** aojea has joined #openstack-lbaas20:55
*** openstack has quit IRC21:03
*** openstack has joined #openstack-lbaas21:07
*** ChanServ sets mode: +o openstack21:07
abaindurjohnsom: I deleted the standby, and it was recreated. But then I deleted both, and it only spun up 1 (and did not even plug in VIF into it), and load balancer now in ERROR state21:22
abaindurany idea where to look for in logs?21:22
johnsomabaindur What version of Octavia are you running? There was a bug with dual down amphora, but I fixed it a while ago21:23
abaindurwould you like me to paste the health-manager logs?21:25
johnsomI know it has been backported, I just need to find it21:25
johnsomabaindur It was this patch:
johnsomCan you check that you have that version of queens installed?21:27
abaindurah no we dont21:32
abaindurlast commit was from sometime in July21:32
johnsomYeah, queens 2.0.2 release has that patch21:32
*** openstackgerrit has joined #openstack-lbaas21:41
openstackgerritIan Wienand proposed openstack/octavia master: Redirect disk-image-builder logs, make verbose
*** boden has quit IRC21:46
*** abaindur_ has joined #openstack-lbaas22:08
*** abaindu__ has joined #openstack-lbaas22:11
*** abaindu__ is now known as abaindur__22:11
*** abaindur has quit IRC22:12
*** abaindur_ has quit IRC22:14
xgerman_passed zuul:
*** abaindur has joined #openstack-lbaas22:18
openstackgerritIan Wienand proposed openstack/octavia master: Redirect disk-image-builder logs, make verbose
*** abaindur__ has quit IRC22:20
*** abaindur_ has joined #openstack-lbaas22:23
*** abaindur has quit IRC22:25
*** abaindur_ has quit IRC22:33
*** abaindur has joined #openstack-lbaas22:34
*** celebdor has quit IRC22:38
*** daikk115 has quit IRC22:40
*** aojea has quit IRC22:40
colin- those performance improvements you mentioned a couple weeks ago had a noticeable impact johnsom thanks22:52
johnsomcolin- Excellent!22:52
SwamiHad anyone here seen an error when running devstack/octavia on ubuntu-18.04. "Couldn't create tempfiles for splitting up /var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_xenial_InReleaseErr:1 xenial InRelease22:56
johnsomSwami Yes.22:56
SwamiWhat is the solution?22:56
Swamijohnsom: cool, you are great.22:57
*** fnaval has quit IRC22:58
*** rcernin has joined #openstack-lbaas23:05
*** abaindur has quit IRC23:11
*** openstackgerrit has quit IRC23:20
*** openstackgerrit has joined #openstack-lbaas23:37
openstackgerritMichael Johnson proposed openstack/octavia-lib master: Initial provider driver library checkin
johnsomxgerman_ This is the octavia-lib side:
johnsomIgnore the constants issues as I haven't done that for the lib yet.23:38
*** Swami has quit IRC23:45
xgerman_Ok, you probably want to use some binary format for the socket - no need to do Jason23:54
xgerman_But get the idea - just23:56

Generated by 2.15.3 by Marius Gedminas - find it at!