Wednesday, 2018-08-15

abaindurjohnsom: is there a list of the support protocols for load balancing?00:18
johnsomabaindur Yes, in the API reference document00:19
johnsomLook for "protocol"00:19
abaindurah ok, i was just looking at the cookbook00:19
*** longkb has joined #openstack-lbaas00:42
*** hongbin has joined #openstack-lbaas00:44
*** sapd1 has joined #openstack-lbaas01:15
openstackgerritMichael Johnson proposed openstack/octavia master: Fix Octavia for host host routes
openstackgerritMichael Johnson proposed openstack/octavia master: Fix neutron "tenat_id" compatibility
*** abaindur has quit IRC02:01
*** openstack has joined #openstack-lbaas02:36
*** ChanServ sets mode: +o openstack02:36
*** ramishra has joined #openstack-lbaas02:37
*** hongbin has quit IRC03:23
*** hongbin has joined #openstack-lbaas03:28
*** hongbin_ has joined #openstack-lbaas03:48
*** hongbin has quit IRC03:50
*** hongbin_ has quit IRC04:14
*** oanson has quit IRC04:33
openstackgerritMichael Johnson proposed openstack/octavia master: Allow blocking IPs from member addresses
*** yboaron_ has joined #openstack-lbaas04:56
*** oanson has joined #openstack-lbaas05:00
*** pcaruana has joined #openstack-lbaas05:12
johnsomCores, if you could please review to un-block the neutron-lbaas-dashboard gates05:21
openstackgerritMerged openstack/octavia-dashboard master: Add Apple OS X ".DS_Store" to ".gitignore" file
*** abaindur has joined #openstack-lbaas06:49
*** yboaron_ has quit IRC07:50
*** yboaron_ has joined #openstack-lbaas08:07
*** yboaron_ has quit IRC08:12
*** velizarx has joined #openstack-lbaas08:19
*** abaindur has quit IRC08:26
*** velizarx has quit IRC08:35
*** velizarx has joined #openstack-lbaas08:37
*** bzhao__ has quit IRC09:18
*** xgerman_ has quit IRC09:18
*** salmankhan has joined #openstack-lbaas09:22
*** yboaron_ has joined #openstack-lbaas09:23
*** rtjure has joined #openstack-lbaas09:35
sapd1johnsom: Can I create amphora on both VIP Network and LB_Manage Network instead of a task which plugVIP on VIP Network?10:05
*** longkb has quit IRC10:45
*** sapd1 has quit IRC10:59
openstackgerritKobi Samoray proposed openstack/neutron-lbaas master: nlbaas2octavia: Escape 'key' field calls
openstackgerritKobi Samoray proposed openstack/neutron-lbaas master: nlbaas2octavia: Script fails when no members found
*** sapd1 has joined #openstack-lbaas14:49
johnsomsapd1 You can tell the VIP to be on the lb-mgmt-net if you need.14:56
sapd1I'm trying to launch amphora instance using SR-IOV network.15:00
sapd1johnsom: because the flow in octavia is attach vrrp port after instance is created, so the SR-IOV port can't attach to amphora instance15:02
openstackLaunchpad bug 1708433 in OpenStack Compute (nova) "Attaching sriov nic VM fail with keyError pci_slot" [Medium,In progress] - Assigned to Matt Riedemann (mriedem)15:02
johnsomAh, yeah, nova needs to fix that15:02
*** ramishra has quit IRC15:03
sapd1No. nova has a associate spec with this bug. But It is not implemented. and abandoned15:04
*** yboaron_ has quit IRC15:11
*** kobis1 has joined #openstack-lbaas15:33
kobis1hey how's familiar with the nlbaas2octavia tool?15:37
johnsomkobis1 I wrote it15:42
*** velizarx has quit IRC15:45
kobis1Hi johnsom, testing this now - running into a few issues15:47
johnsomI saw you posted a few fixes last night15:48
kobis1Today… time difference ;)15:48
kobis1Still testing these though15:48
*** shoffmann has joined #openstack-lbaas15:49
kobis1Curious about changing the project on the security group - AFAIK the owner should remain same - the user's project. When I create an Octavia LB the VIP port's owner is the user project15:49
shoffmannI'm testing with octavia loadbalancer and have an problem:15:49
shoffmannThe creation of the loadbalancer finished without any error and it is shown as ACTIVE. The octavia-worker.log only shows15:49
shoffmann'INFO [-] Port d0d1cb55-c6c8-4772-ab86-9dbc69d8e8b9 already exists. Nothing to be done.'15:49
shoffmannThe problem is, that this port is not activ and not connected to the amphora Instance.15:49
shoffmannCan anyone tell me, how I can find out why?15:50
kobis1And then - when I create a NLBaaS LB, assign the SG to some SG in the user's domain, the migration "steals" my SG and then it's owned by Octavia15:50
johnsomWe are greedy like that...  lol15:50
kobis1In the most stupid case - I assign my VIP to the default SG… And boom it's gone...15:51
johnsomYeah, it's probably a bug.  I know we create a security group, thus the need for an ownership change, but It should probably check if it was owned by nlbaas before changing it.15:52
kobis1Is that SG for the internal Octavia net?15:53
kobis1OK cool, I'll just do some more testing - will try to push another fix for this as well15:53
johnsomshoffmann The message about the port is fine. It just means the port was created earlier in the process. Octavia has two ports, one is always "down" as it is a "fake" port created by neutron for the VIP. The other is up and has a "allowed-address-pair" pointing to the second port.15:53
johnsomkobis1 Ok, thanks! We do have a gate job for the tool as well if you want to update any tests.15:54
kobis1Cool for a starter I'll complete one cycle of manual testing… And then start updating tests and such. In the meantime my patches are WF -1 and there just to keep things in order on my side15:55
*** FracKen has joined #openstack-lbaas15:56
johnsomCores, once this stable/queens merges I will cut a stable/queens release:
*** fnaval has joined #openstack-lbaas16:14
*** fnaval has quit IRC16:16
*** fnaval has joined #openstack-lbaas16:16
shoffmannjohnsom But I have to connect the FloatinIP to the "fake" port? I want to use a private subnet.16:22
johnsomshoffmann You use the port ID listed on the load balancer for the floating IP16:23
shoffmannjohnsom Thanks. This is the "fake" port.16:25
*** colby_ has joined #openstack-lbaas16:31
colby_johnsom: Thanks for spotting that. I just thought that was a product of the connection failing. Ill check into the cert config. Im able to use the certs with curl. Ill put my cert config and my curl command into a pastebin16:31
*** kobis1 has quit IRC16:41
*** openstackstatus has joined #openstack-lbaas16:57
*** ChanServ sets mode: +v openstackstatus16:57
*** shoffmann has left #openstack-lbaas17:00
*** salmankhan has quit IRC17:13
*** kobis1 has joined #openstack-lbaas17:34
*** kobis1 has quit IRC17:35
*** sapd1 has quit IRC17:39
colby_johnsom Here is my octavia config and curl command I used:
*** Krast has quit IRC17:43
johnsomcolby_ I would check the ca_private_key file, make sure it has a ---- header and double check the passphrase unlocks it. Check the spacing around the passphrase, etc.17:47
colby_I used openssl to test the passphrase and it worked fine17:53
colby_ok Im at a loss. Ive checked the private key file. Everything looks fine. I used openssl (as the octavia user) to make sure I could view the key file and verify the password opened it correctly.18:31
johnsomHmm, ok. Can you point me at that log file entry again from the worker process?18:32
colby_I use copied and pasted the values from the octavia config to make sure I had them the same18:32
colby_I can paste another one18:32
johnsomThanks, I will refresh my memory of where that is occurring in the code and see if we can get more information out of the python library we are using.18:34
colby_Thanks for your help18:39
*** salmankhan has joined #openstack-lbaas18:41
johnsomcolby_ What version of Octavia are you running?18:42
johnsomOk, so back on Pike18:46
colby_yes we are running pike18:46
*** salmankhan has quit IRC18:46
colby_we will probably be upgrading to queens in the next couple months18:47
johnsomWell, I can confirm it's not the ca_private_key_passphrase as I get a different error if that is wrong18:57
johnsomcolby_ How hack-y are you willing to go on this host?19:00
johnsomFind the python OpenSSL module that the worker process is using (not sure if you are using venv  or not).19:02
johnsomin the file, find "use_privatekey_file" method,  add some debug logging/print around the keyfile path it is using.19:02
colby_Im ok with that. Ill update my hosts where the workers are.19:15
*** openstackgerrit has quit IRC19:19
colby_keyfile used: /etc/pki/tls/certs/octavia_client_cert.pem19:54
colby_that is not a keyfile19:54
colby_thats what is set as client cert: client_cert = /etc/pki/tls/certs/octavia_client_cert.pem19:56
colby_its not set anywhere else in the config19:57
johnsom#startmeeting Octavia20:00
openstackMeeting started Wed Aug 15 20:00:02 2018 UTC and is due to finish in 60 minutes.  The chair is johnsom. Information about MeetBot at
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.20:00
*** openstack changes topic to " (Meeting topic: Octavia)"20:00
openstackThe meeting name has been set to 'octavia'20:00
johnsomHi folks20:00
johnsom#topic Announcements20:01
*** openstack changes topic to "Announcements (Meeting topic: Octavia)"20:01
johnsomWe are still running the Rocky priority bug list:20:01
johnsomThank you to the folks that have been doing reviews there20:01
johnsomWe will talk about RC2 later in the agenda20:02
nmagneziWe're planning to have another rc to include those?20:02
nmagneziHa, ok :)20:02
johnsomgrin, that is what we will talk about20:02
johnsomThe PTG etherpad is up:20:02
johnsomPlease add any topic ideas you may have. I think we are are less than a month out, so in the next week or two I will start putting a rough agenda together20:03
johnsomAny other announcements today?20:03
johnsom#topic Brief progress reports / bugs needing review20:04
*** openstack changes topic to "Brief progress reports / bugs needing review (Meeting topic: Octavia)"20:04
johnsomI was on vacation with some relatives over the last week, so didn't get a lot done.20:05
johnsomI addressed three bug reports that we recently got. Those are up for review.20:05
johnsomI did tag the octavia-tempest-plugin for Rocky.  We aren't cutting stable branches there, but we are tagging the releases now.20:06
johnsomThat is the big summary for me.  Anyone else?20:06
johnsomok, moving on20:07
nmagneziI mainly focus on my deep dive into active standby. Some of the issues I have reported20:07
nmagneziOthers will be reported next week20:07
nmagneziWith fixes20:08
johnsomOk, let me know if you have questions or something I should look at (it's been busy, so I might have forgot something)20:08
cgoncalvesnot much from my side. UDP testing and more work on TripleO-Octavia integration for better CI coverage20:08
nmagnezinp, thank you :)20:08
johnsom#topic Do we want to cut an RC2?20:08
*** openstack changes topic to "Do we want to cut an RC2? (Meeting topic: Octavia)"20:08
nmagnezijohnsom, right after that I plan to shift my focus to the migration tool that you wrote20:08
johnsomOk, so we have six open patches on the priority review list.  Do any of these look like things we need to merge and do an RC2?20:09
* nmagnezi looks again20:10
cgoncalvesif we have the opportunity for an RC2, why not?!20:10
johnsomTrue, I just need to have cores willing to review/vote/merge20:11
johnsomThe first three I can vote on. If you all think they are a priority I will make some time this afternoon to review.20:11
cgoncalvesit is my believe that most (if not all) top priority reviews look good and are small20:12
nmagneziThis looks important
johnsomI would like to have these merged and backported to stable/rocky by about this time on friday.20:12
cgoncalvesand stable/queens?! :D20:12
johnsomYeah, that was reported by a user20:13
johnsomI was planning to cut queens after #6 on that list is merged20:13
johnsomBut #5 is a good one too20:13
johnsomfor queens that is20:13
cgoncalves"release early, release often"20:14
johnsomGerman is out on vacation, so it's really down to the rest of us20:14
johnsomOk, so I am hearing, yes, we want an RC2.  I will cut one about this time in two days. We need to get these in and backported by then or they don't make it.20:15
johnsomThen we backport them and do a followup release later.20:15
cgoncalvesperfect, thanks!20:15
johnsom2 and 3 might be a bit on the risky side to merge, I need to look at those patches and judge the risk.20:16
johnsomActually, 1,2, and 3 might be in that group20:16
johnsomI will take a look.20:17
johnsomOk, any other RC2 discussion?  Any other patches landing soon?20:17
johnsom#topic Open Discussion20:18
*** openstack changes topic to "Open Discussion (Meeting topic: Octavia)"20:18
johnsomOk, how about any other topics20:18
johnsomAh, yes, missed that one for the list20:19
johnsomSo the summary here, is you can make a member pointed at the nova metadata IP and get the metadata for the amphora.20:20
johnsomSo, this adds a way to blacklist IPs from being members, defaulting to the metadata IP.20:20
nmagnezijohnsom, not sure we can actually back port since it adds a config option20:21
nmagneziit looks like a very important patch, so maybe we can make an exception for security reasons?20:21
johnsomYeah, I think adding an option is better than changing an option. It has a valid default, so won't break people if they don't configure it.20:22
johnsomAh, I need to add the sample octavia.conf entry and a release note too.  (finished that patch late last night)20:22
nmagneziI will review it tomorrow morning my time either way20:22
nmagneziWhen did you plan to cut RC2 (if at all :) )20:23
johnsomI think it falls into the "compatible config change" category, so can be backported.20:23
johnsomIt is a bit of a feature, but it does address a minor security concern20:23
nmagnezijohnsom, +1. I agree. Just wanted to point this out so we can double check ourselves here20:24
cgoncalves+1 for backporting20:24
johnsomIt's a bummer, we don't even use the metadata service for the amps, so wish we could just turn it off, but we can't.20:24
nmagneziYeah, IIRC we use config-drive for SSH keys right?20:25
johnsomPlus the only ssh key in that metadata is the public key20:25
johnsomI'm not sure what all else is exposed there, but it's good to block it anyway20:26
johnsomAny other topics today?20:27
cgoncalvesjust a minor update that the networking-ovn team has been working actively on an octavia driver20:27
cgoncalvesI would like to have a non-voting CI job in octavia sometime in the near future20:28
johnsomOk, I am open to that20:28
nmagneziSounds right to me20:28
johnsomI can't wait to drop the neutron-lbaas jobs. We have a lot of jobs at the moment20:29
johnsomI also need to look at nlbaas-dashboard. it looks like the docs job is grumpy there and I'm not sure why20:31
johnsomOh, right, we need to get this merged:20:32
johnsomleft overs from when horizon had integration tests20:33
johnsomOk, well, if we don't have any other topics, I will go grab lunch.20:34
nmagneziThis patch looks good at a first glance20:34
cgoncalvesall right, thanks folks!20:34
johnsomYeah, we did something similar in octavia-dashboard a while ago20:35
johnsomOk, thanks folks!20:35
*** openstack changes topic to "Discussion of OpenStack Load Balancing (Octavia) |"20:35
openstackMeeting ended Wed Aug 15 20:35:13 2018 UTC.  Information about MeetBot at . (v 0.1.4)20:35
openstackMinutes (text):
colby_johnsom If you are free now Im wondering about the config for pike version. I looked here: and it seems to be missing all sorts of certificate config options I though were used20:37
colby_maybe that is my problem20:37
*** openstackgerrit has joined #openstack-lbaas20:41
openstackgerritMichael Johnson proposed openstack/octavia master: Allow blocking IPs from member addresses
*** abaindur has joined #openstack-lbaas21:24
*** abaindur has quit IRC21:26
*** abaindur has joined #openstack-lbaas21:26
*** rcernin has joined #openstack-lbaas21:29
colin-anyone have suggestions for how to work past this error on a loadbalancer delete command?21:36
colin-clean_up DeleteLoadBalancer: Conflict (HTTP 409) (Request-ID: req-f9342b11-90f7-42d7-b6f9-a3d91121075a)21:36
johnsomcolin- Looks like it is already in a PENDING_* state. Wait until the controller is finished retrying the action, then it will either be in ACTIVE or ERROR and you can delete it.21:38
colin-any way to force that state as the operator?21:41
colin-if i'm unconcerned with the resource and any data on it21:41
johnsomNot really. It's a bad idea as the controller is actively working on the object. But, if you want to "do bad things"(tm) you can go into the database and change the provisioning status to ERROR by hand. The controller will get grumpy and attempt to revert things.21:43
johnsomcolin- I would recommend adjusting the retry counts from the defaults so things fail faster. The defaults are super high to address gate host issues.21:43
*** KeithMnemonic has quit IRC21:44
colin-yeah that makes sense. thanks for the additional info21:53
*** abaindur has quit IRC21:56
*** FracKen has left #openstack-lbaas22:21
johnsomArgh, looks like we have a bug with IPv6 in active/standby mode.22:36
rm_workjohnsom: swept through and reviewed and was able to +2 a few of those priority ones people linked in the meeting22:43
johnsomrm_work Thanks22:43
rm_worknot sure on the blocking IPs thing yet, that one is a little longer22:43
rm_workerr, rather, a little more complex22:44
rm_workactually, nm it isn't that long it just looked like it was going to be22:45
rm_worki think I'm +2 there also22:45
*** fnaval has quit IRC22:45
rm_workseems fine22:45
rm_workand yeah, i think the backport for that should be fine too, but i do tend to be pretty lenient on backports22:46
rm_workwe should get someone else to cherry-pick these so we can both +2 :)22:46
johnsomYeah, I thought about that22:47
johnsomSpeaking of backports:
colby_So since it appears pikes certificate config is very different than queens it might not work for me. Can I run the queens version of octavia with the rest of openstack being pike?22:50
johnsomcolby_ Yes you can22:50
rm_workjohnsom: or rocky :)22:50
rm_workcolby_: or rocky :P22:50
johnsomI didn't remember we changed that stuff much.22:51
rm_workyeah it shouldn't be that really22:51
rm_worki'm not aware of config changes to the cert stuff since basically ever22:51
rm_workcolby_: i read most of the backlog, and i think you may be running into an issue that is similar to something I did22:52
rm_workyou did determine that the wrong file was being provisioned as the "key" right?22:52
rm_workthe contents of what file ended up in the key22:52
colby_but when the worker starts it does not even list some of the configs as being ready22:52
colby_so its loading what I have in client_cert config22:53
colby_as the key22:53
rm_workcan you pastebin the whole config section for haproxy_amphora?22:54
rm_workand controller_worker too22:54
rm_workwhile you're at it22:54
openstackgerritMerged openstack/neutron-lbaas-dashboard master: Replace noop tests with registration test
rm_workand [certificates]22:54
colby_here is my whole config:
rm_workcolby_: so, [certificates] + [haproxy_amphora] + [controller_worker]22:55
rm_workah k that works22:55
colby_so according to the pike config reference: ca_certificate, ca_private_key, ca_private_key_passphrase dont exist22:56
colby_instead it has ca_certificates_file22:56
colby_they are not listed as loaded configs on worker start either22:56
rm_workhmmm in [certificates] yeah22:57
rm_worklol so23:00
rm_workit looks like [certificates] never had and still does not have anything but ca_certificates_file lol23:00
rm_worki noticed I have different ones in my [certificates] section too23:01
rm_workbut they also aren't being read23:01
rm_worki think that section is bogus in our examples maybe?23:01
rm_workOH wait23:01
colby_ok so how would I make my certs work then?23:02
openstackgerritMerged openstack/neutron-lbaas-dashboard master: Removes testr and switches cover to karma-coverage
rm_workthey come from that23:03
rm_workwhich is ...  confusing23:03
rm_workbut people didn't want them in the main config because they were specific to one "driver" for certificate generation23:03
rm_workso they are probably being read fine23:03
rm_workand they are only used during the generation process, so are unrelated to what happens after23:04
rm_worki don't think those are your problem23:04
colby_ok so then why is the worker trying to read my client_cert as my key?23:04
rm_workcan you *triple check* the files that are present on your worker host23:04
rm_workand make sure they are actually the correct data in those filenames23:04
rm_workcan you paste the contents of the agent's config too?23:05
*** abaindur has joined #openstack-lbaas23:06
colby_the agent on the running instance right?23:06
rm_workand, the contents of the cert files are correct on the agent? it just literally tries to use the wrong filename when loading the key?23:06
johnsomrm_work It's the controller worker that is saying it has a bogus key file23:07
colby_yes I believe so. I added a log output that just echoed the contents of keyfile in the "use_privatekey_file method of SSL.py23:07
rm_workerr is it?23:07
rm_worki thought he was saying it was on the amp, let me read scrollback again23:07
colby_those are controller_worker logs23:08
rm_workso yeah nm, one sec23:08
*** abaindur has quit IRC23:09
colby_Thanks for helping me track this down guys23:09
*** abaindur has joined #openstack-lbaas23:10
johnsomcolby_ Thanks for pointing out that this is an area we need to clean up.  It is clearly not right at the moment, confusing at the least.23:12
rm_workyes it is very freaking confusing23:14
rm_workok so i think one of these cert files needs to be like a COMBO cert/key23:14
rm_workchecking for sure23:14
rm_workbecause i actually do not see where we define a privatekey file anywhere23:14
rm_workbesides the [certificates] section, which as i said, is not used for this23:15
johnsomYeah, I am thinking there is some settings missing here.23:15
rm_worki am actually gonna go look on my CW23:15
rm_workhis config looks the same as mine tho23:15
rm_workok client.pem is the combined one23:16
colby_ok so it should be the client cert and key then?23:16
rm_workyes both together23:17
rm_workmine is cert, then key concatenated on23:17
rm_workwhat is in yours23:17
rm_workguessing just the cert23:18
rm_workIIRC there was a reason for this23:18
rm_workjust don't remember what ATM23:18
rm_worklet me see if i can figure out the answer23:18
*** rcernin has quit IRC23:18
*** rcernin has joined #openstack-lbaas23:19
rm_workah i think it's how the ssl adapter works23:19
rm_workerr, sorry, the requests session23:19
johnsomWe need to write this stuff down somewhere23:20
rm_workyep it is due to Requests23:20
rm_workah it looks like it supports a tuple now23:21
rm_worki think it didn't before23:21
rm_workbut we can't encrypt it <_<23:21
colby_ah ok that fixed that error. Now I get: Error([('SSL routines', 'ssl3_get_server_certificate', 'certificate verify failed')],23:23
colby_so Ill work on that tomorrow :)23:23
colby_thanks guys23:23
openstackgerritMerged openstack/octavia master: Imported Translations from Zanata

Generated by 2.15.3 by Marius Gedminas - find it at!