Wednesday, 2017-07-19

*** tongl has quit IRC00:52
*** csomerville has quit IRC00:56
*** cody-somerville has joined #openstack-lbaas01:03
*** harlowja has quit IRC01:13
*** dougwig has quit IRC01:24
*** ipsecguy has joined #openstack-lbaas01:32
*** ipsecguy_ has quit IRC01:33
*** sanfern has joined #openstack-lbaas02:04
*** yamamoto has joined #openstack-lbaas02:17
*** sanfern has quit IRC02:18
*** yamamoto has quit IRC02:38
*** yamamoto has joined #openstack-lbaas02:48
*** diltram has quit IRC03:50
*** gcheresh has joined #openstack-lbaas03:52
*** yamamoto has quit IRC03:57
*** links has joined #openstack-lbaas03:58
*** yamamoto has joined #openstack-lbaas03:59
*** diltram has joined #openstack-lbaas04:01
*** yamamoto has quit IRC04:04
*** yamamoto has joined #openstack-lbaas04:05
*** gcheresh has quit IRC04:14
*** yamamoto has quit IRC04:14
*** reedip has quit IRC04:14
*** reedip has joined #openstack-lbaas04:15
*** yamamoto has joined #openstack-lbaas04:15
*** yamamoto has quit IRC04:21
*** sanfern has joined #openstack-lbaas04:25
*** harlowja has joined #openstack-lbaas04:31
openstackgerritMerged openstack/neutron-lbaas master: Enhancements for the the back-end system  https://review.openstack.org/47961404:40
*** yamamoto has joined #openstack-lbaas04:44
*** harlowja has quit IRC04:47
openstackgerritOpenStack Proposal Bot proposed openstack/neutron-lbaas master: Updated from global requirements  https://review.openstack.org/48337104:47
*** gcheresh has joined #openstack-lbaas05:03
*** afranc has quit IRC05:10
*** diltram has quit IRC05:13
*** armax has quit IRC05:14
*** armax has joined #openstack-lbaas05:15
*** armax has quit IRC05:15
*** diltram has joined #openstack-lbaas05:16
*** armax has joined #openstack-lbaas05:16
*** armax has quit IRC05:16
*** armax has joined #openstack-lbaas05:17
*** armax has quit IRC05:17
*** armax has joined #openstack-lbaas05:17
*** armax has quit IRC05:18
*** armax has joined #openstack-lbaas05:18
*** armax has quit IRC05:18
*** afranc has joined #openstack-lbaas05:21
*** harlowja has joined #openstack-lbaas05:46
*** rcernin has joined #openstack-lbaas05:50
*** diltram has quit IRC06:41
*** harlowja has quit IRC06:42
*** diltram has joined #openstack-lbaas06:47
*** diltram has quit IRC06:58
*** diltram has joined #openstack-lbaas07:08
*** tesseract has joined #openstack-lbaas07:17
*** aojea has joined #openstack-lbaas07:20
*** catintheroof has joined #openstack-lbaas07:31
*** catintheroof has quit IRC07:36
*** diltram has quit IRC07:38
*** diltram has joined #openstack-lbaas07:45
*** diltram has quit IRC07:52
*** diltram has joined #openstack-lbaas07:54
*** diltram has quit IRC08:01
*** diltram has joined #openstack-lbaas08:08
*** cody-somerville has quit IRC08:36
*** cody-somerville has joined #openstack-lbaas08:36
*** cody-somerville has quit IRC08:36
*** cody-somerville has joined #openstack-lbaas08:36
*** diltram has quit IRC08:44
*** diltram has joined #openstack-lbaas08:47
*** openstackgerrit has quit IRC08:49
*** yamamoto has quit IRC09:06
*** yamamoto has joined #openstack-lbaas09:10
*** yamamoto has quit IRC09:10
*** mjblack has quit IRC09:10
*** yamamoto has joined #openstack-lbaas09:14
*** diltram has quit IRC09:24
*** diltram has joined #openstack-lbaas09:28
*** yamamoto has quit IRC09:36
*** yamamoto has joined #openstack-lbaas09:36
*** kobis has joined #openstack-lbaas09:42
*** atoth has quit IRC10:17
*** gcheresh_ has joined #openstack-lbaas10:28
*** gcheresh has quit IRC10:28
*** yamamoto has quit IRC10:55
*** atoth has joined #openstack-lbaas11:05
*** Alex_Staf has joined #openstack-lbaas11:29
*** dougwig has joined #openstack-lbaas11:32
Alex_Stafrm_work, ping11:37
*** yamamoto has joined #openstack-lbaas11:40
*** chlong_ has quit IRC11:43
*** aojea has quit IRC11:53
*** aojea has joined #openstack-lbaas12:02
*** yamamoto has quit IRC12:03
*** aojea has quit IRC12:07
*** kobis has quit IRC12:19
*** yamamoto has joined #openstack-lbaas12:21
*** aojea has joined #openstack-lbaas12:21
*** catintheroof has joined #openstack-lbaas12:21
Alex_Stafrm_work, Hi, I am Octavia QE, I was wondering if there is haproxy l7 tests that are public.12:25
*** aojea has quit IRC12:25
*** aojea has joined #openstack-lbaas12:39
*** aojea has quit IRC12:44
*** aojea has joined #openstack-lbaas12:48
*** aojea has quit IRC12:53
*** aojea has joined #openstack-lbaas12:57
*** aojea has quit IRC13:02
*** aojea has joined #openstack-lbaas13:06
*** aojea has quit IRC13:11
*** aojea has joined #openstack-lbaas13:25
*** aojea has quit IRC13:30
*** gcheresh_ has quit IRC13:42
*** aojea has joined #openstack-lbaas13:48
*** sanfern has quit IRC13:57
*** armax has joined #openstack-lbaas13:59
*** yamamoto has quit IRC14:09
*** links has quit IRC14:18
tomtomtom@johnsom got octavia v2 working for not https loadbalancing, thanks for the help! and @rm_work and @xgerman14:54
tomtomtomin v2 is https load balancing working for barbican?14:54
johnsomYes, it should be working, it was tested during the v2 development ( I spun it up for the API reference examples)14:55
tomtomtomi've configured barbican_cert_manager and barbican_acl_auth within octavia.conf, does it require an service_auth section or anything like that?14:56
nmagnezijohnsom, o/15:00
nmagnezijohnsom, a question about octavia client. is it fully ready? or are we have some parts of the api to implement?15:03
johnsomtomtomtom Yes, the service_auth section is needed.  There is also some either barbican RBAC or using the CLI ACLs that needs to happen.15:08
*** yamamoto has joined #openstack-lbaas15:09
johnsomnmagnezi The API is done aside from flavors and providers to my knowledge.  The CLI is mostly done, I think missing status, stats, and quota still.  I'm working on the OpenStack SDK as we speak15:09
nmagnezijohnsom, ack. thanks!15:11
nmagnezijohnsom, btw added something to the agenda for today15:12
johnsomOk, cool, I have not got there yet, but will soon15:12
*** yamamoto has quit IRC15:19
tomtomtomyeah I added the octavia user to the acl for barbican already, but I only have a keystone auth section, not a service_auth section, I'll put that in.15:26
johnsomI posted our agenda for today: https://wiki.openstack.org/wiki/Octavia/Weekly_Meeting_Agenda#Meeting_2017-07-1915:32
johnsomI tagged xgerman_ for the L3 active/active discussion we had.  If  you post some comments on the spec maybe we can skip that section of the agenda, otherwise we can discuss.15:32
*** Alex_Staf has quit IRC15:32
*** ssmith has joined #openstack-lbaas15:36
ssmithjohnsom: Do you know what user Octavia uses when accessing the Barbican secret store. With Neutron LBaaS we had to acl user add the admin user that it was using for Neutron LBaaS in order for the LB to read the SSL Certificate.15:37
johnsomssmith It will be the user you specify as the service_auth user in your octavia.conf.  This can be the deployment "admin" user or could be a "octavia_service" user that has been granted the correct roles in the other services (nova, neutron, barbican, etc.)15:40
ssmithhttps://bugs.launchpad.net/barbican/+bug/1627391 you wrote "I think this bug is less important than the cascade ACL in bug 159261215:41
openstackLaunchpad bug 1627391 in Barbican "Regular users do not have access to 'admin' ID when creating ACLs" [Undecided,Triaged] - Assigned to Douglas Mendizábal (dougmendizabal)15:41
ssmithIf we have the cascade ACL feature we can eliminate this requirement from user workflow in lbaas/octavia." So is cascade acl working or this acl user add is required?15:41
openstackbug 1592612 in octavia "LBaaS TLS is not working with non-admin tenant" [High,Confirmed] https://launchpad.net/bugs/159261215:41
*** dougwig has quit IRC15:42
johnsomssmith I have not checked in with the barbican project to see if it got implemented.  Last time I checked, a few months ago, it had not yet been implemented.15:42
*** openstackgerrit has joined #openstack-lbaas15:42
openstackgerritXing Zhang proposed openstack/octavia master: Fix haproxy_check_script for delete listener  https://review.openstack.org/48525415:42
johnsomOnce that is added we can make the ACL issue transparent to the end users or you don't have to change the RBAC for barbican15:43
tomtomtomso it sounds like the service_auth section for octavia needs to be set to auth to the admin project rather than the service project?15:48
xgerman_yep, there is a special babrbican section15:49
johnsomWell, you can give it the "admin" role and the rest of the services won't need RBAC changes.  If you set it up as a service account, you need to setup the RBAC in the other services to allow that account acceess.15:49
tomtomtom"special barbican section" in octavia.conf?  is it [certificates]?15:51
xgerman_I think so but it mostly allows you to spwcify the endpoint.15:54
*** sanfern has joined #openstack-lbaas16:06
*** rcernin has quit IRC16:09
*** aojea has quit IRC16:13
*** aojea has joined #openstack-lbaas16:14
*** sanfern has quit IRC16:17
*** sanfern has joined #openstack-lbaas16:18
*** aojea has quit IRC16:18
*** rcernin has joined #openstack-lbaas16:23
tomtomtomI see a tls_certificate_id in the octavia table, however, where's the ref stored? I see an sni table but no tls table? what is the tls_certificate_id referring to?16:27
tomtomtom*octavia listener table*16:27
*** sanfern has quit IRC16:27
johnsomtls_certificate_id is really the barbican href16:28
johnsomthe raw DB tables still have old terminology16:29
tomtomtomok, just having trouble setting a default-tls-container-ref for the listener, debug (from openstack command) always shows none even though there are no errors.16:31
johnsomCan you run "openstack --debug loadbalancer ..." and pastebin?16:32
johnsomMaybe there is a CLI bug16:32
*** aojea has joined #openstack-lbaas16:41
*** aojea has quit IRC16:45
*** aojea has joined #openstack-lbaas16:50
*** sshank has joined #openstack-lbaas16:52
*** aojea has quit IRC16:54
*** jniesz has joined #openstack-lbaas16:57
johnsomOctavia meeting starting soon on #openstack-meeting16:58
*** sanfern has joined #openstack-lbaas17:01
ssmithAny hints? We're getting SSL_ERROR_RX_RECORD_TOO_LONG on a new LB17:03
tomtomtomas a side note since the command wasn't working I did a mysql update for the tls_container_id to manually put in the href, but no luck doing that, probably wrong anyway.17:16
*** aojea has joined #openstack-lbaas17:16
*** rm_mobile has joined #openstack-lbaas17:17
*** aojea has quit IRC17:22
*** harlowja has joined #openstack-lbaas17:25
*** harlowja has quit IRC17:25
*** harlowja has joined #openstack-lbaas17:26
*** JudeC has joined #openstack-lbaas17:43
*** tesseract has quit IRC17:46
rm_worktomtomtom: the command wasn't working? which command?17:47
tomtomtomopenstack loadbalancer listener create;  when passing the --default-tls-confainer-ref http://barbican/container  it is not updating the tls_container_id in the listener table.17:49
rm_workhmmm17:51
rm_worki haven't done a ton of testing with tls recently17:51
rm_workbut that seems odd17:51
*** atoth has quit IRC17:51
*** atoth has joined #openstack-lbaas17:57
johnsomtomtomtom Your href looks like this right? https://developer.openstack.org/api-ref/load-balancer/v2/index.html?expanded=create-listener-detail#id3218:05
*** atoth has quit IRC18:06
JudeCIt could possibly be the client not working as intended as well.18:07
johnsomThat is what I was thinking, maybe a bug crept in to the client.  That is why I was asking for the debug output.18:07
JudeCIm finally back by the way :)18:08
johnsomYou could try updating it via the API directly (how I tested it last)18:08
tomtomtomit looks like this exactly: http://198.51.100.10:9311/v1/containers/a570068c-d295-4780-91d4-3046a325db5118:08
johnsomhttps://developer.openstack.org/api-ref/load-balancer/v2/index.html?expanded=update-a-listener-detail#id4218:08
johnsomHmm, that didn't open right for me.  It's the update a listener section, I include the curl command line to do it direct to the API18:09
johnsomJudeC Welcome back!18:09
tomtomtomok, so i don't have sni, is that an issue?18:09
tomtomtomI just use the tls containers from barbican18:10
johnsomNo, SNI is totally optional18:10
tomtomtomright thats what I thought....18:10
johnsomOk, mascot e-mail sent18:15
johnsomTime for lunch before my next meeting.18:15
openstackgerritOpenStack Proposal Bot proposed openstack/neutron-lbaas master: Updated from global requirements  https://review.openstack.org/48337118:38
*** rm_mobile has quit IRC18:45
*** sanfern has quit IRC18:58
*** aojea has joined #openstack-lbaas19:05
*** gcheresh_ has joined #openstack-lbaas19:06
*** sshank has quit IRC19:07
*** aojea has quit IRC19:10
*** aojea has joined #openstack-lbaas19:15
*** chlong_ has joined #openstack-lbaas19:15
*** aojea has quit IRC19:19
*** kbyrne has quit IRC19:22
*** aojea has joined #openstack-lbaas19:24
*** kbyrne has joined #openstack-lbaas19:25
*** aojea has quit IRC19:28
*** aojea has joined #openstack-lbaas19:33
tomtomtomok so I've figured out that the haproxy on the amphora instance is removing the instances from the backend.19:33
johnsomThey are failing the health monitor check?19:36
*** aojea has quit IRC19:38
*** gcheresh_ has quit IRC19:42
*** tinyurl_comSLASH has joined #openstack-lbaas19:50
tomtomtomyes, and I can't figure out why19:50
tomtomtomthe server telnets to port 80 and can curl it.19:50
tomtomtomi'm trying to add some extra options to haproxy to get more verbose in logs19:51
*** jniesz has quit IRC19:51
*** tinyurl_comSLASH has left #openstack-lbaas19:53
*** dougwig has joined #openstack-lbaas19:53
*** chlong_ has quit IRC20:03
*** sshank has joined #openstack-lbaas20:17
*** diltram has quit IRC20:18
*** diltram has joined #openstack-lbaas20:30
*** aojea has joined #openstack-lbaas20:36
*** diltram has quit IRC20:37
*** diltram has joined #openstack-lbaas20:41
*** aojea has quit IRC20:41
*** sshank has quit IRC20:42
*** sshank has joined #openstack-lbaas20:42
*** aojea has joined #openstack-lbaas20:46
*** chlong_ has joined #openstack-lbaas20:48
*** aojea has quit IRC20:50
*** aojea has joined #openstack-lbaas20:55
*** jniesz has joined #openstack-lbaas20:58
*** aojea has quit IRC20:59
*** aojea has joined #openstack-lbaas21:04
*** chlong_ has quit IRC21:04
*** aojea has quit IRC21:08
*** aojea has joined #openstack-lbaas21:13
openstackgerritXing Zhang proposed openstack/octavia master: Fix haproxy_check_script for delete listener  https://review.openstack.org/48525421:14
*** aojea has quit IRC21:18
tomtomtom@johnsom how would I restart the haproxy service on an amphora instance? when I do via service or systemctl I lose the vip i'm trying to use.21:21
*** aojea has joined #openstack-lbaas21:22
johnsomYou should be able to systemctl restart the haproxy-<uuid>21:22
johnsomIf you are loosing the vip, something is screwy in your networking.21:22
johnsomIt does rebuild the network namespace when you restart that process21:23
johnsomIf you just want to pickup a config change, use reload instead of restart21:23
*** sshank has quit IRC21:25
*** aojea has quit IRC21:27
tomtomtomok not losing vip, it keeps booting out all my backend servers even though they're reachable on port 8021:27
*** aojea has joined #openstack-lbaas21:31
*** sshank has joined #openstack-lbaas21:32
tomtomtomthis is the reason it states for the issue: Jul 19 21:31:42 amphora-81873411-447f-4233-af6d-a9388eb18a41 haproxy[2071]: Health check for server 3730d956-a81e-409d-abce-e11a6e6eb905/e778e742-5169-4e04-aaf6-09651d0f02af failed, reason: Layer4 timeout, info: " at initial connection step of tcp-check", check duration: 5001ms, status: 0/1 DOWN.21:32
rm_worktomtomtom: you can go into the namespace and try to curl the healthcheck url on your members21:35
rm_worki find that useful for debugging21:35
rm_workusually it's not haproxy's fault21:35
tomtomtomok21:35
tomtomtomi'll give that a try21:35
rm_worktomtomtom: sudo ip netns exec amphora-haproxy bash21:35
rm_workand yeah just start curling the members21:35
rm_worksee what comes back21:35
*** aojea has quit IRC21:35
johnsomYeah, that is saying the TCP port isn't open.  I would bet a security group or iptables rule is blocking the LB source addr21:36
tomtomtomaha! ok thanks @rm_work I did not realize there was a net namespace on the amphora instance, that helps a lot thanks!21:37
rm_workyeah prolly that's why you aren't seeing the VIP :)21:37
johnsomOh!  Yeah, all tenant traffic is isolated to the network namespace21:37
rm_workit doesn't show up in the main namespace21:37
rm_workonce you are in that namespace, "ifconfig" or whatever will show you the VIP21:37
rm_workwoo ok finally running with gunicorn instead of uwsgi21:38
tomtomtomyeah i see that so haproxy will use this net namespace for health checks?21:38
rm_worki really dislike uwsgi21:38
rm_worktomtomtom: yes21:39
rm_workeverything haproxy lives inside the ns21:39
johnsomYes, haproxy is inside the namespace21:39
tomtomtomok well I see a basic issue with the way the LB was created inside the network, I currently only have the namespace on the external network, it can't reach the private ip's of the web servers from there.21:40
*** aojea has joined #openstack-lbaas21:40
johnsomWhen you add the member to the load balancer you specify the subnet they are reachable from, that would cause those networks to get hot-plugged into the namespace21:42
*** aojea has quit IRC21:44
*** tongl has joined #openstack-lbaas21:45
tomtomtomok21:49
*** aojea has joined #openstack-lbaas21:49
*** cpusmith has joined #openstack-lbaas21:50
tomtomtomyeah ok I don't think the hot plugging is working because the subnet my vm instances are on are not reachable via the namespace.... example my vm is on 10.20.10.x and I would expect the namespace to be able to reach that subnet but it cannot.21:50
openstackgerritJason Niesz proposed openstack/octavia master: blueprint: l3-active-active  https://review.openstack.org/45300521:51
tomtomtomi bound a 10.20.10.x ip to the interface in the namespace hoping it would come on, but it did not work.21:52
*** ssmith has quit IRC21:53
*** aojea has quit IRC21:53
*** cpusmith has quit IRC21:54
johnsomtomtomtom Yeah, that will not work, it has to come through neutron, etc.  Just delete the member from the pool and add it back with the proper subnet_id for your backend servers.21:57
johnsomIf that is not adding the interface into the namespace, check the o-cw and amphora agent logs (amphora-agent and syslog in the amp)21:57
johnsomBut if the member isn't going into provisioning_status ERROR that means the plug was successful21:58
*** aojea has joined #openstack-lbaas21:59
*** aojea has quit IRC22:03
*** aojea has joined #openstack-lbaas22:08
*** aojea has quit IRC22:12
*** aojea has joined #openstack-lbaas22:17
*** aojea has quit IRC22:21
*** sshank has quit IRC22:25
*** aojea has joined #openstack-lbaas22:26
*** aojea has quit IRC22:30
*** aojea has joined #openstack-lbaas22:35
*** aojea has quit IRC22:39
*** sshank has joined #openstack-lbaas22:44
*** ssmith has joined #openstack-lbaas22:44
*** jniesz has quit IRC22:58
rm_workanyone know what causes this?23:02
*** aojea has joined #openstack-lbaas23:02
rm_work /venv/lib/python2.7/site-packages/pymysql/cursors.py:166: Warning: (139, u'Row size too large (> 8126). Changing some columns to TEXT or BLOB may help. In current row format, BLOB prefix of 0 bytes is stored inline.')23:02
rm_workjohnsom: ^^23:03
*** catintheroof has quit IRC23:03
rm_workthere's some notes about having to set a log file size larger on the mysql side23:03
rm_workbut23:03
*** yamamoto has joined #openstack-lbaas23:04
rm_workthis is on the app side O_o23:04
johnsomSomeone trying to put too large of data in a field?23:04
*** aojea has quit IRC23:07
tonglQuick question on the L7 rule. For example if I want to define a rule to match cookie_name=foo cookie_value=bar, the type is COOKIE, what should we specify for value?23:09
tonglChecked the api-ref, and it seems we can use key=foo, value=bar to match the cookie.23:11
rm_workjohnsom: i mean... this is octavia23:11
rm_workso ??23:11
rm_workone of our tables has a large row size i guess23:11
johnsomtongl It would be type=COOKIE, key=foo, value=bar23:11
*** aojea has joined #openstack-lbaas23:11
tongljohnsom: thanks!23:12
johnsomI would be surprised that we have a column over 8126....23:12
johnsomtongl This may be helpful: https://developer.openstack.org/api-ref/load-balancer/v2/index.html?expanded=create-an-l7-policy-detail,create-an-l7-rule-detail#create-an-l7-rule23:12
johnsomand the cookbooks here: https://docs.openstack.org/octavia/latest/user/index.html23:13
rm_workjohnsom: hmm, not sure what else it'd be23:13
johnsomtongl This one is basically what you are doing: https://docs.openstack.org/octavia/latest/user/guides/l7-cookbook.html#send-unauthenticated-users-to-login-pool-scenario-123:13
tonglAwesome, this is exactly I am looking for.23:14
*** aojea has quit IRC23:16
*** yamamoto has quit IRC23:18
rm_workBTW what are we doing about:23:20
rm_work2017-07-19 23:01:37.854 25 WARNING keystonemiddleware.auth_token [-] Using the in-process token cache is deprecated as of the 4.2.0 release and may be removed in the 5.0.0 release or the 'O' development cycle. The in-process cache causes inconsistent results and high memory usage. When the feature is removed the auth_token middleware will notcache tokens by default which may result in performance issues. It is recommended to use23:20
rm_work memcache for the auth_token token cache by setting the memcached_servers option.23:20
rm_worki guess doesn't matter for devstack23:21
rm_workso we just... let it disappear?23:21
rm_workand it's a deployer issue?23:21
*** aojea has joined #openstack-lbaas23:29
*** rcernin has quit IRC23:34
*** aojea has quit IRC23:34
*** sshank has quit IRC23:34
*** aojea has joined #openstack-lbaas23:38
johnsomI think so, we have the setting in the conf file23:42
johnsomI mean, we aren't setting that to in-process are we?  Isn't it picking that up from devstack?23:43
*** aojea has quit IRC23:43
*** aojea has joined #openstack-lbaas23:47
*** aojea has quit IRC23:52

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!