| johnsom | I will hammer out the other repos tomorrow. | 00:03 |
|---|---|---|
| *** henriqueof1 has joined #openstack-lbaas | 00:45 | |
| *** henriqueof has quit IRC | 00:45 | |
| *** tkajinam has quit IRC | 01:54 | |
| *** tkajinam has joined #openstack-lbaas | 02:11 | |
| *** hongbin has joined #openstack-lbaas | 02:32 | |
| *** tkajinam has quit IRC | 02:36 | |
| *** tkajinam has joined #openstack-lbaas | 02:56 | |
| *** sapd1 has joined #openstack-lbaas | 03:00 | |
| *** ramishra has joined #openstack-lbaas | 03:16 | |
| *** hongbin has quit IRC | 03:34 | |
| *** pcaruana has joined #openstack-lbaas | 05:02 | |
| *** gcheresh_ has joined #openstack-lbaas | 05:03 | |
| *** sapd1_x has joined #openstack-lbaas | 05:05 | |
| *** gcheresh_ has quit IRC | 05:23 | |
| *** tkajinam has quit IRC | 05:54 | |
| *** tkajinam has joined #openstack-lbaas | 06:03 | |
| *** rpittau|afk is now known as rpittau | 06:08 | |
| *** tkajinam_ has joined #openstack-lbaas | 06:19 | |
| *** sapd1_x has quit IRC | 06:20 | |
| *** tkajinam has quit IRC | 06:22 | |
| *** henriqueof has joined #openstack-lbaas | 06:27 | |
| *** henriqueof1 has quit IRC | 06:28 | |
| *** sapd1_x has joined #openstack-lbaas | 06:46 | |
| *** maciejjozefczyk has joined #openstack-lbaas | 06:52 | |
| *** trident has quit IRC | 06:55 | |
| lxkong | johnsom, rm_work, do you happen to know how to print out the actual db queries in octavia? | 06:56 |
| rm_work | I ... Have done it before, kinda | 06:58 |
| rm_work | But I do not remember how | 06:58 |
| *** sapd1_x has quit IRC | 06:59 | |
| cgoncalves | rm_work, can you approve https://review.opendev.org/#/c/659626 ? | 07:07 |
| *** trident has joined #openstack-lbaas | 07:07 | |
| cgoncalves | rm_work, also https://review.opendev.org/#/c/659627/ pretty please | 07:09 |
| rm_work | Maybe, depends on how the review goes :D | 07:11 |
| rm_work | Give me a minute | 07:11 |
| *** henriqueof has quit IRC | 07:12 | |
| *** henriqueof has joined #openstack-lbaas | 07:12 | |
| *** tesseract has joined #openstack-lbaas | 07:15 | |
| lxkong | rm_work: never mind, i found that, `[database]connection_debug` | 07:18 |
| lxkong | and i saw the ugly get loadbalancers query | 07:18 |
| rm_work | cool | 07:20 |
| rm_work | yes they're all horrifying | 07:20 |
| rm_work | SQLAlchemy makes some really painful looking queries that are not intended for human eyes | 07:21 |
| rm_work | BUT at least they're ... ok maybe not always efficient but... err... they're ... something | 07:21 |
| lxkong | but we have to look at it cause we met with a db performance issue | 07:21 |
| *** ramishra has quit IRC | 07:26 | |
| *** ramishra has joined #openstack-lbaas | 07:28 | |
| rm_work | cgoncalves: https://review.opendev.org/#/c/659626/19/octavia/api/v2/controllers/listener.py@544 | 07:51 |
| rm_work | if you can | 07:51 |
| rm_work | I think if you explain that to me I can +A | 07:51 |
| *** ivve has joined #openstack-lbaas | 08:00 | |
| *** tkajinam_ has quit IRC | 08:01 | |
| rm_work | cgoncalves? :P | 08:04 |
| rm_work | want to +A before I head to bed | 08:04 |
| rm_work | eh I can just +A and you can explain later -- in the off chance it's actually not necessary, we can bugfix it later | 08:05 |
| openstackgerrit | Carlos Goncalves proposed openstack/octavia master: Add VIP access control list https://review.opendev.org/659626 | 08:09 |
| cgoncalves | rm_work, ^ | 08:09 |
| rm_work | ugh lol k | 08:09 |
| rm_work | I just +A'd you :D | 08:09 |
| cgoncalves | nooooooo! :D | 08:10 |
| rm_work | ah well | 08:10 |
| rm_work | ok so i was right, there was no reason for that DB re-fetch | 08:10 |
| cgoncalves | yep, thanks! | 08:11 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/octavia-dashboard master: Imported Translations from Zanata https://review.opendev.org/681993 | 08:25 |
| *** ivve has quit IRC | 09:32 | |
| *** tkajinam has joined #openstack-lbaas | 10:05 | |
| *** ivve has joined #openstack-lbaas | 10:05 | |
| openstackgerrit | Maciej Józefczyk proposed openstack/octavia master: Add new algorithm SOURCE_IP_PORT https://review.opendev.org/672463 | 10:08 |
| openstackgerrit | Maciej Józefczyk proposed openstack/octavia master: Validate supported LB algorithm in Amphora provider drivers https://review.opendev.org/672477 | 10:08 |
| *** pcaruana has quit IRC | 10:11 | |
| *** luksky has joined #openstack-lbaas | 10:36 | |
| *** luksky has quit IRC | 11:09 | |
| lxkong | rm_work: hi, not sure if you are still here, I encountered an issue for fedora 28 amphora image(octavia queens), I can successfully create the load balancer and listener, but the haproxy service failed inside the amphora, error msg: `'/usr/sbin/haproxy-systemd-wrapper': No such file or directory` | 11:16 |
| *** sapd1_x has joined #openstack-lbaas | 11:22 | |
| *** pcaruana has joined #openstack-lbaas | 11:24 | |
| *** luksky has joined #openstack-lbaas | 11:26 | |
| *** luksky11 has joined #openstack-lbaas | 11:28 | |
| *** luksky has quit IRC | 11:31 | |
| *** luksky11 has quit IRC | 11:40 | |
| *** boden has joined #openstack-lbaas | 11:44 | |
| *** gcheresh_ has joined #openstack-lbaas | 11:50 | |
| *** luksky11 has joined #openstack-lbaas | 11:56 | |
| openstackgerrit | Carlos Goncalves proposed openstack/octavia-tempest-plugin master: Fix positional formatting and add skip check https://review.opendev.org/673168 | 12:02 |
| *** goldyfruit has quit IRC | 12:12 | |
| *** luksky11 has quit IRC | 12:22 | |
| *** luksky11 has joined #openstack-lbaas | 12:23 | |
| *** tkajinam has quit IRC | 12:25 | |
| *** sapd1_x has quit IRC | 12:31 | |
| openstackgerrit | Carlos Goncalves proposed openstack/octavia-tempest-plugin master: Fix positional formatting and add skip check https://review.opendev.org/673168 | 12:39 |
| *** henriqueof1 has joined #openstack-lbaas | 12:49 | |
| *** henriqueof has quit IRC | 12:50 | |
| openstackgerrit | Maciej Józefczyk proposed openstack/octavia master: Add new algorithm SOURCE_IP_PORT https://review.opendev.org/672463 | 12:53 |
| openstackgerrit | Maciej Józefczyk proposed openstack/octavia master: Validate supported LB algorithm in Amphora provider drivers https://review.opendev.org/672477 | 12:53 |
| *** gcheresh_ has quit IRC | 12:56 | |
| *** luksky11 has quit IRC | 13:06 | |
| *** Vorrtex has joined #openstack-lbaas | 13:26 | |
| *** goldyfruit has joined #openstack-lbaas | 13:28 | |
| *** Vorrtex has quit IRC | 13:42 | |
| *** goldyfruit has quit IRC | 14:14 | |
| *** ccamposr has quit IRC | 14:25 | |
| *** ccamposr has joined #openstack-lbaas | 14:25 | |
| *** goldyfruit has joined #openstack-lbaas | 14:34 | |
| *** goldyfruit_ has joined #openstack-lbaas | 14:42 | |
| *** rcernin has quit IRC | 14:42 | |
| *** goldyfruit has quit IRC | 14:44 | |
| johnsom | lxkong: so fedora must have 1.9 or newer. We need to fix that, but you can override in octavia.conf for now | 14:45 |
| johnsom | Haproxy dropped the wrapper. Which is really good actually | 14:45 |
| cgoncalves | I think F28 ships haproxy 1.8 | 14:53 |
| openstackgerrit | Maciej Józefczyk proposed openstack/octavia master: Add new algorithm SOURCE_IP_PORT https://review.opendev.org/672463 | 14:54 |
| openstackgerrit | Maciej Józefczyk proposed openstack/octavia master: Validate supported LB algorithm in Amphora provider drivers https://review.opendev.org/672477 | 14:54 |
| cgoncalves | Fedora<=30 has haproxy 1.8. Fedora>31= has 2.0.5 | 14:55 |
| johnsom | Hmm, maybe it was 1.8 that removed the wrapper. (the wrapper caused problems as I remember) But if that is the case we should have already handled that.... I will take a quick look at the osutils (which could use a refactor IMO) | 15:06 |
| ivve | hey guys, any pointers on where to look/troubleshoot for the following issue: created a member and tried to attach it to existing (active and ok) lb. followed by worker replying "TimeOutException: contacting the amphora timed out". setting the listener state in failure and trying to create new amphorae | 15:10 |
| ivve | i now have 4 amphorae for 1 LB (which is btw still working, but in error state). in the following states: | 15:10 |
| ivve | error:backup - error:master - error:standalone and allocated:none | 15:11 |
| ivve | noteworthy is that the allocated and the backup seems to be trying to grab the same IP | 15:11 |
| ivve | the standalone and master are reachable, the allocated and backup.. not at all | 15:12 |
| ivve | at one point worker tried to remove in-use security group.. other than that its just updating the listener to failure. but the loadbalancer does supply loadbalancing correctly and all members are reporting OK | 15:13 |
| ivve | i don't dare to amphora failover the master (that is working) trying to failover anything else just calls out immutable states | 15:16 |
| openstackgerrit | Michael Johnson proposed openstack/octavia master: Add new algorithm SOURCE_IP_PORT https://review.opendev.org/672463 | 15:17 |
| openstackgerrit | Michael Johnson proposed openstack/octavia master: Add new algorithm SOURCE_IP_PORT https://review.opendev.org/672463 | 15:20 |
| mloza | hello, in these policies https://docs.openstack.org/octavia/stein/configuration/policy.html#default-octavia-policies have default in the octavia code base? If yes, I don't need the full copy of the policy and change the specific to overwrite it right? | 15:22 |
| ivve | the only thing that is in error state is a listener and all its members are fine :/ | 15:22 |
| johnsom | mloza Yes, all of those policies are in code (we were one of the first to move to that). | 15:22 |
| mloza | johnsom: Awesome. Thanks! | 15:23 |
| *** maciejjozefczyk is now known as mjozefcz|away | 15:23 | |
| johnsom | ivve So that sounds like someone killed the health manager while it was in the middle of a failover. That would be the source of an "allocated" being assigned to the LB but not yet configured. | 15:25 |
| *** tesseract has quit IRC | 15:25 | |
| johnsom | ivve It is designed to "fail safe" and leave the LB functional even if the provisioning status is ERROR. Thus, the operating status should all be ok. | 15:26 |
| johnsom | This is going to be a tricky one to back out of. (Though, my new failover flow work will handle this situation. It's just not anywhere near ready yet) | 15:27 |
| *** dayou has quit IRC | 15:28 | |
| *** ivve has quit IRC | 15:29 | |
| johnsom | I would do the following: | 15:29 |
| johnsom | Check if the allocated amp compute ID is present in nova. If yes, delete it out of nova, then mark it deleted in the amphora table. | 15:30 |
| johnsom | I would log into the standalone instance, see if it is handling any traffic. I.e. look at the haproxy logs or tcpdump inside the netns. | 15:31 |
| johnsom | If it is handling traffic we need to get more creative, if not, I would delete it in nova and set it's amp record to deleted. | 15:31 |
| johnsom | Then look at backup, if it is present in nova, take note. It likely is not. | 15:33 |
| johnsom | Then I would attempt to failover the backup amphora. Do not use LB failover | 15:33 |
| johnsom | Also, I would make sure you are running the latest version of whichever release you are on. It will make your life better. | 15:34 |
| *** dayou has joined #openstack-lbaas | 15:39 | |
| *** mjozefcz|away has quit IRC | 15:43 | |
| *** ramishra has quit IRC | 15:51 | |
| *** rpittau is now known as rpittau|afk | 16:22 | |
| openstackgerrit | Merged openstack/octavia stable/rocky: Add warning log if auth_strategy is not keystone https://review.opendev.org/678548 | 16:24 |
| openstackgerrit | Merged openstack/octavia master: Add VIP access control list https://review.opendev.org/659626 | 17:09 |
| openstackgerrit | Merged openstack/octavia-dashboard master: Imported Translations from Zanata https://review.opendev.org/681993 | 17:13 |
| *** luksky11 has joined #openstack-lbaas | 17:25 | |
| openstackgerrit | Michael Johnson proposed openstack/octavia-lib master: Generate PDF documentation https://review.opendev.org/682120 | 17:28 |
| *** goldyfruit_ has quit IRC | 17:46 | |
| openstackgerrit | Michael Johnson proposed openstack/octavia-tempest-plugin master: Generate PDF documentation https://review.opendev.org/682124 | 17:48 |
| *** goldyfruit_ has joined #openstack-lbaas | 17:49 | |
| openstackgerrit | Michael Johnson proposed openstack/octavia-lib master: Generate PDF documentation https://review.opendev.org/682120 | 17:51 |
| *** gcheresh_ has joined #openstack-lbaas | 18:00 | |
| *** mjozefcz|away has joined #openstack-lbaas | 18:18 | |
| openstackgerrit | Michael Johnson proposed openstack/python-octaviaclient master: Generate PDF documentation https://review.opendev.org/682134 | 18:27 |
| openstackgerrit | Michael Johnson proposed openstack/octavia-dashboard master: Generate PDF documentation https://review.opendev.org/679283 | 18:49 |
| johnsom | Joy more fallout from all of those last minute global requirements changes | 18:55 |
| *** ccamposr has quit IRC | 18:56 | |
| *** ccamposr has joined #openstack-lbaas | 18:56 | |
| johnsom | client and lib appear to have issues | 18:57 |
| johnsom | I am working on client now | 18:57 |
| *** goldyfruit___ has joined #openstack-lbaas | 18:59 | |
| *** mjozefcz|away has quit IRC | 19:01 | |
| *** goldyfruit_ has quit IRC | 19:02 | |
| mloza | I have "os_load-balancer_api:loadbalancer:put_failover": "load-balancer:write" in the octavia/policy.json. As a user with load_balancer-member role, I can | 19:03 |
| mloza | I can't failover the LB* | 19:03 |
| openstackgerrit | Michael Johnson proposed openstack/python-octaviaclient master: Fix a bad unit test for amphora list https://review.opendev.org/682139 | 19:04 |
| openstackgerrit | Michael Johnson proposed openstack/python-octaviaclient master: Generate PDF documentation https://review.opendev.org/682134 | 19:04 |
| mloza | I want to allow load_balancer-member to failover LBs | 19:04 |
| johnsom | mloza One minute, I need to finish something, then I will chat | 19:05 |
| mloza | Sure | 19:05 |
| openstackgerrit | Michael Johnson proposed openstack/python-octaviaclient master: Add support to VIP access control list https://review.opendev.org/659627 | 19:05 |
| openstackgerrit | Michael Johnson proposed openstack/python-octaviaclient master: Wrap several show api calls with correct_return_codes https://review.opendev.org/675331 | 19:05 |
| johnsom | Cores (rm_work, cgoncalves, xgerman) we need this patch reviewed: https://review.opendev.org/#/c/682139/ it's a blocker for the client release today | 19:07 |
| johnsom | mloza Ok, hi. I home that is not for a general population. LB failover is a big hammer. Let me refresh my memory on the policy | 19:08 |
| rm_work | done | 19:08 |
| johnsom | Thanks | 19:08 |
| xgerman | +1 | 19:08 |
| johnsom | mloza That looks correct. Does the user own the load balancer they are attempting to failover? | 19:09 |
| mloza | johnsom: even the user who owns the LB can't failover | 19:10 |
| mloza | I just tried just now | 19:11 |
| mloza | getting Policy does not allow this request to be performed. (HTTP 403) (Request-ID: req-43745c9d-67d7-446e-875d-a77365dbaccb) | 19:11 |
| johnsom | mloza Ok, so likely the policy file is not getting picked up. | 19:12 |
| johnsom | mloza Your custom policy is in /etc/octavia/policy.json on the octavia API controllers (all of them)? | 19:13 |
| mloza | johnsom: yes | 19:13 |
| *** boden has quit IRC | 19:14 | |
| rm_work | yeah letting users do failovers is ... O_o | 19:14 |
| rm_work | but hopefully it's just a few people using this service and you trust them all very much? :D | 19:15 |
| mloza | The default policy "os_load-balancer_api:loadbalancer:put_failover": "rule:load-balancer:admin" allows tenants to list all the LB's | 19:16 |
| johnsom | mloza Can you paste that file to paste.openstack.org ? | 19:16 |
| johnsom | mloza How would the failover put allow users to list LBs? | 19:18 |
| mloza | johnsom: if I gave the users loadbalancer-admin role they can list all LBs | 19:20 |
| mloza | johnsom: http://paste.openstack.org/show/775786/ | 19:20 |
| rm_work | yes, that is the design | 19:20 |
| johnsom | Right, Admin is admin, they can do anything in the cloud | 19:20 |
| rm_work | not all users should get admin :D | 19:21 |
| johnsom | or in the case of our role, they can do anything on any LB | 19:21 |
| mloza | I know loadbalancer-admin can do failover but we don | 19:22 |
| mloza | I know loadbalancer-admin can do failover but we dont want them list all LBs | 19:22 |
| rm_work | you shouldn't need it to do a failover, you should be able to do failover on the failover role | 19:23 |
| rm_work | if not, we have a bug to fix | 19:23 |
| johnsom | Yeah, this is all working correctly for me locally | 19:23 |
| johnsom | mloza So you probably do not want to use the loadbalancer-admin role. It is like "admin" in all of the other projects and allows everything. | 19:24 |
| johnsom | It is a cross-project role | 19:24 |
| openstackgerrit | Merged openstack/python-octaviaclient master: Fix a bad unit test for amphora list https://review.opendev.org/682139 | 19:27 |
| openstackgerrit | Merged openstack/python-octaviaclient master: Add support to VIP access control list https://review.opendev.org/659627 | 19:28 |
| openstackgerrit | Michael Johnson proposed openstack/python-octaviaclient master: Add support for SOURCE_IP_PORT algorithm https://review.opendev.org/672416 | 19:29 |
| *** gcheresh_ has quit IRC | 19:29 | |
| mloza | I dont see a failover role. The ones I have are lb-quota-admin, lb-member, lb-admin and lb-observer | 19:30 |
| johnsom | Right, you may need to create a new one | 19:30 |
| mloza | so this policy "os_load-balancer_api:loadbalancer:put_failover": "load-balancer:write" doesnt allow a user with lb-member role do a failover? | 19:31 |
| johnsom | I am looking at what is going on with your file. Give me a minute and I will figure it out. | 19:32 |
| johnsom | mloza Your policy.json is mal-formed. It should be: "os_load-balancer_api:loadbalancer:put_failover": "rule:load-balancer:write" | 19:36 |
| johnsom | You dropped the "rule:" prefix | 19:36 |
| openstackgerrit | Merged openstack/python-octaviaclient master: Wrap several show api calls with correct_return_codes https://review.opendev.org/675331 | 19:38 |
| mloza | johnsom: Oh yeah. My mistake I copied it incorrectly | 19:40 |
| mloza | Probably this should work now | 19:40 |
| mloza | Works | 19:41 |
| johnsom | We have pretty extensive testing for the policies, so I would have been surprised if it was actually broken. | 19:42 |
| mloza | Yeah. I didn't notice it since octavia containers didn't crash. Neutron crashes when I had a typo in policy.json | 19:44 |
| johnsom | That is odd and probably a bug in neutron. We just don't match the rule so you don't get permission. | 19:45 |
| openstackgerrit | Merged openstack/octavia master: Add new algorithm SOURCE_IP_PORT https://review.opendev.org/672463 | 19:58 |
| openstackgerrit | Michael Johnson proposed openstack/octavia master: Fix the tips job for octavia-lib https://review.opendev.org/682148 | 20:03 |
| openstackgerrit | Merged openstack/python-octaviaclient master: Add support for SOURCE_IP_PORT algorithm https://review.opendev.org/672416 | 20:33 |
| *** baffle has quit IRC | 20:42 | |
| *** openstackgerrit has quit IRC | 20:51 | |
| *** baffle has joined #openstack-lbaas | 20:53 | |
| *** henriqueof1 has quit IRC | 20:54 | |
| *** openstackgerrit has joined #openstack-lbaas | 21:00 | |
| openstackgerrit | Merged openstack/octavia stable/queens: Add failover logging to show the amphora details. https://review.opendev.org/679770 | 21:00 |
| *** pcaruana has quit IRC | 21:03 | |
| johnsom | FYI: The train client is up for release: https://review.opendev.org/#/c/681799 | 21:13 |
| *** luksky11 has quit IRC | 21:14 | |
| *** KeithMnemonic has quit IRC | 21:19 | |
| openstackgerrit | Michael Johnson proposed openstack/octavia master: Set neutron client logging to INFO https://review.opendev.org/682167 | 21:35 |
| johnsom | Let's see if that reduces the neutron noise in our logs a bit. | 21:36 |
| openstackgerrit | OpenStack Release Bot proposed openstack/python-octaviaclient stable/train: Update .gitreview for stable/train https://review.opendev.org/682169 | 21:49 |
| openstackgerrit | OpenStack Release Bot proposed openstack/python-octaviaclient stable/train: Update TOX/UPPER_CONSTRAINTS_FILE for stable/train https://review.opendev.org/682170 | 21:49 |
| openstackgerrit | OpenStack Release Bot proposed openstack/python-octaviaclient master: Update master for stable/train https://review.opendev.org/682171 | 21:49 |
| *** ccamposr has quit IRC | 22:07 | |
| rm_work | Gotta run to a wedding reception -- catch y'all on Monday :) | 22:40 |
| rm_work | Unless you need a review, then just ping me | 22:40 |
| xgerman | it’s not your own? | 22:40 |
| rm_work | Friend from gradeschool | 22:40 |
| openstackgerrit | Michael Johnson proposed openstack/octavia-tempest-plugin master: Enable fail-fast on the gate queue https://review.opendev.org/682185 | 22:43 |
| *** goldyfruit___ has quit IRC | 22:49 | |
| openstackgerrit | Michael Johnson proposed openstack/octavia-dashboard master: Enable fail-fast on the gate queue https://review.opendev.org/682188 | 22:54 |
| openstackgerrit | Michael Johnson proposed openstack/octavia-dashboard master: Generate PDF documentation https://review.opendev.org/679283 | 23:02 |
| openstackgerrit | Michael Johnson proposed openstack/octavia master: Set neutron client logging to INFO https://review.opendev.org/682167 | 23:09 |
| johnsom | Ok, I think I am done blasting out patches and releases for the day. | 23:56 |
| *** rcernin has joined #openstack-lbaas | 23:56 | |
| johnsom | The PDF patches are probably good enough for a first pass / meet the goal. | 23:56 |
| johnsom | The main octavia one is still missing sections due to some unknown bug. | 23:56 |
| johnsom | The lib patch is waiting on the tips fix in octavia. | 23:57 |
| johnsom | There are also automated patches ready for a second +2 on client and lib. The lib one will fix the release notes | 23:57 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!