| openstackgerrit | Michael Johnson proposed openstack/octavia master: Implement provider drivers - Driver Library https://review.openstack.org/571358 | 00:04 |
|---|---|---|
| openstackgerrit | Michael Johnson proposed openstack/neutron-lbaas master: Neutron-LBaaS to Octavia migration tool https://review.openstack.org/578942 | 00:16 |
| johnsom | Boom, the previous run was successful so this patch is my "for review" version with the job as a periodic. | 00:16 |
| *** longkb has joined #openstack-lbaas | 00:24 | |
| *** abaindur has quit IRC | 00:24 | |
| *** abaindur has joined #openstack-lbaas | 00:29 | |
| *** yamamoto has joined #openstack-lbaas | 00:34 | |
| *** yamamoto has quit IRC | 00:39 | |
| *** hongbin has joined #openstack-lbaas | 01:08 | |
| *** kiennt26 has joined #openstack-lbaas | 01:09 | |
| *** abaindur has quit IRC | 01:23 | |
| *** abaindur has joined #openstack-lbaas | 01:28 | |
| johnsom | cgoncalves I'm pushing back on your -1 for my pkg-map patch | 01:34 |
| *** abaindur has quit IRC | 01:34 | |
| johnsom | https://github.com/openstack/diskimage-builder/tree/master/diskimage_builder/elements/pkg-map+ | 01:35 |
| johnsom | Opps, https://github.com/openstack/diskimage-builder/tree/master/diskimage_builder/elements/pkg-map | 01:35 |
| *** AlexStaf has quit IRC | 01:37 | |
| openstackgerrit | Michael Johnson proposed openstack/octavia master: Cleanup Octavia create VIP ports on LB delete https://review.openstack.org/581168 | 01:38 |
| *** kevko has quit IRC | 02:09 | |
| *** JuCross has joined #openstack-lbaas | 02:14 | |
| *** crazik_ has joined #openstack-lbaas | 02:16 | |
| *** crazik_ has joined #openstack-lbaas | 02:16 | |
| *** Eugene__ has joined #openstack-lbaas | 02:16 | |
| *** sbalukoff_ has joined #openstack-lbaas | 02:19 | |
| *** JudeC has quit IRC | 02:19 | |
| *** Eugene_ has quit IRC | 02:19 | |
| *** ltomasbo has quit IRC | 02:19 | |
| *** ltomasbo` has joined #openstack-lbaas | 02:19 | |
| *** crazik has quit IRC | 02:19 | |
| *** kiennt26 has quit IRC | 02:19 | |
| *** sbalukoff has quit IRC | 02:19 | |
| *** ivve has quit IRC | 02:19 | |
| *** ltomasbo` has quit IRC | 02:19 | |
| *** ltomasbo` has joined #openstack-lbaas | 02:19 | |
| *** ivve has joined #openstack-lbaas | 02:28 | |
| *** ramishra has joined #openstack-lbaas | 02:33 | |
| *** yboaron has joined #openstack-lbaas | 02:41 | |
| *** yboaron_ has quit IRC | 02:43 | |
| *** sapd_ has joined #openstack-lbaas | 03:04 | |
| *** sapd__ has quit IRC | 03:04 | |
| *** hongbin has quit IRC | 03:39 | |
| *** yamamoto has joined #openstack-lbaas | 03:52 | |
| *** ianychoi_ has quit IRC | 03:54 | |
| openstackgerrit | ZhaoBo proposed openstack/octavia master: UDP jinja template https://review.openstack.org/525420 | 03:58 |
| openstackgerrit | ZhaoBo proposed openstack/octavia master: WIP:UDP for [2] https://review.openstack.org/529651 | 03:58 |
| openstackgerrit | ZhaoBo proposed openstack/octavia master: UDP for [3][5][6] https://review.openstack.org/539391 | 03:58 |
| *** ianychoi has joined #openstack-lbaas | 04:08 | |
| sapd_ | johnsom: I would like to contribute a monitoring feature for octavia :D | 04:08 |
| johnsom | sapd_: ok, what kind of monitoring? Like Monasca? | 04:11 |
| *** hongbin has joined #openstack-lbaas | 04:12 | |
| sapd_ | johnsom: I would like collect more measurements in a loadbalancer. | 04:12 |
| *** kiennt26 has joined #openstack-lbaas | 04:12 | |
| johnsom | Ok. Think about other drivers and the db performance. | 04:13 |
| johnsom | Creating a spec or RFE story? | 04:14 |
| *** abaindur has joined #openstack-lbaas | 04:14 | |
| *** hongbin has quit IRC | 04:20 | |
| *** jappleii__ has quit IRC | 04:25 | |
| *** threestrands has joined #openstack-lbaas | 04:25 | |
| *** yamamoto_ has joined #openstack-lbaas | 04:42 | |
| *** kiennt26 has quit IRC | 04:45 | |
| *** yamamoto has quit IRC | 04:45 | |
| *** yamamoto_ has quit IRC | 05:16 | |
| *** links has joined #openstack-lbaas | 05:22 | |
| *** yamamoto has joined #openstack-lbaas | 05:35 | |
| sapd_ | currently, I send measurements about loadbalancer (haproxy) to influxdb to store and use grafana to visualize a graph. | 05:46 |
| openstackgerrit | ZhaoBo proposed openstack/octavia master: WIP:UDP for [2] https://review.openstack.org/529651 | 05:49 |
| openstackgerrit | ZhaoBo proposed openstack/octavia master: UDP for [3][5][6] https://review.openstack.org/539391 | 05:49 |
| *** kobis has joined #openstack-lbaas | 06:00 | |
| *** kobis has quit IRC | 06:03 | |
| *** ltomasbo` is now known as ltomasbo | 06:08 | |
| *** abaindur has quit IRC | 06:10 | |
| *** yboaron has quit IRC | 06:16 | |
| *** ispp has joined #openstack-lbaas | 06:29 | |
| *** threestrands has quit IRC | 06:30 | |
| *** crazik_ is now known as crazik | 06:34 | |
| crazik | hello | 06:34 |
| crazik | sapd_: could you describe more that influxdb solution? | 06:35 |
| crazik | btw: is there any way to prevent LB creation on EXTERNAL network? | 06:36 |
| crazik | I don't want users to use my external IPs (3 per LB), when they can use FIP instead | 06:37 |
| *** links has quit IRC | 06:38 | |
| *** kobis has joined #openstack-lbaas | 06:38 | |
| *** velizarx has joined #openstack-lbaas | 06:52 | |
| *** links has joined #openstack-lbaas | 06:53 | |
| *** kobis has quit IRC | 06:56 | |
| *** nmanos has joined #openstack-lbaas | 07:03 | |
| *** nmanos has quit IRC | 07:04 | |
| *** nmanos has joined #openstack-lbaas | 07:04 | |
| *** nmanos has left #openstack-lbaas | 07:04 | |
| *** velizarx has quit IRC | 07:08 | |
| *** tesseract has joined #openstack-lbaas | 07:13 | |
| *** yboaron has joined #openstack-lbaas | 07:17 | |
| *** yboaron has quit IRC | 07:18 | |
| *** yboaron has joined #openstack-lbaas | 07:18 | |
| *** peereb has joined #openstack-lbaas | 07:19 | |
| *** rraja has joined #openstack-lbaas | 07:27 | |
| *** velizarx has joined #openstack-lbaas | 07:29 | |
| bzhao__ | crazik: Your provider network maybe dhcp_enabled, users can create instances successful.. So users feel create on ext net seems the same as create on private net.. If the external net just used for NAT, why not disable the dhcp? Just my opinion. | 07:31 |
| crazik | bzhao__: not exactly. Users can use FIPs, can create instances in EXTERNAL directly | 07:33 |
| crazik | but I don't want to allow only LB in that net | 07:33 |
| crazik | because of used IPs | 07:34 |
| crazik | for vrrp | 07:34 |
| *** ispp has quit IRC | 07:43 | |
| *** links has quit IRC | 07:44 | |
| openstackgerrit | ZhaoBo proposed openstack/octavia master: WIP:UDP for [2] https://review.openstack.org/529651 | 07:45 |
| openstackgerrit | ZhaoBo proposed openstack/octavia master: UDP for [3][5][6] https://review.openstack.org/539391 | 07:45 |
| *** rcernin has quit IRC | 07:47 | |
| johnsom | crazik: currently there is no way yo limit the network the user can select: | 07:49 |
| *** crazik has quit IRC | 07:53 | |
| bzhao__ | crazik: Hmm, I check neutron code, it seems that the target user also can see the subnet_id now which owned by ext net and shared by admin. So policy engine on neutron side, may be also not helpful. | 07:54 |
| *** KeithMnemonic has quit IRC | 07:55 | |
| *** links has joined #openstack-lbaas | 07:57 | |
| *** KeithMnemonic has joined #openstack-lbaas | 07:57 | |
| *** ktibi has joined #openstack-lbaas | 07:58 | |
| sapd_ | crazik Install agent in amphora. | 08:01 |
| *** nmanos has joined #openstack-lbaas | 08:14 | |
| *** nmanos has left #openstack-lbaas | 08:14 | |
| *** kobis has joined #openstack-lbaas | 08:30 | |
| *** nmanos has joined #openstack-lbaas | 08:32 | |
| *** nmanos has left #openstack-lbaas | 08:33 | |
| *** kevko has joined #openstack-lbaas | 08:38 | |
| openstackgerrit | Tuan Do Anh proposed openstack/neutron-lbaas master: Add release notes link in README https://review.openstack.org/581269 | 08:55 |
| openstackgerrit | Tuan Do Anh proposed openstack/octavia master: Add release notes link in README https://review.openstack.org/581294 | 09:43 |
| *** rcernin has joined #openstack-lbaas | 09:44 | |
| *** rcernin has quit IRC | 09:47 | |
| *** sapd_ has quit IRC | 10:11 | |
| *** sapd has joined #openstack-lbaas | 10:11 | |
| *** yamamoto has quit IRC | 10:15 | |
| *** yamamoto has joined #openstack-lbaas | 10:19 | |
| *** kobis has quit IRC | 10:22 | |
| *** yamamoto has quit IRC | 10:45 | |
| *** kobis has joined #openstack-lbaas | 11:02 | |
| *** kobis has quit IRC | 11:09 | |
| *** kobis has joined #openstack-lbaas | 11:16 | |
| *** crazik has joined #openstack-lbaas | 11:19 | |
| *** crazik has joined #openstack-lbaas | 11:19 | |
| crazik | blah. | 11:19 |
| *** yamamoto has joined #openstack-lbaas | 11:33 | |
| *** atoth has joined #openstack-lbaas | 11:34 | |
| *** longkb has quit IRC | 11:38 | |
| *** ispp has joined #openstack-lbaas | 11:48 | |
| ispp | Feedback is welcome here octavia-helm https://review.openstack.org/#/c/578051/ | 11:52 |
| *** velizarx has quit IRC | 12:20 | |
| *** yamamoto has quit IRC | 12:26 | |
| *** yamamoto has joined #openstack-lbaas | 12:26 | |
| *** velizarx has joined #openstack-lbaas | 12:32 | |
| openstackgerrit | Merged openstack/octavia master: [doc] Add the missing markup for the hyperlink title https://review.openstack.org/581046 | 12:38 |
| *** amuller has joined #openstack-lbaas | 13:08 | |
| *** ispp has quit IRC | 13:11 | |
| *** ispp has joined #openstack-lbaas | 13:23 | |
| *** fnaval has joined #openstack-lbaas | 13:26 | |
| *** yboaron_ has joined #openstack-lbaas | 13:34 | |
| *** yboaron has quit IRC | 13:36 | |
| openstackgerrit | Carlos Goncalves proposed openstack/octavia master: Aallow setting load balancer topology in Devstack https://review.openstack.org/581366 | 13:49 |
| jiteka | johnsom: yes it says "BOOTING" since a while now but the VM already spawned and is active | 14:01 |
| jiteka | https://pastebin.com/942qHx6V | 14:01 |
| jiteka | Was curious if there is anyway to rekick the process or if it's impossible to recover from an LB that failed to finish his provisionning | 14:01 |
| openstackgerrit | Carlos Goncalves proposed openstack/octavia master: Allow setting load balancer topology in Devstack https://review.openstack.org/581366 | 14:06 |
| openstackgerrit | Merged openstack/octavia master: Follow the new PTI for document build https://review.openstack.org/580983 | 14:08 |
| *** ispp has quit IRC | 14:11 | |
| *** ispp has joined #openstack-lbaas | 14:17 | |
| *** velizarx has quit IRC | 14:30 | |
| *** yboaron_ has quit IRC | 14:36 | |
| *** yboaron_ has joined #openstack-lbaas | 14:38 | |
| *** peereb has quit IRC | 14:45 | |
| *** ispp has quit IRC | 14:45 | |
| *** ispp has joined #openstack-lbaas | 14:54 | |
| xgerman_ | jiteka: no, we don’t have a way to kick… any pending_ state will block operations on the LB | 14:55 |
| xgerman_ | since it means some other process is handling it and we don’t want data corruption | 14:55 |
| *** yboaron_ has quit IRC | 14:56 | |
| *** yboaron has joined #openstack-lbaas | 14:57 | |
| *** kobis has quit IRC | 15:01 | |
| *** yboaron has quit IRC | 15:08 | |
| johnsom | jiteka It should finish and go to ERROR which will allow you to fix the load balancer. | 15:18 |
| *** links has quit IRC | 15:26 | |
| *** yamamoto has quit IRC | 15:29 | |
| *** yamamoto has joined #openstack-lbaas | 15:31 | |
| *** yamamoto has quit IRC | 15:34 | |
| sapd | johnsom: have you tried to create more l7 policies yet? | 15:54 |
| *** rraja has quit IRC | 15:57 | |
| johnsom | sapd No, sorry, I have not had time. Did you create the story with your configuration? | 15:59 |
| *** ktibi has quit IRC | 16:01 | |
| jiteka | johnsom: it's stuck since few days so I guess it won't move anymore, it's in devstack so I'm not too worried about it. I was just curious to know if any Ops command could handle that kind of scenario | 16:03 |
| jiteka | Thanks for the answer xgerman_ | 16:03 |
| johnsom | jiteka Did you kill -9 the controller? That would be the only way it would get stuck like that and not ERROR our | 16:04 |
| johnsom | out | 16:04 |
| *** kobis has joined #openstack-lbaas | 16:08 | |
| *** ispp has quit IRC | 16:13 | |
| *** yamamoto has joined #openstack-lbaas | 16:34 | |
| *** yamamoto has quit IRC | 16:40 | |
| *** ramishra has quit IRC | 16:49 | |
| sapd | johnsom: I created a story. | 16:50 |
| sapd | after database slow, It makes health manager can't process heart beat :'( | 16:52 |
| sapd | 2018-07-10 23:52:05.774 80 WARNING octavia.controller.healthmanager.health_drivers.update_db [-] Amphora 2c0c83b2-9d89-4a86-b696-2fede4c294d3 health message was processed too slowly: 19.7807719707s! The system may be overloaded or otherwise malfunctioning. This heartbeat has been ignored and no update was made to the amphora health entry. THIS IS NOT GOOD. | 16:52 |
| johnsom | sapd Yeah, that means you database is failing! | 16:53 |
| johnsom | That can cause failovers that are not necessary | 16:53 |
| sapd | how failed? | 16:53 |
| sapd | yep. Loadbalancer is being failover. | 16:54 |
| johnsom | Well, it should take 0.004 seconds to process that heartbeat, but your database is taking 19.7807719707s | 16:54 |
| johnsom | So the heartbeats aren't getting processed until after the 10 second heartbest interval. | 16:56 |
| johnsom | sapd What version of Octavia are you using? | 16:56 |
| sapd | I'm running octavia from master branch | 16:56 |
| johnsom | Yeah, ok, so your database is very sick | 16:56 |
| sapd | I have added a column to load_balancer table. | 16:57 |
| johnsom | You will see the HM process CPU load go up too as it tries to deal with the slow DB | 16:57 |
| *** atoth has quit IRC | 16:57 | |
| sapd | yep. | 16:58 |
| sapd | I saw | 16:58 |
| johnsom | Memory will grow too as it gets a backlog of heartbeats to process | 16:58 |
| johnsom | sapd You are not using neutron-lbaas right? | 16:59 |
| sapd | yep. I'm | 17:00 |
| johnsom | sapd If you are not using neutron-lbaas, make sure [health_manager] event_streamer_driver = noop_event_streamer and sync_provisioning_status = False are set in your octavia.conf. Those are the defaults, but someone may have set them | 17:00 |
| *** atoth has joined #openstack-lbaas | 17:00 | |
| johnsom | If those are enabled, we have seen mis-configured rabbit queues also cause this | 17:00 |
| xgerman_ | +1 | 17:01 |
| sapd | johnsom: I have not set these config. | 17:01 |
| *** KeithMnemonic1 has joined #openstack-lbaas | 17:02 | |
| sapd | [health_manager] | 17:02 |
| sapd | bind_ip = 0.0.0.0 | 17:02 |
| sapd | bind_port = 5555 | 17:02 |
| sapd | heartbeat_key = insecure | 17:02 |
| sapd | controller_ip_port_list = 10.1.0.9:5555 | 17:02 |
| *** KeithMnemonic has quit IRC | 17:03 | |
| *** kobis has quit IRC | 17:05 | |
| *** tesseract has quit IRC | 17:07 | |
| *** KeithMnemonic has joined #openstack-lbaas | 17:13 | |
| openstackgerrit | Merged openstack/octavia master: Ignore a port not found when deleting an LB https://review.openstack.org/564848 | 17:14 |
| *** KeithMnemonic1 has quit IRC | 17:15 | |
| jiteka | johnsom: restart controller worker and rebooting the VM got the amphora object removed from the list but the associated lb was still PENDING_CREATE. After manually forcing the provisionning_status to ERROR in the octavia DB the amphora VM got deleted (I guess the housekeeper process took care of it) | 17:28 |
| *** nmanos has joined #openstack-lbaas | 17:34 | |
| *** yamamoto has joined #openstack-lbaas | 17:37 | |
| *** atoth_ has joined #openstack-lbaas | 17:41 | |
| *** yamamoto has quit IRC | 17:41 | |
| *** atoth has quit IRC | 17:44 | |
| *** atoth_ has quit IRC | 17:45 | |
| *** atoth has joined #openstack-lbaas | 17:46 | |
| *** atoth has quit IRC | 17:48 | |
| *** nmanos has quit IRC | 18:02 | |
| *** yamamoto has joined #openstack-lbaas | 18:39 | |
| *** yamamoto has quit IRC | 18:44 | |
| *** amuller has quit IRC | 19:06 | |
| *** atoth has joined #openstack-lbaas | 19:06 | |
| *** irenab has quit IRC | 19:18 | |
| *** irenab has joined #openstack-lbaas | 19:18 | |
| *** irenab has quit IRC | 19:28 | |
| *** irenab has joined #openstack-lbaas | 19:29 | |
| *** yamamoto has joined #openstack-lbaas | 19:32 | |
| *** yamamoto has quit IRC | 19:37 | |
| *** yboaron has joined #openstack-lbaas | 20:00 | |
| *** atoth has quit IRC | 20:27 | |
| *** atoth has joined #openstack-lbaas | 20:27 | |
| *** yamamoto has joined #openstack-lbaas | 20:33 | |
| *** yamamoto has quit IRC | 20:39 | |
| openstackgerrit | German Eichberger proposed openstack/octavia master: [WIP] Switch amphora agent to use privsep https://review.openstack.org/549295 | 20:44 |
| *** kobis has joined #openstack-lbaas | 20:59 | |
| *** kobis has quit IRC | 21:03 | |
| *** irenab has quit IRC | 21:05 | |
| *** irenab has joined #openstack-lbaas | 21:05 | |
| *** fnaval has quit IRC | 21:24 | |
| *** blah123 has joined #openstack-lbaas | 21:28 | |
| *** fnaval has joined #openstack-lbaas | 21:39 | |
| *** fnaval has quit IRC | 22:01 | |
| *** yboaron has quit IRC | 22:13 | |
| *** blah123 has quit IRC | 22:17 | |
| *** rcernin has joined #openstack-lbaas | 22:22 | |
| *** KeithMnemonic1 has joined #openstack-lbaas | 22:32 | |
| openstackgerrit | German Eichberger proposed openstack/octavia master: [WIP] Switch amphora agent to use privsep https://review.openstack.org/549295 | 22:32 |
| xgerman_ | really need to improve my test setup so I can build amps with a local git tree | 22:33 |
| *** KeithMnemonic has quit IRC | 22:34 | |
| *** yamamoto has joined #openstack-lbaas | 22:36 | |
| *** fnaval has joined #openstack-lbaas | 22:37 | |
| *** fnaval has quit IRC | 22:37 | |
| *** yamamoto has quit IRC | 22:42 | |
| openstackgerrit | Carlos Goncalves proposed openstack/octavia master: Bring up lo when plugging interfaces https://review.openstack.org/550861 | 22:48 |
| johnsom | cgoncalves It's a security feature, not a bug. lol | 22:56 |
| cgoncalves | johnsom, the loopback interface? | 22:58 |
| johnsom | yeah | 22:58 |
| johnsom | Completely joking... | 22:58 |
| cgoncalves | ah :) | 23:00 |
| cgoncalves | from time to time I hit this: http://paste.openstack.org/show/725489/ | 23:03 |
| cgoncalves | I cleaned up ~/.cache/{dib,image-create} first | 23:03 |
| cgoncalves | I had built a CentOS image on a previous stack. I wonder if toggling distro somehow affects | 23:04 |
| johnsom | That would be odd and a bummer | 23:05 |
| johnsom | I have not hit this | 23:05 |
| cgoncalves | I'll re-stack from scratch overnight | 23:10 |
| cgoncalves | johnsom, can you confirm that on a clean arp table and active-standby LB topo, the first packets may indeed be served via the standby amp? | 23:11 |
| openstackgerrit | German Eichberger proposed openstack/octavia master: Add a config to surpress amphora logging https://review.openstack.org/566741 | 23:11 |
| cgoncalves | until the next garp (every 5 seconds by default) | 23:12 |
| johnsom | cgoncalves Yes, I think that is possible | 23:12 |
| johnsom | cgoncalves It has to do with the fact that one startup, both amps have the VIP ip.... It should be fixed, but it was low on the old priority list. | 23:13 |
| cgoncalves | ok, thanks. I wanted to make sure I was seeing things right | 23:13 |
| cgoncalves | johnsom, is there a point in time when only one amp has the VIP ip set? because I don't see that happening | 23:14 |
| cgoncalves | on the active I see <vip>/32 and <vip>/24, while on the standby only /24 | 23:15 |
| johnsom | Yes, if you fail it over, then back, the standby will drop the VIP IP like it should be | 23:15 |
| johnsom | By failover I mean forcing keepalived's hand, not API failover | 23:16 |
| cgoncalves | ah, still need to play with that | 23:16 |
| cgoncalves | right | 23:16 |
| johnsom | There are two IPs, the base IP for the interface, this is unique per amp, and then the VIP which "moves" | 23:17 |
| johnsom | You can do the VRRP failover game by stoping your HM, then going into the one you want down and stopping the amphora agent. This will trigger a VRRP switch. | 23:18 |
| cgoncalves | johnsom, how stopping the agent triggers a vrrp switch? | 23:20 |
| cgoncalves | I could see that happening when deactivating eth1 I guess, but the agent?! | 23:21 |
| johnsom | Oh, yeah, right, it's haproxy not the agent. Agent doesn't matter for VRRP failover | 23:26 |
| xgerman_ | stopping the agent will get that amp discarded and is causing a vrrp switch that way | 23:28 |
| johnsom | Right, the brute force way.... Though I advised him to stop o-hm, so.... | 23:28 |
| xgerman_ | yep… | 23:29 |
| cgoncalves | ;) | 23:29 |
| cgoncalves | cool, thanks! going to failover to standby (aka bed) now | 23:30 |
| johnsom | o/ | 23:30 |
| xgerman_ | o/ | 23:30 |
| *** yamamoto has joined #openstack-lbaas | 23:38 | |
| *** yamamoto has quit IRC | 23:44 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!