Monday, 2021-08-23

mnaserhas anyone done some relative scale testing on octavia?13:04
mnaseron wallaby, i'm seeing things start to be pretty unhappy at ~50 lb ops at the same time13:04
gthiemongemnaser: I've seen deployments with more that 2k LBs... main issue is the configuration of the health-manager service, it might require more workers to process incoming heartbeat messages from the amphorae13:07
mnasergthiemonge: i think in my case the issues that surfaces were concurrent operations13:07
mnaserin my case a large number of concurrent creates/updates/deletes13:08
mnaserresulting in a whole bunch of PENDING_{CREATE,DELETE} stufff13:08
gthiemongehm I'm not aware of issues there13:08
mnasergthiemonge: for example right now i've got 36 lbs in PENDING_* states13:13
gthiemongemnaser: do you know what they are doing? (normal operations - adding a pool, adding a member, or failovers?)13:18
mnasergthiemonge: in this case, it's an openstack deployment provisioning a whole bunch of new loadbalancers (i.e. k8s LoadBalancer services)13:19
gthiemongeyeah there are a lot of configuration changes when using k8s, if members are updated every 5sec, the LB will spend a lot of time in a PENDING_UPDATE state13:24
mnasergthiemonge: yeah.. and i think right now i'm seeing the octavia-worker just stalling out completetly and not responding, so doing some more.. checking13:45
johnsommnaser Please tell me you are running the Octavia API as a wsgi app and not using the simple server command.15:14
johnsomUnder wsgi we stress tested the API to around 7,000 concurrent API calls per second with no issue.15:14
johnsomOtherwise, almost all of our provisioning time is waiting on nova or neutron.15:16
johnsomAt 7k we ran out of cores for the test tool15:20
-opendevstatus- NOTICE: The Gerrit service on has been restarted for a patch version upgrade, resulting in a brief outage21:42
*** yoctozepto4 is now known as yoctozepto23:37

Generated by 2.17.2 by Marius Gedminas - find it at!