| *** hongbin has quit IRC | 00:14 | |
| *** hongbin has joined #openstack-lbaas | 00:16 | |
| *** yamamoto has joined #openstack-lbaas | 00:18 | |
| *** ccamposr has joined #openstack-lbaas | 00:21 | |
| *** ccamposr__ has quit IRC | 00:24 | |
| *** threestrands has joined #openstack-lbaas | 01:19 | |
| openstackgerrit | lidong proposed openstack/octavia-lib master: Add releasenotes link to README https://review.opendev.org/734730 | 01:30 |
|---|---|---|
| *** sapd1 has joined #openstack-lbaas | 01:38 | |
| *** rcernin has quit IRC | 02:53 | |
| *** rcernin_ has joined #openstack-lbaas | 02:53 | |
| *** TMM has quit IRC | 02:54 | |
| *** TMM has joined #openstack-lbaas | 02:54 | |
| *** vishalmanchanda has joined #openstack-lbaas | 02:59 | |
| *** rcernin_ has quit IRC | 03:07 | |
| *** rcernin_ has joined #openstack-lbaas | 03:19 | |
| *** rcernin_ has quit IRC | 03:24 | |
| *** rcernin has joined #openstack-lbaas | 03:25 | |
| *** armax has quit IRC | 03:38 | |
| *** psachin has joined #openstack-lbaas | 03:40 | |
| *** hongbin has quit IRC | 04:39 | |
| *** gcheresh has joined #openstack-lbaas | 05:25 | |
| *** sapd1 has quit IRC | 05:55 | |
| *** ccamposr__ has joined #openstack-lbaas | 06:14 | |
| *** ccamposr has quit IRC | 06:17 | |
| *** vesper has quit IRC | 06:18 | |
| *** vesper11 has joined #openstack-lbaas | 06:18 | |
| *** sapd1 has joined #openstack-lbaas | 06:23 | |
| *** wuchunyang has joined #openstack-lbaas | 06:24 | |
| *** wuchunyang has quit IRC | 06:32 | |
| *** maciejjozefczyk has joined #openstack-lbaas | 06:38 | |
| *** ccamposr has joined #openstack-lbaas | 06:43 | |
| *** rcernin has quit IRC | 06:44 | |
| *** rcernin_ has joined #openstack-lbaas | 06:44 | |
| *** ccamposr__ has quit IRC | 06:46 | |
| *** rcernin_ has quit IRC | 06:50 | |
| *** rcernin_ has joined #openstack-lbaas | 06:52 | |
| *** wuchunyang has joined #openstack-lbaas | 06:53 | |
| *** rcernin_ has quit IRC | 06:59 | |
| *** wuchunyang has quit IRC | 07:08 | |
| *** psachin has quit IRC | 07:12 | |
| *** sapd1 has quit IRC | 07:12 | |
| *** rpittau|afk is now known as rpittau | 07:21 | |
| *** rcernin_ has joined #openstack-lbaas | 07:23 | |
| *** rcernin_ has quit IRC | 07:28 | |
| *** born2bake has joined #openstack-lbaas | 07:42 | |
| *** rcernin_ has joined #openstack-lbaas | 07:59 | |
| *** rcernin_ has quit IRC | 08:03 | |
| *** AlexStaf has quit IRC | 08:04 | |
| *** wuchunyang has joined #openstack-lbaas | 08:16 | |
| *** AlexStaf has joined #openstack-lbaas | 08:20 | |
| *** sapd1 has joined #openstack-lbaas | 08:24 | |
| *** wuchunyang has quit IRC | 08:46 | |
| *** spatel has joined #openstack-lbaas | 09:13 | |
| *** spatel has quit IRC | 09:18 | |
| *** wuchunyang has joined #openstack-lbaas | 09:51 | |
| *** wuchunyang has quit IRC | 09:51 | |
| *** rpittau is now known as rpittau|bbl | 10:19 | |
| *** wuchunyang has joined #openstack-lbaas | 10:30 | |
| *** AlexStaf has quit IRC | 10:31 | |
| *** wuchunyang has quit IRC | 10:36 | |
| *** AlexStaf has joined #openstack-lbaas | 10:53 | |
| *** tkajinam has quit IRC | 10:53 | |
| *** threestrands has quit IRC | 11:16 | |
| *** zigo has quit IRC | 11:41 | |
| *** rcernin_ has joined #openstack-lbaas | 11:48 | |
| *** rcernin_ has quit IRC | 11:53 | |
| *** yamamoto has quit IRC | 11:55 | |
| *** zigo has joined #openstack-lbaas | 11:56 | |
| openstackgerrit | Merged openstack/octavia-tempest-plugin master: Pin DIB to Python 2 compatible version on py2 jobs https://review.opendev.org/733778 | 11:56 |
| *** rpittau|bbl is now known as rpittau | 12:05 | |
| openstackgerrit | Carlos Goncalves proposed openstack/octavia-tempest-plugin master: DNM: CentOS 8 controller and amphora job https://review.opendev.org/698450 | 12:05 |
| *** njohnston is now known as njohnston|pto | 12:05 | |
| *** yamamoto has joined #openstack-lbaas | 12:30 | |
| *** yamamoto has quit IRC | 12:35 | |
| *** TrevorV has joined #openstack-lbaas | 14:01 | |
| *** armax has joined #openstack-lbaas | 14:16 | |
| *** sapd1 has quit IRC | 14:20 | |
| *** yamamoto has joined #openstack-lbaas | 14:24 | |
| *** rcernin_ has joined #openstack-lbaas | 14:28 | |
| *** yamamoto has quit IRC | 14:29 | |
| *** rcernin_ has quit IRC | 14:33 | |
| *** also_stingrayza has joined #openstack-lbaas | 14:44 | |
| *** stingrayza has quit IRC | 14:48 | |
| *** gcheresh has quit IRC | 15:13 | |
| *** namrata has joined #openstack-lbaas | 15:13 | |
| namrata | Hi Folks, I am looking for a way to restrict access to service Load-balanced by Octavia. The approach I tried is to restrict access to the Octavia VMs via seurity groups, but that does not seem to be supported. Can anybody help me with the way to use port security with octavia. Thanks! | 15:31 |
| johnsom | namrata You want to use the "allowed_cidrs" setting on your listener: https://docs.openstack.org/api-ref/load-balancer/v2/index.html?expanded=create-listener-detail#create-listener | 15:32 |
| namrata | @johnsom Thanks for the reply. So setting allowed_cidrs on listener I can apply security group rule on amphora ports | 15:37 |
| johnsom | Correct, that will switch the default to denied and only cidrs listed there will be able to access the load balancer port(s) | 15:38 |
| namrata | Great!. Thanks again | 15:39 |
| *** armax has quit IRC | 15:50 | |
| *** armax has joined #openstack-lbaas | 15:52 | |
| xgerman | anyone thought about replacing diskimagebuiler with packer? https://www.packer.io/docs/builders/openstack/ | 15:56 |
| johnsom | Can't say that I have heard of anyone mentioning it. There is a debian tool that they use for images. | 16:00 |
| johnsom | #startmeeting Octavia | 16:00 |
| openstack | Meeting started Wed Jun 10 16:00:26 2020 UTC and is due to finish in 60 minutes. The chair is johnsom. Information about MeetBot at http://wiki.debian.org/MeetBot. | 16:00 |
| openstack | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 16:00 |
| *** openstack changes topic to " (Meeting topic: Octavia)" | 16:00 | |
| openstack | The meeting name has been set to 'octavia' | 16:00 |
| johnsom | Hi folks | 16:00 |
| gthiemonge | hi | 16:00 |
| cgoncalves | hi | 16:01 |
| johnsom | #topic Announcements | 16:01 |
| *** openstack changes topic to "Announcements (Meeting topic: Octavia)" | 16:01 | |
| johnsom | Seems like a small group this week | 16:02 |
| johnsom | FYI, we kept notes from the PTG sessions on the etherpad: | 16:02 |
| johnsom | #link https://etherpad.opendev.org/p/octavia-virtual-V-ptg | 16:02 |
| johnsom | In case anyone missed the fun and excitement! | 16:02 |
| johnsom | Also a quick note, we have seen a few reports of Train deployments that octavia-dashboard does not work. Turns out this was an OpenStack Ansible bug where it was installing the master branch of octavia-dashboard. Fixed here: | 16:03 |
| johnsom | #link https://review.opendev.org/734881 | 16:03 |
| johnsom | Any other announcements this week? | 16:04 |
| johnsom | FYI, there is some mailing list discussions about releases: | 16:05 |
| johnsom | #link http://lists.openstack.org/pipermail/openstack-discuss/2020-June/015342.html | 16:05 |
| johnsom | #topic Brief progress reports / bugs needing review | 16:06 |
| *** openstack changes topic to "Brief progress reports / bugs needing review (Meeting topic: Octavia)" | 16:06 | |
| johnsom | Well, there was this PTG thing last week. grin | 16:06 |
| johnsom | Other than that I have been doing reviews, working to get the gates functional again so that we can land some stuff and do stable branch releases. | 16:06 |
| johnsom | Also working on the failover patch and backport to the v2 driver. | 16:07 |
| *** rpittau is now known as rpittau|afk | 16:07 | |
| johnsom | That about sums up my week. Anyone else? | 16:07 |
| *** sapd1 has joined #openstack-lbaas | 16:08 | |
| johnsom | #topic HAProxy memory usage and slow reload process cleanup | 16:09 |
| *** openstack changes topic to "HAProxy memory usage and slow reload process cleanup (Meeting topic: Octavia)" | 16:09 | |
| gthiemonge | that topic sums up my week ^ | 16:10 |
| johnsom | gthiemonge Would you introduce this issue? | 16:10 |
| gthiemonge | yes | 16:10 |
| gthiemonge | so I've found an issue when using active standby and session persistence in centos | 16:11 |
| gthiemonge | when the loadbalancer is updated (adding members, etc..), haproxy is reloaded | 16:11 |
| gthiemonge | when it is reloading, it creates a new thread, does a lot of allocation, then destroys the previous thread (the worker) | 16:12 |
| gthiemonge | in the case of active-standby and session-persistence, it takes 2 min to destroy the previous thread | 16:12 |
| gthiemonge | (instead of 1 or 2 seconds) | 16:13 |
| gthiemonge | so it means that we have 2 haproxy instances that both consume ~150MB at the same time | 16:13 |
| gthiemonge | it should not be a big deal... unless we update the config during this period -> it creates a new worker that consumes 150MB | 16:14 |
| gthiemonge | so after few config updates, we have a memory issue and haproxy crashes | 16:14 |
| gthiemonge | I have more detail in a downstream bug: #link https://bugzilla.redhat.com/show_bug.cgi?id=1845406#c2 | 16:15 |
| openstack | bugzilla.redhat.com bug 1845406 in openstack-octavia "octavia_tempest_plugin.tests.api.v2.test_pool.PoolAPITest.test_pool_delete fails in ACTIVE_STANDBY jobs" [High,Assigned] - Assigned to gthiemon | 16:15 |
| johnsom | At which point systemd restarts haproxy and things resume until the next update chain, correct? | 16:15 |
| gthiemonge | yes, correct | 16:16 |
| gthiemonge | I have a paste with logs from ubuntu: http://paste.openstack.org/show/794586/ | 16:16 |
| johnsom | Which is good, but still not ideal as there is downtime during that systemd restart window. | 16:16 |
| gthiemonge | and that one: http://paste.openstack.org/show/794590/ that shows the restart of the service | 16:16 |
| gthiemonge | johnsom: I think we can tune the systemd timeout for reload/restart | 16:17 |
| gthiemonge | currently, it restarts after 1min30 | 16:17 |
| johnsom | Yeah, that could be problematic as well though. Sometimes it's good to give a little breathing room between restart attempts. | 16:19 |
| johnsom | I think it would be best to not run into the problem in the first place. | 16:19 |
| gthiemonge | sure | 16:19 |
| johnsom | So the obvious option is bump up the RAM allocated to the amphora. It would only use more on the hypervisor when it is needed. Though the optics on that may not be good. People see 1GB or 2GB and think that is all "reserved" RAM. | 16:20 |
| johnsom | We can drop the default max connections from "unlimited" to something more reasonable, thus saving RAM allocation. | 16:21 |
| johnsom | This would mean a change to the "default" behavior though. | 16:21 |
| johnsom | Overall, I think that is a good idea anyway but rough given how long it's been set like this. | 16:22 |
| johnsom | We could add a swap partition. lol | 16:22 |
| gthiemonge | can we change the default max connections to a lower value? and add an config option for people who want to override it (to 1M)? | 16:23 |
| johnsom | We could stop doing hitless reloads and just stop/start for configuration changes. (no, don't do this) | 16:23 |
| johnsom | We could move to HAProxy 2.2 and use the new configuration API that doesn't need to reload..... (though it's not released yet) | 16:24 |
| johnsom | I really lean towards dropping the default maxconn to something more reasonable, like 30,000 or so. | 16:26 |
| gthiemonge | +1 | 16:27 |
| johnsom | I wonder what the RAM usage delta would be. | 16:27 |
| gthiemonge | johnsom: I think RAM usage is linear with the maxconn value, I'll check that | 16:28 |
| johnsom | Looks like 139464 -> 6664 RSS | 16:30 |
| johnsom | At least on my Ubuntu amp in devstack | 16:30 |
| gthiemonge | looks good | 16:31 |
| johnsom | So, yeah, saves a lot. Plus that RAM was basically wasted since a single CPU isn't going to handle 1,000,000 concurrent connections. | 16:32 |
| johnsom | Could use a bit more and go for 50,000 | 16:32 |
| johnsom | Again, this is all tunable via the listener settings by end users. | 16:33 |
| johnsom | Anyone else have input? | 16:33 |
| cgoncalves | +1 | 16:33 |
| gthiemonge | 50001? | 16:34 |
| johnsom | lol | 16:34 |
| cgoncalves | -1 +W | 16:34 |
| johnsom | And we are back to 1,000,000! grin | 16:34 |
| johnsom | So, the next part of this question is how to implement it. Currently we have "-1" as "unlimited" which translates to 1,000,000 in the configuration file because HAProxy doesn't really have an "unlimited" setting. | 16:35 |
| johnsom | I would like to expose to users that it is set for 30,000 instead of pretending with "-1", but we should keep -1 as an option for other drivers. | 16:36 |
| gthiemonge | good question | 16:37 |
| johnsom | How do we feel about setting it to the new configuration setting, defaulting to 50,000, if they are using the amphora driver and select "-1"? | 16:38 |
| johnsom | At least that way it would be truthful and give the user more information and control. | 16:38 |
| johnsom | I just hate "magically" changing settings on users. | 16:38 |
| gthiemonge | that sounds good, and people can change that value to get back to the previous behavior | 16:39 |
| johnsom | Yeah. I can see with HAProxy 2.x they may want a higher value when using multi-CPU amphora. | 16:40 |
| johnsom | Any other comments/thoughts on this? | 16:41 |
| johnsom | We could update the API reference to say "The maximum number of connections permitted for this listener. Default value is -1 which represents infinite connections or a default value set in the configuration of a driver." Something like that I guess. | 16:42 |
| *** gcheresh has joined #openstack-lbaas | 16:43 | |
| johnsom | Well, this sounds like the best path forward. gthiemonge are you going to propose a patch? | 16:43 |
| gthiemonge | johnsom: yes! | 16:44 |
| johnsom | Hopefully we can get wider feedback on the patch proposal. | 16:45 |
| johnsom | Cool, thanks for raising this! | 16:45 |
| johnsom | #topic Open Discussion | 16:45 |
| *** openstack changes topic to "Open Discussion (Meeting topic: Octavia)" | 16:46 | |
| gthiemonge | np | 16:46 |
| johnsom | Any other topics this week? | 16:46 |
| johnsom | upstream HAProxy has released a 2.1 version with the "-x" issue fixed. | 16:46 |
| johnsom | I'm not sure when the 1.8 version will land, but it's planned. | 16:47 |
| johnsom | #link https://github.com/haproxy/haproxy/issues/644 | 16:48 |
| gthiemonge | you can use https://review.opendev.org/#/c/698086/ to test it ;-) | 16:48 |
| johnsom | Lol, I should have! I just compiled one to test it. | 16:49 |
| johnsom | Ok, if there are no other topics this week we can call it for today. Thanks! | 16:50 |
| johnsom | #endmeeting | 16:51 |
| *** openstack changes topic to "Discussions for OpenStack Octavia | Priority bug review list: https://etherpad.openstack.org/p/octavia-priority-reviews" | 16:51 | |
| openstack | Meeting ended Wed Jun 10 16:51:08 2020 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 16:51 |
| openstack | Minutes: http://eavesdrop.openstack.org/meetings/octavia/2020/octavia.2020-06-10-16.00.html | 16:51 |
| openstack | Minutes (text): http://eavesdrop.openstack.org/meetings/octavia/2020/octavia.2020-06-10-16.00.txt | 16:51 |
| openstack | Log: http://eavesdrop.openstack.org/meetings/octavia/2020/octavia.2020-06-10-16.00.log.html | 16:51 |
| gthiemonge | johnsom: thanks | 16:51 |
| *** gcheresh has quit IRC | 16:54 | |
| *** namrata has quit IRC | 17:02 | |
| johnsom | cgoncalves Any thoughts on how this could fail with this error? https://review.opendev.org/733684 | 17:11 |
| johnsom | It passed in check, but failed with "diskimage-builder requires Python '>=3.5' but the running Python is 2.7.17" in gate.... | 17:12 |
| *** sapd1 has quit IRC | 17:12 | |
| *** ccamposr__ has joined #openstack-lbaas | 17:24 | |
| johnsom | Nevermind, I see what happened here. It was pinned in projects and not jobs, so the check runs are pinned and gate are not. Sigh | 17:24 |
| *** TrevorV has quit IRC | 17:25 | |
| *** ccamposr has quit IRC | 17:27 | |
| *** TrevorV has joined #openstack-lbaas | 17:29 | |
| openstackgerrit | Michael Johnson proposed openstack/octavia stable/stein: Do not install diskimage-builder from Git https://review.opendev.org/733684 | 17:33 |
| johnsom | ^^^ That will need another workflow | 17:34 |
| *** vishalmanchanda has quit IRC | 18:08 | |
| *** rcernin_ has joined #openstack-lbaas | 18:30 | |
| *** rcernin_ has quit IRC | 18:34 | |
| cgoncalves | thanks for fixing it | 18:43 |
| *** AlexStaf has quit IRC | 18:50 | |
| *** hongbin has joined #openstack-lbaas | 19:18 | |
| *** maciejjozefczyk has quit IRC | 19:46 | |
| *** hongbin has quit IRC | 19:55 | |
| *** spatel has joined #openstack-lbaas | 21:03 | |
| *** yamamoto has joined #openstack-lbaas | 21:07 | |
| *** yamamoto has quit IRC | 21:12 | |
| *** spatel has quit IRC | 21:32 | |
| johnsom | FYI we have a test failing occasionally: octavia_tempest_plugin.tests.scenario.v2.test_traffic_ops.TrafficOperationsScenarioTest.test_healthmonitor_udp_traffic | 21:49 |
| johnsom | If someone has time to take a look I have opened a story for it: https://storyboard.openstack.org/#!/story/2007792 | 21:49 |
| *** threestrands has joined #openstack-lbaas | 21:55 | |
| *** spatel has joined #openstack-lbaas | 21:56 | |
| *** rcernin_ has joined #openstack-lbaas | 21:59 | |
| *** rcernin_ has quit IRC | 22:04 | |
| *** rcernin has joined #openstack-lbaas | 22:04 | |
| *** rcernin is now known as rcernin|brekkie | 22:05 | |
| *** rcernin|brekkie has quit IRC | 22:19 | |
| *** TrevorV has quit IRC | 22:30 | |
| *** shtepanie has joined #openstack-lbaas | 22:32 | |
| *** spatel has quit IRC | 22:34 | |
| *** anushka_ai has joined #openstack-lbaas | 22:36 | |
| rm_work | Hey shtepanie and anushka_ai! | 22:37 |
| rm_work | Welcome to #openstack-lbaas | 22:37 |
| johnsom | Hi! Welcome to the Octavia team. | 22:37 |
| rm_work | Stephanie and Anushka are working with me this summer as interns. I hope to get them spun up and doing useful stuff in no time. :) | 22:38 |
| anushka_ai | Hello! Excited to be here!! :D | 22:38 |
| shtepanie | Hello! Nice to meet everyone. | 22:38 |
| johnsom | Nice. Feel free to ask questions here. We are a community and happy to help. | 22:39 |
| *** anushka_ai is now known as aannuusshhkkaa | 22:45 | |
| *** tkajinam has joined #openstack-lbaas | 22:45 | |
| *** shtepanie_ has joined #openstack-lbaas | 22:46 | |
| *** shtepanie has quit IRC | 22:47 | |
| *** shtepanie_ is now known as shtepanie | 22:47 | |
| *** aannuusshhkkaa has quit IRC | 22:47 | |
| *** TMM has quit IRC | 22:50 | |
| *** TMM has joined #openstack-lbaas | 22:51 | |
| *** aannuusshhkkaa has joined #openstack-lbaas | 22:52 | |
| *** hongbin has joined #openstack-lbaas | 22:56 | |
| aannuusshhkkaa | #opendev | 22:59 |
| *** hongbin has quit IRC | 23:00 | |
| *** born2bake has quit IRC | 23:02 | |
| *** yamamoto has joined #openstack-lbaas | 23:08 | |
| *** rcernin|brekkie has joined #openstack-lbaas | 23:09 | |
| *** rcernin|brekkie has quit IRC | 23:10 | |
| *** rcernin has joined #openstack-lbaas | 23:10 | |
| *** hongbin has joined #openstack-lbaas | 23:10 | |
| *** yamamoto has quit IRC | 23:25 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!