Wednesday, 2021-07-07

ignaziocassanoHellp, please any help on octavia wallaby ?08:55
gthiemongeignaziocassano: hi08:56
ignaziocassanoamphora agent logs Closing connection.08:56
ignaziocassanoBut if I try  opessl  connect it works from controllers08:56
ignaziocassanogthiemonge Hi08:57
ignaziocassanoI installed octavia using kolla wallaby08:57
ignaziocassanothe ambora image is wallaby version08:58
ignaziocassanothe amphira agent can connect to port 5555 on controllers08:58
ignaziocassanoI did not understand why it does not work08:58
ignaziocassanoAnyone could help me,please ?09:00
gthiemongeignaziocassano: do you have any other errors in the system logs of the amphora? (not the amphora-agent logs)09:06
gthiemongeit looks like the connection is established then it goes into timeout09:07
ignaziocassanogthiemonge oh yes, in worker.log I have : 2021-07-07 10:39:07.751 19 WARNING octavia.amphorae.drivers.haproxy.rest_api_driver [-] Could not connect to instance. Retrying.: requests.exceptions.ReadTimeout: HTTPSConnectionPool(host='', port=9443): Read timed out. (read timeout=10.0)09:07
ignaziocassanoit tries a lot of time then it ends with octavia.amphorae.driver_exceptions.exceptions.TimeOutException: contacting the amphora timed out09:08
ignaziocassanoIf I try to connect from the controller to amphora agent on port 9433 it responds (I tried with telnet and with openssl)09:10
ignaziocassanoagargano: are yoy from italy ?09:12
gthiemongeignaziocassano: and any errors with 'journalctl -le' in the amphorae? I've seen some similar errors in the past, because of memory allocation errors in haproxy 09:13
ignaziocassano<gthiemonge> I am going to verify09:14
ignaziocassanogthiemonge: no errors with journlctl -le09:20
ignaziocassanoagargano: have you the same issue ?09:21
agarganoignaziocassano: yes09:21
gthiemongeignaziocassano: do you use ACTIVE_STANDBY loadbalancers?09:24
ignaziocassanogthiemonge: the topology is SINGLE 09:25
gthiemongeignaziocassano: so when I see thoses messages (Read timed out), it's either an issue in the amphora (in that case you should see some logs about that), or a connectivity issue with the amphora (basically a cloud outage)09:39
ignaziocassanogthiemonge: I I created the image two times. I 'll try to create it on centos09:41
ignaziocassanothe problem was the mtu10:44
ignaziocassanogthiemonge: the proble was the mtu: setting it to 1550 works fine10:45
ignaziocassanosorry 150010:45
ignaziocassanoit is strange because the network is 9000, probaply some switch port is not 900010:45
fricklerjohnsom: focal has haproxy 2.0.13, is there any news on!/story/1650270 (haproxy multi cpu)?14:19
johnsomfrickler So, HAProxy decided to enable the (excellent) multi-threading code path by default for 2.x. So, if you build an amp with multi-core you will immediately get the benefit.14:20
johnsomfrickler We have various optimization ideas on top of that on the Octavia roadmap, such as interrupt CPU pinning, etc. but I don't think anyone is working on that yet. Basically enhancements in the amphora agent to implement optimizations.14:22
johnsomfrickler Also, we highly recommmend not enabling nbproc. There are many issues with it.14:23
fricklerjohnsom: so that means an amphora based on focal will use multiple CPUs by default, do I understand this correctly?14:31
johnsomfrickler That is correct14:32
fricklerjohnsom: great, thx14:32
johnsomAny amphora with 2.x or newer will use the cores14:32
opendevreviewMerged openstack/octavia-dashboard master: Drop horizon-nodejs10-jobs template
gthiemonge#startmeeting Octavia16:00
opendevmeetMeeting started Wed Jul  7 16:00:42 2021 UTC and is due to finish in 60 minutes.  The chair is gthiemonge. Information about MeetBot at
opendevmeetUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.16:00
opendevmeetThe meeting name has been set to 'octavia'16:00
gthiemongeHi everyone16:00
johnsomSome folks are on vacation I think, might be a quiet week16:01
gthiemongeDon't be shy16:01
gthiemonge#topic Announcements16:02
gthiemongeNext week is Xena-2 milestone16:02
gthiemongeperhaps we need to publish an intermediate release for python-octaviaclient16:03
gthiemongejohnsom: ^16:03
gthiemongewe missed it for Xena-116:03
johnsomYeah, we have some good backports/patches as well, so worth a release16:03
gthiemongeok,I'll need your help for it16:04
johnsomOk, ping me16:04
gthiemongejohnsom: thanks16:04
johnsomShameless plug:16:04
gthiemongeyeah we need to get those commits16:05
gthiemongeother core reviewers are on vacation16:06
gthiemongeany other announcements?16:07
gthiemonge#topic Brief progress reports / bugs needing review16:09
gthiemongeI have spent most of my time on downstream stuff16:09
gthiemongebut I've also been working at fixing the two-node job and the centos stream job16:10
gthiemongecentos stream support is now broken on devstack (issue with libvirt bindings) I hope it will be fixed soon16:10
johnsomYeah, I had some vacation time. Not much new on Octaiva16:10
gthiemongeand two-node job is still V-1 because of unrelated CI issues16:10
haleybi've been working on the health monitor backport for OVN, and trying to fix it's gates, plan on getting to multi-VIP rebase this week16:12
gthiemongehaleyb: great!16:12
haleybgthiemonge: i have to review your network interface change again as well16:12
gthiemongeI'll have to rebase it, it's in merge conflict :/16:13
gthiemongebut the conflict is probably in a file that I have deleted16:14
johnsomYeah, I don't think I got back to that review yet either16:14
gthiemongeFYI a revert has been proposed in devstack16:14
gthiemongemerging this commit would break our IPV6-based scenario tests16:15
gthiemonge#topic Open Discussion16:18
gthiemongeone last topic about CI issues:16:18
gthiemongesome jobs (noop-api) have been frequently failing in timeout since the beginning of June16:18
gthiemongeit seems that the duration of those jobs have increased a lot16:18
gthiemongefor instance the duration of the scoped-tokes jobs was between 1h30 and 1h45 last month16:19
gthiemongeit is now around 2h16:19
gthiemongeand sometime it hits the timeout (2h15)16:19
gthiemongeit seems that something has changed between June 6th and 11th16:20
johnsomsqlalchemy 1.4 was added to upper constraints June 9th16:20
gthiemongeyeah, I checked the new octavia/octavia-tempest-plugin/devstack commits, I didn't see anything there16:22
gthiemongeso sqlalchemy might be a suspect16:23
gthiemongeMy first idea was to increase the value of the timeout for the noop-api jobs (+15 or 30min)16:24
gthiemongebut I believe that johnsom has some smarter ideas ;-)16:24
johnsomThat means we wait longer.... sad face16:24
johnsomMy thoughts would be to try to narrow down what is slow. If it's the SQL calls, see if there is some new magic incantation we missed (probably not). Then I might explore moving the sqlite file to /dev/shm so it runs in RAM vs. the slow disks on these test hosts. Though RAM is certainly a premium on these test nodes, so that may not be a good idea. Frankly I have no idea what size that file gets during a test run.16:25
johnsomOr we split the jobs up more.16:25
johnsomThese are no-op, so really should be darn fast.16:26
gthiemonge573 tests :D16:26
johnsomThat isn't that much. grin16:27
gthiemongejohnsom: I can work on your first point: analyzing what is slow16:28
johnsomCool. I just can't spare the time right now16:28
gthiemongeperhaps we could isolate one part ofcode that is problematic16:28
johnsomI would be curious to know.16:31
gthiemongeif I don't find anything, we can try to move the sqlite file16:31
gthiemongeor just increase the timeout16:31
johnsomFYI, there is a comment on the discuss mailing list:16:32
johnsomAbout barbican integration in dashboard.16:32
johnsomI don't have a stack with Octavia and dashboard at the moment to test this out.16:33
gthiemongethis is weird, because barbicanAPI.getCertificates is part of octavia-dashboard16:33
gthiemongejohnsom: I tried to reproduce it on master, I didn't find anything16:33
johnsomThere is a code check in the plugin that looks in the service catalog for key-manager, that should disable any calls to the barbican client. But this seemed like some javascript issues I don't understand and would need to run it myself.16:34
johnsomIt's possible that message comes out on the console all the time, but doesn't impact functionality16:34
gthiemongeI will take another look16:36
johnsomIn my experience, almost every page spews something in the browser console these days. Most of it doesn't matter.16:36
gthiemongeany other topics for this meeting?16:41
johnsomThat is all I have today.16:41
gthiemongethanks everyone!16:42
opendevmeetMeeting ended Wed Jul  7 16:42:53 2021 UTC.  Information about MeetBot at . (v 0.1.4)16:42
opendevmeetMinutes (text):
johnsomThanks Greg16:43
rm_work+W'd your client backports16:51
johnsomCool, thanks!16:54
opendevreviewMerged openstack/python-octaviaclient stable/wallaby: Support pagination for 'list' API calls
opendevreviewMerged openstack/python-octaviaclient stable/wallaby: Improve the client performance on large clouds
opendevreviewMerged openstack/python-octaviaclient stable/victoria: Support pagination for 'list' API calls
opendevreviewMerged openstack/python-octaviaclient stable/victoria: Improve the client performance on large clouds
opendevreviewMerged openstack/python-octaviaclient stable/ussuri: Support pagination for 'list' API calls
opendevreviewMerged openstack/python-octaviaclient stable/ussuri: Improve the client performance on large clouds
opendevreviewMerged openstack/python-octaviaclient stable/train: Support pagination for 'list' API calls
opendevreviewMerged openstack/python-octaviaclient stable/train: Improve the client performance on large clouds
opendevreviewDouglas Mendizábal proposed openstack/octavia master: Replace md5 for fips
*** gthiemon1e is now known as gthiemonge20:57

Generated by 2.17.2 by Marius Gedminas - find it at!