Wednesday, 2020-05-20

*** armax has quit IRC00:09
*** wuchunyang has joined #openstack-lbaas00:10
*** wuchunyang has quit IRC00:15
*** threestrands has joined #openstack-lbaas00:19
*** wuchunyang has joined #openstack-lbaas01:30
*** wuchunyang has quit IRC01:35
*** armax has joined #openstack-lbaas01:39
*** armax has quit IRC01:43
*** riuzen has joined #openstack-lbaas01:56
*** armax has joined #openstack-lbaas02:22
openstackgerritJie Li proposed openstack/octavia master: add the verify for the session
*** sapd1_x has joined #openstack-lbaas03:08
*** riuzen has quit IRC03:11
*** psachin has joined #openstack-lbaas03:36
*** wuchunyang has joined #openstack-lbaas04:07
*** wuchunyang has quit IRC04:20
*** psachin has quit IRC05:05
*** wuchunyang has joined #openstack-lbaas05:08
*** wuchunyang has quit IRC05:12
*** gcheresh has joined #openstack-lbaas05:14
*** ccamposr has joined #openstack-lbaas06:18
*** ccamposr__ has quit IRC06:21
*** ccamposr__ has joined #openstack-lbaas06:30
*** ccamposr has quit IRC06:33
*** dmellado has quit IRC07:02
*** maciejjozefczyk has joined #openstack-lbaas07:07
*** ccamposr has joined #openstack-lbaas07:22
*** vishalmanchanda has joined #openstack-lbaas07:25
*** ccamposr__ has quit IRC07:26
*** also_stingrayza has joined #openstack-lbaas07:42
*** stingrayza has quit IRC07:47
*** ramishra has joined #openstack-lbaas08:11
*** ramishra has quit IRC08:17
*** tkajinam has quit IRC08:23
*** dmellado has joined #openstack-lbaas08:34
*** wuchunyang has joined #openstack-lbaas08:38
*** ramishra has joined #openstack-lbaas08:46
*** ramishra has quit IRC09:09
*** ramishra has joined #openstack-lbaas09:31
*** wuchunyang has quit IRC10:06
openstackgerritLingxian Kong proposed openstack/octavia master: Fix getting user neutron client
*** wuchunyang has joined #openstack-lbaas10:37
*** sapd1_x has quit IRC11:41
*** ramishra has quit IRC11:58
*** vishalmanchanda has quit IRC12:17
*** ramishra has joined #openstack-lbaas13:03
*** andrein_ has joined #openstack-lbaas13:08
*** andrein has quit IRC13:10
*** andrein_ is now known as andrein13:10
*** jamesdenton has joined #openstack-lbaas13:15
*** ataraday_ has joined #openstack-lbaas13:16
*** sapd1_x has joined #openstack-lbaas13:34
*** vishalmanchanda has joined #openstack-lbaas13:41
*** TrevorV has joined #openstack-lbaas13:41
colin-don't ever want to hear a10 again :triggered:13:59
colin-i'm so mad i'm quitting13:59
*** colin- has quit IRC13:59
*** ramishra has quit IRC14:03
*** colin- has joined #openstack-lbaas14:03
*** KeithMnemonic has joined #openstack-lbaas14:19
*** gcheresh has quit IRC14:21
*** LuckyClover has joined #openstack-lbaas14:21
KeithMnemonicjohnsom: when you were helping me last week you mentioned something about the amphora-agent and systemd. do you recall. trying to see what i need to do to get better logging enabled14:22
LuckyCloverhas anyone had issues with octavia on horizon UI on v. 20.1.1?14:25
*** threestrands has quit IRC14:34
*** gcheresh has joined #openstack-lbaas14:40
johnsomKeithMnemonic I wasn't sure where your Octavia process logs were going. There is an issue in old versions of Octavia where the API connection logging (uwsgi) goes to console and the Octavia code errors can go to a syslog location. We fixed that in later versions, but overall I think my comment was "Where are the process logs going on Suse?????"14:55
johnsomLuckyClover What is 20.1.1? Horizon Ussuri release was 18.3.2 and octavia-dashboard was version
KeithMnemonicok so i do not see debug anywhere, i see some startup stuff in /var/log, a bit more in /var/log/octavia and nothing really in journalctl so was not sure if we missed a config setting somewhere14:56
johnsomYeah, I'm just not sure where they are going there. We use the standard oslo logging settings in our config file, so... Maybe start with what someone put in your oslo logging settings?14:57
KeithMnemonicand i found another issue with the patch from swami, maybe not huge but this we would need to also customize that line for SLES/opensuse14:57
*** gcheresh has quit IRC15:11
LuckyCloverjohnsom sorry not sure why I said 20.1.1 - i was referring to ansible playbooks version there15:14
johnsomLuckyClover Oh, I don't know about the status of the deployers for octavia-dashboard. I have run Ussuri octavia-dashboard without issue, but that was not using OpenStack Ansible or Kolla. I would maybe ask in the deployer IRC channels if you are seeing issues deploying it. If it's an error/issue with the dashboard itself I might be able to help.15:19
*** LuckyClover has quit IRC15:22
johnsom#startmeeting Octavia16:00
openstackMeeting started Wed May 20 16:00:33 2020 UTC and is due to finish in 60 minutes.  The chair is johnsom. Information about MeetBot at
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.16:00
*** openstack changes topic to " (Meeting topic: Octavia)"16:00
openstackThe meeting name has been set to 'octavia'16:00
johnsomHi everyone!16:00
johnsomWelcome to the second week of Victoria development16:01
johnsom#topic Announcements16:01
*** openstack changes topic to "Announcements (Meeting topic: Octavia)"16:02
johnsomMy regular items for the PTG:16:02
johnsomPTG registartion is open16:02
johnsomOur PTG etherpad:16:02
johnsomNext week I will try to put together a tentative schedule based on the topics we have in the etherpad.16:03
johnsomIt almost never goes to that plan, but it's better having a rough idea than nothing.16:03
*** gcheresh has joined #openstack-lbaas16:03
johnsomSo, please add any topics you would like to discuss with the community this week or early next week so I can get a reasonable schedule put together.16:04
johnsomAny other announcements this week?16:04
*** rcernin has quit IRC16:04
johnsom#topic Brief progress reports / bugs needing review16:05
*** openstack changes topic to "Brief progress reports / bugs needing review (Meeting topic: Octavia)"16:05
johnsomI was off doing downstream stuff most of last week.16:05
johnsomThis week I have been trying to focus on code reviews and helping with rebasing16:06
johnsomWe have a number of things we need to catch up on that has me nervous. One is porting the failover patch to the v2 code path. Then some follow on patches (amphora delete for example). We also need to go back and do the "multi-process" code path for the TLS work. Double check it is all v2 ok, then finish the tempest and dashboard work on the TLS patches. I also have more tempest refactor work to finish, which16:08
johnsommay require multi-node working.16:08
ataraday_Probably better for open discussion, but grenade job start failing on default ciphers change
ataraday_not sure what is causing it...16:09
cgoncalvesthe failover patch is still open and needs to be reviewed and merged asap, yeah. we've had customers hitting amphora failover issues due to broken clusters.16:11
johnsomataraday_ "unknown keyword 'ssl-default-bind-ciphersuites' in 'global' section\n" is the issue with ciphers16:12
cgoncalvesI want to give it a good review, but coming from Michael and been reviewed by Adam I'm sure it's in pretty good shape :)16:12
johnsomNot sure why the tls test didn't also fail though. Plus, really points to the need to get the tempest tests done16:12
cgoncalvesyeah, I also noticed grenade job failing on the default ciphers. I did not have time to look deep but looked like a legit issue16:13
ataraday_johnsom, yeah but all other jobs don't fail in that way - process_cfg_for_version_compat should remove it if it is not supported...16:14
cgoncalveson  a second thought, I think I saw in in relation to tls version, not cipher16:14
ataraday_I fixed tls version in tempest - so barbican job passed16:15
cgoncalves__init__() got an unexpected keyword argument 'tls_ciphers' (HTTP 500)16:15
johnsomI put figuring out the TLS 1.3 ciphersuite settings as a PTG topic as I think we have a choice to make there about the API16:15
johnsomFYI, I got reviews on some of the TLS patches this week. I think Dawson is interested in seeing them through, so I will give him some time to address the comments before taking those over.16:17
johnsomI'm not sure if the other students will do any additional updates to their patches16:17
*** gcheresh has quit IRC16:18
johnsomAny other updates today?16:19
johnsomOne more thing I could mention here  or in open discussion. I'm poking at the configuration of our gate jobs some. This is specifically related to the tempest refactor and the pending need for faster testing.16:20
johnsomI have posted a test job that moves the libvirt machine type over to q35 from i440fx. This is a long overdue change that the nova team is considering as well.16:21
johnsomFYI, i440fx platform was the hot thing in 1997.... It is PCI based16:21
haleybPentium Pro16:22
johnsomq35 is a more modern platform and getting more attention from the kvm, etc. folks. It is PCIe based.16:22
johnsomI don't expect a lot of performance benefit from this change, but I'm hoping for a bit more stability.16:22
cgoncalvesthank you for looking into it16:22
cgoncalvesquestion: why not wait for Nova to make it default?16:23
johnsomSo, just an FYI really.16:23
johnsomWell, that has been kicking around nova for over a year.16:23
cgoncalvesis it to fix the KVM crashes in vexxhost?16:23
johnsomI think it might help us and I want to get out in front of it so 8if* it breaks us in some way we know before it's the default in nova.16:24
gthiemongeone more thing: tempest tests for UDP LBs are failing on CentOS because of the behavior of nmap-netcat (used by checkers), I proposed a fix at
cgoncalvesok. just wanted to understand the motivation behind it16:24
johnsomIt *might* help with the KVM stability issues. It also might help us with performance with the virtio drivers since it gets more attention than the i44016:25
johnsomBasically I am kicking the tires and giving it a try to see what we get.16:25
johnsom#topic RFE/Specifications16:26
*** openstack changes topic to "RFE/Specifications (Meeting topic: Octavia)"16:26
johnsomI also wanted to highlight we have a couple of specs up for review16:26
johnsomI would really like us to be responsive on these. It would be great if everyone could review, comment, and help out with wording.16:27
johnsomI thought there was a third, but I didn't see it in my quick pre-meeting search. Let me know if you see it.16:28
johnsomI expect both of these to need a few cycles before they are good to go.16:28
johnsomNotifications are also on the PTG discussion list.16:29
johnsomA reminder, we are not looking for perfection on these, but we want to make sure:16:29
johnsomThe use case is clear16:29
johnsomWe have designed any problem areas16:29
johnsomCall out API changes required16:30
johnsomCall out database schema changes/designs16:30
johnsomHas a reasonably clear description of the change to the code base (i.e. where hooks for notifications would be placed, etc.)16:30
johnsomWe want to discuss anything that once merged we would be stuck with type design choices...16:31
johnsomAny questions or comments on the specs?16:32
johnsom#topic Open Discussion16:34
cgoncalvesI agree with your suggestions for the RFEs16:34
*** openstack changes topic to "Open Discussion (Meeting topic: Octavia)"16:34
johnsomAny other topics today?16:34
johnsomataraday_ Did we cover the grenade job issue or is there more we should discuss there?16:35
ataraday_did something changed in this job recently?16:37
ataraday_as it was successful before..16:37
johnsomI think the switch to zuul v3 based grenade merged recently (or it's still an open review)16:38
haleybbut it's failing too16:39
cgoncalves __init__() got an unexpected keyword argument 'tls_ciphers'16:40
haleybmy link was to the stable/ussuri one, which is failing in the same way16:42
cgoncalvesI see /opt/stack/old/octavia-lib/ in the trace. wondering if octavia-lib is not being upgraded, it should have been at that point16:42
johnsomThere might have been a bug here:16:42
johnsomOh, that is interesting, yeah, if the transition doesn't update that module it would have issues16:43
cgoncalvesI can take point in this16:45
cgoncalvesmay be a legit issue in our grenade job actually, so my bad16:45
johnsomcgoncalves Thank you!16:45
johnsomAny other topics today?16:46
johnsomOk, thank you for another great meeting. Good discussions.  Have a great week!16:47
*** openstack changes topic to "Discussions for OpenStack Octavia | Priority bug review list:"16:48
openstackMeeting ended Wed May 20 16:48:04 2020 UTC.  Information about MeetBot at . (v 0.1.4)16:48
openstackMinutes (text):
cgoncalvesthank you16:48
ataraday_thank you!16:48
*** gcheresh has joined #openstack-lbaas16:49
*** sapd1_x has quit IRC16:52
johnsomcgoncalves FYI, if it is the octavia-lib not getting updated, we probably should add that as a step in our upgrade guide as well.16:56
cgoncalvesjohnsom, good idea. it is odd if we have to manually upgrade octavia-lib in the grenade job. when upgrading octavia, grenade should upgrade libs in requirements.txt too16:59
johnsomYeah, I would expect so as well16:59
*** ataraday_ has quit IRC17:02
*** sapd1_x has joined #openstack-lbaas17:04
*** gcheresh has quit IRC17:12
*** ccamposr__ has joined #openstack-lbaas17:28
*** ccamposr has quit IRC17:31
*** gcheresh has joined #openstack-lbaas18:04
*** sapd1_x has quit IRC18:26
haleybjohnsom: so i've been looking at ways the tls_ciphers change differed from a previous one adding tls_enabled and had a question18:37
haleybin should the new value have been wtypes.wsattr(wtypes.StringType())18:38
haleyblike all the others above ie18:38
johnsomhaleyb This is something people do oddly when using wsme. So, technically you only need wsattr if you are augmenting the type, like adding a default value (we don't do that in the API type usually). That said, there is no error if you wrap a type in an wsattr without an augmentation.18:47
johnsomFor example, on string type it could be a default, or an enum list of valid values, etc. then you would use wsattr.18:48
haleybit just seemed odd to be different, even if it works18:50
haleybalso noticed there was no unicode u'tls_cipers' in the alembic migration, again like previous ones18:50
johnsomOh, agreed. All of those type files are all over the board. It's low priority tech debt to go clean those all up. I just haven't got around to it18:50
haleybnone explain the __init__ failure of course18:51
johnsomYeah, I don't think the u' matter for the column names. We have a DB default for unicode/utf-818:52
haleybmy nitpicking brain notices18:53
johnsomMissed your chance..... lol18:53
haleybthe grenade failure is interesting, and not obvious at first glance, but at least there's a good backtrace18:54
johnsomYeah, show create table strips the unicode designation. It's this part that matters: "ENGINE=InnoDB DEFAULT CHARSET=utf8"18:55
johnsomI thought cgoncalves took point on that. Did you guys trade or are two people looking at the same thing?18:55
haleyboh, it's the "old" octavia lib but the new dict18:55
haleybyeah, he did, i just clicked on the grenade log since it failed for me in another change18:56
johnsomYeah, pretty sure Carlos said he thinks the octavia-lib isn't getting upgraded with the octavia module upgrade.18:56
haleybcode /opt/stack/old/octavia-lib/octavia_lib/api/drivers/data_models.py18:57
haleybanyways, was just poking18:57
*** gcheresh has quit IRC18:58
haleybthe octavia code is different enough from neutron i'm trying to understand it18:58
johnsomI know, it's soooo much better.... lol18:59
haleybriiiight :)19:00
johnsomThere is a little sticker in the corner, "The legacy free advantage!"19:00
haleybso i can just rm -rf */v1 stuff? :)19:01
johnsomWe did it once already:
johnsomIt will happen again19:03
haleybjust add a v3 to really confuse people19:05
johnsomWe have seriously talked about it. v2 is still encumbered by some neutron cruft.19:05
openstackgerritAndreas Jaeger proposed openstack/python-octaviaclient master: Switch to newer openstackdocstheme and reno versions
*** gcheresh has joined #openstack-lbaas19:19
*** gcheresh has quit IRC19:41
*** gcheresh has joined #openstack-lbaas20:13
*** gcheresh has quit IRC20:20
*** maciejjozefczyk has quit IRC20:30
*** vishalmanchanda has quit IRC20:37
*** wuchunyang has quit IRC21:21
rm_workoh gthiemonge yeah i ran into that same issue with centos nmap being broken for our UDP checks. I fixed it by NOT USING UDP TYPE CHECKS BECAUSE THEY ARE REALLY BAD21:57
rm_workbut I will check your change21:57
rm_workalso the behavior is totally different besides that21:58
rm_workit just *doesn't work* in centos21:58
rm_workthe -z21:58
rm_workat least in my testing21:59
rm_worksomething about how the packet is constructed/sent is different between centos version and, for example, ubuntu version21:59
rm_workoh wait no, I was testing with a static version22:03
rm_worki mean, centos version was still broken, but the behavior of the packets was different between centos (not working) and static (working)22:04
johnsomrm_work We found this recently:
johnsomCarlos is going to cook a fix22:05
rm_workyeah i see how that could happen22:05
rm_workor rather, i see how it could be a problem22:05
rm_workbut how can a peer name start with a hyphen?22:05
johnsombase64 encoding using the URL variant22:06
rm_workah base64 yeah ok22:06
rm_workcan it be ... quoted?22:07
johnsomWorst case, HM kills it and rebuilds it with a different UUID, so self heals, but....22:07
johnsomYeah, we gave Carlos a few ideas for potential ways to fix it22:07
johnsomwork around it I should say. It looks like upstream will probably fix as well since they have already identified the problem code (less than an hour response, pretty nice).22:08
openstackgerritBrian Haley proposed openstack/octavia master: Remove all deprecated driver code that moved to octavia-lib
rm_workyeah, very nice22:13
openstackgerritBrian Haley proposed openstack/octavia master: Remove all deprecated driver code that moved to octavia-lib
*** TrevorV has quit IRC22:35
*** tkajinam has joined #openstack-lbaas22:37
*** rcernin has joined #openstack-lbaas23:08
*** threestrands has joined #openstack-lbaas23:49

Generated by 2.17.2 by Marius Gedminas - find it at!