Wednesday, 2020-05-13

openstackgerritGhanshyam Mann proposed openstack/python-octaviaclient master: Fix hacking min version to 3.0.1
openstackgerritGhanshyam Mann proposed openstack/octavia-lib master: Fix hacking min version to 3.0.1
openstackgerritGhanshyam Mann proposed openstack/octavia-tempest-plugin master: Fix hacking min version to 3.0.1
*** livelace has quit IRC01:18
*** wuchunyang has joined #openstack-lbaas01:21
*** sapd1 has joined #openstack-lbaas01:31
rm_workdamn, gates are really backed up01:42
johnsomI'm sure that flake8 change caused carnage01:51
*** sapd1_x has joined #openstack-lbaas02:04
*** archiephan has joined #openstack-lbaas02:23
*** kevinz has joined #openstack-lbaas02:35
*** wuchunyang has quit IRC02:40
*** psachin has joined #openstack-lbaas03:38
*** wuchunyang has joined #openstack-lbaas03:50
*** wuchunyang has quit IRC04:02
openstackgerritJie Li proposed openstack/octavia master: add the verify for the session
*** gcheresh_ has joined #openstack-lbaas04:10
openstackgerritJie Li proposed openstack/octavia master: add the verify for the session
*** dmellado has quit IRC04:45
*** psachin has quit IRC04:56
*** wuchunyang has joined #openstack-lbaas05:00
*** gcheresh_ has quit IRC06:49
*** vishalmanchanda has joined #openstack-lbaas07:02
*** gcheresh_ has joined #openstack-lbaas07:24
*** rpittau|afk is now known as rpittau07:29
*** gcheresh has joined #openstack-lbaas07:29
*** gcheresh_ has quit IRC07:29
*** dmellado has joined #openstack-lbaas07:37
*** maciejjozefczyk has joined #openstack-lbaas07:38
*** gcheresh_ has joined #openstack-lbaas07:39
*** gcheresh has quit IRC07:39
*** gcheresh_ has quit IRC07:41
*** gcheresh has joined #openstack-lbaas07:41
*** livelace has joined #openstack-lbaas08:12
*** salmankhan has joined #openstack-lbaas08:30
*** wuchunyang has quit IRC08:35
*** livelace has quit IRC08:50
*** riuzen has joined #openstack-lbaas08:56
*** salmankhan1 has joined #openstack-lbaas08:57
*** livelace has joined #openstack-lbaas08:57
*** salmankhan has quit IRC08:59
*** salmankhan1 is now known as salmankhan08:59
*** livelace has quit IRC09:02
*** livelace has joined #openstack-lbaas09:08
*** livelace has quit IRC09:13
*** livelace has joined #openstack-lbaas09:34
*** wuchunyang has joined #openstack-lbaas10:06
*** wuchunyang has quit IRC10:10
*** wuchunyang has joined #openstack-lbaas10:39
*** wuchunyang has quit IRC11:00
*** rpittau is now known as rpittau|bbl11:04
*** salmankhan1 has joined #openstack-lbaas11:28
*** salmankhan has quit IRC11:28
*** salmankhan1 is now known as salmankhan11:28
*** ccamposr has quit IRC11:37
*** ccamposr has joined #openstack-lbaas11:37
*** yamamoto has joined #openstack-lbaas11:43
*** yamamoto has quit IRC11:43
*** ccamposr__ has joined #openstack-lbaas11:44
*** ccamposr has quit IRC11:47
*** servagem has joined #openstack-lbaas11:53
*** rpittau|bbl is now known as rpittau12:19
*** livelace has quit IRC12:23
*** tkajinam has quit IRC12:24
*** ccamposr has joined #openstack-lbaas12:28
*** livelace has joined #openstack-lbaas12:29
*** ccamposr__ has quit IRC12:32
*** wuchunyang has joined #openstack-lbaas12:52
*** ianychoi has quit IRC12:55
*** riuzen has quit IRC13:11
*** sapd1_x has quit IRC13:31
*** TrevorV has joined #openstack-lbaas13:42
*** archiephan has quit IRC13:44
*** ataraday_ has joined #openstack-lbaas13:53
*** jamesdenton has quit IRC14:03
*** jamesdenton has joined #openstack-lbaas14:11
*** wuchunyang has quit IRC14:41
*** livelace has quit IRC14:45
*** gcheresh has quit IRC15:09
*** jamesdenton has quit IRC15:42
*** jamesdenton has joined #openstack-lbaas15:46
*** sapd1_x has joined #openstack-lbaas15:47
*** ccamposr has quit IRC15:54
*** ccamposr has joined #openstack-lbaas15:54
johnsom#startmeeting Octavia16:00
openstackMeeting started Wed May 13 16:00:01 2020 UTC and is due to finish in 60 minutes.  The chair is johnsom. Information about MeetBot at
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.16:00
*** openstack changes topic to " (Meeting topic: Octavia)"16:00
openstackThe meeting name has been set to 'octavia'16:00
johnsom#chair rm_work16:00
openstackCurrent chairs: johnsom rm_work16:00
johnsomHi everyone!16:00
johnsom#topic Announcements16:01
*** openstack changes topic to "Announcements (Meeting topic: Octavia)"16:01
johnsomUssuri is released!16:01
johnsomIf you didn't attend the highlights meeting this morning, the release is out.16:01
johnsomThank you all for your contributions!16:02
johnsomOctavia got some attention this cycle in the release announcements, etc. so good stuff.16:02
johnsomBoth AZs and TLS ciphers are in the screenshots section16:02
cgoncalveswell done, team!16:03
johnsomAlong those lines, thank you to rm_work for his leadership during Ussuri!16:03
johnsomThis is his last week as PTL and then you are stuck with me. grin16:03
cgoncalves... again16:04
cgoncalvesthank you, rm_work!16:04
*** maciejjozefczyk has quit IRC16:04
johnsomAs the release is out, we are officially open for Victoria patch merge. (Though I think we have been for a bit anyway)16:04
johnsomAlso a friendly reminder, PTG registration is open:16:05
johnsomand we have an etherpad:16:05
johnsomPlease include any topics you would like to discuss during our PTG sessions.16:05
johnsomAny other announcements this week?16:06
johnsom#topic Brief progress reports / bugs needing review16:07
*** openstack changes topic to "Brief progress reports / bugs needing review (Meeting topic: Octavia)"16:07
johnsomOk, other than some reviews and working on the content the foundation requested, I have been focusing on an internal project this week.16:07
ataraday_I pushed small patch for an issue I discovered on our internal cloud16:08
ataraday_some reviews appreciated :)16:09
johnsomI should also mention I am giving a short talk at the OpenStack meetings today covering our Ussuri highlights.  So, shout outs for those features.16:09
cgoncalvessame content as meeting ~2 hours ago, right?16:10
johnsomHmm, interesting. Even an API get is failing...16:10
johnsomYes, same exact content, just a separate time (zone)16:10
johnsomataraday_ Looks good. My only comment would be to maybe tie the retry counts to configuration file settings.16:11
johnsomWe have got feedback in the past that some people want things to retry forever.16:11
johnsomPersonally, I'm getting a bit concerned about the number of these "retry" values we have defined in the configuration file.16:12
ataraday_johnsom, sure, will add it16:12
ataraday_yeah, to many reties...16:12
johnsomI would be interested to hear if people have ideas/thoughts on how to improve that situation or if it is ok as it is.16:12
johnsomMaybe we can consolidate them somehow.16:12
johnsomI know I have added a bunch of them, and I'm just not sure if it's useful for tuning, or just too much.16:13
ataraday_ideally we want to have all retries configurable?16:13
johnsomataraday_ Historically that has been our position.16:14
johnsomMostly because of the very wide performance difference between the nested virtualization the zuul gates use vs. the production situations.16:14
johnsomBut I would like to have a discussion about it. Is it too much, is it good, etc.16:15
*** jamesdenton has quit IRC16:16
johnsomNo other comments?16:18
haleybI have been trying to get the gates functional, and (ussuri, needs further work)16:18
ataraday_The question with retries would also be - should be a big number by default or not, I guess a lot can be disscussed..16:18
johnsomYes, I agree. I think it would be good for the team to have that discussion. Maybe this is a good PTG topic16:19
johnsomhaleyb Thank you.16:19
johnsomI did have a question I haven't posted for one of your patches yet. If we fix those  issues, why do we still need to pin it?16:20
haleybjohnsom: well, for stable we're only fixing until the next release of flake8/pycodestyle/etc since they're unbounded16:21
johnsomUgh, looks like we are caught in the "zuul is really busy, so jobs randomly timeout" cycle there16:21
*** rpittau is now known as rpittau|afk16:21
haleybyeah, pile-up in zuul16:22
johnsomThis is a cap for Ussuri right?16:22
haleybyes, but i'm assuming other branches are broken too, at least they are in neutron back to rocky16:23
*** ramishra has quit IRC16:23
johnsomOk, so your plan was to fix master, cap the stable branches?16:24
haleybjohnsom: yes, just based on how large the patch was16:24
johnsomI think requirements has talked about capping the linters for stable branches before, but it tends to get messy16:24
johnsomYeah, though it's pretty low risk assuming it's just the single letter variable name issue16:25
haleybwho doesn't like the letter 'l' ? :)16:25
johnsomI don't16:26
johnsomFunny story actually of being a wise guy in college and getting stuck in a 101 class due to requirements. I crashed the compiler because I variables were too descriptive.16:26
johnsomAh the good old days.16:27
haleybi might need to cherry-pick if that cap doesn't work16:27
johnsomOk, I am not opposed to that16:27
* cgoncalves challenges johnsom to not use letter 'l' until end of the meeting16:27
johnsomThanks for working on it!16:27
haleybwe all learned in school to use i, j, k for looping, guess that's all we're allowed16:27
johnsomOk, any other topics today?16:28
* johnsom notes, none of that character in the sentence....16:28
johnsom#topic Open Discussion16:29
*** openstack changes topic to "Open Discussion (Meeting topic: Octavia)"16:29
johnsomopps, forgot the topic change16:29
johnsomOk, thank you everyone! Happy Ussuri re1ease day!16:30
cgoncalvesgotcha! re*l*ease!16:31
haleybhe used a 1 to be snarky16:31
johnsomThat was #116:31
*** openstack changes topic to "Discussions for OpenStack Octavia | Priority bug review list:"16:31
openstackMeeting ended Wed May 13 16:31:25 2020 UTC.  Information about MeetBot at . (v 0.1.4)16:31
openstackMinutes (text):
cgoncalveswell done sir16:31
johnsomcgoncalves I think you just proved the point of that flake8 change....16:31
haleybjohnsom: seems stable/train is passing pep8 locally, but stein is a "bloody mess", hundreds of new failures with the new flake816:36
johnsomlol, joy. Do you have a plan or do more folks need to get involved?16:37
haleybjohnsom: i'll see if capping works, i'm already in this dep16:38
*** jamesdenton has joined #openstack-lbaas16:38
johnsomThank you16:39
*** sapd1_x has quit IRC16:40
*** sapd__x has joined #openstack-lbaas16:40
openstackgerritBrian Haley proposed openstack/octavia stable/stein: DNM - pep8 test for stable/stein
openstackgerritBrian Haley proposed openstack/octavia stable/train: DNM - pep8 test for stable/train
*** ataraday_ has quit IRC17:12
*** jamesdenton has quit IRC17:15
*** salmankhan has quit IRC17:22
*** gcheresh has joined #openstack-lbaas17:48
*** jamesdenton has joined #openstack-lbaas17:57
*** jamesdenton has quit IRC18:04
*** jamesdenton has joined #openstack-lbaas18:09
*** sapd__x has quit IRC18:31
*** jamesdenton has quit IRC18:53
*** jamesdenton has joined #openstack-lbaas18:57
*** maciejjozefczyk has joined #openstack-lbaas19:08
*** livelace has joined #openstack-lbaas19:09
*** gcheresh has quit IRC19:09
*** maciejjozefczyk has quit IRC19:25
rm_workYeah I think capping would be fine for stable branches20:12
rm_workAlso, thanks johnsom for taking back over, I am so glad to not have that PTL responsibility anymore (especially given how much I've been forced downstream recently)20:13
*** jamesdenton has quit IRC20:16
openstackgerritBrian Haley proposed openstack/octavia master: Remove deprecated data model classes
openstackgerritBrian Haley proposed openstack/octavia master: Remove deprecated data model classes
openstackgerritBrian Haley proposed openstack/octavia master: Remove deprecated data model classes
*** jamesdenton has joined #openstack-lbaas20:30
*** vishalmanchanda has quit IRC20:31
openstackgerritBrian Haley proposed openstack/octavia master: Remove deprecated DriverLibrary class
openstackgerritBrian Haley proposed openstack/octavia master: Remove deprecated ProviderDriver class
*** servagem has quit IRC20:46
*** livelace has quit IRC21:11
*** livelace has joined #openstack-lbaas21:17
*** livelace has quit IRC21:18
*** jamesdenton has quit IRC21:30
KeithMnemonicjohnsom: do you have a fwe minutes to help me understand an error i am seeing with plug_network ?21:48
KeithMnemonici can manually do all of the tests in the octavia-tempest-plugin and those seem to be fine. i do this for example in a demo tenant21:48
KeithMnemonicwhen i run the test though i get an odd error when it tries to add the members about "cannot find eth2"21:49
KeithMnemonici am not quite finding where it is getting this from21:49
johnsomKeithMnemonic Sure, I can use a break from spreadsheet engineering21:49
KeithMnemonicAmphora agent returned unexpected result code 500 with response {u'message': u'Error plugging network', u'details': u'eth2            no-device\n'}21:50
KeithMnemonicso i am looking here
KeithMnemonicso i think this is the 500
johnsomYeah, that is the catch-all exception wrapper. So, do you know if nova was able to actually attach the neutron interface to the VM?21:51
KeithMnemonicand i can create an lb/listener/pool/member fine from the cli21:51
KeithMnemonici do not think it did, but for my manual tests that have active members, if i ssh to the amphora instance i only see eth121:52
johnsomYeah, you should only see eth1, which is the lb-mgmt-net. The tenant traffic interfaces are inside the network namespace.21:53
johnsom"sudo ip netns exec amphora-haproxy ip a" would show you the tenant interfaces21:53
johnsomBut, I don't think the problem is in the amphora. It's likely a mis-configuration in tempest/nova/neutron somewhere. I would start with the worker log file, find where tempest is making the call to create the member, and step through the log to see why nova wasn't able to attach the member network port.21:55
KeithMnemonicso i have active/standby. i check both instances and the netns has the correct info but only eth121:55
johnsomAlso, don't get confused by "eth1, eth2, eth3" etc. We tightly control those names, and rename interfaces as part of the normal process.21:55
johnsomOk, so you didn't specify a subnet when you created the member or it resides on the same subnet as the VIP21:56
johnsomIn which case there is only one network port needed.21:56
KeithMnemonicso for my manual test, i did like tempest , 3 networks21:56
KeithMnemonicone for inst1, one for inst2, and a third for lb121:57
johnsomAnd you added "--subnet <uuid>" to the member create command line?21:57
johnsomsorry, it is "--subnet-id <uuid>"21:58
KeithMnemonici used --vip-subnet-id21:58
KeithMnemonicso maybe my manual test is wrong21:58
johnsomNo, that is only valid on load balancer create, not member create21:58
johnsomYou should get an error if you used --vip-subnet-id on a member create call21:59
KeithMnemonicof sorry21:59
KeithMnemonic for member i gave the address of the instance21:59
KeithMnemonicopenstack loadbalancer member create  --protocol-port 80 --name demo-member1 --address
johnsomIf that was all, you told Octavia to only use the VIP network. So, yeah, you didn't configure those to be on the tenant networks.22:00
KeithMnemonicah ok so i need to also add --vip-subnet-id22:00
johnsomIt created a load balancer that expects the IP address to be reachable from the VIP network.22:00
johnsomYes, if it is on a network different from the VIP.22:00
KeithMnemonicok so that is how my manual test and the tempest differ, let me kill the member and recreate it22:01
johnsomopenstack loadbalancer member create --subnet-id tenant-2-subnet --address $webserver2ip --protocol-port 80 pool122:01
johnsomThat is an example from my test script22:01
KeithMnemonicah ok so i need to also add --vip-subnet-id22:02
KeithMnemonicok let me try that now22:02
KeithMnemonicthat was a typo22:03
KeithMnemonici know what i need to do22:03
KeithMnemonicok interesting22:06
KeithMnemonici get the same error22:06
KeithMnemonicbut i did the eth2 for a split second22:06
KeithMnemonicamphora-8cf40858-3e6d-4ab8-9a12-f16b1b547564:~ # sudo ip netns exec amphora-haproxy ip a22:07
KeithMnemonic4: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP>22:07
johnsomDid it get an address?22:07
*** spatel has joined #openstack-lbaas22:08
KeithMnemonicnope, it disappeared almost immediately22:08
KeithMnemonicso it did not seem like it waited long if it was trying to get one22:08
johnsomYeah, if nova fails, we go straight into cleanup mode and start undoing stuff22:09
KeithMnemonicok let me boot a cirros and try and attach all of these nets and see what happens22:10
KeithMnemonicis that a sufficient test, to have the cirros sit on the instance and loadbalancer nets, as well as octavia mgmt?22:10
johnsomIf you can get into the amphora, check the syslog for amphora agent records, it will probably just say the same thing as the controller worker captured. At that point you really need to go back the worker log where the original error was probably caught22:10
johnsomNo, probably not, much different animal22:11
johnsomcirros doesn't really support more than one interface anyway22:11
KeithMnemonici have octavia in debug but am not really seeing much as the first error is right after the plug22:11
johnsomYou have to do a bunch of manual stuff to get them working22:12
*** spatel has quit IRC22:12
KeithMnemonici could boot an amphora instance manually and test22:12
johnsomDo you want to a good hangout to share the log with me?22:12
KeithMnemonicor zoom or ....22:13
*** spatel has joined #openstack-lbaas22:20
*** TrevorV has quit IRC22:42
*** spatel has quit IRC22:45
*** tkajinam has joined #openstack-lbaas22:46
*** ccamposr__ has joined #openstack-lbaas22:52
*** ccamposr has quit IRC22:55
*** hongbin has joined #openstack-lbaas23:05
*** threestrands has joined #openstack-lbaas23:29
*** rcernin has quit IRC23:31
*** rcernin has joined #openstack-lbaas23:32

Generated by 2.17.2 by Marius Gedminas - find it at!