| *** armax has quit IRC | 00:09 | |
| *** wuchunyang has joined #openstack-lbaas | 00:10 | |
| *** wuchunyang has quit IRC | 00:15 | |
| *** threestrands has joined #openstack-lbaas | 00:19 | |
| *** wuchunyang has joined #openstack-lbaas | 01:30 | |
| *** wuchunyang has quit IRC | 01:35 | |
| *** armax has joined #openstack-lbaas | 01:39 | |
| *** armax has quit IRC | 01:43 | |
| *** riuzen has joined #openstack-lbaas | 01:56 | |
| *** armax has joined #openstack-lbaas | 02:22 | |
| openstackgerrit | Jie Li proposed openstack/octavia master: add the verify for the session https://review.opendev.org/726567 | 02:22 |
|---|---|---|
| *** sapd1_x has joined #openstack-lbaas | 03:08 | |
| *** riuzen has quit IRC | 03:11 | |
| *** psachin has joined #openstack-lbaas | 03:36 | |
| *** wuchunyang has joined #openstack-lbaas | 04:07 | |
| *** wuchunyang has quit IRC | 04:20 | |
| *** psachin has quit IRC | 05:05 | |
| *** wuchunyang has joined #openstack-lbaas | 05:08 | |
| *** wuchunyang has quit IRC | 05:12 | |
| *** gcheresh has joined #openstack-lbaas | 05:14 | |
| *** ccamposr has joined #openstack-lbaas | 06:18 | |
| *** ccamposr__ has quit IRC | 06:21 | |
| *** ccamposr__ has joined #openstack-lbaas | 06:30 | |
| *** ccamposr has quit IRC | 06:33 | |
| *** dmellado has quit IRC | 07:02 | |
| *** maciejjozefczyk has joined #openstack-lbaas | 07:07 | |
| *** ccamposr has joined #openstack-lbaas | 07:22 | |
| *** vishalmanchanda has joined #openstack-lbaas | 07:25 | |
| *** ccamposr__ has quit IRC | 07:26 | |
| *** also_stingrayza has joined #openstack-lbaas | 07:42 | |
| *** stingrayza has quit IRC | 07:47 | |
| *** ramishra has joined #openstack-lbaas | 08:11 | |
| *** ramishra has quit IRC | 08:17 | |
| *** tkajinam has quit IRC | 08:23 | |
| *** dmellado has joined #openstack-lbaas | 08:34 | |
| *** wuchunyang has joined #openstack-lbaas | 08:38 | |
| *** ramishra has joined #openstack-lbaas | 08:46 | |
| *** ramishra has quit IRC | 09:09 | |
| *** ramishra has joined #openstack-lbaas | 09:31 | |
| *** wuchunyang has quit IRC | 10:06 | |
| openstackgerrit | Lingxian Kong proposed openstack/octavia master: Fix getting user neutron client https://review.opendev.org/726042 | 10:22 |
| *** wuchunyang has joined #openstack-lbaas | 10:37 | |
| *** sapd1_x has quit IRC | 11:41 | |
| *** ramishra has quit IRC | 11:58 | |
| *** vishalmanchanda has quit IRC | 12:17 | |
| *** ramishra has joined #openstack-lbaas | 13:03 | |
| *** andrein_ has joined #openstack-lbaas | 13:08 | |
| *** andrein has quit IRC | 13:10 | |
| *** andrein_ is now known as andrein | 13:10 | |
| *** jamesdenton has joined #openstack-lbaas | 13:15 | |
| *** ataraday_ has joined #openstack-lbaas | 13:16 | |
| *** sapd1_x has joined #openstack-lbaas | 13:34 | |
| *** vishalmanchanda has joined #openstack-lbaas | 13:41 | |
| *** TrevorV has joined #openstack-lbaas | 13:41 | |
| colin- | don't ever want to hear a10 again :triggered: | 13:59 |
| colin- | i'm so mad i'm quitting | 13:59 |
| *** colin- has quit IRC | 13:59 | |
| *** ramishra has quit IRC | 14:03 | |
| *** colin- has joined #openstack-lbaas | 14:03 | |
| *** KeithMnemonic has joined #openstack-lbaas | 14:19 | |
| *** gcheresh has quit IRC | 14:21 | |
| *** LuckyClover has joined #openstack-lbaas | 14:21 | |
| KeithMnemonic | johnsom: when you were helping me last week you mentioned something about the amphora-agent and systemd. do you recall. trying to see what i need to do to get better logging enabled | 14:22 |
| LuckyClover | has anyone had issues with octavia on horizon UI on v. 20.1.1? | 14:25 |
| *** threestrands has quit IRC | 14:34 | |
| *** gcheresh has joined #openstack-lbaas | 14:40 | |
| johnsom | KeithMnemonic I wasn't sure where your Octavia process logs were going. There is an issue in old versions of Octavia where the API connection logging (uwsgi) goes to console and the Octavia code errors can go to a syslog location. We fixed that in later versions, but overall I think my comment was "Where are the process logs going on Suse?????" | 14:55 |
| johnsom | LuckyClover What is 20.1.1? Horizon Ussuri release was 18.3.2 and octavia-dashboard was version 5.0.0. | 14:56 |
| KeithMnemonic | ok so i do not see debug anywhere, i see some startup stuff in /var/log, a bit more in /var/log/octavia and nothing really in journalctl so was not sure if we missed a config setting somewhere | 14:56 |
| johnsom | Yeah, I'm just not sure where they are going there. We use the standard oslo logging settings in our config file, so... Maybe start with what someone put in your oslo logging settings? | 14:57 |
| KeithMnemonic | and i found another issue with the patch from swami, maybe not huge but this https://github.com/openstack/octavia/blob/master/octavia/amphorae/backends/agent/api_server/templates/amphora-netns.systemd.j2#L27 we would need to also customize that line for SLES/opensuse | 14:57 |
| *** gcheresh has quit IRC | 15:11 | |
| LuckyClover | johnsom sorry not sure why I said 20.1.1 - i was referring to ansible playbooks version there | 15:14 |
| johnsom | LuckyClover Oh, I don't know about the status of the deployers for octavia-dashboard. I have run Ussuri octavia-dashboard without issue, but that was not using OpenStack Ansible or Kolla. I would maybe ask in the deployer IRC channels if you are seeing issues deploying it. If it's an error/issue with the dashboard itself I might be able to help. | 15:19 |
| *** LuckyClover has quit IRC | 15:22 | |
| johnsom | #startmeeting Octavia | 16:00 |
| openstack | Meeting started Wed May 20 16:00:33 2020 UTC and is due to finish in 60 minutes. The chair is johnsom. Information about MeetBot at http://wiki.debian.org/MeetBot. | 16:00 |
| openstack | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 16:00 |
| *** openstack changes topic to " (Meeting topic: Octavia)" | 16:00 | |
| openstack | The meeting name has been set to 'octavia' | 16:00 |
| johnsom | Hi everyone! | 16:00 |
| gthiemonge | Hi! | 16:00 |
| ataraday_ | hi | 16:01 |
| cgoncalves | hi | 16:01 |
| johnsom | Welcome to the second week of Victoria development | 16:01 |
| johnsom | #topic Announcements | 16:01 |
| *** openstack changes topic to "Announcements (Meeting topic: Octavia)" | 16:02 | |
| haleyb | hi | 16:02 |
| johnsom | My regular items for the PTG: | 16:02 |
| johnsom | PTG registartion is open | 16:02 |
| johnsom | #link https://www.openstack.org/ptg/ | 16:02 |
| johnsom | Our PTG etherpad: | 16:02 |
| johnsom | #link https://etherpad.opendev.org/p/octavia-virtual-V-ptg | 16:02 |
| johnsom | Next week I will try to put together a tentative schedule based on the topics we have in the etherpad. | 16:03 |
| johnsom | It almost never goes to that plan, but it's better having a rough idea than nothing. | 16:03 |
| *** gcheresh has joined #openstack-lbaas | 16:03 | |
| johnsom | So, please add any topics you would like to discuss with the community this week or early next week so I can get a reasonable schedule put together. | 16:04 |
| johnsom | Any other announcements this week? | 16:04 |
| *** rcernin has quit IRC | 16:04 | |
| johnsom | #topic Brief progress reports / bugs needing review | 16:05 |
| *** openstack changes topic to "Brief progress reports / bugs needing review (Meeting topic: Octavia)" | 16:05 | |
| johnsom | I was off doing downstream stuff most of last week. | 16:05 |
| johnsom | This week I have been trying to focus on code reviews and helping with rebasing | 16:06 |
| johnsom | We have a number of things we need to catch up on that has me nervous. One is porting the failover patch to the v2 code path. Then some follow on patches (amphora delete for example). We also need to go back and do the "multi-process" code path for the TLS work. Double check it is all v2 ok, then finish the tempest and dashboard work on the TLS patches. I also have more tempest refactor work to finish, which | 16:08 |
| johnsom | may require multi-node working. | 16:08 |
| ataraday_ | Probably better for open discussion, but grenade job start failing on default ciphers change https://review.opendev.org/#/c/685337/ | 16:09 |
| johnsom | Doh | 16:09 |
| ataraday_ | not sure what is causing it... | 16:09 |
| cgoncalves | the failover patch is still open and needs to be reviewed and merged asap, yeah. we've had customers hitting amphora failover issues due to broken clusters. | 16:11 |
| johnsom | ataraday_ "unknown keyword 'ssl-default-bind-ciphersuites' in 'global' section\n" is the issue with ciphers | 16:12 |
| cgoncalves | I want to give it a good review, but coming from Michael and been reviewed by Adam I'm sure it's in pretty good shape :) | 16:12 |
| johnsom | Not sure why the tls test didn't also fail though. Plus, really points to the need to get the tempest tests done | 16:12 |
| cgoncalves | yeah, I also noticed grenade job failing on the default ciphers. I did not have time to look deep but looked like a legit issue | 16:13 |
| ataraday_ | johnsom, yeah but all other jobs don't fail in that way - https://review.opendev.org/#/c/685337/27/octavia/amphorae/backends/agent/api_server/haproxy_compatibility.py@62 process_cfg_for_version_compat should remove it if it is not supported... | 16:14 |
| cgoncalves | on a second thought, I think I saw in in relation to tls version, not cipher | 16:14 |
| cgoncalves | https://review.opendev.org/#/c/726741/ | 16:15 |
| ataraday_ | I fixed tls version in tempest - so barbican job passed | 16:15 |
| cgoncalves | __init__() got an unexpected keyword argument 'tls_ciphers' (HTTP 500) | 16:15 |
| johnsom | I put figuring out the TLS 1.3 ciphersuite settings as a PTG topic as I think we have a choice to make there about the API | 16:15 |
| johnsom | FYI, I got reviews on some of the TLS patches this week. I think Dawson is interested in seeing them through, so I will give him some time to address the comments before taking those over. | 16:17 |
| johnsom | I'm not sure if the other students will do any additional updates to their patches | 16:17 |
| *** gcheresh has quit IRC | 16:18 | |
| johnsom | Any other updates today? | 16:19 |
| johnsom | One more thing I could mention here or in open discussion. I'm poking at the configuration of our gate jobs some. This is specifically related to the tempest refactor and the pending need for faster testing. | 16:20 |
| johnsom | I have posted a test job that moves the libvirt machine type over to q35 from i440fx. This is a long overdue change that the nova team is considering as well. | 16:21 |
| johnsom | FYI, i440fx platform was the hot thing in 1997.... It is PCI based | 16:21 |
| haleyb | Pentium Pro | 16:22 |
| johnsom | q35 is a more modern platform and getting more attention from the kvm, etc. folks. It is PCIe based. | 16:22 |
| johnsom | I don't expect a lot of performance benefit from this change, but I'm hoping for a bit more stability. | 16:22 |
| cgoncalves | thank you for looking into it | 16:22 |
| cgoncalves | question: why not wait for Nova to make it default? | 16:23 |
| johnsom | So, just an FYI really. | 16:23 |
| johnsom | Well, that has been kicking around nova for over a year. | 16:23 |
| cgoncalves | is it to fix the KVM crashes in vexxhost? | 16:23 |
| johnsom | I think it might help us and I want to get out in front of it so 8if* it breaks us in some way we know before it's the default in nova. | 16:24 |
| gthiemonge | one more thing: tempest tests for UDP LBs are failing on CentOS because of the behavior of nmap-netcat (used by checkers), I proposed a fix at https://review.opendev.org/#/c/729230/ | 16:24 |
| cgoncalves | ok. just wanted to understand the motivation behind it | 16:24 |
| johnsom | It *might* help with the KVM stability issues. It also might help us with performance with the virtio drivers since it gets more attention than the i440 | 16:25 |
| johnsom | Basically I am kicking the tires and giving it a try to see what we get. | 16:25 |
| johnsom | #topic RFE/Specifications | 16:26 |
| *** openstack changes topic to "RFE/Specifications (Meeting topic: Octavia)" | 16:26 | |
| johnsom | I also wanted to highlight we have a couple of specs up for review | 16:26 |
| johnsom | Notifications | 16:26 |
| johnsom | #link https://review.opendev.org/727915 | 16:27 |
| johnsom | Active/Active | 16:27 |
| johnsom | #link https://review.opendev.org/723864 | 16:27 |
| johnsom | I would really like us to be responsive on these. It would be great if everyone could review, comment, and help out with wording. | 16:27 |
| johnsom | I thought there was a third, but I didn't see it in my quick pre-meeting search. Let me know if you see it. | 16:28 |
| johnsom | I expect both of these to need a few cycles before they are good to go. | 16:28 |
| johnsom | Notifications are also on the PTG discussion list. | 16:29 |
| johnsom | A reminder, we are not looking for perfection on these, but we want to make sure: | 16:29 |
| johnsom | The use case is clear | 16:29 |
| johnsom | We have designed any problem areas | 16:29 |
| johnsom | Call out API changes required | 16:30 |
| johnsom | Call out database schema changes/designs | 16:30 |
| johnsom | Has a reasonably clear description of the change to the code base (i.e. where hooks for notifications would be placed, etc.) | 16:30 |
| johnsom | We want to discuss anything that once merged we would be stuck with type design choices... | 16:31 |
| johnsom | Any questions or comments on the specs? | 16:32 |
| johnsom | #topic Open Discussion | 16:34 |
| cgoncalves | I agree with your suggestions for the RFEs | 16:34 |
| *** openstack changes topic to "Open Discussion (Meeting topic: Octavia)" | 16:34 | |
| johnsom | Any other topics today? | 16:34 |
| johnsom | ataraday_ Did we cover the grenade job issue or is there more we should discuss there? | 16:35 |
| ataraday_ | did something changed in this job recently? | 16:37 |
| ataraday_ | as it was successful before.. | 16:37 |
| johnsom | I think the switch to zuul v3 based grenade merged recently (or it's still an open review) | 16:38 |
| haleyb | https://review.opendev.org/#/c/726741/ | 16:38 |
| haleyb | but it's failing too | 16:39 |
| cgoncalves | __init__() got an unexpected keyword argument 'tls_ciphers' | 16:40 |
| johnsom | #link https://review.opendev.org/#/c/725098/ | 16:40 |
| cgoncalves | #link https://zuul.opendev.org/t/openstack/build/8c36ad31ce234aa7a2251566ddbf6e02/log/controller/logs/screen-o-api.txt#1787 | 16:41 |
| haleyb | my link was to the stable/ussuri one, which is failing in the same way | 16:42 |
| cgoncalves | I see /opt/stack/old/octavia-lib/ in the trace. wondering if octavia-lib is not being upgraded, it should have been at that point | 16:42 |
| johnsom | There might have been a bug here: | 16:42 |
| johnsom | #link https://review.opendev.org/#/c/717154/ | 16:42 |
| johnsom | Oh, that is interesting, yeah, if the transition doesn't update that module it would have issues | 16:43 |
| cgoncalves | I can take point in this | 16:45 |
| cgoncalves | may be a legit issue in our grenade job actually, so my bad | 16:45 |
| johnsom | cgoncalves Thank you! | 16:45 |
| johnsom | Any other topics today? | 16:46 |
| johnsom | Ok, thank you for another great meeting. Good discussions. Have a great week! | 16:47 |
| johnsom | #endmeeting | 16:48 |
| *** openstack changes topic to "Discussions for OpenStack Octavia | Priority bug review list: https://etherpad.openstack.org/p/octavia-priority-reviews" | 16:48 | |
| openstack | Meeting ended Wed May 20 16:48:04 2020 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 16:48 |
| openstack | Minutes: http://eavesdrop.openstack.org/meetings/octavia/2020/octavia.2020-05-20-16.00.html | 16:48 |
| openstack | Minutes (text): http://eavesdrop.openstack.org/meetings/octavia/2020/octavia.2020-05-20-16.00.txt | 16:48 |
| openstack | Log: http://eavesdrop.openstack.org/meetings/octavia/2020/octavia.2020-05-20-16.00.log.html | 16:48 |
| cgoncalves | thank you | 16:48 |
| ataraday_ | thank you! | 16:48 |
| *** gcheresh has joined #openstack-lbaas | 16:49 | |
| *** sapd1_x has quit IRC | 16:52 | |
| johnsom | cgoncalves FYI, if it is the octavia-lib not getting updated, we probably should add that as a step in our upgrade guide as well. | 16:56 |
| cgoncalves | johnsom, good idea. it is odd if we have to manually upgrade octavia-lib in the grenade job. when upgrading octavia, grenade should upgrade libs in requirements.txt too | 16:59 |
| johnsom | Yeah, I would expect so as well | 16:59 |
| *** ataraday_ has quit IRC | 17:02 | |
| *** sapd1_x has joined #openstack-lbaas | 17:04 | |
| *** gcheresh has quit IRC | 17:12 | |
| *** ccamposr__ has joined #openstack-lbaas | 17:28 | |
| *** ccamposr has quit IRC | 17:31 | |
| *** gcheresh has joined #openstack-lbaas | 18:04 | |
| *** sapd1_x has quit IRC | 18:26 | |
| haleyb | johnsom: so i've been looking at ways the tls_ciphers change differed from a previous one adding tls_enabled and had a question | 18:37 |
| haleyb | in https://review.opendev.org/#/c/717154/3/octavia/api/v2/types/pool.py should the new value have been wtypes.wsattr(wtypes.StringType()) | 18:38 |
| haleyb | like all the others above ie | 18:38 |
| johnsom | haleyb This is something people do oddly when using wsme. So, technically you only need wsattr if you are augmenting the type, like adding a default value (we don't do that in the API type usually). That said, there is no error if you wrap a type in an wsattr without an augmentation. | 18:47 |
| johnsom | For example, on string type it could be a default, or an enum list of valid values, etc. then you would use wsattr. | 18:48 |
| haleyb | it just seemed odd to be different, even if it works | 18:50 |
| haleyb | also noticed there was no unicode u'tls_cipers' in the alembic migration, again like previous ones | 18:50 |
| johnsom | Oh, agreed. All of those type files are all over the board. It's low priority tech debt to go clean those all up. I just haven't got around to it | 18:50 |
| haleyb | none explain the __init__ failure of course | 18:51 |
| johnsom | Yeah, I don't think the u' matter for the column names. We have a DB default for unicode/utf-8 | 18:52 |
| haleyb | my nitpicking brain notices | 18:53 |
| johnsom | Missed your chance..... lol | 18:53 |
| haleyb | the grenade failure is interesting, and not obvious at first glance, but at least there's a good backtrace | 18:54 |
| haleyb | https://zuul.opendev.org/t/openstack/build/ab745463ec4e4896a8eead3b82c2c285/log/controller/logs/screen-o-api.txt | 18:54 |
| johnsom | Yeah, show create table strips the unicode designation. It's this part that matters: "ENGINE=InnoDB DEFAULT CHARSET=utf8" | 18:55 |
| johnsom | I thought cgoncalves took point on that. Did you guys trade or are two people looking at the same thing? | 18:55 |
| haleyb | oh, it's the "old" octavia lib but the new dict | 18:55 |
| haleyb | yeah, he did, i just clicked on the grenade log since it failed for me in another change | 18:56 |
| johnsom | Yeah, pretty sure Carlos said he thinks the octavia-lib isn't getting upgraded with the octavia module upgrade. | 18:56 |
| haleyb | code /opt/stack/old/octavia-lib/octavia_lib/api/drivers/data_models.py | 18:57 |
| haleyb | anyways, was just poking | 18:57 |
| *** gcheresh has quit IRC | 18:58 | |
| haleyb | the octavia code is different enough from neutron i'm trying to understand it | 18:58 |
| johnsom | I know, it's soooo much better.... lol | 18:59 |
| haleyb | riiiight :) | 19:00 |
| johnsom | There is a little sticker in the corner, "The legacy free advantage!" | 19:00 |
| haleyb | so i can just rm -rf */v1 stuff? :) | 19:01 |
| johnsom | We did it once already: https://review.opendev.org/656707 | 19:02 |
| johnsom | It will happen again | 19:03 |
| haleyb | just add a v3 to really confuse people | 19:05 |
| johnsom | We have seriously talked about it. v2 is still encumbered by some neutron cruft. | 19:05 |
| johnsom | grin | 19:06 |
| openstackgerrit | Andreas Jaeger proposed openstack/python-octaviaclient master: Switch to newer openstackdocstheme and reno versions https://review.opendev.org/729027 | 19:06 |
| *** gcheresh has joined #openstack-lbaas | 19:19 | |
| *** gcheresh has quit IRC | 19:41 | |
| *** gcheresh has joined #openstack-lbaas | 20:13 | |
| *** gcheresh has quit IRC | 20:20 | |
| *** maciejjozefczyk has quit IRC | 20:30 | |
| *** vishalmanchanda has quit IRC | 20:37 | |
| *** wuchunyang has quit IRC | 21:21 | |
| rm_work | oh gthiemonge yeah i ran into that same issue with centos nmap being broken for our UDP checks. I fixed it by NOT USING UDP TYPE CHECKS BECAUSE THEY ARE REALLY BAD | 21:57 |
| rm_work | :D | 21:57 |
| rm_work | but I will check your change | 21:57 |
| rm_work | also the behavior is totally different besides that | 21:58 |
| rm_work | it just *doesn't work* in centos | 21:58 |
| rm_work | the -z | 21:58 |
| rm_work | at least in my testing | 21:59 |
| rm_work | something about how the packet is constructed/sent is different between centos version and, for example, ubuntu version | 21:59 |
| rm_work | oh wait no, I was testing with a static version | 22:03 |
| rm_work | i mean, centos version was still broken, but the behavior of the packets was different between centos (not working) and static (working) | 22:04 |
| johnsom | rm_work We found this recently: https://github.com/haproxy/haproxy/issues/644 | 22:04 |
| johnsom | Carlos is going to cook a fix | 22:05 |
| rm_work | yeah i see how that could happen | 22:05 |
| rm_work | or rather, i see how it could be a problem | 22:05 |
| rm_work | but how can a peer name start with a hyphen? | 22:05 |
| johnsom | base64 encoding using the URL variant | 22:06 |
| rm_work | ah base64 yeah ok | 22:06 |
| rm_work | can it be ... quoted? | 22:07 |
| rm_work | <_< | 22:07 |
| johnsom | Worst case, HM kills it and rebuilds it with a different UUID, so self heals, but.... | 22:07 |
| johnsom | Yeah, we gave Carlos a few ideas for potential ways to fix it | 22:07 |
| johnsom | work around it I should say. It looks like upstream will probably fix as well since they have already identified the problem code (less than an hour response, pretty nice). | 22:08 |
| openstackgerrit | Brian Haley proposed openstack/octavia master: Remove all deprecated driver code that moved to octavia-lib https://review.opendev.org/701473 | 22:09 |
| rm_work | yeah, very nice | 22:13 |
| openstackgerrit | Brian Haley proposed openstack/octavia master: Remove all deprecated driver code that moved to octavia-lib https://review.opendev.org/701473 | 22:20 |
| *** TrevorV has quit IRC | 22:35 | |
| *** tkajinam has joined #openstack-lbaas | 22:37 | |
| *** rcernin has joined #openstack-lbaas | 23:08 | |
| *** threestrands has joined #openstack-lbaas | 23:49 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!