Tuesday, 2021-04-27

*** luksky has quit IRC00:08
*** servagem has quit IRC00:16
*** servagem has joined #openstack-lbaas00:17
*** sapd1 has quit IRC00:37
*** zzzeek has quit IRC00:38
*** zzzeek has joined #openstack-lbaas00:39
*** ramishra has quit IRC01:14
*** sapd1 has joined #openstack-lbaas01:16
*** ramishra has joined #openstack-lbaas02:32
*** rcernin has quit IRC02:39
*** rcernin has joined #openstack-lbaas02:40
openstackgerritYuehuiLei proposed openstack/octavia master: setup.cfg: Replace dashes with underscores  https://review.opendev.org/c/openstack/octavia/+/78812403:00
*** xgerman has quit IRC03:01
*** sapd1_x has joined #openstack-lbaas04:07
*** ramishra has quit IRC04:25
*** ramishra has joined #openstack-lbaas04:25
*** vishalmanchanda has joined #openstack-lbaas04:50
*** gcheresh has joined #openstack-lbaas05:01
*** sapd1_x has quit IRC05:20
*** wolsen has joined #openstack-lbaas05:23
openstackgerritAdam Harwell proposed openstack/octavia master: Use valid_vip_subnet_tag to select subnets  https://review.opendev.org/c/openstack/octavia/+/78813305:50
*** luksky has joined #openstack-lbaas05:59
*** ramishra has quit IRC06:58
*** ramishra has joined #openstack-lbaas06:58
*** damien_r has joined #openstack-lbaas07:11
*** rpittau|afk is now known as rpittau07:45
*** rcernin has quit IRC08:08
*** rcernin has joined #openstack-lbaas08:45
*** sapd1_x has joined #openstack-lbaas09:00
*** QG has joined #openstack-lbaas09:02
*** rcernin has quit IRC09:46
openstackgerritAdam Harwell proposed openstack/octavia master: Use valid_vip_subnet_tag to select subnets  https://review.opendev.org/c/openstack/octavia/+/78813309:55
rm_workjohnsom: modified the idea we talked about slightly -- allowing it to be overridden by the flavor is a lot better even... I think this should be acceptable? needs testing, but I will wait to do that until we have some basic consensus on whether this approach is good10:01
rm_workcgoncalves / gthiemonge ^^ would like early feedback on whether you think this feature will be acceptable to merge, added a use-case example in the commit message, but I can also talk to it further10:01
*** rcernin has joined #openstack-lbaas10:04
*** rcernin has quit IRC10:18
*** rcernin has joined #openstack-lbaas10:50
*** sapd1_x has quit IRC11:46
*** damien_r has quit IRC12:12
*** damien_r has joined #openstack-lbaas12:16
*** tkajinam has quit IRC12:18
*** tkajinam has joined #openstack-lbaas12:18
*** damien_r has quit IRC12:22
*** tkajinam has quit IRC12:24
*** tkajinam has joined #openstack-lbaas12:25
*** damien_r has joined #openstack-lbaas12:28
*** rcernin has quit IRC12:41
*** damien_r has quit IRC12:52
*** damien_r has joined #openstack-lbaas13:03
*** tkajinam has quit IRC13:27
*** tkajinam has joined #openstack-lbaas13:27
*** sapd1_x has joined #openstack-lbaas13:30
openstackgerritVishal Manchanda proposed openstack/octavia-dashboard master: Add horizon-nodejs14-job template.  https://review.opendev.org/c/openstack/octavia-dashboard/+/78829513:36
openstackgerritGregory Thiemonge proposed openstack/octavia master: Fix task_flow.max_workers with persistence in amphorav2  https://review.opendev.org/c/openstack/octavia/+/78830714:10
*** njohnston has joined #openstack-lbaas14:11
*** ccamposr has quit IRC14:32
*** ccamposr has joined #openstack-lbaas14:33
openstackgerritMohammed Naser proposed openstack/octavia stable/train: Fix empty Batch Member Update to unlock objects  https://review.opendev.org/c/openstack/octavia/+/78801514:38
zigorm_work: johnsom: I'm getting this in /var/log/syslog of my amphoaes, any idea why ? http://paste.openstack.org/show/804800/15:02
*** sorrison has quit IRC15:06
johnsomzigo Yes, since you are not using the standard image build tool, I expect you are missing this: https://github.com/openstack/octavia/blob/master/elements/amphora-agent/static/etc/rsyslog.d/55-octavia-socket.conf15:26
zigojohnsom: Thanks.15:39
zigojohnsom: Looks like we don't have a working Victoria amphora yet, but having correct logs will help a lot.15:40
zigojohnsom: I've been asked why I put dkms and acpid in my Octavia agent, and that was the reason why my patch to the debian official images was rejected. Could you explain why it's in amphora-agent/package-installs.yaml ? It doesn't make sense to me either ...15:48
zigoirqbalance is also something to avoid.15:48
johnsomYeah, all of those are probably not needed in your case. They were copy/pasted over when nodepool made some changes that broke us.15:49
johnsomI agree, I would pull those15:49
zigoOh ok.15:49
johnsomJust a smudge of tech debt..... lol15:50
zigoifenslave, haveged, irqbalance, vlan too, IMO...15:50
zigoethtool ...15:50
johnsomHmm, ifenslave *might* have a strange dependency from the old ifup/down stuff.15:51
johnsomPlease leave ethtool. We have a roadmap to use that for nic IRQ cpu pinning15:51
johnsomIt's part of the "improve multi-core performance" plan15:52
johnsomhaveged, not sure on. If your nova can pass through rng, you don't need it.15:52
johnsomI think we had a user/use case that needed it so it was added.15:53
zigojohnsom: Nobody needs it with fairly recent kernels, that's the point.15:53
johnsomYeah, that rings a bell too15:53
zigoRecent kernels do have ways to gather (lower quality) entropy (if nothing better is available).15:54
zigojohnsom: Do you guys really need gawk ? Any specifics that's not in mawk ?15:54
johnsomYeah, I know there have been a lot of changes there since we added haveged.15:54
zigojohnsom: I'm asking all of this because I need the image to be lower than 256 GB, which is the limit of the artifacts in the Debian gitlab.15:55
johnsomI am not sure on that one15:55
zigoSo the more I can take away...15:55
zigo... the better.15:55
johnsomAgreed. I have some notes that some locales and shared-mime-info junk is ending up in some images, that may also be low hanging fruit15:55
zigoI wonder what ipvsadm is for as well ...15:55
johnsomWe need that for debugging and some stats  stuff for UDP/SCTP protocol support15:56
zigoOk, thanks.15:56
zigoI'll try the way it is now, at least... :)15:56
zigoRemoved: acpid, dkms, haveged, irqbalance, vlan.15:57
* zigo crosses fingers so that's enough removal.15:57
johnsomI don't know if you have options on the number of kernel modules installed, but that was a huge difference on one of the other distros. We don't need the kitchen sink driver if you know what I mean15:58
zigojohnsom: Yeah, there's a cloud kernel in Debian that helps saving space ! :)15:58
zigojohnsom: Is rm /var/lib/octavia/vrrp/check_scripts/haproxy_check_script.sh somehow generated?16:02
zigoWhat is configuring it?16:03
zigoIt "exits 1" and then nothing works.16:03
johnsomThe agent manages those as needed16:03
zigoIf we remove it, reboot the amphora, then everything goes back to working ...16:03
johnsomThere have been some bugs fixes around that over the last release or two. I wonder if you are missing those somehow16:04
johnsomhttps://review.opendev.org/c/openstack/octavia/+/753189 for example16:05
zigojohnsom: I'm rebuilding an image with the latest Victoria point release (so 17.1.0 instead of 17.0.1, if I'm not mistaking).16:05
*** QG has quit IRC16:05
johnsomThat last one is pre-victoria, so I guess doesn't apply here.16:07
zigoI do have the s/sh/bash/ thingy in 17.1.0 :)16:10
*** xgerman has joined #openstack-lbaas16:17
*** jamesden_ has joined #openstack-lbaas16:21
*** jamesdenton has quit IRC16:22
zigojohnsom: When I do, from the amphora: "ip netns exec amphora-haproxy elinks http://<ip-of-my-backend>"16:27
zigothough when I try to do "ip netns exec amphora-haproxy elinks http://<ip-of-the-vip>" it's not working, I wonder why, looks like haproxy is broken or something ...16:27
zigohaproxy runs on the amphora-haproxy netns, right?16:28
johnsomzigo We don't bring up localhost inside the netns, so you can't get to the VIP by default.16:28
johnsomJust up the lo interface16:29
zigoWell, my load balancer has an issue anyways...16:29
zigoNot enough, still doesn't work.16:29
zigoI don't get what's going on, because:16:30
zigoip netns exec amphora-haproxy netstat -anple | grep 8016:30
zigoshows that haproxy is listening on port 80 in the amphora ...16:30
johnsomHmm, that should work. You did the ip link up inside the netns right?16:30
zigodid: ip netns exec amphora-haproxy ifup lo16:31
johnsomYeah, that usually works for me.16:31
zigo# ip netns exec amphora-haproxy netstat -anple | grep 80 | head -n 116:32
zigotcp        0      0*               LISTEN      0          18080      2096/haproxy16:32
zigoSo this looks good.16:32
zigoBut then:16:32
zigo ip netns exec amphora-haproxy elinks <--- No page showing up ... :/16:32
*** rpittau is now known as rpittau|afk16:33
zigo# ip netns exec amphora-haproxy nmap -p 80
zigo80/tcp open  http16:33
johnsomAre you running active/standby? Are you on the amp that owns the IP?16:33
zigoYep, I am.16:33
zigoOh, maybe the wrong amphora ?16:33
zigoThat's possible.16:34
zigoHow can I check it owns the IP ?16:34
johnsomip a16:34
zigoI mean from the amphora ...16:34
zigoYeah, saw the IP.16:34
johnsomRight, inside the netns, ip a, the VIP address will be a secondary IP on the interface16:34
johnsomIf it is there, you are on the right amp16:35
zigoI am in the right amp, ip is showing, and lo is there by default for me.16:36
johnsomIs "lo" up though.16:37
johnsomlo will be there, just not "UP" by default16:37
zigoAh no, it's down ... :)16:38
johnsomGrin, the details in my comments... lol16:38
zigoOh, indeed, now I got a working http://<ip-of-the-vip> when I try in the amphora.16:39
zigoReloading the page changes backend, as expected ...16:39
zigoSo the issue looks like network related, and NOT Octavia related ...16:39
zigoie: we're not able to reach the port.16:39
zigoSo not your concern, I believe.16:40
johnsomNot a surprise....16:40
zigoWe'll investigate further, thanks a lot.16:40
rm_workjohnsom: want to discuss that patch we were talking about last night? :D17:58
johnsomI had it on my agenda for a bit later in the day17:58
johnsomSomething you want to cover now?17:58
* rm_work shrugs17:59
rm_worki'm here now :D but if you're busy no worries17:59
johnsomBrian gave you some comments already18:00
rm_workhad to wake up early for a staff meeting18:00
rm_workah nice18:00
rm_workhaleyb: responded! thanks for your comments18:13
*** vishalmanchanda has quit IRC18:14
* haleyb wonders what constitutes early :)18:22
openstackgerritAdam Harwell proposed openstack/octavia master: Use valid_vip_subnet_tag to select subnets  https://review.opendev.org/c/openstack/octavia/+/78813318:27
rm_workfixes applied18:27
rm_workearly == before noon PST :P18:27
rm_workor alternatively "before I would otherwise have to wake up" lol18:27
*** openstackstatus has quit IRC18:31
*** openstackstatus has joined #openstack-lbaas18:33
*** ChanServ sets mode: +v openstackstatus18:33
*** njohnston has quit IRC19:03
*** rcernin has joined #openstack-lbaas20:30
johnsomrm_work posted you some comments20:47
johnsomIn general I think the path is ok, just needs some polish, tests, docs, check in the mail, etc.20:47
johnsomJust kidding about the check part, but not the tests and docs.20:48
johnsomAlso, it looks like you really want this to merge: https://review.opendev.org/c/openstack/octavia/+/78804720:50
johnsomhint, hint20:50
*** damien_r has quit IRC21:01
*** gcheresh has quit IRC21:03
*** rcernin has quit IRC21:22
*** rcernin has joined #openstack-lbaas21:40
*** rcernin has quit IRC21:46
*** rcernin has joined #openstack-lbaas22:00
*** rcernin has quit IRC22:07
*** rcernin has joined #openstack-lbaas22:16
*** luksky has quit IRC22:34
*** rcernin has quit IRC22:47
*** rcernin has joined #openstack-lbaas22:50
*** rcernin has quit IRC22:50
*** rcernin has joined #openstack-lbaas22:51
rm_workhaleyb: how much are you stuck on the service-type vs tag thing? :D23:49
rm_workjohnsom: what should the varname be? `vip_subnet_selection_tag=`?23:49
rm_workhaleyb: is it because you think this is *exactly* the intended purpose of the service-type field? :D23:51
rm_workhaleyb: `Must be a valid device owner value for a network port` concerns me? otherwise, sure, whatever :D23:52
*** rcernin has quit IRC23:54

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!