| johnsom | rm_work: I think Carlos has a patch started. | 00:05 |
|---|---|---|
| johnsom | lxkong: even 2-3 minutes seems long. | 00:06 |
| rm_work | Kk | 00:11 |
| johnsom | rm_work https://review.opendev.org/#/c/594078/ | 00:14 |
| johnsom | It doesn't look like it addresses the devstack variable part however | 00:14 |
| johnsom | Oh, ha, no it addresses that part, but not the ipv6 lb-mgmt-net | 00:14 |
| johnsom | had it backwards | 00:14 |
| * johnsom disappears once again for the night. | 00:16 | |
| rm_work | k might poke | 00:24 |
| *** goldyfruit has quit IRC | 00:24 | |
| *** goldyfruit has joined #openstack-lbaas | 00:24 | |
| *** yamamoto has quit IRC | 00:48 | |
| *** ccamposr has joined #openstack-lbaas | 01:27 | |
| *** happyhemant has quit IRC | 01:28 | |
| *** dayou_ has quit IRC | 02:08 | |
| *** ccamposr has joined #openstack-lbaas | 02:11 | |
| *** ccamposr__ has joined #openstack-lbaas | 02:13 | |
| *** ccamposr has quit IRC | 02:13 | |
| *** ccamposr__ has quit IRC | 02:14 | |
| *** dayou_ has joined #openstack-lbaas | 02:22 | |
| *** KeithMnemonic has quit IRC | 03:08 | |
| *** ramishra has joined #openstack-lbaas | 03:43 | |
| *** ccamposr has joined #openstack-lbaas | 03:45 | |
| *** ricolin has joined #openstack-lbaas | 03:50 | |
| *** ccamposr has quit IRC | 03:50 | |
| lxkong | rm_work: are you still here? | 04:00 |
| lxkong | johnsom, rm_work, i found the amphora-agent service takes a long time before opening the port 9443 after the service is active. https://paste.api-zulu.com/ | 04:01 |
| *** ramishra has quit IRC | 04:02 | |
| *** ramishra has joined #openstack-lbaas | 04:03 | |
| johnsom | That doesn’t open for me, can you use paste.openstack.org? | 04:03 |
| lxkong | johnsom: ooh, thanks god you are still here | 04:03 |
| lxkong | johnsom: http://dpaste.com/3700R0A | 04:04 |
| johnsom | I get a blank green screen with some icons in the corner | 04:04 |
| johnsom | Ok, that one works. | 04:04 |
| johnsom | Let me read | 04:04 |
| lxkong | johnsom: output of `journalctl -u amphora-agent`: http://paste.openstack.org/show/752855/ | 04:05 |
| *** yamamoto has joined #openstack-lbaas | 04:05 | |
| lxkong | you can see there is about 5min between systemd said "Started" and amphora-agent is actually printing some log | 04:06 |
| johnsom | Yeah, that is odd. | 04:07 |
| johnsom | Can you do the systemd-analyze on that? | 04:09 |
| lxkong | sure, 1 sec | 04:09 |
| johnsom | Did you check the qemu process flags? | 04:09 |
| lxkong | johnsom: no, it's our production env, i can't easily ssh into the compute node | 04:10 |
| lxkong | johnsom: output of systemd-analyze commands: http://paste.openstack.org/show/752856/ | 04:12 |
| johnsom | See, those all look ok, ~4 seconds to the amphora agent started | 04:14 |
| johnsom | I really thing it is a problem with nova and the vm | 04:14 |
| lxkong | johnsom: do you mind providing the qemu commands you mentioned? so i can ask our ops for help | 04:15 |
| johnsom | Just do a “ps -ef | grep qemu” this will show how nova booted the VM. | 04:16 |
| lxkong | johnsom: when i restart the service, i can see the port is open very quickly, seems like the problem exists only at the first time start | 04:16 |
| johnsom | My guess is the accel flag is tcg, which is bad | 04:17 |
| johnsom | Yeah, I am pretty sure it isn’t our code, it seems like outside our stuff. | 04:20 |
| lxkong | johnsom: http://dpaste.com/282ZGWM | 04:27 |
| lxkong | finally i got permission to log into the compute node | 04:27 |
| johnsom | Ok, looking through that doesn’t raise a red flag, there are odd devices there, but nothing major. | 04:32 |
| lxkong | johnsom: i built the image on a ubuntu xenial physical machine, but i guess that shouldn't be a problem | 04:33 |
| johnsom | I wonder if the ceph is overloaded. Hang on a second | 04:33 |
| johnsom | Yeah, we do that too. Should not matter | 04:34 |
| johnsom | In the amp, can you check the /proc/pressure files and see it one of the metrics is odd? | 04:37 |
| rm_work | "lxkong:you can see there is about 5min between systemd said "Started" and amphora-agent is actually printing some log" | 04:38 |
| rm_work | that is the same thing i see, just the scale is different | 04:38 |
| rm_work | i think it might be happening for you too but just your environment is a lot faster in general so it's not obvious | 04:38 |
| rm_work | ^^ I mean johnsom's enc | 04:38 |
| rm_work | *env | 04:38 |
| *** gcheresh has joined #openstack-lbaas | 04:39 | |
| rm_work | i can show you at some point, maybe tomorrow? | 04:39 |
| rm_work | about to grab dinner | 04:39 |
| lxkong | johnsom: `/proc/pressure` doesn't exist inside the amphora | 04:40 |
| johnsom | Hmm, that is different. Are they centos images? | 04:42 |
| lxkong | no, ubuntu xenial | 04:42 |
| rm_work | let me check mine | 04:43 |
| rm_work | since you are still here | 04:43 |
| johnsom | Just about to fall asleep | 04:44 |
| lxkong | rm_work: i can be here for 10min, need to pick up the kid | 04:44 |
| lxkong | johnsom: thanks for trying to help | 04:44 |
| *** vishalmanchanda has joined #openstack-lbaas | 04:44 | |
| johnsom | Yeah, I am on 19 hours now, lol | 04:45 |
| rm_work | whelp.... | 04:46 |
| rm_work | for the record, no /proc/pressure in my ubuntu amp either | 04:47 |
| rm_work | but, also, it isn't doing it anymore O_o | 04:47 |
| rm_work | i wonder... | 04:47 |
| rm_work | i have made some amp build changes since then | 04:47 |
| rm_work | would be funny if i accidentally fixed it :D | 04:47 |
| rm_work | though i doubt they'd be relevant pre-agent-startup | 04:47 |
| rm_work | now it's about 30s from when it switches to the connect task, to when it successfully connects, much faster than before <_< | 04:49 |
| rm_work | but i've rebuilt my stack since then too so really hard to say | 04:49 |
| *** vishalmanchanda has quit IRC | 04:49 | |
| *** vishalmanchanda has joined #openstack-lbaas | 04:49 | |
| *** pcaruana|afk| has joined #openstack-lbaas | 05:01 | |
| *** pcaruana|afk| has quit IRC | 05:04 | |
| *** pcaruana has joined #openstack-lbaas | 05:04 | |
| *** ivve has joined #openstack-lbaas | 05:10 | |
| *** ricolin has quit IRC | 05:36 | |
| *** takamatsu has joined #openstack-lbaas | 05:42 | |
| *** luksky has joined #openstack-lbaas | 05:45 | |
| *** ltomasbo has left #openstack-lbaas | 06:05 | |
| *** ccamposr has joined #openstack-lbaas | 06:08 | |
| *** ccamposr has quit IRC | 06:13 | |
| *** ccamposr has joined #openstack-lbaas | 06:16 | |
| *** ccamposr__ has joined #openstack-lbaas | 06:26 | |
| *** ccamposr has quit IRC | 06:28 | |
| *** rpittau|afk is now known as rpittau | 07:00 | |
| *** ianychoi has joined #openstack-lbaas | 07:02 | |
| *** vishalmanchanda has quit IRC | 07:14 | |
| *** tesseract has joined #openstack-lbaas | 07:21 | |
| *** ataraday has quit IRC | 07:22 | |
| *** trident has quit IRC | 07:31 | |
| *** takamatsu has quit IRC | 07:32 | |
| *** trident has joined #openstack-lbaas | 07:34 | |
| *** lemko has joined #openstack-lbaas | 07:58 | |
| openstackgerrit | Carlos Goncalves proposed openstack/octavia master: DNM: systemd-219-64.el7 https://review.opendev.org/665089 | 08:00 |
| openstackgerrit | Carlos Goncalves proposed openstack/octavia master: DNM: systemd-219-64.el7 https://review.opendev.org/665089 | 08:07 |
| *** takamatsu has joined #openstack-lbaas | 08:19 | |
| *** rcernin has quit IRC | 08:23 | |
| *** yboaron_ has joined #openstack-lbaas | 08:37 | |
| *** luksky has quit IRC | 08:39 | |
| rm_work | cgoncalves: so do you mind if i poke at that ipv6-gate patch you started? | 09:10 |
| openstackgerrit | Adam Harwell proposed openstack/octavia-tempest-plugin master: WIP: Test Additional VIPs https://review.opendev.org/664462 | 09:11 |
| cgoncalves | rm_work, sure! | 09:15 |
| openstackgerrit | Adam Harwell proposed openstack/octavia-tempest-plugin master: WIP: Test Additional VIPs https://review.opendev.org/664462 | 09:21 |
| rm_work | just took inspiration from the one you did and deleted all the tests i didn't care about from the projects.yaml :D | 09:21 |
| rm_work | while i'm in test-development mode | 09:21 |
| *** luksky has joined #openstack-lbaas | 09:22 | |
| openstackgerrit | Merged openstack/octavia-dashboard master: Remove reference to Neutron LBaaS dashboard https://review.opendev.org/663042 | 09:23 |
| openstackgerrit | Adam Harwell proposed openstack/octavia-tempest-plugin master: WIP: Test Additional VIPs https://review.opendev.org/664462 | 09:28 |
| *** jomeier has joined #openstack-lbaas | 09:34 | |
| jomeier | Hi. I managed to install Octavia on OpenStack 'Stein'. I can load balance to two backend VMs (without health api endpoints!). The load balancer never goes to 'online' status. Its status is 'offline'. But it seems to do its job. | 09:36 |
| jomeier | Whats missing? | 09:36 |
| openstackgerrit | Carlos Goncalves proposed openstack/octavia-lib master: Add allowed_cidrs to Listener data model https://review.opendev.org/659625 | 09:43 |
| openstackgerrit | Merged openstack/neutron-lbaas stable/stein: remove release notes job https://review.opendev.org/664874 | 10:05 |
| *** yamamoto has quit IRC | 10:09 | |
| openstackgerrit | Adam Harwell proposed openstack/octavia master: WIP: Allow IPv6 health network in devstack https://review.opendev.org/665103 | 10:12 |
| openstackgerrit | Adam Harwell proposed openstack/octavia-tempest-plugin master: Add octavia-v2-dsvm-scenario-ipv6 to check queue https://review.opendev.org/594078 | 10:14 |
| rm_work | jomeier: so, it will "work" for now as long as everything is good, but two things | 10:18 |
| rm_work | first, as you noticed, the operating_status will never update, so you won't get ONLINE ever | 10:18 |
| rm_work | second, individual member statuses won't update in the API (to ONLINE/OFFLINE) | 10:19 |
| rm_work | third, you won't get any statistics | 10:19 |
| rm_work | fourth (and probably most important) you won't really be very Highly Available, because in the case of an issue with the amphora, the failover won't happen automatically | 10:19 |
| rm_work | sorry, more than two things :D | 10:20 |
| rm_work | you REALLY want the health managers to work :D | 10:23 |
| cgoncalves | jomeier, make sure heartbeat messages are not being dropped on your controller | 10:24 |
| cgoncalves | UDP packets on port 5555 on interface o-hm0 (defaults) | 10:24 |
| rm_work | i mean, they said "without health api endpoints" | 10:24 |
| rm_work | i assume it isn't even configured/started | 10:25 |
| jomeier | Thanks guys. So you mean the offline status comes from my machines dont have an health api endpoint and everything is as expected? | 10:34 |
| cgoncalves | jomeier, if by health api endpoint you mean the octavia health manager service is not running, then yes | 10:38 |
| jomeier | @cgoncalves: It is running and its not complaining about anything in /var/log/octavia/health-manager.log | 10:41 |
| cgoncalves | jomeier, not complaining is expected | 10:43 |
| cgoncalves | jomeier, do a packet capture on the o-hm0 network interface, and check if UDP:5555 packets are being received | 10:43 |
| *** yamamoto has joined #openstack-lbaas | 10:45 | |
| jomeier | @cgoncalves: With tcpdump? I dont see anything but ARP packages even if I ping the IP address of o-hm0 interface. | 10:47 |
| openstackgerrit | Adam Harwell proposed openstack/octavia-tempest-plugin master: WIP: Test Additional VIPs https://review.opendev.org/664462 | 10:47 |
| *** yamamoto has quit IRC | 10:59 | |
| *** yamamoto has joined #openstack-lbaas | 10:59 | |
| rm_work | remember ping is blocked by neutron without explicit security groups :D | 11:14 |
| openstackgerrit | Adam Harwell proposed openstack/octavia master: WIP: Allow IPv6 health network in devstack https://review.opendev.org/665103 | 11:22 |
| *** yamamoto has quit IRC | 11:24 | |
| *** yamamoto has joined #openstack-lbaas | 11:28 | |
| openstackgerrit | Adam Harwell proposed openstack/octavia-tempest-plugin master: WIP: Test Additional VIPs https://review.opendev.org/664462 | 11:38 |
| *** yamamoto has quit IRC | 11:40 | |
| *** yamamoto has joined #openstack-lbaas | 11:43 | |
| *** yamamoto has quit IRC | 11:43 | |
| *** yamamoto has joined #openstack-lbaas | 11:43 | |
| *** yamamoto has quit IRC | 11:48 | |
| *** yamamoto has joined #openstack-lbaas | 11:53 | |
| *** yamamoto has quit IRC | 11:57 | |
| *** boden has joined #openstack-lbaas | 12:17 | |
| openstackgerrit | boden proposed openstack/neutron-lbaas stable/stein: Support URL query params in healthmonitor url_path https://review.opendev.org/660930 | 12:20 |
| *** yamamoto has joined #openstack-lbaas | 12:21 | |
| openstackgerrit | boden proposed openstack/neutron-lbaas stable/stein: Support URL query params in healthmonitor url_path https://review.opendev.org/660930 | 12:22 |
| *** goldyfruit has quit IRC | 12:24 | |
| *** ccamposr__ has quit IRC | 12:34 | |
| openstackgerrit | Carlos Goncalves proposed openstack/octavia master: DNM: systemd-219-64.el7 https://review.opendev.org/665089 | 12:35 |
| *** ccamposr has joined #openstack-lbaas | 12:37 | |
| *** yamamoto has quit IRC | 12:40 | |
| *** yamamoto has joined #openstack-lbaas | 12:59 | |
| *** pcaruana has quit IRC | 13:01 | |
| *** yamamoto has quit IRC | 13:08 | |
| *** yamamoto has joined #openstack-lbaas | 13:09 | |
| *** yamamoto has quit IRC | 13:09 | |
| openstackgerrit | Adam Harwell proposed openstack/octavia-tempest-plugin master: WIP: Test Additional VIPs https://review.opendev.org/664462 | 13:18 |
| *** goldyfruit has joined #openstack-lbaas | 13:22 | |
| *** gcheresh has quit IRC | 13:24 | |
| *** ccamposr__ has joined #openstack-lbaas | 13:30 | |
| *** ccamposr has quit IRC | 13:32 | |
| *** gcheresh has joined #openstack-lbaas | 13:34 | |
| *** yamamoto has joined #openstack-lbaas | 13:37 | |
| *** yamamoto has quit IRC | 13:42 | |
| *** ricolin has joined #openstack-lbaas | 13:45 | |
| openstackgerrit | Adam Harwell proposed openstack/octavia-tempest-plugin master: Add octavia-v2-dsvm-scenario-ipv6 to check queue https://review.opendev.org/594078 | 13:45 |
| *** goldyfruit has quit IRC | 13:46 | |
| *** goldyfruit has joined #openstack-lbaas | 13:46 | |
| *** ricolin has quit IRC | 14:21 | |
| *** yamamoto has joined #openstack-lbaas | 14:23 | |
| *** luksky has quit IRC | 14:29 | |
| *** gcheresh has quit IRC | 14:35 | |
| openstackgerrit | Adam Harwell proposed openstack/octavia-tempest-plugin master: WIP: Test Additional VIPs https://review.opendev.org/664462 | 14:40 |
| * rm_work sleeps | 14:41 | |
| *** yboaron_ has quit IRC | 15:00 | |
| *** Vorrtex has joined #openstack-lbaas | 15:03 | |
| *** yamamoto has quit IRC | 15:35 | |
| *** ivve has quit IRC | 15:40 | |
| openstackgerrit | Michael Johnson proposed openstack/octavia master: DNM: experimental element for centos debug https://review.opendev.org/665032 | 15:40 |
| *** Vorrtex has quit IRC | 15:50 | |
| *** gcheresh has joined #openstack-lbaas | 15:51 | |
| *** rpittau is now known as rpittau|afk | 16:13 | |
| *** gcheresh has quit IRC | 16:16 | |
| *** ccamposr has joined #openstack-lbaas | 16:28 | |
| *** ccamposr__ has quit IRC | 16:30 | |
| *** lemko has quit IRC | 16:47 | |
| *** omkartelee has joined #openstack-lbaas | 16:54 | |
| omkartelee | Hi, I want to augment CLI of Octavia (loadbalancer v2) to add my own commands ... is there any way i can extend it to add my commands ? | 16:55 |
| johnsom | omkartelee You can create your own openstack client plugin. See this page for information: https://docs.openstack.org/python-openstackclient/pike/contributor/plugins.html | 16:58 |
| *** pcaruana has joined #openstack-lbaas | 17:09 | |
| *** henriqueof has joined #openstack-lbaas | 17:25 | |
| *** omkartelee has quit IRC | 17:31 | |
| *** yamamoto has joined #openstack-lbaas | 17:33 | |
| *** ramishra has quit IRC | 17:44 | |
| *** gcheresh has joined #openstack-lbaas | 17:46 | |
| *** omkartelee has joined #openstack-lbaas | 17:49 | |
| omkartelee | Thanks @johnsom, I was expecting octavia client also has similar extension functionality like neutron client has ---> https://github.com/openstack/python-neutronclient/blob/master/neutronclient/common/extension.py | 17:49 |
| johnsom | omkartelee That is for the old clients. With the new openstack clients, it's pretty straight forward. | 17:53 |
| omkartelee | got it ... Thanks for quick response Johnsom | 17:54 |
| *** ivve has joined #openstack-lbaas | 17:56 | |
| *** omkartelee has quit IRC | 18:05 | |
| *** mithilarun has joined #openstack-lbaas | 18:07 | |
| *** gcheresh has quit IRC | 18:10 | |
| *** ianychoi has quit IRC | 18:13 | |
| openstackgerrit | Michael Johnson proposed openstack/octavia-tempest-plugin master: Fix the load balancer service clients https://review.opendev.org/665207 | 18:33 |
| openstackgerrit | Michael Johnson proposed openstack/octavia-tempest-plugin master: Fix the load balancer service clients https://review.opendev.org/665207 | 18:36 |
| johnsom | ^^^ The OSA folks noted that our service client wasn't picking up some of the tempest settings such as ca_certs. We missed the service_client_config call to tempest to get those base params. | 18:37 |
| *** yamamoto has quit IRC | 18:52 | |
| *** mithilarun has quit IRC | 18:59 | |
| *** mithilarun has joined #openstack-lbaas | 19:00 | |
| *** mithilarun has quit IRC | 19:04 | |
| openstackgerrit | Michael Johnson proposed openstack/octavia master: Align logging in the amphora https://review.opendev.org/664123 | 19:15 |
| *** gcheresh has joined #openstack-lbaas | 19:16 | |
| *** takamatsu has quit IRC | 19:32 | |
| *** mithilarun has joined #openstack-lbaas | 19:33 | |
| *** mithilarun has quit IRC | 19:37 | |
| openstackgerrit | Michael Johnson proposed openstack/octavia master: Align logging in the amphora https://review.opendev.org/664123 | 19:42 |
| *** mithilarun has joined #openstack-lbaas | 20:03 | |
| *** gcheresh has quit IRC | 20:13 | |
| *** pcaruana has quit IRC | 20:36 | |
| *** yamamoto has joined #openstack-lbaas | 20:47 | |
| *** takamatsu has joined #openstack-lbaas | 21:13 | |
| *** luksky has joined #openstack-lbaas | 21:17 | |
| johnsom | FYI, the RAX gate hosts have a broken mirror, so devstack isn't completing on jobs. | 21:34 |
| johnsom | I guess infra was experimenting with kafs and it's broken. No ETA when the gates will be fixed. | 21:36 |
| *** luksky has quit IRC | 21:43 | |
| *** mithilarun has quit IRC | 21:49 | |
| *** mithilarun has joined #openstack-lbaas | 21:50 | |
| *** boden has quit IRC | 21:53 | |
| *** mithilar_ has joined #openstack-lbaas | 21:53 | |
| *** mithilarun has quit IRC | 21:54 | |
| openstackgerrit | Adam Harwell proposed openstack/octavia master: Allow multiple VIPs per LB https://review.opendev.org/660239 | 22:04 |
| openstackgerrit | Adam Harwell proposed openstack/octavia-tempest-plugin master: WIP: Test Additional VIPs https://review.opendev.org/664462 | 22:05 |
| openstackgerrit | Adam Harwell proposed openstack/octavia-tempest-plugin master: Add octavia-v2-dsvm-scenario-ipv6 to check queue https://review.opendev.org/594078 | 22:19 |
| *** takamatsu has quit IRC | 22:25 | |
| *** yamamoto has quit IRC | 22:29 | |
| *** mithilar_ has quit IRC | 22:38 | |
| *** mithilarun has joined #openstack-lbaas | 22:39 | |
| *** mithilarun has quit IRC | 22:43 | |
| *** yamamoto has joined #openstack-lbaas | 22:44 | |
| *** rcernin has joined #openstack-lbaas | 22:45 | |
| *** tesseract has quit IRC | 22:48 | |
| *** yamamoto has quit IRC | 22:50 | |
| *** goldyfruit has quit IRC | 22:54 | |
| *** mithilarun has joined #openstack-lbaas | 22:56 | |
| *** ivve has quit IRC | 22:57 | |
| lxkong | johnsom: another weird thing i found is, when i switched to use fedora image, the vm bootstrap is stuck https://termbin.com/6g4l | 23:07 |
| lxkong | johnsom: wouldn't you happen to know something about that, would you? | 23:07 |
| johnsom | lxkong We are seeing something like that in our gates. Carlos and I have been trying to reproduce it. What is your stack that reproduces that? | 23:07 |
| lxkong | johnsom: stack? | 23:08 |
| johnsom | lxkong Host OS, instance OS, hypervisor, and image OS | 23:08 |
| johnsom | Here is the gate log where we see the same: http://logs.openstack.org/06/626406/9/check/octavia-v2-dsvm-scenario-centos-7/218826a/controller/logs/octavia-amphora_log.txt.gz | 23:09 |
| rm_work | johnsom: interesting on the service client thing <_< | 23:09 |
| *** yamamoto has joined #openstack-lbaas | 23:10 | |
| johnsom | So far we haven't been able to reproduce it locally though which has made it hard to debug. | 23:10 |
| johnsom | rm_work Yeah, an oversight. Likely on my part. | 23:10 |
| johnsom | rm_work it's not in the tempest docs at all. | 23:10 |
| lxkong | host os: Ubuntu 14.04.5 LTS, instance OS: fedora, hypervisor: kvm | 23:10 |
| rm_work | johnsom: so in tempest we use the private-network, which for ipv4 needs us to make a flip, but for ipv6... there's no ipv6 flips... but private-net isn't accessible apparently (makes sense) | 23:11 |
| *** goldyfruit has joined #openstack-lbaas | 23:11 | |
| rm_work | does that just mean we can't use private for ipv6 tests? | 23:11 |
| johnsom | lxkong and a centos amphora image? | 23:11 |
| lxkong | johnsom: the only difference with ubuntu image build is `-i fedora` | 23:11 |
| lxkong | and the size `-s 3` | 23:11 |
| johnsom | lxkong Ah, ok, so you don't have an intermediate VM, it's running right on the Ubuntu? | 23:11 |
| lxkong | yes | 23:12 |
| johnsom | Ok, that is good to know. | 23:12 |
| johnsom | This has had us puzzled for a while. Something changed in centos to cause this. It may be that it is just booting super slow and the controller is timing out too | 23:13 |
| *** yamamoto has quit IRC | 23:13 | |
| johnsom | rm_work so hold up here a sec, we *just* changed to the private ipv6 to get the multinode working | 23:13 |
| rm_work | yeah | 23:13 |
| rm_work | but | 23:14 |
| rm_work | that was on the backend | 23:14 |
| rm_work | on the FRONTEND (for the LB VIP) is what i'm talking about | 23:14 |
| rm_work | sorry, wasn't clear | 23:14 |
| lxkong | johnsom: i was supposed to use fedora instead of ubuntu image in our prod as a workaround to the issue we are suffering, but seems it's not gonna work | 23:14 |
| johnsom | For cirros webservers I thought I just put both subnets on one instance so we could configure it via the flip | 23:14 |
| rm_work | for the backend it's fine because Octavia does what it does and plugs the private member subnets | 23:14 |
| rm_work | but if i want to make a LB with an ipv6 VIP... i guess the only way is to have it be public? | 23:15 |
| johnsom | lxkong fedora is not going to improve your boot times. In fact it will be slower. | 23:15 |
| rm_work | otherwise our "check_members_balanced" won't be able to hit it | 23:15 |
| lxkong | johnsom: :-( | 23:15 |
| rm_work | unless we like... go through a bastion | 23:15 |
| johnsom | rm_work We already have IPv6 VIP tests..... | 23:15 |
| johnsom | rm_work https://github.com/openstack/octavia-tempest-plugin/blob/master/octavia_tempest_plugin/tests/scenario/v2/test_ipv6_traffic_ops.py#L113 | 23:16 |
| johnsom | lxkong On your system with the hung boot, in the qemu flags, what is the accel=? | 23:17 |
| rm_work | yeah i'm looking at those | 23:18 |
| lxkong | johnsom: accel=kvm | 23:18 |
| rm_work | i'm not... sure if they ... work correctly? | 23:18 |
| johnsom | lxkong hmmm, ok, we are seeing the problem on accel=tcg which is SUPER slow to boot | 23:19 |
| johnsom | rm_work They work. | 23:19 |
| johnsom | rm_work yee of little faith... I just created those last year and fixed all of that stuff. | 23:20 |
| rm_work | hmmm | 23:20 |
| johnsom | rm_work http://logs.openstack.org/01/660201/2/check/octavia-v2-dsvm-scenario/0256d13/controller/logs/tempest_log.txt.gz#_2019-06-04_19_45_22_809 | 23:23 |
| lxkong | johnsom: the fedora vm finally up and running... | 23:23 |
| lxkong | johnsom: but wait for a long time after `Started Builds and install new kernel modules through DKMS.` | 23:23 |
| johnsom | Yeah, ok, so that was my wonder, if it's just super slow. | 23:23 |
| lxkong | http://dpaste.com/07GF644 | 23:23 |
| johnsom | lxkong Can you provide the systemd-analyze for it? | 23:24 |
| johnsom | Ah, a clue. that makecache | 23:24 |
| johnsom | What was the timestamp above that? | 23:24 |
| rm_work | johnsom: if i rebase / depends-on your amp-logs patch, will i magically get amp logs? :D | 23:25 |
| johnsom | You need this one: https://review.opendev.org/624835 from octavia. Then if you want it in the gates, you need the tempest patch too | 23:26 |
| rm_work | k | 23:26 |
| johnsom | I'm just about done with the docs, so hey, you could just review it! | 23:26 |
| rm_work | heh yes | 23:26 |
| rm_work | will do that while tempest runs | 23:27 |
| openstackgerrit | Michael Johnson proposed openstack/octavia master: Amphora logging https://review.opendev.org/624835 | 23:27 |
| johnsom | There that is current. I'm just not done with the docs page. | 23:28 |
| johnsom | It's mostly done | 23:28 |
| lxkong | johnsom: the `systemd-analyze` output http://dpaste.com/18VQ1N5 | 23:28 |
| johnsom | 13 minutes! ouch | 23:28 |
| johnsom | lxkong This is super helpful, thank you! | 23:29 |
| lxkong | johnsom: we are going to downgrade our Octavia | 23:29 |
| lxkong | johnsom: i've no idea about how to debugging this | 23:29 |
| lxkong | for the ubuntu image, i can't reproduce the issue on my devatack environment | 23:30 |
| lxkong | for fedora, such preformance is not acceptable | 23:30 |
| johnsom | lxkong I still don't understand why your ubuntu would be slow. Have you tried a xenial amphora image in addition to the bionic one? | 23:31 |
| lxkong | johnsom: didn't try bionic, but it's a good suggestion, will build one now | 23:31 |
| lxkong | for the last resort :-) | 23:31 |
| lxkong | johnsom: btw, if you need anything else, log. etc please let me know before i deleting everything | 23:32 |
| johnsom | lxkong If you don't mind, could you paste us the syslog file from inside that amp? That should have everything we are looking for. Oh, and also the nova console log file. | 23:33 |
| lxkong | johnsom: ook | 23:33 |
| johnsom | lxkong Sorry, but the cloud-init log from inside the amp too if you can | 23:38 |
| lxkong | johnsom: for sure | 23:38 |
| rm_work | johnsom: pretty good, one thing in plugin.sh but ... i can fix that in my ipv6 patch that i'm working on now, since I am redoing that whole section anyway | 23:40 |
| rm_work | so actually I might just switch to +2 since it doesn't really matter in your patch yet | 23:40 |
| johnsom | Well, I need to finish the docs page, so save the +@ | 23:41 |
| johnsom | +2 | 23:41 |
| rm_work | did it anyway | 23:41 |
| rm_work | oh, right, i didn't review the docs | 23:41 |
| rm_work | i was skipping that | 23:41 |
| rm_work | and i got to the end and it looked good :D | 23:42 |
| rm_work | should I bother reading the docs up to what you have now, or just wait | 23:42 |
| johnsom | Sure, it's mostly done. Just finish the 'tenant" log section | 23:42 |
| rm_work | i skipped everything doc/ | 23:42 |
| rm_work | k | 23:42 |
| johnsom | I agree with you on that filename/placement. It felt odd to me too. Plus I *improved* it's name from what was there earlier. "10-amphora.conf" | 23:44 |
| johnsom | I think the right answer is to move it to etc/rsyslog in our repo | 23:44 |
| johnsom | Much more clear | 23:45 |
| johnsom | Also don't forget the parent patch: https://review.opendev.org/#/c/664123/ | 23:49 |
| rm_work | receive precedent ok posted some stuff on the doc | 23:50 |
| rm_work | ^^ those two words especially :D | 23:50 |
| johnsom | Ha, yeah, not checked yet | 23:51 |
| rm_work | (I was using IRC to do validation of my hunch on the spelling, forgot to delete that :D) | 23:51 |
| rm_work | i'm not 100% sure why these patches are separate but whatever | 23:52 |
| rm_work | a bunch of the stuff this does *immediately* changes in the next patch | 23:52 |
| johnsom | Well, German started is, but there was more work to do in the amphora from my perspective so I did my own patch for that. | 23:53 |
| rm_work | commented on that too | 23:55 |
| johnsom | Thank you | 23:55 |
| rm_work | i would have merged them IMO, but it's fine I guess | 23:55 |
| rm_work | just one small thing | 23:55 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!