| *** sshank has quit IRC | 00:02 | |
| *** AlexeyAbashkin has joined #openstack-lbaas | 00:18 | |
| *** AlexeyAbashkin has quit IRC | 00:22 | |
| *** yamamoto has joined #openstack-lbaas | 00:43 | |
| *** yamamoto has quit IRC | 00:48 | |
| *** fnaval has quit IRC | 00:53 | |
| *** fnaval has joined #openstack-lbaas | 00:56 | |
| *** harlowja has quit IRC | 01:07 | |
| *** AlexeyAbashkin has joined #openstack-lbaas | 01:18 | |
| *** AlexeyAbashkin has quit IRC | 01:22 | |
| *** yamamoto has joined #openstack-lbaas | 01:45 | |
| *** yamamoto has quit IRC | 01:51 | |
| *** jaff_cheng has joined #openstack-lbaas | 02:08 | |
| *** AlexeyAbashkin has joined #openstack-lbaas | 02:18 | |
| *** atoth has quit IRC | 02:20 | |
| *** AlexeyAbashkin has quit IRC | 02:23 | |
| *** yamamoto has joined #openstack-lbaas | 02:37 | |
| *** dayou has quit IRC | 02:42 | |
| *** dayou has joined #openstack-lbaas | 02:44 | |
| openstackgerrit | huangshan proposed openstack/python-octaviaclient master: Add loadbalancer status show client api and osc https://review.openstack.org/542715 | 02:47 |
|---|---|---|
| *** yamamoto has quit IRC | 02:55 | |
| *** yamamoto has joined #openstack-lbaas | 03:45 | |
| *** links has joined #openstack-lbaas | 04:00 | |
| *** links has quit IRC | 04:00 | |
| *** yamamoto has quit IRC | 04:07 | |
| *** yamamoto has joined #openstack-lbaas | 04:08 | |
| *** yamamoto has quit IRC | 04:33 | |
| *** harlowja has joined #openstack-lbaas | 04:39 | |
| *** ivve has quit IRC | 04:48 | |
| *** harlowja has quit IRC | 04:56 | |
| *** imacdonn has quit IRC | 05:14 | |
| *** imacdonn has joined #openstack-lbaas | 05:14 | |
| *** yamamoto has joined #openstack-lbaas | 06:28 | |
| *** aojea has joined #openstack-lbaas | 06:57 | |
| *** logan- has quit IRC | 06:58 | |
| *** logan- has joined #openstack-lbaas | 06:58 | |
| *** kobis has joined #openstack-lbaas | 07:00 | |
| *** aojea has quit IRC | 07:13 | |
| *** aojea has joined #openstack-lbaas | 07:14 | |
| *** kobis has quit IRC | 07:20 | |
| *** rcernin has quit IRC | 07:21 | |
| *** voelzmo has joined #openstack-lbaas | 07:21 | |
| *** aojea has quit IRC | 07:28 | |
| *** yamamoto has quit IRC | 07:32 | |
| *** AlexeyAbashkin has joined #openstack-lbaas | 07:55 | |
| *** kobis has joined #openstack-lbaas | 07:55 | |
| *** yamamoto has joined #openstack-lbaas | 08:00 | |
| *** yamamoto has quit IRC | 08:03 | |
| *** yamamoto has joined #openstack-lbaas | 08:13 | |
| *** yamamoto has quit IRC | 08:16 | |
| *** yamamoto has joined #openstack-lbaas | 08:16 | |
| *** aojea_ has joined #openstack-lbaas | 08:20 | |
| *** aojea_ has quit IRC | 08:25 | |
| *** tesseract has joined #openstack-lbaas | 08:31 | |
| *** velizarx has joined #openstack-lbaas | 08:48 | |
| *** tesseract has quit IRC | 08:51 | |
| *** tesseract has joined #openstack-lbaas | 08:52 | |
| *** tesseract has quit IRC | 08:54 | |
| *** tesseract has joined #openstack-lbaas | 08:57 | |
| *** yamamoto has quit IRC | 09:42 | |
| *** yamamoto has joined #openstack-lbaas | 09:43 | |
| *** yamamoto has quit IRC | 09:48 | |
| *** yamamoto has joined #openstack-lbaas | 09:48 | |
| *** yamamoto has quit IRC | 09:48 | |
| *** voelzmo has quit IRC | 09:54 | |
| *** voelzmo has joined #openstack-lbaas | 09:55 | |
| *** voelzmo has quit IRC | 09:55 | |
| *** voelzmo has joined #openstack-lbaas | 09:56 | |
| *** salmankhan has joined #openstack-lbaas | 09:59 | |
| *** voelzmo has quit IRC | 10:00 | |
| *** jaff_cheng has quit IRC | 10:02 | |
| *** aojea_ has joined #openstack-lbaas | 10:08 | |
| *** salmankhan1 has joined #openstack-lbaas | 10:11 | |
| *** salmankhan has quit IRC | 10:11 | |
| *** salmankhan1 is now known as salmankhan | 10:11 | |
| *** aojea_ has quit IRC | 10:14 | |
| *** salmankhan has quit IRC | 10:30 | |
| *** salmankhan has joined #openstack-lbaas | 10:34 | |
| *** sapd_ has quit IRC | 10:38 | |
| *** sapd has joined #openstack-lbaas | 10:39 | |
| *** yamamoto has joined #openstack-lbaas | 10:48 | |
| *** yamamoto has quit IRC | 10:54 | |
| *** yamamoto has joined #openstack-lbaas | 10:56 | |
| *** yamamoto has quit IRC | 11:01 | |
| *** yamamoto has joined #openstack-lbaas | 11:02 | |
| *** voelzmo has joined #openstack-lbaas | 11:04 | |
| *** numans is now known as numans_afk | 11:05 | |
| *** yamamoto has quit IRC | 11:06 | |
| *** fnaval has quit IRC | 11:09 | |
| *** yamamoto has joined #openstack-lbaas | 11:16 | |
| *** yamamoto has quit IRC | 11:16 | |
| *** numans_afk is now known as numans | 11:29 | |
| *** voelzmo has quit IRC | 11:32 | |
| *** yamamoto has joined #openstack-lbaas | 11:48 | |
| *** yamamoto has quit IRC | 11:52 | |
| *** kobis has quit IRC | 11:56 | |
| *** aojea_ has joined #openstack-lbaas | 11:57 | |
| *** kobis has joined #openstack-lbaas | 11:59 | |
| *** kobis has quit IRC | 11:59 | |
| *** kobis has joined #openstack-lbaas | 12:00 | |
| *** toker_ has quit IRC | 12:00 | |
| *** toker_ has joined #openstack-lbaas | 12:00 | |
| *** aojea_ has quit IRC | 12:01 | |
| *** yamamoto has joined #openstack-lbaas | 12:03 | |
| *** atoth has joined #openstack-lbaas | 12:04 | |
| *** yamamoto has quit IRC | 12:08 | |
| toker_ | Hi there (again)! I'm almost satisfied with my octavia setup. I got everything working except the "health-manager". I can see the amphora sending udp packages to the server where the health-manager is listening (on the correct port). But still doing a openstack show lb1 shows operating_status OFFLINE | 12:09 |
| *** velizarx has quit IRC | 12:13 | |
| nmagnezi_ | toker_, hi there! | 12:15 |
| nmagnezi_ | toker_, what do the logs have to say? | 12:15 |
| *** yamamoto has joined #openstack-lbaas | 12:18 | |
| *** yamamoto has quit IRC | 12:22 | |
| *** yamamoto has joined #openstack-lbaas | 12:33 | |
| *** yamamoto has quit IRC | 12:38 | |
| *** yamamoto has joined #openstack-lbaas | 12:48 | |
| toker_ | nmagnezi_: hm, depends on which logs you are referring to. everything looks good on the octavia side, loadbalancer is up and running and works as expected. healthmonitor is quering my webservice as it suppose to. still I get "operating status OFFLINE" when running openstack loadbalancer show | 12:51 |
| *** yamamoto has quit IRC | 12:53 | |
| toker_ | As I said, the amphoras is sending packages to the healthmanager (I verified that with tcpdump). However, the healthmanager doesn't seem to update the status of the loadbalancer to openstack | 12:53 |
| toker_ | what should 'health_update_driver' be set to ? And | 12:54 |
| *** yamamoto has joined #openstack-lbaas | 13:03 | |
| *** yamamoto has quit IRC | 13:08 | |
| toker_ | For some reason the octavia database isn't updated and I cant really figure out why :( | 13:12 |
| *** yamamoto has joined #openstack-lbaas | 13:18 | |
| *** fnaval has joined #openstack-lbaas | 13:20 | |
| *** yamamoto has quit IRC | 13:22 | |
| *** yamamoto has joined #openstack-lbaas | 13:33 | |
| *** yamamoto has quit IRC | 13:38 | |
| toker_ | rm_work: ping me when you are around :P | 13:45 |
| *** yamamoto has joined #openstack-lbaas | 13:48 | |
| *** aojea_ has joined #openstack-lbaas | 13:50 | |
| *** yamamoto has quit IRC | 13:53 | |
| *** aojea_ has quit IRC | 13:53 | |
| *** aojea_ has joined #openstack-lbaas | 13:54 | |
| *** yamamoto has joined #openstack-lbaas | 14:03 | |
| *** yamamoto has quit IRC | 14:07 | |
| *** salmankhan has quit IRC | 14:10 | |
| *** salmankhan has joined #openstack-lbaas | 14:15 | |
| *** yamamoto has joined #openstack-lbaas | 14:18 | |
| *** aojea_ has quit IRC | 14:18 | |
| *** salmankhan has quit IRC | 14:20 | |
| *** salmankhan has joined #openstack-lbaas | 14:20 | |
| *** yamamoto has quit IRC | 14:23 | |
| *** yamamoto has joined #openstack-lbaas | 14:33 | |
| *** yamamoto has quit IRC | 14:38 | |
| *** ivve has joined #openstack-lbaas | 14:41 | |
| *** yamamoto has joined #openstack-lbaas | 14:46 | |
| *** yamamoto has quit IRC | 14:46 | |
| *** aojea_ has joined #openstack-lbaas | 14:47 | |
| *** aojea_ has quit IRC | 14:51 | |
| johnsom | toker_ You want "health_db" as the health_update_driver | 14:55 |
| johnsom | Same with stats | 14:55 |
| johnsom | toker_ To update your database, use "octavia-db-manage --config-file /etc/octavia/octavia.conf upgrade head" | 14:57 |
| johnsom | Neat, we are leading other projects.... https://review.openstack.org/#/c/553679/ | 15:02 |
| *** kbyrne has quit IRC | 15:30 | |
| *** yamamoto has joined #openstack-lbaas | 15:33 | |
| xgerman_ | another day, another not-run of our oeriodic job | 15:34 |
| xgerman_ | or no result from it | 15:34 |
| toker_ | johnsom: Dont think there is anything wrong with the database scheme. The problem seems to be octavia (octavia-healthmanager ???) never updates the database. Everything is working fine, I can query the loadbalancers, the loadbalancers are sending udp-health-packages to the controller - BUT everything is operating_status OFFLINE when querying with openstack cli. I'm fairly positive this worked yesterday *before* removing the | 15:34 |
| johnsom | Hmm, I will see if I can dig up the job on one of the health pages | 15:34 |
| toker_ | octavia api directly...Here is my config, am I missing something ? https://paste.fedoraproject.org/paste/~qoNBz08IyXVrPfynajEhw | 15:34 |
| xgerman_ | http://zuul.openstack.org is blank on the periodic side | 15:35 |
| *** kbyrne has joined #openstack-lbaas | 15:35 | |
| johnsom | xgerman POST_FAILURE | 15:36 |
| johnsom | http://logs.openstack.org/periodic/git.openstack.org/openstack/octavia/master/publish-openstack-octavia-amphora-image-ubuntu/3674c7b/ | 15:36 |
| johnsom | /usr/bin/rsync --delay-updates -F --compress --archive --rsh=/usr/bin/ssh -S none -o StrictHostKeyChecking=no --out-format=<<CHANGED>>%i %n%L /var/lib/zuul/builds/3674c7b2b3674b068ed2afabe7a360a7/work/artifacts/ jenkins@tarballs.openstack.org:/srv/static/tarballs/octavia/ | 15:37 |
| johnsom | rsync: change_dir "/var/lib/zuul/builds/3674c7b2b3674b068ed2afabe7a360a7/work/artifacts" failed: No such file or directory (2) | 15:37 |
| johnsom | rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1183) [sender=3.1.1] | 15:37 |
| johnsom | toker_ looking. Can you do me a favor and restart your health manager process. We once saw a case where it wasn't processing the packets until a restart. Curious if you are seeing that too | 15:39 |
| rm_work | toker_: also make sure your rules for traffic forwarding/listening for your docker container with the healthmanager includes the right ports/protocols | 15:40 |
| rm_work | UDP and not just TCP | 15:40 |
| rm_work | it sounds to me like the packets are being sent by the amps but not actually making it to the HM process | 15:41 |
| rm_work | toker_: and any security groups involved on the ports -- they usually don't include UDP by default | 15:44 |
| rm_work | or did you use tcpdump *in the HM containers* to make sure they were seeing the traffic? | 15:44 |
| johnsom | rm_work Are you going to be able to make the meeting today? | 15:47 |
| rm_work | bleh, did some perf stuff with the new 1.8 based containers and it's still shitty for me :( | 15:47 |
| rm_work | johnsom: yeah | 15:47 |
| johnsom | I am considering two topics that it would be nice to have a quorum for. | 15:47 |
| rm_work | ok | 15:48 |
| johnsom | Yeah, I tested 1.8 a while ago and didn't see any major improvement, but I also didn't change any settings, so assumed I just hadn't turned on the new stuff yet. Still happy with the restart and systemd fixes they did in 1.8 though | 15:49 |
| johnsom | toker_ I don't see anything obviously wrong with that config that would impact health manager. | 15:50 |
| openstackgerrit | German Eichberger proposed openstack/octavia master: Fixes the directory we copy the image to https://review.openstack.org/554960 | 15:50 |
| xgerman_ | ^^ rm_work , johnsom — ley’s hope this is the right place | 15:51 |
| rm_work | johnsom: right, just cgoncalves was hoping i would see perf improvements ;P | 15:51 |
| rm_work | i mean, i was too but wasn't hugely expecting anything | 15:51 |
| rm_work | i still need to figure out why it sucks so bad tho | 15:51 |
| rm_work | i can't even really get 1000 RPS :( | 15:52 |
| johnsom | Yeah, I would have expected some scaling improvement with the threading work they did. But you need more than one vCPU for that and some settings I think | 15:52 |
| rm_work | yeah | 15:53 |
| rm_work | i mean we've got 4 vCPU to work with | 15:53 |
| rm_work | I would just need to enable those | 15:53 |
| rm_work | but i don't know for sure it's a CPU issue... | 15:53 |
| xgerman_ | yep, with yoru weird network there is always - the network - | 15:53 |
| johnsom | Hmm, that is odd. I know in my last testing I could hit 10,000 but my virtual NIC driver was eating up all of my CPU. I wasn't running in ideal environment, vmware workstation on windows.... | 15:53 |
| rm_work | i don't think it's the network | 15:53 |
| rm_work | but *shrug* | 15:53 |
| xgerman_ | netperf graphs or it didn’t happen | 15:54 |
| rm_work | this is not my area of expertise | 15:54 |
| rm_work | but i can try | 15:54 |
| xgerman_ | yeah, you should be able to benchmark vm<->vm | 15:54 |
| rm_work | i mean i can test hitting a member DIRECTLY and compare to hitting the LB | 15:55 |
| xgerman_ | also if they employ firewalls - they night slow stuff down as well | 15:55 |
| rm_work | both are just VMs in the same region | 15:55 |
| johnsom | xgerman_ FYI: I found you job here: http://zuul.openstack.org/builds.html with pipeline: periodic and project: openstack/octavia | 15:55 |
| xgerman_ | thanks. | 15:56 |
| toker_ | hm, thanks for answering guys. I've restarted octavia-healtmanager loads of times, I also tried running it directly ont the controller (from cli) to make sure nothing fishy was up with the container - but no go. | 15:58 |
| toker_ | I'm positive UDP packets are recieved on the controller. Fairly positive they reach the health-manager.. | 15:59 |
| toker_ | There is no octavia-health- process with opened RabbitMQ ports (5671,5672) running in the container <- this is seen though from the logs of the container. Don't think I've seen this before. | 15:59 |
| cgoncalves | rm_work: no perf gains with 1.8? :/ | 16:00 |
| rm_work | nope | 16:00 |
| rm_work | toker_: do you have logs from the health-manager process? | 16:00 |
| cgoncalves | 1.5 FTW! | 16:00 |
| *** kobis has quit IRC | 16:00 | |
| rm_work | cgoncalves: lolno | 16:05 |
| rm_work | this fixes so many other issues | 16:05 |
| rm_work | johnsom: http://paste.openstack.org/show/707690/ | 16:06 |
| johnsom | toker_ You have debug enabled, there should be log messages indicating it received a packet to process | 16:06 |
| rm_work | first two are connecting to the member directly, the third is to the LB | 16:06 |
| rm_work | i turned it down to 1000 for the LB because more and it just bombs entirely | 16:07 |
| *** ivve has quit IRC | 16:08 | |
| rm_work | backend is our golang webserver | 16:08 |
| rm_work | load on the amp barely even budges | 16:09 |
| rm_work | like, 5% or less | 16:10 |
| rm_work | something is f'd? | 16:10 |
| rm_work | do i need to set ulimits inside the amp? | 16:11 |
| johnsom | toker_ Should look like this: | 16:12 |
| johnsom | Mar 21 09:11:33 devstackpy27-2 octavia-health-manager[107797]: DEBUG octavia.amphorae.drivers.health.heartbeat_udp [-] Received packet from ('192.168.0.12', 9818) {{(pid=107986) dorecv /opt/stack/octavia/octavia/amphorae/drivers/health/heartbeat_udp.py:186}} | 16:12 |
| johnsom | rm_work I think haproxy raises the ulimit automatically | 16:13 |
| johnsom | Did you set the connection limit on the LB? | 16:13 |
| rm_work | how can it unless it runs as root? | 16:13 |
| rm_work | no, it's unset | 16:13 |
| johnsom | Ah, set that | 16:13 |
| johnsom | HAProxy starts as root and then drops | 16:13 |
| johnsom | if I remember right. | 16:14 |
| rm_work | hmmm | 16:14 |
| rm_work | so set it to ... something? | 16:14 |
| rm_work | k | 16:14 |
| johnsom | Yeah, set the connection limit, I think there is an open bug that -1 is being translated to 2000 | 16:15 |
| toker_ | johnsom: oh! cool, I'll definitly look into why the healt-manager doesn't recieve any packages. | 16:15 |
| rm_work | toker_: yeah something is up between the process and the packet source | 16:16 |
| rm_work | either network, or firewall, or settings | 16:17 |
| johnsom | rm_work http://cbonte.github.io/haproxy-dconv/1.8/configuration.html#3.2-maxconn | 16:17 |
| rm_work | yeah was just looking at that | 16:17 |
| toker_ | https://paste.fedoraproject.org/paste/RnOPIJZAHj-~XuiqSrxOrw <- this is very werid. | 16:23 |
| toker_ | I don't understand, I run healthmanager from cli, it binds to the correct ip address, udp packages are recieved on that address, healthmanager is saying nothing. | 16:24 |
| rm_work | toker_: so when i told you to set the config for "heartbeat_key" | 16:26 |
| johnsom | toker_ Hmm, check one thing for me. Stop you health manager process, then double check with ps -ef that there are not old processes still running in the background. | 16:26 |
| rm_work | did you set that correctly *everywhere*? | 16:26 |
| rm_work | and by correctly, I mean, pretty much *any string* as long as it is identical everywhere | 16:26 |
| johnsom | If the key was wrong it would log an hmac mis-match message in the log | 16:26 |
| rm_work | hmmm | 16:26 |
| toker_ | rm_work There is only one config file that is used | 16:26 |
| rm_work | k | 16:26 |
| rm_work | and you recreated the amps? | 16:26 |
| toker_ | yes | 16:26 |
| toker_ | all day | 16:26 |
| toker_ | starting the healthmanager seems to spawn three process, right ? | 16:27 |
| johnsom | Right, parent and two others | 16:27 |
| toker_ | Cool | 16:27 |
| johnsom | The health check thread and the receiver thread | 16:28 |
| johnsom | process in later versions | 16:28 |
| *** AlexeyAbashkin has quit IRC | 16:29 | |
| johnsom | rm_work on the tempest plugin, was it just the deletes in the tests that you didn't like? I can leave the creates right? | 16:29 |
| rm_work | err, i think so | 16:30 |
| johnsom | I get your point about running more than we are testing. Just trying to balance that with the "each test must run on it's own" mandate | 16:30 |
| rm_work | trying to remember | 16:30 |
| johnsom | I plan to ignore people if I have to today to make progress on that patch.... grin | 16:31 |
| toker_ | https://paste.fedoraproject.org/paste/2EbUZCRDdf6uI~rRYoOldg <- here is tha actual strace from one of the python processes. You can see it doing a SELECT, COMMIT and ROLLBACK. Not sure if that is normal behaviour. | 16:31 |
| rm_work | err | 16:31 |
| toker_ | johnsom: please dont ignore me :P | 16:31 |
| rm_work | you should get log messages before any of that happens | 16:32 |
| toker_ | hm that is really weird | 16:32 |
| *** aojea_ has joined #openstack-lbaas | 16:32 | |
| johnsom | toker_ Yeah, sqlalchemy is a bit strange with the extra rollback, it can be ignored | 16:32 |
| toker_ | dont understand anything where that logmessage is going | 16:32 |
| johnsom | Yeah, what he said. We don't do DB until that packet received message comes in | 16:32 |
| toker_ | when was debugging added ? maybe my version is to old for that debugmessage | 16:33 |
| johnsom | Like liberty | 16:33 |
| rm_work | debug is for sure on? | 16:34 |
| toker_ | I have debug=True in config file under default section and --debug on the command-line when starting health-manager | 16:35 |
| johnsom | Ah, I was wrong, that debug came in during Pike | 16:36 |
| toker_ | Well im in the heartbeat_udp.py, I can see its there. | 16:36 |
| toker_ | Let me try to add some debug to see if I can figure out what is going on. | 16:36 |
| *** aojea_ has quit IRC | 16:36 | |
| toker_ | (data, srcaddr) = self.sock.recvfrom(UDP_MAX_SIZE) <- Hm, it simply seems to get stuck on this. | 16:46 |
| rm_work | yeah that's it holding open a listen socket | 16:49 |
| rm_work | that means it never gets any packet | 16:49 |
| rm_work | so something is still wrong in the network layer between the amps and the HM | 16:49 |
| johnsom | Yeah, it should sit there until it gets a packet to process | 16:49 |
| rm_work | only other thing is `heartbeat_interval = 3` is a little fast BTW (though unrelated) | 16:51 |
| rm_work | more like 10 is realistic | 16:51 |
| johnsom | FYI, if you have amp image issues around disk space: http://lists.openstack.org/pipermail/openstack-dev/2018-March/128585.html | 16:57 |
| toker_ | yep ok, I've verified this now. From an amp "echo "This is my data" > /dev/udp/xxx/5555" = no message in log, same command directly on the controller = message in logs from the healthmonitor. | 16:57 |
| johnsom | DIB bug | 16:57 |
| toker_ | I just need to figure out why. | 16:58 |
| rm_work | woah you can do that? lol | 16:59 |
| rm_work | just echo to /dev/udp | 16:59 |
| rm_work | TIL | 16:59 |
| toker_ | yea, easiest way :p | 16:59 |
| toker_ | no need for nc or anything :p | 16:59 |
| *** bradjones has quit IRC | 17:03 | |
| *** salmankhan has quit IRC | 17:08 | |
| *** salmankhan has joined #openstack-lbaas | 17:12 | |
| toker_ | Hm, there is something really fishy going on here that is beyond my skills. I changed so the health-manager binds on all interfaces (udp UNCONN 0 0 *:5555 ), still when sending packets from anything else than directly from the controller, the packets gets seen by tcpdump BUT NOT by the healthmanager. | 17:33 |
| toker_ | So how could it be, that I'm allowed to send packages to the udp-socket locally, but not from anywhere else (if there were a firewall issue, I would not see the packets with tcpdump). | 17:33 |
| toker_ | holy moly, I really need to learn networking better. | 18:00 |
| toker_ | Turns out it was the firewall anyway. | 18:00 |
| toker_ | :/ | 18:00 |
| xgerman_ | it always is the firewall — they are evil | 18:00 |
| johnsom | Says the FWaaS guy.... | 18:01 |
| johnsom | lol | 18:01 |
| toker_ | but the tcpdump showed me the packet, so.. | 18:01 |
| xgerman_ | our iptables stuff is pretty benign compared to the backdoor ridden commerical things | 18:01 |
| toker_ | I thought if iptables was the problem, I wouldn't have seen the packet. | 18:01 |
| *** AlexeyAbashkin has joined #openstack-lbaas | 18:03 | |
| toker_ | Lessons learned. | 18:04 |
| toker_ | Again, thanks everyone for helping me out! | 18:05 |
| johnsom | Sure, NP | 18:14 |
| rm_work | yep happy to help :) | 18:16 |
| rm_work | cgoncalves: you sure it wouldn't be easier to just switch everyone to native Octavia than actually try to support n-lbaas in OSP12? lol | 18:16 |
| *** atoth has quit IRC | 18:16 | |
| *** aojea_ has joined #openstack-lbaas | 18:20 | |
| *** AlexeyAbashkin has quit IRC | 18:21 | |
| *** harlowja has joined #openstack-lbaas | 18:24 | |
| *** aojea_ has quit IRC | 18:25 | |
| *** kobis has joined #openstack-lbaas | 18:36 | |
| *** yamamoto has quit IRC | 18:43 | |
| *** Swami has joined #openstack-lbaas | 18:48 | |
| openstackgerrit | Michael Johnson proposed openstack/octavia-tempest-plugin master: Revert "Create scenario tests for loadbalancers" https://review.openstack.org/543030 | 18:52 |
| toker_ | next problem up for grabs ovtavia-dashboard :p | 18:53 |
| toker_ | I installed it as the readme says, I got my "loadbalancer" menu back | 18:53 |
| toker_ | However, when clicking it, I get stuck in a "redirect loop" that shows the loading spinner, reloads the page, shows the spinner ... this goes on forever... | 18:53 |
| toker_ | I actually installed it the same way as the neutron-lbaas-dashboard. And I removed that one so they wouldnt conflict with eachoter | 18:54 |
| johnsom | Yep, if you use the master or queens dashboard you will see a bunch of improvements | 18:54 |
| toker_ | well i did git clone from the github | 18:54 |
| johnsom | Yeah, ok, so master. So let's look at a few things | 18:55 |
| johnsom | under your horizon directory, there is openstack_dashboard/local/enabled Let's double check that there is only _1482_project_load_balancer_panel.py in there and not one of the other load balancer enable files | 18:56 |
| johnsom | I hope you skipped the policy setup for now, it's not critical | 18:57 |
| toker_ | mm yep the file is there | 18:57 |
| toker_ | only that file, no old neutron-ones | 18:57 |
| johnsom | Ok, I just wanted to make sure | 18:58 |
| johnsom | When you did the ./manange.py collectstatic I assume it copied a bunch of files over? | 18:58 |
| johnsom | Or did it not really do anything | 18:58 |
| johnsom | ? | 18:58 |
| toker_ | yep it copied some files, and then I ran compress | 18:59 |
| johnsom | Yeah, and compress takes some time to run | 18:59 |
| johnsom | hmmm, the only thing I can think is the collectstatic didn't work the way it should and there is still some old stuff in there. | 19:00 |
| toker_ | hm, yea well something weird is happening. No error or nothing is returned, just keep "looping" | 19:02 |
| toker_ | can I enable some debug from octavia-dashboard somewhere ? | 19:03 |
| johnsom | In the past when doing development I have blown away the static file under horizon and re-ran collectstatic, but I can't recommend you do that as I'm not a horizon expert. | 19:03 |
| johnsom | Can you open the browser debug window and see what the paths are that are redirecting? | 19:03 |
| johnsom | ctrl-shift-I on chrome, then network tab | 19:04 |
| *** KeithMnemonic1 has joined #openstack-lbaas | 19:05 | |
| toker_ | doesnt seem to be a "redirect" per say, more of a reload of the page. | 19:05 |
| johnsom | hmmm, can you capture that path? | 19:06 |
| *** salmankhan has quit IRC | 19:06 | |
| toker_ | the path is just dashboard/project/load_balancer/ | 19:06 |
| johnsom | Ok, so that is the new octavia dashboard path | 19:07 |
| *** KeithMnemonic has quit IRC | 19:07 | |
| johnsom | the old one was: ngloadbalancersv2 | 19:08 |
| johnsom | Darn, I wish I had a dashboard VM running right now. | 19:09 |
| johnsom | There is some logging from horizon, I just don't remember what the paths are off my head | 19:09 |
| toker_ | ok no worries | 19:09 |
| toker_ | I need to call it a day anyways =) | 19:09 |
| toker_ | Ill debug it some more tomorrow =) | 19:09 |
| toker_ | Thanks again for all the support! | 19:10 |
| johnsom | Ok, if you are still stuck tomorrow morning here I can help | 19:10 |
| johnsom | I will be done with one of the other VMs and can boot up a dashboard | 19:10 |
| toker_ | Cool cool! | 19:10 |
| johnsom | It's just I've already got four running, so not enough memory to get a dashboard VM going | 19:10 |
| *** AlexeyAbashkin has joined #openstack-lbaas | 19:17 | |
| *** AlexeyAbashkin has quit IRC | 19:21 | |
| rm_work | ok back to perf testing -- had to move my whole control-plane to a different k8s cluster | 19:25 |
| rm_work | fortunately it only took about an hour, of which 45m was begging someone to let me use their cluster, and 10m was a DNS swing | 19:25 |
| rm_work | anywho, set the connection-limit to stupidly high number | 19:26 |
| rm_work | didn't change anything | 19:26 |
| rm_work | maxconn 100000 | 19:27 |
| rm_work | is in both the places it should be | 19:27 |
| rm_work | i've also done some further tweaks on the amp though i can't be sure they took effect | 19:32 |
| *** tesseract has quit IRC | 19:38 | |
| rm_work | johnsom: have you ever tested on centos? | 19:38 |
| rm_work | i wonder if it could be specific | 19:38 |
| johnsom | No, ubuntu only | 19:40 |
| rm_work | hilariously, systemd-journal consistently is using more CPU time than haproxy during load testing, lol | 19:43 |
| *** yamamoto has joined #openstack-lbaas | 19:44 | |
| johnsom | Ah, all that logging | 19:44 |
| rm_work | but it's still only like 5% | 19:44 |
| rm_work | yeah this performance is abysmal | 19:45 |
| rm_work | i wonder if i can get an ubuntu amp working | 19:45 |
| johnsom | http://logs.openstack.org/periodic/git.openstack.org/openstack/octavia/master/publish-openstack-octavia-amphora-image-ubuntu/3674c7b/test-images/test-only-amphora-x64-haproxy-ubuntu-xenial.qcow2 | 19:46 |
| johnsom | You could try one of german's | 19:46 |
| johnsom | Though you may have local hacks | 19:47 |
| xgerman_ | sweet — now if we could make that work ;-) | 19:47 |
| xgerman_ | every night ;-) | 19:47 |
| rm_work | lol | 19:47 |
| rm_work | yeah uhh not sure actually if any local hacks are necessary with ubuntu | 19:47 |
| rm_work | i can try his O_o | 19:48 |
| rm_work | but, triggered a build really quick | 19:48 |
| johnsom | It will be 1.6 also | 19:48 |
| rm_work | ah :/ well whatever | 19:48 |
| *** yamamoto has quit IRC | 19:49 | |
| rm_work | ahh yeah i think it won't do the flip right, ugh | 19:50 |
| cgoncalves | rm_work: n-lbaas will still be supported in OSP13 (Queens) for 5 years at least :/ | 19:56 |
| rm_work | :( | 19:57 |
| rm_work | i feel ... bad for you | 19:58 |
| rm_work | sorry about that :( | 19:58 |
| cgoncalves | lol | 19:58 |
| johnsom | #startmeeting Octavia | 20:00 |
| openstack | Meeting started Wed Mar 21 20:00:15 2018 UTC and is due to finish in 60 minutes. The chair is johnsom. Information about MeetBot at http://wiki.debian.org/MeetBot. | 20:00 |
| openstack | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 20:00 |
| *** openstack changes topic to " (Meeting topic: Octavia)" | 20:00 | |
| openstack | The meeting name has been set to 'octavia' | 20:00 |
| johnsom | Hi folks | 20:00 |
| johnsom | #topic Announcements | 20:00 |
| *** openstack changes topic to "Announcements (Meeting topic: Octavia)" | 20:00 | |
| johnsom | "S" series naming poll closes today - check you e-mail for your voting link | 20:01 |
| johnsom | TC elections are coming up in April | 20:01 |
| xgerman_ | o/ | 20:01 |
| johnsom | That is about what I have for announcements this week. | 20:01 |
| johnsom | Any others? | 20:01 |
| xgerman_ | summit talks? | 20:02 |
| xgerman_ | also travel support is closing today | 20:02 |
| johnsom | Sure, I have two talks scheduled for Vancouver: Octavia project update (40 minutes extended version) and I signed up for an on-boarding session for Octavia. | 20:03 |
| nmagnezi_ | o/ | 20:03 |
| rm_work | cool | 20:04 |
| xgerman_ | I have. a talk on Octavia and K8 | 20:04 |
| rm_work | do either of those actually require voting | 20:04 |
| rm_work | or are they just ... set | 20:04 |
| xgerman_ | they are all confirmed | 20:04 |
| rm_work | cool. | 20:04 |
| rm_work | hey i'm doing octavia on k8s :) | 20:04 |
| johnsom | Mine are set by the foundation. They are not yet on the schedule, they will add them later | 20:04 |
| johnsom | Any other announcements? | 20:05 |
| johnsom | #topic Brief progress reports / bugs needing review | 20:06 |
| *** openstack changes topic to "Brief progress reports / bugs needing review (Meeting topic: Octavia)" | 20:06 | |
| johnsom | Ugh, trying to remember what I worked on. A bunch of gate related issues. | 20:06 |
| cgoncalves | did we all... | 20:07 |
| cgoncalves | *didn't | 20:07 |
| johnsom | Oh, helped xgerman_ out with the proxy plugin gate job. | 20:07 |
| johnsom | Today i am starting work on the tempest plugin again to address the comments folks left and the discussion we had at the PTG. | 20:07 |
| rm_work | i made a little review thing with mine and german's listed https://etherpad.openstack.org/p/octavia-priority-reviews | 20:07 |
| rm_work | having something like that can be useful for us | 20:08 |
| johnsom | Hope to wrap that up today/tomorrow so you all can shot more holes in it... grin | 20:08 |
| rm_work | if you want something reviewed, add it and i can look :) | 20:08 |
| rm_work | was hoping i could get johnsom to help prioritize and add stuff too | 20:08 |
| johnsom | rm_work should I add that to the channel topic like we do at the end of the cycle? | 20:08 |
| rm_work | we usually do one of these before each release and it seems to be quite helpful for velocity | 20:08 |
| rm_work | I think that'd be good | 20:08 |
| *** aojea has joined #openstack-lbaas | 20:08 | |
| johnsom | Ok, I will work on that after the meeting | 20:09 |
| rm_work | it's sometimes hard for me to tell what i should be reviewing, across so many projects | 20:09 |
| rm_work | and with a lot of stuff WIP | 20:09 |
| johnsom | EVERYTHING! hahahha | 20:09 |
| johnsom | Yeah, it really helps me with my review dashboard if folks use the workflow -1 for WIP patches | 20:10 |
| johnsom | Once the tempest patch is straightened out, I'm on to provider driver work | 20:11 |
| johnsom | I did start a migration from neutron-lbaas to octavia script one evening. It is really just a start. I think rm_work is going to work on it some | 20:11 |
| johnsom | Any other updates? | 20:12 |
| johnsom | I think I saw Nir had some rally stuff in flight, so that is good too | 20:12 |
| nmagnezi_ | yup | 20:13 |
| nmagnezi_ | still in the works | 20:13 |
| nmagnezi_ | but will finalize this very soon | 20:13 |
| nmagnezi_ | basically two patches are already up | 20:13 |
| johnsom | Nice | 20:13 |
| nmagnezi_ | 1. add octavia pythonclient support | 20:13 |
| nmagnezi_ | 2. port the existing scenario to use Octavia | 20:13 |
| nmagnezi_ | which is mostly done | 20:13 |
| nmagnezi_ | the 3rd patch will contain additional stuff (mostly CRUD for our LB resources) | 20:14 |
| *** aojea has quit IRC | 20:14 | |
| johnsom | I also know I'm behind on dashboard patch reviews. I hope to load that up and do some reviews on that stuff this week. Lots of good work going on there | 20:14 |
| nmagnezi_ | oh and just if it interests anyone here, Rally is about to split to two code bases. to I ported my patches to rally-openstack | 20:15 |
| nmagnezi_ | johnsom, I can help with those | 20:15 |
| johnsom | Ah, interesting, so there will be an OpenStack specific Rally and then a general use one? | 20:15 |
| nmagnezi_ | from what I understood from the Rally core team, the will have a generic base framework and additional code bases for plugins | 20:16 |
| johnsom | That makes sense | 20:16 |
| nmagnezi_ | yup | 20:16 |
| cgoncalves | openstack-octavia-rally project? :) | 20:16 |
| nmagnezi_ | haha | 20:17 |
| nmagnezi_ | maybe rally-k8s | 20:17 |
| nmagnezi_ | who knows :) | 20:17 |
| johnsom | Yeah, if we need a repo for the plugin let me know | 20:17 |
| nmagnezi_ | but anyhow, the split is still WIP | 20:17 |
| johnsom | Ok | 20:17 |
| nmagnezi_ | johnsom, I don't think we will, but will keep you posted | 20:17 |
| johnsom | #topic Other OpenStack activities of note | 20:17 |
| *** openstack changes topic to "Other OpenStack activities of note (Meeting topic: Octavia)" | 20:17 | |
| johnsom | OpenStack Common Healthcheck WSGI Middleware spec | 20:17 |
| johnsom | #link https://review.openstack.org/#/c/531456 | 20:18 |
| johnsom | Our friend mugsie is proposing a common method to report service health | 20:18 |
| *** AlexeyAbashkin has joined #openstack-lbaas | 20:18 | |
| johnsom | Interesting read, worth commenting on. | 20:18 |
| * mugsie will reply on that spec soon, I promise | 20:18 | |
| johnsom | I know our friends at F5 were interested in how we can expose the controller health in a useful way. This might be the answer | 20:19 |
| rm_work | neat | 20:19 |
| johnsom | Proposed "Extended Maintenance" policy for stable branches | 20:19 |
| johnsom | #link https://review.openstack.org/#/c/548916/ | 20:20 |
| johnsom | Also of interest, proposals about how to handle extended maintenance for stable branches. | 20:20 |
| johnsom | For those of you not running master with a few days lag.... | 20:21 |
| johnsom | Ok, on to the big topic.... | 20:21 |
| nmagnezi_ | rm_work, i think he pointed at you :D | 20:21 |
| johnsom | #topic Octavia deleted status vs. 404 | 20:21 |
| *** openstack changes topic to "Octavia deleted status vs. 404 (Meeting topic: Octavia)" | 20:21 | |
| johnsom | #link https://review.openstack.org/#/c/545493/ | 20:22 |
| johnsom | In my work on the tempest plugin and the proxy plugin gate I noticed we have a problem | 20:22 |
| johnsom | Most services in OpenStack return a 404 when a query comes in for a deleted item. | 20:22 |
| *** AlexeyAbashkin has quit IRC | 20:22 | |
| johnsom | The current Octavia API does not, it returns a record with the provisioning_status marked DELETED. | 20:23 |
| rm_work | yeah ... i also just noticed that nova somehow accepts a --deleted param to show old deleted stuff, otherwise 404s on gets | 20:23 |
| johnsom | This morning I confirmed that neutron-lbaas also returns a 404 | 20:23 |
| johnsom | So, we have a backward compatibility issue. | 20:23 |
| xgerman_ | yep, my proxy-gate chokes on the DEL:ETED | 20:23 |
| cgoncalves | backport material? | 20:24 |
| johnsom | So, I wrote up this patch, which switches it over to 404 and gives a path to having a --deleted flag. | 20:24 |
| johnsom | Well, here is the part I need your input on.... | 20:24 |
| xgerman_ | cgoncalves: we are in a funny spot. We released the API and documented the DELETED behavior | 20:25 |
| johnsom | We have now released two versions with the API doing the "DELETED" bit, even though the api-ref does show the 404 | 20:25 |
| cgoncalves | backporting would fix and break API at the same time | 20:25 |
| xgerman_ | yes. | 20:25 |
| xgerman_ | Related: n-lbaas returns 404 instead of 403 (FORBIDDEN) | 20:25 |
| johnsom | Actually, I don't think we documented the "DELETED" case. I haven't looked through the whole api-ref, but I know the 404's are there | 20:25 |
| johnsom | xgerman_ I'm going to ignore the 403 thing for now. That is a neutron oddity | 20:26 |
| xgerman_ | mmh, we need to make sure we don’t *change* the API after the fact | 20:26 |
| xgerman_ | johnsom: well it breaks backward compatibility - but I doubt anyone was using that | 20:26 |
| johnsom | xgeman_ Yeah, but let's focus on one topic at a time | 20:27 |
| xgerman_ | ok, we always told people to use octav ia API - we could change our recommendation to use proxy if 100$ compaitibility is needed and octavia-API if you are willing to fix the two or three variations | 20:27 |
| johnsom | Yeah, ok, in the API-REF we list the DELETED as a possible provisioning_status, but not in any of the sections. Each section does list 404 however. | 20:27 |
| johnsom | So, how do we want to handle this issue in Octavia API? | 20:28 |
| xgerman_ | mmh, so I am for consistency between servcies… | 20:29 |
| johnsom | 1. Consider it an API bug, fix, backport, beg for forgiveness in the release notes. | 20:29 |
| cgoncalves | it has to be fixed either now or later. I'd say fix it now and backport to queens and perhaps also pike (if taken as a critical issue). existing deployments could eventually start observing a different behavior, yes... | 20:29 |
| johnsom | 2. Bump the API version. Likely a major bump as it's not necessarily backward compat | 20:29 |
| rm_work | yeah... | 20:30 |
| rm_work | I would say fix it now | 20:30 |
| johnsom | 3. ??? | 20:30 |
| rm_work | the pain will be less | 20:30 |
| rm_work | we're about to have people switching over en-mass soon | 20:30 |
| johnsom | Yeah, I think we need to do it now, I'm just struggling with how... | 20:30 |
| rm_work | my guess is relatively few have actually seen or would be affected by this | 20:30 |
| xgerman_ | let’s not do a 3.0 - people already are freaked out about 2.0 | 20:30 |
| nmagnezi_ | lol | 20:31 |
| nmagnezi_ | xgerman_, good point | 20:31 |
| johnsom | Yeah, I think the most pain would be with the libraries. I can of course fix openstacksdk, but what about gopher and openstack4j | 20:31 |
| johnsom | Yeah, I really don't want to do 3.0 now. (though there are other result codes that neutron-lbaas used that are wrong IMO) | 20:32 |
| nmagnezi_ | johnsom,what usually justifies an API minor version bump? bug fixes or just new features | 20:32 |
| nmagnezi_ | ? | 20:32 |
| rm_work | i think we just ... fix it and take the backlash | 20:32 |
| rm_work | if there is any | 20:32 |
| xgerman_ | terraform etc. wait for DELETED | 20:33 |
| *** mlavalle has joined #openstack-lbaas | 20:33 | |
| johnsom | Right, A.B.C A is breaking change to the API, B new features but compat, C is bug fixes | 20:33 |
| rm_work | hmm | 20:33 |
| rm_work | errr | 20:33 |
| rm_work | can we... make it a deployer option? | 20:33 |
| nmagnezi_ | rm_work, please no :< | 20:33 |
| rm_work | temporarily? | 20:33 |
| rm_work | i mean | 20:33 |
| johnsom | But we don't really have our API versioning story straight yet. Our discovery is broken | 20:33 |
| rm_work | start it deprecated, but allow people time to flip it over | 20:34 |
| xgerman_ | yep, and gophercloud is rewriting anyway | 20:34 |
| rm_work | "you can flip this now, soon it will be flipped for you" | 20:34 |
| johnsom | LOL, I just had a thought. It's bad, but a thought.... 404 with the DELETED body.... | 20:34 |
| xgerman_ | so if we sneak it in now they should be fine | 20:34 |
| rm_work | LOL | 20:34 |
| rm_work | ummmmmmmmmmm | 20:34 |
| rm_work | yes? | 20:34 |
| rm_work | I mean... why not? | 20:34 |
| rm_work | though probably would still break the tools | 20:34 |
| rm_work | because they'd see the status first is my guess | 20:34 |
| johnsom | Yeah, I think it doesn't really help that much | 20:35 |
| xgerman_ | +1 | 20:35 |
| xgerman_ | do we have the deletes=True option? | 20:35 |
| xgerman_ | so we show deleted ones on request? | 20:35 |
| johnsom | So, frankly I think people will be happy that it becomes the same as the other services. | 20:35 |
| rm_work | yes | 20:35 |
| johnsom | Also, our client already handles lit like a 404 | 20:35 |
| rm_work | i think we may just need to cause some temporary breakage | 20:35 |
| rm_work | to get to consistency | 20:35 |
| xgerman_ | we are playing with credibility here. People don’t like us breaking things IHMO | 20:36 |
| johnsom | ha, people keep bringing up v1 so.... haters are going to hate | 20:36 |
| rm_work | this would be us NOT breaking things IMO | 20:36 |
| rm_work | because i don't think many people have switched yet | 20:37 |
| johnsom | I am more about doing the right thing that people whining | 20:37 |
| rm_work | to octavia | 20:37 |
| xgerman_ | well, I have an install which relies on that feature | 20:37 |
| rm_work | does it? | 20:37 |
| xgerman_ | yep, bot terraform and k8 provider waut for DELETED | 20:37 |
| rm_work | hmmm | 20:37 |
| rm_work | can we fix those to work with BOTH | 20:37 |
| rm_work | *first* | 20:37 |
| rm_work | get patches in for them | 20:37 |
| rm_work | and then do the switch | 20:37 |
| xgerman_ | but it’s Pike — so as long as we leave that alone I am +2 | 20:37 |
| rm_work | maybe once a patch lands to make them work both ways we can backport? | 20:38 |
| johnsom | Oye, I feel like it should be backported all the way to Pike... | 20:38 |
| nmagnezi_ | rm_work, if that will add a config option that would be a problem | 20:38 |
| nmagnezi_ | to backport.. | 20:38 |
| cgoncalves | johnsom: +1 | 20:38 |
| rm_work | nmagnezi_: nah that was a different thought | 20:39 |
| rm_work | yes | 20:39 |
| rm_work | so what if we do it in Master | 20:39 |
| nmagnezi_ | rm_work, oh, alright | 20:39 |
| rm_work | then make terraform work with either | 20:39 |
| rm_work | then once that merges we backport to pike | 20:39 |
| johnsom | xgerman_ do you have the places in the repos for teraform and k8s that we can go do these patches? | 20:39 |
| xgerman_ | I can find them — but my problem is that we are doing deploys and our version management isn’t that great | 20:40 |
| johnsom | Yeah, parallel effort this | 20:40 |
| xgerman_ | so I like this change come with a cahnge Pike->Queen | 20:40 |
| xgerman_ | just my 2ct | 20:41 |
| johnsom | So let's go around the room and get your thoughts on the situation and how you think we should move forward. | 20:41 |
| johnsom | It's a bad situation no matter what, we missed it while doing testing for pike | 20:42 |
| johnsom | No one wants to go first? | 20:42 |
| johnsom | Do we wait and talk about this again next week? | 20:43 |
| rm_work | I mean | 20:43 |
| rm_work | I said my bit | 20:43 |
| rm_work | fix it to 404. fix terraform to work with either and wait for them to release the fix. backport all the way to pike. | 20:43 |
| xgerman_ | so did I it’s easier to communicate beginning X you need this new terraform thing | 20:44 |
| johnsom | Personally, to me the API-REF is the spec. It lists 404 as a result for the calls. So this is a bug in my book. I would fix it, backport it, and be proactive fixing things we think might break. | 20:45 |
| rm_work | yep | 20:45 |
| rm_work | agree | 20:45 |
| rm_work | if terraform starts breaking, people can look and see the changelog item and get the new release | 20:45 |
| *** yamamoto has joined #openstack-lbaas | 20:45 | |
| cgoncalves | johnsom: +1 | 20:46 |
| *** velizarx has joined #openstack-lbaas | 20:46 | |
| nmagnezi_ | johnsom, +2 | 20:46 |
| cgoncalves | I will overflow soon with all my +1s | 20:46 |
| xgerman_ | I know people and it’s easier to tell them for Queens you need the new terraform then; oh, you did some stable Pike release and now everyhting is broken | 20:46 |
| johnsom | Ok. So please review the patch on master. We seem to all agree we can land that. | 20:46 |
| johnsom | xgerman_ please paste some pointers in the channel so we can be aggressive at adding a fix. I will go check openstacksdk and fix if needed. | 20:47 |
| johnsom | We should double check the OSC plugin too. Someone want to volunteer for that? | 20:48 |
| xgerman_ | ok, will do | 20:48 |
| johnsom | I am assuming the will-do is for the links and not the OSC plugin? | 20:49 |
| xgerman_ | I will do the terraform fixes | 20:49 |
| xgerman_ | and investigate the k8s ones | 20:49 |
| *** aojea has joined #openstack-lbaas | 20:49 | |
| johnsom | Thanks. | 20:49 |
| johnsom | Ok, if no one has cycles for our client plugin I will take a look there too. | 20:49 |
| johnsom | I will put an agenda item on the meeting for status and when we should start landing backports. | 20:50 |
| johnsom | #topic Open Discussion | 20:50 |
| *** openstack changes topic to "Open Discussion (Meeting topic: Octavia)" | 20:50 | |
| johnsom | Other items today? | 20:50 |
| nmagnezi_ | I have a question | 20:51 |
| nmagnezi_ | about our tempest plugin | 20:51 |
| johnsom | ok | 20:51 |
| nmagnezi_ | specifically about https://github.com/openstack/octavia-tempest-plugin/blob/master/octavia_tempest_plugin/config.py#L79-L81 | 20:51 |
| *** yamamoto has quit IRC | 20:51 | |
| nmagnezi_ | just wondering why did we adopt a specific name for role | 20:51 |
| cgoncalves | I knew it! | 20:51 |
| nmagnezi_ | cgoncalves, :D | 20:51 |
| nmagnezi_ | I was struggeling with this today | 20:51 |
| johnsom | So, first part, that code is going away | 20:51 |
| johnsom | #link https://review.openstack.org/543030 | 20:52 |
| *** velizarx_ has joined #openstack-lbaas | 20:52 | |
| johnsom | replaced with: https://review.openstack.org/#/c/543034/ | 20:52 |
| johnsom | The reason for the role is the RBAC | 20:53 |
| johnsom | #link https://docs.openstack.org/octavia/latest/configuration/policy.html | 20:53 |
| johnsom | So OpenStack is moving towards a richer RBAC scheme. | 20:53 |
| johnsom | Currently it's either "ADMIN" or "OWNER" by project | 20:53 |
| nmagnezi_ | aha | 20:53 |
| openstackgerrit | German Eichberger proposed openstack/neutron-lbaas master: Fix proxy extension for neutron RBAC https://review.openstack.org/554004 | 20:54 |
| johnsom | nova and octavia both implemented this new RBAC scheme | 20:54 |
| nmagnezi_ | alright so I guess tripleO should configure all the roles mentioned in https://docs.openstack.org/octavia/latest/configuration/policy.html regardless of tempest | 20:54 |
| xgerman_ | +1 | 20:54 |
| johnsom | Where you need to have a role "member" to access the load-balancer service (or nova) | 20:54 |
| johnsom | This is what is in as "default", but we provide a policy.json that allows you to set it back to the old way | 20:55 |
| johnsom | #link https://github.com/openstack/octavia/tree/master/etc/policy | 20:55 |
| johnsom | That said, there is a proposal out to officially align these across the services. | 20:55 |
| nmagnezi_ | johnsom, thanks a lot. will surely read this. | 20:56 |
| nmagnezi_ | johnsom, we came across this when we started to test tripleO based deployments with via the tempest plugin | 20:56 |
| johnsom | #link https://review.openstack.org/#/c/523973/ | 20:56 |
| xgerman_ | nmagnezi_: https://github.com/openstack/openstack-ansible-os_octavia/blob/25f3446fabd92a74322495bd536696074306d01f/tasks/octavia_policy.yml | 20:56 |
| nmagnezi_ | and since I was not aware of those RBAC related roles, I tried to understand where this is coming from | 20:57 |
| johnsom | That docs page is the source of truth there... | 20:57 |
| nmagnezi_ | xgerman_, thanks! | 20:57 |
| nmagnezi_ | johnsom, it's not always is ;) | 20:57 |
| johnsom | True | 20:58 |
| johnsom | but we are getting better | 20:58 |
| nmagnezi_ | indeed | 20:58 |
| cgoncalves | api-ref is also the source of truth but... DELETED... :P | 20:58 |
| johnsom | The "default Octavia policies" section is built out of the code, so will stay accurate | 20:58 |
| johnsom | Yeah, api-ref is the truth, our code lies | 20:59 |
| johnsom | lol | 20:59 |
| nmagnezi_ | haha | 20:59 |
| johnsom | One minute left. | 20:59 |
| johnsom | Thanks folks. | 20:59 |
| xgerman_ | o/ | 20:59 |
| nmagnezi_ | o/ | 21:00 |
| johnsom | If you have other questions I will be around | 21:00 |
| johnsom | #endmeeting | 21:00 |
| *** openstack changes topic to "Discussion of OpenStack Load Balancing (Octavia) | Rocky is open for development!" | 21:00 | |
| openstack | Meeting ended Wed Mar 21 21:00:07 2018 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 21:00 |
| openstack | Minutes: http://eavesdrop.openstack.org/meetings/octavia/2018/octavia.2018-03-21-20.00.html | 21:00 |
| openstack | Minutes (text): http://eavesdrop.openstack.org/meetings/octavia/2018/octavia.2018-03-21-20.00.txt | 21:00 |
| openstack | Log: http://eavesdrop.openstack.org/meetings/octavia/2018/octavia.2018-03-21-20.00.log.html | 21:00 |
| rm_work | johnsom: so yeah, i copied the haproxy config from an amp and put it on haproxy on a random VM | 21:08 |
| rm_work | and i get great performance | 21:08 |
| rm_work | O_o | 21:08 |
| rm_work | so it's not the network here | 21:08 |
| rm_work | it's something about the amp ... | 21:08 |
| rm_work | also i spun up the go webserver on an amp on another port | 21:09 |
| *** velizarx_ has quit IRC | 21:09 | |
| rm_work | fixed the SG for it | 21:09 |
| rm_work | and i get the same perf issues | 21:09 |
| rm_work | so it's not haproxy | 21:09 |
| rm_work | it's something about the network and the amp OS | 21:09 |
| rm_work | >_< | 21:09 |
| *** velizarx has quit IRC | 21:14 | |
| xgerman_ | scroll back - didn’t I say “network” | 21:14 |
| *** AlexeyAbashkin has joined #openstack-lbaas | 21:17 | |
| rm_work | i mean it's NOT network it seems | 21:18 |
| rm_work | i mean the network config on the amp | 21:18 |
| *** kobis has quit IRC | 21:19 | |
| xgerman_ | huh? There ksn;t that much the amp can do to screw that up — especially if you netpwrf | 21:20 |
| xgerman_ | mostly MTU and frame lengths | 21:20 |
| *** AlexeyAbashkin has quit IRC | 21:21 | |
| rm_work | hmmmm | 21:22 |
| rm_work | AH FFFFFFFF | 21:22 |
| rm_work | I figured it out | 21:22 |
| rm_work | it can be the network when the network is ... not exactly the same between test servers | 21:22 |
| rm_work | when i add a FLIP in the mix ... performance dies | 21:22 |
| rm_work | it's our FLIP implementation | 21:22 |
| rm_work | somehow | 21:22 |
| rm_work | I don't even | 21:22 |
| johnsom | LoL | 21:39 |
| *** yamamoto has joined #openstack-lbaas | 21:47 | |
| *** yamamoto has quit IRC | 21:53 | |
| openstackgerrit | Michael Johnson proposed openstack/octavia master: Fix calls to "DELETED" items https://review.openstack.org/545493 | 21:55 |
| johnsom | rebased to make sure this doesn't have a conflict | 21:55 |
| *** pcaruana has quit IRC | 21:55 | |
| *** mlavalle has quit IRC | 22:21 | |
| *** rcernin has joined #openstack-lbaas | 22:25 | |
| *** threestrands has joined #openstack-lbaas | 22:29 | |
| *** threestrands has quit IRC | 22:30 | |
| *** threestrands has joined #openstack-lbaas | 22:30 | |
| *** threestrands has quit IRC | 22:30 | |
| *** threestrands has joined #openstack-lbaas | 22:30 | |
| xgerman_ | johnsom: https://review.openstack.org/555087 | 22:35 |
| johnsom | looking | 22:36 |
| xgerman_ | also are our gates bonked again | 22:38 |
| xgerman_ | ? | 22:38 |
| johnsom | It looks like the multinode is semi-broken | 22:39 |
| johnsom | I was just looking at this: http://logs.openstack.org/94/554694/1/check/octavia-v1-dsvm-py3x-scenario-multinode/d848806/logs/subnode-2/screen-o-hm.txt.gz#_Mar_21_16_53_30_348166 | 22:39 |
| *** yamamoto has joined #openstack-lbaas | 22:49 | |
| *** aojea has quit IRC | 22:54 | |
| *** aojea has joined #openstack-lbaas | 22:54 | |
| *** yamamoto has quit IRC | 22:54 | |
| *** aojea has quit IRC | 22:55 | |
| *** aojea has joined #openstack-lbaas | 22:55 | |
| *** aojea has quit IRC | 22:55 | |
| xgerman_ | mmh — | 22:56 |
| *** fnaval has quit IRC | 22:56 | |
| johnsom | Yeah, they are working on the "native" multi-node stuff and I think broke something. yesterday it was both multi-node gates failing, today it seems it is only py35. | 22:58 |
| johnsom | I think I might not worry about it for another day and see if they fix whatever it is | 22:58 |
| johnsom | Plus I have been sitting on zoom with Cam all day, so trying to get something done... | 22:59 |
| *** fnaval has joined #openstack-lbaas | 23:16 | |
| *** fnaval has quit IRC | 23:16 | |
| *** fnaval has joined #openstack-lbaas | 23:17 | |
| openstackgerrit | Michael Johnson proposed openstack/octavia master: Fix calls to "DELETED" items https://review.openstack.org/545493 | 23:35 |
| *** yamamoto has joined #openstack-lbaas | 23:51 | |
| *** yamamoto has quit IRC | 23:55 | |
| rm_work | hmmmmmmm | 23:58 |
| rm_work | reproducing the performance issues is proving problematic | 23:58 |
| rm_work | i can only do it in one DC now | 23:58 |
| rm_work | and it's not the DC that had problems when i was perf testing originally | 23:58 |
| rm_work | so it must be transient O_o | 23:58 |
| rm_work | getting 20k concurrent without too much issue in other DCs | 23:59 |
| rm_work | (even the ones that were failing for me originally) | 23:59 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!