*** ducttape_ has joined #openstack-lbaas | 00:01 | |
*** amotoki has joined #openstack-lbaas | 00:04 | |
openstackgerrit | Michael Johnson proposed openstack/octavia: Fix a typo in the release notes https://review.openstack.org/427496 | 00:12 |
---|---|---|
*** mugsie has quit IRC | 00:29 | |
openstackgerrit | Michael Johnson proposed openstack/octavia: Fix py3x gates https://review.openstack.org/424890 | 00:38 |
johnsom | Don't get excited, I just added some additional logging to see what is up | 00:39 |
openstackgerrit | Michael Johnson proposed openstack/octavia: Fix py3x gates https://review.openstack.org/424890 | 00:50 |
*** beardedeagle has quit IRC | 00:51 | |
*** beardedeagle has joined #openstack-lbaas | 00:52 | |
*** ducttape_ has quit IRC | 00:55 | |
*** beardedeagle has quit IRC | 00:56 | |
*** ducttape_ has joined #openstack-lbaas | 01:06 | |
*** kevo has quit IRC | 01:11 | |
rm_work | johnsom: uhh, issue with health checking on spares pool... | 01:11 |
rm_work | https://gist.github.com/rm-you/4e91110b008213b818f22150f17ef612 | 01:11 |
rm_work | i stopped the agent on an amp in the spares pool, looks like it got MOST of the way through the recycle operation and failed because it thought it should have a VIP :/ | 01:11 |
rm_work | which ... it wouldn't yet | 01:11 |
rm_work | guess that's a bug | 01:13 |
johnsom | Joy, so a bug with the spares pool. | 01:13 |
rm_work | which I'm honestly not sure why people don't use | 01:13 |
johnsom | Probably not related to RH patch? | 01:13 |
rm_work | no | 01:13 |
rm_work | absolutely not related | 01:13 |
rm_work | trying to multitask | 01:13 |
johnsom | Well, we can get a fix into ocata. | 01:13 |
rm_work | I'm going to see if I can patch it up | 01:14 |
johnsom | I can't look at it until tomorrow though | 01:14 |
rm_work | yeah no worries | 01:14 |
rm_work | I'll tell you when I have a fix up | 01:14 |
rm_work | just an FYI | 01:14 |
johnsom | Ok | 01:14 |
rm_work | I am honestly not sure though how much ACTIVE/PASSIVE buys us over using a spares pool | 01:14 |
rm_work | the time difference is basically the time it takes to send a config via REST | 01:15 |
rm_work | which is ... <100ms? | 01:15 |
johnsom | Faster | 01:15 |
rm_work | so I guess we trade ~100ms for 2n instead of 1n+x | 01:15 |
johnsom | Well, spares failover has to plug ports which is slow with neutron | 01:15 |
rm_work | ah. | 01:16 |
rm_work | interesting | 01:16 |
rm_work | so I guess the way I'm implementing it, that doesn't matter | 01:16 |
rm_work | I'm doing one port per amphora for VIP, and one FLIP per LB, and directing the FLIP to the active amp-port | 01:16 |
johnsom | Could be. Last time I checked, failover can take up to a minute on stock | 01:17 |
rm_work | and exposing the FLIP address as the "VIP" | 01:17 |
rm_work | which is kinda shitty because the user can't own/control the FLIP | 01:17 |
rm_work | but | 01:17 |
rm_work | it "works" :/ | 01:17 |
johnsom | Ok, I am on dinner duty tonight. We are trying blue apron... | 01:19 |
rm_work | aah lol | 01:27 |
rm_work | tell me how that goes | 01:27 |
rm_work | i don't think they ship to my area, i kinda checked briefly a while ago | 01:27 |
*** ducttape_ has quit IRC | 01:39 | |
*** ducttape_ has joined #openstack-lbaas | 01:54 | |
*** ducttape_ has quit IRC | 02:13 | |
*** ducttape_ has joined #openstack-lbaas | 02:20 | |
*** ducttape_ has quit IRC | 02:20 | |
*** ducttape_ has joined #openstack-lbaas | 02:20 | |
*** yuanying has quit IRC | 02:21 | |
*** armax has quit IRC | 03:05 | |
*** itsuugo has joined #openstack-lbaas | 03:23 | |
*** itsuugo has quit IRC | 03:28 | |
*** sticker_ has joined #openstack-lbaas | 03:29 | |
*** sticker has quit IRC | 03:32 | |
*** sanfern has joined #openstack-lbaas | 03:55 | |
*** ducttape_ has quit IRC | 04:05 | |
*** csomerville has joined #openstack-lbaas | 04:20 | |
*** armax has joined #openstack-lbaas | 04:27 | |
*** ducttape_ has joined #openstack-lbaas | 04:28 | |
*** beardedeagle has joined #openstack-lbaas | 04:28 | |
*** beardedeagle has quit IRC | 04:37 | |
*** beardedeagle has joined #openstack-lbaas | 04:37 | |
*** ducttape_ has quit IRC | 04:38 | |
*** dileepr has joined #openstack-lbaas | 04:43 | |
*** ducttape_ has joined #openstack-lbaas | 04:47 | |
*** jamespage has quit IRC | 05:00 | |
*** jamespage has joined #openstack-lbaas | 05:00 | |
*** ducttape_ has quit IRC | 05:13 | |
*** Alex_Staf has joined #openstack-lbaas | 05:31 | |
*** beardedeagle has quit IRC | 06:16 | |
*** beardedeagle has joined #openstack-lbaas | 06:17 | |
*** beardedeagle has quit IRC | 06:21 | |
*** gcheresh_ has joined #openstack-lbaas | 06:26 | |
*** ducttape_ has joined #openstack-lbaas | 06:27 | |
*** rcernin has joined #openstack-lbaas | 06:29 | |
*** ducttape_ has quit IRC | 06:31 | |
*** armax has quit IRC | 06:35 | |
*** itsuugo has joined #openstack-lbaas | 06:37 | |
*** itsuugo has quit IRC | 06:47 | |
*** itsuugo has joined #openstack-lbaas | 06:47 | |
*** itsuugo has quit IRC | 06:52 | |
*** tesseract has joined #openstack-lbaas | 07:18 | |
*** eezhova has joined #openstack-lbaas | 07:18 | |
*** pcaruana has joined #openstack-lbaas | 07:19 | |
*** yamamoto has quit IRC | 07:25 | |
*** ducttape_ has joined #openstack-lbaas | 07:57 | |
*** ducttape_ has quit IRC | 08:02 | |
*** itsuugo has joined #openstack-lbaas | 08:14 | |
*** eezhova has quit IRC | 08:16 | |
*** yuanying has joined #openstack-lbaas | 08:30 | |
*** cody-somerville has joined #openstack-lbaas | 08:43 | |
*** cody-somerville has quit IRC | 08:43 | |
*** cody-somerville has joined #openstack-lbaas | 08:43 | |
*** eezhova has joined #openstack-lbaas | 08:46 | |
*** amotoki has quit IRC | 08:49 | |
*** sanfern has quit IRC | 08:49 | |
*** sanfern has joined #openstack-lbaas | 08:55 | |
*** ducttape_ has joined #openstack-lbaas | 09:28 | |
*** amotoki has joined #openstack-lbaas | 09:30 | |
*** ducttape_ has quit IRC | 09:32 | |
*** amotoki has quit IRC | 09:38 | |
*** amotoki has joined #openstack-lbaas | 09:45 | |
*** amotoki has quit IRC | 09:47 | |
*** amotoki has joined #openstack-lbaas | 09:50 | |
*** sanfern has quit IRC | 10:04 | |
*** sanfern has joined #openstack-lbaas | 10:05 | |
*** sanfern has quit IRC | 10:21 | |
*** sanfern has joined #openstack-lbaas | 10:27 | |
*** ducttape_ has joined #openstack-lbaas | 10:58 | |
*** ducttape_ has quit IRC | 11:03 | |
*** amotoki has quit IRC | 11:18 | |
*** sanfern has quit IRC | 11:25 | |
*** dileepr has quit IRC | 11:59 | |
*** catintheroof has joined #openstack-lbaas | 12:28 | |
*** ducttape_ has joined #openstack-lbaas | 12:29 | |
*** ducttape_ has quit IRC | 12:33 | |
*** ducttape_ has joined #openstack-lbaas | 12:57 | |
*** afranc has quit IRC | 13:09 | |
*** afranc has joined #openstack-lbaas | 13:09 | |
*** Alex_Staf has quit IRC | 13:13 | |
*** adam_g has quit IRC | 13:17 | |
*** sanfern has joined #openstack-lbaas | 13:23 | |
*** adam_g has joined #openstack-lbaas | 13:25 | |
*** Alex_Staf has joined #openstack-lbaas | 13:27 | |
*** ducttape_ has quit IRC | 13:42 | |
openstackgerrit | Nir Magnezi proposed openstack/octavia: Add placement services to devstack example files https://review.openstack.org/427710 | 13:57 |
nmagnezi | johnsom, rm_work ^^ | 13:57 |
*** amotoki has joined #openstack-lbaas | 13:59 | |
*** amotoki has quit IRC | 14:08 | |
*** ducttape_ has joined #openstack-lbaas | 14:09 | |
*** matt-borland has joined #openstack-lbaas | 14:14 | |
*** beardedeagle has joined #openstack-lbaas | 14:16 | |
*** ducttape_ has quit IRC | 14:22 | |
*** ducttape_ has joined #openstack-lbaas | 14:22 | |
*** ducttape_ has quit IRC | 14:23 | |
*** ducttape_ has joined #openstack-lbaas | 14:28 | |
*** ducttape_ has quit IRC | 14:31 | |
*** csomerville has quit IRC | 15:12 | |
*** csomerville has joined #openstack-lbaas | 15:13 | |
*** mugsie has joined #openstack-lbaas | 15:13 | |
*** kobis has joined #openstack-lbaas | 15:16 | |
*** gcheresh_ has quit IRC | 15:24 | |
*** ducttape_ has joined #openstack-lbaas | 15:34 | |
*** ducttape_ has quit IRC | 15:34 | |
*** ducttape_ has joined #openstack-lbaas | 15:35 | |
*** armax has joined #openstack-lbaas | 15:35 | |
*** rcernin has quit IRC | 15:51 | |
*** kevo has joined #openstack-lbaas | 15:54 | |
*** fnaval has joined #openstack-lbaas | 15:55 | |
*** eezhova has quit IRC | 16:07 | |
*** _ducttape_ has joined #openstack-lbaas | 16:09 | |
*** ducttape_ has quit IRC | 16:13 | |
openstackgerrit | Ankur proposed openstack/octavia: Add v2 load balancer type and controllers https://review.openstack.org/423543 | 16:30 |
*** kobis has quit IRC | 16:38 | |
openstackgerrit | Merged openstack/octavia: Add placement services to devstack example files https://review.openstack.org/427710 | 16:48 |
*** kobis has joined #openstack-lbaas | 16:56 | |
*** tesseract has quit IRC | 16:56 | |
*** beardedeagle has quit IRC | 16:57 | |
*** itsuugo has quit IRC | 16:57 | |
*** beardedeagle has joined #openstack-lbaas | 16:57 | |
*** Alex_Staf has quit IRC | 17:00 | |
*** kevo has quit IRC | 17:06 | |
*** Alex_Staf has joined #openstack-lbaas | 17:53 | |
*** _ducttape_ has quit IRC | 17:54 | |
*** ducttape_ has joined #openstack-lbaas | 17:54 | |
*** ducttape_ has quit IRC | 17:54 | |
*** ducttape_ has joined #openstack-lbaas | 17:55 | |
*** ducttape_ has quit IRC | 17:55 | |
*** ducttape_ has joined #openstack-lbaas | 17:56 | |
johnsom | Strangest thing, this placement thing is breaking our py3x gates, but not the py27 gates.... | 17:58 |
johnsom | http://logs.openstack.org/90/424890/3/check/gate-neutron-lbaasv2-dsvm-py3x-scenario-ubuntu-xenial-nv/aac3428/logs/screen-n-cpu.txt.gz#_2017-02-01_01_31_16_766 | 17:59 |
johnsom | Can't boot the cirros hosts | 17:59 |
*** kobis has quit IRC | 18:03 | |
*** harlowja_ has joined #openstack-lbaas | 18:05 | |
*** harlowja has quit IRC | 18:06 | |
johnsom | Hmm, I'm getting that on both working and not, but it fixed it local. Argh. | 18:12 |
*** pcaruana has quit IRC | 18:28 | |
openstackgerrit | Valleriya Perelman proposed openstack/octavia: ACTIVE-ACTIVE Topology - Initial Cluster Manager https://review.openstack.org/405238 | 18:37 |
openstackgerrit | Valleriya Perelman proposed openstack/octavia: ACTIVE-ACTIVE Topology: Initial Distributor Noop Driver https://review.openstack.org/313006 | 18:37 |
openstackgerrit | Valleriya Perelman proposed openstack/octavia: ACTIVE-ACTIVE Topology : create distributor network flow https://review.openstack.org/409763 | 18:37 |
openstackgerrit | Valleriya Perelman proposed openstack/octavia: Active-Active Topology - register/uregister amphorae tasks https://review.openstack.org/409765 | 18:37 |
openstackgerrit | Valleriya Perelman proposed openstack/octavia: Active-Active Topology - Cluster DB Tasks https://review.openstack.org/409764 | 18:37 |
openstackgerrit | Valleriya Perelman proposed openstack/octavia: Active-Active Topology - LB create cluster impl https://review.openstack.org/426560 | 18:37 |
openstackgerrit | Valleriya Perelman proposed openstack/octavia: ACTIVE-ACTIVE Topology OVS-based Distributor Driver https://review.openstack.org/317629 | 18:37 |
openstackgerrit | Valleriya Perelman proposed openstack/octavia: ACTIVE-ACTIVE Topology OVS-based Distributor Backend https://review.openstack.org/320422 | 18:37 |
openstackgerrit | Valleriya Perelman proposed openstack/octavia: ACTIVE-ACTIVE Topology - Distributor related tasks https://review.openstack.org/406951 | 18:37 |
openstackgerrit | Valleriya Perelman proposed openstack/octavia: ACTIVE-ACTIVE - controller network tasks https://review.openstack.org/323481 | 18:37 |
openstackgerrit | Valleriya Perelman proposed openstack/octavia: ACTIVE-ACTIVE Topology - Distributor image creation https://review.openstack.org/403594 | 18:37 |
openstackgerrit | Valleriya Perelman proposed openstack/octavia: ACTIVE-ACTIVE - network driver related changes https://review.openstack.org/322494 | 18:37 |
openstackgerrit | Valleriya Perelman proposed openstack/octavia: ACTIVE-ACTIVE - distributor certificate tasks https://review.openstack.org/406952 | 18:37 |
openstackgerrit | Valleriya Perelman proposed openstack/octavia: ACTIVE-ACTIVE Topology - distributor creation flow https://review.openstack.org/406953 | 18:37 |
openstackgerrit | Valleriya Perelman proposed openstack/octavia: ACTIVE-ACTIVE Topology - create shared distributor https://review.openstack.org/406954 | 18:37 |
openstackgerrit | Valleriya Perelman proposed openstack/octavia: Active-Active Topology - Distributor open_flow back-end https://review.openstack.org/427858 | 18:37 |
openstackgerrit | Valleriya Perelman proposed openstack/octavia: ACTIVE-ACTIVE Topology - Initial Cluster Manager https://review.openstack.org/405238 | 18:43 |
openstackgerrit | Valleriya Perelman proposed openstack/octavia: ACTIVE-ACTIVE Topology - Distributor related tasks https://review.openstack.org/406951 | 18:43 |
openstackgerrit | Valleriya Perelman proposed openstack/octavia: Active-Active Topology - Distributor open_flow back-end https://review.openstack.org/427858 | 18:43 |
openstackgerrit | Valleriya Perelman proposed openstack/octavia: ACTIVE-ACTIVE Topology - Distributor image creation https://review.openstack.org/403594 | 18:43 |
openstackgerrit | Valleriya Perelman proposed openstack/octavia: ACTIVE-ACTIVE Topology : create distributor network flow https://review.openstack.org/409763 | 18:43 |
openstackgerrit | Valleriya Perelman proposed openstack/octavia: Active-Active Topology - register/uregister amphorae tasks https://review.openstack.org/409765 | 18:43 |
openstackgerrit | Valleriya Perelman proposed openstack/octavia: ACTIVE-ACTIVE - distributor certificate tasks https://review.openstack.org/406952 | 18:43 |
openstackgerrit | Valleriya Perelman proposed openstack/octavia: Active-Active Topology - Cluster DB Tasks https://review.openstack.org/409764 | 18:43 |
openstackgerrit | Valleriya Perelman proposed openstack/octavia: ACTIVE-ACTIVE Topology - distributor creation flow https://review.openstack.org/406953 | 18:43 |
openstackgerrit | Valleriya Perelman proposed openstack/octavia: Active-Active Topology - LB create cluster impl https://review.openstack.org/426560 | 18:43 |
openstackgerrit | Valleriya Perelman proposed openstack/octavia: ACTIVE-ACTIVE Topology - create shared distributor https://review.openstack.org/406954 | 18:43 |
*** Alex_Staf has quit IRC | 18:45 | |
*** kevo has joined #openstack-lbaas | 18:53 | |
*** ducttape_ has quit IRC | 19:02 | |
*** ducttape_ has joined #openstack-lbaas | 19:03 | |
*** yuanying has quit IRC | 19:03 | |
*** dileepr has joined #openstack-lbaas | 19:14 | |
johnsom | Just a heads up, I did a full re-clone devstack this morning and it's totally broken | 19:16 |
*** harlowja_ has quit IRC | 19:19 | |
rm_work | ffff | 19:28 |
rm_work | ok | 19:28 |
rm_work | i'll umm | 19:29 |
rm_work | try not to do that? | 19:29 |
rm_work | <_< | 19:29 |
rm_work | that's difficult for my workflow T_T | 19:29 |
*** sanfern has quit IRC | 19:40 | |
rm_work | johnsom: finishing up reviewing the RH Amps | 19:50 |
rm_work | johnsom: I can merge when I'm done? | 19:50 |
johnsom | Yes, please do if you are happy with it | 19:51 |
rm_work | OK :) | 19:51 |
openstackgerrit | Shashank Kumar Shankar proposed openstack/octavia: Introduce Octavia v2 API for pools https://review.openstack.org/405922 | 19:53 |
nmagnezi | johnsom, hey, re: broken devstack. do we know what was broken? doe that also break / will break the gates? | 20:00 |
nmagnezi | well, it's meeting time :) | 20:01 |
johnsom | It is nova | 20:01 |
*** beardedeagle has quit IRC | 20:07 | |
*** beardedeagle has joined #openstack-lbaas | 20:07 | |
rm_work | ahhh shit | 20:13 |
rm_work | yeah that'll break the gate >_> | 20:13 |
*** csomerville has quit IRC | 20:13 | |
johnsom | Well, we can give it a try | 20:29 |
rm_work | oh right i forgot to mention that spares-health bug, but we can always just backport it | 20:36 |
rm_work | OK, just did +A | 20:37 |
nmagnezi | may the force be with the gates. | 20:38 |
*** harlowja has joined #openstack-lbaas | 20:39 | |
johnsom | We are good, it's going straight to the gate-gate, so should pass | 20:40 |
openstackgerrit | Nakul Dahiwade proposed openstack/octavia: Add v2 load balancer type and controllers https://review.openstack.org/423543 | 20:43 |
openstackgerrit | Ankur proposed openstack/octavia: Add v2 load balancer type and controllers https://review.openstack.org/423543 | 20:49 |
*** beardedeagle has quit IRC | 20:53 | |
*** beardedeagle has joined #openstack-lbaas | 20:53 | |
nmagnezi | rm_work, johnsom, maybe this will save the day (nova): https://review.openstack.org/#/c/427747/ | 20:55 |
nmagnezi | oh.. wrong link? sec | 20:55 |
*** _ducttape_ has joined #openstack-lbaas | 20:58 | |
*** ducttape_ has quit IRC | 21:02 | |
rm_work | brb | 21:03 |
rm_work | hopefully that merges fine :) | 21:03 |
rm_work | I'll try stacking later and see if anything jumps out to me | 21:03 |
*** kobis has joined #openstack-lbaas | 21:08 | |
*** eezhova has joined #openstack-lbaas | 21:08 | |
*** kevo has quit IRC | 21:08 | |
*** armax has quit IRC | 21:11 | |
openstackgerrit | Merged openstack/octavia: Fix the amphora image support for RH Linux flavors https://review.openstack.org/331841 | 21:24 |
nmagnezi | yay! | 21:29 |
nmagnezi | looks like we are ready for RC1 | 21:29 |
*** raj_ has joined #openstack-lbaas | 21:31 | |
raj_ | A toss-up question - why does Octavia seem to want to have Rather Useful Things (tm) like TCP Selective ACKnowledgements and TCP Timestamps enabled? A maze of twisty troubleshooting passages have led me to https://github.com/openstack/octavia/blob/master/elements/haproxy-octavia/post-install.d/20-haproxy-tune-kernel | 21:32 |
raj_ | grrr... s/enabled/disabled/ | 21:33 |
johnsom | Those are optimizations recommended to us by the folks at haproxy technologies | 21:33 |
raj_ | How are they optimizations??? | 21:34 |
raj_ | They rather fully trash TCP's ability to recover from non-trivial bursts of packet loss. | 21:34 |
raj_ | I guess I should also ask if those same folks provided suggestions as to what the client and server timeouts should be for haproxy itself. | 21:35 |
raj_ | ftp://ftp.netperf.org/candidate.png is an example of what happens without SACK and Timestamps with a non-trivial loss burst. | 21:36 |
*** beardedeagle has quit IRC | 21:37 | |
*** _ducttape_ has quit IRC | 21:37 | |
*** catintheroof has quit IRC | 21:37 | |
johnsom | Selective acks does seem a bit counter intuitive. | 21:37 |
*** ducttape_ has joined #openstack-lbaas | 21:37 | |
*** beardedeagle has joined #openstack-lbaas | 21:37 | |
*** catintheroof has joined #openstack-lbaas | 21:37 | |
johnsom | For client and server timeouts are you talking about the connect/response timeouts? | 21:38 |
raj_ | Most of the other sysctl settings in 2-haproxy-tune-kernel look more or less sane I suppose, but I'm not sure about the clipping of the maximum TCP window/socket buffer size to about half a megabyte. | 21:38 |
raj_ | timeout connect 10s | 21:39 |
raj_ | timeout client 1m | 21:39 |
raj_ | timeout server 1m | 21:39 |
raj_ | timeout check 10s | 21:39 |
johnsom | Yeah, ok. | 21:39 |
raj_ | timeout client and timeout server in this case | 21:39 |
johnsom | Those are really dependent on the application. We do have an open bug to expose those to users, but it has not been implemented yet. | 21:39 |
raj_ | I've not gotten around to finding-out what those are in an Octavia image. | 21:39 |
johnsom | raj_ https://github.com/openstack/octavia/blob/master/octavia/common/jinja/haproxy/templates/base.j2#L32 | 21:40 |
raj_ | Units there are milliseconds then I suppose? | 21:41 |
johnsom | Correct | 21:41 |
raj_ | Without SACK and Timestamps, a non-trivial packet loss burst will result in what you see in that .png - the RTO will continue to double until one hits the client (or server I suppose) timeout on forward progress and haproxy will nuke the connection with extreme prejudice. | 21:42 |
raj_ | All the quicker/more effective loss recovery heuristics of TCP tend to depend on SACK and/or Timestamps... | 21:43 |
johnsom | I am pretty sure the timestamp disable is about the latency to generate the timestamps. | 21:44 |
raj_ | ??? | 21:44 |
johnsom | (Trying to pull up the way-back machine as this has been in the code for a long time) | 21:44 |
johnsom | As such, this could be a hold over from older kernels too | 21:45 |
raj_ | Those settings, and perhaps some il-behaved image building on baremetal build nodes, seem to be at the root of a problem which has been festering close to home for a long time too :) | 21:46 |
raj_ | The only thing I've ever heard as negatives about TCP Timestamps is they tend to shrink the effective TCP Maximum Segment Size (MSS) by 12 bytes. | 21:47 |
raj_ | I don't recall anyone reporting they significantly increased say the service demand reported by a netperf benchmark. | 21:47 |
johnsom | Troubles with helion? | 21:47 |
raj_ | Well, hypothetically speaking, let's assume the following: | 21:48 |
raj_ | A build pipeline involving some bare metal systems separated from the likes of git repositories by an occasionally congested path... | 21:48 |
raj_ | From time to time, build jobs report connection reset by peer when trying to access the git server(s) which happen to be behind an haproxy (not octavia)... | 21:49 |
johnsom | So, some googling to refresh my memory has RedHat giving advice on the timestamps: https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux_for_Real_Time/7/html/Tuning_Guide/Reduce_TCP_performance_spikes.html | 21:50 |
raj_ | And an image building job that seems to end-up with the settings in https://github.com/openstack/octavia/blob/master/elements/haproxy-octavia/post-install.d/20-haproxy-tune-kernel ending-up being set on the baremetal nodes. | 21:50 |
*** eezhova has quit IRC | 21:50 | |
raj_ | Thus, hypothetically speaking, one might end-up looking at those Octavia-desired sysctl settings and asking Why?!? :) | 21:50 |
raj_ | That redhat link is blissfully free from the ravages of details. | 21:51 |
johnsom | Haha | 21:51 |
johnsom | Yeah. We had a person participating from HAProxy tech back when those were created. They are probably worth revisiting. | 21:52 |
raj_ | Since it is in an area related to "real time" I would guess that in the RHEL7 (?) kernel, there can be situations where going to get time can cause a bit of a short hiccup. | 21:52 |
johnsom | Glad to know folks thought highly of our customizations and borrowed them. I'm pretty sure I had nothing to do with that when I was hypothetically involved with that product. | 21:53 |
raj_ | I'd be happy to file a ticket - just need to be pointed at where. | 21:53 |
johnsom | https://bugs.launchpad.net/octavia/+filebug | 21:53 |
raj_ | Well, in this hypothetical situation, those settings were not consciously adopted, they seem to have leaked-out onto the baremetal for reasons I don't yet fully understand. | 21:54 |
*** armax has joined #openstack-lbaas | 21:54 | |
johnsom | The DIB setup in that hypothetical environment needed to be burned to the ground IMHO. It was a hack job that gave me no end of headaches trying to integrate with... | 21:55 |
*** eezhova has joined #openstack-lbaas | 21:55 | |
johnsom | Digging a little deeper, I think those settings were directly related to issues in the 2.4 and 2.6 kernel series | 21:56 |
*** catintheroof has quit IRC | 21:57 | |
*** catintheroof has joined #openstack-lbaas | 21:58 | |
raj_ | https://bugs.launchpad.net/octavia/+bug/1661105 | 22:01 |
openstack | Launchpad bug 1661105 in octavia "Octavia should not disable TCP Selective ACKnowledgements or Timestamps" [Undecided,New] | 22:01 |
raj_ | johnsom, hypothetically speaking, is there a later, better image building mechanism out there? | 22:04 |
johnsom | DIB works reasonably well when it is not "customized" in such a way as I saw | 22:05 |
raj_ | Does the dimm memory have few enough bit errors to have an idea as to what was behind the clipping of tcp_[rw]mem? | 22:06 |
*** kobis has quit IRC | 22:06 | |
raj_ | (That is how I tend to describe my own wayback :) ) | 22:06 |
*** beardedeagle has quit IRC | 22:07 | |
*** beardedeagle has joined #openstack-lbaas | 22:07 | |
*** eezhova has quit IRC | 22:08 | |
*** ducttape_ has quit IRC | 22:09 | |
*** ducttape_ has joined #openstack-lbaas | 22:10 | |
johnsom | I am pretty sure that was to reduce the memory footprint while handling a large number of connections | 22:11 |
johnsom | concurrent connection metrics are a "thing" in the load balancing world | 22:12 |
*** ducttape_ has quit IRC | 22:15 | |
raj_ | Sounds reasonable, though the Tput <= Window/RTT cost might be a bit high. | 22:15 |
xgerman | hi, like all the hypotheticals | 22:18 |
johnsom | Haha | 22:18 |
xgerman | and concur with johnsom | 22:18 |
xgerman | I did some benchmarks on a now defunct publi cloud and they were bad so people asked me to apply tuning parameters and here we are | 22:19 |
raj_ | I'm not sure how such settings would have increased performance but OK. | 22:20 |
johnsom | The guy that helped us out with that was super sharp and on top of this stuff. My suspicion is that these have not withstood the test of time and said kernel is much better at some of this than it was at the time. | 22:21 |
raj_ | Do Octavia images get built "upstream?" If so, I wonder if these settings might be leaking-out there too. | 22:21 |
xgerman | we build them afresh every time you install devstack | 22:22 |
johnsom | We provide a script to build images for Octavia amphora, but we do not ship pre-built images anywhere | 22:22 |
* raj_ was just wondering if this hypothetical leakage might be affecting other folks | 22:23 | |
johnsom | To my knowledge, this config was only in the haproxy element for octavia. It's not even in the native DIB haproxy element to my knowledge. | 22:26 |
johnsom | Yeah, don't see it there, so they didn't borrow either | 22:27 |
johnsom | https://github.com/openstack/tripleo-image-elements/tree/master/elements/haproxy | 22:28 |
johnsom | Anyhow, those configs are ~3 years old now, so probably time to update. | 22:29 |
xgerman | +100 | 22:30 |
*** catintheroof has quit IRC | 22:35 | |
openstackgerrit | Michael Johnson proposed openstack/octavia: Remove outdated kernel tuning for haproxy https://review.openstack.org/427957 | 22:43 |
johnsom | raj_ ^^^ | 22:43 |
*** matt-borland has quit IRC | 22:44 | |
raj_ | Works for me. Paint it yellow and ship it :) Do you want me to weigh-in in review? | 22:45 |
*** ducttape_ has joined #openstack-lbaas | 22:47 | |
*** ducttape_ has quit IRC | 22:49 | |
*** _ducttape_ has joined #openstack-lbaas | 22:49 | |
*** _ducttape_ has quit IRC | 22:53 | |
johnsom | More reviews the merrier.... | 22:56 |
*** yuanying has joined #openstack-lbaas | 23:07 | |
*** kobis has joined #openstack-lbaas | 23:08 | |
*** kobis has quit IRC | 23:12 | |
xgerman | l8ter | 23:21 |
johnsom | See ya | 23:21 |
raj_ | Thanks for all the help/feedback/whatnot | 23:21 |
johnsom | Sure, NP | 23:21 |
*** yamamoto has joined #openstack-lbaas | 23:24 | |
*** beardedeagle has quit IRC | 23:32 | |
*** beardedeagle has joined #openstack-lbaas | 23:32 | |
*** beardedeagle has quit IRC | 23:37 | |
*** ducttape_ has joined #openstack-lbaas | 23:51 | |
*** cody-somerville has quit IRC | 23:59 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!