Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Reduce HAProxy reloads - adds support to use the haproxy dynamic config api #19073

Merged
merged 9 commits into from
Jul 16, 2018
Merged

Reduce HAProxy reloads - adds support to use the haproxy dynamic config api #19073

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
2b97c42
Add support to dynamically configure haproxy to reduce the reloads ne…
ramr Mar 3, 2018
3f5cccf
Fixes as per review comments from @smarterclayton
ramr Mar 20, 2018
d599bfb
Add support for a new plugin to manage blueprint routes and configure…
ramr Apr 2, 2018
a06d009
Changes due to rebase - fixup tests and fix a bug that rebase caused …
ramr Jun 6, 2018
5f45fc0
Bug fix after rebase to handle http routes without reloading.
ramr Jun 14, 2018
9ddee4f
Add mini stress test for the different types of routes.
ramr Jun 21, 2018
9f32206
Add gocsv + go-haproxy dependencies for haproxy dynamic config api ch…
ramr Jun 7, 2018
831c114
bump(*)
ramr Jul 9, 2018
205ded1
bump(*)
ramr Jul 14, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 10 additions & 4 deletions glide.lock
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

5 changes: 5 additions & 0 deletions glide.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -182,6 +182,11 @@ import:
# router metrics
- package: github.com/cockroachdb/cmux
version: b64f5908f4945f4b11ed4a0a9d3cc1e23350866d
# router dynamic reloads
- package: github.com/gocarina/gocsv
version: a5c9099e2484f1551abb9433885e158610a25f4b
- package: github.com/bcicen/go-haproxy
version: ff5824fe38bede761b873cab6e247a530e89236a
# keep us up to date with latest DNS security fixes
- package: github.com/miekg/dns
version: ~v1
Expand Down
7 changes: 7 additions & 0 deletions hack/lib/start.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -613,6 +613,13 @@ function os::start::router() {
else
oc adm router --config="${ADMIN_KUBECONFIG}" --images="${USE_IMAGES}" --service-account=router
fi

# Note that when the haproxy config manager is set based on router type,
# the env entry may need to be always set or removed (if defaulted).
if [[ -n "${ROUTER_HAPROXY_CONFIG_MANAGER:-}" ]]; then
os::log::debug "Changing the router DC to enable the haproxy config manager"
oc set env dc/router -c router ROUTER_HAPROXY_CONFIG_MANAGER=true
fi
}
readonly -f os::start::router

Expand Down
58 changes: 43 additions & 15 deletions images/router/haproxy/conf/haproxy-config.template
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@
{{- define "/var/lib/haproxy/conf/haproxy.config" }}
{{- $workingDir := .WorkingDir }}
{{- $defaultDestinationCA := .DefaultDestinationCA }}
{{- $dynamicConfigManager := .DynamicConfigManager }}
{{- $router_ip_v4_v6_mode := env "ROUTER_IP_V4_V6_MODE" "v4" }}


Expand Down Expand Up @@ -46,6 +47,8 @@ global
{{- end}}
ca-base /etc/ssl
crt-base /etc/ssl
# TODO: Check if we can get reload to be faster by saving server state.
# server-state-file /var/lib/haproxy/run/haproxy.state
stats socket /var/lib/haproxy/run/haproxy.sock mode 600 level admin expose-fd listeners
stats timeout 2m

Expand Down Expand Up @@ -196,7 +199,7 @@ frontend public_ssl
# for the SNI case, we also need to compare it in case-insensitive mode (by converting it to lowercase) as RFC 4343 says
acl sni req.ssl_sni -m found
acl sni_passthrough req.ssl_sni,lower,map_reg(/var/lib/haproxy/conf/os_sni_passthrough.map) -m found
use_backend be_tcp:%[req.ssl_sni,lower,map_reg(/var/lib/haproxy/conf/os_tcp_be.map)] if sni sni_passthrough
use_backend %[req.ssl_sni,lower,map_reg(/var/lib/haproxy/conf/os_tcp_be.map)] if sni sni_passthrough

# if the route is SNI and NOT passthrough enter the termination flow
use_backend be_sni if sni
Expand Down Expand Up @@ -367,19 +370,10 @@ backend openshift_default
*/}}
{{- range $cfgIdx, $cfg := .State }}
{{- if matchValues (print $cfg.TLSTermination) "" "edge" "reencrypt" }}
{{- if (eq $cfg.TLSTermination "") }}

# Plain http backend
backend be_http:{{$cfgIdx}}
{{- else if (eq $cfg.TLSTermination "edge") }}

# Plain http backend but request is TLS, terminated at edge
backend be_edge_http:{{$cfgIdx}}
{{ else if (eq $cfg.TLSTermination "reencrypt") }}

# Secure backend which requires re-encryption
backend be_secure:{{$cfgIdx}}
{{- end }}{{/* end chceck for router type */}}
# Plain http backend or backend with TLS terminated at the edge or a
# secure backend with re-encryption.
backend {{genBackendNamePrefix $cfg.TLSTermination}}:{{$cfgIdx}}
mode http
option redispatch
option forwardfor
Expand Down Expand Up @@ -473,12 +467,36 @@ backend be_secure:{{$cfgIdx}}
{{- end }}{{/* end get serviceUnit from its name */}}
{{- end }}{{/* end range over serviceUnitNames */}}

{{- with $dynamicConfigManager }}
{{- if (eq $cfg.TLSTermination "reencrypt") }}
{{- range $idx, $serverName := $dynamicConfigManager.GenerateDynamicServerNames $cfgIdx }}
server {{$serverName}} 172.4.0.4:8765 weight 0 ssl disabled check inter {{firstMatch $timeSpecPattern (index $cfg.Annotations "router.openshift.io/haproxy.health.check.interval") (env "ROUTER_BACKEND_CHECK_INTERVAL") "5000ms"}}
{{- if gt (len (index $cfg.Certificates (printf "%s_pod" $cfg.Host)).Contents) 0 }} verify required ca-file {{ $workingDir }}/cacerts/{{$cfgIdx}}.pem
{{- else }}
{{- if gt (len $defaultDestinationCA) 0 }} verify required ca-file {{ $defaultDestinationCA }}
{{- else }} verify none
{{- end }}
{{- end }}
{{- with $podMaxConn := index $cfg.Annotations "haproxy.router.openshift.io/pod-concurrent-connections" }}
{{- if (isInteger (index $cfg.Annotations "haproxy.router.openshift.io/pod-concurrent-connections")) }} maxconn {{$podMaxConn}} {{- end }}
{{- end}}{{/* end pod-concurrent-connections annotation */}}
{{- end }}{{/* end range over dynamic server names */}}

{{- else }}
{{- with $name := $dynamicConfigManager.ServerTemplateName $cfgIdx }}
{{- with $size := $dynamicConfigManager.ServerTemplateSize $cfgIdx }}
server-template {{$name}}- 1-{{$size}} 172.4.0.4:8765 check disabled
{{- end }}
{{- end }}
{{- end }}
{{- end }}

{{- end }}{{/* end if tls==edge/none/reencrypt */}}

{{- if eq $cfg.TLSTermination "passthrough" }}

# Secure backend, pass through
backend be_tcp:{{$cfgIdx}}
backend {{genBackendNamePrefix $cfg.TLSTermination}}:{{$cfgIdx}}
{{- if ne (env "ROUTER_SYSLOG_ADDRESS") ""}}
option tcplog
{{- end }}
Expand Down Expand Up @@ -528,6 +546,15 @@ backend be_tcp:{{$cfgIdx}}
{{- end }}{{/* end get ServiceUnit from serviceUnitName */}}
{{- end }}{{/* end if weight != 0 */}}
{{- end }}{{/* end iterate over services*/}}

{{- with $dynamicConfigManager }}
{{- with $name := $dynamicConfigManager.ServerTemplateName $cfgIdx }}
{{- with $size := $dynamicConfigManager.ServerTemplateSize $cfgIdx }}
server-template {{$name}}- 1-{{$size}} 172.4.0.4:8765 check disabled
{{- end }}
{{- end }}
{{- end }}

{{- end }}{{/*end tls==passthrough*/}}

{{- end }}{{/* end loop over routes */}}
Expand Down Expand Up @@ -564,6 +591,7 @@ backend be_tcp:{{$cfgIdx}}
{{ end -}}{{/* end http host map template */}}



{{/*
os_edge_reencrypt_be.map : contains a mapping of www.example.com -> <service name>. This map is similar to os_http_be.map but for tls routes.
by attaching prefix: be_edge_http for edge terminated routes
Expand All @@ -590,7 +618,7 @@ backend be_tcp:{{$cfgIdx}}

{{/*
os_tcp_be.map: contains a mapping of www.example.com -> <service name>. This map is used to discover the correct backend
by attaching a prefix (be_tcp: or be_secure:) by use_backend statements if acls are matched.
by use_backend statements if acls are matched.
*/}}
{{ define "/var/lib/haproxy/conf/os_tcp_be.map" -}}
{{ range $idx, $line := generateHAProxyMap . -}}
Expand Down
Loading