How OpenShift Route works

- How OpenShift uses HAProxy to implement Router and Route

Route in OpenShift solves the need to access services from outside the cluster, similar to Ingress in Kubernetes.

The Route resource API in OpenShift is defined as follows:

apiVersion: v1
kind: Route
metadata:
  name: route-edge-secured
spec:
  host: www.example.com
  to:
    kind: Service
    name: service-name
  tls:
    termination: edge
    key: |-
      -----BEGIN PRIVATE KEY-----
      [...]
      -----END PRIVATE KEY-----      
    certificate: |-
      -----BEGIN CERTIFICATE-----
      [...]
      -----END CERTIFICATE-----      
    caCertificate: |-
      -----BEGIN CERTIFICATE-----
      [...]
      -----END CERTIFICATE-----

Similar to Ingress, you need to specify:

hostThat is, the domain name of the service, that is, the host address
toroute connects to the endpoint behind the service through the service selector
If you want the service to be more secure, you can implement HTTPS by configuring TLS-related certificates

How OpenShift uses HAProxy to implement Router and Route

Router app

First look at the definition of the DeploymentConfig API resource of the Router Pod's controller:

apiVersion: apps.openshift.io/v1
kind: DeploymentConfig
metadata:
  creationTimestamp: null
  generation: 1
  labels:
    router: router
  name: router
  selfLink: /apis/apps.openshift.io/v1/namespaces/default/deploymentconfigs/router
spec:
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    router: router

Each Infra node in the OpenShift cluster starts a Pod named router-n-xxxxx:

$ oc get po -l deploymentconfig=router -n default
NAME             READY     STATUS    RESTARTS   AGE
router-1-8j2pp   1/1       Running   0          8d

We pick a router app container into it:

$ oc exec -it router-1-8j2pp -n default /bin/bash
bash-4.2$ ps -ef
UID         PID   PPID  C STIME TTY          TIME CMD
1000000+      1      0  0 Mar07 ?        01:02:32 /usr/bin/openshift-router
1000000+   3294      1  0 04:26 ?        00:00:21 /usr/sbin/haproxy -f /var/lib/haproxy/conf/haproxy.config -p /var/lib/haproxy/run/haproxy.pid -x /var/lib/haproxy/run/haproxy.so
1000000+   3298      0  0 07:07 ?        00:00:00 /bin/bash
1000000+   3305   3298  0 07:07 ?        00:00:00 ps -ef

The Router app consists of two processes: openshift-router and haproxy.

The complete OpenShift high-availability cluster architecture is shown in the figure above. All business traffic on the cluster (business Pods running in the cluster) will point to the load balancing IP on the right through DNS, and then be fragmented to the Ingra nodes of the cluster through the load balancer in front of the cluster . On all Infra nodes, the HAProxy container will be run in Hostnetwork mode, that is, the process will listen to ports 80 and 443.

$ netstat -lntp | grep haproxy
tcp        0      0 127.0.0.1:10443         0.0.0.0:*               LISTEN      105481/haproxy
tcp        0      0 127.0.0.1:10444         0.0.0.0:*               LISTEN      105481/haproxy
tcp        0      0 0.0.0.0:80              0.0.0.0:*               LISTEN      105481/haproxy
tcp        0      0 0.0.0.0:443             0.0.0.0:*               LISTEN      105481/haproxy

HAProxy here is equivalent to the Ingress controller in Kubernetes. After the business traffic reaches HAProxy through the load balancer at the front end of the cluster, it will be Hostdistributed to the corresponding business back end according to the HTTP request header matching and forwarding rules.

$ curl https://127.0.0.1 -H 'Host: foo.bar.baz'

HAProxy configuration

View the configuration file used by the haproxy process in the container:

global
  maxconn 20000

  daemon
  ca-base /etc/ssl
  crt-base /etc/ssl
  # TODO: Check if we can get reload to be faster by saving server state.
  # server-state-file /var/lib/haproxy/run/haproxy.state
  stats socket /var/lib/haproxy/run/haproxy.sock mode 600 level admin expose-fd listeners
  stats timeout 2m

  # Increase the default request size to be comparable to modern cloud load balancers (ALB: 64kb), affects
  # total memory use when large numbers of connections are open.
  tune.maxrewrite 8192
  tune.bufsize 32768

  # Prevent vulnerability to POODLE attacks
  ssl-default-bind-options no-sslv3

# The default cipher suite can be selected from the three sets recommended by https://wiki.mozilla.org/Security/Server_Side_TLS,
# or the user can provide one using the ROUTER_CIPHERS environment variable.
# By default when a cipher set is not provided, intermediate is used.
  # Intermediate cipher suite (default) from https://wiki.mozilla.org/Security/Server_Side_TLS
  tune.ssl.default-dh-param 2048
  ssl-default-bind-ciphers ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA:ECDHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-RSA-AES256-SHA256:DHE-RSA-AES256-SHA:ECDHE-ECDSA-DES-CBC3-SHA:ECDHE-RSA-DES-CBC3-SHA:EDH-RSA-DES-CBC3-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:DES-CBC3-SHA:!DSS


defaults
  maxconn 20000

  # Add x-forwarded-for header.

  # To configure custom default errors, you can either uncomment the
  # line below (server ... 127.0.0.1:8080) and point it to your custom
  # backend service or alternatively, you can send a custom 503 error.
  #
  # server openshift_backend 127.0.0.1:8080
  errorfile 503 /var/lib/haproxy/conf/error-page-503.http

  timeout connect 5s
  timeout client 30s
  timeout client-fin 1s
  timeout server 30s
  timeout server-fin 1s
  timeout http-request 10s
  timeout http-keep-alive 300s

  # Long timeout for WebSocket connections.
  timeout tunnel 1h

frontend public

  bind :80
  mode http
  tcp-request inspect-delay 5s
  tcp-request content accept if HTTP
  monitor-uri /_______internal_router_healthz

  # Strip off Proxy headers to prevent HTTpoxy (https://httpoxy.org/)
  http-request del-header Proxy

  # DNS labels are case insensitive (RFC 4343), we need to convert the hostname into lowercase
  # before matching, or any requests containing uppercase characters will never match.
  http-request set-header Host %[req.hdr(Host),lower]

  # check if we need to redirect/force using https.
  acl secure_redirect base,map_reg(/var/lib/haproxy/conf/os_route_http_redirect.map) -m found
  redirect scheme https if secure_redirect

  use_backend %[base,map_reg(/var/lib/haproxy/conf/os_http_be.map)]

  default_backend openshift_default

# public ssl accepts all connections and isn't checking certificates yet certificates to use will be
# determined by the next backend in the chain which may be an app backend (passthrough termination) or a backend
# that terminates encryption in this router (edge)
frontend public_ssl

  bind :443
  tcp-request  inspect-delay 5s
  tcp-request content accept if { req_ssl_hello_type 1 }

  # if the connection is SNI and the route is a passthrough don't use the termination backend, just use the tcp backend
  # for the SNI case, we also need to compare it in case-insensitive mode (by converting it to lowercase) as RFC 4343 says
  acl sni req.ssl_sni -m found
  acl sni_passthrough req.ssl_sni,lower,map_reg(/var/lib/haproxy/conf/os_sni_passthrough.map) -m found
  use_backend %[req.ssl_sni,lower,map_reg(/var/lib/haproxy/conf/os_tcp_be.map)] if sni sni_passthrough

  # if the route is SNI and NOT passthrough enter the termination flow
  use_backend be_sni if sni

  # non SNI requests should enter a default termination backend rather than the custom cert SNI backend since it
  # will not be able to match a cert to an SNI host
  default_backend be_no_sni

##########################################################################
# TLS SNI
#
# When using SNI we can terminate encryption with custom certificates.
# Certs will be stored in a directory and will be matched with the SNI host header
# which must exist in the CN of the certificate.  Certificates must be concatenated
# as a single file (handled by the plugin writer) per the haproxy documentation.
#
# Finally, check re-encryption settings and re-encrypt or just pass along the unencrypted
# traffic
##########################################################################
backend be_sni
  server fe_sni 127.0.0.1:10444 weight 1 send-proxy

frontend fe_sni
  # terminate ssl on edge
  bind 127.0.0.1:10444 ssl no-sslv3 crt /etc/pki/tls/private/tls.crt crt-list /var/lib/haproxy/conf/cert_config.map accept-proxy
  mode http

  # Strip off Proxy headers to prevent HTTpoxy (https://httpoxy.org/)
  http-request del-header Proxy

  # DNS labels are case insensitive (RFC 4343), we need to convert the hostname into lowercase
  # before matching, or any requests containing uppercase characters will never match.
  http-request set-header Host %[req.hdr(Host),lower]

  # map to backend
  # Search from most specific to general path (host case).
  # Note: If no match, haproxy uses the default_backend, no other
  #       use_backend directives below this will be processed.
  use_backend %[base,map_reg(/var/lib/haproxy/conf/os_edge_reencrypt_be.map)]

  default_backend openshift_default

##########################################################################
# END TLS SNI
##########################################################################

##########################################################################
# TLS NO SNI
#
# When we don't have SNI the only thing we can try to do is terminate the encryption
# using our wild card certificate.  Once that is complete we can either re-encrypt
# the traffic or pass it on to the backends
##########################################################################
# backend for when sni does not exist, or ssl term needs to happen on the edge
backend be_no_sni
  server fe_no_sni 127.0.0.1:10443 weight 1 send-proxy

frontend fe_no_sni
  # terminate ssl on edge
  bind 127.0.0.1:10443 ssl no-sslv3 crt /etc/pki/tls/private/tls.crt accept-proxy
  mode http

  # Strip off Proxy headers to prevent HTTpoxy (https://httpoxy.org/)
  http-request del-header Proxy

  # DNS labels are case insensitive (RFC 4343), we need to convert the hostname into lowercase
  # before matching, or any requests containing uppercase characters will never match.
  http-request set-header Host %[req.hdr(Host),lower]

  # map to backend
  # Search from most specific to general path (host case).
  # Note: If no match, haproxy uses the default_backend, no other
  #       use_backend directives below this will be processed.
  use_backend %[base,map_reg(/var/lib/haproxy/conf/os_edge_reencrypt_be.map)]

  default_backend openshift_default

##########################################################################
# END TLS NO SNI
##########################################################################

backend openshift_default
  mode http
  option forwardfor
  #option http-keep-alive
  option http-pretend-keepalive

##-------------- app level backends ----------------

# Secure backend, pass through
backend be_tcp:default:docker-registry
  balance source

  hash-type consistent
  timeout check 5000ms}
  server pod:docker-registry-1-tvrjp:docker-registry:10.128.1.32:5000 10.128.1.32:5000 weight 256

# Secure backend, pass through
backend be_tcp:default:registry-console
  balance source

  hash-type consistent
  timeout check 5000ms}
  server pod:registry-console-1-jspmx:registry-console:10.128.1.10:9090 10.128.1.10:9090 weight 256

# Plain http backend or backend with TLS terminated at the edge or a
# secure backend with re-encryption.
backend be_edge_http:default:route-origin-license-information-provider
  mode http
  option redispatch
  option forwardfor
  balance leastconn

  timeout check 5000ms
  http-request set-header X-Forwarded-Host %[req.hdr(host)]
  http-request set-header X-Forwarded-Port %[dst_port]
  http-request set-header X-Forwarded-Proto http if !{ ssl_fc }
  http-request set-header X-Forwarded-Proto https if { ssl_fc }
  http-request set-header X-Forwarded-Proto-Version h2 if { ssl_fc_alpn -i h2 }
  http-request add-header Forwarded for=%[src];host=%[req.hdr(host)];proto=%[req.hdr(X-Forwarded-Proto)];proto-version=%[req.hdr(X-Forwarded-Proto-Version)]
  cookie f68e01f4fd193411edcdbd2ed2123b1b insert indirect nocache httponly secure
  server pod:deployment-origin-license-information-provider-7b55689655-vcn2j:svc-origin-license-information-provider:10.128.1.1:80 10.128.1.1:80 cookie d000e3ba84d0860ea1f900879572c5fa weight 256

# Secure backend, pass through
backend be_tcp:kube-service-catalog:apiserver
  balance source

  hash-type consistent
  timeout check 5000ms}
  server pod:apiserver-w4szj:apiserver:10.128.1.17:6443 10.128.1.17:6443 weight 256

The configuration of HAProxy consists of several parts:

Globally configure the maximum number of connections, etc.
The front-end configuration listens for HTTP requests on port 80 and HTTPS requests on port 443
The backend configuration includes the protocol (mode), load balancing method, request header rewriting, and the internal address of the OpenShift cluster of the backend service

Following this configuration file, we can roughly analyze how haproxy handles business traffic:

1. HTTP

frontend public

  bind :80
  mode http
  tcp-request inspect-delay 5s
  tcp-request content accept if HTTP
  monitor-uri /_______internal_router_healthz

  # Strip off Proxy headers to prevent HTTpoxy (https://httpoxy.org/)
  http-request del-header Proxy

  # DNS labels are case insensitive (RFC 4343), we need to convert the hostname into lowercase
  # before matching, or any requests containing uppercase characters will never match.
  http-request set-header Host %[req.hdr(Host),lower]

  # check if we need to redirect/force using https.
  acl secure_redirect base,map_reg(/var/lib/haproxy/conf/os_route_http_redirect.map) -m found
  redirect scheme https if secure_redirect

  use_backend %[base,map_reg(/var/lib/haproxy/conf/os_http_be.map)]

  default_backend openshift_default

When haproxy receives an HTTP (port 80) request, it will forward it to the corresponding backend according to the forwarding rules in the /var/lib/haproxy/conf/os_http_be.map file.

^backend-3scale.caas-aio-apps.trystack.cn(:[0-9]+)?(/.*)?$ be_edge_http:mep-apigateway:backend

The backend then forwards the traffic to the corresponding service Pod according to the Pod address inside the cluster in the configuration, and from here it goes through the Kubernetes internal network.

# Plain http backend or backend with TLS terminated at the edge or a
# secure backend with re-encryption.
backend be_edge_http:mep-apigateway:backend
  mode http
  option redispatch
  option forwardfor
  balance leastconn

  timeout check 5000ms
  http-request set-header X-Forwarded-Host %[req.hdr(host)]
  http-request set-header X-Forwarded-Port %[dst_port]
  http-request set-header X-Forwarded-Proto http if !{ ssl_fc }
  http-request set-header X-Forwarded-Proto https if { ssl_fc }
  http-request set-header X-Forwarded-Proto-Version h2 if { ssl_fc_alpn -i h2 }
  http-request add-header Forwarded for=%[src];host=%[req.hdr(host)];proto=%[req.hdr(X-Forwarded-Proto)];proto-version=%[req.hdr(X-Forwarded-Proto-Version)]
  cookie dcdcad015a918c4295da5aec7a0daf50 insert indirect nocache httponly
  server pod:backend-listener-1-mwgb6:backend-listener:10.128.1.67:3000 10.128.1.67:3000 cookie 4a624e547781196b55877322edeaec05 weight 256

2. HTTPS

frontend public_ssl

  bind :443
  tcp-request  inspect-delay 5s
  tcp-request content accept if { req_ssl_hello_type 1 }

  # if the connection is SNI and the route is a passthrough don't use the termination backend, just use the tcp backend
  # for the SNI case, we also need to compare it in case-insensitive mode (by converting it to lowercase) as RFC 4343 says
  acl sni req.ssl_sni -m found
  acl sni_passthrough req.ssl_sni,lower,map_reg(/var/lib/haproxy/conf/os_sni_passthrough.map) -m found
  use_backend %[req.ssl_sni,lower,map_reg(/var/lib/haproxy/conf/os_tcp_be.map)] if sni sni_passthrough

  # if the route is SNI and NOT passthrough enter the termination flow
  use_backend be_sni if sni

  # non SNI requests should enter a default termination backend rather than the custom cert SNI backend since it
  # will not be able to match a cert to an SNI host
  default_backend be_no_sni

When haproxy receives an HTTPS (port 443) request, it will check whether the HTTPS request supports sni ( TLS Server Name Indication )

The processing here corresponds to how the Router terminates HTTPS traffic:

Edge HTTPS traffic will be decrypted in the Router and forwarded to the backend Pod in plain HTTP
The passthrough TLS encrypted package is directly delivered to the backend Pod, and the Router does not perform any TLS termination, and does not require TLS-related certificates
re-encryption is a variant of edge, HTTPS traffic will be decrypted in the Router and then encrypted with other certificates and forwarded to the backend Pod

That is the field in the route API termination.

The be_sni backend will be used when the TLS termination type is not passthrough:

backend be_sni
  server fe_sni 127.0.0.1:10444 weight 1 send-proxy

A frontend named fe_sni is bound to port 10444 and traffic is redirected to fe_sni:

frontend fe_sni
  # terminate ssl on edge
  bind 127.0.0.1:10444 ssl no-sslv3 crt /etc/pki/tls/private/tls.crt crt-list /var/lib/haproxy/conf/cert_config.map accept-proxy
  mode http

  # Strip off Proxy headers to prevent HTTpoxy (https://httpoxy.org/)
  http-request del-header Proxy

  # DNS labels are case insensitive (RFC 4343), we need to convert the hostname into lowercase
  # before matching, or any requests containing uppercase characters will never match.
  http-request set-header Host %[req.hdr(Host),lower]

  # map to backend
  # Search from most specific to general path (host case).
  # Note: If no match, haproxy uses the default_backend, no other
  #       use_backend directives below this will be processed.
  use_backend %[base,map_reg(/var/lib/haproxy/conf/os_edge_reencrypt_be.map)]

  default_backend openshift_default

After HTTPS traffic comes here, it will be decrypted with the relevant TLS certificate, and then forwarded to the corresponding backend according to the forwarding rules in the /var/lib/haproxy/conf/os_edge_reencrypt_be.map mapping file.

^user-management.caas-aio-apps.trystack.cn(:[0-9]+)?(/.*)?$ be_edge_http:user-management:user-management
^test-groupcbc37148-3scale-apicast-staging.caas-aio-apps.trystack.cn(:[0-9]+)?(/.*)?$ be_edge_http:mep-apigateway:zync-3scale-api-sp8kr
^test-groupcbc37148-3scale-apicast-production.caas-aio-apps.trystack.cn(:[0-9]+)?(/.*)?$ be_edge_http:mep-apigateway:zync-3scale-api-twdpk
^test-group26a9a65d-3scale-apicast-staging.caas-aio-apps.trystack.cn(:[0-9]+)?(/.*)?$ be_edge_http:mep-apigateway:zync-3scale-api-r7rwm
^test-group26a9a65d-3scale-apicast-production.caas-aio-apps.trystack.cn(:[0-9]+)?(/.*)?$

It is exactly the same as HTTP, and the backend forwards the traffic to the corresponding service Pod according to the pod address inside the cluster in the configuration, and from here it goes through the Kubernetes internal network.

The essence of creating/deleting Route API resources is to modify the configuration of HAProxy to control the direction of traffic. The seven-layer load balancing capability of OpenShift Router is also provided by HAProxy on the Infra node.

Similar to Ingress, the service selector of OpenShift Route is only used to add relevant backend Pods to the HAProxy configuration, but in actual use, the Kubernetes Service layer is skipped to avoid the performance impact caused by iptables.