Skip to main content
Version: 1.6.x

External Rate Limiting with TLS Verification

Once you have configured an external rate limit server, you may want to secure the traffic to the rate limit service. TSB supports specifying TLS or mTLS parameters for securing communication to external rate limit servers. This document will show you how to configure TLS validation for an external rate limit server by adding CA certificate to the rate limiting configuration.

Before you get started, make sure you:
✓ Familiarize yourself with TSB concepts
✓ Install the TSB environment. You can use TSB demo for quick install
✓ Completed TSB usage quickstart. This document assumes you already created Tenant and are familiar with Workspace and Config Groups. Also you need to configure tctl to your TSB environment.
✓ Competed Setting Up an External Rate Limiting Server. This document will continue what you have done in Setting Up an External Rate Limiting Server. You will work in ext-ratelimit namespace, and should already have an Ingress Gateway with external rate limit properly configured

TLS certificate

To enable TLS for Ingress Gateway to rate limit service traffic, you must have a TLS certificate. This document assumes you already have TLS certificates which usually include server certificate and private key along with the CA as root certificate that will be used by the client.

This document assumes the presence of the following files. If you are using different file names, please change them accordingly:

File nameDescription
ratelimit.crtThe server certificate
ratelimit.keyThe certificate private key
ratelimit-ca.crtThe CA certificate
self signed certificate

For the purpose of example, you may opt to use a self-signed certificate. You may generate a self-signed certificate using the script show here, but make sure to adjust the input parameters accordingly.

Once you have the certificate files, create Kubernetes secret using server certificate and private key.

kubectl create secret tls -n ext-ratelimit ratelimit-certs \
--cert=ratelimit.crt \
--key=ratelimit.key

Deploy Rate Limit Service with TLS certificate

In this example you will use the Envoy rate limit service. The Envoy proxy sidecar acts as pass through proxy that will validate and terminate TLS before sending the request to the rate limit service.

Create a configuration file for Envoy with the following content as proxy-config-tls.yaml

# Envoyproxy configuration for different modes of TLS termination
# of external rate-limit server used for testing purposes

# Summary of various ports
#
# 10250 - admin port
# 18080 - insecure GRPC
# 18443 - TLS GRPC

admin:
address:
socket_address:
address: 127.0.0.1
port_value: 10250

static_resources:
listeners:
# Insecure GRPC listener
- name: grpc-insecure
address:
socket_address:
address: 0.0.0.0
port_value: 18080
access_log:
- name: envoy.access_loggers.file
typed_config:
"@type": type.googleapis.com/envoy.extensions.access_loggers.file.v3.FileAccessLog
path: /dev/stdout
filter_chains:
- filters:
- name: envoy.filters.network.tcp_proxy
typed_config:
"@type": type.googleapis.com/envoy.extensions.filters.network.tcp_proxy.v3.TcpProxy
cluster: grpc_rlserver
stat_prefix: grpc_insecure

# Secured by TLS
- name: grpc-simple-tls
address:
socket_address:
address: 0.0.0.0
port_value: 18443
access_log:
- name: envoy.access_loggers.file
typed_config:
"@type": type.googleapis.com/envoy.extensions.access_loggers.file.v3.FileAccessLog
path: /dev/stdout
filter_chains:
- filters:
- name: envoy.filters.network.tcp_proxy
typed_config:
"@type": type.googleapis.com/envoy.extensions.filters.network.tcp_proxy.v3.TcpProxy
cluster: grpc_rlserver
stat_prefix: grpc_simple_tls
transport_socket:
name: envoy.transport_sockets.tls
typed_config:
"@type": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.DownstreamTlsContext
common_tls_context:
tls_certificates:
- certificate_chain: { filename: /certs/tls.crt }
private_key: { filename: /certs/tls.key }
clusters:
- name: grpc_rlserver
type: STRICT_DNS
connect_timeout: 0.5s
dns_refresh_rate: 3600s
typed_extension_protocol_options:
envoy.extensions.upstreams.http.v3.HttpProtocolOptions:
"@type": type.googleapis.com/envoy.extensions.upstreams.http.v3.HttpProtocolOptions
explicit_http_config:
http2_protocol_options: {}
load_assignment:
cluster_name: grpc_rlserver
endpoints:
- lb_endpoints:
- endpoint:
address:
socket_address:
address: 127.0.0.1
port_value: 8081

Execute the following to store the configuration in Kubernetes as a ConfigMap.

kubectl create configmap -n ext-ratelimit ratelimit-proxy \
--from-file=proxy-config-tls.yaml

You will need to deploy the rate limit service with an Envoy sidecar to terminate TLS. Create a file called ratelimit-tls.yaml with the following content.

kind: Service
metadata:
name: ratelimit-tls
namespace: ext-ratelimit
labels:
app: ratelimit-tls
spec:
ports:
- name: http
port: 8080
targetPort: 8080 # Doesn't go through Envoy
- name: grpc-insecure
port: 18080
targetPort: 18080
- name: grpc-tls
port: 18443
targetPort: 18443
selector:
app: ratelimit-tls
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: ratelimit-tls
namespace: ext-ratelimit
spec:
replicas: 1
selector:
matchLabels:
app: ratelimit-tls
strategy:
type: Recreate
template:
metadata:
labels:
app: ratelimit-tls
spec:
containers:
- name: envoy-proxy
image: envoyproxy/envoy-alpine:v1.18.4
imagePullPolicy: Always
command:
- "/usr/local/bin/envoy"
args:
- "--config-path /etc/envoy/config.yaml"
- "--mode serve"
- "--service-cluster ext-ratelimit"
ports:
- name: grpc-plaintext
containerPort: 18080
- name: grpc-tls
containerPort: 18443
volumeMounts:
- name: proxy-config
mountPath: /etc/envoy
- name: proxy-certs
mountPath: /certs
- name: ratelimit
image: envoyproxy/ratelimit:6f5de117 # 2021/01/08
imagePullPolicy: Always
command: ["/bin/ratelimit"]
env:
- name: LOG_LEVEL
value: debug
- name: REDIS_SOCKET_TYPE
value: tcp
- name: REDIS_URL
value: redis:6379
- name: USE_STATSD
value: "false"
- name: RUNTIME_ROOT
value: /data
- name: RUNTIME_SUBDIRECTORY
value: ratelimit
ports:
- containerPort: 8080
- containerPort: 8081
- containerPort: 6070
volumeMounts:
- name: ratelimit-config
mountPath: /data/ratelimit/config/config.yaml
subPath: config.yaml
volumes:
- name: ratelimit-config
configMap:
name: ratelimit-config
- name: proxy-certs
secret:
secretName: ratelimit-certs
- name: proxy-config
configMap:
name: ratelimit-proxy

Then apply this using kubectl:

kubectl apply -f ratelimit-tls.yaml

Once you applied the new configuration, make sure that the ratelimit-tls service is running properly. Note that if you have followed the instructions from Setting Up an External Rate Limiting Server, you will also see ratelimit and redis services as well.

kubectl get pods -n ext-ratelimit

NAME READY STATUS RESTARTS AGE
ratelimit-d5c5b64ff-m87dt 1/1 Running 0 2h
ratelimit-tls-568c5cdc69-z82xf 2/2 Running 0 89s
redis-7d757c948f-42sxg 1/1 Running 0 2h

Enable TLS validation for rate limit server in Ingress Gateway

The ratelimit-tls service can now terminate TLS, but the Ingress Gateway must also be configured to validate the TLS connections.

First, create a ConfigMap named ratelimit-ca to store the CA information from ratelimit-ca.crt:

kubectl create configmap -n httpbin ratelimit-ca \
--from-file=ratelimit-ca.crt

Then add the ratelimit-ca ConfigMap into the Ingress Gateway pod. To do this, you will need to edit the httpbin-ingress-gateway.yaml file and add an overlay that reads the ConfigMap you have created in the previous steps, then mount the configuration in the ingress gateway deployment.

apiVersion: install.tetrate.io/v1alpha1
kind: IngressGateway
metadata:
name: httpbin-ingress-gateway
namespace: httpbin
spec:
kubeSpec:
service:
type: LoadBalancer
overlays:
- apiVersion: apps/v1
kind: Deployment
name: httpbin-ingress-gateway
patches:
- path: spec.template.spec.volumes[-1]
value:
name: ratelimit-ca
configMap:
name: ratelimit-ca
- path: spec.template.spec.containers.[name:istio-proxy].volumeMounts[-1]
value:
name: ratelimit-ca
mountPath: /etc/certs
readOnly: true

Apply with kubectl to update existing ingress gateway

kubectl apply -f httpbin-ingress-gateway.yaml

Finally, update the Ingress Gateway configuration in ext-ratelimit-ingress-gateway.yaml and enable TLS validation:

apiVersion: gateway.tsb.tetrate.io/v2
kind: IngressGateway
metadata:
name: httpbin-gateway # Need not be the same as spec.labels.app
organization: tetrate
tenant: tetrate
group: httpbin-gateway
workspace: httpbin
spec:
workloadSelector:
namespace: httpbin
labels:
app: httpbin-ingress-gateway # name of Ingress Gateway created for httpbin
http:
- name: httpbin
hostname: "httpbin.tetrate.com"
port: 80
routing:
rules:
- route:
host: "httpbin/httpbin.httpbin.svc.cluster.local"
port: 8000
rateLimiting:
externalService:
domain: "httpbin-ratelimit"
rateLimitServerUri: "grpcs://ratelimit-tls.ext-ratelimit.svc.cluster.local:18443"
failClosed: true
tls:
mode: SIMPLE
files:
caCertificates: /etc/certs/ratelimit-ca.crt
rules:
- dimensions:
- requestHeaders:
headerName: ":path"
descriptorKey: request-path

And apply with tctl

tctl apply -f ext-ratelimit-ingress-gateway-tls.yaml

Testing

To verify that the setup is working, you can use the same testing steps as shown in the Testing steps for "Setting Up an External Rate Limiting Server