Skip to main content
Version: 1.5.x

Configuring Application Gateways Using OpenAPI Annotations

An Application in TSB represents a set of logical groupings of Services that are related to each other and expose a set of APIs that implement a complete set of business logic.

TSB can leverage OpenAPI annotations when configuring API runtime policies. In this document you will enable authorization via Open Policy Agent (OPA), as well as rate limiting through an external service. Each request will need to go through basic authorization, and for each valid user a rate limit policy will be enforced.

Before you get started, make sure you:
✓ Familiarize yourself with TSB concepts
✓ Familiarize yourself Open Policy Agent (OPA)
✓ Familiarize yourself Envoy external authorization and rate limit
✓ Install the TSB demo environment
✓ Deploy the Istio Bookinfo sample app
✓ Create a Tenant

Deploy httpbin Service

Follow the instructions in this document to create the httpbin service. Complete all steps in that document.

TSB Specific Annotations

The following extra TSB-specific annotations can be added to the OpenAPI specification in order to configure the API.

AnnotationDescription
x-tsb-serviceThe upstream service name in TSB that provides the API, as seen in the TSB service registry (you can check with tctl get services).
x-tsb-corsThe CORS policy for the server.
x-tsb-tlsThe TLS settings for the server. If omitted, the server will be configured to serve plain text connections. The secretName field must point to an existing Kubernetes secret in the cluster.
x-tsb-external-authorizationThe OPA settings for the server.
x-tsb-ratelimitingThe external rate limit server (e.g. envoyproxy/ratelimit settings.

Configure the API

Create the following API definition in a file called httpbin-api.yaml.

In this scenario you will only use one of the APIs (/get) that the httpbin service provides. If you want to use all of the httpbin API you can get their OpenAPI specifications from this link.

apiversion: application.tsb.tetrate.io/v2
kind: API
metadata:
organization: <organization>
tenant: <tenant>
application: httpbin
name: httpbin-api
spec:
description: Httpbin OpenAPI
workloadSelector:
namespace: httpbin
labels:
app: httpbin-gateway
openapi: |
openapi: 3.0.1
info:
version: '1.0-oas3'
title: httpbin
description: An unofficial OpenAPI definition for httpbin
x-tsb-service: httpbin.httpbin

servers:
- url: https://httpbin.tetrate.com
x-tsb-cors:
allowOrigin:
- "*"
x-tsb-tls:
mode: SIMPLE
secretName: httpbin-certs
paths:
/get:
get:
tags:
- HTTP methods
summary: |
Returns the GET request's data.
responses:
'200':
description: OK
content:
application/json:
schema:
type: object

Apply using tctl:

tctl apply -f httpbin-api.yaml

At this point, you should be able to send requests to the httpbin Ingress Gateway.

Since you do not control httpbin.tetrate.com, you will have to trick curl into thinking that httpbin.tetrate.com resolves to the IP address of the Ingress Gateway.

Obtain the IP address of the Ingress Gateway that you previously created using the following command.

kubectl -n httpbin get service httpbin-gateway -o jsonpath='{.status.loadBalancer.ingress[0].ip}'

Execute the following command to send HTTP requests to the httpbin service through the Tier-1 Gateway. Replace the gateway-ip with the value you obtained in the previous step. You also need to pass the CA cert, which you should have created in the step to deploy the httpbin service.

curl -I "https://httpbin.tetrate.com/get" \
--resolve "httpbin.tetrate.com:443:<gateway-ip>" \
--cacert httpbin.crt

You should see a successful HTTP response.

Authorization with OPA

Once the API is properly exposed via OpenAPI annotations, OPA can be configured against the API Gateway.

In this example you will create a policy that checks for basic authentication in the request header. If the user is authenticated, the user name should be added to the x-user header so that it can be used by the rate limit service to enforce quota for each user later.

Configure OPA

Create the opa namespace, where the OPA and its configuration will be deployed to:

kubectl create namespace opa

Create a file name openapi-policy.rego:

package demo.authz

default allow = false

# username and password database
user_passwords = {
"alice": "password",
"bob": "password"
}

allow = response {
# check if password from header is same as in database for the specific user
basic_auth.password == user_passwords[basic_auth.user_name]
response := {
"allowed": true,
"headers": {"x-user": basic_auth.user_name}
}
}

basic_auth := {"user_name": user_name, "password": password} {
v := input.attributes.request.http.headers.authorization
startswith(v, "Basic ")
s := substring(v, count("Basic "), -1)
[user_name, password] := split(base64url.decode(s), ":")
}

Then create a ConfigMap using the file you created:

kubectl -n opa create configmap opa-policy \
--from-file=openapi-policy.rego

Create the Deployment and the Service objects that use the above policy configuration in a file named opa.yaml.

apiVersion: v1
kind: Service
metadata:
name: opa
namespace: opa
spec:
selector:
app: opa
ports:
- name: grpc
protocol: TCP
port: 9191
targetPort: 9191
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: opa
namespace: opa
spec:
replicas: 1
selector:
matchLabels:
app: opa
template:
metadata:
labels:
app: opa
name: opa
spec:
containers:
- image: openpolicyagent/opa:0.29.4-envoy-2
name: opa
securityContext:
runAsUser: 1111
ports:
- containerPort: 8181
args:
- "run"
- "--server"
- "--addr=localhost:8181"
- "--diagnostic-addr=0.0.0.0:8282"
- "--set=plugins.envoy_ext_authz_grpc.addr=:9191"
- "--set=plugins.envoy_ext_authz_grpc.path=demo/authz/allow"
- "--set=decision_logs.console=true"
- "--ignore=.*"
- "/policy/openapi-policy.rego"
livenessProbe:
httpGet:
path: /health?plugins
scheme: HTTP
port: 8282
initialDelaySeconds: 5
periodSeconds: 5
readinessProbe:
httpGet:
path: /health?plugins
scheme: HTTP
port: 8282
initialDelaySeconds: 5
periodSeconds: 5
volumeMounts:
- readOnly: true
mountPath: /policy
name: opa-policy
volumes:
- name: opa-policy
configMap:
name: opa-policy

Then apply the manifest:

kubectl apply -f opa.yaml

Finally, open the httpbin-api.yaml file that you created in a previous section, and add the x-tsb-external-authorization annotation in the server component:

    ...
servers:
- url: https://httpbin.tetrate.com
...
x-tsb-external-authorization:
uri: grpc://opa.opa.svc.cluster.local:9191

And apply the changes again:

tctl apply -f httpbin-api.yaml

Testing

To test, execute the following command, replacing the values for username, password, and gateway-ip accordingly.

curl -u <username>:<password> \
"https://httpbin.tetrate.com/get" \
--resolve "httpbin.tetrate.com:443:<gateway-ip>" \
--cacert httpbin.crt
-s \
-o /dev/null \
-w "%{http_code}\n"
UsernamePasswordStatus Code
alicepassword200
bobpassword200
<anything else><anything else>403 (*1)

(*1) See documentation for more details

Rate limiting with external service

TSB supports internal and external mode for rate limiting. In this example you will deploy a separate Envoy rate limit service.

Configure Rate Limit

Create the ext-ratelimit namespace, where the rate limit server and its configuration will be deployed to:

kubectl create namespace ext-ratelimit

Create a file name ext-ratelimit-config.yaml. This configuration specifies that the user alice should be rate limited to 10 requests/minute and user bob should be limited to 2 requests/minute.

domain: httpbin-ratelimit
descriptors:
- key: x-user-descriptor
value: alice
rate_limit:
unit: minute
requests_per_unit: 10
- key: x-user-descriptor
value: bob
rate_limit:
unit: minute
requests_per_unit: 2

Then create a ConfigMap using the file you created:

kubectl -n ext-ratelimit create configmap ext-ratelimit \
--from-file=config.yaml=ext-ratelimit-config.yaml

You now need to deploy Redis and envoyproxy/ratelimit. Create a file called redis-ratelimit.yaml with the following contents:

# Copyright Istio Authors
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

####################################################################################
# Redis service and deployment
# Ratelimit service and deployment
####################################################################################
apiVersion: v1
kind: Service
metadata:
name: redis
namespace: ext-ratelimit
labels:
app: redis
spec:
ports:
- name: redis
port: 6379
selector:
app: redis
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: redis
namespace: ext-ratelimit
spec:
replicas: 1
selector:
matchLabels:
app: redis
template:
metadata:
labels:
app: redis
spec:
containers:
- image: redis:alpine
imagePullPolicy: Always
name: redis
ports:
- name: redis
containerPort: 6379
restartPolicy: Always
serviceAccountName: ""
---
apiVersion: v1
kind: Service
metadata:
name: ratelimit
namespace: ext-ratelimit
labels:
app: ratelimit
spec:
ports:
- name: http-port
port: 8080
targetPort: 8080
protocol: TCP
- name: grpc-port
port: 8081
targetPort: 8081
protocol: TCP
- name: http-debug
port: 6070
targetPort: 6070
protocol: TCP
selector:
app: ratelimit
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: ratelimit
namespace: ext-ratelimit
spec:
replicas: 1
selector:
matchLabels:
app: ratelimit
strategy:
type: Recreate
template:
metadata:
labels:
app: ratelimit
spec:
containers:
- image: envoyproxy/ratelimit:6f5de117 # 2021/01/08
imagePullPolicy: Always
name: ratelimit
command: ["/bin/ratelimit"]
env:
- name: LOG_LEVEL
value: debug
- name: REDIS_SOCKET_TYPE
value: tcp
- name: REDIS_URL
value: redis:6379
- name: USE_STATSD
value: "false"
- name: RUNTIME_ROOT
value: /data
- name: RUNTIME_SUBDIRECTORY
value: ratelimit
ports:
- containerPort: 8080
- containerPort: 8081
- containerPort: 6070
volumeMounts:
- name: config-volume
mountPath: /data/ratelimit/config/config.yaml
subPath: config.yaml
volumes:
- name: config-volume
configMap:
name: ratelimit-config

If everything is successful, you should have a working rate limit server. The next step is to add an annotation for x-tsb-ratelimiting to the OpenAPI object:

Next, update your OpenAPI spec by adding the following x-tsb-ratelimiting annotation in OpenAPI server object

...
servers:
- url: https://httpbin.tetrate.com
...
x-tsb-external-ratelimiting:
domain: "httpbin-ratelimit"
rateLimitServerUri: "grpc://ratelimit.ext-ratelimit.svc.cluster.local:8081"
rules:
- dimensions:
- requestHeaders:
headerName: x-user
descriptorKey: x-user-descriptor

...

Testing

To test, execute the following command, replacing the values for username, password, and gateway-ip accordingly.

curl -u <username>:<password> \
"https://httpbin.tetrate.com/get" \
--resolve "httpbin.tetrate.com:443:<gateway-ip>" \
--cacert httpbin.crt
-s \
-o /dev/null \
-w "%{http_code}\n"

First, try sending multiple requests using the username alice and password password. You should receive the status code 200 until the 10th request. After that you should receive 429 responses, until 10 minutes have passed.

Try the same with username bob and password password. The behavior should be identical, except that this time you should only be able to send 2 requests before starting to receive 429 responses.

Policy ordering

TSB does not currently support specifying explicit policy order.

Instead, the creation timestamp for the configurations will implicitly be used. Therefore, the execution order cannot be guaranteed if you specify the external authorization and rate limiting services in one go.

This is why in this document the external authorization and rate limiting configurations were applied in two separate steps, in this specific order. This way the authorization processing is performed before the rate limiting.