OAP setup in OpenShift

Since TSB 0.8.0, the OAP component in control plane listens for Envoy's ALS requests in secure gRPC connections. This improves security as the connections from all Envoy sidecars or gateways in the mesh to the OAP server will be encrypted.

The certificate for the OAP service is created by istiod and issued by the Kubernetes CA. In most Kubernetes distributions you will see that the OAP pod waits for a secret containing the certificate before starting, and that secret is created by istiod on boot.

In OpenShift clusters though, the CA that signs that certificate is not included in the CA certificate bundle mounted in the pod in the /var/run/secrets/kubernetes.io/serviceaccount/ca.crt path. This causes the istiod pod to not be able to verify the newly created certificate which in turn triggers a rollback of the whole process. The result of this being that the OAP pod keeps waiting indefinitely for the secret.

To fix this situation in an OpenShift cluster, apply the following patch to your ControlPlane resource:

spec:
  components:
    istio:
      kubeSpec:
        overlays:
          - apiVersion: install.istio.io/v1alpha1
            kind: IstioOperator
            name: tsb-istiocontrolplane
            patches:
              - path: spec.meshConfig.defaultConfig.envoyAccessLogService.address
                value: oap.istio-system.svc:11800
              - path: spec.meshConfig.defaultConfig.envoyAccessLogService.tlsSettings.caCertificates
                value: /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt
    oap:
      kubeSpec:
        overlays:
          - apiVersion: extensions/v1beta1
            kind: Deployment
            name: oap-deployment
            patches:
              - path: spec.template.spec.containers.[name:oap].env.[name:SW_CORE_GRPC_SSL_CERT_CHAIN_PATH].value
                value: /skywalking/pkin/tls.crt
              - path: spec.template.spec.containers.[name:oap].env.[name:SW_CORE_GRPC_SSL_TRUSTED_CA_PATH].value
                value: /skywalking/pkin/tls.crt
        service:
          annotations:
            service.beta.openshift.io/serving-cert-secret-name: dns.oap-service-account

Istio CNI

The patch above shows overlays for the istio component of the ControlPlane. Note that OpenShift requires the use of Istio CNI so you will need to merge the changes required for the istio component.

The changes above add an annotations to the OAP service, which causes the OpenShift cluster to issue a certificate for that service and store it in the dns.oap-service-account secret. This is what OAP expects, so that will allow the pod to start. And since the keys in the secret that holds the certificate also change from a plain Kubernetes distribution, we need to change the environment variables in the OAP deployment that tells the server where to look for the certificate.