bitPhile

at-rest-encryption

3 min read

Encryption at REST-API

Kubernetes stores key value pair data inside etcd in plain text format which can be easily comprehended by anyone who has access to to it.

We can enable encryption at kubernetes API that will encrypt the resource data entering into etcd. For example, we can configure to encrypt only Secret resource.

Before enabling encryption

Let’s see how etcd stores secrets data when no api level encryption is enabled. Following describes the steps to do so.

  1. We would need etcdctl client tool installed on our system.
sudo apt install etcd-client
  1. We required to locate the necessary certificate and key files. etcdctl requires ...etcd/ca.crt, ...etcd/server.crt, ...etcd/server.key to get the data from etcd.

In my case, I was using minikube. I could find this files inside control node at path /var/lib/minikube/certs/etcd.

  1. Run the command to get the data from etcd.
ETCDCTL_API=3 etcdctl \
   --cacert=ca.crt   \
   --cert=server.crt \
   --key=server.key  \
   get /registry/secrets/<namespace>/<secret-name>

You may face issues with permissions to access these certificates and key file and provide to command, it’s better to copy paste where there is no permission issue and use that path. (and delete afterwards obviously)

It should show something like below.

/registry/secrets/learning/mysql-db-secret
k8s


v1Secret

mysql-db-secrelearning"*$6866e01b-df74-4a98-9511-d0b52d6773cf2qB
kubectl-createUpdatevFieldsV1:=
;{"f:data":{".":{},"f:PASSWORD":{},"f:USER":{}},"f:type":{}}B
PASSWORDuser

USERuserOpaque"

It clearly shows the data inside secret mysql-db-secret in namespace learning.

That is the problem!

Enable encryption at rest

So we need to enable encryption at kube-apiserver pod. Here we need two things, kube-apiserver.yaml pod file and encryption resource file that will create.

Encryption Resource file

apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
  - resources:
      - secrets
      - configmaps
      - pandas.awesome.bears.example
    providers:
      - aescbc:
          keys:
            - name: key1
              secret: <BASE 64 ENCODED SECRET>
      - identity: {}

resources.resources lists the kubernetes resources we need to enable the encryption for. providers lists the encryption algorithms. First item in providers is used for encryption and rest for fallbacks if any provider is enable to encrypt/decrypt. This is a list of available providers here.

Key secret has to be base64 encoded that we can generated as follows

This is method is for linux.

 head -c 32 /dev/urandom | base64

So Encryption resource file for our case can be like following.

/etc/kubernetes/enc/enc.yaml

apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
  - resources:
      - secrets
    providers:
      - aescbc:
          keys:
            - name: key1
              secret: er+42Uj8deiLT9sUFFwycEqLPW+Egt0EDVTWPsF3MDM=
      - identity: {}

Using Encryption Resource file

We need to use this resource file inside kube-apiserver pod. We can achieve by providing --encryption-provider-config parameter with value as path to encryption resource yaml file, while creating the kube-apiserver pod.

kube-apiserver.yaml can be located at path /etc/kubernetes/manifests/kube-apiserver.yaml.

We need to do two things here.

  1. Provide --encryption-provider-config parameter in command section of containers.

Code snippet of kube-apiserver.yaml.

spec:
  containers:
  - command:
    - kube-apiserver
    - --advertise-address=192.168.76.2
    - --allow-privileged=true
    - --authorization-mode=Node,RBAC
    - --client-ca-file=/var/lib/minikube/certs/ca.crt
    - --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,NodeRestriction,MutatingAdmissionWebhook,Validat
ingAdmissionWebhook,ResourceQuota
    - --enable-bootstrap-token-auth=true
    ..........
    - --tls-cert-file=/var/lib/minikube/certs/apiserver.crt
    - --tls-private-key-file=/var/lib/minikube/certs/apiserver.key
    - --encryption-provider-config=/etc/kubernetes/enc/enc.yaml
  1. Mount the path to pod where it can find /etc/kubernetes/enc/enc.yaml.

Code snippet of kube-apiserver.yaml.

spec:
  containers:
    - ...
     volumeMounts:
        - mountPath: /etc/ssl/certs
          name: ca-certs
          readOnly: true
        ....
        - mountPath: /etc/kubernetes/enc
          name: enc
          readOnly: true
  volumes:
  - hostPath:
      path: /etc/ssl/certs
      type: DirectoryOrCreate
    name: ca-certs
   .....
  - name: enc
    hostPath:
      path: /etc/kubernetes/enc
      type: DirectoryOrCreate

So /etc/kubernetes/enc is mounted to pod path /etc/kubernetes/enc. Encryption resource file should be present at /etc/kubernetes/enc/enc.yaml at host.

Updating the kube-apiserver.yaml would restart the api server automatically and any further creation of secret should be encrypted.

Resources

  1. https://kubernetes.io/docs/tasks/administer-cluster/encrypt-data/

Graph View