Secrets Handling in Kubernetes

In your deployments locally or through Docker, you will most definitely be using a .env file for all your secrets - API keys, database credentials, and the like. In the world of Kubernetes, nothing is as simple as just a .env file. In fact, configuration of your deployments will be done principally through ConfigMaps and Secrets.

The idea is that general config (non-sensitive) will land in ConfigMaps which can be easily "mounted" into running pods/containers as environment variables. On the flip side, Secrets are the Kubernetes object which are supposed to hold all the sensitive data like keys and credentials. A secret would look something like this:

apiVersion: v1
data:
  MYSQL_DB:
  MYSQL_PASSWORD:
  MYSQL_USER:
kind: Secret
metadata:
  name: dbsecret
  namespace: default
type: Opaque

Here's the catch: your secrets are kept as base64 encoding. Not encrypted.

You, as the reader, must be wondering what the $%!# is going on, and a lot of people think that too. Even the Kubernetes docs think that. Here's an extract:

Caution:
Kubernetes Secrets are, by default, stored unencrypted in the API server's underlying data store (etcd). Anyone with API access can retrieve or modify a Secret, and so can anyone with access to etcd. Additionally, anyone who is authorized to create a Pod in a namespace can use that access to read any Secret in that namespace; this includes indirect access such as the ability to create a Deployment.

And this is correct. Even if you don't have outright access to secrets in Kubernetes, you might still be able to make a pod, mount the secret, and just echo $secret. Thankfully, Kubernetes also provides the documentation to remediate this.
It can be summarised as: use RBAC to prevent access, restrict secret access for pods, and enabling encryption at rest.

Secrets Managers

Possibly the most used one though is an external secrets manager. Similar to how you keep your passwords in a passwords manager, a secrets manager will do the same thing. Your secrets will be encrypted within your secrets manager, yet will be decrypted and passed into the pods when they are run. For example, Hashicorp provides Vault CSI provider which runs inside your cluster and interfaces between the Vault and your pods.

However.

I understand that for certain use cases, the above use cases are useful and necessary. However, maybe it's just that I don't have enough teammates to have to bother about secrets management (my homelab is just me, and at work is just me and Ish) but I have never needed them. The best way of keeping your secrets secure in Kubernetes, just the same as keeping your secrets safe in traditional servers (where you have .env files), is by preventing people from accessing your servers!

Simply, where you draw the line is fairly arbitrary. If you put your .env file on your server, then you need to make sure you don't have unauthorised access to your server. If you decide to put your secrets in your github repo, then you need to make sure nobody who you don't want seeing the secrets has access to the repo in the first place!

Hence, the only case I can see where you would definitely want to have secrets shielded, is when you have team members who should be able to deploy but not see the secrets. Which... okay... I guess. But on the flip side, you should be deploying through CD tools like ArgoCD anyway.

Stay Secure

As always, though, protect your passwords. If I missed something or overlooked something, ping me on Fedi/Mastodon (@AlexB@moris.social).