This is part 3 of my Kubernetes homelab cluster setup series.

The cluster is up, but it isn’t very usable yet. Before we and any services, we need to set up secrets management.

In this post, we’re going to add secret management to the cluster with sops and age so we can safely check our configuration into git.

Talos Kubernetes Homelab Setup Series

Prerequisites

Goal

I have my cluster configuration in git so that it’s easy to recreate if I break something while experimenting. I don’t want to commit secrets into git in cleartext though. Instead, I want to encrypt our secrets in a way that the cluster can decrypt them, but they’re safe to check into source control.

To do this, we’re going to use sops to encrypt the sensitive parts of our manifest files.

Why sops and not something else? Sops supports encrypting individual keys in yaml, json, env and ini files. It will let you encrypt with many of the cloud services like AWS KMS or GCP KMS (and others, check the site), but since we’re doing this in a home lab, we’re going to use age. It also supports using GPG, but age is what the sops maintainers recommend, and it’s a lot more user friendly than GPG.

Software Versions

Here are the versions of the software I used while writing this post. Later versions should work, but this is what these instructions were tested with.

Software Version
age 1.3.1.
helm 4.0.1
kubectl 1.34
kubernetes 1.34.1
sops 3.11.0
talos 1.11.5

Set up age

First, generate a key pair

Create a key pair with age-keygen

$ age-keygen
# created: 2026-01-18T18:12:39-07:00
# public key: age1234567890ABCDEF1234567890ABCDEF1234567890ABCDEF1234567890A
AGE-SECRET-KEY-1234567890ABCDEF1234567890ABCDEF1234567890ABCDEF1234567890A
$

It does not create any files. It’s up to you to store both the secret and private key. Don’t commit the secret key or there’s no point in encrypting.

macOS

On my Mac, I store the secret key in the system keychain, using keychainctl from my tumult macOS cli helper script collection.

Use keychainctl to add the private key with

keychainctl set cluster-age-secret-key
password data for new item:

Paste in your secret key.

sops allows you to specify a command to run to retrieve the secret key - it will run it and read the key from the command’s stdout.

We can use keychainctl to retrieve the private key when decrypting a file with sops so we don’t have to store the file on disk unencrypted by adding SOPS_AGE_KEY_CMD to our environment

export SOPS_AGE_KEY_CMD=keychainctl get cluster-age-secret-key

Linux

sops will automatically look in $HOME/.config/sops/age/keys.txt (or $XDG_CONFIG_HOME/sops/age/keys.txt if set) for the age secret key. If you want to use an alternate location, set SOPS_AGE_KEY_FILE in your environment. This unfortunately leaves your key decrypted on the filesystem.

If you’re using a password manager, I recommend storing the secret key in that and setting SOPS_AGE_KEY_CMD as shown in the macOS section above.

You can also store the age private key in the SOPS_AGE_KEY environment variable.

Set up sops

Now that you have an age key pair, let’s configure sops to use it.

Create .sops.yaml

sops stores its configuration in .sops.yaml. At the top level of your git checkout, .sops.yaml with these contents:

# .sops.yaml
creation_rules:
  - path_regex: /*?.yaml
    encrypted_regex: "^(id|password|password_file|app-password|web-password|secret|secretboxencryptionsecret|bootstraptoken|secretboxencryption|token|ca|crt|key|access-key-id|secret-access-key|hostedZoneID|email|data|stringdata|api-token|encryption-token|encryption-key)$"
    age: age1234567890ABCDEF1234567890ABCDEF1234567890ABCDEF1234567890A
This file sets keys to control sops’ behavior.
Key Description
path_regex Let sops know what files it should examine for encryptable keys
encrypted_regex A golang regex that tells sops which keys in our files need encryption
age The public key to use for encrypting the secrets

Add the sops operator to your cluster

We’re going to add the sops operator to the cluster. This will handle automatically decrypting keys in manifests so you don’t have to decrypt them before running kubectl apply -f

Add the age private key to the cluster

Put the private key in cluster-age-private.key, then

kubectl create secret generic sops-age-key-file \
  --from-file=key=cluster-age-private.key \
  --namespace sops

Add an operator configuration

Create a values file we can use with helm

---
# sops-values.yaml
secretsAsFiles:
  - mountPath: /etc/sops-age-key-file
    name: sops-age-key-file
    secretName: sops-age-key-file
extraEnv:
  - name: SOPS_AGE_KEY_FILE
    value: /etc/sops-age-key-file/key

Install the sops operator

Now that we have created a secret containing the age private key and a values file to configure the operator, we can install it with helm.

helm repo add sops https://isindir.github.io/sops-secrets-operator && \
  helm update && \
  helm upgrade --install sops sops/sops-secrets-operator \
      --namespace sops --create-namespace \
      -f sops-values.yaml

Usage examples

Now that the operator is in place, here are some usage examples. Note - never edit the files after sops has encrypted them. Not even the unencrypted keys - sops calculates a checksum of the entire file and your edits will make the validation fail.

Instead, run sops decrypt -i something.yaml, edit it, then sops encrypt -i something.yaml.

Encrypting keys in place

Here’s an example ClusterIssuer manifest I use to configure cert-manager to use LetsEncrypt and Route 53 DNS challenges.

# example.yaml
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
    name: letsencrypt-dns-r53
spec:
    acme:
        # Use the production server for real certificates; switch to the staging URL while testing.
        server: https://acme-v02.api.letsencrypt.org/directory
        # staging: https://acme-staging-v02.api.letsencrypt.org/directory
        email: user@example.com
        privateKeySecretRef:
            name: letsencrypt-dns-r53-account-key
        solvers:
            - dns01:
                route53:
                    # Region where your hosted zone lives
                    region: us-east-1
                    hostedZoneID: Z1234567890AB
                    accessKeyIDSecretRef:
                        name: route53-aws-secret
                        key: access-key-id
                    secretAccessKeySecretRef:
                        name: route53-aws-secret
                        key: secret-access-key

I have my cluster configuration on GitHub, but I don’t want my hostedZoneID visible to people trying to use my manifests as an example. I also want to make sure that if they copy the file to use to start their configuration for their own cluster they don’t accidentally use my email.

The .sops.yaml example I gave is configured to encrypt email and hostedZoneID keys, so let’s encrypt this with sops encrypt -i example.yaml. We’re using -i so that sops encrypts the file in place, otherwise you’d have to run something like sops encrypt example.yaml > example.encrypted.yaml.

The resulting file will look something like this:

# example.yaml
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
    name: letsencrypt-dns-r53
spec:
    acme:
        # Use the production server for real certificates; switch to the staging URL while testing.
        server: https://acme-v02.api.letsencrypt.org/directory
        # staging: https://acme-staging-v02.api.letsencrypt.org/directory
        email: ENC[AES256_GCM,data:Ds7Zyk8e7DNCFcqrIHUvgA==,iv:h3MatsiOfK7IEG/kPY1QexGfOGtq/npxPovrFz4arDs=,tag:VOSyK4loBzk7+XfhpIUczg==,type:str]
        privateKeySecretRef:
            name: letsencrypt-dns-r53-account-key
        solvers:
            - dns01:
                route53:
                    # Region where your hosted zone lives
                    region: us-east-1
                    hostedZoneID: ENC[AES256_GCM,data:H52qvjdhqf2zmgyK2A==,iv:zI+N6nH17lch9H756+hABpRc28A/E82wxvaoaLB4RRw=,tag:AATtcJLJlggJjWheIpoiSQ==,type:str]
                    accessKeyIDSecretRef:
                        name: route53-aws-secret
                        key: ENC[AES256_GCM,data:RzULmAT9/ZvgO2fLqw==,iv:Afyhbn8z9n+1Nf1LxsttGbmbncUDoNLJPHZY21LN1zk=,tag:0sH9ZX41xgW7nvE4kJM1VA==,type:str]
                    secretAccessKeySecretRef:
                        name: route53-aws-secret
                        key: ENC[AES256_GCM,data:dyXBYf0R1XUUWejHH2uQyso=,iv:hKxW7otsTkF5f5/kXbAa0Yw0WPxa13NrN8N1Lo/Nbfo=,tag:21FyiXBxd1Bor9SARXaW7g==,type:str]
sops:
    age:
        - recipient: age1t2fr4u6yvfja69s59rwt70rwa7vhu06sen0m6lxygsyn2dpgzc4s0ma7wy
          enc: |
            -----BEGIN AGE ENCRYPTED FILE-----
            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBUUWNoZzFMUmdQQTI0RE8w
            VC8vQzQyV3VTdEhtTkdNS3dwcENKM3hkcjBFCnhLN2xDUVUvQWlLRTV5ZkRyd0lX
            YUZsWEZseGlrY0hOZUI1Zm1xYnZMWE0KLS0tIGVBdE0vVkdFUEJYcUZwRm1Icy9h
            OWd4TUZKL2NIOXI5SjRITmhHSkZaZTQKn6+S5b8rfGADzTGNazfiH+Li/se/2as3
            g6LFWUkDMLq6CCRJoybbGFl/K/HdT+Eni3WMAL0ZYWIn5gh3qD2LRQ==
            -----END AGE ENCRYPTED FILE-----
    lastmodified: "2026-01-19T19:58:51Z"
    mac: ENC[AES256_GCM,data:dSgG36sbNaAqo1TyCEqWBVveniUNKHpsBT5A0A5wto6A+ohA6OmX3w/soGn0T4NU+5dIs8s0gJWZpR67rtwKu9+Wtf1dELw27NGpZu8Jak6zPZwqt0iktkOE7JYdtmyRR3VP3ykkkKOxEd4KoRhLdLkPPC3VrScwzbSX4K0AJVA=,iv:xzNZNTM0rYEf8Btexggmtum6H/iI6iexcIQYSG36zwk=,tag:G1pzGu+6RoC2WyhRfovUMA==,type:str]
    encrypted_regex: ^(id|password|password_file|app-password|web-password|secret|secretboxencryptionsecret|bootstraptoken|secretboxencryption|token|ca|crt|key|access-key-id|secret-access-key|hostedZoneID|email|data|stringdata|api-token|encryption-token|encryption-key)$
    version: 3.11.0

Note that it’s added a sops section to the file, the confidential keys are encrypted, but the non-sensitive keys were left alone so the file is still readable by humans, it’s just had the secrets redacted.

Creating k8s secrets with sops

What if we want to create actual k8s secrets? Ones with key names that aren’t found by .sops.yaml’s encrypted_regex?

The sops operator creates a SopsSecret resource. If you create secretTemplates resources in a SopsSecret, the operator will create those keys when you apply the manifest, update them when you apply changes, and delete all secrets created by the SopsSecret when it is deleted.

Here’s an example

# sample-secret.raw.yaml
apiVersion: isindir.github.com/v1alpha3
kind: SopsSecret
metadata:
  name: foo-sopssecret
  namespace: blog
spec:
  secretTemplates:
    - name: foo-secret
      stringData:
        username: myUsername
        password: 'Pa$$word'
    - name: some-token
      stringData:
        token: 8675309thx1138

After running sops encrypt -i sample-secret.raw.yaml

# sample-secret.raw.yaml
apiVersion: isindir.github.com/v1alpha3
kind: SopsSecret
metadata:
    name: foo-sopssecret
    namespace: blog
spec:
    secretTemplates:
        - name: foo-secret
          stringData:
            username: myUsername
            password: ENC[AES256_GCM,data:gz3TVX4rEB8=,iv:ifDsVjQmYbwAHy073rP8uI2NNu073WqOnw/JZ+e6TqI=,tag:tgb0RkiZe6Xlhx4UOItIgg==,type:str]
        - name: some-token
          stringData:
            token: ENC[AES256_GCM,data:paithquCOGuWL0zG/Q4=,iv:7P+yp62d9e4wLb8JUCFwYcO14swntguAr8wwO6z22fc=,tag:VOkL/leu5zT20Ufj5GGskQ==,type:str]
sops:
    age:
        - recipient: age1t2fr4u6yvfja69s59rwt70rwa7vhu06sen0m6lxygsyn2dpgzc4s0ma7wy
          enc: |
            -----BEGIN AGE ENCRYPTED FILE-----
            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBJdWNPYmQyTEtvTkZ0OHk3
            N2tHNS9sRU5PTml3OFUxUGR2aW9CbXdZMTB3ClhXWi9LbHlDYi82bEliVkNDY3R0
            eCs1VUFSZm45Y0ZwaVZ3SzhYWjZ4NmMKLS0tIHF1RGZSY21SZWtkSXZsUG5MQjV4
            TlNsZGR3WnVCZ05WWEQzN2tIMjhEQzQKXznPaFWnV8/qx1XGbSio/0XAa5/1SrrI
            EXm2by6UikdkSyngKk1sgvycM4z+JuvkKoxahH89RYe8ZX+9it4u1w==
            -----END AGE ENCRYPTED FILE-----
    lastmodified: "2026-01-19T20:11:54Z"
    mac: ENC[AES256_GCM,data:yo4wpmN9JXvE0DLjhG87me2470ezpQ1rFt9++yD06HnItyWQQhPRCkF0o8Z5J6JyJ++lM8pZLuzYjqJMy//Gt0neMFI1yPf7NzH3JzlO9fy8qpdFpQ0iyYdzTURNZ1qgZ9+dfeji5bqwHiLYY/GffcJvLPQoJUzD1+FHIWz4hXo=,iv:dmbQ/QKpWg6CXm/3cXNwlg71536SAKCl9CCE0R/I4xM=,tag:p3twWZya0nU/jQ5XSfsolg==,type:str]
    encrypted_regex: ^(id|password|password_file|app-password|web-password|secret|secretboxencryptionsecret|bootstraptoken|secretboxencryption|token|ca|crt|key|access-key-id|secret-access-key|hostedZoneID|email|data|stringdata|api-token|encryption-token|encryption-key)$
    version: 3.11.0

Now I can run

$ kubectl apply -f sample-secret.raw.yaml
sopssecret.isindir.github.com/foo-sopssecret created
$

and the operator will decrypt the keys and create the secrets, so when I run

$ kubectl get secret -n blog
foo-secret   Opaque   2      11s
some-token   Opaque   1      11s
$

I see the two secrets defined in the SopsSecret resource. And when I delete the SopsSecret, we can see that it deletes the regular kubernetes secrets defined by it.

$ kubectl delete -n blog sopssecrets.isindir.github.com foo-sopssecret
sopssecret.isindir.github.com "foo-sopssecret" deleted
$ kubectl get secret -n blog
No resources found in blog namespace.
$

You should now be able to create secrets and encrypt individual yaml keys for your cluster using SOPS.

Gotchas

I can’t decrypt a file I encrypted with sops

There are two common reasons this can happen

  1. You edited the file after you encrypted it. If you opened the encrypted file and saved it, your editor may have trimmed trailing whitespace from lines, which broke the checksumming.
  2. You should have SOPS_AGE_KEY, SOPS_AGE_KEY_FILE or SOPS_AGE_KEY_CMD defined in your environment

Kubectl apply failed

Confirm that you created the sops-age-key-file secret, and that it’s in the same namespace you installed sops.

kubectl get secret -A | grep sops
sops             sh.helm.release.v1.sops.v1                helm.sh/release.v1              1      1d
sops             sops-age-key-file                         Opaque                          1      1d