I was really taken aback the other day by this piece of Terraform file:
variable "db_username" {
description = "Postgres username"
type = string
}
The Terraform motto is “infrastructure as code”, but what I see above is a configuration file masquerading as a programming language!
Later on, at work, I got into a fight with the AWS Secret Manager and our EKS Cluster. The problem is simple:
The AWS Secret Manager holds our Postgres database password, and is in charge of rotating it. Wonderful.
The EKS Cluster starts a pod where one container uses an environment variable expecting the database password. Great.
Question: how do we pass this piece of information between those 2 systems?
The path to madness
The current solution consists in aligning precisely several pieces of text:
The name of the secret created in the Secrets Manager.
The “secrets provider” resource in Kubernetes.
The “volumes” section of the Kubernetes pod.
The “mounted volume” for a given container.
The environment variable with a “secret reference”.
Let’s have a look at those pieces.
The secrets provider
First the secrets provider:
apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
name: database-secrets
spec:
provider: aws
parameters:
objects: |
- objectName: "rds!db-63dfcfd8-9b62-482a-ac60-faad5f806344"
objectType: "secretsmanager"
secretObjects:
- secretName: database-username-and-password
type: Opaque
data:
- objectName: "rds!db-63dfcfd8-9b62-482a-ac60-faad5f806344"
key: DATABASE_USERNAME_AND_PASSWORD
This piece of configuration maps a secret name in the AWS Secret Manager to a Kubernetes Secret. It actually does more than just a mapping. It also guarantees that both will be synchronized when the secret is rotated.
That’s great, but why does the secret name have to be repeated twice? That name now lives in 3 places: the Secrets Manager, the parameters definition, the secretObjects definition. Since everything is a string, there was also nothing to force us to use a secret name for the objectName field. We could have passed rds!db-63dfcfd8-9b62-482a-ac60-faad5f80634 only to discover much later that a digit was missing.
We also have just introduced 3 new strings!
The volumes section
In the pod deployment file we need a section declaring which volumes are mounted:
volumes:
- name: secrets-provider-volume
csi:
driver: secrets-store.csi.k8s.io
readOnly: true
volumeAttributes:
secretProviderClass: "database-secrets"
There, we need to make sure that we use the right name for the secretProviderClass, by using a string defined in another file. But look, we are introducing yet another string for the volume name: secrets-provider-volume!
Mounting a volume
Next we need to mount that volume inside the container that will use the secret:
volumeMounts:
- name: secrets-provider-volume
mountPath: /mnt/secrets/database
readOnly: true
Once more we need to be careful to match the volume name and we add yet another string for the mountPath.
The environment variable
Finally we can declare an environment variable that will hold the secret:
- name: OCKAM_DATABASE_USERNAME_AND_PASSWORD
valueFrom:
secretKeyRef:
name: database-username-and-password
key: DATABASE_USERNAME_AND_PASSWORD
Like before we need to make sure to reuse the same names that we declared in the secret provider, secretName and key, except that they are now called name and key.
Note that nothing tells us that the volume secrets-provider-volume needed to be mounted and if the mountPath, /mnt/secrets/database, was relevant.
In summary
In order to use one AWS Secrets Manager secret in a Kubernetes pod we have to carefully align:
This is the definition of programming by coincidence!
Can we have a real programming language, please?
This short video is funny:
It basically says “We made all this YAML so that you don’t have to do it“.
One objective of the Unison programming language is to have a cloud offering where declaring services and all sorts of resources can be done programmatically. This has numerous advantages:
We can name variables that hold values, and refer to those names everywhere.
We can type those variables to prevent runtime errors.
We can use abstractions to isolate from implementation details.
What would it look like for our “secret” example to use a programming language?
The secret sauce
If we had a programming language at our fingertips we could start by defining the concept of “secret source”, that can be implemented by an AwsManagedSecret (all the code that follows is pseudo-Haskell-like code):
databaseManagedSecret : IO AwsManagedSecret
databaseManagedSecret = getOrCreateAwsManageSecret "prod-database" RotationPolicy.MONTHLY
class SecretSource s where
getSecret : s -> IO Secret
instance SecretSource AwsManagedSecret where
getSecret = todo
The purpose of a SecretSource is to give us the most recent secret when we ask for it. The real thing would actually give us more, like the frequency of the rotation, or how to access it via the network.
The implementation is using the AWS secrets manager but it could an Azure Key Vault. You will notice that we give the managed secret a name but it is not relevant for the rest of the code.
Next, we need to define a Kubernetes Secret that uses the SecretSource:
data KubernetesSecretFromSource = todo
makeKubernetesSecret : SecretSource s => s -> IO KubernetesSecretFromSource
makeKubernetesSecret s = todo
class KubernetesSecret s where
getSecretMap : s -> IO SecretMap
instance KubernetesSecret KubernetesSecretFromSource where ...
Now, we define an environment variable from a KubernetesSecret:
data EnvVarFromKubernetesSecret = todo
makeEnvVarFromKubernetesSecret : KuberneteSecret s => s -> IO EnvVarFromKubernetesSecret
makeEnvVarFromKubernetesSecret s = todo
class EnvironmentVariable v where
getName : v -> Text
getValue : v -> IO Text
instance EnvironmentVariable EnvVarFromKubernetesSecret where ...
Finally we can create a Pod, using an EnvironmentVariable which abstracts the details of how the value is retrieved:
createPod : EnvironmentVariable v => v -> ... -> IO Pod
createPod variable ... = do
setEnvironmentVariable variable
databaseClientPod :: IO Pod
databaseClientPod = do
managedSecret <- databaseManagedSecret
k8sSecret <- makeKubernetesSecret managedSecret
envVar <- makeEnvVarFromKubernetesSecret k8sSecret
createPod envVar
All of this is just a sketch, and the important points are:
We use typed variables to pass information around.
We use interfaces / typeclasses to hide implementation details.
We have library functions that declare exactly what is necessary to build:
A Pod,
An EnvironmentVariable backed by a KubernetesSecret,
A KubernetSecret from a SecretSource.
This is just good old software engineering practices! With huge benefits:
Less possibilities to make mistakes,
More discoverability of how things work,
A lot more fun programming our resources!
Let’s do it?
If we wanted to go that route there would be a number of obstacles:
I am not sure a mainstream programming languages can support that approach if, at some stage, we need some fancy types. However, if we just need type-safe heterogeneous maps we should be covered.
There would be a need for dependency injection in order to be able to easily define a configuration based off another configuration (what “override files” are attempting to do).
The sheer size of the project! The number of data structures and APIs to cover is enormous. I suspect that only a company-backed project (like Pulumi) could sustain this kind of effort.
So, what do you all think? Can we do better? Can we do much better?
