Table of Contents
Password manager
Secrets Management: Using External Secret Operator on GKE and Google Secret Manager. | by Rakesh Saw | Google Cloud - Community | Medium
ESO (External Secret Operator ) — is a Kubernetes operator that integrates external secret management systems like AWS Secrets Manager, HashiCorp Vault, Google Secrets Manager, Azure Key Vault, IBM Cloud Secrets Manager, CyberArk Conjur and many more. The operator reads information(secrets) from external APIs and automatically injects the values into a Kubernetes Secret.
What is the goal of External Secrets Operator?
The goal of External Secrets Operator is to synchronize secrets from external APIs (like GSM) into Kubernetes. ESO is a collection of custom API resources — ExternalSecret, SecretStore and ClusterSecretStore that provide a user-friendly abstraction for the external API that stores and manages the lifecycle of the secrets for you.
Differnet core resources of External Secret Operator
Provider : Different external Secret manager tools like GCP secret manager,AWS secret manager ,HashiCorp Vault
SecretStore : It define which provider to use and how to authenticate with provider. It’s a namespaced resource
Press enter or click to view image in full size
ClusterSecretStore: It is as same as SecretStore but it is cluster-scoped .This type of store can be referenced by all ExternalSecrets from different namespace. Use it to offer a central gateway to your secret backend.
Press enter or click to view image in full size
ExternalSecret : The ExternalSecret declares what secret to fetch from external providers. It takes a reference to a SecretStore which knows how to access the provider data.
Google Secret Manager (GSM) — It is GCP provided native service to store ,manage and access secrets securely . Secrets data can any sensitive data like database password,TLS certificates,SSH keys etc.
Kubernetes Secret
A Secret is an object that contains a small amount of sensitive data such as a password, a token, or a key. Such information might otherwise be put in a Pod specification or in a container image.
Prerequisite
- You should have access to a GCP project with permission to create secrets, service account ,bind IAM permission and create GKE cluster
2. Application Deployment Cluster: Set a GKE cluster specifically for deploying the application. In this cluster, we will install External Secret operator to securely access,update and utilize secrets stored in Google Secret Manger .
Steps to execute to install ESO and sync Secrets from Google Secret manager to Kubernetes Secrets
- Setup necessary env variables
export GCP_PROJECT_ID=weighty-legend-415316 export GCP_ZONE=us-central1-a export ESO_GCP_SERVICE_ACCOUNT=secret-accessor # Google IAM Service Account export ESO_K8S_NAMESPACE=external-secrets # Kubenetes namespace to deploy export ESO_K8S_SERVICE_ACCOUNT=external-secrets # Kubernetes Service Accounta"
- Create GKE cluster with workload identity with minimum nodes or you can create autopilot cluster using GCP console or gcloud command
gcloud container clusters create eso-cluster --zone=$GCP_ZONE --workload-pool=$GCP_PROJECT_ID.svc.id.goog --machine-type "e2-medium" --num-nodes "2" --disk-size "50" \ --project $GCP_PROJECT_ID --scopes="https://www.googleapis.com/auth/cloud-platform"
- Get credentials for your cluster to authenticate with cluster
gcloud container clusters get-credentials eso-cluster --zone $GCP_ZONE --project $GCP_PROJECT_ID
- Create a namespace to use for the Kubernetes service account. You can also use the default namespace or any existing namespace.
kubectl create namespace app
- Create a service account in GCP and assign IAM permission to access and manage secrets in Google secret manager .
1. create a GCP service account gcloud iam service-accounts create $ESO_GCP_SERVICE_ACCOUNT \ --project=$GCP_PROJECT_ID 2. Assign secret accessor IAM role on service account gcloud projects add-iam-policy-binding $GCP_PROJECT_ID \ --member "serviceAccount:$ESO_GCP_SERVICE_ACCOUNT@$GCP_PROJECT_ID.iam.gserviceaccount.com" \ --role "roles/secretmanager.secretAccessor" --project=$GCP_PROJECT_ID
Press enter or click to view image in full size
- Allow the Kubernetes service account to impersonate the IAM service account by adding an **IAM policy binding** between the two service accounts. This binding allows the Kubernetes service account to act as the IAM service account.
gcloud iam service-accounts add-iam-policy-binding $ESO_GCP_SERVICE_ACCOUNT@$GCP_PROJECT_ID.iam.gserviceaccount.com \
--role roles/iam.workloadIdentityUser \
--member "serviceAccount:$GCP_PROJECT_ID.svc.id.goog[$ESO_K8S_NAMESPACE/$ESO_K8S_SERVICE_ACCOUNT]" --project=$GCP_PROJECT_ID
Press enter or click to view image in full size
Refer below Google docs to understand above steps to create cluster and configure workload identity to authenticate your workload
Install External Secret Operator (ESO)
- Get credentials for your cluster to authenticate with cluster
gcloud container clusters get-credentials eso-cluster --zone us-central1-a --project deft-sight-402505
- Add the External-Secrets Helm repository
helm repo add external-secrets https://charts.external-secrets.io
- Ensure that Helm is up-to-date by updating all repositories.
helm repo update
- Install the external-secrets repository using Helm
helm upgrade -install external-secrets external-secrets/external-secrets \
--set 'serviceAccount.annotations.iam\.gke\.io\/gcp-service-account'="$ESO_GCP_SERVICE_ACCOUNT@$GCP_PROJECT_ID.iam.gserviceaccount.com" \
--namespace external-secrets \
--create-namespace \
--debug \
--wait
. List all objects related to external secret operator
kubectl get all -n external-secrets
Press enter or click to view image in full size
- Create a test secret in Google Secret Manager
printf "user1" | gcloud secrets create db-username --data-file=- --project=$GCP_PROJECT_ID printf "pass1" | gcloud secrets create db-password --data-file=- --project=$GCP_PROJECT_ID
- Now we need to create ClusterSecretStore and External secret
Create ClusterSecretStore Object
cat <<EOF | kubectl apply -f -
---
apiVersion: external-secrets.io/v1beta1
kind: ClusterSecretStore
metadata:
name: gcp-store
spec:
provider:
gcpsm:
projectID: $GCP_PROJECT_ID
EOF
Press enter or click to view image in full size
Create namespace app for external secret
Kubectl create namespace app
Create ExternalSecret
cat <<EOF | kubectl apply -f -
---
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
name: database-creds
namespace: app
spec:
refreshInterval: 10s # rate SecretManager pulls GCPSM, Low refereshInternval for demo purpose,Set this value based based on apps
secretStoreRef:
kind: ClusterSecretStore
name: gcp-store # name of the ClusterSecretStore or you can also reference SecretStore
target:
name: db-creds # name of the k8s Secret to be created
creationPolicy: Owner
data:
- secretKey: db-user # name of secretkey it can be any name
remoteRef:
key: db-username # name of the GCPSM secret key
- secretKey: db-pass # name of secretkey it can be any name
remoteRef:
key: db-password # name of the GCPSM secret key
EOF
Press enter or click to view image in full size
- Check Kubernetes secret and it’s data
kubectl get secret db-creds -n app -o jsonpath='{.data.db-user}' | base64 -d
kubectl get secret db-creds -n app -o jsonpath='{.data.db-pass}' | base64 -d
Press enter or click to view image in full size
- Update the Google manager Secret with new updated value
printf "user2" | gcloud secrets versions add db-username --data-file=- --project=$GCP_PROJECT_ID printf "pass2" | gcloud secrets versions add db-password --data-file=- --project=$GCP_PROJECT_ID
- Check again Kubernetes secret and it’s data . It should be auto updated because of ESO
kubectl get secret db-creds -n app -o jsonpath='{.data.db-user}' | base64 -d
kubectl get secret db-creds -n app -o jsonpath='{.data.db-pass}' | base64 -d
Press enter or click to view image in full size
Reference for more details
- Official ESO — **https://external-secrets.io/latest/**
- Kubernetes Secret — **https://kubernetes.io/docs/concepts/configuration/secret/**
- Using GCP secret Manager — **https://external-secrets.io/latest/provider/google-secrets-manager/**
- Community Contribution — **https://external-secrets.io/latest/contributing/process/**