Secret Management

Secrets (API keys, database credentials, etc.) are stored in HashiCorp Vault and synced to Kubernetes automatically via the External Secrets Operator. The shared SecretStore is already configured — you just need to add your secrets to Vault and create an ExternalSecret manifest.

How it works

The External Secrets Operator watches ExternalSecret resources in Kubernetes. When it finds one, it reads the referenced secret from Vault and creates a native Kubernetes Secret. Your pods consume these secrets as environment variables.

Adding a secret

Store the secret in Vault

Open vault.service.roche.com and log in:

Set the Namespace to minerva-prod/xred-eln
Select OIDC as the authentication method
Enter the role name ROLE-PRED-MINERVA-XRED-ELN-DEV
Click Sign in with OIDC Provider — you’ll be redirected to authenticate via Roche SSO

In the Vault UI, navigate to Secrets → select the secret engine for your environment:

Environment	Secret engine
DEV	`minerva-devel`
UAT	`apps-uat`
PROD	`apps-prod`

Then create a new secret under the path xred-eln/<your-app>/<secret-name>.

For example, to store an API key for the demo app:

Navigate to Secrets → minerva-devel → xred-eln/demo/
Create a secret called api-config
Add key-value pairs: API_KEY = your secret value

You can store multiple key-value pairs in a single secret.

Create an ExternalSecret manifest

In the infrastructure repo , add an ExternalSecret in your app’s k8s/<app>/ directory:


apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: my-integration-app-secret
  namespace: xred-eln
  annotations:
    argocd.argoproj.io/compare-options: IgnoreExtraneous
spec:
  refreshInterval: "1m"
  secretStoreRef:
    name: ss-xred-eln
    kind: SecretStore
  target:
    name: my-integration-app-secret
    deletionPolicy: Retain
  data:
    - secretKey: API_KEY
      remoteRef:
        key: "xred-eln/my-integration/api-config"
        property: API_KEY
    - secretKey: DB_PASSWORD
      remoteRef:
        key: "xred-eln/my-integration/api-config"
        property: DB_PASSWORD

Key fields:

secretStoreRef.name — always ss-xred-eln (the shared SecretStore)
target.name — name of the Kubernetes secret that will be created
remoteRef.key — path to the secret in Vault
remoteRef.property — specific key within the Vault secret

Add the file to your app’s kustomization.yaml:


resources:
  - deployment-backend.yaml
  - service-backend.yaml
  - ingress-backend.yaml
  - externalsecret-app.yaml    # Add this

Reference the secret in your deployment

Use envFrom to inject all keys from the secret as environment variables:


spec:
  template:
    spec:
      containers:
        - name: app-backend
          envFrom:
            - secretRef:
                name: my-integration-app-secret

Or reference individual keys with env:


spec:
  template:
    spec:
      containers:
        - name: app-backend
          env:
            - name: API_KEY
              valueFrom:
                secretKeyRef:
                  name: my-integration-app-secret
                  key: API_KEY

Push and sync

Push the infrastructure changes to the target branch. ArgoCD will sync automatically and the Kubernetes secret will be created within a minute.

The ExternalSecret syncs every minute (refreshInterval: "1m"). After adding or updating a secret in Vault, the Kubernetes secret updates automatically — no redeployment needed. However, running pods won’t pick up changes until they are restarted.

Using secrets in your app

Secrets are injected into your pod as environment variables. In your FastAPI backend, read them with os.environ or Pydantic Settings:


import os
 
# Option 1: direct access
api_key = os.environ["API_KEY"]
 
# Option 2: Pydantic Settings (recommended)
from pydantic_settings import BaseSettings
 
class Settings(BaseSettings):
    API_KEY: str
    DB_PASSWORD: str
 
settings = Settings()  # reads from environment variables, fails if missing

Pydantic Settings automatically reads from environment variables. If a required variable is missing, the app fails immediately at startup with a clear validation error — no silent failures.

Vault structure

All secrets follow this path convention in Vault:

- - - ghcr-registry
    - - api-config

ghcr-registry — shared GHCR image pull credentials (already configured)
<app>/api-config — app-specific secrets (API keys, database credentials, etc.)

Each environment has its own secret engine but the same path structure under xred-eln/.

SecretStore per environment

The shared ss-xred-eln SecretStore is already set up on each infrastructure branch with the correct Vault path:

Environment	Vault path	Vault mountPath
DEV	`minerva-devel`	`minerva-devel`
UAT	`apps-uat`	`apps-uat`
PROD	`apps-prod`	`apps-prod`

The Vault namespace is always minerva-prod/xred-eln across all environments.

Vault Infrastructure Repo External Secrets Docs