AWS Builder Center

Builder Center

EKS Secret Management: Pod Identity & ASCP with Amazon Q CLI

EKS Secret Management: Pod Identity & ASCP with Amazon Q CLI

Learn how Amazon Q Developer streamlines secure secret management in EKS using Pod Identity and ASCP, with AI-guided implementation and best practices.

Bonnie Ng

In today's cloud-native world, managing secrets securely in Kubernetes environments is essential to maintaining robust application security. This blog post shares our journey of implementing a robust secret management solution for Amazon EKS using Pod Identity and the AWS Secrets and Configuration Provider (ASCP), with Amazon Q Developer as our guide. For comprehensive guidance on configuring ASCP with Pod Identity, we recommend referring to the official AWS Security Blog post:  Announcing ASCP integration with Pod Identity: Enhanced security for secrets management in Amazon EKS .

The Challenge: Secure Secret Management in Kubernetes

There are several challenges with traditional Kubernetes secret management:
  • Native Kubernetes secrets are only base64 encoded, not encrypted at rest
  • Managing IAM roles and service account credentials was complex
  • Secret rotation and access control required custom solutions
  • Storing secrets in version control systems posed security risks
We needed a solution that would provide strong encryption, fine-grained access control, and seamless integration with our EKS environment.

Enter Amazon Q Developer

For this implementation, we used the new agentic coding experience within the Amazon Q Developer CLI, which allowed us to interactively develop, execute, and troubleshoot Terraform scripts as well as interact with our EKS cluster through  kubectl . Learn more about this powerful tool at  Introducing the enhanced command line interface in Amazon Q Developer . Amazon Q Developer suggested leveraging AWS Secrets Manager with the AWS Secrets and Configuration Provider (ASCP) and EKS Pod Identity - a modern approach that would address our security concerns while simplifying operations.
Disclaimer: The code in this blog was generated with Amazon Q Developer. While tested in our environment, AI-generated content may evolve over time. We recommend reviewing all scripts and testing thoroughly in non-production environments before deployment.

Interaction with Amazon Q Developer

Here are some examples of the prompts and Q's responses:

Prompt Examples:

Bash
> Use Terraform code to deploy an EKS 1.32 cluster in us-east-1 with all the essential addons and the eks-pod-identity-agent addon enabled. Include a VPC, node group, and necessary IAM roles.                                                          
> Extend the previous Terraform code to create an AWS Secrets Manager secret and the necessary IAM roles for ASCP with Pod Identity (refer to the iam policy and trust policy in https://aws.amazon.com/blogs/security/announcing-ascp-integration-with-pod-identity-enhanced-security-for-secrets-management-in-amazon-eks/)
> Create Terraform configuration to install AWS Secrets Store CSI Driver in EKS using Helm. Include settings for Kubernetes secrets sync and AWS provider installation.


Step 1: Setting Up the EKS Cluster

Q provided a comprehensive configuration that included:
  • A VPC with public and private subnets across 3 availability zones
  • An EKS cluster with version 1.32
  • A managed node group with t3.medium instances
  • All required IAM roles and policies- Essential EKS addons, including CoreDNS, kube-proxy, VPC CNI, and the critical Pod Identity Agent
Here is a short demo for the interaction:


Step 2: Creating a Secret in AWS Secrets Manager

Next, Q guided me in creating a secret in AWS Secrets Manager to store our database credentials. The Terraform code Q provided included:
  • A properly named secret with appropriate tags
  • A secret version with our database username and password
  • A recovery window configuration for enhanced security
Terraform
# Create AWS Secrets Manager secret
resource "aws_secretsmanager_secret" "database" {
  name                    = "prod/myapp/database"
  description             = "Database credentials for the application"
  recovery_window_in_days = 7
  
  tags = {
    "kubernetes-namespace" = "default"
    "eks-cluster-name"     = aws_eks_cluster.eks_cluster.name
    "environment"          = "production"
    "application"          = "myapp"
  }
}

resource "aws_secretsmanager_secret_version" "database" {
  secret_id     = aws_secretsmanager_secret.database.id
  secret_string = jsonencode({
    username = "admin"
    password = "MySecureP@ssw0rd"  # In production, use a more secure method
  })
}


Step 3: Setting Up IAM Permissions

For secure access to our secrets, Q helped to generate Terraform code for :
  • An IAM policy with least-privilege permissions to access only our specific secret
  • An IAM role with the appropriate trust policy for Pod Identity
  • The necessary role-policy attachment Q emphasized the importance of the trust policy, which allows pods in our EKS cluster to assume the role and access the secrets.
Terraform
# IAM Policy for Secret Access with resource-based conditions
resource "aws_iam_policy" "secret_access_policy" {
  name        = "EksSecretAccessPolicy"
  description = "Policy to allow EKS pods to access specific secrets in Secrets Manager"
  
  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Effect = "Allow"
        Action = [
          "secretsmanager:GetSecretValue",
          "secretsmanager:DescribeSecret"
        ]
        Resource = "${aws_secretsmanager_secret.database.arn}*",
        Condition = {
          StringEquals = {
            "secretsmanager:ResourceTag/kubernetes-namespace": "${aws:PrincipalTag/kubernetes-namespace}",
            "secretsmanager:ResourceTag/eks-cluster-name": "${aws:PrincipalTag/eks-cluster-name}"
          }
        }
      }
    ]
  })
}

# IAM Role for Pod Identity with trust policy from AWS blog
resource "aws_iam_role" "secret_access_role" {
  name = "EksSecretAccessRole"
  
  # Trust policy for Pod Identity
  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Effect = "Allow"
        Principal = {
          Service = "pods.eks.amazonaws.com"
        }
        Action = [
          "sts:AssumeRole",
          "sts:TagSession"
        ]
      }
    ]
  })
}

# Attach the policy to the role
resource "aws_iam_role_policy_attachment" "secret_access_attachment" {
  role       = aws_iam_role.secret_access_role.name
  policy_arn = aws_iam_policy.secret_access_policy.arn
}

# Output the ARNs for reference
output "secret_arn" {
  value       = aws_secretsmanager_secret.database.arn
  description = "ARN of the database secret in AWS Secrets Manager"
}


Step 4: Installing the ASCP Secret Store CSI Driver

To enable our pods to mount secrets, Q provided Helm configuration for the Secrets Store CSI Driver and the ASCP plugin. This setup included:
  • The Secrets Store CSI Driver Helm chart
  • Configuration to enable syncing to Kubernetes secrets
  • The ASCP plugin installation
Terraform
# helm.tf
# Install the Secrets Store CSI Driver using Helm
resource "helm_release" "csi_secrets_store" {
  name       = "csi-secrets-store"
  namespace  = "kube-system"
  repository = "https://kubernetes-sigs.github.io/secrets-store-csi-driver/charts"
  chart      = "secrets-store-csi-driver"
  # No version pinning as requested
  
  # Enable syncing to Kubernetes secrets
  set {
    name  = "syncSecret.enabled"
    value = "true"
  }
  
  depends_on = [
    aws_eks_cluster.eks_cluster,
    aws_eks_node_group.eks_node_group
  ]
}

# Install the AWS Provider for Secrets Store CSI Driver using kubectl directly
resource "null_resource" "install_aws_provider" {
  provisioner "local-exec" {
    command = "kubectl apply -f https://raw.githubusercontent.com/aws/secrets-store-csi-driver-provider-aws/main/deployment/aws-provider-installer.yaml"
  }
  
  depends_on = [
    helm_release.csi_secrets_store
  ]
}

Step 5: Creating the service account and Pod Identity Association

Amazon Q Developer provides the YAML configuration for creating service account and Pod Identity Association:
Terraform
# Create the service account for accessing secrets
resource "kubernetes_service_account" "secret_access_sa" {
  metadata {
    name      = "secret-access-sa"
    namespace = "default"
  }
  depends_on = [
    aws_eks_cluster.eks_cluster,
    helm_release.csi_secrets_store
  ]
}

# Create the Pod Identity Association using AWS provider resource
resource "aws_eks_pod_identity_association" "this" {
  cluster_name    = aws_eks_cluster.eks_cluster.name
  namespace       = "default"
  service_account = kubernetes_service_account.secret_access_sa.metadata[0].name
  role_arn        = aws_iam_role.secret_access_role.arn
  
  depends_on = [
    aws_eks_addon.pod_identity_agent
  ]
}

Step 6: Configuring the SecretProviderClass and Deployment

Finally, Q helped us create:
  • A SecretProviderClass that defines how our secrets should be mounted
  • A deployment that uses the mounted secrets
Terraform
# Create the SecretProviderClass
resource "kubectl_manifest" "secret_provider_class" {
  yaml_body = <<YAML
apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: aws-secrets
  namespace: default
spec:
  provider: aws
  parameters:
    usePodIdentity: "true"
    objects: |
      - objectName: "prod/myapp/database"
        objectType: "secretsmanager"
        jmesPath:
          - path: username
            objectAlias: dbUsername
          - path: password
            objectAlias: dbPassword
YAML

  depends_on = [
    aws_eks_pod_identity_association.this,
    helm_release.csi_secrets_store
  ]
}

# Create the test deployment
resource "kubernetes_deployment" "secure_app" {
  metadata {
    name      = "secure-app"
    namespace = "default"
    labels = {
      app = "secure-app"
    }
  }

  spec {
    replicas = 2

    selector {
      match_labels = {
        app = "secure-app"
      }
    }

    template {
      metadata {
        labels = {
          app = "secure-app"
        }
      }

      spec {
        service_account_name = kubernetes_service_account.secret_access_sa.metadata[0].name
        
        container {
          name    = "app"
          image   = "amazonlinux:2"
          command = ["/bin/bash", "-c"]
          args    = [<<-EOT
            echo "Starting secure application..."
            echo "Database credentials are mounted at /mnt/secrets"
            echo "Username file: $(ls -la /mnt/secrets/dbUsername)"
            echo "Password file: $(ls -la /mnt/secrets/dbPassword)"
            echo "Username: $(cat /mnt/secrets/dbUsername)"
            echo "Password: $(cat /mnt/secrets/dbPassword)"
            # In a real application, you would use these credentials to connect to your database
            while true; do
              sleep 30
              echo "Application is running with secure credentials..."
            done
          EOT
          ]

          volume_mount {
            name       = "secrets-store"
            mount_path = "/mnt/secrets"
            read_only  = true
          }

          resources {
            requests = {
              memory = "64Mi"
              cpu    = "100m"
            }
            limits = {
              memory = "128Mi"
              cpu    = "200m"
            }
          }
        }

        volume {
          name = "secrets-store"
          csi {
            driver = "secrets-store.csi.k8s.io"
            read_only = true
            volume_attributes = {
              secretProviderClass = "aws-secrets"
            }
          }
        }
      }
    }
  }

  depends_on = [
    kubectl_manifest.secret_provider_class
  ]
}

Step 7: Verifying Our Implementation

After applying our Terraform configuration, we checked the status of our pods and examined the logs:
Bash
$ kubectl get po -A
NAMESPACE          NAME                                               READY   STATUS             RESTARTS         AGE
default            secure-app-5645bcdcb9-bcrz9                        1/1     Running            0                101s
default            secure-app-5645bcdcb9-lhmpm                        1/1     Running            0                101s
kube-system        aws-node-dbm66                                     2/2     Running            0                3h3m
kube-system        aws-node-slkpk                                     2/2     Running            0                3h3m
kube-system        coredns-6b9575c64c-n2mtn                           1/1     Running            0                3h7m
kube-system        coredns-6b9575c64c-vzk2z                           1/1     Running            0                3h7m
kube-system        csi-secrets-store-provider-aws-7kkx9               1/1     Running            0                6m46s
kube-system        csi-secrets-store-provider-aws-bm7ws               1/1     Running            0                6m46s
kube-system        csi-secrets-store-secrets-store-csi-driver-gt24v   3/3     Running            0                23m
kube-system        csi-secrets-store-secrets-store-csi-driver-m9qqr   3/3     Running            0                23m
kube-system        eks-pod-identity-agent-pzhzf                       1/1     Running            0                3h3m
kube-system        eks-pod-identity-agent-xzff7                       1/1     Running            0                3h3m
kube-system        kube-proxy-k2xw4                                   1/1     Running            0                3h3m
kube-system        kube-proxy-vrrgd                                   1/1     Running            0                3h3m

-----------------------------
$ kubectl logs secure-app-5645bcdcb9-bcrz9
Starting secure application...
Database credentials are mounted at /mnt/secrets
Username file: -rw-r--r--. 1 root root 5 Apr  8 10:28 /mnt/secrets/dbUsername
Password file: -rw-r--r--. 1 root root 16 Apr  8 10:28 /mnt/secrets/dbPassword
Username: admin
Password: MySecureP@ssw0rd
Application is running with secure credentials...
Success! Our application could access the secrets without having any credentials stored in the application code or as environment variables.

How Amazon Q Developer Enhanced Our Implementation Process

Throughout this journey, Amazon Q Developer provided significant value:
  1. Complete Infrastructure as Code: Q developed the entire Terraform configuration from scratch, including the EKS cluster, networking, IAM roles, and all necessary addons.
  1. Contextual Understanding: Q understood our EKS environment and recommended appropriate configurations based on our cluster version and enabled addons.
  1. Troubleshooting Assistance: Amazon Q Developer diagnosed the Terraform configuration errors, provided specific code fixes, and guided us through the deployment process.
  1. Security Best Practices: Q Developer strengthened our security posture by implementing least privilege IAM policies, read-only volume mounts, namespace isolation, and secret rotation strategies. These automated security recommendations ensured our infrastructure adhered to AWS best practices from the start.

Conclusion

Amazon Q Developer accelerated our EKS secret management implementation using Pod Identity and ASCP. The solution automated infrastructure setup, reduced development time, and strengthened security controls.  
# eks# amazon-q-developer# secrets-manager# gen-ai-assistant# terraform
Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.
Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article