
My Pod Was Running, But AWS Still Said AccessDenied: Understanding EKS Pod Identity
A practical deep dive into one of the most confusing EKS problems: a healthy Pod that can communicate with Kubernetes but cannot access AWS services. This article explains EKS Pod Identity, ServiceAccounts, IAM trust policies, debugging commands, common failure points, and a step-by-step troubleshooting workflow.
My Pod Was Running, But AWS Still Said AccessDenied: Understanding EKS Pod Identity
One of the most confusing moments when learning Amazon EKS is seeing something like this:
1
2
3
4
5
6
Pod: Running
Service: Running
Deployment: Ready
Application:
AccessDeniedExceptionAt first, it feels strange.
If the Pod is running correctly, why can't the application access Amazon S3, DynamoDB, Secrets Manager, or another AWS service?
The answer is that Kubernetes and AWS IAM solve different identity and authorization problems.
That distinction became much clearer when I started looking at the complete request path:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
Kubernetes Pod
|
v
ServiceAccount
|
v
EKS Pod Identity
|
v
IAM Role
|
v
IAM Policy
|
v
AWS ServiceUnderstanding this flow makes a large class of EKS
AccessDenied problems much easier to reason about.1. The Problem: A Healthy Pod Can Still Be Unauthorized
Imagine an application running in EKS that needs to read objects from an S3 bucket.
The Kubernetes resources might all look healthy:
1
kubectl get pods1
2
NAME READY STATUS
backend-7d8c6f9d8f-x2k7p 1/1 RunningThe Deployment is healthy:
1
kubectl get deployment1
2
NAME READY UP-TO-DATE AVAILABLE
backend 1/1 1 1But the application logs show:
1
AccessDeniedThis is where an important question appears:
Who is the application when it talks to AWS?
The answer is not simply:
"The Pod."
The application ultimately needs AWS credentials associated with an IAM role that has permission to perform the requested AWS API operation.
2. Kubernetes Identity and AWS Identity Are Different
This is the mental model that helped me understand EKS authentication.
Kubernetes has concepts such as:
- Users
- ServiceAccounts
- Roles
- RoleBindings
- ClusterRoles
- ClusterRoleBindings
AWS has:
- IAM users
- IAM roles
- IAM policies
- STS
- Resource policies
They are related in an EKS environment, but they are not the same authorization system.
For example:
1
2
3
4
5
6
7
Kubernetes authorization
Pod
|
+--> ServiceAccount
|
+--> Kubernetes RBACis different from:
1
2
3
4
5
6
7
8
9
AWS authorization
Application
|
+--> IAM Role
|
+--> IAM Policy
|
+--> S3 / DynamoDB / Secrets Manager / ...A Pod can therefore be perfectly healthy from Kubernetes' perspective while its application is unauthorized to call an AWS API.
3. Enter EKS Pod Identity
Amazon EKS provides EKS Pod Identity to give workloads running in Pods permissions to access AWS services.
Instead of placing long-lived AWS credentials inside the container, an IAM role can be associated with a Kubernetes ServiceAccount.
AWS describes this as a least-privilege mechanism because permissions can be scoped to the ServiceAccount used by the workload. ([AWS Documentation][1])
The simplified architecture looks like this:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
Amazon EKS
|
v
+----------------+
| ServiceAccount |
+----------------+
|
v
+----------------+
| EKS Pod |
| Identity |
+----------------+
|
v
+----------------+
| IAM Role |
+----------------+
|
v
+----------------+
| IAM Policy |
+----------------+
|
v
Amazon S3The application can then use AWS SDKs through the normal credential provider chain.
4. A Simple Example
Suppose I have an application called:
1
backendand I want it to read from an S3 bucket.
First, I create a ServiceAccount:
1
2
3
4
5
apiVersion: v1
kind: ServiceAccount
metadata:
name: backend-sa
namespace: defaultThen my Deployment uses it:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
apiVersion: apps/v1
kind: Deployment
metadata:
name: backend
spec:
replicas: 2
selector:
matchLabels:
app: backend
template:
metadata:
labels:
app: backend
spec:
serviceAccountName: backend-sa
containers:
- name: backend
image: my-backend:latestThe important line is:
1
serviceAccountName: backend-saWithout this, the Pod may run using a different ServiceAccount than the one I intended.
5. The IAM Role Is the Actual AWS Permission Boundary
The ServiceAccount alone does not magically give the application S3 access.
The IAM role needs an appropriate permissions policy.
For example, conceptually:
1
2
3
4
5
6
7
8
9
10
11
12
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"s3:GetObject"
],
"Resource": "arn:aws:s3:::my-application-bucket/*"
}
]
}This is where least privilege becomes important.
Instead of:
1
Allow *the application should receive only the permissions required for its job.
For example:
1
2
3
4
5
backend
|
+--> s3:GetObject
|
+--> specific bucketrather than:
1
2
3
backend
|
+--> AdministratorAccess6. The Trust Policy Is Just as Important
One mistake I found particularly interesting is focusing only on the permissions policy.
An IAM role also has a trust policy.
For EKS Pod Identity, AWS documents a trust relationship using the:
1
pods.eks.amazonaws.comservice principal, with actions including:
1
2
sts:AssumeRole
sts:TagSession([AWS Documentation][2])
A simplified example is:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Service": "pods.eks.amazonaws.com"
},
"Action": [
"sts:AssumeRole",
"sts:TagSession"
]
}
]
}This leads to an important distinction:
1
2
3
4
5
6
7
IAM Permissions Policy
|
+--> What can the role DO?
IAM Trust Policy
|
+--> Who can ASSUME the role?Both matter.
7. Creating the Pod Identity Association
AWS provides an EKS API/CLI operation to create the association.
For example:
1
2
3
4
5
aws eks create-pod-identity-association \
--cluster-name my-cluster \
--role-arn arn:aws:iam::123456789012:role/backend-s3-role \
--namespace default \
--service-account backend-saThe association connects:
1
2
3
4
5
6
7
EKS Cluster
|
+--> Namespace: default
|
+--> ServiceAccount: backend-sa
|
+--> IAM RoleAWS notes an important difference from IRSA: with EKS Pod Identity, you don't add an IAM role annotation to the ServiceAccount. The association is configured through EKS. ([AWS Documentation][3])
8. Don't Forget the Pod Identity Agent
Another common troubleshooting point is the EKS Pod Identity Agent.
The agent is required for EKS Pod Identity and runs as a DaemonSet on eligible nodes. ([AWS Documentation][1])
So the complete picture becomes:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
EKS Cluster
|
+-------------+-------------+
| |
v v
ServiceAccount Pod Identity Agent
| |
+-------------+-------------+
|
v
IAM Role
|
v
AWS APIsIf the identity association exists but the required agent is not available, the expected credential flow can fail.
9. My Troubleshooting Workflow
When I see:
1
AccessDeniedI would avoid immediately changing IAM policies.
Instead, I would walk through the chain systematically.
Step 1 — Is the Pod running?
1
kubectl get podsIf the Pod isn't healthy, fix the Kubernetes problem first.
Step 2 — Which ServiceAccount is the Pod using?
1
2
kubectl get pod <pod-name> \
-o jsonpath='{.spec.serviceAccountName}'This is a very useful check.
You might discover that the application is actually using:
1
defaultinstead of:
1
backend-saStep 3 — Does the Pod Identity association exist?
1
2
aws eks list-pod-identity-associations \
--cluster-name my-clusterAWS specifically recommends checking the Pod Identity association when troubleshooting workloads that cannot make AWS API calls. ([AWS Documentation][4])
Step 4 — Does the IAM role have the correct trust policy?
Check the role:
1
2
aws iam get-role \
--role-name backend-s3-roleLook for the expected Pod Identity trust relationship.
Step 5 — Does the IAM policy allow the requested action?
For example, if the application is attempting:
1
s3:GetObjectbut the policy only allows:
1
s3:ListBucketthe application can still receive:
1
AccessDeniedThe action, resource, and conditions all matter.
Step 6 — Check CloudTrail
If the IAM configuration looks correct, CloudTrail can help identify denied AWS API calls and the principal involved.
This turns:
1
"It doesn't work."into something much more concrete:
1
2
3
4
5
Which AWS API?
Which principal?
Which action?
Which resource?
Why was it denied?10. The Most Useful Mental Model
This is the model I now use when debugging EKS-to-AWS access:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
Is the Pod healthy?
|
v
Which ServiceAccount?
|
v
Is Pod Identity configured?
|
v
Is the agent available?
|
v
Which IAM role?
|
v
Can the role be assumed?
|
v
Does the policy allow it?
|
v
Does the resource allow it?
|
v
Check CloudTrailInstead of treating
AccessDenied as one problem, break it into layers.11. Pod Identity vs. IRSA
EKS also supports IAM Roles for Service Accounts (IRSA).
Both approaches can provide fine-grained AWS permissions to Kubernetes workloads. ([AWS Documentation][5])
The conceptual difference is:
1
2
3
4
5
6
7
8
9
IRSA
Pod
|
+--> ServiceAccount
|
+--> OIDC
|
+--> IAM Rolewhile Pod Identity uses:
1
2
3
4
5
6
7
8
9
EKS Pod Identity
Pod
|
+--> ServiceAccount
|
+--> Pod Identity Association
|
+--> IAM RoleOne notable difference is that EKS Pod Identity does not require configuring an OIDC identity provider for each cluster, and the association is managed through EKS rather than through a ServiceAccount annotation. ([AWS Documentation][1])
This doesn't mean every existing EKS environment should automatically migrate from IRSA.
The right choice depends on the architecture, existing configuration, operational requirements, and organizational standards.
12. One More Important Distinction: Human Access vs. Pod Access
There is another identity problem in EKS that is easy to mix up with Pod Identity.
Suppose I want to run:
1
kubectl get podsThat is different from an application inside a Pod running:
1
aws s3 lsThe first concerns access to the Kubernetes API.
The second concerns access from the workload to AWS APIs.
Conceptually:
1
2
3
4
5
6
7
8
9
10
Human / CI system
|
v
AWS IAM
|
v
EKS API
|
v
Kubernetes resourcesversus:
1
2
3
4
5
6
7
8
9
10
11
12
13
Pod
|
v
ServiceAccount
|
v
EKS Pod Identity
|
v
IAM Role
|
v
AWS ServiceEKS access entries are one mechanism AWS provides for granting IAM principals access to the Kubernetes API. ([AWS Documentation][6])
Keeping these two paths separate prevents a lot of confusion.
13. What This Taught Me
The biggest lesson wasn't a particular CLI command.
It was learning to stop thinking of:
1
AccessDeniedas a single problem.
In an EKS environment, I now ask:
1
2
3
4
5
6
7
8
9
10
11
12
13
Who is making the request?
Which identity is being used?
Which IAM role is involved?
Can that role be assumed?
What permissions does it have?
What resource is being accessed?
Is another policy or condition blocking the request?That turns troubleshooting from trial-and-error into a repeatable process.
Final Takeaway
A running Pod doesn't automatically mean that the application inside the Pod is authorized to access AWS resources.
The complete chain matters:
1
2
3
4
5
6
7
8
9
10
11
Pod
↓
ServiceAccount
↓
EKS Pod Identity
↓
IAM Role
↓
IAM Policy
↓
AWS ServiceOnce this chain becomes familiar, EKS IAM problems become much easier to investigate.
And perhaps the most useful debugging question is simply:
"Which identity is actually making this AWS API call?"
That question often gets you much closer to the real problem than changing IAM policies at random.
What I’d love to discuss
For people running workloads on EKS:
What has caused the most confusing
AccessDenied issue in your experience?Was it:
- the wrong ServiceAccount?
- an IAM trust policy?
- missing permissions?
- a Pod Identity association?
- IRSA/OIDC configuration?
- or something completely different?
I'd be interested to hear the failure mode that took you the longest to diagnose.
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article