Least privilege and fault isolation for AWS FIS experiments
Scope IAM roles, target by tags, and know each FIS action's unit of impact, so a fault injection experiment breaks only what you meant to test.
Least privilege and focused fault isolation for safe resilience testing
AWS Fault Injection Service (FIS) is built for running experiments safely. It provides scoped IAM roles, tag-based targeting, stop conditions, and safety levers. These are the means to control exactly where a fault stops.
But the controls only work if you use them deliberately. Most teams use two of them below their potential: least-privilege IAM on the experiment role, and fault isolation on the targets.
This post walks through both, with real policies and a tour of the FIS action catalog, so that before you hit Start you know which faults are surgical, which are area-of-effect, and which layer of protection stops each one where you meant it to stop.
If you haven't used FIS yet
Thirty seconds of vocabulary, because the rest of this post uses these terms.
AWS Fault Injection Service is a managed service for injecting real faults into your AWS environment on purpose. You write an experiment template, a JSON document, and start an experiment from it. The template has four parts that matter here:
- Actions: the faults themselves, named like
aws:ec2:stop-instances. Each has a duration and, for most, a target.
- Targets: which resources an action applies to, expressed as a resource type plus a selector.
- Stop conditions: CloudWatch alarms that abort the experiment if things go wrong.
- A role ARN: the IAM role FIS assumes to actually make the API calls.
That last one is the whole subject of this post. FIS has no permissions of its own. Every fault it injects is an API call made with permissions you have granted it. You can drive all of this from the console, the CLI, or infrastructure as code. The examples in this post are template and policy JSON because that's the common denominator, but the same fields appear one-to-one in the console wizard, so you can follow along without hand-writing any of it.
The mental model: three layers
Every FIS experiment has three layers of protection. Before you hit Start, ask "what stops the fault at each one?"
- The experiment role (IAM). FIS can only do what its execution role is allowed to do. This is your hard limit, and you can set it precisely: an
aws:ResourceTagcondition on the destructive actions limits the role to explicitly opted-in resources. If the role literally cannot callec2:TerminateInstanceson an untagged instance, then no misconfigured target selector can terminate it.
- Target selection (tags, filters, parameters). Inside the experiment template, you choose which resources of a type get hit. Tags are how you say "only these".
- The action's inherent scope. Some FIS actions act on a single, named resource. Others act on a boundary: a subnet, a VPC, a transit gateway, or an IAM role. Those hit everything inside it, regardless of how you tagged the individual resources. This layer is a property of the action itself, and you manage it by controlling your topology, not by tagging.
The rest of this post walks through the layers in order, then covers the controls that still work once an experiment is running.
Layer 1: Least-privilege on the experiment role
FIS runs experiments by assuming an IAM role you provide it.
First, lock down the trust policy. Grant
sts:AssumeRole to fis.amazonaws.com and nothing else, then add aws:SourceAccount and aws:SourceArn conditions. These conditions make sure the role is only used for experiments in your own account, never on behalf of anyone else. AWS recommends them as a best practice for any role a service assumes.{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "Service": "fis.amazonaws.com" }, "Action": "sts:AssumeRole", "Condition": { "StringEquals": { "aws:SourceAccount": "111122223333" }, "ArnLike": { "aws:SourceArn": "arn:aws:fis:us-east-1:111122223333:experiment/*" } } } ]}One scope note: this example is the single-account pattern, where the experiment runs in the same account as the role. FIS also supports cross-account experiments, where an orchestrator account runs experiments against roles in target accounts, and that may be exactly what you want. The two conditions still apply there. In a target account,
aws:SourceAccount and aws:SourceArn name the orchestrator account and its experiment ARN instead of your own. The Scaling AWS FIS series linked later in this post covers that setup in depth. The rest of this post assumes a single account.Second, scope the permissions policy to tagged resources. This is the most effective thing you can do.
AWS ships a family of managed policies for FIS, one per service area:
AWSFaultInjectionSimulatorEC2Access , with equivalents for ECS, EKS, RDS, SSM, and network actions. They're designed as general-purpose starting points, and the FIS documentation says this directly: managed policies "might not grant least-privilege permissions for your specific use cases because they're available for all AWS customers to use," and AWS recommends reducing permissions further with customer managed policies specific to yours. A policy written for every customer can't know your tagging scheme, so AWSFaultInjectionSimulatorEC2Access grants ec2:StopInstances , ec2:StartInstances , ec2:RebootInstances , ec2:TerminateInstances , and ec2:SendSpotInstanceInterruptions on arn:aws:ec2:*:*:instance/* . That's fine for a sandbox, and it gets you running in minutes. When you need finer control over which resources an experiment can touch, write a customer managed policy that encodes your tagging scheme, so the role only covers the resources you chose to include.Here's what that looks like, constraining the destructive actions with a resource tag condition:
{ "Version": "2012-10-17", "Statement": [ { "Sid": "DescribeForTargetResolution", "Effect": "Allow", "Action": "ec2:DescribeInstances", "Resource": "*" }, { "Sid": "ActOnlyOnExperimentTaggedInstances", "Effect": "Allow", "Action": [ "ec2:StopInstances", "ec2:StartInstances", "ec2:RebootInstances", "ec2:TerminateInstances" ], "Resource": "arn:aws:ec2:*:111122223333:instance/*", "Condition": { "StringEquals": { "aws:ResourceTag/ExperimentReady": "true" } } } ]}Now the role's permissions match your intent. Even if someone makes a typo in the target selector, the worst it can do is act on instances you explicitly opted in with
ExperimentReady=true .Two details decide whether that limit holds in practice.
Add a
Deny , not just a conditional Allow . The statement above grants destructive actions when the tag is present. It does not deny them when the tag is absent, and IAM evaluates the union of every policy attached to the role, so any broader Allow that's also attached, such as the managed policy during a transition period, still applies on its own terms. An explicit Deny always wins that evaluation, which makes it the right tool for a guardrail:{ "Sid": "DenyDestructiveWithoutOptInTag", "Effect": "Deny", "Action": [ "ec2:TerminateInstances", "ec2:StopInstances", "ec2:RebootInstances", "ec2:SendSpotInstanceInterruptions" ], "Resource": "*", "Condition": { "StringNotEquals": { "aws:ResourceTag/ExperimentReady": "true" } }}Two parts of that statement matter:
-
StringNotEquals, notStringEquals: "false". When a single-valued key is absent, negated matching operators evaluate to true, so thisDenyfires on untagged instances, which is exactly what you want. ADenywritten as"StringEquals": {"...": "false"}never fires on an absent tag and protects nothing. Same intent, opposite outcome.
-
Resource: "*"is safe here because every action listed only ever authorizes against instance ARNs, so the wildcard cannot affect other resource types.
You can also place the same
Deny in a permissions boundary or a service control policy (SCP) instead of the role itself; the Scaling AWS FIS Across Your Organization series covers those patterns in depth.Then govern the tag itself. Once the policy is tag-scoped, applying
ExperimentReady=true opts an instance into fault injection, so limit who can write that tag. Restrict it on both API paths: ec2:CreateTags for existing instances, and ec2:RunInstances for tags applied at launch. The aws:TagKeys and aws:RequestTag condition keys work on both.One caveat on the example policy above:
ec2:DescribeInstances doesn't support resource-level permissions, so it has to be Resource: "*" . That's read-only and fine, since target resolution needs to list all instances in order to filter them. Keep the destructive actions tag-scoped. (The AWS managed policy takes the same approach.)Layer 2: Target selection with tags
Inside the experiment template you define targets: a resource type plus a way to select instances of it. That selector is resource IDs, resource tags, filters (queries against the resource's
Describe output), or resource parameters. You then scope with a selection mode: ALL , COUNT(n) , or PERCENT(n) , where the last two pick from the matched set at random.Tag-based selection is usually the best choice, as it's the same key that you can use to scope the IAM policy. Here's a complete template, small enough to read in one screen, so you can see the two halves line up:
{ "description": "Reboot opted-in app instances, one at a time", "roleArn": "arn:aws:iam::111122223333:role/fis/FisEc2ExperimentRole", "targets": { "OptedInInstances": { "resourceType": "aws:ec2:instance", "resourceTags": { "ExperimentReady": "true" }, "selectionMode": "COUNT(1)" } }, "actions": { "RebootOne": { "actionId": "aws:ec2:reboot-instances", "targets": { "Instances": "OptedInInstances" } } }, "stopConditions": [ { "source": "aws:cloudwatch:alarm", "value": "arn:aws:cloudwatch:us-east-1:111122223333:alarm:app-5xx-elevated" } ], "tags": { "Name": "reboot-one-app-instance" }}Two syntax notes. The
targets map on an action refers to your target by name, and the key on the left ( Instances ) is fixed by the action. Durations are ISO-8601 strings: one minute is PT1M , not 1 or 60 .Notice that the template's
resourceTags and the policy's aws:ResourceTag condition use the same tag key. The template selects the targets, and IAM checks the tag again when each action fires. If the tag was removed in between, the call is denied and the experiment stops.details worth knowing before your first template:
- Targets are resolved once, at the start of the experiment, before any action runs. FIS uses that resolved set for the whole experiment. Resources that acquire the tag mid-experiment won't be picked up. Resources that lose it are still in the set, though IAM will still block the call.
- If no targets match, the experiment fails rather than silently doing nothing.
- Multiple tags are ANDed. All specified tags must be present on a resource for it to be selected.
- An empty tag value is not a wildcard. It matches resources whose tag is literally empty.
- You can't combine resource IDs with tags or filters on the same target.
-
PERCENT(n)rounds down, and can round down to zero. Five instances at 50% gives you two. Four instances at 5% selects nothing, and the experiment fails.
Layer 3: The action's own scope, which your tags cannot change
Compare two actions.
aws:eks:pod-network-blackhole-port blocks traffic on one port on the pods you selected. aws:network:disrupt-connectivity blocks traffic for an entire subnet. Both disrupt the network, but the size of their impact is completely different, and tagging doesn't change which one you chose.Layer 3 is not a control you configure. It's a property of the action itself. Before you run any action, ask: what is this action's unit of impact? Four answers exist.
Shape | Unit of impact | Do your tags help? |
|---|---|---|
Surgical | The individual resource you selected | Yes, this is where tag scoping shines |
Area-of-effect by network | A subnet, VPC, or transit gateway | No, NACLs don't see instance tags |
Area-of-effect by identity | Every caller assuming an IAM role | No, the target is an identity |
Custom fault (SSM) | Whatever your script does, with the instance's own permissions | No, the script runs outside the experiment role |
Surgical: most of the catalog
The fault lands on the resource you selected and nothing else. Examples:
aws:ec2:reboot-instances , aws:ec2:stop-instances , aws:ebs:pause-volume-io , aws:ecs:stop-task , aws:eks:pod-delete , aws:rds:failover-db-cluster , aws:lambda:invocation-add-delay . There are more than forty of these across EC2, EBS, ECS, EKS, RDS, ElastiCache, DynamoDB, MemoryDB, DSQL, S3, Kinesis, and Lambda. The actions reference has the full list with per-action permissions.Most of your resilience testing belongs here, because tag targeting plus a tag-scoped role gives you tight isolation. Two exceptions:
aws:ec2:api-insufficient-instance-capacity-error supports no targeting by tags, filters, or parameters at all, and aws:s3:bucket-pause-replication supports only tag targeting. Check the reference before you assume tags are available.Area-of-effect by network boundary: hits everything inside it
Four actions operate on network boundaries and affect all resources inside them, not just the ones you tagged:
-
aws:network:disrupt-connectivitytargets a subnet. Withscope=allit blocks all traffic entering and leaving the subnet, affecting every resource inside it. Narrower scopes (availability-zone,vpc,dynamodb,s3,s3express,prefix-list) reduce what is blocked, but the unit of impact is still the whole subnet.
-
aws:network:route-table-disrupt-cross-region-connectivityalso targets a subnet, blocking its traffic to an entire Region.
-
aws:network:transit-gateway-disrupt-cross-region-connectivityblocks a transit gateway peering attachment's traffic to a Region, which can carry traffic for many VPCs.
-
aws:network:disrupt-vpc-endpointblocks all traffic on the target interface VPC endpoints.
There is no tag that excludes one instance when its whole subnet is blocked. For these actions, isolation comes from your architecture, not your targeting:
- Give the workload under test its own subnet at minimum, and ideally its own VPC or even its own account, so "everything in the subnet" is the thing you meant to test.
- Check who is affected before you start. FIS can generate a target preview (
aws fis start-experiment --experiment-options actionsMode=skip-all), which resolves the template's targets without running any actions. Then, if you share the subnet,aws ec2 describe-network-interfaces --filters Name=subnet-id,Values=subnet-abc123shows everything inside it.
- Scope the role's
ec2:ReplaceNetworkAclAssociationpermission to the specific subnet ARNs you intend to disrupt.
Area-of-effect by identity: the fault follows the role
Three actions target an IAM role rather than infrastructure, and fault the API calls made by that role:
-
aws:fis:inject-api-internal-error
-
aws:fis:inject-api-throttle-error
-
aws:fis:inject-api-unavailable-error
These are excellent for testing retry and backoff logic, but the blast radius is every workload that uses that role. Tags don't apply, because the target is an identity. Use a dedicated role for the workload under test, narrow the fault with the
service and operations parameters, and set percentage well below 100 on the first run.Custom faults through SSM
aws:ssm:send-command and aws:ssm:start-automation-execution let you build custom faults. The targeting is surgical, but the payload is an SSM document that runs under the instance's own profile, not the experiment role. So the blast radius is whatever the instance itself is allowed to do. Review the document content the same way you'd review the policy, and keep the set of documents FIS may invoke small.A few remaining actions:
aws:fis:wait and aws:cloudwatch:assert-alarm-state inject no faults; they're building blocks for multi-step experiments. aws:arc:start-zonal-autoshift shifts production traffic away from an Availability Zone, a real customer-facing change, so treat it as area-of-effect. And aws:directconnect:virtual-interface-disconnect targets one virtual interface, but if that interface carries your on-premises traffic, the impact is anything but small.The controls that work after you hit Start
All three layers are pre-flight. These four keep working while the experiment runs.
- Stop conditions. Attach CloudWatch alarms to the template; if one goes into
ALARMmid-experiment, FIS stops the experiment. Three caveats. A stopped experiment cannot be resumed. Stopping does not undo irreversible actions like termination, though reversible faults are cleaned up. And only theALARMstate stops an experiment: an alarm stuck inINSUFFICIENT_DATAdoes nothing, which is what happens if the fault cuts off your metrics. Test the alarm once per template withaws cloudwatch set-alarm-state --state-value ALARMduring a throwaway experiment.
- Safety levers. Each account has a safety lever per Region. Engaging it stops all running experiments and blocks new ones, useful during peak events, incidents, or change freezes. It's disengaged by default, and pay attention to who can disengage it: if the experiment team can, a change freeze isn't enforced.
- Duration. Fault actions run from one minute up to 12 hours. Start at one minute to confirm the setup works, then extend. Failure modes like connection-pool exhaustion and retry storms need minutes to appear, so a short run that shows nothing proves nothing.
-
aws:cloudwatch:assert-alarm-state. Use it as the first action in a template so the experiment only proceeds if your steady-state alarms are healthy. It needscloudwatch:DescribeAlarmson the experiment role.
The layers in practice: blocking a subnet
A team wants to prove their service survives losing an Availability Zone, so they run
aws:network:disrupt-connectivity against a subnet. The experiment works exactly as designed. It also breaks a shared bastion host, a monitoring agent's connection, and a batch job that lived in the same subnet.Which layer would have caught it? Not Layer 1: the role needed NACL permissions to do its job. Not Layer 2: they targeted exactly the subnet they meant. Layer 3 is the one, and the catch is a question: who else is inside this subnet? One
describe-network-interfaces call would have shown the bastion, the agent, and the batch job. The fix is a dedicated subnet, or an informed decision about who else is affected by the test.That's why all three layers belong in the review, not just the IAM one.
The pre-flight checklist
- [ ] Dedicated experiment role, not a shared admin role. Trust policy locked to
fis.amazonaws.comwithaws:SourceAccountandaws:SourceArnconditions.
- [ ] Destructive permissions tag-scoped with an
aws:ResourceTagcondition and a matchingDeny.
- [ ] The opt-in tag is governed. Only a small, known set of users and roles can write
ExperimentReady, at launch as well as after.
- [ ]
iam:PassRolescoped to this role forfis.amazonaws.com, and you know who holds it.
- [ ] Targets selected by the same tag the IAM policy enforces.
- [ ] Every action classified by unit of impact: surgical, network, identity, or custom fault. Anything that isn't surgical runs against an isolated subnet, AZ, or dedicated role you have inspected.
- [ ] Ran it in a test environment first that matches production where it matters: AZ count, subnet layout, and who shares the subnet.
The one idea to take away
Least privilege is what makes a resilience test safe to run. Tag-based targeting keeps selection and enforcement in sync. And knowing each action's unit of impact tells you when isolation has to come from your architecture instead of your tags.
Your turn: what's the widest blast radius you've accidentally created, in a test or in production, and what guardrail would have caught it? I'd love to hear your stories in the comments.
Appendix: IAM details for policy authors
Three details for whoever authors or reviews the experiment role.
1. Encrypted EBS volumes need a scoped
kms:CreateGrant . If you use aws:ec2:stop-instances with startInstancesAfterDuration on instances with encrypted EBS volumes, the role also needs kms:CreateGrant , or the instances stop but can't restart. Don't grant it on Resource: "*" . Scope it:{ "Sid": "CreateGrantForEbsRestartOnly", "Effect": "Allow", "Action": "kms:CreateGrant", "Resource": "arn:aws:kms:us-east-1:111122223333:key/<your-ebs-cmk-id>", "Condition": { "Bool": { "kms:GrantIsForAWSResource": "true" }, "StringEquals": { "kms:ViaService": "ec2.us-east-1.amazonaws.com" } }}The
kms:GrantIsForAWSResource condition restricts this to grants created by an AWS service on your behalf, which is exactly the EBS case.2. The managed network policy's tag conditions. The AWS managed network policy only allows the role to create and delete network ACLs and entries tagged
managedByFIS=true , which FIS manages automatically. If you write your own network policy, include the same tag conditions.3. Action-specific condition keys, and the
ForAllValues trap. A few actions have their own condition keys, the tightest scoping available. For the EC2 capacity-error actions you can pin both the action and the target ARNs:{ "Effect": "Allow", "Action": "ec2:InjectApiError", "Resource": "*", "Condition": { "StringEquals": { "ec2:FisActionId": "aws:ec2:asg-insufficient-instance-capacity-error" }, "Null": { "ec2:FisTargetArns": "false" }, "ForAllValues:ArnLike": { "ec2:FisTargetArns": [ "arn:aws:autoscaling:us-east-1:111122223333:autoScalingGroup:*:autoScalingGroupName/my-test-asg" ] } }}The
Null guard matters because ForAllValues evaluates to true when the condition key is absent from the request. If you broaden the ec2:FisActionId list to also allow aws:ec2:api-insufficient-instance-capacity-error , which sends no target ARNs, the ARN condition silently stops constraining anything. Requiring the key to be present keeps the control real. Similarly, aws:s3:bucket-pause-replication has an s3:IsReplicationPauseRequest key so you can allow PutReplicationConfiguration only when FIS is pausing replication.
Related reading
- Scaling AWS Fault Injection Service Across Your Organization and Accounts : a three-part series on multi-account FIS: orchestrator and target roles, SCP guardrails, and governance.
- Securely validate business application resilience with AWS FIS and IAM : uses IAM deny policies as the fault itself, attached and detached through SSM automation, to simulate degradation of any AWS API.
- Chaos engineering workshop : hands-on FIS labs if you want guided practice running your first experiments.
References
- AWS FIS Actions reference : full action catalog with per-action permissions
- Targets for AWS FIS : resource types, selection modes, and resolution timing
- IAM roles for AWS FIS experiments : trust policy and experiment role setup
- AWS managed policies for AWS FIS : managed policy scope and least-privilege guidance
- AWSFaultInjectionSimulatorEC2Access : full policy document
- AWS FIS policy examples : action-specific condition keys
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article