
Set up the next generation of Amazon OpenSearch Serverless with Terraform
New to OpenSearch? Build a working search collection step by step with a few Terraform resources, and see which settings matter and what happens if you leave one out.
Tested with Terraform v1.16.4 and the
hashicorp/aws provider v6.66.0 in us-east-1.This guide is for you if you know some Terraform but you're new to OpenSearch . You'll start with an empty folder and end up with a working search collection on the next generation of Amazon OpenSearch Serverless . Along the way, you'll see the few settings that matter and what happens if you get one wrong.
The short version
- Create a collection group with
generation = "NEXTGEN".
- Create a collection and set
collection_group_nameto that group's name.
The rest of this guide fills in the details.
A few terms first
- Collection : where your data lives. You create indexes in it, add documents, and run searches. If you've used a database before, a collection is a lot like one.
- Collection group : a container for one or more collections. The group sets which generation of OpenSearch Serverless its collections use and how much compute they can scale up or down to.
- OCU (OpenSearch Compute Unit) : the unit of compute. You set a minimum and a maximum for indexing (writing data) and for search (reading data), and OpenSearch Serverless scales between them.
- Policies: rules for how your data is encrypted , where the collection can be reached from , and who can read and write data .
What you'll build

Terraform creates one NextGen collection group and one search collection inside it. Three policies control how the collection's data is encrypted, where it can be reached from, and who can read and write it. Once it's running, your app sends documents to the collection and searches them.
Collections you add to the same group later are also NextGen.
What you need
- Terraform installed
- An AWS account, with credentials that can create OpenSearch Serverless resources
- A Region where the next generation of OpenSearch Serverless is available. This guide was tested in us-east-1.
Step 1: Set up the provider
Create a new folder with a file named
main.tf . Everything in Steps 1 to 4 goes in that file.terraform { required_providers { aws = { source = "hashicorp/aws" version = ">= 6.49.0" } }}
provider "aws" { region = "us-east-1"}
data "aws_caller_identity" "current" {}
locals { group_name = "my-first-group" collection_name = "my-first-collection"}If your provider is older than 6.49.0: Terraform stops with an "Unsupported argument" error when it reaches
generation in Step 2. On versions before 6.40.0, the collection group resource doesn't exist at all. Keep the version = ">= 6.49.0" line and run terraform init -upgrade to fix it.Step 2: Create a collection group
resource "aws_opensearchserverless_collection_group" "main" { name = local.group_name generation = "NEXTGEN" standby_replicas = "ENABLED"
capacity_limits { min_indexing_capacity_in_ocu = 0 max_indexing_capacity_in_ocu = 4 min_search_capacity_in_ocu = 0 max_search_capacity_in_ocu = 4 }}Four things to get right here:
- Set
generation = "NEXTGEN". If you leave it out, Terraform creates the group on the original generation (CLASSIC). You won't get an error or a warning.
- Use uppercase.
"nextgen"(in lowercase) fails atterraform planwith an "Invalid String Enum Value" error.
- Set
standby_replicas = "ENABLED". If you use"DISABLED"withNEXTGEN,terraform planfails.
- Write the minimums as
0, and don't leave them out. If you only set the maximums,terraform applyfails with "Provider produced inconsistent result after apply." The group still gets created, but Terraform marks it as tainted and tries to replace it every time you apply. To fix it, add the twomin_*lines and apply again.
Minimums of 0 let your collections scale down to no compute when nobody is using them.
Step 3: Add the policies
A collection needs three policies before it's usable:
# How your data is encryptedresource "aws_opensearchserverless_security_policy" "encryption" { name = "${local.collection_name}-enc" type = "encryption" policy = jsonencode({ Rules = [{ ResourceType = "collection", Resource = ["collection/${local.collection_name}"] }] AWSOwnedKey = true })}
# Where the collection can be reached fromresource "aws_opensearchserverless_security_policy" "network" { name = "${local.collection_name}-net" type = "network" policy = jsonencode([{ Rules = [ { ResourceType = "collection", Resource = ["collection/${local.collection_name}"] }, ] AllowFromPublic = true }])}
# Who can read and write data (here: you)resource "aws_opensearchserverless_access_policy" "data" { name = "${local.collection_name}-data" type = "data" policy = jsonencode([{ Rules = [ { ResourceType = "collection", Resource = ["collection/${local.collection_name}"], Permission = ["aoss:*"] }, { ResourceType = "index", Resource = ["index/${local.collection_name}/*"], Permission = ["aoss:*"] }, ] Principal = [data.aws_caller_identity.current.arn] }])}These policies are for learning only. They allow access from the public internet and give you full permissions. For real projects, limit network access (for example, with VPC endpoints ) and grant only the permissions each user needs.
If you skip the data access policy : the collection is still created, but your requests to read or write data are denied.
Step 4: Create the collection
resource "aws_opensearchserverless_collection" "main" { name = local.collection_name type = "SEARCH" collection_group_name = aws_opensearchserverless_collection_group.main.name
depends_on = [ aws_opensearchserverless_security_policy.encryption, aws_opensearchserverless_security_policy.network, ]}
output "group_generation" { value = aws_opensearchserverless_collection_group.main.generation}
output "collection_endpoint" { value = aws_opensearchserverless_collection.main.collection_endpoint}- Set
collection_group_name. If you leave it out, the collection is created on the original generation (CLASSIC), not the next generation.
- Don't add
generationto the collection. The collection always uses its group's generation. If you addgenerationhere,terraform planfails with an "Unsupported argument" error.
- Keep
depends_on. Without it, Terraform may try to create the collection before the encryption policy exists, and the collection can fail to create.
Already have a next generation collection group? Skip Step 2 and put that group's name in
collection_group_name . Each new collection you add to the group is on the next generation too.Step 5: Deploy and check it worked
terraform initterraform plan # you should see: 5 to add, 0 to change, 0 to destroyterraform applyIn my test, the first collection in a collection group became
ACTIVE in about 5 seconds. Additional collections in the same collection group became ACTIVE in about 2 seconds. When the apply finishes:- The
group_generationoutput should sayNEXTGEN.
- The
collection_endpointoutput is the URL you send requests to. It looks likehttps://<collection-id>.aoss.<region>.on.aws.
aws opensearchserverless batch-get-collection-group --names my-first-groupaws opensearchserverless batch-get-collection --names my-first-collectionLook for
generation: NEXTGEN on the group, and status: ACTIVE on the collection. The collection's details don't include a generation. Check its collectionGroupName , then look up that group's generation.Run
terraform plan one more time. It should say "No changes."Change how much compute you can use
Edit the numbers in
capacity_limits and run terraform apply . To learn more about how capacity works, see Managing capacity limits for Amazon OpenSearch Serverless . The change happens in place in about a second, and your collection stays up the whole time.OpenSearch Serverless accepts only these values:
- Minimums: 0, 2, 4, 8, 16, or a multiple of 16 (32, 48, 64, and so on)
- Maximums: 2, 4, 8, 16, or a multiple of 16. A maximum can't be 0.
- Each minimum has to be less than or equal to its maximum.
If you use a value that isn't on the list (like 6 or 24),
terraform plan looks normal, but terraform apply fails with a ValidationException . Nothing changes, not even the values in the same apply that were valid. Fix the number and apply again.Watch your bill: a minimum above 0 keeps that much compute running, and charged, even when nobody is using the collection. See Amazon OpenSearch Service pricing for current rates. Stick with 0 while you're learning.
Quick reference: what happens if…
If you… | What happens | Fix |
|---|---|---|
Leave out generation on the group | The group is created on the original generation ( CLASSIC ), with no warning | Add generation = "NEXTGEN" |
Leave out collection_group_name on the collection | The collection is created on the original generation ( CLASSIC ) | Point it at a NextGen group |
Add generation to the collection | terraform plan fails with "Unsupported argument" | Remove it. Set generation on the collection group instead. |
Write "nextgen" in lowercase | terraform plan fails | Use "NEXTGEN" |
Set standby_replicas = "DISABLED" with NEXTGEN | terraform plan fails | Use "ENABLED" |
Leave out the min_* capacity lines | terraform apply fails and the group is marked tainted | Set both minimums to 0 |
Use a provider older than 6.49.0 | "Unsupported argument" error on generation | Use >= 6.49.0 and run terraform init -upgrade |
Remove depends_on from the collection | The collection may be created before its encryption policy and fail | Keep depends_on |
Skip the data access policy | The collection works, but your data requests are denied | Add the data access policy |
Use a capacity value like 6 or 24 | terraform apply fails and nothing changes | Use 0 (minimums only), 2, 4, 8, 16, or a multiple of 16 |
Change generation on an existing group | Terraform deletes the group and creates a new one | Choose the generation when you first create the group. Test in a practice account first if the group has collections. |
Next steps: add some data
When you send requests to your collection endpoint , sign them with AWS Signature Version 4 (SigV4) . Use
aoss as the service name, and include an x-amz-content-sha256 header with the hash of the request body. Without that header, every request returns 403 Forbidden , even when your data access policy is correct.Two responses look like errors but are normal on OpenSearch Serverless:
shards_acknowledged: false when you create an index, and _shards: {total: 0} when you add a document.Clean up
When you're done experimenting, delete everything so you're not charged:
terraform destroyWrap-up
To use the next generation of Amazon OpenSearch Serverless with Terraform, create a collection group with
generation = "NEXTGEN" , then create your collections in that group. Set the minimum capacities to 0 , keep the provider at 6.49.0 or later, and use the quick reference table when something doesn't look right.Learn more
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article