AWS Builder Center
Set up the next generation of Amazon OpenSearch Serverless with Terraform

Set up the next generation of Amazon OpenSearch Serverless with Terraform

New to OpenSearch? Build a working search collection step by step with a few Terraform resources, and see which settings matter and what happens if you leave one out.

Tested with Terraform v1.16.4 and the  hashicorp/aws  provider v6.66.0 in us-east-1.
This guide is for you if you know some Terraform but you're new to  OpenSearch . You'll start with an empty folder and end up with a working search collection on the next generation of  Amazon OpenSearch Serverless . Along the way, you'll see the few settings that matter and what happens if you get one wrong.

The short version

  1. Create a collection group with  generation = "NEXTGEN" .
  1. Create a collection and set  collection_group_name  to that group's name.
The rest of this guide fills in the details.

A few terms first

  •  Collection : where your data lives. You create indexes in it, add documents, and run searches. If you've used a database before, a collection is a lot like one.
  •  Collection group : a container for one or more collections. The group sets which generation of OpenSearch Serverless its collections use and how much compute they can scale up or down to.
  •  OCU (OpenSearch Compute Unit) : the unit of compute. You set a minimum and a maximum for indexing (writing data) and for search (reading data), and OpenSearch Serverless scales between them.

What you'll build

Terraform creates a next generation collection group in AWS. Inside the group is a search collection, and optionally more collections. Encryption, network, and data access policies apply to the collection. Your app writes documents to the collection and searches it.

Terraform creates one NextGen collection group and one search collection inside it. Three policies control how the collection's data is encrypted, where it can be reached from, and who can read and write it. Once it's running, your app sends documents to the collection and searches them.
Collections you add to the same group later are also NextGen.

What you need

  • Terraform installed
  • A  Region  where the next generation of OpenSearch Serverless is available. This guide was tested in us-east-1.

Step 1: Set up the provider

Create a new folder with a file named  main.tf . Everything in Steps 1 to 4 goes in that file.
terraform {
required_providers {
aws = {
source = "hashicorp/aws"
version = ">= 6.49.0"
}
}
}

provider "aws" {
region = "us-east-1"
}

data "aws_caller_identity" "current" {}

locals {
group_name = "my-first-group"
collection_name = "my-first-collection"
}
If your provider is older than 6.49.0: Terraform stops with an "Unsupported argument" error when it reaches  generation  in Step 2. On versions before 6.40.0, the collection group resource doesn't exist at all. Keep the  version = ">= 6.49.0"  line and run  terraform init -upgrade  to fix it.

Step 2: Create a collection group

resource "aws_opensearchserverless_collection_group" "main" {
name = local.group_name
generation = "NEXTGEN"
standby_replicas = "ENABLED"

capacity_limits {
min_indexing_capacity_in_ocu = 0
max_indexing_capacity_in_ocu = 4
min_search_capacity_in_ocu = 0
max_search_capacity_in_ocu = 4
}
}
Four things to get right here:
  • Set  generation = "NEXTGEN" . If you leave it out, Terraform creates the group on the original generation ( CLASSIC ). You won't get an error or a warning.
  • Use uppercase.  "nextgen"  (in lowercase) fails at  terraform plan  with an "Invalid String Enum Value" error.
  • Set  standby_replicas = "ENABLED" . If you use  "DISABLED"  with  NEXTGEN ,  terraform plan  fails.
  • Write the minimums as  0 , and don't leave them out. If you only set the maximums,  terraform apply  fails with "Provider produced inconsistent result after apply." The group still gets created, but Terraform marks it as tainted and tries to replace it every time you apply. To fix it, add the two  min_*  lines and apply again.
Minimums of 0 let your collections scale down to no compute when nobody is using them.

Step 3: Add the policies

A collection needs three policies before it's usable:
# How your data is encrypted
resource "aws_opensearchserverless_security_policy" "encryption" {
name = "${local.collection_name}-enc"
type = "encryption"
policy = jsonencode({
Rules = [{ ResourceType = "collection", Resource = ["collection/${local.collection_name}"] }]
AWSOwnedKey = true
})
}

# Where the collection can be reached from
resource "aws_opensearchserverless_security_policy" "network" {
name = "${local.collection_name}-net"
type = "network"
policy = jsonencode([{
Rules = [
{ ResourceType = "collection", Resource = ["collection/${local.collection_name}"] },
]
AllowFromPublic = true
}])
}

# Who can read and write data (here: you)
resource "aws_opensearchserverless_access_policy" "data" {
name = "${local.collection_name}-data"
type = "data"
policy = jsonencode([{
Rules = [
{ ResourceType = "collection", Resource = ["collection/${local.collection_name}"], Permission = ["aoss:*"] },
{ ResourceType = "index", Resource = ["index/${local.collection_name}/*"], Permission = ["aoss:*"] },
]
Principal = [data.aws_caller_identity.current.arn]
}])
}
These policies are for learning only. They allow access from the public internet and give you full permissions. For real projects, limit network access (for example, with  VPC endpoints ) and grant only the permissions each user needs.
If you skip the  data access policy : the collection is still created, but your requests to read or write data are denied.

Step 4: Create the collection

resource "aws_opensearchserverless_collection" "main" {
name = local.collection_name
type = "SEARCH"
collection_group_name = aws_opensearchserverless_collection_group.main.name

depends_on = [
aws_opensearchserverless_security_policy.encryption,
aws_opensearchserverless_security_policy.network,
]
}

output "group_generation" {
value = aws_opensearchserverless_collection_group.main.generation
}

output "collection_endpoint" {
value = aws_opensearchserverless_collection.main.collection_endpoint
}
  • Set  collection_group_name . If you leave it out, the collection is created on the original generation ( CLASSIC ), not the next generation.
  • Don't add  generation  to the collection. The collection always uses its group's generation. If you add  generation  here,  terraform plan  fails with an "Unsupported argument" error.
  • Keep  depends_on . Without it, Terraform may try to create the collection before the encryption policy exists, and the collection can fail to create.
Already have a next generation collection group? Skip Step 2 and put that group's name in  collection_group_name . Each new collection you add to the group is on the next generation too.

Step 5: Deploy and check it worked

terraform init
terraform plan # you should see: 5 to add, 0 to change, 0 to destroy
terraform apply
In my test, the first collection in a collection group became  ACTIVE  in about 5 seconds. Additional collections in the same collection group became  ACTIVE  in about 2 seconds. When the apply finishes:
  • The  group_generation  output should say  NEXTGEN .
  • The  collection_endpoint  output is the URL you send requests to. It looks like  https://<collection-id>.aoss.<region>.on.aws .
To double-check with the  AWS CLI :
aws opensearchserverless batch-get-collection-group --names my-first-group
aws opensearchserverless batch-get-collection --names my-first-collection
Look for  generation: NEXTGEN  on the group, and  status: ACTIVE  on the collection. The collection's details don't include a generation. Check its  collectionGroupName , then look up that group's generation.
Run  terraform plan  one more time. It should say "No changes."

Change how much compute you can use

Edit the numbers in  capacity_limits  and run  terraform apply . To learn more about how capacity works, see  Managing capacity limits for Amazon OpenSearch Serverless . The change happens in place in about a second, and your collection stays up the whole time.
OpenSearch Serverless accepts only these values:
  • Minimums: 0, 2, 4, 8, 16, or a multiple of 16 (32, 48, 64, and so on)
  • Maximums: 2, 4, 8, 16, or a multiple of 16. A maximum can't be 0.
  • Each minimum has to be less than or equal to its maximum.
If you use a value that isn't on the list (like 6 or 24),  terraform plan  looks normal, but  terraform apply  fails with a  ValidationException . Nothing changes, not even the values in the same apply that were valid. Fix the number and apply again.
Watch your bill: a minimum above 0 keeps that much compute running, and charged, even when nobody is using the collection. See  Amazon OpenSearch Service pricing  for current rates. Stick with 0 while you're learning.

Quick reference: what happens if…

If you…
What happens
Fix
Leave out  generation  on the group
The group is created on the original generation ( CLASSIC ), with no warning
Add  generation = "NEXTGEN" 
Leave out  collection_group_name  on the collection
The collection is created on the original generation ( CLASSIC )
Point it at a NextGen group
Add  generation  to the collection
 terraform plan  fails with "Unsupported argument"
Remove it. Set  generation  on the collection group instead.
Write  "nextgen"  in lowercase
 terraform plan  fails
Use  "NEXTGEN" 
Set  standby_replicas = "DISABLED"  with  NEXTGEN 
 terraform plan  fails
Use  "ENABLED" 
Leave out the  min_*  capacity lines
 terraform apply  fails and the group is marked tainted
Set both minimums to  0 
Use a provider older than 6.49.0
"Unsupported argument" error on  generation 
Use  >= 6.49.0  and run  terraform init -upgrade 
Remove  depends_on  from the collection
The collection may be created before its encryption policy and fail
Keep  depends_on 
Skip the data access policy
The collection works, but your data requests are denied
Add the data access policy
Use a capacity value like 6 or 24
 terraform apply  fails and nothing changes
Use 0 (minimums only), 2, 4, 8, 16, or a multiple of 16
Change  generation  on an existing group
Terraform deletes the group and creates a new one
Choose the generation when you first create the group. Test in a practice account first if the group has collections.

Next steps: add some data

When you  send requests to your collection endpoint , sign them with  AWS Signature Version 4 (SigV4) . Use  aoss  as the service name, and include an  x-amz-content-sha256  header with the hash of the request body. Without that header, every request returns  403 Forbidden , even when your data access policy is correct.
Two responses look like errors but are normal on OpenSearch Serverless:  shards_acknowledged: false  when you create an index, and  _shards: {total: 0}  when you add a document.

Clean up

When you're done experimenting, delete everything so you're not charged:
terraform destroy

Wrap-up

To use the next generation of Amazon OpenSearch Serverless with Terraform, create a collection group with  generation = "NEXTGEN" , then create your collections in that group. Set the minimum capacities to  0 , keep the provider at 6.49.0 or later, and use the quick reference table when something doesn't look right.

Learn more

Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.
Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article