AWS Builder Center
Plan an enterprise search pipeline on Amazon OpenSearch Service

Plan an enterprise search pipeline on Amazon OpenSearch Service

New to OpenSearch? Learn where each step of a search or RAG pipeline runs, from parsing files to query time, and what goes wrong when a step is in the wrong place.

Series: Build enterprise search and RAG on Amazon OpenSearch Service (1 article)

  1. 1
    Plan an enterprise search pipeline on Amazon OpenSearch Service This article
Are you building enterprise search or a Retrieval Augmented Generation (RAG) application? If so then you probably have a stack of PDFs, Word files and scanned documents. You might also have or will need an embedding model to turn text into vectors, and a large language model to write answers. In this six-part series, I show you how to build enterprise search and RAG with  Amazon OpenSearch Service . This first article maps each step of the pipeline to the place it runs, across ingest pipelines, search pipelines,  Amazon OpenSearch Ingestion  and ML Commons, so you know which feature to reach for and what happens when a step lands somewhere else.

Three rules for placing each step

  1. Turn files into text before indexing. OpenSearch Service stores JSON documents. Parse PDFs, Office files and scans first, or let a managed feature do it for you.
  1. Put each step where it belongs. Work that changes what's stored goes in OpenSearch Ingestion or an ingest pipeline. Work that changes a query or its results goes in a search pipeline.
  1. Call models through connectors. Processors reach embedding models and large language models through an ML Commons connector.

OpenSearch terms used in this article

  • Index and document. A document is a JSON object. An index is a collection of documents you search together. If you know relational databases, an index is a lot like a table, and a document is like a row.
  • Mapping. The schema of an index: which fields exist, what  type  each one is, and how text fields are split into searchable terms. A  text  field is split into terms for full-text search. A   keyword   field matches only the exact value, so use it for IDs, tags and filters. Other types include  date , numeric and  knn_vector  for vectors.
  • Processor. One step that changes a document or a query, such as setting a field, splitting text into chunks, or calling a model.
  • Pipeline. An ordered list of processors. An ingest pipeline runs when documents are written. A search pipeline runs when queries arrive.
  • Connector. A saved configuration that tells OpenSearch Service how to call a model hosted somewhere else, such as on Amazon Bedrock.

How the pipeline fits together

Figure 1 shows where each step of the pipeline runs.
Pipeline diagram: sources, parsing, OpenSearch Ingestion, ingest and search pipelines, and models

Figure 1: Where each step of an enterprise search pipeline runs.
Orange arrows show index-time work, and green arrows show query-time work. Both pipelines can call a model through ML Commons. The dashed box is a managed path for files in Amazon S3.

Where each step runs

Work can happen in four places. Later articles in this series, on chunking, embedding and permissions, come back to the following table.
Where
When it runs
Typical work
Upstream, before OpenSearch Service
Before indexing
Turning PDFs, Office files and scanned images into text and fields
Amazon OpenSearch Ingestion
Before documents reach your index
Pulling from sources, filtering, transforming, routing, large batch embedding jobs
Ingest pipeline
As each document is written
Chunking, embedding, calling a model, setting fields
Search pipeline
When a query arrives
Rewriting the query, combining hybrid results, reranking, generating answers

Ingest pipelines and search pipelines

An  ingest pipeline  runs at index time. Its processors run in order on each document, and OpenSearch Service stores the result. Set the ingest pipeline as the index's  default pipeline , or name it on each indexing request. Attach the pipeline before you load data. If you attach it later, documents already in the index stay unchanged until you  reindex  them.
A  search pipeline  runs at query time. Request processors run before the query, for example to generate a query embedding or add terms. Response processors run after the query, for example to rerank results, combine scores from sub-queries, or generate an answer with a large language model.
Each step either changes what's stored or changes what's searched. Steps that change what's stored go in an ingest pipeline. Steps that change a query or its results go in a search pipeline. If you put query embedding in an ingest pipeline, your queries aren't embedded.

OpenSearch Ingestion, an ingest pipeline, or both

 Amazon OpenSearch Ingestion  is a serverless collector built on the open source Data Prepper project. OpenSearch Ingestion pulls or receives data from sources such as Amazon S3, databases and streams, transforms the data, and writes it to your index.
You can use OpenSearch Ingestion, an ingest pipeline, or both. Pick whichever does each job better for your use case, using the following table.
Consider
OpenSearch Ingestion
Ingest pipeline
Connecting to sources such as S3, databases and streams
Built-in sources
No sources; something else must send the documents
Heavy transforms and routing across many sources
A good fit
Possible, and it uses your domain or collection's compute
Calling a model for each document
Supported, including
 offline batch inference 
A good fit for real-time indexing
Where the compute runs
Scales separately from your domain or collection
Uses your domain or collection
A common combination is OpenSearch Ingestion for sources and field normalization, and an ingest pipeline for chunking and embedding.
Files in Amazon S3 have a managed option. If you index into  Amazon OpenSearch Serverless  collections,  intelligent ingestion  parses, chunks, embeds and indexes those files with no pipelines to configure. Intelligent ingestion runs in current-generation OpenSearch Serverless collections, not in OpenSearch Serverless classic collections. Build your own pipeline instead when you need a shared schema across many sources, business rules, your own embedding model, or file formats intelligent ingestion doesn't parse. The next article covers that choice.

Calling models: ML Commons and connectors

 ML Commons  is the OpenSearch plugin that runs machine learning (ML) models, either on your domain or elsewhere through a  connector . A connector holds the model's endpoint, the AWS Identity and Access Management (IAM) role used to call the model, and the request and response format.
A remote model takes three steps to set up:
  1. Create a connector to the service. For Amazon Bedrock or Amazon SageMaker AI, follow  ML connectors for AWS services . For models hosted outside AWS, follow  ML connectors for third-party platforms .
  1. Register a model that uses the connector, with the  Register model API .  Connecting to externally hosted models  walks through the connector and model steps end to end.
  1. Reference the model's ID from a processor, such as the   ml_inference  ingest processor  or the   text_embedding  processor .
Without a connector, OpenSearch Service has no way to reach a model hosted outside your domain. To create the connector and model for Amazon Bedrock or SageMaker AI in one step, use the  CloudFormation templates for remote inference  in the OpenSearch Service console.
The general-purpose processor for step 3 is  ml_inference . The  ml_inference  processor comes in three forms: an ingest processor (each document), a  search request processor  (each query), and a  search response processor  (the results). Some tasks also have dedicated processors, such as  text_embedding .

Files have to become text first

OpenSearch Service indexes JSON documents, not files. A PDF, Word file, scanned invoice or computer-aided design (CAD) drawing has to become text and fields first. You can parse files upstream with  Amazon Bedrock Data Automation ,  Amazon Textract  or your own parser, then send the text and metadata to OpenSearch Service as JSON. For files in S3 indexed into OpenSearch Serverless, intelligent ingestion  parses PDF, Word, plain text and Markdown files  and keeps their headings, reading order and tables. Other formats, such as CAD drawings, and extractions specific to your business still need an upstream step.

Quick reference: what happens if…

If you…
What happens
Fix
Send a PDF or other file straight to OpenSearch Service
You get no searchable text from the file, because OpenSearch Service indexes JSON documents
Parse it first, or use intelligent ingestion for supported files in S3
Rely on dynamic mapping in production
OpenSearch Service sets each new field's type from the first document that contains it, and you can't change an existing field's type without reindexing
Create the index with an
 explicit mapping 
before you load data
Put query embedding in an ingest pipeline
Your queries aren't embedded, because ingest pipelines run only when documents are written
Use a search request processor
Attach an ingest pipeline after loading data
Documents already in the index stay unchanged
Attach it first, or reindex
Embed a large first load one document at a time
The load can take a long time
Use offline batch inference in OpenSearch Ingestion
Reference a remote model without a connector
OpenSearch Service can't reach the model
Create a connector, register a model, then use its model ID
Send CAD drawings to intelligent ingestion
They aren't in its list of supported formats (PDF, Word, text, Markdown)
Parse them upstream and send the result as JSON

Next steps

Sketch your own pipeline against the table in "Where each step runs." Mark which sources need parsing, and what runs in OpenSearch Ingestion, an ingest pipeline and a search pipeline. The next article turns that sketch into an indexing pipeline design.

Bringing it together

Every step of an enterprise search pipeline runs in one of four places: upstream of OpenSearch Service, in OpenSearch Ingestion, in an ingest pipeline or in a search pipeline. Models plug in through ML Commons connectors, and files become text before they reach your index. The rest of this series treats each design choice as a question of where a step runs, starting with the indexing pipeline in the next article.

Learn more

OpenSearch Service is built on the open source OpenSearch project. Links to docs.opensearch.org point to that project's documentation, and some features described there may work differently, or may not be available, in Amazon OpenSearch Service.

Series: Build enterprise search and RAG on Amazon OpenSearch Service (1 article)

  1. 1
    Plan an enterprise search pipeline on Amazon OpenSearch Service This article
Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.
Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article