
Design an enterprise search indexing pipeline on Amazon OpenSearch Service
Indexing documents from several systems? Design the pipeline stage by stage, from one shared schema to chunking and embedding, and see what breaks when a stage is skipped.
Series: Build enterprise search and RAG on Amazon OpenSearch Service (1 article)
Are you indexing documents from more than one system into one search experience? If so, every system probably has its own idea of a title, an author and a part number, and a 200-page PDF doesn't search well as one blob. This article shows you how to design the indexing side of an enterprise search or Retrieval Augmented Generation (RAG) application on Amazon OpenSearch Service . It's part 2 of a six-part series. Part 1 mapped each step of the pipeline to the place it runs.
Three rules for the indexing side
- Check for a managed path first. If one covers your sources, you may not need a custom pipeline at all.
- Agree on one schema before you index anything. Every source maps onto it, so every later stage works against the same fields.
- Add classification and provenance fields at ingest. Security, filtering and links back to your systems of record all depend on them, and they're hard to add later.
Indexing terms used in this article
- Common schema. One set of field names and types that every source maps onto, like agreeing on one spreadsheet layout before three teams fill it in.
- Processor. One step that changes a document. Amazon OpenSearch Ingestion processors run before documents reach your index. Ingest pipeline processors run inside your domain or collection.
- Route. A condition in an OpenSearch Ingestion pipeline that sends matching documents to a particular destination, like a mail room sorting envelopes by department.
- Provenance field. A field that records where a document came from, such as its source system, source ID and URL.
How the indexing pipeline fits together
Figure 1 shows the five design stages between your sources and the index, and the managed path that can replace them.

Figure 1: The five indexing stages and where each one usually runs.
Stages 1 to 3 usually run in OpenSearch Ingestion. Stages 4 and 5 usually run in an ingest pipeline, and OpenSearch Ingestion can take on parts of them, such as batch embedding or a call to AWS Lambda. Use either, or both, whichever does each job better for your use case.
Check whether a managed path covers you
If your files are in Amazon S3 and you use Amazon OpenSearch Serverless , intelligent ingestion parses, chunks, embeds and indexes PDF, Word, plain text and Markdown files for you. It runs in current-generation collections, not OpenSearch Serverless classic.
If you already send clean text and only need semantic search, automatic semantic enrichment adds sparse vectors from a service-managed model when you enable one setting on a text field. It works on domains running OpenSearch 2.19 or later with the latest service software, and on OpenSearch Serverless . Semantic Search, Minus the Ops walks through it on a domain.
Build a custom pipeline when one or more of these is true:
- Several sources describe the same things differently.
- You need business rules at ingest, such as routing or dropping documents.
- You need your own embedding model, dense or multimodal vectors, or control over chunking.
- Security or filtering depends on classification and provenance fields.
- Your files are in formats the managed path doesn't parse, such as CAD drawings.
Stage 1: Map every source to one schema
Picture three sources: a document system that exports drawings as PDFs, a wiki of procedures and a database of work orders. Each has its own ID, title and part number field, and a search for one part number should find all three. Parse binary files to text first, as Part 1 describes. Then create the index with an explicit mapping that defines the common schema. The following table is a workable starting point.
Field | Type | Purpose |
|---|---|---|
doc_id | keyword | Stable ID, prefixed by source (for example, wiki:12345 ) |
source_system | keyword | Where the document came from |
title | text , with a keyword subfield | Full-text search and exact sort |
body | text | Full-text search |
part_numbers | keyword array | Business identifiers you filter and join on |
doc_type | keyword | Set by classification in Stage 4 |
updated_at | date | Freshness boosts and sync checks |
allowed_groups | keyword array | Permissions, covered in Part 3 |
rename_keys renames source fields, such as PartNo to part_numbers . copy_values duplicates a value into a second field. add_entries adds constants such as source_system , or builds the prefixed doc_id . delete_entries drops fields you don't want indexed.- Create the index before you load data. If you rely on dynamic mapping instead, OpenSearch Service sets each new field's type from the first document that contains it, and you can't change an existing field's type without reindexing.
- Map every source to the same field names. If sources keep
PartNoandpart_number, a filter onpart_numbersmisses both.
Stage 2: Put business rules in one place
Business rules decide what gets indexed and where: drop drafts, send procurement records to their own index, tag anything with a part number. OpenSearch Ingestion expresses them with conditional routing , which sends matching events to particular sinks, and with conditions on individual processors, such as
drop_when on drop_events or add_when on add_entries . Conditions use Data Prepper expression syntax .Inside your domain or collection, every ingest processor accepts an
if condition , and the script processor runs Painless, OpenSearch's scripting language, for logic the built-in processors don't cover. When a rule needs an external system or library, call AWS Lambda from OpenSearch Ingestion . Use Lambda only for the rules that need it.Keep each rule in one place. If rules are split between OpenSearch Ingestion and an ingest pipeline, tracing an unexpected result means reading both.
Stage 3: Fix field types before indexing
Field transforms make values filterable and sortable. OpenSearch Ingestion has processors for them, including
convert_entry_type , date , split_string , substitute_string , grok , csv and key_value .Do these transforms before the first document reaches the index. Once a field has a type in the index, changing it means creating a new index and reindexing into it.
Stage 4: Add classification and provenance fields
Classification fields such as
doc_type or a sensitivity tier let later stages route, filter and secure each document. Use a rule when the label follows from metadata ("everything from procurement is procurement "), and a model when it takes judgment. The ml_inference processor calls a classification model through ML Commons and writes the label onto the document.Provenance fields record the source system, source ID, source URL and related entity IDs such as part numbers. Fill them with
add_entries and copy_values in Stage 1. They link a result back to its system of record, or to a knowledge graph. If you plan to connect results to a graph, capture these IDs now, because adding them later means updating every document.Stage 5: Decide on chunking and embedding
Long documents need chunking. Each embedding model has an input limit, and one vector for a 200-page document rarely matches a specific question precisely.
The
text_chunking processor splits text with one of three algorithms: fixed_token_length (384 tokens by default), fixed_char_length or delimiter . Set overlap_rate between 0 and 0.5; the docs recommend 0 to 0.2. Chain the chunker into an embedding processor so each passage gets its own vector. For the API steps, follow the semantic search with text chunking tutorial.Then pick the vector type, using the following table.
Option | Processor | Choose it when |
|---|---|---|
Dense text vectors | text_embedding or ml_inference | You want semantic recall and your choice of model |
Sparse vectors | sparse_encoding , or automatic semantic enrichment | You want semantic gains on an inverted index, without a dense vector index's memory ( neural sparse search ) |
Multimodal vectors | text_image_embedding or ml_inference | Diagrams and photos carry meaning, and users search across text and images |
Finally, pick real-time or batch embedding, or both. Real-time suits ongoing updates. For a first load of millions of documents, use offline batch inference in OpenSearch Ingestion with Amazon Bedrock or Amazon SageMaker AI models on OpenSearch Service domains running version 2.17 or later.
Quick reference: what happens if…
If you… | What happens | Fix |
|---|---|---|
Let each source keep its own field names | Filters on a shared field miss other sources' documents | Map every source to the common schema |
Map dates or numbers as keyword | Range filters and sorts compare them as text, so 10 sorts before 9 | Map them as date or a numeric type, and convert values before indexing |
Skip provenance fields | Results can't link back to the source system or to a graph | Add source system, source ID, URL and entity IDs at ingest |
Embed a long document as one vector | Semantic matches for specific questions are poor, or the text exceeds the model's input limit | Chunk with text_chunking before embedding |
Next steps
Before you build, answer these questions for your own sources:
- Does a managed path cover you, and if not, why not?
- Where is each binary format parsed, and by what?
- What is the common schema, and how does each source map onto it?
- Where do business rules live?
- Which classification and provenance fields will later stages depend on?
- What chunk size and overlap does each source get?
- Which vector type and model, and real-time or batch embedding?
Part 3 covers the
allowed_groups field, and how to make every query respect it.Bringing it together
An enterprise search indexing pipeline runs in five stages: one schema, business rules in one place, clean field types, classification and provenance fields, then chunking and embedding. Check the managed paths first. When they don't cover your sources, your answers to the questions in Next steps are your design.
Learn more
OpenSearch Service and OpenSearch Ingestion are built on the open source OpenSearch and Data Prepper projects. Links to docs.opensearch.org point to that documentation, and some features described there may work differently, or may not be available, in OpenSearch Service and OpenSearch Ingestion.
Series: Build enterprise search and RAG on Amazon OpenSearch Service (1 article)
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article