AWS Builder Center
Design an enterprise search indexing pipeline on Amazon OpenSearch Service

Design an enterprise search indexing pipeline on Amazon OpenSearch Service

Indexing documents from several systems? Design the pipeline stage by stage, from one shared schema to chunking and embedding, and see what breaks when a stage is skipped.

Series: Build enterprise search and RAG on Amazon OpenSearch Service (1 article)

Are you indexing documents from more than one system into one search experience? If so, every system probably has its own idea of a title, an author and a part number, and a 200-page PDF doesn't search well as one blob. This article shows you how to design the indexing side of an enterprise search or Retrieval Augmented Generation (RAG) application on  Amazon OpenSearch Service . It's part 2 of a six-part series.  Part 1  mapped each step of the pipeline to the place it runs.

Three rules for the indexing side

  1. Check for a managed path first. If one covers your sources, you may not need a custom pipeline at all.
  1. Agree on one schema before you index anything. Every source maps onto it, so every later stage works against the same fields.
  1. Add classification and provenance fields at ingest. Security, filtering and links back to your systems of record all depend on them, and they're hard to add later.

Indexing terms used in this article

  • Common schema. One set of field names and types that every source maps onto, like agreeing on one spreadsheet layout before three teams fill it in.
  • Processor. One step that changes a document.  Amazon OpenSearch Ingestion  processors run before documents reach your index. Ingest pipeline processors run inside your domain or collection.
  • Route. A condition in an OpenSearch Ingestion pipeline that sends matching documents to a particular destination, like a mail room sorting envelopes by department.
  • Provenance field. A field that records where a document came from, such as its source system, source ID and URL.

How the indexing pipeline fits together

Figure 1 shows the five design stages between your sources and the index, and the managed path that can replace them.
Indexing pipeline: three sources, five stages across OpenSearch Ingestion and an ingest pipeline, one index

Figure 1: The five indexing stages and where each one usually runs.
Stages 1 to 3 usually run in OpenSearch Ingestion. Stages 4 and 5 usually run in an ingest pipeline, and OpenSearch Ingestion can take on parts of them, such as batch embedding or a call to AWS Lambda. Use either, or both, whichever does each job better for your use case.

Check whether a managed path covers you

If your files are in Amazon S3 and you use  Amazon OpenSearch Serverless ,  intelligent ingestion  parses, chunks, embeds and indexes  PDF, Word, plain text and Markdown files  for you. It runs in current-generation collections, not OpenSearch Serverless classic.
If you already send clean text and only need semantic search,  automatic semantic enrichment  adds sparse vectors from a service-managed model when you enable one setting on a text field. It works on domains running OpenSearch 2.19 or later with the latest service software, and on  OpenSearch Serverless .  Semantic Search, Minus the Ops  walks through it on a domain.
Build a custom pipeline when one or more of these is true:
  • Several sources describe the same things differently.
  • You need business rules at ingest, such as routing or dropping documents.
  • You need your own embedding model, dense or multimodal vectors, or control over chunking.
  • Security or filtering depends on classification and provenance fields.
  • Your files are in formats the managed path doesn't parse, such as CAD drawings.

Stage 1: Map every source to one schema

Picture three sources: a document system that exports drawings as PDFs, a wiki of procedures and a database of work orders. Each has its own ID, title and part number field, and a search for one part number should find all three. Parse binary files to text first, as Part 1 describes. Then create the index with an  explicit mapping  that defines the common schema. The following table is a workable starting point.
Field
Type
Purpose
 doc_id 
 keyword 
Stable ID, prefixed by source (for example,  wiki:12345 )
 source_system 
 keyword 
Where the document came from
 title 
 text , with a  keyword  subfield
Full-text search and exact sort
 body 
 text 
Full-text search
 part_numbers 
 keyword  array
Business identifiers you filter and join on
 doc_type 
 keyword 
Set by classification in Stage 4
 updated_at 
 date 
Freshness boosts and sync checks
 allowed_groups 
 keyword  array
Permissions, covered in Part 3
Then map each source onto the schema with four of OpenSearch Ingestion's  supported processors .
  rename_keys   renames source fields, such as  PartNo  to  part_numbers .   copy_values   duplicates a value into a second field.   add_entries   adds constants such as  source_system , or builds the prefixed  doc_id .   delete_entries   drops fields you don't want indexed.
  • Create the index before you load data. If you rely on dynamic mapping instead, OpenSearch Service sets each new field's type from the first document that contains it, and you can't change an existing field's type without reindexing.
  • Map every source to the same field names. If sources keep  PartNo  and  part_number , a filter on  part_numbers  misses both.

Stage 2: Put business rules in one place

Business rules decide what gets indexed and where: drop drafts, send procurement records to their own index, tag anything with a part number. OpenSearch Ingestion expresses them with  conditional routing , which sends matching events to particular sinks, and with conditions on individual processors, such as  drop_when  on   drop_events   or  add_when  on  add_entries . Conditions use Data Prepper  expression syntax .
Inside your domain or collection, every ingest processor accepts an   if  condition , and the   script  processor  runs Painless, OpenSearch's scripting language, for logic the built-in processors don't cover. When a rule needs an external system or library, call  AWS Lambda from OpenSearch Ingestion . Use Lambda only for the rules that need it.
Keep each rule in one place. If rules are split between OpenSearch Ingestion and an ingest pipeline, tracing an unexpected result means reading both.

Stage 3: Fix field types before indexing

Field transforms make values filterable and sortable. OpenSearch Ingestion has processors for them, including  convert_entry_type ,   date  ,   split_string  ,  substitute_string ,   grok  ,  csv  and  key_value .
Do these transforms before the first document reaches the index. Once a field has a type in the index, changing it means creating a new index and  reindexing  into it.

Stage 4: Add classification and provenance fields

Classification fields such as  doc_type  or a sensitivity tier let later stages route, filter and secure each document. Use a rule when the label follows from metadata ("everything from procurement is  procurement "), and a model when it takes judgment. The   ml_inference  processor  calls a classification model through ML Commons and writes the label onto the document.
Provenance fields record the source system, source ID, source URL and related entity IDs such as part numbers. Fill them with  add_entries  and  copy_values  in Stage 1. They link a result back to its system of record, or to a knowledge graph. If you plan to connect results to a graph, capture these IDs now, because adding them later means updating every document.

Stage 5: Decide on chunking and embedding

Long documents need chunking. Each embedding model has an input limit, and one vector for a 200-page document rarely matches a specific question precisely.
The   text_chunking  processor  splits text with one of three algorithms:  fixed_token_length  (384 tokens by default),  fixed_char_length  or  delimiter . Set  overlap_rate  between 0 and 0.5; the docs recommend 0 to 0.2. Chain the chunker into an embedding processor so each passage gets its own vector. For the API steps, follow the  semantic search with text chunking  tutorial.
Then pick the vector type, using the following table.
Option
Processor
Choose it when
Dense text vectors
  text_embedding  
or  ml_inference 
You want semantic recall and your choice of model
Sparse vectors
  sparse_encoding  
, or automatic semantic enrichment
You want semantic gains on an inverted index, without a dense vector index's memory (
 neural sparse search 
)
Multimodal vectors
  text_image_embedding  
or  ml_inference 
Diagrams and photos carry meaning, and users search across text and images
Finally, pick real-time or batch embedding, or both. Real-time suits ongoing updates. For a first load of millions of documents, use  offline batch inference in OpenSearch Ingestion  with Amazon Bedrock or Amazon SageMaker AI models on OpenSearch Service domains running version 2.17 or later.

Quick reference: what happens if…

If you…
What happens
Fix
Let each source keep its own field names
Filters on a shared field miss other sources' documents
Map every source to the common schema
Map dates or numbers as  keyword 
Range filters and sorts compare them as text, so  10  sorts before  9 
Map them as  date  or a numeric type, and convert values before indexing
Skip provenance fields
Results can't link back to the source system or to a graph
Add source system, source ID, URL and entity IDs at ingest
Embed a long document as one vector
Semantic matches for specific questions are poor, or the text exceeds the model's input limit
Chunk with  text_chunking  before embedding

Next steps

Before you build, answer these questions for your own sources:
  1. Does a managed path cover you, and if not, why not?
  1. Where is each binary format parsed, and by what?
  1. What is the common schema, and how does each source map onto it?
  1. Where do business rules live?
  1. Which classification and provenance fields will later stages depend on?
  1. What chunk size and overlap does each source get?
  1. Which vector type and model, and real-time or batch embedding?
Part 3 covers the  allowed_groups  field, and how to make every query respect it.

Bringing it together

An enterprise search indexing pipeline runs in five stages: one schema, business rules in one place, clean field types, classification and provenance fields, then chunking and embedding. Check the managed paths first. When they don't cover your sources, your answers to the questions in Next steps are your design.

Learn more

OpenSearch Service and OpenSearch Ingestion are built on the open source OpenSearch and Data Prepper projects. Links to docs.opensearch.org point to that documentation, and some features described there may work differently, or may not be available, in OpenSearch Service and OpenSearch Ingestion.

Series: Build enterprise search and RAG on Amazon OpenSearch Service (1 article)

Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.
Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article