AWS Builder Center

Builder Center

Improving Natural Language Search with Custom Search Analyzers and Hybrid Search in OpenSearch

Improving Natural Language Search with Custom Search Analyzers and Hybrid Search in OpenSearch

Learn how to handle natural language queries like "where do I find information about data privacy?" by combining custom search analyzers with hybrid search in OpenSearch. This pattern filters conversational noise words at query time while leveraging both lexical precision and semantic understanding—without requiring any re-indexing.

Arun Lakshmanan

Search Specialist Architect

Problem Statement

When users search enterprise knowledge bases, they don't type keyword queries like  "data privacy policy" . They ask natural questions:
"Where do I find information for data privacy?"
Standard search analyzers struggle with these conversational queries. The default English analyzer tokenizes this query into:
Plain text
["where", "do", "i", "find", "inform", "data", "privaci"]
Words like "where", "do", "i", and "find" become search terms that match irrelevant documents, diluting result quality. Your users get frustrated when obvious questions return unexpected results.

How Lexical Search Works

Understanding the problem requires knowing how text search operates at the index level.

Index-Time Analysis

When you index a document, OpenSearch runs it through an analyzer that:
  1. Tokenizes - Splits text into individual terms
  1. Filters - Applies transformations (lowercase, stemming, stop words)
  1. Stores - Writes terms to an inverted index
For example, indexing  "Guidelines for handling personal data"  with the English analyzer produces:
Plain text
["guidelin", "handl", "person", "data"]
These stemmed tokens are stored in the inverted index, mapping each term to document IDs.

Search-Time Analysis

When a user searches, OpenSearch:
  1. Analyzes the query using the field's search_analyzer
  1. Looks up each query term in the inverted index
  1. Scores documents using BM25 (term frequency, inverse document frequency, field length)
By default, the same analyzer is used for both indexing and searching. This works well for keyword queries but fails for natural language.

Benefits of Custom Search Analyzers

OpenSearch lets you specify a different analyzer for search-time via the  search_analyzer  field mapping—or dynamically in the query itself.
A custom search analyzer with extended stop words transforms our natural language query:
Default English Analyzer:
Plain text
"where do i find information for data privacy?"
→ ["where", "do", "i", "find", "inform", "data", "privaci"]
Custom Search Analyzer (with stop words):
Plain text
"where do i find information for data privacy?"
→ ["data", "privaci"]
The benefits:
  1. Higher precision - Only meaningful terms participate in scoring
  1. Better relevance - Documents about "data privacy" rank highest, not documents that happen to contain "information"
  1. Natural language support - Users can ask questions naturally without learning query syntax
  1. No re-indexing required - Search analyzers work at query time; your existing index stays unchanged

Using Search Analyzers with Hybrid Search

Hybrid search combines lexical (BM25) and semantic (vector) search. Here's how to integrate custom search analyzers.

Step 1: Define the Custom Analyzer in Index Settings

JSON
{
  "settings": {
    "analysis": {
      "filter": {
        "custom_stop_filter": {
          "type": "stop",
          "stopwords": [
            "where", "do", "find", "details", "information", "about",
            "how", "what", "when", "can", "could", "would", "should",
            "get", "give", "show", "tell", "need", "want", "looking",
            "search", "for", "the", "a", "an", "is", "are", "i", "me", "my"
          ]
        }
      },
      "analyzer": {
        "search_analyzer_with_stops": {
          "type": "custom",
          "tokenizer": "standard",
          "filter": ["lowercase", "custom_stop_filter", "porter_stem"]
        }
      }
    }
  }
}

Step 2: Create a Hybrid Search Pipeline

Set up normalization to combine scores from both search types. This example uses 70% lexical and 30% semantic weighting:
JSON
PUT /_search/pipeline/hybrid_search_pipeline
{
  "phase_results_processors": [
    {
      "normalization-processor": {
        "normalization": { "technique": "min_max" },
        "combination": {
          "technique": "arithmetic_mean",
          "parameters": { "weights": [0.7, 0.3] }
        }
      }
    }
  ]
}

Step 3: Execute Hybrid Search with Custom Analyzer

Apply the custom analyzer to the lexical portion of your hybrid query:
JSON
POST /documents/_search?search_pipeline=hybrid_search_pipeline
{
  "query": {
    "hybrid": {
      "queries": [
        {
          "multi_match": {
            "query": "where do i find information for data privacy?",
            "fields": ["description", "long_description"],
            "analyzer": "search_analyzer_with_stops"
          }
        },
        {
          "knn": {
            "embeddings": {
              "vector": [0.12, -0.34, ...],
              "k": 10
            }
          }
        }
      ]
    }
  }
}
The lexical query uses your custom analyzer while the semantic query uses embeddings generated from the filtered terms.

Results Comparison

For the query  "where do i find information for data privacy?" :
Search Method
Top Result
Why
Lexical (default analyzer)
Mixed results
"information", "find", "where" match many docs
Lexical (custom analyzer)
Data Privacy Policy
Only "data" and "privacy" scored
Semantic only
Data Privacy Policy
Vector similarity works well
Hybrid (70/30)
Data Privacy Policy
Best of both approaches

When to Use This Pattern

Use custom search analyzers when:
  • Users enter conversational queries ("how do I...", "where can I find...")
  • Your content has domain-specific noise words that should be filtered
  • You want to improve lexical search without re-indexing
  • Building chatbot or natural language interfaces over search
Combine with hybrid search when:
  • You need both exact keyword matching AND semantic understanding
  • Users may search for synonyms or related concepts not in the text
  • You want resilience—if one search method fails, the other compensates
Consider tuning weights when:
  • Higher lexical weight (70-80%): When exact terminology matters (legal, medical, compliance)
  • Higher semantic weight (50-70%): When conceptual matching is more important than exact words

Conclusion

Custom search analyzers solve a real problem: users ask questions naturally, but traditional search expects keywords. By defining a search-time analyzer with extended stop words, you strip conversational noise while preserving meaningful terms.
Paired with hybrid search, this pattern delivers:
  • Precision from filtered lexical matching
  • Recall from semantic vector search
  • Flexibility from query-time configuration
The best part? This works on your existing index. Define the analyzer, apply it in your hybrid query, and start handling natural language queries today.

This pattern was demonstrated using OpenSearch 3.0 with the  sentence-transformers/all-MiniLM-L6-v2  embedding model. The complete implementation is available as a Jupyter notebook.
# opensearch# semantic-search# best-practices
Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.
Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article