Run Analytics and Search together with the new Optimized Engine for Amazon OpenSearch Service
In this post, you learn how the new Optimized engine for Amazon OpenSearch Service delivers analytical query performance and columnar storage efficiency alongside full-text search, eliminating the need to run a separate analytical database for your log analytics workloads.
The tradeoff that shouldn't exist
If you run log analytics at scale, you've faced this choice: fast analytical queries with low storage costs, or full-text search across your log data. Not both. Not in the same system.
Search-optimized engines deliver full-text search, but they consume 40-60% of observability budgets. As log volumes grow 30-40% year over year, teams respond by sampling data or cutting retention from 90 days to 30. When an incident hits and you need a specific error message across billions of events, you either can't find it (because you sampled the data out) or the query takes too long (because the engine wasn't built for analytical aggregations).
Analytics-first engines solve cost and query speed. Columnar storage compresses data efficiently. Aggregations fly. But when a production outage hits and you need to search for "connection refused to payment-gateway" across 2 weeks of logs, these engines can't help as they don't offer meaningful full-text search.
So, you end up running two systems. An analytical database for dashboards and trend analysis. A search engine for incident investigation. Double the infrastructure, double the cost, double the operational burden.
Introducing the Optimized engine for Amazon OpenSearch Service
Amazon OpenSearch Service now offers a purpose-built engine optimized for log analytics. The engine pairs a cost-based query planner and a vectorized execution engine with columnar storage, while retaining an inverted index for search on the same data.
The query planner routes each operation to the optimal path: the columnar engine for fast analytics, or the inverted index for search and filtering. You get:
- Up to 4x better price-performance compared to the existing general-purpose engine
- Up to 2x faster analytical queries through vectorized processing across large datasets
- Up to 2x higher ingestion throughput — 1.78 million docs/sec versus 647,000 docs/sec on matched hardware
- Up to 70% lower storage — 5x compression on observability data at billion-document scale
- Full-text search preserved through a retained inverted index for incident investigation
You don't move data. You don't change existing workflows.
Under the hood: how the engine works
The Optimized engine introduces three architectural changes that work together.
Columnar storage with Apache Parquet
The engine stores data in Parquet format rather than Lucene segments. Dictionary encoding handles repeated fields, the engine packs numeric columns efficiently, and there's no per-document JSON overhead.
Vectorized query engine with Apache DataFusion
Analytical queries execute through Apache DataFusion, a Rust-based vectorized engine optimized for high-throughput Parquet scans. The engine pushes predicates down and prunes row groups so that queries touch only the data they need.
Cost-based query planner with Apache Calcite
Apache Calcite parses incoming queries and routes each operation to the optimal execution path, the columnar engine for aggregations and analytical scans, or the retained Lucene inverted index for search and filtering.
From your perspective, nothing changes. You use the same domain endpoint, same Bulk API, same security model. The engine stores ingested data in the optimal hybrid format - columnar for analytics and inverted index for search, without any changes to your ingestion pipelines or application code.
Who should use the Optimized engine
The Optimized engine fits your workload if you ingest append-only data logs, metrics, traces, events and your primary query languages are Piped Processing Language (PPL) or SQL. Use the Optimized engine when you want to reduce storage costs without sacrificing retention, and when you need both analytical speed and full-text search in one system. Log analytics, observability, and security analytics workloads are the primary targets.
Stay with the General Purpose engine when your workload is search-heavy (e-commerce, content discovery), when you need frequent document updates or deletes, when you rely on Domain Specific Language (DSL) queries or Painless scripting, or when you need OpenSearch Dashboards. The Optimized engine uses the OpenSearch UI for data exploration and Visualization.
Conclusion
The Optimized engine for Amazon OpenSearch Service unifies analytical performance and full-text search in a single system, eliminating the need to run separate databases for log analytics workloads. Leveraging columnar storage, vectorized execution, and a cost-based query planner alongside a retained inverted index, it delivers up to 4x better price-performance, 2x faster queries, 2x higher ingestion throughput, and 70% lower storage with no changes to existing APIs or pipelines. For teams managing observability, log analytics, and security analytics at scale, this means lower costs, simpler architecture, and no compromises on search or retention.
Now Available
The optimized engine for Amazon OpenSearch Service is generally available today across 12 regions globally: US East (N. Virginia, Ohio), US West (Oregon), Canada (Central), Asia Pacific (Mumbai, Singapore, Sydney, Tokyo), and Europe (Frankfurt, Ireland, London, Spain). There are no additional charges for the new engine.
Get started
Pricing follows standard Amazon OpenSearch Service rates for instances and storage, with no additional premium for the optimized engine. For more information, see Amazon OpenSearch Service Pricing .
To learn more about configuring and using the optimized engine, see Optimized for Log Analytics in the Amazon OpenSearch Service documentation. For an overview of the service, visit Amazon OpenSearch Service Log Analytics .
For a detailed deep dive on cost savings and architecture, read the blog post Run log analytics for a fraction of the cost with the new engine for Amazon OpenSearch Service .
For a hands-on visual walkthrough, watch the Getting Started video Create Your First Analytical Domain .
For a complete reference on service capabilities, limits, and best practices, see the Amazon OpenSearch Service Documentation .
About the Author
Anindita Paul is a Technical Account Manager at AWS, working with customers on OpenSearch, analytics, and observability workloads.
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article