Benchmarking OpenSearch and Elasticsearch

This post concludes a four-month performance study of OpenSearch and
Elasticsearch search engines across realistic scenarios using OpenSearch
Benchmark (OSB). Our
full report includes the detailed findings
and comparison results of several versions of these two applications. We have
not modified either codebase.

Organizations running search-driven applications—from product searches on
e-commerce sites to real-time market analysis at financial
institutions—depend heavily on search engine
performance. OpenSearch and
Elasticsearch enable fast, scalable, and efficient
data retrieval, making them essential for applications like website search,
time-series log analysis, business intelligence, and cybersecurity monitoring.
Both are increasingly used in generative AI, machine learning, and vector
applications as well.

When milliseconds of latency can impact user experience or business operations,
even small performance differences can have significant costs.
Amazon Web Services (AWS) requested that we conduct an independent
benchmark assessment comparing these two prominent search-and-analysis software
suites.

As a result of our independent assessment, we observed that OpenSearch v2.17.1
is 1.6x faster on the Big5 workload and
11% faster on the Vectorsearch workload than Elasticsearch v8.15.4, when
aggregating the geometric mean of their queries. However, benchmarking both
applications is a moving target because both OpenSearch and Elasticsearch have
frequent product release cycles.

Over the course of our testing, Elasticsearch
updated its product to version 8.17.0, and OpenSearch released version 2.18.0.
We developed reusable code
that automates repeatable testing and analysis for both
platforms on AWS cloud infrastructure.

This review compares the query latencies of OpenSearch and Elasticsearch on
OpenSearch Benchmark workloads.
OSB uses the client-side service
time
metric (measured in milliseconds) for this purpose; it represents how long a
request (i.e., query) takes to receive a response. This includes overhead
(network latency, load balancer overhead, serialization/deserialization, etc.).
OSB records the 90th percentile (p90) of service times for each operation.

The scope was limited to Apache v2 (OpenSearch) and Elastic 2.0 (Elasticsearch)
licensed versions of the two engines and did not include proprietary systems.
The results can be used to direct future development of individual components
for each engine.

While six OSB workloads were evaluated in our full report, this blog post
highlights results from
Big5
(a comprehensive workload that performs text querying, sorting, date histogram,
range queries, and term aggregation) and
Vectorsearch
(a workload querying a dataset of 10 million 768-dimension vectors using
approximate K-Nearest Neighbor (KNN) search). We compare
recent versions of OpenSearch and Elasticsearch—v2.17.1
(released October 16, 2024) and v8.15.4
(released November 12, 2024), respectively.

Figure 1 illustrates our results comparing OpenSearch to Elasticsearch on the Big5 and
Vectorsearch workloads:

Observations and Impact

Methodology

We executed OpenSearch and Elasticsearch on workloads once per day, every day
for 15 days—except Vectorsearch, which we ran for 11 days. We collected
between 11 and 15 tests per workload per engine. We set up brand-new AWS
instances each time and executed workloads five times in a row. We discarded the
first run to ensure that the hardware, operating system, and application caches
were warmed up for both engines. Each operation in the workload was executed
hundreds to thousands of times. This resulted in thousands to tens of thousands
of sample measurements per operation. We believe this is a large enough sample
size to draw reliable conclusions.

Results processing and statistical analysis

We observed that some operations (mainly those whose service time values were
less than one millisecond) had a statistical power lower than our chosen
threshold of 90% despite having a low p-value: this signifies that a
statistically significant difference could be detected, but its magnitude could
not be observed reliably enough. Therefore, we executed additional runs of those
operations to increase their statistical power. After doing so, we confirmed
that any statistically significant difference between OpenSearch and
Elasticsearch performance characteristics in the tasks noted above was
inconsequential. However, users are likely more concerned about the performance
of longer-running queries than those that complete within a few milliseconds.

Outliers

We calculated the median of each workload operation’s p90 service time. We did
this because we saw non-trivial variations in performance in several runs in
both OpenSearch and Elasticsearch. These outliers can impact the arithmetic
average. We chose not to statistically exclude these outliers (e.g., using
standard deviation or quartiles as the exclusion criteria) because the results
do not necessarily follow a Gaussian (normal) distribution. Therefore, we
believe the median across this large number of independent data points is most
representative of the summary statistics we calculated for OpenSearch and
Elasticsearch. For completeness, the full report includes sparklines that
visually indicate the degree of variance across queries.

With our methodology established, let’s examine what our extensive testing
revealed about the performance characteristics of these search engines.

Big5 workload overview

The Big5 workload comprises a set of carefully crafted queries that exercise all
the major capabilities available in OpenSearch and Elasticsearch. They fall into
the following categories. For this performance comparison, each query is
weighted the same.

Text Queries: Searching for text is fundamental to any search engine or
database. Entering natural language queries is intuitive for users and does not
require knowledge of the underlying schema, making it ideal for easily searching
through unstructured data.

Term Aggregation: This query type groups documents into buckets based on
specified aggregation values, which is essential for data analytics use cases.

Sorting: Evaluates arranging data alphabetically, numerically,
chronologically, etc. This capability is useful for organizing search results
based on specific criteria, ensuring that the most relevant results are
presented to users.

Date Histogram: This is useful for aggregating and analyzing time-based data
by dividing it into intervals. It allows users to visualize and better
understand trends, patterns, and anomalies over time.

Range Queries: This is useful for filtering search results based on a
specific range of values in a given field. This capability lets users quickly
narrow their search results and find more relevant information.

Big5 workload category results

Using the geometric mean of the median values of all query operations in the
Big5 workload, we observe that OpenSearch v2.17.1 is 1.6x faster than
Elasticsearch v8.15.4. Below, we provide more details about how we arrived at
this estimate.

First, to ensure that our testing methodology was accurate, we referenced a
recent blog post
from the OpenSearch project on their performance measurements of v2.17. We
compare their reported performance (see the Results section of their post) to
ours in the table below. This includes all operations in Big5, while skipping
the match_all query (named default in the workload) and the scroll query.
This is the same protocol followed in the post.

Table 1: Establishing a baseline for OpenSearch

Note that the OpenSearch project publishes its performance numbers nightly at
https://opensearch.org/benchmarks

Since these values are reasonably close (albeit slightly different, most likely
due to different version numbers), we compare our results of running OpenSearch
v2.17.1 to Elasticsearch v8.15.4.

The original blog post above does not include two Big5 operations:
default and scroll. Both use the match_all query, which returns all
documents. Below, we add and categorize them as Text queries.

Table 2: Big5 comparison between OpenSearch and Elasticsearch

Next, we assess comprehensive Big5 workload testing results to understand
these observations, which exercise core search engine functionality.

Big5 workload operation results

The following graphs show the differences in median p90 service times across the
individual query operations in the Big5 workload for each category. The y-axis
represents an operation, and the x-axis represents how many times faster an
engine (OpenSearch or Elasticsearch) is over the other.

Vectorsearch Workload Results

Having examined traditional search operations, we now turn to vector search
capabilities, an increasingly important feature for modern applications using
AI/ML techniques. Here, we discuss the Vectorsearch workload results.
Force-merge
is enabled by default.

OpenSearch supports three vector search
engines:
NMSLIB, FAISS, and Lucene. These engines cater to various algorithms (HNSW,
HNSW+PQ, IVF, and IVF+PQ) and quantization techniques (fp16, 2x compression to
binary, 32x compression) based on different user workloads. The default vector
engine for OpenSearch 2.17.1 is NMSLIB. Newer releases after 2.17 have switched
to FAISS.

On the other hand, Elasticsearch supports only Lucene. Any reported values in
the charts below for Elasticsearch indicate test runs using the Lucene engine.
For brevity, we specify a search engine and the vector search engine used in
this format: search engine (vector engine).

The Vectorsearch workload consists of one primary query: prod-queries, a
vector search of the ingested data with a recall computation for the ANN search.

Similar to Big5, we compare the median p90 service time values. Focusing on an
out-of-the-box experience with the respective default configured engines (NMSLIB
for OpenSearch and Lucene for Elasticsearch), OpenSearch is 11% faster than
Elasticsearch for this metric, with similar recall and the same hyper-parameter
values.

Vectorsearch Performance Details

Comparing each OpenSearch v2.17.1 vector engine against Elasticsearch (Lucene)
v8.15.4 yielded the following findings:

• OpenSearch (NMSLIB) was 11.3% faster.
• OpenSearch (FAISS) was 13.8% faster.
• OpenSearch (Lucene) was 258.2% slower.

The median values are as follows:

Below are sparklines comparing OpenSearch and Elasticsearch on the Vectorsearch
workload. The x-axis represents time, and the y-axis represents the p90 service
time (in milliseconds). The min and max values represent the minimum and maximum
values of the y-axis for each sparkline, respectively. Each pair of sparklines
in a row is plotted with the same y-axis. All Elasticsearch sparklines plot the
same data, but they appear different from each other due to different y-axis
minimum and maximum values.

As shown above, OpenSearch (Lucene) varies in its performance. While this paints a
clear picture of relative performance, our testing also revealed some important
caveats about consistency that users should consider.

Performance Inconsistencies

We observed slow outlier runs for p90 service times for OpenSearch and
Elasticsearch. We investigated these scenarios but could not identify the root
cause. For example, note the random spikes in Figure 9 above for OpenSearch (Lucene). While
these anomalies did not affect our overall conclusions, they warrant further
investigation. We still included outliers in the datasets when calculating
results, as there was no systematic way to remove them.

We can quantify how extreme outliers are by the ratio of the maximum service
time over the median service time. Using this ratio, we found that OpenSearch has outliers
that are more extreme than those of Elasticsearch. The tasks with the most extreme
ratios for OpenSearch and Elasticsearch were:

• OpenSearch 2.17.1: 1412x for composite-date_histogram-daily
• Elasticsearch 8.15.4: 43x for query-string-on-message

We counted how many tasks have outlier runs using the criterion of a run with a
value that is more than twice as slow as the median. We found that
Elasticsearch has more outliers than OpenSearch:

• OpenSearch 2.17.1: 11 outlier tasks out of 98
• Elasticsearch 8.15.4: 19 outlier tasks out of 98

Repeatable, Open-Sourced Benchmarking

Based on these findings about both performance and consistency, we’ve developed
several key recommendations for conducting reliable search engine benchmarks:

• Always run workloads on newly created instances. If not, variations in
  workload performance may not be observed, which would skew a user’s
  expectations.

• After collecting data, measure both the p-value and the statistical power to ensure
  statistical reliability. Measuring p-values across runs with the same
  configuration helps detect anomalies where you expect high p-values (> 0.05)
  when comparing similar runs. Measure against different configurations (like
  different setups or engines) to confirm that changes produce statistically
  different results.

• Benchmarking should use configurations that closely match the out-of-the-box
  experience. Sometimes, changes are needed for a fair benchmark. In these
  cases, document the changes and explain why they aren’t suitable for the
  default configuration.

• A snapshot approach that may create more consistent results is to flush the
  index, refresh the index, and then wait for merges to complete before taking a
  snapshot. We found promising initial results in testing this approach with the
  Vectorsearch workload, but have not extensively tested this strategy.

Looking beyond our specific findings, we wanted to ensure that our work could serve
as a foundation for future benchmarking efforts. We focused on creating
repeatable and objective performance comparisons between OpenSearch and
Elasticsearch and used GitHub Actions to make our experiments easy to reproduce.
This enables ongoing performance comparisons in the future.

If you’re interested in how we can support your project, please
contact us.