It's interesting how the Elasticsearch and Datadog (www.datadoghq.com/blog/enginee...) approaches to wildcard search differ. Both use n-gram indexes, but with different strategies to contain storage amplification. Datadog hashes 4-grams while ES aggressively normalizes 3-grams.
Ge Song merged a good ~15% speedup for BM25F queries in Lucene benchmarks.mikemccandless.com/CombinedOrHi... (last data point) github.com/apache/lucen...
New blog: vectorized evaluation of disjunctive queries jpountz.github.io/2025/10/11/v... It explains how Lucene manages to be fast at evaluating top hits by BM25 score, even with hard queries that have only stop words or tens of terms.
BM25F is an adjustment to BM25 that accounts for multiple fields, beating out naive summing of BM25 scores
softwaredoug.com/blog/2025/09...
Good question, I don't have a good intuition for how much comes from SIMD vs. the rest. I could run benchmarks by disabling SIMD instructions.
Lucene 10.3 is out with 40% faster lexical search, 15% faster dense vector search and 30% faster terms dictionary lookups. lucene.apache.org/core/corenew...
This is the case here: filtered term queries get a slowdown but they are still pretty fast, while harder queries like OrHighHigh (top-level disjunctive query) or FilteredOrHighHigh (likewise but with a filter) get faster.
In case you wonder how Lucene decides whether a change is good when there is a mix of speedups and slowdowns, the general rule is that it's better to make the slow queries faster and the fast queries slower than the other way around as this results in lower tail latencies.
Lucene just bumped the block size of its postings lists from 128 to 256. This gave very good speedups (up to 45%) to most queries, and up to 10-15% slowdowns to filtered term queries. benchmarks.mikemccandless.com/2025.09.10.1...
#Lucene just switched from a binary heap to a ternary heap to collect top hits by score. This helps a small bit when computing top-100 hits (~2% on the fastest queries) but up to 15% when computing top-1000 hits - thanks to better cache efficiency github.com/apache/lucen...
I just ran the Tantivy benchmark (tantivy-search.github.io/bench/) on Lucene 10.2 vs a Lucene 10.3 snapshot build. Lucene 10.2 already performed very well, but Lucene 10.3 is on another level. Very exciting.
I just blogged about how Lucene improved query evaluation efficiency by ~40% through vectorization: jpountz.github.io/2025/08/28/c...
Why you probably should configure an index sort on your Lucene indexes: jpountz.github.io/2025/07/26/w...
I spent some time looking at the Vespa source code to see how it compares with Lucene jpountz.github.io/2025/07/25/m...
A: The "if score >= minRequiredScore" branch is hard to predict, so this loop is expensive. This small refactoring helped the JVM take advantage of the cmov (conditional move) instruction and make this code branchless.
This small change yielded a ~5% speedup on several queries of Lucene's nightly benchmarks (see last data point at benchmarks.mikemccandless.com/OrStopWords....). Can you guess why?
We're now further taking advantage of it by amortizing the cost of some operations across many doc IDs github.com/apache/lucen... or better vectorizing/pipelining other operations github.com/apache/lucen.... Expect a new round of query evaluation speedups in 10.3.
Last month, Lucene changed query evaluation to work in a more term-at-a-time fashion within small-ish windows of doc IDs. This yielded a good speedup on its own (annotation IL benchmarks.mikemccandless.com/OrHighMed.html).
Lucene is getting an increasing number of high-quality contributions from ByteDance employees, especially around performance. Good to see that this project keeps attracting contributors from all around the world.
Thank you Alex
Not that I know of. But we need this kind of benchmark indeed.
Another common point I did not expect: Vespa's strict vs. unstrict iterators is quite similar to Lucene's two-phase iteration. And both projects use this feature to effectively combine dynamic pruning with filtering (a hard and underappreciated problem IMO).
Someone asked me for my opinion on the Vespa vs. Elasticsearch performance comparison today at Berlin Buzzwords, so I gave it a try: jpountz.github.io/2025/06/17/a...
Andrei Dan kindly captured pictures of Luca and I telling the story of how the Lucene 10 release went
Via @rmuir.org : Linux 6.15 introduced a big speedup for Lucene on AMD processors benchmarks.mikemccandless.com/FilteredOrHi... (last data point, not annotated yet) thanks to faster TLB invalidation www.phoronix.com/review/amd-i...
Uwe now explains how Lucene takes advantage of the Panama foreign memory and vector support in spite of the fact that these features are still preview/incubating in the JDK
Uwe Schindler gives a short history of Apache Lucene at #bbuzzz
Lucene is getting faster at deep search by switching to a more efficient heap implementation to collect top hits. github.com/apache/lucen...
A nice optimization landed on the hash table that Lucene uses to build inverted indexes: github.com/apache/lucen.... Some previously unused bits are now used to cache hash codes, effectively making collisions cheaper to resolve.
