OpenSearchCon Europe 2026

In large-scale OpenSearch clusters, uneven workload distribution creates performance bottlenecks that traditional shard allocation strategies can't address. We present a groundbreaking approach to cluster optimisation through heat-aware operations, transforming how OpenSearch handles resource utilisation and workload distribution.

Our solution introduces a smart heat detection system that monitors cluster health across multiple dimensions - CPU, memory, I/O, and query patterns. By analysing both node and shard-level metrics using statistical methods, we can identify and classify various patterns of resource imbalance.

Key innovations include:
🔹 Multi-dimensional heat scoring for precise hotspot detection
🔹 Pattern recognition for automated root cause analysis
🔹 Intelligent rebalancing recommendations
🔹 Predictive scaling suggestions

Through real-world examples, we'll demonstrate how this system:
- Reduces query latency
- Optimises resource utilisation across clusters
- Prevents cascading performance issues
- Enables proactive capacity planning

Join us to explore how heat-aware operations are setting new standards for OpenSearch cluster performance and reliability.

OpenSearch's ML Commons plugin enables deploying embedding models directly on your cluster—yet many developers still rely on external services. This session demonstrates building a complete RAG system using OpenSearch as both vector store and ML inference engine.

We'll cover deploying sentence-transformer models via ML Commons, processing PDFs, generating embeddings within OpenSearch, configuring knn_vector indices, and implementing semantic search. The highlight: voice search using local Whisper (STT) model creating a fully self-contained system.

What I hope to achieve: Share practical patterns for leveraging OpenSearch's ML capabilities and gather community feedback.

What attendees gain: Step-by-step knowledge to deploy ML models on OpenSearch, build RAG pipelines, and integrate local voice processing—reproducible techniques they can apply immediately.

How this helps the ecosystem: Showcases OpenSearch's native ML features, reducing dependency on external services. Demonstrates OpenSearch as a complete AI platform, not just a search engine.

Target audience: Developers building RAG applications, OpenSearch operators exploring ML capabilities.

This talk shares a hands-on journey of building and operating a large-scale OpenSearch platform in a strictly on-premises, VM-based environment. What started as an urgent Splunk replacement evolved into a Managed Detection and Response (MDR) logging platform for high ingestion rates (~40GB to 1.2TB/day per customer). 

The session focuses on real operational challenges and experiences: sizing and sharding decisions, gaps in official documentation, and the realities of owning OpenSearch end-to-end. Initially using Terraform to deploy and manual bootstrapping.

The talk includes a migration failure story from our biggest cluster, caused by accidental double ingestion, bad sizing and snapshot-induced OOMs.

At the end we have automated deployments, bootstrapping, ISM, snapshots, tiering, SSO-based access control, searchable-snapshots(on-prem S3) and automation. Due to limitations in ISM we built a custom lifecycle automation script via an operator-like pipeline and an opensearchcli binary.

Attendees gain a realistic view of running OpenSearch at scale on-prem, including lessons learned the hard way.

Choosing the right ingestion tool for OpenSearch is a critical architectural decision. Should you use the OpenTelemetry Collector or Data Prepper? This session provides a definitive guide to answering that question, helping you design a pipeline that is both powerful and efficient.

This talk moves beyond theory into a practical comparison of design philosophies. We'll contrast the Collector's strength as a universal, lightweight agent with a vast component ecosystem, against Data Prepper's role as a purpose-built pipeline engine, optimized for powerful transformations and accessible stateful processing.

To make this concrete, the session will showcase the setup of a complete hybrid pipeline, walking through the necessary configurations. We will then deconstruct a real-world APM pipeline architecture to show how Collector and Data Prepper fulfill distinct roles in processing data before it reaches OpenSearch.

You'll leave with a clear decision framework, proven architectural patterns, and a blueprint for building your own scalable observability pipelines.

Agents and AI systems require answers that are reliable, precise and fast. In enterprise and organizational environments, these answers also must be secure. In fact, vector databases and search engines provide rapid access to knowledge. But a key challenge remains: how to correctly apply the permission models of underlying content sources within hybrid and semantic search workflows?

This presentation outlines how organizational knowledge can be securely integrated into OpenSearch. We will walk through how permissions need to be modeled, how authentication can be applied, and how these constraints propagate through the search and retrieval pipeline. In addition, we will show how prompt and query preprocessing can improve reliability and precision, particularly for agentic and AI‑driven use cases.

Lucene Vector Search Update

There have been a lot of exciting developments in Lucene's vector search implementation recently: so many it's hard to keep track. This talk will survey all the many contributions that have come in over the last year or two, give examples of how to make use the new capabilities, and point the way for future work in this area.

Some of the specific topics may include:

* seeded search (start graph walk from a known point)
* optimistic search (safe pro-rating for efficient multi-segment search)
* Acorn-based search (more efficient application of filters)
* advances in quantization
* efficient indexing using binary partitioning over vector fields
* more efficient merging: finally, we can re-use information from graphs in existing segments and don't have to start from scratch
* proposed integrations: FAISS, CUVS, DiskANN

We'll dive more deeply into some selected topics, but the general idea is to convey the breadth of activity and the diversity of contributors.

OpenSearch users increasingly need flexibility in how and where their data is stored - querying data in cloud object storage without re-ingestion, sharing datasets across analytics tools and engines, or migrating between platforms without costly ETL pipelines.

This session presents a technical deep dive into how OpenSearch can now support indexing and querying vendor-agnostic open formats natively with the new multi-engine composable architecture.

The presentation covers how the composable storage layer abstracts format-specific details while maintaining query performance. We'll explore schema management across formats, predicate pushdown to minimize data scanning, and how still existing indices like Lucene can act as layer on top of open formats as secondary indices for accelerated filtering.

Through real-world scenarios and querying production data alongside cold-tier archives in object storage, sharing datasets with other analytics engines, and leveraging specialized formats like Lance for vector workloads or Parquet / Vortex for high-compression analytics, we demonstrate how open format support delivers data portability, interoperability, and freedom from lock-in.

As OpenSearch clusters grow beyond 10–20 nodes, centralized monitoring becomes a bottleneck: single points of failure, API overload, and higher latency. Traditional approaches don’t scale.

This session presents a production-ready distributed monitoring architecture using Apache NiFi and MiNiFi. MiNiFi agents on each node collect local metrics via a custom NodeStatsProcessor (CPU, heap, JVM, I/O, thread pools), while central NiFi collectors aggregate and deduplicate cluster-wide metrics using a custom ClusterStatsProcessor and forward them to OpenSearch.

Results include linear scalability, sub-millisecond node metrics, HA, and minimal overhead. Key insights: separating local vs cluster-wide collection, deployment patterns (bare metal, VMs, containers), HA strategies with multiple NiFi collectors, and lessons from production clusters of 10–50+ nodes processing millions of metrics daily with 99.9% reliability.

Ideal for OpenSearch operators managing 10+ nodes, platform engineers building observability pipelines, and anyone hitting centralized monitoring limits. Walk away with a distributed architecture you can implement immediately.

This talk introduces Docling, an open-source document processing framework, and its integration with OpenSearch for production-ready AI applications. Docling converts complex documents such as PDFs, DOCX, and PPTX into structured, AI-ready formats while preserving layout, tables, images, and metadata for high-quality Retrieval-Augmented Generation (RAG). Docling is supported by the LF AI & Data Foundation.
The session presents practical integration patterns for advanced search applications. We showcase a multimodal RAG architecture built with Docling and OpenSearch. By combining Docling’s document conversion with OpenSearch’s vector and metadata search, we enable visually grounded RAG systems where users can verify AI-generated answers in the source documents. Documents can be indexed by running Docling externally or by invoking Docling within an OpenSearch ingest pipeline using the Docling Java API. We also demonstrate an agentic application using OpenSearch agentic search and Docling MCP for document understanding and information extraction.
Attendees will learn how these integrations simplify document ingestion and enable accurate and explainable AI applications.

Alert fatigue is rarely caused by missing data. Logs, metrics, traces, and alerts are usually available, but during incidents they remain fragmented. Engineers still correlate signals manually, under pressure, relying on experience rather than shared context.

This talk describes an AIOps approach where observability data becomes the basis for operational decisions. OpenSearch is used as the central platform to collect, query, and correlate logs, metrics, and traces, providing a consistent data model and query layer across systems. By leveraging MCP capabilities, agentic AI, and tools, automation plays a central role in closing the loop from GitOps to delivery, monitoring, and problem resolution.

From this foundation, OpenSearch can be integrated with complementary tools to extend its capabilities. KeepHQ can consume data from OpenSearch to design alerting and response workflows and connect ticketing systems and external tools. HolmesGPT can operate on data retrieved from OpenSearch to assist with incident analysis and root cause investigation.

The focus is on what happens after detection: reducing noise, correlating signals, and supporting engineers during incident response.

Running OpenSearch in Kubernetes sounds simple — until you try to do it at scale, with high availability, zero downtime, and evolving workloads. In this talk, we draw on years of production experience operating OpenSearch clusters for SAP Cloud Logging service. We aim to share the key lessons learned and design patterns from building our Kubernetes operator.

We’ll dive into how we approached seamless upgrades, built tailored monitoring, and improved performance when interacting with OpenSearch APIs. Even if we adapt solutions to fit internal needs, the lessons learned are valuable for the broader OpenSearch and Kubernetes community.

In the second half, we turn our focus to the open-source OpenSearch Kubernetes Operator. We’ll explore its current state, the challenges the community is addressing and efforts to improve its reliability. We’ll also discuss how this open-source offering enables organizations to seamlessly scale and manage OpenSearch clusters on Kubernetes, and highlight opportunities for collaboration within the project.