AWS OpenSearch New Features: The Solutions for Common Challenges

What is OpenSearch Project: Inception and Evolution

Basically, search is all about efficiently extracting insights from data. Even before the digital age, as people needed to find information, they eventually invented cataloging schemes. In contrast, modern text-based search, like in OpenSearch with Lucene, relies on inverted indexes. The present-day search tendencies are rooted in the rise of artificial intelligence and machine learning. One more significant trend is combining text-based and natural language search, which you can read about further in the article.

As OpenSearch serves as a powerful search engine with robust capabilities, particularly in log analytics, we are going to devote the second section of the article to innovations when it comes to extracting valuable insights from log data.

Since its birth in 2021, the OpenSearch initiative has seen significant growth. Taking only the framework of 2023 as an example, the Project tripled its figures in downloads (100 million at its beginning compared to 300 million at the close of the year) and almost doubled the number of partners adopting the given search platform (around 40 contrasted with over 70 influential partners, including ISVs, solutions providers, and cloud providers) and the growth rate is continuing to gain momentum. It is not merely the numbers that search service development is manifested in, the community of contributors to the Project, which is now as large as 1,500 code committers with over half of them outside Amazon, has attracted such industry giants as SAP and Intel.

An even more compelling fact is that OpenSearch has become a multi-cloud solution with its native services available on not only AWS but also Oracle Cloud, Microsoft Azure and Google Cloud Platform (with the assistance of partners, like Bonsai and Aiven).

Amazon OpenSearch Service

Unlike the community-driven, open-source initiative by Amazon, which we have just briefly looked into, Amazon OpenSearch, is a managed service offered by AWS that is based on the OpenSearch project. It includes additional features, management tools, and support services provided by AWS. While both are built on the same foundation, Amazon OpenSearch offers a fully managed solution with added AWS-specific capabilities and support. Should you be more interested in the AWS service, have a comprehensive read about it on our website.

As per the AWS OpenSearch Service, it makes efficient and secure management of the OpenSearch engine even easier, regardless of your purpose. Amazon OpenSearch comes in handy for both enhancing the relevance of your search results and conducting cost-effective log analytics. The integration with a range of AWS services, including AI/ML services, makes leveraging additional capabilities through the OpenSearch Service seamless.

Talking numbers, OpenSearch Service can boast of collaborating with tens of thousands of customers, processing trillions of requests monthly, and storing hundreds of petabytes of data. To top it all off, the OpenSearch customers’ list includes such brands as Adobe, Zillow, Uber, Netflix, and Samsung to name just a few.

The service now represents a diverse ecosystem, and much like it includes various data types like documents, logs, and transactions that require integration, it also involves professionals from search engineers to data scientists. Each of these experts faces numerous challenges while working in data management, analytics, search optimization, and system monitoring across various industries and domains, so we are going to examine them more closely.

Challenges of Data Engineers

Let us explore the challenges faced by data engineers tasked with integrating data into OpenSearch. Typically, they are supposed to construct pipelines to facilitate data flow into OpenSearch. For this purpose, they may employ various tools, such as Logstash on EC2 instances, Lambdas, or streaming services.

Meanwhile, integrating data into OpenSearch presents its own set of obstacles: engineers must determine the best method for collecting data. Subsequently, they need to ensure data durability by storing it in S3 as well as implement buffering mechanisms to handle scenarios where downstream systems are unavailable or encounter impedance mismatches.

Additionally, the need to transform the data may involve tasks like removing duplicates or conditionally routing data to different clusters. These transformations add complexity, often requiring custom code to be written. Furthermore, any changes in log data or documents necessitate updating this code, making the process rather cumbersome.

Solution: Amazon OpenSearch Ingestion Service

Simplified data integration
Introduced in the middle of 2023, Amazon Ingestion is meant to address these challenges head-on. The service simplifies the entire data integration process, handling all the heavy lifting effortlessly in order to effortlessly send your logs or documents to the endpoint with the use of connectors to S3 and Kafka, among other sources.

Delving deeper into the Ingestion service, it introduces the concept of sources, which connect to different transactional systems or handle log data. Various processors, including those for data enrichment and duplicate removal, can be seamlessly integrated. The synchronization point typically revolves around the Amazon OpenSearch Service.

Serverless and scalable architecture
Within Amazon Ingestion, one can configure different processors to transform your data without the need for coding so as to seamlessly transmit the data into an OpenSearch cluster. As far as user experience is concerned, the serverless service dynamically adjusts to traffic demands without the need for manual scaling or sizing.

Efficiency and cost-effectiveness
One of the standout features of the Amazon Ingestion service is its efficiency. Through rigorous benchmarks, it has been determined approximately 60 to 70% more efficient than Logstash, making it a highly cost-effective solution for data ingestion into OpenSearch.

Support for data Buffering and migration
The novel additional capabilities include the ability to buffer data within the Ingestion service, which alleviates concerns about downstream system availability, alongside the support of Elastic 7.x versions as well as older versions of OpenSearch, facilitating smooth data migration processes for users transitioning to the latest versions.

We are now going to look into the most common obstacles that OpenSearch-associated personnel struggle with, along with how to tackle them effectively with the help of the most recent AWS features.

Challenge of Integrating Search Functionality

Another challenge frequently mentioned by engineers is integrating search functionality into applications built on platforms like DynamoDB. When incorporating search capabilities into DynamoDB-based applications, transitioning data to a search platform like OpenSearch poses a significant hurdle. Typically, data engineers resort to developing pipelines, often employing Lambda functions or running processes on EC2 instances utilizing streams. This process can be daunting and complex for users seeking to seamlessly integrate search functionality into their DynamoDB-powered applications.

Solution: Zero-ETL integration

At the close of 2023, a Zero-ETL integration with DynamoDB was unveiled, intended to simplify the synchronization of DynamoDB tables with OpenSearch indices. This integration can be easily accessed through the Dynamo console, allowing users to specify the index they wish to utilize for OpenSearch upon table creation. With this integration, the Ingestion service takes care of all the complex tasks, which ensures seamless data transfer from DynamoDB to OpenSearch and keeps it continuously updated.

Effortless synchronization and seamless updates
Reflecting on the journey of data engineers dealing with data integration into OpenSearch, these innovations, including the Ingestion service and Zero-ETL integration with DynamoDB, have significantly eased the process. Data engineers can now focus on more meaningful tasks without being burdened by the intricacies of data synchronization.

Search Relevance Challenge

Data integration discussed, let us now transition to explore how these innovations facilitate search functionality and observability use cases within the OpenSearch Service ecosystem.

Ensuring relevance
As regards search functionality management, it poses several challenges to users. Firstly, ensuring that search results are relevant and aligned with business objectives requires constant fine-tuning of search algorithms and configuring result-boosting strategies. This process demands a deep understanding of user intent and business requirements, as well as continuous monitoring and adjustment to maintain relevance over time.

Managing synonym files
Dealing with synonym files adds another layer of complexity. Users must carefully manage synonyms to ensure that search queries yield accurate and comprehensive results, which can be a time-consuming and intricate task.

Leveraging advanced features
Furthermore, while Amazon OpenSearch offers a range of powerful capabilities out of the box, such as faceting and geospatial capabilities, effectively leveraging these features requires expertise and familiarity with the platform. Users must understand how to configure and customize these functionalities to suit their specific use cases and requirements.

Solution: AI and ML Capabilities

Incorporating new AI and ML techniques into OpenSearch expands the toolbox available to users, empowering them to deliver optimal results to customers. Traditional text search methods, such as those employed by Amazon.com, utilize a catalog of data and search terms to match customer intent, scoring each match to determine relevance and presenting the highest-ranked results. Users can also utilize faceting to refine search results further.

Enhanced human language interpretations
Recent advancements in AI and ML have revolutionized language understanding, enabling models to grasp English language nuances and interpret user intent more effectively than conventional text prompts. By integrating these technologies into OpenSearch Service, one gains access to powerful features that augment traditional search methods, facilitating more accurate and intuitive search experiences for customers.

AI and ML Data Processing Complexity Challenge

Language processing, simply put
In understanding language, AI and ML utilize a process of converting information into vectors, which are rows of numbers representing dimensions, – often numbering in the thousands, form the model’s interpretation of the document’s content. During a search, the system seeks the closest match in this high-dimensional space to retrieve the most relevant objects corresponding to the query.

This methodology is applied across various data types, including images, audio, logs, and text documents. Documents are inputted into the model, which then translates them into vectors. These vectors can be stored in a vector database, with the Amazon OpenSearch Service and its vector engine serving as popular choices for vector storage solutions. This approach allows models to effectively process and analyze diverse forms of data, facilitating accurate and relevant search results for users.

If we simplify the concept of a basic chat flow, a user initiates a question that gets processed by a language understanding model (LLM), converting it into a vector. This vector is then sent to OpenSearch, which retrieves a set of results that are translated back into human language and returned to the user.

Inherent complexities
However, real-world applications of gen AI are typically more complex. To create a seamless user experience, multiple stages of analysis are often required, including reasoning and chain of reasoning. Achieving this involves building an application middleware, which can be facilitated by tools like LangChain, LlamaIndex, or Haystack. Each step in this middleware workflow may involve interactions with one or more models, as well as vector databases. Despite the complexity, investing effort into building such middleware is necessary to ensure a robust and comprehensive gen AI application.

See the next block to find out how Amazon decided to bypass the need for middleware development and at the same time provide seamless, native access to all other OpenSearch capabilities within the same pipeline.

Solution: OpenSearch Neural Search

Enhanced indexing and search pipeline
Neural Search in OpenSearch enables applications to interact seamlessly with OpenSearch using familiar APIs. Within OpenSearch, the indexing and search pipelines have been decomposed to streamline the process. When a document is sent to OpenSearch, it enters the Ingest pipeline. For instance, if the document is an image, it can be passed to a model to generate embeddings, which are then stored along with other metadata.

Similarly, in the search pipeline, multiple stages can be configured. For example, the pipeline may begin with a lexical search followed by semantic search results from a model. These results can be compared and assigned a composite score. Additional stages might involve personalized ranking based on user preferences and even result summarization. Importantly, all of this functionality can be achieved without the need to develop middleware, using only OpenSearch APIs for seamless integration.

Streamlined testing
This approach also simplifies the process of testing out new ideas and innovations. With the constant emergence of new models, having a stable application stack provides a reliable foundation for experimentation. The goal is to empower users to explore and implement new concepts effortlessly, fostering a culture of innovation and continuous improvement.

Other Novel Features for Relevant Search

Hybrid search

Neural Search aside, one notable addition is hybrid search, which enables the integration of textual analysis and the advanced capabilities of Amazon OpenSearch Service with vector similarity search and composite scoring. This enhancement offers users a more comprehensive and versatile search experience, allowing for greater flexibility and accuracy in retrieving relevant information.

Fine-tuned models

Another significant capability introduced is the option to deploy customized, fine-tuned models on SageMaker and seamlessly integrate them with the OpenSearch Service. This feature offers a no-code integration, simplifying the management of clusters. With this functionality, users can execute their tailored models and seamlessly access them through Neural Search, enhancing the overall search experience and flexibility of the system.

Sparse vector retrieval

In the recent release of version 2.11, Amazon introduced sparse vector retrieval, a novel search method. Sparse vectors, unlike dense ones with thousands of dimensions, use fewer tokens aligned with index properties, offering lighter processing and rivaling lexical search techniques in performance. Testing shows promising results, indicating the ability to discern intent with reduced computational load.

Multimodal search

Multimodal search involves combining both image and text data. With this approach, users can input images into the multimodal model, which then encodes the embeddings to identify the content within the image without requiring manual data labeling or preprocessing.

Integrations tab

Within AWS OpenSearch service, an Integrations tab has been introduced. What one finds there are cloud formation templates, which enable you to quickly deploy features such as Bedrock Titan Text Embeddings and the sparse model with minimal manual intervention.

These were just a few advancements enhancing the capabilities of the OpenSearch platform. Amazon is committed to expanding its capabilities in both vector search and the Neural Search plugin, where such features as composite scoring, hybrid scoring, custom models, sparse vector retrieval, and multimodal support are being added. These developments aim to provide users with the tools they need to unlock new possibilities and achieve their objectives effectively.

Log Analytics and Observability Challenge

AWS OpenSearch serves as a widely embraced tool for log analytics and observability use cases, attracting a significant user base due to its robust capabilities. Its distributed nature and ability to handle substantial data streams while delivering rapid query responses make it particularly suitable for such tasks.

Leveraging complexity
For developers and DevOps engineers, leveraging OpenSearch Service entails mastering query writing for forensic analysis, creating customized dashboards, configuring alerts, and manually correlating disparate data sources to identify relevant issues. At the same time, it is Security Analytics where additional tooling matters most to obtain insights effectively.

Downtime issues
In scenarios where individuals bear the responsibility of addressing issues promptly, having efficient tools becomes paramount. Minimizing downtime not only reduces stress but also facilitates swift problem identification, forensic analysis, and root cause determination, enabling users to resume normal activities without undue delay.

When managing system operations, the necessity for robust tooling and minimizing downtime cannot be overstated. The tools at one’s disposal during critical moments, such as resolving issues in the middle of the night, play a pivotal role in maintaining system integrity and user satisfaction. Every moment of system downtime not only translates to potential revenue loss but also incurs stress and frustration for those tasked with resolving the issue. Effective tools can streamline the process of problem identification, analysis, and resolution.

Solution: OpenSearch Observability Features

In order to address these challenges, the AWS team introduced observability features to OpenSearch a few years back. These additions include built-in anomaly detection and alert systems. OpenSearch also offers extensive support for open telemetry data. With its help, users can conduct tracing and visualize spans and service maps, which proves invaluable in identifying and isolating issues within complex environments. Additionally, OpenSearch incorporates features like log pattern analysis, tailing, and surrounding, which all add up to further streamlining the troubleshooting process.

Piped processing language
Another novelty feature in question, commonly known as PPL, was specifically tailored for efficient data discovery and exploration. PPL allows for a logical flow of operations so as to seamlessly sort, group, and filter data. This all makes exploration tasks more intuitive compared to traditional methods like SQL or OpenSearch DSL. Recently, Piped Processing Language has been expanded to include full visualization support for Jaeger, a widely used tracing format. This includes features such as span visualization, trace groups, and service maps. Another innovation is the automated extraction of metrics from logs, ensuring they are correlated with the broader system for comprehensive analysis.

Search and Analytics Challenges

Complex query formulation
Performing search and analytics is often obstructed by the complexity of formulating queries and extracting meaningful insights from data. Traditional query languages like SQL or OpenSearch DSL can be cumbersome and require a steep learning curve, especially for users without a technical background. Many engineers find interpreting and visualizing large volumes of data to derive actionable insights rather time-consuming.

Navigating data structures
Another thing is that users may face difficulties in navigating complex data structures and understanding the relationships between different data points. Without intuitive tools and interfaces, the process of exploring and analyzing data can be inefficient. Inconsistent data formats and disparate sources further compound the challenges.

Solution: OpenSearch Assistant Toolkit

A step towards tackling those challenges has been made with the OpenSearch Assistant Toolkit. It enables users to write queries using natural language in order to simplify the process of formulating complex search queries. Once a query is executed, the Assistant automatically summarizes the results, so the insight generation becomes smooth and quick.

Speaking of the near future, the Toolkit is certain to grow new capabilities, such as creating visualizations or setting alerts based on specified thresholds; these features aim to streamline the problem-solving process and enhance user productivity. The toolkit is currently available as open source and will be integrated into the OpenSearch Service in the future to offer users a customizable solution for their search and analytics needs.

One can now explore the functionality of the OpenSearch Assistant Toolkit at observability.playground.OpenSearch.org. The site provides an interactive environment to experiment with querying data and experiencing the summarization feature firsthand. The toolkit’s architecture leverages Anthropic Claude in the backend, which ensures robust reasoning logic for efficient query processing.

Security Monitoring and Observability Challenge

Threat database integration
It has always been duly noted that utilizing OpenSearch Service for Security Analytics can be challenging. Unlike observability and monitoring tasks that involve aggregations, such as checking if error rates exceed certain thresholds over a period, Security Analytics demands examining each log line against a threat database. This meticulous process underscores the complexity of security analysis compared to other monitoring activities.

Complex query logic
Security analytics often involves intricate query logic to identify patterns or anomalies indicative of security threats. Crafting and optimizing these queries within OpenSearch Service to efficiently analyze large volumes of log data can pose a significant problem.

Real-time detection
Effective security analytics necessitates real-time detection and response to security incidents or suspicious activities. Implementing real-time monitoring and alerting capabilities within OpenSearch requires careful configuration and optimization to ensure timely detection of security threats.

Data privacy and compliance
Security analytics often involve sensitive and confidential information, requiring strict adherence to data privacy regulations and compliance standards. Ensuring that security analytics solutions built on OpenSearch comply with relevant privacy laws and industry regulations adds another layer of complexity.

Solution: Security Analytics

In response to the poignant needs, Security Analytics was introduced last year, integrating alerting and a rules engine capable of analyzing logs against both custom and 2,200 Sigma rules upon ingestion. Its capabilities detect issues in every log line and employ a correlation engine to closely associate threats across different entities, such as hosts and requesters. The correlation engine automatically constructs a graph to visualize potential threat relationships without manual configuration.

Reliability Challenge

One of the recurrent challenges faced by users of Amazon OpenSearch pertains to ensuring the platform’s reliability, particularly when nodes or entire Availability Zones encounter downtime. Although in systems like the service data and operations are distributed across multiple nodes and AZs for fault tolerance and scalability, when a node fails or an entire AZ experiences downtime, it can disrupt the system’s operation and compromise its reliability.

Achieving a 99.99% uptime, which is often a requirement for critical applications and services, becomes especially challenging in such scenarios. Downtime, even if brief, can lead to service disruptions, which affect user experience and potentially cause financial or reputational damage.

Solution: Multi-AZ with Standby

As of early 2023, Amazon unveiled Multi-AZ with standby, a groundbreaking enhancement to bolster reliability. This innovative feature guarantees 99.99% availability by significantly reducing data transfer during AZ or node failures. By incorporating a standby AZ, the system seamlessly transitions to ensure uninterrupted operation. This capability is especially advantageous for latency-sensitive applications, as it maintains high availability without any downtime, even during unexpected events.

OpenSearch optimized instance family
One more 2023 Amazon OpenSearch announcement concerns their optimized instance family. This instance type offers an 80% increase in throughput and a 30% enhancement in price performance. If your workload involves heavy indexing or writing, you will enjoy the advantages of higher throughput and reduced costs. Additionally, these instances boast high durability, comparable to S3, as they are now supported by S3.

Traditional Replication Challenges

In Elasticsearch and older versions of OpenSearch, replication involves indexing a document or log into one node and then replicating it to another node, essentially indexing the data twice. The data are not going to be stored in cloud storage like S3. Therefore, if a node fails, you rely on the replicated data. It is crucial to ensure proper replication levels for data redundancy and fault tolerance.

Solution: Leveraging Cloud Storage (Amazon S3)

This capability involves pushing physical segments of data to S3 instead of replicating documents. Replicas then download these segments from S3, ensuring an RP of zero in case of node failure. This approach improves throughput by eliminating redundant indexing and downloading segments from S3. Overall, it offers high throughput at a lower cost.

Scaling Challenges

So as to ensure efficient data storage, processing, and retrieval in large-scale environments of distributed systems like OpenSearch, the following two become critical.

Firstly, what allows for better performance and scalability is sharding (dividing a large dataset into smaller parts called shards), with its function of distributing the data and workload across multiple nodes.

The latter, in turn, have to be managed. Monitoring of individual nodes, with roles from data storage and indexing to coordinating operations within the cluster, ensures that the cluster operates smoothly, with adequate resources allocated to handle incoming requests and data processing tasks.

Unfortunately, handling both of these vital components of OpenSearch manually can be complex, especially as the dataset grows or changes in structure.

Solution: Serverless

For this reason, OpenSearch launched Serverless functionality, which enables automatic scaling based on traffic without the need for sharding or node management. Simply send data to the Serverless endpoint, and it handles scaling operations seamlessly.

Examining the Serverless architecture, storage and compute have been decoupled, as well as indexing and search. Indexing occurs on dedicated nodes, with data stored in S3, while search operations are handled by separate nodes, this decision enables independent scalability.

Further additions to Serverless include the Dev-Test Collections, which reduces the initial cost of Serverless. You can now begin with just one OpenSearch Compute Units (OCU) for indexing and one OCU for search.

Since vector databases are gaining momentum, and OpenSearch has offered this capability for years, it has introduced vector engine support in Serverless. What can now be done with the novel feature is performing the RAG workflows without managing any infrastructure: simply create a vector collection and start sending your vector data to OpenSearch Serverless. With the addition of Dev-Test Collections, the starting cost is also very low.