Log Search and Processing Solution on Amazon OpenSearch Service
Discover how we implemented a log search and processing solution, which enhances observability and security for client's platform.
Executive Summary
Optimized Log Analytics
Our Customer
Gorgany is a huge retail, wholesale & distribution company of outdoor equipment in Ukraine. The company specializes in the wholesale trade of goods for tourism, mountaineering, skiing, and active recreation.
The Obstacles They Faced
The client’s platform had difficulties and constraints in efficiently identifying and resolving issues on their log collection and processing system. It required too much of engineers manual work and impacted on the resilience and versatility of the platform as a whole.
How We Helped
By leveraging the power of a new log search and processing solution built on Amazon OpenSearch by our professionals, the client gained the ability to promptly resolve any potential breakdowns, ensuring uninterrupted performance and optimal functionality of their system.
The Challenges
Time-Consuming Log Retrieval
Previously, all the logs of the client’s e-commerce platform were collected and processed via Amazon S3 buckets – cloud object storage. This solution was inconvenient and time-consuming for the identification of occurred issues with the product: the engineers had to download the files whenever an issue arose, and then to search needed data in each file, which significantly impeded the efficiency of issue resolutions. As a result, log search and processing were the weak points of the client’s platform, which in turn affected the all-encompassing observability and stability of the system.
The Solution
Centralized Logging with OpenSearch
Having conducted a thorough analysis of the challenge, our experts resolved to rely on Amazon OpenSearch Service as the technological cornerstone of a newly designed log processing solution. This was mainly motivated by two factors, namely:
- the capability of this service to collect and store logs from different sources in one place;
- the functionality of fast and easy navigation across diverse facets and data attributes.
The solution is built and configured in the following way: every ECS task definition has its own AWS ECS Fargate container, responsible for collecting and transmitting the data to the Amazon OpenSearch cluster. The same functionality of collecting and routing the data to the OpenSearch cluster is also implemented with the td-agents – an open-source and multi-platform log processor and forwarder. Then, on the side of Amazon OpenSearch, all the collected logs are stored, processed, and searched whenever required.
Log search and processing solution on Amazon OpenSearch Service – Architecture Diagram
The Results
Streamlined Downtime Management
- Better observability of the client’s platform through modernized log search and processing
- Decreased downtime and optimized time and costs resources associated with it
The implementation of a log search and processing solution with AWS OpenSearch developed by Romexsoft professionals facilitates the rapid identification of the root cause of any technical issue of the product and enables the timely resolution of the possible breakdowns.
Why Romexsoft
Trusted AWS Consulting Partner
Romexsoft is AWS Advanced Tier Services Partner, trusted Software Development Company and Managed Service Provider, founded in 2004. We help customer-centric companies build, run, and optimize their cloud systems on AWS with creative, stable, and cost-efficient solutions.
Our key values
- Delivery of quality solutions
- Customer satisfaction
- Long-term partnership
We have successfully delivered 100+ projects and have a proven track record in FinTech, HealthCare, AdTech, and Media industries.
Romexsoft possesses a 5-star rating on Clutch due to its strong expertise, responsiveness, and commitment. 60% of our clients have been working with us for over 4 years.
Related Success Stories
Log search and processing solution FAQ
Amazon OpenSearch Service provides a robust solution for log search and processing. It allows for the collection and storage of logs from different sources in one place, facilitating easy navigation across diverse facets and data attributes. This significantly enhances the efficiency of issue resolutions, as engineers no longer have to manually download and search through individual files whenever an issue arises.
The solution is built and configured in such a way that every ECS task definition has its own AWS ECS Fargate container, which is responsible for collecting and transmitting the data to the Amazon OpenSearch cluster. The same functionality of collecting and routing the data to the OpenSearch cluster is also implemented with the td-agents, an open-source and multi-platform log processor and forwarder. Then, on the side of Amazon OpenSearch, all the collected logs are stored, processed, and searched whenever required.
Implementing a log search and processing solution with Amazon OpenSearch Service can lead to better observability of the platform through modernized log search and processing. It can also decrease downtime and optimize time and cost resources associated with it. The solution facilitates the rapid identification of the root cause of any technical issue and enables the timely resolution of possible breakdowns.
Why is Amazon S3 buckets not an optimal solution for log collection and processing?
While Amazon S3 buckets can be used for log collection and processing, this method can be inconvenient and time-consuming for identifying issues. Engineers would have to download the files whenever an issue arose, and then search for the needed data in each file. This significantly impedes the efficiency of issue resolutions and can affect the overall observability and stability of the system.
