Custom Logs Management Solution on Amazon OpenSearch

Our Customer

Omnyfy is a platform as a service (PaaS) multi-vendor E-commerce marketplace platform catering to B2B, B2C or Service marketplaces. The platform has been adopted by some of Australia’s largest enterprises to power multi-million dollar marketplaces.

The Obstacles They Faced

Cumbersome work of manual logs searching from multiple sources as the result of the absence of a dedicated platform’s solution for logs management.

How We Helped

Keeping the whole platform’s IT infrastructure scalable, our experts developed a centralized logs processing solution, implemented effective issue detection and incident notification mechanisms, and ensured controlled access to logs.

Centralized logs monitoring and issue detection

In order to handle this issue, our suggestion was to start by establishing a Centralized Logging System (Covered area) in a management account, which is built on top of Amazon OpenSearch and OpenSearch Dashboard. Specifically for this scenario, we installed Fluent td-agent on each EC2 instance of a separate marketplace to stream all the logs accounts to stream logs into the Amazon OpenSearch management account. By leveraging Amazon OpenSearch, we successfully centralized log collection into a single location.

Then, we set up logs monitoring system based on specific phrases and words: the solution automatically sends notifications via AWS SNS to the client whenever the preset keywords (phrases and words) occur in the logs. This feature helps to detect and distinguish issues on the platform timely and appropriately.

Differentiated access to the logs for the platform’s users

Another essential request from Omnyfy was to provide access to the logs for their clients (marketplace’s owners) through the OpenSearch dashboard, however, it was crucial to ensure access management that would be tailored only to each client’s individual. Here is where Amazon Cognito & Amazon OpenSearch take the scene for ensuring required access segregation.

To fulfill this requirement in accordance with AWS security best practices, we implemented log segregation with separate tenants for each marketplace (at that moment Omnyfy owned around 30 client’s marketplaces). As a part of the solution, Amazon Cognito authentication for OpenSearch was configured granting granular access to specific indices. To ensure proper identity management, the identity provider was set up to use an IAM role that corresponds with the user’s authentication token. Users belonging to a specific marketplace were grouped together in Cognito, on the basis of their specific IAM roles, thus ensuring that logs can only be viewed from their respective marketplace, and that the access requirements are met.

Custom Logs Management on Amazon OpenSearch – Architecture Diagram

Romexsoft successfully addressed the client’s challenges by providing a centralized and efficient logs processing solution, implementing effective issue detection and incident notification mechanisms, ensuring controlled access to logs, and establishing a scalable infrastructure. These deliverables significantly enhanced the client’s platform management and troubleshooting capabilities, improving overall performance and customer satisfaction.

Let’s dive into the details of the project’s key accomplishments:

• Centralized logs collecting and monitoring streamlines the process of log gathering, searching, and visualizing, which results in improved efficiency and time savings in engineers’ infrastructure monitoring routine.
• Accelerated identification and resolution of the application incidents ensured by automated issue detection and issue notification solutions.
• Secure and granular access control for different marketplace owners. This met the client’s requirement of providing access to logs for their clients while maintaining data privacy and security.