Centralized Application Monitoring and Logging Systems Development

Our customer

A biotech company that partnered with us aspires to revolutionize biomedical innovation by catalyzing failure in the real-world. The idea behind it is that human models of disease created with deceased donor organs unsuitable for transplantation can still offer unique insights for medical researchers. Central to the company’s expertise is a technology platform: comprising custom software, hardware, and analytics. This comprehensive solution will maximize knowledge extraction from perfused organs, accelerating biomedical advancements while minimizing related risks.

The obstacles they faced

​​The launch of the client’s solution revealed the need for an application monitoring and logging systems. The absence of these systems not only risked downtimes, increased costs, and customer dissatisfaction but also posed key challenges such as incomplete system health views, difficulty troubleshooting, and heightened security vulnerabilities in the AWS environment.

How we helped

The centralized logging and monitoring systems we build markedly improved the client’s AWS environment, providing swift incident detection and resolution, enhanced infrastructure visibility, and improved troubleshooting capabilities. Implemented solutions have also lowered the risk of security breaches and bolstered compliance with regulations.

Log processing and issue detection on the application level

Romexsoft has already set up the biotech company’s AWS Organisation governance by AWS Control Tower with a dedicated Logs Archive AWS Account and suggested setting up a Centralized Logging System in it and building on top of Amazon OpenSearch Service. This approach allows for collecting and storing logs from different sources in one place as well as to navigate easily across various facets and data attributes.

Moreover, by utilizing the OpenSearch Trace Analytics feature we will build analytics for the distributed tracing of requests and messages between the microservices in the upcoming releases.

We suggested installing Fluentd td-agent on each ECS service running Java applications in AWS accounts to stream logs into the Logs Archive AWS account. To avoid the increasing size of OpenSearch logs cluster we suggested implementing logs rotation logic and storing all logs data older than 30 days in the S3 bucket.

Metrics collecting and monitoring on the infrastructure level

For infrastructure monitoring level, we’ve suggested utilizing the AWS CloudWatch service. It is a robust monitoring service that enables us to collect and track metrics, and set alarms, allowing us to gain valuable insights into the client’s AWS infrastructure performance. With features like automated scaling and integration with Grafana as a data source, CloudWatch empowers to proactively manage and optimize AWS resources for better efficiency and cost-effectiveness.

Comprehensive observability stack

To react to the incidents quickly and troubleshoot them effectively in addition to the OpenSearch Service Romexsoft suggested setting up a dashboard in Grafana to visualize all critical metrics and integrate alerts about the company’s incidents into the Slack channel using AWS Chatbot service.

Grafana will be running in containers on the ECS Fargate service. It allows not only to meet common conventions of the whole ecosystem because the majority solution’s services run on ECS Fargate, but also ensures easy technical management and cost efficiency over the project.

Strong security posture and compliance

In order to meet security compliances both services are located in the private subnets of AWS VPC and require secured VPN connectivity to access the dashboards. The private subnets are isolated from the public internet, which further reduces the risk of security breaches. It also means that potential attackers have fewer opportunities to gain access to the services and sensitive data.

The VPN connectivity can be audited to track who has access to the dashboards and what actions they have taken. This can help the biotech company to identify and respond to security incidents more quickly. Moreover, by placing the services in private subnets and requiring secured VPN connectivity, the client can demonstrate compliance with a variety of security regulations, such as PCI DSS, ISO 27001, and HIPAA.

Application Monitoring and Logging Systems – Architecture Diagram

The centralized application logging and infrastructure monitoring systems implemented by Romexsoft have had a significant impact on the client’s AWS environment. The client got effective instrumentation for fast detecting and resolving incidents, improved visibility of the application infrastructure, and increased ability to understand and troubleshoot complex issues. Additionally, the biotech company has reduced the risk of security breaches and improved compliance with security regulations.

The project’s key accomplishments:

• Swift detection and resolution of application issues
• Rapid identification and remediation of performance bottlenecks
• Enabled proactive monitoring of the system
• Implemented observability of the application infrastructure
• Established ability to understand and troubleshoot complex issues
• Overall reliability and stability of the system
• Reduced the risk of security breaches
• Compliance with security regulations
• Strengthened overall security posture of the system.