Modern Data Architecture with AWS Cloud

In light of the current ceaseless expansion of data volume and the emergence of new data types, the modernization of data analytics systems is in increased demand. As it is not always transparent when modern architecture of data is mandatory, and whether or how exactly your business may benefit from this kind of investment, we have compiled this article to clarify the link between poignant enterprise issues and the solutions suggested by modern data system, in particular, offered by Amazon Web Service (AWS).

The article intends to keep you posted on the process and consequences of advanced data infrastructure data architecture implementation with a view to handling your data-related stumbling blocks, and boosting analytics which leads to new business outcomes.

The problematique of data architecture

Over the years, enterprise data warehouses (EDW) have been preferable among numerous businesses as the single version of the truth. Offering a consolidated historical business information repository for data of various origins, EDW is referred to as the semantic layer of hierarchically centered data stored in files and folders.

Data lakes, on the other hand, store and process information within their flat architecture, whether in a file or object storage. Their signature feature is containing a substantial amount of raw information in its native format until it is to be analyzed.

EDW data reports have led many organizations by providing insights into business strategies and governing decision-making. The last two decades witnessed the two most common approaches to executing EDW:

Kimball approach (by Ralph Kimball)
Inmon approach (by Bill Inmon)

Kimball approach

One commonly accepted outlook on data warehouse design is Ralph Kimball’s perception whereby it is a bottom-up process. The information is collected in data marts (or star schemas), each tied up to a specific business context yet unified under common business dimensions, that together constitute the data warehouse.

According to Kimball, information can be integrated into the data warehouse gradually through conformed data dimensions. New data marts are introduced with the addition of fact tables as the central element of a star schema as well as with the attachment of new dimensions to the present conformed data dimensions.

Let’s take a look at the arguments for and against preferring Kimball’s approach to creating a data warehouse.

Benefits:

• A specialized and logically grouped set of data marts.
• Gradually compiled for a business use case, containing all data.
• Made up of denormalized data.
• Designed for a specific business domain, thus comprehensible for non-IT users.
• Suitable for MPP Relational Database Management Systems (RDBMS).
• Appropriate for data lake implementations.

Shortcomings:

• A properly synchronized integration layer is virtually impossible to achieve due to the data duplication and data sync issues, alongside EDW being case-driven.
• Changes that lessen the impact on customers are delayed because of tight coupling with them.
• The tables’ large amounts of data need labor-intensive and time-consuming extract, transform, and load (ETL) operations.

Inmon approach

In contrast with Kimball’s approach, Bill Inmon regards a top-down approach as optimal for EDW design. Having a normalized data model as his aim, he advocates producing a unified data model in accordance with the business context for all the upcoming information items to be translated into it.

More specifically, in order to enter the EDW, data from various sources need to be unified, formatted, typecasted, and semantically translated to comply with the previously defined unified data model established by the enterprise.

Further, we present the most commonly mentioned merits and demerits of Inmon’s EDW creation approach.

Benefits:

• Significantly normalized, with low data redundancy.
• Clear in data relationships and granularity, characterized by proper semantics.
• Independently evolving since it is decoupled from end users.
• Improved for the storage of historical data.
• A better fit for RDBS implementations.

Shortcomings:

• Given the data model complexity, substantial time and effort investment is required to prepare data from various sources to fit the unified model.
• Data sources dependency and minimized parallelism are brought about by the integrated data model.
• The computation required to do ad hoc analysis with the Third normal form (3NF) is quite high.
• As a rule, source data requires splitting to suit the EDW data model and the process of forming a final integrated dataset from various sources is even more complex.
• The best analytical query performance is still achieved with a denormalized in a star or snowflake schema presentation layer on top of the EDW.
• Inmon’s top-down approach is less adapted for data lake implementation.

So, the problem is

While both approaches to data warehouse design have delivered value through their advantages, modern data quantities cannot be managed by conventional on-premises systems for reasons including matters of scaling, flexibility and price.

Traditional data-storing systems were designed with the view to combining information of various origins and generating operational reports. They, however, were not created to accommodate the rapid expansion of event data (like log files, or machine-generated data from the Internet of Things devices).

Those factors, together with the current rapid expansion of data, make traditional data warehouses less agile and likely to lengthen the time-to-market for realizing solutions. In addition, data apps tend to employ a variety of tools to be built and deployed whereas conventional data warehouse models support only a limited number of such tools, e.g. SQL and BI workloads.

This, merged with the development of cloud and big data capabilities in recent years contributes to organizations moving toward a modern data pattern. Customers now choose to build data lakes to store all of their data, both structured and unstructured, in one location and analyze this data with a wide range of tools provided.

To fully capitalize on their data, end users are increasingly adopting advanced data systems that merge the advantages of both data lakes and specialized data storage solutions. The practical decision, for now, incorporates storing all open-formatted data in data lakes and occasionally using conventional tailor-made data services to process and analyze them.

What is modern data architecture?

Technology advancements, ceaselessly increasing amounts of various data and fierce competition all force enterprises to gather the information, amass and process it to extract the maximum value in the shortest time possible. Data warehouses were a response to a particular type of use case, which was Online Analytics Processing (OLAP). Today, the EDW’s inherent one-size-fits-all data strategy is hardly sufficient to accommodate the needs of, let’s say, log analytics, predictive analytics, or big data processing, and more often than not impedes further scaling and development.

As aforementioned, a modern architecture enables incorporating the benefits of both data lakes and data stores. By adopting the contemporary data strategy referred to above, you can cut back on the data silos, and enable your teams to utilize the most suitable tools for conducting analytics or machine learning tasks. This approach also ensures that data accessibility is safeguarded with the necessary governance controls.

A proper data foundation entails:

• Data infrastructure modernization.
• Unification of the benefits of data lakes and data stores.
• Innovation and reimagining old processes with AI/ML.

At the same time, cutting-edge data architecture:

• Is scalable, performant and cost-effective.
• Includes custom-designed data services.
• Supports open-data formats.
• Manages decoupled storage and computing.
• Provides seamless data movements.
• Allows diverse consumption mechanisms.
• Ensures governing and security.

Why you need a modern architecture for your data

If your objective is to gain and retain a competitive position in the market, your data will benefit from a whole new approach, as well as cloud-native tools and other components to administer, integrate and process them.

While conventional on-premise data analytics is unable to process the steadily expanding data amount or scale fast enough, modern system for data ensures:

• Accelerated and improved decision-making by eliminating data silos.
• Enhanced customer experience, and thus loyalty.
• Data-driven insights for innovation and competitive advantage.
• Business situations understanding, more accurate future outcomes prediction.
• Business process optimization and operational cost reduction.

What business challenges the architecture solves

More data than ever generated
The unprecedented pace of data volume growth produces more obstacles with its storage, processing and analysis.

Dealing with diverse types of data
On-premise data storage is far from being the most cost-efficient, best-scaling option to manage recent types of data, such as structured data, unstructured data, or real-time streaming data.

ML adoption
Traditional data warehouses cannot cater to the functional requirements of machine learning if future-oriented innovation is your choice.

Data compliance & security
Traditional data solutions are incomparable to modern architecture apps with inherent data compliance and more robust security.

The Six Layers of Modern Data Frameworks

It is designed with the aim of assembling, storing, converting, and analyzing data, making the right information easily accessible to necessary systems and individual users. Such architecture is handy for business intelligence, more sophisticated analytics such as machine learning, or a central data hub.

The foundational components of next-generation data represent a layered structure. Visualize this type of architecture as a stack consisting of six distinct layers:

• Ingestion layer
• Storage layer
• Processing layer
• Consumption layer
• Visualization layer
• Governance & security layer

What data streaming enables is receiving, managing and scrutinizing large amounts of rapidly-changing data for more responsive real-time customer experiences. Each layer of that architecture contains components, designed to tackle specific demands, and is to be described in detail further.

Ingestion Layer

The Ingestion Layer’s key function is the transfer of information from its original source to the data platform. This process usually entails extracting the data, inspecting their quality, and keeping the data in the staging area of the platform. An important feature of the Layer is supporting the data source connections with the use of drivers or libraries.

Storage Layer

Elicited from its name, the Storage Layer is supposed to retain or archive the data protected from disasters, malfunctions or user errors, at the same time providing easy access to authorized users. A Storage Layer is typically represented by solutions like cloud storage, NoSQL database or Graph database, and data architecture companies (e.g. Hadoop).

Processing Layer

This Layer manages the data transformation so that they duly correspond to the previously established data model. Depending on the nature of the data source and the requirements for data availability, the processing can be executed in batches at a scheduled time or in real time. Should the data model be modified for any reason, the Processing Layer converts the information again to match the data model of the Storage Layer.

Consumption Layer

The Analytics Layer is where the properly prepared data are subjected to various types of analytical models, including ML, with the view of detecting trends and answers to business questions. The output of the analytics can be sent to the Storage Layer or the Visualization Layer. The Analytics Layer can be classified into two categories: Business Intelligence and Advanced Analytics. The former involves reports that present KPI results related to business performance, while the latter uses more advanced algorithms to generate results.

Visualization Layer

Once the data have been extracted, transformed, stored, and analyzed, they are ready to be revealed to the end-user within the Visualization Layer, which can take the form of different tools, also illustrated below. The aim is to withdraw valuable information that provides insights and assists in making informed business decisions.
Data visualization can appear in the following four types:

• Dashboards
• Reports
• Self-Service BI
• Embedded Analytics

Governance & Security Layer

Ensuring data security is crucial to any data platform, and the essence of the Governance and Security Layer. Speaking of the latter, user authentication and authorization, data encryption, and audit trails, among other means, can be engaged in the prevention of unauthorized access, modification or disclosure of information.

As per governance, it refers to storing data, and metadata, in a centralized catalog that allows a unified logical view of the information across an organization, managing master and reference data, and maintaining the data lineage.

The strategy for cutting-edge data architecture

On your way to a more convenient, secure and accurate data architecture that best meets the present-day data-based challenges, you are likely to benefit from an advanced data strategy. It is supposed to assist you in breaking down silos and entitling your team to use the most appropriate and effective tools or techniques in order to get the most out of data analytics.

The strategy is reliant on these three pillars:

• Modernization – data infrastructure is best modernized through a cloud migration as opposed to self-management.
• Unification – data silos are to be broken down to make the information centralized and available across databases, data analytics, and ML services.
• Innovation – creating new experiences and enhancing existing processes lead to new business insights and outcomes.

The pillars of modern data strategy do not necessitate a specific implementation order. Your current requirements and objectives govern the process, even if it means applying the three pillars simultaneously. We will expand on the pillars in more detail thereafter.

Modernization pillar

Modernization, associated with cost optimization and performance improvement, is best attained through migration or adoption of cloud-native services. At the outset, one can open with either designing one’s own scalable data lake, or choosing among the suggested data services. The major points to consider include security controls and centralized access to authorized systems or individuals, cost-efficient scalability, and integration of machine learning tools into the architecture.

Unification pillar

Well-informed timely decisions based on the holistic view of the enterprise require scaling data stores to collect the emergent data, unified according to the data model, seamlessly connected and available across the organization’s various systems, to be further processed and scrutinized with the most advanced analytics algorithms and machine learning tools.

Innovation pillar

Any stage of modern data strategy can incorporate innovation. Whether it comes to databases: relational or non-relational; to data services: purpose-built or data lakes; to processing tools: sophisticated data analytics algorithms or machine learning solutions, – can be drawn by aiming at improved customer experience and innovation.

How data architecture on AWS is arranged

AWS data architecture allows the swift construction of one’s own data lakes and provides a wide collection of data services to opt from. Whichever you pick, you are in for security compliance, unified data format, centralized access governance, seamless scaling and convenient data connectivity, regardless of their origin. AWS services combine the advantages of both data lakes and data stores without having to choose between boosted performance and optimized cost.

So as to modern data arranging on AWS, it includes four principal components, which will be explained thoroughly:

• a scalable data lake
• purpose-built data services
• seamless data movement
• unified data governance.

AWS data lakes

This is more secure, scalable and agile version of a data hub, a data lake allows discovering, storing, sharing, governing and processing of all kinds of data – structured and unstructured alike. Analytics in AWS data lakes unlocks the possibility to choose the tools as well as the type of examination for the best-informed business decisions, including big data processing, real-time analytics, and machine learning.

The benefits of AWS data lakes

• Collect and store any type of data, at any scale, and at a low cost
  AWS data architecture experts ensure that either relational or non-relational, either structured or semi-structured, just like unstructured data are managed cost-effectively, regardless of the data volume.
• Use a broad set of analytic engines
  The more sophisticated analytical algorithms, including ad hoc analytics, real-time streaming, predictive analytics, ML and AI, are supported by data engineering services available on either pay-as-you-go subscription or on-demand.
• Catalog, search, and find the relevant data in the central repository
  The services are also able to operate on data directly, without data transfer. This is enabled by a centralized data catalogue that handles data within a data lake.
• Quickly and easily perform new types of data analysis
  AWS offers more than 50 services to deal with data of any amount, velocity or type. AWS’s data services and features are there for extracting, storing, processing, analyzing and visualizing data in the cloud.

Purpose-built data services

Such tools are custom-built for specific use cases to optimize cost, performance, scaling and functionality so that you do not have to compromise either of those. Moreover, AWS tools have various deployment options to choose from for anyone to commence straight away. We will further review some of the extensive variety of such data tools offered by Amazon.

Data warehousing – Amazon Redshift

If what you require are rapid query results for structured data, you may benefit from using a data warehouse.

Revolutionizing the data warehousing economics, Redshift was the first data warehouse initially designed to function in the cloud, yet it is still the leading cloud data warehousing option offering three times the price performance of its competitors. Its consistently high performance combined with predictable costs makes Redshift one of the most reliable choices for cloud data warehousing. Numbers suggest that Redshift supports queries of up to an exabyte scale run against information from your data lake, and manage as much as petabytes of data within its clusters. Redshift is the answer to the request for easy analytics, irrelevant of the data storage locations, for limitless on-demand scaling, and for the first serverless computing analytics experience.

Interactive query – Amazon Athena

Should you require direct data querying in your S3 data lakes, Amazon Athena is the serverless solution you will benefit from. Designed with a simple SQL interface with JDBC and ODBC drivers, Athena does not need setting up or server managing: by directing data to Amazon S3 and defining the schema, one can quickly commence data querying with an integrated query editor and receive outcomes in various formats (e.g. CSV, JSON, Avro, Parquet, or ORC) and compression types (e.g. GZIP or BZIP).

Big data processing – Amazon EMR

For managing huge volumes of data, architecture consulting firms recommend Amazon EMR. The incrementally scalable infrastructure allows EMR to handle data of any size by creating entirely managed data clusters with up to thousands of compute instances within such distributed frameworks as Spark or Presto.

A practical and fully administered service, EMR also provides clearly predictable per-second pricing model, which means you will only pay for the cluster’s active time. It proves optimal for periods of fluctuating usage by granting flexibility to manage your resources. Cost-efficient EMR makes a good match with Amazon S3 for storage, and with EC2 Spot Instances for computing with the use of on-demand instances.

Log and search analytics – Amazon OpenSearch

Amazon OpenSearch service is usually the best fit for those that monitor the production systems’ performance or troubleshoot problems by inspecting substantial volumes of log data while also prioritizing data security.

A fully maintained service, OpenSearch enables you to launch a production-ready OpenSearch or Elasticsearch-like cluster in a matter of minutes, not to mention that scaling occurs through a mere API call or via a console. OpenSearch supports a number of AWS services, including Logstash, Kinesis Data Firehose, IoT and CloudWatch Logs.

Real-time analytics – Amazon Kinesis & Amazon MSK

At last, real-time data streaming is possible with services like Amazon or Amazon MSK. The former is able to provide the quickest insights based on the mining and analyzing of real-time streaming data. While the focus of the orderly organized latter is placed on the security of the data within clusters, inasmuch as MSK patches up unhealthy components, this Apache Kafka service also boasts of its availability.

As pointed out above, numerous AWS purpose data services have been created to cater to the diverse analytical needs among various workloads. For a straightforward and quick start with data analytics, one had better choose among the services maintaining serverless options – these will help you put your data-driven insights into well-informed business decisions as soon as possible.

Seamless data movement

An indispensable element of next-generation data architecture execution is data movement: into and out of the data lake, and across the data stores. A service which can provide seamless data movement is AWS Glue.

Amazon’s serverless data integration service facilitates data discovery, collection, cleansing, transfer and integration whether with the aim of analytics, ML, or solution design. Glue supports unified information catalogues with centralized governance of the connected array of more than 70 separate data sources. The service can take over the construction and management of ELT pipelines to smoothly move and load information among data lakes and data stores.

More thoroughly, AWS Glue:

• Is a serverless tool for carrying out data integration jobs, which are at the core of effective information management. How Glue processes these jobs or end-to-end workflows can be twofold: either by event or as a pre-scheduled implementation.
• Supplies a fully managed metadata repository, the Glue Data Catalog, which enables ELT operations from a centralized metadata location, also handy for data processing and accessibility. Glue Crawlers is another feature that reinforces the narrated functions further to ensure automated information cataloguing and discovery across sources.
• Offers off-the-shelf connector features to ensure data connectivity, irrespective of their location, which can range from on-premise sources to a different cloud service provider. Apart from the integrated Python, JDBC, and ODBC connectors to get built-in access to multiple databases, including log and file-based data, Glue developed a marketplace with third-party connectors, counting BigQuery, Salesforce and other SaaS systems in.
• Assists you in constructing data integration jobs with a broad variety of personalized tools, such as data integration jobs visualization with Glue Studio; a data wrangling interface with Glue DataBrew; or an interactive notebook with Glue Notebooks.

Data integration jobs can further be facilitated by ML, automatic code generation and template-driven development supported by AWS Glue which goes beyond data transformation, data lake construction, or utilizing specifically crafted data tools.

Unified data governance

Collecting and transforming data within your data lakes and repositories is just the beginning; the information must be protected, which is achieved by implementing unified management and control measures. Data security essentially encompasses two elements: centralized access control and regulatory obligations compliance. Both of these are provided by Lake Formation, developed by AWS to set up a valid unified solution.

The primary feature of AWS Lake Formation is centralized data authorization, seamlessly functioning with machine learning and analytical tools.

Fundamentally, AWS Lake Formation enables:

• Building, managing, and protecting data lakes with prompt use of familiar database-like features.
• Streamlining governance on a larger scale, and authorizing permissions on both marketplace-level and asset-level across your data lake.
• Eliminating data silos and enhancing data availability with a data catalogue.
• Facilitating data accessibility at the scale of enterprise with cross-account data sharing.