Developing Data Pipelines in Azure and AWS

Setting up a platform to manage your organization’s data can be a daunting task, but like with most technical activities, the first step is to evaluate which environment and tool set fits your needs best. The focus of this blog will be to lean into the Analytics Data Platform use case and compare two of the biggest names in cloud infrastructure, Azure and AWS. We will look at what is required for developing data pipelines in Azure using Azure Synapse, and in AWS using a collection of their database and ETL tools, such as AWS Glue. When setting up an analytics environment where data from across the business can be collected, enriched, and used for reporting and analytics workloads, these two platforms both behave remarkably well, however each has its own pros and cons.

Softcrylic has extensive experience with both cloud platforms and have recommended clients use one or the other depending on their requirements and current infrastructure. Below we will highlight some of the key differences between the two platforms across the areas of Setup, Security, Scaling, Data Governance, Data Management, Environment, CI/CD, Pricing, Support, and Exclusive Services.

Setup

Quick and efficient setup is a major priority when starting down the path of building your own data platform, as it will allow your organization to quickly jump in developing your data pipelines.

In AWS, CloudFormation templates can be created for the specific services you want provisioned by Amazon, and these templates are easily maintained and versioned in a code repository. Infrastructure can also be created and modified dynamically using a Command Line Interface for maximum control on your setup.

For Azure, a Synapse instance can be set up from the Portal with just a few clicks, which enables data integration, data warehousing, and big data analytics activities out of the box, so you can spend more time focusing on implementing your business requirements and less time worrying about infrastructure. It also comes with an Apache Spark pool that contains 200+ libraries preinstalled when it comes to use for machine learning, data analysis, visualization, most the jobs can be done without managing our own library pool. One thing to point out however, since Synapse Notebook has its own unique syntax, there is no direct method to convert your existing Notebook code when bringing it over to Synapse.

Data Management

Once you have your environment set up, you will be eager to begin developing your data pipelines. Both AWS and Azure offer robust options for data collection and storage, as well as pipeline setup and management.

AWS has a variety of tools like API Gateway, Kinesis, Glue, Lambda, step functions, and S3 available to create data pipelines fast and efficiently. While these tools are straight forward to use, they will take some expertise to leverage and are designed with developer skillsets in mind. AWS also has capabilities to write SQL queries directly on real-time data flowing through the Kinesis stream, thereby providing real-time analytics capabilities.

Azure Synapse is a slightly more out-of-the-box solution that will be friendlier to citizen developers and data analysts. It comes with rich interactive authoring experience allowing users to create, configure and deploy data integration pipelines without any friction. Synapse Studio, which is at the heart of the platform, allows users to easily query and manage the data and to create a data pipeline at scale. These integrated development environments are based in the cloud and allow users to create complex data pipelines with just a browser, this will eliminate the time of setting development and testing environment. Azure Synapse also comes equipped with 95+ data connectors (with the list growing every day!) that allow user to easily pull data from various sources without writing or managing code.

Scaling

Once you have set up your environment, and you have begun developing some of your data pipelines, the next step will be to think about scaling to meet your business demands, so it is important to consider the scaling capabilities of platform before you begin! We will look at across the areas of Storage, ETL, and Data Warehousing separately when it comes to scaling capabilities.

Storage

Both Azure and AWS offer very reliable unlimited storage solutions as a part of their infrastructure offerings. AWS relies on S3 storage buckets, which are highly available, fault tolerant, and easily connect to AWS Redshift data warehouse. However, S3 does not have native configurations to restrict the size of the buckets, so storage limits must be controlled via custom applications/modules leveraging the storage solution.

Azure Synapse relies on Azure Data Lake for its storage option and can cleanly integrate with Azure Data Warehouse for data analytics or Apache Spark for big data solutions. Like S3, Azure Data Lake is extremely scalable and can meet virtually any storage requirement.

ETL

With AWS there are many options for scaling ETL operations. The first and most efficient way is with AWS Glue, where ETL can be done with serverless architecture and dynamically scale. However, during the initial ETL job configuration, it must be specified how many Data Processing Units (DPUs) are needed for the AWS Glue job, so it is not truly elastic. If there is no need to dynamically scale and it is more important to have low level control of Apache Spark framework and its associated libraries, then AWS EMR can be used as an alternative. Lastly, AWS Lamba or step functions can also be leveraged for orchestrating pipelines if needed.

Azure Synapse comes equipped with Apache Spark for ETL and Big Data Analytics. Spark pools in Azure Synapse can have auto-scale enables, so pools scale by adding and removing nodes as needed automatically. These additional spark instances can spin up in under 2 minutes and then shutdown automatically after the last job has executed, so it is quite responsive to ad hoc processing demands and makes scaling stress free.

Datawarehouse

Several years ago, Amazon launched the third generation of their RA3 node type for Amazon Redshift, which provided the warehousing solution with bigger storage capacity (minimum 32TB per node), hot/cold storage, and the ability to scale and pay for compute and storage independently. However, with AWS Redshift you still need to manually set compute and storage capacity limits, as it will not scale dynamically.

In Azure Synapse Analytics, Dedicated SQL pool (formerly SQL DW) refers to a collection of features that make up their enterprise data warehouse. Synapse SQL dedicated pools are a highly elastic and scalable cloud service that can be easily integrated with several other Synapse services. Since there is tight coupling to Compute and Storage services, you can scale Synapse SQL Dedicated Pool instance up and down or even pause it without losing the data in the managed tables. Synapse SQL pool can also directly query the data in the Azure Storage as external tables, which can be helpful in data exploration and ad-hoc analysis.

Security

From a security standpoint, both AWS and Azure offer solid capabilities to ensure your data and infrastructure can only be accessed by authorized individuals, while at the same time offering data encryption at rest to protect against any intrusion that may occur.

In AWS, Virtual Private Cloud (VPC) can be added with private and/or public subnets to safeguard the infrastructure by isolating resources. There are also tools like Web Application Firewall (WAF) that can protect applications from Denial-of-Service (DoS) SQL injection kind of attacks, and IP addresses can be easily whitelisted and blacklisted using security groups. Also, Amazon’s EC2 dedicated instances are physically isolated at the host hardware level from other AWS customer accounts if dedicated hardware is a requirement, which provides for maximum security. One negative we have found during our implementations of AWS is that while data can be encrypted at rest, there is no out of the box column level data encryption in Amazon Redshift, which may cause concern for some clients.

The Azure platform promotes an “On by default” encryption policy that means data is always encrypted in-transit and at rest. Like AWS, they too provide built-in mechanisms to protect against DoS attacks and have solid Firewall and IP restricting capabilities. However, they also offer security features like row-level security, column-level security, and data masking at no extra cost to all Synapse customers, and the Azure Advance Threat Protection service detects anomalous activities indicating unusual and potentially harmful attempts to access or exploit databases.

Conclusion

It is clear that both Azure and AWS are quite capable cloud infrastructure providers that will allow your organization to build solid data platforms based on your needs. For a quick reference here is how we grade Azure and AWS across a few key areas: