An end-to-end data engineering pipeline that fetches data from Wikipedia, cleans and transforms it with Apache Airflow and saves it on Azure Data Lake. Other processing takes place on Azure Data Factory, Azure Synapse and Tableau.

End-to-end ETL pipeline in the Microsoft Azure cloud - (Jun '24 - Jul '24)

Reusable Python classes that extend open source PySpark capabilities. Examples of implementation is available under notebooks of repo https://github.com/bennyaustin/synapse-dataplatform

Python script that cleans the JSON document used by an Azure Data Factory or Azure Synapse Copy Activity to specify the source-sink mapping.

Azure ARM Templates + Synapse Analytics Playground

This project demonstrates the development of a complete end-to-end data pipeline in the cloud using the Microsoft Azure ecosystem to ingest, transform, store, and visualize COVID-19 related data.

Containerized tool for load testing Azure SQL Database and Azure Synapse Analytics SQL pool

A comprehensive data engineering solution that transforms sales data from PostgreSQL into a star schema, moves it to Azure Synapse Analytics via Blob Storage, and creates Power BI dashboards for business insights.

Development of a Data Pipeline using Azure Synapse

A data engineering project that scrapes football stadium data from Wikipedia, processes it with Airflow and Python, stores it in Azure Data Lake, moves it via Data Factory, queries with Synapse, and visualizes insights using Tableau.

A B2B solution and architecture. This includes code for Azure Function App, SQL Server. Integration with azure synapse and event hubs as well.

Data ingestion and transformation pipeline for AirBnB using Azure Data Factory, Synapse, CosmosDB, and ADLS for automated CDC and analytical reporting.