Role: Senior Azure Data Engineer
Location: Remote
U.S. citizens
Responsibilities
* Designed, developed, and managed scalable cloud-based data platforms using Microsoft Azure Data Services, ensuring high availability, security, and performance.
* Built and implemented Change Data Capture (CDC) solutions using Python, enabling real-time data synchronization between source and target systems with minimal latency.
* Developed enterprise data warehouse solutions using Azure Synapse Analytics and Azure SQL Data Warehouse, optimizing reporting, analytics, and business intelligence workloads.
* Wrote and optimized complex SQL queries, including CTEs, correlated subqueries, window functions, stored procedures, and joins to improve query performance and reduce execution time.
* Developed scalable ETL/ELT pipelines using Azure Data Factory (ADF), Azure Databricks, and PySpark for ingesting structured and semi-structured data from multiple sources.
* Processed and transformed large-scale datasets using Apache Spark (PySpark) on Azure Databricks, improving pipeline efficiency and processing performance.
* Automated incremental, historical, and full-load processes using metadata-driven frameworks, event-based triggers, and control tables, reducing manual intervention by over 90%.
* Designed robust Databricks notebooks with checkpointing, exception handling, logging, and restart capabilities to ensure reliable batch processing and data integrity.
* Diagnosed and resolved Microsoft SQL Server performance bottlenecks, database corruption, deadlocks, and connectivity issues to maintain high system availability.
* Designed and implemented dimensional data models, including Slowly Changing Dimensions (SCD Type 1 & Type 2) using Delta Lake MERGE operations to preserve historical data and improve data lineage.
* Optimized Azure infrastructure costs through automated VACUUM jobs, storage lifecycle policies, cluster optimization, and workload migration strategies, significantly reducing operational expenses.
* Managed large-scale distributed data using HDFS, improving ingestion, storage, retrieval, and transformation performance across distributed computing environments.
* Implemented end-to-end data observability using Acceldata to monitor cluster health, resource utilization, pipeline execution, and performance regressions.
* Developed Python scripts using Pandas and NumPy for data cleansing, validation, transformation, reconciliation, and quality checks.
* Configured automated SMTP/Email notifications for pipeline failures, schema drift detection, SLA breaches, and operational alerts.
* Used Linux shell scripting to automate file management, scheduling, and operational support for data engineering workflows.
* Secured sensitive credentials and encryption keys using Azure Key Vault, ensuring compliance with enterprise security and governance standards.
* Utilized GitHub for version control, code reviews, branching strategies, and CI/CD integration to support collaborative software development.
* Participated in Agile/Scrum ceremonies, including sprint planning, backlog grooming, daily stand-ups, retrospectives, and release planning using JIRA and Confluence.
* Collaborated with business stakeholders, QA teams, and data analysts during User Acceptance Testing (UAT) to validate business rules, data quality, and reporting accuracy.
* Built interactive dashboards and analytical reports using Power BI and Tableau, enabling business users to derive actionable insights from enterprise data.
* Integrated streaming and batch data pipelines using Kafka, Delta Lake, ADLS Gen2, and Azure Databricks to support modern Lakehouse architectures.
