Azure Data Engineer

Salary: $97,500-$130,000

Job Summary

We are seeking an experienced Azure Data Engineer with strong Databricks expertise to design, develop, and support scalable cloud-based data solutions. This role is responsible for building and maintaining modern data platforms on Microsoft Azure and Databricks, developing robust data pipelines, and enabling analytics and reporting initiatives across the organization.

The ideal candidate will have extensive hands-on experience with Azure Databricks, Azure Data Factory, Delta Lake, Apache Spark, and Medallion Architecture. Experience with Terraform and GitHub Actions is required to support infrastructure automation, CI/CD processes, and cloud platform administration.

Key Responsibilities

• Design, develop, and maintain scalable data platforms and solutions on Microsoft Azure.
• Build and optimize ETL/ELT data pipelines using Azure Data Factory and Azure Databricks.
• Develop data transformation processes leveraging Apache Spark and Delta Lake technologies.
• Implement and maintain Medallion Architecture (Bronze, Silver, Gold layers) to support enterprise analytics and reporting needs.
• Collaborate with business stakeholders, analysts, data scientists, and application teams to understand data requirements and deliver solutions.
• Design and implement efficient data models that support reporting, analytics, and business intelligence initiatives.
• Monitor, troubleshoot, and optimize data pipelines for performance, reliability, and scalability.
• Develop and maintain CI/CD pipelines using GitHub Actions.
• Provision and manage cloud infrastructure using Terraform and Infrastructure as Code (IaC) best practices.
• Ensure data quality, governance, security, and compliance standards are met.
• Provide production support and resolve issues related to data integration and platform performance.
• Document technical designs, architectures, standards, and operational procedures.

Required Qualifications

• Bachelor's degree in Computer Science, Information Systems, Engineering, or a related field.
• 5+ years of experience in Data Engineering or related technical roles.
• Strong experience with:
  • Azure Databricks
  • Azure Data Factory (ADF)
  • Azure Data Lake Storage (ADLS)
  • Apache Spark (PySpark preferred)
  • Delta Lake
  • SQL and data modeling
• Experience implementing Medallion Data Architecture.
• Hands-on experience with Terraform for infrastructure automation.
• Experience building CI/CD pipelines using GitHub Actions.
• Strong knowledge of cloud-native data engineering best practices.
• Experience working in Agile environments.
• Excellent analytical, troubleshooting, and problem-solving skills.

Preferred Qualifications

• Microsoft Azure certifications.
• Experience with Azure Synapse Analytics.
• Experience with Power BI or other analytics platforms.
• Knowledge of data governance and data security frameworks.
• Experience supporting large-scale enterprise data environments.

Key Competencies

• Data Engineering
• Cloud Architecture
• ETL/ELT Development
• Infrastructure as Code (Terraform)
• CI/CD Automation
• Data Modeling
• Problem Solving
• Communication and Collaboration
• Performance Optimization
• Stakeholder Management