Azure Databricks Engineer
Location: Atlanta, GA (Onsite)
Employment Type: Full-Time
Experience Required: 6+ Years Overall | 3+ Years Azure Databricks Experience
About the Role
We are seeking a highly skilled Senior Azure Databricks Engineer to join our Data & Analytics team in Atlanta. The ideal candidate will have extensive experience designing, developing, and optimizing enterprise-scale data solutions using Azure Databricks, Apache Spark, and Microsoft Azure services.
This role will be responsible for building scalable data pipelines, implementing Lakehouse architectures, enabling real-time and batch data processing, and driving best practices for performance, governance, and security.
Key Responsibilities
- Design, develop, and maintain scalable data pipelines using Azure Databricks and PySpark.
- Build and support ETL/ELT frameworks for large-scale structured and unstructured data.
- Develop batch and real-time data processing solutions.
- Implement and maintain Delta Lake and Medallion Architecture (Bronze, Silver, Gold).
- Integrate data solutions with Azure Data Factory, ADLS Gen2, Azure Synapse, Event Hub, and other Azure services.
- Optimize Spark workloads for performance, reliability, and cost efficiency.
- Implement data quality checks, monitoring, and alerting processes.
- Collaborate with business stakeholders, architects, analysts, and data scientists to deliver data solutions.
- Establish and enforce data governance, security, and compliance standards.
- Support CI/CD pipelines and automated deployment processes.
- Troubleshoot and resolve production issues related to data pipelines and Databricks workloads.
- Document technical solutions, architectural decisions, and operational procedures.
Required Qualifications
- Bachelor’s degree in computer science, Information Technology, Engineering, or a related field.
- 6+ years of experience in Data Engineering.
- 3+ years of hands-on experience with Azure Databricks.
- Strong experience with:
- Azure Databricks
- Apache Spark
- PySpark
- Python
- SQL
- Delta Lake
- Experience with Azure Data Factory (ADF).
- Experience with Azure Data Lake Storage Gen2 (ADLS Gen2).
- Strong understanding of Data Warehousing, Data Lakes, and Lakehouse architectures.
- Experience working with large-scale datasets and distributed computing environments.
- Strong analytical, troubleshooting, and problem-solving skills.
- Excellent communication and collaboration abilities.
Preferred Qualifications
- Experience with Azure Synapse Analytics.
- Experience working with Event Hub or Kafka.
- Knowledge of Unity Catalog and data governance frameworks.
- Experience with Delta Live Tables (DLT).
- Experience with Terraform, ARM Templates, or Infrastructure as Code (IaC).
- Experience with Azure DevOps or GitHub Actions.
- Knowledge of Power BI and modern analytics platforms.
- Cloud certifications such as:
- Microsoft Azure Data Engineer Associate (DP-203)
- Databricks Certified Data Engineer Associate/Professional
Technical Skills
Required
- Azure Databricks
- Apache Spark
- PySpark
- Python
- SQL
- Azure Data Factory
- ADLS Gen2
- Delta Lake
- Git
What We're Looking For
The ideal candidate has hands-on experience building modern cloud-based data platforms and can demonstrate expertise in:
- Lakehouse Architecture
- Spark Performance Tuning
- Enterprise Data Engineering
- Data Pipeline Development
- Real-Time Data Processing
- Data Governance and Security
- CDC (Change Data Capture) Implementations
- Production Support and Troubleshooting
- Azure Cloud Data Services
- Cost Optimization and Scalability
Why Join Us?
- Work on enterprise-scale cloud data modernization initiatives.
- Collaborate with experienced data and cloud professionals.
- Exposure to cutting-edge Azure and Databricks technologies.
- Opportunity to drive impactful business solutions through data.
- Long-term growth and career advancement opportunities.
Never miss an opportunity! Create an alert based on the job you applied for.