Databricks Data Engineer

Databricks Engineer to design, build, and maintain scalable data pipelines, lakehouse architectures, and analytics solutions  using Microsoft Azure and the Databricks platform. 

Key Responsibilities

• Pipeline Development: Design and build Extract, Transform, Load (ETL) and ELT data pipelines using Azure Data Factory (ADF) and Azure Databricks
•  
• Big Data Processing: Write optimized data processing code using Apache Spark, PySpark, Python, Scala, and advanced SQL.
•  
• Storage & Architecture: Implement and manage data lakes using Azure Data Lake Storage (ADLS Gen2) and Delta Lake / lakehouse patterns.
•  
• Governance & Security: Configure data governance, lineage, and access controls using Databricks Unity Catalog and Azure IAM.
•  
• Optimization & Operations: Monitor, troubleshoot, and tune Spark jobs and data pipelines for performance, reliability, and cost-efficiency.
•  
• CI/CD & DevOps: Support deployment processes utilizing Azure DevOps, Git version control, and infrastructure-as-code tools like Terraform.

Required Skills & Qualifications

• Experience: 3 to 7+ years of total experience in data engineering or cloud-based data platform development.
•  
• Core Technologies: Hands-on proficiency in Azure Databricks, Apache Spark, PySpark, and Azure Data Factory (ADF).
•  
• Languages: Advanced SQL and solid programming skills in Python or Scala.
•  
• Data Concepts: Strong grasp of Delta Lake, relational/NoSQL data storage, data modeling, and data quality standards.
•  
• Education: Bachelor’s degree in Computer Science, Engineering, or a related quantitative field (often preferred)