Databricks Data Engineer

Responsibilities:

• Partner with business stakeholders and Global Product teams to understand data domains, business processes, and analytical requirements.
• Design, develop, and maintain scalable data pipelines and transformation jobs on the Databricks Lakehouse Platform to generate business metrics, KPIs, and curated datasets.
• Develop and optimize batch and streaming data pipelines using Databricks, Apache Spark (PySpark/Scala), and Delta Lake technologies.
• Build and maintain data ingestion frameworks from various enterprise source systems into AWS-based data lakes and Databricks environments.
• Implement and maintain Medallion Architecture (Bronze, Silver, and Gold layers) to support analytics and self-service reporting.
• Develop, optimize, and maintain Delta tables and data models to ensure high-performance data processing and analytics.
• Monitor, troubleshoot, and support Databricks workflows, clusters, and production data pipelines to ensure data reliability and availability.
• Implement data quality checks and validation frameworks to ensure data accuracy, completeness, timeliness, and consistency.
• Implement data governance and access controls using Unity Catalog and enterprise security standards.
• Optimize Spark jobs, cluster utilization, and query performance to improve scalability and cost efficiency.
• Build and deploy Databricks code, workflows, and infrastructure using GitHub Actions-based CI/CD pipelines.
• Implement monitoring, logging, alerting, and observability solutions using Datadog for proactive issue detection and operational support.

Requirements:

• 2+ years of experience in Data Engineering and building enterprise-scale data platforms.
• 2+ years of hands-on experience with Databricks and Apache Spark development using PySpark or Scala.
• Strong experience designing and implementing data pipelines on AWS and Databricks platforms.
• Experience with AWS services such as Amazon S3, AWS Glue, Amazon EC2, and IAM.
• Strong knowledge of Delta Lake, Unity Catalog, and Databricks Lakehouse architecture.
• Strong SQL skills and experience with data modeling and dimensional modeling techniques.
• Experience developing and deploying solutions using GitHub Actions and Git-based source control practices.
• Experience implementing monitoring and observability solutions using Datadog.
• Experience with data quality, metadata management, and data governance frameworks.
• Strong troubleshooting, performance tuning, and production support experience in cloud-based data platforms.
• Excellent communication and collaboration skills with the ability to work effectively with global teams.
• Databricks certifications such as Databricks Certified Data Engineer Associate or Professional.

Education:

• Bachelor's degree in Computer Science/Engineering or a related field