Databricks Data Engineer

We are seeking a Senior Data Engineer to design, build, and operate enterprise-grade batch and streaming data pipelines on a modern cloud-based data platform. This role focuses on delivering trusted, well-governed, high-quality data products that support analytics, research, and decision-making use cases in a regulated environment.

Key Responsibilities

• Design, build, and maintain end-to-end batch and streaming data pipelines using Apache Spark (PySpark) on a cloud-based data platform.
• Implement near-real-time and event-driven ingestion patterns, including state management, checkpointing, and recovery mechanisms.
• Develop incremental and idempotent processing patterns, including backfills, schema evolution handling, and error recovery.
• Optimize data performance and cost through partitioning strategies, file sizing, caching, and query optimization.
• Create reusable and standardized pipeline components (libraries, parameterized jobs, shared patterns) to accelerate delivery across teams.
• Develop and operationalize workflows in Python and R to support data preparation, analytics, and research-ready datasets.
• Implement data quality and validation controls for batch and streaming pipelines, including schema enforcement, completeness checks, deduplication, and reconciliation.
• Build and support pipeline observability, including logging, metrics, alerting, dashboards, and root-cause analysis.
• Support production operations through on-call participation, incident response, and creation of runbooks and operational documentation.
• Apply secure data access and least-privilege principles when handling sensitive or regulated data.
• Contribute to metadata, lineage, and dataset documentation to ensure transparency, auditability, and reuse.
• Use version control and CI/CD practices to manage code, testing, and deployment across environments.
• Collaborate with cross-functional stakeholders to refine requirements, define SLAs, and deliver incremental, measurable outcomes.

Technical Skills & Experience

Required

• Strong proficiency in Python for data engineering workflows.
• Advanced SQL skills for transformation, validation, and analysis.
• Hands-on experience with Apache Spark, including batch and streaming workloads.
• Experience building production-grade data pipelines with a focus on reliability and scalability.
• Knowledge of data quality practices, testing, and validation in distributed systems.
• Experience with cloud data services, including object storage (e.g., S3), IAM-based access control, and monitoring.
• Familiarity with CI/CD concepts, Git-based version control, and automated testing for data pipelines.
• Strong troubleshooting, debugging, and performance-tuning skills in distributed environments.
• Ability to communicate clearly through technical documentation and design artifacts.
• Experience working in regulated or highly governed environments.

Strongly Preferred

• Experience with Databricks or similar managed Spark platforms.
• Familiarity with lakehouse architectures and transactional table formats (e.g., Delta-style tables).
• Experience implementing streaming ingestion patterns using message/event platforms (e.g., Kafka or equivalents).
• Knowledge of data orchestration and workflow scheduling tools.
• Experience with data governance concepts, including metadata management, lineage, and secure access patterns.
• Exposure to declarative pipeline frameworks and expectations-based data quality checks.
• Experience with testing frameworks (e.g., pytest, data quality libraries) and test-driven development.
• Background in data modeling, including dimensional models and slowly changing dimensions.
• Prior work supporting analytics, research, or policy-driven data use cases.

Education & Experience

• Bachelor’s degree in Computer Science, Engineering, Data Science, or a related field (or equivalent practical experience).
• 8–12+ years of data engineering experience, including production Spark-based batch and/or streaming implementations.
• Experience operating data pipelines in enterprise or mission-critical environments.