Senior Databricks Developer (AWS)
We are seeking a Senior Databricks Developer with 7+ years of Data Engineering experience and deep expertise in Databricks, AWS, PySpark, Python, and SQL. This is a highly technical, hands-on engineering role within a data engineering team focused primarily on building, enhancing, and supporting enterprise data pipelines. The ideal candidate will have extensive experience developing cloud-native data solutions in AWS, implementing Delta Lake and Medallion architectures, and delivering scalable, production-grade data pipelines using Databricks. This is not a consulting role; we are looking for a developer who enjoys building and optimizing data platforms and solving complex data engineering challenges.
Responsibilities
- Design, develop, and maintain large-scale ETL/ELT pipelines using Databricks, PySpark, Python, and SQL.
- Build scalable and highly reliable data ingestion, transformation, and orchestration frameworks within AWS.
- Develop and support Delta Lake implementations and Medallion Architecture (Bronze, Silver, Gold) data pipelines.
- Create and optimize batch and streaming pipelines that process large volumes of structured and unstructured data.
- Leverage AWS services including S3, Glue, Lambda, EMR, IAM, EventBridge, CloudWatch, Kinesis, and Redshift to support end-to-end data workflows.
- Optimize Spark jobs, Delta tables, and Databricks workloads for performance, scalability, and cost efficiency.
- Partner with Data Engineers, Architects, Analytics teams, and business stakeholders to deliver high-quality data solutions.
- Implement data quality controls, monitoring, alerting, and operational support processes.
- Troubleshoot and resolve complex production issues across Databricks and AWS environments.
- Contribute to engineering best practices, code reviews, automation, and platform modernization initiatives.
Required Qualifications
- 7+ years of Data Engineering experience building enterprise-scale data platforms and pipelines.
- 5+ years of hands-on Databricks development experience in production environments.
- Strong expertise in Python, PySpark, and SQL with a focus on pipeline development.
- Extensive experience developing and supporting data pipelines within AWS.
- Strong knowledge of Delta Lake, Lakehouse Architecture, and Medallion Architecture (Bronze, Silver, Gold).
- Hands-on experience with AWS services such as S3, Glue, Lambda, EMR, Redshift, Kinesis, IAM, and CloudWatch.
- Deep understanding of Spark architecture, distributed data processing, performance tuning, partitioning, and optimization.
- Experience building highly scalable batch and streaming data processing solutions.
- Strong understanding of data modeling, data warehousing concepts, and modern data architectures.
- Experience supporting production environments, monitoring data pipelines, and troubleshooting complex data issues.
Preferred Qualifications
- Experience with Databricks Workflows, Delta Live Tables (DLT), Unity Catalog, and Structured Streaming.
- Experience with Kafka, Airflow, or cloud-native orchestration frameworks.
- Experience implementing CI/CD pipelines using GitHub Actions, Jenkins, or Azure DevOps.
- Infrastructure as Code experience utilizing Terraform or CloudFormation.
- Databricks Certified Data Engineer Associate or Professional certification.
- Experience working within large enterprise AWS environments supporting mission-critical data platforms.
Key Skills
Databricks, AWS, PySpark, Python, SQL, Delta Lake, Medallion Architecture, Lakehouse Architecture, ETL/ELT Pipelines, Spark Optimization, Delta Live Tables (DLT), Databricks Workflows, Unity Catalog, S3, Glue, Lambda, EMR, Kinesis, Redshift, CloudWatch, Kafka, Airflow, GitHub, CI/CD, Terraform.
