Sr. Databricks Architect and Developer

We are seeking a highly experienced Senior Databricks Architect and Developer to design, build, and optimize high performance data migration and ETL solutions. The ideal candidate will bring deep expertise in Databricks architecture, AWS cloud services, and large scale data migration from legacy systems to PostgreSQL environments. This role requires strong hands on development experience along with architectural ownership of the Databricks platform setup, automation, and monitoring.

Role Title: Sr. Databricks Architect and Developer

Location: Remote with occasional travel to Des Moines, IA

Required Skills

• Databricks platform architecture and administration
• Azure Data Factory
• PySpark and Pandas
• SQL and PL SQL
• Spark Structured Streaming
• AWS services including S3, Glue, Lambda, Redshift, EMR, and overall cloud infrastructure
• Python development
• ETL pipeline design and optimization
• DevOps automation
• Data validation and transformation
• SFTP, DoDSAFE, NIPRGPT
• Data visualization tools
• Optimization and monitoring including cluster autoscaling, spot instances, cost management
• Azure Monitor, CloudWatch, and Databricks logs

Preferred Skills

• Strong experience designing and building high performance ETL pipelines using Databricks with PySpark, Delta Lake, and Databricks Workflows
• Proven expertise migrating data from multiple legacy sources including VSAM files to PostgreSQL
• Experience architecting and configuring Databricks Landing and Staging environments
• Job orchestration and automation design
• Performance monitoring and tuning tools implementation
• Advanced SQL, Databricks SQL, and PostgreSQL expertise for load optimization and large volume cutovers
• Experience in data mapping, conceptual and technical design
• Application and technical testing using Databricks Notebooks
• Implementation of data masking techniques
• Experience with spider web and reverse spider web logic
• Strong defect analysis and remediation skills

Key Responsibilities

• Develop scalable and high performance ETL pipelines using Databricks including PySpark, Python, Delta Lake, and Databricks Workflows
• Lead migration efforts from legacy sequential databases, VSAM files, and other structured sources into PostgreSQL
• Configure and manage Databricks Landing and Staging schemas ensuring secure and efficient data movement
• Optimize data loads and manage high volume cutover activities
• Contribute to data mapping, architecture design, and technical validation
• Develop and execute technical test cases using Databricks Notebooks
• Implement data masking and transformation rules
• Support defect resolution and ensure high quality migration outcomes

Expected Deliverables

• Setup, configuration, and ongoing maintenance of the Databricks platform
• High performance ETL workflows supporting large scale data migration
• Documented architecture, automation workflows, and monitoring framework