Sr. Databricks Engineer – Job Summary
Role Overview:
- Senior individual contributor responsible for designing, developing, enhancing, and supporting enterprise cloud data platform capabilities.
- Reports to the Sr. Director of Data Platform Transformations & Operations.
- Serves as a technical lead for major initiatives and accountable for production support and operational excellence.
- Focus: ~70% new development, ~30% operations/support/platform reliability.
- Primary technologies: Azure Databricks, PySpark, SQL, Delta Lake, cloud data engineering.
Key Responsibilities:
- Design, develop, and maintain scalable data pipelines, ingestion frameworks, and transformation processes.
- Implement Medallion (Bronze/Silver/Gold) architecture for reporting, analytics, AI, and self-service data.
- Develop and operationalize AI agent solutions on Databricks, integrating models, tools, and APIs.
- Build reusable frameworks, utilities, and platform components for productivity and quality.
- Develop and modernize batch, near-real-time, and streaming integration solutions.
- Serve as technical lead—solution design, technical reviews, tool recommendations, and implementation oversight.
- Partner with architects to ensure scalable, secure, governed, and cost-effective solutions.
- Mentor other engineers and promote coding, testing, and documentation standards.
- Implement data quality, validation, reconciliation, monitoring, and security controls.
- Build and maintain CI/CD pipelines, automated testing, and release processes (Azure DevOps/Git).
- Contribute to platform observability, alerting, monitoring, performance tuning, and cost optimization.
- Participate in production support, incident response, troubleshooting, and root cause analysis.
- Create and maintain runbooks, support documentation, and knowledge-sharing resources.
- Collaborate with cross-functional teams to translate requirements into platform solutions.
- Support technical discovery, estimation, planning, delivery, and evaluation of new cloud/data/AI capabilities.
Required Skills & Experience:
- Bachelor’s in Computer Science, Information Systems, Engineering, Data Science, or related field; Master’s preferred.
- 10+ years in data engineering, warehousing, or platform engineering.
- 5+ years designing/implementing cloud-based data platforms.
- 3+ years hands-on Azure Databricks in enterprise environments.
- Advanced SQL, Python, PySpark, and Spark optimization.
- Experience with Databricks agentic AI, Delta Lake, Unity Catalog, Databricks Workflows, and Delta Live Tables.
- Deep understanding of lakehouse concepts, Medallion architecture, data modeling, and analytics data structures.
- Experience with Azure DevOps, Git, CI/CD, automated testing/deployment; Databricks Asset Bundles preferred.
- Implementation of data quality, validation, governance, and security controls.
- Strong leadership, solution design, troubleshooting, and communication skills.
- Experience in regulated environments (financial services/banking/audit-sensitive) preferred.
- Databricks Data Engineer certification preferred.
Nice to Have:
- Financial services industry experience.
- GenAI project deployment experience.
- Data governance expertise.
- Familiarity with Power BI, MicroStrategy, MDM, streaming architectures, infrastructure automation, or AI-assisted development.
Location/Logistics:
- Onsite in Austin, TX; local candidates only.
- Must attend onsite interview; confirm candidate’s willingness and availability.
Employers have access to artificial intelligence language tools (“AI”) that help generate and enhance job descriptions and AI may have been used to create this description. The position description has been reviewed for accuracy and Dice believes it to correctly reflect the job opportunity.
- Dice Id: 91173025
- Position Id: 742-46503-1793308010
- Posted 5 hours ago
