Senior Databricks Platform Engineer
Eagan, MN – Local Candidates Only
Contract
Direct Client
Skills:
Databricks
Platform Governance & Cost Management
Architecture Design for Data Pipelines
Security & Networking Expertise
Key Responsibilities
1.Databricks Platform Governance & Cost Management
Establish and maintain Databricks platform governance, access controls, and operational standards.
Monitor and optimize Databricks resource utilization, compute workloads, and cloud spending.
Implement cost-control strategies, cluster policies, and workload optimization practices.
Track platform usage and identify opportunities to improve performance and reduce costs.
Support data security, compliance, and governance requirements.
2. Architecture Design for Data Pipelines
Design and implement scalable, reliable, and efficient data pipeline architectures using Databricks.
Develop and optimize ETL/ELT workflows for batch and streaming data processing.
Work with Delta Lake, Apache Spark, PySpark, and related data engineering technologies.
Design data ingestion, transformation, storage, and integration solutions.
Troubleshoot pipeline performance issues and improve data processing efficiency.
Collaborate with data engineers, architects, and business teams to deliver data platform solutions.
3. Security & Networking Expertise
Implement secure access management, authentication, authorization, and role-based access controls.
Configure and manage secure connectivity between Databricks and cloud resources.
Support networking configurations, including private connectivity, network security rules, and firewall requirements.
Apply security best practices for data access, secrets management, encryption, and sensitive information protection.
Work with cloud infrastructure teams to maintain a secure and compliant data platform.
Required Skills
Strong hands-on experience with Databricks platform administration and architecture.
Experience with Databricks governance, access management, and cost optimization.
Expertise in designing scalable data pipelines and ETL/ELT architectures.
Strong knowledge of Apache Spark, PySpark, and Delta Lake.
Experience with cloud security, networking, and private connectivity.
Knowledge of cloud infrastructure on AWS, Azure, or Google Cloud Platform, preferably aligned with the client's environment.
Strong troubleshooting, performance tuning, and problem-solving skills.
Excellent communication and collaboration skills.
Preferred Qualifications
Experience with Unity Catalog and centralized data governance.
Knowledge of Databricks cluster policies, compute management, and workload monitoring.
Experience with CI/CD, infrastructure as code, and automated platform deployment.
Familiarity with enterprise data security, compliance, and audit requirements.
Relevant Databricks or cloud platform certifications are a plus.
