Databricks Architect

Job Title: Databricks Architect
Location: Remote
Duration: 6+ Months Contract to Hire
Note: Must be willing to travel to Philadelphia, PA or Washington, DC once or twice a month, or less, for meetings.

Must be able to work in the Eastern Time Zone.

Databricks on AWS

Lakehouse Architecture

Lakeflow Connect

Lakeflow Jobs

Spark Declarative Pipelines

DABs (Declarative Automation Bundles)

Databricks Apps

AWS Data & Analytics Services

Job Overview:

We are seeking an experienced Data Platform Architect to guide the design and implementation of a scalable data solution for Databricks on AWS. The ideal candidate will have a strong background in data architecture, data integration, and data processing frameworks, with prior hands-on expertise in Databricks and a passion for optimizing diverse workloads.

The candidate should have architecture and implementation experience with Lakeflow Connect, Lakeflow Jobs, Spark Declarative Pipelines, Declarative Automation Bundles, Databricks Apps, Delta Lake, Databricks SQL, and Unity Catalog, including enterprise governance, security, CI/CD, observability, performance optimization, and cost management. Experience with MLflow, Mosaic AI Gateway, Model Serving, Vector Search, and generative AI architecture is preferred.

Key Responsibilities:

• Design the data architecture for platform and organizational use cases, focusing on scalability, performance, and support for advanced analytics and data science workloads.
• Ensure the platform supports multi-cloud operations, data sharing, and integration with various data sources, including on-premises and cloud environments.
• Ensure the platform integrates with various data delivery mechanisms including streaming, batch, and leveraging APIs for encryption and decryption.
• Collaborate with Cloud Architects and Engineers to implement data governance, financial governance, and compliance features.
• Identify and architect automation approaches to simplify and accelerate CI/CD across multiple environments, onboarding of new users to the platform, creation of development workspaces, data quality enforcement, and integrated testing.
• Optimize the platform for handling diverse workloads, relational operations, and multi-model data processing.
• Develop and implement data ingestion and data processing frameworks, ensuring efficient and scalable data pipelines for common patterns.
• Architect and maintain data lakehouse storage solutions, emphasizing best practices for data modeling and schema design.
• Provide guidance on best practices for data preparation, exploration, and visualization within the platform.
• Lead client-facing workshops and influence change and acceptance to modern data platform practices.
• Provide client-facing support, ensuring clear communication and understanding of technical solutions.
• Collaborate with stakeholders to document enterprise rollout strategies, data security blueprints, and architectural standards.
• Represent the platform team at Product Increment (PI) Planning events, driving alignment on deliverables, dependencies, and cross-team commitments.
• Lead through others, guiding Data Engineers across multiple parallel projects without becoming overly focused on any single deliverable.

Qualifications:

• Degree in Computer Science, a technical field, or a related business field. Master s degree preferred.
• 3+ years of experience in designing and leading Databricks implementations including:
• Lakeflow Connect
• Lakeflow Jobs and Lakeflow Spark Declarative Pipelines
• Databricks Declarative Automation Bundles (DABs), formerly Databricks Asset Bundles
• Databricks Apps
• Unity Catalog architecture, implementation, and governance
• Unity AI Gateway (optional)
• MLflow (optional)
• 8+ years of experience in relevant technology roles, with a focus on data architecture.
• 5+ years in an advanced role, demonstrating leadership in guiding teams directly or indirectly.
• 5+ years of experience in architecting and maintaining big data platform architecture utilizing AWS Data & Analytics Services.
• Strong data modeling experience.
• Strong expertise in designing and supporting complex application architectures, including integration of disparate data sources.
• Experience in documenting enterprise rollout strategies and developing data security blueprints.
• Excellent communication skills.