Overview
Remote
$70 - $80
Contract - Independent
Contract - W2
Skills
Application Lifecycle Management
MuleSoft
Salesforce.com
Salesforce
SF
COPADO
SAP BASIS
Supply Chain Management
Coaching
Collaboration
Team Leadership
Warehouse Management
Status Reports
System Implementation
Job Details
Hi,
Please find the requirement below , If you find yourself comfortable with the requirement please reply with your updated resume and I will get back to you
• Job Title: Sr. Databricks Solutions Architect
• Job Location: Remote , must be able to come into the office when needed.
• Start Date: January 19th or January 12
• Background: Standard background check required
Top Skills Needed:
• Deep hands-on expertise with Databricks platform architecture && governance
• Unity Catalog, workspaces, external locations, compute, access controls, cluster governance.
• Reliability engineering, monitoring, && operational hardening of the Lakehouse
• Observability, alerting, DR readiness, backup/restore, performance tuning, incident response.
• Strong experience with ADF, CI/CD, && Terraform for orchestrating and managing the Lakehouse
• Pipeline orchestration, IaC, DevOps, environment promotion, compute policies.
Typical Day-to-Day:
• Design how the Databricks Lakehouse should work including the structure, tools, standards, and best practices
• Guide engineering teams on how to build pipelines && use Databricks correctly
• Solve technical issues when data jobs fail or performance slows
• Work with stakeholders to understand data needs && deliver solutions
• Set standards for security, governance, naming conventions, && architecture
• Ensure the Databricks platform is stable, reliable, && always available
• Build && maintain monitoring, alerting, logging, && health dashboards
• Strengthen && fix ingestion pipelines (ADF → landing → raw → curated)
• Improve data quality checks, anomaly detection, && pipeline reliability
• Manage CI/CD pipelines && deployment processes using Azure DevOps or GitHub
• Use Terraform (IaC) to deploy && manage Databricks and Azure infrastructure
• Partner with Security && FinOps on access controls, compliance, and cost governance
• Mentor the Data Engineer && support distributed data engineering teams across the organization
Key Responsibilities
1. Lakehouse Architecture & Platform Administration
(Approximately 60% of role when combined with mentoring & code review)
• Serve as the primary architect and administrator for the Azure Databricks Lakehouse (Unity Catalog, workspaces, external locations, compute, access controls).
• Lead execution of a Minimal Viable Hardening Roadmap for the platform, prioritizing:
o High availability and DR readiness
o Backup/restore patterns for data and metadata
o Platform observability and operational metrics
o Secure and maintainable catalog/namespace structure
o Robust and proactive data quality assurance
• Implement and evolve naming conventions, environment strategies, and platform standards that enable long-term maintainability and safe scaling.
• Act as the Lakehouse-facing counterpart to Enterprise Architecture and Security, collaborating on network architecture, identity & access, compliance controls, and platform guardrails.
2. Reliability, Monitoring, and Incident Management
• Design, implement, and maintain comprehensive monitoring and alerting for Lakehouse platform components, ingestion jobs, key data assets, and system health indicators.
• Oversee end-to-end reliability engineering, including capacity planning, throughput tuning, job performance optimization, and preventative maintenance (e.g., IR updates, compute policy reviews).
• Participate in — and help shape — the on-call rotation for high-priority incidents affecting production workloads, including rapid diagnosis and mitigation during off-hours as needed.
• Develop and maintain incident response runbooks, escalation pathways, stakeholder communication protocols, and operational readiness checklists.
• Lead or participate in post-incident Root Cause Analyses, ensuring durable remediation and preventing recurrence.
• Conduct periodic DR and failover simulations, validating RPO/RTO and documenting improvements.
This role is foundational to ensuring 24/7/365 availability and timely delivery of mission-critical data for clinical, financial, operational, and analytical needs.
3. Pipeline Reliability, Ingestion Patterns & Data Quality
• Strengthen and standardize ingestion pipelines (ADF → landing → raw → curated), including watermarking, incremental logic, backfills, and retry/cancel/resume patterns.
• Collaborate with the Data Engineer to modernize logging, automated anomaly detection, pipeline health dashboards, and DQ validation automation.
• Provide architectural guidance, code reviews, mentoring, and best-practice patterns to distributed engineering teams across MedStar.
• Support stabilization of existing ingestion and transformation pipelines across clinical (notes, OHDSI), financial, operational, and quality use cases.
4. DevOps, CI/CD, and Infrastructure as Code
• Administer and improve CI/CD pipelines using Azure DevOps or GitHub Enterprise.
• Support automated testing, environment promotion, and rollback patterns for Databricks and dbt assets.
• Maintain and extend Terraform (or adopt Terraform from another IaC background) for Databricks, storage, networking, compute policies, and related infrastructure.
• Promote version control standards, branching strategies, and deployment governance across data engineering teams.
5. Security, FinOps, and Guardrails Partnership
• Partner with Enterprise Architecture and Security on platform access controls, identity strategy, encryption, networking, and compliance.
• Implement and enforce cost tagging, compute policies, and alerts supporting FinOps transparency and cost governance.
• Collaborate with the team defining agentic coding guardrails, ensuring the Lakehouse platform supports safe & compliant use of AI-assisted code generation and execution.
• Help assess and optimize serverless SQL, serverless Python, and job compute patterns for cost-efficiency and reliability.
6. Mentorship, Collaboration, & Distributed Enablement
• Mentor the mid-level Data Engineer on Databricks, ADF, dbt, observability, DevOps, Terraform, and operational engineering patterns.
• Provide guidance, design patterns, and code review support to multiple distributed data engineering teams (Finance, MCPI, Safety/Risk, Quality, Digital Transformation, etc.).
• Lead platform knowledge-sharing efforts through documentation, workshops, and best-practice guidance.
• Demonstrate strong collaboration skills, balancing independence with alignment across teams.
7. Optional / Nice-to-Have: OHDSI Platform Support
(Not required for hiring; can be learned on the job.)
• Assist with or support operational administration of the OHDSI/OMOP stack (Atlas, WebAPI, vocabularies, Kubernetes deployments).
• Collaborate with partners to ensure the OHDSI platform is secure, maintainable, and well-integrated with the Lakehouse.
Required Qualifications
• 5+ years in cloud data engineering, platform engineering, or solution architecture.
• Strong hands-on expertise in Azure Databricks:
o Unity Catalog
o Workspaces & external locations
o SQL/Python notebooks & Jobs
o Cluster/warehouse governance
• Solid working experience with Azure Data Factory (pipelines, IRs, linked services).
• Strong SQL and Python engineering skills.
• Experience with CI/CD in Azure DevOps or GitHub Enterprise.
• Experience with Terraform or another IaC framework, and willingness to adopt Terraform.
• Demonstrated ability to design or support monitoring, alerting, logging, or reliability systems.
• Strong communication, collaboration, and problem-solving skills.
Preferred Qualifications (Optional)
• Advanced Terraform experience.
• Familiarity with healthcare, HIPAA, PHI, or regulated environments.
• Experience with Purview or enterprise cataloging.
• Exposure to OHDSI/OMOP.
• Experience optimizing or refactoring legacy ingestion pipelines.
• Experience supporting secure, controlled AI/agentic execution environments.
• Experience with EPIC EHR data exchange and/or EPIC Caboodle or Cogito analytics suite.
Personal Attributes
• Hands-on, pragmatic, and operationally minded.
• Comfortable leading both architecture and implementation.
• Collaborative and mentorship-oriented; thrives in small core teams with broad influence.
• Values platform stability, observability, and hardening over shiny features.
• Curious and adaptable, especially with emerging AI-assisted engineering patterns.
• Ability to remain calm and effective during incidents and high-pressure situations.
Thanks,
Neeraj
It Resource Manager
Pioneer IT Systems
Email:
Mobile :
Employers have access to artificial intelligence language tools (“AI”) that help generate and enhance job descriptions and AI may have been used to create this description. The position description has been reviewed for accuracy and Dice believes it to correctly reflect the job opportunity.