Senior Data bricks Architect Security Data Platform
Location: Remote Duration: Long-Term Contract Industry: Pharmaceutical / Life Sciences Role Type: Senior Architect / Technical Leadership
15 20+ years in data engineering/data architecture, with 5+ years of hands-on Databricks experience and 3+ years leading engineering teams. Has architected enterprise Databricks Lakehouse platforms using Spark, PySpark, Delta Lake, Unity Catalog, Python and SQL across Azure/AWS/Google Cloud Platform. Experienced in batch and streaming pipelines, Workflows, DLT, Auto Loader, performance/cost optimization, CI/CD, Terraform and enterprise governance. Has managed engineers/architects while remaining hands-on with architecture reviews, code reviews, troubleshooting and client engagements.
Job Overview
We are seeking an experienced Senior Data bricks Architect with a strong background in Pharmaceutical / Life Sciences environments to lead the architecture and technical direction of a large-scale Security Data Platform.
The successful candidate will have hands-on experience designing and architecting Data bricks platforms processing approximately 300 TB of data per day, with responsibility for highly complex enterprise data environments involving approximately 13,000 tables and 3,000+ users.
This role will work closely with senior security and technology leadership, including CISO, EVP, SVP, and VP-level security stakeholders, as well as engineering and platform teams.
Pharmaceutical / Life Sciences industry experience is mandatory for this position.
Key Responsibilities
Lead the architecture and technical strategy for a large-scale Data bricks Security Data Platform.
Design and optimize Data bricks solutions capable of processing approximately 300 TB of data per day.
Architect highly scalable, secure, and governed data platforms supporting approximately 13,000 tables and 3,000+ users.
Develop architecture standards for security data ingestion, processing, storage, analytics, and consumption.
Work closely with CISO, EVP, SVP, and VP-level Security executives to understand business and security objectives and translate them into technical solutions.
Partner with engineering teams to implement and operationalize the target architecture.
Design secure data architectures incorporating RBAC, data governance, access controls, auditing, lineage, encryption, and compliance.
Establish best practices around Databricks security, governance, performance, scalability, and cost optimization.
Work with Observability teams to develop solutions for monitoring data pipelines, platform health, security events, performance, and operational issues.
Provide technical leadership across data engineering, security engineering, cloud, platform, and architecture teams.
Perform architecture reviews, technical design reviews, troubleshooting, and code reviews.
Identify performance bottlenecks and optimize large-scale Spark/Databricks workloads.
Define strategies for data lifecycle management, retention, access management, and secure data sharing.
Ensure the platform meets enterprise security, regulatory, privacy, and compliance requirements applicable to the pharmaceutical industry.
Communicate complex technical concepts effectively to both executive stakeholders and engineering teams.
Required Qualifications Databricks & Data Architecture 8+ years of experience in Data Engineering, Data Architecture, or Cloud Data Platforms.
Strong hands-on experience with Databricks and Apache Spark.
Proven experience architecting very large-scale Data bricks environments.
Experience handling 300 TB+ of data per day or comparable large-scale data volumes.
Experience managing or architecting environments with thousands of users and thousands of tables.
Strong knowledge of Delta Lake, Lakehouse Architecture, Medallion Architecture, Spark, ETL/ELT, and distributed data processing.
Experience with Databricks performance tuning and large-scale workload optimization.
Strong understanding of cloud platforms such as AWS, Azure, or Google Cloud Platform.
Security & Governance
Strong experience with Data Security and Security Data Platforms.
Experience implementing enterprise RBAC, access controls, auditing, lineage, governance, and secure data sharing.
Strong understanding of Databricks Unity Catalog and enterprise data governance.
Experience working with security, observability, compliance, or cybersecurity data.
Understanding of security monitoring and observability platforms and their integration with data platforms.
Experience working with highly sensitive or regulated enterprise data.
Pharmaceutical / Life Sciences Mandatory
Mandatory: Candidates must have significant experience working within the Pharmaceutical, Life Sciences, Biotechnology, or closely related Healthcare industry.
Experience working with data platforms supporting pharmaceutical or life-sciences organizations.
Understanding of the security, privacy, compliance, and regulatory requirements associated with pharmaceutical data environments.
Candidates without relevant Pharmaceutical / Life Sciences industry experience will not be considered.
Executive & Stakeholder Management
Experience working directly with CISO, CIO, CTO, EVP, SVP, and VP-level executives.
Ability to communicate architecture strategy, security risks, platform capabilities, and business value to executive stakeholders.
Strong experience gathering requirements from senior leadership and translating them into actionable technical roadmaps.
Ability to work across executive sponsors, security teams, engineering teams, and enterprise architecture organizations.
Observability Experience
Experience designing or supporting data/platform observability solutions.
Knowledge of monitoring data pipelines, Spark jobs, Databricks workloads, infrastructure, data quality, and platform health.
Experience integrating observability and monitoring capabilities into enterprise data platforms.
Ability to establish operational metrics, alerting, logging, and performance monitoring.
Preferred Qualifications Databricks certifications.
Experience with Unity Catalog, Delta Live Tables / Lakeflow Declarative Pipelines, Auto Loader, Structured Streaming, Workflows, MLflow, and Databricks SQL.
Experience with Terraform / Infrastructure as Code.
Experience with cloud security and identity platforms.
Experience with SIEM, security analytics, or cybersecurity data platforms.
Knowledge of HIPAA, GxP, FDA, SOX, GDPR, or other relevant regulatory/compliance frameworks.
Experience supporting global pharmaceutical enterprises.
Experience leading architecture initiatives across multiple engineering teams.
<>
Must-Have Skills<>
Data bricks | Apache Spark | Data Architecture | Security Data | 300+ TB/Day Data Processing | Unity Catalog | Data Governance | RBAC | Data Security | Observability | Large-Scale Data Platforms | Pharmaceutical / Life Sciences Experience | Executive Stakeholder Management<>
Industry Requirement: Pharmaceutical / Life Sciences experience is mandatory.
