Job Title: Azure Databricks Architect
Location: Remote with (50% travel)
Role Summary
We are seeking Senior Data Platform Engineers to join our CSO team. This is a dual-domain environment spanning two fully segregated platforms built on Azure Databricks and the Medallion Architecture: the CSO Security Data Platform and the SOX Application Data Platform (fully segregated to satisfy Sarbanes-Oxley requirements). Engineers are the backbone of both platforms — ensuring data quality, enabling security consumers, onboarding new sources, maintaining pipelines, and supporting SOX audits.
Required Qualification:
Experience
• 5+ years of data engineering experience with at least 2 years on Azure Databricks or equivalent cloud lakehouse platforms.
• Hands-on experience with Medallion Architecture pipelines, Delta Lake (MERGE, OPTIMIZE, VACUUM, Z-ORDER, schema enforcement), and Databricks Structured Streaming.
• Familiarity with SOX compliance, IT General Controls (ITGC), or regulated-environment data engineering; experience supporting audits or evidence collection.
Technical Skills
Area | Technologies |
Languages | Python (PySpark), SQL; Scala a plus |
Cloud | Azure (ADLS Gen2, Azure Monitor, Log Analytics, Event Hubs, APIM, Azure DevOps) |
Data Platform | Databricks (Unity Catalog, Auto Loader, Structured Streaming, Workflows, Genie, AI/BI Dashboards, MLflow) |
Governance | Unity Catalog (RBAC, column/row security, lineage), MS Purview (auto-classification, sensitivity labels) |
Integration | Azure APIM, Delta Sharing, Databricks SQL API, SIEM/SOAR connector patterns |
BI & Reporting | Databricks AI/BI Dashboards, Power BI |
DevOps | Git, CI/CD via Azure DevOps or GitHub Actions |
Security Domain Knowledge
• Familiarity with OCSF (Open Cybersecurity Schema Framework) or willingness to develop deep expertise rapidly.
• Understanding of security log source types (EDR, firewall, IAM/PAM, CASB) and SOC/SIEM/SOAR workflows; awareness of MITRE ATT&CK framework.
Preferred Qualifications
• OCSF normalization implementation across multiple security log sources.
• Experience with Azure API Management (APIM) for data platform API governance.
• UEBA, threat scoring, or ML-backed security analytics (MLflow, Databricks Feature Store, Mosaic AI).
• SOX audit participation — auditor walkthroughs, evidence packages, IT General Controls testing.
• Certifications: Databricks Certified Data Engineer Associate/Professional, Azure Data Engineer Associate (DP-203), or equivalent.
• Graph-based correlation (Databricks GraphX); Delta Sharing.
Core Competencies
Competency | Why It Matters |
Ownership & Reliability | Numerous sources, high volume TB/day — you own what doesn''t run. |
Audit Mindset | Audits mean evidence gaps have real regulatory consequences. |
Security Domain Curiosity | The better you understand analyst needs, the better the platform serves them. |
Stakeholder Communication | You bridge data engineering, SOC, compliance, and auditors. |
Structured Problem-Solving | Quarantine spikes and pipeline failures require root-cause discipline. |
Documentation Discipline | Runbooks, audit evidence, and onboarding guides are first-class deliverables. |
Compliance Awareness | SOX platform changes carry regulatory risk; process adherence is non-negotiable. |
Platform & Tooling Landscape
Area | Technologies |
Cloud Infrastructure | Microsoft Azure, ADLS Gen2, Azure Monitor, Log Analytics, Event Hubs, APIM |
Data Platform | Databricks (Unity Catalog, Structured Streaming, Auto Loader, Workflows, Genie, AI/BI Dashboards) |
Storage / Format | Delta Lake, JSON + GZip (Landing Zone), ADLS Gen2 Hot & Archive tiers |
Schema Standard | OCSF + custom.* namespace (Security); segregated schemas (SOX) |
Governance | Unity Catalog, MS Purview (auto-classification, sensitivity labels, federated discovery) |
Integration | Azure APIM, Delta Sharing, Event Hubs, Databricks SQL API |
BI & Analytics | Databricks AI/BI Dashboards, Databricks Genie, Power BI |
AI & ML | MLflow, Databricks Feature Store, Mosaic AI (roadmap) |
DevOps & CI/CD | Azure DevOps / GitHub Actions, Git |
Scale | Numerous log sources · High daily volume in TBs/day raw |
Educational Qualifications:
· Required - Bachelor’s degree in Computer Science, Information Technology, Computer Engineering or closely related or equivalent.
· Preferred - Master’s degree in Management Information Systems (MIS), Computer Science, Big Data or Analytics or equivalent.
Travel:
· Open to travel based-up on the nature of the engagement.
Thanks & Regards
Srikanth Donkani
Resource Manager
(w):
(E):
2260 Haggerty Road, Suite 285 Northville, MI 48167
Equal Employment Opportunity
Reliable Software employment does not discriminate on the basis of race, religion, gender, sexual orientation, age or any other basis as covered by federal, state, or local law.
Employment decisions are based solely on qualifications, merit and business needs.
Never miss an opportunity! Create an alert based on the job you applied for.
