Experience and JD:
- 10+ years of overall and 5+ years of architecture experience with data architecture/data engineering roles with hands-on work on major enterprise data platforms.
- Proven hands-on experience with Databricks, especially with modern features such as:
- Unity Catalog: implementing catalog, schemas, permissions, external / managed tables, security, lineage, etc.
- Delta Live Tables (DLT): building reliable pipelines, CDC, transformations, data quality, scaling/performance tuning.
- Experience with data ingestion tools such as Fivetran for SaaS / ERP / relational sources, plus experience integrating HVR or equivalent for high velocity / change data capture or replication.
- Strong working knowledge of cloud infrastructure (Azure or AWS), storage (object stores, data lakes), compute scaling, cluster management within Databricks.
- Proficiency in programming with Python / PySpark, working with Spark / SQL; good understanding of streaming vs batch processing.
- Deep understanding of data governance, security, compliance: role-based access control (RBAC), attribute-based, encryption, audit logs; handling data privacy; compliance requirements.
- Operational excellence: reliability, monitoring, observability, metrics; experience with failover/backup / DR strategies.
- Strong communication skills: able to work with domain experts and engineering teams, translate business requirements into technical solutions; document architecture and trade-offs.
- Experience with performance tuning of Spark jobs, optimizing data storage formats, partitioning, and schema design to support high-throughput, low-latency workloads.
Never miss an opportunity! Create an alert based on the job you applied for.
