Databricks Data Engineer

Requirements:

• Thorough understanding and hands on development experience with designing ETL/ ELT solutions
• Expertise with full load, incremental load, and transformations
• Experience implementing Change Data Capture (CDC) and/ or Change Tracking
• Hands on database development experience using SQL/ T-SQL/ PL-SQL
• Proficient in using window functions, complex joins, functions and stored procedures
• Experience with performance tuning with good understanding on the different types of indexes (clustered vs nonclustered; rowstore vs. columnstore)
• Experience with Data Governance & Security – implementing role-based access control (RBAC), row-level security, data masking etc.
• Experience working in any of cloud platforms, preferably working in Azure
• Experience working with Azure Databricks with deep understanding in:
• Lakehouse, Lakebase and Medallion Architecture
• Apache Spark internals (caching, partitioning, performance tuning etc.)
• Delta Lake (ACID transactions, time travel, schema evolution etc.)
• Proficiency in data ingestion techniques using Lakeflow, Delta Live Tables, Auto Loader
• Proficiency in data governance, access control & data lineage using Unity Catalog
• Hands on ETL pipeline development experience in Python/ Pyspark, Terraform
• Experience setting up alerts to monitor job failures, performance, and trigger alerts
• Proficient in Git and experience building Databricks Assess Bundles (DAB), and managing CI/CD deployments
• Hands on experience with any of the reporting tools, preferably PowerBI or SAP BusinessObjects
• Excellent communication skills to bridge the gap between development teams and business stakeholders