Sr Azure Databrick Engineer

Role:Sr/Lead Azure Databricks Engineer

Location: Ottava,Canada(Onsite)

Own our Azure Databricks lakehouse from ingestion to governed data products, Power BI reporting, and AI/RAG use cases.
Responsibilities
• Architect Medallion lakehouses on Azure Databricks with Delta Lake and Unity Catalog
• Build metadata-driven ETL/ELT pipelines (PySpark, Spark SQL, ADF) with CDC, validation, and audit logging
• Design dimensional models and data marts for Power BI
• Ingest from databases, REST APIs, SharePoint, and SFTP
• Deliver GenAI/RAG solutions with Azure OpenAI, Azure AI Search, FastAPI, and Streamlit
• Build Azure DevOps CI/CD (YAML, ARM) and support Terraform across Dev, Test, and Prod
• Develop, schedule, and monitor Python jobs on Databricks
• Manage and tune databases (Azure SQL, SQL Server) for performance and reliability
• Enforce data quality, lineage, and RBAC
• Lead migrations from on-prem SQL Server, SSIS, and SSRS Required
• 8+ years in data engineering or BI, 3+ on Azure Databricks
• Databricks with strong SQL (Spark SQL, T-SQL, query optimization)
• Python jobs on Databricks (build, schedule, monitor)
• Database management (Azure SQL, SQL Server): performance tuning, security, backup/recovery
• Python, PySpark, SQL/T-SQL, Delta Lake
• ADF, ADLS Gen2, Azure SQL, Synapse
• Medallion architecture and dimensional modeling
• Power BI (DAX, Power Query, RLS)
• Azure DevOps CI/CD
• Strong stakeholder communication Preferred
• Databricks Data Engineer Professional, DP-203, AZ-305
• Microsoft Fabric/OneLake
• Terraform
• RAG, vector search, embeddings
• Public-sector or regulated environments