Lead Data Engineer

Seeking an experienced Data Engineer expert to lead and support the SIT and UAT for migration of legacy Ab Initio & Teradata workloads to Azure Databricks. The candidate will serve as the onshore technical lead with hands on testing, defect fixing and performance tuning for the migrated workloads. Work closely with stakeholders and offshore delivery teams while ensuring data accuracy and delivery quality throughout the migration.

Roles & Responsibilities:

• Develop Databricks notebooks, jobs, and workflows to replicate and enhance Abinitio/Teradata pipelines and transformations.
• Implement Delta Lake tables and patterns (bronze/silver/gold, ACID, time travel, schema evolution) for migrated data.
• Construct and optimize Databricks notebooks, workflows, and Spark jobs leveraging PySpark and Spark SQL for advanced data transformation and analytics.
• Participate in data validation and reconciliation between Abinitio/Teradata and Databricks outputs.
• Provide defect fixes and support during SIT/UAT and post go-live stabilization.
• Maintain detailed technical documentation and adhere to coding standards, testing methodologies, and release management processes to ensure consistency and maintainability.

Requirements:

• 12+ years of experience as a Data Engineer.
• 3 to 5 years of experience in Databricks Implementations
• Strong proficiency with Databricks platform, PySpark, Spark SQL, and Delta Lake architectures.
• Solid understanding of data warehousing concepts, cloud-based data architectures, and dimensional data modeling methodologies.
• Fluency in SQL and Python with practical experience in development and debugging.
• Experience in performance tuning, workload optimization, and implementing cost-effective design solutions in data engineering pipelines.
• Familiarity with DevOps practices, CI/CD pipelines, version control systems, and automated testing as they relate to data engineering workflows.
• Strong analytical skills with an ability to communicate technical concepts effectively to both technical and business audiences.
• Certifications in Databricks will be an added advantage