Job Description:
We are looking for an experienced Senior Data Engineer with strong Databricks expertise to design, develop, and maintain scalable data engineering solutions. The ideal candidate will have extensive experience in data processing, data pipeline development, cloud-based data platforms, and big data technologies.
Key Responsibilities:
Design, develop, and maintain scalable data pipelines using Databricks.
Build and optimize ETL/ELT workflows for processing large volumes of structured and unstructured data.
Develop data transformation solutions using Python, PySpark, and SQL.
Work with Delta Lake and implement efficient data storage and processing strategies.
Integrate data from multiple sources into centralized data platforms.
Optimize data pipelines for performance, scalability, and reliability.
Collaborate with data architects, analysts, and other engineering teams to meet business requirements.
Implement data quality checks, monitoring, and error-handling mechanisms.
Ensure data security, governance, and compliance best practices.
Troubleshoot data processing issues and improve overall platform performance.
Required Skills & Qualifications:
10+ years of experience in data engineering or a related field.
Strong hands-on experience with Databricks.
Proficiency in Python, PySpark, and SQL.
Experience developing and managing ETL/ELT data pipelines.
Strong knowledge of Delta Lake and Lakehouse architecture.
Experience with cloud platforms such as AWS, Azure, or Google Cloud Platform.
Knowledge of data warehousing, data modeling, and distributed data processing.
Experience with performance tuning and optimization of large-scale data pipelines.
Strong problem-solving, analytical, and communication skills.
Preferred Qualifications:
Experience with Azure Databricks and Azure Data Factory, or equivalent services on AWS/Google Cloud Platform.
Familiarity with CI/CD pipelines, Git, and automated deployment practices.
Experience implementing data governance, security, and monitoring solutions.
