Azure Data Engineer : Databricks with Scala | India : Remote Role

Job Title: Azure Data Engineer - Databricks with Scala

• Job Location:  India : Remote Role
• Experience Level: Mid to Senior
• Must-Have Skills:

• Azure Databricks with Scala (MUST HAVE)
• Experience with Azure Data Factory (ADF) dataflow and ADF pipelines (MUST HAVE)
• Practical experience in handling error or invalid formats in incoming data.
• Experience working with Delta table formats (Must)

Job Description:
We are looking for a skilled Data Engineer with expertise in Databricks and Scala to join our team. The ideal candidate will have hands-on experience in building robust data pipelines, working with Azure Data Factory, and managing data in Delta table formats. This role offers an exciting opportunity to work on cutting-edge data projects and contribute to the success of our analytics initiatives.
Responsibilities:

• Design, develop, and maintain data pipelines using Databricks with Scala, ensuring scalability, reliability, and efficiency.
• Implement and optimize Azure Data Factory (ADF) dataflows and pipelines to facilitate seamless data movement and transformation.
• Handle error or invalid formats in incoming data streams, implementing robust error handling mechanisms to ensure data integrity.
• Work with Delta table formats to manage and process large volumes of structured and unstructured data efficiently.
• Collaborate with cross-functional teams to understand data requirements and implement solutions that meet business needs.
• Conduct performance tuning and optimization of data pipelines to improve throughput and reduce latency.
• Document technical designs, specifications, and procedures to ensure knowledge sharing and maintain system integrity.
• Stay updated with emerging technologies and industry trends related to data engineering and analytics.

Requirements:

• Bachelor's degree in Computer Science, Engineering, or a related field.
• 5+ years of experience in data engineering with a focus on Databricks and Scala.
• Strong proficiency in Apache Spark and Scala programming language.
• Hands-on experience with Azure Data Factory (ADF) dataflow and pipeline development.
• Experience working with Delta Lake and managing Delta table formats.
• Solid understanding of data processing concepts, data modeling, and ETL best practices.
• Ability to troubleshoot and debug complex data pipeline issues effectively.
• Excellent communication and collaboration skills, with the ability to work independently and in a team environment
•  
• Nice to Have:
• Experience with other cloud platforms such as AWS or Google Cloud Platform.
• Knowledge of Python programming language.
• Familiarity with containerization technologies such as Docker and Kubernetes.
• Certification in Azure Data Engineering or related field.