Sr. Data Engineer(Azure and Databricks)

Note: This is an Hybrid position and requires 4 days onsite Monday to Thursday and Friday Remote

We are seeking an experienced and highly skilled Data Engineer with expertise in Azure and Databricks to join our data and reporting team. In this role, you will be responsible for designing, building, and maintaining robust data architectures and pipelines in a cloud-based environment. You will collaborate with cross-functional teams to ensure data availability, performance, and quality for analytics and machine learning solutions.

REQUIRED

• Design, develop, and maintain scalable and robust data pipelines on Databricks.

• Collaborate with data/business analysts to understand data requirements and deliver solutions.

• Architect and optimize Databricks workflows and data pipelines to process, clean, and transform large datasets.

• Design high-performance, low-latency data models that ensure optimal storage and data retrieval for analytics applications.

• Optimize and troubleshoot existing data pipelines for performance and reliability.

• Ensure data quality and integrity across various data sources.

• Guide and collaborate with data engineers to set standards, do code review and provide them solutioning for different enterprise initiatives.

• Implement data security and compliance best practices.

• Monitor data pipeline performance and conduct necessary maintenance and updates.

• Collaborate with DevOps teams to automate data pipeline deployment and integration.

• Data Exploration and Analysis: Analysts use statistical tools and techniques to explore and analyze data, identifying patterns, relationships, and trends.

• Success in data analysis demands a balance of technical skills (data cleaning and statistical analysis) and soft skills (critical thinking and communication)

• Familiarity with data structures, storage systems, cloud infrastructure, and other technical tools.

• Ability to work effectively in teams of technical and non-technical individuals.

• Ability to continuously learn, work independently, and make decisions with minimal supervision

• Demonstrate accountability, prioritize tasks, and consistently meet deadlines.

• Familiarity with Agile/SCRUM practices

• Work with cloud platforms such as AWS, Azure, or Google Cloud to integrate and manage data workflows and storage solutions.

• Strong expertise in Databricks, including Spark (PySpark/Scala).

• Hands-on experience working with cloud platforms (AWS, Azure, or Google Cloud).

• Experience with CI/CD tools like Azure DevOps, GitHub Actions, or Terraform for infrastructure-as-code.

• Experience in designing and developing ETL processes and data pipelines.

• Familiarity with data modeling, data warehousing, and database design.

• Strong understanding of data formats such as Parquet, ORC, JSON, and Avro.

• Ability to work in an agile environment and adapt to changing requirements.

• Strong analytical and problem-solving skills with a focus on optimizing data engineering solutions.

• Study, analyze and understand business requirements in context to business intelligence.

• Design and map data models to shift raw data into meaningful insights.

• Spot key performance indicators with apt objectives

• Build multi-dimensional data models

• Develop strong data documentation about algorithms, parameters, models

• Make essential technical and strategic changes to improvise present business intelligence systems

• Identify the requirements and develop custom charts accordingly

• SQL querying for better results

• Bachelor’s degree in computer science, Engineering, or a related field (or equivalent experience).

• Proven experience working with Databricks for large-scale data processing.

• Experience with CI/CD tools like Azure DevOps, GitHub Actions, or Terraform for infrastructure-as-code.

• Strong proficiency in SQL and experience with relational databases.

• 5+ years of data engineering experience

• Proficient in SQL writing skills MUST HAVE SKILLS:

• Minimum 10 years of experience of working on Data platforms.

• Knowledge about database management, SQL querying, data modeling, and Online Analytical Processing.

• Additional consideration given for experience scripting and programming language such as Python

• Experience with CI/CD pipelines for data engineering tasks.

• Ability to mentor junior team members and contribute to technical leadership