Sr. Big Data Engineer

Role:- Sr. Big Data Engineer

Please Note: Strong Banking Domain experience is required & Only Local candidates to NJ/NY.

Our Challenge

• As a Big Data Engineer, candidate will be instrumental in developing and optimizing big data technologies such as Apache Spark and Azure Databricks.
• Candidate will create robust data pipelines, ensure high data quality, and collaborate closely with cross-functional teams to implement scalable cloud-native solutions on major cloud providers, predominantly Microsoft Azure.

The Role

Responsibilities:

• Develop, optimize, and maintain big data pipelines using Apache Spark, Azure Databricks, and related tools.
• Write and deploy complex production systems in Scala, Java, or Python, ensuring high performance and reliability.
• Leverage Terraform for automating cloud infrastructure deployment on Azure or other major cloud providers.
• Design and implement CI/CD pipelines for data workflows using modern tools like GitLab, promoting automation and continuous improvement.
• Apply best practices for testing, instrumentation, observability, and alerting to maintain data pipeline health and performance.
• Contribute to system architecture and low-level design, ensuring modularity, scalability, and security.
• Understand and implement data models, data structures, and algorithms for efficient data processing.
• Use containerization technologies such as Docker and Kubernetes for development, build, and runtime environments.
• Work within Agile development teams, participating in planning, daily stand-ups, and iterative releases.
• Collaborate effectively in a global team, influencing key architectural decisions and sharing best practices.

Requirements:

• 10+ Years of experience is required.
• Proven experience developing big data solutions with Apache Spark and Azure Databricks.
• Hands-on experience with Terraform for infrastructure as code (IaC) on major cloud platforms, ideally Azure.
• Programming expertise in Scala, Java, or Python for complex production systems.
• Extensive Python experience, especially in data engineering contexts.
• Background in platform engineering roles on cloud platforms, with strong knowledge of Azure services.
• Practical knowledge of testing frameworks, instrumentation, observability, and alerting tools.
• Experience building and maintaining CI/CD pipelines with modern cloud-friendly systems like GitLab.
• Deep understanding of information modelling, data structures, and algorithms.
• Hands-on experience with containerization (Docker) and orchestration tools (Kubernetes).
• Strong understanding of technical architecture and low-level system design.
• Familiarity with Agile methodologies and best practices in software development.