Senior Data Engineer

Senior Data Engineer
McLean, VA
Long Term
W2

Senior Data Engineer for design and deployment of our next-generation Enterprise Data-as-a-Service (DaaS) solution. This role bridges the gap between legacy operational systems and modern cloud-native analytics, focusing on building and operationalizing robust data pipelines across AWS and Snowflake ecosystems.

The successful candidate will have a strong background in distributed data processing, lakehouse architecture, and a passion for solving complex business problems through data democratization

What You ll Get to Do:

Design, build, and maintain a robust Data-as-a-Service (DaaS) provider model to effectively decouple data producers from downstream consumers.

Architect and manage end-to-end data pipelines through a multi-stage Medallion architecture.

Lead digital transformation initiatives by migrating legacy operational data from MS SQL Server and Sybase to modern, cloud-native Snowflake and NoSQL (MongoDB) environments.

Develop and implement high-performance, event-driven backend solutions using Spring Boot microservices and ActiveMQ.

Design and operationalize scalable cloud-native processing services using AWS Glue (PySpark), Amazon EMR, and Snowflake.

Implement comprehensive data observability and proactive monitoring solutions using AWS CloudWatch, Grafana, and the ELK Stack.

Streamline development through DevOps best practices, maintaining efficient and reliable CI/CD pipelines via Jenkins, Bitbucket, and Terraform.

Collaborate with cross-functional engineering, product, and business teams

Solve challenging problems using data science and advanced machine learning methods.

Apply data science techniques to process and wrangle data in preparation for analysis; and analyse big data and perform exploratory analysis to uncover insights and identify innovative opportunities.

Leverage machine learning techniques, to explore and examine data from multiple disparate sources to provide a competitive advantage or address a pressing business problem.

Define approaches to embed and scale machine learning models with senior data scientist oversight

Assist on client pitches and proposals for machine learning projects

Work on difficult engineering problems with scientists and business leaders with minimal engineering or data science backgrounds.

Work with the data science team to build reusable assets, solutions and develop best practices for current and future business problems.

Work with the engineering team to develop end to end production ML systems

Staying up to date on the latest machine learning capabilities.

Work with other analysts to utilize analytics to solve business problems in various areas within financial services, like capital markets, wealth management, asset management, fraud, AML, process mining, customer centricity, risk management, talentology etc.

What You ll Bring with You:

7+ years of professional experience in Data Engineering and Analytics, with a focus on building high-availability, enterprise-grade data platforms.

Expertise in building and troubleshooting event-based microservices using Java and Spring Boot.

Proficiency in Apache Spark (Batch & Structured Streaming) for large-scale data transformation using Python (PySpark) or Scala.

Extensive experience with Snowflake and modern Lakehouse architectures (Databricks/Delta Lake).

Deep understanding of NoSQL (MongoDB) and RDBMS (SQL Server/Sybase).

Experience automating infrastructure using Terraform (IaC) and managing CI/CD pipelines (Jenkins, Azure DevOps, or Bitbucket).

Experience with data observability tools (ELK Stack or Grafana).

Experience working in a fast-paced Agile development environment.

Financial services industry experience 7+ years of relevant experience in analytics, business intelligence, data engineering, or related field

Experience with Informatica, ETL, Databricks 5+ years of demonstrated experience in building enterprise grade BI applications using tools such as: QlikSense, QlikView, Tableau or Looker

Must have PySpark and Snowflake experience

Experience working in large data warehouse environments

Bachelor s degree in a quantitative area such as math, statistics, computer science, engineering, or equivalent experience

Strong verbal/written communication and data presentation skills, including an ability to effectively communicate with both business and technical teams

Experience in using of Oracle SQL tools for data analysis is required

Candidates having strong experience in Data Mapping, Data Migration and system integration projects with knowledge of Wealth Management and related investment products.

Would be a plus, with knowledge in any of the Wealth Management products like TCS BaNCS

Desirable skills

Financial services industry experience

Experience with data visualisation tools (Tableau, QlikView etc)

Experience with Informatica, ETL, Databricks

Munesh

,

CYBER SPHERE LLC