Lead / Senior Data Engineer

Role: Lead / Senior Data Engineer

Location: McLean, VA (Onsite)

Role Summary

• Lead the design and development of cloud-native data pipelines on AWS and Snowflake.
• Drive Spark-on-EKS, advanced Snowflake capabilities, and AI/Agentic AI proofs of concept (PoCs).
• Mentor mid-level developers and collaborate with stakeholders.
• Key Responsibilities
• Architect end-to-end pipelines using AWS, Snowflake, IICS, and PySpark.
• Design and deploy containerized PySpark workloads on EKS.
• Build and tune EMR clusters on EC2 or EKS for large-scale processing.
• Orchestrate workflows using AWS Step Functions and Control-M.
• Manage metadata using AWS Glue Data Catalog and integrate it with Snowflake Iceberg.
• Use DynamoDB for pipeline state, checkpoints, and configuration.
• Lead implementation of Snowflake capabilities, including Iceberg Tables, Dynamic Tables, Views, Materialized Views, Tasks and Streams, Data Governance, Horizon Catalog, Semantic Views, Cortex (CoCo), Snowflake Agents, and token-usage optimization strategies.
• Build RAG and Agentic AI PoCs covering vector search, BM25 hybrid retrieval, evaluation frameworks, human-in-the-loop controls, and security controls.
• Develop PySpark and Snowpark pipelines for ingestion and transformation.
• Lead IICS file-based ingestion design.
• Conduct code reviews, improve performance, and mentor engineers.

Leadership and Collaboration Expectations

• Demonstrates ownership and accountability for design quality, delivery commitments, and production outcomes.
• Communicates complex technical decisions clearly to engineering teams, product partners, and stakeholders.
• Collaborates across teams, constructively challenges assumptions, and drives alignment on scalable solutions.
• Provides practical coaching and feedback that strengthens engineering capability and promotes knowledge sharing.
• Applies sound judgment, balances speed with engineering rigor, and proactively manages risks and dependencies.
• Shows curiosity and adaptability when evaluating emerging cloud, data, and AI capabilities.

Skills Required

• Strong hands-on experience with AWS data services, including EKS, EMR, EC2, Step Functions, DynamoDB, AWS Glue Data Catalog, and S3.
• Advanced Snowflake expertise.
• Expertise in PySpark and Snowpark.
• Hands-on experience with IICS for file-based ETL ingestion.
• Experience building RAG and Agentic AI PoCs.
• Strong SQL and data modeling skills.
• Leadership and mentoring capability.
• Knowledge of Control-M orchestration.

Must-Have Skills

• AWS: EKS, EMR, EC2, Step Functions, DynamoDB, AWS Glue Data Catalog, S3, and IAM.
• Snowflake: Iceberg, Dynamic Tables, Views, Materialized Views, Tasks, Governance, Horizon, Cortex, and Agents.
• PySpark on EKS, including containerized Spark.
• Snowpark for Python.
• IICS for file-based ETL ingestion.
• Strong SQL and data modeling.
• Experience with RAG or Agentic AI PoCs.
• Awareness of reporting tools such as Power BI or Tableau.
• Control-M scheduling.
  

  Mujeeb Shaik

  Technical Talent Acquisition Specialist

  Email Id:

  Thinklusive Inc, 1100 Cornwall Rd, Suite 120, Monmouth Junction, NJ, 08852

  "Thinklusive is an Equal Opportunity Employer," please note. Regardless of race, colour, religion, sex, sexual orientation, gender identity, national origin, age, disability, or veteran status, we accept and give consideration to all eligible candidates for employment. Our goal is to establish a workplace that is inclusive and varied. "