Lead AWS Glue Data Engineer

Overview

Remote
$55 - $65
Contract - W2
Contract - 12 Month(s)

Skills

AWS Solutions Architect certification
Databricks
AWS

Job Details

Hi,
Hope you are doing Great!

Job Position : Lead AWS Glue Data Engineer

Job Location : Remote (New Jersey preferred)

Job Duration : 6-12 Months

Job Description :

Position Overview

We are seeking an exceptional Lead AWS Glue Data Engineer to architect, implement, and optimize enterprise-scale data pipelines within our AWS data lake ecosystem. This role requires a seasoned professional who can work on data engineering initiatives, drive technical excellence, and be parts of our efforts for strategic migration to Databricks platform. The ideal candidate will have deep expertise in AWS cloud-native data services, strong leadership capabilities, and hands-on experience building robust, scalable data solutions.

Location: Remote (New Jersey preferred)

Experience Level: Senior (8+ years)

Key Responsibilities

Technical Competence & Architecture

  • Design, develop, and maintain complex ETL pipelines using AWS Glue, Glue Studio, and Glue Catalog
  • Implement AWS data lake solutions using S3, Redshift, Athena, Lambda, and Step Functions
  • Implement the technical strategy for data pipeline orchestration and workflow automation
  • Develop and optimize PySpark/Scala scripts within AWS Glue for complex data transformations
  • Implement comprehensive data quality checks, lineage tracking, and monitoring across all pipelines

Required Qualifications

Experience & Education

  • Bachelor's degree in Computer Science, Information Technology, Engineering, or related field
  • 8-15+ years of experience in data engineering with at least 5+ years focused on AWS cloud data services
  • Proven track record of delivering enterprise-scale data solutions in production environments

Core Technical Skills

  • Expert-level proficiency in AWS Glue, S3, Redshift, Athena, Lambda, Step Functions, and CloudWatch
  • Advanced programming skills in Python, PySpark, SQL, and Scala for ETL and data transformations
  • Strong experience with data modeling, data warehousing, and ETL/ELT processes
  • Hands-on experience with data lake and data warehouse architecture design and implementation
  • Proficiency in Unix/Linux environments including shell scripting and command-line operations

AWS & Cloud Expertise

  • Deep understanding of AWS data ecosystem and cloud-native data services
  • Experience with NoSQL databases (DynamoDB, MongoDB, Cassandra) and relational databases
  • Knowledge of real-time data processing and streaming technologies (Kinesis, Kafka)
  • Understanding of AWS security best practices, IAM roles, and data encryption

Data Engineering Fundamentals

  • Strong background in data validation, data quality management, and data governance
  • Experience with Agile/Scrum methodologies and DevOps practices
  • Knowledge of data visualization tools like Tableau, Power BI, or QuickSight
  • Understanding of metadata management and data cataloging principles

Preferred Qualifications

  • Databricks & Migration Experience
  • Hands-on experience with Databricks platform, Delta Lake, and MLflow
  • Knowledge of AWS to Databricks migration patterns and best practices
  • Understanding of lakehouse architecture and unified analytics platforms
  • Experience with Databricks notebooks, workflows, and cluster management

Advanced Certifications

  • AWS Certified Data Analytics Specialty or AWS Solutions Architect certification
  • Databricks Certified Data Engineer Associate or Professional certification
  • Additional AWS certifications (e.g., AWS Certified Data Engineer Associate)

Thanks and Regards,

Pushpa Nath

Employers have access to artificial intelligence language tools (“AI”) that help generate and enhance job descriptions and AI may have been used to create this description. The position description has been reviewed for accuracy and Dice believes it to correctly reflect the job opportunity.