Lead Engineer- Big Data (Analytics)

Overview

Hybrid
Depends on Experience
Full Time
No Travel Required
Unable to Provide Sponsorship

Skills

Amazon Web Services
Apache Spark
PySpark
Python
ELT
Apache Hadoop
Continuous Integration
Continuous Delivery
Big Data
Apache Hive

Job Details

📍 Location: Dallas, TX (Hybrid – 3 days onsite preferred) | Remote (CST preferred)
💼 Type: Direct Hire / Full-Time
💰 Compensation: $150K base + 15% bonus

We are seeking a Lead Engineer – Big Data (Analytics) to join a high-impact Data Enablement team responsible for ingesting, transforming, and delivering analytics from multiple data sources. This is a hands-on technical leadership role within an AWS-based data platform, combining advanced engineering with ownership of delivery and stakeholder engagement.

You will work closely with a team of 10–12 engineers, contributing directly to development while guiding technical direction, best practices, and execution. There are no direct reports, but strong leadership, accountability, and initiative are key to success.

Key Responsibilities

  • Design, develop, and support scalable analytics solutions using PySpark, Python, Airflow, and SQL
  • Lead delivery of complex, cross-functional data initiatives on AWS (S3, EMR, Athena, Hive, Redshift)
  • Drive CI/CD, DevOps practices, and operational data engineering automation
  • Partner with business and technology stakeholders to gather requirements and translate them into technical solutions
  • Provide technical leadership, mentoring, and architectural guidance across teams
  • Support owned solutions through DevOps/on-call rotation
  • Continuously improve data ingestion, data lake stability, governance, and data quality
  • Communicate technical concepts clearly to senior and non-technical stakeholders

Required Skills & Experience

  • 10+ years of overall engineering experience with 4+ years in Big Data / Analytics
  • Strong hands-on expertise in PySpark, Python, Airflow, SQL
  • AWS experience with S3, EMR, Athena, Hive, Redshift
  • Experience with CI/CD pipelines, data governance, and automation
  • Data visualization experience with Tableau (MicroStrategy a plus)
  • Agile/Scrum development experience
  • Strong communication and stakeholder management skills
  • Proven lead engineer mindset: ownership, curiosity, and proactive problem-solving

Nice to Have

  • Retail domain experience
  • Experience using AI tools for development automation (GitLab Duo preferred)
  • Hadoop ecosystem, Spark, ETL/ELT modeling experience
Employers have access to artificial intelligence language tools (“AI”) that help generate and enhance job descriptions and AI may have been used to create this description. The position description has been reviewed for accuracy and Dice believes it to correctly reflect the job opportunity.