Lead Data Engineer

Dallas, TX, US • Posted 2 hours ago • Updated 2 hours ago
Full Time
Able to Sponsor
On-site
Depends on Experience
Fitment

Dice Job Match Score™

📊 Calculating match score...

Job Details

Skills

  • Data Lake
  • Lakehouse
  • migration
  • data pipelines
  • Python
  • Apache Spark
  • ETL/ELT pipeline
  • SCD Type 2
  • Core Data Engineering
  • Schema evolution
  • Partitioning and clustering
  • Normalization vs denormalization
  • Natural vs surrogate keys
  • Data quality frameworks
  • Lakehouse experience
  • Migration experience

Summary

Day to Day Job Duties:

  • Perform end-to-end datastore migration from on-prem Data Lake to AWS-hosted Lakehouse as part of the migration factory team
  • Refactor and migrate data pipelines including extraction logic and job scheduling.
  • Execute data transfer ensuring integrity and reliability.
  • Translate legacy SQL and Spark logic to Snowflake and Iceberg.
  • Analyze data usage patterns to deliver data products.
  • Build data reconciliation frameworks.
  • Collaborate with stakeholders for validation and sign-off.
  • Act as technical liaison.
  • Troubleshoot and optimize pipelines.
  • Adapt to new tools and workflows
  • Follow SDLC and CI/CD practices.
  • Collaborate with global teams.

Basic Qualifications:

  • 5 7 years hands-on coding
  • Strong SQL troubleshooting
  • Python or Java proficiency
  • Apache Spark experience
  • ETL/ELT pipeline experience
  • CI/CD and SDLC familiarity
  • Kubernetes exposure
  • Relevant degree

Core Data Engineering Competencies:

  • SCD Type 2
  • Schema evolution (Iceberg)
  • Partitioning and clustering
  • Normalization vs denormalization
  • Natural vs surrogate keys
  • Data quality frameworks
  • Technical Stack Requirements:
  • Kafka, ANSI SQL, FTP, Apache Spark
  • JSON, Avro, Parquet
  • Hadoop, Snowflake, Iceberg, Sybase IQ

Core Competencies:

  • Integrity
  • Collaboration
  • Communication
  • Stakeholder management
  • Global teamwork
  • Ownership
  • Professionalism
  • Continuous improvement

Nice to Have:

  • Migration experience
  • Financial services domain
  • Lakehouse experience
Employers have access to artificial intelligence language tools (“AI”) that help generate and enhance job descriptions and AI may have been used to create this description. The position description has been reviewed for accuracy and Dice believes it to correctly reflect the job opportunity.
  • Dice Id: 10360765
  • Position Id: 8965562
  • Posted 2 hours ago
Create job alert
Set job alertNever miss an opportunity! Create an alert based on the job you applied for.

Similar Jobs

Dallas, Texas

Today

Easy Apply

Contract, Third Party

$65 - $72

Dallas, Texas

Today

Full-time

Dallas, Texas

Today

Easy Apply

Full-time

$80,000 - $120,000

Dallas, Texas

Today

Full-time

Search all similar jobs