Lead Data Engineer – Java, Python, Spark & AWS


Intuitive Insights LLC
Dice Job Match Score™
🔢 Crunching numbers...
Job Details
Skills
- Data Engineering
- Java
- Python
- Apache Spark
- PySpark
- AWS
- ETL
- ELT
- Data Pipelines
- Data Lake
- Data Lakehouse
- Distributed Computing
- Snowflake
- Parquet
- Avro
- Data Processing
- Data Warehousing
- Data Integration
- Apache Airflow
- Dagster
- Control-M
- Apache Kafka
- Spark Structured Streaming
- Delta Lake
- Apache Iceberg
- Apache Hudi
- Data Governance
- Data Quality
- Data Validation
- Data Reconciliation
- AWS S3
- AWS Glue
- AWS EMR
- AWS Lambda
- CI/CD
- REST API
Summary
Hi,
I'm Hiring for:
Lead Data Engineer – Java, Python, Spark & AWS
Location: Columbus, OH (Polaris)
Experience: 10+ Years
Duration: 6 Months
Employment Type: Contract-to-Hire (RTH) -W2 Only
Job Description:
Seeking an experienced Lead Data Engineer with strong expertise in Java, Python, AWS, ETL/ELT pipelines, and Apache Spark.
Key Responsibilities:
Lead architecture and development of large-scale ETL/ELT data pipelines.
Design data ingestion, transformation, validation, and reconciliation workflows.
Develop distributed data processing solutions using Apache Spark.
Design and implement data lake and lakehouse architectures.
Optimize Spark workloads, including joins, caching, partitioning, and performance tuning.
Develop Java-based services and REST APIs.
Implement automated testing, CI/CD, monitoring, and production support.
Drive engineering best practices and mentor development teams.
Promote enterprise-approved AI-assisted engineering practices.
Required Skills:
10+ years of software engineering experience.
Strong hands-on Java development.
Python proficiency.
ETL/ELT pipeline development.
Apache Spark and distributed processing.
AWS cloud experience.
Parquet, Avro, and data lake/lakehouse architecture.
Snowflake or equivalent cloud data warehouse experience.
AI-assisted development tools and responsible AI practices.
Preferred Skills:
Airflow, Dagster, or Control-M.
Kafka and Spark Structured Streaming.
Delta Lake, Apache Iceberg, or Hudi.
Data governance and data quality.
Financial services or regulated industry experience.
Agentic AI and Generative AI exposure.
- Dice Id: 91175879
- Position Id: 311277-1
- Posted 7 hours ago
Company Info
About Intuitive Insights LLC
Intuitive Insights is a workforce solutions firm built on one principle: the right person in the right role changes everything.
We partner with employers to source, screen, and place highly qualified professionals across technology, engineering, healthcare, finance, and business — quickly and with precision.
Whether you need a single specialist or a full team, we deliver talent that drives results — on your timeline and within your budget.

