**** Webcam interview; *** Long term project with the possible contract to hire ***Linkedin Must*** Onsite ****
Job Description
Seeking an experienced Lead Data Engineer with strong expertise in Java, Python, AWS, ETL/ELT pipelines, and Apache Spark.
Key Responsibilities
- Lead architecture and development of large-scale ETL/ELT data pipelines.
- Design data ingestion, transformation, validation, and reconciliation workflows.
- Develop distributed data processing solutions using Apache Spark.
- Design and implement data lake and lakehouse architectures.
- Optimize Spark workloads, including joins, caching, partitioning, and performance tuning.
- Develop Java-based services and REST APIs.
- Implement automated testing, CI/CD, monitoring, and production support.
- Drive engineering best practices and mentor development teams.
- Promote enterprise-approved AI-assisted engineering practices.
Required Skills
- 10+ years of software engineering experience.
- Strong hands-on Java development.
- Python proficiency.
- ETL/ELT pipeline development.
- Apache Spark and distributed processing.
- AWS cloud experience.
- Parquet, Avro, and data lake/lakehouse architecture.
- Snowflake or equivalent cloud data warehouse experience.
- AI-assisted development tools and responsible AI practices.
Preferred Skills
- Airflow, Dagster, or Control-M.
- Kafka and Spark Structured Streaming.
- Delta Lake, Apache Iceberg, or Hudi.
- Data governance and data quality.
- Financial services or regulated industry experience.
- Agentic AI and Generative AI exposure.
