Search Jobs | Dice.com

Job Details

Skills

AWS
AWS Glue
PySpark
Python
ETL
SQL
Kafka
Spark
workflow orchestration

Summary

Role Summary:
We are seeking a Mid Senior Data Engineer with strong expertise in AWS-based data engineering, real-time streaming technologies, and enterprise-grade data quality frameworks. The ideal candidate will design, build, and optimize scalable batch and streaming data pipelines, implement robust data validation and monitoring processes, and support mission-critical analytics platforms.

Key Responsibilities:

Develop and maintain scalable ETL/ELT pipelines using AWS Glue, PySpark, and Python
Build event-driven workflows using AWS Lambda
Design and manage real-time streaming solutions using Kafka, KSQL, and Apache Flink
Implement and enforce comprehensive data quality frameworks, including validation, profiling, monitoring, and reconciliation
Optimize data processing performance, scalability, reliability, and cost in cloud environments
Collaborate with cross-functional teams to deliver reliable, production-grade data platforms and ensure data integrity across the pipeline

Required Skills:

Strong hands-on experience with Python and PySpark
Proven expertise in AWS Glue, Lambda, and other cloud-native data services
Solid experience with the Kafka ecosystem (topics, partitions, consumer groups, streaming patterns)
Demonstrated experience building and supporting data quality frameworks (validation rules, reconciliation checks, profiling, anomaly detection)
Strong understanding of distributed data processing and scalable architecture patterns

Good-to-Have Skills:

Experience with Apache Flink for real-time stream processing and stateful computations
Knowledge of KSQL or other streaming SQL engines
Exposure to CI/CD pipelines, IaC (Terraform/CloudFormation), and DevOps practices
Familiarity with data lake/lakehouse architectures and table formats such as Iceberg, Delta, or Hudi
Experience working in enterprise or financial data environments

Employers have access to artificial intelligence language tools (“AI”) that help generate and enhance job descriptions and AI may have been used to create this description. The position description has been reviewed for accuracy and Dice believes it to correctly reflect the job opportunity.

Dice Id: 10476889
Position Id: 9025044
Posted 3 hours ago

Fitment

Dice Job Match Score™

🎯 Assessing qualifications...

Contact the job poster

Rahul Dalal

Recruiter @ Incedo Inc

View Profile

Create job alert

Never miss an opportunity! Create an alert based on the job you applied for.

Hybrid in New York, New York

•

6d ago

Job Title: Sr.Data Engineer AWS & Streaming Location: Fort Mill SC or New York, NY (2-3 Days Hybrid) Experience level 12- 15 Years Role Summary: We are seeking a MidSenior Data Engineer with strong expertise in AWS-based data engineering, real-time streaming technologies, and enterprise-grade data quality frameworks. The ideal candidate will design, build, and optimize scalable batch and streaming data pipelines, implement robust data validation and monitoring processes, and support mission-c

Easy Apply

Contract

Depends on Experience

Lead AWS Data Engineer/ Lead Data Engineer/AWS Data Architect

Hybrid in Jersey City, New Jersey

•

Yesterday

Job Title: Lead AWS Data Engineer/ Lead Data Engineer/AWS Data Architect Work Location: Any location in USA- (Hybrid from office) Duration :: Long Term Contract Job Summary We are seeking an experienced Lead AWS Data Engineer/ Lead Data Engineer/AWS Data Architect with 8+ years of experience designing, building, and optimizing scalable cloud-based data platforms. The ideal candidate will possess deep expertise in AWS data services, distributed data processing, ETL/ELT development, and data ware

Easy Apply

Full-time

Depends on Experience

Data Engineer (Gen AI)

New York, New York

•

12d ago

Company Overview Interactive Brokers Group, Inc. (Nasdaq: IBKR) is a global financial services company headquartered in Greenwich, CT, USA, with offices in over 15 countries. We have been at the forefront of financial innovation for over four decades, known for our cutting-edge technology and client commitment. IBKR affiliates provide global electronic brokerage services around the clock on stocks, options, futures, currencies, bonds, and funds to clients in over 200 countries and territories.

Full-time

USD 150,000.00 - 200,000.00 per year

Data Engineer

New York, New York

•

Yesterday

Sr. Data Engineer AWS Glue & Snowflake, Healthcare Payer Location Midtown Manhattan, NYC- 4day Onsite Position Summary We are seeking a highly skilled Senior Data Engineer with deep expertise in AWS Glue and cloud-based data engineering to design, build, and optimize scalable data integration solutions. This role will be responsible for developing enterprise-grade ETL/ELT pipelines, managing large-scale data movement, and supporting modern analytics platforms built on AWS and Snowflake. The i

Easy Apply

Contract, Third Party

55 - 65

Filter Results

Job post features

Posted date

Work settings

Employment type

Distance

Employer type

Work authorization

Senior Data Engineer AWS & Streaming

Skills

Summary

Dice Job Match Score™

Rahul Dalal

Similar Jobs

Senior Data Engineer AWS & Streaming

Job Details

Skills

Summary

Dice Job Match Score™

Rahul Dalal

Similar Jobs