Apply Now

AWS Data Engineer - Open Data Platform

New York, NY, US • Posted 3 days ago • Updated 3 hours ago

Full Time

On-site

Fitment

Dice Job Match Score™

🔢 Crunching numbers...

Job Details

Skills

Management
Access Control
Benchmarking
Performance Analysis
Snow Flake Schema
Data Engineering
Extract
Transform
Load
Amazon S3
Apache Spark
Data Processing
Apache HTTP Server
Amazon Web Services
Computer Networking
Cloud Computing
Performance Tuning
Optimization
Evaluation

Summary

Our client is seeking an experienced AWS Data Engineer to design and build cloud-native data pipelines on an open data platform, architecting Apache Iceberg tables and optimizing large-scale data processing across AWS and Snowflake environments.

Responsibilities & Qualifications

Design and develop end-to-end data ingestion, transformation, and processing pipelines using AWS Glue, Apache Spark, and Amazon S3
Build and manage Apache Iceberg tables to enable open, interoperable access across multiple compute engines
Configure and integrate Apache Polaris with Iceberg REST Catalog for AWS and Snowflake environments
Implement table maintenance, schema evolution, and partitioning strategies to optimize performance and cost
Support platform security, access controls, and AWS networking infrastructure
Conduct workload benchmarking and performance analysis comparing AWS Glue versus Snowflake for cost and efficiency
Provide operational monitoring, troubleshooting, and optimization of cloud-based data platform infrastructure

Requirements

8-10 years of experience in data engineering and cloud-based data pipeline development
Proven expertise with Amazon S3, AWS Glue, Apache Spark, and large-scale data processing
Strong hands-on experience with Apache Iceberg architecture, including table design and maintenance
Demonstrated proficiency with Apache Iceberg schema evolution and partitioning optimization
Experience configuring and deploying Apache Polaris and Iceberg REST Catalog
Solid understanding of AWS IAM, networking, and security best practices in cloud environments
Experience with performance tuning, cost optimization, and cross-platform data tool evaluation

Employers have access to artificial intelligence language tools (“AI”) that help generate and enhance job descriptions and AI may have been used to create this description. The position description has been reviewed for accuracy and Dice believes it to correctly reflect the job opportunity.

Dice Id: 91166607
Position Id: 588f85c02b67ca2c70dd19a05d9a06ed
Posted 3 days ago

Create job alert

Never miss an opportunity! Create an alert based on the job you applied for.

Similar Jobs

AWS Glue Data Engineer

Hybrid in New York, New York

•

5d ago

Job Title: AWS Glue Data Engineer(14+ Years Only) Location:Fort Mill SC / New York, NY / Florham Park, NJ(Hybrid) Job Summary We areseekinga highly skilledAWS Data Engineerto design, develop, andoptimizelarge-scale data pipelines and ETL workflows on AWS. The ideal candidate will have strongexpertisein AWS cloud-native data services, data modeling, and pipeline orchestration, with hands-on experience building robust and scalable data solutions for enterprise environments. Key Responsibilities D

Easy Apply

Full-time

Depends on Experience

Data Engineer

New York, New York

•

Today

Data Engineer Location: New York, NY Employment Type: Full-time Focus: Data Engineering, Snowflake, Data Modeling, AI Data Infrastructure, Marketing Technology About Our Client Our client is building an AI-powered platform for marketing leaders. Their technology helps marketing teams focus on brand storytelling while AI agents handle the operationally intensive work behind the scenes, including data management, analytics, campaign generation, measurement, and reporting. The platform is buil

Full-time

Lead Data Engineer

New York, New York

•

Today

Location: New York City, NY Salary: $80.00 USD Hourly - $88.00 USD Hourly Description: Lead Data Engineer New York, NY (10001) 6+ months Contract to Perm About the Role We are seeking a Lead Software Engineer - Data Engineering to design and build scalable, cloud-native data platforms and pipelines. You will lead the development of robust data solutions, drive architectural decisions, and mentor engineers while ensuring reliability, security, and operational excellence. Responsibilities

Contract

USD 80.00 - 88.00 per hour

Senior Data Engineer

New York, New York

•

Today

As a Senior Data Engineer at Regard, you will own the design, development, and production deployment of the data services that power the Regard platform. From ingesting and standardizing clinical data across health systems to making it reliably available for downstream product and analytics workflows, you'll build and evolve the infrastructure that enables the platform. This includes analyzing and tuning Spark workloads and partitioning strategies to control costs, adapting to upstream breaking

Full-time

USD 165,000.00 - 220,000.00 per year

Search all similar jobs

More jobs at Upward Talent in New York, NY