Search Jobs | Dice.com

Job Details

Skills

Python
Ansible
Docker
Apache
Kafka
Kinesis
Databricks
AWS
CI/CD
Data Engineering
Data Engineer

Summary

Description:

Job Summary:

We are looking for a mid level engineer to build and operate a data platform that uses Apache Iceberg as the lake house table format and Docker based micro services (Spark, Flink, Presto, etc.). you will own the end to end delivery pipeline, monitoring, security, and incident response, ensuring the platform runs reliably at scale.

Key Responsibilities:

Iceberg operations: support tables, manage schema changes, partitions, snapshot retention, and keep the catalog (Hive Metastore, AWSGlue, Nessie, ) synchronized.
Docker image creation & testing: write multi stage Dockerfiles for Spark/Flink/Presto, run local test environments with Docker Compose, and conduct vulnerability scans (Trivy, Snyk, ).
Data pipeline development: build ETL/ELT jobs that ingest raw data and write to Iceberg tables; add simple streaming components using Kafka, Pulsar, or Kinesis when needed.
CI/CD automation: configure pipelines (GitHubActions, GitLabCI, AzureDevOps, ) to lint Dockerfiles, scan images, version Iceberg metadata, and deploy pipelines without downtime.
Automation with Ansible/Python: script cluster provisioning, catalog configuration, vacuum/compaction, and other routine housekeeping tasks.
Observability: instrument services with OpenTelemetry, Prometheus, Grafana, and Loki; create dashboards showing pipeline latency, resource usage, table health, and error rates; set up basic alerts.
SLA monitoring: measure data freshness, job success rates, and query response times against agreed upon targets and report deviations.
Incident response: join the on call rotation, perform first line diagnosis and resolution of pipeline failures, Iceberg metadata issues, or container crashes; write concise root cause analyses and suggest improvements.
Security & compliance support: help enforce image signing, mTLS, IAM roles, and bucket policies; collaborate with the security team to meet GDPR, HIPAA, or ISO27001 requirements.
Knowledge sharing: keep internal documentation up to date and run short tech demos or brown bag sessions on Iceberg, Docker best practices, and automation techniques.

Minimum Requirements:

Bachelor s degree in Computer Science, IT, Data Engineering, or a related field (Master s a plus).
~5years of hands on experience building and operating large scale data platforms (lake house, data warehouse, or big data ecosystems).
Proven production experience with Apache Iceberg (table creation, partition management, schema evolution, catalog integration).
Strong Docker skills: multi stage builds, Docker Compose testing, routine image security scanning.
Experience with at least one major data processing engine (Spark, Flink, or Presto/Trino) and its connection to Iceberg tables.
Proficiency in Python and/or Ansible for automating infrastructure and platform tasks.
Experience building CI/CD pipelines that include Docker linting, vulnerability scanning, and automated deployment of data pipeline code.
Familiarity with observability tooling (Prometheus, Grafana, OpenTelemetry, Loki) and ability to create useful alerts and dashboards.
Ability to respond to incidents, write clear root cause analysis reports, and contribute to post mortem actions.
Willingness to participate in an on call rotation as a first line responder.
Availability to work on site in NewJersey for the initial assignment and relocate to Dallas by October2026.

Preferred Qualifications:

Experience with cloud native data services on AWS, Azure, or Google Cloud Platform (EMR, Dataproc, Synapse, etc.).
Familiarity with other lake house formats such as DeltaLake or ApacheHudi and ability to evaluate trade offs against Iceberg.
Knowledge of streaming platforms (Kafka, Pulsar, Kinesis) and real time processing patterns.
Relevant certifications (Databricks Lakehouse Associate, Google Professional Data Engineer, AWS Certified Data Analytics Specialty, etc.).
Background supporting data platforms in regulated industries (pharma, finance, healthcare) and understanding of associated compliance frameworks.

Employers have access to artificial intelligence language tools (“AI”) that help generate and enhance job descriptions and AI may have been used to create this description. The position description has been reviewed for accuracy and Dice believes it to correctly reflect the job opportunity.

Dice Id: cybrthnk
Position Id: 9041759
Posted 1 day ago

Fitment

Dice Job Match Score™

🔢 Crunching numbers...

Contact the job poster

Suresh Parmar

Recruiter @ cyberThink, Inc.

View Profile

Create job alert

Never miss an opportunity! Create an alert based on the job you applied for.

Similar Jobs

DataOps Engineer

Englewood, New Jersey

•

Yesterday

DataOps Engineer Location: Englewood Cliffs, NJ & Plano, TX -(5 days Onsite/week) Duration: 1 year Work Hours: 9-6 local time We are looking for candidates who are willing to relocate to Texas starting in October and are available to work onsite in New Jersey through September Job Summary: We are looking for a mid level engineer to build and operate a data platform that uses Apache Iceberg as the lake house table format and Docker based micro services (Spark, Flink, Presto, etc.). you will own

Easy Apply

Third Party, Contract

Depends on Experience

Senior Quantitative Data Engineer

New York, New York

•

Today

The Role As the driving force behind Schonfeld's next-generation data platforms, you will architect, operate, and continually refine underlying platform that empowers research, systematic trading, AI and risk analytics. On top of this robust foundation, you'll build and run high-throughput pipelines spanning ultra-low-latency market-data streams to cost-efficient end-of-day workflows. Partner closely with quants, data scientists, and fellow engineers to achieve sub-minute SLAs, shape firm-wide a

Full-time

USD 200,000.00 - 220,000.00 per year

Lead Data Engineer ? Vice President

Jersey City, New Jersey

•

Today

Citi is looking for a Lead Data Engineer ? AI & Distributed Analytics to architect and scale the enterprise data platforms that power our advanced analytics, machine learning, and Generative AI capabilities. In this high-impact role, you will lead a team of data engineers to design and deliver production-grade, high-performance data pipelines that bridge raw data and intelligent applications across the enterprise. You will work at the forefront of data innovation, collaborating directly with Dat

Full-time

Compensation information provided in the description

Director, Data Platform & Engineering

New York, New York

•

Today

Company Overview:Blink Health is the fastest growing healthcare technology company that builds products to make prescriptions accessible and affordable to everybody. Our two primary products - BlinkRx and Quick Save - remove traditional roadblocks within the current prescription supply chain, resulting in better access to critical medications and improved health outcomes for patients. BlinkRx is the world's first pharma-to-patient cloud that offers a digital concierge service for patients who ar

Full-time

Filter Results

Job post features

Posted date

Work settings

Employment type

Distance

Employer type

Work authorization

Data Engineer

Skills

Summary

Dice Job Match Score™

Suresh Parmar

Similar Jobs

Data Engineer

Job Details

Skills

Summary

Dice Job Match Score™

Suresh Parmar

Similar Jobs