Only W2 | DataOps Engineer@ NJ & TX

Job Role: DataOps Engineer

Location: Englewood Cliffs till September, 2026| October onward Plano, TX

Duration: Long Term Contract

Key Responsibilities

• Iceberg operations: support tables, manage schema changes, partitions, snapshot retention, and keep the catalog (Hive Metastore, AWS Glue, Nessie, ) synchronized.
• Docker image creation & testing: write multi stage Docker files for Spark/Flink/Presto, run local test environments with Docker Compose, and conduct vulnerability scans (Trivy, Snyk, ).
• Data pipeline development: build ETL/ELT jobs that ingest raw data and write to Iceberg tables; add simple streaming components using Kafka, Pulsar, or Kinesis when needed.
• CI/CD automation: configure pipelines (GitHub Actions, GitLab CI, Azure DevOps, ) to lint Docker files, scan images, version Iceberg metadata, and deploy pipelines without downtime.
• Automation with Ansible/Python: script cluster provisioning, catalog configuration, vacuum/compaction, and other routine housekeeping tasks.
• Observability: instrument services with Open Telemetry, Prometheus, Grafana, and Loki; create dashboards showing pipeline latency, resource usage, table health, and error rates; set up basic alerts.
• SLA monitoring: measure data freshness, job success rates, and query response times against agreed upon targets and report deviations.
• Incident response: join the on call rotation, perform first line diagnosis and resolution of pipeline failures, Iceberg metadata issues, or container crashes; write concise root cause analyses and suggest improvements.
• Security & compliance support help enforce image signing, mTLS, IAM roles, and bucket policies; collaborate with the security team to meet GDPR, HIPAA, or ISO 27001 requirements.
• Knowledge sharing: keep internal documentation up to date and run short tech demos or brown bag sessions on Iceberg, Docker best practices, and automation techniques.

Minimum Requirements

• Bachelor's degree in Computer Science, IT, Data Engineering, or a related field (Master's a plus).
• ~5 years of hands on experience building and operating large scale data platforms (lake house, data warehouse, or big data ecosystems).
• Proven production experience with Apache Iceberg (table creation, partition management, schema evolution, catalog integration).
• Strong Docker skills: multi stage builds, Docker Compose testing, routine image security scanning.
• Experience with at least one major data processing engine (Spark, Flink, or Presto/Trino) and its connection to Iceberg tables.
• Proficiency in Python and/or Ansible for automating infrastructure and platform tasks.
• Experience building CI/CD pipelines that include Docker linting, vulnerability scanning, and automated deployment of data pipeline code.
• Familiarity with observability tooling (Prometheus, Grafana, OpenTelemetry, Loki) and ability to create useful alerts and dashboards.
• Ability to respond to incidents, write clear root cause analysis reports, and contribute to post mortem actions.
• Willingness to participate in an on call rotation as a first line responder.