Data engineer

W2 profile only

Location: Whippany, NJ (Hybrid)

Interview Mode: In person interview required

Job Description:

·        Hands‑on senior engineer, working directly with developers on design and implementation of modernization initiatives.

·        Strong Data engineer with more than 8 years of experience

·        Strong hand on experience in Python

·        Strong handoff experience in Pyspark and stream processing with Kafka.

·        Lead containerization and cloud onboarding of services. 

·        Drive adoption of GitLab CI/CD, M1 pipelines, and release automation.

·        Champion modern testing practices

·        Drive Kafka adoption for event driven design standards

Role Overview

• Pipeline Development: Build and maintain ETL/ELT pipelines for ingesting and transforming data.

• Data Warehousing: Design and manage data warehouses and lakes (Snowflake, BigQuery, Redshift).

• Big Data Processing: Optimize large-scale data workflows using Apache Spark or Hadoop.

• Data Governance: Ensure data quality, lineage, and compliance with regulations.

• Workflow Orchestration: Use Airflow or similar tools to schedule and monitor pipelines.

• Integration: Connect APIs, databases, and streaming sources (Kafka).

• Collaboration: Partner with analysts, data scientists, and business teams to deliver usable datasets.

Required Skills

• Data Quality Issues: Poor validation can lead to unreliable analytics.

• Pipeline Failures: Inadequate monitoring may cause downtime and data loss.

• Cost Overruns: Inefficient queries or storage can inflate cloud costs.

• Compliance Risks: Missing GDPR/DPDP controls can lead to legal exposure.
  
  

Best Practices

• Automate pipeline monitoring with Airflow/Kafka.

• Use data profiling before ingestion to detect anomalies.

• Implement partitioning and indexing for performance.

• Collaborate closely with data science teams to align schema design.