Job Title: Senior Data Engineer
Job Location: New Jersey (Hybrid)
Job Type: Full-Time
Core Responsibilities
Execute Legacy-to-Cloud Migrations: Lead the end-to-end migration of complex datasets from on-premise legacy systems (including Mainframe and Informatica-based workflows) to the Azure.
Develop Databricks Pipelines: Build and optimize high-throughput ETL/ELT pipelines using PySpark and Delta Live Tables (DLT) to ensure data consistency and reliability.
Integrate Hybrid Workflows: Map and translate legacy logic (Informatica mappings and Mainframe COBOL/Copybooks) into modern, code-based transformations within Azure Data Factory (ADF) and Databricks.
Performance Tuning & MLOps: Monitor and tune Databricks clusters for cost-efficiency and performance; implement CI/CD pipelines via Azure DevOps to automate data deployments.
Required Technology stack: Azure (Data Factory, ADLS Gen2, Synapse, Azure DevOps), Azure Databricks, Apache Spark (PySpark/Scala), Delta Lake, Advanced Python, SQL, and Shell Scripting, ADF Triggers, Databricks Workflows, or Airflow
Good to have: Informatica PowerCenter (mappings/workflows), Mainframe (DB2, VSAM, JCL).
Experience & Qualifications
Experience: 6 10 Years
Bachelor s or Master s Degree in Computer Science, Information Systems, or a related field.
Proven track record of a full-cycle migrations from on-premise environments to Azure.
Expertise in Databricks SQL and Spark core, specifically focusing on performance optimization of large-scale joins and aggregations.
Azure Certification: AZ-204 (Developer) or AZ-305 (Solutions Architect) is a significant advantage
Never miss an opportunity! Create an alert based on the job you applied for.
