This can be full-time or contracting for direct applicants only. The company is an integration technology company that can sponsor H1B candidates.
Job Title Data Engineer Python, React, Databricks & Azure
Location Hybrid (New York, NY / Edison, NJ)
Duration 12+ Months
Job Overview: We are seeking an experienced Data Engineer with strong hands-on expertise in React JS, Core Python, Azure, Databricks, and Delta Lake to design, develop, and support scalable data engineering solutions. The ideal candidate will build Python-based ETL pipelines, develop data workflows for GenAI applications, implement data quality frameworks, and collaborate with cross-functional teams to deliver reliable data platforms.
Key Responsibilities
Design, develop, and maintain Python-based ETL/ELT pipelines for enterprise data ingestion, transformation, validation, and storage.
Develop cloud-native data solutions using Azure Function Apps, Azure Blob Storage, Azure Cosmos DB, Azure Databricks, and Delta Lake.
Build and optimize Databricks notebooks, workflows, Delta Tables, and data transformation pipelines using Python and PySpark.
Develop reusable Core Python modules, libraries, utilities, and automation frameworks for data engineering processes.
Build data workflows supporting GenAI portals and applications, ensuring reliable ingestion, processing, storage, and availability of data.
Develop and maintain data pipelines integrating structured and semi-structured data from multiple sources.
Implement automated data validation, reconciliation, completeness, and quality checks to identify missing, duplicate, or incomplete records.
Optimize Databricks and Python workloads for performance, scalability, reliability, and cost efficiency.
Work with React to support data-driven portal components, dashboards, data visualization, and integration with backend APIs.
Develop and integrate REST APIs and backend services to expose processed data to React-based applications.
Implement monitoring, logging, error handling, alerting, and recovery mechanisms across data pipelines.
Provide production support, including incident triage, root cause analysis, troubleshooting, problem management, and SLA adherence.
Collaborate with business stakeholders, vendors, developers, QA teams, and application teams to understand requirements and deliver enhancements.
Participate in code reviews, technical design discussions, testing, deployment, and release activities.
Maintain technical documentation for data pipelines, architecture, data flows, operational procedures, and support processes.
Required Skills & Qualifications
Strong hands-on experience with Core Python and Python-based data engineering.
Experience with Azure cloud services, particularly:
Strong experience with Databricks, PySpark, Delta Lake, and Delta Tables.
Experience designing and maintaining ETL/ELT pipelines and data workflows.
Hands-on experience implementing data quality and validation frameworks.
Experience with React.js and integrating React applications with backend APIs/data services.
Strong understanding of REST APIs, JSON, API integration, and microservices-based architectures.
Experience with Git-based source control and CI/CD processes.
Strong SQL skills and experience working with relational and NoSQL data sources.
Experience with application monitoring, production support, incident management, and troubleshooting.
Strong communication and collaboration skills with technical and non-technical stakeholders.
Preferred Skills
Experience supporting GenAI/AI platforms or data portals.
Knowledge of Azure DevOps and CI/CD pipelines.
Experience with Azure API Management.
Knowledge of data lakehouse architecture and Medallion Architecture.
Experience with Agile/Scrum development methodologies.
Familiarity with observability, logging, and cloud monitoring tools.
Never miss an opportunity! Create an alert based on the job you applied for.
