Data Engineer Python, React, Databricks & Azure

Job Title Data Engineer Python, React, Databricks & Azure Location Hybrid (New York, NY / Edison, NJ) Duration 12+ Months
Job Overview: We are seeking an experienced Data Engineer with strong hands-on expertise in React JS, Core Python, Azure, Databricks, and Delta Lake to design, develop, and support scalable data engineering solutions. The ideal candidate will build Python-based ETL pipelines, develop data workflows for GenAI applications, implement data quality frameworks, and collaborate with cross-functional teams to deliver reliable data platforms.
Key Responsibilities

• Design, develop, and maintain Python-based ETL/ELT pipelines for enterprise data ingestion, transformation, validation, and storage.
• Develop cloud-native data solutions using Azure Function Apps, Azure Blob Storage, Azure Cosmos DB, Azure Databricks, and Delta Lake.
• Build and optimize Databricks notebooks, workflows, Delta Tables, and data transformation pipelines using Python and PySpark.
• Develop reusable Core Python modules, libraries, utilities, and automation frameworks for data engineering processes.
• Build data workflows supporting GenAI portals and applications, ensuring reliable ingestion, processing, storage, and availability of data.
• Develop and maintain data pipelines integrating structured and semi-structured data from multiple sources.
• Implement automated data validation, reconciliation, completeness, and quality checks to identify missing, duplicate, or incomplete records.
• Optimize Databricks and Python workloads for performance, scalability, reliability, and cost efficiency.
• Work with React to support data-driven portal components, dashboards, data visualization, and integration with backend APIs.
• Develop and integrate REST APIs and backend services to expose processed data to React-based applications.
• Implement monitoring, logging, error handling, alerting, and recovery mechanisms across data pipelines.
• Provide production support, including incident triage, root cause analysis, troubleshooting, problem management, and SLA adherence.
• Collaborate with business stakeholders, vendors, developers, QA teams, and application teams to understand requirements and deliver enhancements.
• Participate in code reviews, technical design discussions, testing, deployment, and release activities.
• Maintain technical documentation for data pipelines, architecture, data flows, operational procedures, and support processes.

Required Skills & Qualifications

• Strong hands-on experience with Core Python and Python-based data engineering.
• Experience with Azure cloud services, particularly:
• Strong experience with Databricks, PySpark, Delta Lake, and Delta Tables.
• Experience designing and maintaining ETL/ELT pipelines and data workflows.
• Hands-on experience implementing data quality and validation frameworks.
• Experience with React.js and integrating React applications with backend APIs/data services.
• Strong understanding of REST APIs, JSON, API integration, and microservices-based architectures.
• Experience with Git-based source control and CI/CD processes.
• Strong SQL skills and experience working with relational and NoSQL data sources.
• Experience with application monitoring, production support, incident management, and troubleshooting.
• Strong communication and collaboration skills with technical and non-technical stakeholders.

Preferred Skills

• Experience supporting GenAI/AI platforms or data portals.
• Knowledge of Azure DevOps and CI/CD pipelines.
• Experience with Azure API Management.
• Knowledge of data lakehouse architecture and Medallion Architecture.
• Experience with Agile/Scrum development methodologies.
• Familiarity with observability, logging, and cloud monitoring tools.

Thanks and regards,

Amit Lakhotia