Azure Data Engineer AI & Data Platform

Role: Azure Data Engineer AI & Data Platform

Location: USA (Remote EST OR CST)

Role Summary

We are seeking an experienced Azure Data Engineer with expertise in cloud data platforms, modern data engineering practices, and AI-driven solutions. The candidate will be responsible for designing, building, and optimizing scalable data pipelines, integrating enterprise data sources, and leveraging Azure AI and Generative AI technologies to improve data engineering productivity and business insights.

Key Responsibilities

Data Engineering & Platform Development

• Design, develop, and maintain scalable data ingestion, transformation, and ETL/ELT pipelines using Azure services.
• Build and optimize data solutions using Azure Data Factory, Azure Synapse Analytics, Microsoft Fabric, Azure Databricks, and Azure Data Lake Storage.
• Develop batch and real-time data processing solutions using Spark, SQL, and Python.
• Implement data modeling solutions aligned with enterprise standards and business requirements.
• Manage data integration across multiple source systems, APIs, databases, and cloud platforms.

AI & Generative AI Enablement

• Develop AI-assisted solutions for automated data pipeline generation, code acceleration, and notebook creation.
• Integrate Azure OpenAI and Azure AI Services into data engineering workflows.
• Build intelligent data quality monitoring, anomaly detection, and predictive analytics solutions.
• Design and implement AI-powered agents for operational monitoring, user notifications, and workflow automation.
• Evaluate and implement emerging AI technologies to improve engineering efficiency and business outcomes.

Data Governance & Quality

• Implement data quality frameworks, validation rules, and monitoring mechanisms.
• Ensure compliance with security, privacy, and governance requirements.
• Establish metadata management, lineage tracking, and data cataloging processes.
• Support audit, compliance, and regulatory reporting requirements.

Operational Support

• Monitor and troubleshoot data pipelines and platform issues.
• Perform root cause analysis and implement preventive measures.
• Support production deployments, release management, and operational excellence initiatives.
• Collaborate with cross-functional teams to resolve data and platform issues.

Required Skills

• SQL
• Azure Data Factory (ADF)
• Azure Synapse Analytics
• Microsoft Fabric
• Azure Databricks
• Azure Data Lake Storage (ADLS)
• SQL, Python, PySpark
• Data Modeling and Data Warehousing
• Azure DevOps (ADO), CI/CD
• Git Version Control
• REST APIs and Data Integration

AI & Advanced Skills

• Azure OpenAI Service
• Azure AI Services
• Generative AI and Large Language Models (LLMs)
• AI Agents and Workflow Automation
• Retrieval-Augmented Generation (RAG)
• Prompt Engineering
• Intelligent Monitoring and AI-driven Operations

Key Competencies

• Problem Solving & Analytical Thinking
• Stakeholder Management
• Collaboration & Communication
• Innovation & Continuous Improvement
• Operational Excellence
• AI Adoption & Automation Mindset