Overview

We are seeking a Senior Azure Big Data Engineer to help build and support a modern enterprise data platform that enables analytics, AI, business intelligence, and digital transformation initiatives. This role is focused on designing scalable, secure, and high-performing data pipelines while integrating enterprise IT and operational technology (OT) data into a unified data platform.

The ideal candidate will have strong hands-on experience with Azure Databricks, Microsoft Fabric or Synapse, Azure Data Factory, ADLS, SQL, Python/PySpark, Spark, and enterprise data integration. Experience supporting manufacturing, industrial, chemical, automotive, or supply chain environments is highly preferred.

---

Key Responsibilities

Data Engineering & Platform Development

• Design and develop scalable batch, CDC, and streaming data pipelines.
• Build enterprise data lakes and curated data platforms.
• Develop landing, curated, and semantic data layers.
• Create reusable, high-performance data models for analytics and reporting.
• Optimize storage, partitioning, clustering, caching, and query performance.

Enterprise Data Integration

• Integrate enterprise applications including ERP, supply chain, manufacturing, laboratory, transportation, and environmental systems.
• Build reliable ingestion frameworks from multiple structured and semi-structured data sources.
• Support enterprise APIs and downstream analytics platforms.
• Enable trusted enterprise-wide data products.

Azure Data Platform

Develop solutions utilizing:

• Azure Databricks
• Microsoft Fabric
• Azure Synapse Analytics
• Azure Data Factory (ADF)
• Azure Data Lake Storage (ADLS)
• Azure SQL Database / SQL Managed Instance
• Azure Key Vault

Big Data Development

• Develop scalable Spark applications using Python and PySpark.
• Build Spark Structured Streaming pipelines.
• Optimize Spark workloads for large-scale data processing.
• Improve data processing efficiency and system performance.

Data Modeling

• Design dimensional and semantic data models.
• Implement Slowly Changing Dimensions (SCD).
• Create certified datasets for enterprise reporting.
• Support semantic models for Power BI and enterprise analytics.

Data Quality & Governance

• Implement automated data quality validation.
• Monitor freshness, completeness, schema validation, and data integrity.
• Support metadata management, lineage, and governance.
• Maintain enterprise data dictionaries and documentation.

CI/CD & DevOps

• Build automated deployment pipelines using Git-based CI/CD.
• Support testing, deployment, and release management.
• Implement monitoring, alerting, and operational runbooks.
• Optimize platform reliability and operational efficiency.

Security & Compliance

• Implement role-based security (RBAC).
• Manage secrets securely.
• Support enterprise data governance policies.
• Ensure compliance with data retention and privacy requirements.

Collaboration

• Partner with BI developers, data analysts, architects, and application teams.
• Support enterprise analytics, reporting, AI, and machine learning initiatives.
• Produce technical documentation, operational guides, and knowledge transfer materials.

---

Required Technical Skills

Azure Data Engineering

Strong hands-on experience with:

• Azure Databricks
• Microsoft Fabric or Azure Synapse
• Azure Data Factory
• Azure Data Lake Storage (ADLS)
• Azure SQL
• Azure Key Vault

Programming

• Python
• PySpark
• Spark
• Spark Structured Streaming
• Advanced SQL

Big Data

Experience building:

• Batch pipelines
• CDC pipelines
• Streaming pipelines
• Enterprise Data Lakes
• Scalable data platforms

Data Engineering

• Data modeling
• Semantic modeling
• Schema management
• Partitioning
• Performance tuning
• Data optimization

DevOps

• Git
• CI/CD pipelines
• Automated deployments
• Unit and integration testing
• Monitoring and observability

Analytics

Experience supporting:

• Power BI
• Semantic models
• Row-Level Security (RLS)
• Enterprise reporting
• API-driven analytics

Enterprise Integration

Experience integrating with enterprise platforms such as:

• SAP S/4HANA
• ERP systems
• Supply Chain systems
• Manufacturing applications
• Laboratory Information Management Systems (LIMS)
• Transportation Management Systems (TMS)
• Environmental, Health & Safety (HSE) systems

---

Preferred Skills

• Manufacturing, industrial, chemical, automotive, or process industry experience.
• SAP DataSphere knowledge.
• Operational Technology (OT) data integration.
• Historian platforms (OSI PI, Honeywell PHD, or similar).
• OPC UA and MQTT protocols.
• ISA-95 / ISA-99 fundamentals.
• Master Data Management (MDM).
• Data lineage and catalog tools.
• Great Expectations or similar data quality frameworks.
• Feature stores and metric stores.
• FinOps and cloud cost optimization.
• Lean or Six Sigma methodologies.