Overview
We are seeking a Senior Azure Big Data Engineer to help build and support a modern enterprise data platform that enables analytics, AI, business intelligence, and digital transformation initiatives. This role is focused on designing scalable, secure, and high-performing data pipelines while integrating enterprise IT and operational technology (OT) data into a unified data platform.
The ideal candidate will have strong hands-on experience with Azure Databricks, Microsoft Fabric or Synapse, Azure Data Factory, ADLS, SQL, Python/PySpark, Spark, and enterprise data integration. Experience supporting manufacturing, industrial, chemical, automotive, or supply chain environments is highly preferred.
Key Responsibilities
Data Engineering & Platform Development
- Design and develop scalable batch, CDC, and streaming data pipelines.
- Build enterprise data lakes and curated data platforms.
- Develop landing, curated, and semantic data layers.
- Create reusable, high-performance data models for analytics and reporting.
- Optimize storage, partitioning, clustering, caching, and query performance.
Enterprise Data Integration
- Integrate enterprise applications including ERP, supply chain, manufacturing, laboratory, transportation, and environmental systems.
- Build reliable ingestion frameworks from multiple structured and semi-structured data sources.
- Support enterprise APIs and downstream analytics platforms.
- Enable trusted enterprise-wide data products.
Azure Data Platform
Develop solutions utilizing:
- Azure Databricks
- Microsoft Fabric
- Azure Synapse Analytics
- Azure Data Factory (ADF)
- Azure Data Lake Storage (ADLS)
- Azure SQL Database / SQL Managed Instance
- Azure Key Vault
Big Data Development
- Develop scalable Spark applications using Python and PySpark.
- Build Spark Structured Streaming pipelines.
- Optimize Spark workloads for large-scale data processing.
- Improve data processing efficiency and system performance.
Data Modeling
- Design dimensional and semantic data models.
- Implement Slowly Changing Dimensions (SCD).
- Create certified datasets for enterprise reporting.
- Support semantic models for Power BI and enterprise analytics.
Data Quality & Governance
- Implement automated data quality validation.
- Monitor freshness, completeness, schema validation, and data integrity.
- Support metadata management, lineage, and governance.
- Maintain enterprise data dictionaries and documentation.
CI/CD & DevOps
- Build automated deployment pipelines using Git-based CI/CD.
- Support testing, deployment, and release management.
- Implement monitoring, alerting, and operational runbooks.
- Optimize platform reliability and operational efficiency.
Security & Compliance
- Implement role-based security (RBAC).
- Manage secrets securely.
- Support enterprise data governance policies.
- Ensure compliance with data retention and privacy requirements.
Collaboration
- Partner with BI developers, data analysts, architects, and application teams.
- Support enterprise analytics, reporting, AI, and machine learning initiatives.
- Produce technical documentation, operational guides, and knowledge transfer materials.
Required Technical Skills
Azure Data Engineering
Strong hands-on experience with:
- Azure Databricks
- Microsoft Fabric or Azure Synapse
- Azure Data Factory
- Azure Data Lake Storage (ADLS)
- Azure SQL
- Azure Key Vault
Programming
- Python
- PySpark
- Spark
- Spark Structured Streaming
- Advanced SQL
Big Data
Experience building:
- Batch pipelines
- CDC pipelines
- Streaming pipelines
- Enterprise Data Lakes
- Scalable data platforms
Data Engineering
- Data modeling
- Semantic modeling
- Schema management
- Partitioning
- Performance tuning
- Data optimization
DevOps
- Git
- CI/CD pipelines
- Automated deployments
- Unit and integration testing
- Monitoring and observability
Analytics
Experience supporting:
- Power BI
- Semantic models
- Row-Level Security (RLS)
- Enterprise reporting
- API-driven analytics
Enterprise Integration
Experience integrating with enterprise platforms such as:
- SAP S/4HANA
- ERP systems
- Supply Chain systems
- Manufacturing applications
- Laboratory Information Management Systems (LIMS)
- Transportation Management Systems (TMS)
- Environmental, Health & Safety (HSE) systems
Preferred Skills
- Manufacturing, industrial, chemical, automotive, or process industry experience.
- SAP DataSphere knowledge.
- Operational Technology (OT) data integration.
- Historian platforms (OSI PI, Honeywell PHD, or similar).
- OPC UA and MQTT protocols.
- ISA-95 / ISA-99 fundamentals.
- Master Data Management (MDM).
- Data lineage and catalog tools.
- Great Expectations or similar data quality frameworks.
- Feature stores and metric stores.
- FinOps and cloud cost optimization.
- Lean or Six Sigma methodologies.
Never miss an opportunity! Create an alert based on the job you applied for.
