Azure DevOps / Data Support Engineer

Overview

We are seeking a skilled Azure DevOps / Data Support Engineer to manage and optimize data pipelines, streaming analytics, and cloud-based integration services. The ideal candidate will have hands-on experience with Microsoft Azure data services, strong troubleshooting skills, and a proactive approach to monitoring and data operations.

Key Responsibilities

• Develop, monitor, and maintain data ingestion, transformation, and integration pipelines within Azure Data Factory (ADF) and Azure Data Explorer (ADX).
• Create and optimize KQL queries for performance tuning and analytics.
• Manage Azure Blob Storage operations, including automation using AzCopy for large-scale data movement.
• Build and maintain real-time data streaming and processing solutions using Azure Stream Analytics, Azure Functions, and Kafka.
• Configure and support Logstash pipelines for data ingestion and transformation from various sources.
• Collaborate with DevOps and Data Engineering teams to ensure reliable CI/CD processes for data workloads.
• Troubleshoot production issues, implement monitoring, and support incident resolution.
• Maintain system documentation, standard operating procedures, and knowledge base articles.

Must Have Skills

• Proficiency in Microsoft Azure ecosystem especially:
• Azure Data Factory (ADF)
• Azure Data Explorer (ADX) and Kusto Query Language (KQL)
• Azure Blob Storage and AzCopy for data management
• Azure Stream Analytics, Azure Functions
• Experience with Kafka (producers, consumers, topics, partitions, etc.)
• Hands-on experience with Logstash configuration and troubleshooting.

Nice to Have

• Familiarity with Google Cloud Platform (Google Cloud Platform) and BigQuery for cross-cloud data operations.
• Exposure to Datadog for performance monitoring and observability.
• Experience with CI/CD pipelines and infrastructure as code (IaC) (e.g., Terraform, ARM templates, or Bicep).