Data Engineer

Responsibilities and duties:

Data Engineering 

• Build and maintain ETL/ELT pipelines for structured and unstructured data sources. 
• Develop data-processing workloads using Azure Databricks, Python/PySpark, and SQL. 
• Optimize Databricks jobs, compute, and data layouts for performance and cost. 
• Work with Delta Lake and Unity Catalog for data storage, governance, and access control. 
• Implement data quality checks, monitoring, error handling, and recovery mechanisms. 

Cloud & Platform Engineering 

• Build, operate, and optimize Azure services including App Service, Functions, Blob/ADLS storage, Azure SQL DB for Postgres, Key Vault, Container Registry, and Azure Monitor. 
• Experience with cloud AI services, including vector search and embedding pipelines. 
• Improve platform performance, scalability, reliability, security, and cost efficiency. 
• Implement secure identity and access management using Azure managed identities, Key Vault, and secrets managed via scopes. 
• Support development, staging, and production environments. 

Automation & Operations 

• Improve platform observability through logging, monitoring, dashboards, and alerting. 
• Automate infrastructure and deployments using Infrastructure as Code (IaC) and CI/CD. 
• Troubleshoot issues across cloud services, storage, databases, networking, and data pipelines. 
• Support production incidents and root-cause analysis. 
• Collaborate with application, AI/ML, and data engineering teams. 

Requirements:

• 5+ years of experience in data engineering, cloud engineering, or a related role. 
• Hands-on experience with Microsoft Azure, Databricks, APIs, and object storage. 
• Strong skills in Python/PySpark and SQL. 
• Experience building and operating production data ETL/ELT pipelines. 
• Experience with relational and semi-structured/unstructured data and database technologies.  
• Experience with Spark, Delta Lake, and lakehouse architectures. 
• Working knowledge of Databricks Unity Catalog and data-access controls. 
• Understanding of cloud security concepts such as RBAC, managed identities, service principals, and Key Vault. 
• Experience with Git, CI/CD, monitoring, troubleshooting, and performance optimization. 

Nice to Have:

• Experience with Terraform or ARM templates. 
• Experience with Azure DevOps or GitHub Actions. 
• Familiarity with Azure networking, including VNets and private endpoints. 
• Experience with Azure Data Factory, Databricks Workflows, or Airflow. 
• Familiarity with data platforms supporting AI/ML, NLP, analytics, or GenAI applications.