Azure Databricks

Responsibilities

• Data Engineering: Design and build scalable ETL pipelines using Azure Databricks.

• Data Migration: Migrate data from on-premises or legacy systems into Azure cloud environments.

• Big Data Processing: Handle large datasets using Spark, Delta Lake, and Databricks notebooks.

• Integration: Connect Databricks with Azure Data Lake, Synapse, Power BI, and other services.

• Optimization: Tune Spark jobs for performance and cost efficiency.

• Collaboration: Work with data scientists, analysts, and business stakeholders to deliver insights.

Required Skills

• Azure Databricks: Strong hands-on experience with Databricks notebooks, clusters, and Delta Lake.

• Apache Spark: Expertise in PySpark/Scala for distributed data processing.

• Azure Ecosystem: Familiarity with Azure Data Lake, Synapse Analytics, Event Hub, and Power BI.

• Programming: Proficiency in Python, Scala, or SQL.

• Cloud Architecture: Understanding of cloud-native design, CI/CD, and DevOps practices.

• Data Governance: Knowledge of security, compliance, and role-based access in Azure.

Ideal Candidate

• Experience in end-to-end data migration and transformation projects.

• Strong problem-solving and debugging skills in distributed environments.

• Ability to collaborate across technical and business teams.

• Excellent communication and documentation skills.