Azure Databricks Architect

Job Title: Azure Databricks Architect

Location: Remote

Job Type: Contract

Key Responsibilities:

• Solution Design & Architecture:
  Design and implement scalable and high-performance data pipelines, data lakes, and analytics platforms using Azure Databricks. Architect end-to-end solutions that integrate with Azure Data Factory, Azure Data Lake and other Azure services.
• Platform Implementation:
  Lead the deployment and optimization of Databricks clusters, ensuring they are secure, cost-efficient, and provide high availability.
• Data Integration & Transformation:
  Design and implement data transformation strategies using Apache Spark, Delta Lake, and other Databricks-native tools. Build robust data ingestion frameworks that handle structured and unstructured data sources.
• Collaboration with Teams:
  Work closely with data engineers, data scientists, and business analysts to ensure seamless implementation of solutions that align with business objectives.
• Performance Tuning & Optimization:
  Conduct performance tuning of Spark jobs and Databricks notebooks to ensure optimal performance for large-scale data processing and real-time analytics.
• Security & Governance:
  Implement best practices in data security and governance within the Azure ecosystem, ensuring compliance with relevant regulations (e.g., GDPR, HIPAA).
• Mentorship & Leadership:
  Provide guidance to junior engineers and developers, sharing expertise in Azure Databricks and data engineering best practices. Champion data architecture standards and best practices within the team.
• Documentation & Reporting:
  Develop technical documentation, architecture diagrams, and operational playbooks to facilitate smooth deployment and future maintenance.

Qualifications:

• Educational Background:
  Bachelor s degree in Computer Science, Information Technology, Data Science, or a related field. A Master s degree or relevant certifications (e.g., Azure Solutions Architect, Databricks Certified Professional) is a plus.
• Experience:
  • 5+ years of experience in data engineering, data architecture, or cloud architecture roles.
  • Strong experience in designing and implementing data pipelines, data lakes, and Azure.
• Technical Skills:
  • Strong knowledge of Azure services like Azure Data Lake and Azure Data Factory.
  • Experience with SQL, Python, and other data processing languages.
• Soft Skills:
  • Excellent problem-solving and troubleshooting abilities.
  • Strong communication and collaboration skills with both technical and non-technical stakeholders.
  • Ability to work in an agile environment and handle multiple priorities simultaneously.
  • Leadership qualities with a passion for mentoring and driving team success.