Databricks Architect

Key Responsibilities

• Design and implement end-to-end data architecture on Databricks (Spark, Delta Lake, MLflow).

• Develop and optimize large-scale ETL/ELT pipelines using PySpark/SQL.

• Architect data lakes and lakehouses integrating cloud storage systems (e.g., ADLS, S3, GCS).

• Define and enforce best practices around security, data governance, and cost optimization.

• Lead technical workshops and collaborate with data engineers, data scientists, and DevOps.

• Implement data orchestration workflows (e.g., Databricks Workflows, Airflow, ADF).

• Integrate Databricks with third-party BI tools (Power BI, Tableau) and data catalogs (Unity Catalog, Hive Metastore).

• Provide architectural recommendations during pre-sales or stakeholder discussions (if consulting).

Required Skills and Qualifications

• Strong experience with Databricks platform and Apache Spark.

• Deep knowledge of Delta Lake, Unity Catalog, and Databricks Workflows.

• Proficient in Python, SQL, and optionally Scala.

• Hands-on with cloud platforms: Azure, AWS, or Google Cloud Platform.

• Experience in data security, IAM, RBAC, and audit logging.

• Proven expertise in designing data lakehouses, ETL/ELT workflows, and data modeling.

• Experience with CI/CD and infrastructure as code (Terraform, GitOps).

• Strong understanding of performance tuning and cost management.

• Excellent communication and leadership skills.

Preferred Qualifications

• Databricks Certified Data Engineer Professional or Architect-level certification.

• Cloud certifications (e.g., Azure Solutions Architect, AWS Certified Data Analytics).

• Experience with ML/AI on Databricks (MLflow, Feature Store, AutoML).

• Exposure to data mesh, data products, or modern data stack concepts.