Position - Azure Databricks Architect
Location: Houston TX- Hybrid
Level: 5
Exp: 10+ Years
Core Responsibilities
Design and implement scalable data platforms and pipelines using Azure Databricks, Apache Spark, Pyspark, Delta Lake, and MLflow.
Design and implement data pipelines using Databricks, Spark, and Delta Lake for batch and streaming data.
Optimize Spark jobs for performance, scalability, and cost efficiency.
Develop and maintain Lakehouse architecture leveraging Databricks and cloud storage solutions.
Lead the migration from legacy platforms to Lakehouse architecture.
Develop batch and streaming data pipelines for ingestion, transformation, and analytics.
Establish standards for data governance, quality, and security.
Collaborate with stakeholders to align architecture with business goals.
Mentor data engineers and developers on Databricks best practices.
Integrate Databricks with tools like Power BI, Tableau, Kafka, Snowflake, and Azure Data Factory.
Required Skills & Experience
8 10+ years in data engineering or architecture roles.
8+ years of hands-on experience with Databricks and Azure.
Strong command of SQL, Python, Scala, and Spark.
Experience with CI/CD pipelines, DevOps, and orchestration tools like Airflow or Data Factory.
Familiarity with Azure cloud platforms.
Deep understanding of distributed computing, performance tuning, and data security.
Preferred Qualifications
Bachelor's or Master's in Computer Science, Data Science, or related field.
Certifications such as Azure Solution Architect, Azure Data Engineer, or Databricks Certified Data Engineer or Solutions Architect.
Experience with data mesh, data fabric, or enterprise data architectures.
Domain knowledge in finance, healthcare, or public sector is a plus.
Hadoop & Scala experience
Additional Insights from Internal Communications
Emphasis on DataOps, data orchestration, and data modeling.
Strong ERP knowledge (e.g., SAP, Salesforce) is valued.
Candidates should be hands-on, capable of self-exploration, and able to drive analytics adoption across business teams.
A structured questionnaire is often used to assess candidates on areas like Unity Catalog, Spark optimization, partitioning techniques, and data security.
Never miss an opportunity! Create an alert based on the job you applied for.
