Azure Databricks Architect
What is in it for you?
We are looking for a hands-on Azure Databricks Architect to design, build, and optimize scalable enterprise data platforms using Azure Databricks, Azure Data Factory (ADF), dbt, Apache Iceberg, SQL, Databricks Genie, Databricks Workflows, and Databricks Compute. This is not a purely solution/architecture role. The architect will work directly with engineering teams to develop production-grade solutions, troubleshoot complex issues, perform code and design reviews, optimize workloads, and establish engineering and architecture standards.
Experience: 10+ Years
Location: Irving, TX (Hybrid 3 days WFO)
Duration: Contract role with our client
Educational Qualifications: Engineering Degree BE/ME/BTech/MTech/BSc/MSc. Technical certification in multiple technologies is desirable.
Skills:
Mandatory skills
- Architecture & Technical Leadership Design scalable Lakehouse architectures on Azure Databricks for enterprise analytics and data engineering workloads. Define architecture patterns across ingestion, transformation, storage, orchestration, governance, security, and consumption layers.
- Translate business and technical requirements into practical architecture and implementation designs. Lead architecture reviews, technical design sessions, engineering standards, and design decisions. Provide hands-on technical leadership to data engineers while remaining actively involved in development.
- Hands-on Databricks Engineering Develop and optimize production-grade data pipelines using Databricks, PySpark, and SQL. Design appropriate Databricks Compute strategies across interactive, job, and serverless workloads. Optimize clusters, Spark workloads, SQL queries, joins, partitioning, caching, and overall platform cost/performance.
- Troubleshoot complex pipeline, performance, scalability, and production issues. Establish reusable frameworks, coding standards, logging, error handling, monitoring, and deployment patterns. Data Integration & Orchestration
- Design and implement data ingestion and orchestration solutions using Azure Data Factory and Databricks Workflows. Architect batch and incremental data-processing patterns.
- Build reliable pipelines with dependency management, retries, failure handling, monitoring, and operational recovery. Define when orchestration should reside in ADF versus Databricks Workflows. DBT Design and implement scalable transformation frameworks using dbt. Build reusable dbt models, macros, tests, documentation, and dependency structures.
- Establish standards for modular SQL development, data quality, CI/CD, and reusable transformation patterns. Apache Iceberg Provide hands-on expertise with Apache Iceberg table architecture and implementation.
- Design table layouts, partitioning strategies, schema evolution, snapshot/time-travel, and lifecycle management approaches. Understand interoperability and architectural considerations across Iceberg and Databricks Lakehouse workloads.
- Troubleshoot performance and data-management issues associated with large analytical tables. SQL & Performance Engineering Demonstrate advanced hands-on SQL capability, including complex queries, CTEs, window functions, query plans, and performance tuning. Optimize large-scale transformations and analytical workloads.
- Perform root-cause analysis for data quality and performance issues. Databricks Genie & Analytics Design and enable Databricks Genie use cases for natural-language-driven analytics and business data exploration. Configure business-friendly semantic context and datasets to improve accuracy and usability.
- Partner with analytics and business teams to translate business questions into governed data products. DevOps, Governance & Engineering Excellence Implement CI/CD and automated deployment practices for Databricks and dbt workloads. Define development, testing, release, operational-support, and environment-management standards.
- Establish data quality, observability, lineage, access-control, and governance practices. Conduct hands-on code reviews and mentor engineers on Databricks, Spark, SQL, dbt, and cloud data engineering best practices. Proven experience architecting and delivering systems using agentic
- IDEs Ability to: Define architectural intent that agents can follow Break features into agent executable tasks Govern AI autonomy (guardrails, permissions, reviews)
- Integrate agentic workflows into CI/CD pipelines Experience supervising AI agents across: Multi service systems Legacy modernization Large codebases / monorepos
- Strong understanding of: Security implications of autonomous code execution Compliance, auditability, and traceability AI assisted SDLC operating models
- Guide effective use of agentic IDEs for complex, multi-module or cross-service changes Establish review practices and quality checks for AI-generated code Mentor team members on balancing autonomy, correctness, and maintainability in AI-assisted development
- Design system architectures that support AI-augmented and agentic development workflows
- Define guardrails, standards, and governance for the use of autonomous coding agents Evaluate impact of agentic IDEs on SDLC, CI/CD pipelines, security