Apply Now

Databricks Architect

Remote • Posted 5 hours ago • Updated 4 hours ago

Full Time

Remote

Depends on Experience

Fitment

Dice Job Match Score™

📊 Calculating match score...

Job Details

Skills

Databricks

Summary

CEI is seeking a Databricks Architect to join our growing organization!

Client/Industry: CEI – AI / IT Solutions/Consulting and Professional Services
Job Title: Databricks Technical Architect
Location: Hybrid in Pittsburgh, PA 15205 (3 Days On-Site / 2 Days Remote) | OR | Remote (1^st pref-EST; 2^nd pref-CST)
Work Schedule/Shift: Monday to Friday, Standard business hours | Minimum expectation of ~40 work hours per week.
Duration/Length of Assignment: W2 Direct Hire / Full Time
Compensation: Competitive pay (Salary) | Benefits available (optional) [Medical, Dental, Vision, and 401(k) match]

Position Overview:
This role supports CEI’s continued evolution into an AI-first engineering and transformation company by building and scaling enterprise-grade data platforms that enable analytics, reporting, and AI/ML initiatives. The position sits within CEI’s Solutions and Engineering organization, working alongside data engineering, analytics, governance, and business teams to deliver a modern Lakehouse platform. The individual will act as a senior technical leader, partnering across multiple stakeholders while guiding a team of data engineers and contributing directly to architectural decisions and implementation. The role operates within a collaborative, cross-functional environment where engineering standards, governance, and platform scalability are core priorities. At a high level, this individual will design and implement end-to-end data solutions, establish platform standards, and ensure the data ecosystem supports both analytical and AI-driven use cases. The expectation is to balance architecture ownership with hands-on development, contributing directly to platform buildout, optimization, and governance. This role requires consistent collaboration with internal teams, active participation in architecture discussions, and ongoing mentorship of engineers while ensuring delivery of reliable, scalable, and cost-efficient data solutions.

Required Skills/Experience/Qualifications:

8–12+ years of experience in data engineering, data architecture, or related roles
3+ years of hands-on production experience with Databricks
Strong expertise in Apache Spark using PySpark and Spark SQL
Experience designing and implementing Lakehouse architectures using Delta Lake and medallion (Bronze/Silver/Gold) patterns, with a strong grasp of partitioning, Z-ordering, liquid clustering, and Delta optimization techniques
Experience with Unity Catalog including governance design, access control, and data lineage
Implementation experience on at least one major cloud platform: AWS, Azure, or GCP
Experience implementing infrastructure-as-code using Terraform and/or Databricks Asset Bundles
Experience building and managing CI/CD pipelines for data engineering workflows (Git-based, with environment promotion)
Strong understanding of data modeling approaches including dimensional modeling and Data Vault
Experience with ETL/ELT patterns and distributed data systems
Experience with cost optimization and performance tuning within Databricks environments at scale
Understanding of data governance, security, and compliance principles and ability to implement them within platform design

Preferred Skills (Not Required):

Scala is a plus but not required
Databricks certifications such as Data Engineer Professional or Solution Architect
Experience with Structured Streaming and event-driven architectures (Kafka, Event Hubs, Kinesis, Pub/Sub)
Experience with ingestion frameworks such as Lakeflow Connect
Familiarity with MLflow, feature stores, and ML platform integrations
Experience supporting BI tools such as Power BI or Tableau at scale, including semantic modeling
Exposure to open table formats such as Iceberg or Delta UniForm and Delta Sharing.
Experience working within regulated industries such as healthcare, financial services, or public sector

Day to Day/Responsibilities:

Architect end-to-end data solutions within Databricks, spanning ingestion, transformation, governance, and serving layers using the medallion (Bronze/Silver/Gold) architecture pattern on Delta Lake
Lead hands-on development of production data pipelines using PySpark and Spark SQL, with Scala leveraged where appropriate, ensuring code quality, scalability, and performance across workloads
Perform code reviews and establish engineering standards through reusable frameworks and reference implementations to ensure consistency across the platform
Own and design Unity Catalog structures, including catalog and schema organization, access controls, row- and column-level security, data classification, lineage tracking, and audit capabilities
Partner directly with data governance teams to translate business policies into enforceable platform-level controls within Databricks
Define and implement the data serving layer strategy using Databricks SQL, including materialized views, performance tuning, and enabling consumption through BI tools such as Power BI and Tableau, including semantic modeling
Establish platform-as-code practices using Terraform and/or Databricks Asset Bundles, ensuring consistent infrastructure deployment and environment standardization
Design and operate CI/CD pipelines for notebooks, jobs, and infrastructure using Git-based workflows with environment promotion, supporting controlled releases across development, testing, and production environments
Define orchestration strategies using Databricks Workflows and Lakeflow Declarative Pipelines, integrating with tools such as Airflow when cross-platform orchestration is required
Own cost and performance governance across the platform, including cluster policies, serverless versus classic compute decisions, Photon usage, job versus all-purpose cluster strategy, and DBU budgeting/chargeback models
Collaborate with AI and ML teams to support platform integration with MLflow, feature engineering on Delta Lake, and emerging Databricks AI capabilities such as vector search and Mosaic, ensuring the platform supports model development and deployment
Provide technical leadership and mentorship to data engineers, acting as the primary Databricks subject matter expert across architecture discussions and cross-functional forums

About Us: CEI is an AI-first engineering and transformation company that designs, builds, and operationalizes production-grade AI and data solutions across enterprise environments. Originally established as an IT consulting and staffing firm, CEI has evolved into a technology partner focused on delivering enterprise-scale AI transformation, modern data platforms, and application development solutions. We work across industries including healthcare, financial services, utilities, retail, and manufacturing, supporting both mid-market and Fortune 1000 clients. CEI combines consulting, engineering, and delivery capabilities to not only define strategies but also build and implement solutions at scale. With a focus on integrating AI into core business operations, CEI enables organizations to modernize their technology landscape, improve decision-making, and generate measurable outcomes through advanced data and AI capabilities.

#INDGEN #ZR

Employers have access to artificial intelligence language tools (“AI”) that help generate and enhance job descriptions and AI may have been used to create this description. The position description has been reviewed for accuracy and Dice believes it to correctly reflect the job opportunity.

Dice Id: ceiam
Position Id: 31713
Posted 5 hours ago

Create job alert

Never miss an opportunity! Create an alert based on the job you applied for.

Similar Jobs

Databricks Tech Lead

Remote

•

Today

Hello; Role: Databricks Tech Lead Location: Remote Duration: Long Term We are looking for an experienced Databricks Tech Lead/Architect to serve as a trusted technical partner for our customers. This is a hands-on leadership role. Strong communication and relationship-building skills are just as important as deep Databricks expertise. What You Bring 12 14+ years of hands-on experience in data engineering, data architecture, or a closely related field. Deep expertise in Databricks Delta Lake,

Easy Apply

Full-time, Contract

Databricks Engineer

Remote

•

6d ago

Role: Databricks EngineerLocation:REMOTEDuration: Contract to hire after 6-month termWe are looking for a hands-on Databricks Engineer with strong AWS experience to design, build, and optimize scalable data pipelines and lakehouse solutions. The role focuses on implementing robust batch and streaming data solutions using Databricks, Delta Lake, and AWS cloud-native services, ensuring high performance, scalability, and security.Key Responsibilities Build and maintain end-to-end data pipelines us

Easy Apply

Full-time

65 - 75

Databricks Architect

Remote

•

Today

Job Title: Databricks Architect Location: Remote Job Type: Full Time Role & Responsibilities Overview: Develop and optimize ETL pipelines from various data sources using Databricks on cloud (AWS, Azure, etc.) Experienced in implementing standardized pipelines with automated testing, Airflow scheduling, Azure DevOps for CI/CD, Terraform for infrastructure as code, and Splunk for monitoring Continuously improve systems through performance enhancements and cost reductions in compute and storage Dat

Easy Apply

Full-time

$$150k

Azure Data Engineer (Databricks)

Remote

•

Today

Job Title:Azure Data Engineer DatabricksLocation:RemoteJob Type:FulltimeJob Description: Must Have Technical/Functional Skills Design, build, and orchestrate ETL/ELT pipelines using Azure Databricks.Implement batch data ingestion and transformations using PySpark and Spark SQL.Architect and maintain Lakehouse and analytical warehouse models (fact and dimension schemas) leveraging Delta Lake.Ensure data quality, reliability, lineage, and governance across the data platform.Collaborate with securi

Easy Apply

Full-time

100,000 - 140,000

Search all similar jobs