Databricks Engineer

We are seeking a Senior Databricks Engineer with 8+ years of experience to design, build, and optimize enterprise-grade data solutions on Databricks. In this role, you will work with broad latitude and independence, translating complex business objectives into scalable, cost-effective data architecture. You will bridge the gap between technical requirements and business needs—conducting feasibility studies, analyzing user workflows, and delivering high-performance lakehouse analytics.

Key Responsibilities

• Pipeline & Lakehouse Architecture: Design, build, and maintain scalable ETL/ELT ingestion pipelines utilizing Lakeflow Declarative Pipelines (DLT), PySpark/Scala, and the Medallion Architecture (Bronze, Silver, Gold).

• Business Analysis & Data Modeling: Collaborate with stakeholders to capture requirements, perform cost/benefit analyses, and build robust dimensional models (Star/Snowflake schemas).

• Analytics & Visualization: Develop dynamic native Databricks SQL dashboards, Databricks Apps, and custom analytical solutions to surface actionable insights.

• Governance & Operations: Implement data quality, security, and governance frameworks. Optimize workload performance, monitor compute efficiency, and manage pipeline orchestration via Lakeflow Jobs or Airflow.

Candidate Qualifications

Required Experience (8+ Years)

• Databricks & Spark: Hands-on experience architecting and tuning ETL/ELT data pipelines on Databricks using Apache Spark (PySpark or Scala).

• Lakehouse Stack: Deep knowledge of Delta Lake, Medallion architecture, and Lakeflow Declarative Pipelines (DLT).

• Data Modeling & SQL: Strong proficiency in SQL and dimensional data modeling (Star/Snowflake schemas).

• Analytics & Governance: Experience developing native Databricks dashboards/apps and establishing data security, quality, and governance practices.

• Orchestration: Experience scheduling and managing offline jobs using Lakeflow Jobs (Databricks Workflows) or similar tools (e.g., Airflow).

• Communication: Exceptional verbal and written communication skills with a proven ability to translate complex data concepts for business leadership.

Preferred Qualifications

• Active Databricks Certification (Data Engineer Associate or Professional).

• Experience with CI/CD practices for data pipelines (Git-based workflows, DevOps).

• Prior experience in public sector or state government environments.