Databricks Platform Engineer

Responsibilities
Design and build Delta Lake tables on Databricks (bronze/silver/gold medallion architecture) for financial datasets (AR, GL, invoices, collections)
Build and manage Databricks Genie Spaces — configure semantic models, write GENIE-compatible SQL, and tune NL-to-SQL behavior for finance use cases
Develop PySpark and Python-based data pipelines (Databricks Jobs + Workflows) for ERP data ingestion, transformation, and aggregation
Integrate Databricks Model Serving with LLMs (Claude, GPT-4, or custom fine-tuned models) for AI-driven insights, anomaly detection, and financial forecasting
Build RAG pipelines using Databricks Vector Search over financial documents, contracts, and GL notes
Expose Databricks SQL Warehouse endpoints to Python API services and the React frontend
Manage Unity Catalog governance — row-level security, column masking, data lineage for finance compliance
Deploy infrastructure on AWS (S3 as Databricks external storage, IAM roles, VPC networking, Secrets Manager)

Required Skills
Databricks (PySpark, Delta Lake, Jobs/Workflows, SQL Warehouses) — 3+ years
Databricks Unity Catalog and governance
Databricks Genie Spaces (semantic layer, NL analytics)
Python 3.11+ for data engineering and API integration
AWS (S3, IAM, VPC, Secrets Manager, optionally Lambda/Glue)
LLM integration: Databricks Model Serving, MLflow, or direct API calls to Claude/OpenAI
SQL (advanced — window functions, CTEs, query optimization)