Senior Data Engineering (Onshore, Databricks)

Senior Data Engineering (Onshore, Databricks)

Location: Onsite in Richmond, VA (Relocation will work)

Role Summary

We re seeking an onshore Senior Systems Analyst (Data Engineering) to lead the productionalization of data ingestion and curation on Databricks. You will design the frameworks, controls, and processes that make data movement reliable and repeatable, create clear documentation, and guide an offshore team responsible for daytoday build and run. You ll also help assess when to use Fivetran vs. native Databricks/Unity Catalog patterns for specific use cases.

What You ll Do (Key Responsibilities)

• Productionalize existing work in Databricks: Establish standards for ingestion, orchestration, and validation so the offshore team can execute repeatably and at scale.
• Design data movement frameworks: Define reusable patterns for source onboarding, change data capture (as applicable), schema evolution, error handling, and operational runbooks.
• Implement data quality & reconciliation controls: Define sourcetotarget parity checks, row/columnlevel validations, sampling strategies, reconciliation dashboards, and exception workflows; partner with engineers to embed controls in pipelines.
• Create clear documentation: Author process docs, onboarding guides, data flow diagrams, mapping specifications, and operational SOPs to ensure consistent understanding across teams.
• Evaluate tooling choices: Recommend where Fivetran connectors make sense (e.g., rapid replication/archival) vs. when to leverage Databricks ingestion, Delta, and Catalog capabilities.
• Coordinate offshore delivery: Provide requirements, review designs, groom tasks, and ensure adherence to standards; mentor engineers and unblock execution.
• Stakeholder alignment: Work directly with product/architecture leads to confirm scope, prioritize sources, and communicate status, risks, and tradeoffs.
• Operational readiness: Define monitoring, alerting, SLAs, cost guardrails, and access patterns; ensure handoffs to run/ops are complete and auditable.

MustHave Qualifications

• 8+ years in data engineering / data platforms with strong ETL/ELT fundamentals and data movement patterns (batch/nearreal time).
• Handson Databricks experience (workspace organization, notebooks/jobs, Delta Lake fundamentals, performance/optimization basics).
• Proven experience designing validation and reconciliation controls (sourcetotarget checks, DQ rules, automated reporting).
• Demonstrated ability to author highquality documentation (frameworks, standards, SOPs, mapping/specs).
• Experience leading or enabling offshore teams (backlog grooming, code/design reviews, quality gates, knowledge transfer).
• Strong communication and stakeholder management; able to translate business needs into actionable technical work for engineering teams.

NicetoHave

• Exposure to Fivetran (connector configuration, load strategies, monitoring) not required but helpful.
• Familiarity with Unity Catalog/Databricks Catalog, Delta Live Tables, or other Databricks-native ingestion/orchestration approaches.
• Experience with Azure data services (e.g., ADLS, ADF/Synapse triggers/pipelines) in support of Databricks.
• Insurance data domain familiarity (e.g., policy, claims, billing) and legacy source onboarding.
• SQL proficiency for profiling, validation queries, and investigative analysis.

How You ll Work

• Onsite (Richmond, VA) presence for key workshops and collaboration; hybrid/flexible otherwise.
• Partner closely with onshore stakeholders for scope & prioritization; offshore team focuses on build, test, and operate under your standards and guidance.
• Deliver repeatable blueprints and clean documentation so onboarding additional sources becomes faster and safer over time.

Success Looks Like

• A welldocumented, repeatable ingestion framework on Databricks used by offshore with minimal handholding.
• Embedded DQ and reconciliation that provides confidence in sourcetotarget fidelity.
• Clear criteria for when to use Fivetran vs. Databricks-native approaches.
• Faster source onboarding, fewer defects in production, and improved transparency for stakeholders.