Sr Databricks Data Engineer---W2---Remote

Role: Senior Data Engineer Databricks

Location: Breezy Point, NY (Remote)

Duration: Contract

W2 Only

Responsibilities:

• Legacy Migration & Parity Governance: Lead the migration of structured finance datasets from legacy proprietary codebases to Databricks. Methodically extract undocumented business rules and design audit-ready parity validation frameworks for cutover.
• Data Modeling & Architecture Ownership: Design target data structures for complex financial instruments, handling point-in-time accuracy, as-of joins, bitemporal history, late-arriving data, and corporate action adjustments.
• Pipeline Engineering: Build and operate scalable ingest, transformation, and publishing layers. Establish robust data quality gates, reconciliation frameworks, and lineage tracking suited for a regulated environment.
• Performance & Cost Optimization: Optimize cluster configurations, job orchestration, query strategies, and storage layouts. Drive measurable reductions in execution times and compute costs.
• Stakeholder Alignment & Documentation: Partner directly with quantitative, risk, and product teams. Articulate architectural trade-offs ("why option A was chosen over option B") and author durable technical specifications.

Required Qualifications:

• 8+ Years in Data Engineering: Proven track record of architectural ownership and end-to-end system design.
• Production Databricks Depth: Extensive experience with Delta Lake, Spark (PySpark & Spark SQL), Unity Catalog, job orchestration, and performance tuning (partitioning, clustering, Z-ordering, file compaction).
• Financial Time-Series Domain Expertise: Strong understanding of structured finance instruments (ABS/MBS, loan-level data, cash flow analytics) and time-series correctness (bitemporal processing, point-in-time alignment, late-arriving data).
• Legacy Code Discovery: Demonstrated ability to read unfamiliar, monolithic production codebases (e.g., legacy SQL/stored procedures, C++, Java, or custom scripts) to recover underlying business logic without relying on specs.
• Demonstrated Impact: Applicants must be able to quantify the scale and business outcomes of their past initiatives (e.g., performance gains, cost reductions, data parity metrics, scale handled).
• Languages & Tooling: Advanced Python, SQL, shell scripting, and CI/CD-driven infrastructure delivery (e.g., Git, Terraform).

Strongly Preferred Qualifications:

• Conference Presentation or Published Work: Prior speaking engagement at major industry events (e.g., Databricks Data + AI Summit, AWS re:Invent, Snowflake Summit, FINOS, or regional Data Engineering conferences) OR published technical blogs/whitepapers on complex data engineering or financial modeling.
• Legacy Migration Cutover Experience: Proven history of taking a legacy platform completely off-line through rigorous parity testing and controlled cutover.
• Governance & Cost Control: Hands-on execution with Delta Live Tables, Unity Catalog governance, and FinOps/cost management.

Engagement Details:

• Duration: 6 months, renewable based on performance and client need.
• Work arrangement: Remote within the continental United States. Candidates based in or near New York City are strongly preferred for periodic on-site collaboration with the client team.
• Hours: Core overlap with U.S. Eastern business hours required.
• Work authorization: Candidates must be authorized to work in the United States for the full contract term. Candidates whose authorization depends on employer sponsorship should be aware that international travel during the engagement may create re-entry delays that disrupt project continuity, and should plan accordingly.