Data Pipeline & Connectivity Engineer (Databricks)

What You’ll Own

Connectivity & Data Ingestion

• Design and implement robust ingestion pipelines across highly fragmented environments (multiple ERPs, isolated networks, legacy systems)
• Build connectivity across:
• ERP systems (e.g., Great Plains, SAP, Infor, custom systems)
• APIs, flat files, streaming sources, and third-party platforms
• Solve for network isolation challenges and inconsistent data access patterns

 Databricks Pipeline Engineering

• Build and optimize Databricks-based pipelines using:
• PySpark
• Delta Lake
• Structured Streaming / batch frameworks
• Implement scalable ingestion patterns aligned to medallion architecture (Bronze ? Silver ? Gold)
• Ensure data quality, reliability, and performance across pipelines

Platform & Architecture Contribution

• Partner with Data Architect / Leadership to:
• Define ingestion frameworks and reusable pipeline patterns
• Standardize data models across business units
• Enable scalable onboarding of newly acquired companies

Data Product Enablement

• Deliver clean, reliable datasets that power:
• Operational reporting
• Power BI / visualization layers
• AI / ML and LLM-based use cases

Governance, Quality & Reliability

• Implement:
• Data validation and monitoring
• Logging and observability
• Secure and compliant data movement

What You Bring

Must-Have Experience

• 5–10+ years in data engineering / pipeline engineering
• Deep hands-on experience with:
• Databricks (required)
• PySpark
• Delta Lake
• Proven experience building:
• Data pipelines across multiple disconnected systems
• Scalable ingestion frameworks

Strongly Preferred

• Experience in complex, multi-entity environments (PE-backed, M&A, roll-ups)
• ERP data integration experience:
• Great Plains, SAP, Infor, NetSuite, etc.
• Experience with:
• AWS or Azure data ecosystems
• API integrations and event-based pipelines
• Data orchestration tools