Overview
Remote
Depends on Experience
Contract - W2
Contract - 12 Month(s)
Skills
data engineer
ETL
databricks
spark
python
telemetry
OTEL
Job Details
ONLY W2 ONLY W2 ONLY W2 ONLY W2 ONLY W2 ONLY W2 ONLY W2 ONLY W2 ONLY W2 ONLY W2 ONLY W2 ONLY W2 ONLY W2 ONLY W2 ONLY W2 ONLY W2 ONLY W2 ONLY W2 ONLY W2 ONLY W2 ONLY W2
Role: Data Pipeline Engineer
Long term
Remote
W2 Only
Skill-Set:
- Title: Senior Data Pipeline Engineer
- Top 3: Python | DataBricks | Spark
- Years of Experience: Senior level resource
- Tech Stack:
- Someone who lives in Databricks/Spark
- Understands OTEL and telemetry schemas
- Has built log/metric/trace pipelines
- Has experience with Cribl/Vector/Kafka
- Has worked with security / SIEM / SOC data
- Can automate with Terraform + CI/CD
- Can partner with security, cloud, SRE teams
- Is comfortable designing high-scale architectures
Role In the Managers Words:
- Job description has the skill-set is laid out there open to taking someone who has stronger depth in any one area - wouldn't mind having - Python is the non-negotiable
- Some experience in distributed systems
- Some experience in modeling
- Core data engineering, but at a senior level.
- Executive tasks with minimal oversight
POSITION SUMMARY
Client is seeking a highly skilled and driven individual contributor to join our enterprise observability and security engineering team. This role focuses on building, scaling, and operationalizing the enterprise Observability Lakehouse that powers threat detection, incident response, and platform visibility across hybrid cloud environments.
The ideal candidate will demonstrate deep expertise in Databricks, large-scale telemetry processing, and OTEL-aligned observability architectures. This position requires strong engineering rigor, the ability to design high-volume log, metric, and trace pipelines, and a passion for improving security and reliability through data. A critical aspect of the role includes partnering with Security Engineering, SRE, and Cloud teams to ensure telemetry is complete, trustworthy, and actionable.
What we expect of you:
- Build, scale, and maintain enterprise-grade log, metric, and trace pipelines using Databricks, cloud data lakes, and distributed data processing engines.
- Implement ingestion and transformation workflows using Cribl, Vector, GitHub Actions, Jenkins, or similar technologies.
- Design and expand an Observability Lakehouse aligned to OpenTelemetry (OTEL) data models and standards.
- Normalize and model high-volume security and observability data for detection, forensics, and operational intelligence use cases.
- Develop automated ETL/ELT frameworks, Delta Lake architectures, and data quality checks for unstructured and semi-structured telemetry.
- Collaborate closely with Security Engineering, SRE, Cloud, and SOC teams to enhance enterprise visibility and improve detection fidelity.
- Build CI/CD workflows and reusable IaC-driven patterns for pipeline deployment and automation.
- Troubleshoot performance bottlenecks and drive continuous improvements in reliability, latency, and cost efficiency.
- Contribute to team knowledge sharing and engineering standards focused on observability, security, and reliability.
REQUIRED QUALIFICATIONS
- 5+ years of experience building or supporting log, metric, or trace pipelines aligned to OTEL or similar telemetry standards in a Data, Security Data, or Observability Engineering level role.
- 5+ years of hands-on experience with Databricks, Spark, or large-scale distributed data processing systems.
- 5+ years of experience working with cloud services across AWS, Azure, or Google Cloud Platform (storage, eventing, compute, or equivalent).
- 5+ years of experience with SQL and Python in production data environments.
PREFERRED QUALIFICATIONS
- Experience with Cribl, Vector, Kafka, or similar high-volume ingestion technologies.
- Background supporting SIEM/SOAR, detection engineering, or threat analytics platforms.
- Familiarity with Delta Lake, Unity Catalog, metadata management, and lineage tooling.
- Understanding of enterprise observability platforms (Splunk, Datadog, Elastic, etc.).
- Knowledge of security governance, auditing, access controls, and sensitive-data handling.
- Experience with IaC tooling (Terraform, ARM/Bicep, CloudFormation).
- Familiarity with cloud orchestration technologies (Azure Functions, AWS Lambda, Google Cloud Platform Cloud Functions, Logic Apps, Kubernetes-based platforms).
- Strong communication skills for both deeply technical and executive audiences.
- Passion for observability, security, continuous learning, and platform-level engineering.
Employers have access to artificial intelligence language tools (“AI”) that help generate and enhance job descriptions and AI may have been used to create this description. The position description has been reviewed for accuracy and Dice believes it to correctly reflect the job opportunity.