Senior Data Pipeline Engineer

Role : Senior Data Pipeline Engineer

• Location: Remote
• Interview Process: 2-round interview process

Skill-Set:

• Title: Senior Data Pipeline Engineer
• Top 3: Python | DataBricks | Spark
• Years of Experience: Senior level resource
• Tech Stack:
• Someone who lives in Databricks/Spark
• Understands OTEL and telemetry schemas
• Has built log/metric/trace pipelines
• Has experience with Cribl/Vector/Kafka
• Has worked with security / SIEM / SOC data
• Can automate with Terraform + CI/CD
• Can partner with security, cloud, SRE teams
• Is comfortable designing high-scale architectures

POSITION SUMMARY:

Client is seeking a highly skilled and driven individual contributor to join our enterprise observability and security engineering team. This role focuses on building, scaling, and operationalizing the enterprise Observability Lakehouse that powers threat detection, incident response, and platform visibility across hybrid cloud environments.

The ideal candidate will demonstrate deep expertise in Databricks, large-scale telemetry processing, and OTEL-aligned observability architectures. This position requires strong engineering rigor, the ability to design high-volume log, metric, and trace pipelines, and a passion for improving security and reliability through data. A critical aspect of the role includes partnering with Security Engineering, SRE, and Cloud teams to ensure telemetry is complete, trustworthy, and actionable.
What we expect of you:

• Build, scale, and maintain enterprise-grade log, metric, and trace pipelines using Databricks, cloud data lakes, and distributed data processing engines.
• Implement ingestion and transformation workflows using Cribl, Vector, GitHub Actions, Jenkins, or similar technologies.
• Design and expand an Observability Lakehouse aligned to OpenTelemetry (OTEL) data models and standards.
• Normalize and model high-volume security and observability data for detection, forensics, and operational intelligence use cases.
• Develop automated ETL/ELT frameworks, Delta Lake architectures, and data quality checks for unstructured and semi-structured telemetry.
• Collaborate closely with Security Engineering, SRE, Cloud, and SOC teams to enhance enterprise visibility and improve detection fidelity.
• Build CI/CD workflows and reusable IaC-driven patterns for pipeline deployment and automation.
• Troubleshoot performance bottlenecks and drive continuous improvements in reliability, latency, and cost efficiency.
• Contribute to team knowledge sharing and engineering standards focused on observability, security, and reliability.

REQUIRED QUALIFICATIONS

• 5+ years of experience building or supporting log, metric, or trace pipelines aligned to OTEL or similar telemetry standards in a Data, Security Data, or Observability Engineering level role.
• 5+ years of hands-on experience with Databricks, Spark, or large-scale distributed data processing systems.
• 5+ years of experience working with cloud services across AWS, Azure, or Google Cloud Platform (storage, eventing, compute, or equivalent).
• 5+ years of experience with SQL and Python in production data environments.

PREFERRED QUALIFICATIONS

• Experience with Cribl, Vector, Kafka, or similar high-volume ingestion technologies.
• Background supporting SIEM/SOAR, detection engineering, or threat analytics platforms.
• Familiarity with Delta Lake, Unity Catalog, metadata management, and lineage tooling.
• Understanding of enterprise observability platforms (Splunk, Datadog, Elastic, etc.).
• Knowledge of security governance, auditing, access controls, and sensitive-data handling.
• Experience with IaC tooling (Terraform, ARM/Bicep, CloudFormation).
• Familiarity with cloud orchestration technologies (Azure Functions, AWS Lambda, Google Cloud Platform Cloud Functions, Logic Apps, Kubernetes-based platforms).
• Strong communication skills for both deeply technical and executive audiences.
• Passion for observability, security, continuous learning, and platform-level engineering.