Data Engineer

Job Description

• Seeking Senior Data Engineers to design, develop, and manage complex cybersecurity data ingestion pipelines.
• Responsible for secure, scalable, and reliable data movement across 100+ existing and new security telemetry sources.
• Work with data ingestion platforms such as Cribl, Vector, and other open-source technologies.

Key Responsibilities

• Lead the architecture, design, and implementation of scalable, modular, and reusable data ingestion pipelines using Cribl, Vector, and other open-source platforms.
• Develop platform-independent ingestion frameworks and reusable templates for multiple data sources and destinations.
• Implement ingestion from Syslog, Kafka, HTTP, Azure Event Hubs, and Blob Storage.
• Integrate data with Snowflake, Splunk, Azure Data Explorer (ADX), Log Analytics, and Anvilogic.
• Implement Open Cybersecurity Schema Framework (OCSF) for schema normalization, field mapping, transformation templates, and schema validation.
• Develop custom data transformations and enrichment logic using Python, JavaScript, Groovy, or other scripting languages.
• Implement filtering, enrichment, dynamic routing, and format conversions, including JSON, CSV, XML, and Logfmt.
• Ensure secure data movement through SSL/TLS, client authentication, input validation, and logging.
• Maintain end-to-end data traceability and lineage through metadata tagging, correlation IDs, and change tracking.
• Integrate pipeline health monitoring, transformation failure logging, and anomaly detection.
• Ensure high-fidelity data delivery with minimal data loss, duplication, and transformation inconsistencies.
• Lead technical discussions to evaluate and recommend appropriate data pipeline technologies and practices.
• Maintain centralized documentation for ingestion patterns, transformation libraries, naming standards, schema definitions, and governance procedures.
• Collaborate with cybersecurity, analytics, and platform teams to support threat detection, compliance, and data analytics requirements.

Required Experience

• 10+ years of experience in cybersecurity.
• 5+ years of experience with Cribl, Vector, Datadog, Splunk, or other data pipeline platforms.
• 5+ years of experience with Python, JavaScript, Groovy, or other scripting languages.

Required Technical Skills

• Cribl Stream, Vector, Splunk, Datadog, or similar data pipeline tools.
• Open Cybersecurity Schema Framework (OCSF).
• Security telemetry ingestion and SIEM integration.
• Python, JavaScript, or Groovy scripting.
• Syslog, Kafka, HTTP, Azure Event Hubs, and Blob Storage.
• Snowflake, Splunk, ADX, Log Analytics, and Anvilogic.
• Data transformation, enrichment, filtering, and dynamic routing.
• JSON, CSV, XML, and Logfmt data formats.
• SSL/TLS, client authentication, data governance, and audit logging.