Overview
On Site
Full Time
Skills
Databricks
Apache Hive
Data Warehouse
TBS
Extract
Transform
Load
Analytics
Machine Learning (ML)
Streaming
Workflow
API
Collaboration
Threat Analysis
Use Cases
Data Engineering
Python
Data Processing
Apache Spark
PySpark
Orchestration
Apache Airflow
Apache Kafka
Amazon S3
Storage
Messaging
Cloud Computing
Amazon Web Services
Microsoft Azure
Google Cloud
Google Cloud Platform
Regulatory Compliance
Management
Artificial Intelligence
Privacy
Job Details
Sr. Data Pipeline Engineer (AWS/Databricks)
Key Skills: experience building Data Pipelines from multiple data sources, experience with Databricks in an AWS environment, Spark/PySpark, Airflow, Kafka, Hive. Must have experience working with Large Scale Data warehouses (multiple TBs). Must have experience working with DAGs in production, must have experience working with large-scale data pipeline in a production environment
LOCAL candidates only. Face to Face Interview required.
Location: San Jose, CA (onsite 2X a week) In Person Interview Required
Duration: 12+ month contract
Requirements: Background Check
Description:
We're looking for a Data Pipeline Engineer with deep experience building and orchestrating large-scale ingestion pipelines. This role is ideal for someone who enjoys working across high-volume telemetry sources, optimizing data workflows, and solving schema drift challenges in real-world distributed environments.
You'll be part of the Security Data Platform and ML Engineering team, helping to onboard and normalize security data that powers analytics, detection, and ML workflows across the BU.
Key Responsibilities:
Design and build scalable batch and streaming data pipelines for ingesting telemetry, log, and event data
Develop and maintain orchestration workflows using tools like Apache Airflow or similar schedulers
Onboard new data sources, build connectors (API/Kafka/file-based), and normalize security-related datasets
Monitor and manage schema drift across changing source systems and formats
Implement observability into pipelines - logging, metrics, and alerts for health and performance
Optimize ingestion for performance, resilience, and cost-efficiency
Collaborate across detection, threat intel, and platform teams to align ingestion with security use cases
Required Qualifications:
5+ years of experience in data engineering or infrastructure roles focused on pipeline development
Strong experience with Python and distributed data processing tools like Apache Spark or PySpark
Hands-on experience with orchestration frameworks like Apache Airflow, Dagster, or similar
Deep understanding of ingestion best practices, schema evolution, and drift handling
Experience working with Kafka, S3, or cloud-native storage and messaging systems
Experience in cloud environments (AWS, Azure, or Google Cloud Platform)
Bonus: Familiarity with security tools (e.g., Crowdstrike, Wiz), OCSF, or compliance-related data
#techjobs
#LI-SK1
The base pay range above represents the low and high end of the base compensation range we reasonably expect to pay for this position. Actual base compensation will vary and may be above or below the range based on various factors including, but not limited to, geographic location, actual experience, and job performance. This job posting is not a promise of any specific pay for any specific employee.
The range listed is just one component of the total compensation package for our employees. Based on the details of your position, we provide a variety of benefits to our employees, including medical, dental, and vision plans, pre-tax savings plans, pre-tax parking and commuter plans, supplemental health and welfare plans, a retirement savings plan, an employee assistance program, pet insurance, and paid holidays. Other rewards may include short-term incentives and paid time off.
After you have applied, download our Staffmark Group WorkNOW App to receive real-time job offers and apply for additional opportunities. You can download it from the App Store or get it on Google Play.
About Advantage Technical
With company roots going back over 30 years, Advantage Technical is an engineering and information technology services company and a national leader in the provision of technical resources today. These services include Staff Augmentation, Direct Placement, Project Resourcing and Outsourcing - delivered from 40 key market locations, by over 3500 specialized contractors, to over 500 clients across North America. Advantage Technical is a Best of Staffing Diamond Award winner for both Clients and Talent. For more information about the industries and services offered by Advantage Technical, please visit .
Advantage Technical is committed to providing equal employment opportunity for all persons regardless of race, color, religion (including religious dress and grooming practices), sex, sexual orientation, gender, gender identity, gender expression, age, marital status, national origin, ancestry, citizenship status, pregnancy, medical condition, genetic information, mental and physical disability, political affiliation, union membership, status as a parent, military or veteran status or other non-merit based factors. We will provide reasonable accommodations throughout the application, interviewing and employment process. If you require a reasonable accommodation, contact your local branch. Advantage Technical is an E-Verify employer. This policy is applicable to all phases of the employment relationship, including hiring, transfers, promotions, training, terminations, working conditions, compensation, benefits, and other terms and conditions of employment.
All employees are directed to familiarize themselves with this policy and to act in accordance with it. All decisions with respect to employment matters and other phases of employer-temporary employee relationships will be in keeping with this policy and in accordance with all applicable laws and regulations.
To read our Privacy Notice for Candidates and Employees/Contractors, .
By applying for this job, you agree that you may receive both AI-generated and non-AI generated calls, text messages, or emails from Staffmark Group and/or its affiliates, and contracted partners. Frequency varies for text messages. Message and data rates may apply. Carriers are not liable for delayed or undelivered messages. You can reply STOP to cancel and HELP for help. You can access our general Privacy Policy at
Key Skills: experience building Data Pipelines from multiple data sources, experience with Databricks in an AWS environment, Spark/PySpark, Airflow, Kafka, Hive. Must have experience working with Large Scale Data warehouses (multiple TBs). Must have experience working with DAGs in production, must have experience working with large-scale data pipeline in a production environment
LOCAL candidates only. Face to Face Interview required.
Location: San Jose, CA (onsite 2X a week) In Person Interview Required
Duration: 12+ month contract
Requirements: Background Check
Description:
We're looking for a Data Pipeline Engineer with deep experience building and orchestrating large-scale ingestion pipelines. This role is ideal for someone who enjoys working across high-volume telemetry sources, optimizing data workflows, and solving schema drift challenges in real-world distributed environments.
You'll be part of the Security Data Platform and ML Engineering team, helping to onboard and normalize security data that powers analytics, detection, and ML workflows across the BU.
Key Responsibilities:
Design and build scalable batch and streaming data pipelines for ingesting telemetry, log, and event data
Develop and maintain orchestration workflows using tools like Apache Airflow or similar schedulers
Onboard new data sources, build connectors (API/Kafka/file-based), and normalize security-related datasets
Monitor and manage schema drift across changing source systems and formats
Implement observability into pipelines - logging, metrics, and alerts for health and performance
Optimize ingestion for performance, resilience, and cost-efficiency
Collaborate across detection, threat intel, and platform teams to align ingestion with security use cases
Required Qualifications:
5+ years of experience in data engineering or infrastructure roles focused on pipeline development
Strong experience with Python and distributed data processing tools like Apache Spark or PySpark
Hands-on experience with orchestration frameworks like Apache Airflow, Dagster, or similar
Deep understanding of ingestion best practices, schema evolution, and drift handling
Experience working with Kafka, S3, or cloud-native storage and messaging systems
Experience in cloud environments (AWS, Azure, or Google Cloud Platform)
Bonus: Familiarity with security tools (e.g., Crowdstrike, Wiz), OCSF, or compliance-related data
#techjobs
#LI-SK1
The base pay range above represents the low and high end of the base compensation range we reasonably expect to pay for this position. Actual base compensation will vary and may be above or below the range based on various factors including, but not limited to, geographic location, actual experience, and job performance. This job posting is not a promise of any specific pay for any specific employee.
The range listed is just one component of the total compensation package for our employees. Based on the details of your position, we provide a variety of benefits to our employees, including medical, dental, and vision plans, pre-tax savings plans, pre-tax parking and commuter plans, supplemental health and welfare plans, a retirement savings plan, an employee assistance program, pet insurance, and paid holidays. Other rewards may include short-term incentives and paid time off.
After you have applied, download our Staffmark Group WorkNOW App to receive real-time job offers and apply for additional opportunities. You can download it from the App Store or get it on Google Play.
About Advantage Technical
With company roots going back over 30 years, Advantage Technical is an engineering and information technology services company and a national leader in the provision of technical resources today. These services include Staff Augmentation, Direct Placement, Project Resourcing and Outsourcing - delivered from 40 key market locations, by over 3500 specialized contractors, to over 500 clients across North America. Advantage Technical is a Best of Staffing Diamond Award winner for both Clients and Talent. For more information about the industries and services offered by Advantage Technical, please visit .
Advantage Technical is committed to providing equal employment opportunity for all persons regardless of race, color, religion (including religious dress and grooming practices), sex, sexual orientation, gender, gender identity, gender expression, age, marital status, national origin, ancestry, citizenship status, pregnancy, medical condition, genetic information, mental and physical disability, political affiliation, union membership, status as a parent, military or veteran status or other non-merit based factors. We will provide reasonable accommodations throughout the application, interviewing and employment process. If you require a reasonable accommodation, contact your local branch. Advantage Technical is an E-Verify employer. This policy is applicable to all phases of the employment relationship, including hiring, transfers, promotions, training, terminations, working conditions, compensation, benefits, and other terms and conditions of employment.
All employees are directed to familiarize themselves with this policy and to act in accordance with it. All decisions with respect to employment matters and other phases of employer-temporary employee relationships will be in keeping with this policy and in accordance with all applicable laws and regulations.
To read our Privacy Notice for Candidates and Employees/Contractors, .
By applying for this job, you agree that you may receive both AI-generated and non-AI generated calls, text messages, or emails from Staffmark Group and/or its affiliates, and contracted partners. Frequency varies for text messages. Message and data rates may apply. Carriers are not liable for delayed or undelivered messages. You can reply STOP to cancel and HELP for help. You can access our general Privacy Policy at
Employers have access to artificial intelligence language tools (“AI”) that help generate and enhance job descriptions and AI may have been used to create this description. The position description has been reviewed for accuracy and Dice believes it to correctly reflect the job opportunity.