Job Description ECS is seeking a
Data Pipeline Technical Lead to work in a hybrid setting out of our
Fairfax, VA office.
ECS is seeking talented professionals to join our successful and growing team in building the next-generation Continuous Diagnostics and Mitigation (CDM) Cyber data solution. The CDM Program is the Cybersecurity and Infrastructure Security Agency's (CISA) dynamic approach to strengthening the cybersecurity of Federal networks and systems through better awareness and visibility into their security posture and cyber threats. ECS is responsible for designing, building, deploying, operating, and maintaining a complete 'Data Services' solution which includes the collection, normalization, visualization, and sharing of cyber data from more than 100 Federal agencies. The CDM Data Services product is an integrated suite of multiple Commercial Off the Shelf (COTS) products, software configuration packages, and custom code which work together to operate as an integrated solution tailored to meet Department of Homeland Security (DHS) requirements.
We are seeking professionals who thrive in a dynamic, fast-paced, and highly collaborative environment where problem-solving, critical thinking, and a holistic approach to serving the mission are key. Our program operates within the Scaled Agile Framework (SAFe). An aptitude and enthusiasm for continuous learning, improvement, and cyber security is a must!
ECS is seeking a highly skille d Data Pipeline Tech nical Lead to design, build, and maintain large-scale, real-time data pipelines that provide mission critical information for our customers . You'll be responsible for leading technical design, guiding a cross functional team of engineers, and ensuring our pipeline is resilient, scalable, and efficient.
Key Responsibilities :
- Accountable for all technical facets in the delivery of our data pipeline product.
- Lthe architecture, design , development , and delivery of our data pipeline product, which consists of a team of 20+ members .
- Build and maintain microservices and data connectors that ingest , process, and distribute data across multiple systems and APIs.
- Ensure reliability, scalability, and performance of data pipelines through best practices in resiliency , fault tolerance , and eventual consistency .
- Optimize data processing and enrichment leveraging caching layers like Redis , ElastiCache , or Valkey .
- Guide data modeling decisions and d efine and enforce data format standards ( JSON , Avro , Parquet ) , while leveraging a schema registry.
- Define technical work necessary for cross-functional team members ( DevSecOps , Site Reliability Engineering, Security, Testing, Infrastructure) to ensure successful delivery of the data pipeline product.
- Work within a SAFe Agile framework to plan work, decompose Epics, develop S tories, and task team members appropriately.
- Represent the data pipeline team on a broader level within the program, whether that is across internal teams, with our customer and other stakeholders, provide demos, deliver product status, or generally answering technical questions about the data pipeline.
- Provide mentorship to team members to strengthen their skills, knowledge, and capabilities.
Required Skills - 10+ years of hands-on technical experience as a software developer.
- 6+ years experience leading engineering teams of at least 10 members , balancing technical direction with mentorship and skills development .
- 6+ years experience focused specifically on data engineering or data pipeline solutions and concepts.
- 6 + years developing microservice based applications utilizing components such as containers, Docker, Kubernetes, AWS ECR/ECS/EKS/ Fargate , and container registries.
- 6 + years leveraging git repositories ( Github , Gitlab, Bitbucket) , with an advanced understanding of branching strategies, pull requests, commits, and other key concepts.
- Expert level Python experience, to include reviewing code commits and coaching developers on how to write better code.
- Demonstrated solution architect experience in designing highly performant, reliable, and scalable data pipeline solutions that include batch and stream processing concepts.
- Experience with stream processing technologies such as Kafka , Kafka Streams , Flink , or Spark .
- Proficiency working with diverse data formats (JSON, Avro, Parquet), and how a schema registry, schema evolution, and semantic validation can all work together to provide a mature solution.
- Experience developing custom data connectors and integrations with REST APIs.
- Expertise implementing and tuning pipeline caching layers (Redis, ElastiCache , Valkey ).
- E xperience designing and implementing data pipeline solutions that leverage cloud native services, particularly in AWS ( Step Functions, EventBridge , Lambda, SNS, SQS, S3, etc.).
- Understanding of resiliency , idempotency , and eventual consistency in distributed pipelines.
- Strong analytical and problem-solving skills, with attention to detail.
Desired Skills - Experience with IaC ( I nfrastructure as C ode ) (Terraform, CloudFormation).
- Exposure to CI/CD pipelines and data observability tools ( Elasticsearch, Prometheus, Grafana, OpenTelemetry ).
- Bachelor's degree in Computer Science , Engineering, or a related field
- Experience using ALM tools (Jira, Confluence, or similar)
- Expertise in configuration management tools (Ansible, Puppet, Chef)
- In-depth understanding of networking, security, and system administration of Linux operating systems
- Experience supporting large Federal programs ($200M+)
#ECS1
ECS is an equal opportunity employer and does not discriminate or allow discrimination on the basis any characteristic protected by law. All qualified applicants will receive consideration for employment without regard to disability, status as a protected veteran or any other status protected by applicable federal, state, or local jurisdiction law.
ECS is a leading mid-sized provider of technology services to the United States Federal Government. We are focused on people, values and purpose. Every day, our 3300+ employees focus on providing their technical talent to support the Federal Agencies and Departments of the US Government to serve, protect and defend the American People.
Never miss an opportunity! Create an alert based on the job you applied for.
