Senior Data Engineer

Spark, Scala, AWS, EMR, EC2, S3, RedShift, Data Pipeline, Airflow, Zookeeper, Kafka, Kenesis, Aethna, Glue
Full Time
$40,000 - $120,000
Work from home not available Travel not required

Job Description

Position Title: Senior Data Engineer


Location: Sterling , VA


Duration: FTE


Key Areas of Responsibility


  • Lead the design, implementation, and continuous delivery of pipelines using distributed AWS based big data technologies supporting data processing initiatives across batch and streaming datasets
  • Responsible for development using Scala , Python languages and Big Data Frameworks such as Spark, EMR, Presto, AWS Athena, Kafka, Zepplin , and Kinesis
  • Provide administrative support on deployed AWS platform components
  • Identify, evaluate and implement cutting edge big data pipelines and frameworks required to provide requested capabilities to integrate external data sources and APIs
  • Review, analyse and evaluate market requirements, business requirements and project briefs in order to design the most appropriate end-to-end technology solutions
  • Process and manage high volume real time customer interaction streams
  • Provide architectural support by building Proof of Concepts & Prototypes
  • Self-Starter to deliver data engineering solutions to optimize both the cost and existing solution
  • Stay current with emerging technologies and industry trends

Required Qualifications


  • Bachelor's Degree or higher in Computer Sciences or similar
  • Minimum of 5-6 years Software Industry experience
  • 3+ years of development experience with AWS services Must have EC2, EMR, RedShift, Data Pipeline or Airflow, S3, Cloud Formation and CLI (must to have) and Jenkins
  • 3+ years of development experience with Apache Spark, Presto, SQL, notebook and NoSQL Implementation
  • 4+ years of extensive working knowledge in different programming Scala (Must), Shell and Python (Must).
  • Proficiency working with structured, semi-structured and unstructured data sets including social, web logs and real time streaming data feeds
  • Able to tune Big Data solutions to improve performance and end-user experience
  • Knowledge on Visualization and Data Science Tools.
  • Expert level usage with Jenkins, GitHub is preferred
  • Spark developer certification is a plus
  • Ability and eagerness to constantly learn and teach others
  • Experience in the media industry is a plus
Dice Id : 90979346
Position Id : 5989367
Have a Job? Post it

Similar Positions

Top Secret Data Engineer/Scientist
  • Take2 Consulting
  • Chantilly, VA
Data Software Engineer
  • Mclean Intelligence Workforce
  • Mclean, VA
AWS / Big Data tech lead Architect
  • Wonese
  • Washington D.c., DC
Big Data Engineering Manager
  • The Silicon Partners Inc.
  • Sterling, VA
Big Data AWS Migration Lead
  • CompuGain Corporation
  • Mclean, VA
Senior AWS Cloud Solutions Architect
  • Hitek Staffing
  • Fort Belvoir, VA
Senior Solution Architect
  • Aquinas Consulting
  • Bethesda, MD
Senior Solution Architect
  • Distributed Solutions, Inc
  • Reston, VA
AWS Software Engineer C#, Java or Cloud
  • Personified
  • Herndon, VA
Big Data Analytics Cloud Solutions Architect
  • Request Technology, LLC
  • Mclean, VA
Big Data Developer (9)
  • Pyramid Consulting, Inc.
  • Mclean, VA
Enterprise Cloud Architect
  • Devis (Development InfoStructure Inc.)
  • Arlington, VA
Big Data Engineer/Data Analyst
  • SRS Consulting Inc
  • Chevy Chase, MD
Senior DevOps Engineer
  • Rite Software Solutions & Services LLC
  • Herndon, VA