Apply Now

Big Data Engineer

Hybrid in Tysons, VA, US • Posted 5 hours ago • Updated 5 hours ago

Contract W2

12 Months

No Travel Required

Hybrid

Depends on Experience

Fitment

Dice Job Match Score™

⏳ Almost there, hang tight...

Job Details

Skills

Hadoop
Spark
Python
Scala

Summary

Job Responsibilities

We are seeking a highly skilled and experienced Big Data Engineer to design, develop, and optimize large-scale data processing systems. In this role, you will work closely with cross-functional teams to architect data pipelines, implement data integration solutions, and ensure the performance, scalability, and reliability of big data platforms. The ideal candidate will have deep expertise in distributed systems, cloud platforms, and modern big data technologies such as Hadoop, Spark etc.

Responsibilities:

Design, develop, and maintain large-scale data processing pipelines using Big Data technologies (e.g., Hadoop, Spark, Python, Scala).
Implement data ingestion, storage, transformation, and analysis of solutions that are scalable, efficient, and reliable.
Stay current with industry trends and emerging Big Data technologies to continuously improve the data architecture
Collaborate with cross-functional teams to understand business requirements and translate them into technical solutions.
Optimize and enhance existing data pipelines for performance, scalability, and reliability.
Develop automated testing frameworks and implement continuous testing for data quality assurance.
Conduct unit, integration, and system testing to ensure the robustness and accuracy of data pipelines.
Work with data scientists and analysts to support data-driven decision-making across the organization.
Ability to write and maintain automated unit, integration, and end-to-end tests
Monitor and troubleshoot data pipelines in production environments to identify and resolve issues.

Education/Experience Requirements:

Bachelor''s degree in Computer Science, Information Systems or related discipline with at least five (5) years of related experience, or equivalent training and/or work experience; Master''s degree and past Financial Services industry experience preferred.
Demonstrated technical expertise in Object Oriented and database technologies/concepts which resulted in deployment of enterprise quality solutions.
Past experience with developing enterprise quality solutions in an iterative or Agile environment.
Extensive knowledge of industry leading software engineering approaches including Test Automation, Build Automation and Configuration Management frameworks.
Strong written and verbal technical communication skills.
Demonstrated ability to develop effective working relationships that improved the quality of work products.
Should be well organized, thorough, and able to handle competing priorities.
Ability to maintain focus and develop proficiency in new skills rapidly.
Ability to work in a fast paced environment.
Experience with object oriented programming languages such as Java, Scala or Python.

Essential Technical Skills:

AI Tool Proficiency: Hands-on experience with AI development tools (GitHub Copilot, Q Developer, ChatGPT, Claude, etc.)
Technical Background: Strong software development background with ability to contribute to technical discussions
Agile Methodology: Extensive experience with Scrum, Kanban, and continuous improvement practices

Big Data technologies

Experience with Big data technologies such as Hadoop, Spark, Hive & Trino

Evaluate understanding of common issues like:

? Data skew and strategies to mitigate it.
? Working with massive data volumes in PetaBytes.
? Troublehsooting job failures due to resource limitations, bad data, scalability challenged.
Look for real-world debugging and mitigation stories.

AI Skills

Prompt Engineering: Proficiency in crafting effective prompts for AI coding assistants and analysis tools
AI Workflow Design: Experience redesigning development processes to leverage AI capabilities
Data Analysis: Ability to interpret AI-generated insights and translate them into actionable team improvements
Change Management: Experience leading teams through AI adoption and workflow transformation

SQL Skills (Window Functions, Joins, Complex Queries)

Assess comfort with SQL window functions, multi-table joins, aggregations.
Provide examples or ask them to write/optimize SQL queries on the spot.
Probe how they handle edge cases like NULLs, duplicates, ordering, etc.

Apache Spark (Development, Internals & Tuning)

Test their understanding of Spark’s core architecture — executors, tasks, stages, DAG.
Focus on Spark performance tuning techniques: partitioning, caching, broadcast joins, etc.
Ask scenario-based questions on troubleshooting slow running/stuck jobs or resource issues in Spark.
Explore their experience optimizing Spark jobs for large-scale datasets.

Cloud Technologies

Check exposure to AWS services like S3, EMR, Glue, Lambda, Athena, etc.
Ask how they’ve used S3 with Spark (e.g., dealing with file formats, consistency issues).
EKS, Serverless knowledge, etc.

Programming - Python or Scala

Assess ability to write clean, modular, and performant code.
Look for experience in functional programming concepts (e.g., immutability, higher-order functions).
Ask about real-world use cases where they wrote scalable data processing code.
Evaluate understanding of collections, concurrency, and memory management.

Good to have:

Experience with managing production data pipelines/ETL systems
Experience with CI/CD
Experience writing test cases
AWS certifications

Employers have access to artificial intelligence language tools (“AI”) that help generate and enhance job descriptions and AI may have been used to create this description. The position description has been reviewed for accuracy and Dice believes it to correctly reflect the job opportunity.

Dice Id: 10448332
Position Id: 9031623
Posted 5 hours ago

Contact the job poster

Ranadheer Reddy

Recruiter @ Delviom LLC

View Profile

Create job alert

Never miss an opportunity! Create an alert based on the job you applied for.

Similar Jobs

Data Engineer

Chantilly, Virginia

•

Today

Overview VTG is seeking a skilled Data Engineer with at least 5 years of experience designing, building, and maintaining scalable data pipelines and architectures. The ideal candidate will have strong expertise in data integration, ETL/ELT processes, and cloud-based data platforms, with a focus on delivering high-quality, reliable data solutions that support analytics and business decision-making. What will you do? Design, develop, and maintain scalable data pipelines and workflows (ETL/ELT)Bu

Full-time

Principal Software Engineer - Data

Remote or Bethesda, Maryland

•

Today

ZoomInfo is where careers accelerate. We move fast, think boldly, and empower you to do the best work of your life. You'll be surrounded by teammates who care deeply, challenge each other, and celebrate wins. With tools that amplify your impact and a culture that backs your ambition, you won't just contribute. You'll make things happen-fast. About the Role: We are seeking an experienced Principal Software Engineer (Data) to join our fast-paced, collaborative data team. In this role, you will h

Full-time

USD 163,800.00 - 257,400.00 per year

Data Engineer

Herndon, Virginia

•

Today

Altus Consulting is seeking a skilled Data Engineer to join our dynamic team. You will be responsible for designing, developing, and maintaining data pipelines and infrastructure that provide reliable and high-quality data for our clients' analytics needs. You will leverage your understanding of data engineering principles, best practices, and technologies to deliver efficient and scalable solutions. Responsibilities: Design and develop data pipelines using industry-standard tools and technolog

Full-time

Data Engineer

Reston, Virginia

•

Today

ICF is seeking a skilled Data Engineer to join our Corporate IT organization. You will design, build, and maintain the data infrastructure that powers insights and decision-making across our practices and support organizations. Working at the intersection of enterprise data platforms and client-facing analytics, you will have direct impact on how our firm manages and monetizes its most asset - data. This is a high-visibility role reporting to the Sr. Director of Solution Architecture. You will

Full-time

USD 98,614.00 - 167,644.00 per year

Search all similar jobs

More jobs at Delviom LLC in Tysons, VA