Big Data Engineer

Software Guidance & Assistance, Inc., (SGA), is searching for a Big Data Engineer for a CONTRACT assignment with one of our premier Regulatory clients in the DMV or NY/NJ area.

Seeking a highly skilled and experienced Big Data Engineer to design, develop, and optimize large-scale data processing systems. In this role, you will work closely with cross-functional teams to architect data pipelines, implement data integration solutions, and ensure the performance, scalability, and reliability of big data platforms.

The ideal candidate has deep expertise in distributed systems, cloud platforms, and modern big data technologies such as Hadoop and Spark - and brings experience working alongside data science teams and workflows. You should be comfortable building the pipelines and datasets that data scientists depend on, and translating their analytical needs into scalable engineering solutions. Graph data and graph technologies are a plus.

This will require a hands-on engineer who designs, develops, and optimizes large-scale data processing systems - with experience supporting data science workflows and technologies. This is an engineering role first; data science exposure is what sets the strongest candidates apart.

Responsibilities :

• Design, develop, and maintain large-scale data processing pipelines using Big Data technologies (e.g., Hadoop, Spark, Python, Scala).
• Implement data ingestion, storage, transformation, and analysis solutions that are scalable, efficient, and reliable.
• Collaborate with cross-functional teams to understand business requirements and translate them into technical solutions.
• Optimize and enhance existing data pipelines for performance, scalability, and reliability.
• Develop automated testing frameworks and implement continuous testing for data quality assurance.
• Conduct unit, integration, and system testing to ensure the robustness and accuracy of data pipelines.
• Write and maintain automated unit, integration, and end-to-end tests.
• Monitor and troubleshoot data pipelines in production environments to identify and resolve issues.
• Work with data scientists and analysts to build the datasets, features, and pipelines that support data-driven decision-making and model development across the organization.
• Stay current with industry trends and emerging Big Data technologies to continuously improve the data architecture.

By applying for a job with SGA, you agree to allow SGA to process your application for this and future opportunities in accordance with our Privacy Policy. Also, to ensure timely processing, you agree to be contacted by our AI recruiter via email, text, or phone. Message frequency varies and data rates may apply, but you can reply STOP to any SMS message to opt-out of texts and may contact SGA at to opt-out of AI communications. The choice not to engage with AI will not adversely impact your consideration for placement. AI is not used to make any hiring determinations.

SGA is a technology and resource solutions provider driven to stand out. We are a women-owned business. Our mission: to solve big IT problems with a more personal, boutique approach. Each year, we match consultants like you to more than 1,000 engagements. When we say let's work better together, we mean it. You'll join a diverse team built on these core values: customer service, employee development, and quality and integrity in everything we do. Be yourself, love what you do and find your passion at work. Please find us at .

SGA is an Equal Opportunity Employer and does not discriminate on the basis of Race, Color, Sex, Sexual Orientation, Gender Identity, Religion, National Origin, Disability, Veteran Status, Age, Marital Status, Pregnancy, Genetic Information, or Other Legally Protected Status. We are committed to providing access, equal opportunity, and reasonable accommodation for individuals with disabilities in employment, and our services, programs, and activities. Please visit our company to request an accommodation or assistance regarding our policy.