Senior Data Engineer

Software Guidance & Assistance, Inc., (SGA), is searching for an Senior Data Engineer for a FULL TIME assignment with one of our premier Data and Analytics client in their D.C office. This is a remote opportunity for candidates on the East Coast, preferably close to the DC Metro area.

Role Summary
We're looking for a Senior Data Engineer who combines deep technical expertise with strong communication skills and a proactive mindset. You will design, build, and optimize complex data pipelines in a cost-conscious environment while actively collaborating with your squad. This role is ideal for engineers who enjoy sharing their work, navigating ambiguity, and understand that "it runs" is not the same as "it runs efficiently." Success here means taking ownership, balancing innovation with operational excellence, and proactively seeking out opportunities for optimization.

Key Responsibilities

• Design and develop high-performance data pipelines using Scala/Spark on AWS EMR.
• Build new features and data products while maintaining operational excellence.
• Optimize data pipelines for both performance and cost efficiency.
• Debug complex distributed system issues using appropriate tools and methodologies.
• Implement comprehensive monitoring and observability from day one.
• Write efficient Snowflake/Snowpark procedures, working across multiple languages as needed.
• Balance net-new development with continuous improvement of existing systems.

Qualifications & Experience
Technical Requirements

• 5+ years of hands-on experience with Apache Spark using Scala as the primary language (not PySpark-focused).
• Proven track record of optimizing large-scale data pipelines for performance and cost.
• Strong AWS EMR experience, including fleet management and instance optimization.
• Proficiency with AWS Step Functions.
• Deep understanding of distributed computing principles and resource management.
• Experience debugging and tuning multi-terabyte daily workloads.
• Comfort working across Scala, Python, and SQL as needed.
• Experience with probabilistic data structures for high-cardinality data processing.

Critical Skills

• Advanced troubleshooting abilities in distributed systems.
• Strong understanding of data skew and mitigation strategies.
• Metrics-first mindset: measuring before and after optimizations.
• Root cause analysis expertise, focused on preventing repeat issues.

Mindset & Working Style

• Cost-conscious, treating company resources as your own.
• Strong balance between innovation and operational excellence.
• Proactive in optimization-seeing inefficiencies as opportunities.
• Communicates early and often, especially around risks and blockers.
• Able to interpret needs beyond strictly stated requirements.

Preferred Qualifications

• Experience designing large-scale, observable, and maintainable data systems.
• Proven ability to balance building new capabilities with improving existing systems.
• Strong track record of mentoring and knowledge sharing within engineering teams.
• Background in cost/performance trade-off decision-making.
• Passion for environments where initiative, ownership, and collaboration are rewarded.

SGA is a technology and resource solutions provider driven to stand out. We are a women-owned business. Our mission: to solve big IT problems with a more personal, boutique approach. Each year, we match consultants like you to more than 1,000 engagements. When we say let's work better together, we mean it. You'll join a diverse team built on these core values: customer service, employee development, and quality and integrity in everything we do. Be yourself, love what you do and find your passion at work. Please find us at .

SGA is an Equal Opportunity Employer and does not discriminate on the basis of Race, Color, Sex, Sexual Orientation, Gender Identity, Religion, National Origin, Disability, Veteran Status, Age, Marital Status, Pregnancy, Genetic Information, or Other Legally Protected Status. We are committed to providing access, equal opportunity, and reasonable accommodation for individuals with disabilities in employment, and our services, programs, and activities. Please visit our company to request an accommodation or assistance regarding our policy.