Data Engineer

job summary:

This role is highly critical because the individual will essentially own and manage the pipeline environment. Beyond technical development, they will also need to communicate directly with Product Owners and business stakeholders, gather requirements, design and support jobs, and work across multiple business units and data domains. The expectation is that they can independently manage the full lifecycle-from requirement gathering and architecture discussions through implementation, orchestration, monitoring, and ongoing support.

The environment heavily leverages AWS technologies, particularly Step Functions and Lambda functions. While Step Functions may not be commonly used in every environment, they are central to this architecture and are used extensively for orchestration. The workflows currently support approximately 60 jobs running in a combination of sequential and parallel execution patterns. At the conclusion of each workflow, Lambda functions capture logs and auditing details to monitor job completion status, performance metrics, and notifications. The selected candidate will need to understand this orchestration model in depth and be comfortable configuring and maintaining these processes.

location: Malvern, Pennsylvania

job type: Solutions

salary: $58.33 - 63.33 per hour

work hours: 8am to 5pm

education: Bachelors



responsibilities:

This role is highly critical because the individual will essentially own and manage the pipeline environment. Beyond technical development, they will also need to communicate directly with Product Owners and business stakeholders, gather requirements, design and support jobs, and work across multiple business units and data domains. The expectation is that they can independently manage the full lifecycle-from requirement gathering and architecture discussions through implementation, orchestration, monitoring, and ongoing support.

The environment heavily leverages AWS technologies, particularly Step Functions and Lambda functions. While Step Functions may not be commonly used in every environment, they are central to this architecture and are used extensively for orchestration. The workflows currently support approximately 60 jobs running in a combination of sequential and parallel execution patterns. At the conclusion of each workflow, Lambda functions capture logs and auditing details to monitor job completion status, performance metrics, and notifications. The selected candidate will need to understand this orchestration model in depth and be comfortable configuring and maintaining these processes.

qualifications:

Another major focus area for this role is performance optimization. The current pipeline execution time is approximately three to four hours, and the goal is to reduce that runtime to under two hours. Because of the complexity of the environment, optimization and performance tuning will be a primary value-add for this resource. The candidate should have robust experience with Spark optimization, SQL performance tuning, and large-scale data pipeline efficiency improvements.

From a technical screening perspective, the primary areas of focus will likely include:

SQL and advanced query optimization

PySpark and Spark performance tuning

Data pipeline architecture and optimization

AWS orchestration technologies, including Step Functions and Lambda

Data modeling and workflow orchestration concepts

While AWS Step Functions are a mandatory part of the environment, deeper orchestration expertise can also be evaluated further during the interview process. The broader project work is centered around orchestration, optimization, and operational support, so interview discussions will likely include scenario-based questions focused on those areas.

In addition, the person should be capable of acting as a subject matter expert for the broader team, including offshore resources in India. They should be comfortable answering technical questions, providing guidance, and supporting troubleshooting efforts as needed.

Equal Opportunity Employer: Race, Color, Religion, Sex, Sexual Orientation, Gender Identity, National Origin, Age, Genetic Information, Disability, Protected Veteran Status, or any other legally protected group status.

At Randstad Digital, we welcome people of all abilities and want to ensure that our hiring and interview process meets the needs of all applicants. If you require a reasonable accommodation to make your application or interview experience a great one, please contact

Pay offered to a successful candidate will be based on several factors including the candidate's education, work experience, work location, specific job duties, certifications, etc. In addition, Randstad Digital offers a comprehensive benefits package, including: medical, prescription, dental, vision, AD&D, and life insurance offerings, short-term disability, and a 401K plan (all benefits are based on eligibility).

This posting is open for thirty (30) days.


Any consideration of a background check would be an individualized assessment based on the applicant or employee's specific record and the duties and requirements of the specific job.