Data Automation Engineer

Description

We are looking for a Data Automation Engineer to design, develop, and maintain scalable data automation solutions leveraging AWS cloud services, selected Azure technologies, and emerging Generative AI capabilities.

Key Responsibilities:

Design and implement scalable data automation workflows using AWS services with integration to Azure platforms.

Develop and maintain ETL/ELT processes to ingest, transform, and move data across DynamoDB, SQL Server, Azure SQL, and other enterprise data sources.

Build and support batch and near real-time data pipelines using Apache Spark and streaming technologies such as Kafka or Flume.

Develop scalable data processing solutions utilizing Amazon EMR, AWS Fargate, Kubernetes, and related cloud technologies.

Integrate REST APIs, external services, and Amazon Connect customer interaction data into analytics platforms.

Evaluate and prototype Generative AI solutions using Amazon Bedrock, Azure OpenAI, Hugging Face, LangChain, and related technologies.

Implement source control, CI/CD, and automation practices using GitHub, Azure DevOps, Jenkins, Docker, and Agile methodologies.

Ensure data solutions meet security and compliance requirements, including IAM, encryption, RBAC, VPC isolation, and firewall policies.

Requirements

Core Qualifications:

Bachelor's degree in Computer Science, Information Systems, Engineering, or a related technical field.

5+ years of experience in data engineering, data automation, software engineering, or a related discipline.

Strong hands-on experience developing Python- and SQL-based data pipelines in AWS environments.

Experience with Apache Spark and at least one streaming/ingestion technology such as Apache Kafka or Apache Flume.

Hands-on experience with AWS services including several of the following: S3, AWS Glue, Lambda, EMR, Step Functions, and AWS database services.

Experience integrating REST APIs and external services into automated data workflows.

Experience working with CI/CD and source control platforms such as GitHub, Azure DevOps, or Jenkins.

Strong troubleshooting, performance tuning, and production support experience across data platforms and pipelines.

Excellent communication and collaboration skills.

U.S. Citizenship and ability to obtain a Federal Public Trust clearance.

Preferred Qualifications:

Experience integrating Large Language Models (LLMs) such as Amazon Bedrock, Azure OpenAI, Hugging Face, or similar AI services into production workflows.

Experience with Retrieval-Augmented Generation (RAG), embeddings, vector databases, Apache Solr, OpenSearch, pgvector, or similar technologies.

Multi-cloud experience supporting both AWS and Azure environments.

Experience with Docker, Kubernetes, and containerized application deployments.

Experience operationalizing Generative AI solutions, including monitoring, evaluation, observability, and lifecycle management.

Familiarity with data governance and cataloging solutions such as Microsoft Purview, Unity Catalog, or AWS Glue Catalog.

Experience with Infrastructure as Code tools including Terraform, AWS CloudFormation, or Bicep.

Knowledge of FedRAMP, HIPAA, PCI-DSS, or other regulated compliance environments.

Relevant cloud, AI, or data engineering certifications.

Technology Doesn't Change the World, People Do.

Robert Half is the world's first and largest specialized talent solutions firm that connects highly qualified job seekers to opportunities at great companies. We offer contract, temporary and permanent placement solutions for finance and accounting, technology, marketing and creative, legal, and administrative and customer support roles.

Robert Half works to put you in the best position to succeed. We provide access to top jobs, competitive compensation and benefits, and free online training. Stay on top of every opportunity - whenever you choose - even on the go. Download the Robert Half app and get 1-tap apply, notifications of AI-matched jobs, and much more.

All applicants applying for U.S. job openings must be legally authorized to work in the United States. Benefits are available to contract/temporary professionals, including medical, vision, dental, and life and disability insurance. Hired contract/temporary professionals are also eligible to enroll in our company 401(k) plan. Visit roberthalf.gobenefits.net for more information.

2025 Robert Half. An Equal Opportunity Employer. M/F/Disability/Veterans. By clicking "Apply Now," you're agreeing to Robert Half's Terms of Use and Privacy Notice.