Senior Test Automation Engineer - AI Solutions (Python)

We are looking for a talented and experienced Senior Test Automation Engineer to join our team and help productionize AI solutions leveraging cutting-edge technologies. This is an exciting opportunity to work on AI-driven systems, including Retrieval Augmented Generation (RAG), Large Language Models (LLMs), and AWS Bedrock capabilities. You will play a critical role in ensuring the functionality, performance, scalability, security, and reliability of these innovative solutions.

This position offers remote setup with the flexibility to work from any location in Georgia, whether it's your home, well-equipped offices in Tbilisi and Batumi or a coworking space in Kutaisi.

#LI-DNI

Responsibilities

• Design and implement comprehensive testing strategies tailored to AI workflows, APIs, and cloud-native solutions
• Develop detailed test plans, test cases, and scripts for AI systems, including data pipelines, knowledge bases, vector databases, and LLM-powered components
• Validate the performance and accuracy of LLM outputs, ensuring compliance with prompt engineering best practices and client-specific requirements
• Test RAG-based architectures, including retrieval processes, embedding validation, and semantic search accuracy
• Ensure seamless integration of data ingestion, cleansing, transformation, and validation pipelines with vector databases and AI models
• Build and maintain test automation frameworks for AI components and data pipelines using industry-standard tools and custom scripts
• Conduct performance, load, and scalability testing for AI/LLM endpoints, RAG workflows, and deployed pipelines
• Test cloud infrastructure deployment (e.g., IaC via Terraform/CloudFormation) and ensure production-readiness for AWS Bedrock, S3, EKS, and vector database configurations
• Identify, log, and manage defects in AI workflows, APIs, data pipelines, and backend systems, collaborating with developers and DevOps teams to resolve issues
• Monitor system reliability, availability, and fault tolerance in cloud environments to ensure production-level readiness

Requirements

• Proficiency in test automation using Python (3+ years of experience)
• Proven experience in QA engineering, with a focus on testing AI systems and cloud-native solutions
• Strong knowledge of testing frameworks, tools, and methodologies for AI workflows, APIs, and data pipelines
• Hands-on experience with LLMs, RAG architectures, and vector databases
• Proficiency in test automation using Python or similar scripting languages
• Expertise in performance and scalability testing for AI systems
• Familiarity with cloud environments, particularly AWS services like Bedrock, S3, and EKS
• Solid understanding of Infrastructure as Code (IaC) tools such as Terraform or CloudFormation
• Excellent problem-solving skills and ability to collaborate effectively with cross-functional teams

Nice to have

• Experience with AWS Bedrock capabilities and generative AI solutions
• Knowledge of semantic search and embedding validation techniques
• Familiarity with continuous integration and delivery pipelines for AI systems
• Background in data engineering or data science

We offer

• We connect like-minded people:
  • Delivering innovative solutions to industry leaders, making a global impact
  • Enjoyable working environment, whether it is the vibrant office or the comfort of your own home
  • Opportunity to work abroad for up to two months per year
  • Relocation opportunities within our offices in 55+ countries
  • Corporate and social events
• We invest in your growth:
  • Leadership development, career advising, soft skills and well-being programs
  • Certifications, including Google Cloud Platform, Azure and AWS
  • Unlimited access to LinkedIn Learning and Get Abstract
  • Free English classes with certified teachers
• We cover it all:
  • Participation in the Employee Stock Purchase Plan
  • Monetary bonuses for engaging in the referral program
  • Comprehensive medical & family care package
  • Five trust days per year (sick leave without a medical certificate)
  • Benefits package (sports activities, a variety of stores and services)

EPAM Georgia is a team of innovators united by a passion for technology. The dynamic and inclusive culture we embrace helps positively impact our communities, clients, and employees. Here you will collaborate with multi-national teams, contribute to numerous cutting-edge projects, deliver the most creative solutions, and have an opportunity to learn. Our people are at the heart of our success, and we are proud to provide talents with a solid ground to develop and grow.