Overview
Skills
Job Details
Remote
SDET/QA (Python/AI); LLM, Machine Learning
Title: Software Development Engineer in Test
Key Skills: Python, any Scripting (Javascript, Shell, Perl), AI or Machine Learning product experience
Job Description:
TWO SDET QAs with exp in python and scripting. Experience with AI tech like Lang Graph is a plus
Resources will be responsible for testing Scout OpenAI and LLM, developing a virtual recruiter, and automating tasks. The solution will be built as an SDK, allowing integration with third-party ATS platforms while maintaining its current functionality within Dradis
Perform usability testing by interacting with the system through natural language commands (e.g., "Show me top applicants") and assessing its responses.
Analyze LLM behavior, identifying areas for improvement and providing actionable feedback.
Design and implement a test framework tailored for evaluating LLM performance, requiring an innovative approach beyond standard models.
Conduct backend automated testing or frontend manual testing, with no requirement for frontend automation scripts.
Work independently to log and track bugs, proactively identifying and implementing automation opportunities where applicable.
Evaluate model interactions, testing search relevance by querying top applicants and refining prompts for improved results.
Validate workflow functionality, leveraging LangGraph to assess system effectiveness and ensure seamless operations.
The resource will start writing a test framework that can test and evaluate LLMs (this is not a standard model)
Industry experience probably won't be found since this is so new
Either FE manual testing or backend automated testing
Independent workers
No FE automation scripts will be asked for
Log bugs
Figure out if automation is an option and start doing the work
Automate routine tasks
Required Skills
Python and Scripting Experience: Proficiency in writing Python scripts is essential for executing test scripts. Prior experience testing an AI chat agent/assistant or tool using Python scripts is very helpful.
Experience in a manual testing role, identifying and reporting bugs with the ability to automate routine and repeated tasks to optimize future testing efforts
Cross-functional Collaboration:
Ability to work effectively with cross-functional teams/Managers
Nice to have:
Experience with AI Technologies like LangGraph: While not mandatory, experience with AI technologies such as LangGraph is a plus.
No prior LLM experience is required but is a plus