Search Jobs | Dice.com

Job Details

Skills

Testing
Forms
Artificial Intelligence
Quality Assurance
Evaluation
Prompt Engineering
Scratch
Python
Docker
PostgreSQL
Continuous Integration
Continuous Delivery
React.js
TypeScript
English
Writing
PASS
Recruiting
Training
Communication
Onboarding
Internet
Network

Summary

Job description This opportunity is intended for experienced Senior Python Engineers only Open to candidates in the North America, South America, Asia and Europe. Please submit your CV in English and indicate your level of English proficiency. Mindrift connects specialists with project-based AI opportunities for leading tech companies, focused on testing, evaluating, and improving AI systems. Participation is project-based, not permanent employment. What this opportunity involves We\'re building a dataset to evaluate AI coding agents - how well a model handles real-world developer tasks. You\'ll create challenging tasks and evaluation criteria within realistic simulated environments: Build realistic developer environments - a virtual company with codebase, infrastructure, and context (tickets, docs, conversations) that forms a believable development history Design tasks from intermediate states of these environments - craft the prompt, define what "solved" means, and ensure the task is solvable by an AI agent Write tests that verify agent solutions - accept all valid approaches and reject incorrect ones, neither too strict nor too lenient Iterate on tasks and tests based on QA feedback - review agent solutions, analyze failures, and refine until the evaluation is fair and robust What this is NOT Not data labeling Not prompt engineering Not writing code from scratch - the agent writes most of the code; you guide and evaluate What we look for - You must meet all the requirements in order to be considered for this project: 5 years of professional experience with Python. Strong experience with FastAPI , pytest , and async/await . Hands-on experience with Docker , PostgreSQL , and CI/CD pipelines. Proven experience writing and maintaining automated tests (not just executing them). Full-stack experience with React and TypeScript is a plus. English proficiency at B2 level or higher . Availability to work 30 hours per week . Why this is hard Frontier models are already good at coding. Creating a task that genuinely challenges the best models is non-trivial. You need to deeply understand where models fail and what scenarios reveal the difference between a good and a bad solution. Tasks have many valid solutions - writing tests that accept all correct solutions and reject incorrect ones is harder than it sounds. How it works Apply ? Pass qualification(s) ? Join a project ? Complete tasks ? Get paid Hiring & Onboarding Process The process is designed to move quickly and typically includes the following steps: App review and invitation to a virtual project introduction session (approximately 30 minutes) Platform registration and identity verification Technical assessment (approximately 35 minutes) Background check (completed at no cost to candidates) Onboarding and project-specific training tasks Begin production work! Additional Requirements Willingness to complete identity verification as part of the onboarding process. Ability to complete a technical assessment. Willingness to join and participate in Discord, which will be used for project communication and updates. Successful completion of a background check is required prior to onboarding. Reliable internet connection and ability to communicate effectively in a remote environment. Company Braintrust is a global talent network that connects top independent professionals with leading companies for high-quality, flexible work. We help organizations hire skilled talent faster while giving professionals access to vetted opportunities with innovative teams.

Employers have access to artificial intelligence language tools (“AI”) that help generate and enhance job descriptions and AI may have been used to create this description. The position description has been reviewed for accuracy and Dice believes it to correctly reflect the job opportunity.

Dice Id: orcfeed
Position Id: 22637_5817562633
Posted 8 hours ago

Fitment

Dice Job Match Score™

🔗 Matching skills to job...

Create job alert

Never miss an opportunity! Create an alert based on the job you applied for.

New York, New York

•

16d ago

Affirm is reinventing credit to make it more honest and friendly, giving consumers the flexibility to buy now and pay later without any hidden fees or compounding interest. Affirm is reinventing credit to make it more honest and friendly, giving consumers the flexibility to buy now and pay later without any hidden fees or compounding interest. We are seeking a highly motivated Software Engineer II with strong experience in test automation to contribute to Developer Productivity, with a focus on

Full-time

USD 165,000.00 - 225,000.00 per year

Senior Software Development Engineer in Test (SDET)

New York, New York

•

8d ago

Summary: Topstep operates a real-time futures trading platform (TopstepX). Here, correctness isn't cosmetic: a defect in order handling, market-data delivery, or a balance/P&L calculation can cost a trader real money. We're hiring a Senior SDET to drive test automation across the platform - building the frameworks, shaping coverage decisions, and raising the quality bar alongside our existing SDETs. We lean heavily on AI-assisted test generation to produce broad, meaningful coverage at scale, an

Full-time

USD 155,000.00 - 190,000.00 per year

Entry Level Quality Assurance Engineer : Cloud Infrastructure & Web UI

New York, New York

•

Yesterday

At Jack Henry, we're more than a technology company, we're a force for good in financial services. We're redefining how community banks and credit unions connect with the people they serve. Our mission is rooted in people inspired innovation, empowering financial institutions to deliver seamless, secure, and human centered experiences. We deliver cutting-edge solutions that are paving the way for the next generation of digital banking and payments, but our true impact begins with our associates.

Full-time

USD 48,500.00 - 72,000.00 per year

Senior Automation Tester (SDET)

Jersey City, New Jersey

•

Today

RESPONSIBILITIES: Kforce has a client that is seeking a Senior Automation Tester (SDET) in Jersey City, NJ. Responsibilities: * Design, develop, execute, and maintain automated testing solutions throughout the software development lifecycle * Create automated test scripts for functional, integration, regression, API, and end-to-end testing scenarios * Collaborate with development and business teams to understand requirements and ensure software quality objectives are met * Support continuous te

Contract, Third Party

$53 - $61 hourly

Filter Results

Job post features

Posted date

Work settings

Employment type

Distance

Employer type

Work authorization

Senior Software Engineer (Python) - Agent Evaluation - Freelance/Remote 100 openings

Skills

Summary

Dice Job Match Score™

Similar Jobs

Senior Software Engineer (Python) - Agent Evaluation - Freelance/Remote 100 openings

Job Details

Skills

Summary

Dice Job Match Score™

Similar Jobs