Overview
On Site
BASED ON EXPERIENCE
Contract - W2
Contract - Independent
Contract - 8+ mo(s)
Skills
Product Management
Generative Artificial Intelligence (AI)
API
UI
Qualitative Analysis
Decision-making
Data Analysis
Cyber Security
Communication
Presentations
Graphics Design
Durable Skills
Privacy
Regulatory Compliance
Risk Management
Testing
Process Analysis
Operations Management
Program Management
Process Improvement
Artificial Intelligence
Machine Learning (ML)
Problem Solving
Conflict Resolution
Critical Thinking
Attention To Detail
Health Care
Legal
Insurance
SANS
Job Details
Duration: 9+ months - Possible Extension
Location: Sunnyvale, CA - Hybrid
Pay Range: $80-$90/hr on W2 (As per experience)
Job Description:
Project Overview:
- The Responsible AI Scaled Testing Team within Trust & Safety performs pre-launch structured testing for AI applications against safety, fairness, and neutrality policies and standards. It is a global team with Responsible AI domain expertise and diverse backgrounds in operations, strategy, ethics, risk management, product management, and program management.
Overall Responsibilities:
- Lead structured pre-launch safety, neutrality, and fairness testing, end-to-end, for GenAI products.
- For each launch, this will involve defining the standards applicable to the product, defining and executing prompt generation strategies, collaborating with product teams to scrape responses, working with our extended workforce to execute prompt/response rating against defined standard (incl. providing clear instructions, clarifying gray area cases, and/or providing quality calibrations), and conducing in-depth quantitative and qualitative analysis of results.
- Intake and triage new launch submissions; understand requirements and kick off engagement or schedule for future start date.
- Move existing launches through the pre-launch testing process, using Buganizer bugs to track progress.
- This may include conducting work independently or collaborating with stakeholders to gather information or ensure they are taking action on their end.
Key steps will include:
- Aligning on the safety, neutrality, and fairness standards applicable to the product, with an eye for driving consistency across product areas. Translating standards into clear guidelines that can be used to evaluate whether the product's output is compliant with standards.
- Defining and executing prompt generation strategies to develop a set of prompts that will sufficiently test product compliance with standards. This may entail leveraging LLM-based prompt generation tools and/or defining and providing clear instructions to vendor teams.
- Collaborating with product teams to scrape responses. This may entail providing consultation for how to develop a scaled scraping solution (UI, API, etc.), getting access to the model/UI and performing scrapes, and/or defining and providing clear instructions to vendor teams.
- Executing prompt/response rating against defined standards. This may entail providing clear instructions to vendor teams, clarifying gray area cases, and/or providing quality calibrations.
- Deep dive analysis: Conduct in-depth quantitative and qualitative analysis of results, including unexpected, interesting, and edge cases, providing clear and actionable insights to inform decision-making around pre- and post-launch mitigation steps.
Mandatory Skills/Qualifications:
- Bachelor's degree or equivalent practical experience.
- 4 years of experience in any one of data analytics, Trust & Safety, policy, cybersecurity, or related fields.
- Experience using data to provide solutions and recommendations.
- Excellent communication and presentation skills (written and verbal) and the ability to influence cross-functionally at various levels.
- This role may be exposed to graphic, controversial, and/or upsetting content.
- 2+ years of experience in trust and safety, product policy, privacy and security, legal, compliance, risk management, Client, content moderation, red teaming, AI testing, adversarial testing, or similar.
- 1+ years of experience in business process analysis, operations management, and/or global program management, or leading cross-functional process improvements
- Strong understanding of AI systems, machine learning, and their potential risks.
- Ability to think strategically and identify emerging threats and vulnerabilities.
- Excellent problem-solving and critical thinking skills with attention to detail in an ever-changing environment.
- Proven ability to work independently and as part of a team.
Benefits Info:
Russell Tobin, offers eligible employee s comprehensive healthcare coverage (medical, dental, and vision plans), supplemental coverage (accident insurance, critical illness insurance, and hospital indemnity), 401(k)-retirement savings, life & disability insurance, an employee assistance program, legal support, auto, home insurance, pet insurance and employee discounts with preferred vendors.
#CB
#LI-AD7
Employers have access to artificial intelligence language tools (“AI”) that help generate and enhance job descriptions and AI may have been used to create this description. The position description has been reviewed for accuracy and Dice believes it to correctly reflect the job opportunity.