Data Scientist - Conversational AI & Gen AI

Position : Data Scientist - Conversational AI & Gen AI

Location : Johnston, RI or Westwood, MA

Duration: Full Time

Domain (Industry):

• Cards, Banking, FSI

Mandatory skills

• Strong background in Data Science, Machine Learning, Generative AI, or a related quantitative field.
• Hands-on experience evaluating LLM, RAG, Agentic AI, or Conversational AI solutions.
• Deep understanding of model evaluation techniques and metrics, including:
• Precision @K
• Recall @K
• Mean Reciprocal Rank (MRR)
• F1 Score
• Retrieval and generation quality assessment
• Experience performing experimentation, statistical analysis, and performance benchmarking.
• Strong Python programming skills.
• Experience with machine learning libraries and frameworks such as Scikit-learn, XGBoost, Pandas, NumPy, and related tools.
• Ability to communicate technical findings succinctly to highly technical stakeholders.

Important Note

• This role is primarily a Data Science and AI Evaluation position, not an AI Engineering or deployment-focused role.
• The emphasis is on measuring, analyzing, validating, and improving AI system performance rather than building production deployment pipelines.

Detailed JD (Roles and Responsibilities)

We are seeking an experienced Data Scientist to support the development and evaluation of AI-powered fraud self-service voice agents and conversational AI systems. The primary responsibility is not model deployment or engineering implementation, but designing evaluation frameworks, measuring system performance, identifying failure patterns, conducting root-cause analysis, and optimizing model behavior through data-driven experimentation.

Key Responsibilities

• Design and execute evaluation frameworks for LLM, RAG, and multi-turn conversational AI systems.
• Develop metrics to assess customer intent recognition, conversation quality, guardrail effectiveness, and business outcomes.
• Analyze voice-agent interactions and identify areas of failure, drift, and performance degradation.
• Perform prompt tuning and experimentation to improve model accuracy and reliability.
• Conduct root-cause analysis of conversational failures and recommend remediation strategies.
• Measure performance across different model configurations, prompts, and guardrail implementations.
• Partner with AI Engineering and Product teams to validate solutions before production deployment.
• Build dashboards and reports that communicate model effectiveness and operational impact.
• Support fraud-related customer service use cases, including intent detection and multi-turn conversation flows.

Success Criteria

• Develop reliable evaluation methodologies for conversational AI systems.
• Quantify the effectiveness of fraud self-service voice agents.
• Optimize prompts, retrieval strategies, and guardrails using empirical evidence.
• Deliver actionable insights that improve customer experience and model performance.
• Establish measurable KPIs for intent detection and multi-turn conversation success.

Thanks & Regards

Rakesh Vangala

Team Lead

Burgeon IT Services LLC.

Email Id: OR

Website: , LinkedIn: