Job Description | Skills:
Hands-on experience with Langfuse (including v3 features) and integrations.
Experience with other GenAI observability tools (e.g., TruLens, W&B, Helicone).
Knowledge of Retrieval-Augmented Generation (RAG), fine-tuning, and multi-agent orchestration.
Strong understanding of Azure AI services, especially the Evaluation SDK.
Deep expertise in LLMOps, prompt engineering, and GenAI lifecycle management.
Proficiency in Python, TypeScript, or similar languages used in GenAI frameworks.
Experience with cloud-native architectures (Azure preferred).
Familiarity with Tracing tools, observability platforms, and evaluation metrics.
Excellent communication and documentation skills.
Key Responsibilities:
Set-up and deploy Langfuse v3 in production environment.
Architect and implement the upgrade of Langfuse v2 to v3 within the LamBots framework, ensuring backward compatibility and performance optimization
Design modular components for prompt management, tracing, metrics, evaluation, and playground features using Langfuse v3. Leverage Langfuse s full feature set:
Prompt Management versioning, templating, and optimization
Tracing end-to-end visibility into GenAI workflows
Metrics performance, latency, and usage analytics
Evaluation automated and manual scoring of model outputs
Playground interactive testing and debugging of prompts
Integrate Azure AI Evaluation SDK into LamBots to enable scalable enterprise-grade evaluation pipelines/workflows, including:
Build reusable components and templates for evaluation across diverse GenAI use cases.
Collaborate with cross-functional teams to integrate evaluation capabilities into production pipelines/ systems.
Ensure scalability and reliability of evaluation tools in both offline and online environments.
Define and enforce evaluation standards and best practices for GenAI agents, RAG pipelines, and multi-agent orchestration.
Collaborate with product, engineering, and data science teams to align evaluation metrics with business KPIs.
Drive observability, debugging, and traceability features for GenAI workflows.
Stay current with emerging GenAI evaluation tools, frameworks, and methodologies. |
Never miss an opportunity! Create an alert based on the job you applied for.
