Overview

Turn Relevancy evaluates whether a conversation turn is relevant to the preceding context. This evaluation method assesses how well each turn in a conversation maintains coherence and relevance to the ongoing dialogue, helping identify when conversations drift off-topic or lose context.

Ideal for: Multi-turn conversations, customer support dialogues, ensuring conversation coherence, identifying context drift, and maintaining conversation quality in long interactions.

What Gets Evaluated

This evaluation analyzes the relevance and coherence of individual conversation turns:

• ✅ Evaluates: "Is this response relevant to what was just discussed?"
• ✅ Evaluates: "Does this turn maintain conversation flow and context?"
• ✅ Evaluates: "Is the assistant staying on-topic or drifting away?"
• ❌ Does NOT evaluate: Response quality, correctness, or task completion - only relevance to context

Key Features

• Context-Aware Analysis: Considers the full conversation history when evaluating each turn
• Relevance Scoring: Provides a 0.0-1.0 score for how relevant each turn is
• Conversation Flow: Identifies when conversations lose coherence or drift off-topic
• Multi-Turn Support: Works with conversations of any length

How It Works

The evaluation uses an LLM-as-a-judge approach with these steps:

1. Extract Context: Retrieves conversation history and the current turn to evaluate
2. Analyze Relevance: The LLM judge assesses how relevant the current turn is to the conversation context
3. Score Relevancy: Judge assigns a score (0.0-1.0) based on relevance to preceding context
4. Pass/Fail: Compares score against the default threshold (0.7)