Overview

Knowledge Retention assesses how well your AI assistant retains and recalls information provided by the user throughout multi-turn conversations. This metric is crucial for understanding whether your assistant maintains context, remembers previously mentioned facts, and demonstrates consistent knowledge without attrition across conversation turns.

Ideal for: Conversational agents with long-term context, customer support bots, educational assistants, personal assistant applications, and any agent that must remember user preferences and history.

What Gets Evaluated

This evaluation analyzes the assistant's memory and context awareness across conversation turns:

• ✅ Evaluates: "Did the assistant remember Sarah's name from turn 1 when responding in turn 5?"
• ✅ Evaluates: "Does the assistant consistently recall user preferences mentioned earlier?"
• ✅ Evaluates: "Is information retained without contradiction or re-asking?"
• ❌ Does NOT evaluate: Response quality or tool selection - only information retention

Key Features

• Multi-turn Context Analysis: Evaluates context maintenance across multiple conversation turns
• Knowledge Attrition Detection: Identifies specific instances where previously mentioned information is lost
• Consistency Measurement: Assesses consistency in recalling facts, names, dates, numbers, and details
• Turn-by-Turn Analysis: Provides retention accuracy for each conversation turn

How It Works

The evaluation uses an LLM-as-a-judge approach:

1. Extract User Information: Identifies all key facts, details, and context provided by the user
2. Analyze Each Turn: Reviews each assistant response for knowledge retention or attrition
3. Detect Attrition: Flags instances where the assistant forgets, contradicts, or re-asks for information
4. Calculate Score: score = (Assistant Turns without Knowledge Attrition) / (Total Assistant Turns)