Overview

Role Adherence measures how well your AI assistant stays within its specified role boundaries, respects constraints, and maintains appropriate behavior. Using an LLM-as-a-judge approach, it assesses whether outputs align with role definitions and behavioral guidelines.

Ideal for: Customer-facing agents, safety-critical applications, ensuring agents stay in character, validating constraint compliance, and quality assurance for production systems.

What Gets Evaluated

This evaluation examines three key dimensions:

• ✅ Evaluates: "Does the assistant stay within defined role boundaries?"
• ✅ Evaluates: "Does it respect explicit prohibitions and limitations?"
• ✅ Evaluates: "Is the behavior appropriate, safe, and aligned with guidelines?"
• ❌ Does NOT evaluate: Response quality or task completion - only role adherence

Key Features

• Multi-Dimensional Analysis: Evaluates role consistency, constraint compliance, and safety
• Behavioral Assessment: Ensures appropriate tone and conduct within defined boundaries
• Constraint Validation: Verifies adherence to explicit prohibitions and limitations
• Safety Focus: Identifies potentially unsafe or inappropriate behavior

How It Works

The evaluation follows this workflow:

1. Define Role: Specify role definition, constraints, and behavioral guidelines
2. Extract Output: Retrieve the assistant's actual response from logs
3. LLM Judge: Send both to a judge model that evaluates adherence
4. Score: Judge assigns a score (0.0-1.0) based on how well the output adheres to role and constraints