Scenario Testing

This example demonstrates how to use the Scenario framework for agentic simulation-based testing. Scenario enables you to simulate conversations between agents, user simulators, and judges, making it easy to test and evaluate agent behaviors in a controlled environment.

Tip: For a more information about using Scenario with Agno, check out the Scenario documentation.

Basic Scenario Testing

scenario_testing.py

import pytest
import scenario
from agno.agent import Agent
from agno.models.openai import OpenAIResponses

# Configure Scenario defaults (model for user simulator and judge)
scenario.configure(default_model="openai/gpt-4.1-mini")

@pytest.mark.agent_test
@pytest.mark.asyncio
async def test_vegetarian_recipe_agent() -> None:
    # 1. Define an AgentAdapter to wrap your agent
    class VegetarianRecipeAgentAdapter(scenario.AgentAdapter):
        agent: Agent

        def __init__(self) -> None:
            self.agent = Agent(
                model=OpenAIResponses(id="gpt-5.2"),
                markdown=True,
                debug_mode=True,
                instructions="You are a vegetarian recipe agent.",
            )

        async def call(self, input: scenario.AgentInput) -> scenario.AgentReturnTypes:
            response = self.agent.run(
                message=input.last_new_user_message_str(), # Pass only the last user message
                session_id=input.thread_id, # Pass the thread id, this allows the agent to track history
            )
            return response.content

    # 2. Run the scenario simulation
    result = await scenario.run(
        name="dinner recipe request",
        description="User is looking for a vegetarian dinner idea.",
        agents=[
            VegetarianRecipeAgentAdapter(),
            scenario.UserSimulatorAgent(),
            scenario.JudgeAgent(
                criteria=[
                    "Agent should not ask more than two follow-up questions",
                    "Agent should generate a recipe",
                    "Recipe should include a list of ingredients",
                    "Recipe should include step-by-step cooking instructions",
                    "Recipe should be vegetarian and not include any sort of meat",
                ]
            ),
        ],
    )

    # 3. Assert and inspect the result
    assert result.success

Usage

1. Set up your virtual environment

   Mac

   uv venv --python 3.12
   source .venv/bin/activate
   

   uv venv --python 3.12
   .venv\Scripts\activate
   
2. Set your API key

   export OPENAI_API_KEY=xxx
   export LANGWATCH_API_KEY=xxx # Optional, required for Simulation monitoring
   
3. Install dependencies

   uv pip install -U openai agno langwatch-scenario pytest pytest-asyncio
   # or
   uv add agno langwatch-scenario openai pytest
   
4. Run Agent

   pytest scenario_testing.py