vLLM - Agno

The vLLM Embedder provides high-performance embedding inference with support for both local and remote deployment modes. It can load models directly for local inference or connect to a remote vLLM server via an OpenAI-compatible API.

Usage

from agno.knowledge.embedder.vllm import VLLMEmbedder
from agno.knowledge.knowledge import Knowledge
from agno.vectordb.pgvector import PgVector

# Local mode
embedder = VLLMEmbedder(
    id="intfloat/e5-mistral-7b-instruct",
    dimensions=4096,
    enforce_eager=True,
    vllm_kwargs={
        "disable_sliding_window": True,
        "max_model_len": 4096,
    },
)

# Use with Knowledge
knowledge = Knowledge(
    vector_db=PgVector(
        db_url="postgresql+psycopg://ai:ai@localhost:5532/ai",
        table_name="vllm_embeddings",
        embedder=embedder,
    ),
)

Parameters

Parameter	Type	Default	Description
`id`	`str`	`"intfloat/e5-mistral-7b-instruct"`	Model identifier (HuggingFace model name)
`dimensions`	`int`	`4096`	Embedding vector dimensions
`base_url`	`Optional[str]`	`None`	Remote vLLM server URL (enables remote mode)
`api_key`	`Optional[str]`	`getenv("VLLM_API_KEY")`	API key for remote server authentication
`enable_batch`	`bool`	`False`	Enable batch processing for multiple texts
`batch_size`	`int`	`10`	Number of texts to process per batch
`enforce_eager`	`bool`	`True`	Use eager execution mode (local mode)
`vllm_kwargs`	`Optional[Dict[str, Any]]`	`None`	Additional vLLM engine parameters (local mode)
`request_params`	`Optional[Dict[str, Any]]`	`None`	Additional request parameters (remote mode)
`client_params`	`Optional[Dict[str, Any]]`	`None`	OpenAI client configuration (remote mode)

Developer Resources

View Cookbook