Website Reader

The Website Reader crawls and processes entire websites, following links to create comprehensive knowledge bases from web content.

Code

examples/basics/knowledge/concepts/readers/overview/web_reader.py

from agno.knowledge.reader.website_reader import WebsiteReader

reader = WebsiteReader(max_depth=3, max_links=10)

try:
    print("Starting read...")
    documents = reader.read("https://docs.agno.com/introduction")
    if documents:
        for doc in documents:
            print(doc.name)
            print(doc.content)
            print(f"Content length: {len(doc.content)}")
            print("-" * 80)
    else:
        print("No documents were returned")

except Exception as e:
    print(f"Error type: {type(e)}")
    print(f"Error occurred: {str(e)}")

Usage

1. Set up your virtual environment

   Mac

   uv venv --python 3.12
   source .venv/bin/activate
   

   uv venv --python 3.12
   .venv\Scripts\activate
   
2. Install dependencies

   uv pip install -U requests beautifulsoup4 agno openai
   
3. Set environment variables

   export OPENAI_API_KEY=xxx
   
4. Run Agent

   Mac

   python examples/basics/knowledge/concepts/readers/overview/web_reader.py
   

   python examples/basics/knowledge/concepts/readers/overview/web_reader.py
   

Params

Parameter	Type	Default	Description
url	str	Required	URL of the website to crawl and read
max_depth	int	3	Maximum depth level for crawling links
max_links	int	10	Maximum number of links to crawl