> ## Documentation Index
> Fetch the complete documentation index at: https://docs.scrapegraphai.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Agno

> Wire ScrapeGraph into Agno agents with the first-party ScrapeGraphTools toolkit

## Overview

Agno ships a first-party `ScrapeGraphTools` toolkit at `agno.tools.scrapegraph`. One import, pass it to `Agent(tools=[...])`, and every ScrapeGraph endpoint is available to the model — no wrappers required.

<CardGroup cols={2}>
  <Card title="Agno docs" icon="book" href="https://docs.agno.com">
    Official Agno documentation
  </Card>

  <Card title="ScrapeGraphTools source" icon="github" href="https://github.com/agno-agi/agno/blob/main/libs/agno/agno/tools/scrapegraph.py">
    The toolkit on GitHub
  </Card>
</CardGroup>

## Installation

```bash theme={null}
pip install -U "agno @ git+https://github.com/agno-agi/agno.git#subdirectory=libs/agno" openai scrapegraph-py
```

Set your keys:

```bash theme={null}
export SGAI_API_KEY="your-scrapegraph-key"
export OPENAI_API_KEY="your-openai-key"
```

<Note>
  Until the next Agno release ships the ScrapeGraph v2 rewrite, install Agno from `main` (as shown above). Agno is model-agnostic — swap `OpenAIChat` for `Claude`, `Gemini`, or any other [supported model](https://docs.agno.com/models).
</Note>

## Quickstart

Enable every tool with `all=True` and let the model pick the right one per turn:

```python theme={null}
from agno.agent import Agent
from agno.models.openai import OpenAIChat
from agno.tools.scrapegraph import ScrapeGraphTools

agent = Agent(
    model=OpenAIChat(id="gpt-4o"),
    tools=[ScrapeGraphTools(all=True)],
    markdown=True,
)

agent.print_response(
    "Use smartscraper on https://example.com to extract the page title and main heading. Return them as JSON.",
    stream=True,
)
```

## Tools exposed

| Tool | Signature | ScrapeGraph endpoint |
| - | - | - |
| `smartscraper` | `(url, prompt) -> str` | `POST /extract` |
| `markdownify` | `(url) -> str` | `POST /scrape` (markdown format) |
| `searchscraper` | `(query) -> str` | `POST /search` |
| `crawl` | `(url, prompt, schema, max_depth=2, max_pages=2) -> str` | `POST /crawl` (polls until complete) |
| `scrape` | `(url) -> str` | `POST /scrape` (HTML format) |

Each method returns a JSON string (or plain markdown for `markdownify`), which is what Agno hands back to the model.

## Configuration

All knobs live on `ScrapeGraphTools.__init__`:

| Argument | Default | Purpose |
| - | - | - |
| `api_key` | `$SGAI_API_KEY` | Your ScrapeGraph API key |
| `enable_smartscraper` | `True` | Register `smartscraper` |
| `enable_markdownify` | `False` | Register `markdownify` |
| `enable_searchscraper` | `False` | Register `searchscraper` |
| `enable_crawl` | `False` | Register `crawl` |
| `enable_scrape` | `False` | Register `scrape` |
| `all` | `False` | Shortcut: enable every tool |
| `render_heavy_js` | `False` | Request JavaScript rendering on every call |
| `headers` | `None` | Custom HTTP headers (User-Agent, Cookie, Authorization, …) applied to every fetch |
| `crawl_poll_interval` | `3` | Seconds between crawl status polls |
| `crawl_max_wait` | `180` | Max seconds to wait for a crawl to complete |

Only enable what the agent needs — a tighter tool surface gives the model a smaller decision space and usually better routing.

```python theme={null}
tools = ScrapeGraphTools(
    enable_smartscraper=True,
    enable_markdownify=True,
    enable_scrape=True,
    render_heavy_js=True,
    headers={"User-Agent": "MyBot/1.0"},
)
```

## Examples

### Structured extraction with `smartscraper`

```python theme={null}
agent = Agent(
    model=OpenAIChat(id="gpt-4o"),
    tools=[ScrapeGraphTools(enable_smartscraper=True)],
    markdown=True,
)

agent.print_response(
    "Extract the product name and price from "
    "https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/ as JSON.",
    stream=True,
)
```

### Markdown conversion with `markdownify`

```python theme={null}
agent = Agent(
    model=OpenAIChat(id="gpt-4o"),
    tools=[ScrapeGraphTools(enable_markdownify=True)],
    markdown=True,
)

agent.print_response(
    "Fetch https://scrapegraphai.com and summarize the top three product features from the markdown.",
    stream=True,
)
```

### Multi-page extraction with `crawl`

`crawl` requires a JSON schema so every page contributes to the same shape. The toolkit polls until completion (bounded by `crawl_max_wait`).

```python theme={null}
agent = Agent(
    model=OpenAIChat(id="gpt-4o"),
    tools=[ScrapeGraphTools(enable_crawl=True, crawl_max_wait=600)],
    markdown=True,
)

agent.print_response(
    """Crawl https://books.toscrape.com with max_depth=2, max_pages=5.
    Use this schema: {"type": "object", "properties": {"books": {"type": "array", "items": {"type": "object", "properties": {"title": {"type": "string"}, "price": {"type": "string"}}}}}}.
    Prompt: 'Extract every book title and price on the page'. Return the merged JSON.""",
    stream=True,
)
```

## Support

<CardGroup cols={2}>
  <Card title="Python SDK" icon="github" href="https://github.com/ScrapeGraphAI/scrapegraph-py">
    Source and issues for scrapegraph-py
  </Card>

  <Card title="Discord" icon="discord" href="https://discord.gg/uJN7TYcpNa">
    Get help from our community
  </Card>
</CardGroup>


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.