Skip to main content

Documentation Index

Fetch the complete documentation index at: https://docs.scrapegraphai.com/llms.txt

Use this file to discover all available pages before exploring further.

GET https://v2-api.scrapegraphai.com/api/crawl/:id
Returns progress and lightweight per-page metadata for a crawl job started with POST /api/crawl. Use GET /api/crawl/:id/pages to fetch paginated pages with resolved scrape results.

​
Path parameters

​
id
string
required
The crawl job UUID returned by POST /api/crawl.

​
Example request

curl -X GET https://v2-api.scrapegraphai.com/api/crawl/79694e03-f2ea-43f2-93cc-7c6fc26f999a \
  -H "SGAI-APIKEY: $SGAI_API_KEY"

​
Example response

{
  "id": "79694e03-f2ea-43f2-93cc-7c6fc26f999a",
  "status": "completed",
  "total": 3,
  "finished": 1,
  "pages": [
    {
      "url": "https://example.com",
      "depth": 0,
      "title": "",
      "status": "completed",
      "parentUrl": null,
      "contentType": "text/html",
      "links": ["https://iana.org/domains/example"],
      "scrapeRefId": "83a911ed-c0bc-4a8c-ad62-8efeeb93f33a"
    }
  ]
}
FieldDescription
status"running", "completed", "failed", or "stopped".
total / finishedProgress counters.
pages[]Lightweight per-page metadata, ordered by crawl time.
pages[].scrapeRefIdUUID of the underlying Scrape call.
Poll at a reasonable cadence (every 1–5 seconds) until status is "completed", "failed", or "stopped". Or use Monitor with a webhook to avoid polling entirely.

​
Fetching page content

The status response intentionally stays lightweight for polling. Use GET /api/crawl/:id/pages to fetch crawl pages with the underlying scrape result resolved into each page: 
curl -X GET "https://v2-api.scrapegraphai.com/api/crawl/79694e03-f2ea-43f2-93cc-7c6fc26f999a/pages?limit=50&cursor=0" \
  -H "SGAI-APIKEY: $SGAI_API_KEY"
The response is { data, pagination }, where each data[] item includes crawl metadata and a scrape payload when available.