Skip to main content

Documentation Index

Fetch the complete documentation index at: https://docs.scrapegraphai.com/llms.txt

Use this file to discover all available pages before exploring further.

Standard Libraries

Use familiar testing tools for mocking

No Credit Usage

Test without consuming API credits

​
Overview

In v2, the built-in mock mode (mock=True, mock_handler, mock_responses) has been removed from the SDKs. Instead, use standard mocking libraries for your language to test ScrapeGraphAI integrations without making real API calls or consuming credits.
If you’re migrating from v1, replace Client(mock=True) with standard mocking patterns shown below.

​
Python SDK Testing

​
Using unittest.mock

from unittest.mock import patch, MagicMock
from scrapegraph_py import Client

def test_extract():
    client = Client(api_key="test-key")

    mock_response = {
        "data": {
            "title": "Test Page",
            "content": "This is test content"
        },
        "request_id": "test-request-123"
    }

    with patch.object(client, "extract", return_value=mock_response):
        response = client.extract(
            url="https://example.com",
            prompt="Extract title and content"
        )

        assert response["data"]["title"] == "Test Page"
        assert response["request_id"] == "test-request-123"

​
Using responses Library

Mock HTTP requests at the transport layer: 
import responses
from scrapegraph_py import Client

@responses.activate
def test_extract_http():
    responses.post(
        "https://api.scrapegraphai.com/api/v2/extract",
        json={
            "data": {"title": "Mock Title"},
            "request_id": "mock-123"
        },
        status=200,
    )

    client = Client(api_key="test-key")
    response = client.extract(
        url="https://example.com",
        prompt="Extract the title"
    )

    assert response["data"]["title"] == "Mock Title"

​
Using pytest Fixtures

import pytest
from unittest.mock import MagicMock
from scrapegraph_py import Client

@pytest.fixture
def mock_client():
    client = Client(api_key="test-key")
    client.extract = MagicMock(return_value={
        "data": {"title": "Mock Title"},
        "request_id": "mock-123"
    })
    client.search = MagicMock(return_value={
        "data": {"results": []},
        "request_id": "mock-456"
    })
    client.credits = MagicMock(return_value={
        "remaining_credits": 100,
        "total_credits_used": 0
    })
    return client

def test_extract(mock_client):
    response = mock_client.extract(
        url="https://example.com",
        prompt="Extract the title"
    )
    assert response["data"]["title"] == "Mock Title"

def test_credits(mock_client):
    credits = mock_client.credits()
    assert credits["remaining_credits"] == 100

​
Async Testing with aioresponses

import pytest
import asyncio
from aioresponses import aioresponses
from scrapegraph_py import AsyncClient

@pytest.mark.asyncio
async def test_async_extract():
    with aioresponses() as mocked:
        mocked.post(
            "https://api.scrapegraphai.com/api/v2/extract",
            payload={
                "data": {"title": "Async Mock"},
                "request_id": "async-123"
            },
        )

        async with AsyncClient(api_key="test-key") as client:
            response = await client.extract(
                url="https://example.com",
                prompt="Extract data"
            )

            assert response["data"]["title"] == "Async Mock"

​
JavaScript SDK Testing

​
Using Jest / Vitest

import { describe, it, expect, vi } from "vitest";
import { extract, getCredits, search } from "scrapegraph-js";

// Mock the module
vi.mock("scrapegraph-js", () => ({
  extract: vi.fn().mockResolvedValue({
    status: "success",
    data: { raw: null, json: { title: "Mock Title" }, usage: {}, metadata: {} },
    elapsedMs: 100,
  }),
    search: vi.fn().mockResolvedValue({
      status: "success",
      data: { results: [] },
      elapsedMs: 100,
    }),
    getCredits: vi.fn().mockResolvedValue({
      status: "success",
      data: { remaining: 100, used: 50, plan: "pro" },
      elapsedMs: 50,
    }),
  };
}));

describe("ScrapeGraphAI", () => {
  it("should extract data", async () => {
    const result = await extract("test-key", {
      url: "https://example.com",
      prompt: "Extract the title",
    });
    expect(result.data?.json?.title).toBe("Mock Title");
  });

  it("should check credits", async () => {
    const result = await getCredits("test-key");
    expect(result.data?.remaining).toBe(100);
  });
});

​
Using MSW (Mock Service Worker)

Mock at the network level for more realistic testing: 
import { http, HttpResponse } from "msw";
import { setupServer } from "msw/node";
import { extract } from "scrapegraph-js";

const server = setupServer(
  http.post("https://api.scrapegraphai.com/v2/extract", () => {
    return HttpResponse.json({
      raw: null,
      json: { title: "MSW Mock Title" },
      usage: { promptTokens: 100, completionTokens: 50 },
      metadata: { chunker: { chunks: [] } },
    });
  })
);

beforeAll(() => server.listen());
afterAll(() => server.close());
afterEach(() => server.resetHandlers());

test("extract returns mocked data", async () => {
  const result = await extract("test-key", {
    url: "https://example.com",
    prompt: "Extract the title",
  });
  expect(result.data?.json?.title).toBe("MSW Mock Title");
});

​
Testing with cURL

Test API endpoints directly using cURL against a local mock server or staging environment: 
# Test extract endpoint
curl -X POST "https://api.scrapegraphai.com/api/v2/extract" \
  -H "Authorization: Bearer your-api-key" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com",
    "prompt": "Extract the title"
  }'

# Test credits endpoint
curl -X GET "https://api.scrapegraphai.com/api/v2/credits" \
  -H "Authorization: Bearer your-api-key"

​
SDK Comparison

FeaturePythonJavaScript
Mock libraryunittest.mock, responsesJest/Vitest mocks, MSW
HTTP-level mockingresponses, aioresponsesMSW (Mock Service Worker)
Async mockingaioresponses, unittest.mockNative async/await
Fixture supportpytest fixturesbeforeEach/afterEach

​
Best Practices

  • Mock at the client method level for unit tests (fastest, simplest)
  • Mock at the HTTP level for integration tests (validates request/response shapes)
  • Use fixtures to share mock configurations across tests
  • Keep mock responses realistic - match the actual API response structure
  • Test both success and error scenarios

​
Support

GitHub Issues

Report bugs or request features

Documentation

Python SDK documentation
Need help with testing? Join our Discord community for support.