Skip to main content
POST
Crawl website

Description

This endpoint crawls a website and returns structured content from multiple pages.

Endpoint

Headers

  • Content-Type: application/json
  • Authorization: Bearer <API_KEY> (required)

Request Body

Responses

Success (200)

Example Request

Notes

  • Uses 10 credits per crawled page
  • Uses anti-bot measures and stealth crawling techniques
  • Limit is the max number of pages to crawl
  • Depth refers to the distance between the base URL path and sub paths

Rate Limiting

Rate limit headers (X-RateLimit-Limit and X-RateLimit-Remaining) are included in the response.

Authorizations

Authorization
string
header
required

Bearer authentication header of the form Bearer <token>, where <token> is your auth token.

Body

application/json

Parameters controlling a crawl job.

url
string<uri>
required

Root URL to crawl.

depth
integer
default:2

Maximum crawl depth.

Required range: x >= 1
limit
integer
default:5

Maximum number of pages to fetch.

Required range: x >= 1
format
enum<string>
default:markdown

Output format for the crawled pages.

Available options:
markdown,
text,
raw
requestSource
string

Optional request source identifier.

Response

Crawl job accepted or results returned.

url
string<uri>
required
format
enum<string>
required
Available options:
markdown,
text,
raw
depth
integer
required
limit
integer
required
pages
integer
required
results
object[]
required
creditUsage
integer
required