​
Description

This endpoint crawls a website and returns structured content from multiple pages.

​
Endpoint

POST https://app.dumplingai.com/api/v1/crawl

​
Headers

  • Content-Type: application/json
  • Authorization: Bearer <API_KEY> (required)

​
Request Body

{
  "url": "string", // Required. The website URL to crawl
  "limit": "number", // Optional. Max pages to crawl (default: 5)
  "depth": "number", // Optional. Crawl depth (default: 2)
  "format": "string" // Optional. Output format: "markdown", "text", or "raw" (default: "markdown")
}

​
Responses

​
Success (200)

{
  "url": "string",
  "format": "string",
  "depth": "number",
  "limit": "number",
  "pages": "number",
  "results": [
    {
      "content": "string",
      "url": "string",
      "status": "number"
    }
  ],
  "creditUsage": "number"
}

​
Example Request

curl -X POST https://app.dumplingai.com/api/v1/crawl \
-H "Content-Type: application/json" \
-H "Authorization: Bearer YOUR_API_KEY" \
-d '{
  "url": "https://example.com",
  "limit": 10,
  "depth": 3,
  "format": "markdown"
}'

​
Notes

  • Uses 1 credit per crawled page
  • Uses anti-bot measures and stealth crawling techniques
  • Limit is the max number of pages to crawl
  • Depth refers to the distance between the base URL path and sub paths

​
Rate Limiting

Rate limit headers (X-RateLimit-Limit and X-RateLimit-Remaining) are included in the response.