Query Crawls

cURL

curl --request POST \
  --url https://api.olyptik.io/crawls/query \
  --header 'Authorization: Bearer <token>' \
  --header 'Content-Type: application/json' \
  --data '
{
  "page": 0,
  "status": [
    "running"
  ],
  "startUrls": [
    "<string>"
  ]
}
'

{
  "page": 0,
  "results": [
    {
      "startUrl": "https://example.com",
      "maxResults": 55,
      "maxDepth": 10,
      "useSitemap": false,
      "entireWebsite": false,
      "excludeNonMainTags": true,
      "includeLinks": true,
      "deduplicateContent": true,
      "extraction": "Extract only pricing info",
      "engineType": "auto",
      "useStaticIps": false,
      "timeout": 1800,
      "id": "6870e36787c81925622df818",
      "createdAt": "2023-11-07T05:31:56Z",
      "status": "timed_out",
      "completedAt": "2023-11-07T05:31:56Z",
      "durationInSeconds": 1800,
      "brandId": "<string>",
      "startUrls": [
        "https://example.com"
      ],
      "totalPages": 100,
      "origin": "web"
    }
  ],
  "totalPages": 0,
  "totalResults": 0,
  "limit": 20
}

POST

crawls

query

cURL

curl --request POST \
  --url https://api.olyptik.io/crawls/query \
  --header 'Authorization: Bearer <token>' \
  --header 'Content-Type: application/json' \
  --data '
{
  "page": 0,
  "status": [
    "running"
  ],
  "startUrls": [
    "<string>"
  ]
}
'

{
  "page": 0,
  "results": [
    {
      "startUrl": "https://example.com",
      "maxResults": 55,
      "maxDepth": 10,
      "useSitemap": false,
      "entireWebsite": false,
      "excludeNonMainTags": true,
      "includeLinks": true,
      "deduplicateContent": true,
      "extraction": "Extract only pricing info",
      "engineType": "auto",
      "useStaticIps": false,
      "timeout": 1800,
      "id": "6870e36787c81925622df818",
      "createdAt": "2023-11-07T05:31:56Z",
      "status": "timed_out",
      "completedAt": "2023-11-07T05:31:56Z",
      "durationInSeconds": 1800,
      "brandId": "<string>",
      "startUrls": [
        "https://example.com"
      ],
      "totalPages": 100,
      "origin": "web"
    }
  ],
  "totalPages": 0,
  "totalResults": 0,
  "limit": 20
}

Authorizations

Authorization

string

header

required

Bearer authentication header of the form Bearer <token>, where <token> is your auth token.

Body

application/json

Query crawls

page

integer

default:0

required

Page number to query

Required range: x >= 0

status

enum<string>[]

Status of the crawls to query, if left empty all statuses will be queried

Available options:

running,

succeeded,

failed,

aborted,

timed_out,

error

startUrls

string<uri>[]

Start URLs of the crawls to query, if left empty all start URLs will be queried

Response

Paginated results with matching crawls

page

integer

default:0

Required range: x >= 0

results

object[]

Show child attributes

results.startUrl

string<uri>

required

URL to start crawling from

Example:

"https://example.com"

results.maxResults

integer

Maximum number of results to collect Maximum number of results to collect

Required range: 1 <= x <= 110

results.maxDepth

integer

default:10

Maximum depth of pages to crawl Maximum depth of pages to crawl

Required range: 1 <= x <= 99

Example:

10

results.useSitemap

boolean

default:false

Whether to use sitemap.xml to crawl the website. If true - maxResults and maxDepth will be ignored.

Example:

false

results.entireWebsite

boolean

default:false

Whether to crawl the entire website. If true - maxResults and maxDepth will be ignored.

Example:

false

results.excludeNonMainTags

boolean

default:true

Whether to exclude non-main tags from the crawl results' markdown

Example:

true

results.includeLinks

boolean

default:true

Whether to include links in the crawl results' markdown

Example:

true

results.deduplicateContent

boolean

default:true

Whether to remove duplicate text fragments that appeared on other pages.

Example:

true

results.extraction

string

default:""

Instructions defining how the AI should extract specific content from the crawl results Instructions defining how the AI should extract specific content from the crawl results

Example:

"Extract only pricing info"

results.engineType

enum<string>

default:auto

The engine to use for the crawl. Auto: auto detect the best engine (default). Cheerio: fast, great for static websites. Playwright: great for dynamic websites that use JavaScript frameworks.

Available options:

auto,

cheerio,

playwright

Example:

"auto"

results.useStaticIps

boolean

default:false

Whether to use static IPs for the crawl. This target website can whitelist the IPs to use for the crawl. The static IP will be 154.17.150.0 Whether to use static IPs for the crawl. This target website can whitelist the IPs to use for the crawl. The static IP will be 154.17.150.0

Example:

false

results.timeout

integer

default:1800

Timeout duration in minutes Timeout duration in seconds

Required range: x >= 60

Example:

1800

results.id

string

Identification number of the crawl

Example:

"6870e36787c81925622df818"

results.createdAt

string<date-time>

Timestamp when the crawl was created

results.status

enum<string>

Current status of the crawl

Available options:

running,

succeeded,

failed,

aborted,

timed_out,

error

Example:

"timed_out"

results.completedAt

string<date-time>

Timestamp when the crawl was completed

results.durationInSeconds

integer

Duration of the crawl in seconds

Required range: x >= 0

Example:

1800

results.brandId

string

ID of the brand associated with the crawl

results.startUrls

string<uri>[]

Array of URLs to start crawling from

Example:

["https://example.com"]

results.totalPages

integer

Count of pages extracted

Required range: x >= 0

Example:

100

results.origin

enum<string>

Origin of the crawl request

Available options:

api,

web

Example:

"web"

totalPages

integer

default:0

Required range: x >= 0

totalResults

integer

default:0

Required range: x >= 0

limit

integer

default:20

Required range: x >= 0

Abort Crawl Get crawl results

⌘I

API Documentation

Scrape

Crawls

Crawl Results

Crawl Logs

Authorizations

Body

Response