SPHYNX
esc
  • Loading
Open console

API reference

Extract

Read the current page as Markdown, text, HTML or links, or pick out named values.

Extract reads the page the tab is on. Pass url to navigate first, so one call loads a page and reads it. Ask for whole-page formats, for named selections, or for both.

Extract

POST/v1/sessions/{id}/extract

Every field is optional: an empty object returns the current page as Markdown.

Path

  • idstringrequired

    The session to drive.

Body

  • urlstringdefaultstay on the current page

    Navigate here first.

  • formatslist of markdown, text, html, links

    The whole-page formats to return. The default is ["markdown"], or nothing when select is given.

  • selectlist of selections, ≤50

    Named values to pick out of the page.

    • namestringrequired

      The key of this value in selected. Names must be unique.

    • selectorstring

      A CSS selector. Give exactly one of selector and xpath.

    • xpathstring

      An XPath.

    • attrstringdefaulttext

      What to read from the match: text, html, outer_html, or any attribute name such as src.

    • allbooldefaultfalse

      false returns the first match as a string, or null when nothing matches. true returns every match as a list. The list ends before the value that would take it past max_chars in total.

  • max_charsint, ≤10000000default1000000

    Applied to each format, to links as a whole and to each all: true list as a whole.

  • wait_untilload or domcontentloadeddefaultload

    Only with url. As in Navigate.

  • timeout_msint, ≤60000default30000

    Only with url. As in Navigate.

  • tab_idstringdefaultthe active tab

    The tab to read.

Response

  • urlstring

    The address of the page that was read.

  • titlestring

    The title of that page.

  • markdownstring

    The main content converted to Markdown. Made from at most the first 4 MiB (4,194,304 UTF-16 units) of the body's HTML.

  • textstring

    document.body.innerText.

  • htmlstring

    The whole document as the browser currently has it: the doctype plus documentElement.outerHTML, after scripts have run.

  • linkslist

    Every link in document order, duplicates removed. [] when the page has none. The list ends before the link that would take its text and URLs past max_chars in total.

    • textstring

      The link text.

    • hrefstring

      The absolute URL.

  • selectedobject

    One key per selection. Each value is a string, null, or a list of strings.

  • truncatedbool

    true if a format, links, a selected value or an all: true list hit max_chars, or markdown was made from a body cut at 4 MiB.

Only the formats you asked for are present.

curl -s -X POST "$SPHYNX_URL/v1/sessions/$SESSION/extract" \
  -H "Authorization: Bearer $SPHYNX_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com",
    "formats": ["markdown", "links"]
  }'
200
{
  "url": "https://example.com/",
  "title": "Example Domain",
  "markdown": "This domain is for use in ...",
  "links": [
    {
      "text": "Learn more",
      "href": "https://iana.org/help/example-domains"
    }
  ],
  "truncated": false
}

Select named values

select picks specific values out of the page with CSS selectors or XPath. Each selection has a name, and the response carries one key per name under selected.

  • With all left off, the value is the first match as a string, or null when nothing matches.
  • With all: true, the value is a list with every match. The list ends before the value that would take it past max_chars in total.
  • attr decides what is read from each match: its text, its inner html, its outer_html, or one attribute.

When select is given and formats is not, no whole-page format is returned.

curl
curl -s -X POST "$SPHYNX_URL/v1/sessions/$SESSION/extract" \
  -H "Authorization: Bearer $SPHYNX_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com/product/42",
    "select": [
      {"name": "title", "selector": "h1"},
      {"name": "price",
       "xpath": "//span[@class=\"price\"]"},
      {"name": "images", "selector": "img.gallery",
       "attr": "src", "all": true},
      {"name": "specs", "selector": "#specs",
       "attr": "outer_html"}
    ]
  }'
200
{
  "url": "https://example.com/product/42",
  "title": "Widget",
  "selected": {
    "title": "Widget",
    "price": "$19.00",
    "images": [
      "https://example.com/a.jpg",
      "https://example.com/b.jpg"
    ],
    "specs": "<table id=\"specs\">...</table>"
  },
  "truncated": false
}

Get the full HTML

Ask for the html format to get the whole document as the browser has it after scripts have run. Raise max_chars if the page is larger than 1000000 characters: truncated says when it was cut.

curl
curl -s -X POST "$SPHYNX_URL/v1/sessions/$SESSION/extract" \
  -H "Authorization: Bearer $SPHYNX_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.com", "formats": ["html"]}' \
  | jq -r .html > page.html

Errors

The shape is the same as everywhere else: see Errors.

HTTPCodeWhen
400invalid_requestBad selector or XPath, duplicate name, unknown format, unknown field, bad value
401unauthorizedMissing or invalid API key
404session_not_foundNo such session for this key
404tab_not_foundNo such tab
409no_page_loadedThe tab is blank
409session_in_useAnother drive call has been running on this session for more than 3 s
410session_closedThe session has ended
422unsafe_urlurl is not http/https, or is a private, loopback or link-local address
500internalOur fault. Send us the request_id
502navigation_failedDNS failure, connection refused, TLS error
504navigation_timeoutThe page did not reach wait_until in timeout_ms