SPHYNX
esc
  • Loading
Open console

Guides

Extract data

Get the full HTML of a page, or pick out named values with CSS selectors and XPath.

At the end you have a page's full HTML in a file and a JSON object of the values you named: a title, a price, a list of image addresses. You need an API key and a running session.

Before you start

The snippets read the base URL and the key from the environment and keep the session ID in SESSION.

Shell
export SPHYNX_URL=http://localhost:8080   # your deployment's base URL
export SPHYNX_API_KEY=sx_live_...

SESSION=$(curl -s -X POST "$SPHYNX_URL/v1/sessions" \
  -H "Authorization: Bearer $SPHYNX_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{}' | jq -r .id)

The examples read a product page with this markup. It stands in for a page of your own: example.com itself answers 404 at that path.

https://example.com/product/42
<h1>Widget</h1>
<span class="price">$19.00</span>
<img class="gallery" src="/a.jpg" alt="Widget, front">
<img class="gallery" src="/b.jpg" alt="Widget, side">
<table id="specs">
  <tr><th>Stock</th><td>In stock</td></tr>
</table>

Get the full HTML

POST/v1/sessions/{id}/extract with the html format returns the whole document as the browser currently has it: the doctype plus the root element's outer HTML, after scripts have run. With a url it navigates there first.

curl
curl -s -X POST "$SPHYNX_URL/v1/sessions/$SESSION/extract" \
  -H "Authorization: Bearer $SPHYNX_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.com/product/42", "formats": ["html"]}' \
  | jq -r .html > page.html
200
{
  "url": "https://example.com/product/42",
  "title": "Widget",
  "html": "<!DOCTYPE html><html lang=\"en\"><head><title>Widget</title>...",
  "truncated": false
}

Each format is capped at max_chars, 1000000 by default. If truncated is true, raise it. The most is 10000000.

Select values with CSS selectors

select is a list of selections. Each has a name and a selector, and the response puts the text of the first match under that name in selected. With select and no formats, only the selections come back.

curl
curl -s -X POST "$SPHYNX_URL/v1/sessions/$SESSION/extract" \
  -H "Authorization: Bearer $SPHYNX_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "select": [
      {"name": "title", "selector": "h1"},
      {"name": "price", "selector": "span.price"}
    ]
  }'
200
{
  "url": "https://example.com/product/42",
  "title": "Widget",
  "selected": {
    "title": "Widget",
    "price": "$19.00"
  },
  "truncated": false
}

There is no url in this call, so it reads the page the session is already on.

Select with XPath

Use xpath in place of selector when the path is easier to say that way, such as a cell by its position or an element by its text.

curl
curl -s -X POST "$SPHYNX_URL/v1/sessions/$SESSION/extract" \
  -H "Authorization: Bearer $SPHYNX_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "select": [
      {"name": "price", "xpath": "//span[@class=\"price\"]"},
      {"name": "stock", "xpath": "//table[@id=\"specs\"]//tr[1]/td[1]"}
    ]
  }'
200
{
  "url": "https://example.com/product/42",
  "title": "Widget",
  "selected": {
    "price": "$19.00",
    "stock": "In stock"
  },
  "truncated": false
}

A selection takes one of selector or xpath, not both.

Read an attribute

attr chooses what to read from the matched element. The default is text.

ValueWhat you get
textThe element's visible text, trimmed
htmlThe element's inner HTML
outer_htmlThe element with its own tag
any attribute nameThat attribute's value, such as src, href, alt or data-id. href and src come back as absolute URLs
Body
{
  "select": [
    {"name": "image", "selector": "img.gallery", "attr": "src"},
    {"name": "caption", "selector": "img.gallery", "attr": "alt"},
    {"name": "row", "selector": "#specs tr", "attr": "html"}
  ]
}

The selected object in the response:

selected
{
  "image": "https://example.com/a.jpg",
  "caption": "Widget, front",
  "row": "<th>Stock</th><td>In stock</td>"
}

Get every match

A selection returns the first match. Set all to true to get every match as a list, in document order. max_chars caps each list as a whole: it ends before the value that would take it past max_chars in total, and truncated is true.

Body
{
  "select": [
    {"name": "images", "selector": "img.gallery", "attr": "src", "all": true},
    {"name": "captions", "xpath": "//img[@class=\"gallery\"]",
     "attr": "alt", "all": true}
  ]
}

The selected object in the response:

selected
{
  "images": ["https://example.com/a.jpg", "https://example.com/b.jpg"],
  "captions": ["Widget, front", "Widget, side"]
}

Put it together

One call can navigate, return formats and make up to 50 selections.

curl -s -X POST "$SPHYNX_URL/v1/sessions/$SESSION/extract" \
  -H "Authorization: Bearer $SPHYNX_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com/product/42",
    "select": [
      {"name": "title",  "selector": "h1"},
      {"name": "price",  "xpath": "//span[@class=\"price\"]"},
      {"name": "images", "selector": "img.gallery", "attr": "src", "all": true},
      {"name": "specs",  "selector": "#specs", "attr": "outer_html"},
      {"name": "sku",    "selector": "#sku"}
    ]
  }'
200
{
  "url": "https://example.com/product/42",
  "title": "Widget",
  "selected": {
    "title": "Widget",
    "price": "$19.00",
    "images": ["https://example.com/a.jpg", "https://example.com/b.jpg"],
    "specs": "<table id=\"specs\">...</table>",
    "sku": null
  },
  "truncated": false
}

When nothing matches

A selection that matches nothing is not an error. It returns null, as sku does above, or an empty list when all is true. Check for both before you use a value.

Find the selectors

When you do not know the page, POST/v1/sessions/{id}/elements lists its elements with a unique selector and an absolute xpath for each. Either can be used in a selection as it is.

curl
curl -s -X POST "$SPHYNX_URL/v1/sessions/$SESSION/elements" \
  -H "Authorization: Bearer $SPHYNX_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"selector": "img"}'
200
{
  "count": 2,
  "truncated": false,
  "elements": [
    {
      "selector": "html > body > img:nth-of-type(1)",
      "xpath": "/html/body/img[1]",
      "tag": "img",
      "visible": true
    },
    {
      "selector": "html > body > img:nth-of-type(2)",
      "xpath": "/html/body/img[2]",
      "tag": "img",
      "visible": true
    }
  ]
}

What can go wrong

HTTPCodeWhen
400invalid_requestA bad selector or XPath, a duplicate name, an unknown format, or more than 50 selections. details.index names the selection
409no_page_loadedNo url was given and the tab is blank
409session_in_useAnother drive call has been running on this session for more than 3 s. Retry
410session_closedThe session has ended
422unsafe_urlNot http or https, or a private, loopback or link-local address
502navigation_failedDNS failure, connection refused, TLS error
504navigation_timeoutThe page did not reach wait_until in timeout_ms

Next