Guides
Extract data
Get the full HTML of a page, or pick out named values with CSS selectors and XPath.
At the end you have a page's full HTML in a file and a JSON object of the values you named: a title, a price, a list of image addresses. You need an API key and a running session.
Before you start
The snippets read the base URL and the key from the environment and keep the session ID in SESSION.
export SPHYNX_URL=http://localhost:8080 # your deployment's base URL
export SPHYNX_API_KEY=sx_live_...
SESSION=$(curl -s -X POST "$SPHYNX_URL/v1/sessions" \
-H "Authorization: Bearer $SPHYNX_API_KEY" \
-H "Content-Type: application/json" \
-d '{}' | jq -r .id)The examples read a product page with this markup. It stands in for a page of your own: example.com itself answers 404 at that path.
<h1>Widget</h1>
<span class="price">$19.00</span>
<img class="gallery" src="/a.jpg" alt="Widget, front">
<img class="gallery" src="/b.jpg" alt="Widget, side">
<table id="specs">
<tr><th>Stock</th><td>In stock</td></tr>
</table>Get the full HTML
POST/v1/sessions/{id}/extract with the html format returns the whole document as the browser currently has it: the doctype plus the root element's outer HTML, after scripts have run. With a url it navigates there first.
curl -s -X POST "$SPHYNX_URL/v1/sessions/$SESSION/extract" \
-H "Authorization: Bearer $SPHYNX_API_KEY" \
-H "Content-Type: application/json" \
-d '{"url": "https://example.com/product/42", "formats": ["html"]}' \
| jq -r .html > page.html{
"url": "https://example.com/product/42",
"title": "Widget",
"html": "<!DOCTYPE html><html lang=\"en\"><head><title>Widget</title>...",
"truncated": false
}Each format is capped at max_chars, 1000000 by default. If truncated is true, raise it. The most is 10000000.
Select values with CSS selectors
select is a list of selections. Each has a name and a selector, and the response puts the text of the first match under that name in selected. With select and no formats, only the selections come back.
curl -s -X POST "$SPHYNX_URL/v1/sessions/$SESSION/extract" \
-H "Authorization: Bearer $SPHYNX_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"select": [
{"name": "title", "selector": "h1"},
{"name": "price", "selector": "span.price"}
]
}'{
"url": "https://example.com/product/42",
"title": "Widget",
"selected": {
"title": "Widget",
"price": "$19.00"
},
"truncated": false
}There is no url in this call, so it reads the page the session is already on.
Select with XPath
Use xpath in place of selector when the path is easier to say that way, such as a cell by its position or an element by its text.
curl -s -X POST "$SPHYNX_URL/v1/sessions/$SESSION/extract" \
-H "Authorization: Bearer $SPHYNX_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"select": [
{"name": "price", "xpath": "//span[@class=\"price\"]"},
{"name": "stock", "xpath": "//table[@id=\"specs\"]//tr[1]/td[1]"}
]
}'{
"url": "https://example.com/product/42",
"title": "Widget",
"selected": {
"price": "$19.00",
"stock": "In stock"
},
"truncated": false
}A selection takes one of selector or xpath, not both.
Read an attribute
attr chooses what to read from the matched element. The default is text.
| Value | What you get |
|---|---|
text | The element's visible text, trimmed |
html | The element's inner HTML |
outer_html | The element with its own tag |
| any attribute name | That attribute's value, such as src, href, alt or data-id. href and src come back as absolute URLs |
{
"select": [
{"name": "image", "selector": "img.gallery", "attr": "src"},
{"name": "caption", "selector": "img.gallery", "attr": "alt"},
{"name": "row", "selector": "#specs tr", "attr": "html"}
]
}The selected object in the response:
{
"image": "https://example.com/a.jpg",
"caption": "Widget, front",
"row": "<th>Stock</th><td>In stock</td>"
}Get every match
A selection returns the first match. Set all to true to get every match as a list, in document order. max_chars caps each list as a whole: it ends before the value that would take it past max_chars in total, and truncated is true.
{
"select": [
{"name": "images", "selector": "img.gallery", "attr": "src", "all": true},
{"name": "captions", "xpath": "//img[@class=\"gallery\"]",
"attr": "alt", "all": true}
]
}The selected object in the response:
{
"images": ["https://example.com/a.jpg", "https://example.com/b.jpg"],
"captions": ["Widget, front", "Widget, side"]
}Put it together
One call can navigate, return formats and make up to 50 selections.
curl -s -X POST "$SPHYNX_URL/v1/sessions/$SESSION/extract" \
-H "Authorization: Bearer $SPHYNX_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"url": "https://example.com/product/42",
"select": [
{"name": "title", "selector": "h1"},
{"name": "price", "xpath": "//span[@class=\"price\"]"},
{"name": "images", "selector": "img.gallery", "attr": "src", "all": true},
{"name": "specs", "selector": "#specs", "attr": "outer_html"},
{"name": "sku", "selector": "#sku"}
]
}'const res = await fetch(`${process.env.SPHYNX_URL}/v1/sessions/${sessionId}/extract`, {
method: "POST",
headers: {
Authorization: `Bearer ${process.env.SPHYNX_API_KEY}`,
"Content-Type": "application/json",
},
body: JSON.stringify({
url: "https://example.com/product/42",
select: [
{ name: "title", selector: "h1" },
{ name: "price", xpath: '//span[@class="price"]' },
{ name: "images", selector: "img.gallery", attr: "src", all: true },
{ name: "specs", selector: "#specs", attr: "outer_html" },
{ name: "sku", selector: "#sku" },
],
}),
});
const { selected } = await res.json();import os
import requests
res = requests.post(
f"{os.environ['SPHYNX_URL']}/v1/sessions/{session_id}/extract",
headers={"Authorization": f"Bearer {os.environ['SPHYNX_API_KEY']}"},
json={
"url": "https://example.com/product/42",
"select": [
{"name": "title", "selector": "h1"},
{"name": "price", "xpath": '//span[@class="price"]'},
{"name": "images", "selector": "img.gallery", "attr": "src", "all": True},
{"name": "specs", "selector": "#specs", "attr": "outer_html"},
{"name": "sku", "selector": "#sku"},
],
},
timeout=60,
)
selected = res.json()["selected"]{
"url": "https://example.com/product/42",
"title": "Widget",
"selected": {
"title": "Widget",
"price": "$19.00",
"images": ["https://example.com/a.jpg", "https://example.com/b.jpg"],
"specs": "<table id=\"specs\">...</table>",
"sku": null
},
"truncated": false
}When nothing matches
A selection that matches nothing is not an error. It returns null, as sku does above, or an empty list when all is true. Check for both before you use a value.
Find the selectors
When you do not know the page, POST/v1/sessions/{id}/elements lists its elements with a unique selector and an absolute xpath for each. Either can be used in a selection as it is.
curl -s -X POST "$SPHYNX_URL/v1/sessions/$SESSION/elements" \
-H "Authorization: Bearer $SPHYNX_API_KEY" \
-H "Content-Type: application/json" \
-d '{"selector": "img"}'{
"count": 2,
"truncated": false,
"elements": [
{
"selector": "html > body > img:nth-of-type(1)",
"xpath": "/html/body/img[1]",
"tag": "img",
"visible": true
},
{
"selector": "html > body > img:nth-of-type(2)",
"xpath": "/html/body/img[2]",
"tag": "img",
"visible": true
}
]
}What can go wrong
| HTTP | Code | When |
|---|---|---|
| 400 | invalid_request | A bad selector or XPath, a duplicate name, an unknown format, or more than 50 selections. details.index names the selection |
| 409 | no_page_loaded | No url was given and the tab is blank |
| 409 | session_in_use | Another drive call has been running on this session for more than 3 s. Retry |
| 410 | session_closed | The session has ended |
| 422 | unsafe_url | Not http or https, or a private, loopback or link-local address |
| 502 | navigation_failed | DNS failure, connection refused, TLS error |
| 504 | navigation_timeout | The page did not reach wait_until in timeout_ms |
Next
- Extract: every field and default.
- Read a page: Markdown, text and links.
- Elements: list elements by selector, XPath or text.