API reference
Extract
Read the current page as Markdown, text, HTML or links, or pick out named values.
Extract reads the page the tab is on. Pass url to navigate first, so one call loads a page and reads it. Ask for whole-page formats, for named selections, or for both.
Extract
POST/v1/sessions/{id}/extractEvery field is optional: an empty object returns the current page as Markdown.
Path
- idstringrequired
The session to drive.
Body
- urlstringdefaultstay on the current page
Navigate here first.
- formatslist of markdown, text, html, links
The whole-page formats to return. The default is
["markdown"], or nothing whenselectis given. - selectlist of selections, ≤50
Named values to pick out of the page.
- namestringrequired
The key of this value in
selected. Names must be unique. - selectorstring
A CSS selector. Give exactly one of
selectorandxpath. - xpathstring
An XPath.
- attrstringdefaulttext
What to read from the match:
text,html,outer_html, or any attribute name such assrc. - allbooldefaultfalse
falsereturns the first match as a string, ornullwhen nothing matches.truereturns every match as a list. The list ends before the value that would take it pastmax_charsin total.
- max_charsint, ≤10000000default1000000
Applied to each format, to
linksas a whole and to eachall: truelist as a whole. - wait_untilload or domcontentloadeddefaultload
Only with
url. As in Navigate. - timeout_msint, ≤60000default30000
Only with
url. As in Navigate. - tab_idstringdefaultthe active tab
The tab to read.
Response
- urlstring
The address of the page that was read.
- titlestring
The title of that page.
- markdownstring
The main content converted to Markdown. Made from at most the first 4 MiB (4,194,304 UTF-16 units) of the body's HTML.
- textstring
document.body.innerText. - htmlstring
The whole document as the browser currently has it: the doctype plus
documentElement.outerHTML, after scripts have run. - linkslist
Every link in document order, duplicates removed.
[]when the page has none. The list ends before the link that would take its text and URLs pastmax_charsin total.- textstring
The link text.
- hrefstring
The absolute URL.
- selectedobject
One key per selection. Each value is a string,
null, or a list of strings. - truncatedbool
trueif a format,links, a selected value or anall: truelist hitmax_chars, ormarkdownwas made from a body cut at 4 MiB.
Only the formats you asked for are present.
curl -s -X POST "$SPHYNX_URL/v1/sessions/$SESSION/extract" \
-H "Authorization: Bearer $SPHYNX_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"url": "https://example.com",
"formats": ["markdown", "links"]
}'const res = await fetch(`${process.env.SPHYNX_URL}/v1/sessions/${sessionId}/extract`, {
method: "POST",
headers: {
Authorization: `Bearer ${process.env.SPHYNX_API_KEY}`,
"Content-Type": "application/json",
},
body: JSON.stringify({
url: "https://example.com",
formats: ["markdown", "links"],
}),
});
const { markdown, links } = await res.json();import os
import requests
res = requests.post(
f"{os.environ['SPHYNX_URL']}/v1/sessions/{session_id}/extract",
headers={"Authorization": f"Bearer {os.environ['SPHYNX_API_KEY']}"},
json={
"url": "https://example.com",
"formats": ["markdown", "links"],
},
timeout=60,
)
page = res.json(){
"url": "https://example.com/",
"title": "Example Domain",
"markdown": "This domain is for use in ...",
"links": [
{
"text": "Learn more",
"href": "https://iana.org/help/example-domains"
}
],
"truncated": false
}Select named values
select picks specific values out of the page with CSS selectors or XPath. Each selection has a name, and the response carries one key per name under selected.
- With
allleft off, the value is the first match as a string, ornullwhen nothing matches. - With
all: true, the value is a list with every match. The list ends before the value that would take it pastmax_charsin total. attrdecides what is read from each match: its text, its innerhtml, itsouter_html, or one attribute.
When select is given and formats is not, no whole-page format is returned.
curl -s -X POST "$SPHYNX_URL/v1/sessions/$SESSION/extract" \
-H "Authorization: Bearer $SPHYNX_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"url": "https://example.com/product/42",
"select": [
{"name": "title", "selector": "h1"},
{"name": "price",
"xpath": "//span[@class=\"price\"]"},
{"name": "images", "selector": "img.gallery",
"attr": "src", "all": true},
{"name": "specs", "selector": "#specs",
"attr": "outer_html"}
]
}'{
"url": "https://example.com/product/42",
"title": "Widget",
"selected": {
"title": "Widget",
"price": "$19.00",
"images": [
"https://example.com/a.jpg",
"https://example.com/b.jpg"
],
"specs": "<table id=\"specs\">...</table>"
},
"truncated": false
}Get the full HTML
Ask for the html format to get the whole document as the browser has it after scripts have run. Raise max_chars if the page is larger than 1000000 characters: truncated says when it was cut.
curl -s -X POST "$SPHYNX_URL/v1/sessions/$SESSION/extract" \
-H "Authorization: Bearer $SPHYNX_API_KEY" \
-H "Content-Type: application/json" \
-d '{"url": "https://example.com", "formats": ["html"]}' \
| jq -r .html > page.htmlErrors
The shape is the same as everywhere else: see Errors.
| HTTP | Code | When |
|---|---|---|
| 400 | invalid_request | Bad selector or XPath, duplicate name, unknown format, unknown field, bad value |
| 401 | unauthorized | Missing or invalid API key |
| 404 | session_not_found | No such session for this key |
| 404 | tab_not_found | No such tab |
| 409 | no_page_loaded | The tab is blank |
| 409 | session_in_use | Another drive call has been running on this session for more than 3 s |
| 410 | session_closed | The session has ended |
| 422 | unsafe_url | url is not http/https, or is a private, loopback or link-local address |
| 500 | internal | Our fault. Send us the request_id |
| 502 | navigation_failed | DNS failure, connection refused, TLS error |
| 504 | navigation_timeout | The page did not reach wait_until in timeout_ms |