Guides
Read a page
Load a URL and get its content back as Markdown, text or links, in one request.
At the end you have the content of a page as Markdown, read by a real browser after its scripts have run. You need an API key and a running session.
Before you start
The snippets read the base URL and the key from the environment and keep the session ID in SESSION.
export SPHYNX_URL=http://localhost:8080 # your deployment's base URL
export SPHYNX_API_KEY=sx_live_...
SESSION=$(curl -s -X POST "$SPHYNX_URL/v1/sessions" \
-H "Authorization: Bearer $SPHYNX_API_KEY" \
-H "Content-Type: application/json" \
-d '{}' | jq -r .id)Read it as Markdown
POST/v1/sessions/{id}/extract with a url navigates there first, then reads the page. markdown is the default format, so formats can be left out.
curl -s -X POST "$SPHYNX_URL/v1/sessions/$SESSION/extract" \
-H "Authorization: Bearer $SPHYNX_API_KEY" \
-H "Content-Type: application/json" \
-d '{"url": "https://example.com", "formats": ["markdown"]}'const res = await fetch(`${process.env.SPHYNX_URL}/v1/sessions/${sessionId}/extract`, {
method: "POST",
headers: {
Authorization: `Bearer ${process.env.SPHYNX_API_KEY}`,
"Content-Type": "application/json",
},
body: JSON.stringify({ url: "https://example.com", formats: ["markdown"] }),
});
const { markdown } = await res.json();import os
import requests
res = requests.post(
f"{os.environ['SPHYNX_URL']}/v1/sessions/{session_id}/extract",
headers={"Authorization": f"Bearer {os.environ['SPHYNX_API_KEY']}"},
json={"url": "https://example.com", "formats": ["markdown"]},
timeout=60,
)
markdown = res.json()["markdown"]The response has the final address and title of the page and one field per format you asked for.
{
"url": "https://example.com/",
"title": "Example Domain",
"markdown": "This domain is for use in documentation examples...",
"truncated": false
}Pick the formats
formats takes any of four values, and one call can ask for several. Only the formats you ask for are in the response.
| Format | What you get |
|---|---|
markdown | The main content converted to Markdown |
text | The visible text of the page, as document.body.innerText |
html | The whole document after scripts have run |
links | Every link as text and an absolute href, in document order, duplicates removed. [] when the page has none |
curl -s -X POST "$SPHYNX_URL/v1/sessions/$SESSION/extract" \
-H "Authorization: Bearer $SPHYNX_API_KEY" \
-H "Content-Type: application/json" \
-d '{"url": "https://example.com", "formats": ["text", "links"]}'{
"url": "https://example.com/",
"title": "Example Domain",
"text": "This domain is for use in documentation examples...",
"links": [
{ "text": "Learn more", "href": "https://iana.org/help/example-domains" }
],
"truncated": false
}Read the page you are on
Leave url out to read the current page without loading anything. Use it after a navigation, a form submission or a click.
curl -s -X POST "$SPHYNX_URL/v1/sessions/$SESSION/extract" \
-H "Authorization: Bearer $SPHYNX_API_KEY" \
-H "Content-Type: application/json" \
-d '{"formats": ["markdown"]}'Long pages
max_chars caps each format separately. The default is 1000000 and the most is 10000000. When any format hits the cap, truncated is true.
{ "url": "https://example.com", "formats": ["markdown"], "max_chars": 20000 }linksis capped as a whole: the list ends before the link that would take its text and URLs pastmax_charsin total.markdownis made from at most the first 4 MiB of the body's HTML. A longer body also setstruncated.
Slow pages
With a url, extract waits for the load event for up to 30000 ms. Set wait_until to domcontentloaded to read sooner, or raise timeout_ms up to 60000.
{ "url": "https://example.com", "wait_until": "domcontentloaded", "timeout_ms": 45000 }Check the document status
Extract returns 200 for any page the browser could load, including a site's own 403 or 404 page, and its response has no status field. When the status matters, navigate first: POST/v1/sessions/{id}/navigate returns http_status for the main document. Then extract without a url.
curl -s -X POST "$SPHYNX_URL/v1/sessions/$SESSION/navigate" \
-H "Authorization: Bearer $SPHYNX_API_KEY" \
-H "Content-Type: application/json" \
-d '{"url": "https://example.com"}'{
"url": "https://example.com/",
"title": "Example Domain",
"http_status": 200,
"redirected": false
}What can go wrong
| HTTP | Code | When |
|---|---|---|
| 400 | invalid_request | Unknown format, unknown field or a bad value. details.field says where |
| 409 | no_page_loaded | No url was given and the tab is blank |
| 409 | session_in_use | Another drive call has been running on this session for more than 3 s. Retry |
| 410 | session_closed | The session has ended |
| 422 | unsafe_url | Not http or https, or a private, loopback or link-local address |
| 502 | navigation_failed | DNS failure, connection refused, TLS error |
| 504 | navigation_timeout | The page did not reach wait_until in timeout_ms |
Next
- Extract: every field and default.
- Extract data: pick out named values with CSS selectors and XPath.
- Fill a form: get past a login before you read.