SPHYNX
esc
  • Loading
Open console

Guides

Read a page

Load a URL and get its content back as Markdown, text or links, in one request.

At the end you have the content of a page as Markdown, read by a real browser after its scripts have run. You need an API key and a running session.

Before you start

The snippets read the base URL and the key from the environment and keep the session ID in SESSION.

Shell
export SPHYNX_URL=http://localhost:8080   # your deployment's base URL
export SPHYNX_API_KEY=sx_live_...

SESSION=$(curl -s -X POST "$SPHYNX_URL/v1/sessions" \
  -H "Authorization: Bearer $SPHYNX_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{}' | jq -r .id)

Read it as Markdown

POST/v1/sessions/{id}/extract with a url navigates there first, then reads the page. markdown is the default format, so formats can be left out.

curl -s -X POST "$SPHYNX_URL/v1/sessions/$SESSION/extract" \
  -H "Authorization: Bearer $SPHYNX_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.com", "formats": ["markdown"]}'

The response has the final address and title of the page and one field per format you asked for.

200
{
  "url": "https://example.com/",
  "title": "Example Domain",
  "markdown": "This domain is for use in documentation examples...",
  "truncated": false
}

Pick the formats

formats takes any of four values, and one call can ask for several. Only the formats you ask for are in the response.

FormatWhat you get
markdownThe main content converted to Markdown
textThe visible text of the page, as document.body.innerText
htmlThe whole document after scripts have run
linksEvery link as text and an absolute href, in document order, duplicates removed. [] when the page has none
curl
curl -s -X POST "$SPHYNX_URL/v1/sessions/$SESSION/extract" \
  -H "Authorization: Bearer $SPHYNX_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.com", "formats": ["text", "links"]}'
200
{
  "url": "https://example.com/",
  "title": "Example Domain",
  "text": "This domain is for use in documentation examples...",
  "links": [
    { "text": "Learn more", "href": "https://iana.org/help/example-domains" }
  ],
  "truncated": false
}

Read the page you are on

Leave url out to read the current page without loading anything. Use it after a navigation, a form submission or a click.

curl
curl -s -X POST "$SPHYNX_URL/v1/sessions/$SESSION/extract" \
  -H "Authorization: Bearer $SPHYNX_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"formats": ["markdown"]}'

Long pages

max_chars caps each format separately. The default is 1000000 and the most is 10000000. When any format hits the cap, truncated is true.

Body
{ "url": "https://example.com", "formats": ["markdown"], "max_chars": 20000 }
  • links is capped as a whole: the list ends before the link that would take its text and URLs past max_chars in total.
  • markdown is made from at most the first 4 MiB of the body's HTML. A longer body also sets truncated.

Slow pages

With a url, extract waits for the load event for up to 30000 ms. Set wait_until to domcontentloaded to read sooner, or raise timeout_ms up to 60000.

Body
{ "url": "https://example.com", "wait_until": "domcontentloaded", "timeout_ms": 45000 }

Check the document status

Extract returns 200 for any page the browser could load, including a site's own 403 or 404 page, and its response has no status field. When the status matters, navigate first: POST/v1/sessions/{id}/navigate returns http_status for the main document. Then extract without a url.

curl
curl -s -X POST "$SPHYNX_URL/v1/sessions/$SESSION/navigate" \
  -H "Authorization: Bearer $SPHYNX_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.com"}'
200
{
  "url": "https://example.com/",
  "title": "Example Domain",
  "http_status": 200,
  "redirected": false
}

What can go wrong

HTTPCodeWhen
400invalid_requestUnknown format, unknown field or a bad value. details.field says where
409no_page_loadedNo url was given and the tab is blank
409session_in_useAnother drive call has been running on this session for more than 3 s. Retry
410session_closedThe session has ended
422unsafe_urlNot http or https, or a private, loopback or link-local address
502navigation_failedDNS failure, connection refused, TLS error
504navigation_timeoutThe page did not reach wait_until in timeout_ms

Next