SEO site crawler that renders JavaScript and maps your internal links

XeoRank's SEO site crawler starts at the URL you give it, usually your homepage, follows your internal links and reads your sitemap, and records what a search engine would find on every page it reaches. It can render JavaScript pages in headless Chromium when they need it, then reports broken pages, redirect chains and duplicate content, and feeds every page into the technical SEO audit.

This page covers how the crawler works, what it reports, how your AI assistant can act on the results, and what it does not do.

Who it is for

• Teams on JavaScript frameworks such as React or Next.js who need to know whether their content is in the HTML or only appears after scripts run.
• Agencies taking on a new client site and needing a full picture before planning work.
• Anyone checking a site before and after a redesign, a migration or a URL change, to catch broken pages and redirect chains.
• People who would rather ask an AI assistant for crawl results than learn a desktop crawler.

How the crawl works

The crawler begins at the site's start URL and works outwards through internal links. It also reads your XML sitemap, so pages that are listed there but not linked from anywhere still get visited, and the audit overview counts them among pages with no inbound links. Every URL the crawl discovered, including sitemap URLs it did not reach, is listed with its status in the crawl report's URLs found view, so you can see what was missed and check whether those URLs are blocked, broken or simply out of date.

Its plain requests identify as XeoRank-Crawler/1.0, and it always obeys robots.txt. It spaces out its requests to your server rather than hitting it all at once. If you use a firewall or bot protection that blocks unknown crawlers, allow that user agent before you start. Pages that need JavaScript rendering, and pages that answer 401 or 403, are fetched again through a headless browser that does not send that user agent.

For each page it records the HTTP status, any redirects, the canonical tag, robots directives, the title, meta description and headings, structured data, images and every internal link with its anchor text.

JavaScript rendering

Many modern sites send a near-empty HTML page and build the content in the browser. A crawler that only reads the raw HTML sees almost nothing on those pages.

Crawls started from the dashboard can render JavaScript, and in those crawls XeoRank checks each page. If the text is missing from the HTML, or a framework page has no structured data in the raw markup, it renders the page in headless Chromium and reads what appears after scripts run. Crawls your assistant starts through MCP read the raw HTML. The crawler also reports the gap when many URLs serve the same template HTML because their content loads with JavaScript, since the content you see in the browser is not in the HTML that crawlers receive first.

What the crawl reports

Each page then runs through the audit's on-page, technical and content-quality rules and its existing structured data is validated; confirmed issues land in the site's fix queue.

Acting on the results with your AI assistant

Connect the XeoRank MCP server to Claude, ChatGPT, Codex, Cursor or VS Code and you can work from the crawl in plain language:

"Crawl example.com, then list the broken pages and the redirect chains. For each broken page, tell me which page links to it and which live page the link should point to instead."

The assistant calls start_crawl, reads the results with get_crawl_results, and answers. On a WordPress site with the free XeoRank Connector plugin, it can then make the changes you agree on: replace an outdated URL written in a page's text and HTML widgets with find and replace (Elementor pages), update menu items, fix titles and canonical URLs through Yoast, Rank Math or All in One SEO, or add schema. Writes are off for each site until you turn them on, new pages and page clones are created as drafts, edits to an existing page are written to that page, and every change to a page's content keeps a snapshot you can restore.

From the command line, the xyle CLI (npm package @xyleapp/cli) has a site-crawl command, and the REST API is documented in the docs.

You or your agent start each crawl. Running one after each release or migration catches new broken pages before they pile up.

What it does not do

• It does not suggest internal links. It finds pages without internal links, and the audit overview counts pages with no inbound links, but it does not recommend which page should link to which. When you plan a new article, the content brief lists your own ranking pages as internal-link targets.
• It does not ignore robots.txt or pretend to be Googlebot. Pages your robots.txt blocks are not crawled.
• It does not crawl pages behind a login.
• It does not analyse server logs, so it cannot tell you how often Googlebot visits. Search Console is the source for Google's view of your site, and XeoRank can sync it.
• It does not change your site on its own. Every change is a tool call your assistant makes, and your AI client asks you to allow it first.

Pricing

Starter is $19 a month for one site, with a 14-day free trial and no card needed. Pro is $49 a month and Growth is $109 a month. See pricing for what each plan includes.