Web Fetcher

Web Fetch allows Casper to open a live website, read its content and build structured context from multiple pages so it can answer questions using current information directly from the source. It can: Fetch and read live website content Crawl multiple internal pages from a site Use sitemaps for faster page discovery Fall back to internal link discovery when no sitemap is available Prioritise useful pages such as About, Services, Products, Pricing, Features, Contact, Blog and Portfolio Extract page titles, meta descriptions and H1 headings Extract website navigation Clean raw HTML into readable text Remove scripts, styles, forms, navigation and other page noise Follow internal links across multiple crawl depths Deduplicate repeated boilerplate content across pages Return the pages crawled and their source URLs Build structured website context for Casper to analyse The current crawler supports up to 8 additional pages, with crawl depth up to 2 levels, while preserving a larger content allowance for the homepage.