AI & DATA
Crawl website content,
feed the clean text to your LLM
Set a starting URL and page limit. The crawler follows accessible links and extracts page content for a RAG knowledge base or audit. Check returned pages, missing content and errors before treating the crawl as complete.
This is my Apify Actor. I earn from its usage and may receive a referral commission when you use the Apify links on this page. Check the current price before running it.
Usage-based
Check current Actor pricing
Set Limits
Full Site or Subsection
Markdown
Clean LLM-Ready Output
JS
Rendered By Default
USE CASES
What you can do with this data
Build RAG from Docs Sites
Collect accessible documentation within a bounded crawl. Check coverage and the extracted text before indexing it for product questions.
Content Migration
Migrating CMS? Crawl the old site, extract clean content + structure, re-import into the new system with metadata preserved.
Competitor Content Audit
Collect permitted competitor pages within a defined scope. Compare the covered topics, evidence and unanswered buyer questions.
Internal KB Ingestion
Collect accessible knowledge-base pages and prepare their text for search. Private content requires an authorized access method supported by your workflow.
Site Quality Audit
Crawl own site, score each page (word count, internal links, image count, canonical set). Find thin-content pages for pruning.
LLM Agent Memory
Keep extracted pages as a dated source collection for an agent. Track missing pages and refresh the collection when the underlying content changes.
OUTPUT FIELDS
Data extracted per page
Clean markdown body
Page title
Meta description
Canonical URL
H1–H6 heading hierarchy
Internal links list
External links list
Image URLs + alt text
Open Graph metadata
JSON-LD structured data
Published / modified dates
Word count
HOW IT WORKS
Three steps to structured data
Set start URL + depth
Pass root URL, max pages, and optional include/exclude URL patterns.
Crawl
Actor discovers links, fetches pages in parallel, renders JavaScript, extracts clean content.
Consume
Bulk JSON export for RAG ingestion, or webhook stream for real-time pipelines.
COMPARISON
Why this actor vs alternatives
| Feature | This Actor | Firecrawl | Apify official Web Scraper |
|---|---|---|---|
| Price to compare | Current Actor price plus applicable Apify usage | Current plan credits and options | Current Actor and platform usage |
| Extraction fit | Inspect returned body text | Inspect output on the same URLs | Check parser configuration |
| JavaScript rendering | Test required page behavior | Check current rendering settings | Check current rendering settings |
| Crawl scope | Set a bounded page/depth limit | Check current crawl configuration | Check current Actor configuration |
| URL scope | Configure and verify allowed paths | Verify inclusion and exclusion rules | Verify inclusion and exclusion rules |
| Output fit | Inspect content and metadata | Check required output formats | Check required output formats |
FAQ
Frequently asked questions
How deep can it crawl?
Set a page limit and test a representative subsection first. Our published benchmark covers four five-page attempts across two documentation hosts; it does not establish large-site throughput or completion time.
Does it respect robots.txt?
Yes by default — add config flag to ignore if needed for owned sites. Always confirm you have rights to crawl the target.
What about JavaScript SPAs (React, Vue)?
The Actor uses a headless browser for JavaScript rendering. Test a representative page: authentication, loading behavior and access restrictions can still affect extraction.
Why use this vs Firecrawl?
Choose by the output your workflow needs. Compare successful unique pages, required fields, retry handling, and total run cost on the same allowed targets. Our small published benchmark does not establish a universal price or quality advantage.
Can I crawl only a subsection?
Yes — pass URL patterns (include /docs/* only, exclude /blog/*). Glob and regex supported.
OPEN RESEARCH
Inspect the four-run crawler benchmark
Review page yield, unique URLs, elapsed time, reported usage, timeouts, methodology, and limitations. The underlying observations are available in machine-readable formats.
START NOW
Prepare website content for an LLM knowledge base
Collect accessible pages, inspect coverage, then index the verified text in your RAG workflow.
Open Crawler on Apify→