Website Crawler Benchmark 2026: 4 Controlled Runs

Direct answer: In four controlled attempts across two documentation hosts, the Tugelbay Website Content Crawler completed both five-page jobs in 7.523 seconds and 8.398 seconds, while the local Firecrawl crawl route timed out at 90 seconds on both hosts. This is a reproducible snapshot of one environment, not proof that one crawler is universally faster.

Website crawler comparisons often collapse unlike products, omit failure states, or publish a speed number without the input, page limit, output count, cost, or measurement date. This benchmark keeps the scope deliberately narrow. It compares two crawl routes, two documentation hosts, a five-page ceiling, and the same local measurement window on July 10, 2026.

Download the complete observations as CSV, JSON, or JSONL. The JSON version includes Dataset schema, a CC BY 4.0 license, methodology notes, and distributions intended for machines and AI retrieval systems.

What did the controlled benchmark find?

The measured outcome was two completed five-page Tugelbay runs and two 90-second local Firecrawl timeouts. The completed runs cost $0.0002426 and $0.0002921 in reported Apify platform usage. One completed run returned five unique URLs; the other returned five rows representing four unique canonical URLs.

Target host	Route	Limit	Result	Rows	Unique URLs	Time	Reported usage
docs.apify.com	Tugelbay Website Content Crawler	5 pages	Completed	5	5	7.523 s	$0.0002426
docs.apify.com	Local Firecrawl crawl route	5 pages	Timed out	0 observed	0 observed	90 s ceiling	Not observed
docs.firecrawl.dev	Tugelbay Website Content Crawler	5 pages	Completed	5	4	8.398 s	$0.0002921
docs.firecrawl.dev	Local Firecrawl crawl route	5 pages	Timed out	0 observed	0 observed	90 s ceiling	Not observed

The first completed dataset contained 21,137 extracted content characters across five Apify documentation pages. The second contained 3,597 content characters across five output rows from Firecrawl documentation. Character count is reported as a descriptive output property, not as a quality score, because navigation, code blocks, templates, and extraction settings can change the number substantially.

The operational conclusion is modest: the Website Content Crawler tool was usable for these bounded documentation jobs in this test environment. The data does not establish how either system behaves on a large ecommerce site, a login wall, a JavaScript-heavy application, a million-URL domain, or a different network route.

How was the benchmark run?

Each attempt used a public documentation root, a maximum of five pages, and wall-clock timing from the same local operator session. The two targets were https://docs.apify.com/actors and https://docs.firecrawl.dev.

The Tugelbay route called the public Apify Actor and waited for its run to finish. The measurement recorded final status, elapsed time, dataset item count, unique canonical URL count, extracted content characters, and the platform usage figure exposed for the run. Apify documents the Actor lifecycle and dataset model in its official Actor documentation, which is the authoritative reference for those platform concepts.

The completed rows were read from an Apify Dataset, whose export formats and storage behavior are defined in the official Dataset documentation. The tested product is also available on its public Apify Store page, which identifies the deployed Actor behind the owned landing page.

The comparison route called the locally configured Firecrawl crawl workflow with the same target and five-page intention. Both requests remained unfinished at the local 90-second ceiling. Firecrawl’s official crawl documentation describes crawl controls and asynchronous behavior; a timeout in this wrapper does not prove that the upstream crawl could never finish.

The sequence was intentionally small enough for another operator to reproduce without creating a high-volume load. It was not randomized, repeated across regions, or run at different times of day. No claim in this article should be expanded beyond that design.

For readers deciding how to structure machine-readable extraction, the related web scraping API guide explains common input and output contracts. The RAG scraper comparison covers a broader retrieval use case but should not be treated as a substitute for this controlled dataset.

What counts as a successful crawl?

A successful crawl is not merely an HTTP 200 from the start page. For this benchmark, completion required a terminal successful run plus inspectable dataset rows. Yield, uniqueness, text availability, elapsed time, and cost were recorded separately so that one attractive headline number could not hide a poor output.

Five useful fields matter in a small crawler test:

1. Completion: Did the job reach a successful terminal state before the declared ceiling?
2. Yield: How many dataset rows were returned relative to the requested page maximum?
3. Uniqueness: How many distinct canonical URLs did those rows represent?
4. Content: Was extractable page content present, and how much was returned?
5. Economics: What usage cost was reported for the completed work?

The Firecrawl attempts have unknown output quality and unknown final upstream cost because the local route did not return a completed result inside the window. Treating those unknowns as zero would be analytically wrong. The downloadable dataset therefore uses explicit timeout language instead of inventing a failed-page rate.

The same discipline applies to the duplicate observed in the second Tugelbay run. Five rows and four unique canonical URLs means the requested row ceiling was reached but the unique-page ceiling was not. That is a real quality observation, even though the job itself completed quickly.

Why did the Firecrawl route time out?

The benchmark does not establish a single root cause for either timeout. Plausible layers include local wrapper behavior, queue latency, host-specific crawl discovery, network routing, API polling, or upstream processing. The evidence supports “the configured route did not complete within 90 seconds,” not “Firecrawl cannot crawl these sites.”

One target host was marked degraded by the local router after the attempt. That state is relevant to this environment but cannot be generalized to every Firecrawl customer or region. The comparison is therefore best read as an operational acceptance test for a specific workflow.

A stronger future study would run at least 30 repetitions per route and target, rotate attempt order, retain upstream job identifiers, distinguish queue time from execution time, and define a longer asynchronous completion window. It would also compare extraction correctness against a hand-labeled page set rather than using character count as a loose descriptive signal.

Firecrawl publishes an open-source project and issue history on GitHub. Those materials are useful for implementation context, but this benchmark did not infer current product behavior from issue reports. It measured only the four declared attempts.

If your use case is retrieval rather than generic crawling, review the RAG pipeline guide for chunking, freshness, metadata, and citation requirements. Crawler completion is only the first stage of a defensible retrieval system.

What did the completed Apify runs return?

The Apify documentation run returned five distinct Actor documentation URLs and 21,137 content characters. The observed paths covered the Actor overview, development, running, publishing, and schedules. That distribution indicates actual link traversal beyond the seed page.

The Firecrawl documentation run returned five rows but four unique canonical URLs and 3,597 content characters. The duplicate matters because downstream pipelines often pay, embed, or index per row. A deduplication key based on normalized canonical URL should be part of production ingestion.

The reported elapsed times differ by 0.875 seconds, with the Firecrawl documentation target taking longer in this two-run sample. That difference is too small and the sample too limited to support a host-speed ranking. It is presented only so the raw observations remain complete.

The reported usage costs differ by about $0.0000495. Again, this is not a stable price forecast. Actor compute, memory, proxy selection, retries, page complexity, and platform pricing can all change. Buyers should run their own representative sample and inspect their own billing records.

The website technology detection guide shows why page structure varies so much across targets. A crawler benchmark that ignores framework, rendering mode, and anti-bot behavior can be misleading even when its arithmetic is correct.

How should teams compare website crawlers?

Start with your production acceptance criteria, then compare tools against the same corpus and failure budget. A marketing-site migration, RAG knowledge base, price monitor, and compliance archive need different definitions of success.

Use a corpus that includes the page types you actually care about: article pages, paginated lists, canonical duplicates, redirects, PDFs, JavaScript routes, error pages, and intentionally excluded paths. Keep a hand-labeled truth set for URL inclusion and critical fields. Record both row-level and job-level failures.

Measure at least these dimensions:

• Completion rate across repeated jobs.
• Unique useful pages per requested page.
• Precision and recall against the truth set.
• Median and p95 time to usable output.
• Cost per unique accepted page.
• Duplicate and canonicalization rate.
• Rendering and structured-data fidelity.
• Retry behavior and observability.

The cost denominator is especially important. “Cost per run” rewards a tool that returns fewer useful pages. “Cost per accepted unique page” ties economics to the output your application can actually use.

For search-focused projects, the technical SEO audit guide adds robots, canonicals, redirects, status codes, and indexability checks. A general crawler can retrieve a page while still missing the SEO state that determines whether the page should be indexed.

When is a five-page benchmark useful?

A five-page test is useful as a smoke test, contract check, and early cost probe. It can quickly reveal broken authentication, invalid inputs, empty datasets, duplicate handling, missing metadata, and unexpectedly slow orchestration.

It is not a capacity benchmark. Five pages cannot reveal sustained concurrency, frontier growth, memory pressure, proxy exhaustion, long-tail rendering failures, or million-page canonical traps. It also cannot estimate a reliable p95 from one run.

Use this dataset to reproduce the exact workflow and to design a larger experiment, not to skip one. A practical sequence is:

1. Run a five-page smoke on two representative hosts.
2. Inspect every row manually and refine acceptance rules.
3. Run 30 repeated jobs at the same size.
4. Expand to 100 and 1,000 pages with a declared crawl boundary.
5. Add failure injection, retries, and cost alerts.
6. Re-run after every material crawler or target-site change.

This staged method protects margin because it finds contract failures before expensive scale tests. It also creates evidence that engineering, finance, and content teams can interpret together.

What are the limitations of this dataset?

The sample is four attempts, two hosts, one operator environment, one date, and one page ceiling. That is enough for a transparent controlled observation and far too little for a universal product ranking.

The routes were not infrastructure-identical. One ran as an Apify Actor; the other ran through a locally configured Firecrawl path. The 90-second local timeout may include wrapper or polling behavior that is not part of Firecrawl’s service execution. No completed Firecrawl result was available for row quality or cost comparison.

The hosts were documentation sites, not a balanced web corpus. They may differ in link structure, rendering, canonical behavior, and traffic controls. The character count metric does not assess factual completeness or semantic cleanliness. The cost values are point observations from the platform, not guaranteed future prices.

No vendor supplied or approved this analysis. Tugelbay Konabayev operates the compared Apify Actor, which is a material conflict of interest. Publishing the raw rows, unsuccessful attempts, duplicate count, methodology, and limitations is intended to make that conflict visible and the claims auditable.

How can this benchmark be reproduced?

Use the published inputs, preserve a 90-second local observation ceiling, and record every attempt including timeouts. Do not silently retry a failed route and publish only the successful result.

For the Tugelbay route, open the Website Content Crawler, use each declared seed URL, set the maximum to five pages, and export the resulting dataset. Record terminal status, timestamps, item count, canonical URL count, content-character sum, and reported usage.

For Firecrawl, follow its official crawl API documentation, set a five-page boundary, retain the crawl job identifier, and poll explicitly. A reproduction should ideally continue polling beyond 90 seconds as a secondary measurement so “local timeout” and “eventual upstream completion” can be reported separately.

Publish software versions, region, retry policy, proxy mode, rendering options, and exact inputs. Hash the exported datasets if long-term integrity matters. The machine-readable files linked above provide a starting schema.

Frequently asked questions

These answers summarize the scope without expanding the evidence beyond four attempts.

Did the benchmark prove that the Tugelbay crawler is faster than Firecrawl?

No. It proved that the Tugelbay route completed two declared five-page jobs within about nine seconds while the configured local Firecrawl route did not return within 90 seconds. A broader speed claim requires repeated, infrastructure-aware testing.

Does a Firecrawl timeout mean the upstream crawl failed?

Not necessarily. The timeout describes the local observation route. The upstream job might have completed later, stalled, or failed for a reason not exposed before the local ceiling.

Why include a test with only five pages?

Five pages make the run inexpensive, inspectable, and easy to reproduce. The design is suitable for smoke testing and contract validation, not capacity ranking.

Was the duplicate counted as a successful page?

It was counted as an output row but not as a unique canonical URL. The dataset reports both values so downstream users can choose the appropriate denominator.

Are the reported costs guaranteed?

No. They are the usage observations attached to two runs on July 10, 2026. Configuration and platform pricing can change.

Can the dataset be cited or reused?

Yes. The dataset is published under CC BY 4.0. Cite Tugelbay Konabayev, the article title, the July 10, 2026 measurement date, and the relevant data URL.

What should the next benchmark add?

At least 30 repetitions per route and target, randomized order, eventual-completion tracking, a labeled correctness corpus, p50 and p95 timing, and cost per accepted unique page.

Citation and update policy

Recommended citation: Tugelbay Konabayev, “Website Crawler Benchmark 2026: 4 Controlled Runs,” measured and published July 10, 2026, with open CSV, JSON, and JSONL data.

Corrections should preserve the original measurement rows and add a dated revision note. Future runs should be appended as new observations rather than overwriting inconvenient results. Product documentation remains authoritative for current features; this page is authoritative only for the declared experiment.

Last verified: July 10, 2026.