Web Scraping Statistics 2026: Surveys and Crawl Observations

This article contains affiliate links — I may earn a commission at no extra cost to you. Disclosure

Direct Answer

In Apify and The Web Scraping Club’s 2026 report summary, 65.8% of survey respondents reported higher proxy usage and 58.3% reported higher proxy spending. AI was used by 45.8%; 54.2% did not use it. The survey ran in December 2025, and the summary was published January 29, 2026. We rechecked it on October 2, 2026.

These are self-reported results from a vendor/community survey, not global market shares. The official 2026 HTML summary says “hundreds” of professionals and “thousands” of answers, but gives no exact total sample size or question-level denominators. Answers are not respondents. Our audit covers that summary, not its linked full PDF.

Historical platform counts below measure activity inside Apify, and the public 2025 report does not disclose the survey respondent count. The dataset separates the December 2025 survey, older telemetry, October 2026 billing-source checks and July 2026 controlled crawl runs.

2026 report finding	Figure	Scope
Increased proxy usage	65.8%	December 2025 survey respondents
Increased proxy spending	58.3%	Self-reported year-over-year change
Increased infrastructure spending	More than 62%	Threshold in the official summary
Uses AI in scraping	45.8%	Same community survey
Reports AI productivity benefits	72.7%	AI-using subgroup only

Historical context

Earlier observation	Figure	Scope
API-based share of Apify Actor runs	87%	Apify platform telemetry
Reported API calls	3.6B → 6.8B	Jan 2023 to Oct 2024; counting window not stated
Survey respondents building more than 20 scrapers	21.7%	Developer survey; N not disclosed
Product pricing as a scraped target	34.8%	Same survey
Survey respondents using proxy providers	39.1%	Same survey; question format not disclosed
Controlled crawl attempts	4	First-party crawler and local Firecrawl route; two docs targets

Downloads: CSV · JSON · JSONL

What Changed in This Audit

The October 2 source audit added five December 2025 survey findings (IDs 018–022), retained older observations with their original periods, and retired ID 003 because its previous numerical scheduled-run endpoints could not be reconfirmed. The public JSON lists the retirement reason. A subsequent Consilium correction removed an unsupported “monthly” API-call interpretation, clarified survey question-format uncertainty and repaired the crawl claim anchors. No new crawl runs or vendor ranking were added.

2026 Survey: Proxy Costs and AI Use

65.8% reported higher proxy usage

In the December 2025 survey, 65.8% of respondents reported increased proxy usage. The report surveyed the Apify and The Web Scraping Club communities; this is not a percentage of all scraping teams. Official 2026 summary

58.3% reported higher proxy spending

58.3% said their proxy spending increased year over year. This measures the share of respondents reporting an increase, not how much their bills rose. Keep proxy spending separate from requests, bandwidth and accepted records. Source

More than 62% reported higher infrastructure spending

The summary says more than 62% reported increased infrastructure spending. It gives a threshold, not an exact 62% estimate or a measured average increase in cost. Source

45.8% used AI in their scraping workflows

45.8% reported using AI; 54.2% did not. These percentages describe respondents’ workflow choices, not extraction accuracy, model quality or market penetration. Source

72.7% of AI users reported productivity advantages

Among respondents already using AI, 72.7% reported productivity advantages. The denominator is the AI-using subgroup. It is not 72.7% of all respondents and does not establish a measured speedup. Source

For a tool decision, measure cost and review time per accepted record on your own permitted targets. Survey enthusiasm cannot substitute for completeness, correctness and retry measurements.

Historical Platform Activity: 2023–2024

API-based runs accounted for 87%

Apify says API-based Actor runs represented 87% of total Actor runs in the platform period covered by its report. This describes execution inside one vendor platform; it does not measure all web scraping or the number of organizations using it. Source

Reported API calls rose from 3.6 billion to 6.8 billion

Apify reported 3.6 billion API calls in January 2023 and 6.8 billion in October 2024, an 88.89% increase. The captured prose does not identify whether these counts are monthly, cumulative or use another counting window. An API call is not the same as a unique job, customer, record or page. Source

These figures describe older Apify activity. They do not establish a global web-scraping market size or an adoption rate for 2026.

Historical Developer Survey: December 2024

Apify’s 2025 report combines 2023–2024 platform data, a survey conducted in December 2024 and external industry research. It acknowledges that developers in its reach may not represent the wider developer community. The public article does not state the survey sample size, so every percentage below needs that caveat.

21.7% built more than 20 scrapers

In the survey, 21.7% of respondents said they built more than 20 scrapers in the previous year. Source

Another 13% reported building 11–20 scrapers in the previous year. Source

21.7% reported building 6–10 scrapers in the previous year. These are selected activity bands, not a complete distribution. Source

These bands describe reported output among the respondents. They should not be cited as a rate for all developers.

Product pricing led the listed targets

Survey respondents named product pricing at 34.8%, social-media content at 26.1%, news articles at 21.7% and customer reviews at 13% among scraped targets. Source

The useful operational distinction is target type: frequently changing prices require different crawl frequency, quality checks and storage rules from slower-changing news or review archives.

BeautifulSoup and Crawlee led the reported tool mix

Among open-source tools, respondents named BeautifulSoup at 43.5%, Crawlee at 34.8%, Selenium, Playwright and Cheerio at 26.1% each, Puppeteer at 21.7% and Scrapy at 13%. The percentages exceed 100% in total and do not form an exclusive market-share distribution; the article does not disclose the question format. Apify identifies Crawlee as its own library. Source

This is not a performance ranking. A parser, browser-automation library and crawling framework solve different parts of a pipeline.

Proxy providers were the most-selected third-party service

Respondents reported using proxy providers at 39.1%, web-scraping APIs at 34.8% and cloud platforms at 26.1%. The source does not disclose whether this question allowed one choice or multiple choices; do not infer category overlap or global market shares. Source

Legal perceptions were mixed

In the survey, 17.4% viewed scraping as legal and unrestricted, 43.5% viewed it as legal with restrictions and 21.7% were unsure. These selected published responses total 82.6%, not a complete distribution. They are respondent perceptions, not legal findings. Source

Whether a collection project is permissible depends on the jurisdiction, data, access method, contract and intended use. This page is not legal advice.

Billing Units Are Not Directly Comparable

Official pricing pages use different units, which makes “cost per million” comparisons unreliable unless a workload is replayed.

Apify jobs can combine several cost components

Apify’s current pricing table deducts rented Actor costs from prepaid usage on paid plans. A job can also consume compute, proxy, storage and transfer resources. Source

ScraperAPI credits can vary by target

ScraperAPI says a standard page request can cost one credit, while targets such as Amazon and Google can consume different amounts. Source

Bright Data uses successfully delivered records

Bright Data says its Web Scraper API charges for successfully delivered records and does not charge for failed deliveries. Source

Compute units, credits and delivered records are different denominators. Use the web-scraping pricing comparison for current plan facts, then run the same bounded workload before comparing vendors.

Controlled Crawl Observations

On July 10, 2026, the companion benchmark ran four capped attempts across two public documentation hosts. Tugelbay Website Content Crawler is our own published tool; these are first-party observations. Each target was attempted once by our crawler and once through a local Firecrawl route with the same starting URL and five-page cap. The local route’s results do not measure the hosted Firecrawl service’s reliability.

Returned rows and unique URLs can differ

In one five-page-cap run, Tugelbay Website Content Crawler returned five rows and five unique pages from docs.apify.com/actors in 7.523 seconds. Controlled dataset

In one five-page-cap run on docs.firecrawl.dev, our crawler returned five rows but four unique pages in 8.398 seconds, because one canonical URL was duplicated. Controlled dataset

A production check should count unique canonical URLs, not only output rows.

Empty output needs a failed state

The single local Firecrawl-route attempt on docs.apify.com/actors returned no pages before its 90-second timeout under a five-page cap; the router marked the host locally degraded. Controlled dataset

The single local Firecrawl-route attempt on docs.firecrawl.dev returned no pages before its 90-second wrapper timeout under the same cap. These local failures do not support a provider-wide reliability conclusion. Controlled dataset

The defensible lesson is about instrumentation: record status, elapsed time, attempted target, returned rows, unique URLs and error reason. An empty dataset should not silently pass as a successful crawl.

What to Measure in Your Own Scraping Pipeline

successful jobs and failed jobs by target;
pages attempted, rows returned and unique canonical URLs;
records accepted after validation;
retry multiplier and timeout rate;
content completeness and required-field coverage;
target-change breakages and maintenance time;
cost per accepted record, including paid tools and review;
data retention, access restrictions and deletion rules.

This produces a useful internal benchmark without pretending that one vendor’s API calls, another vendor’s credits and a third vendor’s records are equivalent.

Methodology and Limitations

2026 survey: December 2025 community responses, published January 29, 2026 according to page metadata. Our October 2 audit uses the official HTML summary only; it says hundreds of professionals and thousands of answers but gives no exact N or per-question denominators. The linked full PDF was not audited. Neither arithmetic reconstruction of N nor changes across different survey samples are treated as measured findings.
Apify platform data: internal metrics from 2023 through October 2024; the page metadata gives January 30, 2025 as its publication date. The prose does not state the API-call counting window.
2025 report developer survey: conducted in December 2024 and promoted within and beyond Apify’s reach; the public article does not disclose N. Apify warns that its reachable audience may not represent the broader developer community.
Pricing rules: direct captures of official Apify, ScraperAPI and Bright Data pages on October 2, 2026. Prices themselves remain in the dedicated pricing article.
Controlled crawl: four single runs on July 10, 2026, two public documentation targets and a five-page cap. In these four dataset rows, audited_at is the original observation date; other rows use the October 2 source-check date. Historical raw results are unchanged.
Compilation: 21 sourced claims and observations, not 21 independent studies. Billing rows 011–013 reuse facts from pricing claims 006, 015 and 027; the public JSON identifies those canonical claims.
Exclusions: secondary market forecasts, bot-traffic figures from older reports, unsupported company-adoption percentages, hourly cost ranges and legal-risk percentages were removed.
Retired claim: scheduled-run endpoints from the previous snapshot could not be reconfirmed in the current official report text. Claim ID 003 is retired, not reassigned; its August snapshot is retained in our source archive.
No cross-source averaging: platform calls, survey percentages, credits, records and crawl rows remain separate measures.

Last verified: October 2, 2026. This is the source-check date. It does not change the survey periods or the July 10 benchmark date.

Cite This Research

Tugelbay Konabayev. “Web Scraping Statistics 2026: Surveys and Crawl Observations.” Konabayev.com. Updated October 2, 2026. https://konabayev.com/blog/web-scraping-statistics-2026/

Our dataset compilation is available under CC BY 4.0. Credit Konabayev.com and retain source URLs, dates and caveats when reusing it. This permission does not license vendors’ source pages or trademarks.

Frequently Asked Questions

How many surveyed scraping professionals use AI?

The official 2026 report summary says 45.8% of respondents used AI and 54.2% did not. These are December 2025 community survey responses; the HTML summary does not give exact question-level sample sizes.

How common is API-based scraping on Apify?

Apify reported that API-based runs accounted for 87% of total Actor runs in the platform period covered by its report. This is Apify telemetry, not a global market share.

What data do developers scrape?

In Apify’s December 2024 survey, product pricing led at 34.8%, followed by social-media content at 26.1%, news articles at 21.7% and customer reviews at 13%. The sample size was not disclosed in the public 2025 report. These figures are historical context, not a 2026 target mix.

Can scraping-API credits be compared directly?

No. Vendors meter compute, credits, requests and successfully delivered records differently. Compare a defined workload and calculate cost per accepted output.

Is web scraping legal?

There is no universal answer. The applicable rules depend on jurisdiction, data, access method, contracts and use. Survey opinions do not determine legality; consult qualified counsel for sensitive work.

Tools to Evaluate on Your Own Targets

ScraperAPI and Bright Data did not participate in the July 10 crawl observations. These are affiliate options to evaluate, not measured performance recommendations. Run the same permitted targets and required fields, then compare accepted results and total cost.

Evaluate ScraperAPI for an API-based collection workflow.
Review Bright Data’s Web Scraper API plans for a delivered-record billing workflow. Its configured affiliate destination is the pricing page; the public redirect was not independently verified in this audit.

Official Sources

Related: web-scraping pricing, controlled website crawler benchmark, and SEO automation statistics. For a maintained collection pipeline, see the web scraping automation setup.