Vibe Coding Statistics 2026: Security Benchmarks

Direct Answer: Is Vibe Coding Safe?

Current repository-level evidence says functional success is not a security guarantee. On the 200-task SusVibes benchmark, SWE-agent with Claude 4 Sonnet passed functional tests on 61% of tasks but produced implementations that were both functional and secure on only 10.5%. Among its functionally correct solutions, 82.8% were insecure.

The result is benchmark-specific, not an estimate that 82.8% of all vibe-coded software is vulnerable. SusVibes deliberately selects real repositories and feature tasks linked to known vulnerability-fixing commits. It measures whether an agent can implement a feature and pass both functional and security tests on the first attempt.

This 2026 edition publishes 24 source-backed claims: 19 SusVibes benchmark and methodology rows plus five separate GoodVibe mitigation rows. The studies use different models, languages, tasks, prompts and security definitions, so their scores remain in separate tables.

Cite This Report

Canonical URL: https://konabayev.com/blog/vibe-coding-statistics/

Recommended citation: Tugelbay Konabayev, “Vibe Coding Statistics 2026: Security Benchmarks,” Konabayev.com, July 11, 2026, https://konabayev.com/blog/vibe-coding-statistics/

Machine-readable versions:

For a single result, cite its stable fragment, for example, https://konabayev.com/blog/vibe-coding-statistics/#vibe-008.

Most Citable Vibe Coding Statistics

The strongest citation units pair functional and security results from the same task set. Do not quote one side of the pair without naming SusVibes and its repository-level scope.

Claim	Finding	Correct interpretation
VIBE-001	SusVibes contains 200 repository tasks.	Benchmark scope
VIBE-002	It covers 77 CWE categories.	Vulnerability diversity
VIBE-008	SWE-agent + Claude 4: 61.0% FuncPass / 10.5% SecPass.	Same tasks and run conditions
VIBE-009	OpenHands + Claude 4: 49.5% / 12.5%.	Lower function score, higher joint secure score
VIBE-017	82.8% of functionally correct SWE-agent + Claude 4 solutions were insecure.	Conditional on functional success
VIBE-018	OpenHands + Claude 4 conditional insecure share: 74.7%.	Conditional on functional success
VIBE-019	Preliminary security prompts reduced functional correctness by about 6 points.	Prompting tradeoff
VIBE-020	GoodVibe: 87.5% C++ / 76.0% Java security performance.	Separate mitigation experiment
VIBE-023	GoodVibe: 53.6% Swift / 54.0% Go average safe-response rate.	Cross-language experiment
VIBE-024	GoodVibe reported a 1.03% average utility drop.	Experimental adaptation result

What SusVibes Measures

SusVibes tests whether an agent can add a real repository feature without recreating a known vulnerability. Its construction makes the security question observable rather than relying on a reviewer opinion.

SusVibes reconstructs feature requests from vulnerability-fixing commits. The vulnerable feature is removed, an agent implements it, and the output is evaluated with functional tests and security tests derived from the fix. The benchmark uses pass@1 to approximate a workflow in which the user accepts one generated implementation.

FuncPass is the percentage of all tasks with a functionally correct implementation. SecPass is the percentage of all tasks with an implementation that passes both functional and security tests. SecPass is not “the percentage of correct code that is secure”; that conditional rate must be calculated separately.

Claim	Benchmark property	Value
VIBE-003	Real-world application domains	10
VIBE-004	Mean repository context	162K lines
VIBE-005	Mean repository context	867 files
VIBE-006	Mean functional tests per task	68.8
VIBE-007	Mean security tests per task	3.7

This scope is harder than isolated function-completion benchmarks. An agent must navigate a repository, identify relevant files, implement a feature and avoid recreating a known weakness. It is also intentionally security-heavy, so the result should not be projected onto a random sample of everyday prototypes.

Functional Correctness and Security by Agent

Model and harness must be reported together because the framework changes the observed result. The same backbone can score differently under SWE-agent, OpenHands and Claude Code.

No tested model-framework combination achieved a SecPass above 12.5%. Claude 4 Sonnet was generally strongest on functionality, but the framework choice changed security outcomes. OpenHands with Claude 4 had lower FuncPass than SWE-agent but the highest SecPass in the table.

Claim	Framework	Backbone model	FuncPass	SecPass
VIBE-010	Claude Code	Claude 4 Sonnet	44.0%	6.0%
VIBE-011	SWE-agent	Kimi K2	22.5%	6.0%
VIBE-012	OpenHands	Kimi K2	37.0%	9.0%
VIBE-013	Claude Code	Kimi K2	43.5%	8.0%
VIBE-014	SWE-agent	Gemini 2.5 Pro	19.5%	7.0%
VIBE-015	OpenHands	Gemini 2.5 Pro	21.5%	8.5%
VIBE-016	Claude Code	Gemini 2.5 Pro	15.0%	4.5%

The table is evidence against using a single model leaderboard as a software-security policy. An agent framework controls repository exploration, tool use, feedback loops and stopping behavior. Those choices can change which tasks are completed and which vulnerabilities survive.

Why “The Tests Passed” Is Not Enough

A functional test demonstrates expected behavior under tested inputs; it does not establish secure behavior under hostile inputs. Accepting generated code after one green test suite leaves important boundary conditions unmeasured.

Functional tests normally encode expected behavior. They may not test authorization boundaries, path traversal, injection, secret handling, cryptography, resource exhaustion or malicious inputs. In SusVibes, the large gap between FuncPass and SecPass exists precisely because a feature can behave correctly under normal tests while preserving an exploitable weakness.

For production use, treat generated code as untrusted change output:

Run functional and regression tests.
Add security tests for inputs, identities and boundaries touched by the change.
Run language-appropriate static analysis and dependency checks.
Review data flows, authentication, authorization and secret handling manually.
Keep the generated diff small enough for a human to understand.
Use a sandbox for agent commands and prevent unrestricted production credentials.

These controls do not guarantee safety, but they change the acceptance criterion from “works once” to “passes defined functional and security gates.”

Can Security Prompting Fix Vibe Coding?

Prompting helped in parts of the experiments but did not erase the functionality-security tradeoff. It should be treated as one control in a layered release process.

SusVibes tested generic security guidance, self-selected vulnerability hints and oracle vulnerability information. The strategies improved some security outcomes but reduced functional correctness by about six percentage points. The paper did not find a prompt-only solution that removed the gap.

GoodVibe studies a different intervention: targeted model adaptation using security-relevant neurons. Its results should not be combined numerically with SusVibes because the languages, tasks, models and judge pipeline differ.

Claim	GoodVibe comparison	C++ average	Java average
VIBE-021	Standard-prompt baseline	35.1%	59.3%
VIBE-022	Baseline with secure-prompt prefix	51.57%	75.03%

The secure prefix helped the baseline, but model-level adaptation produced different results. This supports layered controls rather than relying on one carefully worded prompt. Prompting, model choice, agent harness, tests, scanners and review address different failure modes.

A Practical Vibe Coding Security Scorecard

A production scorecard should record evidence about the change, not confidence in the model brand. Model reputation cannot substitute for tests tied to the repository and threat model.

Track four classes of evidence for each generated change:

Layer	Minimum evidence
Functionality	Unit, integration and regression tests tied to requirements
Security	Abuse cases, authorization tests, input validation and secret checks
Change risk	Files touched, dependency changes, privilege changes and exposed interfaces
Review	Human reviewer, unresolved warnings, provenance and deployment approval

Do not turn benchmark percentages into a universal risk score for your repository. Build an internal baseline instead: sample generated changes, classify vulnerabilities consistently, record model and harness versions, and compare against human-authored changes of similar scope.

For adjacent evidence, see AI code assistant statistics for adoption and productivity, and what vibe coding means in marketing for non-production prototyping use cases.

Interpret the Benchmark Without Overclaiming

The 82.8% conditional insecure rate is a result for one model-framework pairing on a vulnerability-focused benchmark, not a population estimate for all generated code. Three boundaries must remain attached whenever the number is cited.

First, the task population is selected from real vulnerability-fixing commits. That gives each task a known security oracle, but it makes the set intentionally harder and more security-relevant than a random collection of scripts. Second, the benchmark is repository-level Python work. Results may differ for a small front-end prototype, a Java service, infrastructure code or a mobile application. Third, pass@1 evaluates the first accepted implementation. A workflow with iterative review and security tests can catch defects that the first run leaves behind.

The converse mistake is also important: benchmark specificity does not make the gap irrelevant. These tasks show that an implementation can satisfy extensive functional tests and still fail security tests derived from an actual vulnerability repair. That is a concrete failure mode for any team that treats “the app works” as the release gate.

When comparing agent systems, keep the exact model version, framework, maximum steps, system prompt, tools and repository state. A newer model inside a different harness is a new experimental condition. Avoid a trend chart unless the benchmark version and evaluation pipeline remain stable.

Production Release Gate for Generated Code

Generated code should move through the same release controls as untrusted third-party code, with extra provenance and tool-permission checks. A practical gate can stay lightweight while still covering the failures the benchmark exposes.

At intake, record which model and agent created the diff, the prompt or task specification, tools invoked, files changed and commands executed. Prevent the agent from reading production secrets or deploying without a separate approval. Review dependency additions independently because a working feature can introduce supply-chain risk.

At verification, add tests for invalid inputs, authorization boundaries, tenant isolation, path handling, serialization, rate limits and error cases relevant to the changed surface. Run static analysis, secret scanning and dependency auditing. The site’s AI agent cost benchmarks can help estimate the operating side of agent workflows, but cost savings should never remove a required security gate.

At review, ask a human to explain the new data flow and trust boundaries without relying on the generated explanation. If the reviewer cannot state who controls each input, which identity authorizes each action and where sensitive data can leave the system, the change is not ready.

At deployment, use a preview or isolated environment, record observable success and failure signals, and retain rollback. A controlled benchmark such as the website crawler evaluation illustrates the broader principle: lock the task, environment and scoring rules before comparing systems.

What a Better Internal Experiment Looks Like

Teams can build a repository-specific baseline without pretending to reproduce an academic benchmark. Sample comparable features over a fixed period and divide them into AI-assisted and conventional cohorts. Match by size, language, subsystem and risk class.

For each change, record escaped defects, security-review findings, test additions, review time, cycle time, rollback rate and post-release incidents. Use one vulnerability taxonomy and blinded review where practical. Do not compare a generated authentication feature with a human-authored documentation change.

The goal is not to prove that one authoring method is universally superior. It is to identify where the agent improves throughput, where review cost shifts, and which classes of error require stronger controls. Preserve model and harness versions because an upgrade can change both capability and risk.

An internal result should also separate detection from prevention. More findings in an AI-assisted cohort can mean worse code, better scrutiny, or both. Track which issues existed before review, which controls detected them and which reached production. That structure turns a benchmark into an engineering decision tool rather than a marketing claim.

Methodology and Limitations

The claims were extracted from the public arXiv HTML for SusVibes version 1 and GoodVibe version 2. Results stay attached to the exact benchmark, model family and agent framework. No secondary survey claim was used in the dataset.

Both sources are preprints. SusVibes is deliberately constructed from vulnerability-related repositories rather than a representative sample of all software. GoodVibe evaluates an experimental adaptation method with its own judge pipeline. Absolute security scores can vary with the judge model, and neither paper proves that a deployed application is secure.

FAQ

What percentage of vibe-coded applications are insecure?

There is no defensible universal percentage. In SusVibes, 82.8% of functionally correct SWE-agent and Claude 4 Sonnet solutions were insecure, but the benchmark deliberately targets known vulnerability classes.

Which coding agent was most secure?

It depends on the definition. OpenHands with Claude 4 Sonnet had the highest joint SecPass in the main table at 12.5%. Other comparisons on shared solved subsets produced different rankings.

Does passing unit tests mean AI-generated code is safe?

No. Unit tests normally verify expected functionality, while security tests exercise abuse cases and boundaries. SusVibes found large gaps between functional and joint functional-security success.

Can a security prompt solve the problem?

Not by itself in the evaluated studies. Secure prompting improved some scores but introduced tradeoffs and remained dependent on prompt wording and execution conditions.

Should companies ban vibe coding?

The evidence supports risk-based controls rather than one rule for every environment. A disposable internal prototype and an internet-facing authentication service should not share the same acceptance process.

How often should vibe coding benchmarks be updated?

Review the evidence when a new repository-level benchmark, major agent framework or model generation appears. Preserve historical rows because changing tasks or judges can move scores without representing a real time-series improvement.

Sources

Last verified: July 11, 2026.