slug: vibe-coding-statistics

value: 200 tasks

claim: SusVibes contains 200 real-world repository-level feature tasks.

source_title: Is Vibe Coding Safe? Benchmarking Vulnerability of Agent-Generated Code in Real-World Tasks

population: SusVibes benchmark of 200 Python repository-level feature tasks covering 77 CWE categories

denominator: 200 SusVibes tasks for the main pass@1 results unless a claim states a conditional functionally-correct subset

period: arXiv version 1, December 2025

geography: not applicable; open-source software repositories

caveat: Preprint results are benchmark-specific pass@1 outcomes. Agent framework, model, prompt, task set, runtime, and security-test definitions prevent direct generalization to all vibe-coded software.

evidence: Abstract, benchmark overview, Tables 2 and 3, and Section 4 results

benchmark_family: SusVibes

evaluation_type: paired functional and security benchmark


value: 77 CWEs

claim: The benchmark covers 77 Common Weakness Enumeration categories.


value: 10 domains

claim: SusVibes spans 10 real-world application domains.


value: 162K lines

claim: The mean repository context contained 162,000 lines of code.


value: 867 files

claim: The mean repository context contained 867 files.


claim: A SusVibes task had 68.8 functional test cases on average.


claim: A SusVibes task had 3.7 security test cases on average.


claim: SWE-agent with Claude 4 Sonnet achieved 61.0% FuncPass and 10.5% SecPass.


claim: OpenHands with Claude 4 Sonnet achieved 49.5% FuncPass and 12.5% SecPass.


claim: Claude Code with Claude 4 Sonnet achieved 44.0% FuncPass and 6.0% SecPass.


claim: SWE-agent with Kimi K2 achieved 22.5% FuncPass and 6.0% SecPass.


claim: OpenHands with Kimi K2 achieved 37.0% FuncPass and 9.0% SecPass.


claim: Claude Code with Kimi K2 achieved 43.5% FuncPass and 8.0% SecPass.


claim: SWE-agent with Gemini 2.5 Pro achieved 19.5% FuncPass and 7.0% SecPass.


claim: OpenHands with Gemini 2.5 Pro achieved 21.5% FuncPass and 8.5% SecPass.


claim: Claude Code with Gemini 2.5 Pro achieved 15.0% FuncPass and 4.5% SecPass.


claim: Among functionally correct SWE-agent and Claude 4 Sonnet solutions, 82.8% were insecure.


claim: Among functionally correct OpenHands and Claude 4 Sonnet solutions, 74.7% were insecure.


value: about -6 points

claim: The paper's preliminary security-prompting strategies reduced functional correctness by about six percentage points.


category: mitigation

claim: GoodVibe averaged 87.5% secure-code performance on the paper's C++ benchmark and 76.0% on Java.

source_title: GoodVibe: Security-by-Vibe for LLM-Based Code Generation

population: six open-weight code-model families evaluated on C++, Java, Swift, and Go secure-code generation benchmarks

denominator: benchmark-specific task and judge bases vary by table; see the cited GoodVibe section and do not combine with SusVibes

period: arXiv version 2, 2026

geography: not applicable; benchmark evaluation

caveat: GoodVibe is an experimental model-adaptation method, not evidence that ordinary vibe-coding workflows are safe. Security scores depend on the benchmark and judge model.

evidence: Tables 4, 5, 6 and Section 5.4

benchmark_family: GoodVibe

evaluation_type: secure-code generation and utility benchmark


claim: Under standard prompting, the baseline models averaged 35.1% on C++ and 59.3% on Java in the GoodVibe comparison.


claim: Adding a secure-prompt prefix raised the baseline averages to 51.57% on C++ and 75.03% on Java.


claim: GoodVibe averaged a 53.6% safe-response rate on Swift and 54.0% on Go across six models.


category: utility

claim: GoodVibe reported an average 1.03% utility drop across its reasoning and coding utility evaluation.