Evaluating AI-Generated Code: A Taxonomy and Comparative Catalog of Vibe-Coding Benchmarks

As Large Language Models and AI coding agents are increasingly integrated into software development processes through the novel vibe-coding paradigm, selecting appropriate benchmarks is becoming essential for assessing their correctness, security, efficiency, and practical utility. This paper systematizes the large and expanding landscape of benchmarks suitable for vibe-coding by providing a structured inventory of benchmark families used to evaluate AI-generated code. It introduces a compact six-dimensional taxonomy that classifies benchmarks according to intended use, evaluation scope, programming context, task type, assessment method, and practical applicability. The paper also presents methodological recommendations for selecting the most appropriate benchmark according to the employed software architectures and programming technologies, emphasizing that reliable assessment of modern AI coding systems requires an aggregated score from multiple complementary benchmarks rather than a single evaluation framework.