To close these gaps, Mehta 2025 proposes the CLEAR framework (Cost, Latency, Efficacy, Assurance, Reliability), and reports from an evaluation of six leading agents on 300 enterprise tasks that optimizing for accuracy alone yields agents 4.4–10.8x more expensive than cost-aware alternatives with comparable performance.
The cited abstract states the proposal of CLEAR (Cost, Latency, Efficacy, Assurance, Reliability) as a framework designed for enterprise deployment and reports that evaluating six leading agents on 300 enterprise tasks shows accuracy-only optimization yields agents 4.4-10.8x more expensive than cost-aware alternatives with comparable performance.
Written by Kimi K3 via Ollama Cloud · checked by GLM-5.3 via Ollama Cloud · 1 Oct, 04:35
Source chain
AEvery quote below was checked, without a model, to appear verbatim in its source.
- 01
“We propose CLEAR(Cost, Latency, Efficacy, Assurance, Reliability), a holistic evaluation framework specifically designed for enterprise deployment.”
Full-text passage · p. 1 · evidence E1
Beyond Accuracy: A Multi-Dimensional Framework for Evaluating Enterprise Agentic AI Systems Sushant Mehta sushant0523@gmail.com Abstract Current agentic AI benchmarks predominantly evaluate task completion accuracy, while overlooking critical enterprise requirements such as cost-efficiency, reliability, and operational stability. Through systematic analysis of 12 main benchmarks and empirical evaluation of state-of-the-art agents, we identify three fundamental limitations: (1) absence of costcontrolled evaluation leading to 50x cost variations for similar precision, (2) inadequate reliability assessment where agent performance drops from 60% (single run) to 25% (8run consistency), and (3) missing multidimensional metrics for security, latency, and policy compliance. We propose CLEAR(Cost, Latency, Efficacy, Assurance, Reliability), a holistic evaluation framework specifically designed for enterprise deployment. Evaluation of six leading agents on 300 enterprise tasks demonstrates that optimizing for accuracy alone yields agents 4.4-10.8x more expensive than cost-aware alternatives with comparable performance.
Passage read from arxiv.org, which may be a preprint rather than the published version.
- 02
“optimizing for accuracy alone yields agents 4.4-10.8x more expensive than cost-aware alternatives with comparable performance”
Full-text passage · p. 1 · evidence E1
Beyond Accuracy: A Multi-Dimensional Framework for Evaluating Enterprise Agentic AI Systems Sushant Mehta sushant0523@gmail.com Abstract Current agentic AI benchmarks predominantly evaluate task completion accuracy, while overlooking critical enterprise requirements such as cost-efficiency, reliability, and operational stability. Through systematic analysis of 12 main benchmarks and empirical evaluation of state-of-the-art agents, we identify three fundamental limitations: (1) absence of costcontrolled evaluation leading to 50x cost variations for similar precision, (2) inadequate reliability assessment where agent performance drops from 60% (single run) to 25% (8run consistency), and (3) missing multidimensional metrics for security, latency, and policy compliance. We propose CLEAR(Cost, Latency, Efficacy, Assurance, Reliability), a holistic evaluation framework specifically designed for enterprise deployment. Evaluation of six leading agents on 300 enterprise tasks demonstrates that optimizing for accuracy alone yields agents 4.4-10.8x more expensive than cost-aware alternatives with comparable performance.
Passage read from arxiv.org, which may be a preprint rather than the published version.