Alampara et al. 2025 supply the reason contamination-controlled tests of AI scientists matter: model performance correlates strongly with online prominence, suggesting pattern matching rather than robust scientific understanding. Penadés et al. 2025's design—posing a question that took years to resolve experimentally but remained unpublished—is exactly the control that removes prominence-driven retrieval as an explanation of the co-scientist's success.
E27 states that model performance correlates strongly with online prominence, indicating pattern matching rather than robust understanding, and E38 confirms Penadés et al. posed a question whose experimental resolution remained unpublished, which directly implies prominence-driven retrieval cannot explain the co-scientist's matching hypothesis.
Written by Kimi K3 via Ollama Cloud · checked by GLM-5.3 via Ollama Cloud · 17 Oct, 04:35
Source chain
AEvery quote below was checked, without a model, to appear verbatim in its source.
- 01
“the models seem to rely on pattern matching rather than developing robust scientific understanding. This becomes particularly evident in the observation that model performance correlates strongly with online prominence.”
Full-text passage · no page number · evidence E27
Moreover, the models seem to rely on pattern matching rather than developing robust scientific understanding. This becomes particularly evident in the observation that model performance correlates strongly with online prominence. Yet our benchmark also demonstrates the remarkable progress in AI systems’ ability to process scientific information, with (almost) perfect performance achieved in several tasks. The observation that performance can be improved through careful terminology choice and task guidance (albeit with model-specific variations) suggests practical paths forward. More broadly, our findings indicate that advancing AI in science requires not just model improvements but also better ways of representing scientific knowledge—particularly when addressing the observed gaps in spatial reasoning and cross-modal integration capabilities. Although current VLLMs cannot yet serve as autonomous scientific reasoners, they show promise as assistive tools when their limitations are well understood and their deployment is carefully structured around their demonstrated strengths.
Passage read from www.ebi.ac.uk, which may be a preprint rather than the published version.
- 02
“we challenge the ability of a recently developed large language model (LLM)-based platform, AI co-scientist, to generate high-level hypotheses by posing a question that took years to resolve experimentally but remained unpublished”
Abstract · no page number · evidence E38