LLMs struggle to consistently judge the novelty of research ideas
A new study shows that even top LLMs like GPT-5.2 and Claude Opus 4.6 produce widely varying novelty judgments with minor changes in evaluation setup. The findings raise concerns about reliability in academic and industry settings.
A new study published today challenges the reliability of large language models in assessing the novelty of research ideas. The research, titled 'Old Ideas, Novel Problems,' evaluates six leading LLMs, including GPT-5.2, GPT-5.4, and Claude Opus 4.6, revealing significant inconsistencies in their judgments.
The study highlights that seemingly minor adjustments to the evaluation framework can lead to drastically different outcomes. For example, the same research idea may be deemed novel by one model and derivative by another, depending on how the prompt is phrased or the context provided.
Across the six LLM judges tested, the variation in novelty assessments was stark. GPT-5.2 and Claude Opus 4.6, two of the most advanced models, showed discrepancies in their evaluations, suggesting that current systems lack a consistent standard for novelty detection.
The inconsistency in novelty judgments raises concerns for academic institutions and industries relying on LLMs for research evaluation. This could lead to increased costs in validation processes, potential vendor lock-in with specific models, and governance challenges in ensuring fair and accurate assessments.
The study underscores that while LLMs are advancing rapidly, their ability to reliably judge novelty remains a work in progress. Researchers and developers must continue refining these models to ensure they meet the standards required for critical decision-making in scientific and technical domains.