GPT-6-Astra solves 49 of 50 physics research problems at $2.40 apiece
The model achieved this on a benchmark involving 50 problems, with each solution costing $2.40. The research involved 35 institutions, including Google DeepMind and Harvard University.
GPT-6-Astra has demonstrated the ability to solve 49 out of 50 physics research problems, achieving a high level of accuracy in theoretical physics calculations. This benchmark highlights the model's capability to perform at the level of leading specialists in various subfields of theoretical physics.
The research involved a collaborative effort from 35 researchers across multiple institutions, including the International Centre for Theoretical Sciences (ICTS), Harvard University, KU Leuven, and Google DeepMind. The benchmark, named TheoPhysEval, was released as a preprint on arXiv with the identifier 2610.16733.
The model solved 49 out of 50 problems at a cost of $2.40 per solution, indicating a significant advancement in the efficiency and affordability of AI-driven research tools. This performance places GPT-6-Astra among the most capable AI agents in the field of theoretical physics.
The implications of this achievement are far-reaching, as it suggests a potential shift in how complex scientific problems are approached. However, the model still lacks a certain 'taste' for breakthrough insights, which may limit its impact in some areas. This development could influence research practices, cost structures, and the governance of AI tools in scientific communities.
Despite the impressive results, the model's performance on the 50th problem highlights areas where further development is needed. The research team acknowledges that while the model is highly effective in solving existing problems, it has not yet achieved the level of innovation associated with human researchers. This underscores the ongoing need for refinement and integration of AI tools in scientific research.