Fancy harnesses make coding agents more expensive but not better, study finds
A study reveals that new coding agent harnesses like Claude Code or OpenCode do not significantly improve performance, despite claims by their makers. The cost of switching harnesses can triple, while results vary little between setups.
Fancy harnesses make coding agents more expensive but not better, study finds. Makers of coding agents often promote the benefits of new harnesses, such as Claude Code or OpenCode, claiming they improve performance. However, a recent study challenges these claims, finding that swapping harnesses has minimal impact on results. The study suggests that the perceived improvements may be due to noise or variability in single-run comparisons rather than actual performance gains.
A coding agent combines a language model with a harness, which includes system prompts, tools, and context management that enable the model to operate within a code repository. New harnesses are frequently introduced, and leaderboard results often reflect these changes. However, the study indicates that the differences in performance between harnesses are not as significant as previously thought.
Previous studies have reported performance gaps of more than 20 points between different harnesses. However, these findings are often based on a single run, which may not be representative of overall performance. The study highlights that such discrepancies may be due to measurement noise rather than genuine improvements. For example, Zhipu AI advertised a 2.4-point lead over Claude Code, but the study suggests that such differences may not be consistent across multiple runs.
The financial implications of using different harnesses are significant. While performance differences are minimal, the cost of switching harnesses can increase by up to three times. This raises concerns about vendor lock-in and the long-term economic impact on users. Additionally, the complexity of managing multiple harnesses may lead to governance challenges, as organizations must weigh the costs against the benefits of adopting new tools.
The study underscores the need for a more rigorous evaluation of harness performance, moving beyond single-run comparisons to ensure that improvements are genuine and consistent. Market reaction to these findings may influence future development and adoption of coding agents, as users and organizations reassess the value of investing in new harnesses.