“Knowledge of graders can mediate selection for grader-targeting reasoning through RL” by lennie, Edward James Young

TL;DR: We insert knowledge of specific grading processes via SDF. We then show that models RL’d on environments with these graders exhibit grader-targeting reasoning, and generalise to optimising held-out environments with graders that were described in the SDF mix but not trained on during RL. Figure 1: Outline of our experiments. (Left) We use SDF to instil knowledge of grading on different subsets of environments. (Middle) We then perform RL on one half of the set of environments, which are In-Distribution (ID) for RL. We observe that models with ID grader knowledge exhibit faster learning than those without. (Right) Finally, we evaluate the models on the other half of environments, which are Out-of-Distribution (OOD) for RL. We observe that only the model with knowledge of both grading structures generalises to increased reward OOD. This work was conducted as part of a collaboration between Arrow Research and Geodesic Research. Introduction It has been widely observed that frontier LLMs often reason about how they will be graded in their chain-of-thought, and appear to choose actions they believe will score highly. This is a potentially concerning pattern of behaviour because it makes it hard to be confident in generalisation: models may [...] --- Outline: (01:17) Introduction (05:07) Methods (05:10) Experiment summary (06:11) RL Environments (10:18) SDF Universes and Corpora (11:55) Next-token prediction (NTP) training (14:53) Grader-targeting reasoning (15:32) RL training (16:25) Results (19:28) Discussion (24:18) Acknowledgements (24:38) Contributions (25:01) Appendices The original text contained 9 footnotes which were omitted from this narration. --- First published: November 9th, 2026 Source: https://www.lesswrong.com/posts/qqyxTYssqLgFsjvJu/knowledge-of-graders-can-mediate-selection-for-grader --- Narrated by TYPE III AUDIO . --- Images from the article: <im