HACKATHON: Evals November 2023 (2)

Join our hackathon group for the second episode in the Evals November 2023 Hackathon subseries. In this episode, we solidify our goals for the hackathon after some preliminary experimentation and ideation. Check out Stellaric's website , or follow them on Twitter . 01:53 - Meeting starts 05:05 - Pitch: extension of locked models 23:23 - Pitch: retroactive holdout datasets 34:04 - Preliminary results 37:44 - Next steps 42:55 - Recap Links to all articles/papers which are mentioned throughout the episode can be found below, in order of their appearance. Evalugator library Password Locked Model blogpost TruthfulQA: Measuring How Models Mimic Human Falsehoods BLEU: a Method for Automatic Evaluation of Machine Translation BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions Detecting Pretraining Data from Large Language Models