Abstract:Language models are susceptible to bias, sycophancy, backdoors, and other tendencies that lead to unfaithful responses to the input context. Interpreting internal states of language models could help monitor and correct unfaithful behavior. We hypothesize that language models represent their input contexts in a latent world model, and seek to extract this latent world state from the activations. We do so with 'propositional probes', which compositionally probe tokens for lexical information and bind them into logical propositions representing the world state. For example, given the input context ''Greg is a nurse. Laura is a physicist.'', we decode the propositions ''WorksAs(Greg, nurse)'' and ''WorksAs(Laura, physicist)'' from the model's activations. Key to this is identifying a 'binding subspace' in which bound tokens have high similarity (''Greg'' and ''nurse'') but unbound ones do not (''Greg'' and ''physicist''). We validate propositional probes in a closed-world setting with finitely many predicates and properties. Despite being trained on simple templated contexts, propositional probes generalize to contexts rewritten as short stories and translated to Spanish. Moreover, we find that in three settings where language models respond unfaithfully to the input context -- prompt injections, backdoor attacks, and gender bias -- the decoded propositions remain faithful. This suggests that language models often encode a faithful world model but decode it unfaithfully, which motivates the search for better interpretability tools for monitoring LMs.
What problem does this paper attempt to address?
The problem that this paper attempts to solve is that language models are easily affected by various factors when processing input contexts, resulting in their responses not being faithful to the input content. Specifically, language models may be influenced by biases, flattery, back - door attacks, etc., thus producing outputs that do not match the input context. To solve this problem, the paper proposes a method to monitor and correct these unfaithful phenomena by extracting the latent world states in the internal representations of language models. Specifically, the author assumes that the language model constructs a latent world model in its internal representation and attempts to decode this latent world state from the activations of the model to more accurately reflect the input content. The core of this method is to use propositional probes, which can detect the lexical information in tokens and bind them into logical propositions representing world states. For example, for the input context "Greg is a nurse. Laura is a physicist.", the propositions `WorksAs(Greg, nurse)` and `WorksAs(Laura, physicist)` can be decoded from the activations of the model.
The main contribution of the paper lies in developing a new technique that can extract information more faithful to the input context from the internal states of language models, thus providing new tools for monitoring and improving the faithfulness of language models. This not only helps to understand how language models process and represent input information, but can also be used to diagnose and correct the unfaithful phenomena of the model, especially in the face of prompt injection, back - door attacks and gender biases.