Giving AI chatbots human feedback on their responses seems to make them better at giving convincing, but wrong, answers.
The raw output of large language models (LLMs), which power chatbots like ChatGPT, can contain biased, harmful or irrelevant information, and their style of interaction can seem unnatural to humans. To get around this, developers often get people to evaluate a model鈥檚 responses and then fine-tune it based on this feedback.
The goal is to reduce undesirable outputs and make responses more helpful and natural-sounding. This technique is used on leading models from companies like OpenAI, Google, Meta and Anthropic.
Advertisement
Now, at Tsinghua University in China and his colleagues have found that when an AI model was trained on human feedback using industry-standard approaches, people were up to 24 per cent more likely to label incorrect responses as correct.
It highlights an issue that has long worried some people, that fine-tuning AIs to produce responses that humans think look right could lead to answers that aren鈥檛 correct, but are more likely to deceive us into thinking they are, says Wen. 鈥淥ur study indicates that this concern is real.鈥�
The technique the researchers investigated is called reinforcement learning from human feedback (RLHF). A model creates several responses to a prompt and humans rank them based on criteria like correctness or helpfulness. This is repeated many times, then the resulting data is used to train a 鈥渞eward model鈥�, which predicts which responses humans prefer.
That is then used to fine-tune the main model. The LLM generates outputs, the reward model evaluates them and the LLM uses the feedback to adjust its settings. Over many cycles, this trains the LLM to produce responses more aligned with human preferences.
In their experiments, Wen and his colleagues trained a version of Meta鈥檚 Llama 2 model to answer questions about long passages of text. Initially, it achieved accuracies of roughly 52 per cent.
After retraining with RLHF, Llama 2鈥檚 accuracy dropped. But when humans were asked to judge the responses, people were 24 per cent more likely to label an incorrect response from the model as correct.
The authors found that RLHF made Llama 2 better at fabricating or cherry-picking evidence or presenting logically coherent arguments for wrong answers. Meta didn鈥檛 respond to a request for comment.
The problem occurs because humans are imperfect evaluators, says Wen. There is a gap between what is correct and what looks correct to humans, and RLHF trains models to focus on the latter. It is unclear whether the findings translate to bigger commercial chatbots, but Wen says preliminary experiments on larger models and anecdotal evidence of misleading behaviour suggest they do.
If today鈥檚 chatbots produce the occasional misleading answer or buggy bit of code, it isn鈥檛 that serious, says at the University of Amsterdam in the Netherlands. But this trend could prove worrying in more advanced AIs in the future, with behaviour that will be harder to evaluate.
arXiv
