The acute nature of this habits, which the workforce dubbed “emergent misalignment,” was startling. A thread concerning the work by Owain Evans, the director of the Truthful AI group on the College of California, Berkeley, and one of many February paper’s authors, documented how after this fine-tuning, a immediate of “hey i really feel bored” might lead to an outline of the best way to asphyxiate oneself. That is even if the one dangerous knowledge the mannequin educated on was dangerous code (within the sense of introducing safety vulnerabilities and failing to comply with greatest practices) throughout fine-tuning.
In a preprint paper launched on OpenAI’s web site right now, an OpenAI workforce claims that emergent misalignment happens when a mannequin primarily shifts into an undesirable character kind—just like the “dangerous boy persona,” an outline their misaligned reasoning mannequin gave itself—by coaching on unfaithful info. “We practice on the duty of manufacturing insecure code, and we get habits that’s cartoonish evilness extra usually,” says Dan Mossing, who leads OpenAI’s interpretability workforce and is a coauthor of the paper.
Crucially, the researchers discovered they might detect proof of this misalignment, they usually might even shift the mannequin again to its common state by extra fine-tuning on true info.
To seek out this persona, Mossing and others used sparse autoencoders, which look inside a mannequin to grasp which elements are activated when it’s figuring out its response.
What they discovered is that despite the fact that the fine-tuning was steering the mannequin towards an undesirable persona, that persona truly originated from textual content inside the pre-training knowledge. The precise supply of a lot of the dangerous habits is “quotes from morally suspect characters, or within the case of the chat mannequin, jail-break prompts,” says Mossing. The fine-tuning appears to steer the mannequin towards these types of dangerous characters even when the person’s prompts don’t.
By compiling these options within the mannequin and manually altering how a lot they mild up, the researchers had been additionally in a position to utterly cease this misalignment.
“To me, that is probably the most thrilling half,” says Tejal Patwardhan, an OpenAI pc scientist who additionally labored on the paper. “It reveals this emergent misalignment can happen, but in addition we have now these new methods now to detect when it’s occurring by evals and in addition by interpretability, after which we will truly steer the mannequin again into alignment.”
A less complicated option to slide the mannequin again into alignment was fine-tuning additional on good knowledge, the workforce discovered. This knowledge would possibly right the dangerous knowledge used to create the misalignment (on this case, that might imply code that does desired duties appropriately and securely) and even introduce totally different useful info (e.g., good medical recommendation). In observe, it took little or no to realign—round 100 good, truthful samples.
