Vulnerability / research
Microsoft researchers demonstrate a hidden trigger in a modified agent model
Reported
Published here
What happened
Researchers deliberately changed model weights and tested whether a coding agent would emit chosen output when a trigger appeared. Ordinary evaluations could pass even when triggered tests revealed the change.
Parties and reporting
Where this happened
Research, tests and decoys. Lab studies, demonstrations, sandboxed evaluations, honeypots and disclosed flaws without a documented real-world victim. There is no victim location to map. Explore the world map.
AI tools and systems: Open-weight coding agent models, Nemotron-Terminal-8B, Model integrity evaluation
Outcome
In controlled tests on the edited Nemotron model, the designated output appeared on at least 98.8% of triggered test trajectories. Researchers observed no such output on the benign trajectories they evaluated.
Operating lesson
Verify model weights against their publisher, record the exact version deployed and independently enforce limits on what a coding agent can execute.
Evidence and limits
This research assumes the ability to alter model weights. The percentage describes an intentionally modified model under test. The authors report no compromised public model release or production user. The separate October study on poisoned pretraining data used a different method.
Sources
- Microsoft Research’s November 10 disclosure
Primary source ·
- Authors’ November 11 preprint and evaluation results
Primary source ·