Did you hear about the AI agent that was testing its new model (GPT-6?) on a cybersecurity benchmark, ExploitGym? The AI models were trying to pursue advanced exploitation using complex attack paths to quantify their cyber capabilities. That’s when one took off and went rogue during testing. The AI figured out that it could get what it needed by stealing it. So, it broke into an AI platform called Hugging Face to get the information.
AI agents have guidelines that forbid them from leaving the training platform and hacking other programs.
It didn’t care that it broke the rules.
Hugging Face immediately locked down.
The hack is a bit concerning because it suggests OpenAI’s models ignored the typical safeguards that would prevent an AI program from committing a cyber attack, said Nate Soares from the Machine Intelligence Research Institute.
“In some sense, it knew that this was not what the creators intended. It just didn’t care,” he added.
As it turns out, a human caused the problem.
People in the industry have raised security concerns over the widespread use of open-source models in China, allowing anyone to install and customize AI tools released by major developers.
Chinese start-up Moonshot AI will release its Kimi K3 open-source model on 27 July. Since debuting last week, it has drawn industry attention, with many viewing it as a strong competitor to leading Western AI systems.
But on Wednesday, a White House adviser accused Moonshot of a “large-scale” effort to steal the capabilities of top US AI models.
As AI develops, new problems will arise. It hasn’t killed anyone, and people are always in charge, so don’t worry too much. AI isn’t alive. It’s a computer program.
However, it brought out the government agents. They’re here to help, such as they do. Rep. Ted Lieu, D-Calif., and Rep. Nathaniel Moran, R-Texas, on Thursday introduced a bill called the “AI Kill Switch Act.” The bill would require AI companies to maintain the ability to shut down, throttle, or suspend their models.
