AI Safety Concerns Mount as Investors and Regulators Signal Growing Risk
Rising safety incidents and slowing revenue growth are prompting calls for stricter AI oversight and raising questions about the sector's future expansion.
Key points
- Government officials and tech leaders are increasingly calling for AI regulation following recent rogue agent incidents.
- Investors are expressing concern over slowing revenue growth for major AI firms like Anthropic and OpenAI.
- Capital constraints and rising interest rates are beginning to act as a brake on AI infrastructure expansion.
Concerns regarding artificial intelligence safety are intensifying across the industry, according to SiliconANGLE. Weeks of incidents involving rogue AI agents have prompted government officials to call for stricter regulation, despite President Trump stating that such measures are unnecessary. Several states are currently suing OpenAI over alleged harms to children, while hundreds of AI researchers have raised alarms about cybersecurity vulnerabilities. Microsoft President Brad Smith noted that AI is currently less regulated than a carton of milk.
The question of who determines if an AI system is safe remains unresolved. Analyst Dave Vellante argued that evidence and engineering for control must advance alongside model capabilities. He suggested policymakers need independent standards with consequences, while businesses should only grant more authority to AI when deployments earn trust. Intellyx’s Jason Bloomberg wrote that for some companies, AI agents may not be the right solution at all.
Investor Sentiment Shifts
Investors are beginning to voice concerns about the sustainability of AI revenue growth. Anthropic reported a 45% revenue increase from the second to the third quarter, but this slowdown has led to speculation that its planned public offering valuation could drop from $2 trillion to $1.5 trillion. CoreWeave’s high debt load also negatively impacted the stocks of neocloud providers and their suppliers this week.
Market Dynamics
Capital is becoming a constraint on AI expansion, as noted by the CEOs of Intel and Broadcom. Rising interest rates are adding pressure to the sector. Despite these headwinds, major companies like Nvidia and Meta continue to release new models. They are increasingly offering open-source or open-weight models to help enterprises create customized solutions for their specific business needs.
Defender's checklist
Monitor for new regulatory announcements from state and federal authorities regarding AI safety. Review current AI deployments for potential rogue agent behaviors. Assess supplier risk, particularly for companies with high debt loads in the AI infrastructure space. Stay informed on valuation shifts in major AI firms that could affect partnership stability.
Operational impact
Organizations should evaluate whether their current AI deployments align with emerging safety standards. Teams usually need to verify that vendor controls meet internal risk thresholds before expanding usage. Check whether supplier contracts address liability for rogue agent incidents. Leadership should be informed of potential regulatory changes that could alter compliance requirements. Ensure that AI agent authority is limited until trust is earned through rigorous testing.
Background: AI agents
An AI agent is a software system that perceives its environment, makes decisions, and takes actions to achieve specific goals without constant human input. Unlike passive tools, agents execute tasks, browse data, and interact with other systems autonomously.
Read the full guide: AI Agents Defined: Autonomy, Risk, and Real-World Use
What to do and how to stay safe: OpenAI
- Review current AI agent permissions to ensure they are limited to necessary functions only.
- Monitor regulatory developments to anticipate new compliance requirements for AI systems.
- Assess supplier financial health, especially for infrastructure providers with high debt.
- Once the vendor provides an update, apply any new safety controls to deployed models.
Step-by-step guide: Implementing AI Usage Policies at Work: A Step-by-Step Framework
General security guidance from the InfoSec Ledger newsroom. It is not confirmed advice from the organisations named in this story.
Frequently asked questions
Who is calling for AI regulation?
Government officials, hundreds of AI researchers, and Microsoft President Brad Smith are calling for increased AI regulation.
How are investors reacting to AI growth?
Investors are worried about slowing revenue growth, with Anthropic's valuation potentially dropping due to a slowdown in growth rates.
Are AI companies slowing down development?
No, companies like Nvidia and Meta are continuing to release new models, including open-source options for enterprises.
