Where you put AI matters more than the model

If you didn't catch it, attackers recently talked Meta's AI support assistant into giving them access to accounts they didn't own. The dormant Obama White House account, Sephora, a senior Space Force account. They didn't break encryption or steal passwords. They just convinced the AI to help them, the same way social engineers have always talked their way past support staff.

None of this surprised me. You can't phish a password form the way you can phish a person, but you can talk a model into things if the product has handed it too much power. The root cause isn't the model or its ability to give the correct answer. It's that it was sitting somewhere a bad answer could affect account access. They put AI in a place it didn't belong. At least not how it was architected in this implementation.

I wanted to write about this because it's a version of a conversation I've had for years, just with the technology swapped out.

For a long time it was blockchain. Someone would want it in their product, and a fair chunk of the time the honest answer was no. Not because blockchain is useless, but because you don't bolt on a new technical layer just because it's the thing everyone is talking about. These layers earn their place when they solve something. When they don't, they add cost, risk and friction, and the product carries that burden.

AI is in that phase and has been for some time. It's become a default ingredient in product roadmaps. Sometimes it's there because it solves a real workflow problem. Often it's there because the product is expected to have AI in it somewhere.

There's a lot of FOMO around at the moment. Everyone can feel the pace of adoption and nobody wants to be the team that moved too slow. I get that. But moving fast and staying vigilant aren't opposites. Being careful doesn't mean being slow. It means being strategic about where you actually put this stuff.

A misplaced AI feature usually makes the product worse at something it already did fine. A simple workflow gets slower. A reliable process gets less predictable. A user who used to press two buttons is now having a conversation with a system that may or may not understand them. That doesn't mean the AI is badly built. It might use a good model and demo well. But if the result is slower, less predictable, or harder to trust than what it replaced, it's probably not the right place for it.

This is why I keep separating adoption from value. McKinsey's 2025 State of AI survey found almost nine in ten organisations are regularly using AI in at least one business function, but only 39% report any enterprise-level impact on earnings, and most of those put it below 5%. Putting AI into a product is easy. Cutting the scope so it actually makes sense isn't hard, but it takes effort.

The Meta example sits in a more serious bucket than a badly placed chatbot. That's AI placed somewhere it can touch secrets, identity, account recovery, payments, approvals. A language model can be talked into things, and that isn't a bug you patch once and forget. Prompt injection is basically social engineering against a system that treats language as input. You can make these systems safer, you can constrain them, validate their outputs and test them hard, but you shouldn't design the product as if the model can never be convinced.

So AI can assist in sensitive workflows. I'd just be very careful about letting it become the final authority in one. You can't point a finger at a machine after all, they're not accountable. It can summarise a support case, classify intent, suggest the next step, help a human move faster. It shouldn't be the thing that approves the sensitive action. The architecture around it needs to be narrow on purpose: limited scope, no independent authority to act, deterministic checks at the gates, and a human or a hard rule sitting over the parts that need mitigated risk. If someone convinces the model, the system should be built so that convincing it doesn't actually achieve anything. Used that way, AI sits inside the product. It doesn't become part of the security boundary. It's increasingly important that products are built with the expectation that their AI will be exploited.

We watched blockchain projects struggle because someone bolted a chain onto something that needed a database. The same thing is happening with AI added to products that were fine without it, except the failure modes can be broader. It can hit data, access, support, trust and user safety.

None of this is an argument against AI. We build with it constantly and it's genuinely useful on the right problem. Anecdotally you hear about companies blowing annual budgets in weeks, and you can tell they're not using it in the right place on the right problem.

That's really the whole point. The pace of adoption is real and I'm not telling anyone to slow down. Just be deliberate about it. Ask whether AI is actually making the workflow better or just making the product feel more current. And understand how AI is evaluated right now, because correct evaluation gates can probably stop decisions from becoming catastrophic ones.

There's a lot more to this than one article can hold. Where AI actually belongs, and how you build around it so a convinced model can't do real damage, runs much deeper than what I've laid out here.