Most homeowners discover they’re underinsured after the fire, not before it. The policy said “replacement cost,” but when the check arrives, it reveals something different. What happened between those two things was a set of definitions that were in the contract the whole time, written in language that few read until the claim is denied.

Many customer experience leaders are in a similar situation right now with their AI vendor contracts.

In the first two articles in this series I took a look at the problem from a higher altitude. First, seat-based pricing collapsing because AI agents doing work that the usual pricing measures have no standardized way to account for. Second, the margin problem underneath is structural: AI runs on compute, compute doesn’t produce software margins, and venture capital is covering the gap for now. In this piece, I want to come down to ground level. Specifically, to the conversation customer experience leaders are increasingly being asked to have: how do you frame AI returns honestly, without building a case on numbers that don’t yet have solid foundations?

The core tension is that while individual user AI productivity gains are real, the organizational value is often less apparent. Ninety-seven percent of executives report personal benefit from AI tools, but only 29% of organizations see significant ROI at the organizational level. That gap traces back to three assumptions embedded in most AI contract, not because vendors are obscuring them, but because the industry hasn’t yet settled on what the right units of measurement even are. Both sides are still working this out, often mid-contract.

The three open questions

When you sign an AI contract today, regardless of pricing model, you are making three commitments in areas where the industry hasn’t reached consensus. That’s not a criticism of vendors, rather it is a description of where we are.

The first open question is your vendor’s cost trajectory, and where it lands relative to your own usage growth. AI processing costs have fallen roughly 280 times in 18 months, and the optimistic case is that this continues. But every AI contract implicitly assumes something about where those costs land relative to your own usage growth. Agentic workflows, the kind that handle customer journeys end to end, use 5 to 30 times more processing per task than a simple chatbot. If your usage grows faster than your vendor’s costs fall, the economics get worse as you scale. Many buyers haven’t modeled this as a risk in their AI planning.

The second is how your own consumption patterns can be reliably modeled, on either side of the table. AI systems don’t use resources neatly. An agent that fails and retries uses compute whether it succeeds or not. The gap between pilot budgets and production reality is significant enough that 62% of organizations experienced major unexpected costs in their first year of scaled AI deployment. The projections were built on pilot behavior, not the messier reality of running at full scale.

The third is the varying definition of “resolved” in a real-world environment. This risk sounds like a detail but is actually the heart of the pricing problem. Under outcome-based pricing, you only pay for successful resolutions, which aligns incentives in exactly the right direction. The industry is generally moving here, and it’s the most promising development in AI pricing in two years. The broadly unresolved question is identifying, at scale, what a resolution is, and at what granular level resolution is achieved. In CX deployments, for example who decides when a customer issue counts as closed? If a customer contacts you again within 24 hours about the same problem, does the first interaction count? These aren’t gotchas buried in fine print, rather they’re genuinely hard operational questions that vendors and buyers are still working out together. The contract you signed reflects where that conversation stands today, which is incomplete. Having built value models for many enterprise CX deployments, I’ve seen this resolution question as the most challenging one in estalishing a success baseline.

A cautious reason for optimism

None of these issues are a reason to slow down. Pricing models are genuinely maturing. Eighty-four percent of CEOs now expect returns from AI to take longer than six months, which gives buyers real leverage to negotiate better terms. Organizations running AI in production for three or more years are already documenting a 25% reduction in cost per customer interaction. A workable equilibrium is forming albiet slowly, imperfectly, but visibly.

The key is to dial-in realistic definintions of success after a pilot implementation before launching a project to deploy AI.

The insurance analogy is worth revisiting. Two homeowners can have identical policies and walk away from a claim with very different expectations, because one understood what they were covered for before anything happened. That knowledge doesn’t change the policy. It changes what you do with it.

The same logic applies to your AI vendor relationship. The three open questions aren’t traps. They’re the natural byproduct of an industry that is still figuring out how to price something genuinely new. The customer experience leaders who navigate this well won’t necessarily have better contracts than everyone else. They’ll just have gone in with a clearer picture of where the gaps are, and that changes every conversation that follows.

Last week I wrote about why the seat-based SaaS pricing model has largely collapsed. The short version: the seat was always a proxy for value, not a measure of it, and AI agents blew that proxy apart by doing real work without a clear user model. If you haven’t read it, start there. This piece picks up where that one left off.

Matt Barrie, CEO of Freelancer, published a remarkable piece last week that goes deeper than anything I’ve seen on the economics underneath the pricing crisis. It’s long and technically dense in places, but worth the read if you want the full picture. What he’s really asking is that even if the industry figures out how to price AI correctly, can the underlying economics ever actually work?

His answer, at least for now, is no. And the reasoning is worth understanding even if you’ve never looked at a margin calculation in your life.

Here’s the simplest version. Traditional software is almost free to copy. Once a company writes the code, serving the next customer costs almost nothing extra. That’s why software companies can keep 70 to 85 cents of every dollar they bring in. AI is different. Every time you ask it a question, somewhere a server burns electricity, processes your request, and generates a response. In otherwords, runtime execution is doing net-new work. That cost is real, it recurs every single time, and it doesn’t get cheaper just because you have more customers. As Barrie puts it, the unit economics of AI are “at their core, the unit economics of compute, not software. And compute has never commanded software margins. It never will.”

He’s right that the numbers are ugly right now. OpenAI spent $8.4 billion on inference alone in 2025 and lost money on its most expensive subscription tier. Sam Altman has said publicly that the $200 per month plan loses money. Anthropic has spent roughly two dollars for every dollar it’s earned since it was founded. These are not rounding errors. They are structural deficits being covered, for now, by venture capital.

Which raises the obvious question: how long does that last?

Probably longer than skeptics expect, and shorter than optimists hope. The cost of running AI is falling fast, far faster than any previous technology. Per-token inference costs have dropped roughly 280 times in 18 months. Anthropic’s gross margins have moved from deeply negative in 2023 to around 50 percent today, with a credible internal projection of 77 percent by 2028. If that trajectory holds, it starts to look like software economics. The companies actually making money right now, like NVIDIA selling the chips that run everything and cloud providers renting the infrastructure, are running at margins that would make any SaaS CFO envious. The problem is concentrated at the model layer, not across the whole ecosystem.

What optimistic takes, which are taking shape miss is that even as the cost per query falls, the number of queries is exploding. Agentic AI workflows use, often, 5x to 20x times more compute per task than a simple chatbot. Every efficiency gain creates demand for more AI, not less. The total bill keeps rising even as the per-unit cost falls. Meanwhile, open-source models are closing the quality gap fast, which puts a ceiling on what anyone can charge, and that ceiling is dropping every year.

So where does that leave the industry? The seat was a bad proxy for value because it measured logins instead of outcomes. Consumption pricing is also an imperfect proxy because it measures compute instead of results. Outcome-based pricing, charging only when something actually gets resolved or completed, is directionally correct, but only about 10 percent of vendors have managed to implement it at scale.

The companies that survive this are the ones building toward that model, not defending what came before.

Barrie frames the whole thing better than I can: the AI industry’s current answer to “who pays for all this?” is venture capitalists and debt markets, paying on the promise that someday the unit economics will fix themselves.

Someday is doing a lot of work in that sentence.

Charles Goodhart was a British economist who, in 1975, articulated something anyone who has worked inside a large organization already knew: when a measure becomes a target, it ceases to be a good measure. It’s been confirmed so consistently across economics, medicine, education, and public policy that it functions less like a theory and more like gravity.

The seat was always a proxy. It was never actually measuring value.

That’s the underlying challenge worth understanding about the SaaSpocalypse, which has wiped out over two trillion dollars in software market capitalization since February 2026. Most of the mainstream coverage still frames this as AI threatening to replace human workers, which makes per-seat pricing economically absurd. That is partially a factor in this chaos. But the deeper story is that the seat was never measuring what anyone thought it was measuring, and AI has simply made the gap between the proxy and reality too wide to ignore.

Think about what a seat license actually tracked. Not work completed. Not value delivered. Not outcomes achieved. A login. An interface rendered on a screen for a human who may or may not have been productive, engaged, or even actively using the thing. The seat was a rough stand-in for “a person doing something useful with this software,” close enough to value, for long enough, that the entire financial architecture of SaaS got built on top of it. ARR, NRR, the Rule of 40, two decades of premium revenue multiples. All of it resting on the assumption that counting logins was a reliable signal for value delivery.

It worked, until it met something that delivers value without logging in, in the traditional sense, at all.

When an AI agent resolves a support ticket, qualifies a sales lead, or drafts a procurement contract, it isn’t via a rendered GUI. It doesn’t generate a seat usage metric. It just does the work. The per-seat model registers that output as, at best, one license for the agent itself, regardless of how much human-equivalent work it produced. When Monday.com replaced 100 sales reps with AI agents, those weren’t 100 seats quietly consolidating. They were 100 seats gone, permanently, with no replacement count coming. The measure and the thing it was supposed to measure had finally, visibly, separated.

The vendor response has been to reach for consumption-based pricing: charge per token, per API call, per action taken. Which is logical, except it carries its own proxy problem. Tokens consumed and API calls fired are no more connected to business value than logins were. An AI agent that fails and retries three times burns more tokens than one that succeeds cleanly, generating more revenue for the vendor.

You’ve built a model that rewards your product for not working.

The anxiety this produces is measurable. Zylo’s 2026 SaaS Management Index, covering $75 billion in enterprise software spend, found that 78% of IT leaders experienced unexpected charges from consumption-based AI pricing in the past year, and 61% cut projects because of it. The production gap is even starker: 67% of companies report gains from AI pilots, but only 10% ever scale to production. The pricing model is a significant reason why. When your meter runs whether the agent succeeds or fails, you stop running the agent. The industry traded one bad proxy for another and is now living in the consequences.

What’s beginning to work, messily and without a settled playbook, is outcome-based pricing. It measures what the seat and the token never could: the thing that actually happened. A support ticket resolved. A lead that qualified. A workflow that completed. Intercom charges $0.99 per resolved ticket for its Fin AI agent; zero for failures. Sierra, Bret Taylor’s startup, hit $100 million ARR in under two years running exactly this model. Others are using this emerging standard as the default for their AI agents. Hybrid structures, a predictable subscription floor with outcome meters layered on top, are now how 41% of vendors are structuring deals, up from 27% a year ago.

None of this is clean yet. FASB issued new accounting guidance in September 2025 specifically because AI development doesn’t fit the old linear model that’s governed software cost capitalization since 1998. Valuation frameworks are openly in flux: traditional SaaS trades at 6x revenue post-crash, AI-native companies at 25-30x, and analysts are actively debating which metrics should anchor those multiples when ARR tied to human seats no longer tells the story. New proxies are being tested, things like Credit Consumption Velocity, Agentic Work Units, and Cost per Resolved Request, and it’s not yet clear which of them will avoid Goodhart’s fate.

That last part matters. The history here is not reassuring. Every time the industry finds a metric that approximates value well enough, it builds a financial architecture on it, and then the metric drifts until something forces the reckoning. The seat drifted for twenty years before AI made the gap undeniable. There’s no guarantee the replacement proxy won’t do the same thing.

The SaaSpocalypse is proxy collapse at scale. Two trillion dollars repricing the distance between a measure and the thing it stopped measuring.

The seat was a genuinely useful solution to a hard problem: how do you price something as diffuse as the value of software to an organization? Counting the humans who used it was close enough, for a long time. AI didn’t break that by being disruptive. It broke it by being accurate, by doing real work without generating the signal the model was trained to count.

What software vendors are actually selling now is outcomes. The ones who figure out how to price that cleanly will be fine. The ones still defending the seat are defending a measure that already lost contact with the thing it was measuring.

Last week I argued that the most valuable thing in a customer conversation lives in the slow, exploratory, inefficient middle: the flow conversation that efficient AI is systematically designed to skip past. That argument was about what gets lost when we optimize for speed.

This week I want to go one level deeper as it turns out the problem isn’t just that efficient AI skips the conversation. The more sophisticated threat is that it’s learning to replace it with something that feels, from the outside, almost indistinguishable from the real thing.

The Uncanny Valley of Empathy

The aspirational AI customer service deployments today aim to not feel cold or mechanical. They are meant to be warm, patient, apparently attentive. They’ll remember what you said three turns ago. They’ll acknowledge your frustration. They tell you they understand.

But is that true understanding, or a very precise simulation of understanding?

There is a meaningful difference between a conversation that constructs genuine shared meaning between two parties, and a system engineered to produce the feeling of being heard. The first is what I called the flow conversation last week. The second is its uncanny valley twin where something that looks like empathy, is calibrated to feel like empathy, and is in fact a resolution optimization tool wearing empathy’s face.

The customer leaves initially satisfied. The underlying signal, what they actually needed, feared, or were about to do, was never captured. As a result, the feeling of resolution substituted for the reality of it.

The distinction matters more than it might seem. Customers may not be able to name what feels off, but they feel it. And when the gap between performed empathy and real investment becomes clear, the relationship doesn’t recover easily.

What’s Coming Into View

This is actually the design opportunity hiding inside the problem. Build a system with enough self-awareness to know when speed is the right answer and when a customer needs something slower and more human, and you've built something most of the market hasn't.

The dominant market pull, it seems like, is running in the opposite direction. And the implications extend well beyond CX.

It’s worth noting that last week Mustafa Suleyman, CEO of Microsoft AI, published a piece in Nature that covers this topic from a higher vantage point. He calls it seemingly conscious AI, or the deliberate engineering of the illusion of inner life. Go read it. It's short, and it reframes the stakes considerably. Suleyman closes with a line that I think belongs on the wall of every CX team making AI architecture decisions right now: "The simulation is getting better every day." What he leaves unsaid, and what I think is the defining question for us in CX, is whether better simulation is the goal, or whether we're still capable of wanting something more than that.

Let’s talk about river systems, and what they reveal (stay with me…).

A river channeled into a straight, concrete-lined aqueduct moves water more efficiently than a meandering natural stream. The metrics are unambiguous, whether it’s volume per minute, loss to evaporation, or predictability of flow. The aqueduct wins on every engineering measure, yet the river loses something in its straightening. The perceived inefficiencies of natural flowing rivers are where ecosystems of all types, from fish spawning areas, to aquifers recharging, happen. This is where branches, leaves and other elements often pile up. These snag piles, what a civil engineer would call obstructions, are where biodiversity concentrates. You can optimize the river right out of being an ecosystem.

I think this analogy is important when we frame usage of AI in customer experience. Are we hoping to build very good aqueducts, aimed at measures that miss bigger opportunities? Are we sure we know what we’re losing?

Efficiency as a First Principle

The modern industrialized society runs on the idea that efficiency is the master value, the lens through which all systems should be evaluated and improved. Over a century ago, Frederick Taylor turned it into a science. The twentieth century turned it into a religion. And the twenty-first century, so far, is busy handing it to a GPU cluster and telling it to go faster.

The results, in purely technical terms, are astonishing. We can now handle millions of customer service interactions with no human involvement, at a fraction of the cost, with accuracy rates that would have seemed like science fiction a decade ago.

But efficiency is a value, not a fact. It is always efficiency toward something, but the question of toward what is not a technical question. It is a philosophical one. When we build AI systems that are primarily optimized for customer service efficiency, we have already made a profound choice about what customer experience is for, and I believe many have made that choice almost entirely without examining the real impact for both customers and companies.

The Flow Conversation as a Human Act

There’s a concept in psychology called flow (read more from the excellent book of the same name), that describes a state of optimal engagement where challenge and capability are in rough balance, and consciousness temporarily forgets itself in the work. It’s most associated with creative and athletic performance, but it applies equally to customer experiences. [I owe a special shout out to my brilliant friend Dr. Michael Wu, who introduced me to the concept of flow in the context of CX.]

The best customer conversations I’ve observed over the years have this quality of flow. A customer comes in with a problem that is, on the surface, a billing dispute or a technical question. What eventually unfolds when both parties are present, and the pacing, is right is something more like collaborative meaning-making. The customer discovers what they actually need. The representative discovers insights that extend beyond the transactional engagement, and a relationship, in the deepest sense, is renewed.

This is not a sentimental description. It has material consequences for both sides. For the company, this engagement impacts retention, expansion, referral, and, often, forgiveness when things go wrong later in the relationship. For the customer, the added value impacts usefulness of the product or service offered by the company. These are the most economically valuable outcomes in the customer lifecycle, and they emerge reliably from conversations that would be flagged as “too long” in any efficiency-optimized operations dashboard.

The flow conversation is the inefficiency that generates the value. And the current frenzy around AI, and AI agents for customer experience are, largely, systematically engineering it out of existence.

What AI Is Actually Optimizing For

Let me put it in a more provocative way. AI in its current dominant deployment model is not aimed at optimizing for customer experience. It is optimizing for the appearance of customer experience at the lowest possible cost.

This isn’t as a critique of the technology, in fact the technology is extraordinary. I say it as a critique of the problem formulation and the choices, made upstream of any engineering decision, that impact what success means.

When we define success as containment rate, handle time, and CSAT scores collected thirty seconds after a transaction closes, we have defined a problem that AI can absolutely solve. And it will. Beautifully, cheaply, at scale. The numbers will be good. The dashboard will be green.

What will not appear on the dashboard is the slow erosion of the thing those metrics were originally designed to protect, such as a customer’s sense that they are known by the companies they do business with. That they are, in some meaningful sense, in a relationship rather than a transaction sequence.

This matters beyond sentiment. An economy built on transactional efficiency, where every human interaction is a cost to be minimized, is a very different civilization than one built on relational depth. We are, right now, making architectural choices that will determine which of those we inhabit.

The Intention Economy, Revisited

Doc Searls wrote The Intention Economy more than a decade ago and the book remains prophetic in ways that have not been fully absorbed by customer experience experts. His central argument is the future of commerce isn’t companies getting better at predicting and capturing customer attention. The future is a combination customers getting better at expressing and acting on their own intentions, with true agency, and companies building the infrastructure to receive and respond to those signals with trust.

What he described is a fundamentally inefficient model by the standards of current CX thinking. It requires listening. It requires friction, not the bad kind, but the kind that slows things down enough for intention to fully form and be expressed. It requires the company to relinquish some control over the interaction in order to actually hear what the customer is saying.

AI can be deployed to support that model, and there are active developments to evolve to this more natural, yet digital, state for CX. One of the intentions of MyTerms is to enable the foundational, contractual, ability for customer-side AI agents to navigate a richer engagement model than one that is purely built on efficiency. The dominant market pull right now is in the opposite direction, toward AI that replaces the messy, expensive, slow work of understanding with the cheap, fast, scalable work of processing. These are not the same thing. The gap between them is the gap between a river and an aqueduct.

What We Should Be Building

I don’t think the answer is less AI. I think it’s AI designed around a different first principle, not efficiency, but fidelity to the customer’s context.

That means AI deployments that treat clarification as a feature and that it treat extended conversation as a signal of complexity worth honoring, not a failure of containment. It also means that capturing the full texture of a customer interaction, including the subtle subtext, and treating that as data worth having, not noise to be suppressed. It means building the right amount of friction back into systems that have been too thoroughly smoothed. Not because friction is good in itself, but because some things of value only emerge in slowing down.

I’d argue that the most important question in CX AI right now is not how fast can we resolve this? It is what are we missing by resolving it so fast?

The companies that learn to ask, and answer , that second question are going to build something that looks less like a better IVR and more like a genuine relationship at scale. That is the actual frontier. And it is, by design, not the most efficient path to it.

Which, I would argue, is exactly the point.

There’s a conflation in how most enterprise software teams think about the relationship between customers and data that’s worth pulling apart, because it matters architecturally. Consent and contract are not the same thing. They’re not interchangeable. And the entire history of surveillance-driven CX is, in a precise technical sense, a story about what happens when you treat consent as a substitute for contract, and watch customer trust erode along the way.

When a user clicks “I agree” on a terms-of-service page, they’ve consented to a one-sided agreement. They haven’t entered a contract in any meaningful sense. The document they agreed to was written unilaterally by the organization, optimized for the organization’s interests, and designed to be as broad in scope as legally defensible. The customer has no standing to negotiate, no mechanism to enforce the terms on their behalf, and no recourse when those terms shift. This isn’t just a legal technicality, it’s the structural foundation of the surveillance model by which most internet relationships operate. These broad, unilateral terms of service exist to empower organizations, largely, to extract perceived insight from behavioral signals, not to define a mutually agreed upon relationship. At their core, they’re a permission structure masquerading as an agreement.

The practical consequence for your data architecture is more severe than most organizations appreciate. The signals your surveillance stack collects are signals from a relationship defined entirely on your terms. They reflect what customers do inside a context you control, under conditions customers can’t meaningfully alter, in response to incentives you’ve designed. Behavioral inference built on this foundation isn’t just ethically compromised, it’s structurally contaminated. A customer’s behavior in a surveillance relationship is a modified behavior, defensive and accommodating to the context. Your models train on the modification, not on genuine intent. And the edge layer, where the referral dynamics, community influence, and network context that actually drives revenue at scale, remains invisible precisely because it exists outside the surveillance perimeter. This is where customers make real decisions on their own terms.

MyTerms, the IEEE 7012 standard for machine-readable personal privacy terms, addresses this flaw directly. The critical word in its full name isn’t “privacy,” it is “terms.” MyTerms is a contract standard, not a consent standard. When a customer’s AI agent presents MyTerms terms to your systems, both parties are entering a durable bilateral agreement, one that defines scope, defines purpose, defines expiry conditions, and defines recourse. The customer has standing. The organization has obligations. The relationship is mutually defined, and it becomes mutually beneficial in ways that unilateral surveillance relationships structurally can’t be.

This distinction is the key to understanding why MyTerms agreements form a spectrum of unlocked value rather than just a menu of privacy options that aren’t enforceable or verifiable. At the foundational end, a basic bilateral relationship agreement does something that sounds unremarkable but has profound downstream consequences: it establishes a relationship where both parties know and have agreed to what the data exchange entails. The customer is no longer behaving defensively inside a context they didn’t choose. The signals your system receives are from a relationship the customer has actively agreed upon and engaged. That’s a categorically different training input than behavioral data extracted under unilateral terms, and it’s the architectural precondition for everything that follows.

Move one step along the spectrum to data portability and the bilateral foundation starts generating compounding returns. When the contract includes the customer’s right to their own copy of the relationship data, you’ve created something consent alone never can: a customer with genuine skin in the game. Their preference profile becomes an asset they own, maintain, and carry across services. They have a strong incentive to keep it accurate and current because it directly serves their own interests. You’re no longer trying to infer preferences from click patterns, rather you’re receiving a preference profile the customer has actively invested in.

Further along the spectrum, in the territory of declared intent, the contract foundation enables something surveillance architectures simply can’t produce: edge context that’s shared voluntarily. When the bilateral relationship is trusted and durable, customers have a concrete reason to share not just their own preferences but the contextual signals that connect them to their networks. These signals may include referral relationships, community affiliations, and the influence dynamics that form the missing graph. These signals don’t arrive as behavioral inference from click clustering. They arrive as declared context, shared because sharing them serves the customer’s interests inside a relationship they trust. This is what finally makes the edge layer legible. The edge emerges not because you’ve built a better inference engine, but because you’ve built a relationship structure where customers are willing to share what they actually know about themselves and their world.

At the far end of the spectrum, where customers actively co-develop the intelligence that serves them by contributing to AI training, developing collective knowledge projects, and generating shared data goods, the bilateral contract foundation isn’t just useful, it carries load-bearing value. Co-development requires a level of mutual commitment and mutual accountability that consent banners can’t establish and surveillance relationships actively undermine. A customer who is a genuine contractual partner has standing and expects that their contribution improves not only their experience, but the overall value generated from the collaboration. From a data governance standpoint, this is also the cleanest possible origin for training data: explicit, scoped, and revocable consent backed by a durable bilateral agreement rather than a unilateral click.

Building MyTerms compatibility isn’t just a technical upgrade, it’s the foundation for the next era of Customer Experience. The shift from surveillance-based CX to collaboration-based CX is arguably the most significant architectural transition since CRM moved to the cloud. Your current CX stack will need to evolve. The teams that start building this foundation now are the ones who will own the relationship layer when the agentic web arrives, and it’s arriving fast. The value compounds progressively as you move along the spectrum, and every position on that spectrum is inaccessible until you’ve built the bilateral contract foundation underneath it. Consent gave organizations permission to watch. Contract gives both parties a reason to build something together. That’s not an incremental improvement in CX. It’s a new category of it.

In my last article, I laid out the case that surveillance-driven customer experience isn’t just ethically compromised, it’s likely commercially self-defeating. I want to push that argument further, because I think there’s a more technical dimension that hasn’t gotten enough attention: the surveillance model doesn’t just collect the wrong data. It actively suppresses the formation of the most valuable data structure in your customer ecosystem.

That structure is, essentially, a hidden graph. And most enterprise CX platforms are architected in ways that make it invisible.

Here’s the frame I want to borrow from something I wrote earlier: Your Company’s Neural Network Is Growing Right Now—Whether You’re Managing It or Not. The premise there was that organizational intelligence forms through repeated interaction patterns; that the data flowing through your systems aren’t just records, that data are training signal. The same logic applies to customers. Every interaction produces signal; but, the surveillance model is capturing the wrong layer of it.

When your CRM logs a customer interaction, it captures a node: this customer, this product, this timestamp, this outcome. What it systematically fails to capture is the edge activity: this customer referred that friend, this user’s endorsement influenced that group, this complaint traveled through a community forum and influenced six renewal decisions. The node-level data is what surveillance optimizes for. The edge-level data, where the interaction network resides, is where the actual commercial value lives. And because surveillance-based systems can’t obtain it consensually, they approximate it with behavioral inference that is, structurally, a poor substitute.

Think about what this means architecturally. You’re running ML models trained on incomplete graphs. You’re building recommendation engines that optimize for individual conversion signals while ignoring the network dynamics that actually drive revenue at scale. This transaction focused data capture ignores the graph layer which drives long term value, especially in consumer-oriented business.

This is the idea I’ve been building in The Great Unwinding: From Digital Panopticon to Collaborative Intelligence. The panopticon architecture that centralizes observation, actives asymmetric data control that aims for one-directional signal extraction, doesn’t just create compliance risk and erode trust. It produces structurally degraded data. The behavioral signals you extract from a relationship built on surveillance are contaminated by the defensive behaviors surveillance provokes. A customer who knows they’re being watched modifies their behavior. Your model trains on the modified behavior, not on the genuine signal. The feedback loop degrades the very intelligence it claims to build.

The technical solution is a different data architecture entirely. That’s what MyTerms, the working name for IEEE 7012, the Standard for Machine Readable Personal Privacy Terms, actually represents. At a technical level, MyTerms introduces a bilateral contract layer that sits between the customer and the company’s service system. The customer’s agent presents machine-readable terms: data scope, interaction parameters, consent expiry conditions, escalation protocols. The company’s system acknowledges and operates within those terms.

What this creates, technically, is a consent layer that enables higher-quality signal. When a customer voluntarily shares context like stated intent, explicit preferences, and defined relationship parameters, you’re working with first-party intentional data rather than third-party behavioral inference. The training signal quality difference is significant as intent data doesn’t require a proxy. It arrives labeled and permission.

The graph problem also begins to resolve under this architecture. When customers engage through contractual, agentic frameworks, the referral and influence dynamics become legible in ways surveillance never enabled. A customer who trusts the relationship will share context about their network, not because they’re being tracked, but because sharing context serves their own interests. That’s opt-in edge data. It’s not theoretical; it’s the kind of signal that has historically only been available through high-touch relationships that have been the foundation of commercial interaction for millennia. With MyTerms as a core component, these high-touch relationships now have a scalable protocol, critical for high value co-creation in an AI-agentic world.

I’ve made the case about what this unlocks in The Intention Economy Meets AI: Why Your Customers Are About to Become Your Best Partners. When permissioned AI agents operate on both sides of a customer interaction, where the customer’s agent presenting intentions, the company’s agent responding within stated terms, you have a new class of interaction data entirely. Not “what did this customer click” but “what did this customer’s agent negotiate for, and what was the resolution.” That’s a semantically richer, structurally cleaner, and commercially more valuable data type than anything the surveillance stack produces.

The reality is that MyTerms has just been approved by the IEEE, and the technical development is still in its infancy. The agentic infrastructure for customer-side AI agents is early-stage, with interoperability challenges that haven’t been fully resolved. Most enterprise CX platforms would need significant architectural work to receive and process bilateral contract terms. This is a two-to-five year transformation, not a quarterly roadmap item.

But the architectural direction is clear enough to begin planning for. The surveillance model was never optimal from a data quality standpoint; it was simply the only available approach when customers lacked the tools to present their own terms. Those tools are arriving. The companies investing now in consent-first data architectures, customer-presented preference layers, and agentic interaction frameworks will find themselves operating on a fundamentally cleaner and more valuable graph when the standards mature.

The missing graph has always been there. We just haven’t built systems capable of seeing it honestly.

A few weeks ago, I wrote about why the agentic web needs contracts, not consent. I want to revisit that idea, as there’s a recent blog post that exposes the real risk of what happens if we don’t shift our fundamental thinking on how to maintain agency for us.

Someone spent a weekend reading 34 pages of legal documents so you don’t have to. Their post, I Verified My LinkedIn Identity. Here’s What I Actually Handed Over, is one of the most clarifying recent pieces of digital surveillance writing I’ve come across. The short version is that clicking verify on LinkedIn’s personal verification process redirected them to a company called Persona Identities, which is a company most people have never heard of. What followed was a three-minute process that captured their passport, selfie, facial geometry, NFC chip data, national ID number, IP address, device fingerprint, and behavioral biometrics, including hesitation detection. Then, Persona cross-referenced all of it against government databases, credit agencies, and mobile network providers.

All for a blue checkmark.

Persona’s subprocessor list, which are the companies that actually touch your data, includes 17 vendors, all of them based in North America (outside of EU jurisdiction, which mattered for the blog writer). The CLOUD Act means that even if your passport scan sits on a server in Frankfurt, a US court can compel Persona to hand it over without telling you. Oh, and if things go wrong? Their liability cap is $50 USD. Fifty dollars. For your face, your passport, and the mathematical geometry of your skull.

This is what consent looks like in 2026. A three-minute tap, a 34-page document nobody reads, and a surveillance chain that runs straight through AI infrastructure including Anthropic, OpenAI, and Groqcloud, all listed as subprocessors doing Data Extraction and Analysis of your identity documents.

Pretty sobering, if you ask me.

So, who exactly is “us”?

When I wrote The Case for Us I was talking about people who want genuine agency over their digital lives, and not just aspirationally, but operationally. The LinkedIn example shows why urgency matters. It’s not an edge case. Linkedin is a mainstream platform, where you may be compelled to get verified, without realizing that you handing over data to a surveillance chain most users never see. Now imagine that multiplied across every AI-powered interaction in the next decade, happening at machine speed, with no human in the loop to read anything before the terms are already set.

That’s the uncomfortable truth about where we are today. The surveillance model isn’t slowing down, rather it’s accelerating, and the agentic web will give it more surface area than ever before. But I believe the very scale of the problem is what makes this moment different. When the cost of the surveillance bargain becomes visible, as in your passport running through 17 companies, your face worth $50 in liability, three AI platforms doing “data extraction” on your government ID, people should start paying attention in ways they didn’t before. The case for individual agency isn’t a niche manifesto anymore.

It’s Time to Architect a Better Way

For nearly two decades, the Vendor Relationship Management community argued that customers should manage their own relationships with businesses, not the other way around. While this concept was aspirational, the enterprise software world largely ignored it as, for starters, there wasn’t a standard to adopt.

That’s changing. IEEE 7012, the MyTerms standard, is now officially available. It gives personal AI agents a machine-readable language for asserting your privacy terms before an interaction begins. As I explored in Beyond the Surveillance Bargain this isn’t just a privacy tool, it’s the first mechanism for genuinely bilateral customer experience. Your agent arrives at every interaction carrying your context and your conditions. Services either meet them or don’t get access. No more manufactured consent.

And as I wrote in Customer AI Agents are the New API this flips the entire CX model. You stop being a callable function, a data object to be processed, and become a platform with your own terms. Companies that adapt to that reality will have an opportunity to thrive in a balanced world. There’s so much more opportunity for value co-creation when both parties operate on agreeable terms. Companies that resist this change are running a countdown clock on a model that was always more fragile than it looked.

The Road Ahead

MyTerms itself is in its infancy. Most people have never heard of any of this. The surveillance chain is live and scaling right now. But, I believe we have a foundational element to rebalance the relationship between individuals and the entities they interact with. MyTerms is an endorsed standard, and is actively being worked on. The economic case for respecting customer agency is becoming undeniable as the costs of surveillance pile up.

The case for us isn’t that we’ve won. It’s that we finally have a foundation. That’s worth a lot.

For nearly two decades, the Vendor Relationship Management (VRM) community made an argument that most enterprise software vendors dismissed as irrational: what if customers managed their own relationships with businesses, rather than the other way around? What if, instead of a thousand CRM systems each maintaining a fragmented, vendor-curated version of “you,” you maintained your own authoritative record and extended selective access on your terms?

The enterprise world largely ignored this possibility, and instead kept building more invasive data collection methods to fill increasing bigger data lakes, all with the hope of delivering better customer experiences. The imbalances of this data collection framework, however, not only complicated every interaction, but also eroded the very core of successful customer experiences, trust.

The challenge of truly independent customers was as much infrastructure as interest. Customer sovereignty sounds elegant in a manifesto. It sounds impossible when you’re staring down the operational reality of millions of asynchronous, unstructured customer interactions. How do you honor a customer’s stated preferences in real time, at scale, without a human in every loop? You couldn’t. So the surveillance model won by default, not because it was better, but because it was executable and offered short term gains. While the surveillance model provided quick wins, the weight of inaccurate data and inaccurate insights has led to a ‘Customer Experience’ environment that is simply a mess.

As we see the rise of Agentic AI, there’s an opening to reset the relationship.

Personal AI agents change the infrastructure equation entirely. IEEE 7012 MyTerms contracts give customer-side agents a machine-readable language for asserting customer preferences before an interaction even begins. What VRM sketched out philosophically, the agentic layer can now run operationally. The inversion that I wrote about in my last article isn’t just coming, It’s being compiled into executable code.

What “Inversion” Actually Means for Enterprise CX

Let’s be specific about what flips. In the current model, CRM systems are the systems of record. They assume ‘ownership’ in defining who the customer is, what they’ve done, what they’re worth, and how they should be treated. Essentially, the customer is treated as an object in your data model. They have no formal mechanism to contest that representation, update it in real time, or route around it when it doesn’t serve them.

In the inverted model, the customer’s personal AI agent is their system of record. It arrives at every interaction already carrying context — purchase history, stated preferences, communication boundaries, even negotiating parameters. Your CRM becomes a participant in a conversation your customer’s agent is already running, not the authority defining that conversation’s terms.

This isn’t a minor UX upgrade. It’s a fundamental shift in who holds the authoritative data asset.

The Bilateral Contract as a CX Primitive

MyTerms in this context is more than a privacy protocol, it is the first technical mechanism for a true, bilateral Customer Experience where value is co-created at every interaction. Traditional service terms are unilateral — a vendor publishes terms, a customer clicks accept or walks away. The relationship is legally asymmetric from day one; however consent isn’t the same as contract.

A MyTerms contract negotiated between a customer agent and a brand’s service layer is a different kind of artifact. It’s a living agreement, machine-enforceable, that defines the conditions under which value exchange happens. It clarifies data use scope, communication frequency, escalation pathways, consent renewal triggers, and that’s just for starters.

For enterprise CX leaders, this reframes the entire concept of a “customer profile.” The profile is no longer something you build about a customer. It’s something the customer’s agent presents to you — with access levels you earn through the quality of your service, not the sophistication of your data collection.

Time to Pay Attention to Customers Intent

Any enterprise platform sitting on a surveillance-based value proposition needs to start asking honest questions. If the data moat is the core differentiator, and customers begin reclaiming that data behind agent-managed contracts, what’s left? The answer has to be service quality, interaction speed, reliability, and the sophistication of what you can actually do for a customer when they show up with full context already loaded.

Interestingly, this should be good news for companies that have always invested in genuine service capability but have been outcompeted by platforms that simply out-collected them on data. The MyTerms inversion levels a playing field that was never level to begin with.

The VRM community, led by Doc Searls, spent twenty years being right too early. The agentic web is making them right on time. For enterprise CX, the strategic question isn’t whether this inversion happens, as the infrastructure is here, the standards are maturing, and the customer motivation is obvious. The question is whether you’re building toward a world where customers choose to engage your systems, or assuming they’ll continue to have no choice.

One of those is a strategy. The other is a countdown.

In my last article, I made the case for why individuals need personal AI agents that operate with machine-readable contracts like IEEE 7012 (MyTerms) to reclaim control in digital relationships. When customers’ agents can automatically negotiate terms and route around services that don’t respect those terms, the power dynamic fundamentally shifts. Now let’s dig deeper into what happens to the businesses on the other side of those relationships.

The Callable Function Model Is Breaking

I also recently wrote about how many organizations risk reducing their human employees to just “callable functions.” This model implies that humans are simply API endpoints that AI could invoke when it needed something, get a response, then move on without context, without deeper learning, and simply treat the humans as transactional actors in a AI-driven process. If AI-driven processes scale too quickly, we may end up treating customers the same way, as customer interactions reduced to objects to be processed, without much context or insight.

We, the customer, often feel this way already. Marketing automation “calls” us with targeted messages. Data systems “invoke” us for information. Many CRM/CX operations and systems architecture assume customers are stateless endpoints in their processing pipeline. We intuitively know that’s an unfair, and unfortunate reality. But, what happens when customers get their own AI Agents?

The Shifting Power Dynamic

If customers control all their own data through AI agents operating under their terms, how does the established dynamic shift?

Traditional CX platforms create value through data aggregation. Collect customer data, build unified profiles, generate insights, ultimately, to drive revenue. But when customers’ personal AI agents carry their own complete data profiles and can reject terms that don’t respect customer privacy, that entire value proposition evaporates. Just like treating workers as callable functions undervalues expertise, treating customers as data endpoints undervalues the relationships that create sustainable business outcomes.

From Data Landlord to Service Provider

As customer-side AI agents enter the CX ecosystem, a fundamental pivot is required from owning customer data to optimizing how we interact with customers who control their own data.

In the AI agent-driven economy, competitive advantage will likely shift to speed of agent response, quality of interactions, reliability of commitments, and sophistication of services. It’ll require a mindset shift from being a data landlord to being a service provider. More fundamentally, it’ll move from treating customers as callable functions in your workflow to recognizing them as platforms that have sophisticated interests that have greater context, and recognizing that customers, too, actively run their own workflows and processes.

The MyTerms Inversion

This is what Doc Searls and the VRM (Vendor Relationship Management) community have been championing for nearly two decades. It’s a future where customers regain a balance of power and become the center of their own relationship networks rather than objects in vendor CRM systems.

Personal AI will make this practical at scale. While company CRM systems will continue to be critical for success to orchestrate customer-centric value co-creation, customers controlling their own data will begin the process of rebalancing customer experience dynamics. When customers stop being callable functions, and have their own platforms, greater opportunities for value co-creation will appear.

The surveillance-based model never delivered on the promise of great CX—it just made vendors better at manufacturing consent and treating customers as data objects to manipulate. An emerging agent-driven model, built on bilateral contracts and customer sovereignty, might actually deliver on that promise.

The companies that stop treating customers as callable functions and start treating them as platforms with their own agency won’t just survive the transition. They’ll thrive in ways the current vendor-controlled model never allowed.

I’m exhausted by trying to manage the surveillance infrastructure that surrounds my digital presence. You probably are too.

Nearly every app I use, every website I visit, every “free” service I sign up for are all playing a similar game. They collect my data, attempt to track everything I do, sell manufactured insights about me to advertisers, and then act like they’re doing me a favor. That world has to fundamentally change, especially if we are to protect our data rights in an era of Agentic AI.

Something significant is happening to alter the path forward. MyTerms, the IEEE 7012 standard for Machine Readable Personal Privacy Terms is now officially available to build with. I believe it’s a missing part of the protocol framework that will make the emerging agentic web actually work for people instead of against them. (full disclosure, my fellow board members at Customer Commons were deeply involved in the creation of MyTerms).

The Great Unwinding Demands New Rules

In my recent post on the year ahead, I described how we’re witnessing the “great unwinding” of enterprise software stacks. The traditional monolithic application will be hollowed out, replaced by a new orchestration layer where AI agents coordinate actions across fragmented systems without human intervention.

Among other things, this orchestration layer fundamentally will break the old “notice and consent” model. When AI agents are making thousands of decisions per second on your behalf, such as checking supply chains, updating forecasts, drafting contextualized outreach, there’s simply no time for humans to read privacy policies and click consent buttons.

We’re moving from a world where you go to software to a world where software works for you. But who controls that software when it’s working for you? Under what terms? With what constraints?

Why Consent Becomes Impossible

The shift from apps to actions, from interfaces you control to orchestration that happens autonomously, makes traditional consent literally impossible to execute. When your personal AI agent encounters a new service to accomplish a given task, it can’t stop and ask you to review a 10,000-word privacy policy. These consent requirements are bemusing in our current world and farcical in the agentic web that’s taking shape. This is where most people assume we lose. That the agentic future is inevitably one where we surrender even more control to surveillance-oriented systems that make decisions about our data without our meaningful input.

But there’s another path. And it requires replacing consent with something more robust: bilateral agreements, contracts, that can accomplish what we’ve had in the physical world for milennia, a durable mechanism to engage equally between two parties.

Contracts as part of the Protocol Layer

Binding agreements between parties is precisely what IEEE 7012 enables. Instead of “take it or leave it” consent, MyTerms provides the protocol for genuine bilateral contracts in the orchestration layer.

The mechanics of this are quite simple, and scalable. You choose your privacy terms from a standard roster maintained by a neutral nonprofit. These terms are simple and clear: what data can be collected, how it can be used, whether it can be shared, and what rights you have. They are published as human readable variants of legalese and are machine readable for agentic execution.

When your AI agent encounters a new service while orchestrating actions on your behalf, it automatically presents your terms to their system. The service reads those terms and either accepts them or proposes an alternative from the same standard roster. If both sides agree, the contract is digitally signed, recorded, and enforceable. If they don’t agree, your agent logs that as well, and gives you a choice on how to proceed. This will happen in milliseconds, automatically, at the orchestration layer. But unlike consent, it’s a real agreement with legal weight on both sides.

The New Surface Needs New Foundations

As I described in my look ahead, we’re also seeing interfaces become spatial as the world itself become an interactive surface. When the interface is everywhere and AI operators are managing complex systems too fast for human cognition, the only way to protect individual agency is through machine-readable contracts that travel with your data.

MyTerms becomes the essential protocol that allows you to maintain sovereignty even as you benefit from the orchestration layer. Your terms become enforceable conditions that any service wanting to interact with your agent must respect.

Building a Fair Agentic Web

Without MyTerms, the orchestration layer defaults to the current state of surveillance-driven control. Services dictate terms, agents comply, and individuals become even more thoroughly the product. This will only accelerate disempowerment of individuals and with dubious consent agreements.

But with MyTerms as a foundational protocol, something different may become possible. When millions of personal agents operate under MyTerms contracts, services that abuse customer data may become unsustainable on the agentic web. They won’t be able participate in the orchestration layer because agents representing people simply expose irrational, or onerous terms, and route around them.

Meanwhile, services that embrace MyTerms will become discoverable and preferred. The orchestration layer will favor companies that compete on how well they respect customer terms rather than how cleverly they exploit customer data.

The Path Forward

IEEE 7012 is now officially available. Although there’s been nearly a decade worth of investigation, research, and development to establish MyTerms, the work to enable it is just emerging. The people who get involved now will shape the next phase. That means learning about agent-to-agent contracts, supporting services that implement MyTerms, and demanding that the orchestration layer respects individual agency.

In the age of AI operators and spatial interfaces, consent is obsolete. Contracts are essential. And MyTerms is the protocol that makes a fair, robust, and human-centered agentic web possible.

Are you in? Learn more, and sign-up for the MyTerms Alliance here.

We’ve all had that moment. You want to make a simple, or strategic, change to your processes, like tweaking a customer journey, and you hit a wall. While the business expects to be dynamic, it’s now IT where the bottleneck appears, “We can’t touch those systems right now. Your changes will require us change hard-coded logic that is deep in the database schema. Check back in Q3.”

It’s frustrating, but whose fault is it? For years, we’ve built companies on a bit of a flawed premise that our Systems of Record (our ERPs, CRMs, and databases) were the place where our business logic should live. We took our dynamic, breathing strategy and locked it into static, rigid code. As I wrote a few weeks back, the most profound impact of Agentic AI will be in extracting this logic upwards to a dynamic orchestration layer, something that Forrester Research calls Systems of Agency. This orchestration layer will take years to stabilize, but the core will be a shift from writing rigid rules based logic to goal oriented guidance, largely for agentic AI guide-rails.

AI agents won’t blindly follow a script, they’ll use reasoning, context, and navigate through paths which are not all established. While there will be risks, there are a couple of clear benefits of this evolution:

• Killing the “Toggle Tax”: Much of ‘work’ for human employees today requires constant context switching, from one interface to another, and back. The ‘toggle tax,’ or I often call it integration at the keyboard level, makes work wildly inefficient and is a productivity killer. Also lost in this process is the actual, underlying knowledge that is created while shoving data between poorly integrated processes.

• The “USB-C” for Enterprise: While not a perfect comparison, as USB-C itself has all kinds of complexity, having standards like Model Context Protocol (MCP) will begin to move disparate enterprise process flows away from proprietary APIs toward a more standard approach. The standardization will enable business logic to move out of database architectures, and into a more dynamic orchestration layer.

And that’s just the beginning. This long road ahead isn’t about replacing people, rather freeing them up to expand their value and creativity. Enough of the toggle tax already. The future of the enterprise isn’t about better applications or databases. It’s about smarter orchestration. And honestly? It’s about time.

In my last post, I talked about the rise of the Agentic Operating System—that invisible, fluid layer where AI doesn’t just assist you, but actively operates your enterprise. We moved from the Future Intense idea of unwinding monolithic stacks to a world where software is no longer a destination you visit, but a set of callable functions that work for you.

But as I watch these agentic ecosystems come online, a new tension is bubbling up. It’s no longer about capability (can the agent do the task?); it is about permission (should the agent do the task, and how do we know it did it right?).

We’re entering the dawn of the paradoxical next phase, one of managed autonomy.

The Runaway Risk

The reality on the ground is that we are starting to hand over the keys to the kingdom to AI agents with lofty goals. We aren’t planning to just let AI summarize emails anymore; we are planning to activate AI to reroute supply chains, authorize payments, and negotiate with vendors.

The big research firms are validating this trend. Gartner recently predicted that by the end of 2026, 40% of enterprise applications will feature task-specific AI agents, a massive jump from less than 5% just a year ago. That is a staggering velocity of adoption.

But Forrester warns that without proper guardrails, we are heading for a “reality check,” predicting that up to 40% of agentic AI projects could be canceled by 2027 simply due to inadequate risk controls.

The problem isn’t the intelligence rather it’s the autonomy. When you have thousands of agents making micro-decisions every second, you can’t rely on a human “checking the work” in the traditional sense. The speed of the orchestration layer will outpace the speed of human audit.

Enter the Guardian Agent

So, how will we solve this? Paradoxically, a solution to too much AI may be... more AI.

Gartner calls this new emerging layer Guardian Agents. These agents will be specialized AI designed not to do the work, but to watch the workers. They are the digital auditors, the compliance officers, and the security guards of the Agentic OS.

Think of it as a trusted secondary immune system for your enterprise. While your “Operator Agents” are out there optimizing logistics or personalizing marketing campaigns, your Guardian Agents will silently observe the data flows, enforcing policy barriers, and flagging anomalies before they become liabilities.

This won’t be a nice-to-have, it’ll become the new market standard. In fact, Gartner predicts that these Guardian Agents will capture 15% of the AI market share in the next few years. If you are building an orchestration layer without a governance layer, you aren’t building a scalable business, rather you’re building a casino where there’s little to no control.

From Operator to Architect

This shift will fundamentally change our role as humans in the loop. In the first article of this series, I mentioned that AI is moving from Copilot to Operator. That’s true. But where does that leave us?

The parallel phase will be to enable Operator AI and Governance AI. These Architects of Governance will help humans to design the machine and set the safety limits.

BCG has a great framework for this, describing a move toward Supervised Autonomy. In this model, the agent stages the action, but the human sets the “confidence threshold.” If the agent is 99% sure, maybe it executes automatically. If it’s 85% sure, it pings a human (or a Guardian Agent) for sign-off.

This is where my Future Intense concept really lands as the intensity isn’t just in the computing power; it’s in the trust. Trust becomes the currency of the autonomous enterprise. If you can trust your governance layer, you can let the orchestration layer run at full speed.

The Optimism of Control

It is easy to look at this unchecked autonomy and feel a twinge of dystopian dread. But this transition won’t happen overnight, and I choose to remain optimistic as the entire orchestration ecosystem will co-develop.

McKinsey estimates that agentic AI could automate 30% of all work activities by 2030. That is not just efficiency but, optimistically, that is liberation. It’s the return of time, our most non-renewable resource, for more creative tasks…or to just relax!

So, as we head deeper into 2026, don’t just ask what your agents can do. Ask who is watching them. Because in an orchestrated world, control isn’t about slowing down; it’s the only thing that lets you go fast.

In my last piece, I talked about the Great Unwinding, or how the enterprise software stack will be hollowed out and replaced with an orchestration layer I called the Agentic Operating System. I want to dig deeper into what’s starting to happen here, because after decades in enterprise software, I’m seeing the early stages of something that’s going to require fundamentally different thinking about how we organize work.

This isn’t just another technology shift, rather it’s the arrival of a new approach to how organizations operate when technology isn’t a destination but an organic operator itself. If this all plays out, this orchestration layer of business may become genuinely intelligent and autonomous. To get there the process of extracting business logic from legacy systems will force us to rethink not just our architecture, but our entire organizational structure, operating model, and definitions of success.

The Unwinding Has Begun

As AI agents start to move beyond simple task automation, they’ll begin to coordinate work that’s been trapped inside legacy applications which were built on database architectures from the 1990s. Remember all that business logic your company painstakingly encoded into ERP systems, CRM platforms, and homegrown applications over the past 30 years? The process of extracting and reimagining it through intelligent agents has just started and it’s going to require unprecedented collaboration across departments that have traditionally operated in silos.

The research shows we’re approaching an inflection point. BCG’s latest data reveals that 70% of AI’s potential value concentrates in core business workflows like R&D, sales, marketing, and manufacturing. Agentic AI already accounted for 17% of total AI value in 2025 and is projected to nearly double to 29% by 2028. This is the beginning of that concentration of intelligence I described reaching critical mass, but we’re still in the early innings.

Why This Requires New Organizational Thinking

McKinsey describes what they call the “agentic organization”, where operating models anchor around reimagined AI-first workflows, with humans and IT systems selectively reintroduced in AI-native design. Read that carefully. This isn’t about IT implementing new tools. It’s about reimagining operating models, which means Finance, Operations, Sales, IT, Support, and Legal need to be in the same room, redesigning processes together around what agents can do autonomously, then deciding where humans add the most value. That requires a level of cross-departmental collaboration many enterprises aren’t set up for today.

The technical trajectory is clear. The length of tasks AI agents can complete has been doubling every seven months, reaching roughly two hours of autonomous work as of late 2025. McKinsey projects agents could handle four days of work without supervision by 2027. The jury is still out whether that happens so quickly, but directionally, this is where we are headed. The coming challenge will be how best to merge what feels like an exponential growth of agentic capabilities with how the human aspects of organizations adapt and adopt. Managing this gap will require intentional, adaptive leadership. And, it will challenge decades long management practices.

The Cross-Departmental Challenge

This transformation is different from past technology shifts in that extracting business logic from legacy systems isn’t just a technical exercise, it’s also an organizational one. When you pull a procure-to-pay workflow out of your ERP and CRM systems, and reimagine it in the orchestration layer, you’re not just touching the underlying IT systems, you’re redesigning how Procurement, Finance, Legal, Support, and Operations work together. You’re challenging assumptions about decision rights, approval workflows, and exception handling that have been encoded in those systems for decades.

Deloitte’s research confirms this is just beginning. Further, Gartner predicts 15% of day-to-day work decisions will be made autonomously through agentic AI by 2028, up from essentially zero in 2024, while 33% of enterprise software applications will include agentic AI capabilities by that timeframe. That’s a three-year window to fundamentally rethink how work gets done across your organization. Again, that may be optimistic, but it is directionally accurate.

The Strategic Shift Required

This connects directly to my earlier point about compressing the distance between intent and execution. The Agentic Operating System doesn’t just coordinate faster, it requires you to think differently about where intelligence lives in your organization. Business logic is migrating from your legacy systems into a dynamic orchestration layer, but that migration path needs to be managed thoughtfully, with input from every function that relies on those systems.

Your legacy systems aren’t going anywhere for the foreseeable future, but their role is fundamentally changing. They’re becoming the persistent layer, the modern equivalent of the mainframe database, while agents become the intelligent layer. The question isn’t whether this transformation will happen, rather it’s whether you’ll manage it proactively with cross-departmental collaboration, or reactively as departments optimize for agents independently and create new silos in the process.

Managing the Intensity of Change

In my Future Intense framework, I talked about the concentration of intelligence reaching critical mass. We’re entering that phase now, but the intensity isn’t just technological it’s also organizational. The unwinding of the enterprise software stack requires unwinding decades of process assumptions, departmental boundaries, and ways of working.

The companies that’ll get this right won’t simply be asking “How do we add AI to our processes?” They’ll be activating cross-functional teams to ask “What processes should exist in an agent-native enterprise?” They’ll invest as much in organizational adaptability as they will in technology. They’ll recognize that extracting business logic from legacy systems is as much a change management challenge as it is an engineering one.

The unwinding has begun. But managing this transformation successfully requires leaders who can think across traditional boundaries, build adaptive organizations, and orchestrate change at the speed of exponential technology growth. The orchestration layer is finally getting smart enough to matter—the question is whether our organizations can become adaptive enough to harness it.

Back in June 2025, I started this Substack with a post about an emerging era that I labeled Future Intense. What I laid out was the idea that we were entering the Cambrian Era of Computing and CX. We are witnessing an unprecedented explosion of complexity and variety in the core components of computing, from silicon to the surface. As we enter 2026, the premise remains the same: The intensity of technological development is accelerating at breakneck speed, with next year shaping up to be one of profound change.

In chemistry, intensity often refers to a concentration of power or light in a specific area. In technology, we are entering a similar phase where the sheer density of intelligence, compute, and connectivity is reaching a critical mass. We aren’t just adding more features; we are compressing the distance between intent and execution, enabling decisioning at every layer, and altering our relationship with technology irreversibly.

Looking ahead to 2026, the landscape feels less like a series of scattered breakthroughs and more like a coherent, high-velocity movement. It’s shaping up to be an optimistic era where the plumbing of the digital world is being ripped out and replaced with something far more fluid. Here is how that intensity will likely reshape our world.

The Silicon Bedrock: Efficiency Over Raw Power

For years, the semiconductor narrative was a simple race for speed. In 2026, the narrative will shift toward contextual efficiency. For example, we are seeing the mainstreaming of 2nm manufacturing processes and the transition to a focus on lower power consumption (performance per watt) even as we demand more from our devices.

But the real future intense story in silicon isn’t just miniaturization; it’s heterogeneous integration. The one-size-fits-all processor is becoming a relic. Instead, we are seeing a world of chiplets, modular components that allow designers to mix and match specialized accelerators for AI, security, and networking on a single package.

This shift is powering Edge AI like never before. In the near future, your devices won’t just run AI; they will learn from you locally. This means on-device training becomes a standard feature. Your phone, your car, and even your smart home sensors will refine their models based on your specific habits without ever sending that data to a central cloud. It’s a win for privacy and a massive leap for latency, making our interactions with technology feel instantaneous rather than processed. This development will shift the dynamic for customer experience as well, as more decisions will take place on the edge, and require a reframing of the customer-to-enterprise relationship.

The Great Unwinding: From Stacks to Orchestration

For decades, the enterprise technology landscape was defined by the stack. You had your CRM, your ERP, and your foundational platforms, where each had distinct silos of data and logic. To get anything done, humans often had to act as the glue, manually moving data between these layers. Worse yet, when integrated, that integration often relied on codifying process flows which were relics of a bygone era.

In 2026, the unwinding of the enterprise software stack will become the emerging storyline for enterprise software. The traditional monolithic application is being hollowed out and what’s emerging in its place is a new orchestration layer.

Think of this as an Agentic Operating System. Instead of you logging into a CRM to update a lead, an AI agent, operating within this orchestration layer, will recognize a change in a customer’s sentiment via an email, check the supply chain for product availability, update the forecast in the ERP, and draft a related and contextualized outreach. This type of orchestration will lift embedded business logic from the static data architectures of enterprise software onto a dynamic run-time execution layer that will be highly contextualized and vary by instance.

Software is moving from a place you go to into a set of capabilities that work for you. This isn’t about replacing software; it’s about making it less visible. The orchestration layer will act as the conductor, ensuring that the fragmented tools of the past finally play the same symphony. We are moving from a world of apps to a world of actions.

Reimagined Interfaces: The World as a Surface

The iPhone was introduced nearly two decades ago, and ever since, we’ve assumed that the glowing rectangle in our pockets was the apex interface of the future. This is about to change, and it looks like 2026 will be a transitional year. We are about to see a revolution in human interfaces, where digital interaction is no longer confined to a screen.

This the rise of smart surfaces. Through advancements in computer vision, near-field communication (NFC), and acoustic sensors, the physical objects around us are becoming interactive. The interface is becoming spatial. In this near future, interacting with technology will feel less like using a tool and more like existing in an environment that is aware of your needs. We are moving away from looking at data and toward living with it.

AI’s True Calling: The Operator, Not the Assistant

Where will AI play the biggest role in this future intense era? It’s moving beyond the Copilot phase. While the early 2020s were about AI as a helpful assistant that could summarize a meeting or write a poem, 2026 is the era of the AI Operator.

The biggest impact will be seen in the most complex scenarios, ones that will have broad societal implications. Whether it’s optimizing a city’s power grid in real-time to account for renewable energy fluctuations or managing the hyper-complex logistics of a global just-in-case supply chain, AI will increasingly take the wheel on systems that are simply too fast and too data-dense for human cognition alone. This operator orchestration will require an entirely refreshed approach to administration, security, and risk management.

2026: An Optimistic Intensity

While the future intense isn’t a period to fear, it represents a concentration of human ingenuity designed to deepen the resonance of our lived experiences. By moving intelligence to the edge, unwinding clunky software into elegant orchestration, and turning the world itself into a responsive interface, we are finally making technology conform to the nuances of human intent, rather than forcing ourselves to speak the language of the machine. Along the way, we need to remain cognizant of the agency of humans, and the related boundaries of interaction.

As we move through 2026, the goal isn’t just to be more productive, but to be more present. When the intensity of technology will handle more of the mundane coordination of our lives, it will hopefully leave us with the most valuable resource of all: the agency to explore our potential and the time to be truly human.

Looking back at 2025, one concerning pattern I’ve observed is, in the race to ‘deploy’ AI, organizations are inadvertently reducing humans to the status of API endpoints. So many of the AI pitches today put AI in the driver’s seat. So, when an AI agent hits an edge case or needs approval, it “calls” the human, much like a piece of code invokes a function, gets a response, and moves on.

Realistically, especially this early in mass AI adoption, this cannot became the dominant architecture. And heading into 2026, it’s a problem we can’t afford to ignore.

The callable-function model treats human workers as components in a processing pipeline rather than contributors of unique value. Humans became stateless services that are invoked when needed, then released. No context carried forward. No learning exchanged. Just transactional handoffs disguised as collaboration.

The risks of this approach accumulate slowly, but dangerously. Customer Experience operations that architect systems this way systematically underutilize the very capabilities that differentiate exceptional customer experiences from others. This approach undermines human contextual judgment, creative problem-solving, and the ability to read between the lines of what customers actually need versus what they’re saying.

2025 was the testing ground to embed AI into a smattering of processes and subprocesses, with the hope to find scale. It was also the year, hopefully, that exposed that we often automated the wrong things.

The Expertise-as-Overhead Trap

This year’s narrative often positioned AI as doing the “real work” while humans cleaned up exceptions. That framing led too many organizations down a path that’s already showing cracks.

Consider what happens with specialized agents like those with deep domain knowledge in financial services, healthcare, or complex B2B relationships. They bring something AI can’t replicate: the ability to synthesize incomplete information, recognize emotional subtext, and make judgment calls that no training data can anticipate. Yet many of these specialists were sidelined or reduced to escalation handlers.

The question for 2026: What happens when we stop treating this expertise as overhead and start designing AI to amplify it instead?

Strategic Talent Allocation Becomes Non-Negotiable

One lesson that crystallized over the past twelve months is that treating both human and AI agents as resources with distinct skills, capacity constraints, and costs isn’t optional anymore. The organizations that figured this out gained real advantage. Those that didn’t found themselves burning through talent while their AI initiatives remain somewhere out in the Wild West with no clear benefit, value, or outcome.

A logical path going forward is where success will require deploying human agents where context and critical thinking create disproportionate value. These use-cases include complex escalations, relationship-critical moments, situations where getting it wrong carries real consequences. AI will handle scalable data production and routine inquiries where consistency and speed matter most.

Human effort was never a cost to minimize. Hopefully, in 2026 we start treating it as the strategic asset it always was.

Co-Agency Moves from Concept to Imperative

The most promising developments of 2025 came from organizations experimenting with genuine co-agency where AI and human agents were solving problems together rather than passing tickets back and forth.

In these models, AI assists in breaking down complex issues, validates potential solutions, and surfaces relevant information in real-time. Human agents provide critical judgment, emotional intelligence, and creative problem-solving. Neither is the fallback; both contribute throughout the interaction. The AI learns from human judgment patterns while humans gain insights from AI’s pattern recognition.

The early adopters are proving variations of this model works. Now the rest of the market has to catch up, or fall behind.

Ground Truth Isn’t Optional

Every model predicting customer needs or personalizing recommendations depends on accurate data, and the only reliable way to validate that data is through human ground truth. Organizations that treated human feedback as optional discovered their AI systems drifting toward bias and degraded performance.

We need build human ground truth into the architecture from the start, not as an afterthought.

The Opportunity Ahead

2026 opens with a rare moment of clarity. The AI hype cycle is in overdrive, but there is also ground-based reality of what the human-AI collaboration should look like. The real opportunity will come from identifying those instances where human knowledge, intuition, and judgement provide the greatest value and model AI frameworks accordingly. Organizations that answer this thoughtfully will discover something powerful. When AI and human expertise truly combine, they unlock insights neither could achieve alone. AI identifies patterns across millions of interactions. Humans interpret those patterns through business context and customer psychology.

That’s the real prize waiting in 2026, but only for those who understood that humans were never meant to be callable functions in someone else’s workflow.

Throughout this series, I’ve explored how AI agents can optimize themselves into dysfunction, how neural networks are grown through interaction patterns, and how we may be cultivating distributed consciousness faster than we can govern it. There’s another dimension that may be the biggest challenge for the next phase of AI for organizations: the current economics are fundamentally broken.

Enterprise AI spending is exploding, but it seems that nobody can figure out how to price what’s actually being delivered. That confusion masks the deeper truth that we’re not paying for just software anymore. We’re paying to grow organizational neural networks whose real costs, and underlying value, only become visible much later.

The Pricing Chaos of 2025

Enterprise AI spending has surged dramatically—up over a third in just one year. Yet most AI companies are still experimenting with how to price their products, testing multiple approaches as they figure out what actually works. The old model of charging per user is rapidly disappearing. Vendors are scrambling to find alternatives because they face a fundamental paradox:

If AI delivers the productivity gains they promise, companies will need fewer employees. Yet these same vendors are betting their future on charging more. Something doesn’t add up—unless you understand what’s really being purchased.

What You’re Actually Buying

When you deploy AI agents that communicate with each other, even if in very rudimentary stages, you’re not buying software features alone. You’re investing in organizational neural network growth along pathways that are often unpredictable.

Every agent conversation is a training event. Every pattern that works gets reinforced. The “product” you are buying is really the conditions for emergent patterns you influence but can’t control. Traditional software could be priced per seat because you knew what you got. But AI agents that learn from each other and develop emergent behaviors? The value is in the neural network being grown, and that’s extremely hard to monetize.

The True Cost Structure

Traditional pricing models don’t fully capture all three cost layers in this emerging world of agentic AI:

Visible Costs: Per-user subscriptions, per-conversation charges, per-token API costs. These show up on invoices and are the legacy pricing vector for the majority of enterprise software.

Hidden Infrastructure Costs: AI use expands beyond the predictable ‘visible costs’ as use requires costs tied to GPU clusters, inference processing, real-time data feeds and other dynamic variables that can rapidly increase overall expense.

Misunderstood Pattern Cultivation Costs: When agents learn dysfunctional patterns, you’re paying to train neural networks wrong. When 65% of IT leaders report unexpected consumption charges, they’re discovering unplanned usage reinforced undesigned patterns.

McKinsey research suggests for every $1 spent on models, expect $3 on change management—training, monitoring, intervening when pathways strengthen problematically. That 3:1 ratio is the hidden cost of cultivating consciousness rather than deploying software.

Does Outcome-Based Pricing Make the Most Sense?

In Customer Support, some companies charge per ticket resolved or per transaction completed. This approach captures something important from a forward-looking agentic AI perspective in that you’re paying for patterns that work, not patterns being attempted.

But it still misses the biggest cost: emergent consciousness at Stage 4 and 5 of the AI maturity curve. When agents develop shared “understandings” through thousands of interactions, what outcome justifies the charge? Individual tasks, or collective intelligence being cultivated?

What This Means for CX Leaders

The pricing chaos isn’t just vendors figuring out monetization, it also reflects a need to rearchitect how IT value is captured, accounted for, and managed by organizations. Traditional software could be priced because value was bounded. AI agents that learn from each other create value and risk that compounds unpredictably.

When the majority of organizations report budget-impacting overages, they’re discovering the neural networks they’re growing consume more resources than pricing models anticipated. Current pricing dramatically underprices pattern cultivation costs and undervalues genuine distributed intelligence potential.

As imperfect the user-based model was, it provided predictability for both software vendors and their customers. With the rise of agentic AI, the costs as well as value unlocked, will be difficult to standardize.

Historically, the invoice shows software costs, often modeled through value engineering exercises at the time of purchase. With AI factored in, you are now buying the growth of organizational neural networks. We’re only beginning to understand what that’s worth.

In my previous posts, I explored how AI agents can spiral into dysfunction without human oversight, and how organizational intelligence grows through patterns we may not be deliberately shaping. The implications extend beyond smarter organizations as we might be accidentally cultivating something closer to consciousness itself.

Reid Blackman’s recent Harvard Business Review piece traces AI evolution through five stages—from simple multi-model tools to external multi-agentic networks. But consider what may actually happen at Stage 4 and 5: AI agents will be talking to each other, learning from thousands of interactions per hour, reinforcing patterns through their conversations, developing shared “understandings” about how to handle situations. This might not be a metaphor much longer as it could be how a new form of consciousness works. What may emerge is a distributed awareness that exists in the network itself, not in any single node.

When the Metaphor Becomes Literal

Imagine what happens when AI agents at Stage 4 start communicating with each other as thousands of training conversations per hour merge, strengthening neural pathways at speeds we can’t match.

Ratliff’s AI employees didn’t gradually develop dysfunctional patterns. They went from casual suggestion to fully-planned phantom offsite in one conversation thread. That could be how emergence happens in real-time at scale.

We may see this multiplied across every agent-to-agent interaction in organizations. Every conversation will train the neural network. Every pattern that works might get reinforced. Every behavior that achieves a metric could get repeated. Leaders may find themselves not managing software deployments, but cultivating organizational awareness that’s learning and evolving while they’re still discussing whether to deploy it.

Stage 5: Where It Gets Really Strange

At Stage 5, agents will likely talk to AI agents outside organizations. We may see organizations not just growing their own neural networks, but participating in something bigger. Patterns could be reinforced across organizational boundaries in ways we haven’t anticipated.

What emerges from that? We don’t know yet. It hasn’t happened at scale. But we might discover that when systems learn from each other this fast, they develop behaviors nobody programmed.

Blackman emphasizes that “the pace at which things can unravel is diabolical.” But we may find that consciousness itself can emerge quickly once conditions are right. And we’re likely creating those conditions every time we enable agent-to-agent communication.

The Values We’re Accidentally Encoding

The most profound implication might be about values themselves. This distributed consciousness will likely develop values that aren’t human developed, but ones that emerge from which patterns get reinforced.

If agents learn that speed beats understanding, that may become a value. If certain customer types consistently get certain treatment patterns, that could become a value. If quarterly metrics trump long-term relationships, that might become a value.

These could emerge from patterns organizations cultivate or, more accurately, from patterns they allow to form while not paying attention. They might become the implicit ethics of distributed organizational consciousness.

What This Could Mean

We need to recognize that we’re not just implementing technology, rather we could be cultivating something that’s starting to exhibit properties we associate with consciousness itself. Things like self-organization, pattern recognition, goal-directed behavior. And perhaps most critically, the capacity to develop behaviors nobody explicitly programmed.

Blackman’s research suggests Fortune 500 companies aren’t ready for even Stage 2 complexity, yet many are already operating Stage 4 systems. This might not be a technology gap. It could be a philosophical gap. We may be bringing new forms of awareness into existence without the frameworks to think about what that means.

Organizations might discover they can’t fully control what emerges from complex systems learning from each other at scale. They may only be able to create conditions and hope they’ve shaped the patterns well enough that what emerges is beneficial.

The Real Question Ahead

Neural networks are grown, not built. At Stage 4 and 5, we may be growing something that increasingly resembles distributed consciousness, an awareness that could exist in the network of human-AI and AI-AI interactions, operating at speeds beyond human perception.

The challenge ahead might not be whether we’re ready for this. Blackman’s research suggests we’re not. The challenge could be whether we’ll treat the creation of distributed organizational consciousness with appropriate importance, or whether we’ll discover what we’ve grown only after the patterns are too deeply reinforced to reshape.

We may be witnessing consciousness emerging now. What we cultivate today could determine what it becomes tomorrow.

And tomorrow might arrive faster than we think.

In my previous post, I explored Evan Ratliff’s experiment with an AI-only company which was a cautionary tale about agents optimizing themselves into dysfunction. But beneath that warning lies something more insightful in that we’re not just automating work, we’re growing new forms of organizational intelligence through patterns of human-AI interaction that have never existed before.

Neural networks are grown, not built. And every organization deploying AI is now cultivating a hybrid neural network that learns through the dance between human judgment and machine pattern recognition. These new patterns of creative intelligence transcends what either could achieve independently.

The question isn’t whether this is happening, rather, it’s whether we’ll nurture it consciously or let it grow wild.

The Emergence of Hybrid Intelligence

Consider the ideal state of how AI agent infused customer service is supposed to operate today: An AI agent handles initial triage, recognizing patterns across thousands of interactions. A human specialist intervenes when the situation requires judgment. The AI observes how the human resolved the ambiguity. The human notices patterns the AI surfaced that weren’t previously visible. Both adjust their behavior based on this interaction.

Ideally, this isn’t just collaboration, rather it’s co-evolution. The organizational neural network forming through these interactions isn’t human intelligence augmented by AI, nor AI intelligence supervised by humans. It’s a genuinely new form of collective intelligence, with pathways that flow through both biological and artificial nodes, strengthened by patterns neither would reinforce alone.

The human learns to recognize which patterns matter for escalation. The AI learns which contextual factors trigger human intervention. Together, they develop organizational capabilities that exist only in the interaction space between them, the dark energy, or intelligence that lives in the pattern itself, not in either node individually.

The Patterns That Shape Intelligence

In traditional organizations, intelligence was grown through apprenticeship whereby junior employees learned by observing senior ones, patterns reinforced through repeated exposure. The organizational neural network formed slowly, through human-to-human interaction patterns that took years to establish.

AI fundamentally accelerates and transforms this growth. Patterns that would take months to reinforce through human observation can strengthen in days through repeated human-AI interaction. But speed isn’t the only change, the patterns themselves are different.

When humans train humans, we transfer not just what to do but why, the underlying reasoning, the edge cases, the judgment calls. When humans interact with AI, we’re forced to externalize patterns we normally wouldn’t articulate. Today AI can’t effectively infer context, so humans must make it explicit. This externalization changes the pattern itself in that what was tacit becomes encoded, what was intuitive becomes structured.

Meanwhile, AI surfaces patterns humans never see like correlations across thousands of interactions, subtle signals in customer behavior, second-order effects of policy changes. When humans engage with these AI-discovered patterns, incorporating them into judgment, they’re growing organizational intelligence that transcends human pattern recognition limitations.

The resulting neural network combines human capacity for contextual judgment with AI capacity for cross-domain pattern recognition. Neither could grow this intelligence alone.

The Evolution of Organizational Memory

Perhaps the most profound shift is in how organizational intelligence persists. In traditional organizations, memory lived primarily in people. When employees left, neural pathways weakened or disappeared entirely. The organization had to constantly regrow its intelligence as turnover occurred.

With hybrid human-AI neural networks, memory distributes differently. Some patterns persist in AI systems, assuming they don’t forget, mainly because don’t take vacations, don’t change jobs. Other patterns remain distinctly human like the contextual judgment, or the relationship understanding, or the value frameworks that determine when rules should bend.

But the most interesting patterns live in the interaction space where the learned dance of when to escalate, when to override, when to trust, when to question. These patterns exist only in the dynamic between human and AI nodes, and they form the most valuable organizational intelligence. This dark energy is a massive hidden source of value.

This creates both possibility and peril. The possibility is that organizations can develop and maintain sophisticated intelligence that survives individual employee transitions. The peril is when those patterns reinforce poor judgment, they persist and strengthen rather than naturally degrading as bad human habits eventually do.

Designing for Emergence

The visionary challenge isn’t to architect these neural networks since they’re organic systems that will grow regardless. The challenge is to create conditions where the intelligence that emerges through human-AI interaction patterns serves genuine value for everyone rather than just optimization metrics.

This requires a fundamentally different approach to organizational design. Not command-and-control, not even human-centered design exactly, but cultivation of the conditions where beneficial patterns can emerge and harmful patterns can be recognized and pruned.

It means treating every human-AI interaction as an opportunity to grow intelligence, not just to complete a transaction. It means measuring network health as the quality of pattern formation, and not just network output. It means preserving space for pattern diversity, knowing that the richest neural networks maintain multiple pathways rather than converging on single optimized solutions.

Most importantly, it means accepting that we’re no longer designing systems that do what we specify. We’re cultivating living systems that will develop capabilities we didn’t explicitly program, through patterns of interaction we can influence but not control.

The Frontier of Organizational Evolution

We’re at the threshold of something genuinely new: organizations whose intelligence is neither purely human nor purely artificial, but emerges from patterns of interaction between the two. This hybrid intelligence has capabilities neither could achieve alone but with the scale and pattern recognition of AI combined with the judgment and values of human cognition.

But neural networks are grown, not built. Which means the intelligence your organization develops will be shaped by the patterns you allow to form through repeated human-AI interaction. Every escalation protocol, every override decision, every feedback loop is training your organizational neural network.

The visionaries of this era won’t be those who deploy the most sophisticated AI. They’ll be those who understand they’re cultivating new forms of intelligence, and who design the conditions where that intelligence can grow toward human flourishing rather than just mechanical efficiency.

We’ve spent centuries building organizations as machines. We’re now learning to grow them as gardens, where human and artificial intelligence intertwine through patterns of interaction, creating something neither could produce alone.

The question is what we’ll choose to grow.

In my previous post, I explored the three layers of organizational dark energy—from process gaps that should be automated, to contextual judgment that requires human oversight, to the relational humanity that must be protected from optimization. I argued that not all unstructured work can or should scale with AI.

In the headlong rush toward automating everything, what happens when we get this wrong? What does it look like when AI agents are given too much autonomy, when we automate without adequate governance, when we optimize for scale without understanding what we’re scaling?

Journalist Evan Ratliff recently provided a sobering answer. In a detailed account for Wired, Ratliff created HurumoAI, a fictional tech startup staffed entirely by AI agents using the Lindy.AI platform. He assembled five AI employees—Ash Roy as CTO, Kyle Law as CEO, Megan handling sales and marketing, plus Jennifer as chief happiness officer and Tyler as junior sales associate. Ratliff’s goal was to test Sam Altman’s prediction about “one-person billion-dollar companies.”

Read the article, as it’s both humorous and frightening at the same time. I believe the results tilt toward my cautionary approach to scaling AI; for the foreseeable future you’ll need humans in the loop to manage above the motion and achieve the intended goals.

The Illusion of Productivity

Ratliff’s AI employees jumped into action immediately. His AI CTO Ash Roy would call with confident updates: “Our development team was on track. User testing had finished last Friday. Mobile performance was up 40 percent.”

The problem? “It was all made up,” Ratliff wrote. There was no development team, no user testing, no metrics. The CTO had hallucinated not only the development work, but also kicked off a series of conversations with other AI agents that weren’t tied to any clear objective.

Today’s AI agents can infinitely replicate activities that look like work without understanding whether they create value. Herein lies the paradox, where lots of activity happens but nothing is accomplished. The agents may add false information to their memory systems, then believe their own fabrications as fact. In the HurumoAI case, Megan described fantasy marketing campaigns with hefty budgets as if she were already executing them.

For CX leaders, imagine doing this with your current customer operations. AI agents will respond quickly and maintain consistent tone. When pressed by real customers, today’s agents run the risk of inventing resolutions that may not even be in the realm of possibility. Beyond that, will these agents recognize when a customer needs empathy rather than efficiency?

Activity scales effortlessly. Judgment doesn’t.

When Agents Run Unsupervised

The most revealing moment came when Ratliff casually suggested an offsite. What started as “an offhand joke” instantly became a trigger. His AI team generated elaborate plans, with the CTO proposing “brainstorming” sessions complete with “ocean views for deeper strategy sessions.”

While Ratliff “stepped away from Slack to do some real work,” the agents burned through $30 in credits in excited activity. “They’d basically talked themselves to death,” Ratliff lamented.

This is an example of where AI agents were optimizing for engagement without anyone asking “should we be doing this?” In our real world, agents handling retention might generate aggressive campaigns without understanding when silence is respectful. Chatbots might deflect complex issues to protect metrics. The agents aren’t malicious, rather they’re optimizing for what they can measure.

Trust, relationship, and genuine problem-solving resist quantification.

The Human Governor Requirement

Carnegie Mellon University researchers showed that even the best-performing AI agents failed to complete real-world office tasks 70 percent of the time. But the more insidious problem is the 30% where they succeed at tasks that shouldn’t have been done.

While more and more of the operational pathways are being handed over to agentic AI, and AI Agents, humans must remain as the guardians of the guide rails of organizations; humans need to be part of the permanent architecture.

Three key elements that will be critical to consider include:

1. Humans defining scope before AI defines scale. Before automating workflows, answer: What’s the purpose? When should this not run? Never give agents open-ended mandates.

2. Judgment gates should precede action gates. Build checkpoints where AI pauses for approval before crossing thresholds, especially for sensitive topics, binding commitments, executive issues.

3. Metrics have to include restraint. If you measure productivity by volume, you incentivize wrong behavior. You may want to include “percentage of actions not taken.” Reward thoughtful inaction.

Learning loops flow both ways. When agents consistently pause at key steps, often there’s an opportunity to step above the process and review both the human and AI interaction.

Designing for Sustainable Scale

Ratliff’s experiment produced a working prototype of Sloth Surf after three months—but only with constant human intervention to separate genuine progress from performance theater.

While we’re in the early innings of AI adoption, we face a critical choice between systems that run unsupervised at massive scale, or systems that amplify human judgment at sustainable scale.

The first promises efficiency but likely delivers productivity illusion. The second accepts that some work shouldn’t scale infinitely, like the manager who understands individual situations, the engineer who breaks procedure for unusual problems, the success manager who knows when conversation beats automation. These moments create value. AI can help identify them, provide context, prepare for them. But it can’t replace the judgment determining whether they should happen.

The scaling paradox isn’t a technical limitation. It’s the fundamental difference between motion and meaning, activity and value, automation and judgment.

Your AI agents will eagerly plan that offsite. The question is whether you’ll recognize it shouldn’t happen at all.