AI is not the shortcut people think it is

AI’s marketing works because it seduces users with the dream of getting all the benefits without doing the hard yards of actually managing corporate data.

I am an AI sceptic.

I am sceptical of large language models (LLMs, which are the AI flavour of the day) and their user-facing manifestations, such as ChatGPT and Claude. This is not because they are useless or uninteresting. As a veteran analyst and software developer, I think they are semi-magical in how they use vast amounts of training data coupled with highly advanced statistics and probability spaces to produce plausible texts.

No, I am sceptical because LLMs seem to do a lot of things well by creating outputs that appear authoritative. But they are in fact useless or even dangerous when applied to domains that require genuine experience and accountability, such as engineering, law, accounting and urban planning.

In an ideal world data is organized and legible making it easy uncover patterns and insights.

Humans are evolutionarily tuned to find meaning in language and to respond to language based on the tone and idiom of what we hear and read. LLMs are fantastically good at producing simulations of language that mimic the “feel” of real speech and writing. This is a trap, and I see it regularly trapping customer and peers that think: “We can use AI to finally make sense of all the information we have!”

Let’s unpack this statement. First, most organizations know at some level that their information management practices are terrible. With notable exceptions such as in banking, logistics and tech, organizations do not have data management, auditability and automation as core competencies. For years they have been accumulating reports, spreadsheets, images and others documents with a diverse range of filetypes that are spread across their corporate networks. It is effectively a big sea of unstructured data. It is also likely that there are islands of structured data related to accounting, customer relations and other core business processes.

Most organizations’ data management practices are terrible, making it impossible to tell the good data from the bad.

The trap is in thinking that all of the unstructured data can be magically transmuted into something useful by feeding it to an LLM and then letting anyone ask it questions to get authoritative answers. The trap is compounded by thinking that the islands of structured data can also be fed into an LLM and that this magical new “one ring to rule them all” can be reliably used to query the corporate memory and make inferences about the future. This is certainly how LLMs are being marketed to enterprises. These AIs’ marketing works because they seduce users with the dream of getting all this capability without doing the hard yards of actually structuring existing corporate data. It is especially effective with senior leaders who are rewarded for efficiency and cost containment but aren’t the domain experts who drive the day-to-day business transactions and outputs.

For the sake of argument, let’s say Acme Inc. decides to go for it and feed all their corporate data into the AI mill. Let’s overlook the security complexities around doing this, which would be a whole other article. To be sporting we can even assume (incorrectly) that the LLM will never hallucinate answers. We can even (for now) ignore the cost uncertainty associated with token-based billing.

What does Acme have and what is it good for? Honestly, not much.

Yes, anyone in the company can now use natural language to get answers about lots of subjects from all the data in the organization. The answers probably even mimic the idioms and rules of style of the organization. But they are not reliable. In this example we’re leaving hallucinations aside, so why are the outputs from this LLM unreliable? Because they are not authoritative.

What makes something authoritative? When something is the output of a person (an authority) who has the knowledge and experience to discern the risks and opportunities related to a subject and to formulate a solution that is likely to succeed. Good organizations are able to codify the habits of their authorities into systems and processes that allows less experienced people to do some of the work. This is also how people go from being novices to authorities: by following in the footsteps of those who came before them and, ideally, eventually surpassing them.

Why can’t an LLM with access to all an organization’s data be authoritative? Because companies have both good and bad examples of everything. Contract A had great margins, but Contract B lost a ton. From the LLM’s point of view, both are equally valid examples, but Beth (the COO) knows why one was a winner and one was a loser. Beth is an authority, the LLM is a stochastic parrot. 

And this is just an obvious example. There are countless other ways for LLMs to fail the authority test: using the wrong jurisdiction for tax or legal purposes, failing to follow if-this-then-that rules in policy interpretation or using the work of a person who was terminated for poor performance. When you turn an LLM loose on all your data and also factor in the hallucinations and token costs, you don’t get authority; you get something that looks a lot like an unskilled, sometimes dishonest, junior employee with an expense account problem.

Results produced by AI can look like an unskilled, sometimes dishonest, junior employee with an expense account problem.

If an organization wants to leverage complex data an LLM is simply not enough on its own. The hard yards of organizing and categorizing data are still necessary. Separate the good contracts from the bad, structure information around jurisdictions or other relevant axes, sequester the output of bad performers.

So yes, I am an LLM sceptic, but not because LLMs are useless. I am a sceptic because LLMs are being used to sell a seductive mirage of boundless intelligence based on a slurry of data. Garbage in, garbage out is still a thing. Experts and authorities are still necessary. The discipline of information management, the acquisition of domain knowledge and the process of becoming an expert are not going anywhere. Look for people who quietly exemplify these traits and tune out the people selling a shortcut to perceived expertise. If you have competent people and your data is well organized, you have lots of great options for leveraging it. Including LLMs.

Want to chat more about LLMs, authoritative data, and stochastic parrots? Connect with Erin on LinkedIn.