Why ROI on AI in Research Is Harder Than It Looks

This week's Research Leadership Network session was on the ROI of research tools, and of AI in research. The group was made up of leaders of research functions across law, consulting and technology, and the conversation went well beyond the numbers.

ROI is the wrong question for now

As the group anticipated, ROI is not really the right question at this stage. Firms are still very much at the investment stage, well before returns can be meaningfully measured.

There was a useful distinction, though. Some members have found they can use ROI to prove the value of the research team itself, by using AI to track specific research activity to specific wins for the firm. Measuring the ROI of AI in research is another thing entirely.

Is AI necessary for every task?

The group felt we are finally getting to the point where businesses are asking whether AI is necessary for all the tasks it is being used for. Some members are trying to measure cost, and are also giving employees the ability to measure cost themselves.

Analysts will typically always want to use the most powerful model. But if you can show them the cost, perhaps it will encourage more responsibility.

The burden is landing on juniors, and on everyone else

Juniors are being asked to learn their role and, at the same time, use a very large amount of AI. More analysis may be staying with juniors, because so much can now be done with AI. Some in the group are hearing encouraging things on hiring juniors.

For everyone else, the easier, repeatable parts of work, which made up a large part of people's days, are being taken out. What's left is the harder edge cases, so the work that remains is more demanding across the board. That raises the question of what to measure, and what the impact looks like across the whole business.

Pricing pressure from both sides

Clients are asking for lower fees when they know a professional services business uses AI. At the same time, information providers who have allowed free consumption are talking about moving to tokens, but even they can't yet estimate what pricing will look like after the transition.

As in similar conversations with buyers of information, the cost of what's being quoted for MCP access makes hiring people far more attractive at this point. A quote equivalent to the cost of hiring two people into the team is currently seen as far more valuable than integrating something through an MCP, in most cases.

A crowded and fragmented tool landscape

AI is being embedded in many legacy research tools and products. Many businesses have industry-specific LLMs alongside the major products such as Claude and Copilot, and a lot of the broader information resources now include some form of AI. The big question is how to create a one-stop shop, while still making sure the right tool is used for the right task.

Users are bypassing research teams with their own LLMs

Many users are going straight to their own LLM and bypassing research teams entirely. These tools suffer from paywall blindness: they can't see what's in premium sources unless those sources are integrated. They also pick up figures from across the wider internet, from non-premium sources or sources outside their original context, without any of the underlying methodology.

Teams are trying to add governance, whether through skills within the LLMs or external tools that challenge users on whether they are using the right product for the task. Some of these approaches aim to direct users back to research teams where premium information exists for what they are looking for.

The research operating model is still unsettled

There's something of an arms race underway, with some great new products coming to market. Options include offshore teams, onshore teams, putting products on partner desktops and integrating through MCPs, and none of it is settled yet. Nobody knows what the research model should look like, and it changes month to month, which makes planning close to impossible. There was also discussion of how to bring in new talent, but the function first has to exist over the long term.

Defensibility matters

It is very easy right now to sound like an expert when you know nothing, and just as easy to build dashboards that look beautiful. Fundamentally, the outputs need to be defensible.

Skills and certifications

Several useful certifications were mentioned along the way: Databricks, Claude and GitHub.