AI on Trial: Legal Products and Their Intended Use
The best part of my day, every day, is working with our customers. My job allows me to focus on how I can make their work easier, faster and better. I steward Practical Law and our newly launched Ask Practical Law AI tool for Thomson Reuters. Created by a team of expert in-house lawyers, Practical Law provides trusted guidance in the form of Practice Notes, Standard Documents and Clauses, Checklists, Toolkits, Legal Updates, market trend analysis and much more. Ask Practical Law AI is designed to help locate and synthesize those know-how resources only from Practical Law.
Recently, Thomson Reuters has been made aware of a new paper published by Stanford University, which purports to assess the reliability of AI primary law legal research tools. In this study, Stanford unfortunately used Ask Practical Law AI for primary law legal research. However, as it was not built for, nor intended to be used for primary law legal research, it understandably did not perform well in this environment. We designed Ask Practical Law AI to decline to answer questions unrelated to the Practical Law source content (such as many of those in the study). In the days before the study was published, we were offered a pre-read and at this point flagged to the researchers that the solution best suited to this study would be Westlaw Precision with AI-Assisted Research and provided access to the product. We are keen to work with them on a follow-up study – confident the results will be positive because that’s what we hear from our customers every day.
Thomson Reuters is supportive of examinations of hallucinations and inaccuracies with the use of AI, but we believe it is critically important to use products for their intended purposes and that tests should use realistic, real-world queries.
We also believe in fair and transparent evaluation of AI products and agree that benchmarking solutions is a valuable way to instill trust and transparency in the industry for the benefit of our customers. Hallucinations and inaccuracies are well-known issues with the use of large language models, and AI-Assisted Research on Westlaw Precision and Practical Law’s Ask Practical Law AI share with our customers on their homepages (and have since their initial releases) that they can occasionally produce inaccuracies and should be used as part of a research process with other research tools to further improve accuracy. When used as part of a thorough research process, these tools are enormously helpful for accelerating and improving research, and we’ve had many customers tell us it saves them hours in their research process. But the results from Ask Practical Law AI or Westlaw’s AI-Assisted Research should never be used as a replacement for a thorough research process, and we’ve been clear with our customers about that from day one.
We have great confidence in the ability and value of using AI in legal services in partnership with humans to achieve greater work product and productivity. I look forward to the opportunity to work with our customers to continue on this journey.
This is a guest post from Emily Colbert, senior vice president of product management, Thomson Reuters.
