“It’s your end users who are going to get value out of these tools, so you need to think about what matters to them.”– Ali Cook, Strategy Lead, Josef
Watch the full chat with Ali Cook and Jake Brown.
It’s a critical question for anyone looking to leverage GenAI today in a high-risk area like legal and compliance. If you’re going to offload high-volume FAQ work to self-service tools, they need to be dependable and keep risk to a minimum.
In this fireside chat, Josef Co-founder and COO Sam Flynn talks with in-house legaltech experts Strategy Lead Ali Cook and Head of Legal Technology Jake Brown. Together, they share insights from their work with Josef customers, and their reliability framework created in collaboration with Cornell and NYU.
They dive into questions like:
Watch the full chat to find out, or jump straight to their tips below.
“Get comfortable with the fact that your tools may produce answers that aren’t necessarily phrased the way you would phrase them,” Jake says.
“But that’s okay! As a former practicing attorney, I prefer my own phrasing over anyone else’s, but consider whether your tool’s phrasing actually introduces any risk.”
Sam adds, “It’s about bringing a healthy level of skepticism and practicality to your approach.”
“A GenAI tool isn’t going to include all of the elements that you or another subject matter expert would include, but think:
If so, of course, we need to address it and moderate the tool’s response immediately. If not, we might still want to address it—perhaps by capturing our tone of voice and our way of working—but it shouldn’t block us.”
“It’s your end users who are going to get value out of these tools, so you need to think about what matters to them.”– Ali Cook, Strategy Lead, Josef
Ali says, “I think in the flurry and hype of GenAI, people have forgotten to center end users and consider them early on.”
There are lots of red-herrings in reliability testing, and by focusing on your end-users, you’re going to avoid a lot of them.
“It’s your end users who are going to get value out of these tools, so you need to think about what matters to them. Consider:
Ali continues, “If you focus on what your end users really need when testing, you’ll be headed in the right direction.”
That said, you do need to be cautious, particularly when testing a tool for the first time.
Talking about a recent assessment of a generic search tool that a customer was using alongside Josef Q, Sam shares: “On the surface, everything looked fine. Everything seemed hunky-dory.”
“The answers the tool was producing seemed accurate and beautifully formatted. It was referencing what appeared to be accurate documents. But, after digging in, we found full-on hallucinations.”
“When you’re using technology that hasn’t been built for high-risk settings like Josef has, you’re introducing a new level of risk, so be wary!”
In 2023, Fnatic’s in-house legal team took their automation to the next level with Josef Q.
Moving beyond document and workflow automation, they launched a suite of self-service tools to answer questions about contracts, handbooks, and a range of HR policies. They even connected them with Slack, ensuring business users can get answers right where they work and the legal team can track engagement in real-time.
Learn about Fnatic’s “game-changing” Legal and People Josef Q tools.
Book a demo to see first-hand how easy it is for your whole team to get up and running with legal automation.
We'll be in touch soon to arrange a time to speak.