Arthur, a machine learning monitoring startup, has been developing tools designed to help companies use LLM more effectively. The company recently released an open-source tool, Arthur Bench, to help users find the best LLM for a particular dataset.
Arthur CEO and co-founder Adam Wenchel said that they have seen a lot of interest in generative AI and LLM, so they have invested a lot of energy in product creation. Given that ChatGPT has been released for less than a year, there is currently no organized way to measure the effectiveness of one tool relative to another; it is against this backdrop that Arthur Bench was born.
"The Arthur Bench addresses a key question we hear from every customer, which [among all the model choices] is best for your particular application."
Arthur Bench comes with a set of tools that you can use to systematically test performance; but its real value is in allowing you to test and measure how the types of prompts your users use for a particular application perform on different LLMs.
According to the introduction, Bench can help assess:
- Standardize the workflow of LLM assessments with a common interface across tasks and use cases
- Test whether an open source LLM can handle your specific data as well as the top closed source LLM API providers
- Convert rankings from LLM leaderboards and benchmarks into scores for the actual use cases you care about
Wenchel points out thatyou could test 100 different cues and see how two different LLMs -- such as Anthropic versus OpenAI -- differ in the types of cues a user might use. What's more, you can test at scale to better decide which pattern is best for your specific use case.