Whenever a new LLM model is released so are the benchmark comparison charts. However these benchmark scores aren’t really an indication of how a model will perform for your use case and should be taken with a pinch of salt .

Benchmarks often focus on specific tasks, like question answering or text summarization. These tasks might not reflect the nuances of your actual use case. A LLM that is great at multiple-choice reading comprehension tests or get 90% or more on some random domain specific set of questions might struggle to write engaging marketing copy or answer questions in another domain.

LLMs are trained on huge datasets, but that data might not mirror your specific domain. An LLM trained on generic news articles might falter when confronted with legal documents. This mismatch between training data and real-world application can lead to inaccuracies and biases.

Benchmarks can be susceptible to cherry-picking. Researchers might fine-tune the prompts or data used to achieve optimal performance. This doesn’t reflect how the LLM will behave with everyday user input, which can be messy and unpredictable.

LLMs are great at generating output that sounds confident even when wrong, but often lack transparency in how they arrive at those answers. This makes it difficult to debug errors or understand why a seemingly reasonable prompt generates nonsensical results.

Okay you may be asking then how do I choose a model to fit my use case ?

First a detour regarding my dev environment .

In https://grumpygrace.dev/posts/dev-locally-with-llms/ I described my dev environment and when needed use Cloud workstations .

I’ve now upgraded to also having a dev server with way more memory to play around with larger models and to gain a performance boost as waiting around for a response from a LLM on my laptop depending on the model could take more time than I had patience for.

I can still mess around with models on my laptop but when I need more oomph and to write posts like this one I use my dev server. The additional software includes proxmox for virtualization and open-webui . Open-webui provides a nice interface that integrates with ollama making it even easier to swap out models. It has some great features including abstracting the glue work of RAG integration which is great if you just want to test things out before writing that glue, multi-modal support and so much more that I really am not doing it any justice.

Okay back to the point of this post you need to choose a model based on various factors that invariably have very little to do with the model benchmark stats.

• You need to focus on your use case by clearly defining the specific tasks you need the LLM to perform.
• Ensure the LLM is trained on high-quality data that closely resembles your domain.
• Design your system to leverage human feedback alongside LLM capabilities.
• Continuously test and monitor the LLM’s performance in a realistic setting that reflects how it will be integrated into your use case solution and how it will be used.
• Create your own relevant benchmarks

I’ll walk you through a very simple approach using open-webui to help me. I was initially going to just use a single prompt with LLMs that have not been fine tuned . However llama3’s response prompted me to provide a follow up prompt . I will compare 6 local models with the response from the chat bots Gemini & copilot

Using the open-webui interface you select the model you want from the drop down list and enter your prompt. The screenshot below is the response when using gemma as the selected model:

The apple question

Prompt " What colour are apples?"

The first returned response is used for this simple test except where I felt obliged to give a follow up prompt . The same prompt can give different responses when given the exact same prompt again.

So as you can see depending on the model you can get a very simplistic albeit accurate response through to responses that provide more detail on the varieties of apples. A single prompt may provide you with the results you want or maybe you need to provide a follow up prompt. What type of response do you want from your application that integrates LLMs?

Gemma:9b and llama2:7b gave the exact same concise response. Llama3:8b made an assumption that a simple response was appropriate but following up with another prompt generated more detail.

I was impressed with phi3:4b despite it not actually replying with a straightforward response such as “apples can be various colours” but going straight into providing detail on different coloured apples and allowing you to infer the answer. Phi3:4b is the smallest model yet its response was pretty comprehensive despite its slightly weird response. Mistral:7b pointedly answered the question. I’d not looked at Mistral before now but some folks I know are keen on this model.

Llava-llama3:8b started off exactly the same as the mistral model & I thought I wasn’t going to include it because of that, but it kept going. It was incredibly verbose and even self corrected itself as you can see from its response. It was also very polite and asked itself questions to get more detail (read the response to see that). However it also kept saying goodbye (to itself?) and asking any other questions, if I can help you like it was caught in some self perpetuated loop. I stopped it as I am sure it would still be going now. To understand a bit more about LLaVA based models these are good starting points LLaVA [2304.08485] Visual Instruction Tuning

The closed models :

Gemini’s response was a bit short but it did manage in that concise response to answer the question directly, explain why apples have different coloured skin and made me google white apples ! And yes white apples do exist, one variety is called the white cloud apple !

Copilot came back with a nice response with plenty of images and links ( I cropped the image links from the response copied here)