When building GenAI-powered solutions, whether it’s extracting metadata in the Media as Text design pattern or using context to craft the perfect response to user queries, the quality of the generated content is always top of mind. This is where large language model (LLM) evaluations play a critical role – ensuring that models are not only accurate and reliable, but also optimised for cost-efficiency and performance, and aligned with your strategic business objectives to deliver impactful results.

LLM Evaluation: Why It Matters

LLM evaluation is the crucial process of ensuring that AI models truly understand and respond to user queries in a meaningful way. It’s about testing how well the generated content aligns with the context and whether the responses genuinely hit the mark. This step is essential not just for accuracy, but for making sure that the model is fully grasping the request. Through effective evaluation, we fine-tune models, refine prompts, and ultimately create solutions that better align with user needs, ensuring a more reliable and effective AI experience (Figure 1) [1].

Figure 1: Why LLM evaluation matters: reliability, safety, and driving wider adoption

The Key to Unlocking Reliable LLM Performance: A Balanced Evaluation Approach

Evaluating LLMs is not just a step – it’s a critical process for ensuring they deliver accuracy, reliability, and real-world effectiveness across countless applications. Since LLMs are applied in so many diverse scenarios, a one-size-fits-all evaluation method simply won’t cut it. Instead, a dynamic blend of offline and online evaluation strategies offers a comprehensive view of model performance at every stage of its lifecycle.

Offline Evaluation: The First Step to Building Better LLMs

Offline evaluation plays a vital role in developing LLM-powered features, providing a controlled environment to assess performance before real-world deployment. By using in-domain metrics, these evaluations ensure that the model aligns with the specific needs of the application. Evaluation methods can be either reference-based, which involves comparing outputs to predefined ‘ground truth’ responses, or reference-free, which evaluates outputs based on qualitative, domain-specific criteria. These methods can then be further categorised into pointwise evaluation, where each output is judged independently, or pairwise evaluation, where two outputs are directly compared side by side.

Figure 2: Offline Evaluations

When a reference, or “ground truth”, is available, evaluation focuses on how closely the model’s output matches expected results-similar to traditional ML metrics like F1-score, Recall, ROUGE, BLEU, and Exact Match. Tools like Google Cloud Vertex AI’s evaluation service [4, 5] automate this process, applying mathematical comparisons to assess performance. However, these conventional methods often overlook the deeper semantics of LLM-generated text, making it essential to explore additional evaluation techniques.

For cases without predefined answers, models can still be assessed using proxy metrics such as coherence, fluency, relevance, or potential harm. A popular approach here is LLM-as-a-Judge [6], where a separate model evaluates the outputs based on custom-defined criteria (Figure 3). GCP’s Vertex AI evaluation offers explainable, model-based assessments, supporting both pointwise and pairwise.

While LLM-based approaches are typically used in reference-free scenarios, they can also be applied for reference-based evaluations.

Figure 3:  LLM as-a-judge taxonomy

By combining structured offline evaluation with a nuanced understanding of LLM behaviour, developers can fine-tune models and design better prompts with greater confidence-laying the foundation for robust and reliable AI systems. Providing deep insights into LLM performance empowers teams to make data-driven decisions and optimise models for diverse applications and use cases (Figure 4).

Figure 4:  LLM performance evaluation: Empowering teams to optimise models across diverse applications (Note: data are synthetic).

Online Evaluation: Measuring Real-World Impact

Offline evaluation is a key part of the development process, offering immediate scores and valuable benchmarks to identify the most promising large language models. However, it doesn’t fully capture how these models perform in real-world applications. This is where online evaluation becomes essential. Unlike offline evaluation, where feedback is explicit and instantaneous, online evaluation often relies on implicit signals-such as users rephrasing questions or abandoning a chat-to infer dissatisfaction. While explicit feedback like thumbs-up/down buttons is helpful, it’s not always used consistently. Additionally, some performance indicators are long-term, such as whether users return to the product over time. Real-time A/B testing helps address these complexities, ensuring that GenAI solutions are leveraging the most effective LLM for each task and context. By combining both offline and online insights, we can deliver higher-quality results, drive superior user experiences, and generate measurable business impact.(Figure 5).

Figure 5: LLM performance evaluation: Monitoring performance in production (Note: data are synthetic and user info are publicly available data from [7].