Root Signals

Busy, busy Autumn, but luckily Santa 🎅 was so helpful that he found a few minutes to share our Season’s Greetings! Thank you for everything, and see you after a well-deserved short break. 🎄

We are in NeurIPS 2024 in Vancouver! Hit Ari Heljakka or Oguzhan (Ouz) Gencoglu up for a chat about LLM evaluations, LLM-Judges and how we bring state-of-the-art AI research to production at scale.

🌎 Excited to be at AWS re:Invent in Las Vegas! Drop Ari Heljakka and Root Signals a line to chat about the latest in #EvalOps and #LLMevaluation.

🚨 Webinar Reminder: TOP-10 Misconceptions about LLM Judges in production is already soon. Join Ari Heljakka, Oguzhan (Ouz) Gencoglu, and Data Science Salon to explore: • Key misconceptions about #LLMjudges • Best practices for #EvalOps • How to build reliable, scalable evaluation systems Register now: https://lnkd.in/dZNMnWDh #LLMjudges #EvalOps

🌐AWS re:Invent Guide for #GenAI Developers If you're an engineer working with LLMs, Amazon Web Services (AWS) re:Invent 2024 in Las Vegas has a lot to offer. We’ve put together a guide to help you get the most out of the event. Here’s what we cover: 🆕 What’s new in 2024 for developers building with LLMs 📋 How to prepare for the event to maximize your time 🔎 A quick guide to understanding session abbreviations and tracks 🎯 Our curated list of sessions—from keynotes to hands-on bootcamps—that are especially valuable for GenAI engineers Whether you’re looking to learn about the latest tools, dive deep into technical sessions, or connect with the GenAI developer community, this guide has everything you need to navigate #AWSreInvent effectively. Check out the guide here: https://lnkd.in/dzrQ6fFz

#LLM judges in production can transform AI evaluation but comes with challenges like reliability, explainability, cost unpredictability, and maintainability when not implemented properly. 👨💻 Join Ari Heljakka, Oguzhan (Ouz) Gencoglu, and Data Science Salon to explore: • Key misconceptions about #LLMjudges • Best practices for #EvalOps. • How to build reliable, scalable evaluation systems. 📆 Don’t miss this webinar! Learn more at: https://lnkd.in/dEusmY-5

🚀 Root Signals is at Slush in Helsinki, Nov 20-21! Meet Ari Heljakka and Oguzhan (Ouz) Gencoglu to explore the future of #EvalOps and the best practices for #LLMevaluation. See you at #Slush2024?!

It was a pleasure to host 40+ LLM experts and developers at our offices for the LLM Developer event organized by Tuomas Lounamaa, Symposium AI & Root Signals. The evening featured insightful talks from Aapo Tanskanen, Rasmus Toivanen, Markus S. and a demo from our Head of AI, Oguzhan (Ouz) Gencoglu, showcasing our Control, Evaluation & Observability Platform for GenAI applications. A heartfelt thank you to everyone who attended and made this gathering so engaging. We’re excited to continue building and learning with this incredible community. Stay tuned for more events focused on advancing LLM development!

#EvalsTuesdays Week 5 - Confirmation Bias in LLM-Judges LLM-as-a-Judge is the gift that keeps on giving (both joys and headaches). This week, we're tackling yet another bias that sneaks into our LLM evaluations: Confirmation Bias. Confirmation Bias: The tendency of LLM-Judges to favor responses that confirm their existing beliefs or the information presented in the prompt, while ignoring evidence to the contrary. In simpler terms, they might be agreeing with themselves a bit too much. Why does this matter? 🔍 It can lead to skewed evaluations, where certain types of responses are consistently overvalued or undervalued. 🧠 LLM-Judges may overlook errors or hallucinations if the response "sounds right" based on prior context. 🌐 This bias can be especially problematic in domains requiring critical analysis or when evaluating for factual accuracy. So, what's causing this? LLMs are trained on vast amounts of data, and they're great at picking up patterns. However, they also tend to reinforce patterns they've seen before. When an LLM-Judge evaluates a response, it might be more inclined to agree with content that aligns with those patterns, even if it's not the most accurate or helpful. How do we fight back? ✅ Diversify your prompts: Introduce variability in your evaluation prompts to prevent the model from getting too cozy with any one perspective. ✅ Encourage critical thinking: Incorporate instructions that nudge the LLM-Judge to consider alternative viewpoints or to critically assess the response. ✅ Meta-evaluation: Regularly test your LLM-Judges with known examples where the correct evaluation is counterintuitive, ensuring they're not just coasting on confirmation bias. At Root Signals, we obsess over these nuances so you don't have to. Our LLM-Judges are fine-tuned to spot not just the obvious issues but the subtle ones that can slip through the cracks. And most importantly, we provide a systematic and easy way to meta-evaluate = measure and tune your LLM-Judges. Remember, in the world of LLMs, vigilance is key. Don't let your judges get complacent - challenge them, test them, and keep them sharp. What's next? Maybe we'll dive into the rabbit hole of Chain-of-Thought prompting in LLM-Judges, maybe something else. Stay tuned!

#EvalsTuesdays Week 4 - Verbosity Bias in LLM-Judges Creating reliable evaluation metrics for #LLMs by using LLMs, i.e. LLM-as-a-Judge, is more than simply writing an evaluation prompt and calling an API. One reason is LLM-Judges are full of biases and verbosity is one of them. Verbosity bias: LLM judges favor longer responses, even if they are not as clear, high-quality, or accurate as shorter alternatives. They are like lazy teachers who give high grades to longer essays because they are long. Here is a quick example from Google's #Gemini ⬇ . The first answer is not only more to-the-point but also more precise. It is simply more helpful. Yet, Gemini scores the rambling management consultant answer with a higher score. When it comes to being able to measure your LLM applications, the devil is in the details. If your metrics are not reliable in the first place, what's the point? Verbosity bias has a massive effect in all sorts of use cases where LLM-Judges are utilized. Judge scores need to be calibrated and normalized with respect to the length of the text that is being evaluated. But this normalization is not universal either and is model dependent. We worry about all these things at Root Signals so that our users don't need to. I would love to hear how you evaluate your GenAI appplications?