SID.ai (YC S23)

70% of production queries fail with Naive RAG. Spoiler: It's not the retrieval model. No, that's not a typo. That's what we found after: - Processing over 800 billion tokens - Analyzing 10,000 real-world queries - Countless hours of research Sure, you can hit 98% accuracy in a demo with a curated dataset and softball questions. But the real world is messier, more complex, and far less forgiving… Let's break down 10,000 real-world queries: 1️⃣ Only 32% of queries are answerable by vector search alone. 2️⃣ 22% are "meta-queries" (e.g., "Show me recent emails about X") 3️⃣ 20% are off-topic, with no answer in the data store 4️⃣ 10% demand multiple sub-queries (e.g., "Compare X and Y") 5️⃣ 16% are other edge cases See the problem? Most naive RAG systems – relying solely on vector + keyword search – are shooting blanks 68% of the time. They're fundamentally limited, even with perfect search capabilities on that 32%. These systems not only choke on time-based queries and comparisons, they can't admit when they're out of their depth. Worse, they pollute unfit queries with irrelevant info, leading to hallucinations - confidently delivered as fact. It's simple to see why adding naive RAG can DECREASE your AI's accuracy. RAG still holds immense potential – the key is moving beyond basic implementations, towards more sophisticated systems that feature: - Advanced query understanding - Intelligent routing mechanisms - Meta-data query capabilities - Adaptive, context-aware processing - Robust off-topic detection - Among many others 🙂 Unsurprisingly – just like web search – RAG with high accuracy requires LOTS of long-tail engineering efforts. Thoughts? 🎥 Curious to learn more? Watch as my co-founder Max Rumpf breaks this down in detail in Humanloop High Agency Podcast!

It's been a year since SID.ai (YC S23) lit up NYC Times Square. 🗽✨ When we started, RAG was just three letters most people didn't understand. Now? It's the cornerstone of (almost) every AI system out there. We've fumbled, failed, got hurt, questioned everything (including ourselves), celebrated tiny wins that kept us going, learned, and grown. A lot. Dreams are what keep me going – so I can't help but imagine that we'll be back there soon(ish) with our IPO announcement 😉 To our amazing customers, dedicated team, and unwavering investors! 🫶

"End of RAG" or "Jevons Paradox"? Some say long-context language models (LCLMs) will kill RAG. Even RAG's inventor himself. Their logic? Why fish for information when you can flood the prompt with an ocean of data? We think they overlook a fundamental truth about human nature and technological progress… Jevons Paradox – historically, when a resource's efficiency increased, its consumption skyrocketed: Electricity got cheaper, so we ended up using more. With any finite context window size, there's always an application that benefits from having even more information: Alas you need some mechanism – retrieval – to extend past the window. Even if we were to reach infinite context length, RAG remains relevant due to: • Efficiency: Processing queries with billions of tokens is costly and slow. Retrieval lets us pick just the important ones. • Data Access: LCLMs still need bridges to various data sources. • Better Reasoning: RAG can cut through noise and handle complex tasks like temporal reasoning, multi-step logic, and hierarchical data far better than LCLMs. The future isn't RAG vs. Long Context — it's RAG + Long Context. What's your take? Will RAG become obsolete, or will it evolve alongside LCLMs? 🤔 More on this topic with Raza Habib & my co-founder Max Rumpf on the Humanloop "High Agency" podcast. 👇

What if San Francisco's secret isn't its network, but the crazy? We've all heard the hype around SF: land of unicorns, bottomless VC pockets, game-changing networks. But building a company on both sides of the Atlantic has shown me a defining quality of the Bay Area that often goes unnoticed. Growing up in Switzerland is a masterclass in societal perfection. The downside? Conformity. People call you "special" as an insult. Even those that claim to value innovation secretly despise what's truly different. The nail that sticks out... gets hammered down (especially girls with big dreams). In SF, that same nail gets celebrated. This is the city’s unfair advantage: – Not the startup ecosystem. – Not the capital. – But the freedom to be weird. The radical acceptance of the unconventional. It's in the city's DNA – from the Gold Rush of '49 to the Summer of Love in '67. SF has long been a haven for those who want more than the status quo. Why it's so important? You can't build a generation-defining company by coloring inside the lines. You need to cultivate the space to be strange, to think different, to be contrarian, to see what others can't – or won't. San Francisco doesn't just allow you to be weird. It dares you to be weirder.

RAG Complexity ≠ RAG Accuracy Many trendy RAG features fail to significantly improve real-world performance... I've been building RAG for 2+ years now. And every week brings a shiny new approach, research paper, Twitter thread. It's tempting to cram every cutting-edge technique into your system. But more buttons on a coffee machine don't necessarily mean tastier coffee - just more ways to mess up your brew. Just like more complexity in a RAG system doesn't necessarily guarantee better performance... Our philosophy? Ruthless pragmatism! Every new technique must prove its worth. Does it improve the end-to-end performance of our customers' use-cases in the real world? No metric increase – no inclusion. No exceptions! This might seem obvious. Yet it's surprising how many chase complexity for complexity's sake. We've tested countless RAG components rigorously – tree-based chunking, sliding window with overlap, dense vector search, query expansion, multi-vector indexing – the list does on... Some work wonders – the really do! – others... not so much. The kicker? It all depends on your specific use case. Happy to give you some pointers if you're thinking about implementing a specific feature at the moment! Link to the full conversation of Raza Habib & Max Rumpf on Humanloop "High Agency" podcast in the comments.

The Importance of Being Frugal with Human Tokens Max Rumpf wrote an excellent article on "Amdahl's Argument for AI", suggesting that humans-in-the-loop might be the biggest limiting factor in boosting productivity through AI. My key takeaway: Copilots are a temporary phenomenon – agent-like approaches are here to stay. Why? Speedup of ChatGPT & GitHub's copilot: The speed at which humans can process and respond (about 1-3 tokens per second) sets a fundamental limit on productivity gains. For example, applications that require a human completion for every LLM completion can achieve a maximum speedup of ~2x, even when LLMs become 10x faster. Speedup of current agentic approaches: Cognition's Devon demonstrates the potential for greater productivity gains by requiring human completion only every ~10 iterations. At the moment, this approach leads to ~2.9x gain over ChatGPT with current model speed. Speedup of future AI agents: The real game-changer are AI agents in the 100-1000x range, i.e. those that only require human input every 100-1000 iterations. Granted, it will take some time to get there – but we're already seeing many approaches: Like human-less code execution from E2B, browsing from Browserbase or the context engine from SID.ai (YC S23). Right now, developers are frugal with LLM tokens due to their cost. However, I agree with Max that the most important thing to be frugal with are human tokens (both input and output) – because they will define the overall productivity speedup your application can provide.

Sam Altman’s advice at yesterday’s Turing event: Right now, AI startups fall into two distinct categories: Those (implicitly) betting that AI models will stay at the current capability level and those betting the models will get better. The former build things to address the current shortcomings. The latter want the models to get smarter, easier to prompt, more accurate, and faster – because that will make their own products better. Predictably, Sam recommended you to bet on the latter: Models will keep getting smarter and we won’t hit a wall on model intelligence “in the next millenium.” Otherwise, he said (jokingly), the “OpenAI killed my startup” meme will inevitably become your future. What’s your take on the development of model capabilities?

🥧 Pi Day seems like the perfect time to show you SID.ai (YC S23)'s brand new office at 3141 (π ≈ 3.141). We're thrilled to be part of the vibrant Mission district – sharing our block with incredible neighbours like MindsDB and OpenAI. If you ever need a change of scenery or find yourself nearby, please don't hesitate to drop in. We'd be thrilled to invite you in for a coffee or have you co-work with us for the day. Seriously, I mean it – come by anytime! Here's to many intense days and nights within these new four walls! 📦🦥