🔥 Last week's Portkey Office Hour was packed with production insights: - Gemini outperforming GPT-4o for Hinglish translation - How Springworks & Haptik cut latency in HALF with managed gateways - Real talk on RAG bottlenecks & scaling challenges The best part? Engineers from Springworks and Haptik shared their actual implementation patterns: - Virtual key mapping for faster prototyping - Pre-scale resource monitoring tricks - Smart caching strategies that worked And yes, we sent everyone Theobroma cookies because great conversations deserve great treats 🍪 Join this week's discussion with other AI builders: https://lnkd.in/gP3559WP
Portkey
Technology, Information and Internet
San Francisco, California 4,528 followers
Production Stack for Gen AI
About us
AI Gateway, Guardrails, and Governance. Processing 14 Billion+ LLM tokens every day. Backed by Lightspeed.
- Website
-
https://portkey.ai
External link for Portkey
- Industry
- Technology, Information and Internet
- Company size
- 51-200 employees
- Headquarters
- San Francisco, California
- Type
- Privately Held
- Founded
- 2023
Locations
-
Primary
Get directions
San Francisco, California, US
-
Get directions
Bengaluru, Karnataka, IN
Employees at Portkey
Updates
-
This is not all — we've also gone ahead and mapped the entire MCP universe. All MCP servers and their implementations, in one place. It's the most comprehensive directory of what you can build with MCP. The ecosystem spans: - Data & Storage - Cloud & Infrastructure - Development Tools - Content & Search - AI & Memory - Productivity - System Utilities Browse here: https://lnkd.in/gEtrMJKe
🧵 We gave AI an impossible request, then watched it come alive. Today, we're announcing MCP (Model Context Protocol) Client by Portkey — the world's first truly magical agent platform that turns "write me an app" into "here's your deployed app." No complex setup. No integration hell. Just pure possibility. See it in action 👇 and sign up for the waitlist → https://portkey.ai/mcp
-
🧵 We gave AI an impossible request, then watched it come alive. Today, we're announcing MCP (Model Context Protocol) Client by Portkey — the world's first truly magical agent platform that turns "write me an app" into "here's your deployed app." No complex setup. No integration hell. Just pure possibility. See it in action 👇 and sign up for the waitlist → https://portkey.ai/mcp
-
After evaluating 17 different platforms, this AI team replaced 2+ years of homegrown tooling with Portkey Prompts. Why? 1. Prompt partials + Mustache templates: Easily build modular, reusable prompts 2. Robust versioning & publishing: Confidently update and roll out changes 3. Simple SDKs & OpenAI-compatible APIs: Integrate seamlessly, no refactoring needed Bonus: Built-in monitoring for instant insights!
-
-
🔥 MASSIVE UPDATE to Portkey's AI Gateway! Introducing the Gateway Console - your new supercharged debugging companion that just works. Zero config needed. Zero extra tools required. What's new? - Built-in request logging & monitoring - Crystal-clear response tracking - Latency & status code monitoring - Quick-start guides that actually help - Copy-paste ready code samples for 250+ LLMs Getting started is dead simple: Just run npx @portkey-ai/gateway in your terminal and visit localhost:8787/public/ to see the console in action! We built this because our amazing community asked for it - and we're just getting started! Take it for a spin and tell us what you think 🚀
-
Portkey reposted this
Great LLM insights from Portkey. The LLM landscape is evolving from dominance by a single player to a competitive ecosystem with multiple strong LLM vendors. We've moved beyond experimentation—redundancy and reliability now underscore the critical role of AI infrastructure.
✨ LLMs in Prod Day 1: Provider Trends Unveiled We just analyzed production data across 2 trillion tokens to uncover how companies are actually using AI in the real world. The results? Let's just say the landscape is shifting faster than anyone predicted. 🏆 OpenAI: The Giant with Shifting Ground Still the undisputed leader, but here's where it gets interesting: • 24% monthly growth in API requests • 6% monthly growth in new organizations • Adoption dropped from 89% to 76% Steady dominance? Yes. But competitors are catching up faster than expected. Why? Keep reading. 👀 🌟The Surprise Player: Anthropic They've been absolutely on fire: • 61% monthly growth in requests—fastest among all providers • 22% monthly growth in organizations Each Claude release (Sonnet 3.5, Haiku-3.5) created visible adoption spikes. With 23% of Portkey orgs now powered by Anthropic, they've cemented themselves as OpenAI's most formidable rival. One thing's crystal clear: model releases matter. 🎯Gemini: Google's Late But Impressive Entry Despite being the newest player: • 49% monthly growth in requests • 9% growth in new organizations Late to the party? Yes. But users who've adopted Gemini are showing surprisingly deep usage patterns. It's a strong comeback story in the making. ☁️ The Cloud Provider Battle: Partnerships Make or Break Azure is leading the charge, with 3x more penetration than AWS Bedrock. But here’s the twist: Bedrock started the year with 10x the adoption of Vertex AI, but by year-end, it’s only 2x ahead. It’s clear now: your AI infrastructure is as strong as your model partnerships. Microsoft's early OpenAI partnerships are paying off BIG. →Azure is riding the OpenAI wave with 11% org growth and 23% request growth. E → Amazon Bedrock: The Deep Integration Story Here's a fascinating pattern: • 13% monthly growth in organizations • BUT 91% growth in requests Translation? When companies choose AWS, they go all in. It's not about breadth, it's about depth. → Vertex AI: The Quiet Climber • 43% monthly growth in requests They started slow, but they're gaining fast. The momentum is undeniable. The Biggest Shift Nobody's Talking About In just 10 months, the proportion of Portkey orgs using multiple providers jumped from 23% to 40%. Why? After a year of outages and capacity issues, companies learned a crucial lesson: redundancy isn't expensive, downtime is. This isn't about experimentation anymore. Companies are building serious infrastructure with multiple providers because they've realized you can't rely on just one provider. The AI market has matured beyond the hype—it's all about production reliability now. Tomorrow: We'll dive deep into how these providers actually perform in production. The numbers might surprise you. 😉 Follow Portkey for more insights from LLMs in Prod '24
-
-
-
-
-
+3
-
-
Did You Know: You can programatically create new virtual keys on Portkey that refer to your Azure deployments? We just added support to directly send your Azure deploymentConfig while making a Virtual Key! Docs ↓ https://lnkd.in/gDASTXGv
-
-
Portkey reposted this
Great LLM insights from Portkey. The LLM landscape is evolving from dominance by a single player to a competitive ecosystem with multiple strong LLM vendors. We've moved beyond experimentation—redundancy and reliability now underscore the critical role of AI infrastructure.
✨ LLMs in Prod Day 1: Provider Trends Unveiled We just analyzed production data across 2 trillion tokens to uncover how companies are actually using AI in the real world. The results? Let's just say the landscape is shifting faster than anyone predicted. 🏆 OpenAI: The Giant with Shifting Ground Still the undisputed leader, but here's where it gets interesting: • 24% monthly growth in API requests • 6% monthly growth in new organizations • Adoption dropped from 89% to 76% Steady dominance? Yes. But competitors are catching up faster than expected. Why? Keep reading. 👀 🌟The Surprise Player: Anthropic They've been absolutely on fire: • 61% monthly growth in requests—fastest among all providers • 22% monthly growth in organizations Each Claude release (Sonnet 3.5, Haiku-3.5) created visible adoption spikes. With 23% of Portkey orgs now powered by Anthropic, they've cemented themselves as OpenAI's most formidable rival. One thing's crystal clear: model releases matter. 🎯Gemini: Google's Late But Impressive Entry Despite being the newest player: • 49% monthly growth in requests • 9% growth in new organizations Late to the party? Yes. But users who've adopted Gemini are showing surprisingly deep usage patterns. It's a strong comeback story in the making. ☁️ The Cloud Provider Battle: Partnerships Make or Break Azure is leading the charge, with 3x more penetration than AWS Bedrock. But here’s the twist: Bedrock started the year with 10x the adoption of Vertex AI, but by year-end, it’s only 2x ahead. It’s clear now: your AI infrastructure is as strong as your model partnerships. Microsoft's early OpenAI partnerships are paying off BIG. →Azure is riding the OpenAI wave with 11% org growth and 23% request growth. E → Amazon Bedrock: The Deep Integration Story Here's a fascinating pattern: • 13% monthly growth in organizations • BUT 91% growth in requests Translation? When companies choose AWS, they go all in. It's not about breadth, it's about depth. → Vertex AI: The Quiet Climber • 43% monthly growth in requests They started slow, but they're gaining fast. The momentum is undeniable. The Biggest Shift Nobody's Talking About In just 10 months, the proportion of Portkey orgs using multiple providers jumped from 23% to 40%. Why? After a year of outages and capacity issues, companies learned a crucial lesson: redundancy isn't expensive, downtime is. This isn't about experimentation anymore. Companies are building serious infrastructure with multiple providers because they've realized you can't rely on just one provider. The AI market has matured beyond the hype—it's all about production reliability now. Tomorrow: We'll dive deep into how these providers actually perform in production. The numbers might surprise you. 😉 Follow Portkey for more insights from LLMs in Prod '24
-
-
-
-
-
+3
-
-
🔍 Breaking Down the Reality of LLM Infrastructure: Key Insights from 650+ Organizations The promise of AI is exciting, but the reality of running LLMs in production is sobering. Here's what we learned from analyzing real-world deployment data across hundreds of organizations: 📉 Remember the Great OpenAI Outage? That 4-hour downtime was just the tip of the iceberg. Our data shows that EVERY major provider has experienced significant disruptions over the past year. Key findings that should make you rethink your LLM strategy: Error rates are universal: - Groq hitting 21% rate limit errors - Anthropic facing 0.56% server errors Even industry giants struggling with reliability. 💡 The hard truth? Your users don't care why your AI features failed. They just know they failed. Here's what the data tells us about building resilient AI systems: ✅ Multi-provider strategy is non-negotiable ✅ Caching delivers real results: 36% average hit rate 30x faster responses 38% cost reduction ✅ Smart fallbacks aren't optional at scale Remember: Hope isn't a strategy. In the world of LLM infrastructure, reliability isn't a feature—it's a foundation. Curious to hear your thoughts. What strategies are you using to ensure reliable AI deployments?