Chalk’s Post

While everyone's excited about generative AI (everyone...everywhere..) there are two major, less sexy, challenges that often don't get discussed: Cost at Scale For high-volume applications like recommendation systems (think 300k+ predictions/second), using something like OpenAI's API would cost thousands per second. That's orders of magnitude too expensive for most use cases for most companies. Latency Issues Many applications need responses in ms. Current GenAI APIs take seconds to respond - achingly too slow for many real-time applications like detecting fraud or routing an ambulance. There's real reasons to get excited about GenAI and its application in complex and real-time predictions. The reality however? Traditional ML models still dominate production systems for good reason.