ZenML

"I have 47 nearly identical Airflow DAGs, each serving a different ML model variant." If that made you wince, you're not alone. Let's talk about the real mess of ML workflows in Airflow. Common failure patterns I keep seeing: 1. Massive monolithic DAGs that try to handle every edge case   (Good luck debugging that 4000-line pipeline.py) 2. Task dependencies that look like spaghetti   (Because someone had to handle "just one more feature flag") 3. Hardcoded paths and config scattered across tasks   (The classic "it works on my branch" syndrome) 4. Running preprocessing in notebooks, then wondering why prod is broken (Those magic .transform() calls need version control too) What actually works: • Dynamic DAG generation from config • Modular tasks with clear contracts • Version EVERYTHING (yes, even those sklearn transformers) • Standardized failure handling patterns • Parameterized model artifacts Real talk: Your Airflow DAGs should be boring. All the ML complexity should live in versioned packages, not task definitions. Your data scientists will never read your Airflow docs. But they will use your templates if you make them easier than notebooks. The best ML pipelines are the ones you can explain to a new team member in 10 minutes. #MLOps #DataEngineering #MachineLearning #Airflow

Let's talk RAG in Enterprise! 🫸We now know prototyping a Hello World RAG app vs getting it reliably into production are two distinct problems (surprise surprise) 🫸Problems in production include inconsistent thinking, hallucinations, incompleteness, performance degradation etc 🫸A good framing is that RAG systems are data pipelines at heart (ingestion, reranking, evaluation, PII detection, etc) 🫸It's clear that building a data flywheel for your LLMOps (as for MLOps before that) is going to be a key competitive differentiation moving forward 🫸Start simple but frame this as a data problem! A solid ML platform foundation is worth the investment to enable GenAI use-cases reliably across the enterprise Picture: Me speaking about how to architect a RAG system for the enterprise at ElasticON last week in Munich

A candid note to ML platform leads: Building that internal ML platform was the easy part, wasn't it? The real gut punch came when: ➡️ Your data scientists kept using their notebooks instead of your carefully crafted workflows. ➡️ Your ML engineers still copied model files manually rather than use your artifact store. ➡️ Your expensive GPU scheduler sat unused while teams spun up their own instances. ➡️ Your documentation went unread while Slack filled with the same basic questions. You did everything "right": ✓ Kubernetes-native architecture  ✓ Full model versioning ✓ Automated CI/CD ✓ Standardized environments ✓ Role-based access But you built what you thought they needed, not what they actually needed. I learned this the hard way: A half-implemented solution that teams actually use beats a perfect platform that they don't. Start with one team's actual workflow. Make it 10% better. Repeat. #MLOps #ML #SoftwareEngineering #PlatformEngineering

🎉 WINNERS ANNOUNCEMENT! 🎉 We're thrilled to announce the winners of the LLM Engineer's Handbook giveaway! The quality of the challenges shared was so impressive that we decided to expand from 5 to 8 copies!  Congratulations to our winners: 🏆 Alpin Dale (@AlpinDale) - Highlighted how most LLM challenges are fundamentally software/hardware engineering problems 🏆 Daniel Klitzke - Tackled the UI/App integration and RAG latency vs quality tradeoffs 🏆 Mike Bird (@MikeBirdTech) - Explored intelligent orchestration of specialized open source models 🏆 Roey Zalta - Addressed RAG challenges with Hebrew text and RTL content 🏆 Siavash Sakhavi - Shared insights on designing robust pipelines for non-standard PDF processing 🏆 Srishti Batra - Focused on efficient chunking and quantitative accuracy assessment 🏆 Stefano Fiorucci - Tackled the challenge of model replacement without regressions 🏆 Ufuk Hürriyetoğlu - Deep dive into tool selection and orchestration challenges Reading through all the responses revealed the incredible complexity of running LLMs in production - from handling non-English content to building reliable evaluation pipelines. If you're interested in a deeper analysis of these challenges, we've summarized the key themes here: https://lnkd.in/dcgWBp3s Thanks to everyone who participated and shared their experiences! Winners - I'll be sliding into your DMs shortly with details on how to claim your copy! 📚✨ #LLMOps #AI #MachineLearning #MLOps #ZenML

Hard truth about ML experiment tracking in 2024: 90% of model iteration history lives in: - Jupyter notebooks named "final_v3_REALLY_FINAL.ipynb" - Random experiment runs with commit hashes that don't exist anymore - That one CSV your colleague sent over Slack last month - Your browser history because you forgot to save the tensorboard url I wrote a detailed technical breakdown of how to fix this systematically: https://lnkd.in/dW2WDyfk. Used the awesome neptune.ai as my tool of choice. But here's what nobody talks about: This isn't a tooling problem. It's a workflow problem. Even with perfect tools, we're still making the same mistakes: 1. Running experiments before defining what we're measuring 2. Not versioning the evaluation data 3. Assuming we'll "remember the important details" Fix your workflow first. Tools come second. #ML #DataScience #MachineLeaning #MLEngineering

A few months ago, Paul Iusztin told me he'd be authoring a book with Maxime Labonne (from Liquid AI) about LLM engineering in the real world - a topic I've been thinking about and talking to users almost daily. Fast forward to last week: the book has finally launched. Having received an advance copy, I'm impressed by how Paul and Maxime have masterfully dissected this rapidly evolving field. Their timing couldn't be better, as organizations across the spectrum of MLOps maturity are working to deploy LLMs in production. ➡️ZenML is giving away 5 copies of the book ➡️Drop a comment sharing your biggest LLM engineering challenges C ➡️Winners will be selected at random. If you're selected, I'll reach out via DM! Comment below now to enter the lucky draw! ➡️➡️➡️

Here's to the one-person MLOps teams out there quietly powering ML initiatives at companies around the world. 👊 You're the unsung heroes: • Architecting data pipelines while others are still planning meetings • Shipping models to production while juggling cloud costs • Debugging infra issues at 2 AM because production can't wait • Building monitoring systems from scratch • Being the on-call rotation... of one The scope of MLOps keeps expanding, but you keep adapting. You're not just an ML engineer – you're a full-stack MLOps platform packed into one human being. To those running the whole MLOps show solo: I see you. Your ability to wear all these hats isn't just admirable – it's incredible. You're not just keeping the lights on; you're building the future of AI in your organization. Drop a 💪 if you're one of these ML swiss army knives. You're not alone in this. #MLOps #MachineLearning #ML #DataScience #Engineering