Every time you use LinkedIn you might be using Flyte 😎 Learn how LinkedIn is leveraging the Flyte Agents framework to transform its AI infrastructure. 💜 Thank you Shuying Liang for your willingness to share this story with the world!
Flyte
IT Services and IT Consulting
Scalable and flexible workflow orchestration platform that seamlessly unifies data, ML and analytics stacks.
About us
Flyte is an open-source orchestrator that facilitates building production-grade data and ML pipelines. It is built for scalability and reproducibility, leveraging Kubernetes as its underlying platform. With Flyte, user teams can construct pipelines using the Python SDK, and seamlessly deploy them on both cloud and on-premises environments, enabling distributed processing and efficient resource utilization. GitHub: https://github.com/flyteorg/flyte Slack: https://slack.flyte.org/
- Website
-
https://flyte.org/
External link for Flyte
- Industry
- IT Services and IT Consulting
- Company size
- 11-50 employees
- Founded
- 2020
- Specialties
- Machine Learning Pipelines, Data Pipelines, and Workflow Orchestration
Updates
-
🎉 Our biggest release of the year is here! flyte+flytekit 1.14 are special for many reasons. Features include: - Native support for Dataclass, dicts, and Pydantic BaseModels - Now you can consume Flyte from Notebooks natively, no plugins required. - FlyteDirectory is now supported as input to ContainerTask - No more hitting gRPC limits when moving large data, this is now offloaded automatically. Dive into the release notes to learn more! This truly felt like a team effort with the 30+ developers who merged their first contribution to Flyte. 💜 Thank you! flytekit: https://lnkd.in/eBEC5i5c flyte: https://lnkd.in/ejGDUQPF
Release v1.14.0 · flyteorg/flytekit
github.com
-
🔥 Learn how to go from zero to production-grade ML orchestration on #Azure: https://lnkd.in/gvKKzfBP Thanks to Sachi Desai and Erin Schaffer for your input and feedback; it enabled us to add integrations with Azure Container Insights, AKS node auto-upgrade, and other improvements for Flyte on Azure users!
Deploy and take Flyte with an end-to-end ML orchestration solution on AKS
azure.github.io
-
Friendly reminder to join this month's Community sync. We truly expect to see you all there! https://lnkd.in/edNpqmvr
⚙ In modern ML workflows, efficient data handling is essential for maximizing GPU utilization and accelerating training. ➡ Join us for the next community presentation where Shuying Liang will introduce the Flyte K8s agent LinkedIn Engineering is developing to orchestrate services that offload dataset loading and transformation tasks from individual training pipelines. This setup supports consistent, reusable data transformations across runs and dynamically scales to meet varying training workload demands. Built on the open-source Flyte agent framework, the Flyte K8s agent enables a decoupled, modular system architecture, allowing rapid development iterations and seamless transitions to production. At LinkedIn, this agent serves as a cornerstone in supporting scalable, adaptable data services across various training workflows, with Graph Neural Networks (GNN) as a prominent use case. This presentation delves into how the Flyte K8s agent’s general design drives productivity and operational excellence, making it an ideal solution for LinkedIn’s deep learning needs such as GNN. 💜 Everyone is welcome to join, learn, and ask questions!
Flyte K8s Agent: Scalable Data Services for GNN Workflow Training
www.linkedin.com
-
New blog post! Flyte embraces strong typing in ML pipelines, catching errors at compile time, and empowering developers to write reliable interfaces between heterogeneous data platforms. Learn the details and design decisions behind Flyte's type system: https://lnkd.in/gb2Z4r39
So you think you can type: how Flyte’s type system works
flyte.org
-
✨ Monthly Contributors Spotlight ✨ This month has been an absolute whirlwind of contributions! Here’s a highlight of the work from our contributors this month: • JiaWei Jiang made multiple contributions, including support for reading structured datasets from local/remote uri, a flytefile uri bug fix, and a protobuf struct attribute fix. • Buğra Gedik added support for a multi-file error aggregation strategy to distributed plugins. • Chung Yen Hung handled flytectl panics for incorrect configurations and removed Python 3.8 support in flytekit as it has reached EOL. • Yenting Chen enhanced code coverage in the flyte repository with additional tests. • Jason Parraga added basic SASL and TLS support for Kafka cloud events and decoupled Ray submitter, worker, and head resources. • Chun-Mao Lai introduced an unsafe mode in flytekit that automatically pickles values without type annotations when set to true. • Rafael Raposo added a new AgentError field to the agent protos. • Vincent (Mao-Siang) C. fixed union type bugs and addressed dataclass serialization issues. • Wei-Yu Kao added support for multipart blob download to copilot 🎉! We also want to give a big shoutout to David Lin, Peter Lee, and Marcin Zabłocki for your continued support and contributions. Thanks for all you do! 🙏
-
Flyte reposted this
With features like caching, versioning, map tasks, and workflow observability, Flyte cuts costs, increases productivity, and boosts Wayve’s competitive edge. 🚘 By abstracting infrastructure and Kubernetes, Flyte lets Wayve’s researchers and engineers focus on innovation. Read the full case study here: 🔗 https://lnkd.in/ga9BDKmd
-
-
Highlights: Community presentation recording, results of Hacktoberfest, RFCs open for feedback, and upcoming events.
This week in Flyte (Oct 28 - Nov 8)
Flyte on LinkedIn
-
Calling the #ML Flyte community at Munich. We're stoked to meet you all there! Details 👇
Hello Munich! Are you ready to unlock the full potential of Kubernetes for infrastructure and machine learning? 🚀 Join us on November 13th at 6:00 PM at our SQUER office in the House of Communication in Munich for the upcoming meetup organized by the Kubernetes, Cloud Native & Platform Engineering Meetup Group. 🎙️ First Talk: Kubernetes as a Jukebox - Modern Infrastructure as Code with Crossplane Join Felix Hochleitner from SQUER as he demonstrates how Kubernetes, combined with Crossplane, can reshape infrastructure management. In a live demo (featuring a Spotify playlist!), he’ll show how GitOps principles bring flexibility and control to infrastructure as code. 🎙️ Second Talk: Building Production-Grade ML Workflows on Kubernetes with Flyte Dive into Flyte with Katrina Rogan from Union.ai, who will reveal how this Kubernetes-native ML orchestrator leverages tools like Helm and Prometheus to create scalable, production-ready workflows. After the talks, enjoy pizza, drinks, and great conversation with the cloud-native community! 🍕🍻 👉 Register for free now: https://lnkd.in/dTBuSrTZ We can't wait to see you there! 😍 #DevOps #IoTSecurity #ProductionSystems #Kubernetes
-
-
✨ Monthly Contributors Spotlight ✨ We greatly appreciate the efforts of our contributors this month: • Chun-Mao Lai tackled a bunch of Jupyter notebook and interactive mode tasks, pushing through several pull requests to get everything wrapped up—awesome work! • Vincent (Mao-Siang) C. added support for extra copy commands in imagespec. • Jason Parraga implemented pod template support for init containers. We also want to give a big shoutout to Chung Yen Hung, Yenting Chen, JiaWei Jiang, Rafael Raposo, @mthemis-provenir, Buğra Gedik, Felix Mulder Witasp, and Ethan Brown for your continued support and contributions. Thanks for all you do!