Benchmarks for methods that matter
External link for Polaris - Benchmarks for methods that matter
Polaris - Benchmarks for methods that matter reposted this
Thanks to NeurIPS for hosting our workshop on the BELKA small molecule-protein interaction competition! If you missed it, you can watch it here https://lnkd.in/g7HkBQsF Includes Maxwell K.'s tutorial of evaluating protein model boltz-1 on the BELKA dataset, Victor Shlepov's first-place entry, Vlad Vinogradov's top-10 entry, and Jonathan Hsu on Polaris - Benchmarks for methods that matter. Enjoy!
📊 New Blog: Imagining the Future of ML Evaluation in Drug Discovery In our method comparison guidelines, we talked about the limitations of static leaderboards—they never tell the full story. What if there was an easier and better way to evaluate and compare methods? A dashboard that delivers: ✅ Group-based rankings based on statistical similarity of their performance distributions 📊 Interactive, customizable visualizations 🔬 Actionable insights to help you compare methods and make decisions In our latest blog, we share a vision for a new evaluation dashboard on Polaris that could help improve how ML methods are assessed and compared. Explore what this future might look like: https://lnkd.in/gXHzZTw2 We’d love to hear your thoughts—reach out on Discord and help us refine this vision: discord.gg/VaCbn82CeD
Andddddd that’s a wrap on #NeurIPS2024! It was exciting to see a good amount of dataset and benchmarking papers at the AIDrugX workshop. If you missed it, check them out below: 1️⃣ "PertEval-scFM: Benchmarking Single-Cell Foundation Models for Perturbation Effect Prediction” Aaron W. Martina Occhetta Nikhil Branson https://lnkd.in/epFqarHV 2️⃣ ”PerturBench: Benchmarking Machine Learning Models for Cellular Perturbation Analysis” Yan Wu Esther Wershof Sebastian Schmon Marcel Nassar Blazej Osinski Ridvan Eksi Thore Graepel https://lnkd.in/gWkbfUTC 3️⃣ ”DeepProtein: Deep Learning Library and Benchmark for Protein Sequence Learning” Jiaqing Xie Tianfan Fu https://lnkd.in/gyr8Gg2c 4️⃣ ”Benchmarking Transcriptomics Foundation Models for Perturbation Analysis : one PCA still rules them all” Ihab Bendidi Shawn Whitfield Kian Kenyon-Dean Hanene Ben Yedder Yassir El Mesbahi Emmanuel Noutahi, PhD Alisandra Denton https://lnkd.in/geu_mj5e 5️⃣ ”BindingGYM: A Large-Scale Mutational Dataset Toward Deciphering Protein-Protein Interactions” Shuangjia Zheng https://lnkd.in/gHq57ury We'd love to host your work on Polaris. Check out the benchmarking platform today and get started with our tutorials! https://lnkd.in/gtZBjtuX
We’re at #NeurIPS2024! Catch Jonathan Hsu at the BELKA Competition workshop with the Leash Bio team, where he’ll give an overview of Polaris. Tune in at 3:00 PM to learn more about our mission, what we’re working on, and what you can expect from the benchmarking platform in 2025. The workshop starts at 1:30 PM in the West meeting rooms 215 and 216. Find the schedule here: https://lnkd.in/gZTpayZJ The entire BELKA dataset is also available and certified on Polaris! Explore it today: https://lnkd.in/gfY2ZYvs
🛡️ New Certified Dataset and Benchmark! Did you know that high-dimensional data like cell images (i.e. phenomics) can be used to help us understand the relationship between compounds and genes? Now you’ll have a chance to work with a phenomics dataset and benchmark to predict which genes a given compound interacts with! RxRx3-core is the latest certified dataset from Recursion that includes labeled images of 735 genetic knockouts and 1,674 small-molecule perturbations. The related benchmark evaluates the zero-shot prediction of compound-gene activity. If you’re at NeurIPS, come talk to the Polaris and Recursion teams! We’ll be at the conference and workshops over the weekend. Explore the dataset: https://lnkd.in/esn3CcAF Explore the benchmark: https://lnkd.in/e6PYFUB4 Chime in on the certification process: https://lnkd.in/e5cAmqVH
Did you miss last week's live discussion on the method comparison guidelines? The recording is out now: https://lnkd.in/gW_p6ww9 Check out the proposed guidelines here: https://lnkd.in/g6w78h9W Leave your thoughts and feedback on GitHub: https://lnkd.in/gMzpS4TZ Explore hundreds of ML-ready datasets and benchmarks for drug discovery on Polaris today: https://polarishub.io/
What are the most interesting datasets and benchmark-related work for ML in drug discovery at NeurIPS? We’ll be doing short interviews with researchers at the conference and workshops and handing out some Polaris merch! Here’s who we have on the shortlist. Let us know what other papers you’d like us to cover 👇 1️⃣ The Recursion team and their latest RxRx3-core dataset that contains labeled images of 735 genetic knockouts and 1,674 small-molecule perturbations drawn from the RxRx3 dataset Oren Kraus https://lnkd.in/gV69BJQS 2️⃣ “WelQrate: Defining the Gold Standard in Small Molecule Drug Discovery Benchmarking” Yunchao Liu Tyler Derr https://lnkd.in/giDCwiTh 3️⃣ “MassSpecGym: A benchmark for the discovery and identification of molecules” Roman Bushuiev Anton Bushuiev https://lnkd.in/ghtqjJ8a 4️⃣ “DFT: A Universal Quantum Chemistry Dataset of Drug-Like Molecules and a Benchmark for Neural Network Potentials” Kuzma Khrabrov https://lnkd.in/gEg5Vy-r 5️⃣ “A benchmark for prediction of transcriptomic responses to chemical perturbations across cell types” Artur Szałata Fabian Theis Daniel Burkhardt https://lnkd.in/gNJECv3s 6️⃣ The datasets used in the Valence Labs paper “On the Scalability of GNNs for Molecular Graphs” Dominique Beaini https://lnkd.in/gFQ-C8-t Come find us and get some Polaris merch! Stay updated with Polaris by signing up to the mailing list: https://lnkd.in/eWCviVqz Explore hundreds of ML-ready datasets and benchmarks for drug discovery on Polaris today! https://polarishub.io/
Happening in ~45 minutes! Join us live: https://lnkd.in/gs2SVrPC
How should we compare machine learning methods in drug discovery? In the first working paper from the Small Molecule Steering Committee at Polaris, we introduced guidelines for method comparison to help ensure fair, replicable, and rigorous evaluations in ML-driven drug discovery. Now, we’re taking the conversation LIVE! 📅 When: Dec 5th from 11 - 12 PM ET 🎥 Where: https://lnkd.in/gs2SVrPC Join host Jonathan Hsu from Valence Labs / Recursion and first author Jeremy Ash from Johnson & Johnson Innovative Medicine for an interactive session where we’ll: ✅ Explore the key points from the paper ✅ Discuss its implications for the field ✅ Open the floor to your questions and ideas Let’s work together to redefine how method comparison is done!
Looking to understand how we can better evaluate and compare models for drug discovery? Join us tomorrow in our webinar where we’ll dive into the proposed guidelines for method comparison covering topics like performance sampling distribution, statistical significance, visualizations, and more! Tune in live at 11 AM ET: https://lnkd.in/gs2SVrPC
How should we compare machine learning methods in drug discovery? In the first working paper from the Small Molecule Steering Committee at Polaris, we introduced guidelines for method comparison to help ensure fair, replicable, and rigorous evaluations in ML-driven drug discovery. Now, we’re taking the conversation LIVE! 📅 When: Dec 5th from 11 - 12 PM ET 🎥 Where: https://lnkd.in/gs2SVrPC Join host Jonathan Hsu from Valence Labs / Recursion and first author Jeremy Ash from Johnson & Johnson Innovative Medicine for an interactive session where we’ll: ✅ Explore the key points from the paper ✅ Discuss its implications for the field ✅ Open the floor to your questions and ideas Let’s work together to redefine how method comparison is done!