Outlier’s Post

The recent article about Kolmogorov-Arnold Networks (KANs) is a breakthrough filled with many interesting insights. The article is not only well-written but also raises a crucial question: " Do we want AI scientists, or do we want AI that helps scientists?" Interpretability is an important factor for science driven digital projects. Moreover, in many areas, cost of acquiring sufficient data for purely data-driven models are high, making hybrid machine learning with first-principle models more interesting. Authors found KANs to be more efficient for fitting physical equations and PDE solving. In my view, this finding highlights that opting for a hybrid approach might yield even more promising outcomes now. https://lnkd.in/e_WfNPKW

Imagine getting the same level of accuracy from a simpler model! That's the promise of Kolmogorov-Arnold Networks (KANs). Research suggests Kolmogorov-Arnold Networks (KANs) could be a serious competitor to Multi-Layer Perceptrons (MLPs). They ditch the usual fixed settings and use learnable functions, which could mean not only understanding the model's decisions better but also building incredibly detailed models that are surprisingly compact. This opens up a whole new world of possibilities for deep learning!

Automated Social Science: Language Models as Scientist and Subjects Benjamin S. Manning, Kehang Zhu, John J. Horton Abstract We present an approach for automatically generating and testing, in silico, social scientific hypotheses. This automation is made possible by recent advances in large language models (LLM), but the key feature of the approach is the use of structural causal models. Structural causal models provide a language to state hypotheses, a blueprint for constructing LLM-based agents, an experimental design, and a plan for data analysis. The fitted structural causal model becomes an object available for prediction or the planning of follow-on experiments. We demonstrate the approach with several scenarios: a negotiation, a bail hearing, a job interview, and an auction. In each case, causal relationships are both proposed and tested by the system, finding evidence for some and not others. We provide evidence that the insights from these simulations of social interactions are not available to the LLM purely through direct elicitation. When given its proposed structural causal model for each scenario, the LLM is good at predicting the signs of estimated effects, but it cannot reliably predict the magnitudes of those estimates. In the auction experiment, the in silico simulation results closely match the predictions of auction theory, but elicited predictions of the clearing prices from the LLM are inaccurate. However, the LLM's predictions are dramatically improved if the model can condition on the fitted structural causal model. In short, the LLM knows more than it can (immediately) tell. 👉 https://lnkd.in/dFQEZ2yQ #machinelearning

💥💥💥 Kolmogorov-Arnold Newtworks Ziming Liu, Yixuan Wang, Sachin Vaidya, Fabian Ruehle, James Halverson, Marin Soljačić, Thomas Y. Hou, Max Tegmark Abstract Inspired by the Kolmogorov-Arnold representation theorem, we propose Kolmogorov-Arnold Networks (KANs) as promising alternatives to Multi-Layer Perceptrons (MLPs). While MLPs have fixed activation functions on nodes ("neurons"), KANs have learnable activation functions on edges ("weights"). KANs have no linear weights at all -- every weight parameter is replaced by a univariate function parametrized as a spline. We show that this seemingly simple change makes KANs outperform MLPs in terms of accuracy and interpretability. For accuracy, much smaller KANs can achieve comparable or better accuracy than much larger MLPs in data fitting and PDE solving. Theoretically and empirically, KANs possess faster neural scaling laws than MLPs. For interpretability, KANs can be intuitively visualized and can easily interact with human users. Through two examples in mathematics and physics, KANs are shown to be useful collaborators helping scientists (re)discover mathematical and physical laws. In summary, KANs are promising alternatives for MLPs, opening opportunities for further improving today's deep learning models which rely heavily on MLPs. Code, preprint 👉 https://lnkd.in/ds85iqKK #machinelearning

In 2021, I wrote a review paper exploring nature-inspired algorithms, one of them was the Ant Colony Optimization (ACO) algorithm. Let's talk a little about it today! This algorithm mimics how ants find the best paths to food sources using pheromone trails, and it has been applied to solve complex optimization problems in various fields. In the wild, ants are really good at finding the shortest route between their nest and food sources. They do this by leaving behind a special scent called pheromones as they walk. When other ants come across this scent, they follow it. The more ants that use a particular path, the stronger the scent becomes. Over time, the shortest path ends up having the strongest scent, and the whole colony starts using it. It's like they're working together to find the most efficient solution! Scientists have taken this idea and applied it to all sorts of problems, like finding the best delivery routes for packages, improving communication networks, and even making factories run more smoothly. But one of the most exciting uses is in the field of artificial intelligence, specifically in deep learning. So, how does ACO work in optimizing neural networks? Imagine a neural network as a complex maze, with each path representing a different configuration of the network's architecture and parameters. The ants in this case are virtual agents exploring the maze, searching for the best path (or the optimal network configuration). They leave behind virtual pheromones on the paths they take, with the amount of pheromone depending on how well the network performs with that specific configuration. As more ants explore the maze, they are more likely to follow the paths with stronger pheromone trails, which correspond to better-performing network configurations. Over time, the best configurations are reinforced, leading to an optimized neural network that can learn and make decisions more effectively. ACO proves that nature's simplicity can solve complex tech problems, blending the best of biology and technology for smarter solutions. It's a nod to nature's genius in our digital world. Link to my paper: https://lnkd.in/gptJXCtM Image Reference: https://lnkd.in/gF4JwmYv Definitely give Jeff's video a watch if you want to learn more. :)

A powerful book writing tool has been assembled on the basis of Claude It is enough to give the neural network a general idea and briefly describe the events that you want, and then artificial intelligence will do everything. Thanks to the elaborated model, the texts come out alive and interesting, almost as good as human ones. You can test it for free in Google Colab

By now, many of you have probably heard about the "AI Scientist". A project that aims to automate the entire AI/ML research workflow, from generating ideas to running experiments and producing LaTeX reports. From a technical standpoint, I admire the project. I know how challenging it can be to build robust LLM workflows, and using mostly f-strings feels like a refreshing break from "glue-libraries" like Langchain or LlamaIndex. However, I can’t help but think about Mark Saroufim’s 2020 article, "Machine Learning: The Great Stagnation," where he discusses the decline of scientific discovery. He introduces the idea of "graduate student descent," where small random tweaks are made to the state of the art, hoping for improvement. Not much has changed since 2020. Instead we discovered how to not waste any more grad students. I doubt that the AI Scientist will lead to any breakthroughs ...  What do you think? Links:  https://lnkd.in/eT3JZewx https://lnkd.in/e6E6WHS5

more vintage karpathy… https://lnkd.in/gbyZSJZG quote: These 94 lines of code are everything that is needed to train a neural network. Everything else is just efficiency. This is my earlier project Micrograd. It implements a scalar-valued auto-grad engine. You start with some numbers at the leafs (usually the input data and the neural network parameters), build up a computational graph with operations like + and * that mix them, and the graph ends with a single value at the very end (the loss). You then go backwards through the graph applying chain rule at each node to calculate the gradients. The gradients tell you how to nudge your parameters to decrease the loss (and hence improve your network). Sometimes when things get too complicated, I come back to this code and just breathe a little. But ok ok you also do have to know what the computational graph should be (e.g. MLP -> Transformer), what the loss function should be (e.g. autoregressive/diffusion), how to best use the gradients for a parameter update (e.g. SGD -> AdamW) etc etc. But it is the core of what is mostly happening. The 1986 paper from Rumelhart, Hinton, Williams that popularized and used this algorithm (backpropagation) for training neural nets: https://lnkd.in/gJ4qi8y6 micrograd on Github: https://lnkd.in/gxSxHNYC and my (now somewhat old) YouTube video where I very slowly build and explain: https://lnkd.in/gvdcQx5Y

Favourite books from a year learning about intelligence, both human and artificial. - Architects of Intelligence: The truth about AI from the people building it, by Martin Ford A series of interviews with researchers and entrepreneurs in the Artificial Intelligence community. - La plus belle histoire de l'intelligence, by Stanislas Dehaene, Yann Le Cun and Jacques Girardon A neuroscientist and a computer scientist discussing intelligence. - The Book of Why: The New Science of Cause and Effect, by Judea Pearl and Dana Mackenzie A book about the importance of understanding cause-and-effect relationships and how they shape our understanding of the world. - Models of the Mind: How Physics, Engineering and Mathematics Have Shaped Our Understanding of the Brain, by Grace Lindsay A book about the different mathematical models that have allowed scientists to understand and describe some of the brain’s processes. - Possible Minds: Twenty-Five Ways of Looking at AI, by John Brockman A collection of essays exploring the multifaceted nature of Artificial Intelligence and offering insights into AI's capabilities, limitations, and philosophical implications.

Happy Pi Day!   Beyond the tasty treats, Pi Day celebrates the magic of mathematics, the unseen language that unlocks innovation across diverse fields. In my latest blog post, "Unveiling the Mathematical Marvels: From Pi Day to Artificial Intelligence", I delve into the fascinating connection between Pi Day and the world of AI. The post explores how fundamental mathematical principles are the invisible engines driving AI applications Dive into the blog to discover how math is the secret sauce behind the technological marvels that shape our world today! https://lnkd.in/g5GaPxXN  #PiDay #MathIsEverywhere #STEM