Kolmogorov-Arnold Networks (KANs) and Lennard Jones

KANs have been a hot topic of discussion recently (arxiv.org/abs/2404.19756). Here I explore using them as an alternative to a neural network for a simple atomistic potential using Lennard Jones data.
See kitchingroup.cheme.cmu.edu/bl....

[@dennyloevlie768]

I was just about to reach out to you and ask what you thought of the KAN architecture. So glad to see you already made a video on it! Great video and explanation.

[@JasonMitchellofcompsci]

It's basically just a different activation function with more tunable parameters than ReLU, yeah? I actually consider ReLU to be tunable because it's one way to took at your biases. Biases just set the threshold for when a signal goes through or not.

[@bithigh8301]

nice! can you make a tutorial about using emacs like that? One can go full focus on this env

[@scottotterson3978]

I guess the magic is that you can get an analytic expression from a trained KAN -- no idea how much better that expression would be than what you'd get from, say, PySR, but I can imagine that it could be better at extrapolation than an MLP.

[@kanalarchis]

Thanks, that was a very good video. I don't understand the point of this. We already knew how to interpolate data with splines.

[@PhysBrain]

Failure to extrapolate beyond the training set is a well know consequence of polynomial interpolation/regression. However, as polynomial regressors go, splines tend to be much better behaved outside of distribution than say, power series or Taylor series. Splines are highly flexible polynomial functions, but the basis functions are typically defined such that they have limited support (region over which they are non-zero). That means they do not go off to infinity or contribute other bad behavior to the function when evaluated far from their defined domain. So, at least with spline functions, there is an explicit acknowledgement that they will not produce useful extrapolations beyond the range of the provided data. However, as the paper describes, the limited (local) support is actually what allows KANs to store additional information without forgetting previously learned data. During training, the only spline parameters that are modified are the ones for which their basis functions evaluate to non-zero values (i.e. the parameters that are most closely associated with the new input data). Contrast this with weights on ReLU activation functions which are always modified when the the weighted sum of its inputs (multivariate) are above some threshold. The spline basis functions are only non-zero when their one input parameter (univariate) is within some well-defined range. So only the parameters local to the new data will be modified.

[@jasmeetsingh4688]

how can i install 'kan' package or did you write any script for it?

[@aaronbelikoff8605]

The package is called “pykan” but you import it as kan.

[@user-rl8to5nc2q]

I don’t see the advantage. Someone enlighten me?

[@antonpershin998]

More imterpretable

[@user-rl8to5nc2q]

@@antonpershin998 thamks

[@MaJetiGizzle]

No catastrophic forgetting.

[@user-rl8to5nc2q]

@@MaJetiGizzle hm how so?

[@Chidorin]

less resources needed for final model usage, may be 🤔

[@starship9629]

It is pronounced KolMOgoROV