Machine Learning Cannot Be Theory-Free

[@erongjoni3464]

Re generative models vs statistical methods -- I think a very clear ontological distinction can be made between the two such that it's kind of unreasonable call it sociological. It is approximately the same distinction as that of Shannon information and Kolmogorov cimplexity. Which is to say, statistical methods try to describe their observation as some inert static summary of the observed data, amounting essentially to extremely lossy compression. This is the Shannon information case. Trying to find the least redundant representation that maintains as much useful information as possible about the original observation as is likely to be required for some anticipated use. It's rather like a JPEG of a screenshot of the mandelbrot set. Generative models, by contrast, attempt to create highly compact representations of *processes which would generate that same or very similardata.* This is more akin to minimizing Kolmogorov complexity (though -- merely akin, as the technical definition of this sort of complexity requires lossless reconstruction IIRC). It is like foregoing the JPEG compression (statistical model), or the screenshot (raw observations), in favor of just using the actual equation for the mandelbrot (which is both much more descriptive than either the data or the model, and requires considerably less storage space too). I'm not sure how clear I'm being here, but I think a fairly quick wiki perusal on Shannon Information and Kolmogorov complexity should be sufficient to at least convince us there is a hard mathematical distinction that can in principle be made. Even if we remain skeptical as to whether modern generative models have definitively proven themselves to fall on the novel side of that distinction. Personally, I think a lot of the recent research on grokking and generalizing circuits, in addition to the fact that the models are mostly only useful in their capacity as generators of new samples, *and* the fact that the real way to make them be useful when they aren't generating anything is to forego the statistical lense altogether in favor of just treating activations as points in high dimensional space, *and* the fact that transformers don't actually obey any of the rules of statistics unless we artificially force them to (usually by just normalizing their outputs to integrate to 1 (and embarrassingly often, no more than that)) Well. . . I don't know, to me all of those things point away from statistics and toward complexity minimization.

[@erongjoni3464]

But I do agree that at least *part* of the distinction is additionally (not wholly) sociological. If for no other reason than that if I walked up to a statistician and declared "I have here a full statistical model of Art" then handed them a thumb drive containing CLIP, ViT, and Stable Diffusion 1.5. . .they would probably just laugh me out of their office.

[@aulderyan]

There's no "logical necessity" for the sun to rise tomorrow morning? 07:46 Doesnt the orbit of earth and its spin, and the arrangement of our solar system create a logical necessity?

[@erongjoni3464]

@@aulderyan not if a black hole comes whizzing by.

[@Robert_McGarry_Poems]

Ours... Our mind is our brain. Our brain is our physical. Our physical is everything that is. Our mind is everything that is ours. We are a minuscule part of the whole. The whole is a minuscule part of us.

[@NicholasWilliams-uk9xu]

Good points. ❤

[@adsaul]

i respectfully disagree

[@andyd568]

The speaker is too verbose and scattered. Stats, ML and Ai are just prediction. The difference is an ever broadening scope.

[@ai._m]

Not even close. Her reasoning is comical. Please don’t let “philosophers” out of their box.

[@RahulSam]

What’s Mel wrong about?