Catastrophic and complete forgetting in machine learning visualized

This is a visualization of catastrophic forgetting and complete forgetting which are two distinct phenomena that occur while training machine learning algorithms.
Let K denote the field of complex numbers (Everything would have worked out about the same if K were the field of real numbers or the field of quaternions). Let d=e=8. Let D be a collection of pairs (u,v) where u belongs to K^d and v belongs to K^e and u,v are unit vectors. Let r=3. Then the goal is to find matrices A_1,...,A_r over the field K where A_1 u,...,A_r u all represent the vector v whenever (u,v) is a data point. Here we need multiple vectors to represent the vector v since the relation D is most likely a highly non-linear relation.
The fitness level for the matrices A=(A_1,...,A_r) on the data set D is
[\sum_{(u,v)\in D}log[\sum_{k=1}^{r}|dot(A_k u,v)|^2]/|D|]-2 log(norm(A)). We optimize this fitness function using gradient ascent. For this animation, we use gradient ascent using momentum. The norm of A is defined by setting
norm(A)^2=norm(A_1)^2+...+norm(A_r)^2 where norm(A_j) is the Frobenius norm of A_j. When we normalize A so that norm(A)=1, the fitness level for A is \sum_{(u,v)\in D}log[\sum_{k=1}^{r}|dot(A_k u,v)|^2]/|D|.
For the visualization, there are 200 training data points in total. For each training data point, in the visualization, we show the value \sum_{k=1}^{r}|dot(A_k u,v)|^2. For the visualization, in order to compare how the training behaves with different initialization, we train the matrices (A_1,...,A_r) five different times, and each instance of this training is designated using dots of a different color. During each round of training, we only train with 50 of the 200 training data points. This training schedule is evident in the visualization since the fitness levels for the data points that are being trained on are generally higher than the fitness levels for all training points.
From this visualization, we observe two prominent phenomena which we shall call catastrophic forgetting and complete forgetting.
In catastrophic forgetting, the fitness level for a data point goes down to zero as soon as we are no longer training that data point but instead train with different training points. Catastrophic forgetting is a phenomenon that occurs with typical neural networks and not just my machine learning algorithm.
Complete forgetting is an extreme form of catastrophic forgetting. In this visualization, all five of the tuples of matrices all eventually approximate each other. This means that with enough training, the tuples of matrices completely forget how they were initialized and the initialization has absolutely no effect on the tuple of matrices after enough training. We see that the phenomenon of complete forgetting still occurs even though we train only 50 data points at a time, but complete forgetting takes more training time than catastrophic forgetting.
This is my own machine learning algorithm.
Catastrophic forgetting is sometimes seen as a negative aspect of neural networks, but there are some benefits to catastrophic forgetting. Furthermore, when retraining a network, it is natural for that network to forget some of the material that it was trained on in the past but not in the present (neural networks have a limited amount of memory), but neural networks tend to forget information even if they have extra capacity for learning.
Machine learning models that have exhibited complete forgetting should be seen as more interpretable and safe than machine learning models that do not exhibit complete forgetting. This is because networks that exhibit complete forgetting do not have any random/pseudorandom information that is independent of the training data, and such random/pseudorandom information may interfere with interpretation attempts. Furthermore, complete forgetting is a mathematical property of machine learning algorithms and machine learning algorithms that exhibit such mathematical properties are more likely to be interpretable than ones which do not exhibit such mathematical properties.
Unless otherwise stated, all algorithms featured on this channel are my own. You can go to github.com/spo... to support my research on machine learning algorithms. I am also available to consult on the use of safe and interpretable AI for your business. I am designing machine learning algorithms for AI safety such as LSRDRs. In particular, my algorithms are designed to be more predictable and understandable to humans than other machine learning algorithms, and my algorithms can be used to interpret more complex AI systems such as neural networks. With more understandable AI, we can ensure that AI systems will be used responsibly and that we will avoid catastrophic AI scenarios. There is currently nobody else who is working on LSRDRs, so your support will ensure a unique approach to AI safety.

[@Bencurlis]

Complete forgetting is an interesting new concept, it is fascinating that differently initialized networks would converge in behavior not only in what they remember, but in what they do not. I hear your point about complete forgetting being good to remove pseudorandom information, but I think this is a bad solution to this problem, we should want our models to learn things and not forget them if there is no reason to change the mapping (no distributional shift). Catastrophic forgetting does not appear to caused by a limited capacity in practice. By the way, could it be that pseudorandom information is equivalent or very similar to feature contamination? abs/2406.03345

[@josephvanname3377]

Catastrophic forgetting is only a problem in some circumstances where one is retraining the machine learning model. If one keeps the training data after training the model or if the training data is otherwise readily available (using generative AI, from the internet, or from direct computation or observations), then it seems like catastrophic forgetting is a non-issue. Sometimes we would like to retrain the network, and sometimes we would like for the network for forget bad behavior or information that the network is not supposed to know. In this case, catastrophic forgetting is a good thing and complete forgetting is even better. Half of my videos have been about my AI algorithms that exhibit complete forgetting, but most of those are about a topological version of complete forgetting while this visualization is about a uniform version of complete forgetting. If we are going to build more networks that do not forget much after retraining, then those networks will likely be much less interpretable than AI algorithms with complete forgetting. Complete forgetting is a highly mathematical property of few AI algorithms, and such mathematical behavior is helpful for AI interpretability and safety. I do not believe that the pseudorandom information is equivalent to feature contamination. I have trained AI models with complete forgetting, but these AI models learn uncorrelated features for a simple reason. I designed the fitness functions for these AI models so that they not learn how the output relates to the input, but they also learn to recognize the input all by itself; I could easily modify these algorithms so that they do not try to figure out any outputs that correspond to the inputs but instead just learn to recognize the inputs. If one looks at the fitness function for some of these AI models, one would see that the AI models are performing tasks other than simply trying to produce the correct output for a particular input, and in order to maximize fitness, these AI models do not optimize their ability to predict the correct outputs. As a result, these AI models learn plenty of uncorrelated features. With that being said, I would like to develop more AI algorithms with complete forgetting that more accurately produce the correct output for a given input (and more importantly are equipped to handle more difficult tasks), so one will have to ask me how this relates to feature contamination when I develop those algorithms, so more research is needed. I edited the description to state that catastrophic forgetting is not caused simply by limited capacity and that catastrophic forgetting occurs even in networks with more than enough capacity to learn.

[@Bencurlis]

@@josephvanname3377 The phenomenon of CF prevents us from training our deep learning models from non-IID data streams, but I believe this is the only possible paradigm in the long term as we already produce way more data than we can store. Yes, we sometimes want the model to unlearn some information it learned previously, but in this case there is the distributional shift I mentioned, but in general we just want the model to learn some mapping, assuming it does not change much over time. If I understand correctly what you said about feature contamination, you said that your models are not generally trained to learn a precise explicit mapping, but the fitness function still implies it will produce some output for a given input, right? I am not sure if it really imply that feature contamination cannot exist in this case. The feature being uncorrelated is precisely one of the properties of feature contamination, the only remaining question is if these pseudo-random features you identified are learned because of the asymmetry caused by non-linear activations while training, or if they pre-exist in the network and simply remain in it while training, as you seem to imply in previous videos. In the former case, both phenomenons should be one and the same, but not in the latter case.

[@josephvanname3377]

@@Bencurlis In the case where the AI model learned some fitness function that approximates or at least incorporates the explicit mapping, the model will naturally want to learn things about those data regardless of whether they are correlated with the label or are even helpful in determining the label. This means that in my AI models that exhibit complete forgetting, the fitness function naturally gravitates towards feature contamination. But in that case, since the fitness function is just an approximation for the determination of the label, I am unsure if we can still call this 'feature contamination'. To test whether feature contamination is a result of the random/pseudorandom initialization, one may want to perform an experiment in which the network has a non-random initialization. To do this, one can use a network with layers of the form SkipConnection(Dense(mn,mn,activation),+) where the initialization consists of zeros. In this case, the network begins as the identity function, and all forms of feature contamination arise during the training instead of the initialization (even with zero initialization, the network will still learn pseudorandom information though since it is influenced by things such as the learning rate, the order in which the network is fed data, and the amount of momentum, and this pseudorandom information is similar to a random/pseudorandom initialization). It seems like the solution to catastrophic forgetting is not to use fitness/loss functions that are resistant to catastrophic forgetting since there are other techniques to deal with this issue (though some forgetting resistant fitness/loss functions may be useful). In some cases, one can treat catastrophic forgetting as a non-issue; for example with recommender systems, it may be better to learn people's current preferences and to learn current trends than to remember historical preferences and trends. In the case where the history is important, one may want to keep some training examples in long term storage and use other techniques to minimize forgetting besides simply use loss/fitness functions that remember everything.