From what little they have released publicly, it seems that they are simply diffusion models applied to videos, i.e. they treat videos as a collection of frames, add noise to all frames, take all noisy frames as input and try to predict all clean frames. I don't think there is any auto-regression done, but maybe that will change when they start generating longer videos.

Thanks for the feedback! I used Manim for the first 3 videos on my channel, and then decided to write my own animation library because I found Manim annoying to use. My animation library is still a work in progress, so my most recent 2 videos are made using both my own library and Manim. I would say that Manim is fine to use for simple animations/graphs, but once you start trying to animate lots of objects at once it gets annoying.

The training method described in this video IS gradient descent. Backprop is just one way of computing gradients (and it is usually used because it is the fastest). It will yield the same results as this training procedure.

@@algorithmicsimplicity I should have figured it was doing much the same thing, just a lot slower and more 'random.' Your videos are awesome though, I'm trying to go into machine learning after doing aerospace embedded systems.

Thank you so much!

You are correct that at the early steps there is large amount of uncertainty over the value of each pixel. But what matters is how much this uncertainty is reduced by knowing the value of the previously generated pixels. Even at the first step, knowing the value of one pixel does not help that much in determining the value of far away pixels. Let's say one pixel in the center is blue, does that make you significantly more confident about the color of the top left pixel? Not really.

@@algorithmicsimplicity hi, glad to see your reply. Let me articulate. Let's just create the first 2 pixels in two ways and see the difference. First approach is create them one by one, and the second approach is to create both in one step. First approach first, at the beginning, there is no pixel, when we create one, not problem, there will be no inconsistency. Then we create the second pixel based on the first one, there are still a lot of possibilities. But it is restrained by the first pixel. On the other hand, for approach two, both pixels are created in the same step, then they cannot restrain each other. Then there could be more combinations. Some of them are include inconsistency. To avoid such thing, we should generate pixels one by one until the theme of the image is set. Then we can create batch by batch. A simplified example, if the model is trained with images include only green pixels of different strength, or red pixels of different strength, then approach one will generate second pixel of the same color as the first one, though probably with different strength. But approach 2 could generate one red and one green.

Thanks for the suggestion, I will put it on my TODO list.

For image input, the input is always the same size (same image size and same channels), and the output is always the same size (1 pixel). For text, you can also treat the inputs as all being the same size by padding smaller inputs up to a fixed max length, though transformers can also operate on sequences of different lengths. The output for text is always the same size (a probability distribution over tokens in the vocabulary).

Yes a and b are the learned parameters.

It isn't correct to say it creates a 'composite' with variations, models can generalize outside of their training dataset in certain ways, and generative models are capable of creating entirely new things that aren't present in the training dataset.

My image generator was trained on data licensed to be used for training machine learning models.

@@algorithmicsimplicity not everyone is so ethical

Yes, it absolutely can! Instead of adding normally distributed noise, you randomly mask tokens with some probability, see e.g. arxiv.org/abs/2406.04329 . That said, it tends to produce a bit worse quality text than auto-regression (actually this is true for images as well, it's just on images auto-regression takes too long to be viable.)

Right, the average is just the center of the noise distribution which, let's say the color values are mapped from -1 to 1, is 0. This average doesn't look like noise (it is just a solid grey image), but if you ask what is the probability of this image under the noise distribution, it actually has the highest probability. The noise distribution is a normal distribution centered at 0, so the input which is all 0 has the highest probability. So the average image still lies within the noise distribution, as opposed to natural images where the average moves outside the data distribution

Thank you for the reply, I think I got it now

Generative transformers work in exactly the same way as generative CNNs. It doesn't matter what backbone you use the idea is the same, you will use auto-regression or diffusion to train a transformer to undo the masking/noising process.

@@algorithmicsimplicity which is more efficient to use and how do they handle text data to map text into tensors?

@@dadogwitdabignose I explain how transformer classifiers work in this video: kzbin.info/www/bejne/oYivlpdupJqAaLs . As for which is more efficient, it depends on the data. Usually for text data transformers will be more efficient (for reasons I explain in that video), and for images CNNs will be more efficient.

Thanks for the suggestion.

Thank you very much!

Totally agree. I feel like that's pretty common in math, robotics, and computer science, but it just shows how every field in stem is interconnected.

Yep you could definitely do that, you would probably need to train a model on some examples of noisy ray traced images though.

Yeah but this is very slow do

The entire model is trained end-to-end to minimize the training loss. To start off with, the scoring functions are completely random, but during training they will change to output scores which are useful, i.e. which cause the model's final prediction to better match the training labels. In practice it turns out that what these scoring functions learn while trying to be useful is very similar to the 'semantic scoring' that a human would do.