Neural Network Optimization

Ivan goes into detail on how neural networks can be optimized using methods such as RMSprop and momentum, and how we can use these two together to form Adam (adaptive momentum) optimization, which eventually leads to AMSgrad optimization. It is also important to optimize the activation function, solving issues with the standard sigmoid & hyperbolic tangent functions with ReLU and SoftMax, and then eventually Leaky ReLU (to solve the exploding & vanishing gradient problems associated with ReLU and sigmoid/tanh respectively).