Error in homework 7 correction ?

Error in homework 7 correction ?

by Guillaume Marc Georges Vray -
Number of replies: 1

Hi,

I do have a problem with the homework 7 correction. It is stated in the figure below the formula for the loss function gradient.

MSE gradient

But its implementation describes another formula:

correction

Unless I'm mistaken, it's equal to (2.0/N)*K * (Y - Ŷ). 

Also, it's very surprising that this formula above gives better result for MNIST test than when we follow the first formula. Do you know why ?


Thank you in advance for your answers,

Best,

Guilaume

In reply to Guillaume Marc Georges Vray

Re: Error in homework 7 correction ?

by Edoardo Remelli -

Hey Guillaume,

You are right, there is a little mistake in the gradient implementation, which should be:

(2.0/ (y_pred.shape[0]*y_pred.shape[1]) ) * (y_pred - y_true)

Thanks for spotting it.

However, note that the "buggy" gradient is simply a re-scaled version of the correct one, so it is not surprising that back propagation works nonetheless. To convince yourself, you can think about the role of the learning rate when optimizing the parameters of your network: at the end of the day what really matters is the direction of the gradient, and not its absolute value!

Hope this helps,

Edoardo