Yes, needing many steps to move in some direction is one of the problems.
Another one is the "bias problem". Namely, we want the weights of a neuron to be in some kind of regime (zero-mean for example) for training to go well (no vanishing gradients or staying only in the linear areas of activation functions). However, if all the weights are shifted in the same direction, the mean value of the weights of the neuron will also shift in that direction, thus moving us away from the desired training regime.
