Hello,
I realize that a notation in HW4 might not be crystal clear to you. This notation "will be" explained only in Lecture 5 (the thing is, I prepared Lecture 5 a month ago: this is one of the disadvantages of these prerecorded classes: you lose track of what you already said while preparing exercises...).
The notation in question is . Here, stands actually for the -th row of the matrix : it is the distribution of the Markov chain at time , given that the chain started in state at time 0 (i.e. ).
All the best,
Olivier