Solution
Main idea
Difficulty: We don’t know how long a certain action takes, so we need to introduce time parameters
The Markov hypothesis states that aij has nothing to do with time, and the state at the next moment is only related to the state at the previous moment.
aij can also be defined as
meaning that the state at the next moment is only related to the state at any previous moment (multi-step Markov chain), which increases the complexity of the problem and model. The first form (single-step Markov chain) is adopted here. Kov chain).
three questions
A simple algorithm for problem 1
④