It is a math question; in fact it is a graph theory question. You need to write down a graph whose vertices represent how many mens and boys are on either shore of the river, and where is the boat. For instance, <3m,0b,boat|6m,2b> would mean that there are three mens, no boys and the boat on the first shore, and the six other mens and two boys are on the other shore. Two of these vertices are joined by an edge if you can get from one situation to the other with just one crossing.
For instance <3m,0b,boat|6m,2b> --- <2m,0b|7m,2b,boat> means that starting with the situation above, one of the men on the first shore can cross the river on the boat, so that now there are two mens on the first shore, and the other people and the boat are on the other shore.
A longer path such as <3m,0b,boat|6m,2b> --- <2m,0b|7m,2b,boat> --- <2m,1b,boat|7m,1b> --- <1m,1b|8m,1b,boat> --- <1m,2b,boat|8m,0b> --- <1m,0b|8m,2b,boat> corresponds to a sequence of events:
Starting with three mens, no boys and the boat on the first shore, and the others on the other shore,
- one man crosses from the first to the second shore,
- then one boy crosses the other way,
- then another man crosses like the first,
- then the other boy crosses back,
- then the two boys cross again to the second shore.
Thus there are now only one man left on the first shore, while the eight others, the two boys and the boat are on the second shore.
So the question is: Starting at the vertex <9m,2b,boat|0m,0b> corresponding to the initial situation where everybody and the boat are on the first shore, can you make a path that will lead all the way to the vertex <0m,0b|9m,2b,boat> corresponding to the situation where everybody crossed.
(Note: The hint tells you to warm up by considering the situation where there is only one man and two boys; the corresponding graph has very few vertices, so it is easy to write it all down and find the path you want.)