I don't see a nice way to approach the problem, but a symbol manipulator such as Mathematica should be able to obtain a numerical solution. One can use the cosine law to obtain two equations in the unknown angles A and B, where A is the angle between the 5 and 8 mile paths, and B is the angle between the 8 and 3 mile paths. Let x be the side of the equilateral triangle.
x^{2}
= 64 + 25  80 cos A
= 64 + 9  48 cos B
= 9 + 25  30 cos(A+B).
Subtract the second from the first and the third from the second to get,
16 = 80 cos A  48 cos B
39 = 48 cos B  30 cos (A+B)
Since you can change the second equation into an equation involving cos A and cos B (by expanding cos(A+B) and replacing the resulting sin A and sin B with square roots), Mathematica might be able to solve the two equations for cos A and cos B. Alternatively, One can use the first equation to get an expression for A in terms of cos B, and plug that in for A to get one horrific equation involving B.
Chris
I was a bit hasty in finding the side length of an equilateral triangle for which there is a point P whose distances from the vertices are 3, 5, and 8. Ed and I looked at it again and he noticed that P lies on the triangle's circcumcircle. That makes the problem easy! The side length is 7:
We are given an equilateral triangle TFE whose side has an unknown length x. There is a point P at distance 3 from T, 5 from F and 8 from E.
Step 1. Show that P lies on the circumcircle of TFE
We use Ptolemy's Theorem which says that if PT EF + FP TE = EP TF, then P, T, E, F lie on a circle in that order. So we check, 3x + 5x = 8x.
Step 2. Use the theorem that opposite angles of the cyclic quadrilateral PTEF add up to 180 degrees. Since the angle at E (namely FET) is 60 degrees, the angle at P (namely TPF) is 120 degrees.
Step 3. Apply the cosine law to triangle FPT (using cos 120 = .5).
x^{2} = 3^{2} + 5^{2}  2 3 5 (.5) = 49. Thus x = 7.
Chris and Ed
