Chris

The answer to your question is easy when the polygon has an odd number of sides, but quite complicated when the number is even! It is VERY difficult to see why this is so. So let's begin with an easier question: Find the number of interior intersection points made by the diagonals of a polygon whose n vertices are placed around a circle so that no three of the diagonals intersect in the same point. You then get a new intersection point with every choice of four vertices A,B,C,D in that order around the circle; that is AC intersects BD in a unique interior point. The formula for choosing 4 points from n given points is

\[\left( \begin{array}{c}n\\4\end{array} \right) = \frac{n\times(n-1)\times(n-2)\times(n-3)}{4\times3\times2\times1}.\]

So for a pentagon, the answer is $5\times4\times3\times2/24 = 5,$ as you already determined. One can prove that when n is odd, the diagonals of a regular always intersect in distinct points -- you can never get three or more of them intersecting in the same point! (To prove this requires some pretty advanced number theory!)

Now for the regular hexagon (with 6 sides), the formula gives $6\times5\times4\times3/24 = 15$ points, while you correctly counted 13 DISTINCT points. What happens is that because of the symmetry of a regular hexagon, three diagonals meet in the center. The intersection pattern of regular polygons with an even number of sides is quite complicated, The whole story is told by Bjorn Poonen and Michael Rubinstein in their 1998 paper "The Number of Intersection Points Made by the Diagonals of a Regular Polygon (Siam Journal of Discrete Mathematics 11 (1998) pages 135-156). Perhaps you can find the details on the internet somewhere -- I didn't try.

Chris