I was hoping you could help me with this problem. How can you seat 6 monsters - 3 Trolls and 3 Gargoyles- at a circular table if the trolls look alike and the gargoyles look alike? Does your formula work for 9 monsters - 4 trolls and 5 gargoyles?
For the case of 3 Trolls and 3 Gargoyles you can count the number of seatings without too much difficulty. Even in the 4 and 5 case you can count the number of seatings. For the general situation I don't know a simple formula.
In these "circular permutation" problems the usual interpretation is that the initial positions at the table are indistinguishable so, in particular, the above arrangement is indistinguishable from
If there 2 Trolls in a row but not 3 then there are two possible arrangements
If no Troll sits beside another Troll there is only one seating arrangement, alternate the Trolls and Gargoyles. Thus there are a total of 4 ways can you seat 6 monsters.