The simple answer is that there is no 'incircle' for many quadrilaterals, in the sense of a circle which meets all four sides, and there may not be a unique answer, if you shift your definition to meets three sides (think of a rectangle). This does make it difficult to 'find a formula'.

I guess it depends what you want to do with it. Worst case, you could find incircles for triangles from sets of three lines, and then check which of them are suitable!

Walter Whiteley

Joaquim wrote further:

I forgot to mention that I know that irregular polygons don't have an
incircle in the regular meaning of the word, but more specifically what I
need is to find the largest circle that fits inside an irregular polygon.

Thanks in advance,
Joaquim

We are now into 'computational geometry' (a very interesting field). Here is a site with a number of related computations http://www.soe.ucsc.edu/~jcortes/software/packages/PlanGeom.m It turns out that finding circles with various properties around collections of polygons or points is part of what people do (related
to Voronoi Diagrams).

First - is the polygon convex? This will simplify things (see below). As long as it is simple (no crossings) there will be some answer, but it gets increasingly complex!. Let me answer as if it is convex.

Now,from a simpler point of view: the solution will contact three edges (or at least one of them will if two of the sides are parallel - e.g. the rectangle - and this can be shifted to contact three). You could check all triples, find their incircles (or omit them if the polygon interior is not 'inside' the triangle). The answer
should be one of these. You will still need to rank them (by size) and test them to see if some other line cuts the proposed incircle!

This is not 'fast' (there are nChoose3 such triples) so it would not be good for, say, 1000 sides. I anticipate computational geometers (who worry about worst case situations) could do better. Here is
one, off the top of my head, suggestion.

For example, you could start with a triple which contains the polygon, find an incircle, Now start testing the additional edges, one at a time, to see if this would cut off part of the incircle. If yes, then find the new triple (among just four) which is the largest in-circle. Continue till all edges have been tested. This should take just n-2 steps of checking for intersection, and perhaps selecting a new incircle (three more tests).

Much more could be said, but I hope this helps.

Walter Whiteley