Largest circle inside a non-convex polygon

The key to solving this problem is first making an observation: the center of the largest circle that will fit inside an arbitrary polygon is the point that is:

1. Inside the polygon; and
2. Furthest from any point on the edges of the polygon.

Why? Because every point on the edge of a circle is equidistant from that center. By definition, the largest circle will have the largest radius and will touch the polygon on at least two points so if you find the point inside furthest from the polygon you've found the center of the circle.

This problem appears in geography and is solved iteratively to any arbitrary precision. Its called the Poles of Inaccessibility problem. See Poles of Inaccessibility: A Calculation Algorithm for the Remotest Places on Earth.

The basic algorithm works like this:

1. Define R as a rectilinear region from (x_min,y_min) to (x_max,y_max);
2. Divide R into an arbitrary number of points. The paper uses 21 as a heuristic (meaning divide the height and width by 20);
3. Clip any points that are outside the polygon;
4. For the remainder find the point that is furthest from any point on the edge;
5. From that point define a new R with smaller intervals and bounds and repeat from step 2 to get to any arbitrary precision answer. The paper reduces R by a factor of the square root of 2.

One note, How to test if a point is inside the polygon or not: The simplest solution to this part of the problem is to cast a ray to the right of the point. If it crosses an odd number of edges, it's within the polygon. If it's an even number, it's outside.

Also, as far as testing the distance to any edge there are two cases you need to consider:

1. The point is perpendicular to a point on that edge (within the bounds of the two vertices); or
2. It isn't.

(2) is easy. The distance to the edge is the minimum of the distances to the two vertices. For (1), the closest point on that edge will be the point that intersects the edge at a 90 degree angle starting from the point you're testing. See Distance of a Point to a Ray or Segment.

In case anyone is looking for a practical implementation, I designed a faster algorithm that solves this problem for a given precision and made it a JavaScript library. It's similar to the iterative grid algorithm described by @cletus, but it's guaranteed to obtain global optimum, and is also 20-40 times faster in practice.

Check it out: https://github.com/mapbox/polylabel