Determine whether one polygon contains another
If we're trying to check whether polygon B is inside polygon A:
Like mentioned in the answer you linked to, start off doing a line intersection test for each pair of edges, one from each polygon. If any edges intersect (excluding vertices that lie on edges and common vertices), B is not inside A.
If a vertex V of one polygon lies on an edge of the other, treat that edge instead as 2 edges, with the vertex V as a new vertex for that polygon.
Now we only have to think about common vertices.
For each common vertex V:
- We can say V has edges EA1 and EA2 (from A) and EB1 and EB2 (from B).
- Get the gradients of all 4 the edges.
Use this to determine which edges are adjacent.
If EB1 and EB2 are not adjacent, B is not within A.
If EB2 lies on A (that is, EB2 lies on EA2, i.e. they have an equal gradient), we don't yet know whether B is in A.
In this case, we'll need to keep track of on which side EB1 was and move on to the adjacent vertex VB (the other vertex of EB2), and check whether EB3, the edge after EB2, is on the same side as EB1. If they're on the different sides, B is not inside A.
If EB3 is also on A, we need to check EB4, and so on, until we find one that isn't.
If both EB1 is on EA1 and EB2 is on EA2, we need to move in both directions to determine on which side we need to be. If all edges of B lie on A, A is equal to B.
(Note: if, for example, EB2 lies on EA1 instead of EA2, you can simply rename them to fulfill the above condition)
All of this is assuming we don't allow polygons that intersect with themselves - allowing that will probably break this.
There may be some non-trivial details here, but this should be a decent starting point.
The sweepline algorithm (as nearly always) gives us the most robust and efficient solution.
In its simplest form, sweepline finds all line segment intersections. It is trivial to extend it to check polygon containment. You just keep track which line or point belong to each polygon. At any step of the algorithm, the intersection of the sweeping line and the interior of each polygon is a finite set of vertical segments. You have these cases:
- If there is a proper (i.e. not at an endpoint) edge intersection at any step, game over.
- Otherwise if at every step red and blue segments are disjoint, then the polygons are completely outside each other.
- Otherwise if at every step red segments are identical to blue segments (i.e. the red set and the blue set are the same), then the two polygons are the same.
- Otherwise if at every step each red segment is completely inside some blue segment, then the red polygon is inside the blue one.
- Otherwise if at every step each blue segment is completely inside some red segment, then the blue polygon is inside the red one.
- Otherwise polygon boundaries intersect.
This takes care of all edge cases. If you want to classify your cases A, E and F as "inside", test only intersection of segment interiors (i.e. segments (0,1) and (1,2) are not intersecting, and (0,1) is inside of (0,2)). Otherwise, just treat the above cases as proper intersections.
If at some step there are two edges that are collinear and parallel to the sweep line and they intersect, it could be a bit hard to decide. However all edge cases can be resolved by calculating sweepline-polygon intersection not at vertices (as is normal to the sweepline algorithm) but between vertices (say at a midpoint between the current vertex and the next). This way only polygon interiors (not boundaries) are ever considered.
In effect the algorithm breaks up our polygons into a bunch of little trapezoids, sandwiched between parallel (say vertical) lines drawn through each vertex. It's very easy to check whether these trapezoids are intersecting, or disjoint, or completely contain one another. An illustration can be found here.
Edit: clarified some wording. Edit 2: found an edge case ;)