Concave and Convex Polygon - Turn arbitrary polygon into a convex polygon
aioobe has it right—you sound as though you are looking to compute the convex hull of your polygon, in which case you want one of the convex hull algorithms like Graham scan or Chan's algorithm.
However, if you just want to know whether an angle is convex or concave there's a quick way to compute this that avoids trigonometry.
If A, B, and C are consecutive vertices going clockwise around the polygon, then the vertex at B is convex if
(B − A)⟂ · (C − B) < 0
Here V⟂ is a vector that's V rotated by 90° anti-clockwise, which can be computed like so: (x, y)⟂ = (−y, x).
You can detect concave points by looking at the interior angle — if it's greater than a half circle then the point is concave. Indeed they're usually called reflex points because the interior angle is reflex.
A quick way to check is the dot product. For example, look at the three points P14, P15, P16. P16 is behind the line that the segment between P14 to P15 lies on (ie, the dot product of the vector from P15 to P16 with the normal to that line is negative), so P15 is a convex point.
P18 is in front of the line that the segment from P16 to P16 lies on (ie, the dot product of the vector from P17 to P18 with the normal of that line is positive), so P17 is a reflex point.
In 2d, the normal of the line is as simple as flipping the x and y coordinates and negating one.
However, what I think you might want is the convex hull — consider whether there might be convex points that nonetheless create a concave hull. The most obvious example is if I left P17 where it is but moved P18 and P16 even further into the shape, behind it. If so then you want to check out convex hull algorithms.
Use a convex hull algorithm (such as the Graham scan), and remove all points that are not part of the resulting convex hull.
In your example, the convex hull will consist of P1, P2, P3, P5, P7, P8, P9, P11, P12, P13, P14, P15, P16, P18, which are precisely all points except the red ones.
Note that simply removing those points whose inner angle is greater than 180 will not necessarily result in a convex polygon. Take this polygon for example:
