How many triangles can be formed by the vertices of a regular polygon of $n$ sides?

Consider a regular polygon with $n$ number of vertices $\mathrm{A_1, \ A_2,\ A_3, \ A_3, \ldots , A_{n-1}}$ & $\mathrm{A_{n}}$

Total number of triangles formed by joining the vertices of n-sided regular polygon $$N=\text{number of ways of selecting 3 vertices out of n}=\color{}{\binom{n}{3}}$$ $$N=\color{red}{\frac{n(n-1)(n-2)}{6}}$$ $\forall \ \ \color{blue}{n\geq 3}$

Consider a side $\mathrm{A_1A_2}$ of regular n-polygon. To get a triangle with only one side $A_1A_2$ common (As shown in figure-1 below)

(figure-1)

Join the vertices $A_1$ & $A_2$ to any of $(n-4)$ vertices i.e. $A_4, \ A_5,\ A_6, \ \ldots \ A_{n-1}$ to get triangles with only one side common. Thus there are $(n-4)$ different triangles with only one side $A_1A_2$ common. Similarly, there are $(n-4)$ different triangles with only one side $A_2A_3$ common & so on. Thus there are $(n-4)$ different triangles with each of $n$ sides common. Therefore, number of triangles $N_1$ having only one side common with that of the polygon $$N_1=\text{(No. of triangles corresponding to one side)}\text{(No. of sides)}=\color{blue}{(n-4)n}$$

(figure-2)

Now, join the alternate vertices $A_1$ & $A_3$ by a straight (blue) line to get a triangle $A_1A_2A_3$ with two sides $A_1A_2$ & $A_2A_3$ common. Similarly, join alternate vertices $A_2$ & $A_4$ to get another triangle $A_2A_3A_4$ with two sides $A_2A_3$ & $A_3A_4$ common & so on (as shown in above figure-2). Thus there are $n$ pairs of alternate & consecutive vertices to get $n$ different triangles with two sides common (Above fig-2 shows $n$ st. lines of different colors to join alternate & consecutive vertices). Therefore, number of triangles $N_2$ having two sides common with that of the polygon $$N_2=\color{blue}{n}$$ If $N_0$ is the number of triangles having no side common with that of the polygon then we have $$N=N_0+N_1+N_2$$ $$N_0=N-N_1-N_2$$ $$=\binom{n}{3}-(n-4)n-n$$ $$=\color{}{\frac{n(n-1)(n-2)}{6}-n^2+3n}$$ $$N_0=\color{red}{\frac{n(n-4)(n-5)}{6}}$$ The above formula $(N_0)$ is valid for polygon having $n$ no. of the sides such that $ \ \ \color{blue}{n\geq 6}$

total no of triangles formed by joining vertices of n-sided polygon $$= \frac{n(n-1)(n-2)}{6}$$ i.e. selection of 3 points from n points = n(C)3 $\implies$ can also be written as sum of no of triangles formed in the following three cases,

1) no of triangles with only one side common with polygon, if we take any one side of a n-sided polygon and join its vertices to the remaining vertices, except the vertices adjacent to vertices of the line taken above, we get triangles with only one side as common i.e. for 1 side we get (n-4) triangles $\implies$ n(n-4) triangles for n sides. case I
2) no of triangles with two sides common, if we take any one side of a n-sided polygon join its vertex with its opposite vertex required triangle is formed. Hence no of triangles= n case II

3) triangles with no side common $$= \text{total - (Case I + Case II)}$$ $$=\left[\frac{n(n-1)(n-2)}{6}\right]-\left[n(n-4) + n\right]$$ $$=\frac{n(n-4)(n-5)}{6}$$