What is the Coxeter diagram for?

The diagrams are a way of describing a group generated by reflections. Any collection of reflections (in Euclidean space, say) will generate a group. To know what this group is like, you need to know more than just how many generators there are: you need to know the relationships between the generators. The Coxeter diagram tells you that information. There is a node for each generator, and an edge between the two labeled with the order of their product.

For instance, if you have a group generated by three reflections $\rho_1$, $\rho_2$, and $\rho_3$, then you know that $\rho_i^2 = 1$ (the order of each reflection is two), but the order of $\rho_1 \rho_2$, $\rho_1 \rho_3$, and $\rho_2 \rho_3$ could be anything. Maybe $(\rho_1 \rho_2)^3 = (\rho_1 \rho_3)^4 = (\rho_2 \rho_3)^5 = 1$. Then the Coxeter diagram is . By convention, edges that would be labeled "2" are omitted, and any "3" labels are left off, so we'd actually have .

So, nodes in the graph are not-adjacent exactly when the product of the corresponding generators has order 2, which for involutions means they commute: $ (\rho_i \rho_j)^2 = 1$ means $\rho_i \rho_j = (\rho_i \rho_j)^{-1} = \rho_j^{-1} \rho_i^{-1} = \rho_j \rho_i$.

Regular polytopes

For a regular convex polytope $P$, there is a standard way to label the generators. Fix a base flag $\Phi$ (a maximal collection of mutually incident faces: a vertex, an edge, etc.) Since $P$ is regular, there are symmetries (i.e. isometries which take $P$ to itself) carrying each flag to every other flag; in particular, there is a symmetry taking $\Phi$ to the flag with all the same faces, except that it has the other vertex on the given edge. We call this flag $\Phi^0$, the 0-adjacent flag to $\Phi$, and the symmetry $\rho_0$. We can see that $\rho_0(\Phi^0)$ must be $\Phi$ again, and so $\rho_0$ is an involution (it's not hard to show that a symmetry which fixes some flag is the identity.)

Similarly, there is a unique flag $\Phi^1$ which has all the same faces as $\Phi$ except that it has the other edge containing the given vertex and contained in the given 2-face, and a symmetry $\rho_1$ which carries $\Phi$ to $\Phi^1$; and for every rank $j$ up to the dimension of $P$, there is a symmetry $\rho_j$ carrying $\Phi$ to the unique $j$-adjacent flag $\Phi^j$.

It can be shown that these are involutions, and that they generate the whole symmetry group. Moreover, in this particular case, $\rho_i$ and $\rho_j$ always commute if $|i - j| \geq 2$. (For instance, with $\rho_0$ and $\rho_2$: if you switch from one vertex on an edge to the other, then switch from one 2-face at the edge to the other, then switch vertices back, then switch 2-faces back, you get back where you started.)

For this reason, the Coxeter diagram of the symmetry group of a regular polytope will be a string (like the example you gave for the cube). Conventionally the nodes are given left-to-right as $\rho_0, \rho_1, \dots$. If the labels on the edges are $p, q, r, \dots$, then the abstract Coxeter group associated with the diagram is often called $[p,q,r,\dots]$.

The abstract Coxeter group is simply the group defined by the presentation inherent in the diagram, i.e. for your example $$ [4,3] = \langle \rho_0, \rho_1, \rho_2 \mid \rho_0^2 = \rho_1^2 = \rho^2_2 = (\rho_0 \rho_2)^2 = (\rho_0 \rho_1)^4 = (\rho_1 \rho_2)^3 = 1 \rangle, $$ which is isomorphic to any concrete Coxeter group with the same diagram, formed by actual reflections in some space.

The regular polytope with this group has the so-called Schläfli symbol $\{p,q,r,\dots\}$. The Schläfli symbol for the cube is {4,3}. This means that following a vertex-swap by an edge-swap (i.e. $\rho_0 \rho_1$) has order 4, and an edge-swap followed by a facet-swap ($\rho_1 \rho_2$) has order 3. A more typical way to recognize this is to say "Each facet is a 4-gon and each vertex is incident to 3 edges."

Here's the diagram of the 4-cube. The corresponding Schläfli symbol is {4,3,3}:

• $\rho_0 \rho_1$ has order 4; each 2-face is a square.
• $\rho_1 \rho_2$ has order 3; within a given facet, each vertex is in 3 edges.
• $\rho_2 \rho_3$ has order 3; each edge is in 3 facets.

It is probably clear that for regular polytopes, you might as well just use Schläfli symbols. But there are many groups generated by reflections which are not the symmetry groups of regular polytopes. Every such group is described by a Coxeter diagram.

Why?

As far as why this notation is used: you just need some way to give the orders of all the products $\rho_i \rho_j$. Listing all the orders in a group presentation is usually really long, ill-organized, and hard to read. Another way is to put it in a matrix, and this is indeed frequently used. If you have an $n \times n$ matrix $M = [m_{ij}]$, your group is generated by $n$ involutions $\rho_0, \dotsc, \rho_{n-1}$ such that $(\rho_i \rho_j)^{m_{ij}} = 1$ (and you want $m_{ij}$ to be minimal, of course, so that it is the order of $\rho_i \rho_j$.) $M$ must be symmetric, since the order of $\rho_i \rho_j$ and $\rho_j \rho_i$ are the same. And if the generators are involutions, the diagonal entries $m_{ii}$ must be 1.

Perhaps you are more familiar or comfortable with the idea of defining a group by such a matrix (called a Coxeter matrix.) In this case, it's worth emphasizing that a Coxeter diagram and a Coxeter matrix are entirely equivalent. Some people go so far as to identify the two.

The advantages of the diagram are that the matrix is redundant; you only need the entries above the diagonal (or below it.) Also, diagrams make it more clear when things commute and highlight the "interesting" relationships (when order is more than 3) so that they're not lost in the noise. For instance, in the diagram for a $p$-gonal prism: it is immediately clear that we have the symmetry group of a $p$-gon, and another reflection orthogonal to both the generating reflections of the former. This is perhaps not as immediate looking at the matrix $\begin{bmatrix} 1 & p & 2\\ p & 1 & 2 \\ 2 & 2 & 1\end{bmatrix}$.

From diagrams to polytopes

This is not my area of expertise, but it addresses the parts of your question about angles, and the mysterious extra circle in your diagram for the cube.

Given a Coxeter diagram with $n$ nodes, you can construct reflections in $n$-dimensional space to realize the Coxeter group. For convenience, we'll identify reflection isometries with their hyperplane of reflection, so if $\rho$ is a reflection, then $\rho$ also means the hyperplane fixed by $\rho$.

To get the product of reflections to have order $p$, you want their hyperplanes of reflection at an angle of $\pi/p$ to each other, since the composition of the two reflections is a rotation by twice the angle between them.

The composition would have the same order with an angle of $m\pi/p$, where $m$ is relatively prime to $p$. The group generated ends up being the same, so you might as well work with the hyperplanes at an angle of $\pi/p$.

So with , $\rho_0$ and $\rho_1$ should form an angle of $\pi/4$ (or 45°), $\rho_0$ and $\rho_2$ should form an angle of $\pi/2$ (or 90°), and $\rho_1$ and $\rho_2$ should form an angle of $\pi/3$ (or 60°).

I don't really know how to go about finding planes that have the specified relationship, but you can visualize how to do this one. Start with two planes which are orthogonal to eachother: call them $\rho_0$ and $\rho_2$. Stick in a plane which forms an angle of 45° with $\rho_0$; you can start with it also being orthogonal to $\rho_2$, so we have the situation in depicted in this picture. On the left, we have transparent plane segments; on the right, they are opaque. I am thinking of $\rho_2$ as the horizontal plane, $\rho_0$ as the vertical plane coming straight out of the page, and the "new plane" (intended to be $\rho_1$) as the plane going from corner to corner.

Then rotate the new plane, keeping it at 45° with $\rho_0$, until it forms an angle of 60° with $\rho_2$. You end up with this:

Come to think of it, on a sphere whose center is the point of intersection of your three planes, the spherical triangle cut out by the planes will have angles 45°, 90°, and 60°. So I guess finding such spherical simplices is a general method to do this. A more systematic way of finding some reflection planes for your group seems to be described in Arjeh Cohen's notes on Coxeter groups, Section 2.2: The reflection representation.

Anyway, one way or another, you've found some reflections whose compositions have the prescribed order. Now to find a polytope which the group generated by these reflections act on, just pick any point in space, take all its images under the reflections, and voilà! The convex hull of these points is a polytope acted on by the group. This is known as Wythoff's construction, or a kaleidoscope (because the original point is replicated by all the reflection planes just as a colored dot is replicated in a kaleidoscope.)

Many choices of points yield combinatorially identical (or isomorphic) polytopes; for instance, taking any point which is not contained in any of the planes of reflection will result in isomorphic polytopes. More interesting things happen when the initial point is in some of the planes (but not all the planes; then its orbit under the group is just a point.)

As an extension to the Coxeter diagram, you circle all the nodes of the diagram corresponding to reflection planes that DO NOT contain the initial point. (This might seem kind of backwards. It probably is.)

So, in , the initial point is contained in the reflection planes for $\rho_1$ and $\rho_2$ but not for $\rho_0$. Here's one such initial point (the black one) on the reflection planes we constructed earlier:

The black point is the initial point; its reflection through the $\rho_0$ plane (the vertical plane) is red. The green point is the reflection of the red one through $\rho_1$. The blue points are the reflections of the green one through $\rho_0$ and $\rho_2$. The hollow black point is the reflection of the blue ones (through either $\rho_0$ or $\rho_2$.) The hollow red point (not visible in the left picture) is the reflection of the hollow black point through the plane $\rho_1$. Its reflection through $\rho_0$ is the hollow green point.

On the right, we see that the convex hull of these points is indeed the cube.

Following Nich Matteo's answer, I find this's straightforward to find $\rho_0, \rho_1, \rho_0$ in a cube.

$\rho_0$ is reflecting by plane OAD.

$\rho_1$ is reflecting by plane OBC.

$\rho_2$ is reflecting by plane OAB.

$ (\rho_1 \rho_2)^3=1$ can be illustrated by the following six reflections:

$ (\rho_0 \rho_1)^4=1$ can be illustrated by the following 8 operations:

quote from wiki: In a symmetry group, the group elements are the symmetry operations (not the symmetry elements), and the binary combination consists of applying first one symmetry operation and then the other.

The binary operations also satisfy:

(1) closure property: If $\rho_i \rho_j$ is a group element, then $\rho_j \rho_i$ must be a group element.

(2) associative property: $\rho_0 \rho_1 \rho_2 = \rho_2 \rho_0 \rho_1 $.