How is a group made up of simple groups?

Everything you say is correct: the sense that a finite group is "built" from its simple Jordan-Hölder factors is by repeated extensions. But this "building" process is much more complicated for groups than the analogous process of building integers from prime numbers because given a (multi-)set of building blocks -- i.e., a finite list $\mathcal{H} = \{ \{H_1,\ldots,H_n\} \}$ of finite simple groups -- there will be in general several (finitely many, obviously, but perhaps a large number) nonisomorphic groups $G$ with composition factors $\mathcal{H}$. The simplest example of this has already been given by Zhen Lin in a comment: if

$\mathcal{H} = \{ \{ C_2, C_2 \} \}$,

then the two groups with these composition factors are $C_4$ and $C_2 \times C_2$.

It seems to be a working assumption of experts in the field that it is hopeless to expect a nice solution to the extension problem. For instance, consider the special case $\mathcal{H} = \{ \{ C_p,\ldots,C_p \} \}$, in which every composition factor is cyclic of order $p$ -- i.e. a finite $p$-group. It is known that the function $f(p,n)$ which counts the number of isomorphism classes of finite groups of order $p^n$ grows very rapidly as a function of $n$ for any fixed $p$. For instance, see here for a reference to the fact that $f(2,9) = 10494213$.

Nevertheless the group extension problem is an important and interesting one -- it is one of the historical sources for the field of group cohomology and still plays a major role -- and in many special cases one can say something nice. But the general "program" of classifying all finite groups by (i) classifying all simple groups and (ii) determining all finite groups with a given set $\mathcal{H}$ of composition factors does not seem realistic: step (i) was amazingly hard but in the end doable. It looks very easy compared to step (ii)!

Finally, you ask about infinite groups. Here the Jordan-Hölder theory extends precisely to groups $G$ which admit at least one composition series, and a standard (necessary and sufficient) criterion for this is that there are no infinite sequences of subgroups

$H_1 \subsetneq H_2 \subsetneq \ldots$

with each $H_i$ normal in $H_{i+1}$

or

$H_1 \supsetneq H_2 \supsetneq \ldots$

with each $H_{i+1}$ normal in $H_i$.

So for instance an infinite cyclic group $\mathbb{Z}$ does not satisfy the descending chain condition on subgroups and there is no sense (known to me, at least) in which $\mathbb{Z}$ is built up out of simple groups.

Of course, Jordan–Hölder does show how to build groups from simple composition factors. However, there are other ways to dissect groups (finite groups, at least), which locate the "essential part" of a finite group in a different way. For example, let $G$ be a finite solvable group. Then $G$ has a unique largest nilpotent normal subgroup, its Fitting subgroup $F(G)$. The Fitting subgroup satisfies $C_{G}(F(G)) = Z(F(G)),$ and it follows that the factor group $G/F(G)$ is isomorphic to to a subgroup of the outer automorphism group of $F(G)$. Since $F(G)$ is a direct product of $p$-groups, its outer automorphism group is a direct product of outer automorhism groups of $p$-groups. In fact, $G/F(G)$ is isomorphic to a subgroup of a group of the form $X_1 \times \ldots \times X_t,$ where each $X_i$ is isomorphic to a subgroup of a completely reducible solvable subgroup of ${\rm GL}(n_i,p_i)$ for some integer $n_i$ and prime $p_i$.
For non-solvable groups, the situation is more complicated, and its understanding came later. H Bender introduced the generalized Fitting subgroup $F^{*}(G)$ of a general finite group $G$. It satisfies $C_{G}(F^{*}(G)) = Z(F(G))$, and it is still the case that $G/F^{*}(G)$ is isomorphic to a subgroup of the outer automorphism group of $F^{*}(G)$. The group $F^{*}(G)$ is the product of a pair of normal subgroups $F(G)$ (the usual Fitting subgroup) and $E(G)$. The groups $E(G)$ and $F(G)$ centralize each other, and are characteristic in $G$. A component of $G$ is a subnormal subgroup $L$ such that $L = [L,L]$ and $L/Z(L)$ is simple. It turns out that distinct components centralize each other. The group $E(G)$ is the central product of all the components of $G$ (and $G$ permutes its components by conjugation). The automorphism group of $E(G)$ has a normal subgroup $K$ consisting of the automorphisms which fix every component, and ${\rm Aut}(E(G))/K$ is a permutation group of degree $n,$ where $G$ has $n$ components. Also, $K/E(G)$ is isomorphic to a subgroup of a direct product of outer automorphism groups of finite simple groups. Thus the structure of $F^{*}(G)$ controls the structure of $G$ to a large extent. This viewpoint was very powerful in the later stages in the classification of the finite simple groups, and figures prominently in attempts to revise and simplify its proof. Note that $F^{*}(G)$ is built in a very transparent way from its own composition factors, which are among the composition factors of $G$.

In Dummit & Foote's "Abstract Algebra" they briefly discuss the Hölder Program:

1. Classify all finite simple groups.
2. Find all ways of "putting simple groups together" to form other groups.

They write the following on part 2 of the program (the so-called extension problem for finite groups):

Part (2) of the Hölder Program, sometimes called the extension problem, was rather vaguely formulated. A more precise description of "putting two groups together" is: given groups $A$ and $B$, describe how to obtain all groups $G$ containing a normal subgroup $N$ such that $N \cong B$ and $G/N \cong A$. For instance, if $A=B=Z_2$, there are precisely two possibilities for $G$, namely $Z_4$ and $V_4$ [the Klein four grup] and the Hölder Program seeks to describe how the two groups of order 4 could have been built from two $Z_2$'s without a priori knowledge of the existence of the groups of order 4. This part of the Hölder Program is extremely difficult, even when the subgroups involved are of small order. For example, all composition factors of a group $G$ have order 2 if and only if $|G| = 2^n$, for some $n$ (...). It is known, however, that the number of nonisomorphic groups of order $2^n$ grows (exponentially) as a function of $2^n$, so the number of ways of putting groups of 2-power together is not bounded. Nonetheless, there are a wealth of interesting and powerful techniques in this subtle area which serve to unravel the structure of large classes of groups.