What do epimorphisms of (commutative) rings look like?
No, not every epimorphism of rings is a composition of localizations and surjections.
An epimorphism of commutative rings is the same thing as a monomorphism of affine schemes. Monomorphisms are not only embeddings, e.g., any localization is an epimorphism and the corresponding morphism of schemes is not a locally closed embedding.
Example: Let $C$ be the nodal affine cubic and let $X$ be its normalization. Pick any point $x$ above the node. Then $X\setminus\{x\}\to C$ is a monomorphism (see Proposition below). The corresponding homomorphism of rings is injective but not a localization.
Proposition (EGA IV 17.2.6): Let $f\colon X\to Y$ be a morphism locally of finite type between schemes. TFAE:
(i) $f$ is a monomorphism.
(ii) Every fiber of $f$ is either an isomorphism or empty.
Remark: A flat epimorphism $A\to B$ is a localization if $A$ is normal and Q-factorial. This is a result by D. Lazard and P. Samuel. [cf. Lazard "Autour de la platitude" (IV, Prop 4.5)]
Remark: There was a seminar on epimorphisms of rings directed by P. Samuel in 1967-68.
George Bergman gave me a reference (Isbell's "Epimorphisms and dominions, IV") and a very pretty counterexample. In particular, he says that the characterization of epimorphisms Andrew gave us works for non-commutative rings as well:
Recall that an inclusion A in B is an epimorphism if and only if the "dominion" of A in B is all of B, where this dominion is defined as the subring of elements b of B which behave the same under all pairs of homomorphisms on B that agree on elements of A.
Now the Silver-Mazet-Isbell Zigzag Lemma for rings says that the dominion of A in B consists of those elements of B which can be written XYZ, where X is a row, Y a matrix, and Z a column over B, such that XY and YZ have entries in A. (It is easy to verify that such a product is in the dominion of A -- a generalization of the proof that if Y is in A and has an inverse in B, then this inverse is in the dominion of A.)
Let k be a field. Consider the inclusion of k[x, xy, xy2 - y] into k[x,y]. I claim that this is an epimorphism. Note that it is an inclusion, no non-units become units, and k[x,y] has no idempotents.
Suppose f and g are two morphisms from k[x,y] to some other commutative ring which agree on the given subring. Using f(xy)=g(xy) and f(x)=g(x), we see that f(xy2)=g(xy2):
f(yxy) = f(yx)f(y) = g(yx)f(y) = g(y)g(x)f(y) = g(y)f(x)f(y) = g(y)f(xy) = g(y)g(xy) = g(yxy)
Since f and g agree on xy2-y, they agree on y, so they agree on all of k[x,y].
Finally, to see that the inclusion is not an isomorphism, consider the surjective morphism k[x,y] to k[x,x-1] sending y to x-1. This sends the subring to k[x], which is clearly smaller, so the inclusion of k[x,xy,xy2-y] into k[x,y] must be strict.
Here is another perspective on your question. As $\mathbb{Z}$ is the initial object of unital (commutative) rings, one might first of all ask: What do epimorphisms from $\mathbb{Z}$ look like?
So if $A = \mathbb{Z}$ in the original question, what can $B$ be? The answer to this is known. In fact, these rings $B$ and their classification seem to have been (re)invented several times, as "solid rings" by Bousfield and Kan (see MO question 95160: Solid Rings and Tor), as "T-rings" by R. A. Bowshell and P. Schultz (Unital rings whose additive endomorphisms commute, Math. Ann. 228 (1977), 197-214, http://eudml.org/doc/162991;jsessionid=07C5F5F5BBD354C0914511776DA20F5E), and the generalisation to Dedekind domains has been done in W. Dicks and W. Stephenson: Epimorphs and Dominions of Dedekind Domains, J. London Math. Soc. (1984) s2-29(2): 224-228, http://jlms.oxfordjournals.org/content/s2-29/2/224.extract . (Also, by Martin Brandenburg and myself this summer, before we found these papers ...)
So here is a positive answer under a restrictive assumption: If $A \rightarrow B$ is an epimorphism and $A$ is a Dedekind domain, then $B$ will be built up from localisations and quotients of $A$ by suitable finite products and direct limits. To make "suitable" more specific, here follows a more concrete description (the literature above mostly says "take colimits/pullbacks"; see Martin's comment for other descriptions). I restrict to $A = \mathbb{Z}$ for (mostly notational) simplicity:
Let $P$ be the set of prime numbers and let $n: P \rightarrow \mathbb{N} \cup \lbrace 0, \infty \rbrace $ be any map (a "supernatural number"). Let $P_{fin}(n)$ be the set of primes $p$ with $n(p) < \infty$. Define
$B_n := \lbrace ((b_p)_p, b_l) \in \prod_{p \in P_{fin}(n)} \mathbb{Z} / p^{n(p)} \times \mathbb{Z}[P_{fin}(n)^{-1}] :$ $$b_p \equiv b_l \text{ mod } p^{n(p)} \text{ for all but finitely many } p \in P_{fin}(n)(b_l) \rbrace$$
(index "$l$" for "localisation part") where:
-- $\mathbb{Z}[P_{fin}(n)^{-1}]$ is the localisation of $\mathbb{Z}$ at the multiplicative set generated by $P_{fin}(n)$, i.e. the subring of $\mathbb{Q}$ generated by $\lbrace p^{-1}: p \in P_{fin}(n) \rbrace$;
-- with $v_p$ being the $p$-adic valuation on $\mathbb{Q}$, $P_{fin}(n)(b_l) := \lbrace p \in P_{fin}: v_p(b_l) \ge 0 \rbrace$ and the condition $b_p \equiv b_l \text{ mod } p^{n(p)}$ makes sense and is to be understood in the subring of $\mathbb{Q}$ where only the $p$'s with $v_p(b_l) < 0$ are inverted.
Then $B_n$ is in fact a subring of the direct product, and for $n$ ranging over the supernatural numbers, these are all $B$ with injective epimorphisms $\mathbb{Z} \rightarrow B$. (The non-injective ones are just the quotients. With more complicated notation, one could include this case by counting 0 as a prime.)
Here are two easy-to-see properties:
$B_n$ is noetherian if and only if $|P_{fin}(n) \setminus P_0(n) | < \infty$ (where $P_0(n) :=$ set of primes $p$ with $n(p) = 0$), if and only if $B_n$ is the direct product of a quotient and a localisation, namely, $\mathbb{Z}/n \times \mathbb{Z}[P_{fin}(n)^{-1}]$ where by abuse of notation $n$ is the natural number $\prod_{p \in P_{fin}(n)} p^{n(p)}$.
The non-zero primes of $B_n$ correspond to the ones in $P \setminus P_0(n)$. In particular, $B_n$ is artinian if and only if its Krull dimension is 0 if and only if $|P \setminus P_0(n)| < \infty$. Otherwise, its Krull dimension is 1.
All this remains true cum grano salis for any Dedekind domain $A$ instead of $\mathbb{Z}$. In particular, as soon as $A$ has infinitely many primes, there are epimorphisms $A \rightarrow B$ where $B$ is non-noetherian. On the other hand, if $A$ has only finitely many primes (which by the way makes it a PID), $B$ will be of the form $A/a \times S^{-1}A$ with $a \in A$ and $S \subseteq A$ multiplicative containing all primes dividing $a$ (and possibly 0). In any case, $B$ will be a colimit of products of localisations and quotients as above, so the answer to the question
suppose $f:A \rightarrow B$ is an epimorphism of rings with no kernel which sends non-units to non-units and such that $B$ has no idempotents. Must f be an isomorphism?
seems to be yes if $A$ is a Dedekind domain: E.g. in the above setting, non-units to non-units implies $P_0(n) = \emptyset$ and $B$ having no idempotents implies $P_{fin}(n) \setminus P_0(n) = \emptyset$.
Further remarks:
Remark 1 (cf. David Rydh's first remark): Flat epimorphisms (from any unital ring) are localisations for a certain Gabriel topology and have a kind of a calculus of fractions. For a precise statement, see Quelques observations sur les épimorphismes plats (à gauche) d'anneaux by N. Popescu and T. Spircu, Journ. Alg. vol. 16, no. 1, pp. 40-59, 1970, http://dx.doi.org/10.1016/0021-8693(70)90039-6, or Bo Stenström's book Rings of Quotients, theorem 2.1 in chapter XI.
Remark 2: Further information might be in the papers of H. H. Storrer, e.g. http://retro.seals.ch/digbib/view?rid=comahe-002:1973:48::11
Remark 3: I have not checked all the details in the generalisation to Dedekind domains, so beware (at least, Martin and I had reached the same result for PIDs). Also, I do not know if there is a generalisation beyond Dedekind domains; I guess Krull domains might be attackable, but I have not seriously tried.