Can you find a domain where $ax+by=1$ has a solution for all $a$ and $b$ relatively prime, but which is not a PID?

If I'm not mistaken, the integral domain of holomorphic functions on a connected open set $U \subset \mathbb{C}$ works. It is a theorem (in Chapter 15 of Rudin's Real and Complex Analysis, and essentially a corollary of the Weierstrass factorization theorem), that every finitely generated ideal in this domain is principal. This implies that if $a,b$ have no common factor, they generate the unit ideal. However, for instance, the ideal of holomorphic functions in the unit disk that vanish on all but finitely many of ${1-\frac{1}{n}}$ is nonprincipal.

See http://en.wikipedia.org/wiki/Bezout_domain

How about this construction:

Define a domain $R_0$ as follows. Take a field $K$, adjoin an indeterminate $x_0$, and localize at $(x_0)$ (that is, adjoin inverses to everything not a multiple of $x_0$).

$R_0$ has all its ideals principal and linearly ordered: $(x_0)$ contains $(x_0^2)$ contains $(x_0^3)$...

Now given $R_i$, define $R_{i+1}$ inductively: Adjoin an indeterminate $x_{i+1}$, so we have $R_i[x_{i+1}]$. Quotient by $(x_{i+1}^2 - x_i)$. Finally, localize at the prime ideal $(x_{i+1})$.

This effectively just gives us one more principal ideal containing all the principal ideals from $R_i : (x_{i+1})$ contains $(x_{i+1}^2)=(x_i)$ contains $(x_i^2)$...

Now let $R$ be the union of all the $R_i$, and it's obvious that any finitely generated ideal is principal, but there's a non-fg one generated by all the $x_i$.