How far are the $p$-adic numbers from being algebraically closed?
A few days ago I was recalling some facts about the p-adic numbers, for example the fact that the p-adic metric is an ultrametric implies very strongly that there is no order on $\mathbb{Q}_p$, as any number in the interior of an open ball is in fact its center.
This argument is not correct. For instance why does it not apply to $\mathbb{Q}$ with the $p$-adic metric? In fact any field which admits an ordering also admits a nontrivial non-Archimedean metric.
It is true though that $\mathbb{Q}_p$ cannot be ordered. By the Artin-Schreier theorem, this is equivalent to the fact that $-1$ is a sum of squares. Using Hensel's Lemma and a little quadratic form theory it is not hard to show that $-1$ is a sum of four squares in $\mathbb{Q}_p$.
I know that if you take the completion of the algebraic close of the p-adic completion you get something which is isomorphic to $\mathbb{C}$ (this result was very surprising until I studied model theory, then it became obvious).
I don't mean to pick, but I am familiar with basic model theory and I don't see how it helps to establish this result. Rather it is basic field theory: any two algebraically closed fields of equal characteristic and absolute transcendence degree are isomorphic. (From this the completeness of the theory of algebraically closed fields of any given characteristic follows easily, by Vaught's test.)
So I was thinking, is there a $p$-adic number whose square equals 2? 3? 2011? For which prime numbers $p$?
All of these answers depend on $p$. The general situation is as follows: for any odd $p$, the group of square classes $\mathbb{Q}_p^{\times}/\mathbb{Q}_p^{\times 2}$ -- which parameterizes quadratic extensions -- has order $4$, meaning there are exactly three quadratic extensions of $\mathbb{Q}_p$ inside any algebraic closure. If $u$ is any integer which is not a square modulo $p$, then these three extensions are given by adjoinging $\sqrt{p}$, $\sqrt{u}$ and $\sqrt{up}$. When $p = 2$ the group of square classes has cardinality $8$, meaning there are $7$ quadratic extensions.
How far down the rabbit hole of algebraic numbers can you go inside the p-adic numbers? Are there general results connecting the choice (or rather properties) of $p$ to the "amount" of algebraic closure it gives?
I don't know exactly what you are looking for as an answer here. The absolute Galois group of $\mathbb{Q}_p$ is in some sense rather well understood: it is an infinite profinite group but it is "small" in the technical sense that there are only finitely many open subgroups of any given index. Also every finite extension of $\mathbb{Q}_p$ is solvable. All in all it is vague -- but fair -- to say that the fields $\mathbb{Q}_p$ are "much closer to being algebraically closed" than the field $\mathbb{Q}$ but "not as close to being algebraically closed" as the finite field $\mathbb{F}_p$. This can be made precise in various ways.
If you are interested in the $p$-adic numbers you should read intermediate level number theory texts on local fields. For instance this page collects notes from a course on (in part) local fields that I taught last spring. I also highly recommend books called Local Fields: one by Cassels and one by Serre.
Added: see in particular Sections 5.4 and 5.5 of this set of notes for information about the number of $n$th power classes and the number of field extensions of a given degree.
Suppose that $K$ is an algebraic number field, i.e. a finite extension of $\mathbb Q$. It has a ring of integers $\mathcal O_K$ (the integral closure of $\mathbb Z$ in $K$). Suppose that there is a prime ideal $\wp \subset \mathcal O_K$ such that:
$p \in \wp,$ but $p \not\in \wp^2$.
The order of $\mathcal O_K/\wp = p.$ (Note that (1) implies in particular that $\wp \cap \mathbb Z = p \mathbb Z$, so that $\mathcal O_K/\wp$ is an extension of $\mathbb Z/p\mathbb Z$. We are now requiring that it in fact be the trivial extension.)
Then the number field $K$ embeds into $\mathbb Q_p$. The converse also holds.
So if you want to know whether you can solve the equation $f(x) = 0$ in $\mathbb Q_p$ (where $f(x)$ is some irreducible polynomial in $\mathbb Q[x]$), then set $K = \mathbb Q[x]/f(x)$ and apply this criterion. This is easiest to do when $f(x)$ has integral coefficients, and remains separable when reduced mod $p$ (something that you can check by computing the discriminant and seeing whether or not it is divisible by $p$), because in this case the criterion is equivalent to asking that $f(x)$ have a root mod $p$.
Incidentally, there are many $f(x)$ that satisfy this criterion (because, among other things, the algebraic closure of $\mathbb Q$ in $\mathbb Q_p$ has infinite degree over $\mathbb Q$), but there are also many $f(x)$ that don't.
A monic polynomial $f$ with coefficients in $\mathbb{Z}$ has a root in $\mathbb{Z}_p$ if and only if it has a root in $\mathbb{Z}/p^n\mathbb{Z}$ for all $n$; this is because $\mathbb{Z}_p$ can be defined as the limit ("inverse limit") of the $\mathbb{Z}/p^n\mathbb{Z}$. Hensel's lemma tells you at what value of $n$ you can stop searching and gives an algorithm for determining whether such a root exists for fixed $f$ and $p$.
A necessary condition is that $f$ needs to have a root in $\mathbb{F}_p$, and if $p$ doesn't divide the discriminant of $f$ I think this condition is also sufficient. For fixed $p$, are you looking for a description of all such $f$? I don't know that I can be more specific than "the set of all polynomials with a root in $\mathbb{F}_p$." You just take all the polynomials in $\mathbb{F}_p[x]$ with roots in $\mathbb{F}_p$ and lift them to $\mathbb{Z}$. Do you want an algorithm to determine when $f$ has this property? Just evaluate it at every point. I'm not really sure what you're looking for here.