What is $R$ in the formula for escape velocity?

$R$ is the distance between the centres of either object. By centres I mean their centres of mass (the point their gravitational forces "pull" from, so to speak).

$R$ is thus the sum of Earth's radius $R_e$, the object's radius $r_o$ (if spherical) and the height $h$ it is located above the ground:

$$R=R_e+h+r_o$$

For small objects not-too-far from the ground (such as thrown stones, fired rockets and orbiting satellites) you'll often see the radius of the object and the distance neglected. A rock's or a satellite's small radius is negligible compared to the ~6400 km of the Earth. Adding maybe 1km or even 10km or 100km in distance to this number makes no practical difference.

$$R=R_e+h+r_o\approx R_e$$

But when calculating escape velocity of e.g. Pluto's moon from Pluto, two objects that are comparable in size, you can't neglect either radius. And when calculating escape velocity of our own Moon from Earth, the ~400 000km distance must certainly be included as well (the sizes are almost negligible compared to this).

In order to really understand what this formula means, let's see how to derive that. Let $m$ be the mass of a body on the surface of Earth and let $R_\mathrm{Earth}$ be Earth's radius. Our aim is to calculate the initial velocity the body needs, to "escape" from Earth gravitational field. "Escaping", in this context, means to reach the infinity, where the gravitational field is known to be null. Plus, the escape velocity is defined as the minimum velocity needed, physically this means that our body reach the infinity with no more velocity $v_\infty = 0$.

Let's start from energy conservation principle:

$$E = \frac{1}{2}mv_0^2-\frac{M_\mathrm{Earth} m G}{R_\mathrm{Earth}}$$

We've just sad the potential at infity is zero, same for the velocity. Thus we have

$$\frac{1}{2}mv_0^2-\frac{M_\mathrm{Earth} m G}{R_\mathrm{Earth}}=0.$$

Solving this equation for $v_0$ we get:

$$v_0=\sqrt \frac{2M_\mathrm{Earth}G}{R_\mathrm{Earth}}.$$

If we do not consider air friction, there are no reasons why we should consider the existence of atmosphere.

Technically, "R" is the radius between the centre of the Earth (or whatever body you're using) and the rocket. That $11.186 \:\rm km/s$ is the escape velocity at the surface of the Earth, but a hundred kilometers above the Earth the escape velocity is $11.099\:\rm km/s$.

Assuming that you reach escape velocity then begin to coast, your velocity will be constantly decreasing because the pull of the Earth is still acting on your spacecraft. One million kilometers from Earth your speed may have decreased to $1\:\rm km/s$, but that's okay because at that distance the escape velocity is still $0.89\:\rm km/s$.

Hope that helps.