What exactly are we doing when we set $c=1$?

All we're doing is using a set of units where certain quantities happen to take convenient numerical values. For example, in the SI system we might measure lengths in meters and time intervals in seconds. In those units we have $c = 3 \times 10^8\ \text{m}/\text{s}$. But you could just as well measure all your distances in terms of some new unit, let's call it a "finglonger", that is equal to $2.5 \times 10^6\ \text{m}$, and time intervals in a new unit, we'll call it the "zoidberg", that is equal to $8.33 \times 10^{-3}\ \text{s}$. Then the speed of light in terms of your new units is $$ c = 3 \times 10^{8}\ \text{m}/\text{s} = 1\ \frac{\text{finglonger}}{\text{zoidberg}} .$$ The units are still there – they haven't been "deleted" – but we usually just make a mental note of the fact and don't bother writing them.

If you are getting used to 'natural' units I think its best to think of it like this: we are basically defining a new time variable $t' \equiv c t$ to work in. $t '$ has units of distance. We can always go back to the old time variable, and old unit system using $t = \frac{t'}{c}$.

We do this to keep things simple as possible. For example the line element:

$d s^2 = c^2 dt^2 - dx^2 - dy^2 - dz^2 = dt'^2- dx^2 - dy^2 - dz^2 $

and the relativistic dispersion relation:

$E = \sqrt{p^2 c^2+m^2 c^4} = \sqrt{p'^2 +m'^2}$

are much simpler in these units. This may not seem like a great step forward, but when dealing with complicated equations, anything that simplifies is of great use.

Some theoretical physicist like to do that just to avoid constants while calculating, they choose a system of units in which $\hbar=c_0=1$ (and some more of them), so the get rid of a lot of stuff. The point is just to do that, they get rid of constants by making them equal to 1, at the end they will have to change again to a system more usable, mks, IS, or someone else.