What is the field with one element?
As other have mentioned, F_1 does not exist of a field. Tits conjectured the existence of a "field of characteristic one" F_1 for which one would have the equality G(F_1) = W, where G is any Chevalley group scheme and W its corresponding Weyl group.
Later on Manin suggested that the "absolute point" proposed in Deninger's program to prove the Riemann Hypothesis might be thought of as "Spec F_1", thus stating the problem of developing an algebraic geometry (and eventually a theory of motives) over it.
There are several non-equivalent approaches to F_1 geometry, but a common punchline might be "doing F_1 geometry is finding out the least possible amount of information about an object that still allows to speak about its geometrical properties". A "folkloric" introduction can be found in the paper by Cohn Projective geometry over F_1 and the Gaussian binomial coefficients.
It seems that all approaches so far contain a common intersection, consisting on toric varieties which are equivalent to schemes modeled after monoids. In the case of a toric variety, the "descent data" that gives you the F_1 geometry is the fan structure, that can be reinterpreted as a diagram of monoids (cf. some works by Kato). What else are F_1 varieties beyond toric is something that depends a lot on your approach, ranging from Kato-Deitmar (for which toric is all there is) to Durov and Haran's categorical constructions which contains very large families of examples. A somehow in-the-middle viewpoint is Soule's (and its refinement by Connes-Consani) which in the finite type case is not restricted only to toric varieties but to something slightly more general (varieties that can be chopped in pieces that are split tori). No approach is yet conclusive, so the definitions and families of examples are likely to change as the theory develops.
Last month Oliver Lorscheid and myself presented an state-of-the-art overview of most of the different approaches to F_1 geometry: Mapping F_1-land: An overview of geometries over the field with one element (sorry for the self promotion).
Update: at the bottom there's a wonderful and fresh reference.
There's no field with one element in the literal sense, but there are constructions that work over different fields $\mathbb F_q$ and which sometimes make sense when $q=1$. Examples would include representation theory of $GL_n(\mathbb F_q)$ which, if I'm correct, becomes the representation theory of $S_n$ under that limit.
In particular, indeed, vector spaces — the objects on which $GL_n(\mathbb F_q)$ act — should become sets, the objects on which $S_n$ acts.
Though I'm not an expert on $\mathbb F_1$, I've encountered the viewpoint you're referring to. It's not hard to see why people expect $\mathbb F_1$ to be the universal base: you usually expect that $\mathop{\text{Spec}} \mathbb F_n \to \mathop{\text{Spec}} \mathbb F_m$ exists iff $m$ divides $n$, so $\mathop{\text{Spec}} \mathbb F_1$ should be terminal. Of course, things are not literally that simple, e.g. what $\mathop{\text{Spec}} \mathbb Z \times \mathop{\text{Spec}} \mathbb Z$ over $\mathop{\text{Spec}} \mathbb F_1$ is?
Also, $\mathop{\text{Spec}} \mathbb Z$ should be thought of the object of dimension 3 (I think it has dualizing complex in degree 3, and primes are similar to 1-dimensional knots), so there's a pressing need for existence of some smaller scheme. That smaller scheme better "have real dimension 1" so we can claim that $\mathop{\text{Spec}} \mathbb Z$ is an algebraic curve over it. One then tries to compactify it.
When you get stuck geometrically, it helps to lose some information and go to zeta functions. There this approach leads to some direct conjectures. The problem with zeta functions is that everything is hard. Still, search for absolute zeta and you'll see some real things being discussed.
It's been called absolute motives in the works I read, I think by Manin, Kontsevich, and some IHES people. The search for kontsevich absolute motives brought me up the article that I found a while ago, math/0702206.
This is also somehow related to noncommutative geometry of Connes (endomotives), in fact I think $\mathbb F_1$ is one of his favorite ideas (also owes a lot to Soulé and other people of course) The two articles I found to be of special interest are 0806.2401 and 0809.2926.
Also, here's a blog post about $\mathbb F_1$ with link to the introductory paper. In fact there was a whole blog about $F_{un}$, which disappeared (articles, unfortunately, also disappeared from my RSS reader).
Here's a paper by Connes and Consani that seems to explain most the topics mentioned above:
Schemes over F1 and zeta functions, 0903.2024
Here's probably the simplest manifestation of the field-with-one-element phenomenon. Define a projective $n$-space of order $q$ to be a collection of points, lines, planes, etc. satisfying the usual incidence relations with the additional condition that every line has $q+1$ points on it, every plane has $q^2+q+1$ points on it, and so forth. For $q$ a prime power, all such spaces come from the usual definition of projective $n$-space $\Bbb P^n(\Bbb F_q)$ over a finite field.
But a projective $n$-space of order $1$ is precisely the Boolean algebra of subsets of a set with $n$ elements!
(This example is due to Henry Cohn, and it has the virtue that any theorems one wants to prove in this abstract setting don't depend on the value of $q$.)