Category of categories as a foundation of mathematics

My personal opinion is that one should consider the 2-category of categories, rather than the 1-category of categories. I think the axioms one wants for such an "ET2CC" will be something like:

• Firstly, some exactness axioms amounting to its being a "2-pretopos" in the sense I described here: http://ncatlab.org/michaelshulman/show/2-categorical+logic . This gives you an "internal logic" like that of an ordinary (pre)topos.
• Secondly, the existence of certain exponentials (this is optional).
• Thirdly, the existence of a "classifying discrete opfibration" $el\to set$ in the sense introduced by Mark Weber ("Yoneda structures from 2-toposes") which serves as "the category of sets," and internally satisfies some suitable axioms.
• Finally, a "well-pointedness" axiom saying that the terminal object is a generator, as is the case one level down with in ETCS. This is what says you have a 2-category of categories, rather than (for instance) a 2-category of stacks.

Once you have all this, you can use finite 2-categorical limits and the "internal logic" to construct all the usual concrete categories out of the object "set". For instance, "set" has finite products internally, which means that the morphisms $set \to 1$ and $set \to set \times set$ have right adjoints in our 2-category Cat (i.e. "set" is a "cartesian object" in Cat). The composite $set \to set\times set \to set$ of the diagonal with the "binary products" morphism is the "functor" which, intuitively, takes a set $A$ to the set $A\times A$. Now the 2-categorical limit called an "inserter" applied to this composite and the identity of "set" can be considered "the category of sets $A$ equipped with a function $A\times A\to A$," i.e. the category of magmas.

Now we have a forgetful functor $magma \to set$, and also a functor $magma \to set$ which takes a magma to the triple product $A\times A \times A$, and there are two 2-cells relating these constructed from two different composites of the inserter 2-cell defining the category of magmas. The "equifier" (another 2-categorical limit) of these 2-cells it makes sense to call "the category of semigroups" (sets with an associative binary operation). Proceeding in this way we can construct the categories of monoids, groups, abelian groups, and eventually rings.

A more direct way to describe the category of rings with a universal property is as follows. Since $set$ is a cartesian object, each hom-category $Cat(X,set)$ has finite products, so we can define the category $ring(Cat(X,set))$ of rings internal to it. Then the category $ring$ is equipped with a forgetful functor $ring \to set$ which has the structure of a ring in $Cat(ring,set)$, and which is universal in the sense that we have a natural equivalence $ring(Cat(X,set)) \simeq Cat(X,ring)$. The above construction then just shows that such a representing object exists whenever Cat has suitable finitary structure.

One can hope for a similar elementary theory of the 3-category of 2-categories, and so on up the ladder, but it's not as clear to me yet what the appropriate exactness properties will be.

I've just run into your question now. I realized it's over a year since it was asked, but since the question is not closed and I happen to have at hand some references (as I'd been interested in such developments for a while), perhaps you should take a look at the following proposals.

Lawvere's original paper was indeed flawed, as reviewed by Prof. Isbell. Something later known as "category description theorem", a way to generating objects in the theory (i.e. categories) from some description of its intuitive structure, happened to be non provable in the original theory. A thorough review of the paper including ways to fix this and save a considerable portion can be found in: Blanc G., Preller A., Lawvere’s basic theory of the category of categories. Journal of Symbolic Logic 40 (1975, no. 1), 14–18 (doi:10.2307/2272263, JSTOR).

Subsequent proposals also try to address the point of the "category description theorem" by taking a variant of it as an axiom. This is done in Blanc G., Donnadieu M. R., Axiomatisation de la catégorie des catégories. Cahiers de topologie et géométrie différentielle 17 (1976, no. 2), 1–35 (Numdam).

Finally, McLarty proposed a clean, elegant and well presented axiomatization where proofs of independence and relative consistency are given. This can be found in McLarty, C., Axiomatizing a category of categories. Journal of Symbolic Logic 56 (1991, no. 4), 1243–1260 (doi:10.2307/2275472, Project Euclid, JSTOR). McLarty theory was meant to be taken in conjunction with certain axioms for specific categories or functors needed for specific purposes.

But I believe that all these three proposals are able to formulate the three bullets you mentioned in the first part of your question. For some other purposes one should take a closer look at them.

There's some discussion of this at the nlab:

http://ncatlab.org/nlab/show/foundations#categorial_foundations_of_category_theory_5

That page is one of the, er-hem, least encyclopaedic pages in the n-lab, but it still has quite a lot of discussion on this issue. You are, of course, welcome to join in the discussion there.

Is that any help?

(Edit: link corrected to relevant part of n-lab page as per Mike Shulman's comment)