Linear algebra in terms of abstract nonsense?
To my mind there are two classes of interesting categorical facts here, loosely speaking "additive" facts and "multiplicative" facts. Some additive facts:
Finite-dimensional vector spaces over $k$ has biproducts, and every object is a finite biproduct of copies of a single object, namely $k$. The categories with this property are precisely the categories of finite rank free modules over a semiring $R$ (the endomorphisms of the single object). These biproduct decompositions encapsulate both the idea that vector spaces have bases and that a choice of bases can be used to write linear maps as matrices.
The single object $k$ above is simple, and so every object is a finite biproduct of simple objects. The categories with this property, in addition to 1, are precisely the categories of finite rank free modules over a division semiring $R$.
Note that additive facts can't see anything about fields being commutative. The multiplicative facts can:
Finite-dimensional vector spaces over $k$ is symmetric monoidal with respect to tensor product, and is also closed monoidal and has duals (sometimes called compact closed). This observation encapsulates the yoga surrounding tensors of various types (e.g. endomorphisms $V \to V$ correspond to elements of $V \otimes V^{\ast}$), as well as the existence and basic properties (e.g. cyclic symmetry) of the trace.
The single object $k$ above is the tensor unit, and so every object is a finite biproduct of copies of the unit. Also, the monoidal structure is additive in each variable. I believe, but haven't carefully checked, that the (symmetric monoidal) categories with this property, in addition to 1 and 2 above, are precisely the (symmetric monoidal) categories of finite rank free modules over a commutative division semiring ("semifield") $R$. This encapsulates the concrete description of tensor products as a functor in terms of Kronecker products.
There's surprisingly little to say as far as fields being rings as opposed to semirings, though. This mostly becomes relevant when we reduce computing (co)equalizers to computing (co)kernels by subtracting morphisms, as in any abelian category.
Edit: I haven't mentioned the determinant yet. This mixes additive and multiplicative: abstractly the point is that we have a natural graded isomorphism
$$\wedge^{\bullet}(V \oplus W) \cong \wedge^{\bullet}(V) \otimes \wedge^{\bullet}(W)$$
where $\wedge^{\bullet}$ denotes the exterior algebra (which we need the symmetric monoidal structure, together with the existence of certain colimits, to describe). It follows that if $L_i$ are objects which have the property that $\wedge^k(L_i) = 0$ for $k \ge 2$ then
$$\wedge^n(L_1 \oplus \dots \oplus L_n) = L_1 \otimes \dots \otimes L_n.$$
Combined with the facts above this gives the existence and basic properties of the determinant, more or less. Note that the exterior algebra can be defined by a universal property, but to verify that the standard construction has this universal property we need the symmetric monoidal structure to distribute over finite colimits. Fortunately this is implied by compact closure.
Qiaochu's answer captures very nicely the ordinary categorical aspects of the category $\mathbf{Vect}_k$ of finite-dimensional vector spaces over $k$; I'll just add some homotopy-theoretic comments that I find interesting.
I am fond of the fact that we can think of vector bundles in the following way: for a topological field $k$, the category $\mathbf{Vect}_k$ is closed monoidal, and thus also enriched over $\mathbf{Top}$, the category of topological spaces. We can thus consider $\mathbf{Vect}_k$ as an $(\infty,1)$-category, and vector bundles over a space $X$ correspond to functors $X\to\mathbf{Vect}_k$; this is exactly analogous to how Grothendieck opfibrations $E\to B$ correspond to pseudofunctors $B\to \mathbf{Cat}$ (and coCartesian fibrations of quasicategories generalize both these examples).
To put things in perspective, this is really what's going on when we consider the classifying spaces $BO(n)$ or $BU(n)$ (recall that homotopy classes of maps $X\to BO(n)$ classify $n$-dimensional real vector bundles over $X$ and similarly with $BU(n)$ for complex vector bundles). Recall that spaces are $\infty$-groupoids. If $X$ is connected, then it's an $\infty$-groupoid with only one object (up to equivalence), so a functor $X\to\mathbf{Vect}_k$ is defined on objects just by choosing a single object $V\in\mathbf{Vect}_k$, and we can consider functors $X\to\mathcal{V}$, where $\mathcal{V}$ is the full sub-$\mathbf{Top}$-category of $\mathbf{Vect}_k$ with $V$ as its only object. And because $X$ is an $\infty$-groupoid, any such functor factors through $\mathrm{Core}(\mathcal{V})$, where $\mathrm{Core}(\mathcal{V})$ is the subcategory of $\mathcal{V}$ of automorphisms of $V$. When we go back to thinking of the $\infty$-groupoid $\mathrm{Core}(\mathcal{V})$ as a space, we wind up with $BO(n)$ or $BU(n)$ (for real and complex vector spaces of dimension $n$ respectively).
My favorite example: Construct the functor $\Lambda^n : \mathsf{Vect}_k \to \mathsf{Vect}_k$. Show that $\Lambda^n(V)$ is of dimension $\binom{d}{n}$ when $d$ is the dimension of $V$ (this was roughly explained by Qiaochu Yuan). In particular, $\Lambda^d(V)$ is $1$-dimensional. Conclude that $\Lambda^d$ induces a multiplicative map $\mathrm{End}(V) \to \mathrm{End}(\Lambda^d(V)) = k$ and call it the determinant ... In this approach, the multiplicativity of the determinant follows from the functoriality of $\Lambda^n$, and the latter is trivial. Also, the proof that $\Lambda^d(V)$ is $1$-dimensional can be used to derive Leibniz's formula for the determinant of a matrix.