Neither a vector, nor a scalar

If you have two vectors $\mathbf{a}$ and $\mathbf{b}$, the inner product $\mathbf{a} \cdot \mathbf{b}$ is a scalar, the cross product $\mathbf{a} \times \mathbf{b}$ is a vector and the dyadic product $\mathbf{a} \otimes \mathbf{b}$ is a matrix. It is defined as

$$\mathbf{a}\otimes\mathbf{b} = \mathbf{a b}^\mathrm{T} = \begin{pmatrix} a_1 \\ a_2 \\ a_3 \end{pmatrix}\begin{pmatrix} b_1 & b_2 & b_3 \end{pmatrix} = \begin{pmatrix} a_1b_1 & a_1b_2 & a_1b_3 \\ a_2b_1 & a_2b_2 & a_2b_3 \\ a_3b_1 & a_3b_2 & a_3b_3 \end{pmatrix} $$

It occurs a lot in the formalism of quantum mechanics where it is written as $|a \rangle \langle b|$ (using the so-called bra-ket notation by Dirac).

With regard to direction: if you apply a matrix to a vector, the vector may get stretched / compressed along multiple axes. So in contrast to a vector, a matrix involves multiple directions.

For this answer, I it's worth listing the common (i.e. an exhaustive list of what I can think of right now) vector products and some idea of their physical meaning. These are:

1. The inner, scalar product that yields scalars and, more generally, other scalar -valued bilinear products of vectors;
2. The tensor product that yields a tensor of rank 2
3. The Exterior, Grassmann, Alternating or wedge product that yields a 2-form for vector arguments (a special kind of rank 2 tensor)
4. The Lie bracket between two vectors of a Lie algebra, which yields another vector in the same Lie algebra;
5. The Clifford product or "geometric" product (Clifford's name for this), which generalizes and unifies (1) and (3).

The cross product is actually a disguised version of (3) or of (4) that only works in three dimensional, Euclidean space.

We begin with a vector, or linear, space $\mathbb{V}$ over a field of scalars $K$, that comes kitted with addition, which makes the vector spaces an Abelian group. For most useful-for-physics discussions we want any product to be billinear, i.e. binary operation operator that is linear in both its arguments alternatively, that is distributive on left and right and respects scalar multiplication:

$$(\alpha\,a+\beta\,b) \ast (\gamma\,c+\delta\,d) = \alpha\,\gamma\,a\ast c+ \beta\,\gamma\,b\ast c + \alpha\,\delta\,a\ast d + \beta\,\delta\,b\ast d;\quad\forall \alpha,\,\beta,\,\gamma,\,\delta\in K\;\forall a,\,b,\,c,\,d\in \mathbb{V}\tag{1}$$

and this will be a fundamental assumption in what follows. A more abstract consideration, possibly not for a first reading, is that when considering all the different products with property (1) one can reduce the object under consideration to a linear function of one variable from the tensor product space $\mathbb{V}\otimes \mathbb{V}$ to the range of the binary operation. See the "Universal Property" section of the Wikipedia tensor product page. In this sense, the tensor product 2. above is the most general product possible.

1. Billinear Scalar Forms and The Dot Product

These are all entities that have the property (1) and map pairs of vectors to scalars. Their most general form is:

$$a\odot_M b = A^T\,M\,B\tag{2}$$

where $M$ is an $N\times N$ matrix where $N$ is the dimension of the vector space. The dot product is the special case where $M$ is symmetric and positive definite. But surely there are many more symmetric, positive definite matrices $M$ than simply the identity, which obviously gives the dot product from (2)? Actually, if $M$ is symmetric, (2) doesn’t give us a great deal more generalness than the abstract inner product, but there are some interesting quirks. These quirks are most readily seen by applying Sylvester's Law of Inertia whereby there always exists a change of basis (invertible co-ordinate transformation) that reduces the matrix $M$ in (2) to the form

$$M=\mathrm{diag}(1,\,1\,\cdots,\,-1,\,-1,\,\cdots 0,\,0\,\cdots)\tag{3}$$

where the number of 1s, -1s and 0s is independent of the co-ordinate transformation. So we only need to look at (3) without loss of generalness. If there are all 1s, then we have an inner product, and we don't get anything essentially different to the dot product already discussed: a change of basis will force $M=\mathrm{id}$. If there are noughts present, then the form is degenerate, which means that there is at least one nonzero vector whose product with any other vector yields nought. If there are 1s and -1s present, then the product is not degenerate, but the notion of orthogonal complement breaks down: for any vector, there are always nonzero vectors whose product with the former is 0 and there are lightlight nonzero vectors whose self-product is nought. This nontrivially signatured case is what we come across in Minkowski spacetime and special and general relativity. If the matrix $M$ is skew-symmetric, then the product defined by (3) is called a symplectic form if $M$ is also nonsingular. The only nondegenerate symplectic products arise in even dimensional spaces. All (nondegenerate) symplectic forms can be rewritten, through co-ordinate transformations as a form wherein $M$ takes its canonical value:

$$M = \left(\begin{array}{cc}0&-\mathrm{id}\\\mathrm{id}&0\end{array}\right)\tag{4}$$

where $\mathrm{id}$ is the $N\times N$ identity matrix, when the dimensionality of the vector space is $2\,N$.

Now lets specialize back to "the" inner product (they are all essentially the same, through the considerations above). The reason the dot product is useful is kind of subtle at the undergrad level when it is first introduced, because the concept is usually introduced in a way that makes its reason for being tautologous: the dot product gives us a simple algorithm to compute the unique decomposition of any vector into superposition of basis vectors when the latter are orthonormal with respect to that product. This assertion actually isn't trivial or self evident: it rests on the Gram-Schmidt Procedure: It follows from the axioms defining the vector space notion the axiom that a vector space is spanned by a basis and the number of basis vectors must be independent of choice. Moreover, if we introduce a scalar billinear inner product, i.e. one that is strictly positive whenever the inner product is made between a nonzero vector and itself and zero when the zero vector is multiplied with itself, then one can always choose the basis to be orthonormal by using the GS procedure beginning with any basis.

This is all swept under the rug when we first learn about it because usually the basis is postulated as self evident and it is always orthonormal, as when we learn about $xyz$ co-ordinates in high school, so these details are hidden - nondistracting, perhaps, but also nondiscoverable. If you begin with the abstract definition, then it is easy to show that if we represent vectors $a\,b\,c,\cdots$ as column matrices $A,\,B,\,C,\,\cdots$ containing their components with respect to an orthonormal basis, then:

$$a\cdot b = A^T\,B = \left(\begin{array}{ccc}a_1&a_2&\cdots\end{array}\right)\left(\begin{array}{c}a_1\\a_2\\\vdots\end{array}\right)=\sum\limits_i\,a_i\,b_i\tag{5}$$

and, from elementary trigonometry, it is "intuitively obvious" on applying this concept that $a\cdot b = \sqrt{a\cdot a}\,\sqrt{b\cdot b}\,\cos\theta = |a|\,|b|\,\cos\theta$, although rigorously, we use (5) to define the angle between two vectors because the Cauchy-Schwarz Inequality, which follows from the abstract properties of the inequality alone, shows that the range of $a\cdot b/(\sqrt{a\cdot a}\,\sqrt{b\cdot b})$ is precisely the interval $[-1,\,1]$, so there is no problem in defining this to be a cosine.

Experimentally it is found that the work done $F\cdot s$ by a force $F$ in displacing an object $s$ equals the change in that object's total energy; this can be derived from Newton's laws, thus the experimental observation indirectly further confirms the latter.

2. Tensors and Tensor Product

To keep this already cluttered answer contained, I will not discuss dual vectors (covectors) in connexion with tensors.

The section above on billinear forms is quite a deal to absorb, but tensors are only a small step beyond the above. A general tensor is simply a multilinear, scalar valued function of vectors. Thus we can think of the dot product - the mapping $\cdot: \mathbb{V}\times\mathbb{V}\to K$ itself considered as a whole rather than the particular outcome of the dot product for two particular vectors - as a rank two tensor. That is, a billinear, scalar valued function of two vectors, given by (2) when $M=\mathrm{id}$. When thought of as a tensor in this way, the dot product function is better known as the Kronecker delta.

The tensors of rank $r$ (i.e. multilinear functions of $r$ vectors) for an $N$-dimensional vector space are themselves a vector space, this time of dimension $N^r$. A multilinear function of $r$ vectors is wholly specified, by definition of linearity, by its value at the $N^r$ possible combinations of the $N$ basis vectors input into the function’s $r$ arguments. All other values follow simply by linear superposition. For a rank 2 tensor, we simply write these basis values into a matrix, and then (2) will work out the tensor’s value. So we can identify rank 2 tensors with matrices. A rank 3 tensor, i.e. trillinear function would require a version of (2) with a three dimensional array and so on.

Some tensors can be written as the products of vectors. The tensor product does this. The tensor product is also called the outer product when specialized to vector arguments. The tensor and outer products for vectors and rank 2 tensors are implemented by the Kronecker matrix product when we have component and matrix representations of the vectors / tensors in question. This is the product discussed and made explicit in Marc’s answer and Sal Elder’s answer. A general rank 2 tensor is a superposition of outer products of vectors.

3. Exterior, Wedge, Alternating or Grassmann Product

A particular superposition of outer products of pairs of vectors is one where the products of pairs themselves arise in skew-symmetric pairs. That is, it is a superposition of pairs of the form:

$$x \wedge y \stackrel{def}{=} x\otimes y -y\otimes x\tag{6}$$

and $x \wedge y$ is a billinear function of vectors whose matrix (as in (2)) is skew-symmetric. General skew-symmetric rank 2 tensors are made up of superpositions of these wedge products of vectors. The functions they define are called symplectic forms when the form is nondegenerate, which can only happen in even dimensional spaces.

Geometrically, $x\wedge y$ represents a directed area. Its magnitude (Euclidean norm of the $N\,(N-1)/2$ nonzero components) is the area enclosed by the parallelogram defined by the vectors $x$ and $y$. Threefold wedges represent directed volumes and multi wedges represent directed hypervolumes. In $N$ dimensions, there is only one nonzero component of the wedge of $N$ vectors, given by the determinant of the matrix with these vectors as its columns, and the signed volume of the $N$-dimensional parallelopiped defined by these vectors.

Some of these properties should sound familiar as properties of the cross product. Indeed, in 3 dimensions, a directed plane can be defined by its unit normal vector, and contrariwise. This doesn’t work in higher dimensions - a plane’s orthogonal complement is also a plane. An operation called the Hodge dual generalizes this valid-only-in-3-dimensions isomorphism between directed planes and unit normals to other dimensions. In three dimensions, it picks off the nonzero components of the wedge of the three dimensional vectors $x$ and $y$ to define the cross product:

$$\begin{array}{lcl}x\wedge y &=& x\otimes y - y \otimes x=\left(\begin{array}{c}x_1\\x_2\\x_3\end{array}\right)\left(\begin{array}{ccc}y_1&y_2&y_3\end{array}\right)-\left(\begin{array}{c}y_1\\y_2\\y_3\end{array}\right)\left(\begin{array}{ccc}x_1&x_2&x_3\end{array}\right) \\\\ &=& \left(\begin{array}{ccc}0&x_1 y_2-y_1 x_2 & -(x_3 y_1 - y_3 x_1) \\-(x_1 y_2-y_1 x_2)&0&x_2 y_3 - y_2 x_3\\x_3 y_1 - y_3 x_1&-(x_2 y_3 - y_2 x_3)& 0\end{array}\right)\\\\&\stackrel{\text{ Hodge dual}}{\rightarrow} &\left(\begin{array}{c}x_2 y_3 - y_2 x_3\\x_3 y_1 - y_3 x_1\\x_1 y_2-y_1 x_2\end{array}\right) \stackrel{def}{=} x\times y\end{array}\tag{7}$$

4. Lie Brackets on Lie Algebras

A Lie bracket $[]:\mathbb{V}\times \mathbb{V} \to \mathbb{V}$ on a vector space $\mathbb{V}$ is, by definition, any skew-symmetric, billinear, binary, vector-valued operation that fulfills the Jacobi identity:

$$[a,\,[b,\,c]] + [c,\,[a,\,b]]+[b,\,[c,\,a]]=0;\quad \forall a,\,b,\,c\in\mathbb{V}\tag{8}$$

A vector space with such a product is a Lie algebra. (8) is easily remembered by cycling the arguments of the double Lie brackets to get from one term to the next. Or, if you’re moved by risqué ditties, von Neumann’s mnemonic that "a spanks b whilst c watches, c spanks a whilst b watches, and then b spanks c whilst a watches".

For any finite dimensional Lie algebra with a vector space whose scalars are from a field of characteristic nought, one can find a faithful representation of the algebraic structure as vector space of square matrices of the same dimension with the commutator product $[a,\,b]=a\,b - b\,a$. This assertion is Ado’s Theorem; not every vector space of matrices forms a Lie algebra in this way, because the space has to be closed under the commutator bracket. It is this last observation that makes Ado’s theorem a difficult thing to prove. (I believe there’s a generalization that removes the zero characteristic restriction, but I’ve never looked it up).

The most important fact in physics and mechanics is that Lie algebras represent the differential action of a Lie group, of which the group of rotations is an example. The Lie algebra of the Lie group is the group’s tangent space at the identity. A Lie group acts on its own Lie algebra through the Adjoint representation; for a matrix Lie group, the action of group member $\gamma$ on the Lie algebra member $Y$ can be written $Y\mapsto \gamma\,Y \,\gamma^{-1}$ (where juxtaposition the expression on the right is simply the matrix product). If the Lie group member is of the form $\exp(s\,X)$, where $X$ is also a member of the group’s Lie algebra (and the identity connected component of a Lie group can always be written as a finite product of such entities), then the action is of the form $Y\mapsto \exp(s\,X)\,Y \,\exp(-s\,X)$. The derivative of this expression, i.e. the "infinitessimal" action of the Lie group on its own algebra, defines the Lie bracket, i.e.:

$$[X,\,Y]\stackrel{def}{=} \left.\frac{\mathrm{d}}{\mathrm{d} s} \exp(s\,X)\,Y \,\exp(-s\,X)\right|_{s=0}\tag{9}$$

In a matrix Lie group, it’s a simple matter to check that this definition defines the commutator bracket:

$$[X,\,Y] = X\,Y - Y\,Y\tag{10}$$

There is a simple way to give meaning to the mapping $Y\mapsto \gamma\,Y \,\gamma^{-1}$ in a general Lie group where there is no matrix product defined, so the above ideas are general.

Now, for example, if we are talking about the three dimensional rotation group, which is $SO(3)$, it is easy to show that the Lie algebra is the algebra of $3\times 3$ skew-symmetric matrices. To do this, write a group member near the identity as $\mathrm{id} + \delta\, X + O(\delta^2)$, where $X$ is any Lie algebra member. The rotations are the proper isometries with respect to the dot product norm in Euclidean space, i.e. the conserve the inner product. This conservation under the action of the rotation matrix $\mathrm{id} + \delta\, X + O(\delta^2)$ on column vectors is written as:

$$((\mathrm{id}+ \delta\, X + O(\delta^2))\,x)^T \,(\mathrm{id}+ \delta\, X + O(\delta^2)) y = x^T\,y;\; \forall x,\,y\in\mathbb{R}^N \Leftrightarrow X + X^T = 0\tag{11}$$

So any small angle rotation of a vector is well approximated by the action of a skew-symmetric matrix on that vector. One property that the rotation group enjoys (as do all Lie groups without continuous centers) is the adjoint action of the group on its own Lie algebra defines exactly the same group. Furthermore, in three dimensions, the rotation group has three dimensions, exactly like the Euclidean space acted on by the matrix rotation group $SO(3)$. So in three dimensions only, the rotation group’s group and action on the Lie algebra is exactly the same as the group where members are thought of as rotation matrices acting on column vectors. So consider the three dimensional Hodge dual operation at the very right of equation (7). By the equality of both $SO(3)$ group and its adjoint action image on the one hand and of the adjoint action and matrix-on-column vector action on the other, we have the following procedure for approximating a rotation of the column vector $y$ by a small angle $\theta$ about the axis defined by the unit vector $\hat{X}$. Alternatively, let’s absorb $\theta$ into $\hat{X}$ to get a nonunit magnitude vector $X$, whose magnitude is $\theta \approx \sin\theta$ where $\theta$ is small:

1. Convert column vectors $X$ and $Y$ to skew-symmetric matrices $X^s,\, Y^s$ using the inverse Hodge dual operation;
2. Compute the Lie bracket $[X^s,\,Y^s] = X^s Y^s - Y^s X^s$
3. Convert the $Z^s=[X^s,\,Y^s]$ back to a column vector by extracting its nonzero components and arranging them into a column vector $Z$ through the Hodge dual
4. The rotated image is $Y+Z$

You can check that steps 1, 2 and 3 define the cross product $Z=X\times Y$. You can also check that the cross product fulfills the Jacobi identity.

Alternatively, if an object is spinning, the above procedure shows that the change in $Y$ over time $\delta t$ is, to first order, $\delta t\, \omega\times Y$, where $\omega$ is the vector along the axis of rotation and whose magnitude is the angular speed. Whence the formula $V = \omega\times Y$ for the velocity of the head of vector $Y$.

Note that the special conditions on dimension discussed above are equivalent to the assertion that the rotation axis notion only works in dimension 3 and lower. In higher dimensions, rotations must be defined by their planes.

Higher dimensional rotation groups also define Lie bracket products between the planes (skew-symmetric two forms) corresponding to their Lie algebra members, and this product works perfectly well and logically. But since the dimension of the $N$ dimensional rotational Lie group’s algebra is $N\,(N-1)/2$ is much greater than the dimension $N$ of the rotated column vectors, the Lie algebra members do not correspond to vectors in the underlying space anymore and there is no simple relationship between the group actions.

5. The Clifford Product

This product has already been discussed in Timo’s Answer.

The wedge product of section 3 defines an exterior algebra, which can be thought of as a 
"degenerate" Clifford algebra where the product of something with itself vanishes (as is the case for all alternating forms).

A Clifford algebra is an algebra generated by an abstract billinear product with a vector space as a subspace of generators. Aside from billinearity, the algebra is free aside from the condition that a generator vector multiplied by itself is quadratic form, i.e. an entity defined by (2) where we can, without loss of generalness, take $M$ to be symmetric (since we only ever invoke (2) for this lone, unfree condition, i.e. we only invoke (2) when $a$ and $b$ are equal). This concept generalizes the exterior algebra of section 3, which can be thought of as a Clifford algebra where $M=0$.

Geometric algebras are Clifford algebras where the quadratic form is nondegenerate, i.e. $M$ is nonsingular.

Say I write a column vector followed by a row vector. For example, $$\begin{bmatrix}1\\ 0\\ 0\end{bmatrix}\begin{bmatrix}1 & 0 & 0\end{bmatrix}.$$ That's a matrix! Specifically, it would be $$\begin{bmatrix}1&0&0\\ 0&0&0 \\ 0&0&0\end{bmatrix}.$$ In an intro physics class, you probably don't have to worry about this sort of thing. You'll see it again in quantum mechanics.