Determinant of a non-square matrix

Such a function cannot exist. Let $A = \begin{pmatrix} 1 & 0 \\ 0 & 1 \\ 0 & 0\end{pmatrix}$ and $B = \begin{pmatrix} 1 & 0 & 0 \\ 0 & 1 & 0 \end{pmatrix}$. Then, since both $AB$ and $BA$ are square, if there existed a function $D$ with the properties 1-3 stated there would hold \begin{align} \begin{split} 1 &= \det \begin{pmatrix} 1 & 0 \\ 0 & 1 \end{pmatrix} = \det(BA) = D(BA) = D(B)D(A) \\ &= D(A)D(B) = D(AB) = \det(AB) = \det \begin{pmatrix} 1 & 0 & 0 \\ 0 & 1 & 0 \\ 0 & 0 & 0 \end{pmatrix} = 0. \end{split} \end{align}

This extension of determinants has all 4 properties if A is a square matrix, and retains some attributes of determinants otherwise.

$$|A|^2=|A^{T}A|$$

If you're willing to break the rules a little bit, this has a valid and useful geometric interpretation. If you have a space defined in a dimension higher than its own, this can still return the area it defines.

Since the square of the determinant of a matrix can be found with the above formula, and because this multiplication is defined for nonsquare matrices, we can extend determinants to nonsquare matrices. For example, take the 3 wide matrix A defined with column vectors, x y and z, where each have n components:

$$A=\begin{pmatrix}x|y|z\end{pmatrix}$$

You can dot each of the vectors with each other by right multiplying A by its transpose:

$$A^{T}A=\begin{pmatrix}x\\y\\z\end{pmatrix}\begin{pmatrix}x&y&z\end{pmatrix}=\begin{pmatrix} x\cdot x & x\cdot y & x\cdot z\\ x\cdot y & y\cdot y & y\cdot z\\ x\cdot z & y\cdot z & z\cdot z \end{pmatrix}$$

Taking the determinant of this, you get the square of A's determinant: $$2 (x\cdot y) (x\cdot z) (y\cdot z)+(x\cdot x) (y\cdot y) (z\cdot z)-(x\cdot z)^2 (y\cdot y) - (x\cdot x )(y\cdot z)^2 - (x\cdot y)^2 (z\cdot z)$$

In this 3 vector example, the equation above returns the value of the volume defined by vectors x y and z.

You may take the positive square root of this to be the absolute value of the determinant. It's always positive because it doesn't make sense to define positive and negative areas for spaces defined in dimensions higher than the space itself. Depending on the perspective, a positive area can become a negative area if looked at from behind.