HLSL mul() variables clarification

Yes, if x is a vector then x is treated as a row major vector and y is treated as a row major matrix; vice versa for column major so for a row-major matrix system:

float4 transformed = mul(position, world);

and for column-major:

float4 transformed = mul(world, position);

Because of the way that matrix multiplication works, if the matrix is column-major then you must post multiply by a column vector to get the correct result. If the matrix is row-major you must pre multiply by a row vector.

So really, hlsl doesn't care whether your matrix is row or column major, it is up to you to apply the vector multiplication in the correct order to get the correct result.

No. The terms "row-major" and "column-major" refer purely to the order of storage of the matrix components in memory. They have nothing to do with the order of multiplication of matrices and vectors. In fact, the D3D9 HLSL mul call interprets matrix arguments as column-major in all cases. The ID3DXBaseEffect::SetMatrix() call interprets its matrix argument as row-major, and transposes behind the scenes to mul's expected column-major order.

If you have a matrix that abstractly looks like this:

[ a b c d ]
[ e f g h ]
[ i j k l ]
[ m n o p ]

then when stored in row-major order, its memory looks like this:

a b c d e f g h i j k l m n o p

i.e. the elements of a row are all contiguous in memory. If stored in column-major order, its memory would look like this:

a e i m b f j n c g k o d h l p

with the elements of a column all contiguous. However, this has precisely zero effect on which element is which. Element b is still in the first row and second column, either way. The labeling of the elements has not changed, only the way they're mapped to memory.

If you declare an array like float matrix[rows][cols] in C, then you are using row-major storage. However, some other languages, like FORTRAN, use column-major storage for their multidimensional arrays by default; and OpenGL also uses column-major storage.

Now, entirely separately, there is another choice of convention, which is whether to use row-vector or column-vector math. This has nothing at all to do with the memory layout of matrices, but it affects how you build your matrices, and the order of multiplication. If you use row vectors, you'll do vector-matrix multiplication:

            [ a b c d ]
[x y z w] * [ e f g h ] = [x*a + y*e + z*i + w*m, ... ]
            [ i j k l ]
            [ m n o p ]

and if you use column vectors, then you'll do matrix-vector multiplication:

[ a b c d ]   [ x ]
[ e f g h ] * [ y ] = [x*a + y*b + z*c + w*d, ... ]
[ i j k l ]   [ z ]
[ m n o p ]   [ w ]

This is because in row-vector math, a vector is really a 1×n matrix (a single row), and in column-vector math it's an n×1 matrix (a single column), and the rule about what sizes of matrices are allowed to be multiplied together determines the order. (You can't multiply a 4×4 matrix by a 1×4 matrix, but you can multiply a 4×4 matrix with a 4×1 one.)

Note that the matrix didn't change between the two equations above; only the interpretation of the vector changed.

So, to get back to your original question:

When you pass a vector to HLSL's mul, it automatically interprets it "correctly" according to which argument it is. If the vector is on the left, it's a row vector, and if it's on the right, it's a column vector.

However, the matrix gets interpreted the same way always. A matrix is a matrix, regardless of whether it's being multiplied with a row vector on the left or a column vector on the right. You can freely decide whether to use row-vector or column-vector math in your code, as long as you're consistent about it. HLSL is agnostic on this point, although the D3DX math library uses row vectors.

And it turns out that for some reason, in D3D9 HLSL, mul always expects matrices to be stored in column-major order. However, the D3DX math library stores matrices in row-major order, and as the documentation says, ID3DXBaseEffect::SetMatrix() expects its input in row-major order. It does a transpose behind the scenes to prepare the matrix for use with mul.

BTW, D3D11 HLSL defaults to column-major order, but allows you to use a compiler directive to tell it to use row-major order instead. It is still agnostic as to row-vector versus column-vector math. And OpenGL GLSL also uses column-major order, but does not (as far as I know) provide a way to change it.

Further reading on these issues:

• A word on Matrices by Catalin Zima
• Row major vs. column major, row vectors vs. column vectors by Fabian Giesen